注意用root用户执行该命令。
-p 可以限制罗列的进程，比如 lsof -p 23543
-u 可以限制罗列的用户，比如 lsof -u leo，加^可对条件取反，比如 lsof -u ^leo
-i 可以列举网络连接信息，-n可以禁止地址解析，比如 lsof -i -n
-i4 和 -i6 可以限制列举的地址类型
-a 可以把条件组合起来，比如 lsof -n -i -a -u leo
参考 http://www.serverwatch.com/tutorials/article.php/3914236/Using-lsof-to-Find-Open-Files.htm
另外参考 fuser
2010年11月25日星期四
通过mac/ip地址的绑定控制访问公网
根据iptables的过滤规则，网关机器转发包的三个链(chain)是
PREROUTING -> FORWARD -> POSTROUTING
iptables的mac模块可以工作在 PREROUTING/FORWARD/INPUT 三个链，因此可考虑下面的配置规则
假设内部员工的网段是 10.0.0.0/255.255.255.0
临时上网人员的网段是 10.0.1.0/255.255.255.0
# 缺省不转发所有包
iptables -P FORWARD DROP
# 转发临时上网人员产生的包
iptables -A FORWARD -d 10.0.1.0/24 -j ACCEPT
iptables -A FORWARD -s 10.0.1.0/24 -j ACCEPT
# 员工只允许ip/mac匹配的数据包通过
iptables -A FORWARD -d 10.0.0.0/24 -j ACCEPT
iptables -A FORWARD -s 10.0.0.1 -m mac --mac-source 08:00:27:E8:02:52 -j ACCEPT
iptables -A FORWARD -s 10.0.0.2 -m mac --mac-source ... -j ACCEPT
iptables -A FORWARD -s 10.0.0.3 -m mac --mac-source ... -j ACCEPT
...
配置的时候注意考虑数据包的入方向和出方向。
PREROUTING -> FORWARD -> POSTROUTING
iptables的mac模块可以工作在 PREROUTING/FORWARD/INPUT 三个链，因此可考虑下面的配置规则
假设内部员工的网段是 10.0.0.0/255.255.255.0
临时上网人员的网段是 10.0.1.0/255.255.255.0
# 缺省不转发所有包
iptables -P FORWARD DROP
# 转发临时上网人员产生的包
iptables -A FORWARD -d 10.0.1.0/24 -j ACCEPT
iptables -A FORWARD -s 10.0.1.0/24 -j ACCEPT
# 员工只允许ip/mac匹配的数据包通过
iptables -A FORWARD -d 10.0.0.0/24 -j ACCEPT
iptables -A FORWARD -s 10.0.0.1 -m mac --mac-source 08:00:27:E8:02:52 -j ACCEPT
iptables -A FORWARD -s 10.0.0.2 -m mac --mac-source ... -j ACCEPT
iptables -A FORWARD -s 10.0.0.3 -m mac --mac-source ... -j ACCEPT
...
配置的时候注意考虑数据包的入方向和出方向。
2010年11月20日星期六
使用Oracle分析函数
虽然分析函数可以用原始SQL，结合join和子查询完成，但分析函数运行更快，且省掉了不必要的开发。
Oracle 11g支持的分析函数有（参考书籍 Expert Oracle Database 11g Administration）：
排名函数(Ranking functions)： 根据某个规则对数据集中的数据项排名，常见函数有RANK, DENSE_RANK, CUME_DIST, PERCENT_RANK, NTILE；
移动时间窗合计(Moving-window aggregates)： 可分析累计值和移动平均数；
时间段比较（Period-over-period comparisons）： 可对两个时间段进行比较，比如2010年Q1的利润比2009年Q1增长了百分之几？
比例报告（Ratio-to-report comparisons）：可用于比例对比，比如8月份注册人数占全年注册人数的百分之几？
统计函数（Statistical functions）： 相关性分析和回归函数，可从数据中挖掘因果关系；
反向百分比（Inverse percentiles）： 可根据一个百分比值找到对应的数据，比如找中等收入家庭的户主名称，中等收入值即百分比，从该值判断出对应的家庭户主名称；
推测排名和分布（Hypothetical ranks and distributions）： 可刻划新数据在既有数据中的排名和分布；
直方图（Histograms）： 根据表中的行值返回适当的直方图数据值；
第一/最后聚合（Fist/last aggregates）： 适用于GROUP BY语句排序数据
这里有一篇分析函数入门教程，原文参考 http://www.orafaq.com/node/55
Analytic functions by Example
This article provides a clear, thorough concept of analytic functions and its various options by a series of simple yet concept building examples. The article is intended for SQL coders, who for might be not be using analytic functions due to unfamiliarity with its cryptic syntax or uncertainty about its logic of operation. Often I see that people tend to reinvent the feature provided by analytic functions by native join and sub-query SQL. This article assumes familiarity with basic Oracle SQL, sub-query, join and group function from the reader. Based on that familiarity, it builds the concept of analytic functions through a series of examples.
It is true that whatever an analytic function does can be done by native SQL, with join and sub-queries. But the same routine done by analytic function is always faster, or at least as fast, when compared to native SQL. Moreover, I am not considering here the amount of time that is spent in coding the native SQLs, testing, debugging and tuning them.
The general syntax of analytic function is:
Function(arg1,..., argn) OVER ( [PARTITION BY <...>] [ORDER BY <....>] [] )
is like "ROW " or "RANK "
All the keywords will be dealt in details as we walk through the examples. The script for creating the schema (SCOTT) on which the example queries of this article are run can be obtained in ORACLE_HOME/sqlplus/demo/demobld.sql of any standard Oracle installation.
How are analytic functions different from group or aggregate functions?
SELECT deptno,
COUNT(*) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30)
GROUP BY deptno;
DEPTNO DEPT_COUNT
---------------------- ----------------------
20 5
30 6
2 rows selected
Query-1
Consider the Query-1 and its result. Query-1 returns departments and their employee count. Most importantly it groups the records into departments in accordance with the GROUP BY clause. As such any non-"group by" column is not allowed in the select clause.
SELECT empno, deptno,
COUNT(*) OVER (PARTITION BY
deptno) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30);
EMPNO DEPTNO DEPT_COUNT
---------- ---------- ----------
7369 20 5
7566 20 5
7788 20 5
7902 20 5
7876 20 5
7499 30 6
7900 30 6
7844 30 6
7698 30 6
7654 30 6
7521 30 6
11 rows selected.
Query-2
Now consider the analytic function query (Query-2) and its result. Note the repeating values of DEPT_COUNT column.
This brings out the main difference between aggregate and analytic functions. Though analytic functions give aggregate result they do not group the result set. They return the group value multiple times with each record. As such any other non-"group by" column or expression can be present in the select clause, for example, the column EMPNO in Query-2.
Analytic functions are computed after all joins, WHERE clause, GROUP BY and HAVING are computed on the query. The main ORDER BY clause of the query operates after the analytic functions. So analytic functions can only appear in the select list and in the main ORDER BY clause of the query.
In absence of any PARTITION or inside the OVER( ) portion, the function acts on entire record set returned by the where clause. Note the results of Query-3 and compare it with the result of aggregate function query Query-4.
SELECT empno, deptno,
COUNT(*) OVER ( ) CNT
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, 1;
EMPNO DEPTNO CNT
---------- ---------- ----------
7782 10 8
7839 10 8
7934 10 8
7369 20 8
7566 20 8
7788 20 8
7876 20 8
7902 20 8
Query-3
SELECT COUNT(*) FROM emp
WHERE deptno IN (10, 20);
COUNT(*)
----------
8
Query-4
How to break the result set in groups or partitions?
It might be obvious from the previous example that the clause PARTITION BY is used to break the result set into groups. PARTITION BY can take any non-analytic SQL expression.
Some functions support the inside the partition to further limit the records they act on. In the absence of any analytic functions are computed on all the records of the partition clause.
The functions SUM, COUNT, AVG, MIN, MAX are the common analytic functions the result of which does not depend on the order of the records.
Functions like LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE depends on order of records. In the next example we will see how to specify that.
How to specify the order of the records in the partition?
The answer is simple, by the "ORDER BY" clause inside the OVER( ) clause. This is different from the ORDER BY clause of the main query which comes after WHERE. In this section we go ahead and introduce each of the very useful functions LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE and show how each depend on the order of the record.
The general syntax of specifying the ORDER BY clause in analytic function is:
ORDER BY [ASC or DESC] NULLS [FIRST or LAST]
The syntax is self-explanatory.
ROW_NUMBER, RANK and DENSE_RANK
All the above three functions assign integer values to the rows depending on their order. That is the reason of clubbing them together.
ROW_NUMBER( ) gives a running serial number to a partition of records. It is very useful in reporting, especially in places where different partitions have their own serial numbers. In Query-5, the function ROW_NUMBER( ) is used to give separate sets of running serial to employees of departments 10 and 20 based on their HIREDATE.
SELECT empno, deptno, hiredate,
ROW_NUMBER( ) OVER (PARTITION BY
deptno ORDER BY hiredate
NULLS LAST) SRLNO
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, SRLNO;
EMPNO DEPTNO HIREDATE SRLNO
------ ------- --------- ----------
7782 10 09-JUN-81 1
7839 10 17-NOV-81 2
7934 10 23-JAN-82 3
7369 20 17-DEC-80 1
7566 20 02-APR-81 2
7902 20 03-DEC-81 3
7788 20 09-DEC-82 4
7876 20 12-JAN-83 5
8 rows selected.
Query-5 (ROW_NUMBER example)
RANK and DENSE_RANK both provide rank to the records based on some column value or expression. In case of a tie of 2 records at position N, RANK declares 2 positions N and skips position N+1 and gives position N+2 to the next record. While DENSE_RANK declares 2 positions N but does not skip position N+1.
Query-6 shows the usage of both RANK and DENSE_RANK. For DEPTNO 20 there are two contenders for the first position (EMPNO 7788 and 7902). Both RANK and DENSE_RANK declares them as joint toppers. RANK skips the next value that is 2 and next employee EMPNO 7566 is given the position 3. For DENSE_RANK there are no such gaps.
SELECT empno, deptno, sal,
RANK() OVER (PARTITION BY deptno
ORDER BY sal DESC NULLS LAST) RANK,
DENSE_RANK() OVER (PARTITION BY
deptno ORDER BY sal DESC NULLS
LAST) DENSE_RANK
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, RANK;
EMPNO DEPTNO SAL RANK DENSE_RANK
------ ------- ----- ----- ----------
7839 10 5000 1 1
7782 10 2450 2 2
7934 10 1300 3 3
7788 20 3000 1 1
7902 20 3000 1 1
7566 20 2975 3 2
7876 20 1100 4 3
7369 20 800 5 4
8 rows selected.
Query-6 (RANK and DENSE_RANK example)
LEAD and LAG
LEAD has the ability to compute an expression on the next rows (rows which are going to come after the current row) and return the value to the current row. The general syntax of LEAD is shown below:
LEAD (, , ) OVER ()
is the expression to compute from the leading row.
is the index of the leading row relative to the current row.
is a positive integer with default 1.
is the value to return if the points to a row outside the partition range.
The syntax of LAG is similar except that the offset for LAG goes into the previous rows.
Query-7 and its result show simple usage of LAG and LEAD function.
SELECT deptno, empno, sal,
LEAD(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) NEXT_LOWER_SAL,
LAG(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) PREV_HIGHER_SAL
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, sal DESC;
DEPTNO EMPNO SAL NEXT_LOWER_SAL PREV_HIGHER_SAL
------- ------ ----- -------------- ---------------
10 7839 5000 2450 0
10 7782 2450 1300 5000
10 7934 1300 0 2450
20 7788 3000 3000 0
20 7902 3000 2975 3000
20 7566 2975 1100 3000
20 7876 1100 800 2975
20 7369 800 0 1100
8 rows selected.
Query-7 (LEAD and LAG)
FIRST VALUE and LAST VALUE function
The general syntax is:
FIRST_VALUE() OVER ()
The FIRST_VALUE analytic function picks the first record from the partition after doing the ORDER BY. The is computed on the columns of this first record and results are returned. The LAST_VALUE function is used in similar context except that it acts on the last record of the partition.
-- How many days after the first hire of each department were the next
-- employees hired?
SELECT empno, deptno, hiredate ? FIRST_VALUE(hiredate)
OVER (PARTITION BY deptno ORDER BY hiredate) DAY_GAP
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, DAY_GAP;
EMPNO DEPTNO DAY_GAP
---------- ---------- ----------
7369 20 0
7566 20 106
7902 20 351
7788 20 722
7876 20 756
7499 30 0
7521 30 2
7698 30 70
7844 30 200
7654 30 220
7900 30 286
11 rows selected.
Query-8 (FIRST_VALUE)
注： hiredate ? FIRST_VALUE 应为 hiredate - FIRST_VALUE
FIRST and LAST function
The FIRST function (or more properly KEEP FIRST function) is used in a very special situation. Suppose we rank a group of record and found several records in the first rank. Now we want to apply an aggregate function on the records of the first rank. KEEP FIRST enables that.
The general syntax is:
Function( ) KEEP (DENSE_RANK FIRST ORDER BY) OVER ()
Please note that FIRST and LAST are the only functions that deviate from the general syntax of analytic functions. They do not have the ORDER BY inside the OVER clause. Neither do they support any clause. The ranking done in FIRST and LAST is always DENSE_RANK. The query below shows the usage of FIRST function. The LAST function is used in similar context to perform computations on last ranked records.
-- How each employee's salary compare with the average salary of the first
-- year hires of their department?
SELECT empno, deptno, TO_CHAR(hiredate,'YYYY') HIRE_YR, sal,
TRUNC(
AVG(sal) KEEP (DENSE_RANK FIRST
ORDER BY TO_CHAR(hiredate,'YYYY') )
OVER (PARTITION BY deptno)
) AVG_SAL_YR1_HIRE
FROM emp
WHERE deptno IN (20, 10)
ORDER BY deptno, empno, HIRE_YR;
EMPNO DEPTNO HIRE SAL AVG_SAL_YR1_HIRE
---------- ---------- ---- ---------- ----------------
7782 10 1981 2450 3725
7839 10 1981 5000 3725
7934 10 1982 1300 3725
7369 20 1980 800 800
7566 20 1981 2975 800
7788 20 1982 3000 800
7876 20 1983 1100 800
7902 20 1981 3000 800
8 rows selected.
Query-9 (KEEP FIRST)
注： 分区deptno为10的记录中，均值为 3725 = (2450 + 5000) / 2，因为前两条记录的HIRE值相同。
How to specify the Window clause (ROW type or RANGE type windows)?
Some analytic functions (AVG, COUNT, FIRST_VALUE, LAST_VALUE, MAX, MIN and SUM among the ones we discussed) can take a window clause to further sub-partition the result and apply the analytic function. An important feature of the windowing clause is that it is dynamic in nature.
The general syntax of the is
[ROW or RANGE] BETWEEN AND
can be any one of the following
1. UNBOUNDED PECEDING
2. CURRENT ROW
3. PRECEDING or FOLLOWING.
1. can be any one of the following UNBOUNDED FOLLOWING or
2. CURRENT ROW or
3. PRECEDING or FOLLOWING.
For ROW type windows the definition is in terms of row numbers before or after the current row. So for ROW type windows must evaluate to a positive integer.
For RANGE type windows the definition is in terms of values before or after the current ORDER. We will take this up in details latter.
The ROW or RANGE window cannot appear together in one OVER clause. The window clause is defined in terms of the current row. But may or may not include the current row. The start point of the window and the end point of the window can finish before the current row or after the current row. Only start point cannot come after the end point of the window. In case any point of the window is undefined the default is UNBOUNDED PRECEDING for and UNBOUNDED FOLLOWING for .
If the end point is the current row, syntax only in terms of the start point can be can be
[ROW or RANGE] [ PRECEDING or UNBOUNDED PRECEDING ]
[ROW or RANGE] CURRENT ROW is also allowed but this is redundant. In this case the function behaves as a single-row function and acts only on the current row.
ROW Type Windows
For analytic functions with ROW type windows, the general syntax is:
Function( ) OVER (PARTITIN BY ORDER BY ROWS BETWEEN AND )
or
Function( ) OVER (PARTITON BY ORDER BY ROWS [ PRECEDING or UNBOUNDED PRECEDING]
For ROW type windows the windowing clause is in terms of record numbers.
The query Query-10 has no apparent real life description (except column FROM_PU_C) but the various windowing clause are illustrated by a COUNT(*) function. The count simply shows the number of rows inside the window definition. Note the build up of the count for each column for the YEAR 1981.
The column FROM_P3_TO_F1 shows an example where start point of the window is before the current row and end point of the window is after current row. This is a 5 row window; it shows values less than 5 during the beginning and end.
-- The query below has no apparent real life description (except
-- column FROM_PU_C) but is remarkable in illustrating the various windowing
-- clause by a COUNT(*) function.
SELECT empno, deptno, TO_CHAR(hiredate, 'YYYY') YEAR,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) FROM_P3_TO_F1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM_PU_TO_C,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 PRECEDING) FROM_P2_TO_P1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 1 FOLLOWING AND 3 FOLLOWING) FROM_F1_TO_F3
FROM emp
ORDEDR BY hiredate
EMPNO DEPTNO YEAR FROM_P3_TO_F1 FROM_PU_TO_C FROM_P2_TO_P1 FROM_F1_TO_F3
------ ------- ---- ------------- ------------ ------------- -------------
7369 20 1980 1 1 0 0
7499 30 1981 2 1 0 3
7521 30 1981 3 2 1 3
7566 20 1981 4 3 2 3
7698 30 1981 5 4 3 3
7782 10 1981 5 5 3 3
7844 30 1981 5 6 3 3
7654 30 1981 5 7 3 3
7839 10 1981 5 8 3 2
7900 30 1981 5 9 3 1
7902 20 1981 4 10 3 0
7934 10 1982 2 1 0 1
7788 20 1982 2 2 1 0
7876 20 1983 1 1 0 0
14 rows selected.
Query-10 (ROW type windowing example)
The column FROM_PU_TO_CURR shows an example where start point of the window is before the current row and end point of the window is the current row. This column only has some real world significance. It can be thought of as the yearly employee build-up of the organization as each employee is getting hired.
The column FROM_P2_TO_P1 shows an example where start point of the window is before the current row and end point of the window is before the current row. This is a 3 row window and the count remains constant after it has got 3 previous rows.
The column FROM_F1_TO_F3 shows an example where start point of the window is after the current row and end point of the window is after the current row. This is a reverse of the previous column. Note how the count declines during the end.
RANGE Windows
For RANGE windows the general syntax is same as that of ROW:
Function( ) OVER (PARTITION BY ORDER BY RANGE BETWEEN AND )
or
Function( ) OVER (PARTITION BY ORDER BY RANGE [ PRECEDING or UNBOUNDED PRECEDING]
For or we can use UNBOUNDED PECEDING, CURRENT ROW or PRECEDING or FOLLOWING. However for RANGE type windows must evaluate to value compatible with ORDER BY expression .
is a logical offset. It must be a constant or expression that evaluates to a positive numeric value or an interval literal. Only one ORDER BY expression is allowed.
If evaluates to a numeric value, then the ORDER BY expr must be a NUMBER or DATE datatype. If evaluates to an interval value, then the ORDER BY expr must be a DATE datatype.
Note the example (Query-11) below which uses RANGE windowing. The important thing here is that the size of the window in terms of the number of records can vary.
-- For each employee give the count of employees getting half more that their
-- salary and also the count of employees in the departments 20 and 30 getting half
-- less than their salary.
SELECT deptno, empno, sal,
Count(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN UNBOUNDED PRECEDING AND (sal/2) PRECEDING) CNT_LT_HALF,
COUNT(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN (sal/2) FOLLOWING AND UNBOUNDED FOLLOWING) CNT_MT_HALF
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, sal
DEPTNO EMPNO SAL CNT_LT_HALF CNT_MT_HALF
------- ------ ----- ----------- -----------
20 7369 800 0 3
20 7876 1100 0 3
20 7566 2975 2 0
20 7788 3000 2 0
20 7902 3000 2 0
30 7900 950 0 3
30 7521 1250 0 1
30 7654 1250 0 1
30 7844 1500 0 1
30 7499 1600 0 1
30 7698 2850 3 0
11 rows selected.
Query-11 (RANGE type windowing example)
注：
CTN_LT_HALF的计算规则 - 当前分区中，与当前记录的sal值偏差范围大于sal/2的之前的记录，即符合记录的sal值为 0 ~ (sal - sal/2)之间，且必须在当前记录之前；
CTN_MT_HALF的计算规则 - 当前分区中，与当前记录的sal值偏差范围大于sal/2的之后的记录，即符合记录的sal值大于 sal + sal/2，且必须在当前记录之后
另参考 http://www.club-oracle.com/articles/analytic-functions-ii-windowing-clause-168/
Order of computation and performance tips
Defining the PARTITOIN BY and ORDER BY clauses on indexed columns (ordered in accordance with the PARTITION CLAUSE and then the ORDER BY clause in analytic function) will provide optimum performance. For Query-5, for example, a composite index on (deptno, hiredate) columns will prove effective.
It is advisable to always use CBO for queries using analytic functions. The tables and indexes should be analyzed and optimizer mode should be CHOOSE.
Even in absence of indexes analytic functions provide acceptable performance but need to do sorting for computing partition and order by clause. If the query contains multiple analytic functions, sorting and partitioning on two different columns should be avoided if they are both not indexed.
Conclusion
The aim of this article is not to make the reader try analytic functions forcibly in every other complex SQL. It is meant for a SQL coder, who has been avoiding analytic functions till now, even in complex analytic queries and reinventing the same feature much painstakingly by native SQL and join query. Its job is done if such a person finds analytic functions clear, understandable and usable after going through the article, and starts using them.
Oracle 11g支持的分析函数有（参考书籍 Expert Oracle Database 11g Administration）：
排名函数(Ranking functions)： 根据某个规则对数据集中的数据项排名，常见函数有RANK, DENSE_RANK, CUME_DIST, PERCENT_RANK, NTILE；
移动时间窗合计(Moving-window aggregates)： 可分析累计值和移动平均数；
时间段比较（Period-over-period comparisons）： 可对两个时间段进行比较，比如2010年Q1的利润比2009年Q1增长了百分之几？
比例报告（Ratio-to-report comparisons）：可用于比例对比，比如8月份注册人数占全年注册人数的百分之几？
统计函数（Statistical functions）： 相关性分析和回归函数，可从数据中挖掘因果关系；
反向百分比（Inverse percentiles）： 可根据一个百分比值找到对应的数据，比如找中等收入家庭的户主名称，中等收入值即百分比，从该值判断出对应的家庭户主名称；
推测排名和分布（Hypothetical ranks and distributions）： 可刻划新数据在既有数据中的排名和分布；
直方图（Histograms）： 根据表中的行值返回适当的直方图数据值；
第一/最后聚合（Fist/last aggregates）： 适用于GROUP BY语句排序数据
这里有一篇分析函数入门教程，原文参考 http://www.orafaq.com/node/55
Analytic functions by Example
This article provides a clear, thorough concept of analytic functions and its various options by a series of simple yet concept building examples. The article is intended for SQL coders, who for might be not be using analytic functions due to unfamiliarity with its cryptic syntax or uncertainty about its logic of operation. Often I see that people tend to reinvent the feature provided by analytic functions by native join and sub-query SQL. This article assumes familiarity with basic Oracle SQL, sub-query, join and group function from the reader. Based on that familiarity, it builds the concept of analytic functions through a series of examples.
It is true that whatever an analytic function does can be done by native SQL, with join and sub-queries. But the same routine done by analytic function is always faster, or at least as fast, when compared to native SQL. Moreover, I am not considering here the amount of time that is spent in coding the native SQLs, testing, debugging and tuning them.
The general syntax of analytic function is:
Function(arg1,..., argn) OVER ( [PARTITION BY <...>] [ORDER BY <....>] [
All the keywords will be dealt in details as we walk through the examples. The script for creating the schema (SCOTT) on which the example queries of this article are run can be obtained in ORACLE_HOME/sqlplus/demo/demobld.sql of any standard Oracle installation.
How are analytic functions different from group or aggregate functions?
SELECT deptno,
COUNT(*) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30)
GROUP BY deptno;
DEPTNO DEPT_COUNT
---------------------- ----------------------
20 5
30 6
2 rows selected
Query-1
Consider the Query-1 and its result. Query-1 returns departments and their employee count. Most importantly it groups the records into departments in accordance with the GROUP BY clause. As such any non-"group by" column is not allowed in the select clause.
SELECT empno, deptno,
COUNT(*) OVER (PARTITION BY
deptno) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30);
EMPNO DEPTNO DEPT_COUNT
---------- ---------- ----------
7369 20 5
7566 20 5
7788 20 5
7902 20 5
7876 20 5
7499 30 6
7900 30 6
7844 30 6
7698 30 6
7654 30 6
7521 30 6
11 rows selected.
Query-2
Now consider the analytic function query (Query-2) and its result. Note the repeating values of DEPT_COUNT column.
This brings out the main difference between aggregate and analytic functions. Though analytic functions give aggregate result they do not group the result set. They return the group value multiple times with each record. As such any other non-"group by" column or expression can be present in the select clause, for example, the column EMPNO in Query-2.
Analytic functions are computed after all joins, WHERE clause, GROUP BY and HAVING are computed on the query. The main ORDER BY clause of the query operates after the analytic functions. So analytic functions can only appear in the select list and in the main ORDER BY clause of the query.
In absence of any PARTITION or
SELECT empno, deptno,
COUNT(*) OVER ( ) CNT
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, 1;
EMPNO DEPTNO CNT
---------- ---------- ----------
7782 10 8
7839 10 8
7934 10 8
7369 20 8
7566 20 8
7788 20 8
7876 20 8
7902 20 8
Query-3
SELECT COUNT(*) FROM emp
WHERE deptno IN (10, 20);
COUNT(*)
----------
8
Query-4
How to break the result set in groups or partitions?
It might be obvious from the previous example that the clause PARTITION BY is used to break the result set into groups. PARTITION BY can take any non-analytic SQL expression.
Some functions support the
The functions SUM, COUNT, AVG, MIN, MAX are the common analytic functions the result of which does not depend on the order of the records.
Functions like LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE depends on order of records. In the next example we will see how to specify that.
How to specify the order of the records in the partition?
The answer is simple, by the "ORDER BY" clause inside the OVER( ) clause. This is different from the ORDER BY clause of the main query which comes after WHERE. In this section we go ahead and introduce each of the very useful functions LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE and show how each depend on the order of the record.
The general syntax of specifying the ORDER BY clause in analytic function is:
ORDER BY
The syntax is self-explanatory.
ROW_NUMBER, RANK and DENSE_RANK
All the above three functions assign integer values to the rows depending on their order. That is the reason of clubbing them together.
ROW_NUMBER( ) gives a running serial number to a partition of records. It is very useful in reporting, especially in places where different partitions have their own serial numbers. In Query-5, the function ROW_NUMBER( ) is used to give separate sets of running serial to employees of departments 10 and 20 based on their HIREDATE.
SELECT empno, deptno, hiredate,
ROW_NUMBER( ) OVER (PARTITION BY
deptno ORDER BY hiredate
NULLS LAST) SRLNO
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, SRLNO;
EMPNO DEPTNO HIREDATE SRLNO
------ ------- --------- ----------
7782 10 09-JUN-81 1
7839 10 17-NOV-81 2
7934 10 23-JAN-82 3
7369 20 17-DEC-80 1
7566 20 02-APR-81 2
7902 20 03-DEC-81 3
7788 20 09-DEC-82 4
7876 20 12-JAN-83 5
8 rows selected.
Query-5 (ROW_NUMBER example)
RANK and DENSE_RANK both provide rank to the records based on some column value or expression. In case of a tie of 2 records at position N, RANK declares 2 positions N and skips position N+1 and gives position N+2 to the next record. While DENSE_RANK declares 2 positions N but does not skip position N+1.
Query-6 shows the usage of both RANK and DENSE_RANK. For DEPTNO 20 there are two contenders for the first position (EMPNO 7788 and 7902). Both RANK and DENSE_RANK declares them as joint toppers. RANK skips the next value that is 2 and next employee EMPNO 7566 is given the position 3. For DENSE_RANK there are no such gaps.
SELECT empno, deptno, sal,
RANK() OVER (PARTITION BY deptno
ORDER BY sal DESC NULLS LAST) RANK,
DENSE_RANK() OVER (PARTITION BY
deptno ORDER BY sal DESC NULLS
LAST) DENSE_RANK
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, RANK;
EMPNO DEPTNO SAL RANK DENSE_RANK
------ ------- ----- ----- ----------
7839 10 5000 1 1
7782 10 2450 2 2
7934 10 1300 3 3
7788 20 3000 1 1
7902 20 3000 1 1
7566 20 2975 3 2
7876 20 1100 4 3
7369 20 800 5 4
8 rows selected.
Query-6 (RANK and DENSE_RANK example)
LEAD and LAG
LEAD has the ability to compute an expression on the next rows (rows which are going to come after the current row) and return the value to the current row. The general syntax of LEAD is shown below:
LEAD (
The syntax of LAG is similar except that the offset for LAG goes into the previous rows.
Query-7 and its result show simple usage of LAG and LEAD function.
SELECT deptno, empno, sal,
LEAD(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) NEXT_LOWER_SAL,
LAG(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) PREV_HIGHER_SAL
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, sal DESC;
DEPTNO EMPNO SAL NEXT_LOWER_SAL PREV_HIGHER_SAL
------- ------ ----- -------------- ---------------
10 7839 5000 2450 0
10 7782 2450 1300 5000
10 7934 1300 0 2450
20 7788 3000 3000 0
20 7902 3000 2975 3000
20 7566 2975 1100 3000
20 7876 1100 800 2975
20 7369 800 0 1100
8 rows selected.
Query-7 (LEAD and LAG)
FIRST VALUE and LAST VALUE function
The general syntax is:
FIRST_VALUE(
The FIRST_VALUE analytic function picks the first record from the partition after doing the ORDER BY. The
-- How many days after the first hire of each department were the next
-- employees hired?
SELECT empno, deptno, hiredate ? FIRST_VALUE(hiredate)
OVER (PARTITION BY deptno ORDER BY hiredate) DAY_GAP
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, DAY_GAP;
EMPNO DEPTNO DAY_GAP
---------- ---------- ----------
7369 20 0
7566 20 106
7902 20 351
7788 20 722
7876 20 756
7499 30 0
7521 30 2
7698 30 70
7844 30 200
7654 30 220
7900 30 286
11 rows selected.
Query-8 (FIRST_VALUE)
注： hiredate ? FIRST_VALUE 应为 hiredate - FIRST_VALUE
FIRST and LAST function
The FIRST function (or more properly KEEP FIRST function) is used in a very special situation. Suppose we rank a group of record and found several records in the first rank. Now we want to apply an aggregate function on the records of the first rank. KEEP FIRST enables that.
The general syntax is:
Function( ) KEEP (DENSE_RANK FIRST ORDER BY
Please note that FIRST and LAST are the only functions that deviate from the general syntax of analytic functions. They do not have the ORDER BY inside the OVER clause. Neither do they support any
-- How each employee's salary compare with the average salary of the first
-- year hires of their department?
SELECT empno, deptno, TO_CHAR(hiredate,'YYYY') HIRE_YR, sal,
TRUNC(
AVG(sal) KEEP (DENSE_RANK FIRST
ORDER BY TO_CHAR(hiredate,'YYYY') )
OVER (PARTITION BY deptno)
) AVG_SAL_YR1_HIRE
FROM emp
WHERE deptno IN (20, 10)
ORDER BY deptno, empno, HIRE_YR;
EMPNO DEPTNO HIRE SAL AVG_SAL_YR1_HIRE
---------- ---------- ---- ---------- ----------------
7782 10 1981 2450 3725
7839 10 1981 5000 3725
7934 10 1982 1300 3725
7369 20 1980 800 800
7566 20 1981 2975 800
7788 20 1982 3000 800
7876 20 1983 1100 800
7902 20 1981 3000 800
8 rows selected.
Query-9 (KEEP FIRST)
注： 分区deptno为10的记录中，均值为 3725 = (2450 + 5000) / 2，因为前两条记录的HIRE值相同。
How to specify the Window clause (ROW type or RANGE type windows)?
Some analytic functions (AVG, COUNT, FIRST_VALUE, LAST_VALUE, MAX, MIN and SUM among the ones we discussed) can take a window clause to further sub-partition the result and apply the analytic function. An important feature of the windowing clause is that it is dynamic in nature.
The general syntax of the
[ROW or RANGE] BETWEEN
1. UNBOUNDED PECEDING
2. CURRENT ROW
3.
1.
2. CURRENT ROW or
3.
For ROW type windows the definition is in terms of row numbers before or after the current row. So for ROW type windows
For RANGE type windows the definition is in terms of values before or after the current ORDER. We will take this up in details latter.
The ROW or RANGE window cannot appear together in one OVER clause. The window clause is defined in terms of the current row. But may or may not include the current row. The start point of the window and the end point of the window can finish before the current row or after the current row. Only start point cannot come after the end point of the window. In case any point of the window is undefined the default is UNBOUNDED PRECEDING for
If the end point is the current row, syntax only in terms of the start point can be can be
[ROW or RANGE] [
[ROW or RANGE] CURRENT ROW is also allowed but this is redundant. In this case the function behaves as a single-row function and acts only on the current row.
ROW Type Windows
For analytic functions with ROW type windows, the general syntax is:
Function( ) OVER (PARTITIN BY
or
Function( ) OVER (PARTITON BY
For ROW type windows the windowing clause is in terms of record numbers.
The query Query-10 has no apparent real life description (except column FROM_PU_C) but the various windowing clause are illustrated by a COUNT(*) function. The count simply shows the number of rows inside the window definition. Note the build up of the count for each column for the YEAR 1981.
The column FROM_P3_TO_F1 shows an example where start point of the window is before the current row and end point of the window is after current row. This is a 5 row window; it shows values less than 5 during the beginning and end.
-- The query below has no apparent real life description (except
-- column FROM_PU_C) but is remarkable in illustrating the various windowing
-- clause by a COUNT(*) function.
SELECT empno, deptno, TO_CHAR(hiredate, 'YYYY') YEAR,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) FROM_P3_TO_F1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM_PU_TO_C,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 PRECEDING) FROM_P2_TO_P1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 1 FOLLOWING AND 3 FOLLOWING) FROM_F1_TO_F3
FROM emp
ORDEDR BY hiredate
EMPNO DEPTNO YEAR FROM_P3_TO_F1 FROM_PU_TO_C FROM_P2_TO_P1 FROM_F1_TO_F3
------ ------- ---- ------------- ------------ ------------- -------------
7369 20 1980 1 1 0 0
7499 30 1981 2 1 0 3
7521 30 1981 3 2 1 3
7566 20 1981 4 3 2 3
7698 30 1981 5 4 3 3
7782 10 1981 5 5 3 3
7844 30 1981 5 6 3 3
7654 30 1981 5 7 3 3
7839 10 1981 5 8 3 2
7900 30 1981 5 9 3 1
7902 20 1981 4 10 3 0
7934 10 1982 2 1 0 1
7788 20 1982 2 2 1 0
7876 20 1983 1 1 0 0
14 rows selected.
Query-10 (ROW type windowing example)
The column FROM_PU_TO_CURR shows an example where start point of the window is before the current row and end point of the window is the current row. This column only has some real world significance. It can be thought of as the yearly employee build-up of the organization as each employee is getting hired.
The column FROM_P2_TO_P1 shows an example where start point of the window is before the current row and end point of the window is before the current row. This is a 3 row window and the count remains constant after it has got 3 previous rows.
The column FROM_F1_TO_F3 shows an example where start point of the window is after the current row and end point of the window is after the current row. This is a reverse of the previous column. Note how the count declines during the end.
RANGE Windows
For RANGE windows the general syntax is same as that of ROW:
Function( ) OVER (PARTITION BY
or
Function( ) OVER (PARTITION BY
For
If
Note the example (Query-11) below which uses RANGE windowing. The important thing here is that the size of the window in terms of the number of records can vary.
-- For each employee give the count of employees getting half more that their
-- salary and also the count of employees in the departments 20 and 30 getting half
-- less than their salary.
SELECT deptno, empno, sal,
Count(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN UNBOUNDED PRECEDING AND (sal/2) PRECEDING) CNT_LT_HALF,
COUNT(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN (sal/2) FOLLOWING AND UNBOUNDED FOLLOWING) CNT_MT_HALF
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, sal
DEPTNO EMPNO SAL CNT_LT_HALF CNT_MT_HALF
------- ------ ----- ----------- -----------
20 7369 800 0 3
20 7876 1100 0 3
20 7566 2975 2 0
20 7788 3000 2 0
20 7902 3000 2 0
30 7900 950 0 3
30 7521 1250 0 1
30 7654 1250 0 1
30 7844 1500 0 1
30 7499 1600 0 1
30 7698 2850 3 0
11 rows selected.
Query-11 (RANGE type windowing example)
注：
CTN_LT_HALF的计算规则 - 当前分区中，与当前记录的sal值偏差范围大于sal/2的之前的记录，即符合记录的sal值为 0 ~ (sal - sal/2)之间，且必须在当前记录之前；
CTN_MT_HALF的计算规则 - 当前分区中，与当前记录的sal值偏差范围大于sal/2的之后的记录，即符合记录的sal值大于 sal + sal/2，且必须在当前记录之后
另参考 http://www.club-oracle.com/articles/analytic-functions-ii-windowing-clause-168/
Order of computation and performance tips
Defining the PARTITOIN BY and ORDER BY clauses on indexed columns (ordered in accordance with the PARTITION CLAUSE and then the ORDER BY clause in analytic function) will provide optimum performance. For Query-5, for example, a composite index on (deptno, hiredate) columns will prove effective.
It is advisable to always use CBO for queries using analytic functions. The tables and indexes should be analyzed and optimizer mode should be CHOOSE.
Even in absence of indexes analytic functions provide acceptable performance but need to do sorting for computing partition and order by clause. If the query contains multiple analytic functions, sorting and partitioning on two different columns should be avoided if they are both not indexed.
Conclusion
The aim of this article is not to make the reader try analytic functions forcibly in every other complex SQL. It is meant for a SQL coder, who has been avoiding analytic functions till now, even in complex analytic queries and reinventing the same feature much painstakingly by native SQL and join query. Its job is done if such a person finds analytic functions clear, understandable and usable after going through the article, and starts using them.
如何解决ssh登录太慢的问题
登录sshd时发现等待的时间很长，怀疑是 dns 反向解析的问题，于是编辑 /etc/ssh/sshd_config把
UseDNS yes
改成
UseDNS no
重启sshd服务，登录还是很慢。于是在运行登录命令时打开消息输出选项
ssh -vvv myusername@myhost
发现卡在
...
debug1: Next authentication method: gssapi-with-mic
...
于是把 /etc/ssh/sshd_config里的
GSSAPIAuthentication yes
改成
GSSAPIAuthentication no
重启sshd服务，问题解决。
UseDNS yes
改成
UseDNS no
重启sshd服务，登录还是很慢。于是在运行登录命令时打开消息输出选项
ssh -vvv myusername@myhost
发现卡在
...
debug1: Next authentication method: gssapi-with-mic
...
于是把 /etc/ssh/sshd_config里的
GSSAPIAuthentication yes
改成
GSSAPIAuthentication no
重启sshd服务，问题解决。
CentOS 如何设置缺省网关
假设缺省网关是74.86.49.129，通常做法
# vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=wan.nixcraft.in
GATEWAY=74.86.49.129
# /etc/init.d/networking restart
也可以通过setup程序设置网卡的缺省网关，但在有多块网卡的情况下，总是最后一块网卡的缺省网关有效，有时候这并不符合我们的期望。
根据网卡设置路由的方法是编辑 /etc/sysconfig/network-scripts/route-eth* 文件，
Open /etc/sysconfig/network and append
GATEWAY=eth0.gateway.IP.address
Open /etc/sysconfig/network-scripts/route-eth1 and add routing for eth1 and restart it. For example:
202.54.1.2/29 via 202.54.2.254
Also, set static route for /etc/sysconfig/network-scripts/route-eth0:
10.1.2.3/8 via 10.10.38.95
Do not add gateway entries to /etc/sysconfig/network-scripts/ifcfg-eth1 and /etc/sysconfig/network-scripts/ifcfg-eth0
参考 http://www.cyberciti.biz/faq/howto-rhel-fedora-linux-setup-default-gateway/
# vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=wan.nixcraft.in
GATEWAY=74.86.49.129
# /etc/init.d/networking restart
也可以通过setup程序设置网卡的缺省网关，但在有多块网卡的情况下，总是最后一块网卡的缺省网关有效，有时候这并不符合我们的期望。
根据网卡设置路由的方法是编辑 /etc/sysconfig/network-scripts/route-eth* 文件，
Open /etc/sysconfig/network and append
GATEWAY=eth0.gateway.IP.address
Open /etc/sysconfig/network-scripts/route-eth1 and add routing for eth1 and restart it. For example:
202.54.1.2/29 via 202.54.2.254
Also, set static route for /etc/sysconfig/network-scripts/route-eth0:
10.1.2.3/8 via 10.10.38.95
Do not add gateway entries to /etc/sysconfig/network-scripts/ifcfg-eth1 and /etc/sysconfig/network-scripts/ifcfg-eth0
参考 http://www.cyberciti.biz/faq/howto-rhel-fedora-linux-setup-default-gateway/
实现跨机房NAT转发
场景描述：有两台服务器 HostA 和 HostB，位于不同的机房。需要把 HostA 的邮件连接请求(tcp/25)通过 HostB转发到公网其他机器，在其他机器看来，该邮件连接请求仿佛是从HostB发出来的。
这种需求是有的，比如HostA被列为邮件服务器黑名单，把HostA的邮件服务迁移到HostB又比较麻烦，最省心的应急处理方式是把HostB作为邮件发送跳板。
基本思路是在HostA和HostB之间建立一个ip通道(ssl通道、pptp或者ipsec)，然后把对外连接tcp/25的请求转发到HostB，再由HostB做来源地址改写（SNAT）。
1. 建立 HostA与HostB之间的ip通道
为简单起见，我们用ipip方式建立通道。在HostA做如下配置
modprobe ipip
ip tu add ipiptun mode ipip local remote ttl 64 dev eth0
ip ad ad dev ipiptun 172.33.1.1 peer 172.33.1.2/32
ip li se dev ipiptun up
在HostB做如下配置
modprobe ipip
ip tu add ipiptun mode ipip local remote ttl 64 dev eth0
ip ad ad dev ipiptun 172.33.1.2 peer 172.33.1.1/32
ip li se dev ipiptun up
2. 设置HostA的Netfilter Marking & Routing
echo 201 mail.out >> /etc/iproute2/rt_tables
iptables -A OUTPUT -t mangle -p tcp --dport 25 -j MARK --set-mark 1
ip rule add fwmark 1 table mail.out
ip rule ls
ip route add default via 172.33.1.2 dev ipiptun table mail.out
3. 设置HostB的SNAT
iptables -A INPUT -i ipiptun -j ACCEPT
iptables -t nat -A POSTROUTING -s 172.33.1.1/32 -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -s/32 -o eth1 -j MASQUERADE
这种需求是有的，比如HostA被列为邮件服务器黑名单，把HostA的邮件服务迁移到HostB又比较麻烦，最省心的应急处理方式是把HostB作为邮件发送跳板。
基本思路是在HostA和HostB之间建立一个ip通道(ssl通道、pptp或者ipsec)，然后把对外连接tcp/25的请求转发到HostB，再由HostB做来源地址改写（SNAT）。
1. 建立 HostA与HostB之间的ip通道
为简单起见，我们用ipip方式建立通道。在HostA做如下配置
modprobe ipip
ip tu add ipiptun mode ipip local
ip ad ad dev ipiptun 172.33.1.1 peer 172.33.1.2/32
ip li se dev ipiptun up
在HostB做如下配置
modprobe ipip
ip tu add ipiptun mode ipip local
ip ad ad dev ipiptun 172.33.1.2 peer 172.33.1.1/32
ip li se dev ipiptun up
2. 设置HostA的Netfilter Marking & Routing
echo 201 mail.out >> /etc/iproute2/rt_tables
iptables -A OUTPUT -t mangle -p tcp --dport 25 -j MARK --set-mark 1
ip rule add fwmark 1 table mail.out
ip rule ls
ip route add default via 172.33.1.2 dev ipiptun table mail.out
3. 设置HostB的SNAT
iptables -A INPUT -i ipiptun -j ACCEPT
iptables -t nat -A POSTROUTING -s 172.33.1.1/32 -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -s
如何删除单个iptables规则
见 http://www.webhostingtalk.com/showthread.php?t=518035
With command line :
iptables -L INPUT -n --line-numbers
You'll get the list of all blocked IP. Look at the number on the left, then :
iptables -D INPUT <<number>>
注：INPUT是iptables的过滤链名称，常见的有PREROUTING/INPUT/FORWARD/OUTPUT/POSTROUTING等
With command line :
iptables -L INPUT -n --line-numbers
You'll get the list of all blocked IP. Look at the number on the left, then :
iptables -D INPUT <<number>>
注：INPUT是iptables的过滤链名称，常见的有PREROUTING/INPUT/FORWARD/OUTPUT/POSTROUTING等
如何使用MySQL执行每秒75万次的NOSQL查询
原文见 http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html
随着Web应用变得越来越复杂，单纯的MySQL + Memcached似乎已满足不了数据存储的需求，一些企业纷纷转向NoSQL方案，比如MongoDB、CouchDB、 TokyoCabinet/Tyrant、Cassandra等。在他们看来，如果数据访问模式不是很复杂，用不上SQL数据库。然而，DeNA公司截然相反，他们选择了 "only MySQL" 的方案，且获得了远远超越NoSQL的性能。
该公司仍在使用MySQL + Memcached，Memcached主要用于前端Cache，比如预处理的HTML、计数和摘要信息等，但数据行并不放在Cache里，而是直接从数据库查，因为普通的服务器就可以获得75万次每秒的查询，当前又有哪种NoSQL可以做到呢？
可以使用sysbench、super-smack、mysqlslap等工具测试MySQL性能，比如
[matsunobu@host ~]$ mysqlslap --query="select user_name,..
from test.user where user_id=1" \
--number-of-queries=10000000 --concurrency=30 --host=xxx -uroot
然后使用如下命令得到每秒读取的行数，
[matsunobu@host ~]$ mysqladmin extended-status -i 1 -r -uroot \
| grep -e "Com_select"
...
| Com_select | 107069 |
| Com_select | 108873 |
| Com_select | 108921 |
| Com_select | 109511 |
| Com_select | 108084 |
| Com_select | 108483 |
| Com_select | 108115 |
...
可以使用vmstat和Oprofile等工具诊断系统瓶颈。
MySQL Cluster因为性能问题一直受人批判，为改善这种情况引入了NDBAPI，使得性能提升了N倍。但对于非集群情况怎么优化呢？通过MySQL瓶颈分 析，发现大部分时间花费在SQL解析和表操作上，如果绕过这层操作直接存取存储引擎，可大大提升性能，MySQL的插件HandlerSocket正是由 此获得了每秒75万次查询操作的性能，这个评测数据无疑会颠覆整个NoSQL世界。另外，HandlerSocket支持批量读取和写操作，这进一步提升 了它的性能。
原文评论部分提到使用支持多个Tx/Rx队列的网卡可提升IO性能，方便时可测试一下。
随着Web应用变得越来越复杂，单纯的MySQL + Memcached似乎已满足不了数据存储的需求，一些企业纷纷转向NoSQL方案，比如MongoDB、CouchDB、 TokyoCabinet/Tyrant、Cassandra等。在他们看来，如果数据访问模式不是很复杂，用不上SQL数据库。然而，DeNA公司截然相反，他们选择了 "only MySQL" 的方案，且获得了远远超越NoSQL的性能。
该公司仍在使用MySQL + Memcached，Memcached主要用于前端Cache，比如预处理的HTML、计数和摘要信息等，但数据行并不放在Cache里，而是直接从数据库查，因为普通的服务器就可以获得75万次每秒的查询，当前又有哪种NoSQL可以做到呢？
可以使用sysbench、super-smack、mysqlslap等工具测试MySQL性能，比如
[matsunobu@host ~]$ mysqlslap --query="select user_name,..
from test.user where user_id=1" \
--number-of-queries=10000000 --concurrency=30 --host=xxx -uroot
然后使用如下命令得到每秒读取的行数，
[matsunobu@host ~]$ mysqladmin extended-status -i 1 -r -uroot \
| grep -e "Com_select"
...
| Com_select | 107069 |
| Com_select | 108873 |
| Com_select | 108921 |
| Com_select | 109511 |
| Com_select | 108084 |
| Com_select | 108483 |
| Com_select | 108115 |
...
可以使用vmstat和Oprofile等工具诊断系统瓶颈。
MySQL Cluster因为性能问题一直受人批判，为改善这种情况引入了NDBAPI，使得性能提升了N倍。但对于非集群情况怎么优化呢？通过MySQL瓶颈分 析，发现大部分时间花费在SQL解析和表操作上，如果绕过这层操作直接存取存储引擎，可大大提升性能，MySQL的插件HandlerSocket正是由 此获得了每秒75万次查询操作的性能，这个评测数据无疑会颠覆整个NoSQL世界。另外，HandlerSocket支持批量读取和写操作，这进一步提升 了它的性能。
原文评论部分提到使用支持多个Tx/Rx队列的网卡可提升IO性能，方便时可测试一下。
Linux配置 WPA-PSK无线上网
WPA-PSK(Pre-Shared Key)，也叫做WPA Personal，是一种比较理想的家庭或小公司无线上网方案。这种方法基于共享口令，只要上网环境没有大嘴先生或女士，还算比较安全。
Linux存取WPA-PSK需要wpa_supplicant软件，Debian和它的派生系列叫做wpasupplicant，配置文件一般保存为/etc/wpa_supplicant.conf。
运行如下命令生成配置文件
$ wpa_passphrase myssid
network={
ssid="myssid"
#psk="waylongkeythelongerthebetterbecausewecareaboutsecurityalot"
psk=ef82e334d941fd88ee8e6d6ef9d112eed40e93e2aa560fcaa326c29659ad375d
}
输入ssid的口令waylongkeythelongerthebetterbecausewecareaboutsecurityalot，把以上命令的输出保存为配置文件 /etc/wpa_supplicant.conf，假设无线接口为eth1，可用下面的命令测试一下
# iwconfig eth1 essid "myessid"
# ifconfig eth1 up
# wpa_supplicant -ieth1 -c/etc/wpa_supplicant.conf
-i参数指定无线网络接口，-c指定配置文件，参数值前不要留空格。
运行下面的命令检查是否无线正常
$ iwconfig eth1
如果一切顺利，可把该过程加到开机启动脚本里，如果是Debian或者Ubuntu，编辑 /etc/network/interfaces
auto eth1
iface eth1 dhcp
up wpa_supplicant -ieth1 -c/etc/wpa_supplicant.conf -B
down killall wpa_supplicant
参数-B指定以后台方式运行
如果是 RHEL或者Fedora，编辑 /etc/sysconfig/network-scripts/ifcfg-eth1，如下
DEVICE=eth1
BOOTPROTO=dhcp
HWADDR=11:22:33:44:55:66
ONBOOT=yes
TYPE=Ethernet
把如下行加入 /etc/sysconfig/network-scripts/ifup-wireless
wpa_supplicant -iath1 -c/etc/wpa_supplicant.conf -B
killall wpa_supplicant
参考 http://www.enterprisenetworkingplanet.com/netsecur/article.php/10952_3594946_1
Linux存取WPA-PSK需要wpa_supplicant软件，Debian和它的派生系列叫做wpasupplicant，配置文件一般保存为/etc/wpa_supplicant.conf。
运行如下命令生成配置文件
$ wpa_passphrase myssid
network={
ssid="myssid"
#psk="waylongkeythelongerthebetterbecausewecareaboutsecurityalot"
psk=ef82e334d941fd88ee8e6d6ef9d112eed40e93e2aa560fcaa326c29659ad375d
}
输入ssid的口令waylongkeythelongerthebetterbecausewecareaboutsecurityalot，把以上命令的输出保存为配置文件 /etc/wpa_supplicant.conf，假设无线接口为eth1，可用下面的命令测试一下
# iwconfig eth1 essid "myessid"
# ifconfig eth1 up
# wpa_supplicant -ieth1 -c/etc/wpa_supplicant.conf
-i参数指定无线网络接口，-c指定配置文件，参数值前不要留空格。
运行下面的命令检查是否无线正常
$ iwconfig eth1
如果一切顺利，可把该过程加到开机启动脚本里，如果是Debian或者Ubuntu，编辑 /etc/network/interfaces
auto eth1
iface eth1 dhcp
up wpa_supplicant -ieth1 -c/etc/wpa_supplicant.conf -B
down killall wpa_supplicant
参数-B指定以后台方式运行
如果是 RHEL或者Fedora，编辑 /etc/sysconfig/network-scripts/ifcfg-eth1，如下
DEVICE=eth1
BOOTPROTO=dhcp
HWADDR=11:22:33:44:55:66
ONBOOT=yes
TYPE=Ethernet
把如下行加入 /etc/sysconfig/network-scripts/ifup-wireless
wpa_supplicant -iath1 -c/etc/wpa_supplicant.conf -B
killall wpa_supplicant
参考 http://www.enterprisenetworkingplanet.com/netsecur/article.php/10952_3594946_1
让Mailman的web GUI使用 HTTPS
修改 apache 的配置，比如 /etc/httpd/conf.d/mailman.conf，把 RedirectMatch 行替换成
RedirectMatch ^/mailman[/]*$ https:///mailman/listinfo
修改 mailman 配置文件 mm_cfg.py，替换如下配置项
# 确保新列表的 web_page_url 属性使用该url模式
DEFAULT_URL_PATTERN = 'https://%s/mailman/'
# 让公开的归档也通过https访问
PUBLIC_ARCHIVE_URL = 'https://%(hostname)s/pipermail/%(listname)s'
修复现有的列表url地址
$prefix/bin/withlist -l -r fix_url
或者修复所有既有的列表地址
$prefix/bin/withlist -l -a -r fix_url
参考 http://wiki.list.org/display/DOC/4.27+Securing+Mailman's+web+GUI+by+using+Secure+HTTP-SSL
RedirectMatch ^/mailman[/]*$ https://
修改 mailman 配置文件 mm_cfg.py，替换如下配置项
# 确保新列表的 web_page_url 属性使用该url模式
DEFAULT_URL_PATTERN = 'https://%s/mailman/'
# 让公开的归档也通过https访问
PUBLIC_ARCHIVE_URL = 'https://%(hostname)s/pipermail/%(listname)s'
修复现有的列表url地址
$prefix/bin/withlist -l -r fix_url
或者修复所有既有的列表地址
$prefix/bin/withlist -l -a -r fix_url
参考 http://wiki.list.org/display/DOC/4.27+Securing+Mailman's+web+GUI+by+using+Secure+HTTP-SSL
使用TELNET进行POP3会话
原文参考 http://techhelp.santovec.us/pop3telnet.htm 可用的命令如下
USER userid
This must be the first command after the connect. Supply your e-mail userid (this may or may not not the full e-mail address). Example: USER john.smith
PASS password
This must be the next command after USER. Supply your e-mail password. The password may be case sensitive.
The following commands may be used as needed:
STAT
The response to this is: +OK #msgs #bytes Where #msgs is the number of messages in the mail box and #bytes is the total bytes used by all messages. Sample response: +OK 3 345910
LIST
The response to this lists a line for each message with its number and size in bytes, ending with a period on a line by itself. Sample response:
+OK 3 messages
1 1205
2 305
3 344400
.
RETR msg#
This sends message number msg# to you (displays on the Telnet screen). You probably don't want to do this in Telnet (unless you have turned on Telnet logging). Example: RETR 2
TOP msg# #lines
This is an optional POP3 command. Not all POP3 servers support it. It lists the header for msg# and the first #lines of the message text. For example, TOP 1 0 would list just the headers for message 1, where as TOP 1 5 would list the headers and first 5 lines of the message text.
DELE msg#
This marks message number msg# for deletion from the server. This is the way to get rid a problem causing message. It is not actually deleted until the QUIT command is issued. If you lose the connection to the mail server before issuing the QUIT command, the server should not delete any messages. Example: DELE 3
RSET
This resets (unmarks) any messages previously marked for deletion in this session so that the QUIT command will not delete them.
QUIT
This deletes any messages marked for deletion, and then logs you off of the mail server. This is the last command to use. This does not disconnect you from the ISP, just the mailbox.
There are other POP3 commands. For some more information on this topic, see:
RFC 1939 which defines POP3 commands and error codes
How to Enable and Interpret the Pop3.log File article, which is specific to the Microsoft Internet Mail program (the logging only works with that program), but it provides general information on what goes on with a POP3 mail server.
USER userid
This must be the first command after the connect. Supply your e-mail userid (this may or may not not the full e-mail address). Example: USER john.smith
PASS password
This must be the next command after USER. Supply your e-mail password. The password may be case sensitive.
The following commands may be used as needed:
STAT
The response to this is: +OK #msgs #bytes Where #msgs is the number of messages in the mail box and #bytes is the total bytes used by all messages. Sample response: +OK 3 345910
LIST
The response to this lists a line for each message with its number and size in bytes, ending with a period on a line by itself. Sample response:
+OK 3 messages
1 1205
2 305
3 344400
.
RETR msg#
This sends message number msg# to you (displays on the Telnet screen). You probably don't want to do this in Telnet (unless you have turned on Telnet logging). Example: RETR 2
TOP msg# #lines
This is an optional POP3 command. Not all POP3 servers support it. It lists the header for msg# and the first #lines of the message text. For example, TOP 1 0 would list just the headers for message 1, where as TOP 1 5 would list the headers and first 5 lines of the message text.
DELE msg#
This marks message number msg# for deletion from the server. This is the way to get rid a problem causing message. It is not actually deleted until the QUIT command is issued. If you lose the connection to the mail server before issuing the QUIT command, the server should not delete any messages. Example: DELE 3
RSET
This resets (unmarks) any messages previously marked for deletion in this session so that the QUIT command will not delete them.
QUIT
This deletes any messages marked for deletion, and then logs you off of the mail server. This is the last command to use. This does not disconnect you from the ISP, just the mailbox.
There are other POP3 commands. For some more information on this topic, see:
RFC 1939 which defines POP3 commands and error codes
How to Enable and Interpret the Pop3.log File article, which is specific to the Microsoft Internet Mail program (the logging only works with that program), but it provides general information on what goes on with a POP3 mail server.
通过TELNET存取IMAP邮箱
原文见 http://bobpeers.com/technical/telnet_imap
The basics.
About IMAP.
IMAP is an email protocol for organizing, storing and retrieving emails on a remote server. It wasdeveloped after POP and is a much more advanced system, one of the main differences being that all the mail is stored on the server so it remains accessible from manydifferent locations. With POP you have to download the mail to your local computer in order to read it and therefore you cannot synchronize your mail across many different machines.It may be more complex than POP but there are still only a few core commands we need to know in order to access our mail on an IMAP server.
Before starting it's important to know a few things:
IMAP command syntax.
Before the actual command is typed into the terminal we need to type a command tag, this could be anything (without spaces) and the server will tag its response with the tag we give it. This seems to be because IMAP allows multiple connections and so multiple commands, by tagging you know which response refers to which command.
In our case we have only 1 connection and we send single commands so it's not really relevant, however we need to type something as a tag. I usually just use a period '.' but you could use a number or whatever suits you. To demonstrate the command tag see the two server responses here with the tag (don't worry about the command itself, it will be explained soon), in the first one we send '. fetch' and the second one 'a01a fetch' getting the same tag back to identify the response:
. fetch 1 fast
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013)
. OK Completed (0.000 sec)
ao1a fetch 1 fast
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013)
a01a OK Completed (0.000 sec)
Finally, the IMAP commands are not case sensitive, so 'SELECT inbox' will work just as well as 'select INBOX'. For clarity in the code I have typed the commands in uppercase and the word INBOX in uppercase also.
Mail server address.
The address of your mail server, this will usually be of the form mail.domain.com. You should look at the settings in your email client or documentation about your email account to get this information.
Security.
In this demonstration we will be sending our account username and password unencrypted over the internet, if this is a major concern to you then you should not follow this exercise.
Another alternative, if your email provider supports SSL, is to use OpenSSL (which most if not all Linux computers will have), see the 'Connecting to the host' section
below for the syntax.
Using telnet.
If you make a mistake in a telnet session you cannot use backspace to delete the entry, you may have to press enter to get an error and then re-type the command or quit and start again.
Connecting to the host.
Insecure login - login using telnet.
By insecure I just mean that your username and password are sent unencrypted over the internet so potentially could be intercepted on the route between your computer and the mail server.
First open up a terminal and type the following, of course replacing mail.myserver.com with the address of your IMAP server, note that the IMAP port used is 143:
telnet mail.myserver.com 143
This should return something like:
telnet mail.myserver.com 143
Trying 66.111.4.160...
Connected to mail.myserver.com (66.111.4.160).
Escape character is '^]'.
* OK IMAP4 ready
Secure login - login using OpenSSL.
To open an SSL session that encrypts all data sent between your computer and the mail server open a teminal and follow these steps, note that we use port 993 here:
openssl s_client -connect mail.myserver.com:993
CONNECTED(00000003)
depth=0 /C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine .com
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN= Thawte Premium Server CA/emailAddress=premium-server@thawte.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDeDCCAuGgAwIBAgIDQBYSMA0GCSqGSIb3DQEBBAUAMIHOMQswCQYDVQQGEwJa
..........................
-----END CERTIFICATE-----
subject=/C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingeng ine.com
issuer=/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/C N=Thawte Premium Server CA/emailAddress=premium-server@thawte.com
---
No client certificate CA names sent
---
SSL handshake has read 1054 bytes and written 340 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: Session ID
Session-ID-ctx:
Master-Key: Key
Key-Arg : None
Krb5 Principal: None
Start Time: 1140271254
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
* OK IMAP4 ready
Once this step is carried out the IMAP commands are identical to those for a normal telnet session.
Logging in.
Next we need to log in using the login command. Type '. login' followed by your username and password separated by spaces.
. login accountname@myserver.com *********
. OK User logged in
LIST command.
To see a list of all the mailboxes on the server we use the list command. The arguments "" "*" simply get all the mailboxes including sub folders.
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.test1"
* LIST (\HasNoChildren) "." "INBOX.test2"
. OK Completed (0.460 secs 7 calls)
We can see from this output how the mailboxes are arranged like a tree with INBOX being the 'trunk'. My IMAP provider uses a period (.) as a separator between parent and child folders so INBOX.Drafts is a child of the INBOX. The \HasChildren simply tells us that this folder has sub folders whereas the other folders do not. The way IMAP works means that all folders are created as subfolders of the INBOX even if your email client is configured not to show it that way.
STATUS command.
This command return some basic information on the folder without selecting the folder, it takes arguments depending on what information you would like returned.
Here are 3 example showing total messages, recent messages and unseen messages.
. status INBOX (messages)
* STATUS INBOX (MESSAGES 2)
. OK Completed
. status INBOX (recent)
* STATUS INBOX (RECENT 0)
. OK Completed
. status INBOX (unseen)
* STATUS INBOX (UNSEEN 0)
. OK Completed
EXAMINE and SELECT commands.
These two commands basically do the same thing, they return information on the folder chosen and then allow us to access the mails stored inside the folder. The
main difference is that EXAMINE returns a read-only reference whereas SELECT is read-write.
. examine INBOX.test2
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1138801117]
* OK [UIDNEXT 1]
. OK [READ-ONLY] Completed
. select INBOX.test2
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1138801117]
* OK [UIDNEXT 1]
. OK [READ-WRITE] Completed
Note that the only difference in response is the [READ-ONLY] and [READ-WRITE] text. Basically this command just tells us the possible IMAP flags we can set, EXIST is how many mails are in the folder, RECENT is how many recent mails there are (the SELECT command will remove the RECENT flag since the folder has now been viewed, this is not the same as the \Seen IMAP flag, also note that the EXAMINE command will not reset the RECENT flag).
The RECENT data is what tells an IMAP email client if you have new mails, by clicking on the folder the client sends the SELECT command and the new mail icon disappears even though the mails are still unread.
CREATE, DELETE and RENAME folders.
It's very easy to create and delete folders, just make sure you create them as subfolders of the INBOX. For example to create a top level folder called test3 do do the following.
. create INBOX.test3
. OK Completed
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.test1"
* LIST (\HasNoChildren) "." "INBOX.test2"
* LIST (\HasNoChildren) "." "INBOX.test3" #we created this
. OK Completed (0.420 secs 8 calls)
Conversely we can delete our new folder using the DELETE command. Note that you cannot delete a folder that had subfolders without first deleting the subfolders, also
deleting a folder containing mails will delete all the mails inside so beware!
. delete INBOX.test3
. OK Completed
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.test1"
* LIST (\HasNoChildren) "." "INBOX.test2"
. OK Completed (0.430 secs 7 calls)
Renaming a folder is just as easy, just type RENAME [current name] [new name]. This will not delete mails as they will just exist in the new folder. Here we rename
folder test1 to linux.
. rename INBOX.test1 INBOX.test3
* OK rename user.accountname.test1 user.accountname.test3
. OK Completed
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.linux" #this was test1
* LIST (\HasNoChildren) "." "INBOX.test2"
. OK Completed (0.410 secs 7 calls)
FETCH command.
This command is the main command we use to actually access our emails. It has many possible options depending in what you wish to see, message flags, email headers,
text of the body etc. Here we select the INBOX and fetch the emails in a few different ways.
. select INBOX
* FLAGS (\Answered \Flagged \Draft \DeleteCLOSE and EXPUNGE commands.d \Seen hasatt)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt \*)]
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1138801043]
* OK [UIDNEXT 3]
. OK [READ-WRITE] Completed
. fetch 1 flags
* 1 FETCH (FLAGS (\Seen hasatt))
. OK Completed (0.000 sec)
First we shall fetch the message IMAP flags for all the messages in the folder.
. fetch 1:2 flags
* 1 FETCH (FLAGS (\Seen hasatt))
* 2 FETCH (FLAGS (\Seen hasatt))
. OK Completed
Note that with all the commands that act upon messages we can select either 1 message by using the message number as in 'fetch 1 command' or we can select a range
of messages in the format 'fetch first:last command' or all the messages 'fetch 1:last command'. Also note that we can use '*' to indicate all messages so fetch 1:* will get all the messages from the first to the last without us knowing how many messages are in the folder.
First we shall fetch using fast, all and full options (these refer to the headers).
. fetch 1 fast
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013)
. OK Completed (0.000 sec)
. fetch 1 all
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013 ENVELOPE ("Wed, 1 Feb 2006 13:37:19 UT" "IMPORTANT: Click here to begin using your account" (("Email Administrator" NIL "bounce" "myserver.com")) (("Email Administrator" NIL "bounce" "myserver.com")) ((NIL NIL "webmaster" "myserver.com")) (("Joe Bloggs" NIL "accountname" "myserver.com")) NIL NIL NIL ""))
. OK Completed (0.000 sec)
. fetch 1 full
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013 ENVELOPE ("Wed, 1 Feb 2006 13:37:19 UT" "IMPORTANT: Click here to begin using your account" (("Email Administrator" NIL "bounce" "myserver.com")) (("Email Administrator" NIL "bounce" "myserver.com")) ((NIL NIL "webmaster" "myserver.com")) (("Joe Bloggs" NIL "accountname" "myserver.com")) NIL NIL NIL "") BODY ((("TEXT" "PLAIN" NIL NIL NIL "8BIT" 5599 137)("TEXT" "HTML" NIL NIL NIL "8BIT" 7434 141) "ALTERNATIVE")("TEXT" "PLAIN" ("NAME" "This_is_how_attachments_appear.txt") NIL NIL "8BIT" 247 6) "MIXED"))
. OK Completed (0.000 sec)
As you can see this returns differing amounts of data about the IMAP flags, size and ENVELOPE information. It's maybe more informative to use either 'fetch message body[header]' or 'fetch message rfc822.header' both of which return the data below.
. fetch 1 rfc822.header
* 1 FETCH (RFC822.HEADER {824}
Return-Path:
Received: from web2.internal (web2.internal [10.202.2.211])
by server2.messagingengine.com (Cyrus v2.3-alpha) with LMTPA;
Wed, 01 Feb 2006 08:37:23 -0500
X-Sieve: CMU Sieve 2.3
X-Attached: This_is_how_attachments_appear.txt
X-Resolved-to: accountname
X-Mail-from: nobody
Content-Transfer-Encoding: 8bit
Content-Type: multipart/mixed; boundary="_----------=_1138801039165120"
MIME-Version: 1.0
X-Mailer: MIME::Lite 5022 (F2.73; T1.15; A1.64; B3.05; Q3.03)
Date: Wed, 1 Feb 2006 13:37:19 UT
From: "Email Administrator"
Reply-To: webmaster@myserver.com
To: "Joe Bloggs"
Subject: IMPORTANT: Click here to begin using your account
Message-ID:
)
. OK Completed (0.000 sec)
To fetch only some headers we can select the header fields we wish to see.
. fetch 1 (body[header.fields (from to subject date)])
* 1 FETCH (BODY[HEADER.FIELDS (from to subject date)] {195}
Date: Wed, 1 Feb 2006 13:37:19 UT
From: "Email Administrator"
To: "Joe Bloggs"
Subject: IMPORTANT: Click here to begin using your account
)
. OK Completed (0.000 sec)
To read the body of the email message we can use either 'fetch message body[text]' or 'fetch message rfc822.text' as shown here.
. fetch 2 rfc822.text
* 2 FETCH (RFC822.TEXT {11658}
This is a multi-part message in MIME format.
--_----------=_1138865560223950
Content-Disposition: inline
Content-Length: 5194
Content-Transfer-Encoding: binary
Content-Type: text/plain
more text here.............
. OK Completed (0.000 sec)
STORE command.
This command allows us to add, remove or replace the IMAP flags on the messages. These are flags that denote a message as replied to, deleted, seen etc. and allow
the message information, as well as the message itself, to be synchronized across different computers. Note that the STORE command causes an automatic FETCH
command of the message flags so we can see the change immediately. There are 3 ways to use STORE:
* STORE message +flags [flag list] - this adds the [flag list] flags to the chosen messages.
* STORE message -flags [flag list] - this removes the [flag list] flags from the chosen messages.
* STORE message flags [flag list] - resets the flags to [flag list] on the chosen messages (the same as removing all flags and then adding [flag list].
The list of flags to add include \Answered \Flagged \Draft \Deleted \Seen and many more. All the IMAP flags used as part of the standard installation have the
backslash in front of them. However some email clients (Thunderbird is one) also allow you to set labels or mark a message as junk, if you add labels do not use
the backslash. First we shall mark all the messages as deleted.
. store 1:2 flags \Deleted
* 1 FETCH (FLAGS (\Recent \Deleted))
* 2 FETCH (FLAGS (\Recent \Deleted))
. OK Completed
Next replace the flags with $label1
. store 1:* flags $label1
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt Junk label1)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt Junk $label1 \*)]
* 1 FETCH (FLAGS ($label1))
* 2 FETCH (FLAGS ($label1))
. OK Completed
Finally we can add the flag NonJunk so that Thunderbird recognises them as not being junk mail.
. store 1:* +flags NonJunk
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt NonJunk Junk label1)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt NonJunk Junk $label1 \*)]
* 1 FETCH (FLAGS ($label1 NonJunk))
* 2 FETCH (FLAGS ($label1 NonJunk))
. OK Completed
Note that the \Deleted flag is used by an IMAP server to mark an email ready for deletion, it is not actually deleted until the server receives either the CLOSE
or EXPUNGE command shown below.
CLOSE and EXPUNGE commands.
Both these commands have the effect of permanently deleting any messages in the current folder marked for deletion with the \Deleted flag. EXPUNGE just deletes the
messages but does nothing else (this command is the equivalent of compacting folders in Thunderbird), while CLOSE deletes the messages and deselects the current
folder (you cannot carry out more action on messages until you select a new folder). Assuming the two messages in our INBOX had the \Deleted flag set then the output
looks like the following.
. expunge
* 1 EXPUNGE
* 1 EXPUNGE
* 0 EXISTS
* 0 RECENT
. OK Completed
COPY command.
IMAP has no built in move command, when you move a message you actually copy it to another folder and then delete the original. We can easily copy any number of messages using the COPY message [destination] format. Here I copy both messages from the INBOX (that I already have selected) to INBOX.test2 folder, after that I
select INBOX.test2 to confirm the messages are there. Note that after copying the RECENT flag is reset.
. copy 1:2 inbox.test2
. OK [COPYUID 1138801117 1:2 1:2] Completed
. select inbox.test2
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt \*)]
* 2 EXISTS
* 2 RECENT
* OK [UIDVALIDITY 1138801117]
* OK [UIDNEXT 3]
. OK [READ-WRITE] Completed
IDLE command.
IDLE allows us to constantly monitor a folder so that we will be instantly be notified if a new message arrives in the current folder. This is of little use
while in a telnet session but I'll show it here just so you know how it works. First start IDLE on the folder.
. idle
+ idling
The server responds with +idling and will stay this way until either a message is received, we stop idling or carry out another command to break the idle. If non
of these things happen then the connection will eventually time out after a pre-set period depending on your IMAP provider (30 minutes in my case). To stop the IDLE
command use DONE (note this is the only command WITHOUT the preceding command tag).
done
. OK Completed
LSUB, SUBSCRIBE and UNSUBSCRIBE commands.
These are more commands that only really apply to email clients since they involve subscribing to folders, however they are shown here for completeness. First LSUB
works like LIST with the same arguments but returns a list of the currently subscribed folders.
. lsub "" "*"
* LSUB (\HasChildren) "." "INBOX"
* LSUB () "." "INBOX.Drafts"
* LSUB () "." "INBOX.Sent Items"
* LSUB () "." "INBOX.Trash"
* LSUB () "." "INBOX.test2"
. OK Completed (0.000 secs 6 calls)
This shows that all folders except INBOX.test3 are currently subscribed, to subscribe a new folder use SUBSCRIBE [foldername].
. subscribe INBOX.test3
. OK Completed
To unsubscribe from this folder use UNSUBSCRIBE [foldername].
. unsubscribe INBOX.test3
. OK Completed
LOGOUT command.
Of course we need to log out of the server, we do this with the LOGOUT command.
. logout
* BYE LOGOUT received
. OK Completed
That's the main commands covered however there are a few more just 3 of which I'll mention here as they could be useful.
CAPABILITY, GETQUOTAROOT AND GETACL commands.
These 3 commands return general information on the server environment and your account information. CAPABILITY returns a long list of the mail servers option most of
which are not very exciting, the most important one listed is probably IDLE letting you know that your provider supports the IDLE command. The CHILDREN entry we saw
returned when we did a LIST command (\HasChildren or \HasNoChildren depending on whether a folder has subfolders).
. capability
* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE IDLE LOGINDISABLED
. OK Completed
GETQUOTAROOT return the amount of space you are using and the amount you have available.
. getquotaroot inbox
* QUOTAROOT inbox user.accountname
* QUOTA user.accountname (STORAGE 31306 2048000)
As you can see I'm using 31Mb but have 2Gb capacity, so plenty to spare! Finally GETACL returns the access control list, basically a list of permission you have on
your mail folders.
. getacl inbox
* ACL inbox accountname lrswipcd admin lrswipcda anyone p
. OK Completed
These letters each refer to a different permission, the letters after the user are that users rights, the full list is explained here:
* l - lookup_flag: mailbox is visible to LIST/LSUB commands
* r - read_flag: SELECT the mailbox, perform CHECK, FETCH, PARTIAL SEARCH, COPY from mailbox
* s - seen_flag: keep seen/unseen information across session
* w - write_flag: STORE flags other than SEEN and DELETED
* i - insert_flag: perform APPEND, COPY into mailbox
* p - post_flag: send mail to submission address for mailbox
* c - create_flag: CREATE new sub-mailboxes in any implementation defined hierarchy
* d - delete_flag: STORE DELETED flag perform EXPUNGE
* a - administer_flag: perform SETACL
The basics.
About IMAP.
IMAP is an email protocol for organizing, storing and retrieving emails on a remote server. It wasdeveloped after POP and is a much more advanced system, one of the main differences being that all the mail is stored on the server so it remains accessible from manydifferent locations. With POP you have to download the mail to your local computer in order to read it and therefore you cannot synchronize your mail across many different machines.It may be more complex than POP but there are still only a few core commands we need to know in order to access our mail on an IMAP server.
Before starting it's important to know a few things:
IMAP command syntax.
Before the actual command is typed into the terminal we need to type a command tag, this could be anything (without spaces) and the server will tag its response with the tag we give it. This seems to be because IMAP allows multiple connections and so multiple commands, by tagging you know which response refers to which command.
In our case we have only 1 connection and we send single commands so it's not really relevant, however we need to type something as a tag. I usually just use a period '.' but you could use a number or whatever suits you. To demonstrate the command tag see the two server responses here with the tag (don't worry about the command itself, it will be explained soon), in the first one we send '. fetch' and the second one 'a01a fetch' getting the same tag back to identify the response:
. fetch 1 fast
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013)
. OK Completed (0.000 sec)
ao1a fetch 1 fast
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013)
a01a OK Completed (0.000 sec)
Finally, the IMAP commands are not case sensitive, so 'SELECT inbox' will work just as well as 'select INBOX'. For clarity in the code I have typed the commands in uppercase and the word INBOX in uppercase also.
Mail server address.
The address of your mail server, this will usually be of the form mail.domain.com. You should look at the settings in your email client or documentation about your email account to get this information.
Security.
In this demonstration we will be sending our account username and password unencrypted over the internet, if this is a major concern to you then you should not follow this exercise.
Another alternative, if your email provider supports SSL, is to use OpenSSL (which most if not all Linux computers will have), see the 'Connecting to the host' section
below for the syntax.
Using telnet.
If you make a mistake in a telnet session you cannot use backspace to delete the entry, you may have to press enter to get an error and then re-type the command or quit and start again.
Connecting to the host.
Insecure login - login using telnet.
By insecure I just mean that your username and password are sent unencrypted over the internet so potentially could be intercepted on the route between your computer and the mail server.
First open up a terminal and type the following, of course replacing mail.myserver.com with the address of your IMAP server, note that the IMAP port used is 143:
telnet mail.myserver.com 143
This should return something like:
telnet mail.myserver.com 143
Trying 66.111.4.160...
Connected to mail.myserver.com (66.111.4.160).
Escape character is '^]'.
* OK IMAP4 ready
Secure login - login using OpenSSL.
To open an SSL session that encrypts all data sent between your computer and the mail server open a teminal and follow these steps, note that we use port 993 here:
openssl s_client -connect mail.myserver.com:993
CONNECTED(00000003)
depth=0 /C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingengine .com
i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN= Thawte Premium Server CA/emailAddress=premium-server@thawte.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDeDCCAuGgAwIBAgIDQBYSMA0GCSqGSIb3DQEBBAUAMIHOMQswCQYDVQQGEwJa
..........................
-----END CERTIFICATE-----
subject=/C=AU/ST=New South Wales/L=Crows Nest/O=Optimal Decisions Group Pty Ltd/CN=mail.messagingeng ine.com
issuer=/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/C N=Thawte Premium Server CA/emailAddress=premium-server@thawte.com
---
No client certificate CA names sent
---
SSL handshake has read 1054 bytes and written 340 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: Session ID
Session-ID-ctx:
Master-Key: Key
Key-Arg : None
Krb5 Principal: None
Start Time: 1140271254
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
* OK IMAP4 ready
Once this step is carried out the IMAP commands are identical to those for a normal telnet session.
Logging in.
Next we need to log in using the login command. Type '. login' followed by your username and password separated by spaces.
. login accountname@myserver.com *********
. OK User logged in
LIST command.
To see a list of all the mailboxes on the server we use the list command. The arguments "" "*" simply get all the mailboxes including sub folders.
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.test1"
* LIST (\HasNoChildren) "." "INBOX.test2"
. OK Completed (0.460 secs 7 calls)
We can see from this output how the mailboxes are arranged like a tree with INBOX being the 'trunk'. My IMAP provider uses a period (.) as a separator between parent and child folders so INBOX.Drafts is a child of the INBOX. The \HasChildren simply tells us that this folder has sub folders whereas the other folders do not. The way IMAP works means that all folders are created as subfolders of the INBOX even if your email client is configured not to show it that way.
STATUS command.
This command return some basic information on the folder without selecting the folder, it takes arguments depending on what information you would like returned.
Here are 3 example showing total messages, recent messages and unseen messages.
. status INBOX (messages)
* STATUS INBOX (MESSAGES 2)
. OK Completed
. status INBOX (recent)
* STATUS INBOX (RECENT 0)
. OK Completed
. status INBOX (unseen)
* STATUS INBOX (UNSEEN 0)
. OK Completed
EXAMINE and SELECT commands.
These two commands basically do the same thing, they return information on the folder chosen and then allow us to access the mails stored inside the folder. The
main difference is that EXAMINE returns a read-only reference whereas SELECT is read-write.
. examine INBOX.test2
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1138801117]
* OK [UIDNEXT 1]
. OK [READ-ONLY] Completed
. select INBOX.test2
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1138801117]
* OK [UIDNEXT 1]
. OK [READ-WRITE] Completed
Note that the only difference in response is the [READ-ONLY] and [READ-WRITE] text. Basically this command just tells us the possible IMAP flags we can set, EXIST is how many mails are in the folder, RECENT is how many recent mails there are (the SELECT command will remove the RECENT flag since the folder has now been viewed, this is not the same as the \Seen IMAP flag, also note that the EXAMINE command will not reset the RECENT flag).
The RECENT data is what tells an IMAP email client if you have new mails, by clicking on the folder the client sends the SELECT command and the new mail icon disappears even though the mails are still unread.
CREATE, DELETE and RENAME folders.
It's very easy to create and delete folders, just make sure you create them as subfolders of the INBOX. For example to create a top level folder called test3 do do the following.
. create INBOX.test3
. OK Completed
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.test1"
* LIST (\HasNoChildren) "." "INBOX.test2"
* LIST (\HasNoChildren) "." "INBOX.test3" #we created this
. OK Completed (0.420 secs 8 calls)
Conversely we can delete our new folder using the DELETE command. Note that you cannot delete a folder that had subfolders without first deleting the subfolders, also
deleting a folder containing mails will delete all the mails inside so beware!
. delete INBOX.test3
. OK Completed
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.test1"
* LIST (\HasNoChildren) "." "INBOX.test2"
. OK Completed (0.430 secs 7 calls)
Renaming a folder is just as easy, just type RENAME [current name] [new name]. This will not delete mails as they will just exist in the new folder. Here we rename
folder test1 to linux.
. rename INBOX.test1 INBOX.test3
* OK rename user.accountname.test1 user.accountname.test3
. OK Completed
. list "" "*"
* LIST (\HasChildren) "." "INBOX"
* LIST (\HasNoChildren) "." "INBOX.Drafts"
* LIST (\HasNoChildren) "." "INBOX.Sent Items"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\HasNoChildren) "." "INBOX.linux" #this was test1
* LIST (\HasNoChildren) "." "INBOX.test2"
. OK Completed (0.410 secs 7 calls)
FETCH command.
This command is the main command we use to actually access our emails. It has many possible options depending in what you wish to see, message flags, email headers,
text of the body etc. Here we select the INBOX and fetch the emails in a few different ways.
. select INBOX
* FLAGS (\Answered \Flagged \Draft \DeleteCLOSE and EXPUNGE commands.d \Seen hasatt)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt \*)]
* 2 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1138801043]
* OK [UIDNEXT 3]
. OK [READ-WRITE] Completed
. fetch 1 flags
* 1 FETCH (FLAGS (\Seen hasatt))
. OK Completed (0.000 sec)
First we shall fetch the message IMAP flags for all the messages in the folder.
. fetch 1:2 flags
* 1 FETCH (FLAGS (\Seen hasatt))
* 2 FETCH (FLAGS (\Seen hasatt))
. OK Completed
Note that with all the commands that act upon messages we can select either 1 message by using the message number as in 'fetch 1 command' or we can select a range
of messages in the format 'fetch first:last command' or all the messages 'fetch 1:last command'. Also note that we can use '*' to indicate all messages so fetch 1:* will get all the messages from the first to the last without us knowing how many messages are in the folder.
First we shall fetch using fast, all and full options (these refer to the headers).
. fetch 1 fast
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013)
. OK Completed (0.000 sec)
. fetch 1 all
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013 ENVELOPE ("Wed, 1 Feb 2006 13:37:19 UT" "IMPORTANT: Click here to begin using your account" (("Email Administrator" NIL "bounce" "myserver.com")) (("Email Administrator" NIL "bounce" "myserver.com")) ((NIL NIL "webmaster" "myserver.com")) (("Joe Bloggs" NIL "accountname" "myserver.com")) NIL NIL NIL "
. OK Completed (0.000 sec)
. fetch 1 full
* 1 FETCH (FLAGS (\Seen hasatt) INTERNALDATE " 1-Feb-2006 08:37:23 -0500" RFC822.SIZE 15013 ENVELOPE ("Wed, 1 Feb 2006 13:37:19 UT" "IMPORTANT: Click here to begin using your account" (("Email Administrator" NIL "bounce" "myserver.com")) (("Email Administrator" NIL "bounce" "myserver.com")) ((NIL NIL "webmaster" "myserver.com")) (("Joe Bloggs" NIL "accountname" "myserver.com")) NIL NIL NIL "
. OK Completed (0.000 sec)
As you can see this returns differing amounts of data about the IMAP flags, size and ENVELOPE information. It's maybe more informative to use either 'fetch message body[header]' or 'fetch message rfc822.header' both of which return the data below.
. fetch 1 rfc822.header
* 1 FETCH (RFC822.HEADER {824}
Return-Path:
Received: from web2.internal (web2.internal [10.202.2.211])
by server2.messagingengine.com (Cyrus v2.3-alpha) with LMTPA;
Wed, 01 Feb 2006 08:37:23 -0500
X-Sieve: CMU Sieve 2.3
X-Attached: This_is_how_attachments_appear.txt
X-Resolved-to: accountname
X-Mail-from: nobody
Content-Transfer-Encoding: 8bit
Content-Type: multipart/mixed; boundary="_----------=_1138801039165120"
MIME-Version: 1.0
X-Mailer: MIME::Lite 5022 (F2.73; T1.15; A1.64; B3.05; Q3.03)
Date: Wed, 1 Feb 2006 13:37:19 UT
From: "Email Administrator"
Reply-To: webmaster@myserver.com
To: "Joe Bloggs"
Subject: IMPORTANT: Click here to begin using your account
Message-ID:
)
. OK Completed (0.000 sec)
To fetch only some headers we can select the header fields we wish to see.
. fetch 1 (body[header.fields (from to subject date)])
* 1 FETCH (BODY[HEADER.FIELDS (from to subject date)] {195}
Date: Wed, 1 Feb 2006 13:37:19 UT
From: "Email Administrator"
To: "Joe Bloggs"
Subject: IMPORTANT: Click here to begin using your account
)
. OK Completed (0.000 sec)
To read the body of the email message we can use either 'fetch message body[text]' or 'fetch message rfc822.text' as shown here.
. fetch 2 rfc822.text
* 2 FETCH (RFC822.TEXT {11658}
This is a multi-part message in MIME format.
--_----------=_1138865560223950
Content-Disposition: inline
Content-Length: 5194
Content-Transfer-Encoding: binary
Content-Type: text/plain
more text here.............
. OK Completed (0.000 sec)
STORE command.
This command allows us to add, remove or replace the IMAP flags on the messages. These are flags that denote a message as replied to, deleted, seen etc. and allow
the message information, as well as the message itself, to be synchronized across different computers. Note that the STORE command causes an automatic FETCH
command of the message flags so we can see the change immediately. There are 3 ways to use STORE:
* STORE message +flags [flag list] - this adds the [flag list] flags to the chosen messages.
* STORE message -flags [flag list] - this removes the [flag list] flags from the chosen messages.
* STORE message flags [flag list] - resets the flags to [flag list] on the chosen messages (the same as removing all flags and then adding [flag list].
The list of flags to add include \Answered \Flagged \Draft \Deleted \Seen and many more. All the IMAP flags used as part of the standard installation have the
backslash in front of them. However some email clients (Thunderbird is one) also allow you to set labels or mark a message as junk, if you add labels do not use
the backslash. First we shall mark all the messages as deleted.
. store 1:2 flags \Deleted
* 1 FETCH (FLAGS (\Recent \Deleted))
* 2 FETCH (FLAGS (\Recent \Deleted))
. OK Completed
Next replace the flags with $label1
. store 1:* flags $label1
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt Junk label1)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt Junk $label1 \*)]
* 1 FETCH (FLAGS ($label1))
* 2 FETCH (FLAGS ($label1))
. OK Completed
Finally we can add the flag NonJunk so that Thunderbird recognises them as not being junk mail.
. store 1:* +flags NonJunk
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt NonJunk Junk label1)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt NonJunk Junk $label1 \*)]
* 1 FETCH (FLAGS ($label1 NonJunk))
* 2 FETCH (FLAGS ($label1 NonJunk))
. OK Completed
Note that the \Deleted flag is used by an IMAP server to mark an email ready for deletion, it is not actually deleted until the server receives either the CLOSE
or EXPUNGE command shown below.
CLOSE and EXPUNGE commands.
Both these commands have the effect of permanently deleting any messages in the current folder marked for deletion with the \Deleted flag. EXPUNGE just deletes the
messages but does nothing else (this command is the equivalent of compacting folders in Thunderbird), while CLOSE deletes the messages and deselects the current
folder (you cannot carry out more action on messages until you select a new folder). Assuming the two messages in our INBOX had the \Deleted flag set then the output
looks like the following.
. expunge
* 1 EXPUNGE
* 1 EXPUNGE
* 0 EXISTS
* 0 RECENT
. OK Completed
COPY command.
IMAP has no built in move command, when you move a message you actually copy it to another folder and then delete the original. We can easily copy any number of messages using the COPY message [destination] format. Here I copy both messages from the INBOX (that I already have selected) to INBOX.test2 folder, after that I
select INBOX.test2 to confirm the messages are there. Note that after copying the RECENT flag is reset.
. copy 1:2 inbox.test2
. OK [COPYUID 1138801117 1:2 1:2] Completed
. select inbox.test2
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen hasatt \*)]
* 2 EXISTS
* 2 RECENT
* OK [UIDVALIDITY 1138801117]
* OK [UIDNEXT 3]
. OK [READ-WRITE] Completed
IDLE command.
IDLE allows us to constantly monitor a folder so that we will be instantly be notified if a new message arrives in the current folder. This is of little use
while in a telnet session but I'll show it here just so you know how it works. First start IDLE on the folder.
. idle
+ idling
The server responds with +idling and will stay this way until either a message is received, we stop idling or carry out another command to break the idle. If non
of these things happen then the connection will eventually time out after a pre-set period depending on your IMAP provider (30 minutes in my case). To stop the IDLE
command use DONE (note this is the only command WITHOUT the preceding command tag).
done
. OK Completed
LSUB, SUBSCRIBE and UNSUBSCRIBE commands.
These are more commands that only really apply to email clients since they involve subscribing to folders, however they are shown here for completeness. First LSUB
works like LIST with the same arguments but returns a list of the currently subscribed folders.
. lsub "" "*"
* LSUB (\HasChildren) "." "INBOX"
* LSUB () "." "INBOX.Drafts"
* LSUB () "." "INBOX.Sent Items"
* LSUB () "." "INBOX.Trash"
* LSUB () "." "INBOX.test2"
. OK Completed (0.000 secs 6 calls)
This shows that all folders except INBOX.test3 are currently subscribed, to subscribe a new folder use SUBSCRIBE [foldername].
. subscribe INBOX.test3
. OK Completed
To unsubscribe from this folder use UNSUBSCRIBE [foldername].
. unsubscribe INBOX.test3
. OK Completed
LOGOUT command.
Of course we need to log out of the server, we do this with the LOGOUT command.
. logout
* BYE LOGOUT received
. OK Completed
That's the main commands covered however there are a few more just 3 of which I'll mention here as they could be useful.
CAPABILITY, GETQUOTAROOT AND GETACL commands.
These 3 commands return general information on the server environment and your account information. CAPABILITY returns a long list of the mail servers option most of
which are not very exciting, the most important one listed is probably IDLE letting you know that your provider supports the IDLE command. The CHILDREN entry we saw
returned when we did a LIST command (\HasChildren or \HasNoChildren depending on whether a folder has subfolders).
. capability
* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE IDLE LOGINDISABLED
. OK Completed
GETQUOTAROOT return the amount of space you are using and the amount you have available.
. getquotaroot inbox
* QUOTAROOT inbox user.accountname
* QUOTA user.accountname (STORAGE 31306 2048000)
As you can see I'm using 31Mb but have 2Gb capacity, so plenty to spare! Finally GETACL returns the access control list, basically a list of permission you have on
your mail folders.
. getacl inbox
* ACL inbox accountname lrswipcd admin lrswipcda anyone p
. OK Completed
These letters each refer to a different permission, the letters after the user are that users rights, the full list is explained here:
* l - lookup_flag: mailbox is visible to LIST/LSUB commands
* r - read_flag: SELECT the mailbox, perform CHECK, FETCH, PARTIAL SEARCH, COPY from mailbox
* s - seen_flag: keep seen/unseen information across session
* w - write_flag: STORE flags other than SEEN and DELETED
* i - insert_flag: perform APPEND, COPY into mailbox
* p - post_flag: send mail to submission address for mailbox
* c - create_flag: CREATE new sub-mailboxes in any implementation defined hierarchy
* d - delete_flag: STORE DELETED flag perform EXPUNGE
* a - administer_flag: perform SETACL
通过TELNET发送邮件
以下是一次典型的发送邮件过程，粗体部分是手工输入的
220 mx1.hudong.com ESMTP Postfix
EHLO localhost
250-mx1.hudong.com
250-PIPELINING
250-SIZE 204800000
250-VRFY
250-ETRN
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH PLAIN sldfkZedEwxddLSXVFDssW=
235 2.0.0 Authentication successful
MAIL FROM:
250 2.1.0 Ok
RCPT TO:
250 2.1.5 Ok
DATA
354 End data with.
mail test.
.
250 2.0.0 Ok: queued as E78388824003
QUIT
221 2.0.0 Bye
Connection closed by foreign host.
AUTH PLAIN 后面接用户ID和密码的base64编码，可用如下命令产生
$ printf 'kdent\0kdent\0Rumpelstiltskin' | mmencode
a2RlbnQAa2RlbnQAcnVtcGxlc3RpbHRza2lu
其中用户名是kdent，密码是Rumpelstiltskin
也可用如下perl脚本生成
#!/usr/bin/perl
use strict;
use MIME::Base64;
if ( $#ARGV != 1 ) {
die "Usage: encode_sasl_plain.pl \n";
}
print encode_base64("$ARGV[0]\0$ARGV[0]\0$ARGV[1]");
exit 0;
DATA表示要发送邮件消息，邮件消息以 回车换行加点再回车换行 结尾。
220 mx1.hudong.com ESMTP Postfix
EHLO localhost
250-mx1.hudong.com
250-PIPELINING
250-SIZE 204800000
250-VRFY
250-ETRN
250-AUTH LOGIN PLAIN
250-AUTH=LOGIN PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
AUTH PLAIN sldfkZedEwxddLSXVFDssW=
235 2.0.0 Authentication successful
MAIL FROM:
250 2.1.0 Ok
RCPT TO:
250 2.1.5 Ok
DATA
354 End data with
mail test.
.
250 2.0.0 Ok: queued as E78388824003
QUIT
221 2.0.0 Bye
Connection closed by foreign host.
AUTH PLAIN 后面接用户ID和密码的base64编码，可用如下命令产生
$ printf 'kdent\0kdent\0Rumpelstiltskin' | mmencode
a2RlbnQAa2RlbnQAcnVtcGxlc3RpbHRza2lu
其中用户名是kdent，密码是Rumpelstiltskin
也可用如下perl脚本生成
#!/usr/bin/perl
use strict;
use MIME::Base64;
if ( $#ARGV != 1 ) {
die "Usage: encode_sasl_plain.pl
}
print encode_base64("$ARGV[0]\0$ARGV[0]\0$ARGV[1]");
exit 0;
DATA表示要发送邮件消息，邮件消息以 回车换行加点再回车换行 结尾。
EMAIL系统配置花絮
本文配置环境为 CentOS 5.4 for x86_64，用户帐号信息存储在OpenLDAP数据库，MTA为Postfix，MDA为Cyrus Imapd，认证通过saslauthd服务，因为内外网的关系使用dnsmasq做域名分割(domain split)，为支持内网专用域解析使用了bind。
1 OpenLDAP
1.1 为安全起见会使用 TLS，但因为证书的问题经常导致认证失败，解决办法是修改 /etc/openldap/ldap.conf里的如下配置项
TLS_REQCERT allow
1.2 有些帐号希望能修改LDAP数据库，但只允许特定的ip地址来源，可在 slapd.conf做如下配置
access to dn.base="cn=Subschema" by * read
access to *
by self write
by dn.exact="uid=auth,ou=sysusers,dc=intra,dc=example,dc=com" peername.regex=127\.0\.0\.1 write
by dn.subtree="ou=sysusers,dc=intra,dc=example,dc=com" read
by anonymous auth
1.3 为提供LDAP服务的可用性，通常会配置复制(replication)，以下是主端(master) slapd.conf相关配置
# For replication
index entryCSN,entryUUID eq,pres
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
limits dn.subtree="ou=sysusers,dc=intra,dc=example,dc=com" size=unlimited time=unlimited
从端(slave)slapd.conf的相关配置
index entryCSN,entryUUID eq,pres
syncrepl rid=101
provider=ldap://172.16.35.11:389
type=refreshAndPersist
retry="60 +"
searchbase="dc=intra,dc=example,dc=com"
filter="(objectclass=*)"
attrs="*,+"
scope=sub
schemachecking=off
updatedn="cn=Manager,dc=intra,dc=example,dc=com"
bindmethod=simple
binddn="uid=lsync,ou=sysusers,dc=intra,dc=example,dc=com"
credentials=111111
为使用replication功能需安装rpm包 openldap-servers-overlays
1.4 为调试ldap交互信息，可打开日志，按如下配置slapd.conf即可
# 为让配置生效需配置syslog
# local4.* /var/log/ldap.log
#
#loglevel Logging description
#-1 enable all debugging
#0 no debugging
#1 trace function calls
#2 debug packet handling
#4 heavy trace debugging
#8 connection management
#16 print out packets sent and received
#32 search filter processing
#64 configuration file processing
#128 access control list processing
#256 stats log connections/operations/results
#512 stats log entries sent
#1024 print communication with shell backends
#2048 print entry parsing debugging
loglevel 928
1.5 初始化ldap数据库。编辑如下文件 base.ldif
dn: dc=intra,dc=example,dc=com
dc: intra
objectClass: domain
启动ldap服务后，用ldapadd命令导入。
2 Postfix
2.1 配置从ldap数据库获取收信用户，可如下配置main.cf
local_recipient_maps = ldap:ldapintra $alias_maps
ldapintra_server_host = 172.16.35.10
ldapintra_search_base = ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapintra_bind_dn = uid=mail,ou=sysusers,dc=intra,dc=example,dc=com
ldapintra_bind_pw = 111111
ldapintra_start_tls = yes
ldapintra_version = 3
#ldapintra_scope = sub
ldapintra_query_filter = (&(mail=%s)(employeeType=staff))
ldapintra_result_attribute = uid
2.2 配置smtp认证，可如下配置main.cf
smtpd_sasl_auth_enable = yes
smtpd_sender_login_maps = pcre:/etc/postfix/sender_login
smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_destination
reject_sender_login_mismatch
broken_sasl_auth_clients = yes
/usr/lib64/sasl2/smtp.conf内容如下
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN
ldapdb_uri: ldap://172.16.35.10/
ldapdb_version: 3
ldapdb_start_tls: 0
ldapdb_bind_dn: cn=mail,ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapdb_bind_pw: 111111
ldapdb_search_base: ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapdb_filter: uid=%U
ldapdb_password_attr: userPassword
ldapdb_sasl: 0
/etc/postfix/sender_login内容如下
/^(.*)@example.com/ $1
通过sender_login配置我们可确保本域用户发给本域用户的信件是不可伪造的(我们不希望有人冒充CEO发给all邮件列表不该发的信 ^_^)。
2.3 配置备用邮件服务器。首先设置DNS服务器的MX记录，权值数高（优先级低）的为备用邮件服务器。备用邮件服务器的main.cf相关选项如下
relay_domains = example.com
transport_maps = hash:/etc/postfix/transport
relay_recipient_maps = ldap:ldapintra $alias_maps
ldapintra_server_host = 172.16.35.10
ldapintra_search_base = ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapintra_bind_dn = uid=mail,ou=sysusers,dc=intra,dc=example,dc=com
ldapintra_bind_pw = 111111
ldapintra_start_tls = yes
ldapintra_version = 3
#ldapintra_scope = sub
ldapintra_query_filter = (&(mail=%s)(employeeType=staff))
ldapintra_result_attribute = uid
/etc/postfix/transport内容如下
example.com relay:[mx1.example.com]
以上配置会把收到的邮件转发到mx1.example.com，注意relay_domains参数指定的域不能出现在mydestination配置项里。
2.4 要配置邮件域改写，比如把发给 test.com域的邮件改写成 example.com域，可如下配置main.cf。
recipient_canonical_maps = pcre:/etc/postfix/domain_rewrite
/etc/postfix/domain_rewrite内容如下
/^(.*)@test\.com/ $1@example.com
2.5 配置把收到的信发给 cyrus imapd，可如下配置main.cf
mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp
以上unix套接口会被cyrus imapd创建。
2.6 相关工具。
postalias 修改别名文件后，运行它重建别名数据库
postmap 修改查询表后，运行它重建查询表，也可用它做查询表排错
postqueue 显示或flush邮件队列
postsuper 邮件队列管理程序
postcat 查看队列里的邮件内容
2.7 性能优化。参考 http://wiki.list.org/display/DOC/MTA+Performance+Tuning+Tips+for+Postfix 的一篇文章，相关main.cf参数为
default_destination_concurrency_limit=50 default_destination_recipient_limit=50 default_process_limit=200 smtp_mx_session_limit=100 smtpd_client_connection_count_limit=100 smtp_destination_concurrency_limit=100 maximal_backoff_time = 1000s minimal_backoff_time = 300s
3 Cyrus Imapd
3.1 /etc/imapd.conf内容如下
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: cyrus leo
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
fulldirhash: true
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN
sasl_saslauthd_path: /var/run/saslauthd/mux
tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
virtualdomains: yes
defaultdomain: example.com
lmtp_downcase_rcpt: 1
munge8bit: no
allowusermoves: 1
allowplaintext: 1
allowplainwithouttls: 1
autocreatequota: 2048000
unixhierarchysep: no
quotawarn: 90
createonpost: 1
autocreateinboxfolders: Drafts|Sent|Trash
autosubscribeinboxfolders: Drafts|Sent|Trash
lmtp_over_quota_perm_failure: 1
3.2 配置saslauthd，/etc/saslauthd.conf内容如下
ldap_servers: ldap://172.16.35.10/
ldap_bind_dn: uid=mail,ou=sysusers,dc=intra,dc=example,dc=com
ldap_bind_pw: 111111
ldap_search_base: ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldap_filter: uid=%u
ldap_password_attr: userPassword
/etc/sysconfig/saslauthd里配置项
MECH=ldap
3.3 要对pop/imap做分布，可用cyrus imapd murder，cyrus-imapd包已具有相关程序和文档。这个话题比较复杂，相关内容参考所附链接。
3.4 要删除邮件，比如只留7天的邮件，可用如下命令
su - cyrus -c '/usr/lib/cyrus-imapd/ipurge -f -d 7 user.myuser
参考 http://comments.gmane.org/gmane.mail.imap.cyrus/21988
3.4 性能调优。
安装cyrus-imapd后会附有调优文档参考，主要瓶颈是磁盘IO，可考虑多购买几块磁盘分布负载。
pop协议会创建锁文件，可把锁文档目录放到内存文件系统(tmpfs)，一般64M的内存文件系统足够使用。
当出现DBERROR db4: Logging region out of memory错误时，应该是因为cyrus-imapd依赖的Berkeley DB默认内存参数太小。可编辑 /var/lib/imap/db/DB_CONFIG，内容如下（根据需要调整）：
set_cachesize 0 2097152 1
set_lg_regionmax 1048576
停掉cyrus-imapd服务，运行如下命令重建数据库文件
# db_recover -h /var/lib/imap/db
重建后注意把/var/lib/imap/db目录下的文件所有者改成cyrus，组改成mail。启动cyrus-imapd服务，运行如下命令查看修改是否生效
# db_stat -m -h /var/lib/imap/db
# db_stat -l -h /var/lib/imap/db
细节参考 http://www.cjc.org/blog/archives/2006/08/22/cyrus-imap-log-and-cache-settings/ 。
Cyrus imapd遇到的另一个常见问题是报错Mailbox is locked by POP server，结果是邮件客户端反复要求用户输入密码。问题产生于如下方面：
同时有多个pop客户端收信，解决办法是把客户端收信方式由pop改成imap
连接非正常中断，往往发生在物理网络切换时，比如正在收信突然拔掉有线网换成无线，再次收信就会被锁，默认超时时间为10分钟(RFC规范要求)。不幸的是这种情况无法避免，可以从锁文件目录里（/var/lib/imap/proc/）找到被锁的用户，手工杀掉服务器端的pop3d进程，然后删除锁文件。另一种解决方法是在/etc/imapd.conf里把参数 poptimeout设短，但cyrus-imapd要求超时时间不能低于10分钟。如果你被这个问题困扰，可考虑把cyrus-imapd的源码找到，修改imap/pop3d.c的代码，找到登录后检查锁的位置，把之前已经登录的用户注销掉，然后重新尝试锁的检测。要生成rpm包可能还需安装 redhat-rpm-config 包
cyrus-imapd的性能遇到瓶颈，锁文件来不及释放，可参考安装包里的优化文档进行优化
4 DNS
4.1 要配置bind主从复制，且监听在定制的端口，比如153，可如下配置主服务的named.conf
acl nsservers { 172.16.35.10; 172.16.35.11; };
options {
…
allow-notify { nsservers; 127.0.0.1; };
allow-transfer { nsservers; 127.0.0.1; };
also-notify { 172.16.35.11 port 153; };
listen-on port 153 { any; };
…
};
从服务配置如下
acl nsservers { 172.16.35.10; 172.16.35.11; };
options {
…
allow-notify { nsservers; 127.0.0.1; };
allow-transfer { nsservers; 127.0.0.1; };
listen-on port 153 { any; };
…
};
view ... {
…
zone … {
…
masters port 153 { 172.16.35.10; };
...
};
...
};
每次修改zone数据后，必须增加序号，否则同步无法进行。
5 相关资料
http://dns-learning.twnic.net.tw/bind/toc.html
http://www.linuxmail.info/openldap-setup-howto/
http://tldp.org/HOWTO/Cyrus-IMAP.html
http://www.akadia.com/services/ssh_test_certificate.html
http://blogwords.neologix.net/neils/?p=140 OpenLDAP同步
http://oreilly.com/catalog/mimap/chapter/ch09.html 管理IMAP
http://www.linuxmail.info/smtp-authentication-postfix-centos-5/
http://www.postfix.org/documentation.html
http://www.yolinux.com/TUTORIALS/LinuxTutorialMailman.html
http://onlamp.com/pub/a/onlamp/2005/10/06/cyrus_imap.html?page=1
http://wpkg.org/Integrating_Mailman_with_a_Swish-e_search_engine
http://www.linuxjournal.com/article/9804 建立可扩展的邮件系统
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Design
http://download.oracle.com/docs/cd/E17076_02/html/api_reference/C/frame_main.html Berkeley DB配置参数说明
1 OpenLDAP
1.1 为安全起见会使用 TLS，但因为证书的问题经常导致认证失败，解决办法是修改 /etc/openldap/ldap.conf里的如下配置项
TLS_REQCERT allow
1.2 有些帐号希望能修改LDAP数据库，但只允许特定的ip地址来源，可在 slapd.conf做如下配置
access to dn.base="cn=Subschema" by * read
access to *
by self write
by dn.exact="uid=auth,ou=sysusers,dc=intra,dc=example,dc=com" peername.regex=127\.0\.0\.1 write
by dn.subtree="ou=sysusers,dc=intra,dc=example,dc=com" read
by anonymous auth
1.3 为提供LDAP服务的可用性，通常会配置复制(replication)，以下是主端(master) slapd.conf相关配置
# For replication
index entryCSN,entryUUID eq,pres
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
limits dn.subtree="ou=sysusers,dc=intra,dc=example,dc=com" size=unlimited time=unlimited
从端(slave)slapd.conf的相关配置
index entryCSN,entryUUID eq,pres
syncrepl rid=101
provider=ldap://172.16.35.11:389
type=refreshAndPersist
retry="60 +"
searchbase="dc=intra,dc=example,dc=com"
filter="(objectclass=*)"
attrs="*,+"
scope=sub
schemachecking=off
updatedn="cn=Manager,dc=intra,dc=example,dc=com"
bindmethod=simple
binddn="uid=lsync,ou=sysusers,dc=intra,dc=example,dc=com"
credentials=111111
为使用replication功能需安装rpm包 openldap-servers-overlays
1.4 为调试ldap交互信息，可打开日志，按如下配置slapd.conf即可
# 为让配置生效需配置syslog
# local4.* /var/log/ldap.log
#
#loglevel Logging description
#-1 enable all debugging
#0 no debugging
#1 trace function calls
#2 debug packet handling
#4 heavy trace debugging
#8 connection management
#16 print out packets sent and received
#32 search filter processing
#64 configuration file processing
#128 access control list processing
#256 stats log connections/operations/results
#512 stats log entries sent
#1024 print communication with shell backends
#2048 print entry parsing debugging
loglevel 928
1.5 初始化ldap数据库。编辑如下文件 base.ldif
dn: dc=intra,dc=example,dc=com
dc: intra
objectClass: domain
启动ldap服务后，用ldapadd命令导入。
2 Postfix
2.1 配置从ldap数据库获取收信用户，可如下配置main.cf
local_recipient_maps = ldap:ldapintra $alias_maps
ldapintra_server_host = 172.16.35.10
ldapintra_search_base = ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapintra_bind_dn = uid=mail,ou=sysusers,dc=intra,dc=example,dc=com
ldapintra_bind_pw = 111111
ldapintra_start_tls = yes
ldapintra_version = 3
#ldapintra_scope = sub
ldapintra_query_filter = (&(mail=%s)(employeeType=staff))
ldapintra_result_attribute = uid
2.2 配置smtp认证，可如下配置main.cf
smtpd_sasl_auth_enable = yes
smtpd_sender_login_maps = pcre:/etc/postfix/sender_login
smtpd_recipient_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_destination
reject_sender_login_mismatch
broken_sasl_auth_clients = yes
/usr/lib64/sasl2/smtp.conf内容如下
pwcheck_method: saslauthd
mech_list: PLAIN LOGIN
ldapdb_uri: ldap://172.16.35.10/
ldapdb_version: 3
ldapdb_start_tls: 0
ldapdb_bind_dn: cn=mail,ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapdb_bind_pw: 111111
ldapdb_search_base: ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapdb_filter: uid=%U
ldapdb_password_attr: userPassword
ldapdb_sasl: 0
/etc/postfix/sender_login内容如下
/^(.*)@example.com/ $1
通过sender_login配置我们可确保本域用户发给本域用户的信件是不可伪造的(我们不希望有人冒充CEO发给all邮件列表不该发的信 ^_^)。
2.3 配置备用邮件服务器。首先设置DNS服务器的MX记录，权值数高（优先级低）的为备用邮件服务器。备用邮件服务器的main.cf相关选项如下
relay_domains = example.com
transport_maps = hash:/etc/postfix/transport
relay_recipient_maps = ldap:ldapintra $alias_maps
ldapintra_server_host = 172.16.35.10
ldapintra_search_base = ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldapintra_bind_dn = uid=mail,ou=sysusers,dc=intra,dc=example,dc=com
ldapintra_bind_pw = 111111
ldapintra_start_tls = yes
ldapintra_version = 3
#ldapintra_scope = sub
ldapintra_query_filter = (&(mail=%s)(employeeType=staff))
ldapintra_result_attribute = uid
/etc/postfix/transport内容如下
example.com relay:[mx1.example.com]
以上配置会把收到的邮件转发到mx1.example.com，注意relay_domains参数指定的域不能出现在mydestination配置项里。
2.4 要配置邮件域改写，比如把发给 test.com域的邮件改写成 example.com域，可如下配置main.cf。
recipient_canonical_maps = pcre:/etc/postfix/domain_rewrite
/etc/postfix/domain_rewrite内容如下
/^(.*)@test\.com/ $1@example.com
2.5 配置把收到的信发给 cyrus imapd，可如下配置main.cf
mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp
以上unix套接口会被cyrus imapd创建。
2.6 相关工具。
postalias 修改别名文件后，运行它重建别名数据库
postmap 修改查询表后，运行它重建查询表，也可用它做查询表排错
postqueue 显示或flush邮件队列
postsuper 邮件队列管理程序
postcat 查看队列里的邮件内容
2.7 性能优化。参考 http://wiki.list.org/display/DOC/MTA+Performance+Tuning+Tips+for+Postfix 的一篇文章，相关main.cf参数为
default_destination_concurrency_limit=50 default_destination_recipient_limit=50 default_process_limit=200 smtp_mx_session_limit=100 smtpd_client_connection_count_limit=100 smtp_destination_concurrency_limit=100 maximal_backoff_time = 1000s minimal_backoff_time = 300s
3 Cyrus Imapd
3.1 /etc/imapd.conf内容如下
configdirectory: /var/lib/imap
partition-default: /var/spool/imap
admins: cyrus leo
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
hashimapspool: true
fulldirhash: true
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN
sasl_saslauthd_path: /var/run/saslauthd/mux
tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt
virtualdomains: yes
defaultdomain: example.com
lmtp_downcase_rcpt: 1
munge8bit: no
allowusermoves: 1
allowplaintext: 1
allowplainwithouttls: 1
autocreatequota: 2048000
unixhierarchysep: no
quotawarn: 90
createonpost: 1
autocreateinboxfolders: Drafts|Sent|Trash
autosubscribeinboxfolders: Drafts|Sent|Trash
lmtp_over_quota_perm_failure: 1
3.2 配置saslauthd，/etc/saslauthd.conf内容如下
ldap_servers: ldap://172.16.35.10/
ldap_bind_dn: uid=mail,ou=sysusers,dc=intra,dc=example,dc=com
ldap_bind_pw: 111111
ldap_search_base: ou=users,ou=accounts,dc=intra,dc=example,dc=com
ldap_filter: uid=%u
ldap_password_attr: userPassword
/etc/sysconfig/saslauthd里配置项
MECH=ldap
3.3 要对pop/imap做分布，可用cyrus imapd murder，cyrus-imapd包已具有相关程序和文档。这个话题比较复杂，相关内容参考所附链接。
3.4 要删除邮件，比如只留7天的邮件，可用如下命令
su - cyrus -c '/usr/lib/cyrus-imapd/ipurge -f -d 7 user.myuser
参考 http://comments.gmane.org/gmane.mail.imap.cyrus/21988
3.4 性能调优。
安装cyrus-imapd后会附有调优文档参考，主要瓶颈是磁盘IO，可考虑多购买几块磁盘分布负载。
pop协议会创建锁文件，可把锁文档目录放到内存文件系统(tmpfs)，一般64M的内存文件系统足够使用。
当出现DBERROR db4: Logging region out of memory错误时，应该是因为cyrus-imapd依赖的Berkeley DB默认内存参数太小。可编辑 /var/lib/imap/db/DB_CONFIG，内容如下（根据需要调整）：
set_cachesize 0 2097152 1
set_lg_regionmax 1048576
停掉cyrus-imapd服务，运行如下命令重建数据库文件
# db_recover -h /var/lib/imap/db
重建后注意把/var/lib/imap/db目录下的文件所有者改成cyrus，组改成mail。启动cyrus-imapd服务，运行如下命令查看修改是否生效
# db_stat -m -h /var/lib/imap/db
# db_stat -l -h /var/lib/imap/db
细节参考 http://www.cjc.org/blog/archives/2006/08/22/cyrus-imap-log-and-cache-settings/ 。
Cyrus imapd遇到的另一个常见问题是报错Mailbox is locked by POP server，结果是邮件客户端反复要求用户输入密码。问题产生于如下方面：
同时有多个pop客户端收信，解决办法是把客户端收信方式由pop改成imap
连接非正常中断，往往发生在物理网络切换时，比如正在收信突然拔掉有线网换成无线，再次收信就会被锁，默认超时时间为10分钟(RFC规范要求)。不幸的是这种情况无法避免，可以从锁文件目录里（/var/lib/imap/proc/）找到被锁的用户，手工杀掉服务器端的pop3d进程，然后删除锁文件。另一种解决方法是在/etc/imapd.conf里把参数 poptimeout设短，但cyrus-imapd要求超时时间不能低于10分钟。如果你被这个问题困扰，可考虑把cyrus-imapd的源码找到，修改imap/pop3d.c的代码，找到登录后检查锁的位置，把之前已经登录的用户注销掉，然后重新尝试锁的检测。要生成rpm包可能还需安装 redhat-rpm-config 包
cyrus-imapd的性能遇到瓶颈，锁文件来不及释放，可参考安装包里的优化文档进行优化
4 DNS
4.1 要配置bind主从复制，且监听在定制的端口，比如153，可如下配置主服务的named.conf
acl nsservers { 172.16.35.10; 172.16.35.11; };
options {
…
allow-notify { nsservers; 127.0.0.1; };
allow-transfer { nsservers; 127.0.0.1; };
also-notify { 172.16.35.11 port 153; };
listen-on port 153 { any; };
…
};
从服务配置如下
acl nsservers { 172.16.35.10; 172.16.35.11; };
options {
…
allow-notify { nsservers; 127.0.0.1; };
allow-transfer { nsservers; 127.0.0.1; };
listen-on port 153 { any; };
…
};
view ... {
…
zone … {
…
masters port 153 { 172.16.35.10; };
...
};
...
};
每次修改zone数据后，必须增加序号，否则同步无法进行。
5 相关资料
http://dns-learning.twnic.net.tw/bind/toc.html
http://www.linuxmail.info/openldap-setup-howto/
http://tldp.org/HOWTO/Cyrus-IMAP.html
http://www.akadia.com/services/ssh_test_certificate.html
http://blogwords.neologix.net/neils/?p=140 OpenLDAP同步
http://oreilly.com/catalog/mimap/chapter/ch09.html 管理IMAP
http://www.linuxmail.info/smtp-authentication-postfix-centos-5/
http://www.postfix.org/documentation.html
http://www.yolinux.com/TUTORIALS/LinuxTutorialMailman.html
http://onlamp.com/pub/a/onlamp/2005/10/06/cyrus_imap.html?page=1
http://wpkg.org/Integrating_Mailman_with_a_Swish-e_search_engine
http://www.linuxjournal.com/article/9804 建立可扩展的邮件系统
http://www.cyrusimap.org/mediawiki/index.php/Cyrus_Murder_Design
http://download.oracle.com/docs/cd/E17076_02/html/api_reference/C/frame_main.html Berkeley DB配置参数说明
TCP Keepalive HOWTO
原文参考 http://www.tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/
TCP连接有很多定时器，其中一些与Keepalive有关。当这些定时器计数为0时，就会发送Keepalive探帧包（我记得在最早的BSD代码上该探帧包占一个序列号Sequence）。如果这些包没有得到对端的ACK回应，则可采取一定的控制策略，比如继续发送探帧包或认为对端已非正常中断连接。
Keepalive的用途主要有两个：检查对端是否死掉或者防止因网络无活动而连接中断。假设A和B建立了一个TCP连接，B突然死掉了，A会在一定时间后发送Keepalive探帧包，如果B重启后收到了探帧包，因TCP套接字对不上或TCP序列号对不上，会导致B给A发送RST包，此时连接马上中断。如果B一直没活过来，则发送一定数量的探帧包后A认为连接中断。这是有效检查对端死掉的情况，还有一种情况是通过NAT或代理上网，如果A和B之间隔着NAT或代理，这些NAT或者代理通常有一定的连接超时控制，发现A和B在一段时间后没有交换数据，就会切断二者的通信，此时用Keepalive可避免这种情况。
Linux下控制Keepalive的参数有三个：
tcp_keepalive_time 在连接多长时间没有数据交换时发送探帧包
tcp_keepalive_intvl 发送下一个探帧包距离发送本次探帧包的时间间隔
tcp_keepalive_probes 发送多少次没有回应的探帧包后认为对端死掉
对应在proc文件系统的位置是
/proc/sys/net/ipv4/tcp_keepalive_time
/proc/sys/net/ipv4/tcp_keepalive_intvl
/proc/sys/net/ipv4/tcp_keepalive_probes
注意：仅仅配置内核参数是不够的，还必须在编程的时候设置套接字的选项，调用函数是
int setsockopt(int s, int level, int optname,
const void *optval, socklen_t optlen)
选项为SO_KEEPALIVE，此外还可以设置TCP选项（设置的level为SOL_TCP而不是SOL_SOCKET）覆盖系统全局设置
TCP_KEEPCNT 与tcp_keepalive_probes对应
TCP_KEEPIDLE 与tcp_keepalive_time对应
TCP_KEEPINTVL 与tcp_keepalive_intvl对应
如果有些程序无法修改源码，可采用libkeepalive覆盖程序的socket调用，达到设置这些选项的目的，为socket调用提供覆盖的是ld的PRELOAD机制。假设待覆盖的程序为test，则
$ LD_PRELOAD=libkeepalive.so \
KEEPCNT=20 \
KEEPIDLE=180 \
KEEPINTVL=60 \
test
可修改test程序的socket选项。
关于libkeepalive项目可参考 http://libkeepalive.sourceforge.net/
TCP连接有很多定时器，其中一些与Keepalive有关。当这些定时器计数为0时，就会发送Keepalive探帧包（我记得在最早的BSD代码上该探帧包占一个序列号Sequence）。如果这些包没有得到对端的ACK回应，则可采取一定的控制策略，比如继续发送探帧包或认为对端已非正常中断连接。
Keepalive的用途主要有两个：检查对端是否死掉或者防止因网络无活动而连接中断。假设A和B建立了一个TCP连接，B突然死掉了，A会在一定时间后发送Keepalive探帧包，如果B重启后收到了探帧包，因TCP套接字对不上或TCP序列号对不上，会导致B给A发送RST包，此时连接马上中断。如果B一直没活过来，则发送一定数量的探帧包后A认为连接中断。这是有效检查对端死掉的情况，还有一种情况是通过NAT或代理上网，如果A和B之间隔着NAT或代理，这些NAT或者代理通常有一定的连接超时控制，发现A和B在一段时间后没有交换数据，就会切断二者的通信，此时用Keepalive可避免这种情况。
Linux下控制Keepalive的参数有三个：
tcp_keepalive_time 在连接多长时间没有数据交换时发送探帧包
tcp_keepalive_intvl 发送下一个探帧包距离发送本次探帧包的时间间隔
tcp_keepalive_probes 发送多少次没有回应的探帧包后认为对端死掉
对应在proc文件系统的位置是
/proc/sys/net/ipv4/tcp_keepalive_time
/proc/sys/net/ipv4/tcp_keepalive_intvl
/proc/sys/net/ipv4/tcp_keepalive_probes
注意：仅仅配置内核参数是不够的，还必须在编程的时候设置套接字的选项，调用函数是
int setsockopt(int s, int level, int optname,
const void *optval, socklen_t optlen)
选项为SO_KEEPALIVE，此外还可以设置TCP选项（设置的level为SOL_TCP而不是SOL_SOCKET）覆盖系统全局设置
TCP_KEEPCNT 与tcp_keepalive_probes对应
TCP_KEEPIDLE 与tcp_keepalive_time对应
TCP_KEEPINTVL 与tcp_keepalive_intvl对应
如果有些程序无法修改源码，可采用libkeepalive覆盖程序的socket调用，达到设置这些选项的目的，为socket调用提供覆盖的是ld的PRELOAD机制。假设待覆盖的程序为test，则
$ LD_PRELOAD=libkeepalive.so \
KEEPCNT=20 \
KEEPIDLE=180 \
KEEPINTVL=60 \
test
可修改test程序的socket选项。
关于libkeepalive项目可参考 http://libkeepalive.sourceforge.net/
Ubuntu 10.10与Lenovo V460
蓝牙功能需要在Windows下启用(Fn + F5)才可检测到。
要启用 nVidia显卡驱动，需在Bios设置里把显卡模式由switchable改成dicrete，然后运行 nvidia-xconfig。要打开HDMI输出，选择 系统 > 系统管理 > NVIDIA X Server Settings，点击 Detect Display按钮，应该可以发现HDMI屏幕。
要启用桌面3D，需安装compiz软件。
摄像头/读卡器默认即可支持。
要自己编译Kernel，可参考 http://en.gentoo-wiki.com/wiki/Lenovo_Ideapad_V460
感觉Ubuntu 10.10 x86_64和Lenovo V460搭配太完美了。
要启用 nVidia显卡驱动，需在Bios设置里把显卡模式由switchable改成dicrete，然后运行 nvidia-xconfig。要打开HDMI输出，选择 系统 > 系统管理 > NVIDIA X Server Settings，点击 Detect Display按钮，应该可以发现HDMI屏幕。
要启用桌面3D，需安装compiz软件。
摄像头/读卡器默认即可支持。
要自己编译Kernel，可参考 http://en.gentoo-wiki.com/wiki/Lenovo_Ideapad_V460
感觉Ubuntu 10.10 x86_64和Lenovo V460搭配太完美了。
Linux/UNIX DNS查找命令
原文参考 http://www.cyberciti.biz/faq/unix-linux-dns-lookup-command/
两条命令可帮你解决DNS服务器和域名解析的问题：host 和 dig 。
host命令的基本格式：
host example.com
host -t TYPE example.com
host -t a example.com
host例子（美元符号 $ 表示 shell提示符）
$ host -t a cyberciti.biz
cyberciti.biz has address 75.126.153.206
$ host -t mx cyberciti.biz
cyberciti.biz mail is handled by 2 CYBERCITI.BIZ.S9A2.PSMTP.com.
cyberciti.biz mail is handled by 3 CYBERCITI.BIZ.S9B1.PSMTP.com.
cyberciti.biz mail is handled by 4 CYBERCITI.BIZ.S9B2.PSMTP.com.
cyberciti.biz mail is handled by 1 CYBERCITI.BIZ.S9A1.PSMTP.com.
$ host -t ns cyberciti.biz
cyberciti.biz name server ns2.nixcraft.net.
cyberciti.biz name server ns1.nixcraft.net.
cyberciti.biz name server ns5.nixcraft.net.
cyberciti.biz name server ns4.nixcraft.net.
$ host -t txt cyberciti.biz
cyberciti.biz descriptive text "v=spf1 a mx ip4:74.86.48.99 ip4:74.86.48.98 ip4:74.86.48.102 ip4:74.86.48.101 ip4:74.86.48.100 ip4:72.26.218.170 ip4:93.89.92.12 ip4:180.92.186.178 include:_spf.google.com ~all"
$ host -t cname files.cyberciti.biz
files.cyberciti.biz is an alias for files.cyberciti.biz.edgesuite.net.
$ host -t soa cyberciti.biz
cyberciti.biz has SOA record ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600
可在命令的尾部加上域名服务器地址，以查询特定的DNS服务器，比如从 ns2.nixcraft.net 服务器查询 cyberciti.biz 的记录
$ host cyberciti.biz ns2.nixcraft.net
Using domain server:
Name: ns2.nixcraft.net
Address: 75.126.168.152#53
Aliases:
cyberciti.biz has address 75.126.153.206
cyberciti.biz has IPv6 address 2607:f0d0:1002:51::4
cyberciti.biz mail is handled by 3 CYBERCITI.BIZ.S9B1.PSMTP.com.
cyberciti.biz mail is handled by 4 CYBERCITI.BIZ.S9B2.PSMTP.com.
cyberciti.biz mail is handled by 1 CYBERCITI.BIZ.S9A1.PSMTP.com.
cyberciti.biz mail is handled by 2 CYBERCITI.BIZ.S9A2.PSMTP.com.
可以查询关于某个域的所有信息，
$ host -a cyberciti.biz
或者
$ host -t any cyberciti.biz
输出为
Trying "cyberciti.biz"
;; Truncated, retrying in TCP mode.
Trying "cyberciti.biz"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34079
;; flags: qr rd ra; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cyberciti.biz. IN ANY
;; ANSWER SECTION:
cyberciti.biz. 3423 IN AAAA 2607:f0d0:1002:51::4
cyberciti.biz. 3600 IN SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600
cyberciti.biz. 3600 IN TXT "v=spf1 a mx ip4:74.86.48.99 ip4:74.86.48.98 ip4:74.86.48.102 ip4:74.86.48.101 ip4:74.86.48.100 ip4:72.26.218.170 ip4:93.89.92.12 ip4:180.92.186.178 include:_spf.google.com ~all"
cyberciti.biz. 3600 IN MX 2 CYBERCITI.BIZ.S9A2.PSMTP.com.
cyberciti.biz. 3600 IN MX 3 CYBERCITI.BIZ.S9B1.PSMTP.com.
cyberciti.biz. 3600 IN MX 4 CYBERCITI.BIZ.S9B2.PSMTP.com.
cyberciti.biz. 3600 IN MX 1 CYBERCITI.BIZ.S9A1.PSMTP.com.
cyberciti.biz. 2805 IN A 75.126.153.206
cyberciti.biz. 3423 IN NS ns2.nixcraft.net.
cyberciti.biz. 3423 IN NS ns5.nixcraft.net.
cyberciti.biz. 3423 IN NS ns1.nixcraft.net.
cyberciti.biz. 3423 IN NS ns4.nixcraft.net.
cyberciti.biz. 84092 IN RRSIG NSEC 8 2 86400 20101125013720 20101026010313 50568 biz. OjDv09mccTZR2bYCl4D57QcnNEkBq6bNEa20ExsI6NC2sI9pmiKLnq+w UnCYxWMnkMi7WNXwIhhUWtNhV48X3wJGj1Mufrhq8MnO25JIcRE6UJF2 y12TTZHHE0UJV6HSkw1sac3XlZKXLi/oSvE/IXTsdj2SckPh+pMlaieQ jAA=
cyberciti.biz. 84092 IN NSEC CYBERCITIZEN.biz. NS RRSIG NSEC
Received 749 bytes from 192.168.1.254#53 in 0.1 ms
可以使用ipv6查询
$ host -6 cyberciti.biz
$ host -6 -a cyberciti.biz
$ host -6 cyberciti.biz ns1.nixcraft.net
$ host -6 -t ns cyberciti.biz ns3.nixcraft.net
输出为
Using domain server:
Name: ns3.nixcraft.net
Address: 2001:48c8:10:1::2#53
Aliases:
cyberciti.biz name server ns2.nixcraft.net.
cyberciti.biz name server ns4.nixcraft.net.
cyberciti.biz name server ns5.nixcraft.net.
cyberciti.biz name server ns1.nixcraft.net.
反向地址解析的格式为
$ host {IP-Address-Here}
比如
$ host 75.126.153.206
206.153.126.75.in-addr.arpa domain name pointer www.cyberciti.biz.
得到TTL信息的格式是
$ host -v -t {TYPE} {example.com}
比如
$ host -v -t a cyberciti.biz
Trying "cyberciti.biz"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17431
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 7
;; QUESTION SECTION:
;cyberciti.biz. IN A
;; ANSWER SECTION:
cyberciti.biz. 1866 IN A 75.126.153.206
;; AUTHORITY SECTION:
cyberciti.biz. 3850 IN NS NS1.NIXCRAFT.NET.
cyberciti.biz. 3850 IN NS NS4.NIXCRAFT.NET.
cyberciti.biz. 3850 IN NS NS5.NIXCRAFT.NET.
cyberciti.biz. 3850 IN NS NS2.NIXCRAFT.NET.
;; ADDITIONAL SECTION:
NS1.NIXCRAFT.NET. 85669 IN A 72.26.218.170
NS1.NIXCRAFT.NET. 85689 IN AAAA 2001:48c8:7::2
NS2.NIXCRAFT.NET. 85669 IN A 75.126.168.152
NS2.NIXCRAFT.NET. 85669 IN AAAA 2607:f0d0:1002:51::3
NS4.NIXCRAFT.NET. 85669 IN A 93.89.92.12
NS4.NIXCRAFT.NET. 85669 IN AAAA 2a01:348:0:15:5d59:50c:0:1
NS5.NIXCRAFT.NET. 85669 IN AAAA 2001:48c8:10:1::2
Received 291 bytes from 10.0.80.11#53 in 2 ms
dig命令比host功能更多，典型用法
dig @{ns1.example.com} {example.com}
dig @{ns1.example.com} {example.com} {TYPE}
dig cyberciti.biz a
dig cyberciti.biz mx
dig cyberciti.biz ns
dig cyberciti.biz txt
dig @ns1.nixcraft.net cyberciti.biz a
trace选项可跟踪从根域开始的递归查询过程，比如
$dig +trace cyberciti.biz
<<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> +trace cyberciti.biz
;; global options: printcmd
. 41219 IN NS b.root-servers.net.
. 41219 IN NS e.root-servers.net.
. 41219 IN NS i.root-servers.net.
. 41219 IN NS d.root-servers.net.
. 41219 IN NS g.root-servers.net.
. 41219 IN NS k.root-servers.net.
. 41219 IN NS l.root-servers.net.
. 41219 IN NS c.root-servers.net.
. 41219 IN NS m.root-servers.net.
. 41219 IN NS a.root-servers.net.
. 41219 IN NS h.root-servers.net.
. 41219 IN NS j.root-servers.net.
. 41219 IN NS f.root-servers.net.
;; Received 436 bytes from 10.0.80.11#53(10.0.80.11) in 2 ms
biz. 172800 IN NS h.gtld.biz.
biz. 172800 IN NS c.gtld.biz.
biz. 172800 IN NS e.gtld.biz.
biz. 172800 IN NS b.gtld.biz.
biz. 172800 IN NS g.gtld.biz.
biz. 172800 IN NS a.gtld.biz.
biz. 172800 IN NS f.gtld.biz.
;; Received 316 bytes from 192.228.79.201#53(b.root-servers.net) in 34 ms
cyberciti.biz. 7200 IN NS NS5.NIXCRAFT.NET.
cyberciti.biz. 7200 IN NS NS1.NIXCRAFT.NET.
cyberciti.biz. 7200 IN NS NS2.NIXCRAFT.NET.
cyberciti.biz. 7200 IN NS NS4.NIXCRAFT.NET.
;; Received 115 bytes from 2001:503:8028:ffff:ffff:ffff:ffff:ff7e#53(h.gtld.biz) in 23 ms
cyberciti.biz. 3600 IN A 75.126.153.206
cyberciti.biz. 3600 IN NS ns4.nixcraft.net.
cyberciti.biz. 3600 IN NS ns5.nixcraft.net.
cyberciti.biz. 3600 IN NS ns1.nixcraft.net.
cyberciti.biz. 3600 IN NS ns2.nixcraft.net.
;; Received 307 bytes from 2001:48c8:10:1::2#53(NS5.NIXCRAFT.NET) in 222 ms
得到dns的简短回答
$ dig +short cyberciti.biz
75.126.153.206
显示所有记录
$ dig +noall +answer cyberciti.biz any
cyberciti.biz. 3490 IN A 75.126.153.206
cyberciti.biz. 2733 IN NS NS2.NIXCRAFT.NET.
cyberciti.biz. 2733 IN NS NS1.NIXCRAFT.NET.
cyberciti.biz. 2733 IN NS NS4.NIXCRAFT.NET.
cyberciti.biz. 2733 IN NS NS5.NIXCRAFT.NET.
cyberciti.biz. 85668 IN RRSIG NSEC 8 2 86400 20101125013720 20101026010313 50568 biz. OjDv09mccTZR2bYCl4D57QcnNEkBq6bNEa20ExsI6NC2sI9pmiKLnq+w UnCYxWMnkMi7WNXwIhhUWtNhV48X3wJGj1Mufrhq8MnO25JIcRE6UJF2 y12TTZHHE0UJV6HSkw1sac3XlZKXLi/oSvE/IXTsdj2SckPh+pMlaieQ jAA=
cyberciti.biz. 85668 IN NSEC CYBERCITIZEN.biz. NS RRSIG NSEC
反向地址解析格式为
$ dig -x +short {IP-Address-here}
比如
$ dig -x 75.126.153.206 +short
www.cyberciti.biz.
找 SOA 记录
$ dig +nssearch cyberciti.biz
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns5.nixcraft.net in 81 ms.
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns4.nixcraft.net in 216 ms.
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns1.nixcraft.net in 347 ms.
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns2.nixcraft.net in 316 ms.
找 TTL 值格式为
$ dig +nocmd +noall +answer {TYPE} {example.com}
比如
$ dig +nocmd +noall +answer a cyberciti.biz
cyberciti.biz. 1642 IN A 75.126.153.206
两条命令可帮你解决DNS服务器和域名解析的问题：host 和 dig 。
host命令的基本格式：
host example.com
host -t TYPE example.com
host -t a example.com
host例子（美元符号 $ 表示 shell提示符）
$ host -t a cyberciti.biz
cyberciti.biz has address 75.126.153.206
$ host -t mx cyberciti.biz
cyberciti.biz mail is handled by 2 CYBERCITI.BIZ.S9A2.PSMTP.com.
cyberciti.biz mail is handled by 3 CYBERCITI.BIZ.S9B1.PSMTP.com.
cyberciti.biz mail is handled by 4 CYBERCITI.BIZ.S9B2.PSMTP.com.
cyberciti.biz mail is handled by 1 CYBERCITI.BIZ.S9A1.PSMTP.com.
$ host -t ns cyberciti.biz
cyberciti.biz name server ns2.nixcraft.net.
cyberciti.biz name server ns1.nixcraft.net.
cyberciti.biz name server ns5.nixcraft.net.
cyberciti.biz name server ns4.nixcraft.net.
$ host -t txt cyberciti.biz
cyberciti.biz descriptive text "v=spf1 a mx ip4:74.86.48.99 ip4:74.86.48.98 ip4:74.86.48.102 ip4:74.86.48.101 ip4:74.86.48.100 ip4:72.26.218.170 ip4:93.89.92.12 ip4:180.92.186.178 include:_spf.google.com ~all"
$ host -t cname files.cyberciti.biz
files.cyberciti.biz is an alias for files.cyberciti.biz.edgesuite.net.
$ host -t soa cyberciti.biz
cyberciti.biz has SOA record ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600
可在命令的尾部加上域名服务器地址，以查询特定的DNS服务器，比如从 ns2.nixcraft.net 服务器查询 cyberciti.biz 的记录
$ host cyberciti.biz ns2.nixcraft.net
Using domain server:
Name: ns2.nixcraft.net
Address: 75.126.168.152#53
Aliases:
cyberciti.biz has address 75.126.153.206
cyberciti.biz has IPv6 address 2607:f0d0:1002:51::4
cyberciti.biz mail is handled by 3 CYBERCITI.BIZ.S9B1.PSMTP.com.
cyberciti.biz mail is handled by 4 CYBERCITI.BIZ.S9B2.PSMTP.com.
cyberciti.biz mail is handled by 1 CYBERCITI.BIZ.S9A1.PSMTP.com.
cyberciti.biz mail is handled by 2 CYBERCITI.BIZ.S9A2.PSMTP.com.
可以查询关于某个域的所有信息，
$ host -a cyberciti.biz
或者
$ host -t any cyberciti.biz
输出为
Trying "cyberciti.biz"
;; Truncated, retrying in TCP mode.
Trying "cyberciti.biz"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34079
;; flags: qr rd ra; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cyberciti.biz. IN ANY
;; ANSWER SECTION:
cyberciti.biz. 3423 IN AAAA 2607:f0d0:1002:51::4
cyberciti.biz. 3600 IN SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600
cyberciti.biz. 3600 IN TXT "v=spf1 a mx ip4:74.86.48.99 ip4:74.86.48.98 ip4:74.86.48.102 ip4:74.86.48.101 ip4:74.86.48.100 ip4:72.26.218.170 ip4:93.89.92.12 ip4:180.92.186.178 include:_spf.google.com ~all"
cyberciti.biz. 3600 IN MX 2 CYBERCITI.BIZ.S9A2.PSMTP.com.
cyberciti.biz. 3600 IN MX 3 CYBERCITI.BIZ.S9B1.PSMTP.com.
cyberciti.biz. 3600 IN MX 4 CYBERCITI.BIZ.S9B2.PSMTP.com.
cyberciti.biz. 3600 IN MX 1 CYBERCITI.BIZ.S9A1.PSMTP.com.
cyberciti.biz. 2805 IN A 75.126.153.206
cyberciti.biz. 3423 IN NS ns2.nixcraft.net.
cyberciti.biz. 3423 IN NS ns5.nixcraft.net.
cyberciti.biz. 3423 IN NS ns1.nixcraft.net.
cyberciti.biz. 3423 IN NS ns4.nixcraft.net.
cyberciti.biz. 84092 IN RRSIG NSEC 8 2 86400 20101125013720 20101026010313 50568 biz. OjDv09mccTZR2bYCl4D57QcnNEkBq6bNEa20ExsI6NC2sI9pmiKLnq+w UnCYxWMnkMi7WNXwIhhUWtNhV48X3wJGj1Mufrhq8MnO25JIcRE6UJF2 y12TTZHHE0UJV6HSkw1sac3XlZKXLi/oSvE/IXTsdj2SckPh+pMlaieQ jAA=
cyberciti.biz. 84092 IN NSEC CYBERCITIZEN.biz. NS RRSIG NSEC
Received 749 bytes from 192.168.1.254#53 in 0.1 ms
可以使用ipv6查询
$ host -6 cyberciti.biz
$ host -6 -a cyberciti.biz
$ host -6 cyberciti.biz ns1.nixcraft.net
$ host -6 -t ns cyberciti.biz ns3.nixcraft.net
输出为
Using domain server:
Name: ns3.nixcraft.net
Address: 2001:48c8:10:1::2#53
Aliases:
cyberciti.biz name server ns2.nixcraft.net.
cyberciti.biz name server ns4.nixcraft.net.
cyberciti.biz name server ns5.nixcraft.net.
cyberciti.biz name server ns1.nixcraft.net.
反向地址解析的格式为
$ host {IP-Address-Here}
比如
$ host 75.126.153.206
206.153.126.75.in-addr.arpa domain name pointer www.cyberciti.biz.
得到TTL信息的格式是
$ host -v -t {TYPE} {example.com}
比如
$ host -v -t a cyberciti.biz
Trying "cyberciti.biz"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17431
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 7
;; QUESTION SECTION:
;cyberciti.biz. IN A
;; ANSWER SECTION:
cyberciti.biz. 1866 IN A 75.126.153.206
;; AUTHORITY SECTION:
cyberciti.biz. 3850 IN NS NS1.NIXCRAFT.NET.
cyberciti.biz. 3850 IN NS NS4.NIXCRAFT.NET.
cyberciti.biz. 3850 IN NS NS5.NIXCRAFT.NET.
cyberciti.biz. 3850 IN NS NS2.NIXCRAFT.NET.
;; ADDITIONAL SECTION:
NS1.NIXCRAFT.NET. 85669 IN A 72.26.218.170
NS1.NIXCRAFT.NET. 85689 IN AAAA 2001:48c8:7::2
NS2.NIXCRAFT.NET. 85669 IN A 75.126.168.152
NS2.NIXCRAFT.NET. 85669 IN AAAA 2607:f0d0:1002:51::3
NS4.NIXCRAFT.NET. 85669 IN A 93.89.92.12
NS4.NIXCRAFT.NET. 85669 IN AAAA 2a01:348:0:15:5d59:50c:0:1
NS5.NIXCRAFT.NET. 85669 IN AAAA 2001:48c8:10:1::2
Received 291 bytes from 10.0.80.11#53 in 2 ms
dig命令比host功能更多，典型用法
dig @{ns1.example.com} {example.com}
dig @{ns1.example.com} {example.com} {TYPE}
dig cyberciti.biz a
dig cyberciti.biz mx
dig cyberciti.biz ns
dig cyberciti.biz txt
dig @ns1.nixcraft.net cyberciti.biz a
trace选项可跟踪从根域开始的递归查询过程，比如
$dig +trace cyberciti.biz
<<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> +trace cyberciti.biz
;; global options: printcmd
. 41219 IN NS b.root-servers.net.
. 41219 IN NS e.root-servers.net.
. 41219 IN NS i.root-servers.net.
. 41219 IN NS d.root-servers.net.
. 41219 IN NS g.root-servers.net.
. 41219 IN NS k.root-servers.net.
. 41219 IN NS l.root-servers.net.
. 41219 IN NS c.root-servers.net.
. 41219 IN NS m.root-servers.net.
. 41219 IN NS a.root-servers.net.
. 41219 IN NS h.root-servers.net.
. 41219 IN NS j.root-servers.net.
. 41219 IN NS f.root-servers.net.
;; Received 436 bytes from 10.0.80.11#53(10.0.80.11) in 2 ms
biz. 172800 IN NS h.gtld.biz.
biz. 172800 IN NS c.gtld.biz.
biz. 172800 IN NS e.gtld.biz.
biz. 172800 IN NS b.gtld.biz.
biz. 172800 IN NS g.gtld.biz.
biz. 172800 IN NS a.gtld.biz.
biz. 172800 IN NS f.gtld.biz.
;; Received 316 bytes from 192.228.79.201#53(b.root-servers.net) in 34 ms
cyberciti.biz. 7200 IN NS NS5.NIXCRAFT.NET.
cyberciti.biz. 7200 IN NS NS1.NIXCRAFT.NET.
cyberciti.biz. 7200 IN NS NS2.NIXCRAFT.NET.
cyberciti.biz. 7200 IN NS NS4.NIXCRAFT.NET.
;; Received 115 bytes from 2001:503:8028:ffff:ffff:ffff:ffff:ff7e#53(h.gtld.biz) in 23 ms
cyberciti.biz. 3600 IN A 75.126.153.206
cyberciti.biz. 3600 IN NS ns4.nixcraft.net.
cyberciti.biz. 3600 IN NS ns5.nixcraft.net.
cyberciti.biz. 3600 IN NS ns1.nixcraft.net.
cyberciti.biz. 3600 IN NS ns2.nixcraft.net.
;; Received 307 bytes from 2001:48c8:10:1::2#53(NS5.NIXCRAFT.NET) in 222 ms
得到dns的简短回答
$ dig +short cyberciti.biz
75.126.153.206
显示所有记录
$ dig +noall +answer cyberciti.biz any
cyberciti.biz. 3490 IN A 75.126.153.206
cyberciti.biz. 2733 IN NS NS2.NIXCRAFT.NET.
cyberciti.biz. 2733 IN NS NS1.NIXCRAFT.NET.
cyberciti.biz. 2733 IN NS NS4.NIXCRAFT.NET.
cyberciti.biz. 2733 IN NS NS5.NIXCRAFT.NET.
cyberciti.biz. 85668 IN RRSIG NSEC 8 2 86400 20101125013720 20101026010313 50568 biz. OjDv09mccTZR2bYCl4D57QcnNEkBq6bNEa20ExsI6NC2sI9pmiKLnq+w UnCYxWMnkMi7WNXwIhhUWtNhV48X3wJGj1Mufrhq8MnO25JIcRE6UJF2 y12TTZHHE0UJV6HSkw1sac3XlZKXLi/oSvE/IXTsdj2SckPh+pMlaieQ jAA=
cyberciti.biz. 85668 IN NSEC CYBERCITIZEN.biz. NS RRSIG NSEC
反向地址解析格式为
$ dig -x +short {IP-Address-here}
比如
$ dig -x 75.126.153.206 +short
www.cyberciti.biz.
找 SOA 记录
$ dig +nssearch cyberciti.biz
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns5.nixcraft.net in 81 ms.
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns4.nixcraft.net in 216 ms.
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns1.nixcraft.net in 347 ms.
SOA ns1.nixcraft.net. vivek.nixcraft.com. 2008072353 10800 3600 604800 3600 from server ns2.nixcraft.net in 316 ms.
找 TTL 值格式为
$ dig +nocmd +noall +answer {TYPE} {example.com}
比如
$ dig +nocmd +noall +answer a cyberciti.biz
cyberciti.biz. 1642 IN A 75.126.153.206
Windows outlook 邮件列表Sender字段显示的误区
有些人的MUA显示类似如下的信息
From listname-admin@mailman.server.com 代表 fred@poster.domain.com
或者
From listname-bounces@mailman.server.com 代表 fred@poster.domain.com
原因在 http://wiki.list.org/display/DOC/From+field+displayed+by+Microsoft+Outlook 有说明，这是由于Windows下的一些MUA解释邮件头字段Sender造成的，相关的RFC说明
=================================
The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message. For example, if a secretary were to send a message for another person, the mailbox of the secretary would appear in the "Sender:" field and the mailbox of the actual author would appear in the "From:" field. If the originator of the message can be indicated by a single mailbox and the author and transmitter are identical, the "Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD appear.
http://www.faqs.org/rfcs/rfc2822.html
=================================
从这个说明看，Windows的MUA在显示Sender字段的准确性上有待商榷。
From listname-admin@mailman.server.com 代表 fred@poster.domain.com
或者
From listname-bounces@mailman.server.com 代表 fred@poster.domain.com
原因在 http://wiki.list.org/display/DOC/From+field+displayed+by+Microsoft+Outlook 有说明，这是由于Windows下的一些MUA解释邮件头字段Sender造成的，相关的RFC说明
=================================
The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message. For example, if a secretary were to send a message for another person, the mailbox of the secretary would appear in the "Sender:" field and the mailbox of the actual author would appear in the "From:" field. If the originator of the message can be indicated by a single mailbox and the author and transmitter are identical, the "Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD appear.
http://www.faqs.org/rfcs/rfc2822.html
=================================
从这个说明看，Windows的MUA在显示Sender字段的准确性上有待商榷。
今天的Java程序员是明天的比萨投递员？
几年前看了一篇文章，说今天的Java程序员是明天的比萨投递员，google了一下这篇文章，链接如下
http://rixstep.com/2/2/20080127,00.shtml
大致意思是说面向对象和Java编程隐藏了太多的细节，让人不再去思考计算机科学的细节，比如指令寄存器，段和偏移，堆分配，栈，编译器，微代码……这会让自己的职业毫无竞争力。
这种说法也许欠缺考虑，Java的发展一片美好，Java 7/8要出来了，有望成为一种兼性能、生产力与优雅于一体的首选语言，见
http://www.developer.com/java/java-7-and-8-begin-to-take-shape-whats-in-whats-out.html
我比较看好的几个功能：
Lambda和闭包的支持，有望取代inner class；
集合的自动化并行处理；
JVM的模块化和Jigsaw(一种通过模块化JVM减肥Java程序，从而提升加载速度和资
源开销的思路)；
fork/join的框架体系，可发挥多核和多线程的优势；
真正的异步I/O API，应对I/O密集型应用
http://rixstep.com/2/2/20080127,00.shtml
大致意思是说面向对象和Java编程隐藏了太多的细节，让人不再去思考计算机科学的细节，比如指令寄存器，段和偏移，堆分配，栈，编译器，微代码……这会让自己的职业毫无竞争力。
这种说法也许欠缺考虑，Java的发展一片美好，Java 7/8要出来了，有望成为一种兼性能、生产力与优雅于一体的首选语言，见
http://www.developer.com/java/java-7-and-8-begin-to-take-shape-whats-in-whats-out.html
我比较看好的几个功能：
Lambda和闭包的支持，有望取代inner class；
集合的自动化并行处理；
JVM的模块化和Jigsaw(一种通过模块化JVM减肥Java程序，从而提升加载速度和资
源开销的思路)；
fork/join的框架体系，可发挥多核和多线程的优势；
真正的异步I/O API，应对I/O密集型应用
如何从rpm包提取文件
原文参考 http://www.cyberciti.biz/tips/how-to-extract-an-rpm-package-without-installing-it.html
Most of you may know to how extract a tarballs and/or a zip files. Someone recently PM me with a question:
How do I extract an RPM package without installing it on my Fedora Core Linux v5?
Extract rpm File
To be frank there is no direct RPM option available via rpm command to extract an RPM file. But there is a small nifty utility available called rpm2cpio. It Extract cpio archive from RPM Package Manager (RPM) package. With the following hack you will be able to extract an RPM file.
So rpm2cpio converts the .rpm file specified as a single argument to a cpio archive on standard out. If a - argument is given, an rpm stream is read from standard in.
Syntax is as follows:
rpm2cpio myrpmfile.rpm
rpm2cpio - < myrpmfile.rpm
rpm2cpio myrpmfile.rpm | cpio -idmv
Examples - Extract files from rpm
Download an RPM file:
$ mkdir test
$ cd test
$ wget http://www.cyberciti.biz/files/lighttpd/rhel4-php5-fastcgi/php-5.1.4-1.esp1.x86_64.rpm
Extract RPM file using rpm2cpio and cpio command:
$ rpm2cpio php-5.1.4-1.esp1.x86_64.rpm | cpio -idmv
Output:
/etc/httpd/conf.d/php.conf ./etc/php.d ./etc/php.ini ./usr/bin/php ./usr/bin/php-cgi ./usr/lib64/httpd/modules/libphp5.so ./usr/lib64/php ./usr/lib64/php/modules .... ..... .. ./var/lib/php/session ./var/www/icons/php.gif 19188 blocks
Output of rpm2cpio piped to cpio command (see how to use cpio) with following options:
i: Restore archive
d: Create leading directories where needed
m: Retain previous file modification times when creating files
v: Verbose i.e. display progress
Verify that you have extracted an RPM file in current directory:
$ ls
Output:
etc php-5.1.4-1.esp1.x86_64.rpm usr var
This is useful if you want to extract configuration file or other file w/o installing an RPM file.
Most of you may know to how extract a tarballs and/or a zip files. Someone recently PM me with a question:
How do I extract an RPM package without installing it on my Fedora Core Linux v5?
Extract rpm File
To be frank there is no direct RPM option available via rpm command to extract an RPM file. But there is a small nifty utility available called rpm2cpio. It Extract cpio archive from RPM Package Manager (RPM) package. With the following hack you will be able to extract an RPM file.
So rpm2cpio converts the .rpm file specified as a single argument to a cpio archive on standard out. If a - argument is given, an rpm stream is read from standard in.
Syntax is as follows:
rpm2cpio myrpmfile.rpm
rpm2cpio - < myrpmfile.rpm
rpm2cpio myrpmfile.rpm | cpio -idmv
Examples - Extract files from rpm
Download an RPM file:
$ mkdir test
$ cd test
$ wget http://www.cyberciti.biz/files/lighttpd/rhel4-php5-fastcgi/php-5.1.4-1.esp1.x86_64.rpm
Extract RPM file using rpm2cpio and cpio command:
$ rpm2cpio php-5.1.4-1.esp1.x86_64.rpm | cpio -idmv
Output:
/etc/httpd/conf.d/php.conf ./etc/php.d ./etc/php.ini ./usr/bin/php ./usr/bin/php-cgi ./usr/lib64/httpd/modules/libphp5.so ./usr/lib64/php ./usr/lib64/php/modules .... ..... .. ./var/lib/php/session ./var/www/icons/php.gif 19188 blocks
Output of rpm2cpio piped to cpio command (see how to use cpio) with following options:
i: Restore archive
d: Create leading directories where needed
m: Retain previous file modification times when creating files
v: Verbose i.e. display progress
Verify that you have extracted an RPM file in current directory:
$ ls
Output:
etc php-5.1.4-1.esp1.x86_64.rpm usr var
This is useful if you want to extract configuration file or other file w/o installing an RPM file.
MongoDB GridFS
MongoDB GridFS规范参考 http://www.mongodb.org/display/DOCS/GridFS+Specification，
使用的时候加上索引
db.fs.files.ensureIndex({filename:1}));
db.fs.chunks.ensureIndex({files_id:1, n:1}, {unique: true});
做类似于文件系统的目录扫描时，可用正则查找，查找更高级的功能见
http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-ConditionalOperators
为了获取较好的性能，最好只做路径前缀扫描，且不要带其他正则标记（比如不区分大小写的i），这样可以使用filename索引，比如
db.fs.files.find( { filename: /^myfiles/ } )
GridFS的性能比本地文件系统要差很多，在台式机上测试了一下写入约2MB/s，读取约5MB/s，数据规模为1300万个10K ~ 100K的小文件，另参考
http://www.ypass.net/solaris/nginx-gridfs-benchmarks/index.html
据说读取50K以上的文件可以把千兆网卡填满。
数据每2G建一个文件，当新建文件时如果MongoDB刚好很繁忙会有超时（测试遇到的情况是有45秒钟阻塞)，网上建议用ext4文件系统，这样可加速文件创建过程。
如果打算产品环境使用，建议做好数据冗余(Replication/ReplSet)，由于MongoDB没有日志恢复功能（据说版本1.8支持，但目前还没稳定版释出），数据恢复过程异常缓慢。
使用的时候加上索引
db.fs.files.ensureIndex({filename:1}));
db.fs.chunks.ensureIndex({files_id:1, n:1}, {unique: true});
做类似于文件系统的目录扫描时，可用正则查找，查找更高级的功能见
http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-ConditionalOperators
为了获取较好的性能，最好只做路径前缀扫描，且不要带其他正则标记（比如不区分大小写的i），这样可以使用filename索引，比如
db.fs.files.find( { filename: /^myfiles/ } )
GridFS的性能比本地文件系统要差很多，在台式机上测试了一下写入约2MB/s，读取约5MB/s，数据规模为1300万个10K ~ 100K的小文件，另参考
http://www.ypass.net/solaris/nginx-gridfs-benchmarks/index.html
据说读取50K以上的文件可以把千兆网卡填满。
数据每2G建一个文件，当新建文件时如果MongoDB刚好很繁忙会有超时（测试遇到的情况是有45秒钟阻塞)，网上建议用ext4文件系统，这样可加速文件创建过程。
如果打算产品环境使用，建议做好数据冗余(Replication/ReplSet)，由于MongoDB没有日志恢复功能（据说版本1.8支持，但目前还没稳定版释出），数据恢复过程异常缓慢。
订阅：
博文 (Atom)