一、NameNode、Fsimage 、Edits 和 SecondaryNameNode 概述
NameNode：在内存中储存 HDFS 文件的元数据信息（目录）
如果节点故障或断电，存在内存中的数据会丢失，显然只在内存中保存是不可靠的 实际在磁盘当中也有保存：Fsimage 和 Edits，一个 NameNode 节点在重启后会根据这磁盘上的这两个文件来恢复到之前的状态
Fsimage（镜像文件） 和 Edits（编辑日志）：记录内存中的元数据
如果每次对 HDFS 的操作都实时的把内存中的元数据信息往磁盘上传输，这样显然效率不够高，也不稳定 这时就出现了 Edits 文件，用来记录每次对 HDFS 的操作，这样在磁盘上每次就只用做很小改动（只进行追加操作） 当 Edits 文件达到了一定大小或过了一定的时间，就需要把 Edits 文件转化 Fsimage 文件，然后清空 Edits 这样的 Fsimage 文件不会和内存中的元数据实时同步，需要加上 Edits 文件才相等
SecondaryNameNode：负责 Edits 转化成 Fsimage
SecondaryNameNode 不是 NameNode 的备份 SecondaryNameNode 会定时定量的把集群中的 Edits 文件转化为 Fsimage 文件，来保证 NameNode 中数据的可靠性
二、NameNode 工作机制
磁盘上的元数据文件
三、查看 Edits 与 Fsimage
Edits 和 Fsimage 并非明文存储，需要转换后才能查看，使用 hdfs 命令进行转换
# 部分命令参数 Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND where COMMAND is one of: # 转换 fsimage 文件 oiv apply the offline fsimage viewer to an fsimage # 转换旧版 fsimage 文件 oiv_legacy apply the offline fsimage viewer to an legacy fsimage # 转换 edits 文件 oev apply the offline edits viewer to an edits file
1.转换 Fsimage 文件
hdfs oiv：查看参数
部分参数说明
Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE -i 要转换的文件 -o 转换后文件路径 -p 转换格式 (XML|FileDistribution|ReverseXML|Web|Delimited)
转换文件
hdfs oiv -p XML -i ./fsimage_0000000000000000326 -o ./fsimage.xml
转换后的文件，记录了文件信息
<?xml version="1.0"?> <fsimage> <version> <layoutVersion>-63</layoutVersion> <onDiskVersion>1</onDiskVersion> <oivRevision>Unknown</oivRevision> </version> <NameSection> <namespaceId>1803101069</namespaceId> <genstampV1>1000</genstampV1> <genstampV2>1015</genstampV2> <genstampV1Limit>0</genstampV1Limit> <lastAllocatedBlockId>1073741839</lastAllocatedBlockId> <txid>326</txid> </NameSection> <INodeSection> <lastInodeId>16400</lastInodeId> <numInodes>11</numInodes> <inode> <id>16385</id> <type>DIRECTORY</type> <name></name> <mtime>1555218314075</mtime> <permission>root:supergroup:0755</permission> <nsquota>9223372036854775807</nsquota> <dsquota>-1</dsquota> </inode> <inode> <id>16388</id> <type>FILE</type> <name>javafx-src.zip</name> <replication>2</replication> <mtime>1554996304962</mtime> <atime>15550 84059617</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>root:supergroup:0644</permission> <blocks> <block> <id>107 3741829</id> <genstamp>1005</genstamp> <numBytes>5211154</numBytes> </block> </blocks> <storagePolicyId>0</storagePolicyId> </inode> <inode> <id>16389</id> <type>DIRECTORY</type> <name>0529</name> <mtime>1554999968472</mtime> <permission>root:supergroup:0755</permission> <nsquota>-1</nsquota> <dsquota>-1</dsquota> </inode> <inode> <id>16390</id> <type>DIRECTORY</type> <name>mkmk</name> <mtime>1555000275046</mtime> <permission>root:supergroup:0755</permission> <nsquota>-1</nsquota> <dsquota>-1</dsquota> </inode> <inode> <id>16391</id> <type>DIRECTORY</type> <name>10088</name> <mtime>1555000763405</mtime> <permission>root:supergroup:0755</permission> <nsquota>-1</nsquota> <dsquota>-1</dsquota> </inode> <inode> <id>16394</id> <type>FILE</type> <name>Writage-1.12.msi</name> <replication>3</replication> <mtime>1555086090236</mtime> <atime>155 5086085824</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>root:supergroup:0644</permission> <blocks> <block> <id>1 073741832</id> <genstamp>1008</genstamp> <numBytes>12771328</numBytes> </block> </blocks> <storagePolicyId>0</storagePolicyId> </inode> <inode> <id>16395</id> <type>FILE</type> <name>hadoop-2.9.2-win10-64.tar.gz</name> <replication>3</replication> <mtime>1555086717860</mtime> <atime>1555086612935</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>root:supergroup:0644</permission> <blocks> <block> <id>1073741833</id> <genstamp>1009</genstamp> <numBytes>134217728</numBytes> </block> <block> <id>1073741834</id> <genstamp>1010</genstamp> <numBytes>134217728</numBytes> </block> <block> <id>1073741835</id> <genstamp>1011</genstamp> <numBytes>43080138</numBytes> </block> </blocks> <storagePolicyId>0</storagePolicyId> </inode> <inode> <id>16396</id> <type>DIRECTORY</type> <name>Priority</name> <mtime>1555219450567</mtime> <permission>root:supergroup:0755</permission> <nsquota>-1</nsquota> <dsquota>-1</dsquota> </inode> <inode> <id>16398</id> <type>FILE</type> <name>AAA.msi</name> <replication>3</replication> <mtime>1555219070073</mtime> <atime>155521906501 6</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>root:supergroup:0644</permission> <blocks> <block> <id>1073741837</id> <genstamp>1013</genstamp> <numBytes>12771328</numBytes> </block> </blocks> <storagePolicyId>0</storagePolicyId> </inode> <inode> <id>16399</id> <type>FILE</type> <name>BBB.msi</name> <replication>1</replication> <mtime>1555219285187</mtime> <atime>155521928012 9</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>root:supergroup:0644</permission> <blocks> <block> <id>1073741838</id> <genstamp>1014</genstamp> <numBytes>12771328</numBytes> </block> </blocks> <storagePolicyId>0</storagePolicyId> </inode> <inode> <id>16400</id> <type>FILE</type> <name>CCC.msi</name> <replication>2</replication> <mtime>1555219455139</mtime> <atime>155521945056 7</atime> <preferredBlockSize>134217728</preferredBlockSize> <permission>root:supergroup:0644</permission> <blocks> <block> <id>1073741839</id> <genstamp>1015</genstamp> <numBytes>12771328</numBytes> </block> </blocks> <storagePolicyId>0</storagePolicyId> </inode> </INodeSection> <INodeReferenceSection></INodeReferenceSection> <SnapshotSection> <snapshotCounter>0</snapshotCounter> <numSnapshots>0</numSnapshots> </SnapshotSection> <INodeDirectorySection> <directory> <parent>16385</parent> <child>16389</child> <child>16391</child> <child>16396</child> <child>16394</child> <child>16395</child> <child>16388</child> <child>16390</child> </directory> <directory> <parent>16396</parent> <child>16398</child> <child>16399</child> <child>16400</child> </directory> </INodeDirectorySection> <FileUnderConstructionSection></FileUnderConstructionSection> <SecretManagerSection> <currentId>0</currentId> <tokenSequenceNumber>0</tokenSequenceNumber> <numDelegationKeys>0</numDelegationKeys> <numTokens>0</numTokens> </SecretManagerSection> <CacheManagerSection> <nextDirectiveId>1</nextDirectiveId> <numDirectives>0</numDirectives> <numPools>0</numPools> </CacheManagerSection> </fsimage>
2.转换 Edits 文件
hdfs oev：查看参数
参数与上面类似，-p 有些不同
-p binary (native binary format that Hadoop uses), xml (default, XML format), stats (prints statistics about edits file)
转换文件
hdfs oev -p xml -i ./edits_inprogress_0000000000000000329 -o ./edits.xml
转换后的文件，记录 hdfs 的操作记录
<?xml version="1.0" encoding="UTF-8"?> <EDITS> <EDITS_VERSION>-63</EDITS_VERSION> <RECORD> <OPCODE>OP_START_LOG_SEGMENT</OPCODE> <DATA> <TXID>16</TXID> </DATA> </RECORD> <RECORD> <OPCODE>OP_ADD</OPCODE> <DATA> <TXID>17</TXID> <LENGTH>0</LENGTH> <INODEID>16386</INODEID> <PATH>/javafx-src.zip._COPYING_</PATH> <REPLICATION>3</REPLICATION> <MTIME>1554995796290</MTIME> <ATIME>1554995796290</ATIME> <BLOCKSIZE>134217728</BLOCKSIZE> <CLIENT_NAME>DFSClient_NONMAPREDUCE_1941526907_1</CLIENT_NAME> <CLIENT_MACHINE>192.168.8.136</CLIENT_MACHINE> <OVERWRITE>true</OVERWRITE> <PERMISSION_STATUS> <USERNAME>root</USERNAME> <GROUPNAME>supergroup</GROUPNAME> <MODE>420</MODE> </PERMISSION_STATUS> <RPC_CLIENTID>5b80319d-463f-4591-b090-2d1462563e48</RPC_CLIENTID> <RPC_CALLID>3</RPC_CALLID> </DATA> </RECORD> <RECORD> <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE> <DATA> <TXID>18</TXID> <BLOCK_ID>1073741825</BLOCK_ID> </DATA> </RECORD> <RECORD> <OPCODE>OP_SET_GENSTAMP_V2</OPCODE> <DATA> <TXID>19</TXID> <GENSTAMPV2>1001</GENSTAMPV2> </DATA> </RECORD> <RECORD> <OPCODE>OP_ADD_BLOCK</OPCODE> <DATA> <TXID>20</TXID> <PATH>/javafx-src.zip._COPYING_</PATH> <BLOCK> <BLOCK_ID>1073741825</BLOCK_ID> <NUM_BYTES>0</NUM_BYTES> <GENSTAMP>1001</GENSTAMP> </BLOCK> <RPC_CLIENTID></RPC_CLIENTID> <RPC_CALLID>-2</RPC_CALLID> </DATA> </RECORD> <RECORD> <OPCODE>OP_UPDATE_BLOCKS</OPCODE> <DATA> <TXID>21</TXID> <PATH>/javafx-src.zip._COPYING_</PATH> <RPC_CLIENTID></RPC_CLIENTID> <RPC_CALLID>-2</RPC_CALLID> </DATA> </RECORD> <RECORD> <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE> <DATA> <TXID>22</TXID> <BLOCK_ID>1073741826</BLOCK_ID> </DATA> </RECORD> <RECORD> <OPCODE>OP_SET_GENSTAMP_V2</OPCODE> <DATA> <TXID>23</TXID> <GENSTAMPV2>1002</GENSTAMPV2> </DATA> </RECORD> <RECORD> <OPCODE>OP_ADD_BLOCK</OPCODE> <DATA> <TXID>24</TXID> <PATH>/javafx-src.zip._COPYING_</PATH> <BLOCK> <BLOCK_ID>1073741826</BLOCK_ID> <NUM_BYTES>0</NUM_BYTES> <GENSTAMP>1002</GENSTAMP> </BLOCK> <RPC_CLIENTID></RPC_CLIENTID> <RPC_CALLID>-2</RPC_CALLID> </DATA> </RECORD> <RECORD> <OPCODE>OP_CLOSE</OPCODE> <DATA> <TXID>25</TXID> <LENGTH>0</LENGTH> <INODEID>0</INODEID> <PATH>/javafx-src.zip._COPYING_</PATH> <REPLICATION>3</REPLICATION> <MTIME>1554995797095</MTIME> <ATIME>1554995796290</ATIME> <BLOCKSIZE>134217728</BLOCKSIZE> <CLIENT_NAME></CLIENT_NAME> <CLIENT_MACHINE></CLIENT_MACHINE> <OVERWRITE>false</OVERWRITE> <BLOCK> <BLOCK_ID>1073741826</BLOCK_ID> <NUM_BYTES>5211154</NUM_BYTES> <GENSTAMP>1002</GENSTAMP> </BLOCK> <PERMISSION_STATUS> <USERNAME>root</USERNAME> <GROUPNAME>supergroup</GROUPNAME> <MODE>420</MODE> </PERMISSION_STATUS> </DATA> </RECORD> <RECORD> <OPCODE>OP_RENAME_OLD</OPCODE> <DATA> <TXID>26</TXID> <LENGTH>0</LENGTH> <SRC>/javafx-src.zip._COPYING_</SRC> <DST>/javafx-src.zip</DST> <TIMESTAMP>1554995797102</TIMESTAMP> <RPC_CLIENTID>5b80319d-463f-4591-b090-2d1462563e48</RPC_CLIENTID> <RPC_CALLID>11</RPC_CALLID> </DATA> </RECORD> <RECORD> <OPCODE>OP_DELETE</OPCODE> <DATA> <TXID>27</TXID> <LENGTH>0</LENGTH> <PATH>/javafx-src.zip</PATH> <TIMESTAMP>1554995877570</TIMESTAMP> <RPC_CLIENTID>847eb009-9524-4f1c-a5a5-5253908ec1b5</RPC_CLIENTID> <RPC_CALLID>3</RPC_CALLID> </DATA> </RECORD> </EDITS>
四、CheckPoint 时间设置
1.以时间为准
<property> <name>dfs.namenode.checkpoint.period</name> <value>3600</value> </property>
SecondaryNameNode 默认每隔一小时执行一次
查看默认配置：http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
2.以次数为准
<property> <name>dfs.namenode.checkpoint.txns</name> <value>10000</value> <description>作次数到达一万次就发起合并</description> </property> <property> <name>dfs.namenode.checkpoint.check.period</name> <value>600</value> <description>每间隔10分钟检查一次操作次数是否到达</description> </property>
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html