清除HDFS后，某些数据节点仍显示使用的块池 [英] Some datanodes still showing block pool used after clearing the HDFS

问题描述

Hadoop版本:2.7.3数据节点:32面积:512m复制:3

Hadoop Version: 2.7.3 Datanodes: 32 BlockSize: 512m Replication: 3

我使用以下命令清除了HDFS中的所有数据hdfs dfs -rm -r/*

I Cleared all the data in HDFS using the command hdfs dfs -rm -r /*

清除HDFS群集后，尽管块为零，但很少有数据节点仍显示已使用的块池.

After clearing the HDFS cluster, few datanodes still showing the Block pool used, though blocks are zero.

hdfs webui的屏幕截图

The screen shot of hdfs webui

hdfs fsck/的输出如下

The output of hdfs fsck / is as follows

Connecting to namenode via http://ip-10-0-2-7:50070/fsck?ugi=ubuntu&path=%2F
FSCK started by ubuntu (auth:SIMPLE) from /10.0.2.7 for path / at Tue Jul 18 04:34:19 UTC 2017
Status: HEALTHY
 Total size:    0 B
 Total dirs:    1
 Total files:   0
 Total symlinks:        0
 Total blocks (validated):  0
 Minimally replicated blocks:   0
 Over-replicated blocks:    0
 Under-replicated blocks:   0
 Mis-replicated blocks:     0
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        0
 Missing replicas:      0
 Number of data-nodes:      32
 Number of racks:       1
FSCK ended at Tue Jul 18 04:34:19 UTC 2017 in 1 milliseconds
The filesystem under path '/' is HEALTHY

我知道块删除是异步过程，但是具有 Block Pool Used 的数据节点未设置为0，这需要大量时间来释放块.有人可以帮我弄清楚为什么在这些服务器上块删除速度很慢，或者由于使用这些块而导致的其他任何问题.

I know the block deletion is asynchronous process, but the datanodes which are having the Block Pool Used not set to 0, taking a lot of time to release the blocks. Can anybody help me figure out, why on those servers, block deletion is slow, or any other issue because of which the blocks are being used.

推荐答案

有以下几种潜在原因:

1. 最近使用 HDFS滚动升级功能，并且升级尚未完成.在滚动升级期间，文件删除实际上不会删除DataNode上的基础块文件.相反，将保留阻止文件，以便如果操作员选择回滚升级，则可以将群集的数据还原回升级前的状态.因此，在滚动升级窗口期间，DataNode指标(例如已使用的块池")继续显示消耗的空间.检查是否正在进行滚动升级的一种简单方法是转到NameNode Web UI.概述选项卡将显示有关滚动升级已开始"的消息.要完成升级，请运行 hdfs dfsadmin -rollingUpgrade .之后，DataNodes将开始(异步)删除块文件.
2. 集群的用户使用了 HDFS快照.创建快照时，快照会维护文件系统路径的状态.为此，如果用户选择读取包含数据的旧快照，DataNode可能需要保留先前删除文件的块.使用 -includeSnapshots 参数运行 hdfs fsck 可以显示是否正在使用快照.
3. 由于某种I/O错误(磁盘故障，基础块文件的权限冲突)，DataNode无法删除块文件.这不太可能，但是如果出现问题，则DataNode日志将显示更多信息.

1. The cluster software was upgraded recently using the HDFS Rolling Upgrade feature, and the upgrade has not been finalized. During a rolling upgrade, file deletions do not actually delete the underlying block files at the DataNode. Instead, the block files are maintained, so that if the operator chooses to roll back the upgrade, the cluster's data can be reverted back to pre-upgrade state. As a consequence, during the rolling upgrade window, DataNode metrics such as "Block pool used" continue to show space consumed. An easy way to check if a rolling upgrade is in progress is to go to the NameNode web UI. The Overview tab will show a message about "Rolling upgrade started". To finalize the upgrade, run hdfs dfsadmin -rollingUpgrade. Afterwards, DataNodes will start deleting block files (asynchronously).
2. A user of the cluster made use of HDFS Snapshots. Snapshots maintain the state of a file system path at the moment the snapshot was created. To support this, DataNodes may need to keep blocks for previously deleted files, in case a user chooses to read an old snapshot containing the data. Running hdfs fsck with the -includeSnapshots argument can show if there are snapshots in use.
3. Due to some kind of I/O error (failed disk, permissions violation on underlying block files), the DataNode is failing to delete the block files. This is unlikely, but if there is a problem, then the DataNode logs would show more information.

这篇关于清除HDFS后，某些数据节点仍显示使用的块池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！