无法删除HDFS损坏的文件 [英] Unable to delete HDFS Corrupt files

问题描述

我无法删除HDFS中存在的损坏文件. Namenode已进入安全模式.块总数为980，其中已报告978.当我运行以下命令时，

I am unable to delete corrupt files present in my HDFS. Namenode has run into Safe mode. Total number of blocks are 980, out of which 978 have reported. When I run the following command,

sudo -u hdfs hdfs dfsadmin -report

生成的报告是

Safe mode is ON
Configured Capacity: 58531520512 (54.51 GB)
Present Capacity: 35774078976 (33.32 GB)
DFS Remaining: 32374509568 (30.15 GB)
DFS Used: 3399569408 (3.17 GB)
DFS Used%: 9.50%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (1):

Name: 10.0.2.15:50010 (quickstart.cloudera)
Hostname: quickstart.cloudera
Decommission Status : Normal
Configured Capacity: 58531520512 (54.51 GB)
DFS Used: 3399569408 (3.17 GB)
Non DFS Used: 19777388544 (18.42 GB)
DFS Remaining: 32374509568 (30.15 GB)
DFS Used%: 5.81%
DFS Remaining%: 55.31%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 14 10:39:58 IST 2017

执行以下命令时，

sudo -u hdfs hdfs fsck /

输出为

Connecting to namenode via http://quickstart.cloudera:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /10.0.2.15 for path / at Tue Nov 14 10:41:25 IST 2017
/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728: CORRUPT blockpool BP-1914853243-127.0.0.1-1500467607052 block blk_1073743141

/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728: MISSING 1 blocks of total size 83 B..
/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta: CORRUPT blockpool BP-1914853243-127.0.0.1-1500467607052 block blk_1073743142

/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta: MISSING 1 blocks of total size 83 B
Status: CORRUPT
Total size: 3368384392 B (Total open files size: 166 B)
Total dirs: 286
Total files:    966
Total symlinks:     0 (Files currently being written: 3)
Total blocks (validated):   980 (avg. block size 3437126 B) (Total open file blocks (not validated): 2)
********************************
UNDER MIN REPL'D BLOCKS:    2 (0.20408164 %)
dfs.namenode.replication.min:   1
CORRUPT FILES:  2
MISSING BLOCKS: 2
MISSING SIZE:       166 B
CORRUPT BLOCKS:     2
********************************
Minimally replicated blocks:    978 (99.79592 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks:    0 (0.0 %)
Mis-replicated blocks:      0 (0.0 %)
Default replication factor: 1
Average block replication:  0.9979592
Corrupt blocks:     2
Missing replicas:       0 (0.0 %)
Number of data-nodes:       1
Number of racks:        1
FSCK ended at Tue Nov 14 10:41:26 IST 2017 in 774 milliseconds
The filesystem under path '/' is CORRUPT

任何人都可以帮助纠正损坏的块，(或)将其删除吗?预先感谢.

Can anyone please help in either correcting the corrupted blocks, (or) deleting them? Thanks in advance.

推荐答案

据说Namenode处于安全模式，请先将其关闭.

As it's said that Namenode is in Safe mode, first turn it off.

hdfs dfsadmin -safemode leave

然后执行任一命令

hdfs fsck / | egrep -v '^\.+$' | grep -v replica | grep -v Replica

或

hdfs fsck hdfs://quickstart.cloudera:50070/ | egrep -v '^\.+$' | grep -v replica | grep -v Replica

输出将类似于

/path/to/filename.fileextension: CORRUPT blockpool BP-1016133662-10.29.100.41-1415825958975 block blk_1073904305

/path/to/filename.fileextension: MISSING 1 blocks of total size 15620361 B

在您的情况下，已列出损坏的文件.因此，请执行以下命令

In your case corrupted files are already listed. So execute below commands

hdfs dfs -rm /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728
hdfs dfs -rm /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta

也不要进入安全模式.只是继续工作. 呀！

And don't enter into Safemode. Just continue working. Yippee!!

这篇关于无法删除HDFS损坏的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！