如何修复损坏的 HDFS 文件 [英] How to fix corrupt HDFS FIles

问题描述

有人如何修复损坏的 HDFS?我查看了 Apache/Hadoop 网站，它说它的 fsck 命令，它没有修复它.希望之前遇到过这个问题的人可以告诉我如何解决这个问题.

How does someone fix a HDFS that's corrupt? I looked on the Apache/Hadoop website and it said its fsck command, which doesn't fix it. Hopefully someone who has run into this problem before can tell me how to fix this.

与用于本机文件系统的传统 fsck 实用程序不同，此命令不会纠正它检测到的错误.通常 NameNode 会自动纠正大部分可恢复的故障.

Unlike a traditional fsck utility for native file systems, this command does not correct the errors it detects. Normally NameNode automatically corrects most of the recoverable failures.

当我运行 bin/hadoop fsck/-delete 时，它列出了损坏或丢失块的文件.我如何使它不腐败?这是在练习机上，所以我可以把所有东西都吹掉，但是当我们上线时，我将无法通过吹掉所有东西来修复"它，所以我现在想弄清楚.

When I ran bin/hadoop fsck / -delete, it listed the files that were corrupt or missing blocks. How do I make it not corrupt? This is on a practice machine so I COULD blow everything away but when we go live, I won't be able to "fix" it by blowing everything away so I'm trying to figure it out now.

推荐答案

您可以使用

  hdfs fsck /

以确定哪些文件有问题.查看输出中是否有丢失或损坏的块(暂时忽略复制不足的块).这个命令真的详细，尤其是在大型 HDFS 文件系统上，所以我通常会深入了解有意义的输出与

to determine which files are having problems. Look through the output for missing or corrupt blocks (ignore under-replicated blocks for now). This command is really verbose especially on a large HDFS filesystem so I normally get down to the meaningful output with

  hdfs fsck / | egrep -v '^.+$' | grep -v eplica

它忽略了除了点和线之外什么都没有谈论复制的行.

which ignores lines with nothing but dots and lines talking about replication.

一旦发现文件损坏

  hdfs fsck /path/to/corrupt/file -locations -blocks -files

使用该输出来确定块可能存在的位置.如果文件是大于您的块大小，它可能有多个块.

Use that output to determine where blocks might live. If the file is larger than your block size it might have multiple blocks.

您可以使用报告的块编号来访问搜索机器或机器的数据节点和名称节点日志街区居住的地方.尝试查找文件系统错误在那些机器上.缺少挂载点，数据节点未运行，重新格式化/重新配置文件系统.如果你能发现问题以这种方式并将块重新联机，该文件将是健康的再次.

You can use the reported block numbers to go around to the datanodes and the namenode logs searching for the machine or machines on which the blocks lived. Try looking for filesystem errors on those machines. Missing mount points, datanode not running, file system reformatted/reprovisioned. If you can find a problem in that way and bring the block back online that file will be healthy again.

泡沫冲洗并重复，直到所有文件都健康或耗尽寻找块的所有替代方案.

Lather rinse and repeat until all files are healthy or you exhaust all alternatives looking for the blocks.

一旦你确定发生了什么并且你不能再恢复任何块，只需使用

Once you determine what happened and you cannot recover any more blocks, just use the

  hdfs fs -rm /path/to/file/with/permanently/missing/blocks

命令让你的 HDFS 文件系统恢复健康，这样你就可以开始了跟踪出现的新错误.

command to get your HDFS filesystem back to healthy so you can start tracking new errors as they occur.

这篇关于如何修复损坏的 HDFS 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！