Hadoop 3:如何配置/启用擦除编码? [英] Hadoop 3 : how to configure / enable erasure coding?
问题描述
我正在尝试设置Hadoop 3集群.
I'm trying to setup an Hadoop 3 cluster.
有关删除编码功能的两个问题:
Two questions about the Erasure Coding feature :
- 如何确保启用擦除编码?
- 我仍然需要将复制因子设置为3吗?
请指出与擦除编码/复制相关的相关配置属性，以便获得与Hadoop 2相同的数据安全性(复制因子3)，但具有Hadoop 3擦除编码的磁盘空间优势(仅50％的开销，而不是200％).
Please indicate the relevant configuration properties related to erasure coding/replication, in order to get the same data security as Hadoop 2 (replication factor 3) but with the disk space benefits of Hadoop 3 erasure coding (only 50% overhead instead of 200%).
推荐答案
在Hadoop3中，我们可以对HDFS中的任何folder启用Erasure编码策略.默认情况下，在Hadoop3中未启用擦除编码，您可以使用setPolicy命令并指定所需的文件夹路径来启用它.
In Hadoop3 we can enable Erasure coding policy to any folder in HDFS. By default erasure coding is not enabled in Hadoop3, you can enable it by using setPolicy command with specifying desired path of folder.
1:要确保启用擦除编码，可以运行getPolicy命令.
1: To ensure erasure coding is enabled, you can run getPolicy command.
2:在Hadoop3中，复制因子设置将仅影响其他未由擦除代码setPolicy配置的文件夹.您可以在单个群集中同时使用Erasure编码和复制因子设置.
2: In Hadoop3 Replication factor setting will affect only to other folders which is not configured by erasure code setPolicy. You can use both Erasure coding and replication factor settings in single cluster.
列出支持的擦除策略的命令:
Command to List the supported erasure policies:
./bin/hdfs ec -listPolicies
启用XOR-2-1-1024k擦除策略的命令:
Command to Enable XOR-2-1-1024k Erasure policy:
./bin/hdfs ec -enablePolicy -policy XOR-2-1-1024k
将擦除策略设置为HDFS目录的命令:
Command to Set Erasure policy to HDFS directory:
./bin/hdfs ec -setPolicy -path /tmp -policy XOR-2-1-1024k
将策略设置为给定目录的命令:
Command to Get the policy set to the given directory:
./bin/hdfs ec -getPolicy -path /tmp
从目录中删除策略的命令.即未设置策略:
Command to Remove the policy from the directory.i.e unset policy:
./bin/hdfs ec -unsetPolicy -path /tmp
禁用策略命令:
./bin/hdfs ec -disablePolicy -policy XOR-2-1-1024k
这篇关于Hadoop 3:如何配置/启用擦除编码?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
