自动加载基于HADOOP_HOME的HDFS配置？ [英] Automatically loading HDFS Configuration based on HADOOP_HOME?

问题描述

我正在开发一个Java程序来与已经运行的hadoop集群进行交互。该程序将 HADOOP_HOME 作为环境变量传递给它。

I am working on a Java program to interface with an already running hadoop cluster. The program has HADOOP_HOME passed to it as an environment variable.

基于此值，我需要加载所有在开始与HDFS / MapReduce交互之前，必要的配置资源。我认为我需要的文件基于 apache文档。我目前的解决方案如下所示：

Based on this value, I need to load all of the necessary configuration resources before I start interacting with HDFS/MapReduce. The files that I think I need are based on the apache documentation. My current solution looks like this:

final String HADOOP_HOME = System.getEnv("HADOOP_HOME");
Configuration conf = new Configuration();

conf.addResource(new Path(HADOOP_HOME, "src/core/core-default.xml"));
conf.addResource(new Path(HADOOP_HOME, "src/hdfs/hdfs-default.xml"));
conf.addResource(new Path(HADOOP_HOME, "src/mapred/mapred-default.xml"));
conf.addResource(new Path(HADOOP_HOME, "conf/core-site.xml"));
conf.addResource(new Path(HADOOP_HOME, "conf/hdfs-site.xml"));
conf.addResource(new Path(HADOOP_HOME, "conf/mapred-site.xml"));

FileSystem hdfs = new FileSystem(conf);

有没有更简单的方法可以做到这一点？希望有一种不涉及显式设置每个资源的方法？

Is there a cleaner way to do this? Hopefully a way that does not involve setting each resource explicitly?

推荐答案

使用 hadoop jar <您-罐子> 。这会自动设置所有内容。

You run your jar using hadoop jar <your-jar>. This sets up everything automatically.

这篇关于自动加载基于HADOOP_HOME的HDFS配置？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！