远程运行Hadoop作业 [英] Running Hadoop Job Remotely
问题描述
我尝试从集群外部运行MapReduce作业。
例如。 Hadoop集群正在Linux机器上运行。
我们有一个在Windows机器上运行的Web应用程序。
我们想从这个远程Web应用程序运行hadoop作业。
我们希望检索hadoop输出目录并将其显示为图形。
我们编写了以下代码片段:
Configuration conf = new Configuration();
工作职位=新职位(conf);
conf.set(mapred.job.tracker,192.168.56.101:54311);
conf.set(fs.default.name,hdfs://192.168.56.101:54310);
job.setJarByClass(Analysis.class);
//job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
//job.set
job.setInputFormatClass(CustomFileInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job,new Path(args [0]));
FileOutputFormat.setOutputPath(job,new Path(args [1]));
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.waitForCompletion(true);
这是我们得到的错误。即使我们关闭了hadoop 1.1.2集群,错误仍然是一样的。
14/03/07 00: 23:37 WARN util.NativeCodeLoader:无法为您的平台加载native-hadoop库......在适用的情况下使用builtin-java类
14/03/07 00:23:37 ERROR security.UserGroupInformation:PriviledgedActionException as:用户原因:java.io.IOException:无法设置路径的权限:\tmp\hadoop -user\mapred\staging\user818037780\.staging为0700
线程mainjava中的异常.io.IOException:未能在org.apache.hadoop.fs.FileUtil中设置路径的权限:\ tmp \hadoop-user\mapred\staging\user818037780\.staging为0700
。 checkReturnValue(FileUtil.java:691)
在org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
在org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem。 java:514)
在org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349 )
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
在org.apache.hadoop.mapred.JobClient $ 2.run(JobClient.java:942)
在org.apache.hadoop.mapred.JobClient $ 2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native方法)
位于javax.security.auth.Subject.doAs(Subject.java:396)
位于org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation .java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550 )
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at LineCounter.main(LineCounter.java:86)
从远程系统运行时,应该以远程用户身份运行。你可以在你的主类中做如下工作:
public static void main(String a []){
UserGroupInformation ugi
= UserGroupInformation.createRemoteUser(root);
try {
ugi.doAs(new PrivilegedExceptionAction< Void>(){
public void run()throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf);
conf.set(hadoop.job.ugi, root);
//在这里写下你的剩余代码
return null;
}
});
} catch(Exception e){
e.printStackTrace();
}
}
在提交mapreduce作业时,它应该将你的java类与它们的依赖jar包复制到hadoop集群,在那里它执行mapreduce作业。你可以阅读更多 here 。
所以你需要创建一个可运行的jar包你的代码(在你的情况下有主类分析)与清单类路径中的所有相关jar文件。然后使用命令行运行你的jar文件: java -jar job-jar -with-dependencies.jar arguments
HTH!
I am trying to run a MapReduce job from outside the cluster.
e.g. Hadoop cluster is running on Linux machines. We have one web application running on a Windows machine. We want to run the hadoop job from this remote web application. We want to retrieve the hadoop output directory and present it as a Graph.
We have written the following piece of code:
Configuration conf = new Configuration();
Job job = new Job(conf);
conf.set("mapred.job.tracker", "192.168.56.101:54311");
conf.set("fs.default.name", "hdfs://192.168.56.101:54310");
job.setJarByClass(Analysis.class) ;
//job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
//job.set
job.setInputFormatClass(CustomFileInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.waitForCompletion(true);
And this is the error we get. Even if we shut down the hadoop 1.1.2 cluster, the error is still the same.
14/03/07 00:23:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/03/07 00:23:37 ERROR security.UserGroupInformation: PriviledgedActionException as:user cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-user\mapred\staging\user818037780\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-user\mapred\staging\user818037780\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at LineCounter.main(LineCounter.java:86)
While running from a remote system, you should run as remote user. You can do it in your main class as follows:
public static void main(String a[]) {
UserGroupInformation ugi
= UserGroupInformation.createRemoteUser("root");
try {
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf);
conf.set("hadoop.job.ugi", "root");
// write your remaining piece of code here.
return null;
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
Also while submitting a mapreduce job, it should copy your java classes with their dependent jars to hadoop cluster, where it execute mapreduce job.You can read more here.
So you need to create a runnable jar of your code (with main class Analysis in your case) with all dependent jar files inits manifest classpath. Then run your jar file from your commandline using
java -jar job-jar-with-dependencies.jar arguments
HTH!
这篇关于远程运行Hadoop作业的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!