通过我的PHP网页运行用Java编写的MapReduce作业 [英] Running MapReduce job written in Java through my PHP web page

问题描述

我的PHP服务器托管在Job Tracker机器上，我试图通过调用执行jar命令的命令行来通过我的网页运行map reduce作业

My PHP server is hosted on Job Tracker machine and I am trying to run the map reduce job through my web page by calling the command line executing the jar command,

但是我没有任何回应，工作也没有开始.

but I am getting no response and job is not starting.

但是，如果我使用相同的方法运行命令以列出hdfs，则运行良好.请引导我.

However if I run a command to list the hdfs using same methodology it is running fine. Please guide me.

以下命令对我没有任何响应，并且作业未运行:

Following command is not responding me anything and job is not running:

exec("HADOOP_DIR/bin/hadoop jar /usr/local/MapReduce.jar Mapreduce [input Path] [output Path]");

但是，如果我这样做:

exec("HADOOP_DIR/bin/hadoop dfs -ls /user/hadoop");

运行正常.

推荐答案

我通过将php服务器用户更改为hduser(有权在hdfs中写入文件的用户)解决了此问题.在不更改该用户的情况下，只有从hdfs读取的命令有效，而无需创建文件或在hdfs上写入的命令有效.

I solved this problem by changing the php server user to hduser (user which has permission to write files in hdfs). without changing this user only the commands which reads from the hdfs were working and not the one which needs to create the files or write on hdfs.

当我尝试通过php脚本运行用于在hdfs中创建目录的命令时，我的php服务器日志(/var/log/apache2/error.log)中出现以下错误:

When i tried to run the command for creating the directory in hdfs through my php script, I got the following error in my php server logs (/var/log/apache2/error.log):

mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: user=www-data, access=WRITE, inode="hduser":hduser:supergroup:rwxr-xr-x

在运行Jar命令来触发MapRed程序时，出现以下错误:

And on running the Jar command to trigger MapRed program I got the following error:

Exception in thread "main" java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:1879)
at org.apache.hadoop.util.RunJar.main(RunJar.java:115)

然后我所做的是将/etc/apache2/apache2.conf中的用户更改为我的hadoop用户，然后重新启动服务器，一切正常.

Then what i did is i changed the user in /etc/apache2/apache2.conf to my hadoop user and then restarted my server and every thing was working fine now.

我应该引用从PHP Server执行hadoop jar失败.权限被拒绝帖子，对我解决这个问题有很大帮助.我希望这篇文章也能对其他人有所帮助.

I should reference Execute hadoop jar from PHP Server fails. Permission denied post which helped me alot in solving this problem. I hope this post helps others too.

这篇关于通过我的PHP网页运行用Java编写的MapReduce作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！