Apache Pig:无法运行我自己的 pig.jar 和 pig-withouthadoop.jar [英] Apache Pig: unable to run my own pig.jar and pig-withouthadoop.jar

问题描述

我有一个运行 Hadoop 0.20.2 和 Pig 0.10 的集群.我有兴趣在 Pig 的源代码中添加一些日志并在集群上运行我自己的 Pig 版本.

I have a cluster running Hadoop 0.20.2 and Pig 0.10. I'm interested to add some logs to Pig's source code and to run my own Pig version on the cluster.

我做了什么:

1. 使用ant"命令构建项目

1. built the project with 'ant' command

得到 pig.jar 和 pig-withouthadoop.jar

got pig.jar and pig-withouthadoop.jar

将 jars 复制到集群名称节点上的 Pig 主目录

copied the jars to Pig home directory on the cluster's namenode

运行作业

然后我有以下标准输出:

Then I've got following std output:

2013-03-25 06:35:05,226 [main] WARN  org.apache.pig.backend.hadoop20.PigJobControl -   falling back to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
    at java.lang.Class.getDeclaredField(Class.java:1882)
    at org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
    at  org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:97)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:287)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1320)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1305)
    at org.apache.pig.PigServer.execute(PigServer.java:1295)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:480)
    at org.apache.pig.Main.main(Main.java:157)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
2013-03-25 06:35:05,229 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-03-25 06:35:05,260 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-03-25 06:35:05,272 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2013-03-25 06:35:06,041 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job9091543475518322185.jar
2013-03-25 06:35:10,974 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job9091543475518322185.jar created
2013-03-25 06:35:10,995 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-03-25 06:35:11,006 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2013-03-25 06:35:11,006 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2013-03-25 06:35:11,006 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2013-03-25 06:35:11,181 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;

猪堆栈跟踪:

ERROR 2998: Unhandled internal error. org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;

java.lang.NoSuchMethodError: org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:298)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1320)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1305)
    at org.apache.pig.PigServer.execute(PigServer.java:1295)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
    at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
    at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:480)
    at org.apache.pig.Main.main(Main.java:157)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

出了什么问题?除了替换namenode安装目录下的pig.jar和pig-withouthadoop.jar，我还需要做些什么吗?

what went wrong? Should I do anything else except replacing pig.jar and pig-withouthadoop.jar in installation directory of namenode?

帮助...

推荐答案

我错过的一点是:pig-withouthadoop.jar 应该使用特定的 Hadoop 版本编译.我按照以下方式编译了 jar 并且它工作正常:

the point I missed was: pig-withouthadoop.jar should be compiled with specific Hadoop version. I compiled the jar in following way and it worked:

% ant clean jar-withouthadoop -Dhadoopversion=23

这篇关于Apache Pig:无法运行我自己的 pig.jar 和 pig-withouthadoop.jar的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！