通过Oozie HDP 2.1工作不创建job.splitmetainfo [英] Job via Oozie HDP 2.1 not creating job.splitmetainfo

问题描述

当试图执行一个sqoop作业时，我的Hadoop程序作为一个jar文件在 -jarFiles 参数中传递，执行过程中出现以下错误。任何解决方案似乎都不可用。

  org.apache.hadoop.yarn.exceptions.YarnRuntimeException：java.io .FileNotFoundException：文件不存在：hdfs：//sandbox.hortonworks.com：8020 / user / root / .staging / job_1423050964699_0003 / job.splitmetainfo 
 at org.apache.hadoop.mapreduce.v2.app.job .impl.JobImpl $ InitTransition.createSplits（JobImpl.java:1541）
 at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl $ InitTransition.transition（JobImpl.java:1396）
 at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl $ InitTransition.transition（JobImpl.java:1363）
 at org.apache.hadoop.yarn.state.StateMachineFactory $ MultipleInternalArc。 doTransition（StateMachineFactory.java:385）
位于org.apache.hadoop.yarn.state.StateMachineFactory.doTransition（StateMachineFactory.java:302）
位于org.apache.hadoop.yarn.state.StateMachineFactory。在org.apache.hadoop.yarn.state.Sta访问$ 300（StateMachineFactory.java:46）
 teMachineFactory $ InternalStateMachine.doTransition（StateMachineFactory.java:448）
 at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle（JobImpl.java:976）
 at org。 apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle（JobImpl.java:135）
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster $ JobEventDispatcher.handle（MRAppMaster.java ：1241）
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart（MRAppMaster.java:1041）
 at org.apache.hadoop.service.AbstractService.start（AbstractService.java ：193）
 at org.apache.hadoop.mapreduce.v2.app.MRAppMaster $ 1.run（MRAppMaster.java:1452）
 at java.security.AccessController.doPrivileged（Native Method）
在javax.security.auth.Subject.doAs（Subject.java:415）
在org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1548）
在org.apache。 hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster（MRAppMaster.java:1448）
在org.apache.hadoop .mapreduce.v2.app.MRAppMaster.main（MRAppMaster.java:1381）
  

解决方案

这就是我解决问题的方法。我们使用CDH5来运行加缪以从卡夫卡提取数据。我们运行CamusJob，它负责使用comman行从kafka获取数据：

  hadoop jar ... 
  code> 

问题是新主机没有得到所谓的纱线网关。 Cloudera命名与服务相关的一组配置，并将其复制到 / etc / hadoop / conf
中作为网关。所以我在CM UI中点击了 deploy client configuration 。 YARN客户端conf已被复制到每个YARN NodeManager节点，并解决了问题。


When trying to execute a sqoop job which has my Hadoop program passed as a jar file in -jarFiles parameter, the execution blows off with below error. Any resolution seems to be not available. Other jobs with same Hadoop user is getting executed successfully.

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://sandbox.hortonworks.com:8020/user/root/.staging/job_1423050964699_0003/job.splitmetainfo
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1541)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1396)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1363)
    at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:976)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:135)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1241)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1041)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1452)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1448)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1381)

解决方案

So here is the way I solved it. We are using CDH5 to run Camus to pull data from kafka. We run CamusJob which is responsible for getting data from kafka using comman line:

hadoop jar...

The problem is that new hosts didn't get so-called "yarn-gateway". Cloudera names pack of configs related to service and copied to /etc/hadoop/conf as "gateway". So I just clicked "deploy client configuration" in CM UI. YARN client conf has been copied to each YARN NodeManager node and it solved problem.

这篇关于通过Oozie HDP 2.1工作不创建job.splitmetainfo的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！