使用Oozie的移动文件在S3 [英] Move files in S3 using oozie

问题描述

我想使用AWS了Oozie在S3移动文件。我想运行

I want to move files in S3 using AWS oozie. I want to run

aws s3 mv s3://temp/*.zip s3://temp/processed_files/. --recursive

我怎么能在Oozie的做到这一点？

How I can do this in oozie?

2015-11-12 10:18:55,758  WARN ShellActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] Launcher exception: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:93)
    at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:55)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
    at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:65)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:452)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 17 more

2015-11-12 10:18:55,838  INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] ERROR is considered as FAILED for SLA
2015-11-12 10:18:55,880  INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@killemail] Start action [0000118-151029144311676-oozie-oozi-W@killemail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

截屏色调Oozie的屏幕截图和错误日志。

Screen shot of Hue Oozie screen shot and error log.

推荐答案

编写一个Java类移动在S3中的文件并执行它作为一个Java行动Oozie的。

Write a java class to move files in s3 and implement it as a java action in oozie.

这篇关于使用Oozie的移动文件在S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！