将参数传递给sqoop作业 [英] Passing parameter to sqoop job

问题描述

我想根据Date将增量加载到配置单元中一个参数，将被传递给sqoop作业

经过研究，我无法找到一种方法将参数传递给Sqoop作业

解决方案

您可以通过将日期传递给两个阶段来完成此操作： li>协调员到工作流程

在您的协调员中，您可以将日期作为 < property> ，如下所示：

 < coordinator-app name =schedulefrequency =$ {coord：days（1）}
 start =2015-01-01T00：00Zend =2025-01-01T00：00Z
 timezone =Etc / UTCxmlns = URI：Oozie的：协调器：0.2 > 
 ... 
 <工作流程> 
< app-path> $ {nameNode} /your/workflow.xml</app-path> 
<配置> 
<属性> 
<名称> workflow_date< / name> 
< value> $ {coord：formatTime（coord：nominalTime（），'yyyyMMdd'）}< / value> 
< / property> 
< / configuration> 
< / workflow> 
< / action> 
 ... 
< / coordinator-app> 
  

2. 工作流到Sqoop

在您的工作流中，您可以使用 $ {workflow_date} 变量在您的Sqoop调用中引用该属性，像这样：

 < sqoop xmlns =uri：oozie：sqoop-action：0.2> 
 ... 
< command> import --connect jdbc：connect：string：here --table tablename --target-dir / your / import / dir / $ {workflow_date} / -m 1< ; /命令> 
 ... 
< / sqoop> 
  

I'm crceating a sqoop job which will be scheduled in Oozie to load daily data into Hive.

I want to do incremental load into hive based on Date as a parameter, which will be passed to sqoop job

After researching lot I'm unable to find a way to pass a parameter to Sqoop job

解决方案

You do this by passing the date down through two stages:

1. Coordinator to workflow

In your coordinator you can pass the date to the workflow that it executes as a <property>, like this:

<coordinator-app name="schedule" frequency="${coord:days(1)}"
                 start="2015-01-01T00:00Z" end="2025-01-01T00:00Z"
                 timezone="Etc/UTC" xmlns="uri:oozie:coordinator:0.2">
    ...
    <action>
        <workflow>
            <app-path>${nameNode}/your/workflow.xml</app-path>
            <configuration>
                <property>
                    <name>workflow_date</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'yyyyMMdd')}</value>
                </property>
            </configuration>
        </workflow>
    </action>
    ...
</coordinator-app>

2. Workflow to Sqoop

In your workflow you can reference that property in your Sqoop call using the ${workflow_date} variable, like this:

<sqoop xmlns="uri:oozie:sqoop-action:0.2">
    ...
    <command>import --connect jdbc:connect:string:here --table tablename --target-dir /your/import/dir/${workflow_date}/ -m 1</command>
    ...
</sqoop>

这篇关于将参数传递给sqoop作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！