从oozie传递用户名和密码到sqoop meta连接 [英] pass username and password to sqoop meta connect from oozie

问题描述

<sqoop xmlns="uri:oozie:sqoop-action:0.3">
  <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
      <arg>job</arg>
      <arg>--meta-connect</arg>
      <arg>jdbc:mysql://FQDN:3306/sqoop</arg>
      <arg>--exec</arg>
      <arg>fabric_inventory</arg>
</sqoop>

现在，在此处传递--meta-connect的用户名和密码，

Now, to pass username and password for the --meta-connect here,

如果我在oozie.xml中按以下方式通过它:

if I pass it as following in oozie.xml:

<arg>jdbc:mysql://FQDN:3306/sqoop?user=sqoop&amp;password=sqoop</arg>

或

<arg>jdbc:mysql://FQDN:3306/sqoop?user=sqoop&password=sqoop</arg>

它给出以下异常:

遇到IOException正在运行导入作业:java.io.IOException:没有要为ClassWriter生成的列 2.它不需要

Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter 2. it does not take

--username and --password prop and val in arg tag.

如何以正确的方式传递值?

How do I pass the values in the correct manner?

这是完整异常的样子:

2016-05-11 10:21:44,920 ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
    at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
    at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
    at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

推荐答案

如果您曾经解决过Sqoop 1 和Sqoop 2 之间的混淆，那么您将不得不也跳过Oozie障碍.

If you ever sort out the confusion between Sqoop1 and Sqoop2, then you will have to jump over the Oozie hurdle too.

我的建议:停止使用命令行参数，而使用标准的Hadoop配置文件.

My advice: stop toying with command-line arguments and use standard Hadoop config files.

1..在网关节点上(用于单元测试)，编辑/etc/sqoop/conf/sqoop-site.xml以设置客户端属性

1. On your Gateway node (for unit tests), edit /etc/sqoop/conf/sqoop-site.xml to set Client properties

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <!-- some stuff here that does not matter for Metastore -->
  ...

  <!-- now the Metastore config -->
  <property>
    <name>sqoop.metastore.client.enable.autoconnect</name>
    <value>true</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.url</name>
    <value>jdbc:hsqldb:hsql://FQDN:16000/sqoop</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>sa</value>
  </property>
  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
  </property>
  <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>false</value>
  </property>
</configuration>

1b..将该文件上传到某处的HDFS(用于Oozie作业)

1b. Upload that file to HDFS somewhere (for use with Oozie jobs)

2..在将实际运行全局Metastore DB的节点上，还编辑文件，并添加两个额外的Server属性(在本示例中，数据库文件存储在/var/lib/...)

2. On the node that will actually run the global Metastore DB, also edit the file, and also add two extra Server properties (in this example the DB files are stored in /var/lib/...)

  <property>
     <name>sqoop.metastore.server.port</name> 
     <value>16000</value> 
  </property>
  <property>
     <name>sqoop.metastore.server.location</name> 
     <value>/var/lib/sqoop/data/shared.db</value> 
  </property>

2b..请确保您不时检查该数据库(以重置脚本"文件并刷新重做日志"文件)，然后备份脚本"文件作为当前数据库状态的快照，放在安全的地方，以防丢失节点及其磁盘-是的，这些事情会发生

2b. Make sure you CHECKPOINT that database from time to time (to reset the "script" file and flush the "redo log" file) then backup the "script" file as a snapshot of the current DB state, somewhere safe, in case you lose the node and its disk -- yes, these things happen

3..在您的Oozie Sqoop操作中，使用针对HDFS中配置文件的<job-xml>条目设置客户端属性.

3. In your Oozie Sqoop actions, set the Client properties with a <job-xml> entry targeting the config file in HDFS.

如果您对处理这些道具和Metastore客户端的实际Sqoop源代码感兴趣，请查看

If you are interested in the actual Sqoop source code that handles these props and the Metastore client, look there

这篇关于从oozie传递用户名和密码到sqoop meta连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！