Sqoop和Java 7 [英] Sqoop and Java 7
问题描述
我试图使用sqoop将一个MySQL表导入HDFS。我正在使用JDK 1.7.0_45和CDH4.4。我实际上使用了cloudera的预建VM，不过我将JDK更改为1.7，因为我想使用pydev插件进行eclipse。我的sqoop版本是1.4.3-cdh4.4.0。
当我运行sqoop时，出现以下异常：
错误：商品：不支持major.minor版本51.0
过去我看到过这个错误：
1.编译为java 7
2.用java 6运行应用程序。
但这不是我现在所做的。我相信我的sqoop版本被编译为java 6，而且我正在使用java 7来运行它，这应该是非常好的。我想可能hadoop正在用JDK 6启动映射程序，我不知道如何改变它。我浏览了mapred配置文档，并没有看到设置Java版本以用于map任务的任何方式。
以下是相关的控制台输出：
[cloudera @ localhost〜] $ echo $ JAVA_HOME
/ usr / java / latest
[cloudera @ localhost〜 ] $ java -version
java版本1.7.0_45
Java™SE运行时环境（build 1.7.0_45-b18）
Java HotSpot™64位服务器虚拟机构建24.45-b08，混合模式）
[cloudera @ localhost〜] $ sqoop版本
Sqoop 1.4.3-cdh4.4.0
git commit id 2cefe4939fd464ba11ef63e81f46bbaabf1f5bc6
由jenkins在星期二编译Sep 3 20:41:55 PDT 2013
[cloudera @ localhost〜] $ hadoop version
Hadoop 2.0.0-cdh4.4.0
Subversion file：/// data / 1 / jenkins / workspace / generic-package-rhel64-6-0 / topdir / BUILD / hadoop-2.0.0-cdh4.4.0 / src / hadoop-common-project / hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79
编译jenkins于星期二九月3 19:33:17 PDT 2013
来自智慧h checksum ac7e170aa709b3ace13dc5f775487180
此命令使用/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar
[cloudera @ localhost〜] $ cat mysqooper.sh
＃！/ bin / bash
sqoop import -m 1 --connect jdbc：mysql：// localhost / $ 1 \
--username root --table $ 2 --target-dir $ 3
[cloudera @ localhost〜] $ ./mysqooper.sh云商品/用户/ cloudera /商品/ csv / sqooped
14/01/16 16:45:10信息manager.MySQLManager：准备使用MySQL流式结果集。
14/01/16 16:45:10 INFO tool.CodeGenTool：开始代码生成
14/01/16 16:45:11 INFO manager.SqlManager：执行SQL语句：SELECT t。* FROM `商品`as t LIMIT 1
14/01/16 16:45:11 INFO manager.SqlManager：执行SQL语句：SELECT t。* FROM`commodity` as t LIMIT 1
14/01 / 16 16:45:11 INFO orm.CompilationManager：HADOOP_MAPRED_HOME是/usr/lib/hadoop-0.20-mapreduce
14/01/16 16:45:11信息orm.CompilationManager：找到hadoop核心jar：/ usr /lib/hadoop-0.20-mapreduce/hadoop-core.jar
注意：/tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.java使用或覆盖弃用的API。
注意：使用-Xlint：deprecation重新编译以获取详细信息。
14/01/16 16:45:15 INFO orm.CompilationManager：编写jar文件：/tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.jar
14/01/16 16:45： 15 WARN manager.MySQLManager：看起来你是从mysql导入的。
14/01/16 16:45:15 WARN manager.MySQLManager：这个传输可以更快！使用--direct
14/01/16 16:45:15 WARN manager.MySQLManager：选项来执行特定于MySQL的快速路径。
14/01/16 16:45:15信息manager.MySQLManager：将零DATETIME行为设置为convertToNull（mysql）
14/01/16 16:45:15信息mapreduce.ImportJobBase：开始导入商品
14/01/16 16:45:17警告mapred.JobClient：使用GenericOptionsParser解析参数。应用程序应该实现相同的工具。
14/01/16 16:45:20信息mapred.JobClient：正在运行的作业：job_201401161614_0001
14/01/16 16:45:21信息mapred.JobClient：map 0％reduce 0％
14/01/16 16:45:38信息mapred.JobClient：任务ID：attempt_201401161614_0001_m_000000_0，状态：FAILED
错误：商品：不受支持major.minor版本51.0
14/01/16 16： 45:46信息mapred.JobClient：任务ID：attempt_201401161614_0001_m_000000_1，状态：FAILED
错误：商品：不受支持的major.minor版本51.0
14/01/16 16:45:54信息mapred.JobClient：Task Id：attempt_201401161614_0001_m_000000_2，状态：FAILED
错误：商品：不受支持major.minor版本51.0
14/01/16 16:46:07信息mapred.JobClient：作业完成：job_201401161614_0001
14 / 01/16 16:46:07信息mapred.JobClient：计数器：6
14/01/16 16:46:07信息mapred.JobClient：作业计数器
14/01/16 16:46： 07信息mapred.JobClient：失败的地图任务= 1
14/01/16 16:46:07信息mapred.JobClient：启动的地图任务= 4
14/01/16 16:46:07信息mapred.JobClient：占用插槽中所有地图花费的总时间（ms）= 23048
14/01/16 16:46:07 INFO mapred。 JobClient：占用插槽中所有缩减花费的总时间（毫秒）= 0
14/01/16 16:46:07信息mapred.JobClient：预留插槽后等待的所有地图花费的总时间（毫秒）= 0
14/01/16 16:46:07信息mapred.JobClient：所有减少等待时间（ms）= 0
14/01/16 16:46:07警告mapreduce .Counters：组FileSystemCounters已弃用。使用org.apache.hadoop.mapreduce.FileSystemCounter而不是
14/01/16 16:46:07 INFO mapreduce.ImportJobBase：在51.0252秒（0字节/秒）内传输0个字节
14/01 / 16 16:46:07 WARN mapreduce.Counters：Group org.apache.hadoop.mapred.Task $ Counter已弃用。使用org.apache.hadoop.mapreduce.TaskCounter而不是
14/01/16 16:46:07 INFO mapreduce.ImportJobBase：检索到0条记录。
14/01/16 16:46:07错误tool.ImportTool：导入期间出错：导入作业失败！
我试着用JDK 1.6运行它，但它真的不想切换回每次我需要使用sqoop。
有人知道我需要改变吗？
我认为问题的根本原因在于您的Hadoop发行版仍然在JDK6上运行，而不是JDK7，因为您认为是这样。
Sqoop进程将生成使用当前使用的JDK编译的Java代码。因此，如果您在JDK7上执行Sqoop，它将使用此JDK7生成并编译代码。生成的代码随后会作为mapreduce作业的一部分提交给您的hadoop集群。因此，如果您在JDK7上运行Sqoop时收到不支持的major.minr异常，很可能您的Hadoop集群正在JDK6上运行。
我强烈建议在jinfo上调用您的hadoop deamons来验证它们正在运行的JDK。
I'm trying to use sqoop to import a MySQL table into HDFS. I'm using JDK 1.7.0_45 and CDH4.4. I'm actually using cloudera's pre-built VM, except I changed the JDK to 1.7 because I wanted to use the pydev plugin for eclipse. My sqoop version is 1.4.3-cdh4.4.0.
When I run sqoop I get this exception:
Error: commodity : Unsupported major.minor version 51.0
I have seen this error in the past when I did this: 1. compiled to java 7 2. ran an application with java 6.
but that is not what I am doing this time. I believe my sqoop version was compiled to java 6, and I'm running it with java 7, which should be perfectly fine. I think maybe hadoop is launching mapper processes with JDK 6, I have no idea how to change that. I skimmed through the mapred configuration documentation, and did not see any way to set the java version to use for map tasks.
Here is the relevant console output:
[cloudera@localhost ~]$ echo $JAVA_HOME
/usr/java/latest
[cloudera@localhost ~]$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
[cloudera@localhost ~]$ sqoop version
Sqoop 1.4.3-cdh4.4.0
git commit id 2cefe4939fd464ba11ef63e81f46bbaabf1f5bc6
Compiled by jenkins on Tue Sep 3 20:41:55 PDT 2013
[cloudera@localhost ~]$ hadoop version
Hadoop 2.0.0-cdh4.4.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79
Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013
From source with checksum ac7e170aa709b3ace13dc5f775487180
This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar
[cloudera@localhost ~]$ cat mysqooper.sh
#!/bin/bash
sqoop import -m 1 --connect jdbc:mysql://localhost/$1 \
--username root --table $2 --target-dir $3
[cloudera@localhost ~]$ ./mysqooper.sh cloud commodity /user/cloudera/commodity/csv/sqooped
14/01/16 16:45:10 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/01/16 16:45:10 INFO tool.CodeGenTool: Beginning code generation
14/01/16 16:45:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `commodity` AS t LIMIT 1
14/01/16 16:45:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `commodity` AS t LIMIT 1
14/01/16 16:45:11 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-0.20-mapreduce
14/01/16 16:45:11 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar
Note: /tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/01/16 16:45:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/f75bf6f8829e8eff302db41b01f6796a/commodity.jar
14/01/16 16:45:15 WARN manager.MySQLManager: It looks like you are importing from mysql.
14/01/16 16:45:15 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
14/01/16 16:45:15 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
14/01/16 16:45:15 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
14/01/16 16:45:15 INFO mapreduce.ImportJobBase: Beginning import of commodity
14/01/16 16:45:17 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/01/16 16:45:20 INFO mapred.JobClient: Running job: job_201401161614_0001
14/01/16 16:45:21 INFO mapred.JobClient: map 0% reduce 0%
14/01/16 16:45:38 INFO mapred.JobClient: Task Id : attempt_201401161614_0001_m_000000_0, Status : FAILED
Error: commodity : Unsupported major.minor version 51.0
14/01/16 16:45:46 INFO mapred.JobClient: Task Id : attempt_201401161614_0001_m_000000_1, Status : FAILED
Error: commodity : Unsupported major.minor version 51.0
14/01/16 16:45:54 INFO mapred.JobClient: Task Id : attempt_201401161614_0001_m_000000_2, Status : FAILED
Error: commodity : Unsupported major.minor version 51.0
14/01/16 16:46:07 INFO mapred.JobClient: Job complete: job_201401161614_0001
14/01/16 16:46:07 INFO mapred.JobClient: Counters: 6
14/01/16 16:46:07 INFO mapred.JobClient: Job Counters
14/01/16 16:46:07 INFO mapred.JobClient: Failed map tasks=1
14/01/16 16:46:07 INFO mapred.JobClient: Launched map tasks=4
14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=23048
14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/01/16 16:46:07 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/01/16 16:46:07 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
14/01/16 16:46:07 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 51.0252 seconds (0 bytes/sec)
14/01/16 16:46:07 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
14/01/16 16:46:07 INFO mapreduce.ImportJobBase: Retrieved 0 records.
14/01/16 16:46:07 ERROR tool.ImportTool: Error during import: Import job failed!
I tried running with JDK 1.6 and it works, but I really don't want to switch back to that every time I need to use sqoop.
Does anybody know what I need to change?
I belive that root cause of your problem is that your Hadoop distribution is still running on JDK6 and not JDK7 as you believe so.
Sqoop process will generate Java code that is compiled with currently used JDK. Therefore if you execute Sqoop on JDK7, it will generate and compile code with this JDK7. The generated code is then submitted to your hadoop cluster as a part of mapreduce job. Therefore if you are getting this unsupported major.minr exception while running Sqoop on JDK7 is very likely that your Hadoop cluster is running on JDK6.
I would strongly suggest calling jinfo on your hadoop deamons to verify which JDK they are running on.
这篇关于Sqoop和Java 7的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
