Couchbase / hadoop连接器：sqoop作业失败“Found interface org.apache.hadoop.mapreduce.TaskAttemptContext，but class was expected” [英] Couchbase/hadoop connector: sqoop job FAILED "Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected"
问题描述
我的配置：
- CouchBase server 2.0
- Sqoop 1.4.2（为hadoop编译版本2.0.0）
- Stack Hadoop CDH4.1.2
我想使用CouchBase / Hadoop连接器（ http://www.couchbase.com/develop/connectors/hadoop ），但是当我启动一个导入时，我有以下错误：
$ b $ pre $ sqoop-import --connect http：// localhost： 8091 / pools --table DUMP
13/01/07 11:00:34 INFO tool.CodeGenTool：开始代码生成
13/01/07 11:00:34 INFO orm.CompilationManager：HADOOP_HOME is / usr / lib / hadoop
注意：/tmp/sqoop-agent/compile/fccb413cf9eec92f9a716320cf268a0a/DUMP.java使用或覆盖弃用的API。
注意：使用-Xlint：deprecation重新编译以获取详细信息。
13/01/07 11:00:42 INFO orm.CompilationManager：编写jar文件：/tmp/sqoop-agent/compile/fccb413cf9eec92f9a716320cf268a0a/DUMP.jar
13/01/07 11:00： 42 INFO mapreduce.ImportJobBase：开始导入DUMP
13/01/07 11:00:45 WARN util.Jars：没有这样的类couchbase不使用可用的jdbc驱动程序。
13/01/07 11:00:50警告mapred.JobClient：使用GenericOptionsParser解析参数。应用程序应该实现相同的工具。
2013-01-07 11：00：55.065 INFO net.spy.memcached.MemcachedConnection：添加了{QA sa = / 127.0.0.1：11210，＃Rops = 0，＃Wops = 0，＃iq = 0， topRop = null，topWop = null，toWrite = 0，interested = 0}连接队列
2013-01-07 11：00：55.098 INFO net.spy.memcached.MemcachedConnection：为sun.nio更改了连接状态。 ch.SelectionKeyImpl@41e22632
2013-01-07 11：00：56.266 INFO net.spy.memcached.MemcachedConnection：关闭memcached客户端
13/01/07 11:02:39信息mapred.JobClient ：正在运行的作业：job_201212111702_0131
13/01/07 11:02:40信息mapred.JobClient：map 0％reduce 0％
13/01/07 11:02:59信息mapred.JobClient：Task Id：attempt_201212111702_0131_m_000001_0，状态：FAILED
错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但期望类
13/01/07 11:03:02信息mapred.JobClient：Task Id ：attempt_201212111702_0131_m_000002_0，状态：FAILED
错误：找到的接口org.apache.hadoop.mapreduce.TaskAttemptContext，但类是exp ected
13/01/07 11:03:02信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000000_0，状态：FAILED
错误：找到的接口org.apache.hadoop.mapreduce.TaskAttemptContext，但类是预期的
13/01/07 11:03:09信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000003_0，状态：失败
错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但期望类为
13/01/07 11:03:12信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000001_1，状态：失败
错误：已找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但期望类为
13/01/07 11:03:13信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000002_1，状态：失败
错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但期望类
13/01/07 11:03:20信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000000_1，状态：FAILED
错误：找到的接口org.apache.hadoop.ma preduce.TaskAttemptContext，但期望类
13/01/07 11:03:24信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000003_1，状态：失败
错误：找到的接口org.apache.hadoop.mapreduce .TaskAttemptContext，但期望类
13/01/07 11:03:24信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000001_2，状态：FAILED
错误：找到接口org.apache.hadoop.mapreduce。 TaskAttemptContext，但期望类
13/01/07 11:03:30信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000002_2，状态：FAILED
错误：找到的接口org.apache.hadoop.mapreduce.TaskAttemptContext ，但类预计
13/01/07 11:03:33信息mapred.JobClient：任务ID：attempt_201212111702_0131_m_000000_2，状态：失败
错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但班级预期
13/01/07 11:03:39信息mapred.JobClient：工作完成：job_201212111702_0131
13/01/07 11:03:40信息地图red.JobClient：计数器：7
13/01/07 11:03:40信息mapred.JobClient：作业计数器
13/01/07 11:03:40信息mapred.JobClient：地图任务失败= 1
13/01/07 11:03:40信息mapred.JobClient：启动地图任务= 14
13/01/07 11:03:40信息mapred.JobClient：机架本地地图任务= 14
13/01/07 11:03:40信息mapred.JobClient：所有地图在占用插槽中花费的总时间（毫秒）= 74621
13/01/07 11:03:40信息mapred.JobClient：占用时隙所有缩减花费的总时间（ms）= 0
13/01/07 11:03:40信息mapred.JobClient：预留槽后等待的所有地图花费的总时间（毫秒） = 0
13/01/07 11:03:40信息mapred.JobClient：所有花费的时间都减少了预留插槽后的等待时间（毫秒）= 0
13/01/07 11:03:40 INFO mapreduce.ImportJobBase：在174.4617秒（0字节/秒）中传输0个字节
13/01/07 11:03:40警告mapreduce.Counters：Group org.apache.hadoop.mapred.Task $ Counter已弃用。使用org.apache.hadoop.mapreduce.TaskCounter而不是
13/01/07 11:03:40信息mapreduce.ImportJobBase：检索到0条记录。
13/01/07 11:03:40错误tool.ImportTool：导入过程中出错：导入作业失败！
是否有人使用此连接器进行了相同的配置并成功实现了它的功能？
Thx为您提供帮助！
马德琳
Hadoop经历了从Hadoop 1.0到Hadoop 2.0的巨大代码重构（相应地从CDH3到CDH4）。一个副作用是针对Hadoop 1.0（CDH3）编译的代码与Hadoop 2.0（CDH4）不兼容，反之亦然。但是源代码是兼容的，因此只需要用目标Hadoop发行版重新编译代码。
例外情况Found interfaces X，but class was expectedis often common when you运行Hadoop 2.0（CDH4）上为Hadoop 1.0（CDH3）编译的代码，反之亦然。
完全相同的约束适用于连接器。 Hadoop版本CouchBase连接器的编译没有明确的信息，所以我猜测它是针对Hadoop 1.0的，因此它不适用于Hadoop 2.0（CDH4）。
My Configuration:
- CouchBase server 2.0
- Sqoop 1.4.2 (compile for hadoop version 2.0.0)
- Stack Hadoop CDH4.1.2
I want to use the CouchBase/Hadoop connector (http://www.couchbase.com/develop/connectors/hadoop), but when I launch an import I have the following error:
sqoop-import --connect http://localhost:8091/pools --table DUMP
13/01/07 11:00:34 INFO tool.CodeGenTool: Beginning code generation
13/01/07 11:00:34 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
Note: /tmp/sqoop-agent/compile/fccb413cf9eec92f9a716320cf268a0a/DUMP.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/01/07 11:00:42 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-agent/compile/fccb413cf9eec92f9a716320cf268a0a/DUMP.jar
13/01/07 11:00:42 INFO mapreduce.ImportJobBase: Beginning import of DUMP
13/01/07 11:00:45 WARN util.Jars: No such class couchbase doesn't use a jdbc driver available.
13/01/07 11:00:50 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-01-07 11:00:55.065 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-01-07 11:00:55.098 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@41e22632
2013-01-07 11:00:56.266 INFO net.spy.memcached.MemcachedConnection: Shut down memcached client
13/01/07 11:02:39 INFO mapred.JobClient: Running job: job_201212111702_0131
13/01/07 11:02:40 INFO mapred.JobClient: map 0% reduce 0%
13/01/07 11:02:59 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000001_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:02 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000002_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:02 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000000_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:09 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000003_0, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:12 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000001_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:13 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000002_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:20 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000000_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:24 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000003_1, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:24 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000001_2, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:30 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000002_2, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:33 INFO mapred.JobClient: Task Id : attempt_201212111702_0131_m_000000_2, Status : FAILED
Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
13/01/07 11:03:39 INFO mapred.JobClient: Job complete: job_201212111702_0131
13/01/07 11:03:40 INFO mapred.JobClient: Counters: 7
13/01/07 11:03:40 INFO mapred.JobClient: Job Counters
13/01/07 11:03:40 INFO mapred.JobClient: Failed map tasks=1
13/01/07 11:03:40 INFO mapred.JobClient: Launched map tasks=14
13/01/07 11:03:40 INFO mapred.JobClient: Rack-local map tasks=14
13/01/07 11:03:40 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=74621
13/01/07 11:03:40 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
13/01/07 11:03:40 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/01/07 11:03:40 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/01/07 11:03:40 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 174.4617 seconds (0 bytes/sec)
13/01/07 11:03:40 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
13/01/07 11:03:40 INFO mapreduce.ImportJobBase: Retrieved 0 records.
13/01/07 11:03:40 ERROR tool.ImportTool: Error during import: Import job failed!
Does anyone have used this connector with the same configuration and have succeeded to make it works?
Thx for your help!
Madeleine
Hadoop has gone through a huge code refactoring from Hadoop 1.0 to Hadoop 2.0 (correspondingly from CDH3 to CDH4). One side effect is that code compiled against Hadoop 1.0 (CDH3) is not compatible with Hadoop 2.0 (CDH4) and vice-versa. However source code is compatible and thus one just need to recompile code with target Hadoop distribution.
The exception "Found interface X, but class was expected" is very common when you're running code that is compiled for Hadoop 1.0 (CDH3) on Hadoop 2.0 (CDH4) or vice-versa.
Exactly the same constraint applies to connectors. There is no explicit information for which Hadoop version CouchBase connector was compiled, so I would guess that it was against Hadoop 1.0 and thus it's not working on Hadoop 2.0 (CDH4).
这篇关于Couchbase / hadoop连接器：sqoop作业失败“Found interface org.apache.hadoop.mapreduce.TaskAttemptContext，but class was expected”的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
