sqoop无法将增量状态存储到元存储 [英] sqoop fails to store incremental state to the metastore
问题描述
我在保存增量导入状态时得到了这个
I get this on saving incremental import state
16/05/15 21:43:05 INFO tool.ImportTool: Saving incremental import state to the metastore
16/05/15 21:43:56 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Error communicating with database
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:426)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.update(HsqldbJobStorage.java:445)
at org.apache.sqoop.tool.ImportTool.saveIncrementalState(ImportTool.java:164)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:518)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.setV0Property(HsqldbJobStorage.java:707)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:391)
... 12 more
我已将默认元存储更改为MySQL.其他的一切都很好.我了解到一些消息来源说sqoop不支持它.在这种情况下，我还需要知道在--metaconnect参数中指定的默认HSQL数据库url是什么?
I have changed default metastore to MySQL. Everything else has been working just fine . I understand that some of the sources say that sqoop does not support it. In that case, I also need to know what is the default HSQL db url that I need to specify in --metaconnect argument?
在sqoop site.xml中，我看到它是:
In sqoop site.xml, I see that it is :
jdbc:hsqldb:file:/tmp/sqoop-meta/meta.db;shutdown=true
但是我要在--meta-connect中指定什么?我确定这里没有指定.
But what do I specify in --meta-connect? I am sure this is not to be specified in there.
我需要明确指定meta-connect，因为我正在通过oozie运行作业，因此需要指定meta-connect.
I need to specify meta-connect explicitly because I am running jobs through oozie and so will need to specify meta-connect.
我已经检查并重新检查，但是端口16000上没有任何运行.sqoop-metastore命令可以启动它，但是在那种情况下它就不能作为服务运行.
I have checked and rechecked but nothing is running on port 16000. sqoop-metastore command can ofcourse start it but then it's not running as a service in that case.
所以我要问的是要么将sqoop metastore作为服务运行，要么需要知道为什么在为mysql配置它时无法将状态存储在metastore中.
so what I am asking is either a way to run sqoop metastore as a service or need to know why is this failing to store state in metastore when it is configured for mysql.
推荐答案
在某个地方将HSQLDB实例作为Linux服务启动并不难.即使您坚持使用Sqoop1附带的过时的V1.8.
It is not very difficult to start a HSQLDB instance as a Linux service somewhere. Even if you stick to the obsolete V1.8 that is packaged with Sqoop1.
然后，您必须定期备份数据=>与JDBC客户端连接，运行 CHECKPOINT ，备份脚本"文件，其中包含重建数据库快照所需的所有SQL.
Then you have to backup the data periodically => connect with JDBC client, run a CHECKPOINT, backup the "script" file that contains all SQL necessary to rebuild the DB snapshot.
Might be much easier than trying to hack a MySQL JDBC connection into that source code that expects HSQLDB.
Ah, and for no additional cost, you will find here some explanations about how to use Java properties in the default conf files instead of command-line arguments.
这篇关于sqoop无法将增量状态存储到元存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
