如何更改sqoop metastore? [英] How to change sqoop metastore?
问题描述
我正在使用sqoop 1.4.2版本。
我试图将默认hsqldb的sqoop Metastore更改为mysql。
我在sqoop-site.xml文件中配置了以下属性。
< property>
< name> sqoop.metastore.client.enable.autoconnect< / name>
<值> false< /值>
< description>如果为true,则当没有提供其他元存储参数
时,Sqoop将连接到作业管理的本地元数据
。
< / description>
< / property>
<属性>
< name> sqoop.metastore.client.autoconnect.url< / name>
< value> jdbc:mysql:// ip:3206 / sqoop?createDatabaseIfNotExist = true< / value>
< / property>
<属性>
< name> sqoop.metastore.client.autoconnect.username< / name>
<值> userName< /值>
< / property>
<属性>
<名称> sqoop.metastore.client.autoconnect.password< / name>
<值>密码< /值>
< / property>
< / configuration>
当我尝试使用元连接url创建sqoop作业时,它无法连接到配置的mysql db 。
sqoop job --create --meta-connect {mysql_jdbc_url} sqoop作业定义
$ c $ 14/06/06 15:04:54信息sqoop.Sqoop:运行Sqoop版本:1.4.4.2.0.6.1-101
14/06/06 15:04:55警告hsqldb.HsqldbJobStorage:无法解释数字:空
14/06/06 15时04分55秒错误hsqldb.HsqldbJobStorage:无法解释的元数据架构
14/06/06 15时04分55秒错误hsqldb.HsqldbJobStorage:元数据架构版本为null
14/06/06 15时04分55秒ERROR hsqldb.HsqldbJobStorage:支持的最高版本是0
14/06/06 15时04分55秒错误hsqldb.HsqldbJobStorage:要使用此版本的Sqoop,您必须降级您的元数据模式。
14/06/06 15:04:55错误tool.JobTool:执行作业操作的I / O错误:java.io.IOException:无效的元数据版本。在org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.init(HsqldbJobStorage.java:202)
在org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.open(HsqldbJobStorage.java:161)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:274)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org .apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
在org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
在org.apache.sqoop.Sqoop .runTool(Sqoop.java:222)
在org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
在org.apache.sqoop.Sqoop.main(Sqoop.java:240 )
是否sqoop 1.4.2支持hsql db以外的元存储?
请提出建议。
解决方案答案是肯定的,对我来说我使用的是PostgreSQL。我最近碰到了这个,我正在使用1.4.4版本。我不知道我所做的是推荐的方式,但它的工作原理。以下是我遵循的步骤:
-
在sqoop-site.xml中,我配置了连接字符串到我的数据库,用户名因为Sqoop失败了。
在数据库中创建以下对象> CREATE TABLE SQOOP_ROOT(
版本INT,
PROPNAME VARCHAR(128)NOT NULL,
PROPVAL VARCHAR(256),
约束SQOOP_ROOT_unq UNIQUE(版本, propname)
);
-
插入以下行(这似乎是您的脚本失败的原因) / p>
INSERT INTO
SQOOP_ROOT
VALUES(
NULL,
'sqoop.hsqldb.job。 storage.version',
'0'
);
我认为正确的做法可能是下载源代码,并使用数据库实现来扩展
org.apache.sqoop.metastore.JobStorage
。
I am using sqoop 1.4.2 version.
I am trying to change the sqoop metastore from default hsqldb to mysql.
I have configured following properties in sqoop-site.xml file.
<property>
<name>sqoop.metastore.client.enable.autoconnect</name>
<value>false</value>
<description>If true, Sqoop will connect to a local metastore
for job management when no other metastore arguments are
provided.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:mysql://ip:3206/sqoop?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>userName</value>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value>password</value>
</property>
</configuration>
When I try to create a sqoop jobs with meta-connect url it fails to connect to configured mysql db.
sqoop job --create --meta-connect {mysql_jdbc_url} sqoop job defination
it is throwing following exception.
14/06/06 15:04:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.0.6.1-101
14/06/06 15:04:55 WARN hsqldb.HsqldbJobStorage: Could not interpret as a number: null
14/06/06 15:04:55 ERROR hsqldb.HsqldbJobStorage: Can not interpret metadata schema
14/06/06 15:04:55 ERROR hsqldb.HsqldbJobStorage: The metadata schema version is null
14/06/06 15:04:55 ERROR hsqldb.HsqldbJobStorage: The highest version supported is 0
14/06/06 15:04:55 ERROR hsqldb.HsqldbJobStorage: To use this version of Sqoop, you must downgrade your metadata schema.
14/06/06 15:04:55 ERROR tool.JobTool: I/O error performing job operation: java.io.IOException: Invalid metadata version.
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.init(HsqldbJobStorage.java:202)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.open(HsqldbJobStorage.java:161)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:274)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
Does sqoop 1.4.2 supports metastore other than hsql db?
Please suggest.
解决方案 The answer is Yes, in my case I am using PostgreSQL. I ran into this recently and I am using Version 1.4.4. I am not sure if what I did is the recommended way, but it works. Here are the steps I followed
In sqoop-site.xml I configured it with, the connect string to my database, username and password.
Created the following object in the database, as Sqoop was failing at it.
CREATE TABLE SQOOP_ROOT (
version INT,
propname VARCHAR(128) NOT NULL,
propval VARCHAR(256),
CONSTRAINT SQOOP_ROOT_unq UNIQUE (version, propname)
);
Inserted the following row (This seems to be the reason your script is failing)
INSERT INTO
SQOOP_ROOT
VALUES(
NULL,
'sqoop.hsqldb.job.storage.version',
'0'
);
I think the correct way might be is to download the source, and extend
org.apache.sqoop.metastore.JobStorage
with you DB implementation.
这篇关于如何更改sqoop metastore?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!