将负载数据发布到HIVE中 [英] Issue with load data into HIVE

查看:278
本文介绍了将负载数据发布到HIVE中的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我们在AWS中启动了两个EMR,并在一个EMR中安装了hadoop和hive-0.11.0,另外一个配置了hive-0.13.1。

工作正常,但在尝试将数据加载到TABLE时,它给出了以下错误,并且它发生在两个Hive服务器中。

错误信息:


执行SQL命令时发生错误:load数据inpath
's3:// buckername / export / employee_1 /'到表employee_2中查询
返回非零代码:10028,cause:FAILED:SemanticException [Error
10028]:Line 1 :17路径不合法
''s3:// buckername / export / employee_1 /'':从以下位置移动:
s3:// buckername / export / employee_1 to:
hdfs:/ /XXX.XX.XXX.XX:X000/mnt/hive_0110/warehouse/employee_2不是
有效。请检查paramsdefault.fs.name和
hive.metastore.warehouse.dir的值是否有冲突。 [SQL状态= 42000,DB
Errorcode = 10028]


我搜索了这条消息的原因和意思,发现此链接,但是当试图执行 link ,它也会给出下面的错误信息。



命令:

   -  service metatool -updateLocation hdfs://XXX.XX.XXX.XX:X000 hdfs:// XXX .XX.XXX.XX:X000 




初始化HiveMetaTool .. HiveMetaTool:解析失败。原因:
无法识别的选项:-hiveconf


b

解决方案

LOAD不支持S3。最好的做法是将数据保留在S3中,并将其用作Hive外部表,而不是将数据复制到HDFS。一些引用 http://docs.aws。 amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hive-additional-features.html 当您在Hive中使用S3位置创建外部表时,传输的数据是?


We have launched two EMR in AWS and installed the hadoop and hive-0.11.0 in one EMR and hive-0.13.1 other one.

Everything seems to be working fine but while trying to loading data into TABLE it's giving the below error and it happening in both the Hive Servers.

ERROR MESSAGE:

An error occurred when executing the SQL command: load data inpath 's3://buckername/export/employee_1/' into table employee_2 Query returned non-zero code: 10028, cause: FAILED: SemanticException [Error 10028]: Line 1:17 Path is not legal ''s3://buckername/export/employee_1/'': Move from: s3://buckername/export/employee_1 to: hdfs://XXX.XX.XXX.XX:X000/mnt/hive_0110/warehouse/employee_2 is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict. [SQL State=42000, DB Errorcode=10028]

I searched for the reason and mean of this message, I found this link but when tried to execute command suggested in the given link it's also giving the below error.

Command:

--service metatool -updateLocation hdfs://XXX.XX.XXX.XX:X000 hdfs://XXX.XX.XXX.XX:X000

Initializing HiveMetaTool.. HiveMetaTool:Parsing failed. Reason: Unrecognized option: -hiveconf

Any help in this will be really appreciated.

解决方案

LOAD does not support S3. It is best practice to leave data in S3 and just use it as a Hive external table instead of copying the data to HDFS. Some references http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hive-additional-features.html and When you create an external table in Hive with an S3 location is the data transfered?

这篇关于将负载数据发布到HIVE中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆