蜂巢-使用外部或本地s3而不是AWS s3 [英] hive - use external or local s3 instead of aws s3

问题描述

我有自己的s3在本地运行，而不是aws s3.有没有办法覆盖 s3.amazonaws.com ?
我创建了hive-site.xml并将其放在${HIVE_HOME}/conf/中.
这就是我在.xml中得到的:

I have my own s3 running locally instead of aws s3. Is there a way to overwrite s3.amazonaws.com?
I have created hive-site.xml and put it in ${HIVE_HOME}/conf/.
This is what I have got in .xml:

<configuration>
<property>
    <name>fs.s3n.impl</name>
    <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
</property>
<property>
    <name>fs.s3n.endpoint</name>
    <value>local_s3_ip:port</value>
</property>
<property>
    <name>fs.s3n.awsAccessKeyId</name>
    <value>VALUE</value>
</property>
<property>
    <name>fs.s3n.awsSecretAccessKey</name>
    <value>VALUE</value>
</property>    

现在，我想创建表，如果我输入:

Now I want to create table and if I put:

LOCATION('s3n://hive/sample_data.csv')

我有一个错误:
org.apache.hadoop.hive.ql.exec.DDLTask. java.net.UnknownHostException: hive.s3.amazonaws.com :名称解析暂时失败

I have an error:
org.apache.hadoop.hive.ql.exec.DDLTask. java.net.UnknownHostException: hive.s3.amazonaws.com: Temporary failure in name resolution

对于s3和s3n均无效.

It doesn't work neither for s3 nor s3n.

是否可以覆盖默认的 s3.amazonaws.com 并使用我自己的s3?

Is it possible to overwrite default s3.amazonaws.com and use my own s3?

推荐答案

1. 切换到S3A连接器(和Hadoop 2.7+ JAR)
2. 将"fs.s3a.endpoint"设置为服务器的主机名
3. and"fs.s3a.path.style.access" = true(而不是期望每个存储桶都具有DNS)

期望花时间在身份验证选项上，因为签名始终是第三方商店中的麻烦.

Expect to spend time working on authentication options as signing is always a troublespot in third-party stores.

这篇关于蜂巢-使用外部或本地s3而不是AWS s3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！