当您在 Hive 中创建具有 S3 位置的外部表时，数据何时传输? [英] When is the data transferred when you create an external table in Hive with an S3 location?

问题描述

当您使用 Amazon S3 源位置在 Hive(在 Hadoop 上)创建外部表时，数据何时传输到本地 Hadoop HDFS?是否开启:

When you create an external table in Hive (on Hadoop) with an Amazon S3 source location, when is the data transferred to the local Hadoop HDFS? Is it on:

• 创建外部表
• 在外部表上运行查询(MR 作业)时
• 从不(不传输任何数据)和 MR 作业读取 S3 数据.

这里产生的 S3 读取成本是多少?将数据传输到 HDFS 是否有单一成本，或者是否没有数据传输成本，但是当 Hive 创建的 MapReduce 作业在此外部表上运行时，会产生读取成本.

What are the costs incurred here for S3 reads? Is there a single cost for the transfer of data to HDFS, or is there no data transfer costs but when the MapReduce job created by Hive runs on this external table the read costs are incurred.

一个示例外部表定义是:

An example external table definition would be:

CREATE EXTERNAL TABLE mydata (key STRING, value INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '='
LOCATION 's3n://mys3bucket/';

推荐答案

Map 任务将直接从 S3 读取数据.在 Map 和 Reduce 步骤之间，数据将写入本地文件系统，在 mapreduce 作业之间(在需要多个作业的查询中)，临时数据将写入 HDFS.

Map tasks will read the data directly from S3. Between the Map and Reduce steps, data will be written to the local filesystem, and between mapreduce jobs (in queries that require multiple jobs) the temporary data will be written to HDFS.

如果您担心 S3 读取成本，创建另一个存储在 HDFS 上的表并从 S3 表一次性复制到 HDFS 表可能是有意义的.

If you are concerned about S3 read costs, it might make sense to create another table that is stored on HDFS, and do a one-time copy from the S3 table to the HDFS table.

这篇关于当您在 Hive 中创建具有 S3 位置的外部表时，数据何时传输?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！