Hive:合并配置设置不起作用 [英] Hive: Merging Configuration Settings not working
问题描述
在 Hive 2.2.0 上，我使用查询从另一个大小为 1.34 GB 的源表中填充一个 orc 表
On Hive 2.2.0, I am filling an orc table from another source table of size 1.34 GB, using the query
INSERT INTO TABLE TableOrc SELECT * FROM Table; ---- (1)
该查询创建了包含 6 个 orc 文件的 TableORC 表，这些文件远小于 256MB 的块大小.
The query creates TableORC table with 6 orc files, which are much smaller than the block size of 256MB.
-- FolderList1
-rwxr-xr-x user1 supergroup 65.01 MB 1/1/2016, 10:14:21 AM 1 256 MB 000000_0
-rwxr-xr-x user1 supergroup 67.48 MB 1/1/2016, 10:14:55 AM 1 256 MB 000001_0
-rwxr-xr-x user1 supergroup 66.3 MB 1/1/2016, 10:15:18 AM 1 256 MB 000002_0
-rwxr-xr-x user1 supergroup 63.83 MB 1/1/2016, 10:15:41 AM 1 256 MB 000003_0
-rwxr-xr-x user1 supergroup 69.11 MB 1/1/2016, 10:15:57 AM 1 256 MB 000004_0
-rwxr-xr-x user1 supergroup 23.83 MB 1/1/2016, 10:16:02 AM 1 256 MB 000005_0
为了可能消除这种情况，请按照多个其他 stackoverflow 用户的建议使用以下配置，并截断 TableORC 并再次执行查询 (1).
To potentially remove this, the configurations below is used as suggested by multiple other stackoverflow users and TableORC is truncated and query (1) is executed again.
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET hive.merge.size.per.task=128000000; -- (128MB)
SET hive.merge.smallfiles.avgsize=128000000; -- (128MB)
这次的结果是 TableOrc 有 3 个 ORC 文件，LOGS 显示了 3 个额外的 MR 作业
This time the result is a TableOrc having 3 ORC files and LOGS show 3 additional MR jobs
-- FolderList2
-rwxr-xr-x user1 supergroup 132.49 MB 1/1/2016, 11:47:01 PM 1 256 MB 000000_0
-rwxr-xr-x user1 supergroup 130.12 MB 1/1/2016, 11:47:02 PM 1 256 MB 000001_0
-rwxr-xr-x user1 supergroup 92.93 MB 1/1/2016, 11:47:03 PM 1 256 MB 000002_0
我的问题是当配置设置设置为使用 128MB 时，为什么 ORC 文件大小大于 128MB (https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties)
My question is why are the ORC file sizes greater than 128MB when the configuration settings are set to use 128MB (https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties)
hive.merge.size.per.task
Default Value: 256000000
Added In: Hive 0.4.0
Size of merged files at the end of the job.
推荐答案
您的初始平均文件大小小于 hive.merge.smallfiles.avgsize，这就是合并任务开始合并它们的原因.合并的前两个文件 65.01 MB + 67.48 MB = 132.49 MB 这比 hive.merge.size.per.task 大，这就是为什么合并任务将停止将此结果文件与更多文件合并.它不会被拆分为恰好 128M.它的工作方法很简单.
Your initial average file size is smaller than hive.merge.smallfiles.avgsize, that is why merge task started to merge them.
First two files merged 65.01 MB + 67.48 MB = 132.49 MB this is bigger than hive.merge.size.per.task that is why merge task will stop to merge this resulted file with more files. It will not be splitted to be exactly 128M. The method it works is quite simple.
这篇关于Hive:合并配置设置不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
