蜂巢:合并配置设置不起作用 [英] Hive: Merging Configuration Settings not working
问题描述
在Hive 2.2.0上，我使用查询从另一个大小为1.34 GB的源表中填充一个兽人表
On Hive 2.2.0, I am filling an orc table from another source table of size 1.34 GB, using the query
INSERT INTO TABLE TableOrc SELECT * FROM Table; ---- (1)
该查询使用6个orc文件创建TableORC表，这些文件比256MB的块大小小得多.
The query creates TableORC table with 6 orc files, which are much smaller than the block size of 256MB.
-- FolderList1
-rwxr-xr-x user1 supergroup 65.01 MB 1/1/2016, 10:14:21 AM 1 256 MB 000000_0
-rwxr-xr-x user1 supergroup 67.48 MB 1/1/2016, 10:14:55 AM 1 256 MB 000001_0
-rwxr-xr-x user1 supergroup 66.3 MB 1/1/2016, 10:15:18 AM 1 256 MB 000002_0
-rwxr-xr-x user1 supergroup 63.83 MB 1/1/2016, 10:15:41 AM 1 256 MB 000003_0
-rwxr-xr-x user1 supergroup 69.11 MB 1/1/2016, 10:15:57 AM 1 256 MB 000004_0
-rwxr-xr-x user1 supergroup 23.83 MB 1/1/2016, 10:16:02 AM 1 256 MB 000005_0
要潜在地删除此问题，请使用多个其他stackoverflow用户建议的以下配置，并将TableORC截断并再次执行查询(1).
To potentially remove this, the configurations below is used as suggested by multiple other stackoverflow users and TableORC is truncated and query (1) is executed again.
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET hive.merge.size.per.task=128000000; -- (128MB)
SET hive.merge.smallfiles.avgsize=128000000; -- (128MB)
这一次结果是一个TableOrc具有3个ORC文件，并且LOGS显示了3个其他MR作业
This time the result is a TableOrc having 3 ORC files and LOGS show 3 additional MR jobs
-- FolderList2
-rwxr-xr-x user1 supergroup 132.49 MB 1/1/2016, 11:47:01 PM 1 256 MB 000000_0
-rwxr-xr-x user1 supergroup 130.12 MB 1/1/2016, 11:47:02 PM 1 256 MB 000001_0
-rwxr-xr-x user1 supergroup 92.93 MB 1/1/2016, 11:47:03 PM 1 256 MB 000002_0
我的问题是，当配置设置设置为使用128MB时，为什么ORC文件大小大于128MB( https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties )
My question is why are the ORC file sizes greater than 128MB when the configuration settings are set to use 128MB (https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties)
hive.merge.size.per.task
Default Value: 256000000
Added In: Hive 0.4.0
Size of merged files at the end of the job.
推荐答案
您的初始平均文件大小小于hive.merge.smallfiles.avgsize，这就是为什么合并任务开始合并它们的原因.
合并的前两个文件65.01 MB + 67.48 MB = 132.49 MB，这比hive.merge.size.per.task大，这就是为什么合并任务将停止将此文件与更多文件合并的原因.它将不会被精确地分割为128M.它的工作方法非常简单.
Your initial average file size is smaller than hive.merge.smallfiles.avgsize, that is why merge task started to merge them.
First two files merged 65.01 MB + 67.48 MB = 132.49 MB this is bigger than hive.merge.size.per.task that is why merge task will stop to merge this resulted file with more files. It will not be splitted to be exactly 128M. The method it works is quite simple.
这篇关于蜂巢:合并配置设置不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
