仅加载 PIG 中的特定字段? [英] Load only particular field in PIG?

问题描述

这是我的文件:

Col1、Col2、Col3、Col4、Col5

我只需要 Col2 和 Col3.

目前我正在这样做:

a = load 'input' as (Col1:chararray, 
                     Col2:chararray, 
                     Col3:chararray, 
                     Col4:chararray);
b = foreach a generate Col2, Col3;

有没有办法直接只加载Col2和Col3，而不是加载整个input然后generate 必填列?

Is there a way to do directly load only Col2 and Col3 instead of loading the whole input and then generate required columns?

推荐答案

您只GENERATE生成您想要的列的方法是满足您的要求的有效方法.请记住，您的所有数据都存储在 HDFS 上，并且您在启动脚本时并没有将其全部加载到内存中.即使您没有将它们保留在处理中使用，您仍然必须从磁盘中读取这些字节，因此从不加载该数据没有性能优势.优点是永远不必将其发送到减速器，这是您用您的方法完成的.

Your method of only GENERATEing the columns you want is an effective way to do just what you ask. Remember that all of your data is stored on HDFS, and you're not loading it all into memory when you start your script. You still will have to read those bytes off the disk even if you are not keeping them around for use in your processing, so there is no performance advantage to never loading that data. The advantage comes in never having to send it to a reducer, which you have accomplished with your method.

如果 Pig 可以判断某个列将不会被使用，它会立即修剪"它，基本上为您执行 b = foreach a generate Col2, Col3; .但是，如果您使用的 UDF 可能访问其他字段，则不会发生这种情况，因为 Pig 不会查看 UDF 内部以查看它们是否被使用.例如，假设 Col3 是一个 int.如果你有

In cases where Pig can tell that a column won't be used, it will "prune" it immediately, essentially doing for you what you did with your b = foreach a generate Col2, Col3;. This won't happen, however, if you are using a UDF that might access other fields, because Pig doesn't look inside the UDF to see if they get used. For example, suppose Col3 is an int. If you have

b = group a by Col2;
c = foreach b generate group, SUM(a.Col3);

然后 Pig 会自动为您修剪第 1 和第 4 列，因为它可以看到它们从未被使用过.但是，如果您改为这样做

then Pig will automatically prune the 1st and 4th columns for you, since it can see they're never used. However, if you instead did

b = group a by Col2;
c = foreach b generate group, COUNT(a);

那么 Pig 不能修剪，因为它看不到 COUNT UDF 内部，也不知道其他字段不会被使用.当不确定 Pig 是否会进行这种修剪时，您可以使用您已有的 foreach/generate 方法.当您启动脚本时，Pig 应该会打印一条诊断消息，列出它能够删除的所有列.

then Pig can't prune, because it doesn't see inside the COUNT UDF and doesn't know that the other fields won't be used. When in doubt of whether Pig will do this pruning, you can use the foreach/generate method you already have. And Pig should print a diagnostic message when you start your script listing all the columns it was able to prune out.

如果您的问题是当您只对几列感兴趣时不想提供完整架构，则可以完全跳过架构并将其放入 GENERATE:

If instead your problem is that you don't want to have to provide a full schema when you're interested in just a few columns, you can skip the schema entirely and put it in the GENERATE:

a = load 'input';
b = foreach a generate (chararray) $1 as Col2, (chararray) $2 as Col3;

这篇关于仅加载 PIG 中的特定字段?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！