猪:从包装袋中取出所有元组 [英] PIG: Get all tuples out of a grouped bag

问题描述

我正在使用PIG从元组生成组，如下所示:

I am using PIG to generate groups from tuples as follows:

a1, b1
a1, b2
a1, b3
...

->

a1, [b1, b2, b3]
...

这很容易且有效.但是我的问题是要获得以下信息:从获得的组中，我想生成一组在该组包中的所有元组:

This is easy and working. But my problem is to get the following: From the obtained groups, I would like to generate a set of all tuples in the group's bag:

a1, [b1, b2, b3]

->

b1,b2
b1,b3
b2,b3

如果我可以嵌套"foreach"，然后首先遍历每个组，然后遍历其包，这将很容易.

This would be easy if I could nest "foreach" and firstly iterate over each group and then over its bag.

我想我对这个概念有误解，请多多关照.

I suppose I am misunderstanding the concept and I will appreciate your explanation.

谢谢.

推荐答案

在包与包之间似乎需要笛卡尔积.为此，您需要使用FLATTEN(bag)两次.

It looks like you need a Cartesian product between the bag and itself. To do this you need to use FLATTEN(bag) twice.

代码:

inpt = load '.../group.txt' using PigStorage(',') as (id, val);
grp = group inpt by (id);
id_grp = foreach grp generate group as id, inpt.val as value_bag;
result = foreach id_grp generate id, FLATTEN(value_bag) as v1, FLATTEN(value_bag) as v2; 
dump result;

请注意，大袋子会产生很多行.为了避免这种情况，您可以在FLATTEN之前使用TOP(...):

Be aware that large bags will produce a lot of rows. To avoid it you could use TOP(...) before FLATTEN:

inpt = load '....group.txt' using PigStorage(',')  as (id, val);
grp = group inpt by (id);
id_grp = foreach grp generate group as id, inpt.val as values;
result = foreach id_grp {
    limited_bag = TOP(50, 0, values); -- all sorts of filtering could be done here
    generate id, FLATTEN(limited_bag) as v1, FLATTEN(limited_bag) as v2; 
};
dump result;

对于您的特定输出，可以在FLATTEN之前使用一些过滤:

For your specific output you could use some filtering before FLATTEN:

inpt = load '..../group.txt' as (id, val);
grp = group inpt by (id);
id_grp = foreach grp generate group as id, inpt.val as values;
result = foreach id_grp {
    l = filter values by val == 'b1' or val == 'b2';
    generate id, FLATTEN(l) as v1, FLATTEN(values) as v2; 
};
result = filter result by v1 != v2;

希望对您有帮助.

欢呼

这篇关于猪:从包装袋中取出所有元组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！