Apache Pig - 在相同关系上嵌套 FOREACH [英] Apache Pig - nested FOREACH over same relation

问题描述

我有许多袋子，我想计算袋子之间的成对相似性.

I have a number of bags and I want to compute the pairwise similarities between the bags.

sequences = FOREACH raw GENERATE gen_bag(logs);

关系描述如下:

sequences: {t: (type: chararray, value:charray)}

相似度由 Python UDF 计算，该 UDF 将两个包作为参数.我试图对序列变量进行嵌套的 foreach，但我不能在同一关系上循环两次.我也尝试过两次定义序列，但我无法访问 foreach 中的副本.我也不确定如何想出一个允许我做这样的事情的数据结构.我该怎么做?

The similarity is computed by a Python UDF that takes two bags as arguments. I have tried to do a nested foreach over the sequences variable, but I cant loop over the same relation twice. I've also tried to define the sequences twice, but I cant access the copy in the foreach. I'm also unsure how I can come up with a data structure that allows me to do things like this. How can I do this?

推荐答案

听起来您可以加载源数据的两个副本，执行交叉乘积，然后通过对每一对调用 UDF 进行迭代.类似的东西:

It sounds like you could load up two copies of the source data, perform a cross-product, then iterate through calling your UDF on each pair. Something like:

sequences_A = FOREACH raw GENERATE gen_bag(logs);
sequences_B = FOREACH raw GENERATE gen_bag(logs);
all_pairs = CROSS sequences_A, sequences_B;
FOREACH all_pairs GENERATE myudf(first_bag, second_bag);

这篇关于Apache Pig - 在相同关系上嵌套 FOREACH的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！