Apache Pig - 通过相同的关系嵌套FOREACH [英] Apache Pig - nested FOREACH over same relation

问题描述

我有很多包包，我想计算包包之间的两两相似之处。

I have a number of bags and I want to compute the pairwise similarities between the bags.

sequences = FOREACH raw GENERATE gen_bag(logs);

该关系如下所述：

The relation is described as follows:

sequences: {t: (type: chararray, value:charray)}

相似度是通过一个Python UDF来计算的，该UDF以两个袋子为参数。我试图对序列变量做一个嵌套的foreach，但是我无法循环两次相同的关系。我也尝试过两次定义序列，但是我不能在foreach中访问这个副本。我也不确定我怎么能想出一个数据结构来让我做这样的事情。我怎么能这样做？

The similarity is computed by a Python UDF that takes two bags as arguments. I have tried to do a nested foreach over the sequences variable, but I cant loop over the same relation twice. I've also tried to define the sequences twice, but I cant access the copy in the foreach. I'm also unsure how I can come up with a data structure that allows me to do things like this. How can I do this?

推荐答案

听起来你可以加载两个源数据副本，执行跨产品，然后通过每对调用你的UDF进行迭代。例如：

It sounds like you could load up two copies of the source data, perform a cross-product, then iterate through calling your UDF on each pair. Something like:

sequences_A = FOREACH raw GENERATE gen_bag(logs);
sequences_B = FOREACH raw GENERATE gen_bag(logs);
all_pairs = CROSS sequences_A, sequences_B;
FOREACH all_pairs GENERATE myudf(first_bag, second_bag);

这篇关于Apache Pig - 通过相同的关系嵌套FOREACH的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！