hadoop pig 加入任何匹配的元组值 [英] hadoop pig joining on any matching tuple values

问题描述

我是 Pig 的新手，并试图用它来处理数据集.我有一组看起来像

I'm new to pig and trying to use it to process a dataset. I have a set of records that looks like

id    elements
--------------
1     ["a","b","c"]
2     ["a","f","g"]
3     ["f","g","h"]

这个想法是我想创建具有任何重叠元素的元素元组.如果元素只是一个项目而不是数组，我可以做一个简单的连接:

The idea is that I want to create tuples of elements that have any overlapping elements. If elements was just a single item instead of array, I could do a simple join like:

A = LOAD 'mydata' ...
B = FOREACH A GENERATE id as id_2, elements as elements_2;
C = JOIN A BY elements, B BY elements_2;

但是由于 elements 是一个数组，如果只有部分重叠，这将不起作用.关于如何在猪身上做到这一点的任何想法?

But since elements is an array, this won't work if there is only a partial overlap. Any thoughts on how to do this in pig?

预期输出将给出重叠的元组:

The intended output would give the tuples that have overlap:

(1,2)
(2,3)

推荐答案

我认为不可能为此使用 JOIN.一种(不太优雅的)解决方案是 CROSS 两个关系，然后执行 FILTER 操作.FILTER 条件可以是 UDF 或某种 regex_extract_all 以及生成的字段的匹配.如果数组的大小始终为 3，我可能会选择 regex_extract_all 解决方案.

I don't think it's possible to use JOIN for this. One (not so elegant) solution is to CROSS both relations and then do a FILTER operation. The FILTER condition could either be a UDF or some kind of regex_extract_all and a matching of the produced fields. If the size of the array is always 3 I would probably go for the regex_extract_all solution.

这篇关于hadoop pig 加入任何匹配的元组值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！