PIG 中整套记录的最大值/最小值 [英] Max/Min for whole sets of records in PIG

问题描述

我有一组正在从文件加载的记录，我需要做的第一件事是获取列的最大值和最小值.在 SQL 中，我会用这样的子查询来做到这一点:

I have a set set of records that I am loading from a file and the first thing I need to do is get the max and min of a column. In SQL I would do this with a subquery like this:

   select c.state, c.population, 
(select max(c.population) from state_info c) as max_pop, 
(select min(c.population) from state_info c) as min_pop
from state_info c

我认为在 PIG 中也必须有一种简单的方法可以做到这一点，但我找不到它.它有一个 MAX 和 MIN 函数，但是当我尝试执行以下操作时它不起作用:

I assume there must be an easy way to do this in PIG as well but I'm having trouble finding it. It has a MAX and MIN function but when I tried doing the following it didn't work:

records=LOAD '/Users/Winter/School/st_incm.txt'  AS (state:chararray, population:int);
with_max = FOREACH records GENERATE state, population, MAX(population);

这没有用.我有更好的运气为每一行添加一个具有相同值的额外列，然后将它们分组在该列上.然后在那个新组上获得最大值.这似乎是一种获得我想要的东西的复杂方式，所以我想我会问是否有人知道更简单的方法.

This didn't work. I had better luck adding an extra column with the same value to each row and then grouping them on that column. Then getting the max on that new group. This seems like a convoluted way of getting what I want so I thought I'd ask if anyone knows a simpler way.

预先感谢您的帮助.

推荐答案

正如您所说，您需要将所有数据组合在一起，但如果您使用 全部分组.

As you said you need to group all the data together but no extra column is required if you use GROUP ALL.

猪

records = LOAD 'states.txt'  AS (state:chararray, population:int);
records_group = GROUP records ALL;
with_max = FOREACH records_group 
           GENERATE
               FLATTEN(records.(state, population)), MAX(records.population);

输入

CA  10
VA  5
WI  2

输出

(CA,10,10)
(VA,5,10)
(WI,2,10)

这篇关于PIG 中整套记录的最大值/最小值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！