如何在Dataset< Row>上的groupby之后获取所有列在Spark SQL 2.1.0中 [英] How to get all columns after groupby on Dataset<Row> in spark sql 2.1.0
问题描述
首先,我是SPARK的新手
First, I am very new to SPARK
我的数据集中有数百万条记录,我想使用名称"列进行分组,并查找具有最长使用期限的名称.我得到正确的结果,但是我的结果集中需要所有列.
I have millions of records in my Dataset and i wanted to groupby with name column and finding names which having maximum age. I am getting correct results but I need all columns in my resultset.
Dataset<Row> resultset = studentDataSet.select("*").groupBy("name").max("age");
resultset.show(1000,false);
我在结果集数据集中仅获得姓名和max(age).
I am getting only name and max(age) in my resultset dataset.
推荐答案
对于您的解决方案,您必须尝试其他方法.您几乎在那儿寻求解决方案,但让我帮助您理解.
For your solution you have to try different approach. You was almost there for solution but let me help you understand.
Dataset<Row> resultset = studentDataSet.groupBy("name").max("age");
现在您可以做的是可以将resultset
与studentDataSet
now what you can do is you can join the resultset
with studentDataSet
Dataset<Row> joinedDS = studentDataset.join(resultset, "name");
groupBy
的问题是,应用groupBy后您会得到RelationalGroupedDataset
,因此这取决于您要执行的下一个操作(如sum, min, mean, max
等),然后这些操作的结果与groupBy
结合在一起
The problem with groupBy
this that after applying groupBy you get RelationalGroupedDataset
so it depends on what next operation you perform like sum, min, mean, max
etc then the result of these operation joined with groupBy
在您的情况下,name
列与age
的max
连接在一起,因此它将仅返回两列,但是如果在age
上应用apply groupBy
然后在'age'上应用max
列,您将获得两列,第一列是age
,第二列是max(age)
.
As in you case name
column is joined with the max
of age
so it will return only two columns but if use apply groupBy
on age
and then apply max
on 'age' column you will get two column one is age
and second is max(age)
.
注意:-代码未经测试,请根据需要进行更改 希望这可以清除您的查询
Note :- code is not tested please make changes if needed Hope this clears you query
这篇关于如何在Dataset< Row>上的groupby之后获取所有列在Spark SQL 2.1.0中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!