按键排序，但是使用Scala值具有多个元素 [英] Sort by a key, but value has more than one element using Scala

问题描述

我是Spark上的Scala的新手，想知道如何创建键值对，而键具有多个元素.例如，我有一个用于婴儿名字的数据集:

I'm very new to Scala on Spark and wondering how you might create key value pairs, with the key having more than one element. For example, I have this dataset for baby names:

年份，名称，县，号码

2000，JOHN，KINGS，50岁

2000, JOHN, KINGS, 50

2000，BOB，KINGS，40

2000, BOB, KINGS, 40

2000，玛丽，NASSAU，60岁

2000, MARY, NASSAU, 60

2001，JOHN，KINGS，14

2001, JOHN, KINGS, 14

2001，简，金斯，30岁

2001, JANE, KINGS, 30

2001，BOB，NASSAU，45

2001, BOB, NASSAU, 45

我想找到每个县最常发生的事，与年份无关.我该怎么做呢?

And I want to find the most frequently occurring for each county, regardless of the year. How might I go about doing that?

我确实使用循环来完成此任务.请参考以下内容.但是我想知道是否有更短的方法利用Spark和Scala对偶性. (即我可以减少计算时间吗?)

I did accomplish this using a loop. Refer to below. But I'm wondering if there is shorter way to do this that utilizes Spark and Scala duality. (i.e. can I decrease computation time?)

val names = sc.textFile("names.csv").map(l => l.split(","))

val uniqueCounty = names.map(x => x(2)).distinct.collect

for (i <- 0 to uniqueCounty.length-1) {
    val county = uniqueCounty(i).toString;
    val eachCounty = names.filter(x => x(2) == county).map(l => (l(1),l(4))).reduceByKey((a,b) => a + b).sortBy(-_._2);
    println("County:" + county + eachCounty.first)
}

推荐答案

以下是使用RDD的解决方案.我假设您需要每个县名列前茅的名字.

Here is the solution using RDD. I am assuming you need top occurring name per county.

val data = Array((2000, "JOHN", "KINGS", 50),(2000, "BOB", "KINGS", 40),(2000, "MARY", "NASSAU", 60),(2001, "JOHN", "KINGS", 14),(2001, "JANE", "KINGS", 30),(2001, "BOB", "NASSAU", 45))
val rdd = sc.parallelize(data)
//Reduce the uniq values for county/name as combo key
val uniqNamePerCountyRdd = rdd.map(x => ((x._3,x._2),x._4)).reduceByKey(_+_)
// Group names per county.
val countyNameRdd = uniqNamePerCountyRdd.map(x=>(x._1._1,(x._1._2,x._2))).groupByKey()
// Sort and take the top name alone per county
countyNameRdd.mapValues(x => x.toList.sortBy(_._2).take(1)).collect

输出:

res8: Array[(String, List[(String, Int)])] = Array((KINGS,List((JANE,30))), (NASSAU,List((BOB,45))))

这篇关于按键排序，但是使用Scala值具有多个元素的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！