星火Combinebykey JAVA拉姆达前pression [英] Spark Combinebykey JAVA lambda expression

问题描述

我想使用lambda函数，以便通过一个关键的计算平均（ JavaPairRDD＆LT;整数，双＆GT;对）。出于这个原因，我制定了以下code：

  java.util.function.Function＆LT;双，Tuple2＆LT;双，整数GT;＆GT; createAcc = X  - ＆GT;新Tuple2＆LT;双，整数GT;（X，1）;双功能＆LT; Tuple2＆LT;双，整数＆gt;中双，Tuple2＆LT;双，整数GT;＆GT; addAndCount =（Tuple2＆LT;双，整数GT; X，双Y） -  GT; {返回新Tuple2（x._1（）+ Y，x._2（）+ 1）; };双功能＆LT; Tuple2＆LT;双，整数＆gt;中Tuple2＆LT;双，整数＆gt;中Tuple2＆LT;双，整数GT;＆GT;结合=（Tuple2＆LT;双，整数X的催化剂，Tuple2＆LT;双，整数GT; Y） -  GT; {返回新Tuple2（x._1（）+ y._1（），x._2（）+ y._2（））; };JavaPairRDD＆LT;整数，Tuple2＆LT;双，整数GT;＆GT; avgCounts = pairs.combineByKey（createAcc，addAndCount，结合）;
 

然而，蚀 diplays这个错误

 的方法combineByKey（功能＆LT;四，C＆gt;中功能2＆LT; C，双，C＆gt;中功能2＆LT; C，C，C＆GT;）在类型JavaPairRDD＆LT;整数，双＆GT;不适用于参数（功能＆LT;双，Tuple2＆LT;双，整数GT;＆gt;中
 双功能＆LT; Tuple2＆LT;双，整数GT;双，Tuple2＆LT;双，整数GT;＆gt;中BiFunction<Tuple2<Double,Integer>,Tuple2<Double,Integer>,Tuple2<Double,Integer>>)
 

解决方案

该combineByKey方法需要 org.apache.spark.api.java.function.Function2 而不是 java.util.function.BiFunction 。因此，无论你写的：

  java.util.function.Function＆LT;双，Tuple2＆LT;双，整数GT;＆GT; createAcc =
    点¯x - ＆GT;新Tuple2＆LT;双，整数GT;（X，1）;功能2＆LT; Tuple2＆LT;双，整数＆gt;中双，Tuple2＆LT;双，整数GT;＆GT; addAndCount =
    （Tuple2＆LT;双，整数GT; X，双Y） -  GT; {返回新Tuple2（x._1（）+ Y，x._2（）+ 1）; };功能2＆LT; Tuple2＆LT;双，整数＆gt;中Tuple2＆LT;双，整数＆gt;中Tuple2＆LT;双，整数GT;＆GT;结合=
    （Tuple2＆LT;双，整数X的催化剂，Tuple2＆LT;双，整数GT; Y） -  GT; {返回新Tuple2（x._1（）+ y._1（），x._2（）+ y._2（））; };JavaPairRDD＆LT;整数，Tuple2＆LT;双，整数GT;＆GT; avgCounts =
    pairs.combineByKey（createAcc，addAndCount，结合）;
 

I want to use lambda function in order to compute the average by key of a (JavaPairRDD<Integer, Double> pairs). For that reason, I developed the following code:

java.util.function.Function<Double, Tuple2<Double, Integer>> createAcc = x -> new Tuple2<Double, Integer>(x, 1);

BiFunction<Tuple2<Double, Integer>, Double, Tuple2<Double, Integer>>  addAndCount = (Tuple2<Double, Integer> x, Double y) -> {  return new Tuple2(x._1()+y, x._2()+1 );   };

BiFunction<Tuple2<Double, Integer>, Tuple2<Double, Integer>, Tuple2<Double, Integer>>  combine = (Tuple2<Double, Integer> x, Tuple2<Double, Integer> y) -> {  return new Tuple2(x._1()+y._1(), x._2()+y._2() );   };

JavaPairRDD<Integer, Tuple2<Double, Integer>> avgCounts = pairs.combineByKey(createAcc, addAndCount, combine);

However, eclipse diplays this error:

The method combineByKey(Function<Double,C>, Function2<C,Double,C>, Function2<C,C,C>) in the type JavaPairRDD<Integer,Double> is not applicable for the arguments (Function<Double,Tuple2<Double,Integer>>,
 BiFunction<Tuple2<Double,Integer>,Double,Tuple2<Double,Integer>>, BiFunction<Tuple2<Double,Integer>,Tuple2<Double,Integer>,Tuple2<Double,Integer>>) 

解决方案

The combineByKey method expects org.apache.spark.api.java.function.Function2 instead of java.util.function.BiFunction. So either you write:

java.util.function.Function<Double, Tuple2<Double, Integer>> createAcc =
    x -> new Tuple2<Double, Integer>(x, 1);

Function2<Tuple2<Double, Integer>, Double, Tuple2<Double, Integer>>  addAndCount = 
    (Tuple2<Double, Integer> x, Double y) -> {  return new Tuple2(x._1()+y, x._2()+1 );   };

Function2<Tuple2<Double, Integer>, Tuple2<Double, Integer>, Tuple2<Double, Integer>>  combine = 
    (Tuple2<Double, Integer> x, Tuple2<Double, Integer> y) -> {  return new Tuple2(x._1()+y._1(), x._2()+y._2() );   };

JavaPairRDD<Integer, Tuple2<Double, Integer>> avgCounts = 
    pairs.combineByKey(createAcc, addAndCount, combine);

这篇关于星火Combinebykey JAVA拉姆达前pression的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！