星火：创建新的蓄电池类型将无法正常工作（斯卡拉） [英] Spark: Create new accumulator type won't work (Scala)

问题描述

我要建立[（字符串，字符串）类型的List列表累加器。我第一次创建以下对象：

I want to create an accumulator for lists of type List[(String, String)]. I first created the following object:

object ListAccumulator extends AccumulatorParam[List[(String, String)]] {
  def zero(initialValue: List[(String, String)]): List[(String, String)] = {
    Nil
  }

  def addInPlace(list1: List[(String, String)], list2: List[(String, String)]): List[(String, String)] = {
    list1 ::: list2
  }
}

在同一个文件中（SparkQueries.scala）我想在我的类中的函数中使用它：

In the same file (SparkQueries.scala) I tried to use it within a function in my class:

val resultList = sc.accumulator(Nil)(ListAccumulator)

不过，在这里我的编译器在抱怨（ListAccumulator）。出现以下错误：

However, here my compiler complains at (ListAccumulator). The following error occurs:

类型不匹配;发现：sparkMain.ListAccumulator.type要求：org.apache.spark.AccumulatorParam [scala.collection.immutable.Nil.type]注：列表[（字符串，字符串）]>：scala.collection.immutable.Nil.type（和sparkMain.ListAccumulator.type＆LT ;: org.apache.spark.AccumulatorParam [列表[（字符串，字符串）]]），但特点是AccumulatorParam型T.不变的你不妨​​来定义T作为-T来代替。

type mismatch; found : sparkMain.ListAccumulator.type required: org.apache.spark.AccumulatorParam[scala.collection.immutable.Nil.type] Note: List[(String, String)] >: scala.collection.immutable.Nil.type (and sparkMain.ListAccumulator.type <: org.apache.spark.AccumulatorParam[List[(String, String)]]), but trait AccumulatorParam is invariant in type T. You may wish to define T as -T instead.

sparkMain是包.scala文件是什么我做错了？是否有可能编译器不知道ListAccumulator对象的存在？

sparkMain is the package the .scala file is in. What am I doing wrong? Is it possible the compiler doesn't know of the existence of the ListAccumulator object?

在此先感谢！

推荐答案

您可以修复你的错误类型是这样的：

You can fix your type error like this:


VAL resultList = sc.accumulator（ListAccumulator.zero（无））（ListAccumulator）


在斯卡拉类型inferencer有过错的地方，假设最具体类型（无，空列表的类型）是你想为你的蓄电池类型。通过使用零，具有显式的返回类型列表[（字符串，字符串）] ，你帮它足够知道你的意思。

The type inferencer in Scala is at fault where, assuming that the most specific type (Nil, the type of empty lists) is the type that you want for your accumulator. By using zero, with an explicit return type of List[(String, String)], you help it enough to know what you mean.

一个侧面说明：您正在使用列表连接为 addInPlace ，这是在列表的大小呈线性关系。如果你的列表可以得到大的，你除了会很慢。如果你需要高效的追加使用 ListBuffer ， ArrayBuffer 或矢量如果你想要一个不可变的序列。

A side remark: you are using list concatenation for addInPlace, which is linear in the size of the list. If your lists can get large, your addition will be slow. If you need efficient appends use a ListBuffer, ArrayBuffer, or a Vector if you want an immutable sequence.

这篇关于星火：创建新的蓄电池类型将无法正常工作（斯卡拉）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！