对DStream进行类型参数化 [英] Type-parameterize a DStream

问题描述

DStream 可以具有 type参数 s吗?

如果是，怎么办?

当我在 myDStream上尝试 lazy val qwe = mStream.mapWithState(stateSpec)时:DStream [(A，B)] (类参数)，我得到:

When I try lazy val qwe = mStream.mapWithState(stateSpec) on myDStream: DStream[(A, B)] (class parameter), I get:

value mapWithState is not a member of org.apache.spark.streaming.dstream.DStream[(A, B)]
    lazy val qwe = mStream.mapWithState(stateSpec)

推荐答案

Spark API的大量子集需要隐式的 ClassTags (请参阅类定义:

Substantial subset of the Spark API requires implicit ClassTags (see Scala: What is a TypeTag and how do I use it?) and PairDStreamFunctions.mapWithState is no different. Check class definition:

class PairDStreamFunctions[K, V](self: DStream[(K, V)])
  (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K])

和:

def mapWithState[StateType: ClassTag, MappedType: ClassTag](
    spec: StateSpec[K, V, StateType, MappedType]
  ): MapWithStateDStream[K, V, StateType, MappedType] = {
  ...
}

如果要创建对通用对流进行操作并使用 mapWithState 的函数，则应至少为 KeyType 和 ValueType 类型:

If want to create a function which operates on a generic pair streams and uses mapWithState you should at least provide ClassTags for KeyType and ValueType types:

def foo[T : ClassTag, U : ClassTag](
  stream: DStream[(T, U)], f: StateSpec[T, U, Int, Int]) = stream.mapWithState(f)

如果同时设置了 StateType 和 MappedType 的参数，则您还需要 ClassTags :

If StateType and MappedType are parametrized as well you'll need ClassTags for these too:

def bar[T : ClassTag, U : ClassTag, V : ClassTag,  W : ClassTag](
  stream: DStream[(T, U)], f: StateSpec[T, U, V, W]) = stream.mapWithState(f)

这篇关于对DStream进行类型参数化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！