类型参数化 DStream [英] Type-parameterize a DStream

问题描述

DStream 可以有类型参数s吗?

如果是，如何?

当我在 myDStream: DStream[(A, B)](类参数)上尝试 lazy val qwe = mStream.mapWithState(stateSpec) 时，我得到:

When I try lazy val qwe = mStream.mapWithState(stateSpec) on myDStream: DStream[(A, B)] (class parameter), I get:

value mapWithState is not a member of org.apache.spark.streaming.dstream.DStream[(A, B)]
    lazy val qwe = mStream.mapWithState(stateSpec)

推荐答案

Spark API 的大量子集需要隐式 ClassTags(请参阅 Scala:什么是 TypeTag，我该如何使用它?) 和 PairDStreamFunctions.mapWithState 没有什么不同.检查 类定义:

Substantial subset of the Spark API requires implicit ClassTags (see Scala: What is a TypeTag and how do I use it?) and PairDStreamFunctions.mapWithState is no different. Check class definition:

class PairDStreamFunctions[K, V](self: DStream[(K, V)])
  (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K])

和:

def mapWithState[StateType: ClassTag, MappedType: ClassTag](
    spec: StateSpec[K, V, StateType, MappedType]
  ): MapWithStateDStream[K, V, StateType, MappedType] = {
  ...
}

如果要创建一个对通用对流进行操作并使用 mapWithState 的函数，您至少应该为 KeyType 和 ClassTags 提供ClassTagscode>ValueType 类型:

If want to create a function which operates on a generic pair streams and uses mapWithState you should at least provide ClassTags for KeyType and ValueType types:

def foo[T : ClassTag, U : ClassTag](
  stream: DStream[(T, U)], f: StateSpec[T, U, Int, Int]) = stream.mapWithState(f)

如果 StateType 和 MappedType 也被参数化了，你也需要 ClassTags:

If StateType and MappedType are parametrized as well you'll need ClassTags for these too:

def bar[T : ClassTag, U : ClassTag, V : ClassTag,  W : ClassTag](
  stream: DStream[(T, U)], f: StateSpec[T, U, V, W]) = stream.mapWithState(f)

这篇关于类型参数化 DStream的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！