星火并行化？ （无法与名称“ID”找到creator属性） [英] Spark Parallelize? (Could not find creator property with name 'id')

问题描述

在Apache的星火1.4.0打电话时，是什么原因导致这种序列化错误：

What causes this Serialization error in Apache Spark 1.4.0 when calling:

sc.parallelize(strList, 4)

这抛出异常：

com.fasterxml.jackson.databind.JsonMappingException: 
Could not find creator property with name 'id' (in class org.apache.spark.rdd.RDDOperationScope)

从 addBeanProps 抛出的杰克逊：com.fasterxml.jackson.databind.deser.BeanDeserializerFactory＃addBeanProps

Thrown from addBeanProps in Jackson: com.fasterxml.jackson.databind.deser.BeanDeserializerFactory#addBeanProps

该RDD是一个序列[字符串]和#partitions似乎并不重要（试过1，2，4）。

The RDD is a Seq[String], and the #partitions doesn't seem to matter (tried 1, 2, 4).

有没有序列化堆栈跟踪，作为普通工人封闭不能被序列化。

There is no serialization stack trace, as normal the worker closure cannot be serialized.

什么是另一种方式来追踪下来？

What is another way to track this down?

推荐答案

@Interfector是正确的。我遇到了这个问题也，这里是从我的SBT文件和dependencyOverrides部分，其中固定它的一个片段。

@Interfector is correct. I ran into this issue also, here's a snippet from my sbt file and the 'dependencyOverrides' section which fixed it.

libraryDependencies ++= Seq(
  "com.amazonaws" % "amazon-kinesis-client" % "1.4.0",
  "org.apache.spark" %% "spark-core" % "1.4.0",
  "org.apache.spark" %% "spark-streaming" % "1.4.0",
  "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.4.0",
  "com.amazonaws" % "aws-java-sdk" % "1.10.2"
)

dependencyOverrides ++= Set(
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4"
)

这篇关于星火并行化？ （无法与名称“ID”找到creator属性）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！