星火 - 斯卡拉 - saveAsHadoopFile抛出错误 [英] Spark - Scala - saveAsHadoopFile throwing error
本文介绍了星火 - 斯卡拉 - saveAsHadoopFile抛出错误的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想解决问题,但无法进一步移动。任何人都可以请帮助
I would like to troubleshoot the issue but couldn't move further. Can anyone please help
import org.apache.hadoop.mapred.lib.MultipleTextOutputFormat
class KeyBasedOutput[T >: Null, V <: AnyRef] extends MultipleTextOutputFormat[T , V] {
override def generateFileNameForKeyValue(key: T, value: V, leaf: String) = {
key.toString
}
override def generateActualKey(key: T, value: V) = {
null
}
}
val cp1 =sqlContext.sql("select * from d_prev_fact").map(t => t.mkString("\t")).map{x => val parts = x.split("\t")
val partition_key = parts(3)
val rows = parts.slice(0, parts.length).mkString("\t")
("date=" + partition_key.toString, rows.toString)}
cp1.saveAsHadoopFile(FACT_CP)
我已经得到了如下错误,不能够调试
I have got an error as below and not able to debug
scala> cp1.saveAsHadoopFile(FACT_CP,classOf[String],classOf[String],classOf[KeyBasedOutput[String, String]])
java.lang.RuntimeException: java.lang.NoSuchMethodException: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$KeyBasedOutput.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.mapred.JobConf.getOutputFormat(JobConf.java:709)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:742)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:674)
的想法是基于所述密钥中的值写入到多个文件夹
The idea is to write the values into multiple folder based on the Key
推荐答案
我不能肯定,但我认为类型擦除与反思相结合可能对您造成这个问题。尝试定义 KeyBasedOutput
的非通用的子类,硬codeS类型的参数和使用。
I'm not certain, but I think type erasure combined with reflection may be causing this problem for you. Try defining a non generic subclass of KeyBasedOutput
that hard codes the type parameters and use that.
class StringKeyBasedOutput extends KeyBasedOutput[String, String]
这篇关于星火 - 斯卡拉 - saveAsHadoopFile抛出错误的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文