斯卡拉/星火应用与＆QUOT;无＆QUOT TypeTag;错误＆QUOT;高清主＆QUOT;风格的应用程序 [英] Scala/Spark App with "No TypeTag available" Error in "def main" style App

问题描述

我是新来斯卡拉/星火栈，我试图找出如何使用SparkSql到地图中TempTables，反之亦然RDDS测试我的基本技能。

I'm new to Scala/Spark stack and I'm trying to figure out how to test my basic skills using SparkSql to "map" RDDs in TempTables and viceversa.

我有同样的code 2不同.scala文件：一个简单的对象（DEF主...）和扩展应用程序的对象

I have 2 distinct .scala files with the same code: a simple object (with def main...) and an object extending App.

在简单的对象就是我得到一个错误，由于连接到我的案例类日志没有可用TypeTag

In the simple object one I get an error due to "No TypeTag available" connected to my case class Log:

object counter {
  def main(args: Array[String]) {
.
.
.
   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
   import sqlContext.createSchemaRDD
   case class Log(visitatore: String, data: java.util.Date, pagina: String, count: Int)
   val log = triple.map(p => Log(p._1,p._2,p._3,p._4))
   log.registerTempTable("logs")
   val logSessioni= sqlContext.sql("SELECT visitor, data, pagina, count FROM logs")
   logSessioni.foreach(println)
}

在行错误： log.registerTempTable（日志）说：可用于日志中没有TypeTag

The error at line: log.registerTempTable("logs") says "No TypeTag available for Log".

在其他文件（对象扩展应用程序），一切工作正常：

In the other file (object extends App) all works fine:

object counterApp extends App {
.
.
.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    import sqlContext.createSchemaRDD
    case class Log(visitatore: String, data: java.util.Date, pagina: String, count: Int)
    val log = triple.map(p => Log(p._1,p._2,p._3,p._4))
    log.registerTempTable("logs")
    val logSessioni= sqlContext.sql("SELECT visitor, data, pagina, count from logs")
    logSessioni.foreach(println)
}

由于我刚刚开始，我没有收到两个要点：
1）为什么在第二个文件的同一code做工精细（对象扩展应用程序），而在第一个（简单对象）我得到的错误？

Since I've just started, I'm not getting two main points: 1) Why does the same code work fine in the second file (object extend App) while in the first one (simple object) I get the error?

2）（最重要的），我应该在code（简单对象文件），以应对案例类和TypeTag（我几乎不知道）来解决这个错误怎么办？

2) (and most important) What should I do in my code (simple object file) to fix this error in order to deal with case class and TypeTag (which I barely know)?

每一个答案，code例子将大大AP preciated！

Every answer, code examples will be much appreciated!

在此先感谢

FF

推荐答案

TL; DR;

只是将你的case类该方法定义的

Just move your case class out of the method definition

问题是，你的案例类登录是，它正在使用的方法中定义。因此，只需将您的案件类定义的方法之外，也将努力。我将不得不看看如何编译下来，但我的猜测是，这更是一个鸡生蛋蛋的问题。在 TypeTag （用于反射）是不能够被隐含定义，因为它尚未完全在该点限定。 <一href=\"https://stackoverflow.com/questions/19507760/typetags-not-working-inside-of-$c$c-block-scope\">Here 中有两个<一个href=\"https://stackoverflow.com/questions/18866866/why-there-is-no-typetag-available-in-nested-instantiations-when-inter$p$pted-by\">SO问题与表现出的火花将需要使用 WeakTypeTag 同样的问题。而且，这里是JIRA 解释这更正式

The problem is that your case class Log is defined inside of the method that it is being used. So, simply move your case class definition outside of the method and it will work. I will have to take a look at how this compiles down, but my guess is that this is more of a chicken-egg problem. The TypeTag (used for reflection) is not able to be implicitly defined as it has not been fully defined at that point. Here are two SO questions with the same problem that exhibit that Spark would need to use a WeakTypeTag. And, here is the JIRA explaining this more officially

这篇关于斯卡拉/星火应用与＆QUOT;无＆QUOT TypeTag;错误＆QUOT;高清主＆QUOT;风格的应用程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！