udf没有TypeTag可用于类型字符串 [英] udf No TypeTag available for type string

问题描述

我不了解火花的行为.

I don't understand a behavior of spark.

我创建了一个udf，返回如下所示的整数

I create an udf wich return an Integer like below

import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}

object Show {

  def main(args: Array[String]): Unit = {


    val (sc,sqlContext) = iniSparkConf("test")
    val testInt_udf = sqlContext.udf.register("testInt_udf", testInt _)

  }

  def iniSparkConf(appName: String): (SparkContext, SQLContext) = {
    val conf = new SparkConf().setAppName(appName)//.setExecutorEnv("spark.ui.port", "4046")
    val sc = new SparkContext(conf)
    sc.setLogLevel("WARN")
    val sqlContext = new SQLContext(sc)

    (sc, sqlContext)
  }
  def testInt() : Int= {
    return 2
  }
}

我工作得很好，但是如果我将方法测试的返回类型从Int更改为String

I work perfectly but if I change the return type of method test from Int to String

val testString_udf = sqlContext.udf.register("testString_udf", testString _)
def testString() : String = {
  return "myString"
}

我收到以下错误

Error:(34, 43) No TypeTag available for String
    val testString_udf = sqlContext.udf.register("testString_udf", testString _)
Error:(34, 43) not enough arguments for method register: (implicit evidence$1: reflect.runtime.universe.TypeTag[String])org.apache.spark.sql.UserDefinedFunction.
Unspecified value parameter evidence$1.
    val testString_udf = sqlContext.udf.register("testString_udf", testString _)

这是我的嵌入式jars

here are my embedded jars:

datanucleus-api-jdo-3.2.6
datanucleus-core-3.2.10
datanucleus-rdbms-3.2.9
spark-1.6.1-yarn-shuffle
spark-assembly-1.6.1-hadoop2.6.0
spark-examples-1.6.1-hadoop2.6.0

我有点迷路了...你有什么主意吗?

I am a little bit lost... Do you have any idea?

推荐答案

由于我无法复制仅将示例代码复制粘贴到新文件中的问题，所以我敢打赌，在您的真实代码中String实际上被遮盖了通过别的东西.要验证这一理论，您可以尝试将您的签名更改为

Since I can't reproduce the issue copy-pasting just your example code into a new file, I bet that in your real code String is actually shadowed by something else. To verify this theory you can try to change you signature to

def testString() : scala.Predef.String = {
  return "myString"
}

或

def testString() : java.lang.String = {
  return "myString"
}

如果此版本可编译，请搜索字符串"以查看如何隐藏标准类型.如果您使用IntelliJ Idea，则可以尝试使用"Ctrl + B"(转到)找出来.最明显的候选人是您使用String作为泛型类型参数的名称，但可能还有其他选择.

If this one compiles, search for "String" to see how you shadowed the standard type. If you use IntelliJ Idea, you can try to use "Ctrl+B" (GoTo) to find it out. The most obvious candidate is that you used String as a name of generic type parameter but there might be some other choices.

这篇关于udf没有TypeTag可用于类型字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！