SPARK:failed:``union''expected but`('found [英] SPARK : failure: ``union'' expected but `(' found
问题描述
df.registerTempTable(d_f)
val query =SELECT *,ROW_NUMBER ()OVER(ORDER BY employee_id)row_number FROM d_f
val result = Spark.getSqlContext()。sql(query)
但要获得以下问题。任何帮助?
[1.29] failure:``union''expected but`('found
SELECT * ROW_NUMBER()OVER(ORDER BY employee_id)row_number FROM d_f
^
java.lang.RuntimeException:[1.29] failure:``union''expected but`('found
SELECT *, ROW_NUMBER()OVER(ORDER BY employee_id)row_number FROM d_f
Spark 2.0 +
Spark 2.0引入了窗口函数的本机实现( SPARK-8641 )所以不必再需要 HiveContext
,但与窗口函数无关的类似错误可以是仍然归结于SQL解析器之间的差异。
Spark <= 1.6
在Spark 1.4.0中引入了Window函数,并且需要 HiveContext
工作。 SQLContext
将不会在这里工作。
确保您使用Spark> = 1.4.0并创建 HiveContext
:
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)
I have a dataframe called df with column named employee_id. I am doing:
df.registerTempTable("d_f")
val query = """SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f"""
val result = Spark.getSqlContext().sql(query)
But getting following issue. Any help?
[1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f
^
java.lang.RuntimeException: [1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f
Spark 2.0+
Spark 2.0 introduces native implementation of window functions (SPARK-8641) so HiveContext
should be no longer required. Nevertheless similar errors, not related to window functions, can be still attributed to the differences between SQL parsers.
Spark <= 1.6
Window functions have been introduced in Spark 1.4.0 and require HiveContext
to work. SQLContext
won't work here.
Be sure you you use Spark >= 1.4.0 and create the HiveContext
:
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)
这篇关于SPARK:failed:``union''expected but`('found的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!