SPARK:失败:“联合"预期但“("找到 [英] SPARK : failure: ``union'' expected but `(' found

问题描述

我有一个名为 df 的数据框，其中包含名为 employee_id 的列.我在做:

I have a dataframe called df with column named employee_id. I am doing:

 df.registerTempTable("d_f")
val query = """SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f"""
val result = Spark.getSqlContext().sql(query)

但是遇到以下问题.有什么帮助吗?

But getting following issue. Any help?

[1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f
                            ^
java.lang.RuntimeException: [1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f

推荐答案

Spark 2.0+

Spark 2.0 引入了窗口函数的原生实现 (SPARK-8641) 所以 HiveContext 应该不再需要了.尽管如此，与窗口函数无关的类似错误仍然可以归因于 SQL 解析器之间的差异.

Spark 2.0 introduces native implementation of window functions (SPARK-8641) so HiveContext should be no longer required. Nevertheless similar errors, not related to window functions, can be still attributed to the differences between SQL parsers.

火花 <= 1.6

Spark 1.4.0 中引入了窗口函数，并且需要 HiveContext 工作.SQLContext不能在这里工作.

Window functions have been introduced in Spark 1.4.0 and require HiveContext to work. SQLContext won't work here.

确保您使用 Spark >= 1.4.0 并创建 HiveContext:

Be sure you you use Spark >= 1.4.0 and create the HiveContext:

import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)

这篇关于SPARK:失败:“联合"预期但“("找到的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！