SPARK：failed：``union''expected but`（'found [英] SPARK : failure: ``union'' expected but `(' found

问题描述

我有一个名为df的数据框，名为employee_id。我在做：

  df.registerTempTable（d_f）
 val query =SELECT *，ROW_NUMBER （）OVER（ORDER BY employee_id）row_number FROM d_f
 val result = Spark.getSqlContext（）。sql（query）
  

但要获得以下问题。任何帮助？

  [1.29] failure：``union''expected but`（'found 
 SELECT * ROW_NUMBER（）OVER（ORDER BY employee_id）row_number FROM d_f 
 ^ 
 java.lang.RuntimeException：[1.29] failure：``union''expected but`（'found 
 SELECT *， ROW_NUMBER（）OVER（ORDER BY employee_id）row_number FROM d_f 
  

解决方案

Spark 2.0 +

Spark 2.0引入了窗口函数的本机实现（ SPARK-8641 ）所以不必再需要 HiveContext ，但与窗口函数无关的类似错误可以是仍然归结于SQL解析器之间的差异。

Spark <= 1.6

在Spark 1.4.0中引入了Window函数，并且需要 HiveContext 工作。 SQLContext 将不会在这里工作。

确保您使用Spark> = 1.4.0并创建 HiveContext ：

  import org.apache.spark.sql.hive.HiveContext 
 val sqlContext = new HiveContext（sc）
  




I have a dataframe called df with column named employee_id. I am doing:

 df.registerTempTable("d_f")
val query = """SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f"""
val result = Spark.getSqlContext().sql(query)

But getting following issue. Any help?

[1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f
                            ^
java.lang.RuntimeException: [1.29] failure: ``union'' expected but `(' found
SELECT *, ROW_NUMBER() OVER (ORDER BY employee_id) row_number FROM d_f

解决方案

Spark 2.0+

Spark 2.0 introduces native implementation of window functions (SPARK-8641) so HiveContext should be no longer required. Nevertheless similar errors, not related to window functions, can be still attributed to the differences between SQL parsers.

Spark <= 1.6

Window functions have been introduced in Spark 1.4.0 and require HiveContext to work. SQLContext won't work here.

Be sure you you use Spark >= 1.4.0 and create the HiveContext:

import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc)

这篇关于SPARK：failed：``union''expected but`（'found的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！