您必须使用Hive构建Spark.导出'SPARK_HIVE = true' [英] You must build Spark with Hive. Export 'SPARK_HIVE=true'

问题描述

我试图在运行于Bluemix上的Apache Spark的Analytics(分析)上运行笔记本，但遇到以下错误:

I'm trying to run a notebook on Analytics for Apache Spark running on Bluemix, but I hit the following error:

Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and 
run build/sbt assembly", Py4JJavaError(u'An error occurred while calling 
None.org.apache.spark.sql.hive.HiveContext.\n', JavaObject id=o38))

错误是间歇性的-并非总是发生.有问题的代码行是:

The error is intermittent - it doesn't always happen. The line of code in question is:

df = sqlContext.read.format('jdbc').options(
            url=url, 
            driver='com.ibm.db2.jcc.DB2Driver', 
            dbtable='SAMPLE.ASSETDATA'
        ).load()

关于stackoverflow也有一些类似的问题，但是他们并没有询问bluemix上的spark服务.

There are a few similar questions on stackoverflow, but they aren't asking about the spark service on bluemix.

推荐答案

在使用sqlContext之前创建一个新的SQLContext对象:

Create a new SQLContext object before using sqlContext:

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

，然后再次运行代码.

如果您使用开箱即用的sqlContext有多个笔记本，则会发生此错误.

This error happens if you have multiple notebooks using the out of box sqlContext.

这篇关于您必须使用Hive构建Spark.导出'SPARK_HIVE = true'的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！