sc未在SparkContext中定义 [英] sc is not defined in SparkContext
本文介绍了sc未在SparkContext中定义的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我的Spark软件包是spark-2.2.0-bin-hadoop2.7.
My Spark package is spark-2.2.0-bin-hadoop2.7.
我将spark变量导出为
I exported spark variables as
export SPARK_HOME=/home/harry/spark-2.2.0-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH
我通过
pyspark
我能够从Spark加载程序包
I am able to load packages from spark
from pyspark import SparkContext, SQLContext
from pyspark.ml.regression import LinearRegression
print(SQLContext)
输出为
<class 'pyspark.sql.context.SQLContext'>
但是我的错误是
print(sc)
"sc is undefined"
plz谁能帮助我...!
plz can anyone help me out ...!
推荐答案
在 pysparkShell 中, SparkContext
已初始化为 SparkContext(app = PySparkShell,master =local [*])
,因此您只需要使用 getOrCreate()
将 SparkContext
设置为变量
In pysparkShell, SparkContext
is already initialized as SparkContext(app=PySparkShell, master=local[*])
so you just need to use getOrCreate()
to set the SparkContext
to a variable as
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
出于在简单的本地模式下进行编码的目的,您可以执行以下操作
For coding purpose in simple local mode, you can do the following
from pyspark import SparkConf, SparkContext, SQLContext
conf = SparkConf().setAppName("test").setMaster("local")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
print(sc)
print(sqlContext)
这篇关于sc未在SparkContext中定义的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文