无法初始化火花上下文 [英] Couldn't initialize spark context

问题描述

我正在尝试在 Python 中初始化 Spark 上下文变量.

I am trying to initialize Spark Context variable in Python.

from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("test").setMaster("local")
sc = SparkContext(conf=conf)

但我收到以下错误:

py4j.protocol.Py4JJavaError: 调用时发生错误None.org.apache.spark.api.java.JavaSparkContext.:java.lang.NoClassDefFoundError:无法初始化类org.apache.spark.internal.config.package$
在 org.apache.spark.SparkConf.validateSettings(SparkConf.scala:546)在 org.apache.spark.SparkContext.(SparkContext.scala:373)在 org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)在 sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)在 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)在 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)在 java.lang.reflect.Constructor.newInstance(Constructor.java:423)在 py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)在 py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)在 py4j.Gateway.invoke(Gateway.java:236)在 py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)在 py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)在 py4j.GatewayConnection.run(GatewayConnection.java:214)在 java.lang.Thread.run(Thread.java:748)

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.internal.config.package$
at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:546) at org.apache.spark.SparkContext.(SparkContext.scala:373) at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:236) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748)

我四处寻找解决方案，但没有得到确切的解决方案.请帮忙.

I have looked around for solution but not did not get exact solution. Please help.

推荐答案

将 SPARK_LOCAL_IP 环境变量设置为 localhost 解决了我的错误.

Setting SPARK_LOCAL_IP environment variable to localhost solved my error.

这篇关于无法初始化火花上下文的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！