在 Zeppelin 0.7.1 中运行 Spark 代码时获取 NullPointerException [英] Getting NullPointerException when running Spark Code in Zeppelin 0.7.1

问题描述

我已经安装了 Zeppelin 0.7.1.当我尝试执行示例 spark 程序(可用于 Zeppelin Tutorial 笔记本)时，出现以下错误

I have installed Zeppelin 0.7.1. When I tried to execute the Example spark program(which was available with Zeppelin Tutorial notebook), I am getting the following error

java.lang.NullPointerException
    at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
    at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
    at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:391)
    at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:380)
    at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
    at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:828)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
    at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

我还设置了配置文件 (zeppelin-env.sh) 以指向我的 Spark 安装 &Hadoop配置目录

I have also setup the config file(zeppelin-env.sh) to point to my Spark installation & Hadoop configuration directory

export SPARK_HOME="/${homedir}/sk"
export HADOOP_CONF_DIR="/${homedir}/hp/etc/hadoop"

我使用的 Spark 版本是 2.1.0 &Hadoop是2.7.3

The Spark version I am using is 2.1.0 & Hadoop is 2.7.3

我也使用默认的 Spark 解释器配置(所以 Spark 设置为在 Local mode 下运行)

Also I am using the default Spark Interpreter Configuration(so Spark is set to run in Local mode)

我在这里遗漏了什么吗?

Am I missing something here?

PS:我可以使用 spark-shell

推荐答案

我刚刚为 Zeppelin-0.7.2 解决了这个问题:

Just now I got solution of this issue for Zeppelin-0.7.2:

根本原因是:Spark 尝试设置 Hive 上下文，但 hdfs 服务没有运行，这就是为什么 HiveContext 变为 null 并抛出 null指针异常.

Root Cause is : Spark trying to setup Hive context, but hdfs services is not running, that's why HiveContext become null and throwing null pointer exception.

解决方案:
1.设置Saprk Home [可选]和HDFS.
2.运行HDFS服务
3.重启zeppelin服务器
或
1. 转到 Zeppelin 的解释器设置.
2. 选择 Spark 解释器
3. zeppelin.spark.useHiveContext = false

这篇关于在 Zeppelin 0.7.1 中运行 Spark 代码时获取 NullPointerException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！