在 Windows 10 或 7 中的 Apache Zeppelin 中运行 Spark 程序时出错 [英] Getting error while running spark programs in Apache Zeppelin in Windows 10 or 7
问题描述
我是 Apache Zeppelin 的新手.安装 0.8.0 并使用 7000 端口访问 Zeppelin.配置了几个路径，如下所述.
I am new to Apache Zeppelin. Installed 0.8.0 and using 7000 port to access Zeppelin. Configured few paths as mentioned below.
JAVA_HOME:C:\Program Files\Java\jdk1.8.0_144HADOOP_HOME: C:\winutilsZEPPELIN_HOME: C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all
JAVA_HOME: C:\Program Files\Java\jdk1.8.0_144 HADOOP_HOME: C:\winutils ZEPPELIN_HOME: C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all
所有这些变量都包含在路径变量和各自的 bin 文件夹中
all these variables are included in path variable and respective bin folders
尝试运行 spark 程序并收到以下错误.尝试了多种选择来修复它，但无法解决.请帮忙.
Tried to run spark program and getting the below error. Tried with multiple options to fix it, but unable. Please help.
Spark 计划:％火花println(sc.appName)
Spark Program: %spark println(sc.appName)
错误:DEBUG [2018-07-29 00:06:05,371] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:153) - callbackServer 正在服务信息 [2018-07-29 00:06:05,380] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:190) - 运行解释器进程 [C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all\bin\interpreter.cmd, -d, C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all/interpreter/spark,-c, 10.120.44.23, -p, 57136, -r, :, -l, C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all/local-repo/spark, -g, 火花]DEBUG [2018-07-29 00:06:09,625] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 警告:本地 jar C:\Users\vvellabo\57136 不存在，跳过.DEBUG [2018-07-29 00:06:09,626] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 警告:本地 jar C:\Users\vvellabo\10.120.44.23 不存在，跳过.调试 [2018-07-29 00:06:09,627] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - java.lang.ClassNotFoundException: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer调试 [2018-07-29 00:06:09,628] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 java.net.URLClassLoader.findClass(URLClassLoader.java:381)调试 [2018-07-29 00:06:09,630] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 java.lang.ClassLoader.loadClass(ClassLoader.java:424)调试 [2018-07-29 00:06:09,632] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 java.lang.ClassLoader.loadClass(ClassLoader.java:357)DEBUG [2018-07-29 00:06:09,633] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 java.lang.Class.forName0(Native Method)调试 [2018-07-29 00:06:09,634] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 java.lang.Class.forName(Class.java:348)调试 [2018-07-29 00:06:09,635] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 org.apache.spark.util.Utils$.classForName(Utils.scala:238)调试 [2018-07-29 00:06:09,641] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)调试 [2018-07-29 00:06:09,642] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)调试 [2018-07-29 00:06:09,644] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)调试 [2018-07-29 00:06:09,645] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)调试 [2018-07-29 00:06:09,647] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 在 org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)DEBUG [2018-07-29 00:06:09,656] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 2018-07-29 00:06:09 INFO ShutdownHookManager:54 - 调用关闭钩子调试 [2018-07-29 00:06:09,659] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 2018-07-29 00:06:09 INFO ShutdownHookManager:54 - 删除目录 C:\Users\vvellabo\AppData\Local\Temp\spark-427c3202-c243-4761-86ce-ea51a27a881c信息 [2018-07-29 00:06:09,747] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessFailed]:250) - 解释器进程失败 {}org.apache.commons.exec.ExecuteException:进程退出并出现错误:101(退出值:101)在 org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)在 org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)在 org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)在 java.lang.Thread.run(Thread.java:748)错误 [2018-07-29 00:07:05,382] ({pool-2-thread-2} Job.java[run]:190) - 作业失败java.lang.RuntimeException: 警告:本地 jar C:\Users\vvellabo\57136 不存在，跳过.警告:本地jar C:\Users\vvellabo\10.120.44.23 不存在，跳过.
Error: DEBUG [2018-07-29 00:06:05,371] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:153) - callbackServer is serving now INFO [2018-07-29 00:06:05,380] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:190) - Run interpreter process [C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all\bin\interpreter.cmd, -d, C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all/interpreter/spark, -c, 10.120.44.23, -p, 57136, -r, :, -l, C:\zeppelin\zeppelin-0.8.0-bin-all\zeppelin-0.8.0-bin-all/local-repo/spark, -g, spark] DEBUG [2018-07-29 00:06:09,625] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - Warning: Local jar C:\Users\vvellabo\57136 does not exist, skipping. DEBUG [2018-07-29 00:06:09,626] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - Warning: Local jar C:\Users\vvellabo\10.120.44.23 does not exist, skipping. DEBUG [2018-07-29 00:06:09,627] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - java.lang.ClassNotFoundException: org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer DEBUG [2018-07-29 00:06:09,628] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at java.net.URLClassLoader.findClass(URLClassLoader.java:381) DEBUG [2018-07-29 00:06:09,630] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at java.lang.ClassLoader.loadClass(ClassLoader.java:424) DEBUG [2018-07-29 00:06:09,632] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at java.lang.ClassLoader.loadClass(ClassLoader.java:357) DEBUG [2018-07-29 00:06:09,633] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at java.lang.Class.forName0(Native Method) DEBUG [2018-07-29 00:06:09,634] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at java.lang.Class.forName(Class.java:348) DEBUG [2018-07-29 00:06:09,635] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at org.apache.spark.util.Utils$.classForName(Utils.scala:238) DEBUG [2018-07-29 00:06:09,641] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851) DEBUG [2018-07-29 00:06:09,642] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) DEBUG [2018-07-29 00:06:09,644] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) DEBUG [2018-07-29 00:06:09,645] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) DEBUG [2018-07-29 00:06:09,647] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) DEBUG [2018-07-29 00:06:09,656] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 2018-07-29 00:06:09 INFO ShutdownHookManager:54 - Shutdown hook called DEBUG [2018-07-29 00:06:09,659] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:298) - 2018-07-29 00:06:09 INFO ShutdownHookManager:54 - Deleting directory C:\Users\vvellabo\AppData\Local\Temp\spark-427c3202-c243-4761-86ce-ea51a27a881c INFO [2018-07-29 00:06:09,747] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessFailed]:250) - Interpreter process failed {} org.apache.commons.exec.ExecuteException: Process exited with an error: 101 (Exit value: 101) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) at java.lang.Thread.run(Thread.java:748) ERROR [2018-07-29 00:07:05,382] ({pool-2-thread-2} Job.java[run]:190) - Job failed java.lang.RuntimeException: Warning: Local jar C:\Users\vvellabo\57136 does not exist, skipping. Warning: Local jar C:\Users\vvellabo\10.120.44.23 does not exist, skipping.
谢谢vvell
推荐答案
我遇到了类似的问题.如果有人解决了这个问题，请继续回到这个问题.
此链接将为您提供帮助！
I faced a similar issue. Kept coming back to this question in case someone solves it.
This link will help you!
如果您不想执行这些步骤，只需删除 SPARK_HOME 环境变量即可.Zeppelin 拥有自己的火花罐库.接下来去
If you do not wish to go through the steps, simply delete the SPARK_HOME environment variable. Zeppelin has it's own library of spark jars. Next go to
%Zeppelin_HOME%\conf\
%Zeppelin_HOME%\conf\
并重命名
zeppelin-env.cmd.template到zeppelin-env.cmd
zeppelin-env.cmd.template to zeppelin-env.cmd
并添加以下几行:
set JAVA="C:\Program Files\Java\jdk1.8.0_181"
set JAVA_HOME="%JAVA%"
确保输入正确的 jdk 路径.保存并使用命令 bin\zeppelin.cmd 启动 zepplin
您的带 Spark 的齐柏林飞艇已准备就绪！通过运行任何简单代码进行确认:例如，sc.version
希望这会有所帮助！
Make sure you enter the correct path to your jdk.
Save it and start zepplin with the command bin\zeppelin.cmd
Your zeppelin with Spark is ready! Confirm by running any simple code: For example, sc.version
Hope this helps!
这篇关于在 Windows 10 或 7 中的 Apache Zeppelin 中运行 Spark 程序时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
