用齐柏林飞艇在Kubernetes上火花 [英] Spark on kubernetes with zeppelin
问题描述
我正在按照本指南在使用minikube设置的本地kubernetes集群中运行一个齐柏林飞艇容器.
https://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html
我能够设置齐柏林飞艇并在其中运行一些示例代码.我已经下载了spark 2.4.5& 2.4.0源代码，并使用以下命令将其构建为支持kubernetes:
./build/mvn -Pkubernetes -DskipTests clean package
一旦构建了spark，就按照文章中的说明创建了一个docker容器:
bin/docker-image-tool.sh -m -t 2.4.X build
我将齐柏林飞艇配置为使用由kubernetes支持构建的spark图像.上面的文章解释了spark解释器将自动在kubernetes上配置spark以在客户端模式下运行并运行作业.
但是每当我尝试使用spark运行任何参数时，都会收到以下错误消息
Exception in thread "main" java.lang.IllegalArgumentException: basedir must be absolute: ?/.ivy2/local
我尝试在齐柏林飞艇中将火花配置spark.jars.ivy设置为指向临时目录，但这也不起作用.
我在这里发现了类似的问题: basedir必须是绝对的:?/.ivy2/local
但是我似乎无法将spark配置为与spark.jars.ivy /tmp/.ivy配置一起运行.在构建Spark时，我尝试使用 spark-defaults.conf 来构建Spark，但这似乎也不起作用.
很困惑这个问题，以及如何解决它的任何指导将不胜感激.
谢谢！
我也遇到了这个问题，但是我用于设置spark.jars.ivy=/tmp/.ivy的一种解决方法是将其设置为环境变量.
在您的Spark解释器设置中，添加以下属性:SPARK_SUBMIT_OPTIONS并将其值设置为--conf spark.jars.ivy=/tmp/.ivy.
这应该通过其他选项来触发提交，您的工作应该继续.
I am following this guide to run up a zeppelin container in a local kubernetes cluster set up using minikube.
https://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html
I am able to set up zeppelin and run some sample code there. I have downloaded spark 2.4.5 & 2.4.0 source code and built it for kubernetes support with the following command:
./build/mvn -Pkubernetes -DskipTests clean package
Once spark is built I created a docker container as explained in the article:
bin/docker-image-tool.sh -m -t 2.4.X build
I configured zeppelin to use the spark image which was built with kubernetes support. The article above explains that the spark interpreter will auto configure spark on kubernetes to run in client mode and run the job.
But whenever I try to run any parahgraph with spark I receive the following error
Exception in thread "main" java.lang.IllegalArgumentException: basedir must be absolute: ?/.ivy2/local
I tried setting the spark configuration spark.jars.ivy in zeppelin to point to a temp directory but that does not work either.
I found a similar issue here: basedir must be absolute: ?/.ivy2/local
But I can't seem to configure spark to run with the spark.jars.ivy /tmp/.ivy config. I tried building spark with the spark-defaults.conf when building spark but that does not seems to be working either.
Quite stumped at this problem and how to solve it any guidance would be appreciated.
Thanks!
I have also run into this problem, but a work-around I used for setting spark.jars.ivy=/tmp/.ivy is to rather set it is as an environment variable.
In your spark interpreter settings, add the following property: SPARK_SUBMIT_OPTIONS and set its value to --conf spark.jars.ivy=/tmp/.ivy.
This should pass additional options to spark submit and your job should continue.
这篇关于用齐柏林飞艇在Kubernetes上火花的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
