使用 zeppelin 在 kubernetes 上激发火花 [英] Spark on kubernetes with zeppelin

问题描述

我正在按照本指南在使用 minikube 设置的本地 kubernetes 集群中运行 zeppelin 容器.

I am following this guide to run up a zeppelin container in a local kubernetes cluster set up using minikube.

https://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html

我能够设置 zeppelin 并在那里运行一些示例代码.我已经下载了 spark 2.4.5 &2.4.0 源代码并使用以下命令构建它以支持 kubernetes:

I am able to set up zeppelin and run some sample code there. I have downloaded spark 2.4.5 & 2.4.0 source code and built it for kubernetes support with the following command:

./build/mvn -Pkubernetes -DskipTests clean package

spark 构建完成后，我按照文章中的说明创建了一个 docker 容器:

Once spark is built I created a docker container as explained in the article:

bin/docker-image-tool.sh -m -t 2.4.X build

我将 zeppelin 配置为使用基于 kubernetes 支持构建的 spark 映像.上面的文章解释了spark解释器会在kubernetes上自动配置spark以客户端模式运行并运行作业.

I configured zeppelin to use the spark image which was built with kubernetes support. The article above explains that the spark interpreter will auto configure spark on kubernetes to run in client mode and run the job.

但是每当我尝试使用 spark 运行任何段落时，我都会收到以下错误

But whenever I try to run any parahgraph with spark I receive the following error

Exception in thread "main" java.lang.IllegalArgumentException: basedir must be absolute: ?/.ivy2/local

我尝试在 zeppelin 中设置 spark 配置 spark.jars.ivy 以指向临时目录，但这也不起作用.

I tried setting the spark configuration spark.jars.ivy in zeppelin to point to a temp directory but that does not work either.

我在这里发现了一个类似的问题:basedir 必须是绝对的:?/.ivy2/local

I found a similar issue here: basedir must be absolute: ?/.ivy2/local

但我似乎无法将 spark 配置为使用 spark.jars.ivy/tmp/.ivy 配置运行.我在构建 spark 时尝试使用 spark-defaults.conf 构建 spark，但这似乎也不起作用.

But I can't seem to configure spark to run with the spark.jars.ivy /tmp/.ivy config. I tried building spark with the spark-defaults.conf when building spark but that does not seems to be working either.

对这个问题以及如何解决它感到非常困惑，我们将不胜感激.

Quite stumped at this problem and how to solve it any guidance would be appreciated.

谢谢！

推荐答案

我也遇到了这个问题，但是我用于设置 spark.jars.ivy=/tmp/.ivy 是将其设置为环境变量.

I have also run into this problem, but a work-around I used for setting spark.jars.ivy=/tmp/.ivy is to rather set it is as an environment variable.

在您的 spark 解释器设置中，添加以下属性:SPARK_SUBMIT_OPTIONS 并将其值设置为 --conf spark.jars.ivy=/tmp/.ivy.

In your spark interpreter settings, add the following property: SPARK_SUBMIT_OPTIONS and set its value to --conf spark.jars.ivy=/tmp/.ivy.

这应该传递额外的选项来触发提交，你的工作应该继续.

This should pass additional options to spark submit and your job should continue.

这篇关于使用 zeppelin 在 kubernetes 上激发火花的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！