在Kubernetes上运行Spark示例失败 [英] Run Spark example on Kubernetes failed

问题描述

• 我的Mac OS/X版本:10.15.3
• Minikube版本:1.9.2

我使用以下命令启动minikube，无需任何额外配置.

I start the minikube use the following command without any extra configuration.

minikube start --driver=virtualbox
--image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' --cpus 4 --memory 4096 --alsologtostderr

然后我从Spark官方网站下载spark-2.4.5-bin-hadoop2.7并通过以下命令构建spark图像

And I download spark-2.4.5-bin-hadoop2.7 from the Spark official website and build spark images by the following command

eval $(minikube docker-env)
./bin/docker-image-tool.sh -m -t 2.4.5 build

然后我在存储Spark 2.4.5的本地计算机中使用follwing命令运行Spark-pi.

then I run Spark-pi using the follwing command within my local machine where store the Spark 2.4.5.

kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=admin --serviceaccount=default:spark --namespace=default
./bin/spark-submit \
--master k8s://https://192.168.99.104:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.container.image=spark:2.4.5 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar

我收到以下错误

完整日志可以在完整日志中找到

谁能解释这个错误以及如何解决?

Can anyone explain this error and how to solve it?

推荐答案

请检查您使用Minikube启动的Kubernetes版本.

Please check the Kubernetes version you launched with Minikube.

Spark v2.4.5 fabric8 Kubernetes客户端v4.6.1与直至1.15.2的Kubernetes API兼容(请参阅 answer ).

Spark v2.4.5 fabric8 Kubernetes client v4.6.1 is compatible with Kubernetes API up to 1.15.2 (refer answer).

您可以通过在minikube start命令中添加--kubernetes-version标志来使用Minikube启动特定的Kubernetes API版本(请参考

You can launch the specific Kubernetes API version with Minikube by adding --kubernetes-version flag to minikube start command (refer docs).

此问题也可能是由此问题的注释中描述的OkHttp库错误引起的.

Also the issue might be caused by OkHttp library bug described in the comment of this qustion.

这篇关于在Kubernetes上运行Spark示例失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！