SparkException:Python worker没有及时返回 [英] SparkException: Python worker did not connect back in time

问题描述

我正在尝试将Python作业提交到2个工作节点的Spark集群，但始终遇到以下问题，最终导致spark-submit失败:

I am trying to submit a Python job to a 2 worker node Spark cluster but am persistently seeing the following problem, which eventually causes spark-submit to fail:

15/07/04 21:30:40 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID
 2, workernode0.rhom-spark.b9.internal.cloudapp.net):    
org.apache.spark.SparkException: Python worker did not connect back in time
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:135)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:64)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:102)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:278)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:245)
    at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:305)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:278)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:245)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:56)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Accept timed out
    at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)

    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
    at java.net.ServerSocket.implAccept(ServerSocket.java:530)
    at java.net.ServerSocket.accept(ServerSocket.java:498)
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:130)
    ... 15 more

我要提交以下内容的工作

I'm submitting the job with the following

spark-submit --master yarn --py-files tile.py --num-executors 1 --executor-memory 2g main.py

有什么想法吗?

推荐答案

因此，当python worker进程无法连接到spark执行程序JVM时，就会发生这种情况. Spark使用套接字与辅助进程进行通信.发生这种情况的原因有很多，确切的详细信息可能会在执行者/工人计算机的日志中.

So this happens when the python worker process fails to connect to the spark executor JVM. Spark uses sockets to communicate with the worker process. There are a large number of reasons why this could happen, and the exact specific details will likely be in the logs on the executor/worker machines.

这篇关于SparkException:Python worker没有及时返回的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！