为什么启动我的pyspark时它会像接受的那样悬挂在纱线上？ [英] Why does my pyspark just hang as ACCEPTED in yarn when I launch it?

问题描述

我只是在 Linux 中启动了一个新的 AWS 实例。并且，我在上面安装了 pyspark 。它具有 spark 1.6 。

I just spun up a new AWS instance in Linux. And, I installed pyspark on it. It has spark 1.6.

我正在运行 pyspark 和 yarn 。当我在终端中执行命令 pyspark 时，它首先启动，但随后我收到消息：

I'm running pyspark with yarn. When I do the command pyspark in the terminal, it launches initially, but then I get the message:

dd/mm/YY HH:MM:SS INFO yarn.Client: Application report for application_XXXXXXXXXXX_XXXX (state: ACCEPTED)

.....然后这将永远持续下去。

.....and then this just continues for forever.

因此，我检查了纱线看看是否还有其他东西在运行：

So, I checked yarnto see if anything else was running:

yarn application -list

显示我的应用程序正在运行。如何打开 pyspark 外壳并使我的应用程序启动，而不仅仅是接受？

And ONLY shows my application running. How do I open up the pyspark shell and get my application to start rather than just being ACCEPTED?

推荐答案

您可以尝试运行spark-shell并查看其是否进入运行状态吗？

Can you try to run spark-shell and see if that goes into running state or not?

当yarn没有从其请求资源时，会发生这种情况。

This happens when yarn doesn't have requested resources from it.

示例：假设yarn具有5gb的可用内存，而您请求的是10gb。您的工作将停留在接受阶段，直到获得请求的内存为止。

Example: Lets say yarn has 5gb of free memory available and you are requesting 10gb. Your job would be stuck in Accepted phase till it gets the requested memory.

尝试使用以下配置运行您的工作：

Try running your job with below config:

pyspark --total-executor-cores 2 --executor-memory 1G

这篇关于为什么启动我的pyspark时它会像接受的那样悬挂在纱线上？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！