纱，客户端模式和纱线群集模式之间的区别 [英] Difference between yarn-client mode and yarn-cluster mode

问题描述

我运行在纱线上的客户端模式类似code以及纱线集群模式，而有一个小问题。我的code执行完全当我在客户端模式下运行它，但作出纱线群集节点上运行时失败。

I am having a little problem while running similar code on the yarn-client mode as well as the yarn-cluster mode. My code executes perfectly when I run it in the client mode, but fails when made to run on the yarn-cluster node.

它抛出一个文件而不是文件例外，指出pyspark.zip文件无法找到。任何深入了解这将是有益的。

It throws a file not file exception, stating that pyspark.zip file could not be found. Any insight into this would be helpful.

推荐答案

在纱线集群模式下，司机在申请硕士（纱线容器内）运行。在纱客户模式，它运行在客户端。

In yarn-cluster mode, the driver runs in the Application Master (inside a YARN container). In yarn-client mode, it runs in the client.

在纱线集群模式下，火花外壳不被支持。

In yarn-cluster mode, the spark-shell is not supported.

现在回到你的问题：你使用的星火版本？在下面1.4版本，纱线运行pyspark目前仅限于纱线的客户端模式（见火花5162 ）

Coming back to your problem: which version of Spark are you using ? In version below 1.4, running pyspark in yarn is currently limited to yarn-client mode (see SPARK-5162)

这篇关于纱，客户端模式和纱线群集模式之间的区别的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！