Spark pyspark与Spark提交 [英] Spark pyspark vs spark-submit

问题描述

有关spark-submit的文档中的内容如下:

The documentation on spark-submit says the following:

Spark的bin目录中的spark-submit脚本用于启动 集群上的应用程序.

The spark-submit script in Spark’s bin directory is used to launch applications on a cluster.

关于pyspark，它表示以下内容:

Regarding the pyspark it says the following:

您还可以使用bin/pyspark启动交互式Python Shell.

You can also use bin/pyspark to launch an interactive Python shell.

这个问题听起来很愚蠢，但是当我通过pyspark运行命令时，它们也在集群"上运行，对吗?它们不只在主节点上运行，对吧?

This question may sound stupid, but when i am running the commands though pyspark they also run on the "cluster", right? They do not run on the master node only, right?

推荐答案

这两者之间没有实际区别.如果未配置，则两者都将在本地模式下执行代码.如果配置了master(通过--master命令行参数或spark.master配置)，则将使用相应的群集执行程序.

There is no practical difference between these two. If not configured otherwise both will execute code in a local mode. If master is configured (either by --master command line parameter or spark.master configuration) corresponding cluster will be used to execute the program.

这篇关于Spark pyspark与Spark提交的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！