spark命令中--driver-class-path的用途是什么? [英] What is the use of --driver-class-path in the spark command?

问题描述

根据spark 文档，

要开始使用，您将需要在spark类路径上包含特定数据库的JDBC驱动程序.例如，要从Spark Shell连接到postgres，您可以运行以下命令:

To get started you will need to include the JDBC driver for you particular database on the spark classpath. For example, to connect to postgres from the Spark Shell you would run the following command:

bin/spark-shell --driver-class-path postgresql-9.4.1207.jar --jars postgresql-9.4.1207.jar

在没有--driver-class-path 的情况下工作正常.那么，spark命令中--driver-class-path的用途是什么?

推荐答案

--driver-class-path或spark.driver.extraClassPath只能用于修改Spark驱动程序的类路径.这对于执行者不需要的库(例如，仅在本地使用的任何代码)很有用.

--driver-class-path or spark.driver.extraClassPath can be used for to modify class path only for the Spark driver. This is useful for libraries which are not required by the executors (for example any code that is used only locally).

与此相比，--jars或spark.jars不仅将jar添加到驱动程序和执行程序类路径，而且还将在群集上分发归档文件.如果特定的jar仅由驱动程序使用，则这是不必要的开销.

Compared to that, --jars or spark.jars will not only add jars to both driver and executor classpath, but also distribute archives over the cluster. If particular jar is used only by the driver this is unnecessary overhead.

这篇关于spark命令中--driver-class-path的用途是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！