Spark - 将 JDBC 驱动程序 JAR 添加到 Google Dataproc [英] Spark - Adding JDBC Driver JAR to Google Dataproc

问题描述

我正在尝试通过 JDBC 编写:

I am trying to write via JDBC:

df.write.jdbc("jdbc:postgresql://123.123.123.123:5432/myDatabase", "myTable", props)

Spark 文档 解释了配置选项 spark.driver.extraClassPath 如果在客户端模式(Dataproc 运行的模式)下运行，则不能用于添加 JDBC 驱动程序 JAR，因为 JVM 已经启动.

The Spark docs explain that the configuration option spark.driver.extraClassPath cannot be used to add JDBC Driver JARs if running in client mode (which is the mode Dataproc runs in) since the JVM has already been started.

我尝试在 Dataproc 的提交命令中添加 JAR 路径:

I tried adding the JAR path in Dataproc's submit command:

gcloud beta dataproc jobs submit spark ... 
     --jars file:///home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar

我还添加了加载驱动程序的命令:

I also added the command to load the driver:

  Class.forName("org.postgresql.Driver")

但我仍然收到错误:

java.sql.SQLException: No suitable driver found for jdbc:postgresql://123.123.123.123:5432/myDatabase 

推荐答案

根据我的经验，将 driver 添加到属性通常可以解决问题:

From my experience adding driver to the properties usually solves the problem:

props.put("driver", "org.postgresql.Driver")
db.write.jdbc(url, table, props)

这篇关于Spark - 将 JDBC 驱动程序 JAR 添加到 Google Dataproc的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！