Spark-将JDBC驱动程序JAR添加到Google Dataproc [英] Spark - Adding JDBC Driver JAR to Google Dataproc

问题描述

我正在尝试通过JDBC进行编写:

I am trying to write via JDBC:

df.write.jdbc("jdbc:postgresql://123.123.123.123:5432/myDatabase", "myTable", props)

Spark文档解释了配置选项spark.driver.extraClassPath无法如果由于已启动JVM而在客户端模式(Dataproc所运行的模式)下运行，则可用于添加JDBC驱动程序JAR.

The Spark docs explain that the configuration option spark.driver.extraClassPath cannot be used to add JDBC Driver JARs if running in client mode (which is the mode Dataproc runs in) since the JVM has already been started.

我尝试在Dataproc的Submit命令中添加JAR路径:

I tried adding the JAR path in Dataproc's submit command:

gcloud beta dataproc jobs submit spark ... 
     --jars file:///home/bryan/org.postgresql.postgresql-9.4-1203-jdbc41.jar

我还添加了用于加载驱动程序的命令:

I also added the command to load the driver:

  Class.forName("org.postgresql.Driver")

但是我仍然收到错误消息:

But I still get the error:

java.sql.SQLException: No suitable driver found for jdbc:postgresql://123.123.123.123:5432/myDatabase 

推荐答案

根据我的经验，将driver添加到属性通常可以解决问题:

From my experience adding driver to the properties usually solves the problem:

props.put("driver", "org.postgresql.Driver")
db.write.jdbc(url, table, props)

这篇关于Spark-将JDBC驱动程序JAR添加到Google Dataproc的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！