Zeppelin 中的 AWS Redshift 驱动程序 [英] AWS Redshift driver in Zeppelin
问题描述
我想使用笔记本 Zeppelin 在 Redshift 中探索我的数据.一个带有 Spark 的小型 EMR 集群正在运行.我正在加载 databricks 的 spark-redshift 库
I want to explore my data in Redshift using notebook Zeppelin. A small EMR cluster with Spark is running behind. I am loading databricks' spark-redshift library
%dep
z.reset()
z.load("com.databricks:spark-redshift_2.10:0.6.0")
然后
import org.apache.spark.sql.DataFrame
val query = "..."
val url = "..."
val port=5439
val table = "..."
val database = "..."
val user = "..."
val password = "..."
val df: DataFrame = sqlContext.read
.format("com.databricks.spark.redshift")
.option("url", s"jdbc:redshift://${url}:$port/$database?user=$user&password=$password")
.option("query",query)
.option("tempdir", "s3n://.../tmp/data")
.load()
df.show
但我收到错误
java.lang.ClassNotFoundException: Could not load an Amazon Redshift JDBC driver; see the README for instructions on downloading and configuring the official Amazon driver
我添加了选项
option("jdbcdriver", "com.amazon.redshift.jdbc41.Driver")
但不是为了更好.我想我需要在某个地方指定 redshift 的 JDBC 驱动程序,就像我将 --driver-class-path 传递给 spark-shell 一样,但是如何使用 zeppelin 来做到这一点?
but not for the better. I think I need to specify redshift's JDBC driver somewhere like I would passing --driver-class-path to spark-shell, but how to do that with zeppelin?
推荐答案
您可以使用 Zeppelin 的 依赖加载机制 或者,在 Spark 的情况下,使用 %dep
动态依赖加载器
You can add external jars with dependencies like the JDBC driver using either Zeppelin's dependency-loading mechanism or, in case of Spark, using %dep
dynamic dependency loader
当您的代码需要外部库时,您可以使用 %dep 解释器轻松完成以下工作,而不是下载/复制/重启 Zeppelin.
When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using %dep interpreter.
- 从 Maven 存储库递归加载库
- 从本地文件系统加载库
- 添加额外的 Maven 仓库
- 自动向 SparkCluster 添加库(您可以关闭)
后者看起来像:
%dep
// loads with all transitive dependencies from Maven repo
z.load("groupId:artifactId:version")
// or add artifact from filesystem
z.load("/path/to.jar")
并且按照惯例必须在注释的第一段中.
and by convention have to be in the first paragraph of the note.
这篇关于Zeppelin 中的 AWS Redshift 驱动程序的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!