％AddJar加载的Spark-cloudant软件包1.6.4未被笔记本使用 [英] Spark-cloudant package 1.6.4 loaded by %AddJar does not get used by notebook

问题描述

我正尝试在笔记本电脑上使用最新的spark-cloudant软件包：

I'm trying to use the latest spark-cloudant package with a notebook:

%AddJar -f https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.4/cloudant-spark-v1.6.4-167.jar

哪些输出：

Starting download from https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.4/cloudant-spark-v1.6.4-167.jar
Finished download of cloudant-spark-v1.6.4-167.jar

之后：

val dfReader = sqlContext.read.format("com.cloudant.spark")
dfReader.option("cloudant.host", sourceDB.host)
if (sourceDB.username.isDefined && sourceDB.username.get.nonEmpty) dfReader.option("cloudant.username", sourceDB.username.get)
if (sourceDB.password.isDefined && sourceDB.password.get.nonEmpty) dfReader.option("cloudant.password", sourceDB.password.get)
val df = dfReader.load(sourceDB.database).cache()

哪些输出：

Use connectorVersion=1.6.3, dbName=ratingdb, indexName=null, viewName=null,jsonstore.rdd.partitions=5, + jsonstore.rdd.maxInPartition=-1,jsonstore.rdd.minInPartition=10, jsonstore.rdd.requestTimeout=900000,bulkSize=20, schemaSampleSize=1

连接器是1.6.3。我的笔记本是：

The connector is 1.6.3. My notebook is:

Scala 2.10 with Spark 1.6

我尝试重新启动内核，但这没有帮助。

I've tried restarting the kernel but that didn't help.

其他调试信息：

Server Information:

You are using Jupyter notebook.

The version of the notebook server is 4.2.0 and is running on:
Python 2.7.11 (default, Jun 24 2016, 12:41:03) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

Current Kernel Information:

IBM Spark Kernel

---

更新

我尝试了以下操作：

import sys.process._

"test -d ~/data/libs/scala-2.10" #|| "mkdir -p ~/data/libs/scala-2.10" !
"wget -c -O ~/data/libs/scala-2.10/cloudant-spark-v1.6.4-167.jar https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.4/cloudant-spark-v1.6.4-167.jar" !
"ls ~/data/libs/scala-2.10/" !

println("Now restart the kernel")

不幸的是，这没有't work-1.6.3仍在使用。

Unfortunately, this didn't work - 1.6.3 is still being used.

更新2

在上面的代码中似乎无法将tilda解析到我的HOME文件夹中。

It appears that the tilda was not getting resolved to my HOME folder in the above code.

请参阅工作解决方案的答案。

See the answer for the working solution.

推荐答案

从scala笔记本运行以下代码对我有用：

Running the following code from a scala notebook worked for me:

import sys.process._

val HOME = sys.env("HOME")
val DESTDIR = s"${HOME}/data/libs/scala-2.10"

s"test -d ${DESTDIR}" #|| s"mkdir -p ${DESTDIR}" !
s"wget -q -c -O ${DESTDIR}/cloudant-spark-v1.6.4-167.jar https://github.com/cloudant-labs/spark-cloudant/releases/download/v1.6.4/cloudant-spark-v1.6.4-167.jar" !
s"ls ${DESTDIR}/" !

我还要求Spark服务的产品管理人员正式升级此库。

I have also requested product management for the spark service to officially upgrade this library.

这篇关于％AddJar加载的Spark-cloudant软件包1.6.4未被笔记本使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！