如何在Data Science Experience上实施TensorFrames Spark软件包? [英] How do I implement the TensorFrames Spark package on Data Science Experience?
问题描述
我已经能够导入软件包:
I've been able to import the package:
import pixiedust
pixiedust.installPackage("databricks:tensorframes:0")
但是当我尝试一个简单的例子时:
But when I try a simple example:
import tensorflow as tf
import tensorframes as tfs
from pyspark.sql import Row
data = [Row(x=[float(x), float(2 * x)],
key=str(x % 2),
z = float(x+1)) for x in range(1, 6)]
df = spark.createDataFrame(data)
tfs.print_schema(df)
我收到以下错误:
...
Py4JJavaError: An error occurred while calling o97.loadClass.
: java.lang.NoClassDefFoundError:com.typesafe.scalalogging.slf4j.LazyLogging
...
我已经查看了这个问题,似乎有依赖项树中的较旧的scala-logging-slf4j
工件. 如何删除此工件?假定删除后,我可以使用PixieDust添加新版本:
I've looked up the issue and it seems like there is an older scala-logging-slf4j
artifact in the dependency tree. How do I remove this artifact? Once it's removed, I assume I can add a newer version with PixieDust:
pixiedust.installPackage("https://mvnrepository.com/artifact/com.typesafe.scala-logging/scala-logging-slf4j_2.10/2.1.2")
pixiedust.installPackage("https://mvnrepository.com/artifact/com.typesafe.scala-logging/scala-logging-api_2.10/2.1.2")
pixiedust.installPackage("https://mvnrepository.com/artifact/com.typesafe.scala-logging/scala-logging-slf4j_2.10/2.1.2")
pixiedust.installPackage("https://mvnrepository.com/artifact/com.typesafe.scala-logging/scala-logging-api_2.10/2.1.2")
推荐答案
IBM支持小组的查尔斯帮助我找到了要包括的罐子.
Charles from IBM support helped me find which jars to include:
pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-slf4j_2.10/2.1.2/scala-logging-slf4j_2.10-2.1.2.jar")
pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-api_2.10/2.1.2/scala-logging-api_2.10-2.1.2.jar")
pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-slf4j_2.10/2.1.2/scala-logging-slf4j_2.10-2.1.2.jar")
pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-api_2.10/2.1.2/scala-logging-api_2.10-2.1.2.jar")
这从技术上解决了第一个错误,但是TesnsorFrames仍然无法正常工作.我将发布另一个更具体的问题.
This technically solves the first error, however TesnsorFrames still aren't working. I'll post another question that is more specific.
这篇关于如何在Data Science Experience上实施TensorFrames Spark软件包?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!