如何在 Apache Spark 预构建版本中添加任何新库,如 spark-csv [英] How to add any new library like spark-csv in Apache Spark prebuilt version
问题描述
我已经构建了 Spark-csv 并且能够使用以下命令从 pyspark shell 中使用它
bin/spark-shell --packages com.databricks:spark-csv_2.10:1.0.3
获取错误
<预><代码>>>>df_cat.save("k.csv","com.databricks.spark.csv")回溯(最近一次调用最后一次):文件<stdin>",第 1 行,在 <module> 中文件/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/pyspark/sql/dataframe.py",第209行,保存self._jdf.save(source, jmode, joptions)文件/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",第538行,__call__文件/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py",第300行,get_return_valuepy4j.protocol.Py4JJavaError我应该将 jar 文件放在我的 Spark 预构建设置中的什么位置,以便我也可以直接从 python 编辑器访问 spark-csv
.
在我使用 spark-csv 的时候,我也不得不下载 commons-csv
jar(不确定它是否仍然相关).两个 jar 都在 spark 分发文件夹中.
我下载的 jars 如下:
wget http://search.maven.org/remotecontent?filepath=org/apache/commons/commons-csv/1.1/commons-csv-1.1.jar -O commons-csv-1.1.jar<br/>wget http://search.maven.org/remotecontent?filepath=com/databricks/spark-csv_2.10/1.0.0/spark-csv_2.10-1.0.0.jar -O spark-csv_2.10-1.0.0.jar
然后使用参数启动 python spark shell:
./bin/pyspark --jars "spark-csv_2.10-1.0.0.jar,commons-csv-1.1.jar"
并从 csv 文件中读取 spark 数据帧:
from pyspark.sql import SQLContextsqlContext = SQLContext(sc)df = sqlContext.load(source="com.databricks.spark.csv", path = "/path/to/you/file.csv")df.show()
I have build the Spark-csv and able to use the same from pyspark shell using the following command
bin/spark-shell --packages com.databricks:spark-csv_2.10:1.0.3
error getting
>>> df_cat.save("k.csv","com.databricks.spark.csv")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/pyspark/sql/dataframe.py", line 209, in save
self._jdf.save(source, jmode, joptions)
File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError
Where should I place the jar file in my spark pre-built setup so that I will be able to access spark-csv
from python editor directly as well.
At the time I used spark-csv, I also had to download commons-csv
jar (not sure it is still relevant). Both jars where in the spark distribution folder.
I downloaded the jars as follow:
wget http://search.maven.org/remotecontent?filepath=org/apache/commons/commons-csv/1.1/commons-csv-1.1.jar -O commons-csv-1.1.jar<br/> wget http://search.maven.org/remotecontent?filepath=com/databricks/spark-csv_2.10/1.0.0/spark-csv_2.10-1.0.0.jar -O spark-csv_2.10-1.0.0.jar
then started the python spark shell with the arguments:
./bin/pyspark --jars "spark-csv_2.10-1.0.0.jar,commons-csv-1.1.jar"
and read a spark dataframe from a csv file:
from pyspark.sql import SQLContext sqlContext = SQLContext(sc) df = sqlContext.load(source="com.databricks.spark.csv", path = "/path/to/you/file.csv") df.show()
这篇关于如何在 Apache Spark 预构建版本中添加任何新库,如 spark-csv的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!