在pyspark到其他节点运输Python模块？ [英] shipping python modules in pyspark to other nodes?

问题描述

我怎么能舰C编译的模块（例如，蟒蛇 - 莱文斯坦）在火花群集中的每个节点？

How can I ship C compiled modules (for example, python-Levenshtein) to each node in a spark cluster?

我知道我可以使用独立的python脚本（例如code以下）船舶火花Python文件：

I know that I can ship python files in spark using a standalone python script (example code below):

from pyspark import SparkContext
sc = SparkContext("local", "App Name", pyFiles=['MyFile.py', 'MyOtherFile.py'])

但在情况下不存在的.py'我怎么运输模块？

but in situations where there is no '.py' how do I ship the module?

推荐答案

如果你可以打包模块插入 .egg 或 .zip文件文件，你应该能够列出它在 pyFiles 构建您的SparkContext（或者你可以通过<以后添加时href=\"https://spark.apache.org/docs/latest/api/python/pyspark.context.SparkContext-class.html#addPyFile\">sc.addPyFile).

If you can package your module into a .egg or .zip file, you should be able to list it in pyFiles when constructing your SparkContext (or you can add it later through sc.addPyFile).

有关使用setuptools的Python库，你可以运行蟒蛇setup.py bdist_egg 来建立一个鸡蛋分布。

For Python libraries that use setuptools, you can run python setup.py bdist_egg to build an egg distribution.

另一个选择是群集范围，通过使用PIP / easy_install的每台机器上或通过在集群范围的文件系统（如NFS）共享一个Python安装安装库。

Another option is to install the library cluster-wide, either by using pip/easy_install on each machine or by sharing a Python installation over a cluster-wide filesystem (like NFS).

这篇关于在pyspark到其他节点运输Python模块？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！