如何在同一个Spark项目中同时使用Scala和Python? [英] How to Use both Scala and Python in a same Spark project?

问题描述

是否可以将 Spark RDD 传递给Python?

Is that possible to pipe Spark RDD to Python?

因为我需要一个python库来对数据进行一些计算，但是我的主要Spark项目基于Scala. 有没有办法将它们混在一起或让python访问相同的spark上下文?

Because I need a python library to do some calculation on my data, but my main Spark project is based on Scala. Is there a way to mix them both or let python access the same spark context?

推荐答案

您确实可以使用Scala和Spark以及常规Python脚本通过管道传递到python脚本.

You can indeed pipe out to a python script using Scala and Spark and a regular Python script.

test.py

#!/usr/bin/python

import sys

for line in sys.stdin:
  print "hello " + line

火花壳(scala)

val data = List("john","paul","george","ringo")

val dataRDD = sc.makeRDD(data)

val scriptPath = "./test.py"

val pipeRDD = dataRDD.pipe(scriptPath)

pipeRDD.foreach(println)

输出

你好约翰

你好林格

你好乔治

你好保罗

这篇关于如何在同一个Spark项目中同时使用Scala和Python?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！