是否可以从 Scala(spark) 调用 python 函数 [英] Is it possible to call a python function from Scala(spark)

问题描述

我正在创建一个 spark 作业，该作业需要使用 Python 编写的函数将一列添加到数据帧中.其余的处理使用 Scala 完成.

I am creating a spark job that requires a column to be added to a dataframe using a function written in python. The rest of the processing is done using Scala.

我找到了如何从 pyspark 调用 Java/Scala 函数的示例:

I have found examples of how to call a Java/Scala function from pyspark:

• https://community.hortonworks.com/questions/110844/is-it-possible-to-call-a-scala-function-in-pythonp.html
• http://aseigneurin.github.io/2016/09/01/spark-calling-scala-code-from-pyspark.html

我发现以另一种方式发送数据的唯一示例是使用 pipe

The only examples I have found to send data the other way is using pipe

• 如何同时使用 Scala和 Python 在同一个 Spark 项目中?

我是否可以将整个数据帧发送到 python 函数，让函数处理数据并添加额外的列，然后将生成的数据帧发送回调用 Scala 函数?

Is it possible for me to send the entire dataframe to a python function, have the function manipulate the data and add additional columns and then send the resulting dataframe back to the calling Scala function?

如果这不可能，我当前的解决方案是运行 pyspark 进程并调用多个 Scala 函数来操作数据帧，这并不理想.

If this isn't possible my current solution is to run a pyspark process and call multiple Scala functions to manipulate the dataframe, this isn't ideal.

推荐答案

只需从 Python 注册一个 UDF，然后从 Scala 评估使用该函数对 DataFrame 的 SQL 语句 - 就像一个魅力，刚刚尝试过;)https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook 是在 Toree 中运行笔记本的好方法，该笔记本混合了调用相同 Spark 上下文的 Scala 和 Python 代码.

Just register a UDF from Python, and then from Scala evaluate an SQL statement that uses the function against a DataFrame - works like a charm, just tried it ;) https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook is a good way to run a notebook in Toree that mixes Scala and Python code calling the same Spark context.

这篇关于是否可以从 Scala(spark) 调用 python 函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！