Pyspark UDF 中自定义 Python 对象的使用 [英] Usage of custom Python object in Pyspark UDF

问题描述

运行以下 PySpark 代码时:

When running following piece of PySpark code:

nlp = NLPFunctions()

def parse_ingredients(ingredient_lines):
    parsed_ingredients = nlp.getingredients_bulk(ingredient_lines)[0]
    return list(chain.from_iterable(parsed_ingredients))


udf_parse_ingredients = UserDefinedFunction(parse_ingredients, ArrayType(StringType()))

我收到以下错误:_pickle.PicklingError:无法序列化对象:TypeError:无法pickle _thread.lock 对象

我想这是因为 PySpark 无法序列化这个自定义类.但是如何避免在每次运行 parse_ingredients_line 函数时实例化这个昂贵的对象的开销?

I imagine this is because PySpark can not serialize this custom class. But how can I avoid the overhead of instantiating this expensive object on every run of the parse_ingredients_line function?

推荐答案

我基于 (https://github.com/scikit-learn/scikit-learn/issues/6975) 通过使 NLPFunctions 类的所有依赖项可序列化.

I solved it based on (https://github.com/scikit-learn/scikit-learn/issues/6975) by making all dependencies of the NLPFunctions class serializable.

这篇关于Pyspark UDF 中自定义 Python 对象的使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！