在Pyspark UDF中使用自定义Python对象 [英] Usage of custom Python object in Pyspark UDF

问题描述

在运行以下PySpark代码时:

When running following piece of PySpark code:

nlp = NLPFunctions()

def parse_ingredients(ingredient_lines):
    parsed_ingredients = nlp.getingredients_bulk(ingredient_lines)[0]
    return list(chain.from_iterable(parsed_ingredients))


udf_parse_ingredients = UserDefinedFunction(parse_ingredients, ArrayType(StringType()))

我收到以下错误: _pickle.PicklingError: Could not serialize object: TypeError: can't pickle _thread.lock objects

我想这是因为PySpark无法序列化此自定义类.但是，如何避免在每次parse_ingredients_line函数运行时实例化此昂贵的对象的开销?

I imagine this is because PySpark can not serialize this custom class. But how can I avoid the overhead of instantiating this expensive object on every run of the parse_ingredients_line function?

推荐答案

我基于(

I solved it based on (https://github.com/scikit-learn/scikit-learn/issues/6975) by making all dependencies of the NLPFunctions class serializable.

这篇关于在Pyspark UDF中使用自定义Python对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！