在Pyspark UDF中使用自定义Python对象 [英] Usage of custom Python object in Pyspark UDF
本文介绍了在Pyspark UDF中使用自定义Python对象的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
在运行以下PySpark代码时:
When running following piece of PySpark code:
nlp = NLPFunctions()
def parse_ingredients(ingredient_lines):
parsed_ingredients = nlp.getingredients_bulk(ingredient_lines)[0]
return list(chain.from_iterable(parsed_ingredients))
udf_parse_ingredients = UserDefinedFunction(parse_ingredients, ArrayType(StringType()))
我收到以下错误:
_pickle.PicklingError: Could not serialize object: TypeError: can't pickle _thread.lock objects
我想这是因为PySpark无法序列化此自定义类.但是,如何避免在每次parse_ingredients_line
函数运行时实例化此昂贵的对象的开销?
I imagine this is because PySpark can not serialize this custom class. But how can I avoid the overhead of instantiating this expensive object on every run of the parse_ingredients_line
function?
推荐答案
I solved it based on (https://github.com/scikit-learn/scikit-learn/issues/6975) by making all dependencies of the NLPFunctions class serializable.
这篇关于在Pyspark UDF中使用自定义Python对象的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文