气流因 TypeError“无法腌制模块对象"而下降当任务返回 kwargs [英] Airflow falls with TypeError "can't pickle module objects" when task returns kwargs

问题描述

当任务返回kwargs且provide_context= True时，为什么气流会因TypeError: can't pickle module objects而下降?

Why airflow falls with TypeError: can't pickle module objects when task returns kwargs with provide_context= True?

但是当我在同一任务中执行 print kwargs 时 - 那么一切都很好.

But when I do print kwargs in same task - then everything is ok.

推荐答案

这里的问题很可能出在 kwargs 对象中.出现错误是因为当您从气流中的操作员返回时，它会作为 xcom 设置的一部分保存在气流数据库中.

The problem here is most likely something in the kwargs object. The error arises because when you return from an operator in airflow it is saved in the airflow database as part of the xcom set up.

我建议您查看 kwargs 中的内容，并找出哪些项目是模块对象"而不是更琐碎的数据类型.解决问题本身的一个简单方法是只返回您需要返回的内容，而不是整个 kwargs 对象.

I would suggest you take a look at what's in your kwargs and figure out which items are "module objects" rather than more trivial data types. A simple way around the problem itself would be to only return the things you need to return, rather than the whole kwargs object.

这篇关于气流因 TypeError“无法腌制模块对象"而下降当任务返回 kwargs的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！