是否可以遍历一个prefect.Parameter? [英] Is it possible to loop over a prefect.Parameter?

问题描述

我有以下流:

 ，其中Flow(我的流程")为流程:urls =参数("urls"，默认= ['url1'，'url2'])对于网址中的网址:结果= get_url(urls) 

当我通过 flow.run()运行此代码时，出现以下错误:

  TypeError:'Parameter'对象不可迭代 

是否可以对其进行迭代，如果可以，怎么做?

解决方案

对于刚接触延迟计算范例的人们来说，这是一个非常常见的绊脚石.

Prefect Parameter (通常是 any Prefect任务)代表将来将运行 的工作单元.构建流时，您要指定工作单元之间的依赖关系，这些依赖关系将在以后运行.这是一个功能强大的抽象，因为它使您可以在运行工作流之前先对其工作流的某些属性进行推理.

需要注意的是，您需要区分在构建时间(在运行Flow之前)可用的内容和在运行时间(在运行期间)可用的内容.流运行的上下文).在您的情况下，您正在使用一些知识，即Prefect无法知道哪个参数将返回可迭代的值.此外，即使Prefect可以推断出参数输出 是可迭代的，它也无法知道将返回多少个元素(每次运行时可能会有所不同！).鉴于所有这些，Prefect会处理这些延迟迭代".的概念称为映射": https://docs.prefect.io/core/concepts/mapping.html

根据您的情况，您可以将代码重构为:

 ，其中Flow(我的流程")为流程:urls =参数("urls"，默认= ['url1'，'url2'])结果= get_url.map(urls) 

，它将为参数的返回值中的每个值动态生成 get_url 的副本.

I have the following flow:

with Flow("My flow") as flow:
    urls = Parameter("urls", default=['url1', 'url2'])
    for url in urls:
        result = get_url(urls)

When I run this by flow.run(), I get the following error:

TypeError: 'Parameter' object is not iterable

Is it possible to iterate over it and if yes, how?

解决方案

This is a very common stumbling block for people new to deferred computing paradigms.

Prefect Parameters (and more generally, any Prefect task) represent units of work that will run in the future. When you build a flow, you are specifying dependencies between units of work that will run later. This is a powerful abstraction because it lets you reason about certain properties of your workflow before you ever run it.

The caveat is that you need to distinguish between what is available at build time (prior to ever running the Flow) and what is available at run time (during the context of a flow run). In your case, you are using a piece of knowledge that Prefect has no way of knowing about which is that your Parameter is going to return something iterable. Moreover, even if Prefect could infer that the parameter output will be iterable, it has no way of knowing how many elements will be returned (which could be different with every run of the flow!).

Given all of that, Prefect handles these "deferred iterations" with a concept called "mapping": https://docs.prefect.io/core/concepts/mapping.html

In your case, you could refactor your code to:

with Flow("My flow") as flow:
    urls = Parameter("urls", default=['url1', 'url2'])
    result = get_url.map(urls)

which will dynamically spawn copies of get_url for each value in the parameter's return value.

这篇关于是否可以遍历一个prefect.Parameter?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！