ADF V2 - 批量自定义活动：在池上的每个节点上创建任务 [英] ADF V2 - Batch Custom Activity: create tasks on each node on pool
问题描述
我正在使用ADF 2和Batch自定义活动处理多个文件。
I am using ADF 2 and Batch custom activity for multiple files processing.
管道链如下所示： -
Pipeline chain looks like this: -
1.复制数据活动（加载文件串说50从FTP到Blob）
1.Copy Data activity (Load bunch of files say 50 from FTP to Blob)
2。自定义批处理活动（从＃1读取blob文件，将它们转换并加载到另一个blob位置）
2. Custom Batch activity (Read blob files from #1, transform them & load then to another blob location)
在自定义活动中，我希望为每个文件创建一个50个任务，分配给池中的5个节点。
Here in custom activity i would like to have 50 tasks created one for each file and assigned to 5 nodes in Pool.
如何将数据集从＃1传递到＃2，以便创建50个任务并在5个节点上并行运行文件处理。
How can I pass dataset from #1 to #2 so that it creates 50 tasks and runs file processing on 5 nodes in parallel.
推荐答案
嗨Sudheer，
Hi Sudheer,
如何将forEach活动与复制活动结合使用。我建议您实施此结构：
How about having a forEach activity in conjunction with the Copy Activity. I would recommend you to have this structure implemented :
- 复制活动。
- forEach活动。
- 转换活动。
要阅读有关forEach活动的更多信息，请参阅此文档：
To read more about forEach activity, pleas refer this doc :
https://docs.microsoft。 com / zh-cn / azure / data-factory / control-flow-for-activity-blob-dataset-definition
https://docs.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity#blob-dataset-definition
您可以将blob存储配置为forEach活动的数据集如文档所示。
You can configure blob storage as the dataset for the forEach activity as shown in the doc.
如果有帮助，请告诉我们。另外，我们可以继续进一步探讨。
Let us know if this helps. Else we can continue to probe in further.
这篇关于ADF V2 - 批量自定义活动：在池上的每个节点上创建任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
