ForEach BatchCount枚举器或项属性 [英] ForEach BatchCount Enumerator or Item Property

问题描述

我正在从API中提取大量数据。 结果是分页的，但当值为< url> ;; rel = next格式时，ADF不能（据我所知）读取Headers.Link属性。

I'm extracting a large amount of data from an API.  Results are paginated, but ADF cannot (as far as I can tell) read the Headers.Link property when the value is in a <url>;rel=next format.

我是使用ForEach循环作为解决方法。 我在1小时内提取数据，以确保我保持在每页限制200条记录之下。 这意味着类似于28000个单独的查询，所以我试图批处理它们。

I am using a ForEach loop as a workaround.  I'm pulling data in 1 hour chunks to ensure I stay under the 200 records per page limit.  This means something like 28000 separate queries, so I am trying to batch them.

但是我遇到了API速率限制和缺少大量时间的问题。  

But I'm running into issues with the APIs rate limit and missing chunks of time.  

如果能够识别当前批次的哪个项目正在运行，我可以添加等待活动，时间段设置为100ms * Batch.Item（）。 然后我可以显着增加批量大小并更快地处理工作。

If there is the ability to identify which Item of the current batch is running, I can add a Wait Activity with the time period set to 100ms * Batch.Item().  I could then increase my batch size significantly and process the job faster.

这样的属性是否存在？

_________________ Stephen Lloyd DW Architect

_________________ Stephen Lloyd DW Architect

推荐答案

Stephen Stephen Lloyd先生，谢谢你的提问。 我无法找到这样的房产，但我确实有一个解决方法。 您可以创建一个数组变量，并在ForEach中包含AppendVariable活动。 然后你可以利用
数组的长度模数（％）你的批量大小来找出已经完成了多少次批量迭代。

Hello Stephen Lloyd and thank you for your question.  I have not been able to find such a property, however I do have a workaround.  You can create an array variable, and have an AppendVariable activity inside your ForEach.  Then you can leverage the Length of the array modulo (%) your batch size to find out how many batch iterations have completed, roughly.

我建议的长度而不是一个简单的计数器变量，是由于并行处理的竞争条件，禁止变量自引用（myVar = myVar + 1;被禁止，但是myVarX = myVarY + 1; myVarY = myVarX;
不是禁止。）

The reason I suggested Length instead of a simple counter variable, is that due to the race conditions of parallel processing, a variable self-reference is prohibited ( myVar = myVar + 1; is prohibited, but myVarX = myVarY + 1 ; myVarY = myVarX; is not prohibited.)

这篇关于ForEach BatchCount枚举器或项属性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！