包含异步和同步的并行循环 [英] Parallel loop containing both async and synchronous

问题描述

我有一个需要并行运行的循环，因为每次迭代都很慢且占用大量处理器，但是我还需要在循环中的每次迭代中调用一个异步方法.

I've got a loop that needs to be run in parallel as each iteration is slow and processor intensive but I also need to call an async method as part of each iteration in the loop.

我已经看到了有关如何在循环中处理异步方法的问题，但是我并没有看到异步和同步的组合.

I've seen questions on how to handle an async method in the loop but not a combination of async and synchronous, which is what I've got.

我的(简化)代码如下-我知道由于将异步操作传递给foreach，因此无法正常工作.

My (simplified) code is as follows - I know this won't work properly due to the async action being passed to foreach.

protected IDictionary<int, ReportData> GetReportData()
{
    var results = new ConcurrentDictionary<int, ReportData>();
      
    Parallel.ForEach(requestData, async data =>
    {
        // process data synchronously
        var processedData = ProcessData(data);

        // get some data async
        var reportRequest = await BuildRequestAsync(processedData);

        // synchronous building
        var report = reportRequest.BuildReport();

        results.TryAdd(data.ReportId, report);
     });

     // This needs to be populated before returning
     return results;
}

在必须异步执行操作以等待单个异步调用时，是否有任何方法可以并行执行该操作.

Is there any way to get execute the action in parallel when the action has to be async in order to await the single async call.

将同步功能转换为异步操作不是可行的选择.

It's not a practical option to convert the synchronous functions to async.

我不想拆分动作并使用Parallel.ForEach，然后再使用whenAll和另一个Parallel.ForEach进行异步调用，因为每个阶段的速度在不同的迭代之间可能会有很大的不同，所以拆分它会导致效率低下因为更快的会在继续之前等待更慢的.

I don't want to split the action up and have a Parallel.ForEach followed by the async calls with a WhenAll and another Parallel.ForEach as the speed of each stage can vary greatly between different iterations so splitting it would be inefficient as the faster ones would be waiting for the slower ones before continuing.

我确实想知道是否可以使用PLINQ ForAll代替Parallel.ForEach，但从未使用过PLINQ，并且不确定是否在返回之前是否会等待所有迭代完成，即Tasks是否仍在运行该过程的结束.

I did wonder if a PLINQ ForAll could be used instead of the Parallel.ForEach but have never used PLINQ and not sure if it would wait for all of the iterations to be completed before returning, i.e. would the Tasks still be running at the end of the process.

推荐答案

在必须异步执行操作以等待单个异步调用时，是否有任何方法可以并行执行该操作.

Is there any way to get execute the action in parallel when the action has to be async in order to await the single async call.

是的，但是您需要了解采用其他方法时 Parallel 给您带来的损失.具体来说， Parallel 将自动确定适当的线程数并根据使用情况进行调整.

Yes, but you'll need to understand what Parallel gives you that you lose when you take alternative approaches. Specifically, Parallel will automatically determine the appropriate number of threads and adjust based on usage.

将同步功能转换为异步操作不是可行的选择.

It's not a practical option to convert the synchronous functions to async.

对于受CPU约束的方法，您不应对其进行转换.

For CPU-bound methods, you shouldn't convert them.

我不想拆分动作并使用Parallel.ForEach，然后再使用whenAll和另一个Parallel.ForEach进行异步调用，因为每个阶段的速度在不同的迭代之间可能会有很大的不同，所以拆分它会导致效率低下因为更快的会在继续之前等待更慢的.

I don't want to split the action up and have a Parallel.ForEach followed by the async calls with a WhenAll and another Parallel.ForEach as the speed of each stage can vary greatly between different iterations so splitting it would be inefficient as the faster ones would be waiting for the slower ones before continuing.

我要提出的第一个建议是研究

The first recommendation I would make is to look into TPL Dataflow. It allows you to define a "pipeline" of sorts that keeps the data flowing through while limiting the concurrency at each stage.

我确实想知道是否可以使用PLINQ ForAll代替Parallel.ForEach

I did wonder if a PLINQ ForAll could be used instead of the Parallel.ForEach

不.PLINQ在工作方式上与 Parallel 非常相似.它们在CPU利用率方面的积极性方面存在一些差异，并且在API方面存在一些差异-例如，如果您最终获得了一系列结果，则PLINQ通常比 Parallel 干净-但在很大程度上级别的视图，它们非常相似.两者都只能在同步代码上工作.

No. PLINQ is very similar to Parallel in how they work. There's a few differences over how aggressive they are at CPU utilization, and some API differences - e.g., if you have a collection of results coming out the end, PLINQ is usually cleaner than Parallel - but at a high-level view they're very similar. Both only work on synchronous code.

不过，您可以将简单的 Task.Run 与 Task.WhenAll 一起使用:

However, you could use a simple Task.Run with Task.WhenAll as such:

protected async Task<IDictionary<int, ReportData>> GetReportDataAsync()
{
  var tasks = requestData.Select(async data => Task.Run(() =>
  {
    // process data synchronously
    var processedData = ProcessData(data);

    // get some data async
    var reportRequest = await BuildRequestAsync(processedData);

    // synchronous building
    var report = reportRequest.BuildReport();

    return (Key: data.ReportId, Value: report);
  })).ToList();
  var results = await Task.WhenAll(tasks);
  return results.ToDictionary(x => x.Key, x => x.Value);
}

您可能需要应用并发限制( Parallel 会为您完成).在异步世界中，这看起来像:

You may need to apply a concurrency limit (which Parallel would have done for you). In the asynchronous world, this would look like:

protected async Task<IDictionary<int, ReportData>> GetReportDataAsync()
{
  var throttle = new SemaphoreSlim(10);
  var tasks = requestData.Select(data => Task.Run(async () =>
  {
    await throttle.WaitAsync();
    try
    {
      // process data synchronously
      var processedData = ProcessData(data);

      // get some data async
      var reportRequest = await BuildRequestAsync(processedData);

      // synchronous building
      var report = reportRequest.BuildReport();

      return (Key: data.ReportId, Value: report);
    }
    finally
    {
      throttle.Release();
    }
  })).ToList();
  var results = await Task.WhenAll(tasks);
  return results.ToDictionary(x => x.Key, x => x.Value);
}

这篇关于包含异步和同步的并行循环的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！