在 dotnet core 中批量使用 Parallel.For [英] Use Parallel.For in batches in dotnet core

问题描述

我在 dotnet core 中使用 httptrigger 函数，在那里我以 Json 格式获取 httprequest 数据.我需要在 Google Merchant Center 帐户中插入这个值.几乎有 9000 行(每次都是动态数据)需要插入.我如何实现 Parallel.for 逻辑，该逻辑将执行得更快.目前我正在使用如下所示的每个循环，但需要更多时间.下面是代码.

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();动态体 = JsonConvert.DeserializeObject(requestBody);for (int i =0;i

解决方案

我创建了一个小例子，也许你可以找到最适合你情况的最佳方法.

I am using httptrigger function in dotnet core where i am getting httprequest data in Json format.I need to insert this value in Google Merchant center account. There are almost 9000 rows (dynamic data each time) that needs to be inserted. How i can implement the Parallel.for logic which will execute faster. Currently i am using for each loop like below but it is taking more time. Below is the code.

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic body = JsonConvert.DeserializeObject(requestBody);
for (int i =0;i<body.Count;i++)
{
  Product newProduct = InsertProduct(merchantId, websiteUrl,body[i]);
}

解决方案

I created a small example maybe there you can find the best way which fits your case best.

dotnet fiddle Example

There are 3 options:

In sequence

As the title says every item is processed in sequence. Very save method but not the fastest one to process 9000 items :)

var list = GenerateItems();
var count = list.Count();
for(var i = 0; i < count; i++) 
{
    InsertInDatabaseAsync($"{i}", list.ElementAt(i)).GetAwaiter().GetResult();
}

With Parallel.For Library

Like said from the comments its good for CPU bound processing but has some lacks on async methods (here)

var list = GenerateItems();
var count = list.Count();
var options = new ParallelOptions{MaxDegreeOfParallelism = MAX_DEGREE_OF_PARALLELISM};
Parallel.For(0, count, options, (i) => 
{
    InsertInDatabaseAsync($"{i}", list.ElementAt(i)).GetAwaiter().GetResult();
});

With Async-Await

I think in your example this fits best for you. Every item is processed in parallel, starts the processing directly and spinns up a Task. (Copied the async-extension from here)

var list = GenerateItems();
var count = list.Count();

// Extensions method see in referenced SO answer
ForEachAsync(count, list, async (item, index) => 
{
    await InsertInDatabaseAsync($"{index}", item);
}).GetAwaiter().GetResult();

...Updated

Thanks for the comments. I have updated the async-await implementation to a more simpler one:

private static async Task ForEachAsync<T>(IEnumerable<T> enumerable, Func<T, int, Task> asyncFunc)
{
    var itemsCount = enumerable.Count();
    var tasks = new Task[itemsCount];
    int i = 0;
    foreach (var t in enumerable)
    {
        tasks[i] = asyncFunc(t, i);
        i++;
    }
    await Task.WhenAll(tasks);
}

And also added the MAX_DEGREE_OF_PARALLELISM set to 1. This has a huge impact on the parallel processing like described in the commends.

这篇关于在 dotnet core 中批量使用 Parallel.For的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！