等待数千个任务 [英] await thousands of Tasks

问题描述

我有一个可以转换一些数据的应用程序，通常有1.000-30.000个文件.

I have an application which converts some data often there are 1.000 - 30.000 files.

我需要执行3个步骤:

1. 复制文件(在其中替换一些文本)
2. 与WebClient进行Web请求以下载文件(我将复制的文件发送到WebServer，后者将文件转换为另一种格式)
3. 获取下载的文件并更改一些内容

因此，所有三个步骤都包含一些I/O，并且我使用了async/await方法:

So all three steps include some I/O and I used async/await methods:

var tasks = files.Select(async (file) =>
{
    Item item = await createtempFile(file).ConfigureAwait(false);
    await convert(item).ConfigureAwait(false);
    await clean(item).ConfigureAwait(false);
}).ToList();

await Task.WhenAll(tasks).ConfigureAwait(false);

我不知道这是否是最佳实践，因为我创建了数千个任务.我考虑过将这三个步骤分开:

I don´t know if this is the best practice, because I create more than thousand tasks. I thought about splitting the three steps like:

List<Item> items = new List<Item>();
var tasks = files.Select(async (file) =>
{
    Item item = await createtempFile(file, ext).ConfigureAwait(false);
    lock(items)
        items.Add(item);
}).ToList();

await Task.WhenAll(tasks).ConfigureAwait(false);

var tasks = items.Select(async (item) =>
{
    await convert(item, baseAddress, ext).ConfigureAwait(false);
}).ToList();

await Task.WhenAll(tasks).ConfigureAwait(false);

var tasks = items.Select(async (item) =>
{
    await clean(targetFile, item.Doctype, ext).ConfigureAwait(false);
}).ToList();

await Task.WhenAll(tasks).ConfigureAwait(false);

但这似乎并没有更好或更快，因为我创建了3千个任务.

But that doesn´t seem to be better or faster, because I create 3 times thousands of tasks.

我应该限制任务的创建吗?像大块的100个任务? 还是我只是想得太多而已，创建数千个任务就可以了.

Should I throttle the creation of tasks? Like chunks of 100 tasks? Or am I just overthinking it and the creation of thousands of tasks is just fine.

CPU处于2-4％峰值的空闲状态，因此我考虑了太多等待或上下文切换.

The CPU is idling with 2-4% peak, so I thought about too many awaits or context switches.

也许WebRequest调用太多了，因为WebServer/WebService无法同时处理数千个请求，而我只应限制WebRequest?

Maybe the WebRequest calls are too many, because the WebServer/WebService can´t handle thousands of Requests simultaneously and I should only throttle the WebRequests?

我已经在app.config文件中增加了.NET maxconnection.

I already increased the .NET maxconnection in the app.config file.

推荐答案

正如评论者已正确指出的那样，您正在考虑过多. .NET运行时绝对没有问题，可以跟踪数千个任务.

As commenters have correctly noted, you're overthinking it. The .NET runtime has absolutely no problem tracking thousands of tasks.

但是，您可能要考虑使用TPL Dataflow管道，这将使您可以轻松地对管道中的不同操作(块")使用不同的并发级别.

However, you might want to consider using a TPL Dataflow pipeline, which would enable you to easily have different concurrency levels for different operations ("blocks") in your pipeline.

这篇关于等待数千个任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！