如何限制并发异步 I/O 操作的数量? [英] How to limit the amount of concurrent async I/O operations?

问题描述

// let's say there is a list of 1000+ URLs
string[] urls = { "http://google.com", "http://yahoo.com", ... };

// now let's send HTTP requests to each of these URLs in parallel
urls.AsParallel().ForAll(async (url) => {
    var client = new HttpClient();
    var html = await client.GetStringAsync(url);
});

这是问题所在，它同时启动了 1000 多个 Web 请求.有没有一种简单的方法来限制这些异步 http 请求的并发数量?以便在任何给定时间下载的网页不超过 20 个.如何以最有效的方式做到这一点?

Here is the problem, it starts 1000+ simultaneous web requests. Is there an easy way to limit the concurrent amount of these async http requests? So that no more than 20 web pages are downloaded at any given time. How to do it in the most efficient manner?

推荐答案

您绝对可以使用 .NET 4.5 Beta 在最新版本的 async for .NET 中做到这一点.来自 'usr' 的前一篇文章指向了 Stephen Toub 写的一篇好文章，但鲜为人知的消息是异步信号量实际上已进入 .NET 4.5 的 Beta 版

You can definitely do this in the latest versions of async for .NET, using .NET 4.5 Beta. The previous post from 'usr' points to a good article written by Stephen Toub, but the less announced news is that the async semaphore actually made it into the Beta release of .NET 4.5

如果你看看我们心爱的 SemaphoreSlim 类(您应该使用它，因为它比原始的 Semaphore)，它现在拥有 WaitAsync(...) 系列重载，包括所有预期的参数 - 超时间隔、取消标记、您所有常用的调度朋友 :)

If you look at our beloved SemaphoreSlim class (which you should be using since it's more performant than the original Semaphore), it now boasts the WaitAsync(...) series of overloads, with all of the expected arguments - timeout intervals, cancellation tokens, all of your usual scheduling friends :)

Stephen 最近还写了一篇关于新的 .NET 4.5 好东西的博客文章，它是测试版的，参见 .NET 4.5 Beta 中并行的新功能.

Stephen's also written a more recent blog post about the new .NET 4.5 goodies that came out with beta see What’s New for Parallelism in .NET 4.5 Beta.

最后，这里有一些关于如何使用 SemaphoreSlim 进行异步方法节流的示例代码:

Last, here's some sample code about how to use SemaphoreSlim for async method throttling:

public async Task MyOuterMethod()
{
    // let's say there is a list of 1000+ URLs
    var urls = { "http://google.com", "http://yahoo.com", ... };

    // now let's send HTTP requests to each of these URLs in parallel
    var allTasks = new List<Task>();
    var throttler = new SemaphoreSlim(initialCount: 20);
    foreach (var url in urls)
    {
        // do an async wait until we can schedule again
        await throttler.WaitAsync();

        // using Task.Run(...) to run the lambda in its own parallel
        // flow on the threadpool
        allTasks.Add(
            Task.Run(async () =>
            {
                try
                {
                    var client = new HttpClient();
                    var html = await client.GetStringAsync(url);
                }
                finally
                {
                    throttler.Release();
                }
            }));
    }

    // won't get here until all urls have been put into tasks
    await Task.WhenAll(allTasks);

    // won't get here until all tasks have completed in some way
    // (either success or exception)
}

最后，但可能值得一提的是使用基于 TPL 的调度的解决方案.您可以在尚未启动的 TPL 上创建委托绑定任务，并允许自定义任务调度程序来限制并发.事实上，这里有一个 MSDN 示例:

Last, but probably a worthy mention is a solution that uses TPL-based scheduling. You can create delegate-bound tasks on the TPL that have not yet been started, and allow for a custom task scheduler to limit the concurrency. In fact, there's an MSDN sample for it here:

另请参阅TaskScheduler .

这篇关于如何限制并发异步 I/O 操作的数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！