C#同时运行许多异步任务 [英] C# Running many async tasks the same time

问题描述

我是异步任务的新手.

我有一个功能，可以获取学生ID，并使用所需的ID从特定大学的网站上抓取数据.

I've a function that takes student ID and scrapes data from specific university website with the required ID.

    private static HttpClient client = new HttpClient();
    public static async Task<Student> ParseAsync(string departmentLink, int id, CancellationToken ct)
    {
        string website = string.Format(departmentLink, id);
        try
        {
            string data;
            var stream = await client.GetAsync(website, ct);
            using (var reader = new StreamReader(await stream.Content.ReadAsStreamAsync(), Encoding.GetEncoding("windows-1256")))
                data = reader.ReadToEnd();

            //Parse data here and return Student.
        } catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }

它可以正常工作.有时候，尽管我需要为很多学生运行此功能，所以我使用以下内容

And it works correctly. Sometimes though I need to run this function for a lot of students so I use the following

        for(int i = ids.first; i <= ids.last; i++)
        {
            tasks[i - ids.first] = ParseStudentData.ParseAsync(entity.Link, i, cts.Token).ContinueWith(t =>
            {
                Dispatcher.Invoke(() =>
                {
                    listview_students.Items.Add(t.Result);
                    //Students.Add(t.Result);
                    //lbl_count.Content = $"{listview_students.Items.Count}/{testerino.Length}";
                });
            });
        }

我将任务存储在数组中，以待稍后等待.

I'm storing tasks in an array to wait for them later.

只要学生人数介于(0，〜600?)之间，这也是可以的，这有点随机. 然后，对于尚未解析的其他每个学生，抛出任务已取消.

This also works finely as long as the students count is between (0, ~600?) it's kinda random. And then for every other student that still hasn't been parsed throws A task was cancelled.

请记住，我从不使用取消令牌.

Keep in mind that, I never use the cancellation token at all.

我需要在这么多的学生上运行此功能，它总共可以完成约9000个异步任务.那是怎么回事?

I need to run this function on so many students it can reach ~9000 async task altogether. So what's happening?

推荐答案

在如此短的时间内排队9000个请求时，您基本上是在网站上发起拒绝服务攻击.这不仅会导致您出错，而且可能会导致网站瘫痪.最好将并发请求的数量限制为一个更合理的值(例如30).尽管可能有几种方法可以做到这一点，但我想到的是以下一种方法:

You are basically creating a denial of service attack on the website when you are queuing up 9000 requests in such a short time frame. Not only is this causing you errors, but it could take down the website. It would be best to limit the number of concurrent requests to a more reasonable value (say 30). While there are probably several ways to do this, one that comes to mind is the following:

private async Task Test()
{
  var tasks = new List<Task>();
  for (int i = ids.first; i <= ids.last; i++)
  {
    tasks.Add(/* Do stuff */);
    await WaitList(tasks, 30);
  }
}

private async Task WaitList(IList<Task> tasks, int maxSize)
{
  while (tasks.Count > maxSize)
  {
    var completed = await Task.WhenAny(tasks).ConfigureAwait(false);
    tasks.Remove(completed);
  }
}

其他方法可能使用.Net类(例如

Other approaches might leverage the producer/consumer pattern using .Net classes such as a BlockingCollection

这篇关于C#同时运行许多异步任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！