C# 控制台在多处理模式 (Parallel.ForEach) 下调用 REST API 10 万次以上 [英] C# Console call rest API 100k plus times in multiprocessing mode (Parallel.ForEach)

问题描述

我正在使用 Parallel.ForEach 以批处理模式调用 rest api 服务，例如一次 1000 个请求.我的 MaxDegreeOfParallelism 设置为 1000 ，但似乎系统一次只创建 10-15 个请求，尽管系统 CPU 利用率非常正常(15%)

I am using Parallel.ForEach to call a rest api service in batch mode like 1000 request at a time. My MaxDegreeOfParallelism is set to 1000 , But it seems system only creating 10-15 request at a time althoug system CPU utilization is very normal (15%)

var maxDegree = new ParallelOptions() { MaxDegreeOfParallelism = MaxDegreeOfParallelism };
        Parallel.ForEach(parsedLines, maxDegree, obj =>
        {


            processIndividualRecord(obj);

            var currentCount = Interlocked.Increment(ref paraaleCounter);
            if (currentCount % 100 == 0)
            {
                logger.Debug("Reaming records to process is:" + (parsedLines.Count - currentCount));
            }
        });

有什么办法可以一次发出 1000 个请求，以避免在非阻塞模式下等待时间过长.

Is there any way I can make 1000 request at a time to avoid large wait time in non blocking mode.

推荐答案

ThreadPool 线程不足.一个快速的解决方法是使用 ThreadPool.SetMinThreads 方法.

You are running short of ThreadPool threads. A quick workaround is to increase the initial pool of workers by using the ThreadPool.SetMinThreads method.

ThreadPool.SetMinThreads(1000, 10);

更好的方法是异步调用 REST API.这样，在请求进行中时对工作线程的需求将被消除.协调异步过程的一个极好的工具是 TPL 数据流 库.您可以在此处.

A better approach is to call the REST API asynchronously. This way the need for worker threads while the requests are in flight will be removed. An excellent tool for coordinating the asynchronous process is the TPL Dataflow library. You can see a usage example of this library here.

这篇关于C# 控制台在多处理模式 (Parallel.ForEach) 下调用 REST API 10 万次以上的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！