动态更改HttpClient中的代理，而无需使用大量CPU [英] Dynamically change proxy in HttpClient without hard cpu usage

问题描述

我需要创建一个发出请求(发布，获取等)的多线程应用程序 为此，我选择了Httpclient.

I need to create a multithreaded application which makes requests (Post, get etc) For this purpose i chose Httpclient.

默认情况下，它不支持Socks代理.所以我发现可以使用Sockshandler( https://github.com/extremecodetv/SocksSharp )而不是基本的HttpClientHandler.它可以让我用袜子.

By default it does not support Socks proxies. So I find Sockshandler (https://github.com/extremecodetv/SocksSharp) can be used instead of basic HttpClientHandler. It allows me to use socks.

但是我有一个问题.我所有的请求都应通过我从互联网解析的不同代理发送.但是httpclient处理程序不支持动态更改代理.如果我没有有效的代理，则需要重新创建一个httclient，这没关系，但是如果我有200个线程，则需要大量的cpu.那么在这种情况下我该怎么办?

But I have a problem. All my requests should be send through different proxies which I have parsed from the internet. But httpclient handler doesn't support changing proxies dynamically. If I don't have valid proxy, I need to recreate a httclient, this is ok, but if I have 200 threads, it takes a lot of cpu. So what should I do in this situation?

第二个问题.我找到了这篇文章( https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/)讨论将HttpClient作为单个实例使用以提高性能，但是在多线程程序中是不可能的.在这种情况下哪种方法更好?

And second problem. I found this article (https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/) which talks to use HttpClient as a single instance to better performance, but it's impossible in multithreaded program. Which way is better in this case?

寻求帮助

推荐答案

httpclient处理程序不支持动态更改代理.

httpclient handler doesn't support changing proxies dynamically.

我不确定这在技术上是否正确. 代理是可读写的属性，因此我相信您可以更改它(除非导致运行时错误...我实际上并没有尝试过说实话).

I'm not sure if that's technically true. Proxy is a read/write property so I believe you could change it (unless that results in a runtime error...I haven't actually tried it to be honest).

更新: 我已经尝试过，并且您的断言在技术上是正确的.在下面的示例中，更新UseProxy的行将失败，并显示"System.InvalidOperationException:'此实例已启动一个或多个请求.只能在发送第一个请求之前修改属性.' >已在.NET Core和完整框架上确认.

UPDATE: I have tried it now and your assertion is technically true. In the sample below, the line that updates UseProxy will fail with "System.InvalidOperationException: 'This instance has already started one or more requests. Properties can only be modified before sending the first request.'" Confirmed on .NET Core and full framework.

var hch = new HttpClientHandler { UseProxy = false };
var hc = new HttpClient(hch);
var resp = await hc.GetAsync(someUri);

hch.UseProxy = true; // fail!
hch.Proxy = new WebProxy(someProxy);
resp = await hc.GetAsync(someUri);

但是真实的是，您不能以线程安全的方式为每个请求设置不同的属性，这很不幸.

But what is true is that you can't set a different property per request in a thread-safe way, and that's unfortunate.

如果我有200个线程，则需要大量的CPU

if I have 200 threads, it takes a lot of cpu

并发异步HTTP调用不应占用额外的线程或CPU.使用await Task.WhenAll或类似的方法和没有线程将它们解雇消耗，直到返回响应.

Concurrent asynchronous HTTP calls should not consume extra threads nor CPU. Fire them off using await Task.WhenAll or similar and there is no thread consumed until a response is returned.

第二个问题.我找到了这篇文章...

And second problem. I found this article...

这绝对是您需要注意的事情.但是，即使您可以为每个请求设置一个不同的代理，基础网络堆栈仍将需要为每个代理打开一个套接字，因此您不会从每个实例的HttpClient实例中获得任何收益套接字耗尽问题方面的代理.

That's definitely something you need to look out for. However, even if you could set a different proxy per request, the underlying network stack would still need to open a socket for each proxy, so you wouldn't be gaining anything over an HttpClient instance per proxy in terms of the socket exhaustion problem.

最好的解决方案取决于您在这里谈论多少代理.在本文中，作者描述了当服务器命中4000-5000个开放套接字时遇到的问题，而在400个或更少的问题上没有问题. YMMV，但是如果代理的数量不超过几百个，则应该安全地为每个代理创建一个新的HttpClient实例.如果更多，我会查看限制您的并发性并对其进行测试，直到找到一个可以在服务器资源中找到的数字为止可以跟上.无论如何，请确保如果需要对同一代理进行多次调用，请为它们重新使用HttpClient实例. ConcurrentDictionary对于延迟创建和重用这些实例可能很有用.

The best solution depends on just how many proxies you're talking about here. In the article, the author describes running into problems when the server hit around 4000-5000 open sockets, and no problems around 400 or less. YMMV, but if the number of proxies is no more than a few hundred, you should be safe creating a new HttpClient instance per proxy. If it's more, I would look at throttling your concurrency and test it until find a number where your server resources can keep up. In any case, make sure that if you need to make multiple calls to the same proxy, you're re-using HttpClient instances for them. A ConcurrentDictionary could be useful for lazily creating and reusing those instances.

这篇关于动态更改HttpClient中的代理，而无需使用大量CPU的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！