控制台应用程序中的最大并行任务，以及如何处理“将内容复制到流时出错"调用API时 [英] Max parallel tasks in Console App, and how to handle "Error while copying content to stream" when calling API

问题描述

我正在处理多个大型数据集(大约 300 万条记录，每天增长)的集成，这些数据集是从供应商 OData API 中分块提取的.当我在本地运行/调试时，集成将运行完成而不会出错.但是，一旦我将它放在我们的 PROD 服务器上，它就会有时在给定的数据块上以将内容复制到流时出错"而失败.我无法找出原因，需要一些帮助.

I'm working on an integration that handles several large datasets (around 3M records or so, growing daily) which are pulling from a vendor OData API in chunks. When I run it/debug locally, the integration will run to completion without errors. However, as soon as I put it out on our PROD server, it will sometimes fail with "Error while copying content to stream" on a given chunk of data. I'm having trouble figuring out the reason, and could use some help.

每个请求都包含在 using 语句中，并且应该干净地处理自己.通常，确实如此.同样，这一切都在我的本地机器上完美运行，但是服务器在提取数百万条记录后遇到了一些麻烦.我几乎在每一步都添加了额外的日志记录以捕获特定错误，但仍然不清楚如何处理(或避免)此错误.完全不一致……完全是断断续续和不一致的.

Each request is wrapped in a using statement and should be disposing of itself cleanly. Usually, it does. Again, this all works perfectly on my local machine, but the server has a bit of trouble after it's into pulling millions of records. I have added extra logging at virtually every step to catch the specific error, but it's still not clear how to handle (or avoid) this error. It isn't consistent at all... totally intermittent and inconsistent.

错误抛出的代码:

using (WebRequestHandler webRequestHandler = new WebRequestHandler())
{
    using (HttpClient httpClient = new HttpClient(webRequestHandler))
    {
        httpClient.Timeout = TimeSpan.FromMinutes(15);

        ConfigureJsonClient(httpClient, syncConfig.ApiEndpoint, syncConfig.ApiAuthKey);

        apiCommand = "http://foo/with/filtering";

        responseMessage = Task.Run(async () => await httpClient.GetAsync(apiCommand)
            .ConfigureAwait(true)).Result;
    }
}

错误:

API Error Occurred - RETRYING... Foo: 2 | API CALL: https://foo... | EXCEPTION: One or more errors occurred. | INNER EXCEPTION: Error while copying content to a stream. | STACKTRACE:    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
       at FooNamespace.Processor.GetDataFromApi(SyncConfig syncConfig, String collection, List`1 nonDatedCollectionList, Int32 skipIndex, Int32 retryCount, Int32 taskId)

我还尝试运行尽可能多的并行任务，以使用跳过/获取将这些大型数据集提取到预定义的块(数据页)中.无论我做什么，即使在本地，我可以并行启动的最大任务数是 40.这个限制有什么原因吗?

I am also trying to run as many parallel tasks as I can to pull these large datasets in pre-defined chunks (data pages) using skip/take. No matter what I do, even locally, the max number of Tasks I can spin up in parallel is 40. Is there a reason for this limitation?

在访问 API 时，并行任务是否可能以某种方式相互冲突?这些应该显示为完全独立的、孤立的调用...

Is it possible the parallel tasks are conflicting with each other somehow when hitting the API? These should appear as completely separate, isolated calls...

非常感谢任何和所有帮助.

Any and all help is much appreciated.

推荐答案

我遇到了类似的异常将内容复制到流时出错.在内部异常中，我有 System.IO.IOException: Thedecryption operation failed.我发现服务器正在发送没有标题 content-encoding: gzip 的压缩内容.更改HttpClientHandler上AutomaticDecompression属性的默认值即可解决问题.

I've got a similar exception Error while copying content to a stream. On the inner exception I've got System.IO.IOException: The decryption operation failed. I found that the server is sending the content compressed without the header content-encoding: gzip. Change de default value of the AutomaticDecompression property on HttpClientHandler solve the problem.

HttpClientHandler httpClientHandler = new HttpClientHandler();
httpClientHandler.AutomaticDecompression = DecompressionMethods.All;
using (var client = new HttpClient(httpClientHandler))

这篇关于控制台应用程序中的最大并行任务，以及如何处理“将内容复制到流时出错"调用API时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！