WCF流式System.OutOfMemoryException [英] WCF Streaming System.OutOfMemoryException

问题描述

在流式WCF应用程序中，我在服务中得到System.OutOfMemoryExceptions来处理大型消息(使用数据读取器查询> 7GB的MSSQL表，消息甚至可能超过7GB)，而小型消息则可以正常工作.我可以观察到在执行DataReaderToExcelXml的过程中内存使用量不断增长(请参见下文).奇怪的是，它通常会快速增长到2GB，在2GB + -1GB的状态下保持一会儿(1-5分钟)，然后又非常快地提高到〜6.5GB，这会导致异常(机器具有8GB内存).此时，在我看来，好像Stream不再通过而是被缓冲了.

In my streamed WCF application I get System.OutOfMemoryExceptions in my service for large messages (querying >7GB MSSQL tables using a datareader, messages might even exceed 7GB) while small messages work just fine. I can observe the memory usage grow constantly during the execution of DataReaderToExcelXml (see below). The weird thing is, that it usually grows fast to 2GB, stays at 2GB +-1GB for a while (1-5 minutes) and then again raises very fast to ~6.5GB which leads to the exception (machine has 8GB memory). At this point it looks to me, as if the Stream isn't passed through anymore but buffered.

我已经启用了跟踪日志记录，但是似乎异常停止了. DataReaderToExcelXml函数调用是跟踪日志中的最后一个可见事件.

I already enabled trace logging, but it seems to stop with the exception. The DataReaderToExcelXml function call is the last visible event in the trace log.

在WCF消息约定中，我确保消息仅包含流.在客户端，只需读取返回的Stream，将其写入文件流并进行处理即可.但是，收到异常后，我再也看不到正在执行的客户端代码或正在写入的文件.

In the WCF message contract I ensured that the message only contains the Stream. On client side, the returned Stream is simply read, written to a filestream and disposed. However, I can never observe the client code being executed or the file being written when I get the exception.

我已经尝试在客户端和服务器端将maxBufferPoolSize设置为零，如下所述 https://stackoverflow .com/a/595871/4166885 .没有成功.

I already tried setting the maxBufferPoolSize to zero, both on the client and server side, as described here https://stackoverflow.com/a/595871/4166885. No success.

WCF服务端的流编写器功能:

Stream writer function on WCF-service side:

 Public Shared Function DataReaderToExcelXml(ByRef dr As SqlDataReader) As Stream
    Dim ms As New MemoryStream
    Dim tw As New IO.StreamWriter(ms)

    For Each row As DbDataRecord In dr
            'Embed row in ExcelXml, detailed function omitted
            tw.write(row.toString()) 'row.toString is just a simplification
    End While

    tw.Flush()
    dr.Close()
    ms.Seek(0, SeekOrigin.Begin)

    Return ms
 End Function

Web.config绑定

Web.config bindings

    <bindings>
      <basicHttpBinding>
        <binding receiveTimeout="24.00:00:00" sendTimeout="24.00:00:00"
          maxBufferPoolSize="9223372036854775807" maxReceivedMessageSize="9223372036854775807"
          messageEncoding="Mtom" transferMode="Streamed" bypassProxyOnLocal="True">
          <readerQuotas maxStringContentLength="2147483647" maxArrayLength="2147483647"
            maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
          <security mode="Transport">
            <transport clientCredentialType="None" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>

App.config绑定

App.config bindings

    <binding name="BasicHttpBinding_IFileService" closeTimeout="00:01:00"
        openTimeout="00:01:00" receiveTimeout="24.00:00:00" sendTimeout="24.00:00:00"
        allowCookies="false" bypassProxyOnLocal="true" hostNameComparisonMode="StrongWildcard"
        maxBufferSize="2147483647" maxBufferPoolSize="2147483647"
        maxReceivedMessageSize="8589934592" messageEncoding="Mtom"
        textEncoding="utf-8" transferMode="Streamed">
        <readerQuotas maxDepth="32" maxStringContentLength="2147483647"
            maxArrayLength="2147483647" maxBytesPerRead="4096" maxNameTableCharCount="2147483647" />
        <security mode="Transport">
            <transport clientCredentialType="None" proxyCredentialType="None"
                realm="" />
            <message clientCredentialType="UserName" algorithmSuite="Default" />
        </security>
    </binding>

edit:在我的服务上使用.NET内存分析器，我发现，可以将增加的内存量追溯到byte[]类型.就个人而言，这并没有告诉我任何事情，但这也许是有用的信息.

edit: Using .NET memory profiler on my service I found out, that the increasing amount of memory can be traced back to the type byte[]. Personally, that's not telling anything to me, but maybe it's useful information.

edit2:在检查System.OutOfMemoryException时，我刚注意到它是在ms.Capacity = ms.length = 1073741824 = 1GB时发生的.因此，就在内存流即将将其容量增加一倍之前.我仍然不确定，为什么w3wp首先会消耗这么多的内存，但是现在很明显，内存流会触发Exception.

edit2: While inspecting the System.OutOfMemoryException I just noticed that it occurs when ms.Capacity = ms.length = 1073741824 = 1GB. So just before the memory stream would double its capacity. I'm still not sure, why w3wp consumes such large amounts of memory in the first place, but now it's clear that the memory stream triggers the Exception.

推荐答案

好吧，阅读您的问题，我认为您的错误源是关键.错误到底在哪里?在运输中还是在服务器中? 如果在DataReaderToExcel中引发了异常，也许您可​​以考虑其他想法(我认为这不是WCF问题): 到现在为止，我认为您在Windows窗体应用程序中会遇到同样的问题.

Well, reading your question, I think that the source of your error, is the key. Where is exactly the error? In transport, or in your server? If the Exception is raised in DataReaderToExcel, maybe you could consider another ideas (I think it isn't a WCF issue): By now, I think you would have the same problem in a Windows forms app.

您需要内存，因此只能用SqlDataReader填充它，必须根据范围按块查询.

You need memory, so you can fill it only with your SqlDataReader, you have to query by blocks, based on ranges.

我们将开始计算"TotalRecords":

We will start computing the "TotalRecords":

SELECT COUNT(*)
FROM YOUR_TABLE  
WHERE WATHEVER_YOU_WANT 

例如，查询 10.000 条记录，您的TotalRecords/10.000页(查询)为+1. (例如:30.001行，每块10.000行= 4个查询)

For example, querying blocks of 10.000 records, you have TotalRecords/10.000 pages (querys) +1. (Example: 30.001 rows, 10.000 row per block = 4 queries )

如此反复构建它们:

select top **10.000** * from 
(SELECT TOP (100) PERCENT  ROW_NUMBER() OVER (ORDER BY YOUR_TABLE_FIELD_1 DESC) ROW_PAGINATED, YOUR_TABLE_FIELD_1, YOUR_TABLE_FIELD_2 , ... , YOUR_TABLE_FIELD_N 
FROM YOUR_TABLE  
WHERE WATHEVER_YOU_WANT 
ORDER BY YOUR_TABLE_FIELD_1 DESC
) YOUR_ALIAS
WHERE YOUR_ALIAS.ROW_PAGINATED BETWEEN min_records_per_page AND max_records_per_page 
)

其中min_records_per_page和max_records_per_page具有以下值:

Where min_records_per_page AND max_records_per_page have this values:

Query 1:
  min_records_per_page= 1  
  max_records_per_page  = 10000
Query 2:
  min_records_per_page= 10001  
  max_records_per_page  = 20000
  ...
 Query N:
  min_records_per_page= (N-1)* +1
  max_records_per_page  = TotalRecords

在每次迭代中，您都将每个数据行映射到一个ExcelXml类中.

On each iteration, you will map each datarow into a ExcelXml class.

这样做，您将避免消耗所有内存，前提是您将要处置所使用的对象.例如，您可以每2次迭代使用Function SetProcessWorkingSetSize. 要在大量已处理的行之后释放内存，请执行以下操作:

Doing so, you will avoid to consume all the memory, assuming you will be Disposing the objects you use. You can use the Function SetProcessWorkingSetSize each 2 iterations, for example. To free memory after a significant number of processed rows:

Private Declare Auto Function SetProcessWorkingSetSize Lib "kernel32.dll" (ByVal procHandle As IntPtr, ByVal min As Int32, ByVal max As Int32) As Boolean

 Dim Mem As Process
 Mem = Process.GetCurrentProcess()
 SetProcessWorkingSetSize(Mem.Handle, -1, -1)

然后，现在您有了一个包含映射对象的数组.第1部分完成.

Then, now you have an array with mapped objects. Part 1 finished.

第2部分.您的客户端WCF可能会收到此数据.如果要发送的数据准备好了，请告诉我们是否有问题.然后，我们可以探索通信参数，然后说增加该参数，或添加该参数".

Part 2. Your client WCF may receive this data. Tell us then if you have problems once the data to be sent are ready. Then we can explore communication parameters, and then we would say "increase that parameter, or add this one".

希望有帮助

这篇关于WCF流式System.OutOfMemoryException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！