WCF流大数据(500MB / 1GB)在自我托管服务 [英] WCF Streaming large data (500MB / 1GB) on a self hosted service
问题描述
我目前遇到的问题试图发送使用WCF自托管的服务(不IIS)大数据。 使用流式结果与用的System.OutOfMemoryException我的服务崩溃传输500MB。 是否有可能在所有传输的数据量等?
下面是我的WCF配置:
< system.serviceModel>
<服务>
<服务名称=CIServicebehaviorConfiguration =CIBehavior>
<主机>
< baseAddresses>
<新增baseAddress =的net.tcp://本地主机:6547 / CIService / CIService.svc/>
< / baseAddresses>
< /主机>
<端点绑定=NetTcpBinding的
bindingConfiguration =netTcpBindingConfig
behaviorConfiguration =CIBehavior.EndpointBehavior
合同=CIService.ICreatable/>
<端点地址=MEX
绑定=mexHttpBinding
NAME =mexTcpBinding
合同=IMetadataExchange接口/>
< /服务>
< /服务>
< serviceHostingEnvironment multippleSiteBindingEnabled =真/>
<绑定>
< NetTcpBinding的>
<绑定名称=netTcpBindingConfigcloseTimeout =零点01分00秒openTimeout =零点01分00秒
receiveTimeout =1点00分○○秒的SendTimeout =○点10分00秒
hostNameComparisonMode =StrongWildcard将ListenBackLog =10MAXCONNECTIONS =10
maxBufferSize =2147483647maxBufferPoolSize =2147483647maxReceivedMessageSize =2147483647
transferMode =流媒体>
< readerQuotas MAXDEPTH =2147483647maxStringContentLength =2147483647maxArrayLength =2147483647
maxBytesPerRead =2147483647maxNameTableCharCount =2147483647/>
< /装订>
< / NetTcpBinding的>
< /绑定>
<行为>
< serviceBehaviors>
<行为NAME =CIBehavior>
< serviceMetadata httpGetEnabled =FALSE/>
< serviceDebug includeExceptionDetailInFaults =真/>
< serviceThrottling maxConcurrentCalls =200maxConcurrentInstances =2147483647maxConcurrentSessions =100/>
< DataContractSerializer的maxItemsInObjectGraph =2147483647/>
< /行为>
< / serviceBehaviors>
< endpointBehavior>
<行为NAME =CIBehavior.EndpointBehavior>
< DataContractSerializer的maxItemsInObjectGraph =2147483647/>
< /行为>
< / endpointBehavior>
< /行为>
< /system.serviceModel>
我的客户端配置:
< system.serviceModel>
<绑定>
< NetTcpBinding的>
<绑定名称=NetTcpBinding_ICreatable
closeTimeout =00:01:00openTimeout =00:01:00
receiveTimeout =1点00分○○秒的SendTimeout =○点10分00秒
transactionFlow =假
transferMode =流媒体
transactionProtocol =OleTransactions
hostNameComparisonMode =StrongWildcard
将ListenBackLog =10
maxBufferPoolSize =2147483647
maxBufferSize =2147483647
MAXCONNECTIONS =10
maxReceivedMessageSize =2147483647>
< readerQuotas
MAXDEPTH =2147483647
maxStringContentLength =2147483647
maxArrayLength =2147483647
maxBytesPerRead =2147483647
maxNameTableCharCount =2147483647/>
<有序的ReliableSession =真正的inactivityTimeout =〇时10分00秒anabled =FALSE/>
< /装订>
< / NetTcpBinding的>
< /绑定>
<客户端>
<端点名称=NetTcpBinding_ICreatable
地址=的net.tcp://本地主机:6547 / CIService / CIService.svc
绑定=NetTcpBinding的
bindingConfiguration =NetTcpBinding_ICreatable
behaviorConfiguration =CIBehavior.EndpointBehavior
合同=ICreatable/>
< /客户>
<行为>
< endpointBehavior>
<行为NAME =CIBehavior.EndpointBehavior>
< DataContractSerializer的maxItemsInObjectGraph =2147483647/>
< /行为>
< / endpointBehavior>
< /行为>
< /system.serviceModel>
您不需要的 maxBufferSize 或 maxBufferPoolSize 设置如此之高,这些都可能导致你的内存溢出异常。默认值应该就可以了。
查看大数据和MSDN上的流,特别是部分特别安全注意事项大型数据这一块的文字是很重要的。
该MaxBufferSize属性要求来约束内存 WCF缓冲区。将它设置为一个安全值(或保持它是非常重要的 为默认值),当流。例如,假设您的 服务必须接收文件多达4 GB的大小,并将其存储在 本地磁盘。还假设你的内存受到限制以这样的方式 你只能缓冲64KB的数据在一个时间。然后您将设置 该MaxReceivedMessageSize至4 GB和MaxBufferSize为64 KB。也, 在你的服务实现,你必须确保你只读 从64 KB块的输入流和不读下一 之前的previous一个块已被写入到磁盘,并丢弃 从内存中。
我放在一起流数据的一个非常简单的例子,从自托管服务控制台客户端。为了保持短后我只加了客户端的服务器code和组成部分。
服务合同
使用System.IO;
使用System.ServiceModel;
命名空间服务
{
[的ServiceContract]
公共接口的IStream
{
[OperationContract的]
流GetLargeObject();
}
}
服务实现
使用系统;
使用System.IO;
使用System.ServiceModel;
命名空间服务
{
[ServiceBehavior]
公共类StreamService:IStream的
{
公共流GetLargeObject()
{
//添加路径,一个大的文件,这一个是2.5 GB
字符串的文件路径= Path.Combine(Environment.CurrentDirectory,C:\\ \\温度BFBC2_PC_Client_R11_795745_Patch.exe);
尝试
{
镜像文件的FileStream = File.OpenRead(文件路径);
返回镜像文件;
}
赶上(IOException异常前)
{
Console.WriteLine(的String.Format(试图打开文件出现异常{0},文件路径));
Console.WriteLine(例外情况是:);
Console.WriteLine(ex.ToString());
扔;
}
}
}
}
该服务主要
使用系统;
使用System.ServiceModel;
命名空间服务
{
类节目
{
静态无效的主要(字串[] args)
{
尝试
{
使用(VAR的ServiceHost =新的ServiceHost(typeof运算(StreamService)))
{
serviceHost.Open();
Console.WriteLine(preSS任意键结束);
Console.ReadKey();
}
}
赶上(例外前)
{
Console.WriteLine(ex.ToString());
}
}
}
}
该服务的app.config
< XML版本=1.0&GT?;
<结构>
<启动>
< supportedRuntime版本=4.0版的SKU =NETFramework,版本= V4.0/>
< /启动>
< system.serviceModel>
<行为>
< serviceBehaviors>
<行为NAME =StreamServiceBehavior>
< serviceMetadata httpGetEnabled =真/>
< /行为>
< / serviceBehaviors>
< /行为>
<绑定>
< NetTcpBinding的>
<绑定名称=NewBinding0transferMode =流媒体/>
< / NetTcpBinding的>
< /绑定>
<服务>
<服务behaviorConfiguration =StreamServiceBehaviorNAME =Service.StreamService>
<端点地址=的net.tcp://本地主机:9000 / streamserver绑定=NetTcpBinding的
bindingConfiguration =NewBinding0bindingName =合同=Service.IStream/>
<端点地址=MEX绑定=mexHttpBinding
合同=IMetadataExchange接口/>
<主机>
< baseAddresses>
<新增baseAddress =HTTP://本地主机:8080 / StreamService/>
< / baseAddresses>
< /主机>
< /服务>
< /服务>
< /system.serviceModel>
< /结构>
启动服务,可能需要在管理员帐户下运行,打开插座。创建一个客户端控制台应用程序,使用URL http添加服务引用://本地主机:8080 / StreamService,使用服务的命名空间生成的客户端
客户端主
使用系统;
使用System.IO;
使用Client.Service;
命名空间客户端
{
类节目
{
静态无效的主要(字串[] args)
{
尝试
{
使用(StreamClient streamClient =新StreamClient())
{
streamClient.Open();
使用(的FileStream FILESTREAM =新的FileStream(C:\\ \\温度bigfile.exe,FileMode.Create))
{
。streamClient.GetLargeObject()CopyTo从(FILESTREAM);
}
}
Console.WriteLine(preSS任意键结束);
Console.ReadKey();
}
赶上(例外前)
{
Console.WriteLine(前);
}
}
}
}
生成的客户端配置文件将需要稍加修改,增加的 receiveTimeout ,然后将 maxReceivedMessageSize =4294967295
< system.serviceModel>
<绑定>
< NetTcpBinding的>
<绑定名称=NetTcpBinding_IStreamcloseTimeout =00:01:00
openTimeout =〇时01分00秒receiveTimeout =○时30分○○秒的SendTimeout =〇时01分00秒
transactionFlow =假transferMode =流媒体transactionProtocol =OleTransactions
hostNameComparisonMode =StrongWildcard将ListenBackLog =10
maxBufferPoolSize =524288maxBufferSize =65536MAXCONNECTIONS =10
maxReceivedMessageSize =4294967295>
< readerQuotas MAXDEPTH =32maxStringContentLength =8192maxArrayLength =16384
maxBytesPerRead =4096maxNameTableCharCount =16384/>
<有序的ReliableSession =真正的inactivityTimeout =00:10:00
启用=FALSE/>
<安全模式=运输>
<交通运输clientCredentialType =窗口的ProtectionLevel =EncryptAndSign/>
<消息clientCredentialType =的Windows/>
< /安全>
< /装订>
< / NetTcpBinding的>
< /绑定>
<客户端>
<端点地址=的net.tcp://本地主机:9000 / streamserver绑定=NetTcpBinding的
bindingConfiguration =NetTcpBinding_IStream合同=Service.IStream
NAME =NetTcpBinding_IStream>
< /端点>
< /客户>
< /system.serviceModel>
启动该服务,然后客户端。它会流大量的文件没有问题。
I'm currently experiencing an issue trying to send large data using WCF self hosted service (no IIS). Transferring 500MB using streaming results with my service crashing with System.OutOfMemoryException. Is it possible at all to transfer such amount of data?
Here is my WCF configuration:
<system.serviceModel>
<services>
<service name="CIService" behaviorConfiguration="CIBehavior">
<host>
<baseAddresses>
<add baseAddress="net.tcp://localhost:6547/CIService/CIService.svc" />
</baseAddresses>
</host>
<endpoint binding="netTcpBinding"
bindingConfiguration="netTcpBindingConfig"
behaviorConfiguration="CIBehavior.EndpointBehavior"
contract="CIService.ICreatable" />
<endpoint address="mex"
binding="mexHttpBinding"
name="mexTcpBinding"
contract="IMetadataExchange" />
</service>
</services>
<serviceHostingEnvironment multippleSiteBindingEnabled="True" />
<bindings>
<netTcpBinding>
<binding name="netTcpBindingConfig" closeTimeout="00:01:00" openTimeout="00:01:00"
receiveTimeout="01:00:00" sendTimeout="00:10:00"
hostNameComparisonMode="StrongWildcard" listenBacklog="10" maxConnections="10"
maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"
transferMode="Streamed">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647"
maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
</binding>
</netTcpBinding>
</bindings>
<behaviors>
<serviceBehaviors>
<behavior name="CIBehavior">
<serviceMetadata httpGetEnabled="False" />
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceThrottling maxConcurrentCalls="200" maxConcurrentInstances="2147483647" maxConcurrentSessions="100" />
<dataContractSerializer maxItemsInObjectGraph="2147483647"/>
</behavior>
</serviceBehaviors>
<endpointBehavior>
<behavior name="CIBehavior.EndpointBehavior">
<dataContractSerializer maxItemsInObjectGraph="2147483647" />
</behavior>
</endpointBehavior>
</behaviors>
</system.serviceModel>
My client configuration:
<system.serviceModel>
<bindings>
<netTcpBinding>
<binding name="NetTcpBinding_ICreatable"
closeTimeout="00:01:00" openTimeout="00:01:00"
receiveTimeout="01:00:00" sendTimeout="00:10:00"
transactionFlow="false"
transferMode="Streamed"
transactionProtocol="OleTransactions"
hostNameComparisonMode="StrongWildcard"
listenBacklog="10"
maxBufferPoolSize="2147483647"
maxBufferSize="2147483647"
maxConnections="10"
maxReceivedMessageSize ="2147483647">
<readerQuotas
maxDepth="2147483647"
maxStringContentLength="2147483647"
maxArrayLength="2147483647"
maxBytesPerRead="2147483647"
maxNameTableCharCount="2147483647" />
<reliableSession ordered="true" inactivityTimeout="00:10:00" anabled="false" />
</binding>
</netTcpBinding>
</bindings>
<client>
<endpoint name="NetTcpBinding_ICreatable"
address="net.tcp://localhost:6547/CIService/CIService.svc"
binding="netTcpBinding"
bindingConfiguration="NetTcpBinding_ICreatable"
behaviorConfiguration="CIBehavior.EndpointBehavior"
contract="ICreatable" />
</client>
<behaviors>
<endpointBehavior>
<behavior name="CIBehavior.EndpointBehavior">
<dataContractSerializer maxItemsInObjectGraph="2147483647" />
</behavior>
</endpointBehavior>
</behaviors>
</system.serviceModel>
You dont need maxBufferSize or maxBufferPoolSize set so high these are possibly causing your out of memory exception. The defaults should be fine.
Check out Large Data and Streaming on MSDN, specifically the section Special Security Considerations for Large Data this piece of the text is important
The MaxBufferSize property is required to constrain the memory that WCF buffers. It is important to set this to a safe value (or keep it at the default value) when streaming. For example, suppose your service must receive files up to 4 GB in size and store them on the local disk. Suppose also that your memory is constrained in such a way that you can only buffer 64 KB of data at a time. Then you would set the MaxReceivedMessageSize to 4 GB and MaxBufferSize to 64 KB. Also, in your service implementation, you must ensure that you read only from the incoming stream in 64-KB chunks and do not read the next chunk before the previous one has been written to disk and discarded from memory.
I put together a very simple example of streaming data from a self hosted service to a console client. To keep the post short I only added the server code and part of the client.
The service contract
using System.IO;
using System.ServiceModel;
namespace Service
{
[ServiceContract]
public interface IStream
{
[OperationContract]
Stream GetLargeObject();
}
}
The service implementation
using System;
using System.IO;
using System.ServiceModel;
namespace Service
{
[ServiceBehavior]
public class StreamService : IStream
{
public Stream GetLargeObject()
{
// Add path to a big file, this one is 2.5 gb
string filePath = Path.Combine(Environment.CurrentDirectory, "C:\\Temp\\BFBC2_PC_Client_R11_795745_Patch.exe");
try
{
FileStream imageFile = File.OpenRead(filePath);
return imageFile;
}
catch (IOException ex)
{
Console.WriteLine(String.Format("An exception was thrown while trying to open file {0}", filePath));
Console.WriteLine("Exception is: ");
Console.WriteLine(ex.ToString());
throw;
}
}
}
}
The service main
using System;
using System.ServiceModel;
namespace Service
{
class Program
{
static void Main(string[] args)
{
try
{
using (var serviceHost = new ServiceHost(typeof(StreamService)))
{
serviceHost.Open();
Console.WriteLine("Press Any Key to end");
Console.ReadKey();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
}
}
The service app.config
<?xml version="1.0"?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
</startup>
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior name="StreamServiceBehavior">
<serviceMetadata httpGetEnabled="True" />
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<netTcpBinding>
<binding name="NewBinding0" transferMode="Streamed"/>
</netTcpBinding>
</bindings>
<services>
<service behaviorConfiguration="StreamServiceBehavior" name="Service.StreamService">
<endpoint address="net.tcp://localhost:9000/streamserver" binding="netTcpBinding"
bindingConfiguration="NewBinding0" bindingName="" contract="Service.IStream" />
<endpoint address="mex" binding="mexHttpBinding"
contract="IMetadataExchange" />
<host>
<baseAddresses>
<add baseAddress="http://localhost:8080/StreamService" />
</baseAddresses>
</host>
</service>
</services>
</system.serviceModel>
</configuration>
Launch the service, may need to run under admin account to open the socket. Create a client console application and add a service reference using the url http:// localhost:8080 / StreamService, using Service as the namespace for the generated client.
The client main
using System;
using System.IO;
using Client.Service;
namespace Client
{
class Program
{
static void Main(string[] args)
{
try
{
using (StreamClient streamClient = new StreamClient())
{
streamClient.Open();
using (FileStream fileStream = new FileStream("c:\\temp\\bigfile.exe",FileMode.Create))
{
streamClient.GetLargeObject().CopyTo(fileStream);
}
}
Console.WriteLine("Press any key to end");
Console.ReadKey();
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
}
}
}
The generated client config file will need to be modified slightly, increase receiveTimeout and set maxReceivedMessageSize="4294967295"
<system.serviceModel>
<bindings>
<netTcpBinding>
<binding name="NetTcpBinding_IStream" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:30:00" sendTimeout="00:01:00"
transactionFlow="false" transferMode="Streamed" transactionProtocol="OleTransactions"
hostNameComparisonMode="StrongWildcard" listenBacklog="10"
maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"
maxReceivedMessageSize="4294967295">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<reliableSession ordered="true" inactivityTimeout="00:10:00"
enabled="false" />
<security mode="Transport">
<transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
<message clientCredentialType="Windows" />
</security>
</binding>
</netTcpBinding>
</bindings>
<client>
<endpoint address="net.tcp://localhost:9000/streamserver" binding="netTcpBinding"
bindingConfiguration="NetTcpBinding_IStream" contract="Service.IStream"
name="NetTcpBinding_IStream">
</endpoint>
</client>
</system.serviceModel>
Launch the service then the client. It will stream a large file without issue.
这篇关于WCF流大数据(500MB / 1GB)在自我托管服务的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!