OutOfMemoryException异常上写的MemoryStream [英] OutOfMemoryException on MemoryStream writing
问题描述
我有我正在试图获得一些新的.NET 4.0并行扩展会稍微示例应用程序（它们是很不错的）。我正在与一个OutOfMemoryException（可能是非常愚蠢的）问题。我说我在找这个样本插到主要的应用程序读取一些数据和大量的文件，确实对他们的一些处理，然后写到某处出来。我当时正在与文件挺大的一些问题（可能是GB的），并担心内存，所以我想并行的东西，害得我沿着这条道路。
现在下面的代码获取较小的文件的OOME，我想我只是失去了一些东西。它会在10-15文件中读取，并在中并行很好地把它们写出来，但随后扼流圈下一个。它看起来像它的读取和写入650MB左右。第二组的眼睛，将不胜感激。
我读到从一个的FileStream因为MemorySteam这是什么是需要的主要应用程序，我只是想复制该到一定程度。它从所有类型的地方读取数据和文件，并对其工作原理MemoryStreams。
这是使用.NET 4.0 Beta 2时，VS 2010。
命名空间ParellelJob
{
类节目
{
BlockingCollection< FileHolder> serviceToSolutionShare;
静态无效的主要（字串[] args）
{
程序P =新计划（）;
p.serviceToSolutionShare =新BlockingCollection< FileHolder>（）;
ServiceStage SVC =新ServiceStage（REF p.serviceToSolutionShare）;
SolutionStage溶胶=新SolutionStage（REF p.serviceToSolutionShare）;
变种svcTask = Task.Factory.StartNew（（）=> svc.Execute（））;
变种solTask = Task.Factory.StartNew（（）=> sol.Execute（））;
，而（！solTask.IsCompleted）
{
}
}
}
类ServiceStage
{
BlockingCollection< FileHolder> outputCollection;
公共ServiceStage（REF BlockingCollection< FileHolder>输出）
{
outputCollection =输出;
}
公共无效执行（）
{
变种DI =新的DirectoryInfo（@C：\temp\testfiles）;
var中的文件= di.GetFiles（）;
的foreach（FileInfo的网络连接文件）
{
使用（VAR FS =新的FileStream（fi.FullName，FileMode.Open，FileAccess.Read））
{
INT b：
变种MS =新的MemoryStream（）;
，而（（B = fs.ReadByte（））！= -1）
{
ms.WriteByte（（字节），B）; // OutOfMemoryException异常发生在这里，
}
变种F =新FileHolder（）;
f.filename = fi.Name;
f.contents =毫秒;
outputCollection.TryAdd（F）;
}
}
outputCollection.CompleteAdding（）;
}
}
类SolutionStage
{
BlockingCollection< FileHolder> inputCollection;
公共SolutionStage（REF BlockingCollection< FileHolder>输入）
{
inputCollection =输入;
}
公共无效执行（）
{
FileHolder电流;
，而
{
使用（！inputCollection.IsCompleted）
{
如果（inputCollection.TryTake（出电流））（VAR FS =新的FileStream（的String.Format （@C：\temp\parellel\ {0}，current.filename），FileMode.OpenOrCreate，FileAccess.Write））
{使用
（MemoryStream的MS =（MemoryStream的）电流.contents）
{
ms.WriteTo（FS）;
current.contents.Close（）;
}
}
}
}
}
}
类FileHolder
{
公字符串的文件名{获得;组; }
公共流内容{搞定;组; }
}
}
主要的逻辑似乎不错，但如果空while循环中主要是文字，那么你正在燃烧unnecesary CPU周期。更好地利用solTask.Wait（）代替。
但是，如果单个文件可以运行千兆字节，你仍然有完全在内存中至少持有1的问题，通常2 （1被读取，1个正在处理/写
PS1：我刚刚意识到你不预先分配的MemStream这是不好的，它必须重新大小往往一个大文件，并花费了大量的内存更好地利用这样的：
VAR毫秒=新的MemoryStream （fs.Length）;
然后，在大文件，你必须要考虑的大对象堆（LOH）你确定你不能在细分打破文件并处理它们。
PS2：你不需要在构造函数的参数裁判的，但是这不是问题。
I have a little sample application I was working on trying to get some of the new .Net 4.0 Parallel Extensions going (they are very nice). I'm running into a (probably really stupid) problem with an OutOfMemoryException. My main app that I'm looking to plug this sample into reads some data and lots of files, does some processing on them, and then writes them out somewhere. I was running into some issues with the files getting bigger (possibly GB's) and was concerned about memory so I wanted to parallelize things which led me down this path.
Now the below code gets an OOME on smaller files and I think I'm just missing something. It will read in 10-15 files and write them out in parellel nicely, but then it chokes on the next one. It looks like it's read and written about 650MB. A second set of eyes would be appreciated.
I'm reading into a MemorySteam from the FileStream because that is what is needed for the main application and I'm just trying to replicate that to some degree. It reads data and files from all types of places and works on them as MemoryStreams.
This is using .Net 4.0 Beta 2, VS 2010.
namespace ParellelJob
{
class Program
{
BlockingCollection<FileHolder> serviceToSolutionShare;
static void Main(string[] args)
{
Program p = new Program();
p.serviceToSolutionShare = new BlockingCollection<FileHolder>();
ServiceStage svc = new ServiceStage(ref p.serviceToSolutionShare);
SolutionStage sol = new SolutionStage(ref p.serviceToSolutionShare);
var svcTask = Task.Factory.StartNew(() => svc.Execute());
var solTask = Task.Factory.StartNew(() => sol.Execute());
while (!solTask.IsCompleted)
{
}
}
}
class ServiceStage
{
BlockingCollection<FileHolder> outputCollection;
public ServiceStage(ref BlockingCollection<FileHolder> output)
{
outputCollection = output;
}
public void Execute()
{
var di = new DirectoryInfo(@"C:\temp\testfiles");
var files = di.GetFiles();
foreach (FileInfo fi in files)
{
using (var fs = new FileStream(fi.FullName, FileMode.Open, FileAccess.Read))
{
int b;
var ms = new MemoryStream();
while ((b = fs.ReadByte()) != -1)
{
ms.WriteByte((byte)b); //OutOfMemoryException Occurs Here
}
var f = new FileHolder();
f.filename = fi.Name;
f.contents = ms;
outputCollection.TryAdd(f);
}
}
outputCollection.CompleteAdding();
}
}
class SolutionStage
{
BlockingCollection<FileHolder> inputCollection;
public SolutionStage(ref BlockingCollection<FileHolder> input)
{
inputCollection = input;
}
public void Execute()
{
FileHolder current;
while (!inputCollection.IsCompleted)
{
if (inputCollection.TryTake(out current))
{
using (var fs = new FileStream(String.Format(@"c:\temp\parellel\{0}", current.filename), FileMode.OpenOrCreate, FileAccess.Write))
{
using (MemoryStream ms = (MemoryStream)current.contents)
{
ms.WriteTo(fs);
current.contents.Close();
}
}
}
}
}
}
class FileHolder
{
public string filename { get; set; }
public Stream contents { get; set; }
}
}
The main logic seems OK, but if that empty while-loop in main is literal then you are burning unnecesary CPU cycles. Better use solTask.Wait() instead.
But if individual files can run in Gigabytes, you still have the problem of holding at least 1 completely in memory, and usually 2 (1 being read, 1 being processed/written.
PS1: I just realized you don't pre-allocate the MemStream. That's bad, it will have to re-size very often for a big file, and that costs a lot of memory. Better use something like:
var ms = new MemoryStream(fs.Length);
And then, for big files, you have to consider the Large Object Heap (LOH). Are you sure you can't break a file up in segments and process them?
PS2: And you don't need the ref's on the constructor parameters, but that's not the problem.
这篇关于OutOfMemoryException异常上写的MemoryStream的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
