“块状”内存流 [英] "Chunked" MemoryStream
问题描述
我正在寻找MemoryStream的实现，该实现不将内存分配为一个大块，而是分配一组块。我想在内存（64位）中存储几GB的数据，并避免内存碎片的限制。
首先确定虚拟地址碎片是否是问题所在。
如果您使用的是64位计算机（似乎表明您在使用），则我对此表示严重怀疑。每个64位进程几乎都具有全部可用的64位虚拟内存空间，您唯一担心的是虚拟地址空间碎片而不是物理内存碎片（这是操作系统必须担心的）。 OS内存管理器已经在内存页下分页了。在可预见的将来，您将不会用完虚拟地址空间，然后再用尽物理内存。
如果您有一个32位地址空间，那么在GB内存中分配连续的大内存块时，您将遇到碎片问题很快。 CLR中没有分配内存流的库存块。 ASP.NET的幕后有一个（出于其他原因），但是不可访问。如果您必须走这条路，那么您最好还是自己写一个，因为您的应用程序的使用模式不太可能与许多其他应用程序相似，并且尝试将数据装入32位地址空间可能是您的性能瓶颈。如果要处理GB的数据，我强烈建议要求64位处理。不管您是如何砍刀，与32位地址空间分段的手动解决方案相比，它将做得更好。
I'm looking for the implementation of MemoryStream which does not allocate memory as one big block, but rather a collection of chunks. I want to store a few GB of data in memory (64 bit) and avoid limitation of memory fragmentation.
You need to first determine if virtual address fragmentation is the problem.
If you are on a 64 bit machine (which you seem to indicate you are) I seriously doubt it is. Each 64 bit process has almost the the entire 64 bit virtual memory space available and your only worry is virtual address space fragmentation not physical memory fragmentation (which is what the operating system must worry about). The OS memory manager already pages memory under the covers. For the forseeable future you will not run out of virtual address space before you run out of physical memory. This is unlikely change before we both retire.
If you are have a 32 bit address space, then allocating contiguous large blocks of memory in the GB ramge you will encounter a fragmentation problem quite quickly. There is no stock chunk allocating memory stream in the CLR. There is one in the under the covers in ASP.NET (for other reasons) but it is not accessable. If you must travel this path you are probably better off writing one youself anyway because the usage pattern of your application is unlikely to be similar to many others and trying to fit your data into a 32bit address space will likely be your perf bottleneck.
I highly recommend requiring a 64 bit process if you are manipulating GBs of data. It will do a much better job than hand-rolled solutions to 32 bit address space fragmentation regardless of how cleaver you are.
这篇关于“块状”内存流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
