大对象堆和String对象从队列中来了 [英] Large Object Heap and String Objects coming from a queue

问题描述

我有一个Windows控制台应用程序，应该是没有重新启动的日期和月份运行。这个应用程序获取从MSMQ和处理它的工作。有迹象表明，同时处理一个工作组块30个线程。

I have a windows console app that is supposed to run without restarts for days and months. The app retrieves "work" from an MSMQ and process it. There are 30 threads that process a work chunk simultaneously.

每个工作块从MSMQ未来大约为200KB其中大部分被分配在一个String对象。

Each work chunk coming from the MSMQ is approximately 200kb most of which is allocated in a single String object.

我注意到，处理约3-4个这样的工作组块后，应用程序的内存消耗高的离谱消耗1 - 1.5 GB的内存。

I have noticed that after processing about 3-4 thousands of these work chunks the memory consumption of the application is ridiculously high consuming 1 - 1.5 gb of memory.

我运行通过探查应用程序，并注意到，大多数这种内存（可能是演出左右）是不用在大对象堆，但结构是分散的。

I run the app through a profiler and noticed that most of this memory (maybe a gig or so) is unused in the large object heap but the structure is fragmented.

我还发现，90％的未使用的（垃圾回收）个字节为previously分配的字符串。我开始怀疑然后从MSMQ进来琴弦被分配，使用和随后释放，因此破碎的原因。

I have found that 90% of these unused (garbage collected) bytes were previously allocated String. I started suspecting then that the strings coming in from the MSMQ were allocated, used and then deallocated and are therefore the cause of the fragmentation.

据我所知，像GC.Collect的（2或GC.Max ...）不会帮助，因为它们GC大对象堆，但不压缩它（这是这里的问题）。因此，我认为我需要的是缓存这些字符串，并重新使用它们以某种方式，但由于字符串是不可变的，我将不得不使用StringBuilders。

I understand that things like GC.Collect(2 or GC.Max...) wont help since they gc the large object heap but don't compact it (which is the problem here). So I think that what I need is to cache these Strings and re-use them somehow but since Strings are immutable I would have to use StringBuilders.

我的问题是：反正有没有改变底层结构（即使用MSMQ，因为这是我不能改变），仍然避免初始化一个新的字符串每次以避免分裂蕙

My question is: Is there anyway to not change the underlying structure (i.e. using the MSMQ as this is something I cant change) and still avoid initializing a new String everytime to avoid fragmenting the LOH?

谢谢， 雅尼斯

更新：关于如何将这些工作块正在检索

目前这些被存储为在MSMQ WorkChunk对象。每个对象包含一个字符串名为内容与另一个字符串被称为头。这些都是实际的文本数据。如果需要，如果需要超过一个MSMQ别的东西，我可以改变存储结构到别的东西和潜在的底层存储机制。

Currently these are stored as WorkChunk objects in the MSMQ. Each of these objects contains a String called Contents and another String called Headers. These are actual textual data. I can change the storage structure to something else if needed and potentially the underlying storage mechanism if needed to something else than an MSMQ.

在工作节点端目前我们做的

On the worker nodes side currently we do

WorkChunk块= _Queue.Receive（）;

WorkChunk chunk = _Queue.Receive();

因此​​，有一点我们可以在这个阶段缓存。如果我们改变了结构（S）莫名其妙地话，我想我们可以做一点进步。在任何情况下，我们必须弄清楚这个问题，所以我们会尽一切是为了避免抛出个月的工作。

So there is little we can cache at this stage. If we changed the structure(s) somehow then I suppose we could do a bit of progress. In any case, we will have to sort out this problem so we will do whatever is needed to avoid throwing out months of work.

更新：我去尝试一些下面的建议，并注意到这个问题不能被复制我的本地机器（运行Windows 7 64位和64位应用程序）。这使得事情这么多困难 - 如果有谁知道为什么那么它会真正帮助repdocung这个问题在本地

UPDATE: I went on to try some of the suggestions below and noticed that this issue cannot be reproduced on my local machine (running Windows 7 x64 and 64bit app). this makes things so much more difficult - if anyone knows why then it would really help repdocung this issue locally.

推荐答案

看来你的问题是由于对大对象堆内存分配 - 大对象堆不板结等都可以成为碎片的来源。有一个很好的文章在这里，将详细在内的一些调试步骤，您可以按照确认的大对象堆的碎片正在发生的事情：

Your problem appears to be due to memory allocation on the large object heap - the large object heap is not compacted and so can be a source of fragmentation. There is a good article here that goes into more detail including some debugging steps that you can follow to confirm that fragmentation of the large object heap is happening:

大对象堆揭秘

您似乎有<打击>两个三种解决方法：

You appear to have two three solutions:

1. 在改变你的应用程序在块/更短的字符串，其中每个块较小超过85000字节进行处理 - 这样就避免了大对象的分配
。
2. 在改变你的应用程序分配内存的前期，并通过复制新邮件到所分配的内存，而不是重新使用这些块几大块。请参见堆碎片使用字节数组的时候。
3. 将事物的本来面目 - 只要你不会遇到内存不足异常出来，应用程序与系统不你应该心存因为它们是在运行其他应用程序的干扰。

1. Alter your application to perform processing on chunks / shorter strings, where each chunk is smaller than 85,000 bytes - this avoids the allocation of large objects.
2. Alter your application to allocate a few large chunks of memory up-front and re-use those chunks by copying new messages into the allocated memory instead. See Heap fragmentation when using byte arrays.
3. Leave things as they are - As long as you don't experience out of memory exceptions and the application isn't interfering with other applications running on the system you should probably leave things as they are.

及其重要此处理解虚拟存储器和物理存储器之间的区别 - 尽管该过程使用大量的虚拟内存，如果分配的对象的数量相对低则凸轮是因为的，实际的存储器使用工艺低（未使用的内存分页到磁盘），这意味着在系统上的其他进程影响不大。您还可能会发现，VM囤积选项帮助 - 读大对象堆揭秘的文章以了解更多信息。

Its important here to understand the distinction between virtual memory and physical memory - even though the process is using a large amount of virtual memory, if the number of objects allocated is relatively low then it cam be that the physical memory use of that process is low (the un-used memory is paged to disk) meaning little impact on other processes on the system. You may also find that the "VM Hoarding" option helps - read "Large Object Heap Uncovered" article for more information.

要么改变涉及到改变你的应用程序来执行一些或者全部使用字节数组和短子，而不是一个单一的大串的处理 - 多么困难这将是对你将取决于它是一个什么样的处理你在做什么。

Either change involves changing your application to perform either some or all of its processing using byte arrays and short substrings instead of a single large string - how difficult this is going to be for you will depend on what sort of processing it is that you are doing.

这篇关于大对象堆和String对象从队列中来了的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！