控制占用大量内存的多线程应用程序 [英] Control Memory-Hungy Multi-Threaded App
问题描述
这是一个非常开放的问题.
This a VERY open question.
基本上，我有一个计算应用程序，可以为 N 个场景启动测试组合.
每个测试都在一个专用线程中进行，包括读取大型二进制数据、处理它并将结果放入数据库.
Basically, I have a computing application that launches test combinations for N Scenarios.
Each test is conducted in a single dedicated thread, and involves reading large binary data, processing it, and dropping results to DB.
如果线程数太大，应用程序会变得流氓并吃掉所有可用内存并挂起..利用所有 CPU+RAM 功能的最有效方法是什么(高性能计算，即 12 核/16GB RAM)而不会使系统瘫痪(如果太多"同时启动的线程，太多"当然是一个相对的概念)
If the number of threads is too large, the app gets rogue and eats out all available memory and hangs out.. What is the most efficient way to exploit all CPU+RAM capabilities (High Performance computing i.e 12Cores/16GB RAM) without putting the system down to its knees (which happens if "too many" simultaneous threads are launched, "too many" being a relative notion of course)
我必须指定我有一个带有 N 个工作器的工作器缓冲队列，每次完成并死亡时，都会通过队列启动一个新的工作器.到目前为止，这工作得很好.但我想避免手动"和凭经验"设置并发线程数，并拥有一个智能可扩展系统，在系统可以正确处理的情况下一次丢弃尽可能多的线程，并在合理"的内存使用(目标服务器专用于应用程序，因此除系统外的其他应用程序没有问题)
I have to specify that I have a workers buffer queue with N workers, every time one finishes and dies a new one is launched via a Queue. This works pretty fine as of now. But I would like to avoid "manually" and "empirically" setting the number of simultaneous threads and have an intelligent scalable system that drops as many threads at a time that the system can properly handle, and stop at a "reasonable" memory usage (the target server is dedicated to the app so there is no problem regarding other applications except the system)
PS:我知道 .Net 3.5 带有线程池，而 .Net 4 有有趣的 TPL 功能，我现在仍在考虑中(到目前为止我从未深入研究过).
PS : I know that .Net 3.5 comes with Thread Pools and .Net 4 has interesting TPL capabilites, that I am still considering right now (I never went very deep into this so far).
PS 2 : 在阅读了这篇文章后，我对不要这样做"的回答.虽然我认为这样的要求对于需要内存的计算程序来说是公平的.
PS 2 : After reading this post I was a bit puzzled by the "don't do this" answers. Though I think such request is fair for a memory-demanding computing program.
推荐答案
.NET 中的所有内置线程功能都不支持根据内存使用情况进行调整.您需要自己构建.
All built-in threading capabilities in .NET do not support adjusting according to memory usage. You need to build this yourself.
您可以预测内存使用情况或对低内存情况做出反应.替代方案:
You can either predict memory usage or react to low memory conditions. Alternatives:
- 在启动新任务之前查看系统上的可用内存量.如果它低于 500mb，请等待释放足够的空间.
- 在任务到来时启动任务，并在其中一些任务因 OOM 开始失败时立即进行限制.稍后重新启动它们.这种替代方法很费时间，因为您的进程会疯狂地进行垃圾收集以避免 OOM.
我推荐 (1).
您可以查看空闲系统内存或您自己的进程内存使用情况.为了获得内存使用情况，我建议使用 Process 类查看私有字节.
You can either look at free system memory or your own processes memory usage. In order to get the memory usage I recommend looking at private bytes using the Process class.
如果您在 16GB 系统上留出 1GB 缓冲区，那么您的运行效率为 94%，并且非常安全.
If you set aside 1GB of buffer on your 16GB system you run at 94% efficiency and are pretty safe.
这篇关于控制占用大量内存的多线程应用程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
