在具有动态大小的内存池的多线程C/C ++中实现内存管理器? [英] Implementing a memory manager in multithreaded C/C++ with dynamically sized memory pool?

问题描述

背景:我正在开发一种多平台框架，该框架将用作 game 和 util/tool 创建的基础.基本思想是要有一组工作程序，每个工作程序都在自己的线程中执行. (此外，工作人员还可以在运行时生成.)每个线程都有其自己的内存管理器.

Background: I'm developing a multiplatform framework of sorts that will be used as base for both game and util/tool creation. The basic idea is to have a pool of workers, each executing in its own thread. (Furthermore, workers will also be able to spawn at runtime.) Each thread will have it's own memory manager.

我一直在考虑创建自己的内存管理系统，并且我认为这个项目将是一个完美的尝试.由于该框架的使用类型，我发现这种系统很合适，通常需要实时分配内存(游戏和纹理编辑工具).

I have long thought about creating my own memory management system, and I think this project will be perfect to finally give it a try. I find such a system fitting due to the types of usages of this framework will often require memory allocations in realtime (games and texture edition tools).

问题:

• 没有通用的解决方案(?)-该框架将用于游戏/可视化(不是AAA，而是独立游戏)和工具/应用程序创建.我的理解是，对于游戏开发，通常(至少对于主机游戏而言)在初始化时只分配一次大块内存，然后在内存管理器内部使用此内存.但是，这种技术是否适用于更一般的应用?

• No generally applicable solution(?) - The framework will be used for both games/visualization (not AAA, but indie/play) and tool/application creation. My understanding is that for game development it is usual (at least for console games) to allocate a big chunk of memory only once in the initialization, and then use this memory internally in the memory manager. But is this technique applicable in a more general application?

理论上，您可以在游戏中知道场景和资源需要多少存储空间，但是例如，照片编辑应用程序将加载所有不同大小的资源...因此，在后一种情况下，动态存储空间的大小会更大"是否需要?这导致我遇到下一个问题:

In a game you could theoretically know how much memory your scenes and resources will need, but for example, a photo editing application will load resources of all different sizes... So in the latter case a more dynamic memory "chunk size" would be needed? Which leads me to the next problem:

移动已分配的数据并保留有效的指针-通常，在堆上进行分配时，您将获得指向内存块的简单指针.据我所知，在自定义内存管理器中，一种类似的方法是将指针返回到预分配块中空闲的某个位置.但是，如果预分配的块太小并且需要调整大小甚至进行碎片整理，会发生什么?数据将需要在内存中移动，旧指针将无效.有没有办法以某种方式透明地包装这些指针，但是仍然像通常的C ++指针一样将它们用作内存管理的外部"?

Moving already allocated data and keeping valid pointers - Normally when allocating on the heap, you will acquire a simple pointer to the memory chunk. In a custom memory manager, as far as I understand it, a similar approach is then to return a pointer to somewhere free in the pre-allocated chunk. But what happens if the pre-allocated chunk is too small and needs to be resized or even defragmentated? The data would be needed to be moved around in the memory and the old pointers would be invalid. Is there a way to transparently wrap these pointers in some way, but still use them as normally "outside" the memory management as if they were usual C++ pointers?

第三方库-如果无法透明地将自定义内存管理系统用于应用程序中的所有内存分配，则我链接的每个第三方库仍会在内部使用旧的" OS内存分配.我了解到，库公开函数以设置库将使用的自定义分配函数是很常见的，但是并不能保证我将使用的每个库都具有此功能.

Third party libraries - If there is no way to transparently use a custom memory management system for all memory allocation in the application, every third party library I'm linking with, will still use the "old" OS memory allocations internally. I have learned that it is common for libraries to expose functions to set custom allocation functions that the library will use, but it is not guaranteed every library I will use will have this ability.

问题:实现可以使用动态大小的内存块池的内存管理器是否可行?如果是这样，如何在不破坏当前正在使用的指针的情况下进行碎片整理和调整内存大小?最后，如何最好地实现这种系统以与第三方库一起使用?

Questions: Is it possible and feasible to implement a memory manager that can use a dynamically sized memory chunk pool? If so, how would defragmentation and memory resize work, without breaking currently in-use pointers? And finally, how is such a system best implemented to work with third party libraries?

我也感谢任何相关的阅读材料，论文，文章和其他内容！ :-)

I'm also thankful for any related reading material, papers, articles and whatnot! :-)

推荐答案

1. 准备一个以上的解决方案，并让框架的用户采用任何特定的解决方案.您开发的通用分配器的策略类可以很好地做到这一点.

1. Prepare more than one solution and let the user of the framework adopt any particular one. Policy classes to the generic allocator you develop would do this nicely.

一种解决此问题的好方法是将指针包装在带有重载*运算符的类中.使该类的内部数据仅是内存池的索引.现在，您可以在后台线程将数据复制完之后快速更改索引.

A nice way to get around this is to wrap up pointers in a class with overloaded * operator. Make the internal data of that class only an index to the memory pool. Now, you can just change the index quickly after a background thread copies the data over.

大多数good C ++库支持分配器，您应该实现一个分配器.您还可以重载全局新文件，以便使用您的版本.并且请记住，通常不需要考虑分配或取消分配大量数据的库，这通常是客户端代码的职责.

Most good C++ libraries support allocators and you should implement one. You can also overload the global new so your version gets used. And keep in mind that you generally won't need to think about a library allocating or deallocating a large amount of data, which is generally a responsibility of client code.

这篇关于在具有动态大小的内存池的多线程C/C ++中实现内存管理器?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！