跨共享/静态库集成C ++自定义内存分配器 [英] Integrating C++ custom memory allocators across shared/static libraries

问题描述

我开始使用一些自定义内存分配器，例如 rpmalloc 和

I started to use some custom memory allocators such rpmalloc and ltmalloc into my project, but I have some concern about the integration, my project has various internal modules built as shared libraries or static libraries (depending how I configure them in my build system) and should build/run for the Windows/Linux/FreeBSD/Mac OS X, and architectures such x86 and ARM, and I don't know if I should make the calls of my memory allocator integrations inside of the header file or should remain inside of the cpp file.

如果内存分配器调用保留在头文件中，则每个模块都应链接内存分配器的静态库，如果它保存在.cpp文件中，则调用将包含在包含它们的库中，并且仅包含该模块应该链接自定义内存分配器，但是该模块应该包含一个接口，每个模块都可以分配它们(避免内存分配不一致)

If the memory allocator calls stay in the header files, every module should link the static library of the memory allocator, if it is kept in the .cpp file the calls are contained in the library who contains them, and only that module should link the custom memory allocator, but that module should contain an interface to every module can allocate them (avoiding memory allocation inconsistency)

我已经阅读了此处如果正常分配了内存(如malloc/free/syscalls一样)，则每个共享库都有自己的堆，但是如果使用mmap，则会分配不属于程序堆的内存.

I've read here if the memory is allocated normally (like malloc/free/syscalls does) every shared library has his own heap, but if uses mmap allocates memory which doesn't belong to the program's heap.

我的问题是，如果共享/静态库保存在一个库中，是否会对任何危害(但其他所有库都应链接它才能访问其内存分配接口)?还是应该将所有内容都内联到头文件中，并且每个库都应链接内存分配器库?

My question is does it introduce any hazard in my shared/static libraries if they are kept into one library (but every other library should link it in order to access their memory allocation interfaces)? or should everything inline in the header files, and every library should link the memory allocator library?.

推荐答案

内存分配的完成方式很大程度上取决于操作系统.您需要了解共享库在这些操作系统中的工作方式，C语言如何与那些操作系统以及共享库的概念相关.

How memory allocation is done heavily depends on the operating system. You need to understand how shared libraries work in those operating systems, how C language relates to those operating systems and to the concept of shared libraries.

首先，我想提一下C语言不是不是模块化语言，例如它不支持模块或模块化编程.对于诸如C和C ++之类的语言，模块化编程的实现由底层操作系统决定.共享库是用于用C和C ++实现模块化编程的机制的示例，因此我将它们称为 modules .

First of all, I want to mention that C language is not a modular language e.g. it has no support for modules or modular programming. For languages like C and C++ implementation of modular programming is left to the underlying operating system. Shared libraries is an example of mechanism that is used to implement modular programming with C and C++, therefore I will refer to them as modules.

模块=共享库和可执行文件

最初，Unix系统上的所有内容都是静态链接的.共享库稍后发布.而且，由于Unix是C语言的起点，因此这些系统试图提供与C语言编程相近的共享库编程接口.

Initially everything on Unix systems was statically linked. Shared libraries came later. And as Unix was a starting point for the C langauge, those systems try to provide shared library programming interface that is close to what programming in C feels like.

这个想法是，应该在不考虑共享库的情况下构建最初编写的C代码，并且应该在不更改源代码的情况下工作.结果，所提供的环境通常具有由所有加载的模块共享的单个进程范围的符号名称空间，例如在整个过程中，只有一个名称为foo的函数，除了static函数(以及某些使用OS特定机制的模块中的hidden函数).基本上与静态链接相同，在静态链接中，不允许使用重复的符号.

The idea is that C code written originally without shared libraries in mind should be build and should work without changes made to the source code. As the result, provided environment usually has single process-wide symbol namespace shared by all loaded modules e.g. there can only be a single function with a name foo in the whole process, except for static functions (and some functions that are hidden in moduels using OS-specific mechanisms). Basically it is the same as with static linking where you are not allowed to have duplicate symbols.

这对您而言意味着在整个过程中始终使用一个名为malloc的函数，并且每个模块都在使用它，例如所有模块共享相同的内存分配器.

What this means for your case is that there is always a single function named malloc in use in the whole process and every module is using it e.g. all modules share the same memory allocator.

现在，如果进程碰巧具有多个malloc功能，则仅选择一个功能，所有模块都将使用该功能.这里的机制非常简单-由于共享库不知道每个引用函数的位置，因此它们通常会通过某些表(GOT，PLT)来调用它们，这些表在第一次调用或加载时会懒洋洋地填充所需的地址时间.提供原始功能的模块采用了相同的规则-即使在同一表中也将在内部调用此功能，即使在提供该功能的原始模块中也可以覆盖该功能(这是许多与效率低下有关的原因在Linux上使用共享库，请搜索-fno-semantic-interposition，-fno-plt来克服此问题.

Now if process happens to have multiple malloc functions, only a single one is picked and will be used by all modules. Mechanism here is very simple - as shared libraries do not know location of every referenced function, they will usually call them though some table (GOT, PLT) that will be filled with required addresses lazily on the first call or at load time. The same rule is applied to the module that provides original function - even internally this function will be called though the same table, making it possible to override that function even in the original module that provides it (which is the source of many ineffeciencies related to usage of shared libraries on Linux, search for -fno-semantic-interposition, -fno-plt to overcome this).

此处的一般规则是，引入符号的第一个模块将是提供符号的模块.因此，原始流程可执行文件在此处具有最高优先级，如果它定义了malloc函数，则该malloc函数将在流程中的任何地方使用.功能calloc，realloc，free和其他功能也相同.使用LD_PRELOAD之类的技巧，可以覆盖应用程序的默认内存分配器".由于存在一些极端情况，因此不能保证这能正常工作.在执行此操作之前，您应该查阅图书馆的文档.

The general rule here is that the first module to introduce symbol will be the one providing it. Therefore original process executable has the highest priority here and if it defines malloc function, that malloc function will be used everywhere in the process. The same applies to functions calloc, realloc, free and others. Using this trick and tricks like LD_PRELOAD allow you to override "default memory allocator" of your application. This is not guaranteed to work thought as there are some corner cases. You should consult documentation for your library before doing this.

我要特别指出，这意味着在所有模块共享的进程中只有一个堆，这是有充分理由的.类似于Unix的系统通常提供两种在进程中分配内存的方式:

I want to specifically note that this means there is a single heap in the process shared by all modules and there is a good reason for that. Unix-like systems usually provide two ways of allocating memory in a process:

1. brk，sbrk系统调用
2. mmap syscall

1. brk, sbrk syscalls
2. mmap syscall

第一个可让您访问通常在可执行映像之后直接分配的单个进程内存区域.由于只有一个这样的区域，因此这种内存分配方式只能由进程中的单个分配器使用(并且您的C库通常已经使用了它.)

The first one provides you an access to a single per-process memory region usually allocated directly after the executable image. Because of the fact that there is only one such region, this way of memory allocation can only be used by a single allocator in a process (and it is usually already used by your C library).

在将任何自定义内存分配器放入进程之前，必须了解这一点-它不应该使用brk，sbrk，或者应该覆盖C库的现有分配器.

This is important to understand before you throw any custom memory allocator into your process - it either should not use brk, sbrk, or should override existing allocator of your C library.

第二个可用于直接从底层内核请求内存块.由于内核知道进程虚拟内存的结构，因此它能够分配内存页面，而不会干扰任何用户空间分配器.这也是在进程中具有多个完全独立的内存分配器(堆)的唯一方法.

The second one can be used to request chunk of memory directly from the underlying kernel. As kernel know the structure of your process virtual memory, it is able to allocate pages of memory without interfering with any user-space allocator. This is also the only way to have multiple fully independent memory allocators (heaps) in the process.

Windows不像类似Unix的系统那样依赖C运行时.相反，它提供了自己的运行时-Windows API.

Windows does not rely on C runtime the same way Unix-like systems do. Instead it provides its own runtime - Windows API.

使用Windows API分配内存的方式有两种:

There are two ways of allocating memory with Windows API:

1. 使用VirtualAlloc，MapViewOfFile之类的功能.
2. 和堆分配功能-HeapCreate，HeapAlloc.

1. Using functions like VirtualAlloc, MapViewOfFile.
2. And heap allocation functions - HeapCreate, HeapAlloc.

第一个等效于mmap，而第二个等效于malloc的更高级版本，该内部基于(我相信)基于VirtualAlloc.

The first one is an equivalent to mmap, while the second one is a more advanced version of malloc which is based internally (as I believe) on VirtualAlloc.

现在，由于Windows与C语言的关系不如Unix之类，因此它无法为您提供malloc和free函数.而是由在Windows API之上实现的C运行时库提供的.

Now because Windows does not have the same relation to C language as Unix-likes have, it does not provide you with malloc and free functions. Instead, those are provided by C runtime library which is implemented on top of Windows API.

关于Windows的另一件事-它没有每个进程符号名称空间单一的概念，例如您不能像在类似Unix的系统上那样重写函数.这样，您可以在同一进程中同时存在多个C运行时，并且这些运行时中的每一个都可以提供其malloc，free等的独立实现，每个运行在单独的堆上.

Another thing about Windows - it does not have a concept of single per process symbol namespace e.g. you cannot override function here the same way you do on Unix-like systems. This allows you to have multiple C runtimes co-existing in the same process, and every of those runtimes can provide its independent implementation of malloc, free etc, each operating on a separate heap.

因此，在Windows上，所有库都将共享一个进程特定于Windows API的堆(可通过GetProcessHeap获得)，同时它们将共享该进程中C个运行时之一的堆.

Therefore on Windows all libraries will share a single process Windows API-specific heap (can be obtained through GetProcessHeap), at the same time they will share heap of one of C runtimes in the process.

这取决于.您需要了解您要实现的目标.

It depends. You need understand what you are trying to achieve.

您是否需要替换进程中每个人使用的内存分配器，例如默认分配器?仅在类似Unix的系统上才有可能.

Do you need to replace memory allocator used by everyone in your process e.g. the default allocator? This is only possible on Unix-like system.

这里唯一可移植的解决方法是显式使用您的特定分配器接口.这样做并不重要，您只需要确保Windows上的所有库共享同一堆即可.

The only portable solition here is to use your specific allocator interface explicitly. It doesn't really matter how you do this, you just need to make sure the same heap is shared by all libraries on Windows.

这里的一般规则是，要么所有内容都应该静态链接，要么所有内容都应该动态链接.两者之间的某种混合可能真的很复杂，并且需要您将整个体系结构保持在头脑中，以避免在程序中混合堆或其他数据结构(如果您没有很多模块，这不是一个大问题) .如果需要混合使用静态链接和动态链接，则应将分配器库构建为共享库，以使其在进程中具有单个实现更加容易.

The general rule here is that either everything should be statically linked or everything should be dynamically linked. Having some sort of mix between the two might be really complicated and requires you to keep the whole architecture in your head to avoid mixing heaps or other data structures in your program (which is not a big problem if you don't have many modules). If you need to mix static and dynamic linking, you should build you allocator library as a shared library to make it easier having single implementation of it in a process.

类似Unix和Windows之间的另一个区别是Windows没有静态链接的可执行文件"的概念.在Windows上，每个可执行文件都依赖于Windows特定的动态库，例如ntdll.dll.对于ELF，可执行文件具有静态链接"和动态链接"可执行文件不同的类型.

Another difference between Unix-alikes and Windows is that Windows does not have a concept of "statically linked executable". On Windows every executable has dependencies on Windows-specific dynamic libraries like ntdll.dll. While with ELF executables have separate types for "statically linked" and "dynamically linked" executables.

这主要是由于每个进程的单个符号名称空间导致在Unix-like上混合共享和静态链接很危险，但允许Windows很好地混合静态和动态链接(几乎，不是真的).​​

This is mostly due to single per-process symbol namespace which makes it dangerous to mix shared and static linking on Unix-alikes, but allows Windows to mix static and dynamic linking just fine (almost, not really).

如果您使用一个库，则应确保使用动态链接的可执行文件动态链接它.想象一下，如果您将分配器静态链接到共享库，但是进程中的另一个库也使用相同的库-您可能是偶然使用了另一个分配器，而不是您所期望的.

If you use one of your libraries, you should make sure you link it dynamically with dynamically linked executables. Imagine if you link your allocator statically into your shared library, but another library in your process uses the same library too - you might be using another allocator by accident, not the one you were expecting.

这篇关于跨共享/静态库集成C ++自定义内存分配器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！