jemalloc，MMAP和共享内存？ [英] jemalloc, mmap and shared memory?

问题描述

可以 jemalloc 被修改为共享内存分配？ FreeBSD的功能 dallocx（） 意味着你可以提供一个指针用于分配，但我没有看到一个明显的方式告诉 jemalloc 来限制来自内存的所有的分配（也不设置大小，等等）。

  

的 dallocx（）函数使通过 PTR 引用的内存提出，要可用于将来的分配。

如果没有，什么是努力为这样的功能水平？我在努力寻找能够从我提供了一个共享的内存部分分配一个现成的，货架分配方案。

同样，可 jemalloc 被配置为从内存锁定区域分配给prevent交换？

随意点我需要修改和提供任何意见或建议有关code部分。

我正在探索的想法是 - 因为你可以分配在多线程环境，为 jemalloc 并最大限度地减少争创建阿里纳斯/堆，该概念似乎扩展到分配在多环境共享内存区域，即我创建一个使用的mmap（）共享内存的N个区域，我想利用的力量jemalloc （或任何分配方案），以尽可能高效地分配，以最小线程争，从这些其中一个共享区域，即如果线程/进程不能访问同一共享区域和场馆，机会争是最小和的malloc 运行速度提高。

这是不是与的malloc（） API一个全局池ALLOC不同，因为通常这些都需要一个全局锁有效序列用户空间。我想避免这种情况。

编辑2：

在理想情况下像这样的API：

  //初始化的页头背景下两个SHMEM池
CTX1 = alloc_init（shm_region1_ptr）;
ctx2 = alloc_init（shm_region2_ptr）;（......一堆code的决定池2，应使用基于一些方法
池选择的它可以最大限度地减少锁争用的可能性
与其他进程分配SHMEM缓冲区）//从POOL2分配
PTR =的malloc（ctx2，大小）
 

解决方案

是的。但是，当你问的问题，这是不正确的。

Jemalloc 4（以2015年8月发布）有几个 mallctl 的命名空间，这将是用于此目的的;它们允许你指定每个领域，应用程序特定的组块分配挂钩。尤其是，<一href=\"http://www.canonware.com/download/jemalloc/jemalloc-latest/doc/jemalloc.html#arena.i.chunk_hooks\"相对=nofollow> 舞台＆LT; I＆GT; .chunk_hooks 命名空间和的 arenas.extend mallctl 选项使用。一个集成测试存在演示如何消费这一API

至于理由，我希望这需要有效的消息的开销来理解任何特定的内存段谎言争将是类似于只是百家争鸣的开销，因为你会降解成竞争的缓存线准确地更新特定的舞台的争夺的价值。

由于jemalloc已经采用了多种技术来减少争，那么您可以使用的 opt.narenas 。这将减少争用较少的线程将被映射成一个舞台，但由于线程有效地圆robined，它，你得到的热点反正是可能的。

要解决这个问题，你可以做你争计数和热点检测，并简单地使用的 thread.arena mallctl 界面切换线程与上少争的舞台。

Can jemalloc be modified to allocate from shared memory? The FreeBSD function dallocx() implies you can provide a pointer to use for allocation, but I don't see an obvious way to tell jemalloc to restrict all allocations from that memory (nor set a size, etc).

The dallocx() function causes the memory referenced by ptr to be made available for future allocations.

If not, what is the level of effort for such a feature? I'm struggling to find an off-the-shelf allocation scheme that can allocate from a shared memory section that I provided.

Similarly, can jemalloc be configured to allocate from a locked region of memory to prevent swapping?

Feel free to point me to relevant code sections that require modification and provide any ideas or suggestions.

The idea I am exploring is — since you can create arenas/heaps for allocating in a threaded environment, as jemalloc does to minimize contention, the concept seems scalable to allocating regions of shared memory in a multiprocessing environment, i.e. I create N regions of shared memory using mmap(), and I want to leverage the power of jemalloc (or any allocation scheme) to allocate as efficiently as possible, with minimum thread contention, from those one of those shared regions, i.e. if threads/processes are not accessing the same shared regions and arenas, the chance for contention is minimal and speed of the malloc operation is increased.

This is different than a global pool alloc with malloc() API since usually these require a global lock effectively serializing the user-space. I'd like to avoid this.

edit 2:

Ideally an api like this:

// init the alloc context to two shmem pools
ctx1 = alloc_init(shm_region1_ptr);
ctx2 = alloc_init(shm_region2_ptr);

(... bunch of code determines pool 2 should be used, based on some method
of pool selection which can minimize possibility of lock contention
with other processes allocating shmem buffers)

// allocate from pool2
ptr = malloc(ctx2, size)

解决方案

Yes. But this was not true when you asked the question.

Jemalloc 4 (released in August of 2015) has a couple of mallctl namespaces that would be useful for this purpose; they allow you to specify per-arena, application-specific chunk allocation hooks. In particular, the arena.<i>.chunk_hooks namespace and the arenas.extend mallctl options are of use. An integration test exists that demonstrates how to consume this API.

Regarding the rationale, I would expect that the effective "messaging" overhead required to understand where contention on any particular memory segment lies would be similar to the overhead of just contending, since you're going to degrade into contending on a cache line to accurately update the "contention" value of a particular arena.

Since jemalloc already employs a number of techniques to reduce contention, you could get a similar behavior in a highly threaded environment by creating additional arenas with opt.narenas. This would reduce contention as fewer threads would be mapped to an arena, but since threads are effectively round-robined, it's possible you get to hot-spots anyway.

To get around this, you could do your contention counting and hotspot detection, and simply use the thread.arena mallctl interface to switch a thread onto an arena with less contention.

这篇关于jemalloc，MMAP和共享内存？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！