CPython的静态对象地址和碎片 [英] CPython's static object address and fragmentation

问题描述

我读过

对于CPython，id（x）是存储x的内存地址。

For CPython, id(x) is the memory address where x is stored.

这是给定对象的 id 永不改变的意思，这意味着对象始终存储在给定的位置生存期内的内存地址。这就引出了一个问题：（虚拟）内存碎片怎么办？

And it's a given the id of an object never changes, which means an object always stored at a given memory address in its lifetime. This leads to the question: what about fragmentation of (virtual) memory?

说一个对象 A 在地址中1（具有 id 1），占用10个字节，因此占用地址1-10。对象 B 具有 id 11并占用字节11-12，而对象 C 占用地址13-22。一旦B超出范围并获得GC证明，我们将产生碎片。

Say an object A is at address 1 (has id 1), takes up 10 bytes, so it takes up addresses 1-10. Object B has id 11 and takes up bytes 11-12, and object C takes up addresses 13-22. Once B goes out of scope and gets GC'd, we'll have fragmentation.

这个难题如何解决？

推荐答案

CPython使用自己的方法小对象的内存分配器， pymalloc-allocator 。在代码本身。

CPython uses its own memory allocator for small objects, the pymalloc-allocator. A quite good description can be found in the code itself.

此分配器非常擅长避免内存碎片，因为它可以有效地重用释放的内存。但是，这只是一种试探法，可以提出导致内存碎片的方案。

This allocator is pretty good at avoiding memory fragmentation, because it reuses the freed memory efficiently. However, it is only a heuristic and one could come up with scenarios which lead to memory fragmentation.

让我们来看看分配一个1字节大小的对象时会发生什么。

Let's play through what happens when we allocate an object with size of 1byte.

对于小于512字节的对象，CPython拥有自己的所谓的竞技场。显然，1个字节的请求将由其分配器管理。

CPython has its own so called arena for objects smaller than 512 bytes. Obviously, 1 byte request will be managed by its allocator.

请求的大小分为64个不同的类：第0个类用于1..8字节，第1个字节-class用于大小或9..16字节，依此类推-这是由于需要对齐8个字节。以上每个类都有或多或少的独立/专用内存。我们的要求是0级。

Requested sizes are divided in 64 different classes: 0th-class is for sizes of 1..8 bytes, 1th-class is for sizes or 9..16 bytes and so on - this is due to required alignment of 8 bytes. Every of the above classes has its own more or less independent/dedicated memory. Our request is for the 0th-class.

假设这是此尺寸等级的第一个请求。将创建一个新的池或重用一个空池。池 4KB大，因此具有空间512个8字节块。尽管请求只有1个字节，但我们将阻塞所占用块的另外7个字节，因此它们不能用于其他对象。所有可用块均保存在列表中-首先，所有512个块均在该列表中。分配器从该空闲块列表中删除第一个块，并返回其地址作为指针。

Let's assume this is the first request for this size-class. A new "pool" will be created or an empty pool reused. A pool is 4KB big, and thus has space for 512 8-byte "blocks". Despite requestion only 1 byte we will be blocking another 7 bytes of the occupied block, so they cannot be used for another objects. All free blocks are kept in a list - in the beginning all 512 blocks are in this list. The allocator deletes the first block from this free-block-list and returns its address as pointer.

该池本身被标记为已使用，并添加到列表中。

The pool itself is marked as "used" and added to a list of used pools for 0th-class.

现在，分配另一个大小小于等于8个字节的对象的情况如下。首先，我们查看第0类的已用池列表，然后找到一个已在使用中的池，即具有一些已用和一些空闲块的池。分配器使用第一个空闲块，将其从空闲块列表中删除并返回其地址作为指针。

Now, allocation another object with size <=8 bytes happens as follows. At first we look at the list of used pools for 0th-class and will find a pool which already in use, i.e. has some used and some free blocks. Allocator uses the first free block, deletes it from the list of free blocks and returns its address as pointer.

删除第一个对象很容易-我们将占用的块添加为head

Deleting first object is easy - we add the occupied block as head of the list of free blocks in the (so far single) used pool.

创建8字节的新对象时，因此空闲块中的第一个块-list被使用，这是第一个现在已删除的对象所使用的块。

When a new object of 8 byte is created, so the first block in the free-block-list is used, and this is the block which was used by the first, now deleted, object.

您可以看到内存被重用，因此内存碎片大大减少了。这并不意味着不会有内存碎片：

As you can see the memory gets reused, and thus memory fragmentation is vastly reduced. This doesn't mean there cannot be memory fragmentation:

分配了512个1字节对象后，第一个池变为满，而新的池用于第0类-sizes将被创建/使用。同样，我们还要添加另外512个对象，第二个池也将变为满。等等。

After allocating 512 1-byte-objects, the first pool becomes "full" and a new pool for 0th-class-sizes will be created/used. Once also we add another 512 objects also the second pool becomes "full". And so on.

现在，如果删除了前511个元素-仍然有一个字节会阻塞整个4KB，不能用于 other 类。

Now, if the first 511 elements are deleted - there will be still one byte which blocks whole 4KB, which cannot be used for other classes.

仅当释放了最后一个块时，池才变为空，因此可以重用于其他大小级别。

Only when the last block is freed, the pool becomes "empty" and thus can be reused for other size-classes.

空池不会返回到操作系统，而是留在竞技场中并被重用。但是，pymalloc 管理多个竞技场，如果竞技场变为未使用状态，它可能被释放，并且占用的内存（即池）被返回到操作系统。

The empty pools are not returned to the OS, but stay in the arena and are reused. However, the pymalloc manages multiple arenas, and if an arena becomes "unused" it might be freed and occupied memory (i.e. pools) is returned to the OS.

这篇关于CPython的静态对象地址和碎片的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！