缓存302MB对象 [英] Caching a 302MB object
问题描述
我有一个302MB的对象（实际上是一个对象数组）。当我尝试使用memcached对其进行缓存时，无论我给memcached分配了多少内存，它都无法工作，这显然是因为memcached对其可缓存的对象有1MB的限制。 （最后一部分可能是错误的。我找不到很好的文档。）
I have an object (actually, an array of objects) that's 302MB. When I try to cache it with memcached it doesn't work, no matter how much memory I give memcached, apparently because memcached has a 1MB limit on objects that it can cache. (I might be wrong about that last part. I haven't been able to find great documentation.)
关于如何缓存此内容的任何建议？我在Linux上使用PHP / symfony。
Any suggestions on how to cache this thing? I'm using PHP/symfony on Linux.
推荐答案
15.5.5.4：可以存储在内存缓存中的对象的最大大小是多少，并且可以配置吗？
默认最大对象大小为1MB。在memcached 1.4.2和更高版本中，您可以使用-I命令行选项更改对象的最大大小。
The default maximum object size is 1MB. In memcached 1.4.2 and later you can change the maximum size of an object using the -I command line option.
对于此之前的版本，要增大此大小，您可以必须重新编译memcached。您可以在源代码中的slabs.c文件中修改POWER_BLOCK的值。
For versions before this, to increase this size, you have to re-compile memcached. You can modify the value of the POWER_BLOCK within the slabs.c file within the source.
在memcached 1.4.2及更高版本中，您可以使用以下命令配置支持的最大对象大小-I命令行选项。例如，要将最大对象大小增加到5MB：
In memcached 1.4.2 and higher you can configure the maximum supported object size by using the -I command-line option. For example, to increase the maximum object size to 5MB:
$ memcached -I 5m
但是，即使增加内存，这也不是IMO的好选择。更好的主意是将对象分解成较小的碎片，然后缓存其中的独立部分。
However, even when increasing the memory, this is hardly a good choice IMO. A better idea would be to break the object apart into smaller pieces and then cache indididual parts of it.
简短的回答：由于内存分配器算法的工作原理。
Short answer: Because of how the memory allocator's algorithm works.
长答案：Memcached的内存存储引擎（可插拔/将来进行调整...），使用平板方法进行内存管理。内存被分解为大小不同的平板块，从最小数量开始，然后以阶乘递增，直到最大可能值。
Long answer: Memcached's memory storage engine (which will be pluggable/adjusted in the future...), uses a slabs approach to memory management. Memory is broken up into slabs chunks of varying sizes, starting at a minimum number and ascending by a factorial up to the largest possible value.
说最小值是400字节，最大值为1 MB，阶乘为1.20：
Say the minimum value is 400 bytes, and the maximum value is 1 megabyte, and the factorial is 1.20:
slab 1-400字节slab 2-480字节slab 3-576字节...等
slab 1 - 400 bytes slab 2 - 480 bytes slab 3 - 576 bytes ... etc.
板坯越大，与前一个板坯之间的缝隙就越大。因此，最大值越大，内存存储的效率越低。 Memcached还必须为存在的每个平板预分配一些内存，因此设置较小的阶乘和较大的最大值将需要更多的开销。
The larger the slab, the more of a gap there is between it and the previous slab. So the larger the maximum value the less efficient the memory storage is. Memcached also has to pre-allocate some memory for every slab that exists, so setting a smaller factorial with a larger max value will require even more overhead.
为什么您不想这样做的原因...如果我们在谈论网页，而您试图存储/加载那么大的值，则可能是在做错事。以这种大小，将需要大量时间来将数据结构加载和解压缩到内存中，并且您的站点可能无法很好地运行。
There're other reason why you wouldn't want to do that... If we're talking about a web page and you're attempting to store/load values that large, you're probably doing something wrong. At that size it'll take a noticeable amount of time to load and unpack the data structure into memory, and your site will likely not perform very well.
如果您真的如果确实要存储大于1MB的项目，则可以使用已编辑的slabs.c：POWER_BLOCK值重新编译memcached，或使用效率低下的malloc / free后端。其他建议包括数据库，MogileFS等。
If you really do want to store items larger than 1MB, you can recompile memcached with an edited slabs.c:POWER_BLOCK value, or use the inefficient malloc/free backend. Other suggestions include a database, MogileFS, etc.
这篇关于缓存302MB对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
