Memcached:检索所有密钥时如何打破限制 [英] Memcached: How to break the limitation when retrieving all keys

问题描述

由于事实是，一旦在平板中的密钥计数> 500k，我就无法在网络上找到任何可检索所有内存缓存密钥的可靠程序，我想用PHP编写自己的程序.

Due to the fact I could not find any reliable program in the net which retrieves all memcache keys once the key count in a slab > 500k I wanted to write my own program in PHP.

然后我意识到，为什么所有程序对于如此高的键数都无法真正起作用，是有原因的.

Then I realized there is a reason why all of the programs did not really work for this high amount of keys.

当我用stats cachedump <slab-id> <limit>读取密钥时，它仅返回一定数量的密钥，在我的情况下，密钥的终止时间约为30k.

When I read the keys with stats cachedump <slab-id> <limit> it returns only a certain amount of keys which ends - in my case - by around 30k.

如何至少将这500k放在一块板中?

How can I get at least these 500k which may be in one slab?

所有建议使用memdump的帖子"和答案"均无效.他们也有这个限制.

All the "posts" and "answers" which suggest to use memdump do not work. They have this limitation as well.

我正在使用memcached 1.4.25

推荐答案

在调试memcached源代码后，我意识到限制是在items.c/item_cachedump()中，由以下行引起:

After debugging the memcached source I realized the limitation is in items.c/ item_cachedump() caused by this line:

unsigned int memlimit = 2 * 1024 * 1024;   /* 2MB max response size */

由于此分配，分配的缓冲区限制了返回的键的数量，因为所有键的总长度(包括每个键名称后的\r\n)不得超过此大小.

Due to this assignment the allocated buffer limits the amount of the returned keys because the total length of all keys [including \r\n after each key name] must not exceed this size.

我可以通过将问题更改为以下方式来解决我的问题:

I could solve for my case the problem in that way that I changed it to:

unsigned int memlimit = 128 * 1024 * 1024;

并重新编译memcached.现在，我可以读取大约70万个按键[我没有更多的按键可以最大化该功能].

and recompiling memcached. Now I can read around 700k keys [I did not have more keys to max the function].

我发现很高的值会使系统不稳定.值为500 * 1024 * 1024 [e.q. 500MB]，我的系统几乎崩溃了，并显示以下消息:

I found out that a very high value makes the system instable. With a value of 500 * 1024 * 1024 [e.q. 500MB] my system almost crashed with this messages:

shell-init:检索当前目录时出错:getcwd:无法访问 父目录:没有这样的文件或目录

shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

即使我的服务器具有5GB RAM，我也不是很清楚为什么，但是最后128MB的值仍然是可用的值.

Even my server has 5GB RAM I don't really understand why but in the end a value of 128MB is a usable value.

我要求memcached 开发人员团队添加一个可以设置命令行中的值.

I asked the memcached developer team to add a switch which can set the value in the command line.

这篇关于Memcached:检索所有密钥时如何打破限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！