在django memcached中存储大键的明显错误 [英] Apparent bug storing large keys in django memcached
问题描述
我一直在遵循有关设置内存缓存以支持大值的说明.
I've been following instructions on setting up memcached to support large values.
用例基本上是我想在内存中保留一个相当大的机器学习模型，并用它来响应查询.模型的大小可能在0.5到100 MB之间.
The use case is basically that I want to keep a fairly large machine-learning model in memory, and use it to respond to queries. The size of the model might vary between 0.5 and 100 MB.
这些是我的/etc/memcached.conf中的自定义键
These are the custom keys in my /etc/memcached.conf
# custom flags
-I 10M
-m 200
-vvv
以下是重启memcached后设置的键:
Here's the keys set after restarting memcached:
> sudo service memcached restart
Restarting memcached: memcached.
> echo "stats settings" | nc localhost 11211
STAT maxbytes 67108864
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 11211
STAT inter 127.0.0.1
STAT verbosity 3
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 4
STAT num_threads_per_udp 4
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT item_size_max 10485760
STAT maxconns_fast no
STAT hashpower_init 0
STAT slab_reassign no
STAT slab_automove no
END
因此似乎有好消息和坏消息:ITEMSIZEMAX已更改为大约10 MB，但maxbytes仍保持默认值67 MB.
So there appears to be good news and bad news: ITEMSIZEMAX has changed to about 10 MB, but maxbytes has remained at default 67 MB.
真正的问题是我仍然无法存储1 MB以上的密钥！
The real problem is that I still can't store keys > 1 MB!
如果我运行以下Python脚本，则在尝试存储大小约为0.4 MB的numpy数组时会失败
If I run the following Python script, it fails when trying to store a numpy array of size ~0.4 MB
from django.core.management import setup_environ
from myapp import settings
setup_environ(settings)
key = "test_key"
def store_test(val):
cache.set(key, val)
if cache.get(key) is not None:
cache.delete(key)
return True
else:
return False
from django.core.cache import cache
from numpy import arange
i = 10
while i <= 1000000:
yes = store_test(arange(i))
if yes:
print "stored", i, "successfully"
else:
print "failed to store", i
print "bytes:", arange(i).nbytes
i *= 10
当然，要运行此代码，您将需要用包含settings.py文件的有效Django模块替换"myapp".
Of course, to run this code, you will need to replace 'myapp' with a valid Django module that contains a settings.py file.
任何人都可以复制问题，帮助我调试问题或提出解决方案吗?
Can anyone replicate the problem, help me debug this, or suggest a solution?
** 编辑 **
我写了一个bash脚本来直接测试memcached而不是通过Python，奇怪的是它似乎可以工作！
I wrote a bash script to test memcached directly instead of going through Python, and oddly enough it seems to work!
for i in 10 1000 10000 100000 1000000 2000000 10000000 11000000
do
bytes=$(yes | head -n $i | tr -d '\n')
echo doing $i bytes
(echo delete key; echo set key 0 900 $i; echo $bytes; echo get key; sleep 1) | telnet localhost 11211 > output_$i
outputsize=$(stat -c%s "output_$i")
echo output size is $outputsize - should be about $(($i+110))
done
echo delete key | telnet localhost 11211
~
推荐答案
所以我以前似乎找不到它，但是显然这个问题已经得到回答.
So I couldn't seem to find it before, but apparently this question has already been answered.
问题是memcached和Django的memcached绑定都将限制设置为1 MB.我很确定这是一个错误-python绑定应该尊重memcached的实际配置.显然，新的价值必须扎根.
The problem is that both memcached and Django's memcached binding set limits of 1 MB. I'm pretty sure this is a bug - the python binding should respect memcached's actual configuration. The new value has to be hardwired in, apparently.
这篇关于在django memcached中存储大键的明显错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
