Django缓存大量列表 [英] Django caching a large list

问题描述

我的django应用程序处理25MB二进制文件.它们每个都有大约100,000个记录"，每个256个字节.

My django application deals with 25MB binary files. Each of them has about 100,000 "records" of 256 bytes each.

我花了大约7秒钟的时间从磁盘读取二进制文件并使用python的struct模块对其进行解码.我将数据转换成大约100,000个项目的列表，其中每个项目都是一个字典，具有各种类型(浮点数，字符串等)的值.

It takes me about 7 seconds to read the binary file from disk and decode it using python's struct module. I turn the data into a list of about 100,000 items, where each item is a dictionary with values of various types (float, string, etc.).

我的django视图需要在此列表中进行搜索.显然7秒太长了.

My django views need to search through this list. Clearly 7 seconds is too long.

我曾尝试使用django的低级缓存API来缓存整个列表，但是这是行不通的，因为任何单个缓存项的最大大小限制为1MB.我已经尝试过分别缓存100,000个列表项，但这花费了7秒钟以上的时间-大多数时间都花在了拆开这些项上.

I've tried using django's low-level caching API to cache the whole list, but that won't work because there's a maximum size limit of 1MB for any single cached item. I've tried caching the 100,000 list items individually, but that takes a lot more than 7 seconds - most of the time is spent unpickling the items.

是否有方便的方法将两次请求之间的大列表存储在内存中?您能想到另一种缓存对象供我的Django应用程序使用的方法吗?

Is there a convenient way to store a large list in memory between requests? Can you think of another way to cache the object for use by my django app?

推荐答案

将商品尺寸限制修改为10m(大于1m)，然后添加

edit the item size limit to be 10m (larger than 1m), add

-I 10m

到/etc/memcached.conf并重新启动memcached

to /etc/memcached.conf and restart memcached

还要在/usr/lib/python2.7/dist-packages/django/core/cache/backends中的memcached.py中编辑此类，如下所示:

also edit this class in memcached.py located in /usr/lib/python2.7/dist-packages/django/core/cache/backends to look like this:

class MemcachedCache(BaseMemcachedCache):
"An implementation of a cache binding using python-memcached"
def __init__(self, server, params):
    import memcache
    memcache.SERVER_MAX_VALUE_LENGTH = 1024*1024*10 #added limit to accept 10mb
    super(MemcachedCache, self).__init__(server, params,
                                         library=memcache,
                                         value_not_found_exception=ValueError)

这篇关于Django缓存大量列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！