Django Memcached缓存消失 [英] Django Memcached Cache Disappears

问题描述

我有我的django应用程序配置了memcached，一切都很顺利。

我正在尝试随着时间的推移填充缓存，随着新的数据来自外部API，这是我所做的一切：

主视图

  api_query，more_results = apiQuery（** params）
 cache_key =mystring
 cache.set（cache_key，data_list，600）
 
如果more_results：
t = Thread（target ='apiMoreResultsQuery'，args =（param1，param2，param3））
 t.daemon = True 
 t.start（）
  pre> 
 
 
更多结果功能
  cache_key =mystring
 my_cache = cache.get（cache_key）
 api_query，more_results = apiQuery（** params）
 new_cache = my_cache + api_query 
 cache.set（cache_key，new_cache，600）
 
如果more_results：
 apiMoreResultsQuery（param1，param2，param3）
  
方法通过 apiMoreResultsQuery 进行几次迭代，但是在某些时候，缓存返回无导致整个循环崩溃。我已经尝试增加缓存过期，但没有改变任何东西。为什么缓存会突然消失？
 
 
 
为了澄清我在一个独特的线程中运行 apiMoreResultsQuery 因为我需要从初始调用返回一个响应更快，那么完整的数据集将被填充，所以我想保持填充在后台运行，而响应仍然可以返回。
解决方案
当您设置一个特定的缓存密钥，并且您正在设置的项目大于为缓存项目分配的大小时，它会以静默方式失败，并且您的密钥设置为无。 （我知道这是因为我被咬了）。
 
 
 
 Memcached使用 pickle 来缓存对象，所以在某些点 new_cache 正在获得 pickled ，它比为缓存项目分配的大小更大。
 
 
 
 memcached默认大小为 1MB ，您可以增加它，但是似乎有点奇怪的更大的问题是，您一次又一次使用相同的键再次，您的单个缓存项目越来越大。 
 
 
 
在缓存中设置新项目并确保这些项目足够小以缓存？不会有更好的策略？
 
 
 
无论如何，如果你想看看你的项目有多大，你可以测试它是否进入缓存，你可以执行以下一些操作：
 >>>进口泡菜
>>> some_object = [1，2，3] 
>>> len（pickle.dumps（some_object，-1））
 22 
>>> new_object = list（range（1000000））
>>> len（pickle.dumps（new_object，-1））
 4871352＃哇，那很漂亮！ 
  
请注意，如果您正在腌制Django模型实例，这可能会增长得更大，在这种情况下，建议只要从实例中挑选你想要的值。
 
 
 
有关更多阅读，请参阅这个其他答案：
 
 
 
 如何获取Google AppEngine上的python对象的大小（以字节计）？ 
 
I had my django application configured with memcached and everything was working smoothly. 

I am trying to populate the cache over time, adding to it as new data comes in from external API's. Here is the gist of what I have going on:

main view
api_query, more_results = apiQuery(**params)
cache_key = "mystring"
cache.set(cache_key, data_list, 600)

if more_results:
    t = Thread(target = 'apiMoreResultsQuery', args = (param1, param2, param3))
    t.daemon = True
    t.start()
more results function
cache_key = "mystring"
my_cache = cache.get(cache_key)
api_query, more_results = apiQuery(**params)
new_cache = my_cache + api_query
cache.set(cache_key, new_cache, 600)

if more_results:
    apiMoreResultsQuery(param1, param2, param3)
This method works for several iterations through the apiMoreResultsQuery but at some point the cache returns None causing the whole loop to crash. I've tried increasing the cache expiration but that didn't change anything. Why would the cache be vanishing all of a sudden?

For clarification I am running the apiMoreResultsQuery in a distinct thread because I need to return a response from the initial call faster then the full data-set will populate so I want to keep the populating going in the background while a response can still be returned.
解决方案
When you set a particular cache key and the item you are setting is larger than the size allotted for a cached item, it fails silently and your key gets set to None. (I know this because I have been bitten by it.)

Memcached uses pickle to cache objects, so at some point new_cache is getting pickled and it's simply larger than the size allotted for cached items.

The memcached default size is 1MB, and you can increase it, but the bigger issue that seems a bit odd is that that you are using the same key over and over again and your single cached item just gets bigger and bigger. 

Wouldn't a better strategy be to set new items in the cache and to be sure that those items are small enough to be cached?

Anyway, if you want to see how large your item is growing, so you can test whether or not it's going to go into the cache, you can do some of the following:
>>> import pickle
>>> some_object = [1, 2, 3]
>>> len(pickle.dumps(some_object, -1))
22
>>> new_object = list(range(1000000))
>>> len(pickle.dumps(new_object, -1))
4871352   # Wow, that got pretty big!
Note that this can grow a lot larger if you are pickling Django model instances, in which case it's probably recommended just to pickle the values you want from the instance.

For more reading, see this other answer:

How to get the size of a python object in bytes on Google AppEngine?

                        
这篇关于Django Memcached缓存消失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！