避免 Memcache “长度为 1000000 字节"数值限制 [英] Avoiding Memcache "1000000 bytes in length" limit on values

问题描述

我的模型有不同的实体，我想像公司的员工一样计算一次.为了避免一次又一次地进行相同的查询，计算出的列表会保存在 Memcache 中(持续时间 = 1 天).问题是该应用程序有时会给我一个错误，即 Memcache 中存储的字节数超出了允许的范围:

My model has different entities that I'd like to calculate once like the employees of a company. To avoid making the same query again and again, the calculated list is saved in Memcache (duration=1day).. The problem is that the app is sometimes giving me an error that there are more bytes being stored in Memcache than is permissible:

Values may not be more than 1000000 bytes in length; received 1071339 bytes

是否应该使用 Memcache 来存储对象列表?如果是这样，避免上述错误的最佳做法是什么?我目前正在拉 1000 个对象.您是否将值限制为 <200?在内存中检查对象的大小似乎不是一个好主意，因为它们可能在进入 Memcache 之前已被处理(序列化或类似的处理).

Is storing a list of objects something that you should be doing with Memcache? If so, what are best practices in avoiding the error above? I'm currently pulling 1000 objects. Do you limit values to < 200? Checking for an object's size in memory doesn't seem like too good an idea because they're probably being processed (serialized or something like that) before going into Memcache.

推荐答案

大卫，你没有说你使用的是哪种语言，但在 Python 中你可以像 Ibrahim 建议的使用 pickle 那样做同样的事情.您需要做的就是编写两个小辅助函数，用于将大对象读取和写入内存缓存.这是(未经测试的)草图:

David, you don't say which language you use, but in Python you can do the same thing as Ibrahim suggests using pickle. All you need to do is write two little helper functions that read and write a large object to memcache. Here's an (untested) sketch:

def store(key, value, chunksize=950000):
  serialized = pickle.dumps(value, 2)
  values = {}
  for i in xrange(0, len(serialized), chunksize):
    values['%s.%s' % (key, i//chunksize)] = serialized[i : i+chunksize]
  return memcache.set_multi(values)

def retrieve(key):
  result = memcache.get_multi(['%s.%s' % (key, i) for i in xrange(32)])
  serialized = ''.join([v for k, v in sorted(result.items()) if v is not None])
  return pickle.loads(serialized)

这篇关于避免 Memcache “长度为 1000000 字节"数值限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！