如何在多个进程之间共享缓存? [英] How to share a cache between multiple processes?

问题描述

我正在使用 LRU缓存来加快某些繁重的处理速度.它运作良好，并大大加快了工作速度.但是...

I'm using a LRU cache to speed up some rather heavy duty processing. It works well and speeds things up considerably. However...

当我进行多进程处理时，每个进程都会创建自己的单独缓存，并且同一事物有8个副本.直到盒子用完了内存，结果发生了坏事，这似乎才不是问题.

When I multiprocess, each process creates it's own separate cache and there are 8 copies of the same thing. That doesn't appear to be a problem, until the box runs out of memory and bad things happen as a result...

理想情况下，对于该应用程序，我只需要约300个项目的缓存大小，并且1 * 300可以容纳我必须使用的7GB，但是8 * 300则不合适.

Ideally I only need a cachesize of around 300 items for the application, and 1*300 will fit in the 7GB i have to work with, but the 8*300 just doesn't fit.

如何使所有进程共享同一缓存?

How do I get all the processes to share the same cache?

推荐答案

我相信您可以使用Manager在进程之间共享命令.从理论上讲，这应该使您对所有功能使用相同的缓存.

I believe you can use a Manager to share a dict between processes. That should in theory let you use the same cache for all functions.

但是，我认为更合理的逻辑是让一个进程通过在缓存中查找查询来响应查询，如果不存在，则将工作委派给子进程，并且返回结果之前将其缓存.您可以轻松地做到这一点

However, I think a saner logic would be to have one process that responds to queries by looking them up in the cache, and if they are not present then delegating the work to a subprocess, and caching the result before returning it. You could easily do that with

with concurrent.futures.ProcessPoolExecutor() as e:
    @functools.lru_cache
    def work(*args, **kwargs):
        return e.submit(slow_work, *args, **kwargs)

请注意，work将返回Future对象，使用者必须等待. lru_cache将缓存将来的对象，因此它们将被自动返回；我相信您可以多次访问他们的数据，但现在无法对其进行测试.

Note that work will return Future objects, which the consumer will have to wait on. The lru_cache will cache the future objects so they will be returned automatically; I believe you can access their data more than once but can't test it right now.

如果您不使用Python 3，则必须安装concurrent.futures和functools.lru_cache的向后移植版本.

If you're not using Python 3, you'll have to install backported versions of concurrent.futures and functools.lru_cache.

这篇关于如何在多个进程之间共享缓存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！