如何将Python对象存储在内存中以供不同进程使用? [英] How do I store a Python object in memory for use by different processes?

问题描述

在这种情况下:我有一个巨大的对象需要加载到内存中.如此之大，以至于如果将其加载两次，它将超出计算机上的可用内存(不，我无法升级内存).我也不能将它分成任何较小的部分.为了简单起见，我们仅假设对象为600 MB，而我只有1 GB的RAM.我需要使用Web应用程序中的该对象，该应用程序在多个进程中运行，并且我无法控制它们的生成方式(第三方负载均衡器可以做到这一点)，所以我不能仅仅依靠创建对象在某些主线程/进程中，然后生成子级.这也消除了使用POSH之类的东西的可能性，因为它依赖于它自己的自定义fork调用.我也不能使用SQLite内存数据库，mmap或posix_ipc，sysv_ipc和shm模块之类的东西，因为它们充当内存中的文件，并且此数据必须是我可以使用的对象.使用其中之一，我将不得不将其读取为文件，然后在每个单独的进程和BAM中将其变成一个对象，这是由于我尝试加载第二个副本而导致的超出计算机内存限制的分段错误.

Here's the situation: I have a massive object that needs to be loaded into memory. So big that if it is loaded in twice it will go beyond the available memory on my machine (and no, I can't upgrade the memory). I also can't divide it up into any smaller pieces. For simplicity's sake, let's just say the object is 600 MB and I only have 1 GB of RAM. I need to use this object from a web app, which is running in multiple processes, and I don't control how they're spawned (a third party load balancer does that), so I can't rely on just creating the object in some master thread/process and then spawning off children. This also eliminates the possibility of using something like POSH because that relies on it's own custom fork call. I also can't use something like a SQLite memory database, mmap or the posix_ipc, sysv_ipc, and shm modules because those act as a file in memory, and this data has to be an object for me to use it. Using one of those I would have to read it as a file and then turn it into an object in each individual process and BAM, segmentation fault from going over the machine's memory limit because I just tried to load in a second copy.

必须有某种方式将Python对象存储在内存中(而不是作为文件/字符串/序列化/可插入)，并且可以从任何进程访问它.我只是不知道那是什么.我到处都是StackOverflow和Google，却找不到答案，所以我希望有人可以帮助我.

There must be someway to store a Python object in memory (and not as a file/string/serialized/pickled) and have it be accessible from any process. I just don't know what it is. I've looked all over StackOverflow and Google and can't find the answer to this, so I'm hoping somebody can help me out.

推荐答案

寻找共享内存或服务器进程.重新阅读后，Server的过程听起来更接近于您想要的.

Look for shared memory, or Server process. After re-reading your post Server process sounds closer to what you want.

http://en.wikipedia.org/wiki/Shared_memory

这篇关于如何将Python对象存储在内存中以供不同进程使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！