持久对象 [英] Persistent objects
问题描述
我已经有了这个反复出现的半生不长的渴望，我已经认为我会张贴它，即使我没有任何具体的
提案和整个想法充满了危险。
基本上我希望有一种方法可以拥有持久的内存中对象
in a Python应用程序，可能是一个多进程的应用程序。所以你可以有一个
持久字典d，如果你说
d [x] = Frob（foo = 9，bar = 23）
，创建一个Frob实例并将其存储在d [x]中。然后，如果你退出应用程序然后重新启动它，那么就有办法将d /
带回进程并让那个Frob实例在那里。
请不要建议使用泡菜或搁架;我已经了解了那些
。我正在追求更高性能的东西。基本上d将
存在于一个内存区域，可以对磁盘文件进行mmap'以及与其他进程共享的
。一个d植根于该区域，
在其中创建的任何条目也将在该区域中，并且分配给条目的任何
对象也将移动到该区域。
使用
信号量可能必须有办法锁定区域进行更新。普通的下标分配会自动锁定，
但有时你可能需要更新
a单笔交易中的几个结构。
这样的事情可以在忙碌的
服务器应用程序中节省大量SQL流量。您在网站上看到了各种各样的虚假限制
网站，每个html页面上看不到10个以上的项目，或者
无论如何，因为他们没有'不想加载一个页面导致太多
数据库命中。使用内存中的方法，所有数据都可以在这个过程中直接存在，不需要TCP消息，也不需要上下文
需要切换，只需普通的内存字典引用。很多
的机器现在拥有多GB的物理内存，这足以支持除了最大的网站之外的所有内容。例如，像
Slashdot这样的网站每天可能会获得100,000次登录和10,000条消息
的帖子。每次登录1k字节（方式太多）和10k字节
每封邮件（也太多），那仍然只有200兆字节
一整天的活动。如今，即使是低端笔记本电脑也需要更多的ram，而且多GB的工作站也没什么大不了的。
更多。偶尔有人可能会看一个可能会导致一些磁盘流量的几天的线程和
，但即使这样也可以留在
内存中（分页系统可以处理）它）。
另一方面，要么必须要翻译才能将持久对象与非持久性对象分开，或者
让一切都持久，然后有一些方法来保持进程
共享内存互相踩踏。也许PyPy中的抽象
机器可以让这很简单。
嗯，正如你所看到的，这个想法留下了很多细节尚未
想出来了。但它足够诱人，以为我会问，是否
其他人在这里看到了什么。
I''ve had this recurring half-baked desire for long enough that I
thought I''d post about it, even though I don''t have any concrete
proposals and the whole idea is fraught with hazards.
Basically I wish there was a way to have persistent in-memory objects
in a Python app, maybe a multi-process one. So you could have a
persistent dictionary d, and if you say
d[x] = Frob(foo=9, bar=23)
that creates a Frob instance and stores it in d[x]. Then if you
exit the app and restart it later, there''d be a way to bring d back
into the process and have that Frob instance be there.
Please don''t suggest using a pickle or shelve; I know about those
already. I''m after something higher-performance. Basically d would
live in a region of memory that could be mmap''d to a disk file as well
as shared with other processes. One d was rooted into that region,
any entries created in it would also be in that region, and any
objects assigned to the entries would also get moved to that region.
There''d probably have to be a way to lock the region for update, using
semaphores. Ordinary subscript assignments would lock automatically,
but there might be times when you want to update several structures in
a single transaction.
A thing like this could save a heck of a lot of SQL traffic in a busy
server app. There are all kinds of bogus limitations you see on web
sites, where you can''t see more than 10 items per html page or
whatever, because they didn''t want loading a page to cause too many
database hits. With the in-memory approach, all that data could be
right there in the process, no TCP messages needed and no context
switches needed, just ordinary in-memory dictionary references. Lots
of machines now have multi-GB of physical memory which is enough to
hold all the stuff from all but the largest sites. A site like
Slashdot, for example, might get 100,000 logins and 10,000 message
posts per day. At a 1k bytes per login (way too much) and 10k bytes
per message post (also way too much), that''s still just 200 megabytes
for a full day of activity. Even a low-end laptop these days comes
with more ram than that, and multi-GB workstations are no big deal any
more. Occasionally someone might look at a several-day-old thread and
that might cause some disk traffic, but even that can be left in
memory (the paging system can handle it).
On the other hand, there''d either have to be interpreter hair to
separate the persistent objects from the non-persistent ones, or else
make everything persistent and then have some way to keep processes
sharing memory from stepping on each other. Maybe the abstraction
machinery in PyPy can make this easy.
Well, as you can see, this idea leaves a lot of details not yet
thought out. But it''s alluring enough that I thought I''d ask if
anyone else sees something to pursue here.
推荐答案
Paul Rubin写道：
Paul Rubin wrote:
基本上我希望有一种方法可以在Python应用程序中拥有持久的内存中对象，也许是一个多进程的对象。所以你可以有一个
持久字典d，如果你说
d [x] = Frob（foo = 9，bar = 23）
创建一个Frob实例并将其存储在d中[X]。然后，如果您退出应用程序并稍后重新启动它，那么就有办法将d /
带回到流程中并让Frob实例在那里。
Basically I wish there was a way to have persistent in-memory objects
in a Python app, maybe a multi-process one. So you could have a
persistent dictionary d, and if you say
d[x] = Frob(foo=9, bar=23)
that creates a Frob instance and stores it in d[x]. Then if you
exit the app and restart it later, there''d be a way to bring d back
into the process and have that Frob instance be there.
您是否考虑过使用Zope的独立ZODB？
-
hilsen /问候Max M，丹麦
http://www.mxm.dk/
IT''s Mad Science
Have you considdered using the standalone ZODB from Zope?
--
hilsen/regards Max M, Denmark
http://www.mxm.dk/
IT''s Mad Science
Max M< ma ** @ mxm.dk>写道：
Max M <ma**@mxm.dk> writes:
基本上我希望有一种方法可以在Python应用程序中拥有持久的内存中对象，也许是一个多进程对象。所以你可以有一个
持久字典d，如果你说d [x] = Frob（foo = 9，bar = 23）
创建一个Frob实例并将其存储在d [x]中。然后，如果您退出应用程序并稍后重新启动它，那么就有办法将d /
带回到流程中并让Frob实例在那里。
Basically I wish there was a way to have persistent in-memory objects
in a Python app, maybe a multi-process one. So you could have a
persistent dictionary d, and if you say d[x] = Frob(foo=9, bar=23)
that creates a Frob instance and stores it in d[x]. Then if you
exit the app and restart it later, there''d be a way to bring d back
into the process and have that Frob instance be there.
<你有没有考虑过使用Zope的独立ZODB？
Have you considdered using the standalone ZODB from Zope?
不，我听说它很慢，并且工作方式有点像搁置
。我错了吗？我希望对象永远不会留下内存除了
通过mmap。
No. I''ve heard that it''s quite slow, and works sort of the way shelve
does. Am I mistaken? I want the objects to never leave memory except
through mmap.
Paul Rubin写道：
Paul Rubin wrote:
嗯，正如你所看到的，这个想法留下了许多尚未考虑的细节。但它足够诱人，以为我会问，是否有其他人在这里看到了什么。
Well, as you can see, this idea leaves a lot of details not yet
thought out. But it''s alluring enough that I thought I''d ask if
anyone else sees something to pursue here.
你看过ZODB和ZEO？它完成了你要求的大部分内容，虽然
不一定按照你建议的方式。
它不会试图将所有内容都保存在内存中，但只要大部分
你的对象都是缓存命中，这就不重要了。它也不使用共享
内存：使用ZEO你可以有一个客户端服务器方法，所以你不会只限于一台机器。
而不是锁定方案，而不是每个线程在一个事务中工作，并且只有在提交事务时才会确定
你是否接受了你的更改
或拒绝。如果它们被拒绝，那么你只需再试一次。
只要你的大多数访问都是只读的，并且事务很快就会被提交给b $ b这个方案可以比锁定更好。
Have you looked at ZODB and ZEO? It does most of what you ask for, although
not necessarily in the way you suggest.
It doesn''t attempt to hold everything in memory, but so long as most of
your objects are cache hits this shouldn''t matter. Nor does it use shared
memory: using ZEO you can have a client server approach so you aren''t
restricted to a single machine.
Instead of a locking scheme each thread works within a transaction, and
only when the transaction is committed do you find out whether your changes
are accepted or rejected. If they are rejected then you simply try again.
So long as most of your accesses are read-only, and transactions are
committed quickly this scheme can work better than locking.
这篇关于持久对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
