在进程之间共享具有文件句柄属性的对象 [英] Share objects with file handle attribute between processes
问题描述
我对进程之间具有文件句柄的共享资源有疑问. 这是我的测试代码:
I have a question about shared resource with file handle between processes. Here is my test code:
from multiprocessing import Process,Lock,freeze_support,Queue
import tempfile
#from cStringIO import StringIO
class File():
def __init__(self):
self.temp = tempfile.TemporaryFile()
#print self.temp
def read(self):
print "reading!!!"
s = "huanghao is a good boy !!"
print >> self.temp,s
self.temp.seek(0,0)
f_content = self.temp.read()
print f_content
class MyProcess(Process):
def __init__(self,queue,*args,**kwargs):
Process.__init__(self,*args,**kwargs)
self.queue = queue
def run(self):
print "ready to get the file object"
self.queue.get().read()
print "file object got"
file.read()
if __name__ == "__main__":
freeze_support()
queue = Queue()
file = File()
queue.put(file)
print "file just put"
p = MyProcess(queue)
p.start()
然后我得到一个KeyError，如下所示:
Then I get a KeyError like below:
file just put
ready to get the file object
Process MyProcess-1:
Traceback (most recent call last):
File "D:\Python26\lib\multiprocessing\process.py", line 231, in _bootstrap
self.run()
File "E:\tmp\mpt.py", line 35, in run
self.queue.get().read()
File "D:\Python26\lib\multiprocessing\queues.py", line 91, in get
res = self._recv()
File "D:\Python26\lib\tempfile.py", line 375, in __getattr__
file = self.__dict__['file']
KeyError: 'file'
我认为当我将File()对象放入队列时，该对象被序列化，并且文件句柄无法被序列化，因此，我得到了KeyError:
I think when I put the File() object into queue , the object got serialized, and file handle can not be serialized, so, i got the KeyError:
有人对此有任何想法吗?如果要共享具有文件句柄属性的对象，该怎么办?
Anyone have any idea about that? if I want to share objects with file handle attribute, what should I do?
推荐答案
我不得不反对@Mark反复断言@@@@@@@@反复断言文件处理不能只是在运行的进程之间传递" –在真正的现代操作系统中(例如，Unix(包括免费的BSD变体，MacOSX和Linux，包括在内)，这根本不是真的)–嗯，我想知道遗漏了哪些OS此列表中的...?-)-当然 sendmsg 可以做到这一点(通过使用SCM_RIGHTS标志在"Unix套接字"上).
I have to object (at length, won't just fit in a commentl;-) to @Mark's repeated assertion that file handles just can't be "passed around between running processes" -- this is simply not true in real, modern operating systems, such as, oh, say, Unix (free BSD variants, MacOSX, and Linux, included -- hmmm, I wonder what OS's are left out of this list...?-) -- sendmsg of course can do it (on a "Unix socket", by using the SCM_RIGHTS flag).
现在可怜的，有价值的multiprocessing是完全不使用此功能的完全正确的方法(即使假设也可能在Windows上实现此功能也是如此)–大多数开发人员无疑会滥用它(让多个进程访问该功能).同时打开同一文件并进入竞争状态).唯一正确的使用方法是，对于一个拥有专有权的进程，该进程有权打开某些文件，以将打开的文件句柄传递给另一个特权降低的进程，然后再也不使用该句柄.无论如何，无法在multiprocessing模块中强制执行该操作.
Now the poor, valuable multiprocessing is fully right to not exploit this feature (even assuming there might be black magic to implement it on Windows too) -- most developers would no doubt misuse it anyway (having multiple processes access the same open file concurrently and running into race conditions). The only proper way to use it is for a process which has exclusive rights to open certain files to pass the opened file handles to another process which runs with reduced privileges -- and then never use that handle itself again. No way to enforce that in the multiprocessing module, anyway.
回到@Andy的原始问题，除非他只打算在Linux上工作(并且也只与本地进程一起工作)并且愿意在/proc文件系统上耍花招，否则他将不得不定义自己的应用程序级需求更加尖锐，并相应地序列化file对象.大多数文件都有一个路径(或者可以做成一个路径:无路径文件非常少见，我相信Windows上实际上不存在)，因此可以通过它进行序列化-许多其他文件足够小，可以通过发送它们来进行序列化内容-等等，等等.
Back to @Andy's original question, unless he's going to work on Linux only (AND with local processes only, too) and willing to play dirty tricks with the /proc filesystem, he's going to have to define his application-level needs more sharply and serialize file objects accordingly. Most files have a path (or can be made to have one: path-less files are pretty rare, actually non-existent on Windows I believe) and thus can be serialized via it -- many others are small enough to serialize by sending their content over -- etc, etc.
这篇关于在进程之间共享具有文件句柄属性的对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
