多处理模块和pyro的比较? [英] Comparison of the multiprocessing module and pyro?

问题描述

我将 pyro 用于计算群集上并行作业的基本管理.我刚刚移到一个群集，负责在每个计算节点上使用所有内核. (在以前的群集中，每个核心都是一个单独的节点.)python multiprocessing 模块似乎就像一个很好的选择.我注意到它也可以用于远程过程通信.如果有人使用这两种框架进行远程过程通信，我将不胜感激，听听他们如何相互竞争.多处理模块的明显好处是它是从2.6内置的.除此之外，我很难说出哪个更好.

I use pyro for basic management of parallel jobs on a compute cluster. I just moved to a cluster where I will be responsible for using all the cores on each compute node. (On previous clusters, each core has been a separate node.) The python multiprocessing module seems like a good fit for this. I notice it can also be used for remote-process communication. If anyone has used both frameworks for remote-process communication, I'd be grateful to hear how they stack up against each other. The obvious benefit of the multiprocessing module is that it's built-in from 2.6. Apart from that, it's hard for me to tell which is better.

推荐答案

我正在更改答案，以免造成痛苦.多重处理还不成熟，BaseManager上的文档是 INCORRECT ，如果您是一个面向对象的思想家，想在运行时动态创建共享对象，请使用PYRO或您将认真地注册它！如果您只是使用共享队列来进行功能编程，那么您就需要像所有愚蠢的示例"GOOD FOR YOU"一样预先注册.

I'm changing my answer so you avoid pain. multiprocessing is immature, the docs on BaseManager are INCORRECT, and if you're an object-oriented thinker that wants to create shared objects on the fly at run-time, USE PYRO OR YOU WILL SERIOUSLY REGRET IT! If you are just doing functional programming using a shared queue that you register up front like all the stupid examples GOOD FOR YOU.

多重处理:

• 做面向对象的远程对象感到尴尬
• 轻松的加密(authkey)
• 通过网络或仅进行进程间通信
• 没有像Pyro这样的名称服务器带来额外的麻烦(有多种方法可以解决此问题)
• 实例化管理器后就无法注册"对象！
• 如果未启动服务器，则客户端会抛出一些无效参数"异常，而不仅仅是说无法连接" WTF！?
• BaseManager文档不正确！没有启动"方法！?！
• 编辑:关于如何使用它的示例很少.

• Feels awkward doing object-oriented remote objects
• Easy breezy crypto (authkey)
• Over a network or just inter-process communication
• No nameserver extra hassle like in Pyro (there are ways to get around this)
• Can't "register" objects once the manager is instantiated!!??
• If a server isn't not started, the client throws some "Invalid argument" exception instead of just saying "Failed to connect" WTF!?
• BaseManager documentation is incorrect! There is no "start" method!?!
• Very little examples as to how to use it.

火焰兵:

• 简单的远程对象
• 仅网络通信(仅本地时回送)
• 这只是工作，它喜欢面向对象的对象共享，这让我很喜欢
• 为什么这不是标准库的一部分，而不是试图复制它并惨遭失败的多处理废话?

• Simple remote objects
• Network comms only (loopback if local only)
• This thing just WORKS, and it likes object-oriented object sharing, which makes me LIKE it
• Why isn't THIS a part of the standard library instead of that multiprocessing piece of crap that tried to copy it and failed miserably?

编辑:第一次回答这个问题时，我刚刚涉足2.6多处理.在下面显示的代码中，Texture类被注册并共享为代理，但是其中的"data"属性不是.因此，请猜测会发生什么，尽管您可能会期望，但每个进程在Texture代理内部都有一个单独的"data"属性副本.我花了无数的时间试图弄清楚如何在运行时创建共享对象的良好模式，而且我一直都在碰壁.这一直令人感到困惑和沮丧.也许只有我一个人，但是环顾四周人们尝试过的例子并不多.

The first time I answered this I had just dived into 2.6 multiprocessing. In the code I show below, the Texture class is registered and shared as a proxy, however the "data" attribute inside of it is NOT. So guess what happens, each process has a separate copy of the "data" attribute inside of the Texture proxy, despite what you might expect. I just spent untold amount of hours trying to figure out how a good pattern to create shared objects during run-time and I kept running in to brick walls. It has been quite confusing and frustrating. Maybe it's just me, but looking around at the scant examples people have attempted it doesn't look like it.

我不得不做出一个痛苦的决定，那就是删除多处理库，并选择Pyro，直到多处理变得更加成熟为止.起初，我很高兴学习内置在python中的多处理程序，但现在我对此感到非常厌恶，宁愿安装Pyro软件包很多次，但高兴的是它为python提供了一个漂亮的库.

I'm having to make the painful decision of dropping multiprocessing library and preferring Pyro until multiprocessing is more mature. While initially I was excited to learn multiprocessing being built into python, I am now thoroughly disgusted with it and would rather install the Pyro package many many times with glee that such a beautiful library exists for python.

我在过去的项目中一直使用Pyro，对此感到非常满意.我还开始使用2.6中的新增多处理功能.

I have used Pyro in past projects and have been very happy with it. I have also started to work with multiprocessing new in 2.6.

通过多处理，我发现允许根据需要创建共享对象有点尴尬.似乎在年轻时，多处理模块更适合于函数式编程，而不是面向对象的.但是，这并非完全正确，因为有可能做到，我只是受到"register"调用的限制.

With multiprocessing I found it a bit awkward to allow shared objects to be created as needed. It seems like, in its youth, the multiprocessing module has been more geared for functional programming as opposed to object-oriented. However this is not entirely true because it is possible to do, I'm just feeling constrained by the "register" calls.

例如:

manager.py:

manager.py:

from multiprocessing import Process
from multiprocessing.managers import BaseManager

class Texture(object):
   def __init__(self, data):
        self.data = data

   def setData(self, data):
      print "Calling set data %s" % (data)
      self.data = data

   def getData(self):
      return self.data

class TextureManager(BaseManager):
   def __init__(self, address=None, authkey=''):
      BaseManager.__init__(self, address, authkey)
      self.textures = {}

   def addTexture(self, name, texture):
      self.textures[name] = texture

   def hasTexture(self, name):
      return name in self.textures

server.py:

server.py:

from multiprocessing import Process
from multiprocessing.managers import BaseManager
from manager import Texture, TextureManager

manager = TextureManager(address=('', 50000), authkey='hello')

def getTexture(name):
   if manager.hasTexture(name):
      return manager.textures[name]
   else:
      texture = Texture([0]*100)
      manager.addTexture(name, texture)
      manager.register(name, lambda: texture)

TextureManager.register("getTexture", getTexture)


if __name__ == "__main__":
   server = manager.get_server()
   server.serve_forever()

client.py:

client.py:

from multiprocessing import Process
from multiprocessing.managers import BaseManager
from manager import Texture, TextureManager

if __name__ == "__main__":
   manager = TextureManager(address=('127.0.0.1', 50000), authkey='hello')
   manager.connect()
   TextureManager.register("getTexture")
   texture = manager.getTexture("texture2")
   data = [2] * 100
   texture.setData(data)
   print "data = %s" % (texture.getData())

我要描述的尴尬来自server.py，其中我注册了getTexture函数以从TextureManager中检索某个名称的函数.当我要解决此问题时，如果将TextureManager设为创建/检索可共享纹理的可共享对象，则可以消除尴尬.嗯，我还在玩，但是你明白了.我不记得使用pyro遇到这种尴尬，但是可能有比上面的示例更干净的解决方案.

The awkwardness I'm describing comes from server.py where I register a getTexture function to retrieve a function of a certain name from the TextureManager. As I'm going over this the awkwardness could probably be removed if I made the TextureManager a shareable object which creates/retrieves shareable textures. Meh I'm still playing, but you get the idea. I don't remember encountering this awkwardness using pyro, but there probably is a solution that's cleaner than the example above.

这篇关于多处理模块和pyro的比较?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！