在基于Trio的Python应用程序中生成进程并在进程之间进行通信 [英] Spawn processes and communicate between processes in a trio based Python application

问题描述

对于在Python库 fluidimage 上的实习，我们正在研究编写它是否是一个好主意使用库 trio 的具有客户端/服务器模型的HPC并行应用程序.

For an internship on the Python library fluidimage, we are investigating if it could be a good idea to write a HPC parallel application with a client/servers model using the library trio.

对于异步编程和I/O，三重奏确实很棒！

For asynchronous programming and i/o, trio is indeed great!

然后，我想知道如何

1. 生成进程(服务器完成CPU-GPU的工作)
2. 在进程之间通信复杂的Python对象(可能包含大的numpy数组).

我在其文档中没有找到推荐的方法来使用三重奏(即使

I didn't find what was the recommended way to do this with trio in its documentation (even if the echo client/server tutorial is a good start).

在Python中产生进程并进行通信的一种显而易见的方法是使用多重处理

One obvious way for spawning processes in Python and communicate is using multiprocessing.

在HPC环境中，我认为一个好的解决方案是使用MPI( rpyc (

In the HPC context, I think one good solution would be to use MPI (http://mpi4py.readthedocs.io/en/stable/overview.html#dynamic-process-management). For reference, I also have to mention rpyc (https://rpyc.readthedocs.io/en/latest/docs/zerodeploy.html#zerodeploy).

我不知道是否可以将这些工具与三重奏一起使用，什么是正确的方法.

I don't know if one can use such tools together with trio and what would be the right way to do this.

• 在python3中的多进程之间共享python对象

在我看来， PEP 574 (请参见 https://pypi.org/project/pickle5/)也可能是此解决方案的一部分问题.

It seems to me that the PEP 574 (see https://pypi.org/project/pickle5/) could also be part of a good solution to this problem.

推荐答案

不幸的是，截至今天(2018年7月)，Trio尚不支持生成和与子流程或任何类型的MPI高包装器进行通信或其他高级进程间协调协议.

Unfortunately, as of today (July 2018), Trio doesn't yet have support for spawning and communicating with subprocesses, or any kind of high-wrappers for MPI or other high-level inter-process coordination protocols.

这绝对是我们最终要达到的目标，如果您想更详细地讨论需要实现的内容，则可以此问题概述了核心子流程支持所需的内容.但是，如果您的目标是要在几个月内为您的实习工作，说实话，您可能想考虑使用更成熟的HPC工具，例如黄昏.

This is definitely something we want to get to eventually, and if you want to talk in more detail about what would need to be implemented, then you can hop in our chat, or this issue has an overview of what's needed for core subprocess support. But if your goal is to have something working within a few months for your internship, honestly you might want to consider more mature HPC tools like dask.

这篇关于在基于Trio的Python应用程序中生成进程并在进程之间进行通信的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！