分布式python编程 [英] distributed python programming

问题描述

我正在尝试将python程序的执行分为两个不同的机器。我想知道是否有一种方法可以在一台机器上从另一台机器上调用python解释器。不要在另一台机器上运行脚本，而是将执行任务分成两台机器。

I am trying to split the execution of a python program to two different machines. I am wondering if there's a way to call the python interpreter on one machine from another. Not running a script on another machine, but rather split the task of execution to two machines.

在接下来的几个月中，我将自学分布式编程，我认为这将是一个很好的开始。

Over the course of the next couple of months, I will be teaching my self distributed programming, and I thought this would be a good way to start.

我认为第一步是使用一台计算机调用另一台计算机并将其发送给另一台计算机。该程序。然后，下一步将是两台机器一起执行相同的程序并进行通信以避免出现问题。第三步将是三台机器，等等。

I think the first step is to use one machine to call another machine and send it a piece of the program. Then the next step would be that both machines execute the same program together and communicate to avoid problems. The third step would be three machines, etc.

建议，技巧和想法都受到欢迎...

Advice, tips, and thoughts are all welcome...

推荐答案

Disclamer：我是SCOOP的开发人员。

Disclamer: I am a developer of SCOOP.

您可能想了解分布式处理的基于数据的技术将是MPI标准（对于多计算机，使用mpi4py [首选]或pympi）和允许远程计算的标准多处理模块（从我的角度来看，这很尴尬）。

Data-based technologies you may want to get acquainted with for distributed processing would be the MPI standard (for multi-computers, using mpi4py [prefered] or pympi) and the standard multiprocessing module allowing remote computation (but awkward, from my point of view).

不过，您应该从基于任务的框架开始。它们提供了一种简单且用户友好的用法。创建 SCOOP 时，这两项都是最大的重点。您可以使用 pip -U瓢尝试一下。在Windows上，您可能希望先使用其可执行安装程序安装PyZMQ。您可以检查提供的示例并使用各种参数来了解是什么原因导致性能降低或轻松提高。我鼓励您将其与类似的替代方案进行比较，例如 Celery 。

You should begin with task-based frameworks, though. They provides a simple and user-friendly usage. Both of these were an utmost focus while creating SCOOP. You can try it with pip -U scoop. On Windows, you may wish to install PyZMQ first using their executable installers. You can check the provided examples and play with the various parameters to understand what causes performance degradation or increase with ease. I encourage you to compare it to its alternatives such as Celery for similar work.

这两个框架都允许远程启动Python程序。更重要的是，它只为您提供任务的并行处理。

Both of these frameworks allow remote launching of Python programs. More importantly, it does the parallel processing for you while you only need to feed them with your tasks.

您可能需要检查 Fabric 提供了一种简便的方法来设置远程环境，甚至远程控制或启动脚本。

You may want to check Fabric for an easy way to setup your remote environments or even control or launch scripts remotely.

这篇关于分布式python编程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！