python:在多个实例上并行调用方法 [英] python: calling method over multiple instances parallelly

问题描述

我试图在多个实例并行引用同一方法的情况下，实例引用的是同一对象.

I am trying to parallel calling the same method over multiple instances, where instances are referring to the same object.

很抱歉造成这种混淆.

Sorry for this confusion statements.

具体来说，我想将以下for循环更改为并行执行:

Specifically, I want to change the following for-loop to parallel execution:

for i in range(len(instances)):#instances is a list of instances
   instances[i].do_some_computation_over_a_dataset()

有可能吗?

给以后的读者的注意事项:

以上代码不是在Python中迭代实例集合的方法.这是一种以顺序(即非并行)方式进行迭代的方法:

The above code is not the way to iterate over a collection of instances in Python. This is how to iterate in a sequential (ie non-parallel) way:

for i in instances:
    i.do_some_computation_over_a_dataset()

推荐答案

好的，让我们开始吧.首先是代码(多处理文档):

Ok, let's do it. First the code(multiprocessing docs):

In [1]: from multiprocessing import Process

In [2]: def f():
   ...:     print(1)
   ...:     for i in range(100):
   ...:         # do something
   ...:         pass
   ...:

In [3]: p1 = Process(target=f)

In [4]: p1.start()

1
In [5]: p2 = Process(target=f)

In [6]: p2.start()

1
In [7]: import time

In [8]: def f():
   ...:     for i in range(100):
   ...:         print(i)
   ...:         # do something
   ...:         time.sleep(1)
   ...:         pass
   ...:
In [9]: p1 = Process(target=f)
In [9]: p1 = Process(target=f)

In [10]: p1.start()

0
In [11]: p2 1
= Process(target=f)2
3
4
5
In [11]: p2 = Process(target=f)

In [12]: 6
p2.7
start8
In [12]: p2.start()

0
In [13]: 9

这是一个如何并行调用函数的示例.从In [10]: p1.start()您可以看到输出混乱，因为我们在运行程序p2时并行运行了程序p1.

This is an example of how a function can be called in parallel. From In [10]: p1.start() you can see the output gets jumbled because program p1 is running in parallel while we run program p2.

在Python脚本中运行程序时，您要确保脚本仅在所有程序成功执行后才结束.您可以通过

When running the program in a Python script you want to make sure script only ends when all the programs have executed successfully. You can do this by

def multi_process(instance_params, *funcs):
   process = []
   for f in funcs:
       prog = Process(target=f, args=instance_params)
       prog.start()
       process.append(prog)
   for p in process:
       p.join()

multi_process(params, f, f)

由于GIL，Python不具有C ++或Java之类的多线程支持. 在此处阅读.尽管如果您的程序能够执行更多的I/O操作，则需要占用大量CPU资源的任务，那么您可以使用多线程.为了执行CPU密集型任务，建议进行多处理.

Python doesn't have C++ or Java like multithreading support because of GIL. Read about it here. Though if your program is such that it does more I/O operations then CPU intensive tasks then you can use multithreading. For performing CPU intensive tasks multiprocessing is recommended.

@ytutow在评论中询问工人池和过程有什么区别.来自 Pymotw :

In comment @ytutow asked what is difference between pool of workers and process. From Pymotw:

Pool类可用于管理固定数量的工作器，用于 简单的情况下，可以分解要完成的工作，并且 在工人之间独立分布.

The Pool class can be used to manage a fixed number of workers for simple cases where the work to be done can be broken up and distributed between workers independently.

收集作业的返回值并作为列表返回.

The return values from the jobs are collected and returned as a list.

池参数包括进程数和要执行的函数 在开始任务过程时运行(每个孩子调用一次).

The pool arguments include the number of processes and a function to run when starting the task process (invoked once per child).

您可以将Pool用作:

You can use Pool as:

def your_instance_method(instance):
   instances.do_some_computation_over_a_dataset()

with Pool(3) as p:
    instances = [insatnce_1, instance_2, instance_3]
    print(p.map(your_instance_method, instances))

关于正确的工人数，一般建议使用2 * cpu_cores的工人数.

About the correct number of workers, it's gereral recommendation to have 2*cpu_cores number of workers.

这篇关于python:在多个实例上并行调用方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！