Python的 - 处理多安装一个数组 [英] Python - Multi processing to mount an array

问题描述

我米使用的GridData以摩的阵列形状的巨大数量和
我想知道如果我能以加速这一进程计算每个我的4个核心职能（各部分）？

 进口numpy的大小= 8。
Y =（人气指数（2000））
X =（人气指数（2000））
（XX，YY）= meshgrid（X，Y）阵=零（（Y.shape [0]，X.shape [0]，大小））阵[：，：，0] = 0
数组[：，：，1] = X + Y
数组[：，：，2] = X ** 2 + Y ** 2 + X + Y
数组[：，：3] = X ** 3 + Y ** 3 + X ** 2 + Y ** 2 + X + Y数组[：，：4] = X ** 4 + Y ** 4 + X ** 3 + Y ** 3 + X ** 2 + Y ** 2 + X + Y
数组[：，：5] = X ** 5 + Y ** 5 + X ** 4 + Y ** 4 + X ** 3 + Y ** 3 + X ** 2 + Y ** 2 + X + Y
数组[：，：6] = X ** 6 + Y ** 6 + X ** 5 + Y ** 5 + X ** 4 + Y ** 4 + X ** 3 + Y ** 3 + X ** 2 + Y ** 2 + X + Y
数组[：，：6] = X ** 7 + Y ** 7 + X ** 6 + Y ** 6 + X ** 5 + Y ** 5 + X ** 4 + Y ** 4 + X ** 3 + Y ** 3 + X ** 2 + Y ** 2 + X + Y
 

所以在这里我想计算数组[：，：0]放大器;数组[：，：，1]第一核心，然后阵列[：，：，2]安培;数组[：，：3]与第二个核心...

----编辑LATER ---

有不同的片的...我的不同功能独立

没有链接

 数组[：，：0] = 0
数组[：，：，1] = X + Y
数组[：，：，2] = X * np.cos（X）+ Y * np.sin（Y）
数组[：，：3] = X ** 3 + np.sin（X）+ X ** 2 + Y ** 2 + np.sin（Y）
...
 

解决方案

如果您想通过数据数组应用功能单一，然后使用例如一个 multiprocessing.Pool 是一个很好的解决方案，的提供的，无论是输入和计算输出都比较小。

您想要做很多不同的计算以两个输入数组，这会导致数组被返回的这些计算中的每一个。

自独立的进程不共享存储器，X和Y阵列具有当它被开始被输送到每个工作进程。和各计算的结果（这也是一个numpy的数组大小相同的X和Y）必须返回给父进程

根据如该数组的大小和内核从通过进程间通信（IPC）的工作进程与父进程之间的所有这些阵列的传输量，的开销的将耗费时间，的减少的使用多个芯的优点。

请记住，父进程监听并处理的所有的工作进程IPC请求。所以，你已经从移动瓶颈的计算的到的通信的

因此​​，这不是一个给定的多处理实际上会提高在这种情况下的性能。这取决于实际问题（核的数量，阵列的大小，物理存储器等等的量）的细节。

您将不得不做使用例如一些细致的性能测试泳池或过程与现实的数组的大小。

I m using griddata to "mount" array with a great number of shapes and i would like to know if i can calculate functions (on each slice) on each my 4 cores in order to accelerate the process?

import numpy

size = 8.
Y=(arange(2000))
X=(arange(2000))
(xx,yy)=meshgrid(X,Y)

array=zeros((Y.shape[0],X.shape[0],size))

array[:,:,0] = 0
array[:,:,1] = X+Y
array[:,:,2] = X**2+Y**2+X+Y
array[:,:,3] = X**3+Y**3+X**2+Y**2+X+Y

array[:,:,4] = X**4+Y**4+X**3+Y**3+X**2+Y**2+X+Y
array[:,:,5] = X**5+Y**5+X**4+Y**4+X**3+Y**3+X**2+Y**2+X+Y
array[:,:,6] = X**6+Y**6+X**5+Y**5+X**4+Y**4+X**3+Y**3+X**2+Y**2+X+Y
array[:,:,6] = X**7+Y**7+X**6+Y**6+X**5+Y**5+X**4+Y**4+X**3+Y**3+X**2+Y**2+X+Y

So here i would like to calculate array[:,:,0] & array[:,:,1] with the first core, then array[:,:,2] & array[:,:,3] with the second core...?

----EDIT LATER---

There is no link between different "slices"...My different functions are independent

array[:,:,0] = 0
array[:,:,1] = X+Y
array[:,:,2] = X*np.cos(X)+Y*np.sin(Y)
array[:,:,3] = X**3+np.sin(X)+X**2+Y**2+np.sin(Y)
...

解决方案

If you want to apply a single function over an array of data, then using e.g. a multiprocessing.Pool is a good solution, provided that both the input and output of the calculation are relatively small.

You want to do many different calculations to two input arrays, which results in an array being returned for every one of those calculations.

Since separate processes do not share memory, the X and Y arrays have to be transported to each worker process when it is are started. And the result of each calculation (which is also a numpy array the same size as X and Y) has to be returned to the parent process.

Depending on e.g. the size of the arrays and the amount of cores, the overhead from the transfer of all those array between worker processes and the parent process via interprocess communication ("IPC") will cost time, reducing the advantages of using multiple cores.

Keep in mind that the parent process has to listen for and handle IPC requests from all the worker processes. So you've shifted the bottleneck from calculation to communication.

So it is not a given that multiprocessing will actually improve performance in this case. It depends on the details of the actual problem (number of cores, array size, amount of physical memory et cetera).

You will have to do some careful performance measurements using e.g. Pool or Process with realistic array sizes.

这篇关于Python的 - 处理多安装一个数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！