如何在 PyTorch 中使用多处理? [英] How to use multiprocessing in PyTorch?

问题描述

我正在尝试使用具有复杂损失函数的 PyTorch.为了加速代码，希望可以使用PyTorch多处理包.

I'm trying to use PyTorch with complex loss function. In order to accelerate the code, I hope that I can use the PyTorch multiprocessing package.

第一次试验，我将 10x1 的特征放入神经网络并得到 10x4 的输出.

The first trial, I put 10x1 features into the NN and get 10x4 output.

之后，我想将 10x4 参数传递给函数来做一些计算.(以后的计算会很复杂.)

After that, I want to pass 10x4 parameters into a function to do some calculation. (The calculation will be complex in the future.)

计算后，该函数总共会返回一个 10x1 的数组.该数组将被设置为 NN_energy 并计算损失函数.

After calculating, the function will return a 10x1 array in total. This array will be set as NN_energy and calculate loss function.

另外，我也想知道有没有另一种方法可以创建一个backwardable数组来存储NN_energy数组，而不是使用

Besides, I also want to know if there is another method to create a backward-able array to store the NN_energy array, instead of using

NN_energy = net(Data_in)[0:10,0]

非常感谢.

完整代码:

import torch
import numpy as np
from torch.autograd import Variable 
from torch import multiprocessing

def func(msg,BOP):
    ans = (BOP[msg][0]+BOP[msg][1]/BOP[msg][2])*BOP[msg][3]
    return ans

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden_1, n_hidden_2, n_output):
        super(Net, self).__init__()
        self.hidden_1 = torch.nn.Linear(n_feature , n_hidden_1)  # hidden layer
        self.hidden_2 = torch.nn.Linear(n_hidden_1, n_hidden_2)  # hidden layer
        self.predict  = torch.nn.Linear(n_hidden_2, n_output  )  # output layer

    def forward(self, x):
        x = torch.tanh(self.hidden_1(x))      # activation function for hidden layer
        x = torch.tanh(self.hidden_2(x))      # activation function for hidden layer
        x = self.predict(x)                   # linear output
        return x

if __name__ == '__main__': # apply_async
    Data_in      = Variable( torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() )
    Ground_truth = Variable( torch.from_numpy( np.asarray(list(range(20,30))).reshape(10,1) ).float() )

    net = Net( n_feature=1 , n_hidden_1=15 , n_hidden_2=15 , n_output=4 )     # define the network
    optimizer = torch.optim.Rprop( net.parameters() )
    loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss 
    NN_output = net(Data_in)   
    args = range(0,10)
    pool = multiprocessing.Pool()
    return_data = pool.map( func, zip(args, NN_output) )
    pool.close()
    pool.join()

    NN_energy = net(Data_in)[0:10,0]  
    for i in range(0,10):
        NN_energy[i] = return_data[i]

    loss = torch.sqrt( loss_func( NN_energy , Ground_truth ) )     # must be (1. nn output, 2. target) 
    print(loss)

错误信息:

文件"C:ProgramDataAnaconda3libsite-packages orchmultiprocessing eductions.py",第 126 行，在 reduce_tensor 中raise RuntimeError("怯懦地拒绝序列化需要 grad 的非叶张量，"

File "C:ProgramDataAnaconda3libsite-packages orchmultiprocessing eductions.py", line 126, in reduce_tensor raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "

RuntimeError: Cowardly 拒绝序列化非叶张量requires_grad，因为 autograd 不支持交叉进程边界.如果你只是想传输数据，调用 detach() on序列化前的张量(例如，将其放入队列).

RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries. If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).

推荐答案

首先，Torch Variable API 已经弃用很久了，请不要使用它.

First of all, Torch Variable API is deprecated since a very long time, just don't use it.

接下来，torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() 在很多层面上都是错误的:list 的 >np.asarray 没有用，因为无论如何都会执行一个副本，并且 np.array 将 list 作为输入设计.然后，np.arange 可用于将范围作为 numpy 数组返回，它也可用于 Torch.接下来，为 reshape 指定两个维度既无用又容易出错，你可以简单地做 reshape((-1, 1))，或者更好的 unsqueeze(-1).这是简化的表达式 torch.arange(10, dtype=torch.float32, requires_grad=True).unsqueeze(-1).

Next, torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() is wrong at many levels: np.asarray of list is useless since a copy will be performed anyway, and np.array takes list as input by design. Then, np.arange is available to return a range as numpy array, and it is also available on Torch. Next, specifying both dimension for reshape is useless and error prone, you could simply do reshape((-1, 1)), or even better unsqueeze(-1). Here is the simplified expression torch.arange(10, dtype=torch.float32, requires_grad=True).unsqueeze(-1).

如果可以使用批处理，则使用多处理池是一种不好的做法.它将更加高效和可读.事实上，并行执行 N 个小代数运算总是更慢，而且是更大的单个代数运算，在 GPU 上甚至更多.更重要的是，多处理不支持计算梯度，因此会出现错误.然而，这部分是正确的，因为它自 1.6.0 以来就支持 cpu 上的张量.祝您好运，查看官方发布更新日志.

Using multiprocessing pool is a bad practice if using batch processing is possible. It will be both way more efficient and readable. Indeed, performing N small algebraic operations in parallel is always slower and a larger single algebraic operation, and even more on GPU. More importantly, computing the gradient is not supported by multiprocessing, hence the error that you get. Yet, this is partially true, because it is supports for tensors on cpu since 1.6.0. Have a lok, to the official release changelog.

您能否发布一个更具代表性的示例，说明 func 方法可能是什么，以确保您真的需要它?

Could you post a more representative example of what func method could be to make sure you really need it ?

注意:您正在寻找的分布式 autograd 现在在 Pytorch 中可用，作为自 1.6.0 以来在 beta 中可用的实验性功能.查看官方文档.

NB: Distributed autograd as you are looking is now available in Pytorch as an experimental feature available in beta since 1.6.0. Have a look to the official documentation.

这篇关于如何在 PyTorch 中使用多处理?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！