如何屏蔽 PyTorch 权重参数中的权重? [英] How to mask weights in PyTorch weight parameters?

问题描述

我试图在 PyTorch 中屏蔽(强制为零)特定的权重值.我试图屏蔽的权重在 def __init__

I am attempting to mask (force to zero) specific weight values in PyTorch. The weights I am trying to mask are defined as so in the def __init__

class LSTM_MASK(nn.Module):
        def __init__(self, options, inp_dim):
            super(LSTM_MASK, self).__init__()
            ....
            self.wfx = nn.Linear(input_dim, curernt_output, bias=add_bias)

掩码也在def __init__中定义为

self.mask_use = torch.Tensor(curernt_output, input_dim)

掩码是一个常量，掩码参数的 .requires_grad_() 是 False.现在在类的 def forward 部分，我尝试在线性运算完成之前对权重参数和掩码进行逐元素乘法

The mask is a constant and the .requires_grad_() is False for the mask parameter. Now in the def forward part of the class I attempt to do an element-wise multiplication of the weight parameter and the mask before the linear operation is completed

def forward(self, x):
    ....
    self.wfx.weight = self.wfx.weight * self.mask_use
    wfx_out = self.wfx(x)

我收到一条错误消息:

self.wfx.weight = self.wfx.weight * self.mask_use
  File "/home/xyz/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 537, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

但是当我使用 .type() 检查这两个参数时，它们都显示为 torch.cuda.FloatTensor.我不知道为什么这里有错误.

But when I check on the both parameters with .type() both of them come up as torch.cuda.FloatTensor. I am not sure why there is an error here.

推荐答案

元素操作总是返回一个 FloatTensor.无法将正常张量分配为层的 weight.

The element-wise operation always returns a FloatTensor. It is not possible to assign normal tensors as weight of layers.

有两种可能的选择来处理它.您可以将其分配给您的权重的 data 属性，在那里可以分配正常的张量.

There are two possible options to deal with it. You can assign it to the data attribute of your weight, there it is possible assign normal tensors.

或者您将结果转换为 nn.Parameter 本身，然后您可以将其分配给 wfx.weight.

Or alternatively you convert your result to an nn.Parameter itself, then you can assign it to wfx.weight.

这是一个显示两种方式的示例:

Here is an example which shows both ways:

import torch
import torch.nn as nn

wfx = nn.Linear(10, 10)
mask_use = torch.rand(10, 10)
#wfx.weight = wfx.weight * mask_use #your example - this raises an error

# Option 1: write directly to data
wfx.weight.data = wfx.weight * mask_use

# Option 2: convert result to nn.Parameter and write to weight
wfx.weight = nn.Parameter(wfx.weight * mask_use)

免责声明:在权重上使用 =(赋值)时，您将替换参数的权重张量.这可能会对图形产生不良影响.优化步骤.

Disclaimer: When using an = (assignment) on the weights you are replacing the weights tensor of your parameter. This may have unwanted effects on the graph resp. the optimization step.

这篇关于如何屏蔽 PyTorch 权重参数中的权重?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！