如何在PyTorch权重参数中屏蔽权重? [英] How to mask weights in PyTorch weight parameters?

问题描述

我正在尝试在PyTorch中屏蔽(强制为零)特定的重量值.我尝试掩盖的权重在 def __init __

I am attempting to mask (force to zero) specific weight values in PyTorch. The weights I am trying to mask are defined as so in the def __init__

class LSTM_MASK(nn.Module):
        def __init__(self, options, inp_dim):
            super(LSTM_MASK, self).__init__()
            ....
            self.wfx = nn.Linear(input_dim, curernt_output, bias=add_bias)

掩码在 def __init __ 中也定义为

self.mask_use = torch.Tensor(curernt_output, input_dim)

mask是一个常数，mask参数的 .requires_grad _()是 False .现在，在该类的 def forward 部分中，我尝试在线性操作完成之前对weight参数和mask进行逐元素乘法

The mask is a constant and the .requires_grad_() is False for the mask parameter. Now in the def forward part of the class I attempt to do an element-wise multiplication of the weight parameter and the mask before the linear operation is completed

def forward(self, x):
    ....
    self.wfx.weight = self.wfx.weight * self.mask_use
    wfx_out = self.wfx(x)

我收到一条错误消息:

self.wfx.weight = self.wfx.weight * self.mask_use
  File "/home/xyz/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 537, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

但是当我使用 .type()检查这两个参数时，它们都以 torch.cuda.FloatTensor 出现.我不确定为什么这里有错误.

But when I check on the both parameters with .type() both of them come up as torch.cuda.FloatTensor. I am not sure why there is an error here.

推荐答案

基于元素的操作始终返回 FloatTensor .无法将法线张量分配为图层的 weight .

The element-wise operation always returns a FloatTensor. It is not possible to assign normal tensors as weight of layers.

有两种可能的解决方法.您可以将其分配给权重的 data 属性，也可以分配法线张量.

There are two possible options to deal with it. You can assign it to the data attribute of your weight, there it is possible assign normal tensors.

或者将结果转换为 nn.Parameter 本身，然后可以将其分配给 wfx.weight .

Or alternatively you convert your result to an nn.Parameter itself, then you can assign it to wfx.weight.

以下是同时显示两种方式的示例:

Here is an example which shows both ways:

import torch
import torch.nn as nn

wfx = nn.Linear(10, 10)
mask_use = torch.rand(10, 10)
#wfx.weight = wfx.weight * mask_use #your example - this raises an error

# Option 1: write directly to data
wfx.weight.data = wfx.weight * mask_use

# Option 2: convert result to nn.Parameter and write to weight
wfx.weight = nn.Parameter(wfx.weight * mask_use)

免责声明:在权重上使用 = (分配)时，您正在替换参数的权重张量.这可能会对图形的响应产生不良影响.优化步骤.

Disclaimer: When using an = (assignment) on the weights you are replacing the weights tensor of your parameter. This may have unwanted effects on the graph resp. the optimization step.

这篇关于如何在PyTorch权重参数中屏蔽权重?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！