BERT令牌重要性衡量问题.毕业无 [英] BERT token importance measuring issue. Grad is none

问题描述

我正在尝试通过比较令牌嵌入grad值来衡量BERT的令牌重要性.因此，要获得毕业证书，我已经复制了BertModel的2.8.0并进行了一些更改:

I am trying to measure token importance for BERT via comparing token embedding grad value. So, to get the grad, I've copied the 2.8.0 forward of BertModel and changed it a bit:

huggingface transformers 2.8.0 BERT https://github.com/holdingface/transformers/blob/11c3257a18c4b5e1a3c1746eefd96f180358397b/src/transformers/modeling_bert.py

代码:

        embedding_output = self.embeddings(
            input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
        )
        embedding_output = embedding_output.requires_grad_(True) # my code
        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
        )
        sequence_output = encoder_outputs[0]
        sequence_output.mean().backward() # my code
        assert(embedding_output.grad is not None) # my code

Colab链接: https://colab.research.google.com/drive/1MggBUaDWAAZNuXbTDM11E8jvdMGEkuRD 但这会导致断言错误.我不明白为什么，这对我来说似乎是个错误. 请帮忙！

Colab link: https://colab.research.google.com/drive/1MggBUaDWAAZNuXbTDM11E8jvdMGEkuRD But it gives assertion error. I do not understand why and it seems to be a bug for me. Please, help!

推荐答案

我需要添加以下行:

embedding_output = torch.tensor(embedding_output, requires_grad=True)

似乎我错误地使用了.requires_grad_方法.

It seems, that I used .requires_grad_ method incorrectly.

这篇关于BERT令牌重要性衡量问题.毕业无的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！