如何在bert模型的顶部添加Bi-LSTM层? [英] How can i add a Bi-LSTM layer on top of bert model?
问题描述
我正在使用 pytorch ,并且正在使用基础预训练的伯特对仇恨言论进行句子分类.我想实现一个 Bi-LSTM 层,该层将最新的所有输出作为输入bert模型的变压器编码器作为一种新模型(实现 nn.Module 的类),我对 nn.LSTM 参数感到困惑.我使用
I'm using pytorch and I'm using the base pretrained bert to classify sentences for hate speech. I want to implement a Bi-LSTM layer that takes as an input all outputs of the latest transformer encoder from the bert model as a new model (class that implements nn.Module), and i got confused with the nn.LSTM parameters. I tokenized the data using
bert = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=int(data['class'].nunique()),output_attentions=False,output_hidden_states=False)
我的数据集有2列:类(标签),句子.有人可以帮我弄这个吗?预先谢谢你.
My data-set has 2 columns: class(label), sentence. Can someone help me with this? Thank you in advance.
修改:同样,在处理bi-stm中的输入之后,网络会将最终的隐藏状态发送到使用softmax激活功能执行分类的完全连接的网络.我该怎么办?
Edit: Also, after processing the input in the bi-lstm, the network sends the final hidden state to a fully connected network that performs classication using the softmax activation function. how can I do that ?
推荐答案
您可以执行以下操作:
from transformers import BertModel
class CustomBERTModel(nn.Module):
def __init__(self):
super(CustomBERTModel, self).__init__()
self.bert = BertModel.from_pretrained("bert-base-uncased")
### New layers:
self.lstm = nn.LSTM(768, 256, batch_first=True,bidirectional=True)
self.linear = nn.Linear(256*2, <number_of_classes>, batch_first=True)
def forward(self, ids, mask):
sequence_output, pooled_output = self.bert(
ids,
attention_mask=mask)
# sequence_output has the following shape: (batch_size, sequence_length, 768)
lstm_output, (h,c) = self.lstm(sequence_output) ## extract the 1st token's embeddings
hidden = torch.cat((lstm_output[:,-1, :256],lstm_output[:,0, 256:]),dim=-1)
linear_output = self.linear(lstm_output[:,-1].view(-1,256*2)) ### assuming that you are only using the output of the last LSTM cell to perform classification
return linear_output
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
model = CustomBERTModel()
这篇关于如何在bert模型的顶部添加Bi-LSTM层?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!