BERT输出不确定 [英] BERT output not deterministic

问题描述

BERT输出不确定. 当我输入相同的输入时，我期望输出值是确定性的，但是在我的bert模型中，这些值正在变化.听起来很尴尬，相同的值一次返回两次.也就是说，一旦出现另一个值，就会出现相同的值并重复执行. 如何使输出具有确定性? 让我显示我的代码片段. 我使用以下模型.

BERT output is not deterministic. I expect the output values are deterministic when I put a same input, but my bert model the values are changing. Sounds awkwardly, the same value is returned twice, once. That is, once another value comes out, the same value comes out and it repeats. How I can make the output deterministic? let me show snippets of my code. I use the model as below.

对于BERT实现，我使用拥抱面实现的BERT pytorch实现.这是在pytorch地区非常有名的模型ri实现. [link] https://github.com/huggingface/pytorch-pretrained-BERT/

For the BERT implementation, I use huggingface implemented BERT pytorch implementation. which is quite fameous model ri implementation in the pytorch area. [link] https://github.com/huggingface/pytorch-pretrained-BERT/

        tokenizer = BertTokenizer.from_pretrained(self.bert_type, do_lower_case=self.do_lower_case, cache_dir=self.bert_cache_path)
        pretrain_bert = BertModel.from_pretrained(self.bert_type, cache_dir=self.bert_cache_path)
        bert_config = pretrain_bert.config

获得这样的输出

        all_encoder_layer, pooled_output = self.model_bert(all_input_ids, all_segment_ids, all_input_mask)

        # all_encoder_layer: BERT outputs from all layers.
        # pooled_output: output of [CLS] vec.

pooled_output

pooled_output

tensor([[-3.3997e-01,  2.6870e-01, -2.8109e-01, -2.0018e-01, -8.6849e-02,

tensor([[ 7.4340e-02, -3.4894e-03, -4.9583e-03,  6.0806e-02,  8.5685e-02,

tensor([[-3.3997e-01,  2.6870e-01, -2.8109e-01, -2.0018e-01, -8.6849e-02,

tensor([[ 7.4340e-02, -3.4894e-03, -4.9583e-03,  6.0806e-02,  8.5685e-02,

对于所有编码器层，情况都是相同的-一次两次相同.

for the all encoder layer, the situation is same, - same in twice an once.

我从bert中提取单词嵌入功能，情况是一样的.

I extract word embedding feature from the bert, and the situation is same.

wemb_n
tensor([[[ 0.1623,  0.4293,  0.1031,  ..., -0.0434, -0.5156, -1.0220],

tensor([[[ 0.0389,  0.5050,  0.1327,  ...,  0.3232,  0.2232, -0.5383],

tensor([[[ 0.1623,  0.4293,  0.1031,  ..., -0.0434, -0.5156, -1.0220],

tensor([[[ 0.0389,  0.5050,  0.1327,  ...,  0.3232,  0.2232, -0.5383],

推荐答案

请尝试设置种子.我遇到了同样的问题，并设置了种子以确保每次都获得相同的值.可能的原因之一可能是BERT退出了.

Please try to set the seed. I faced the same issue and set the seed to make sure we get same values every time. One of the possible reasons could be dropout taking place in BERT.

这篇关于BERT输出不确定的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！