如何从预训练模型加载保存的分词器 [英] How to load the saved tokenizer from pretrained model
问题描述
我使用 Huggingface Transformer 在 Pytorch 中微调了一个预训练的 BERT 模型.所有训练/验证均在云端 GPU 上完成.
I fine-tuned a pretrained BERT model in Pytorch using huggingface transformer. All the training/validation is done on a GPU in cloud.
在训练结束时,我保存模型和分词器,如下所示:
At the end of the training, I save the model and tokenizer like below:
best_model.save_pretrained('./saved_model/')
tokenizer.save_pretrained('./saved_model/')
这会在 saved_model
目录中创建以下文件:
This creates below files in the saved_model
directory:
config.json
added_token.json
special_tokens_map.json
tokenizer_config.json
vocab.txt
pytorch_model.bin
现在,我将 saved_model
目录下载到我的计算机中,并想加载模型和标记器.我可以像下面这样加载模型
Now, I download the saved_model
directory in my computer and want to load the model and tokenizer. I can load the model like below
model = torch.load('./saved_model/pytorch_model.bin',map_location=torch.device('cpu'))
但是我如何加载分词器?我是 pytorch 的新手,不确定因为有多个文件.可能我没有以正确的方式保存模型?
But how do I load the tokenizer? I am new to pytorch and not sure because there are multiple files. Probably I am not saving the model in the right way?
推荐答案
如果你看语法,它是你应该通过的预训练模型的目录.因此,加载分词器的正确方法必须是:
If you look at the syntax, it is the directory of the pre-trained model that you are supposed to pass. Hence, the correct way to load tokenizer must be:
tokenizer = BertTokenizer.from_pretrained(<Path to the directory containing pretrained model/tokenizer>)
就你而言:
tokenizer = BertTokenizer.from_pretrained('./saved_model/')
./saved_model
这里是您将保存预训练模型和标记器的目录.
./saved_model
here is the directory where you'll be saving your pretrained model and tokenizer.
这篇关于如何从预训练模型加载保存的分词器的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!