如何从Pytorch中的预训练模型加载保存的令牌生成器 [英] How to load the saved tokenizer from pretrained model in Pytorch

问题描述

我使用 Huggingface Transformer 在 Pytorch 中微调了一个预训练的 BERT 模型.所有训练/验证都是在云中的GPU上完成的.

I fine-tuned a pretrained BERT model in Pytorch using huggingface transformer. All the training/validation is done on a GPU in cloud.

在培训结束时，我将模型和令牌生成器保存如下:

At the end of the training, I save the model and tokenizer like below:

best_model.save_pretrained('./saved_model/')
tokenizer.save_pretrained('./saved_model/')

这将在 saved_model 目录中创建以下文件:

This creates below files in the saved_model directory:

config.json
added_token.json
special_tokens_map.json
tokenizer_config.json
vocab.txt
pytorch_model.bin

现在，我在计算机上下载了 saved_model 目录，并希望加载模型和令牌生成器.我可以像下面那样加载模型

Now, I download the saved_model directory in my computer and want to load the model and tokenizer. I can load the model like below

model = torch.load('./saved_model/pytorch_model.bin'，map_location = torch.device('cpu'))

但是如何加载令牌生成器?我是pytorch的新手，不确定，因为有多个文件.可能我没有以正确的方式保存模型?

But how do I load the tokenizer? I am new to pytorch and not sure because there are multiple files. Probably I am not saving the model in the right way?

请提出建议.

推荐答案

如果您查看语法，则它是您应该传递的经过预先训练的模型的目录.因此，加载令牌生成器的正确方法必须是:

If you look at the syntax, it is the directory of the pre-trained model that you are supposed to pass. Hence, the correct way to load tokenizer must be:

tokenizer = BertTokenizer.from_pretrained(<包含预训练模型/tokenizer的目录的路径>)

在您的情况下:

tokenizer = BertTokenizer.from_pretrained('./saved_model/')

./saved_model 这是您将保存预训练的模型和令牌生成器的目录.

./saved_model here is the directory where you'll be saving your pretrained model and tokenizer.

这篇关于如何从Pytorch中的预训练模型加载保存的令牌生成器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！