为特定领域微调 Bert(无监督) [英] Fine-tune Bert for specific domain (unsupervised)

问题描述

我想在与特定领域(在我的情况下与工程相关)相关的文本上微调 BERT.培训应该是无人监督的，因为我没有任何标签或任何东西.这可能吗?

I want to fine-tune BERT on texts that are related to a specific domain (in my case related to engineering). The training should be unsupervised since I don't have any labels or anything. Is this possible?

推荐答案

您实际上想要的是继续对来自您特定域的文本进行 BERT 预训练.在这种情况下，您要做的是继续将模型训练为掩码语言模型，但使用您的领域特定数据.

What you in fact want to is continue pre-training BERT on text from your specific domain. What you do in this case is to continue training the model as masked language model, but on your domain-specific data.

您可以使用 run_mlm.py 来自 Huggingface 的变形金刚的脚本.

You can use the run_mlm.py script from the Huggingface's Transformers.

这篇关于为特定领域微调 Bert(无监督)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！