针对特定域微调Bert(无人监管) [英] Fine-tune Bert for specific domain (unsupervised)

问题描述

我想微调与特定领域(在我的情况下与工程有关)相关的文本上的BERT.培训应该不受监督，因为我没有任何标签或任何东西.这可能吗?

I want to fine-tune BERT on texts that are related to a specific domain (in my case related to engineering). The training should be unsupervised since I don't have any labels or anything. Is this possible?

推荐答案

您实际上想要的是继续对来自您特定域的文本进行BERT的预培训.在这种情况下，您要做的是继续将模型训练为掩蔽语言模型，但要针对您的域特定数据.

What you in fact want to is continue pre-training BERT on text from your specific domain. What you do in this case is to continue training the model as masked language model, but on your domain-specific data.

您可以使用 run_mlm.Huggingface的变形金刚中的py 脚本.

You can use the run_mlm.py script from the Huggingface's Transformers.

这篇关于针对特定域微调Bert(无人监管)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！