训练我们的CNN进行文本分类时，如何微调word2vec? [英] how to fine-tune word2vec when training our CNN for text classification?
问题描述
我有3个关于微调字向量的问题.拜托，帮帮我.我会很感激的！提前非常感谢！
I have 3 Questions about fine-tuning word vectors. Please, help me out. I will really appreciate it! Many thanks in advance!
-
当我训练自己的CNN进行文本分类时，我使用Word2vec初始化单词，然后我将这些预先训练的向量用作输入特征来训练CNN，因此，如果我从来没有嵌入层，它肯定不能通过反向传播进行任何微调.我的问题是是否要进行微调，是否意味着要创建嵌入层?如何创建它?
When I train my own CNN for text classification, I use Word2vec to initialize the words, then I just employ these pre-trained vectors as my input features to train CNN, so if I never had a embedding layer, it surely can not do any fine-tunes through back-propagation. my question is if I want to do fine-tuning, does it means to create a Embedding layer?and how to create it?
当我们训练Word2vec时，我们使用无监督的训练对吗?就像我的情况一样，我使用skip-gram模型来获取预先训练的word2vec；但是，当我拥有vec.bin并将其用于文本分类模型(CNN)中作为单词初始化程序时，如果我可以在vec.bin中微调单词到矢量的映射，是否就意味着我必须CNN网络结构与训练我的Word2vec时的网络结构完全相同?微调的东西会改变vec.bin还是微调计算机内存?
When we train Word2vec, we use unsupervised training right? as in my case, I use the skip-gram model to get my pre-trained word2vec; But when I had the vec.bin and use it in the text classification model (CNN) as my words initialiser, if I could fine-tune the word-to-vector map in vec.bin, does it means that I have to have a CNN net structure exactly same as the one when training my Word2vec? and does the fine-tunes stuff would change the vec.bin or just fine-tune in computer memory?
skip-gram模型和CBOW模型仅用于无监督的Word2vec培训吗?或者他们也可以申请其他常规文本分类任务?和Word2vec无监督训练与监督微调之间的网络有何不同?
Are the skip-gram model and CBOW model are only used for unsupervised Word2vec training? Or they could also apply for other general text classification tasks? and what's the different of the network between Word2vec unsupervised training supervised fine-tuning?
@Franck Dernoncourt，谢谢您的提醒.我在这里很绿，希望能从强大的社区中学到一些东西.有空的时候请看看我的问题，再次感谢您！
@Franck Dernoncourt thank you for reminding me. I'm green here, and hope to learn something from the powerful community. Please have a look at my questions when you have time, thank you again!
推荐答案
第一个问题的答案-
在Embedding构造函数中设置trainable=True时.您的预训练嵌入设置为该embedding layer的weights.现在，在那些weights上发生的任何微调都与w2v(CBOW or SG)无关.如果要微调，则必须使用以下任何一种技术来微调w2v模型. 请参阅以下答案.
When you set trainable=True in your Embedding constructor. Your pretrained-embeddings are set as weights of that embedding layer. Now any fine-tuning that happens on those weights has nothing to do with w2v(CBOW or SG). If you want to finetune you will have to finetune your w2v model using any of these techniques. Refer to these answers.
答案2-
对embedding layer的weights进行的任何微调都不会影响您的vec.bin.这些更新的weights与模型一起保存，因此从理论上讲您可以将它们删除.
Any finetuning on weights of embedding layer does not affect your vec.bin. These updated weights are saved along with the model though so theoretically you can get them out.
答案3-
gensim仅实现这两种方法(SG和CBOW).但是，有多种新方法用于训练单词向量，例如MLM(屏蔽语言建模).
glove尝试对单词co-occurences的概率建模.
gensim implements only these two methods (SG and CBOW). However there are multiple new methods being used for training word vectors like MLM(masked language modeling).
glove tries to model probabilities of co-occurences of words.
如果要使用自己的自定义方法进行微调.您只需要指定一个任务(例如text classification)，然后保存更新的嵌入层权重即可.您将必须适当地进行索引编制，才能为每个单词分配相应的向量.
If you want to fine-tune using your own custom method. You will just have to specify a task(like text classification) and then save your updated embedding layer weights. You will have to take proper care of indexing to assign each word its corresponding vector.
这篇关于训练我们的CNN进行文本分类时，如何微调word2vec?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
