Tensorflow 负采样 [英] Tensorflow negative sampling

问题描述

我正在尝试遵循 tensorflow 上的 udacity 教程，在那里我遇到了以下两行词嵌入模型:

I am trying to follow the udacity tutorial on tensorflow where I came across the following two lines for word embedding models:

  # Look up embeddings for inputs.
  embed = tf.nn.embedding_lookup(embeddings, train_dataset)
  # Compute the softmax loss, using a sample of the negative labels each time.
  loss = tf.reduce_mean(tf.nn.sampled_softmax_loss(softmax_weights, softmax_biases, 
                        embed, train_labels, num_sampled, vocabulary_size))

现在我明白了第二个语句是用于抽样负面标签.但问题是它如何知道负面标签是什么?我提供的第二个函数是当前输入及其相应的标签以及我想要(负)采样的标签数量.本身就没有从输入集采样的风险吗?

Now I understand that the second statement is for sampling negative labels. But the question is how does it know what the negative labels are? All I am providing the second function is the current input and its corresponding labels along with number of labels that I want to (negatively) sample from. Isn't there the risk of sampling from the input set in itself?

这是完整示例:https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/5_word2vec.ipynb

推荐答案

您可以找到 tf.nn.sampled_softmax_loss() 此处.TensorFlow here (pdf).

You can find the documentation for tf.nn.sampled_softmax_loss() here. There is even a good explanation of Candidate Sampling provided by TensorFlow here (pdf).

它如何知道负面标签是什么?

How does it know what the negative labels are?

TensorFlow 会在所有可能的类中随机选择否定类(对你来说，所有可能的词).

TensorFlow will randomly select negative classes among all the possible classes (for you, all the possible words).

是否存在从输入集本身采样的风险?

Isn't there the risk of sampling from the input set in itself?

当你想为你的真实标签计算 softmax 概率时，你计算:logits[true_label]/sum(logits[negative_sampled_labels].因为类的数量很大(词汇量)，将 true_label 采样为负标签的可能性很小.
无论如何，我认为 TensorFlow 在随机采样时完全消除了这种可能性.(@Alex 确认 TensorFlow 默认执行此操作)

When you want to compute the softmax probability for your true label, you compute: logits[true_label] / sum(logits[negative_sampled_labels]. As the number of classes is huge (the vocabulary size), there is very little probability to sample the true_label as a negative label.
Anyway, I think TensorFlow removes this possibility altogether when randomly sampling. ( @Alex confirms TensorFlow does this by default)

这篇关于Tensorflow 负采样的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！