在 Pytorch 中嵌入 3D 数据 [英] Embedding 3D data in Pytorch
问题描述
我想实现字符级嵌入.
这是通常的词嵌入.
词嵌入
Input: [ [‘who’, ‘is’, ‘this’] ]
-> [ [3, 8, 2] ] # (batch_size, sentence_len)
-> // Embedding(Input)
# (batch_size, seq_len, embedding_dim)
这就是我想做的.
字符嵌入
Input: [ [ [‘w’, ‘h’, ‘o’, 0], [‘i’, ‘s’, 0, 0], [‘t’, ‘h’, ‘i’, ‘s’] ] ]
-> [ [ [2, 3, 9, 0], [ 11, 4, 0, 0], [21, 10, 8, 9] ] ] # (batch_size, sentence_len, word_len)
-> // Embedding(Input) # (batch_size, sentence_len, word_len, embedding_dim)
-> // sum each character embeddings # (batch_size, sentence_len, embedding_dim)
The final output shape is same as Word embedding. Because I want to concat them later.
虽然我尝试过,但我不确定如何实现 3-D 嵌入.你知道如何实现这样的数据吗?
Although I tried it, I am not sure how to implement 3-D embedding. Do you know how to implement such a data?
def forward(self, x):
print('x', x.size()) # (N, seq_len, word_len)
bs = x.size(0)
seq_len = x.size(1)
word_len = x.size(2)
embd_list = []
for i, elm in enumerate(x):
tmp = torch.zeros(1, word_len, self.embd_size)
for chars in elm:
tmp = torch.add(tmp, 1.0, self.embedding(chars.unsqueeze(0)))
以上代码出错,因为self.embedding
的输出是Variable
.
Above code got an error because output of self.embedding
is Variable
.
TypeError: torch.add received an invalid combination of arguments - got (torch.FloatTensor, float, Variable), but expected one of:
* (torch.FloatTensor source, float value)
* (torch.FloatTensor source, torch.FloatTensor other)
* (torch.FloatTensor source, torch.SparseFloatTensor other)
* (torch.FloatTensor source, float value, torch.FloatTensor other)
didn't match because some of the arguments have invalid types: (torch.FloatTensor, float, Variable)
* (torch.FloatTensor source, float value, torch.SparseFloatTensor other)
didn't match because some of the arguments have invalid types: (torch.FloatTensor, float, Variable)
更新
我可以做到这一点.但是 for
对批处理无效.你们知道更有效的方法吗?
Update
I could do this. But for
is not effective for batch. Do you guys know more efficient way?
def forward(self, x):
print('x', x.size()) # (N, seq_len, word_len)
bs = x.size(0)
seq_len = x.size(1)
word_len = x.size(2)
embd = Variable(torch.zeros(bs, seq_len, self.embd_size))
for i, elm in enumerate(x): # every sample
for j, chars in enumerate(elm): # every sentence. [ [‘w’, ‘h’, ‘o’, 0], [‘i’, ‘s’, 0, 0], [‘t’, ‘h’, ‘i’, ‘s’] ]
chars_embd = self.embedding(chars.unsqueeze(0)) # (N, word_len, embd_size) [‘w’,‘h’,‘o’,0]
chars_embd = torch.sum(chars_embd, 1) # (N, embd_size). sum each char's embedding
embd[i,j] = chars_embd[0] # set char_embd as word-like embedding
x = embd # (N, seq_len, embd_dim)
更新 2
这是我的最终代码.谢谢你,瓦西艾哈迈德!
Update2
This is my final code. Thank you, Wasi Ahmad!
def forward(self, x):
# x: (N, seq_len, word_len)
input_shape = x.size()
bs = x.size(0)
seq_len = x.size(1)
word_len = x.size(2)
x = x.view(-1, word_len) # (N*seq_len, word_len)
x = self.embedding(x) # (N*seq_len, word_len, embd_size)
x = x.view(*input_shape, -1) # (N, seq_len, word_len, embd_size)
x = x.sum(2) # (N, seq_len, embd_size)
return x
推荐答案
我假设你有一个形状为 BxSxW
的 3d 张量,其中:
I am assuming you have a 3d tensor of shape BxSxW
where:
B = Batch size
S = Sentence length
W = Word length
并且您已如下声明嵌入层.
And you have declared embedding layer as follows.
self.embedding = nn.Embedding(dict_size, emsize)
地点:
dict_size = No. of unique characters in the training corpus
emsize = Expected size of embeddings
所以,现在您需要将 BxSxW
形状的 3d 张量转换为 BSxW
形状的 2d 张量,并将其提供给嵌入层.
So, now you need to convert the 3d tensor of shape BxSxW
to a 2d tensor of shape BSxW
and give it to the embedding layer.
emb = self.embedding(input_rep.view(-1, input_rep.size(2)))
emb
的形状将是 BSxWxE
,其中 E
是嵌入大小.您可以将生成的 3d 张量转换为 4d 张量,如下所示.
The shape of emb
will be BSxWxE
where E
is the embedding size. You can convert the resulting 3d tensor to a 4d tensor as follows.
emb = emb.view(*input_rep.size(), -1)
emb
的最终形状将是 BxSxWxE
,这正是您所期望的.
The final shape of emb
will be BxSxWxE
which is what you are expecting.
这篇关于在 Pytorch 中嵌入 3D 数据的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!