如何以清洁高效的方式在火炬中获得迷你批次? [英] How to get mini-batches in pytorch in a clean and efficient way?
问题描述
我试图做一个简单的事情，就是使用火炬训练带有随机梯度下降(SGD)的线性模型:
import numpy as np
import torch
from torch.autograd import Variable
import pdb
def get_batch2(X,Y,M,dtype):
X,Y = X.data.numpy(), Y.data.numpy()
N = len(Y)
valid_indices = np.array( range(N) )
batch_indices = np.random.choice(valid_indices,size=M,replace=False)
batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
batch_ys = torch.FloatTensor(Y[batch_indices]).type(dtype)
return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)
def poly_kernel_matrix( x,D ):
N = len(x)
Kern = np.zeros( (N,D+1) )
for n in range(N):
for d in range(D+1):
Kern[n,d] = x[n]**d;
return Kern
## data params
N=5 # data set size
Degree=4 # number dimensions/features
D_sgd = Degree+1
##
x_true = np.linspace(0,1,N) # the real data points
y = np.sin(2*np.pi*x_true)
y.shape = (N,1)
## TORCH
dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU
X_mdl = poly_kernel_matrix( x_true,Degree )
X_mdl = Variable(torch.FloatTensor(X_mdl).type(dtype), requires_grad=False)
y = Variable(torch.FloatTensor(y).type(dtype), requires_grad=False)
## SGD mdl
w_init = torch.zeros(D_sgd,1).type(dtype)
W = Variable(w_init, requires_grad=True)
M = 5 # mini-batch size
eta = 0.1 # step size
for i in range(500):
batch_xs, batch_ys = get_batch2(X_mdl,y,M,dtype)
# Forward pass: compute predicted y using operations on Variables
y_pred = batch_xs.mm(W)
# Compute and print loss using operations on Variables. Now loss is a Variable of shape (1,) and loss.data is a Tensor of shape (1,); loss.data[0] is a scalar value holding the loss.
loss = (1/N)*(y_pred - batch_ys).pow(2).sum()
# Use autograd to compute the backward pass. Now w will have gradients
loss.backward()
# Update weights using gradient descent; w1.data are Tensors,
# w.grad are Variables and w.grad.data are Tensors.
W.data -= eta * W.grad.data
# Manually zero the gradients after updating weights
W.grad.data.zero_()
#
c_sgd = W.data.numpy()
X_mdl = X_mdl.data.numpy()
y = y.data.numpy()
#
Xc_pinv = np.dot(X_mdl,c_sgd)
print('J(c_sgd) = ', (1/N)*(np.linalg.norm(y-Xc_pinv)**2) )
print('loss = ',loss.data[0])
代码运行良好，尽管我的get_batch2方法看起来真的很简单/天真，这可能是因为我对pytorch陌生，但是我没有找到一个很好的地方来讨论如何检索数据批.我浏览了他们的教程( http://pytorch.org/tutorials/beginner/pytorch_with_examples.html)并通过数据集( http://pytorch.org/tutorials/beginner/data_loading_tutorial. html ).所有教程似乎都假设一个人在开始时就已经具有批处理和批处理大小，然后继续训练该数据而不更改它(具体请参见解决方案
如果我正确地理解了您的代码，则您的get_batch2函数似乎正在从数据集中获取随机迷你批，而没有跟踪您使用了哪些索引已经到了一个时代.此实现的问题在于，它可能不会利用您的所有数据.
我通常进行批处理的方法是使用torch.randperm(N)为所有可能的顶点创建随机排列，然后分批循环遍历它们.例如:
n_epochs = 100 # or whatever
batch_size = 128 # or whatever
for epoch in range(n_epochs):
# X is a torch Variable
permutation = torch.randperm(X.size()[0])
for i in range(0,X.size()[0], batch_size):
optimizer.zero_grad()
indices = permutation[i:i+batch_size]
batch_x, batch_y = X[indices], Y[indices]
# in case you wanted a semi-full example
outputs = model.forward(batch_x)
loss = lossfunction(outputs,batch_y)
loss.backward()
optimizer.step()
如果您要复制和粘贴，请确保在纪元循环开始之前的某个位置定义优化器，模型和损失函数.
关于您的错误，请尝试使用torch.from_numpy(np.random.randint(0,N,size=M)).long()而不是torch.LongTensor(np.random.randint(0,N,size=M)).我不确定这是否可以解决您遇到的错误，但可以解决将来的错误.
I was trying to do a simple thing which was train a linear model with Stochastic Gradient Descent (SGD) using torch:
import numpy as np
import torch
from torch.autograd import Variable
import pdb
def get_batch2(X,Y,M,dtype):
X,Y = X.data.numpy(), Y.data.numpy()
N = len(Y)
valid_indices = np.array( range(N) )
batch_indices = np.random.choice(valid_indices,size=M,replace=False)
batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
batch_ys = torch.FloatTensor(Y[batch_indices]).type(dtype)
return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)
def poly_kernel_matrix( x,D ):
N = len(x)
Kern = np.zeros( (N,D+1) )
for n in range(N):
for d in range(D+1):
Kern[n,d] = x[n]**d;
return Kern
## data params
N=5 # data set size
Degree=4 # number dimensions/features
D_sgd = Degree+1
##
x_true = np.linspace(0,1,N) # the real data points
y = np.sin(2*np.pi*x_true)
y.shape = (N,1)
## TORCH
dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU
X_mdl = poly_kernel_matrix( x_true,Degree )
X_mdl = Variable(torch.FloatTensor(X_mdl).type(dtype), requires_grad=False)
y = Variable(torch.FloatTensor(y).type(dtype), requires_grad=False)
## SGD mdl
w_init = torch.zeros(D_sgd,1).type(dtype)
W = Variable(w_init, requires_grad=True)
M = 5 # mini-batch size
eta = 0.1 # step size
for i in range(500):
batch_xs, batch_ys = get_batch2(X_mdl,y,M,dtype)
# Forward pass: compute predicted y using operations on Variables
y_pred = batch_xs.mm(W)
# Compute and print loss using operations on Variables. Now loss is a Variable of shape (1,) and loss.data is a Tensor of shape (1,); loss.data[0] is a scalar value holding the loss.
loss = (1/N)*(y_pred - batch_ys).pow(2).sum()
# Use autograd to compute the backward pass. Now w will have gradients
loss.backward()
# Update weights using gradient descent; w1.data are Tensors,
# w.grad are Variables and w.grad.data are Tensors.
W.data -= eta * W.grad.data
# Manually zero the gradients after updating weights
W.grad.data.zero_()
#
c_sgd = W.data.numpy()
X_mdl = X_mdl.data.numpy()
y = y.data.numpy()
#
Xc_pinv = np.dot(X_mdl,c_sgd)
print('J(c_sgd) = ', (1/N)*(np.linalg.norm(y-Xc_pinv)**2) )
print('loss = ',loss.data[0])
the code runs fine and all though my get_batch2 method seems really dum/naive, its probably because I am new to pytorch but I have not found a good place where they discuss how to retrieve data batches. I went through their tutorials (http://pytorch.org/tutorials/beginner/pytorch_with_examples.html) and through the data set (http://pytorch.org/tutorials/beginner/data_loading_tutorial.html) with no luck. The tutorials all seem to assume that one already has the batch and batch-size at the beginning and then proceeds to train with that data without changing it (specifically look at http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-variables-and-autograd).
So my question is do I really need to turn my data back into numpy so that I can fetch some random sample of it and then turn it back to pytorch with Variable to be able to train in memory? Is there no way to get mini-batches with torch?
I looked at a few functions torch provides but with no luck:
#pdb.set_trace()
#valid_indices = torch.arange(0,N).numpy()
#valid_indices = np.array( range(N) )
#batch_indices = np.random.choice(valid_indices,size=M,replace=False)
#indices = torch.LongTensor(batch_indices)
#batch_xs, batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)
#batch_xs,batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)
even though the code I provided works fine I am worried that its not an efficient implementation AND that if I were to use GPUs that there would be a considerable further slow down (because my guess it putting things in memory and then fetching them back to put them GPU like that is silly).
I implemented a new one based on the answer that suggested to use torch.index_select():
def get_batch2(X,Y,M):
'''
get batch for pytorch model
'''
# TODO fix and make it nicer, there is pytorch forum question
#X,Y = X.data.numpy(), Y.data.numpy()
X,Y = X, Y
N = X.size()[0]
batch_indices = torch.LongTensor( np.random.randint(0,N+1,size=M) )
pdb.set_trace()
batch_xs = torch.index_select(X,0,batch_indices)
batch_ys = torch.index_select(Y,0,batch_indices)
return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)
however, this seems to have issues because it does not work if X,Y are NOT variables...which is really odd. I added this to the pytorch forum: https://discuss.pytorch.org/t/how-to-get-mini-batches-in-pytorch-in-a-clean-and-efficient-way/10322
Right now what I am struggling with is making this work for gpu. My most current version:
def get_batch2(X,Y,M,dtype):
'''
get batch for pytorch model
'''
# TODO fix and make it nicer, there is pytorch forum question
#X,Y = X.data.numpy(), Y.data.numpy()
X,Y = X, Y
N = X.size()[0]
if dtype == torch.cuda.FloatTensor:
batch_indices = torch.cuda.LongTensor( np.random.randint(0,N,size=M) )# without replacement
else:
batch_indices = torch.LongTensor( np.random.randint(0,N,size=M) ).type(dtype) # without replacement
pdb.set_trace()
batch_xs = torch.index_select(X,0,batch_indices)
batch_ys = torch.index_select(Y,0,batch_indices)
return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)
the error:
RuntimeError: tried to construct a tensor from a int sequence, but found an item of type numpy.int64 at index (0)
I don't get it, do I really have to do:
ints = [ random.randint(0,N) for i i range(M)]
to get the integers?
It would also be ideal if the data could be a variable. It seems that it torch.index_select does not work for Variable type data.
this list of integers thing still doesn't work:
TypeError: torch.addmm received an invalid combination of arguments - got (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor), but expected one of:
* (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
* (torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
* (float beta, torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
* (torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
* (float beta, torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
* (torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
* (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor)
* (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor)
If I'm understanding your code correctly, your get_batch2 function appears to be taking random mini-batches from your dataset without tracking which indices you've used already in an epoch. The issue with this implementation is that it likely will not make use of all of your data.
The way I usually do batching is creating a random permutation of all the possible vertices using torch.randperm(N) and loop through them in batches. For example:
n_epochs = 100 # or whatever
batch_size = 128 # or whatever
for epoch in range(n_epochs):
# X is a torch Variable
permutation = torch.randperm(X.size()[0])
for i in range(0,X.size()[0], batch_size):
optimizer.zero_grad()
indices = permutation[i:i+batch_size]
batch_x, batch_y = X[indices], Y[indices]
# in case you wanted a semi-full example
outputs = model.forward(batch_x)
loss = lossfunction(outputs,batch_y)
loss.backward()
optimizer.step()
If you like to copy and paste, make sure you define your optimizer, model, and lossfunction somewhere before the start of the epoch loop.
With regards to your error, try using torch.from_numpy(np.random.randint(0,N,size=M)).long() instead of torch.LongTensor(np.random.randint(0,N,size=M)). I'm not sure if this will solve the error you are getting, but it will solve a future error.
这篇关于如何以清洁高效的方式在火炬中获得迷你批次?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
