PyTorch:损失保持不变 [英] PyTorch: Loss remains constant

问题描述

我已经用自己实现的损失函数focal_loss_fixed在PyTorch中编写了代码.但是我的损失价值在每个时期都保持不变.权重似乎没有被更新.这是我的代码段:

I've written a code in PyTorch with my own implemented loss function focal_loss_fixed. But my loss value stays fixed after every epoch. Looks like weights are not being updated. Here is my code snippet:

optimizer = optim.SGD(net.parameters(),
                          lr=lr,
                          momentum=0.9,
                          weight_decay=0.0005)


for epoch in T(range(20)):
    net.train()
    epoch_loss = 0
    for n in range(len(x_train)//batch_size):
        (imgs, true_masks) = data_gen_small(x_train, y_train, iter_num=n, batch_size=batch_size)
        temp = []
        for tt in true_masks:
            temp.append(tt.reshape(128, 128, 1))
        true_masks = np.copy(np.array(temp))
        del temp
        imgs = np.swapaxes(imgs, 1,3)
        imgs = torch.from_numpy(imgs).float().cuda()
        true_masks = torch.from_numpy(true_masks).float().cuda()
        masks_pred = net(imgs)
        masks_probs = F.sigmoid(masks_pred)
        masks_probs_flat = masks_probs.view(-1)
        true_masks_flat = true_masks.view(-1)
        print((focal_loss_fixed(tf.convert_to_tensor(true_masks_flat.data.cpu().numpy()), tf.convert_to_tensor(masks_probs_flat.data.cpu().numpy()))))
        loss = torch.from_numpy(np.array(focal_loss_fixed(tf.convert_to_tensor(true_masks_flat.data.cpu().numpy()), tf.convert_to_tensor(masks_probs_flat.data.cpu().numpy())))).float().cuda()
        loss = Variable(loss.data, requires_grad=True)
        epoch_loss *= (n/(n+1))
        epoch_loss += loss.item()*(1/(n+1))
        print('Step: {0:.2f}% --- loss: {1:.6f}'.format(n * batch_size* 100.0 / len(x_train), epoch_loss), end='\r')
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print('Epoch finished ! Loss: {}'.format(epoch_loss))

这是我的"focal_loss_fixed"函数:

And this is my `focal_loss_fixed' function:

def focal_loss_fixed(true_data, pred_data):
    gamma=2.
    alpha=.25
    eps = 1e-7
    # print(type(y_true), type(y_pred))
    pred_data = K.clip(pred_data,eps,1-eps)
    pt_1 = tf.where(tf.equal(true_data, 1), pred_data, tf.ones_like(pred_data))
    pt_0 = tf.where(tf.equal(true_data, 0), pred_data, tf.zeros_like(pred_data))
    with tf.Session() as sess:
        return sess.run(-K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1))-K.sum((1-alpha) * K.pow( pt_0, gamma) * K.log(1. - pt_0)))

在每个时期之后，损耗值保持恒定(5589.60328).怎么了?

After each epoch the loss value stays constant(5589.60328). What's wrong with it?

推荐答案

我认为问题出在您体重的下降.

I think the problem lies in your heavy weight decay.

从本质上讲，您不是在上减去x 来减少权重，而是将上的权重乘以x ，这意味着您瞬时只是在做较小的增量，导致(看似)平稳损失函数.

Essentially, you are not reducing the weight by x, but rather you multiply the weights by x, which means that you are instantaneously only doing very small increments, leading to a (seemingly) plateauing loss function.

有关此问题的更多解释，请参见PyTorch论坛(例如，此处 ).
不幸的是，仅仅用于SGD的源也无法实现告诉您有关其实现的更多信息. 只需将其设置为较大的值，即可获得更好的更新.您可以先将其完全忽略掉，然后迭代地减少它(从1.0开始)，直到获得更好的结果为止.

More explanation on this can be found in the PyTorch discussion forum (e.g., here, or here).
Unfortunately, the source for SGD alone also does not tell you much about its implementation. Simply setting it to a larger value should result in better updates. You can start by leaving it out completely, and then iteratively reducing it (from 1.0), until you get more decent results.

这篇关于PyTorch:损失保持不变的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！