逐渐衰减损失函数的权重 [英] Gradually decay the weight of loss function

问题描述

我不确定是问这个问题的合适地点，请随时告诉我是否需要删除帖子.

I am not sure is the right place to ask this question, feel free to tell me if I need to remove the post.

我是pyTorch的新手，目前正在我的项目中使用CycleGAN(pyTorch实现)，并且我了解cycleGAN的大多数实现.

I am quite new in pyTorch and currently working with CycleGAN (pyTorch implementation) as a part of my project and I understand most of the implementation of cycleGAN.

我阅读了名为"CycleGAN with Better Cycles"的论文，并尝试应用本文中提到的修改.修改之一是周期一致性权重衰减，我不知道该如何应用.

I read the paper with the name ‘CycleGAN with better Cycles’ and I am trying to apply the modification which mentioned in the paper. One of modification is Cycle consistency weight decay which I don’t know how to apply.

optimizer_G.zero_grad()

# Identity loss
loss_id_A = criterion_identity(G_BA(real_A), real_A)
loss_id_B = criterion_identity(G_AB(real_B), real_B)

loss_identity = (loss_id_A + loss_id_B) / 2

# GAN loss
fake_B = G_AB(real_A)
loss_GAN_AB = criterion_GAN(D_B(fake_B), valid)
fake_A = G_BA(real_B)
loss_GAN_BA = criterion_GAN(D_A(fake_A), valid)

loss_GAN = (loss_GAN_AB + loss_GAN_BA) / 2

# Cycle consistency loss
recov_A = G_BA(fake_B)
loss_cycle_A = criterion_cycle(recov_A, real_A)
recov_B = G_AB(fake_A)
loss_cycle_B = criterion_cycle(recov_B, real_B)

loss_cycle = (loss_cycle_A + loss_cycle_B) / 2

# Total loss
loss_G =    loss_GAN + 
            lambda_cyc * loss_cycle + #lambda_cyc is 10
            lambda_id * loss_identity #lambda_id is 0.5 * lambda_cyc

loss_G.backward()
optimizer_G.step()

我的问题是如何逐渐减小循环一致性损失的权重?

My question is how can I gradually decay the weight of cycle consistency loss?

在实现此修改方面的任何帮助将不胜感激.

Any help in implementing this modification would be appreciated.

这是从论文中得出的: 周期一致性损失有助于在初期阶段稳定很多训练，但在后期阶段却成为获得逼真的图像的障碍.我们建议随着训练的进展逐渐降低循环一致性损失λ的权重.但是，我们仍应确保λ为 不会衰减到0，这样生成器就不会变得不受约束并完全消失了.

This is from the paper: Cycle consistency loss helps to stabilize training a lot in early stages but becomes an obstacle towards realistic images in later stages. We propose to gradually decay the weight of cycle consistency loss λ as training progress. However, we should still make sure that λ is not decayed to 0 so that generators won’t become unconstrained and go completely wild.

谢谢.

推荐答案

以下是您可以使用的原型函数！

Below is a prototype function you can use!

def loss (other params, decay params, initial_lambda, steps):
    # compute loss
    # compute cyclic loss
    # function that computes lambda given the steps
    cur_lambda  = compute_lambda(step, decay_params, initial_lamdba) 

    final_loss = loss + cur_lambda*cyclic_loss 
    return final_loss

---

compute_lambda函数以50步从10线性衰减到1e-5

---

compute_lambda function for linearly decaying from 10 to 1e-5 in 50 steps

def compute_lambda(step, decay_params):
    final_lambda = decay_params["final"]
    initial_lambda = decay_params["initial"]
    total_step = decay_params["total_step"]
    start_step = decay_params["start_step"]

    if (step < start_step+total_step and step>start_step):
        return initial_lambda + (step-start_step)*(final_lambda-initial_lambda)/total_step
    elif (step < start_step):
        return initial_lambda 
    else:
        return final_lambda
# Usage:
compute_lambda(i, {"final": 1e-5, "initial":10, "total_step":50, "start_step" : 50})    

这篇关于逐渐衰减损失函数的权重的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！