在张量流中计算多个批次的精确移动平均值 [英] Computing exact moving average over multiple batches in tensorflow

问题描述

在训练期间，我想将最近 N 个小批量的平均损失写入 SummaryWriter，作为平滑非常嘈杂的批量损失的一种方式.在 python 中计算并打印它很容易，但我想将它添加到摘要中，以便我可以在 tensorboard 中看到它.这是我现在正在做的事情的一个过于简化的例子.

During training, I would like to write the average loss over the last N mini-batches to SummaryWriter as a way of smoothing the very noisy batch loss. It's easy to compute this in python and print it, but I would like to add this to a summary so that I can see it in tensorboard. Here's an overly simplified example of what I'm doing now.

losses = []
for i in range(10000):
  _, loss = session.run([train_op, loss_op])
  losses.append(loss)
  if i % 100 == 0:
    # How to produce a scalar_summary here?
    print sum(losses)/len(losses)
    losses = []

我知道我可以使用衰减为 1.0 的 ExponentialMovingAverage，但我仍然需要某种方法来每 N 批重置一次.真的，如果我只关心张量板中的损失可视化，则可能不需要重置，但我仍然很好奇出于其他原因(例如，计算过于复杂的测试数据集的总准确度)如何跨批次聚合大批量运行).

I'm aware that I could use ExponentialMovingAverage with a decay of 1.0, but I would still need some way to reset this every N batches. Really, if all I care about is visualizing loss in tensorboard, the reset probably isn't necessary, but I'm still curious how one would go about aggregating across batches for other reasons (e.g. computing total accuracy over a test dataset that is too big to run in a single batch).

推荐答案

可以使用占位符和 feed_dict 将数据从 python 传递到像 tf.scalar_summary 这样的图形函数.

Passing data from python to a graph function like tf.scalar_summary can be done using a placeholder and feed_dict.

average_pl = tf.placeholder(tf.float32)
average_summary = tf.summary.scalar("average_loss", average_pl)
writer = tf.summary.FileWriter("/tmp/mnist_logs", sess.graph_def)

losses = []
for i in range(10000):
  _, loss = session.run([train_op, loss_op])
  losses.append(loss)
  if i % 100 == 0:
    # How to produce a scalar_summary here?
    feed = {average_pl: sum(losses)/len(losses)}
    summary_str = sess.run(average_summary, feed_dict=feed)
    writer.add_summary(summary_str, i)
    losses = []

我还没有尝试过，这是从可视化数据中匆忙复制过来的，但我希望这样的事情会奏效.

I haven't tried it and this was hastily copied from the visualizing data how to but I expect something like this would work.

这篇关于在张量流中计算多个批次的精确移动平均值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！