Tensorflow (1.4.1) Tensorboard 可视化图可以追溯到过去吗? [英] Tensorflow (1.4.1) Tensorboard visualization plot goes back in time?

问题描述

我在整个图表中创建了一些摘要操作，如下所示:

I created a few summary ops throughout my graph like so:

tf.summary.scalar('cross_entropy', cross_entropy)
tf.summary.scalar('accuracy', accuracy)

当然合并了一个作家:

sess = tf.InteractiveSession()
summaries = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(TENSORBOARD_TRAINING_DIR, sess.graph)
tf.global_variables_initializer().run()

我在每次训练迭代中都写了这些:

and I write these in each training iteration:

summary, acc = sess.run([summaries, accuracy], feed_dict={...})
train_writer.add_summary(summary, i)

当我加载张量板时，我得到了一些奇怪的结果:

when I load the tensorboard, I get some weird results:

这很奇怪，原因有几个:

this is weird for a couple reasons:

1. cross_entropy 图上的 Y 轴没有增加(或不同)的刻度线
2. 线图似乎会自行折叠或回到过去

我确实检查过 - 我的训练摘要文件夹中有一些以前的事件文件:

I did check - there are a few previous event files in my training summaries folder:

$ ls /tmp/tv_train/
events.out.tfevents.1517210066.xxxxxxx.local    
events.out.tfevents.1517210097.xxxxxxx.local    
...
events.out.tfevents.1517210392.xxxxxxx.local

我想我一定是在某个时候重新启动了火车循环，导致在(0、1 等)索引处记录了多个摘要.

I think I must have restarted the train loop at some point, causing there to be multiple summaries logged at (0, 1, etc) indices.

如何附加到旧的训练日志?我可以将我的作者指向特定的 tfevents 文件以从我离开的地方重新开始"吗?

How can I append to old training logs? Can I point my writer to a specific tfevents file to "start back where I left off"?

推荐答案

您不能(轻松地)重新打开和附加"到现有的事件文件，但这不是必需的.Tensorboard 会显示顺序事件文件就好了，只要记录中的步长值一致.保存摘要时，您指定一个 step 值，该值指示应在 x 轴上的哪个点绘制摘要.

You can't (easily) reopen and "append" to an existing events file, but that's not necessary. Tensorboard will display sequential event files just fine, as long as the step value in the records is consistent. When you save a summary, you specify a step value, which indicates at which point on the x axis the summary should be plotted.

图形会回到过去"，因为在每次新运行时，您都会从 0 重新启动计步器.要使其在多次运行中保持一致，您应该定义一个 global_step 变量，该变量保存到保存网络时的检查点.这样，当您在下一次训练运行中恢复网络时，您的全局步长将从它离开的地方开始，您的图表将不再看起来很奇怪.

The graph goes "back in time" because at every new run you restart the step counter from 0. To have it consistent in multiple runs, you should define a global_step variable that is saved to the checkpoint when you save the network. This way, when you restore the network in the next training run, your global step will pick up from where it left and your graphs will not look weird anymore.

这篇关于Tensorflow (1.4.1) Tensorboard 可视化图可以追溯到过去吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！