TensorFlow：训练for循环中的每次迭代都比较慢 [英] TensorFlow: Each iteration in training for-loop slower

问题描述

我正在训练一个标准，简单的多层感知器ANN，在TensorFlow中有三个隐藏层。我添加了一个文本进度条，所以我可以观察迭代遍历的进度。我发现，在最初几个时代之后，每次迭代的处理时间都会增加。下面是一个示例截图，显示了每次迭代的增加情况：

在这种情况下，最初的几次迭代花费了大约1.05s / 100％，花费了4.01s / it。

相关的代码在这里列出：

 ＃------------------- ------构建TensorFlow图------------------------- 
 
与tf.Graph（）。 as_default（）：
 
（一堆用于指定图形的语句）
 
＃------------------- --------------培训---------------------------------- 
 $ b $ sess = tf.InteractiveSession（）
 sess.run（tf.initialize_all_variables（））
 
 print开始训练
 
 pbar = tqdm（total = training_epochs）
为范围内的纪元（training_epochs）：
 avg_成本= 0.0 
 batch_iter = 0 
 
 while batch_iter< batch_size：
 train_features = [] 
 train_labels = [] 
 batch_segments = random.sample（train_segments，20）
 for batch_segments：
 train_features.append（segment 
 train_labels.append（segment [1]）$ ​​b $ b sess.run（optimizer，feed_dict = {x：train_features，y_：train_labels}）
 line_out =，+ str （batch_iter）+\\\

 train_outfile.write（line_out）
 line_out =,,+ str（sess.run（tf.reduce_mean（weights ['h1']），feed_dict = {x：train_features，y_：train_labels}））
 line_out + =，+ str（sess.run（tf.reduce_mean（weights ['h2']），feed_dict = {x：train_features，y_：train_labels }））
 line_out + =，+ str（sess.run（tf.reduce_mean（weights ['h3']），feed_dict = {x：train_features，y_：train_labels}））+\\\
 
 train_outfile.write（line_out）
 avg_cost + = sess.run（cost，feed_dict = {x：train_features，y_：train_labels}）/ ba tch_size 
 
 batch_iter + = 1 
 
 pbar.update（1）＃将进度条增加一个
 
 train_outfile.close（）
打印完成的培训
  

在搜索stackoverflow时，我发现每次迭代后处理时间越来越长，而其他人也遇到问题每次迭代的时间比最后一次长。不过，我相信我可能会有所不同，因为他们明确地使用像这样的语句添加操作：

  distorted_image = tf。虽然我是TensorFlow的新手，但我不相信我因为在我的循环中唯一的东西是sess.run（）调用。
 
 
 
任何帮助非常感谢。
解决方案
您有三个地方：
 
 
  sess。 run（tf.reduce_mean（weights ['h1']），...）
  
新的 tf.reduce_mean（）节点添加到while循环的每次迭代中的图，这会增加开销。尝试在while循环之外创建它们：使用tf.Graph（）。
 
 
  as_default（）：
。 .. 
 m1 = tf.reduce_mean（权重['h1']）
 
，而batch_iter< batch_size：
 ... 
 line_out =,,+ str（sess.run（m1，feed_dict = {x：train_features，y_：train_labels}））
  
 
I'm training a standard, simple multilayer perceptron ANN with three hidden layers in TensorFlow. I added a text progress bar so I could watch the progress of iterating through the epochs. What I'm finding is that the processing time per iteration increases after the first few epochs. Here's an example screenshot showing the increase with each iteration:




In this case, the first few iterations took roughly 1.05s/it and by 100% it was taking 4.01s/it.

The relevant code is listed here:
# ------------------------- Build the TensorFlow Graph -------------------------

with tf.Graph().as_default():

    (a bunch of statements for specifying the graph)

# --------------------------------- Training ----------------------------------

    sess = tf.InteractiveSession()
    sess.run(tf.initialize_all_variables())

    print "Start Training"

    pbar = tqdm(total = training_epochs)
    for epoch in range(training_epochs):
        avg_cost = 0.0
    batch_iter = 0

    while batch_iter < batch_size:
        train_features = []
        train_labels = []
        batch_segments = random.sample(train_segments, 20)
        for segment in batch_segments:
            train_features.append(segment[0])
            train_labels.append(segment[1])
        sess.run(optimizer, feed_dict={x: train_features, y_: train_labels})
        line_out = "," + str(batch_iter) + "\n"
        train_outfile.write(line_out)
        line_out = ",," + str(sess.run(tf.reduce_mean(weights['h1']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h2']), feed_dict={x: train_features, y_: train_labels}))
        line_out += "," + str(sess.run(tf.reduce_mean(weights['h3']), feed_dict={x: train_features, y_: train_labels})) + "\n"
        train_outfile.write(line_out)
        avg_cost += sess.run(cost, feed_dict={x: train_features, y_: train_labels})/batch_size

        batch_iter += 1

    pbar.update(1)  # Increment the progress bar by one

train_outfile.close()
print "Completed training"
In searching stackoverflow, I found Processing time gets longer and longer after each iteration where someone else was also having problems with each iteration taking longer than the last. However, I believe mine may be different since they were clearly adding ops to the graph using statements like so:
distorted_image = tf.image.random_flip_left_right(image_tensor)
While I'm new to TensorFlow, I don't believe that I'm making the same mistake because the only stuff in my loop are sess.run() calls.

Any help is much appreciated.
解决方案
The three places where you have:
sess.run(tf.reduce_mean(weights['h1']), ...)
each append a new tf.reduce_mean() node to the graph at each iteration of the while loop, which adds overhead. Try to create them outside of the while loop:
with tf.Graph().as_default():
  ...
  m1 = tf.reduce_mean(weights['h1'])

while batch_iter < batch_size:
  ...
  line_out = ",," + str(sess.run(m1, feed_dict={x: train_features, y_: train_labels}))


                        
这篇关于TensorFlow：训练for循环中的每次迭代都比较慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！