Tensorflow 的内存成本在非常简单的“for 循环"中逐渐增加 [英] Tensorflow's memory cost gradually increasing in very simple “for loop”
问题描述
我在 tensorflow 上有一个很奇怪的问题.我将我的问题简化为以下版本:
I have a very strange problem on tensorflow. I simplified my question to the following version:
我问这个是因为我需要运行一系列训练,我只是将它们放在一个 for 循环中,然后我为每次迭代使用不同的参数.
I ask this because I need run a series of training, I just put them in a for loop, then I use different parameters for each iteration.
为了简化问题,我只用tensorflow语言写了一个简单的矩阵乘法,然后我把这个矩阵乘法训练"放在一个for循环"中(当然你可以把其他复杂的函数放在for循环中,结论一样).
To simplify the question, I just write a simple Matrix multiplication in tensorflow language, then I put this "Matrix multiplication training" in a "for loop"(Of course you can put other complicated functions in the for loop, the conclusion is same).
我设置了 100000 次迭代,这意味着我将运行 10000 个训练示例.并打印每个循环中的耗时,然后我可以观察到每次迭代的时间消耗是相同的,这是没有问题的.但是内存成本增加得很快,最后我得到了错误:内存不足"(我期望每次迭代的内存应该保持不变)
I set 100000 iterations times,which means I will run 10000 training examples. and print the time-consuming in each loop, then I can observe that the consumption of time is same for each iteration, this is no problem. But the memory cost increase very quickly and finally I got the Error:" run out of memory "(what I expect is the memory should keep same for each iteration)
import tensorflow as tf
import numpy as np
import datetime
for i in range(100000): # I must put the following code in this for loop
starttime = datetime.datetime.now()
graph=tf.Graph()
with graph.as_default():
with tf.device("/cpu:0"):
a=np.arange(100).reshape(1,-1)
b=np.arange(10000).reshape(100,100)
A = tf.placeholder(tf.float32, [1,100])
B = tf.placeholder(tf.float32, [100,100])
sess = tf.InteractiveSession()
RESULT =tf.matmul(A,B)
RESULT_o=sess.run(RESULT,feed_dict={A: a, B: b})
endtime = datetime.datetime.now()
print(endtime-starttime)
我知道原因是在每次迭代中,程序都会创建一个新的操作,这会增加内存.我想知道有没有办法在每次迭代后释放内存成本?(这个内存问题对于 GPU 情况是一样的)
I know the reason is that in each iteration, the program created a new operation,this will increase memory. I want to know is there any way to release the memory cost after each iteration? (this memory problem is same for GPU situation)
推荐答案
你的代码结构应该是这样的:
Your code should be structured like this:
import tensorflow as tf
import numpy as np
A = tf.placeholder(tf.float32, [1,100])
B = tf.placeholder(tf.float32, [100,100])
result = tf.matmul(A,B)
init_op = tf.global_variables_initializer()
# Later, when launching the model
with tf.Session() as sess:
# Run the init operation.
# This will make sure that memory is only allocated for the variable once.
sess.run(init_op)
for i in range(100000):
a = np.arange(100).reshape(1,-1)
b = np.arange(10000).reshape(100,100)
sess.run(result, feed_dict={A: a, B: b})
if i % 1000 == 0:
print(i, "processed")
在这里,您将为第一次迭代分配一次内存,并在后续迭代中继续重用相同的内存块.
Here you will allocate memory once for the first iteration and keep reusing the same memory block in successive iterations.
这篇关于Tensorflow 的内存成本在非常简单的“for 循环"中逐渐增加的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!