Tensorflow 计算时内存不足:如何查找内存泄漏? [英] Tensorflow runs out of memory while computing: how to find memory leaks?
问题描述
我正在使用 Google 的 TensorFlow DeepDream 实现(https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb).
I'm iteratively deepdreaming images in a directory using the Google's TensorFlow DeepDream implementation (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb).
我的代码如下:
model_fn = tensorflow_inception_graph.pb
# creating TensorFlow session and loading the model
graph = tf.Graph()
sess = tf.InteractiveSession(graph=graph)
with tf.gfile.FastGFile(model_fn, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
t_input = tf.placeholder(np.float32, name='input') # define the input tensor
imagenet_mean = 117.0
t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)
tf.import_graph_def(graph_def, {'input':t_preprocessed})
def render_deepdream(t_obj, img0=img_noise,
iter_n=10, step=1.5, octave_n=4, octave_scale=1.4):
t_score = tf.reduce_mean(t_obj) # defining the optimization objective
t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
# split the image into a number of octaves
img = img0
octaves = []
for i in range(octave_n-1):
hw = img.shape[:2]
lo = resize(img, np.int32(np.float32(hw)/octave_scale))
hi = img-resize(lo, hw)
img = lo
octaves.append(hi)
# generate details octave by octave
for octave in range(octave_n):
if octave>0:
hi = octaves[-octave]
img = resize(img, hi.shape[:2])+hi
for i in range(iter_n):
g = calc_grad_tiled(img, t_grad)
img += g*(step / (np.abs(g).mean()+1e-7))
#print('.',end = ' ')
#clear_output()
#showarray(img/255.0)
return img/255.0
def morphPicture(filename1,filename2,blend,width):
img1 = PIL.Image.open(filename1)
img2 = PIL.Image.open(filename2)
if width is not 0:
img2 = resizePicture(filename2,width)
finalImage= PIL.Image.blend(img1, img2, blend)
del img1
del img2
return finalImage
def save_array(arr, name,direc, ext="png"):
img = np.uint8(np.clip(arr, 0, 1)*255)
img =cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
cv2.imwrite("{d}/{n}.{e}".format(d=direc, n=name, e=ext), img)
del img
framesDir = "my directory"
os.chdir(framesDir)
outputDir ="my directory"
for file in os.listdir(framesDir):
img0 = PIL.Image.open(file)
img0 = np.float32(img0)
dreamedImage = render_deepdream(tf.square(T('mixed4c')),img0,iter_n=3,octave_n=6)
save_array(dreamedImage,1,outputDir,'jpg')
break
i=1
j=0
with tf.device('/gpu:0'):
for file in os.listdir(framesDir):
if j<=1: #already processed first image so we skip it here
j+=1
continue
else:
dreamedImage = "my directory"+str(i)+'.jpg' # get the previous deep dreamed frame
img1 = file # get the next undreamed frame
morphedImage = morphPicture(dreamedImage,img1,0.5,0) #blend the images
morphedImage=np.float32(morphedImage)
dreamedImage = render_deepdream(tf.square(T('mixed4c')),morphedImage,iter_n=3,octave_n=6) #deep dream a
#blend of the two frames
i+=1
save_array(dreamedImage,i,outputDir,'jpg') #save the dreamed image
del dreamedImage
del img1
del morphedImage
time.sleep(0.5)
每当我运行代码超过一个小时时，脚本就会因 MemoryError 而停止.我假设某处一定存在内存泄漏，但我找不到它.我认为通过包含多个 del 语句，我可以摆脱阻塞 RAM/CPU 的对象，但它似乎不起作用.
Whenever I run the code for more than an hour, the script stops with a MemoryError. I'm assuming there must be a memory leak somewhere, but I'm unable to find it. I thought that by including multiple del statements, I would get rid of the objects that were clogging up the RAM/CPU, but it doesn't seem to be working.
我的代码中是否有明显的对象堆积?或者是在我的代码下方的某个地方，即在 tensorflow 中构建的?
Is there an obvious build up of objects that I am missing within my code? Or is the build up somewhere beneath my code, i.e. within tensorflow?
任何帮助/建议将不胜感激.谢谢.
Any help/suggestions would be much appreciated. Thanks.
仅供参考，目录中有 901 张图片.我正在使用带有 NVIDIA GeForce GTX 980 Ti 的 Windows 7.
FYI there are 901 images in the directory. I am using Windows 7 with NVIDIA GeForce GTX 980 Ti.
推荐答案
99% 的情况下，在使用 tensorflow 时，内存泄漏"实际上是由于迭代时不断添加到图中的操作 - 而不是构建先绘制图形，然后循环使用.
99% of the time, when using tensorflow, "memory leaks" are actually due to operations that are continuously added to the graph while iterating — instead of building the graph first, then using it in a loop.
您为循环指定设备(使用 tf.device('/gpu:0) 的事实暗示了这种情况:您通常为新节点指定设备为这不会影响已经定义的节点.
The fact that you specify a device (with tf.device('/gpu:0) for your loop is a hint that it is the case: you typically specify a device for new nodes as this does not affect nodes that are already defined.
幸运的是，tensorflow 有一个方便的工具来发现这些错误:tf.Graph.finalize.调用时，此函数可防止将更多节点添加到图形中.在迭代之前调用此函数是一种很好的做法.
Fortunately, tensorflow has a convenient tool to spot those errors: tf.Graph.finalize. When called, this function prevents further nodes to be added to your graph. It is good practice to call this function before iterating.
所以在你的情况下，我会在你的循环之前调用 tf.get_default_graph().finalize() 并查找它可能抛出的任何错误.
So in your case I would call tf.get_default_graph().finalize() before your loop and look for any error it may throw.
这篇关于Tensorflow 计算时内存不足:如何查找内存泄漏?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
