Tensorflow分配内存:38535168的分配超过系统内存的10％ [英] Tensorflow Allocation Memory: Allocation of 38535168 exceeds 10% of system memory

问题描述

使用ResNet50预先训练的权重我正在尝试建立分类器.该代码库已在Keras高级Tensorflow API中完全实现.完整的代码发布在下面的GitHub链接中.

Using ResNet50 pre-trained Weights I am trying to build a classifier. The code base is fully implemented in Keras high-level Tensorflow API. The complete code is posted in the below GitHub Link.

源代码:使用RestNet50体系结构进行分类

经过预先训练的模型的文件大小为 94.7mb .

The file size of the pre-trained model is 94.7mb.

我加载了预训练的文件

new_model = Sequential()

new_model.add(ResNet50(include_top=False,
                pooling='avg',
                weights=resnet_weight_paths))

并拟合模型

train_generator = data_generator.flow_from_directory(
    'path_to_the_training_set',
    target_size = (IMG_SIZE,IMG_SIZE),
    batch_size = 12,
    class_mode = 'categorical'
    )

validation_generator = data_generator.flow_from_directory(
    'path_to_the_validation_set',
    target_size = (IMG_SIZE,IMG_SIZE),
    class_mode = 'categorical'
    )

#compile the model

new_model.fit_generator(
    train_generator,
    steps_per_epoch = 3,
    validation_data = validation_generator,
    validation_steps = 1
)

在Training数据集中，我有两个文件夹dog和cat，每个文件夹可容纳近10,000张图像.编译脚本时，出现以下错误

and in the Training dataset, I have two folders dog and cat, each holder almost 10,000 images. When I compiled the script, I get the following error

Epoch 1/1 2018-05-12 13:04:45.847298:W tensorflow/core/framework/allocator.cc:101]分配38535168 超过系统内存的10％. 2018-05-12 13:04:46.845021:W tensorflow/core/framework/allocator.cc:101]分配37171200 超过系统内存的10％. 2018-05-12 13:04:47.552176:W tensorflow/core/framework/allocator.cc:101]分配37171200 超过系统内存的10％. 2018-05-12 13:04:48.199240:W tensorflow/core/framework/allocator.cc:101]分配37171200 超过系统内存的10％. 2018-05-12 13:04:48.918930:W tensorflow/core/framework/allocator.cc:101]分配37171200 超过系统内存的10％. 2018-05-12 13:04:49.274137:W tensorflow/core/framework/allocator.cc:101]分配19267584 超过系统内存的10％. 2018-05-12 13:04:49.647061:W tensorflow/core/framework/allocator.cc:101]分配19267584 超过系统内存的10％. 2018-05-12 13:04:50.028839:W tensorflow/core/framework/allocator.cc:101]分配19267584 超过系统内存的10％. 2018-05-12 13:04:50.413735:W tensorflow/core/framework/allocator.cc:101]分配19267584 超过系统内存的10％.

Epoch 1/1 2018-05-12 13:04:45.847298: W tensorflow/core/framework/allocator.cc:101] Allocation of 38535168 exceeds 10% of system memory. 2018-05-12 13:04:46.845021: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:47.552176: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:48.199240: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:48.918930: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:49.274137: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:49.647061: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:50.028839: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:50.413735: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory.

有什么想法可以优化加载预训练模型的方式(或摆脱这种警告消息吗?)

Any ideas to optimize the way to load the pre-trained model (or) get rid of this warning message?

谢谢！

推荐答案

尝试将batch_size属性减小为一个较小的数字(例如1,2或3). 示例:

Try reducing batch_size attribute to a small number(like 1,2 or 3). Example:

train_generator = data_generator.flow_from_directory(
    'path_to_the_training_set',
    target_size = (IMG_SIZE,IMG_SIZE),
    batch_size = 2,
    class_mode = 'categorical'
    )

这篇关于Tensorflow分配内存:38535168的分配超过系统内存的10％的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！