构建Keras项目以在GPU中获得可重现的结果 [英] Structuring a Keras project to achieve reproducible results in GPU

问题描述

我正在编写一个tensorflow.Keras包装器来进行ML实验.

I am writing a tensorflow.Keras wrapper to perform ML experiments.

我需要我的框架能够执行配置yaml文件中指定的实验并在GPU中并行运行.

I need my framework to be able to perform an experiment as specified in a configuration yaml file and run in parallel in a GPU.

然后，我需要保证，如果我再次进行实验，即使结果不完全相同，也会得到相当接近的结果.

Then I need a guarantee that if I ran the experiment again I would get if not the exact same results something reasonably close.

为确保这一点，我的训练脚本首先按照官方文档:

To try to ensure this, my training script contains these lines at the beginning, following the guidelines in the official documentation:

# Set up random seeds
random.seed(seed)
np.random.seed(seed)
tf.set_random_seed(seed)

事实证明这还不够.

我运行相同的配置4次，并绘制了结果:

I ran the same configuration 4 times, and plotted the results:

如您所见，每次运行之间的结果差异很大.

As you can see, results vary a lot between runs.

如何在Keras中设置培训课程，以确保在GPU中进行培训时获得合理相似的结果?这有可能吗?

How can I set up a training session in Keras to ensure I get reasonably similar results when training in a GPU? Is this even possible?

完整的培训脚本可以在此处找到.

The full training script can be found here.

一些同事正在使用只是纯TF ，他们的结果似乎更加一致.而且，除了确保训练和验证拆分始终相同之外，他们似乎没有任何随机性.

Some of my colleagues are using just pure TF, and their results seem far more consistent. What is more, they do not seem to be seeding any randomness except to ensure that the train and validation split is always the same.

推荐答案

Keras + Tensorflow.

Keras + Tensorflow.

第1步，禁用GPU.

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = ""

第2步，植入代码中包含的库，例如"tensorflow，numpy，random".

Step 2, seed those libraries which are included in your code, say "tensorflow, numpy, random".

import tensorflow as tf
import numpy as np
import random as rn

sd = 1 # Here sd means seed.
np.random.seed(sd)
rn.seed(sd)
os.environ['PYTHONHASHSEED']=str(sd)

from keras import backend as K
config = tf.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)
tf.set_random_seed(sd)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
K.set_session(sess)

确保这两段代码都包含在代码的开头，那么结果将是可重复的.

Make sure these two pieces of code are included at the start of your code, then the result will be reproducible.

这篇关于构建Keras项目以在GPU中获得可重现的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！