构建 Keras 项目以在 GPU 中实现可重现的结果 [英] Structuring a Keras project to achieve reproducible results in GPU

问题描述

我正在编写一个 tensorflow.Keras 包装器来执行机器学习实验.

我需要我的框架能够执行配置 yaml 文件中指定的实验并在 GPU 中并行运行.

然后我需要保证，如果我再次运行该实验，即使不是完全相同的结果，我也会得到一些合理接近的结果.

为了确保这一点，我的训练脚本在开头包含这些行，遵循

如您所见，运行之间的结果差异很大.

如何在 Keras 中设置训练课程以确保在 GPU 中训练时获得相当相似的结果?这甚至可能吗?

可以在此处找到完整的训练脚本.

我的一些同事正在使用纯TF，他们的结果似乎更加一致.更重要的是，他们似乎没有播种任何随机性，只是为了确保训练和验证拆分始终相同.

解决方案

Keras + Tensorflow.

第 1 步，禁用 GPU.

导入操作系统os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"os.environ["CUDA_VISIBLE_DEVICES"] = ""

第 2 步，为代码中包含的库设置种子，比如tensorflow、numpy、random".

 将 tensorflow 导入为 tf将 numpy 导入为 np将随机导入为 rnsd = 1 # 这里 sd 表示种子.np.random.seed(sd)rn.种子(sd)os.environ['PYTHONHASHSEED']=str(sd)从 keras 导入后端为 Kconfig = tf.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)tf.set_random_seed(sd)sess = tf.Session(graph=tf.get_default_graph(), config=config)K.set_session(sess)

确保在代码的开头包含这两段代码，然后结果将是可重现的.

I am writing a tensorflow.Keras wrapper to perform ML experiments.

I need my framework to be able to perform an experiment as specified in a configuration yaml file and run in parallel in a GPU.

Then I need a guarantee that if I ran the experiment again I would get if not the exact same results something reasonably close.

To try to ensure this, my training script contains these lines at the beginning, following the guidelines in the official documentation:

# Set up random seeds
random.seed(seed)
np.random.seed(seed)
tf.set_random_seed(seed)

This has proven to not be enough.

I ran the same configuration 4 times, and plotted the results:

As you can see, results vary a lot between runs.

How can I set up a training session in Keras to ensure I get reasonably similar results when training in a GPU? Is this even possible?

The full training script can be found here.

Some of my colleagues are using just pure TF, and their results seem far more consistent. What is more, they do not seem to be seeding any randomness except to ensure that the train and validation split is always the same.

解决方案

Keras + Tensorflow.

Step 1, disable GPU.

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = ""

Step 2, seed those libraries which are included in your code, say "tensorflow, numpy, random".

import tensorflow as tf
import numpy as np
import random as rn

sd = 1 # Here sd means seed.
np.random.seed(sd)
rn.seed(sd)
os.environ['PYTHONHASHSEED']=str(sd)

from keras import backend as K
config = tf.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)
tf.set_random_seed(sd)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
K.set_session(sess)

Make sure these two pieces of code are included at the start of your code, then the result will be reproducible.

这篇关于构建 Keras 项目以在 GPU 中实现可重现的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！