在Google Cloud上训练不同的Keras模型的结果 [英] Results of training a Keras model different on Google Cloud

问题描述

我创建了一个脚本来训练keras神经网络，并已在我的机器上成功运行了该脚本(在训练结束时，验证准确度约为0.8).但是，当我尝试在Google Cloud VM实例上运行完全相同的代码(对相同的数据)时，结果却大大恶化(〜0.2验证精度).

I've created a script to train a keras neural net and have run it successfully on my machine (at the end of training there is roughly 0.8 validation accuracy). However, when I try to run the exact same code (on the same data) on a Google Cloud VM instance I get drastically worse results (~0.2 validation accuracy).

Git状态确认VM中的存储库是最新版本的master(与我的本地计算机相同)，并且我已验证其tf和keras版本是最新的(与我的本地计算机相同).在导入Keras之前，我还设置了numpy和tensorflow随机种子.

Git status confirms that the repo in the VM is up to date with master (same with my local machine), and I have verified that its versions of tf and keras are up to date (and same as my local machine). I've also set the numpy and tensorflow random seeds before importing Keras.

有人以前遇到过这样的问题吗?我对可能导致这种情况的原因不知所措...我能想到的唯一区别是，我的机器运行的是Python 3.6，而VM运行的是Python 2.7.能否说明培训结果的巨大差异?

Has anyone run into a problem like this before? I'm at a loss for what could be causing this... the only difference I can think of is that my machine is running Python 3.6 whereas the VM is running Python 2.7. Could that account for the vast difference is training results?

推荐答案

我在tensorflow 1.10(当前的gcloud版本)中发现了Keras和Estimator API之间的错误交互，但在> = 1.11中却没有(我在本地使用的是)

I found a buggy interaction between Keras and the Estimator API in tensorflow 1.10 (current gcloud version), but not in >=1.11 (what I was using locally).

不确定是否适用于您(您是否使用Keras + Estimator和tensorflow> = 1.11来表示本地变量?)

Not sure if it applies to you (do you use Keras+Estimator and tensorflow >=1.11 for local?)

我在此处提交了错误报告: https://github.com/tensorflow/tensorflow/Issues/24299

I filed a bug report here: https://github.com/tensorflow/tensorflow/issues/24299

这篇关于在Google Cloud上训练不同的Keras模型的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！