分布式 Tensorflow:检查失败:size>=0 [英] Distributed Tensorflow: check failed: size>=0

问题描述

我使用的是 keras 2.0.6.tensorflow 的版本是 1.3.0.

I'm using keras 2.0.6. The version of tensorflow is 1.3.0.

我的代码可以在 theano 后端运行，但在 tensorflow 后端运行失败:

My code can run with theano backend, but failed with tensorflow backend:

F tensorflow/core/framework/tensor_shape.cc:241] 检查失败:大小 >= 0 (-14428307456 vs. 0)

F tensorflow/core/framework/tensor_shape.cc:241] Check failed: size >= 0 (-14428307456 vs. 0)

我想知道是否有人能想到可能导致这种情况的任何可能原因.

I was wondering if anyone can thought of any possible reason that might cause this.

谢谢！

----更新-----

----UPDATE-----

我使用 tensorflow 在我的 PC 上测试了完全相同的代码.它运行完美.

I tested exactly the same code on my PC with tensorflow. It runs perfectly.

但是，当我在超级计算机上运行它时，它会抛出此错误.

However, it throw out this error when I run it on a Supercomputer.

虽然这个错误看起来像溢出，但不可能不是在我的 PC 上溢出，而是在超级计算机上溢出.

Although this error looks like overflow, there is no way that it didn't overflow on my PC, but overflow on a supercomputer.

我怀疑它来自用于分布式计算的 tensorflow 上的错误.

I suspect that it comes from a bug on tensorflow for distributed computation.

推荐答案

它出现了同样的错误，但在我缩小批量大小之后它运行正常.

it came out the same bug, but it ran ok after that I shrimped the batch size.

我认为原因是它的 GPU 内存不足.

I think the reason is it running out of GPU memories.

这篇关于分布式 Tensorflow:检查失败:size>=0的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！