TensorFlow:极简程序在分布式模式下失败 [英] TensorFlow: minimalist program fails on distributed mode
问题描述
我写了一个非常简单的程序,它在没有分发的情况下运行得很好,但在分布式模式下挂在 CheckpointSaverHook
上(尽管我的本地主机上的一切!).我看到有一些关于在分布式模式下挂起的问题,但似乎没有一个与我的问题相符.
I wrote a very simple program that runs just fine without distribution but hangs on CheckpointSaverHook
in distributed mode (everything on my localhost though!). I've seen there's been a few questions about hanging in distributed mode, but none seem to match my question.
这是脚本(使用新的图层 API 制作):
Here's the script (made to toy with the new layers API):
import numpy as np
import tensorflow as tf
from tensorflow.contrib.learn.python.learn import learn_runner
from tensorflow.contrib import layers
DATA_SIZE=10
DIMENSION=5
FEATURES='features'
def generate_input_fn():
def _input_fn():
mid = int(DATA_SIZE/2)
data = np.array([np.ones(DIMENSION) if x < mid else -np.ones(DIMENSION) for x in range(DATA_SIZE)])
labels = ['0' if x < mid else '1' for x in range(DATA_SIZE)]
table = tf.contrib.lookup.string_to_index_table_from_tensor(tf.constant(['0', '1']))
label_tensor = table.lookup(tf.convert_to_tensor(labels, dtype=tf.string))
return dict(zip([FEATURES], [tf.convert_to_tensor(data, dtype=tf.float32)])), label_tensor
return _input_fn
def build_estimator(model_dir):
features = layers.real_valued_column(FEATURES, dimension=DIMENSION)
return tf.contrib.learn.DNNLinearCombinedClassifier(
model_dir=model_dir,
dnn_feature_columns=[features],
dnn_hidden_units=[20,20])
def generate_exp_fun():
def _exp_fun(output_dir):
return tf.contrib.learn.Experiment(
build_estimator(output_dir),
train_input_fn=generate_input_fn(),
eval_input_fn=generate_input_fn(),
train_steps=100
)
return _exp_fun
if __name__ == '__main__':
tf.logging.set_verbosity(tf.logging.DEBUG)
learn_runner.run(generate_exp_fun(), 'job_dir')
要测试分布式模式,我只需使用环境变量启动它 TF_CONFIG={"cluster": {"ps":["localhost:5040"], "worker":["localhost:5041"]}, "task":{"type":"worker","index":0}, "environment": "local"}
(这是给worker的,同ps
code> 类型用于启动参数服务器.
To test distributed mode, I simply launch it with the environment variable TF_CONFIG={"cluster": {"ps":["localhost:5040"], "worker":["localhost:5041"]}, "task":{"type":"worker","index":0}, "environment": "local"}
(this is for the worker, the same with ps
type is used to launch the parameter server.
我在 windows-64 上使用 tensorflow-1.0.1(但与 1.0.0 具有相同的行为),仅使用 CPU.我实际上从未收到任何错误,它只是在 INFO:tensorflow:Create CheckpointSaverHook.
之后一直挂起.我无法打印本机部分发生的事情的堆栈.
I use tensorflow-1.0.1 (but had the same behavior with 1.0.0) on windows-64, only CPU. I actually never get any error, it just hang on after INFO:tensorflow:Create CheckpointSaverHook.
forever... I've tried to attach VisualStudio C++ debugger to the process but with little success so far, so I can't print a stack for what's happening in the native part.
P.S.:DNNLinearCombinedClassifier
不是问题,因为使用简单的 tf.contrib.learn.LinearClassifier
也会失败.正如评论中所指出的,这不是因为两个进程都在 localhost
上运行,因为它在单独的 VM 上运行时也会失败.
P.S.: it's not a problem with DNNLinearCombinedClassifier
because it fails as well with a simple tf.contrib.learn.LinearClassifier
. And as noted in the comments, it's not due to both process running on localhost
, since it fails also when running on separate VMs.
我认为服务器启动实际上存在问题.当您处于本地模式时(无论是否分布式),服务器似乎未启动,参见.tensorflow/contrib/learn/python/learn/experiment.py l.250-258
:
I think there's actually an issue with server launching. It looks like the server is not launched when you're in local mode (no matter if distributed or not), cf. tensorflow/contrib/learn/python/learn/experiment.py l.250-258
:
# Start the server, if needed. It's important to start the server before
# we (optionally) sleep for the case where no device_filters are set.
# Otherwise, the servers will wait to connect to each other before starting
# to train. We might as well start as soon as we can.
config = self._estimator.config
if (config.environment != run_config.Environment.LOCAL and
config.environment != run_config.Environment.GOOGLE and
config.cluster_spec and config.master):
self._start_server()
这将阻止服务器以本地模式为工作人员启动......有人知道这是一个错误还是我遗漏了什么?
This will prevent the server from being started in local mode for the workers... Anyone has an idea if it's a bug or there's something I'm missing?
推荐答案
So this has been answer in: https://github.com/tensorflow/tensorflow/issues/8796.最后,任何分布式操作都应该使用CLOUD
.
So this has been answered in: https://github.com/tensorflow/tensorflow/issues/8796. Finally, one should use CLOUD
for any distributed operation.
这篇关于TensorFlow:极简程序在分布式模式下失败的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!