在分布式 TensorFlow 中，是否可以在不同的工作人员之间共享相同的队列? [英] In distributed TensorFlow, is it possible to share the same queue across different workers?

问题描述

在 TensorFlow 中，我希望在不同机器上的不同工作人员之间共享一个文件名队列，这样每台机器都可以获得要训练的文件子集.查了很多，貌似只有变量可以放到一个PS任务上共享.有人有任何例子吗?谢谢.

In TensorFlow, I want to have a filename queue shared across different workers on different machines, such that each machine can get a subset of files to train. I searched a lot, and it seems that only variables could be put on a PS task to be shared. Does anyone have any example? Thanks.

推荐答案

通过在创建队列时设置可选的 shared_name 参数，可以跨工作人员共享同一个队列.与 tf.Variable 对象一样，您可以将队列放置在可以从不同工作人员访问的任何设备上.例如:

It is possible to share the same queue across workers, by setting the optional shared_name argument when creating the queue. Just as with tf.Variable objects, you can place the queue on any device that can be accessed from different workers. For example:

with tf.device("/job:ps/task:0"):  # Place queue on parameter server.
  q = tf.FIFOQueue(..., shared_name="shared_queue")

一些注意事项:

• shared_name 的值对于您共享的特定队列必须是唯一的.不幸的是，Python API 目前没有使用范围界定或自动名称统一来简化此操作，因此您必须手动确保这一点.

• The value for shared_name must be unique to the particular queue that you are sharing. Unfortunately, the Python API does not currently use scoping or automatic name uniqification to make this easier, so you will have to ensure this manually.

您不需要将队列放在参数服务器上.一种可能的配置是设置一个额外的输入作业"(例如 "/job:input")，其中包含一组执行预处理的任务，并为工作人员导出一个共享队列使用.

You do not need to place the queue on a parameter server. One possible configuration would be to set up an additional "input job" (e.g. "/job:input") containing a set of tasks that perform pre-processing, and export a shared queue for the workers to use.

这篇关于在分布式 TensorFlow 中，是否可以在不同的工作人员之间共享相同的队列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！