Tensorflow和线程 [英] Tensorflow and threading

问题描述

下面是Tensorflow网站上的简单mnist教程(即单层softmax)，我尝试通过多线程训练步骤对其进行扩展:

Below is the simple mnist tutorial (i.e. single layer softmax) from the Tensorflow website, which I tried to extend with a multi-threaded training step:

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import threading

# Training loop executed in each thread
def training_func():
  while True:
      batch = mnist.train.next_batch(100)
      global_step_val,_ = sess.run([global_step, train_step], feed_dict={x: batch[0], y_: batch[1]})
      print("global step: %d" % global_step_val)
      if global_step_val >= 4000:
        break

# create session and graph
sess = tf.Session()

x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])

W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
global_step = tf.Variable(0, name="global_step")
y = tf.matmul(x,W) + b

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y, y_))

inc = global_step.assign_add(1)
with tf.control_dependencies([inc]):
  train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# initialize graph and create mnist loader
sess.run(tf.global_variables_initializer())
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

# create workers and execute threads
workers = []
for _ in range(8):
  t = threading.Thread(target=training_func)
  t.start()
  workers.append(t)

for t in workers:
  t.join()

# evaluate accuracy of the model
print(accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels},
  session=sess))

我一定会缺少一些东西，因为下面的8个线程会产生不一致的结果(精度大约= 0.1)，而只有1个线程只能获得预期的精度(大约0.92).有人知道我的错误吗?谢谢！

I must be missing something, as 8 threads as below yield inconsistent results (accuracy approx. = 0.1), when with 1 thread only the expected accuracy is obtained (approx. 0.92). Does anybody have a clue about my mistake(s)? Thanks!

推荐答案

请注意，由于

Note that unfortunately, threadingwith python doesn't create real parallelism because of the GIL. So what happens here is that you will have multiple threads which are all running on the same CPU where in reality they are running sequentially. Therefore, I would suggest using Coordinator in Tensorflow. More information about Coordinator can be found here:

https://www.tensorflow.org/programmers_guide/threading_and_queues
https://www.tensorflow.org/programmers_guide/reading_data

最后，我建议您说:

with tf.device('/cpu:0'):
    your code should go here... 'for the first thread'

然后为另一个线程使用另一个cpu，依此类推... 希望这个答案能找到你！！

Then use another cpu for the other thread and so on... Hope this answer finds you well!!

这篇关于Tensorflow和线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！