python多线程比串行慢? [英] python multi-threading slower than serial?

问题描述

我正在尝试找出python中的多线程编程.这是我要比较串行和并行速度的简单任务.

I'm trying to figure out multi-threading programming in python. Here's the simple task with which I want to compare serial and parallel speeds.

import threading
import Queue
import time
import math

def sinFunc(offset, n):
  result = []
  for i in range(n):
    result.append(math.sin(offset + i * i))
  return result

def timeSerial(k, n):
  t1 = time.time()    
  answers = []
  for i in range(k):
    answers.append(sinFunc(i, n))
  t2 = time.time()
  print "Serial time elapsed: %f" % (t2-t1)

class Worker(threading.Thread):

  def __init__(self, queue, name):
    self.__queue = queue
    threading.Thread.__init__(self)
    self.name = name

  def process(self, item):
    offset, n = item
    self.__queue.put(sinFunc(offset, n))
    self.__queue.task_done()
    self.__queue.task_done()

  def run(self):
    while 1:
        item = self.__queue.get()
        if item is None:
            self.__queue.task_done()
            break
        self.process(item)

def timeParallel(k, n, numThreads):
  t1 = time.time()    
  queue = Queue.Queue(0)
  for i in range(k):
    queue.put((i, n))
  for i in range(numThreads):
    queue.put(None)    
  for i in range(numThreads):
    Worker(queue, i).start()
  queue.join()
  t2 = time.time()
  print "Serial time elapsed: %f" % (t2-t1)

if __name__ == '__main__':

  n = 100000
  k = 100
  numThreads = 10

  timeSerial(k, n)
  timeParallel(k, n, numThreads)

#Serial time elapsed: 2.350883
#Serial time elapsed: 2.843030

有人可以向我解释发生了什么吗?我已经习惯了C ++，使用此模块的类似版本可以看到我们期望的加速.

Can someone explain to me what's going on? I'm used to C++, and a similar version of this using the module sees the speed-up we would expect.

推荐答案

其他答案提到GIL是cpython中的问题.但是我觉得这里缺少一些信息.在线程中运行的代码受CPU约束的情况下，这将导致性能问题.就您的情况而言，是的，在线程中执行许多计算很可能会导致性能急剧下降.

Other answers have referred to the issue of the GIL being the problem in cpython. But I felt there was a bit of missing information. This will cause you performance issues in situations where the code you are running in threads is CPU bound. In your case here, yes doing many calculations in threads is going to most likely result in dramatically degraded performance.

但是，如果您执行的操作受到IO的更多限制，例如从网络应用程序中的许多套接字中读取数据，或者调出子进程，则可以从线程中获得性能提升.上面代码的一个简单示例是向外壳添加一个愚蠢的简单调用:

But, if you were doing something that was more IO bound, such as reading from many sockets in a network application, or calling out to subprocess, you can get performance increases from threads. A simple example for your code above would be to add a stupidly simple call out to the shell:

import os

def sinFunc(offset, n):
  result = []
  for i in xrange(n):
    result.append(math.sin(offset + i * i))
  os.system("echo 'could be a database query' >> /dev/null; sleep .1")
  return result

该调用可能确实是真正的事情，例如在文件系统上等待.但是您可以看到，在此示例中，线程将开始被证明是有益的，因为当线程在等待IO时可以释放GIL，而其他线程将继续处理.即便如此，当创建和同步它们的开销使更多的线程开始被否定时，仍然是一个甜蜜的地方.

That call might have been something real like waiting on the filesystem. But you can see that in this example, threading will start to prove beneficial, as the GIL can be released when the thread is waiting on IO and other threads will continue to process. Even so, there is still a sweet spot for when more threads start to become negated by the overhead of creating them and synchronizing them.

对于受CPU约束的代码，您可以使用多重处理

For CPU bound code, you would make use of multiprocessing

摘自文章: http://www.informit.com /articles/article.aspx?p=1850445&seqNum=9

...线程更适合于受I/O约束的应用程序(I/O版本 GIL，以实现更多的并发性)...

...threading is more appropriate for I/O-bound applications (I/O releases the GIL, allowing for more concurrency)...

关于线程与进程的类似问题参考:
https://stackoverflow.com/a/1227204/496445
https://stackoverflow.com/a/990436/496445

Similar question references about threads vs processes:
https://stackoverflow.com/a/1227204/496445
https://stackoverflow.com/a/990436/496445

这篇关于python多线程比串行慢?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！