Python:线程+锁定大大降低了我的应用程序运行速度 [英] Python: threading + lock slows my app down considerably

问题描述

说我有一个写入文件的函数.我也有一个循环循环读取该文件的功能.我有这两个函数在单独的线程中运行. (实际上，我正在通过MDIO读取/写入寄存器，这就是为什么我不能让两个线程同时执行，只能一个或另一个同时执行，但是为了简单起见，我们只说它是一个文件)

现在，当我单独运行write函数时，它的执行速度非常快.但是，当我运行线程并在运行之前让它获取锁时，它的运行速度似乎非常慢.这是因为第二个线程(读取功能)正在轮询以获取锁吗?有什么办法可以解决这个问题?

我目前仅使用一个简单的RLock，但是可以进行任何会提高性能的更改.

作为一个例子，我将举个例子说明发生了什么.读线程基本上总是在运行，但是偶尔会有一个单独的线程来调用加载.如果我通过从cmd提示符运行负载进行基准测试，则在线程中运行的速度至少要慢3倍.

写线程:

import usbmpc # functions I made which access dll functions for hardware, etc

def load(self, lock):
    lock.acquire()
    f = open('file.txt','r')
    data = f.readlines()
    for x in data: 
        usbmpc.write(x)
    lock.release()

读取线程:

import usbmpc

def read(self, lock): 
    addr = START_ADDR
    while True: 
        lock.acquire()
        data = usbmpc.read(addr)
        lock.release()
        addr += 4
        if addr > BUF_SIZE: addr = START_ADDR

解决方案

您是否在多核计算机上使用线程?

如果答案是肯定的，那么除非您的Python版本是3.2+，否则运行线程应用程序时，性能将降低.

David Beazly 投入了大量精力来寻找多核GIL的发展趋势，并做出了我们其他人也很容易理解它.检查他的网站和那里的资源.另外，您可能希望在PyCon 2010上看到他的演示文稿.这很有趣.

长话短说，在Python 3.2中，Antoine Pitrou 编写一个在单核和多核计算机上具有相同性能的新GIL .在以前的版本中，您拥有的内核/线程越多，性能损失就会增加...

希望它会有所帮助:)

Say I have a function that writes to a file. I also have a function that loops repeatedly reading from said file. I have both of these functions running in separate threads. (Actually I am reading/writing to registers via MDIO which is why I can't have both threads executing concurrently, only one or the other, but for the sake of simplicity, let's just say it's a file)

Now when I run the write function in isolation, it executes fairly quickly. However when I'm running threaded and have it acquire a lock before running, it seems to run extremely slow. Is this because the second thread (read function) is polling to acquire the lock? Is there any way to get around this?

I am currently just using a simple RLock, but am open to any change that would increase performance.

Edit: As an example, I will put a basic example of what's going on. The read thread is basically always running, but occasionally a separate thread will make a call to load. If I benchmark by running load from cmd prompt, running in a thread is at least 3x slower.

write thread:

import usbmpc # functions I made which access dll functions for hardware, etc

def load(self, lock):
    lock.acquire()
    f = open('file.txt','r')
    data = f.readlines()
    for x in data: 
        usbmpc.write(x)
    lock.release()

read thread:

import usbmpc

def read(self, lock): 
    addr = START_ADDR
    while True: 
        lock.acquire()
        data = usbmpc.read(addr)
        lock.release()
        addr += 4
        if addr > BUF_SIZE: addr = START_ADDR

解决方案

Do you use threading on a multicore machine?

If the answer is yes, then unless your Python version is 3.2+ you are going to suffer reduced performance when running threaded applications.

David Beazly has put considerable effort to find what is going on with GIL on multicores and has made it easy for the rest of us to understand it too. Check his website and the resources there. Also you might want to see his presentation at PyCon 2010. It is rather intresting.

To make a long story short, in Python 3.2, Antoine Pitrou wrote a new GIL that has the same performance on single and multicore machines. In previous versions, the more cores/threads you have, the performance loss increases...

hope it helps :)

这篇关于Python:线程+锁定大大降低了我的应用程序运行速度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！