python多线程进程，用于运行可执行文件 [英] python multiple threaded processes for running executables

问题描述

我正在尝试升级一个Python脚本，该脚本在Windows上运行一个可执行文件，并将文本输出文件管理为使用多个线程进程的版本，因此我可以利用多个内核.我有四个独立版本的可执行文件，每个线程都可以访问该版本.这部分工作正常.我遇到问题的地方是它们同时运行并尝试打开(不同的)输出文件，以确保它们正确运行并根据输出文件的内容做出反应.

I am trying to upgrade a python script that runs an executable on windows and manages the text output files to a version that uses multiple threaded processes so I can utilize more than one core. I have four separate versions of the executable which each thread knows to access. This part works fine. Where I run into problems is when they are running simultaneously and try to open the (different) output files to ensure they ran correctly and react depending on the contents of the output file.

具体来说，当运行三个线程时，两个将崩溃并显示以下错误，而一个继续工作:

Specifically, when running three threads, two will crash with the following error, while one continues to work:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 552, in __bootstrap_inner
    self.run()
  File "E:\HDA\HDA-1.0.1\Hm-1.0.1.py", line 782, in run
    conf = self.conf_file(Run)
  File "E:\HDA\HDA-1.0.1\Hm-1.0.1.py", line 729, in conf_file
    l = open(self.run_dir(Run)+Run, 'r').readlines()     #list of file lines
IOError: [Errno 2] No such file or directory: 'Path/to/Outputfile'

这是由于线程未正确运行可执行文件导致的(即为什么未创建"Path/to/Outputfile"，因此无法找到它).但是其中一个线程可以正确执行此操作，而其他两个则不能.为什么我不能让多个线程运行不同版本的可执行文件?

This results from the thread not correctly running the executable (i.e. why 'Path/to/Outputfile' was not created and hence can't be found). But one of the threads does this correctly while the other two cannot. Is there a reason why I can't get multiple threads running different versions of an executable?

推荐答案

我不认为GIL本身不会杀死它，除非打开文件使您陷入一些怪异的死锁或自旋锁状态. 通常，在这种情况下，如果您受到I/O约束，您会想要线程.实际上，线程能够同时运行的事实可能会导致其他线程失败，而不是多次成功打开文件.

I don't think GIL by itself wouldn't kill this by itself unless opening a file gets you into some weird deadlock or spinlock condition. In general, you want threads in cases like this where you're I/O-bound. In fact, the fact that the threads are able to run concurrently probably contributes to the other threads failing rather than successfully opening a file several times.

在本演示文稿的幻灯片15中，作者指出GIL在阻止I/O调用，使其他线程有机会.

On slide fifteen of this presentation, the author points out that the GIL releases on blocking I/O calls to give other threads a chance.

这里的真正问题似乎是对文件资源的锁定.我不太确定Windows的工作原理，所以我无法说出为什么这个错误会逐渐蔓延，但是似乎只有一个线程实际上在文件资源上有锁.

The real problem here seems to be a lock on a file resource. I'm not really sure about how Windows works, so I can't speak to why this error is creeping up, but it seems like only one thread actually has a lock on a file resource.

另一个发布者关于多个内核和GIL的观点可能正在发挥作用，因为在其他两个线程都处于饥饿状态时，您可能会进行某种优先级倒置，但是鉴于上述介绍，我认为这不太可能说在阻塞操作中间的线程将锁释放给其他线程.

The other poster's point about multiple cores and the GIL might be coming into play, in that you could have some sort of priority inversion going on where the other two threads are getting starved, but I find it unlikely given that the above presentation says that threads in the middle of a blocking operation free the lock for other threads.

一种想法是尝试 multiprocessing .我怀疑通过跨多个进程而不是线程读取文件会更好.

One thought is to try multiprocessing. I suspect you'll have better luck with reading the file across multiple processes rather than with threads.

这是我在OS 10.7.3计算机上编写并尝试过的示例，它打开一个文件test，其内容为lol\n:

Here is an example I wrote and tried on my OS 10.7.3 machine, it opens up a file test whose contents are lol\n:

import multiprocessing
import os

def open_file(x):
   with open(x, 'r') as file_obj:
     return file_obj.readlines()

a = multiprocessing.Pool(4)
print a.map(open_file, ['test']*4)

执行结果如下:

➜  ~ git:(master) ✗ python open_test.py
[['lol\n'], ['lol\n'], ['lol\n'], ['lol\n']]

这篇关于python多线程进程，用于运行可执行文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！