在Python线程中运行子进程以实时读取输出 [英] Run subprocess inside Python thread reading the output in realtime

问题描述

考虑以下Python代码:

import io
import time
import subprocess
import sys

from thread import start_new_thread

def ping_function(ip):

    filename = 'file.log'
    command = ["ping", ip]

    with io.open(filename, 'wb') as writer, io.open(filename, 'rb', 1) as reader:
        process = subprocess.Popen(command, stdout=writer)
        while process.poll() is None:
            line = reader.read()
            # Do something with line
            sys.stdout.write(line)
            time.sleep(0.5)
        # Read the remaining
        sys.stdout.write(reader.read())

ping_function("google.com")

目标是运行shell命令(在本例中为 ping ，但此处与此无关)，并实时处理输出，该输出也保存在日志文件中. /p>

换句话说， ping 在后台运行，并且每秒在终端上产生输出.我的代码将读取此输出(每0.5秒)一次，对其进行解析并实时(几乎)采取一些措施.

这里的实时表示我不想等待过程结束来读取输出.在这种情况下，实际上 ping 永远不会完成，因此像我刚刚描述的方法是强制性的.

我已经测试了上面的代码，它实际上可以正常运行:)

现在我想在一个单独的线程中对此进行调整，因此我用以下内容替换了最后一行:

from thread import start_new_thread
start_new_thread(ping_function, ("google.com", ))

由于某种原因，它不再起作用，并且阅读器始终返回空字符串. 特别是， reader.read()返回的字符串始终为空.

使用Queue或其他全局变量将无济于事，因为我什至在检索数据时都遇到问题(即获取shell命令的输出)

我的问题是:

• 我该如何解释这种行为?

• 在单独的线程中运行进程是一个好主意还是应该使用其他方法? 本文表示不是. .

• 如何修复代码?

谢谢！

解决方案

启动线程后切勿分叉.您可以在启动叉子后进行穿线，因此可以让线程处理I/O管道，但是...

让我重复一遍:启动线程后切勿分叉

那篇文章很好地解释了它.启动线程后，您将无法控制程序的状态.尤其是在Python中，事情在后台进行.

要修复代码，只需从主线程启动子进程，然后开始线程化.在线程中处理来自管道的I/O完全可以.

Consider the following Python code:

import io
import time
import subprocess
import sys

from thread import start_new_thread

def ping_function(ip):

    filename = 'file.log'
    command = ["ping", ip]

    with io.open(filename, 'wb') as writer, io.open(filename, 'rb', 1) as reader:
        process = subprocess.Popen(command, stdout=writer)
        while process.poll() is None:
            line = reader.read()
            # Do something with line
            sys.stdout.write(line)
            time.sleep(0.5)
        # Read the remaining
        sys.stdout.write(reader.read())

ping_function("google.com")

The goal is to run a shell command (in this case ping, but it is not relevant here) and to process the output in real time, which is also saved on a log file.

In other word, ping is running in background and it produces output on the terminal every second. My code will read this output (every 0.5 seconds), parse it and take some action in (almost) real time.

Realtime here means that I don't want to wait the end of the process to read the output. In this case actually ping never completes so an approach like the one I have just described is mandatory.

I have tested the code above and it actually works OK :)

Now I'd like to tun this in a separate thread, so I have replaced the last line with the following:

from thread import start_new_thread
start_new_thread(ping_function, ("google.com", ))

For some reason this does not work anymore, and the reader always return empty strings. In particular, the string returned by reader.read() is always empty.

Using a Queue or another global variable is not going to help, because I am having problems even to retrieve the data in the first place (i.e. to obtain the output of the shell command)

My questions are:

• How can I explain this behavior?

• Is it a good idea to run a process inside a separate thread or I should use a different approach? This article suggests that it is not...

• How can I fix the code?

Thanks!

解决方案

You should never fork after starting threads. You can thread after starting a fork, so you can have a thread handle the I/O piping, but...

Let me repeat this: You should never fork after starting threads

That article explains it pretty well. You don't have control over the state of your program once you start threads. Especially in Python with things going on in the background.

To fix your code, just start the subprocess from the main thread, then start threading. It's perfectly OK to process the I/O from the pipes in a thread.

这篇关于在Python线程中运行子进程以实时读取输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！