如何在不阻塞Rust的情况下读取子进程的输出? [英] How do I read the output of a child process without blocking in Rust?
问题描述
我正在Rust中制作一个小的ncurses应用程序，该应用程序需要与子进程进行通信.我已经有一个用Common Lisp编写的原型.我试图重写它，因为CL占用了这么小的工具大量的内存.
我在弄清楚如何与子流程进行交互时遇到了一些麻烦.
我目前正在做的大致是这样:
-
创建过程:
let mut program = match Command::new(command) .args(arguments) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::piped()) .spawn() { Ok(child) => child, Err(_) => { println!("Cannot run program '{}'.", command); return; } }; -
将其传递到一个无限(直到用户退出)循环，该循环读取和处理输入，并监听这样的输出(并将其写入屏幕):
fn listen_for_output(program: &mut Child, output_viewer: &TextViewer) { match program.stdout { Some(ref mut out) => { let mut buf_string = String::new(); match out.read_to_string(&mut buf_string) { Ok(_) => output_viewer.append_string(buf_string), Err(_) => return, }; } None => return, }; }
对read_to_string的调用将阻止程序，直到进程退出.从我可以看到，read_to_end和read似乎也被阻止了.如果我尝试运行立即退出的ls之类的东西，它可以工作，但是对于没有退出的诸如python或sbcl的东西，它只有在我手动杀死该子进程后才继续.
基于此答案，我将代码更改为使用BufReader:
fn listen_for_output(program: &mut Child, output_viewer: &TextViewer) {
match program.stdout.as_mut() {
Some(out) => {
let buf_reader = BufReader::new(out);
for line in buf_reader.lines() {
match line {
Ok(l) => {
output_viewer.append_string(l);
}
Err(_) => return,
};
}
}
None => return,
}
}
但是，问题仍然存在.它将读取所有可用的行，然后进行阻止.由于该工具应该与任何程序一起使用，因此在尝试读取之前，无法猜测输出何时结束.似乎也没有办法为BufReader设置超时.
默认情况下，流被阻止. TCP/IP流，文件系统流，管道流都被阻塞.当您告诉流为您提供大量字节时，它将停止并等待，直到它具有给定的字节数或发生其他事情为止(
当您需要处理多个流(例如子进程的 类似地，当您不希望程序无限期地等待子进程时，也不能使用 如果您的操作系统恰好不急于将数据返回给进程(优选完全读取"而不是短读取")，则不能使用基于 您可能想知道为什么在这里分块读取数据如此重要，为什么 要在不阻塞的情况下等待流，您需要一个非阻塞流. MIO 承诺为管道提供所需的非阻塞流支持，最有可能使用
在没有非阻塞流的情况下，您将不得不使用生成线程，以便阻塞读取将在单独的线程中执行，因此不会阻塞您的主线程.您可能还希望逐字节读取流，以便在操作系统不喜欢短读取"的情况下立即对行分隔符做出反应.这是一个工作示例: https://gist.github.com/ArtemGr/db40ae04b431a95f2b78 ./p>
P.S.这是一个函数示例，该函数允许通过共享的字节向量监视程序的标准输出: 与几个助手一起，我正在使用它来控制SSH会话: P.S.请注意，读取的 I'm making a small ncurses application in Rust that needs to communicate with a child process. I already have a prototype written in Common Lisp. I'm trying to rewrite it because CL uses a huge amount of memory for such a small tool. I'm having some trouble figuring out how to interact with the sub-process. What I'm currently doing is roughly this: Create the process: Pass it to an infinite (until user exits) loop, which reads and handles input and listens for output like this (and writes it to the screen): The call to Based on this answer, I changed the code to use However, the problem still remains the same. It will read all lines that are available, and then block. Since the tool is supposed to work with any program, there is no way to guess out when the output will end, before trying to read. There doesn't appear to be a way to set a timeout for Streams are blocking by default. TCP/IP streams, filesystem streams, pipe streams, they are all blocking. When you tell a stream to give you a chunk of bytes it will stop and wait till it has the given amout of bytes or till something else happens (an interrupt, an end of stream, an error). The operating systems are eager to return the data to the reading process, so if all you want is to wait for the next line and handle it as soon as it comes in then the method suggested by Shepmaster in Unable to pipe to or from spawned child process more than once (and also in his answer here) works. This simple Similarly, you can't use You can't use You might be wondering why reading data in chunks is so important here, why can't the To wait on a stream without blocking you'd need a non-blocking stream. MIO promises to have the required non-blocking stream support for pipes, most probably with PipeReader, but I haven't checked it out so far. The non-blocking nature of a stream should make it possible to read data in chunks regardless of whether the operating system prefers the "short reads" or not. Because non-blocking stream never blocks. If there is no data in the stream it simply tells you so. In the absense of a non-blocking stream you'll have to resort to spawning threads so that the blocking reads would be performed in a separate thread and thus won't block your primary thread. You might also want to read the stream byte by byte in order to react to the line separator immediately in case the operating system does not prefer the "short reads". Here's a working example: https://gist.github.com/ArtemGr/db40ae04b431a95f2b78. P.S. Here's an example of a function that allows to monitor the standard output of a program via a shared vector of bytes: With a couple of helpers I'm using it to control an SSH session: P.S. Note that 这篇关于如何在不阻塞Rust的情况下读取子进程的输出?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！stdout和stderr)或多个进程时，这种基于BufReader的简单方法变得更加危险.例如，当子进程等待您清空其stderr管道而阻塞您的进程等待其为空的stdout时，基于BufReader的方法可能会死锁.BufReader.也许您想在孩子还在工作时显示进度条或计时器，而没有任何输出.BufReader的方法，因为在这种情况下，最后几行子进程打印的结果可能会出现在灰色区域:操作系统已将其捕获，但它们的大小不足以填充BufReader的缓冲区.BufReader仅限于Read接口允许它处理流的内容，它的阻塞程度不亚于基础流.为了提高效率，它会阅读以分块形式输入，告诉操作系统将其缓冲区尽可能多地填满.BufReader不能只逐字节读取数据.问题是要从流中读取数据，我们需要操作系统的帮助.另一方面，我们不是操作系统，我们与它隔离工作，以免在我们的过程出现问题时避免混乱.因此，为了调用操作系统，需要过渡到内核模式"，这也可能会导致上下文切换".这就是为什么调用操作系统读取每个字节很昂贵的原因.我们希望尽可能少地调用OS，因此我们可以分批获取流数据.use std::io::Read;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
/// Pipe streams are blocking, we need separate threads to monitor them without blocking the primary thread.
fn child_stream_to_vec<R>(mut stream: R) -> Arc<Mutex<Vec<u8>>>
where
R: Read + Send + 'static,
{
let out = Arc::new(Mutex::new(Vec::new()));
let vec = out.clone();
thread::Builder::new()
.name("child_stream_to_vec".into())
.spawn(move || loop {
let mut buf = [0];
match stream.read(&mut buf) {
Err(err) => {
println!("{}] Error reading from stream: {}", line!(), err);
break;
}
Ok(got) => {
if got == 0 {
break;
} else if got == 1 {
vec.lock().expect("!lock").push(buf[0])
} else {
println!("{}] Unexpected number of bytes: {}", line!(), got);
break;
}
}
}
})
.expect("!thread");
out
}
fn main() {
let mut cat = Command::new("cat")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("!cat");
let out = child_stream_to_vec(cat.stdout.take().expect("!stdout"));
let err = child_stream_to_vec(cat.stderr.take().expect("!stderr"));
let mut stdin = match cat.stdin.take() {
Some(stdin) => stdin,
None => panic!("!stdin"),
};
}
try_s! (stdin.write_all (b"echo hello world\n"));
try_s! (wait_forˢ (&out, 0.1, 9., |s| s == "hello world\n"));
await 在async-std中的调用也处于阻塞状态.它只是阻塞一个期货链(本质上是一个无堆栈的绿色线程)，而不是阻塞系统线程. poll_read 是非阻塞接口.在 async-std#499 中，我问过开发人员是否可以从这些API获得简短的阅读保证.
let mut program = match Command::new(command)
.args(arguments)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
{
Ok(child) => child,
Err(_) => {
println!("Cannot run program '{}'.", command);
return;
}
};
fn listen_for_output(program: &mut Child, output_viewer: &TextViewer) {
match program.stdout {
Some(ref mut out) => {
let mut buf_string = String::new();
match out.read_to_string(&mut buf_string) {
Ok(_) => output_viewer.append_string(buf_string),
Err(_) => return,
};
}
None => return,
};
}
read_to_string however blocks the program until the process exits. From what I can see read_to_end and read also seem to block. If I try running something like ls which exits right away, it works, but with something that doesn't exit like python or sbcl it only continues once I kill the subprocess manually.BufReader: fn listen_for_output(program: &mut Child, output_viewer: &TextViewer) {
match program.stdout.as_mut() {
Some(out) => {
let buf_reader = BufReader::new(out);
for line in buf_reader.lines() {
match line {
Ok(l) => {
output_viewer.append_string(l);
}
Err(_) => return,
};
}
}
None => return,
}
}
BufReader either.
Though in theory it doesn't have to work, because an operating system is allowed to make the BufReader wait for more data in read, but in practice the operating systems prefer the early "short reads" to waiting.BufReader-based approach becomes even more dangerous when you need to handle multiple streams (like the stdout and stderr of a child process) or multiple processes. For example, BufReader-based approach might deadlock when a child process waits for you to drain its stderr pipe while your process is blocked waiting on it's empty stdout.BufReader when you don't want your program to wait on the child process indefinitely. Maybe you want to display a progress bar or a timer while the child is still working and gives you no output.BufReader-based approach if your operating system happens not to be eager in returning the data to the process (prefers "full reads" to "short reads") because in that case a few last lines printed by the child process might end up in a gray zone: the operating system got them, but they're not large enough to fill the BufReader's buffer.BufReader is limited to what the Read interface allows it to do with the stream, it's no less blocking than the underlying stream is. In order to be efficient it will read the input in chunks, telling the operating system to fill as much of its buffer as it has available.BufReader just read the data byte by byte. The problem is that to read the data from a stream we need the operating system's help. On the other hand, we are not the operating system, we work isolated from it, so as not to mess with it if something goes wrong with our process. So in order to call to the operating system there needs to be a transition to "kernel mode" which might also incur a "context switch". That is why calling the operating system to read every single byte is expensive. We want as few OS calls as possible and so we get the stream data in batches.use std::io::Read;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
/// Pipe streams are blocking, we need separate threads to monitor them without blocking the primary thread.
fn child_stream_to_vec<R>(mut stream: R) -> Arc<Mutex<Vec<u8>>>
where
R: Read + Send + 'static,
{
let out = Arc::new(Mutex::new(Vec::new()));
let vec = out.clone();
thread::Builder::new()
.name("child_stream_to_vec".into())
.spawn(move || loop {
let mut buf = [0];
match stream.read(&mut buf) {
Err(err) => {
println!("{}] Error reading from stream: {}", line!(), err);
break;
}
Ok(got) => {
if got == 0 {
break;
} else if got == 1 {
vec.lock().expect("!lock").push(buf[0])
} else {
println!("{}] Unexpected number of bytes: {}", line!(), got);
break;
}
}
}
})
.expect("!thread");
out
}
fn main() {
let mut cat = Command::new("cat")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("!cat");
let out = child_stream_to_vec(cat.stdout.take().expect("!stdout"));
let err = child_stream_to_vec(cat.stderr.take().expect("!stderr"));
let mut stdin = match cat.stdin.take() {
Some(stdin) => stdin,
None => panic!("!stdin"),
};
}
try_s! (stdin.write_all (b"echo hello world\n"));
try_s! (wait_forˢ (&out, 0.1, 9., |s| s == "hello world\n"));
await on a read call in async-std is blocking as well. It's just instead of blocking a system thread it only blocks a chain of futures (a stack-less green thread essentially). The poll_read is the non-blocking interface. In async-std#499 I've asked the developers whether there's a short read guarantee from these APIs.
