流媒体命令输出进度 [英] Streaming commands output progress

问题描述

我正在编写一个服务，该服务必须将执行的命令的输出同时传输给父节点和日志。当进程很长时，问题是 cmd.StdoutPipe 给了我一个最终的（字符串）结果。

是否可以部分输出正在发生的事情，如shell中所示

  func main（）{
 cmd：= exec.Command（sh，-c，some long runnig task）
 
 stdout，_：= cmd.StdoutPipe（）
 cmd.Start（） 
 
 scanner：= bufio.NewScanner（stdout）
 for scanner.Scan（）{
m：= scanner.Text（）
 fmt.Println（m）
 log.Printf（m）
} 
 
 cmd.Wait（）
} 
  

PS只是输出将是：

  cmd.Stdout = os.Stdout 
  

但在我的情况下，这还不够。

解决方案

您发布的代码可以工作（执行合理的命令）。

这是一个简单的长时间运行的任务调用并测试你的代码：

  func main（）{
 fmt.Println（Child started。） 
 time.Sleep（time.Second * 2）
 fmt.Println（Tick ...）
 time.Sleep（time.Second * 2）
 fmt.Println （Child ended。）
} 
  

编译并将其命名为您的命令。您会看到不同的行会立即显示为由子流程写入的流式传输。

为什么它不适用于您的原因

扫描仪 由 bufio.NewScanner（）返回如果遇到换行符（如 bufio.ScanLines（） 函数）。

如果执行的命令不打印换行符，它的输出将不会立即返回（只有当打印换行符时，内部缓冲区已填满或进程结束）。

可能的解决方法

如果您不能保证子进程打印换行符，但您仍然希望流转th e输出，您无法读取整行。一种解决方案是用文字阅读，甚至用字符（符文）阅读。您可以使用拆分功能来实现此目的。 //golang.org/pkg/bufio/#Scanner.Splitrel =nofollow noreferrer> Scanner.Split（） 方法：

  scanner：= bufio.NewScanner（stdout）
 scanner.Split（bufio.ScanRunes）
  

bufio.ScanRunes 函数通过 rune s读取输入，所以 Scanner.Scan（） 会在新的符文可用。

或者在没有扫描仪的情况下手动读取（在本例中每个字节）：

$ $ $
$ $ b _，err：= stdout.Read（oneByte）
if err！= nil {
break
}
fmt.Printf（％c，oneByte [0]）
}

注上面的代码会读取 rune s，表明UTF-8编码中的多个字节不正确。要读取多个UTF-8字节符文，我们需要一个更大的缓冲区：

  oneRune：= make（[] byte，utf8 .UTFMax）
为{
 count，err：= stdout.Read（oneRune）
 if err！= nil {
 break 
} 
 fmt。 Printf（％s，oneRune [：count]）
} 
  

考虑到标准输出和标准错误（通常是几KB的大小），流程有默认的缓冲区。如果一个进程写入标准输出或标准错误，它将进入相应的缓冲区。如果此缓冲区已满，则进一步写入将阻止（在子进程中）。如果您没有读取子进程的标准输出和标准错误，那么如果缓冲区已满，您的子进程可能会挂起。

因此建议始终阅读标准输出和子进程的错误。即使您知道该命令不会正常写入其标准错误，但如果发生某些错误，它可能会开始将错误消息转储到其标准错误。

编辑：由于Dave C默认提到子进程的标准输出和错误流被丢弃，如果不读取则不会导致块/挂起。但是，由于没有阅读错误流，您可能会错过流程中的一两件事。

I'm writing a service that has to stream output of a executed command both to parent and to log. When there is a long process, the problem is that cmd.StdoutPipe gives me a final (string) result.

Is it possible to give partial output of what is going on, like in shell

func main() {
    cmd := exec.Command("sh", "-c", "some long runnig task")

    stdout, _ := cmd.StdoutPipe()
    cmd.Start()

    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
        log.Printf(m)
    }

    cmd.Wait()
}

P.S. Just to output would be:

cmd.Stdout = os.Stdout

But in my case it is not enough.

解决方案

The code you posted works (with a reasonable command executed).

Here is a simple "some long running task" written in Go for you to call and test your code:

func main() {
    fmt.Println("Child started.")
    time.Sleep(time.Second*2)
    fmt.Println("Tick...")
    time.Sleep(time.Second*2)
    fmt.Println("Child ended.")
}

Compile it and call it as your command. You will see the different lines appear immediately as written by the child process, "streamed".

Reasons why it may not work for you

The Scanner returned by bufio.NewScanner() reads whole lines and only returns something if a newline character is encountered (as defined by the bufio.ScanLines() function).

If the command you execute doesn't print newline characters, its output won't be returned immediately (only when newline character is printed, internal buffer is filled or the process ends).

Possible workarounds

If you have no guarantee that the child process prints newline characters but you still want to stream the output, you can't read whole lines. One solution is to read by words, or even read by characters (runes). You can achieve this by setting a different split function using the Scanner.Split() method:

scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanRunes)

The bufio.ScanRunes function reads the input by runes so Scanner.Scan() will return whenever a new rune is available.

Or reading manually without a Scanner (in this example byte-by-byte):

oneByte := make([]byte, 1)
for {
    _, err := stdout.Read(oneByte)
    if err != nil {
        break
    }
    fmt.Printf("%c", oneByte[0])
}

Note that the above code would read runes that multiple bytes in UTF-8 encoding incorrectly. To read multi UTF-8-byte runes, we need a bigger buffer:

oneRune := make([]byte, utf8.UTFMax)
for {
    count, err := stdout.Read(oneRune)
    if err != nil {
        break
    }
    fmt.Printf("%s", oneRune[:count])
}

Things to keep in mind

Processes have default buffers for standard output and for standard error (usually the size of a few KB). If a process writes to the standard output or standard error, it goes into the respective buffer. If this buffer gets full, further writes will block (in the child process). If you don't read the standard output and standard error of a child process, your child process may hang if the buffer is full.

So it is recommended to always read both the standard output and error of a child process. Even if you know that the command don't normally write to its standard error, if some error occurs, it will probably start dumping error messages to its standard error.

Edit: As Dave C mentions by default the standard output and error streams of the child process are discarded and will not cause a block / hang if not read. But still, by not reading the error stream you might miss a thing or two from the process.

这篇关于流媒体命令输出进度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！