如何正确地从大型过程输出中读取 [英] How to read from large process output correctly

问题描述

已经有很少 答案关于如何从流程流中进行读取的信息，但据我所知，它们并不涵盖从以下流程中读取的内容：

• 可以运行进行任何输出之前很长时间


• 在其生命周期结束时会产生巨大的输出（超出流缓冲区的容纳能力）


• 应该不超过给定时间范围运行


• 结果输出需要作为一个整体（长字符串）

因此，使用上述解决方案可能会导致流程周期浪费，因为即使没有输出，循环也会尝试读取流，或者由于解决方案无法打印其全部输出而导致解决方案未终止（由于到完整缓冲区）

我当前的解决方案如下所示（受批量读取大文件的解决方案的启发）

 （取消控制过程（过程超时）
（sb-ext：with-timeout超时
（处理程序情况
（do（（（output-stream（sb-ext：process-output process））
（string nil））
（（and（equalp（sb-ext：process-status process）：exited ）
（equalp（peek-char nil output-stream nil：eof）：eof））
（值字符串：exited））
（cond 
（（equalp（sb- ext：process-status进程）：已信号）
（错误'意外的进程完成：finish-status：已信号））
（（等于（sb-ext：process-status进程）：已停止） 
（错误'unexpected-process-finish：finish-status：stopped）））
（let（（seq（make-string（file-length output-stream））））
（读序列seq输出-s tream）
（setf字符串（连接'string string seq））
（sleep 1）））
（sb-ext：timeout（err）
（declare（ignore err）） ）
（值nil：timeout））
（未预期的处理完成（err）
（值nil（finish-status err））））））
  

该函数通过以下过程调用：

 （sb-ext：运行程序 / path / to / programm 
（列表 --params foo bar）
：output：stream：wait nil） 
  

但是此解决方案有其缺点：

• 由于流未与文件关联（错误），因此无法正常工作


• 即使没有输出，它的通用睡眠也为1当时


• 它做了很多级联，这似乎是一个不太好的解决方案

最终处理/清理已退出/已停止/运行时间过长的流程

如何从以下过程中读取内容：

• 可能会运行很长时间（并且在其生命周期结束时会输出其结果）


• 可能会使输出大于流缓冲区


• 运行时间不得超过给定时间


• 需要整体输出

？

解决方案

字符串输出流对您有用吗？输出将存储在您随后返回的字符串中，因此缓冲不会有太大问题。例如，

  *（带有输出到字符串（输出）
（sb-ext：run-program  / bin / ls'（ /）：输出））
 
 bin 
引导
…
 vmlinuz 
 vmlinuz.old 
 
  

如果要预分配字符串，可以使用 with-output-to-string 的 string-form 参数。 / p>

不过，您不必使用with-output-to-string来使用string-output-stream。您也可以使用 make-string-output-stream 并将其传递给sb-ext：run-program。最终，您将使用 get-output-stream从文本中提取文字-string 。

There are already a few answers on how to read from a process stream but as far as I can see they do not cover reading from a process which:

• may run for a long time before making any output
• makes a huge output at the end of its lifetime (more than the stream buffer can hold)
• should not run longer than a given time frame
• the resulted output is needed as a whole (long character string)

Therefore using the mentioned solutions would either result in wasted process cycles as the loop will try to read the stream even though there is no output, or the non termination of the solution as the process cannot print its whole output (due to full buffer) and the output handler waits for process termination prior to reading.

My current solution looks like this (inspired by an solution for bulk reading large files)

(defun control-process (process timeout)
  (sb-ext:with-timeout timeout
    (handler-case
      (do ((output-stream (sb-ext:process-output process))
           (string nil))
          ((and (equalp (sb-ext:process-status process) :exited)
            (equalp (peek-char nil output-stream nil :eof) :eof))
           (values string :exited))
        (cond 
          ((equalp (sb-ext:process-status process) :signaled)
           (error 'unexpected-process-finish :finish-status :signaled))
          ((equalp (sb-ext:process-status process) :stopped)
           (error 'unexpected-process-finish :finish-status :stopped)))
        (let ((seq (make-string (file-length output-stream))))
          (read-sequence seq output-stream)
          (setf string (concatenate 'string string seq))
          (sleep 1)))
     (sb-ext:timeout (err)
      (declare (ignore err))
      (values nil :timeout))
     (unexpected-process-finish (err)
      (values nil (finish-status err))))))

The function is called with a process:

(sb-ext:run-program "/path/to/programm"
                           (list "--params" "foo" "bar") 
                           :output :stream :wait nil)

But this solution has its drawbacks:

• it does not work as the stream is not associated with a file (error)
• it does a generic sleep of 1 even though there might not be a output at that time
• it does a lot of concatenation which seems to an inelegant solution

Final handling/clean-up of an exited/stopped/too long running process is handled by the calling function.

How can I read from a process which:

• may run for a long time (and does its output at the end of its lifetime)
• may make a output larger than the stream buffer
• must not run longer than a given time span
• output is needed as a whole

?

解决方案

Might a string output stream work for you? The output will be stored in the string that you get back afterward, so the buffering shouldn't be too much of a problem. E.g.,

* (with-output-to-string (out)
    (sb-ext:run-program "/bin/ls" '("/") :output out))

"bin
boot
…
vmlinuz
vmlinuz.old
"

If you want to preallocate the string, you can do that to, with with-output-to-string's string-form argument.

You don't have to use with-output-to-string to use a string-output-stream, though. You could also create one with make-string-output-stream and pass it to sb-ext:run-program. You'd get the text out of it eventually with get-output-stream-string.

这篇关于如何正确地从大型过程输出中读取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！