可以生成和与外部进程通信，而无需为每个外部进程启动一个操作系统线程？ [英] Can Go spawn and communicate with external processes without starting one OS-thread per external process?

问题描述

短版本：

在Golang中可以并行生成多个外部进程/ strong>，以便它不会对每个外部进程启动一个操作系统线程 ...仍然能够在完成后接收其输出？

较长版本：

在Elixir中，如果使用ports，可以生成数千个外部进程，的Erlang虚拟机中的线程。

例如在Erlang VM下，只有20个操作系统线程管理启动2500个外部 sleep 进程的以下代码段：

  defmodule Exmultiproc do 
 for _<  -  1..2500 do 
 cmd =sleep 3600
 IO.puts启动另一个进程...
 Port.open（{：spawn，cmd}，[：exit_status，：stderr_to_stdout]）
 end 
 System.cmd ，[3600]）
 end 
  

> ulimit -n 到一个较高的数字，例如10000）

另一方面，Go中的以下代码做同样的事情 - 启动2500外部 sleep 进程 - 也启动2500 操作系统线程。因此，它显然会启动一个操作系统线程（阻塞？）系统调用（为了不阻塞整个CPU或类似的，如果我理解正确）：

  package main 
 
 import（
fmt
os / exec
sync 
）
 
 func main（）{
 wg：= new（sync.WaitGroup）
 for i：= 0; i< 2500; i ++ {
 wg.Add（1）
 go func（i int）{
 fmt.Println（Starting sleep，i，...）
 cmd： = exec.Command（sleep，3600）
 _，err：= cmd.Output（）
如果err！= nil {
 panic（err）
} 
 fmt.Println（Finishing sleep，i，...）
 wg.Done（）
}（i）
} 
 fmt.Println （Waiting for WaitGroup ...）
 wg.Wait（）
 fmt.Println（WaitGroup finished！）
} 
  

因此，我想知道是否有一种方法来编写Go代码，使它做类似于Elixir代码，不打开

我基本上在寻找一种方法来管理至少几千个外部长期运行（最多10天） ）过程，以尽可能少的问题，在操作系统中的任何虚拟或物理限制。

（对不起，代码中的任何错误，因为我是Elixir的新人，并且相当新的Go。我很想知道我在做什么错误。）

编辑：澄清关于并行运行长时间运行进程的要求。

解决方案

我发现如果我们不 wait 运行时将不会启动 2500操作系统线程。所以请使用cmd.Start（）而不是cmd.Output（）。

但是似乎不可能读取进程的 stdout ，而不消耗golang os软件包的操作系统线程。我认为这是因为os包不使用非阻塞io读取管道。

下面的程序在我的Linux上运行良好，虽然它阻止进程的stdout因为@JimB在评论中说，也许是因为我们有小的输出，它适合系统缓冲区。

  func main {
 concurrentProcessCount：= 50 
 wtChan：= make（chan * result，concurrentProcessCount）
 for i：= 0; i< concurrentProcessCount; i ++ {
 go func（i int）{
 fmt.Println（Starting process，i，...）
 cmd：= exec.Command（bash， -c，for i in 1 2 3 4 5; do echo to sleep $ i seconds; sleep $ i; echo done; done;）
 outPipe，_：= cmd.StdoutPipe b err：= cmd.Start（）
如果err！= nil {
 panic（err）
} 
< -time.Tick（time.Second）
 fmt.Println（Finishing process，i，...）
 wtChan < - & result {cmd.Process，outPipe} 
}（i）
} 
 
 fmt.Println（root：，os.Getpid（））; 
 
 waitDone：= 0 
 forLoop：
 for {
 select {
 case r：=< -wtChan：
 rpWait ）
 waitDone ++ 
 output：=& bytes.Buffer {} 
 io.Copy（output，rb）
 fmt.Println（waitDone，output.String（））
 if waitDone == concurrentProcessCount {
 break forLoop 
} 
} 
} 
} 
  p> 
Short version:

Is it possible in Golang to spawn a number of external processes (shell commands) in parallel, such that it does not start one operating system thread per external process ... and still be able to receive its output when it is finished?

Longer version:

In Elixir, if you use ports, you can spawn thousands of external processes without really increasing the number of threads in the Erlang virtual machine.

E.g. the following code snippet, which starts 2500 external sleep processes, is managed by only 20 operating system threads under the Erlang VM:
defmodule Exmultiproc do
  for _ <- 1..2500 do
    cmd = "sleep 3600"
    IO.puts "Starting another process ..."
    Port.open({:spawn, cmd}, [:exit_status, :stderr_to_stdout])
  end
  System.cmd("sleep", ["3600"])
end
(Provided you set ulimit -n to a high number, such as 10000)

On the other hand, the following code in Go, which is supposed to do the same thing - starting 2500 external sleep processes - does also start 2500 operating system threads. So it obviously starts one operating system thread per (blocking?) system call (so as not to block the whole CPU, or similar, if I understand correctly):
package main

import (
    "fmt"
    "os/exec"
    "sync"
)

func main() {
    wg := new(sync.WaitGroup)
    for i := 0; i < 2500; i++ {
        wg.Add(1)
        go func(i int) {
            fmt.Println("Starting sleep ", i, "...")
            cmd := exec.Command("sleep", "3600")
            _, err := cmd.Output()
            if err != nil {
                panic(err)
            }
            fmt.Println("Finishing sleep ", i, "...")
            wg.Done()
        }(i)
    }
    fmt.Println("Waiting for WaitGroup ...")
    wg.Wait()
    fmt.Println("WaitGroup finished!")
}
Thus, I was wondering if there is a way to write the Go code so that it does the similar thing as the Elixir code, not opening one operating system thread per external process?

I'm basically looking for a way to manage at least a few thousand external long-running (up to 10 days) processes, in a way that causes as little problems as possible with any virtual or physical limits in the operating system.

(Sorry for any mistakes in the codes, as I'm new to Elixir and, and quite new to Go. I'm eager to get to know any mistakes I'm doing.)

EDIT: Clarified about the requirement to run the long-running processes in parallel.
解决方案
I find that if we not wait processes, the Go runtime will not start 2500 operating system threads. so please use  cmd.Start() other than cmd.Output().

But seems it is impossible to read the process's stdout without consuming a OS thread by golang os package. I think it is because os package not use non-block io to read the pipe.

The bottom, following program runs well on my Linux, although it block the process's stdout as @JimB said in comment, maybe it is because we have small output and it fit the system buffers.
func main() {
    concurrentProcessCount := 50
    wtChan := make(chan *result, concurrentProcessCount)
    for i := 0; i < concurrentProcessCount; i++ {
        go func(i int) {
            fmt.Println("Starting process ", i, "...")
            cmd := exec.Command("bash", "-c", "for i in 1 2 3 4 5; do echo to sleep $i seconds;sleep $i;echo done;done;")
            outPipe,_ := cmd.StdoutPipe()
            err := cmd.Start()
            if err != nil {
                panic(err)
            }
            <-time.Tick(time.Second)
            fmt.Println("Finishing process ", i, "...")
            wtChan <- &result{cmd.Process, outPipe}
        }(i)
    }

    fmt.Println("root:",os.Getpid());

    waitDone := 0
    forLoop:
    for{
        select{
        case r:=<-wtChan:
            r.p.Wait()
            waitDone++
            output := &bytes.Buffer{}
            io.Copy(output, r.b)
            fmt.Println(waitDone, output.String())
            if waitDone == concurrentProcessCount{
                break forLoop
            }
        }
    }
}


                        
这篇关于可以生成和与外部进程通信，而无需为每个外部进程启动一个操作系统线程？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！