随时可以运行x个goroutine [英] Always have x number of goroutines running at any time

问题描述

我看到了很多有关如何使Go等待x数量的goroutine完成的教程和示例，但是我想做的是确保始终有x数量在运行，因此将尽快启动一个新的goroutine.作为结束.

I see lots of tutorials and examples on how to make Go wait for x number of goroutines to finish, but what I'm trying to do is have ensure there are always x number running, so a new goroutine is launched as soon as one ends.

具体来说，我有几十万个要做的事情"正在处理MySQL产生的一些东西.所以它是这样的:

Specifically I have a few hundred thousand 'things to do' which is processing some stuff that is coming out of MySQL. So it works like this:

db, err := sql.Open("mysql", connection_string)
checkErr(err)
defer db.Close()

rows,err := db.Query(`SELECT id FROM table`)
checkErr(err)
defer rows.Close()

var id uint
for rows.Next() {
    err := rows.Scan(&id)
    checkErr(err)
    go processTheThing(id)
    }
checkErr(err)
rows.Close()

当前，这将启动数十万个processTheThing()线程.我需要的是启动了最多x个数字(我们称其为20)goroutines.因此，它从在前20行中启动20开始，然后从当前goroutine中的一个完成后的下一个id启动新的goroutine.因此，在任何时间点总是有20个正在运行.

Currently that will launch several hundred thousand threads of processTheThing(). What I need is that a maximum of x number (we'll call it 20) goroutines are launched. So it starts by launching 20 for the first 20 rows, and from then on it will launch a new goroutine for the next id the moment that one of the current goroutines has finished. So at any point in time there are always 20 running.

我确定这是非常简单/标准的，但是我似乎无法在任何教程或示例中或如何完成此操作中找到很好的解释.

I'm sure this is quite simple/standard, but I can't seem to find a good explanation on any of the tutorials or examples or how this is done.

推荐答案

感谢大家为我提供帮助.但是，尽管您都帮助我理解了该技术，但我认为没有人真正提供既可行又简单/易于理解的东西.

Thanks to everyone for helping me out with this. However, I don't feel that anyone really provided something that both worked and was simple/understandable, although you did all help me understand the technique.

最后我做的是，我认为这是对我的特定问题的解答，更加容易理解和实用，因此，如果其他人有相同的问题，我将在此处张贴.

What I have done in the end is I think much more understandable and practical as an answer to my specific question, so I will post it here in case anyone else has the same question.

以某种方式，最终看起来很像OneOfOne发布的内容，这很棒，因为现在我明白了.但是我最初发现OneOfOne的代码很难理解，因为将函数传递给函数使人们很难理解到底是什么.我认为这种方式更有意义:

Somehow this ended up looking a lot like what OneOfOne posted, which is great because now I understand that. But OneOfOne's code I found very difficult to understand at first because of the passing functions to functions made it quite confusing to understand what bit was for what. I think this way makes a lot more sense:

package main

import (
"fmt"
"sync"
)

const xthreads = 5 // Total number of threads to use, excluding the main() thread

func doSomething(a int) {
    fmt.Println("My job is",a)
    return
}

func main() {
    var ch = make(chan int, 50) // This number 50 can be anything as long as it's larger than xthreads
    var wg sync.WaitGroup

    // This starts xthreads number of goroutines that wait for something to do
    wg.Add(xthreads)
    for i:=0; i<xthreads; i++ {
        go func() {
            for {
                a, ok := <-ch
                if !ok { // if there is nothing to do and the channel has been closed then end the goroutine
                    wg.Done()
                    return
                }
                doSomething(a) // do the thing
            }
        }()
    }

    // Now the jobs can be added to the channel, which is used as a queue
    for i:=0; i<50; i++ {
        ch <- i // add i to the queue
    }

    close(ch) // This tells the goroutines there's nothing else to do
    wg.Wait() // Wait for the threads to finish
}

这篇关于随时可以运行x个goroutine的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！