#Problem with unknown number of goroutines

31 messages · Page 1 of 1 (latest)

wraith yacht
#

Hi, I have a program like this.
I have a un-buffered data channel that accept slice of string.
I put an initial slice with one string into the channel.
Then I listen on the channel with <- operator, and with the data, i create go routines for each of the elements.
These go routines put slices back into the channel, and wait if the channal if filled.
And it continues. Some go routines will terminate without putting data back in. And there will be a point where all routine will stop and I ended up waiting on the channel.

My problem is, i want to close the channel, when there is no go routine running of a particular group. You can assume there are other groups, since this is a part of a larger program. How should I deal with it?

molten ridge
#

sounds like a logic problem, can you share code?

neat fulcrum
#

why do you want to close the channel?

#

it's unclear what 'each of the elements' are

wraith yacht
#

There is a loop like this
for data := range channel {
for _, item := range data {
go func(item) {
// do something with item so that the result is a slice, sometimes, there is no result, so the routine terminate.
channel <- result
}(item)
}
}

#

sort of something like this. Go routines put the data into the channel and from those data, we create more go routine. If there is always data. They will keep doing it. In my case, there will be a time all of the running go routines doesn't produce result. And the loop just hangs there. so i though if i close the channel, it might quit the loop.

neat fulcrum
#

you are mixing two different things there: the spawning of new goroutines to perform some job, and the control of those goroutines

wraith yacht
#

So, how should I seperate them, if I want a operation like this. To spawn new goroutines based on the result of the same routine.

neat fulcrum
#

you could remove the channel, and have the main goroutine control the number of spawned goroutines, you do not need to close a channel that do not exist

wraith yacht
#

so my primary operation is like this. I have some initial data say, ["abc"], then spawn a goroutine the takes it and it will produce a result like, say, ["a", "b", "c"]. Then for each of them, perform the same operation. so spawn three more go routines. Each of them will continues the same operation. So, I was stuck trying to control the number of spawned goroutines.

neat fulcrum
#

that's what I was already understanding, no new info there

#

main goroutine:
keep a slice of slices of strings as local state (or just a slice of strings)
add the first slice of strings (or just the first string)
while the slice is not empty: spawn goroutine, increase counter
then wait for the first response of a goroutine, decrease counter

wraith yacht
#

ok. I do follow that up to that point. So, to get the response of the goroutine so that I can add it back into the local state, I will need to use a channel, right?

neat fulcrum
#

yeah

#

unless you can decouple the generation of the new strings from the actual concurrent work

#

actually, you don't need the internal variable for the strings, you spawn new goroutines as you collect the strings, you only need the counter

wraith yacht
#

So, right now what is happing is that, go routines produces data, and using those data, I was creating the same go routines. The reason I was doing
for data := range channel is that sometimes, all of the goroutines are taking a while to produce data, that is performing api request. Sorry for confusion, first time interacting in forums. So I want to wait for one of the go routine to produce data and then spawn new ones based on them. Sometimes, none of them will return data, and in that case, don't wait anymore. and quit. My initial thought was having a counter with mutexes. For each go routine that is spawned, increase it. For those that are done, decrease it. And in each of those go routine, I check: "are there any go routines that are running, if none, close the channel". Something like that.

molten ridge
#

sounds like a waitgroup

wraith yacht
#

yes, I have tried with waitgroup. But like this

for data := range channel {
  for _, item := range data {
    wg.add(1)
    go func() {
      defer wg.done()
      if result != nil {
        channel <- result
      }
    }()
  }
}
wg.wait()

but, because it is outside of range channel, it nevers get to it.

#

wg blocks the process, but I thought if i can just know, if there are some go routine working, it just terminates without touching the channel. If it is the last go routine then close the channel.

dense berry
molten ridge
#

what did you want to do with the result of the worker?

molten ridge
#

based on what you've said, here's my scrub attempt at interpreting it..

package main

import (
    "fmt"
    "sync"
    "unicode/utf8"
)

func main() {
    var wg sync.WaitGroup

    // initial data set from whereever
    datalist := []string{"abc", "def", "世界"}

    for _, data := range datalist {
        wg.Add(1)
        go func() {
            defer wg.Done()
            process(data)
        }()
    }
    wg.Wait()
}

// process handles each entry from the data source
func process(values string) {
    numValues := utf8.RuneCountInString(values)
    result := make(chan string, numValues)
    output := make([]string, numValues)

    // Create workerFuncs for each entry of our data
    for _, elem := range values {
        go workerFunc(elem, result)
    }

    // workerFuncs will pause until the result is read
    for i := range output {
        output[i] = <-result
        fmt.Println(output[i])
    }
}

// workerFunc gets the actual work done
func workerFunc(l rune, r chan string) {
    r <- fmt.Sprintf("got element %c", l)
}
dense berry
molten ridge
#

it didn't sound like it needed to be

dense berry
#

I’m just saying because this could lead to strange bugs

molten ridge
#

nah fair

dense berry
#

It would be better to put the index within the result

#

At least

molten ridge
#

tbh depending on what james is actually doing it sounds like a recipe for hammering the absolute bejeezus out of some resource

dense berry
#

That might be the problem here. He is already asking for a specific solution