#Issue with channels

25 messages · Page 1 of 1 (latest)

mental kestrel
#

why if i, ok := siteURLCh; ok { ?

#

i assume addToDBCh is a result channel, where you collect result from the workers and safe it to database.
close it in main func.

tall parcel
#

routines putting stuff into channels should close the channels

#

producers

#

it's not mandatory to close channels

#

or you could close them in the function context where you defined them (e.g. main func)

lusty yoke
#

generally the rule of thumb is that writers should close the channel
closed channels are STILL readable if there's content so you dont have to worry about the reader

#

in this case it seems like all the channels are defined in main
and there's multiple writers(since all routine shares the same channel)
if you insist on closing, you use a waitgroup done to collaborate and close it once all writers are done

#

but like said before, you dont have to

mental kestrel
#

goroutine addToDBWorker is not needed, you can do that in main.
i'll make it clearer.

tall parcel
#

no

#

or wait

#

well

#

you'd have to close them earlier

#

or use two different waitgroups

#

one for producer routines and one for consumer routines

#

and wg.Wait() for consumers should be called after closing the channels after the wg.Wait() for producer routines

mental kestrel
#

you've two group of goroutines that have different life time. you need 2 wait group.
scrapeSiteWorker would stop when siteURLCh is closed. but addToDBWorker only stop when addToDBCh is closed, the problem is that scrapeSiteWorker is the producer of addToDBCh.
you need to wait for all scrapeSiteWorker to return before closing addToDBCh (sending to closed channel cause panic)

lusty yoke
#

it should be

       for i := 0, i < 10; i += 1 {
               wgScrape.Add(1)
               go scrapeSiteWorker(siteURLCh, addToDBCh)
       }
mental kestrel
#

it deadlock because wg.Wait never return. because addToDBWorker still waiting on addToDBCh but it never closed/

lusty yoke
#

also the scrape site worker needs wg to call wg done

#

if you add 10 to wg, wg.wait will wait until there's 10 calls to wg.done

#

wouldnt it work even with wgScrape.Add(10) though
yes but what if you changed the amount of routines to say 20?
wg.Add(1) will always work regardless of that

#

i have the wait groups set as global variables
generally using global is best avoided

#

for example instead it should be passed rather then just be a global value