#troubles with concurrency

32 messages · Page 1 of 1 (latest)

serene spindle
#

Noob question here. I have a script that takes in a list of usernames and turns on the vacation responder for them in gmail. I'm not actually sure if my code is using concurrency the way it's meant to be used. I.e. did I call go in the right area? Or am I supposed to do it somewhere else (e.g. in one of the for loops in processVacationResponders)? Would it be better if I write it in a producer/consumer pattern or would this good be enough?

I ran this against 2000 test users but it took about 14 min. Not sure if my code is the bottleneck or if maybe the rpcs I'm using are just slow. On the bright side, its a hell of lot faster than the python script (10min / 100 users 😮 ) I have that does the same thing.

main()

c := make(chan userVacationResponderSettings, len(users))
    go processVacationResponders(ctx, users, c)
    failedUsers := []string{}
    for updatedSetting := range c {
        if updatedSetting.err != nil {
            fmt.Printf("failed to update for %q: %v\n", updatedSetting.email, updatedSetting.err)
            failedUsers = append(failedUsers, updatedSetting.email)
            continue
        }
        if updatedSetting.settings == nil {
            continue
        }
        ... // printing out stuff
    }
    if len(failedUsers) > 0 {
        fmt.Printf("some updates failed. affected users: %q\n", failedUsers)
    }
    fmt.Printf("Finished processing in %v\n", time.Since(startTime))

processVacationResponders()

func processVacationResponders(ctx context.Context, users []userVacationResponderSettings, userResponseChannel chan userVacationResponderSettings) {
    for _, user := range users {
        updatedSettings, err := updateVacationResponder(ctx, user)
        if err != nil {
            userResponseChannel <- userVacationResponderSettings{email: user.email, err: err}
        }
        userResponseChannel <- userVacationResponderSettings{email: user.email, settings: updatedSettings}
    }
    close(userResponseChannel)
}

updateVacationResponder()

func updateVacationResponder(ctx context.Context, user userVacationResponderSettings) (*gmail.VacationSettings, error) {
    gmailClient, err := createGmailClient(ctx, user.email)
    if err != nil {
        return nil, err
    }
    settings, err := gmailClient.Users.Settings.UpdateVacation(user.email, user.settings).Do()
    if err != nil {
        return nil, err
    }
    return settings, err
}
bitter pine
#

ouh, closing channel from the receiver side.

#

you arent processing each user concurrently

#

you don't need to rework anything. try just be spawning multiple go processVacationResponders(ctx, users, c)
and spread the task by subslicing users slice.

serene spindle
#

oof, thanks for taking a look.

and spread the task by subslicing users slice.

excellent suggestion.

you don't need to rework anything. try just be spawning multiple go processVacationResponders(ctx, users, c)

ah, damn thats what I was afraid of. I'm guessing I could a for loop of that right? instead of doing

go processVacationResponders(ctx, users, c)
go processVacationResponders(ctx, users, c)
go processVacationResponders(ctx, users, c)
go processVacationResponders(ctx, users, c)
bitter pine
serene spindle
#

got it, ty!

serene spindle
#

I'm trying to do something like this but when I'm ranging over c and printing values, it hangs on the last value (I also removed the close() in the receiver side)

main()

startTime := time.Now()
chunks := chunkSlice(users, 10)
c := make(chan userVacationResponderSettings)
var wg sync.WaitGroup
for n := 0; n < len(chunks); n++ {
  wg.Add(1)
  go func(n int) {
    processVacationResponders(ctx, chunks[n], c)
  }(n)
  wg.Done()
 }
wg.Wait()
#

I tried closing the channel after ranging over it but I'm still getting the same issue

bitter pine
#

you didnt use wait group correctly

   wg.Add(1)
        go func(n int) {
            processVacationResponders(ctx, chunks[n], c)
        }(n)
        wg.Done()
serene spindle
#

give me a sec I'll get it tho lol

#

am I supposed to be calling wg.done in the processVacationResponders()? or is that unrelated?

bitter pine
serene spindle
#

ah yeah I subsliced it before. chunks is a [][]processVacationResponders. updated code is above

bitter pine
#

about wg.Wait and close(c)
you want to wait after range or do the range in separate goroutine and close c after the wait.
dont worry about the range being in separate goroutine and does not hold a wait group. since its lifetime is depend on c, it'll be terminated once the producer return.

serene spindle
#

thanks boku, will give those a try

serene spindle
#

can't figure it out haha. I tried using wait after the range and then closing but it still hangs

func main() {
    // setup code
    startTime := time.Now()
    chunks := chunkSlice(users, 10)
    updatedSettingsC := make(chan userVacationResponderSettings)
    var wg sync.WaitGroup
    for n := 0; n < len(chunks); n++ {
        wg.Add(1)
        go func(n int) {
            processVacationResponders(ctx, chunks[n], updatedSettingsC)
        }(n)
        wg.Done()
    }

    failedUsers := []string{}
    usersUpdated := 0.0
    for updatedSetting := range updatedSettingsC {
        if updatedSetting.err != nil {
            fmt.Printf("failed to update for %q: %v\n", updatedSetting.email, updatedSetting.err)
            failedUsers = append(failedUsers, updatedSetting.email)
            continue
        }
        if updatedSetting.settings == nil {
            continue
        }
        usersUpdated++
        fmt.Printf("User: %q\n"+
            "Subject: %q\n"+
            "Message: %q\n"+
            "Start Time: %q\n"+
            "End Time: %q\n"+
            "Restrict to Contacts: %v\n"+
            "Restrict to Domain: %v\n\n",
            updatedSetting.email,
            updatedSetting.settings.ResponseSubject,
            updatedSetting.settings.ResponseBodyPlainText,
            time.UnixMilli(updatedSetting.settings.StartTime),
            time.UnixMilli(updatedSetting.settings.EndTime),
            updatedSetting.settings.RestrictToContacts,
            updatedSetting.settings.RestrictToDomain)
        logger.Infof("len of c: %v", len(updatedSettingsC))
        logger.Infof("usersUpdated: %v", usersUpdated)
        logger.Infof("time elapsed: %v", time.Since(startTime))
        logger.Infof("users/sec: %v", usersUpdated/time.Since(startTime).Seconds())
    }
    wg.Wait()
    close(updatedSettingsC)
    if len(failedUsers) > 0 {
        fmt.Printf("some updates failed. affected users: %q\n", failedUsers)
    }
    fmt.Printf("Finished processing in %v\n", time.Since(startTime))
}
rain pike
serene spindle
#

function now looks like this per the example, I left wait() in the same location:

for n := 0; n < len(chunks); n++ {
        wg.Add(1)
        go func(n int) {
            defer wg.Done()
            processVacationResponders(ctx, chunks[n], updatedSettingsC)
        }(n)
    }
#

still having the same problem tho

#

re chunking - not sure, but when I chunk it down to where the chunks have a len of 5-10 the program updates the users settings like really fast. 2k users in 4 secs. probably might get throttled if I have to do 10-20k plus but I can control it through a flag right?

#

troubles with concurrency

bitter pine
#

based on your first code:

Time := time.Now()
defer func() {
    fmt.Printf("Finished processing in %v\n", time.Since(startTime))
}()

c := make(chan userVacationResponderSettings, len(users))

var wg sync.WaitGroup

const numberOfWorkers = 10
chunk := len(users) / numberOfWorkers
for i := 0; i < numberOfWorkers; i++ {
    wg.Add(1)
    go func(wg *sync.WaitGroup, n int) {
        defer wg.Done()
        if n == numberOfWorkers-1 {
            processVacationResponders(ctx, users[n*chunk:], c)
            return
        }
        processVacationResponders(ctx, users[n*chunk:(n+1)*chunk], c)
    }(&wg, i)
}

go func() {
    failedUsers := []string{}
    for updatedSetting := range c {
        if updatedSetting.err != nil {
            fmt.Printf("failed to update for %q: %v\n", updatedSetting.email, updatedSetting.err)
            failedUsers = append(failedUsers, updatedSetting.email)
            continue
        }
        if updatedSetting.settings == nil {
            continue
        }
        //... // printing out stuff
    }
    if len(failedUsers) > 0 {
        fmt.Printf("some updates failed. affected users: %q\n", failedUsers)
    }
}()

wg.Wait()
close(c)
#

almost missed the last part

serene spindle
#

wow thanks dude. I was just not getting it for some reason. I hate that I couldn't figure it out.

This whole bit is making my head whirl

go func(wg *sync.WaitGroup, n int) {
        defer wg.Done()
        if n == numberOfWorker-1 {
            processVacationResponders(ctx, users[n*chunk:], c)
            return
        }
        processVacationResponders(ctx, users[n*chunk:(n+1)*chunk], c)
    }(&wg, i)

got a couple questions

  1. how come you need to pass in waitgroup to the go func? what was wrong with the old pattern?
  2. what is this bit of code doing?
if n == numberOfWorker-1 {
            processVacationResponders(ctx, users[n*chunk:], c)
            return
        }
bitter pine
serene spindle
#

yeah thats the better way for me to learn lol, I got spoonfed enough

#

ah without it one of the users doesn't get processed

#

the very last one I believe

#

plz tell me your a SWE and not just some high school kid haha

bitter pine