Concurrency and HTTP

[brazen berry]

Hey Go community!
How do you deal with HTPP connection wait times and tune concurrent requests for the best throughput?

I'm really interested in finding practical ways to reduce handshake and connection time, make the most of reusable connections, and determine the right number of parallel requests to run.

Thanks!

Would this be the best way to do it?

package main

import (
    "fmt"
    "net/http"
    "sync"
)

func main() {
    numWorkers := 10

    jobs := make(chan int, 200)

    var wg sync.WaitGroup

    for w := 0; w < numWorkers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := range jobs {
                url := fmt.Sprintf("https://domain.tld/someting/%d", i)
                resp, err := http.Head(url)
                if err != nil {
                    fmt.Printf("Error checking %d: %v\n", i, err)
                    continue
                }
                resp.Body.Close()
                if resp.StatusCode == 200 {
                    fmt.Printf("URL with number %d works\n", i)
                }
            }
        }()
    }

    for i := 0; i < 200; i++ {
        jobs <- i
    }
    close(jobs)

    wg.Wait()
}

[summer plaza]

are the defaults, when reusing a shared http client object, not working for you ?

[brazen berry]

Hi _dl!
No, I have not tried using a

http.Client

Should I do that, instead of that above?