Pagination with Goroutines | Discord Gophers | Page 1

Alright, so I have a thing I'd like to figure out.

We have a api endpoint where we get our products from, however getting a lot of products at once takes time, but fewer the shorter, therefore I'd like to add some pagination for it, and also use goroutines to make multiple requests at once and then merge all the responses into one list/slice.

I'll start of with my initial try:

func GetServiceLayerProducts() ([]api.Product, error) {
    page := 0
    var products []api.Product

    limiter := make(chan int, 7)
    done := make(chan bool) // Channel to signal loop termination
    var wg sync.WaitGroup   // Wait group to wait for all goroutines to finish

    for {
        fmt.Println("Retrieving page", page)
        limiter <- 1
        endpoint := fmt.Sprintf("%s%s?pageSize=100&pageReference=%d&productClass=Global", serviceLayerBaseEndpoint, serviceLayerProductEndpoint, page)

        wg.Add(1) // Increment the wait group counter

        go func() {

            defer wg.Done() // Decrement the wait group counter when done

            // making the request
                        // ....
            if len(response.Items) == 0 {
                done <- true // Signal that there are no more items
                return
            }

            products = append(products, response.Items...)

            <-limiter // Release the limiter after successful processing
        }()

        page++

        wg.Wait() // Wait for all goroutines to finish

        // break the loop somehow
        select {
        case <-done:
            fmt.Println("Loop terminated after retrieving all items.")
            return products, nil
        default:
            continue
        }
    }

}

however, this seems very slow and it logs out 0 for the page reference over and over again initially.

Not sure about the mental model either. Let me know if I should clear something up!

and 2: should I add a sync.Mutex too?

#Pagination with Goroutines