daniel_connect-pagination

[earnest inlet]

👋 Welcome to your new thread!

⏲️ We'll be here soon! We typically respond in a few minutes, but in some cases we might need a bit more time (e.g., server's busy, you've got a complex question, etc.).

⏱️ We close idle threads, which makes them read-only. Once a thread is closed it won't be reopened, but you can start a new thread if you have another question.

🔗 This thread will always be available, even after it's closed. You can find it again using Discord's search, or you can save this link: https://discord.com/channels/841573134531821608/1250104538673713356

📝 Have more to share? Add details, code, screenshots, videos, etc. below.

[edgy ivy]

Hi there 👋 our logs are indicating that request returned 100 results. Can you elaborate on what is leading you to believe the limit value isn't being respected?

[quiet tulip]

Our logs emit another log whenever they've received 100 requests. So there should only be one Stripe API call in between each of our logs, but there are tons of Stripe API calls before we get to the the next batch of 100.

[edgy ivy]

Our logs emit another log whenever they've received 100 requests.
Our list endpoints give you 100 objects in a single request, so logging after 100 requests doesn't sound like it aligns with our behavior.

Are yoyu able to share the code you're running?

[quiet tulip]

Sure, just a sec.

        """List stripe entities for a given stripe account."""
        logger.debug("Listing %s for account %s", entity_name, self.stripe_account_id)
        return (
            getattr(stripe, entity_name)
            .list(stripe_account=self.stripe_account_id, limit=MAX_STRIPE_RESPONSE_LIMIT, **kwargs)
            .auto_paging_iter()
        )```

[edgy ivy]

Hm, I see you also have the .auto_paging_iter() in there, so I think you're also using our auto-pagination functionality. So depending on how you're calling that, you may be making a large number of requests to retrieve all payments for that account.
https://docs.stripe.com/api/pagination/auto

[quiet tulip]

Right, we're not logging 100 after making 100 requests. We're logging 100 after we've received 100 objects.

Let me ask another question. We've done this process for over tons of accounts. This is actually the very last one, and for some reason it's going much slower than the others. Can you tell why?

Is it just that this account has many more objects?

[edgy ivy]

What part is taking a long time? The code you shared seems to be creating an iterator, how are you consuming/using that iterator after it's returned?

I don't have insight into what other accounts you're referring to, so I don't know if it's because this account has more objects than the other accounts you're referring to offhand. Though that would be my initial guess.

[quiet tulip]

Is there a way through Stripe's API to tell how many of a certain object exist?

[edgy ivy]

No, there aren't count APIs, you'd list all the objects and count the results.

[quiet tulip]

It looks lke since last night we've made 4651 API calls to /v1/payment_intents -- so that would be 46,510 payment intents received so far.

anyway to tell if those numbers make sense?

[edgy ivy]

Not that I can offer. Since this is a Connected Account, you may be able to view the Stripe dashboard as that account and see how many payments we list in the dashboard view.

[quiet tulip]

It just says 10,000+ 😉

I'll get back to this, but any other ideas of how we can speed this along?

[edgy ivy]

Not offhand, but I still don't have a great understanding about what part is seen as being slow. If you're retrieving large amounts of data then the process will take time.

[quiet tulip]

Probably not the forum forum for this, but I really wish Stripe had bulk APIs. 🙂

It would be nice if it didn't take days/weeks to get our data.

[edgy ivy]

I'm not sure if it works with Connect, but it may be worth exploring what our Data Pipeline product offers:
https://stripe.com/data-pipeline

It's not a product my team is too familiar with, but I think it may be a better fit here. Because I expect even a "bulk API" would be trying to shove the same amount of data through the same medium currently used.

[quiet tulip]

I've looked into it and data pipeline would have to be used separately with all 100+ of our Connected Standard accounts. And that's not viable for us.

Thanks for your help today.

[small vessel]

daniel_connect-pagination