long ibex
Hi guys, I need to import 20millions documents into the database, I am currently running some scripts that add the documents one by one. Its has been a week and I only managed to import 1million due to hitting rate limits very often.
What are the options to populate my collections a bit faster? I've seen some mentions that says the rate limit only applies for 1 connection and that I could have different connections for increasing it? in practice, how you guys implement that?
tropic girder
On the roadmap for 1.7 is bulk inserting, other than that I think this is all you can do. Depending on how you're inserting them you can look at doing concurrent insertions in your code.
long ibex
concurrent inserts will not work since it would hit the rate limits much faster.
I see this in the docs: Rate limits only apply to Client SDKs. Rate limits do not apply when accessing Appwrite with a Server SDK authenticated using an API key.
I am using SDK with API key and I still keep hitting rate limit.
tired bough
I see this in the docs: Rate limits only apply to Client SDKs. Rate limits do no...
You're using appwrite cloud?
long ibex
Yes I am.
long ibex
ops that was actually my fault. changing to server api I see the rate errors gone now. thank you guys.