#convex runMutation continuously times out

9 messages · Page 1 of 1 (latest)

inner scaffold
#

Hello, I'm currently hitting a wall while trying out convex. I have to write quite a bit of data, about millions of tiny documents. I haven't been able to finish the download, but the size of the data looks like around writing a gb of data in minutes.

The initial runMutations run fine, but at some point of the download script just time out. At first, each runMutation was inserting 7000 documents. I did try to batch it to 500 docs, but I still run into timeout issues.

this happened with the free plan and with the paid plan, on both development and production environments.

Should I just retry the mutations when they timeout?

cyan sparrowBOT
#

Thanks for posting in #1088161997662724167.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.

    - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
    - Use [search.convex.dev](https://search.convex.dev) to search Docs, Stack, and Discord all at once.
    - Additionally, you can post your questions in the Convex Community's #1228095053885476985 channel to receive a response from AI.
    - Avoid tagging staff unless specifically instructed.

    Thank you!
inner scaffold
#

getting also this screen many times on the dashboard.

image.png
#

Hum, I've tried batching with 100 docs, and it runs better, but a bit slow. With 300 docs seems to be better, but also slow. I wonder if convex can handle heavy write spikes if I batch well enough + handle timeouts with exponential backoffs?

marsh marlin
#

Mutations are limited to 1 second execution times. Sounds like you may want to use an action? Are you performing some download logic in your function as well? Calls to third-party APIs or services should be handled in actions. Actions can run for 10 minutes, and inside an action you can schedule/fire any number of mutations etc. So inside your action, you could prepare the data that you want to write, and then perform many invocations of your mutation with small batch sizes (to keep each mutation <1 sec)

inner scaffold
#

I want to syncronize all filenames for a few large repos for many commits. The download part is quite fast, the part that's very slow is the writing.

The mutations being limited to 1 second might be the issue, as some mutations are approaching 700ms+ of query time

#

I might want to change my schema though, it's a bit inneficient, each filename has a key to the commit id.

inner scaffold
marsh marlin
#

If you could share your mutation code maybe I can help more 🙂 i dont get 100% what you mean by synchronizing filenames across repos, but sounds interesting!