wet field
Hi, I'm facing an issue since one or two weeks.
When i run a task in production mode, it takes 20 seconds to start which is a lot for my use case as the user is waiting...
It seems the task is put to queued state during 20 secs even if there are no other job running at the same time, so its not an issue about concurrency.
I'm using trigger.dev on remote, not self hosted.
Can someone help me about this ?
Thanks
hushed wharf
Can you share a link to a run? Are you doing infrequent runs or lots of new deploys?
If you haven't done a run in a while for that deploy then the Docker image will need to be fetched and booted up again. We cache them when they've been run.
At the moment when doing consistent runs you can expect start times of 2–5s.
We're going to launch "warm starts" soon. When a run finishes it will look for more runs from the same deploy, which will give much faster start times when doing lots of runs.
Then we will work on faster cold starts. Our aim is to get p95 cold starts under 1 second by taking CPU/memory snapshots (a technology we already use for our wait functions) to do very fast starts for all runs/
finite tusk
I see the same on my dashboard but it was always linked to a new deployment.
I will follow what happens with consecutive runs 🙂
wet field
Thanks for the information @hushed wharf , 2s to 5s is totally OK, but 30s is a bit much especially when these runs are chained, the user has to wait for multiple cold starts.
I get the issue every time I run whatever task, not depending about the deploy date.
For instance :
run_id : run_u6d7vnoh5umcfa9t3xv9s
And then I ran the same job again using replay run :
run_id : run_2xmr3pxt87ib9pbgkp5ro
But i'm also experiencing that on other tasks :
run_id : run_rq3onop9xsy6mmjnefsoj
hushed wharf
Yeah this sucks, currently the caching is at the server level so if you do a handful of runs a day like this you will get slow start times.
You can see this one was with a cached docker image: run_eqee1f5td67n1jn3cet6g
Unfortunately, warm starts which launch soon probably won't help much with start times if you're doing less than 100 runs per day.
Faster cold starts however will have a big impact on this. I'm hoping we'll ship the first version of that in March.
wet field
Ok thanks for your answer, and just to I get it right, was there any release that changed how it works ?
Because 2 weeks ago, I did not seem to have these issues.
For instance this run : run_q4pz775sj3q1gxncreauk for the same task, and all the tasks from the same day did not have this latency and I did not run more jobs than today.
My bad, this one may have gotten from cache, because i can experience the same effect from 1 month ago, I just did not notice it.
So you if I launch more runs, I should get more cached starts ? Or is it going to be that way until march ?
hushed wharf
More runs means you're more likely to have a cached start. Most of our customers get p95 start times under 5 seconds because they're doing many runs per minute.
We'll announce any improvements we make to start times, it's our highest priority. The Run Engine is the first part of us being able to have consistently fast start times.
wet field
Perfect, well, I m gonna look for more customers then 🙂