tawdry rover
Most of our tasks are still in the running state, even when all sub tasks are completed. This is completely blocking the queue.
@vocal harness @glass tree Been facing too many issues with respect to the task blocking the queue, concurrency since we upgraded to v4. Can you please let me know by when can we get a resolution for this ?
delicate ploverBOT
Hi @tawdry rover. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Based on the knowledge sources, there are several important changes and behaviors in Trigger.dev v4 that could be related to your issue with tasks remaining in the "running" (executing) state even after all subtasks have completed, leading to blocked queues and concurrency exhaustion:
Concurrency Release Behavior Changed in v4
In v4, concurrency is only released when a run is checkpointed and shut down, not when it is first blocked via a waitpoint. This means that if a parent task is waiting (e.g., after triggering subtasks), the concurrency slot is not released until the run is checkpointed and the process is shut down. If the parent task does not reach a checkpoint or is not properly shut down after subtasks complete, it may remain in the executing state, blocking the queue and preventing new runs from starting. This is a significant change from previous versions and could explain your symptoms if your workflow relies on concurrency being released earlier in the process Gotchas Breaking changes.
Queue and Concurrency Model Updates
EXECUTING or DEQUEUED state count against the limit.WAITING state do not count against concurrency, but a run only transitions to WAITING after being checkpointed.EXECUTING and block the queue Breaking changes.Potential Deadlocks with Shared Queues
If parent and subtask share the same queue and the concurrency limit is low, improper handling of wait/checkpoint can cause deadlocks or blocked queues. The system is designed to release the parent’s slot to the queue when waiting for a subtask, but only up to the concurrency limit. If the process is not checkpointed or if there’s a recursive wait without enough concurrency, tasks can get stuck Waiting for a subtask on the same queue.
Recent v4 Changes and Known Issues
The v4 upgrade introduces stricter queue definitions and concurrency management. There are also notes about possible unknown issues with process persistence and memory leaks, which could cause tasks to remain in a running state if not handled correctly. The documentation recommends testing in staging before production due to these changes Gotchas.
What you can do:
EXECUTING state.Summary:
Your issue is likely related to the new concurrency and checkpointing model in v4, where concurrency is only released when a run is checkpointed and shut down. If the parent task does not reach this state after subtasks complete, it will remain in the running state and block the queue. Review your workflow for proper checkpointing and consider the new concurrency behaviors in v4 Gotchas Breaking changes.
tawdry rover
Not sure if billing is also affected due to this
glass tree
Hey @tawdry rover, I'm looking into this now.
You are over your 100 concurrency so that might explain why it's not continuing but let me look properly at this.
It's definitely not counting towards billed usage. You're on just 3.9s for that run, you can see that and the billed compute on the detail tab: https://cloud.trigger.dev/orgs/adamx-96f2/projects/adamx-v1-BUdA/env/prod/runs/run_cme3qc8ik1ioz27loug44cdgy?span=b24a0b69e015289e&tab=detail&showDebug=true
tawdry rover
All 100 tasks are blocked due to this issue
All subtasks associated with tasks in the executing state have completed hours ago itself
glass tree
I can confirm it's nothing to do with concurrency limits. The run should have continued, got scheduled to execute in the cluster, acknowledged that is was continuing, but then stopped responding.
It should have at least been marked as "stalled" after 2 mins and then retried.
What do you do in your code after the extract-outreach-calls triggerAndWait returns?
I'm looking through the logs to try and figure out why it didn't get marked as stalled
glass tree
I would go ahead and cancel that run @tawdry rover, it's not actually executing and won't recover at this point. I'm looking into why the stall detection didn't work – it certainly should have.
tawdry rover
What do you do in your code after the `extract-outreach-calls` triggerAndWait re...
We mark the call as processed once call is done
I can cancel runs manually as of now, but I guess we need to find a solution
glass tree
Yep I'll figure out what happened and try get a fix in
Are you seeing this happening a lot now?
glass tree
OK I figured out why this happened 👍
That server stopped responding, which can happen and is a failure condition the system is built to withstand. Servers fail sometimes with networking issues or other problems like disk failures etc.
The real issue here is that it didn't recover. There was a race condition where two things happened within 1ms of each other with our stall detection. Submitting a PR for this now.
I'm very interested if you're seeing a lot of v4 bugs with the newer package versions. Reliability should be very high now and so please share any runs that you think have issues.
tawdry rover
It is happening a lot. The business is getting affected a lot. We are doing backfilling everyday
What is the remedy now ?
glass tree
I'm testing this PR now and it will get released to prod tomorrow AM: https://github.com/triggerdotdev/trigger.dev/pull/2380
That will mean those runs get reattempted.
But what I don't understand is why you're experiencing so many stalls. There must be a package you're using or something that's going on that's causing stalls. Could be blocking the Node.js event loop by doing lots of synchronous work. Maybe you don't have enough connections to your database.
I'll download your bundle and look through to see if I can figure it out, but this is not a common issue on v4 so there's something odd here
quartz geode
@tawdry rover I think you might be explicitly setting runtime: "node-22" is that correct? If so, could you deploy with the default of "node" instead which currently points at node 21? Would be great if we could exclude the node version as the cause of this.
tawdry rover
We are using sentry profiling, which is not supported on node21.. node 22 is there in type def
glass tree
I just wanted to follow up to say I've deployed the change which will mean these runs that got stuck would have been retried (assuming you have retries enabled for the task).
You've enabled Sentry profiling from your Trigger tasks? Does this work? Do you see CPU/memory benchmarks?
We're still looking into why you're getting runs hanging a lot, this is not the case for other users.
tawdry rover
We have just started monitoring, I'll keep this thread updated, but looks like the stalling issue is solved. But, he is some other issue, not sure whether it is related to this
If you look at the screenshot below, the message says, "Requeued run after a failure". But none of the sub-tasks associated with that main task have failed.
Works fine in node21 runtime, the issue seems only on node-22 runtime
glass tree
You're self-hosting right? When it goes from PENDING_EXECUTING to QUEUED it means that the run failed to start on a machine.
tawdry rover
No, using trigger cloud