#Parent task frozen when all batch triggered child tasks finished(completed/failed)

1 messages · Page 1 of 1 (latest)

ember venture
#

Hey team,

I'm using V3, and below is an affected task example:
https://cloud.trigger.dev/orgs/hallidai-858b/projects/v3/junrs-CQAB/runs/run_qtm42x4mrjve3wki2t7g2?span=cd874a5d40bb3ec5

The problem is:
I have 3 task A, B and C,

  1. A will batchTriggerAndWait B
  2. each B will batchTriggerAndWait C
  3. I have cases like the above one, when one C task failed and the B triggers the C displayed as completed, the batchTriggerAndWait on B never finish and make A frozen.
  4. I have logic in B to filter our failed Cs from batch trigger, and I want the A and B to be able to keep executing instead of being blocked by one of C's failure. How could I do that? Did I setup anythingwrong? Or should I add some failure handle on C task definition?

Thanks for any help! Best

image.png image.png
ember venture
#

@feral crystal Hey Matt, appreciate if I could get some help on this one.

feral crystal
#

Hey, it looks like you had a bunch of consecutive runs that were impacted by this (including the one you linked above). And then you made a change and it's now working better?

Were you using Promise.all wrapped around tasks? it would be great to know what you changed if you did?

I am working on some edge cases at the moment with lots of triggerAndWait and batchTriggerAndWait especially when at the concurrency limits.

ember venture
#

yeah, I'm using promise.all to wrap tasks, and I did some change but the freeze keeps happening. even though I add error handle and continue the process, the parent task still stuck

feral crystal
#

Unfortunately you can't use Promise.all around triggerAndWait or batchTriggerAndWait at the moment.

That definitely won't work properly.

I am working on some other edge cases for frozen tasks, but it won't allow Promise.all.

ember venture
#

oooo sorry my bad, I mis-read the message, I didn't use promise.all, but await only

feral crystal
#

Ok cool, I'm working on a fix for this right now. I've finally managed to reproduce this problem.

Do you only experience this when one of the subtasks fails? That's the bug I've found, there's an edge case where when a subtask fails the parent doesn't continue properly.

ember venture
#

yep, most of cases when I see the error is when the subtask failed, then the parent keeps executing instead of complete or goes into frozen afte ra while

#

thank matt, do we have more details such as ETA for your fix?

#

I also see cases like attached
seems all tasks in batchTriggerAndWait are finished but the triggering keeps going for another 20-30s. Is this expected?

image.png
#

the second one is even longer, wait for another 1 min

feral crystal
#

I'm hoping my fix will go live on Monday after extensive testing.

#

Are you doing a lot of other runs when you experience those slow resumes? I think it's because your queue is full so the run can't continue because you're at your concurrency limit.

#

But not 100% sure without looking into it properly

ember venture
#

hah, I see, that could be an issue. thanks for sharing.

#

I do have some other runs

feral crystal
#

We need to figure out how to make that clearer. I have a plan for some of the statuses a run can be in so it's clear when it's "requeued" after a wait of some kind.

exotic oxide
#

@feral crystal
we do self hosting, and encountered this problem with a very basic example

We do limit execution on staging to only 1 to keep execution time free for prod use. but when we do triggerAndWait the sub job works and the parent freezes and it never run again which caused to queue staging jobs and it kept queuing because the freezed job was still in-progress. But when we increase the limit to 2 everything work perfectly.

It's not critical now for us, but I just want to know if this is happening with everyone else. Thanks

#

@unreal jackal @tranquil rain

feral crystal