#how to increase reliability of tasks

1 messages · Page 1 of 1 (latest)

devout cradle
#

I have a workflow where I trigger 100 tasks and then it triggers a few tasks within it. They aren’t super heavy, just API calls and DB calls. But when I run them, I usually get 3-5 system failures/tasks frozen. I tried increasing the machine size to be medium, which i feel like should be more than enough. Is there a way to make sure they all run reliably? Right now, if I retry the frozen tasks which had system failure in the console, it’ll usually work after the first or second retry

pseudo crypt
#

Hey, sorry about this.

Could you share a link to a run that suffers from this? Also any code you can provide would be really helpful?

Are you using Promise.all in this code? triggering tasks from inside a Promise.all isn't supported at the moment and will cause problems like this.

devout cradle
#

Not using Promise.all anywhere to call a task. i am using it within a function but i highly doubt that's causing the problem

I used to use it and spent a day trying to debug why things were getting stuck on prod until I read in the docs where it says we should use batch instead

#

ill DM you some runs

pseudo crypt
#

Thanks, I'll take a look at these. I'm working on some reliability improvements with runs getting frozen and Nick is investigating some system failures.

potent lily
#

I have some similar problems, i had some System Failures task for some reason that I don't understand and when I retry them they work fine. Another thing is that I had alerts when the task run fails and when is System Failure I dont get the alarm so i have to rememeber to came here and filter the task sometimes in the week to make sure all of them are processed

image.png
#

It would be really helpful to know in the list if the task was retried or not because I also have to remember if I retry them

#

Maybe there is a better way to do this, and I just don't know how

pseudo crypt
#

Could you drop a link in here @potent lily? I’ll look at this tomorrow as well

#

You can share links to the dashboard publicly btw

potent lily
#

this one is wierd

#

it look like is still "running"

image.png
#

it has and end date but the other spinner still running

#

let me know if i can share more info

devout cradle
#

ya, worst case if there was a way to make it so if theres a system failure, it'll auto retry, that could be helpful

but the most ideal would be that it just never fails due to system failure

devout cradle
#

@pseudo crypt wondering if there were any updates on this if you had a chance to look through logs on why this is happening/how to prevent

#

also random feedback but I noticed some others in discord potentailly using Promise.all. It might be helpful to have a print out in terminal or in the logs on the dashboard that mention this as something to look out for

#

i know its in the docs as well

pseudo crypt
#

Good idea 👍

#

Yes, I just got off the phone with Eric about this. I've been digging into it and we think we know why the runs got permanently stuck. I think it's the same problem for @potent lily as well.

So there are a few problems that need addressing:

SYSTEM_FAILURE
These runs are failing with SYSTEM_FAILURE because Node is exiting unexpectedly. This means that a process.exit(-1) is happening somewhere.

A lot of things can cause this including libraries you're using throw certain errors.

Runs getting stuck

When the SYSTEM_FAILURE happens on a subtask it's breaking the run continuing.

Things I'm working on now:

  1. Fix runs continuing (this should be the easiest fix)
  2. Get errors to come through properly to the attempt so you can actually see the error message for these…
#

Then next:

  1. If these errors aren't "SYSTEM_FAILURE" (which is an error on our side) then we should give them a different status, like CRASHED if they're out of memory for example. This seems like it would be easy but is surprisingly hard to do, we capture most of them but are constantly finding new ones.
  2. If the error isn't obviously "fatal" where doing another attempt is pointless, we should retry (if you have retries left).
  3. If they are genuine system failures then we need to fix those.
#

This is my top priority to get sorted

devout cradle
#

Thanks! Lmk once you think you’ve made a change that would make it potentially work and I’ll try running a ton of them again to see if I still get errors

pseudo crypt
#

Thanks a lot 🙂

pseudo crypt
#

OK I have some major improvements to resuming runs after waits which should go live on Monday, it's being tested now.

devout cradle
#

Ok cool! Also to clarify, my code doesn’t use Waits. It’s mainly just a task which calls a substack multiple times with batchTriggerAndWait.

#

Oh I guess you’re referring to the waits after batchTriggerAndWait, which in that case yes

pseudo crypt
#

Yep I mean trigger and waits 👍

pseudo crypt
#

I have confirmed a fix for this on our test environment today and it will get deployed tomorrow morning UTC

#

I’ll update here when it’s live

pseudo crypt
#

Ok fixes for frozen runs are now live.

Assuming you're on a fairly recent SDK you don't need to do anything. This was a pure server-side change.

Let me know if you have any runs that get frozen and don't continue when all the tasks they're waiting on are finished.

devout cradle
#

I ran into an issue with a frozen run

#

its weird cuz the task basically reached the very end but then it froze

pseudo crypt
#

Could you share a link? Ideally here because DMs make it hard to track. No one will be able to access the content

devout cradle
pseudo crypt
#

Thanks, I'll investigate.

devout cradle
#

Thanks, lmk if you find anything @pseudo crypt 🙏

#

Could i also get a higher concurrency limit for my acct? Not sure what the limit is but like 500 or 1000 okay?

devout cradle
pseudo crypt
#

Higher concurrency will actually help with this problem in the short term. I think it’s happening when you’re at your concurrency limit and doing a lot of triggerAndWaits. I’ll increase your concurrency tomorrow.

We’re working on changes to how we do triggerAndWait to make it impossible for these problems to occur.

devout cradle
#

Awesome. Lmk when it gets updated

pseudo crypt
#

I've increased your concurrency for prod to 250. Let's see how that goes and we can increase it some more soon