Wondering what happens if functions fail in V3 tasks.

[static salmon]

Hi all,

I was reading an article on the Trigger.dev website about writing durable tasks and have a question about a specific section. This is the one I'm talking about: https://trigger.dev/blog/v3-announcement#writing-regular-async-code-with-no-timeouts-and-durability

It shows a code snippet that uses a rollback function to revert a certain action if any errors happen after the rollback function is used. I can't seem to find this function anywhere. I happen to have a use case for it, because I am calling an API of a third party service in my task, but if any errors happen after that API call, I have to revert the changes (or do I?).

I remember that in V2, you could use the io.runTask function to "pick up where you left off" if anything goes wrong, so that certain functions are only executed once, even if the job fails. I am wondering how this works in V3. I somewhat understand the article mentioned above, but I am still a bit struggling to understand how this works exactly in V3. Should I implement this rollback function (as mentioned in the article)? If so, where do I find this rollback function? If not, will the task automatically skip the functions that were successfully executed and start at the failing functions?

Hopefully someone can help me understand this!

[kindred saddle]

You can use an idempotencyKey when calling a task, which is the equivalent of v2 using a cacheKey: https://trigger.dev/docs/idempotency#idempotency

We haven't launched the rollback function yet, feature request is here: https://feedback.trigger.dev/p/rollback-mechanism-for-failed-run

[static salmon]

So, to be sure, should I create a child task for every "must-revert if fails" API call and create idempotency keys when triggering those child tasks?

[woven cedar]

Do you mean "must-retry"? Because we don't do anything with reverting

[static salmon]

I mean that certain API calls inside my tasks should not be re-executed if the tasks fails and is retried. For example, if, inside a task, I make an API call to transfer funds to a user and the next function after that API call fails, I do want the whole task to be retried, but that API call specifically should not be retried. It shouldn't transfer the funds on every retry, only on the first one. Not all of the AP calls are like this, but some shouldn't be executed after the first call in cases like the example above.

In V2, this was possible by wrapping these important API calls or functions inside a io.runTask, so that these won't be executed again if the job fails and is retried. I'm wondering how I accomplish this in V3 tasks.

The example in the article has a different approach though. It uses a rollback function, which I can't seem to find in the SDK

[woven cedar]

Oh yea we haven't implemented the rollback feature yet

But yea the way to accomplish what you want is to wrap the API call to transfer funds in a subtask, trigger that using triggerAndWait , and passing an idempotency key

That way if something fails further down, and the parent task is retried, the transfer funds task will not be retried

The other option is to use our idempotency key stuff and do your own caching

Using something like https://www.unkey.com/blog/announcing-unkey-cache-package

[static salmon]

Understood!, Thank you!

[woven cedar]

Actually, we're considering slightly changing the idempotency key semantics and I wonder if I could pick your brain on how you think it should work. Basically right now, if you are in a task.run function, and you triggerAndWait a child task passing in an idempotency key, we will cache that result no matter if the child task succeeds or fails. So if further down in the parent task there is an error, and you retry the parent task, when the child task is triggered again, the cached success or failure result will be returned.

The change I want to make is to change it so the child task will be rerun if it failed the first time

[static salmon]

I have somewhere to run to right now, but let me get back on this after giving it some thought. We're using these "only run once" functions quite a lot in our codebase, so getting to a nice solution definitely benefits us. I'll get back on this as soon as I can!

[static salmon]

I agree that the child task should be rerun if it failed, because I feel like a child task should be treated like any other task. It's something that can be triggered and might fail. If it does fail, it should be rerun for the maximum amount of attempts (can be manually set per task as per documentation). It is the developer's responsibility (mine in this case) to make sure to set a maximum amount of attempts if it's an expensive operation. The result of a task should only be cached if it succeeds and has an idempotency key.

[woven cedar]

Currently, the child task will be rerun for the maximum amount of attempts, but after that the run ends in the FAILED status. What I'm taking about changing is doing a NEW run for the child task if the first run ends in FAILED status. So it would go through another round of attempts up to the maximum amount of attempts.

(Also thanks for getting back with your thoughts 🫡)

[static salmon]

Aaah I think I understand now. What I'm wondering now, though, is that the run of the child task failed for a reason (after a number of attempts). If it goes through another round of attempts, would anything change?

What I'm thinking now is, if a child task fails multiple attempts and ends its run in the FAILED status, shouldn't the parent task automatically be aborted as well?

I'm not sure if that's the current behaviour, but what do you think?

[woven cedar]

Yea exactly, because if you retried the parent run after a child task failed, you will just continue to get cached failed status.

We do have this AbortTaskRunError that allows basically stopping a task from re-attempting that could be used, but it's a lot to know/think about: https://trigger.dev/docs/errors-retrying#using-aborttaskrunerror

[static salmon]

I saw this in the documentation indeed. I think the current behaviour is correct, though. If the child task ends in a FAILED status, it should abort the parent run. And in the case that the child task succeeds (and has an idempotency key), but something else in the parent run throws an error, the parent run SHOULD re-attempt, but use the cached result of the child task.

[woven cedar]

So when re-attempting the parent run, you want to get back the cached failed result of the child task?

[static salmon]

I think as soon as a child task fails multiple attempts, the parent task should be aborted and not even be re-attempted.

Because in my opinion, there is a solid reason for that child task to fail multiple attempts.

[woven cedar]

But we return a result type from triggerAndWait, which allows you to handle the result (success or fail) of the child task

[static salmon]

Aah I see, that gives the developer more control over what to do

So in this case, I can manually throw the AbortTaskRunError

[woven cedar]

Yea maybe we just need to document this better

[static salmon]

But this is only after multiple attempts, right? Or do you immediately get the result from triggerAndWait after one (possibly failed) attempt?

[woven cedar]

You only get the result after all the attempts are finished