Declarative way to handle retries?

[upbeat scaffold]

Don't think this exists right now, but any thoughts on a declarative way to handle retries?
For example, I'm using Dagger primarily for a bunch of CI tests. Even if my tests are 1% flaky, at scale that adds up to a ton a day.
I could manually implement retries, but a large benefit of Dagger is that everything's declarative.

Maybe something like:

    return dag
      .container()
      .from("hexpm/elixir:1.18.4")
      .withExec(["mix", "test"], {retries: 5})
      .exitCode()

or

    return dag
      .container()
      .from("hexpm/elixir:1.18.4")
      .withRetries(5)
      .withExec(["mix", "test"])
      .exitCode()

or

    return dag
      .container()
      .from("hexpm/elixir:1.18.4")
      .withExec(["mix", "test"])
      .exitCodeWithRetries(5)

etc.

Not really sure how it'd fit in best.

[queen umbra]

@upbeat scaffold you can use the with((c) => {}) helper function for this

so you can do something like:

dag.container()
.from("foo")
.with(retries(["mix","test"],5))
.exitCode()

you'll have to define the retires function to run a loop and perform whatever check you need to either retry, fail or continue

[upbeat scaffold]

hm

[queen umbra]

it's very hard to implement this in a way that works for everyone in the engine given that everyone will very likely have dfferent conditions for which they'd like to retry. So implenenting this in userspace makes the most sense

[upbeat scaffold]

any docs on how to use this with bit?
not quite understanding

[queen umbra]

let me give you a basic example

[upbeat scaffold]

maybe something like this?

function retries(command: string[], maxRetries: number) {
  return (container: Container): Container  => {
    let result: Container;
    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      result = container.withExec(command);
      const exitCode = result.exitCode();

      if (exitCode === 0) {
        break;
      }
    }
    return result;
  };
}

[queen umbra]

yes, I was doing something around those lines.

thing is exitCode currently returns a promise and the with helper doesn't allow to return Promise<Container>` so that makes it quite tricky

[upbeat scaffold]

ah, yeah

bigger problem I might be having is withExec failing just takes down everything, so it can't even check the status to decide to retry or not

[queen umbra]

you can catch the error with a try{} catch{} block

that allows preventing it to fail everything

[upbeat scaffold]

looks like I might also be able to pass expect to withExec somehow

[queen umbra]

it's possible to make this approach but will require a bit of work. We should definitely add an async version of this with helper

[upbeat scaffold]

that'd be cool

[queen umbra]

https://github.com/dagger/dagger/issues/10900

[wicked canopy]

I agree that we won't be able to support every kind of retry, but withExec(retry:int) seems like a pretty good solution to the basic case?

wdyt @queen umbra ?

[queen umbra]

In my experience usually these kind of optimizations end up being a shot in the foot rather than a true benefit.

Reason being that it generally ends up being more detrimental to your pipelines as you end up retrying for any errors that occur regardless if they're a flake or not.

What I generally needed in the past is a way to express "only retry in under these conditions" and that's where I think the current with helper method becomes handy

[wicked canopy]

but what would the conditions for retry be?

What you're saying makes sense, a retry on exec would basically make every failed test N times slower, even if they're not flakes.

But I don't know what kind of custom heuristic could solve that in the module code.

[queen umbra]

but what would the conditions for retry be?

that's what I'm saying it really depends on each use-case and it's generally better IMO if module authors come up with their custom chaining retry helper

[wicked canopy]

OK but what would be an example?

[queen umbra]

But I don't know what kind of custom heuristic could solve that in the module code.

we could do something like this automatically in Dagger Cloud with our runners once we have the metric

[wicked canopy]

Yeah we could solve this at the platform level, with a lot of Dagger Cloud magic (as you know I love this idea). But that's different from users implementing a custom heuristic in their module

[queen umbra]

this is the example Solomon. In this case it's only checking for status_code != 0 but you can do whatever you want there to flag the retry

[wicked canopy]

To me a realistic custom heuristic would be: "we know this particular test is flakey, so let's implement it with withExec(retry: 3)

[wicked canopy]

OK but that's just reimplementing a simple retry count. What does "whatever I want" really get me?

It's not a trick question, there may be a great answer, I just don't see it right now

[queen umbra]

get the stdout / stderr from that execution and look for any specific value / test name which hints a retry

[wicked canopy]

Ah I see, like if there's a known flake, but I don't want to split out the whole flaky part from the test suite

[queen umbra]

exactly. I was just about to reply that

[wicked canopy]

But when we had flakes, we ended up just splitting out right?

[queen umbra]

we didn't get to the point of splitting individual flakes, no. We usually address the underlying issue before getting to that point

once it starts becoming very annoying, we invest the time to fix it