#WithExec caching

1 messages · Page 1 of 1 (latest)

idle inlet
#

Are there any circumstances where an exec will be cached or skipped? I have a number of WithExec commands and one is being skipped over by dagger for some reason...

idle inlet
#

That or the commands are being executed out of order?

thorn saffron
#

Execs are always be cached if they have the exact same inputs as before. If you need it to always run no matter what, you can attach a randomly generated value as an environment variable, like container.WithEnvVariable("BUST", time.Now().String())

idle inlet
#

came across that solution in another thread, oddly its still being skipped with that :/

#

will see if I can get a repro

idle inlet
#

Things seem to work as expected if I explicitly request Stdout from each WithExec.

maiden nebula
#

that's because Stdout gets the output of the withExec where it was invoked. You can try this:

package main

import (
    "context"
    "fmt"
    "time"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    client, err := dagger.Connect(ctx)
    if err != nil {
        panic(err)
    }
    defer client.Close()

    c := client.Pipeline("test").
        Container().
        From("alpine").
        WithEnvVariable("BUST", time.Now().String())

    printOut(c.WithExec([]string{"echo", "xxx1"}).Stdout(ctx))
    printOut(c.WithExec([]string{"echo", "xxx2"}).Stdout(ctx))
    printOut(c.WithExec([]string{"echo", "xxx3"}).Stdout(ctx))

    if err != nil {
        panic(err)
    }
}

func printOut(o string, err error) {
    if err == nil {
        fmt.Println(o)
    }
}
idle inlet
#

Thanks beat me to it @maiden nebula 🙂 Im still a bit confused on the behaviour of dagger when compared to more traditional CI engines...

Is it the case that without the calls to .Stdout in each step that the order is undetermined?

maiden nebula
#

if you add logging to your client: client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr)) you'll have more visibility about how things are currently being executed.

#

the only way to get "undefined" execution is if you run multiple WithExecsin different goroutines to make them run in parallel

thorn saffron
idle inlet
#

My original use case was a basic pipeline with terraform, something like this:

src := client.Host().Directory(".")                
terraform := client.Container().From(toolbox)                                                       
terraform = terraform.WithMountedDirectory("/work", src).WithWorkdir("/work")                                                                
terraform = terraform.WithExec([]string{"terraform", "init"})                                                
terraform = terraform.WithExec([]string{"terraform", "apply", "--auto-approve")

But what I was seeing (before explicitly fetching and print Stdout after each WithExec), was that it would skip over the 'init' step entirely and go straight to the apply, which of course failed because init hasn't run...

It seems the Stdout call is forcing dagger to wait for the init to run before proceeding, without that it just fails pretty quickly. This is even with the cache bust workaround...

Sorry to be persistent but I feel I need to understand this to have a future with dagger 😄

maiden nebula
#

sure! that's strange since it shouldn't happen if you used the cache bust command. I'll try to repro this a bit later 🙏

#

you shouldn't have to do Stdout for each step

maiden nebula
#

@idle inlet managed to get a simple example which doesn't show the behavior you mention:

package main

import (
    "context"
    "os"
    "time"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    client.Pipeline("test").
        Container().
        From("hashicorp/terraform").
        WithMountedDirectory("/app", client.Host().Workdir()).
        WithWorkdir("/app").
        WithEnvVariable("BUST", time.Now().String()).
        WithExec([]string{"init"}).
        WithExec([]string{"apply", "-auto-approve"}).ExitCode(ctx)
}

resource "local_file" "foo" {
  content  = "foo!"
  filename = "${path.module}/foo.bar"
}
#

if you run go run main.go multiple times, init and apply are both executed

#

LMK if you have a repro example from your side so we can take a look

idle inlet
#

Thanks very much for checking - your example also works as expected on my end so Im kind of stumped as to why I was having issues in my other pipeline. At least I know a little more about how its supposed to work, if I see this issue again I'll let you know