[SOLVED] Pattern to avoid duplicate pipelines even with cache buster

[late pulsar]

We have a pipeline that includes a long-running build. At the beginning of the build, the build tool checks multiple Git remote repositories for changes.

To invalidate the cache during runs and re-run the check against remotes, we added a cache buster as described here. We also need the pipeline to export artifacts back to the client before exiting with the exit code returned by the build as described here:

dagger shell <<'EOM'
    result=$(build-latest)
    $result | artifacts | export "build"
    .exit $($result | exit-code)
EOM

The issue I'm having now is that the build is executed twice, which seems logical given that we added a cache buster.

Is there a pattern or solution for stopping the cache buster to break at a certain point in the DAG? For example, once I have the build result—which is just a struct holding the artifacts directory and the exit-code integer—can I prevent the complete pipeline from being executed again once Sync is called? Are there recommendations besides the current time to use as cache buster variable (e.g some session id) so that we only invalide the cache once per session?

[grave mantle]

👋 are you sure the build is effectively being called twice? When running commands in the shell the cache buster works a bit different than when you do multiple dagger call calls given that once you get a reference to a an object, it should return the cached reference each time

here's a quick example:

func (m *Cache) Test(ctx context.Context) *Result {
    c := dag.Container().
        From("alpine:latest").
        WithEnvVariable("CACHE", time.Now().String()).
        WithExec([]string{"date"})
    result, _ := c.Stdout(ctx)
    code, _ := c.ExitCode(ctx)

    return &Result{
        Artifacts: c.Directory("/tmp"),
        ExitCode:  code,
        Stdout:    result,
    }
}

and then

dagger shell <<'EOM'
    test | stdout
        test | stdout
        test | stdout
EOM

will print the same output three times

[late pulsar]

Yes, I am sure the build is call twice as the build tool is outputting timestamps, which are different between the runs, e.g. I see the following lines:

...
101 : ┆ ┆ [3m53s] | 2025-06-29 20:41:45 - INFO     - OK - All required tests passed (successes=2, skipped=0, failures=0, errors=0)
...
173 : ┆ ┆ [2m5s] | 2025-06-29 20:44:23 - INFO     - OK - All required tests passed (successes=2, skipped=0, failures=0, errors=0)
...

However, I tried to reproduce the issue with a MRE using this example from the cookbook as a base and just adding a cache buster variable before the test run, and it works as expected. So I guess the issue is somewhere else in my code. Doing some more debugging atm.

[grave mantle]

perfect. If you run multiple build-latest | exit-code commands in a shell session, does it execute them every time?

if that's the the case then as you mentioned, something else is probably invalidating the cache

[late pulsar]

I believe I've found the issue. The source for the build is passed as follows:

// +optional
// +defaultPath="/"
// +ignore=["build"]
source *dagger.Directory,

The issue stems from unexpected behavior of defaultPath and ignore. When I call the build with explicitly setting the source, it works as expected. For example, this works:

dagger shell <<'EOM'
    result=$(build-latest --source ".")
    $result | artifacts | export "build"
    .exit $($result | exit-code)
EOM

Without passing the source explicitly it causes the job to run twice:

dagger shell <<'EOM'
    result=$(build-latest)
    $result | artifacts | export "build"
    .exit $($result | exit-code)
EOM

As I understand it, when exporting the artifacts, the source directory—which is also an input for the pipeline that evaluates the exit code—gets changed, which invalidates the cache. This is where the behavior of passing the directory explicitly and using defaultPath seem to differ. In the first case, the directory seems to be passed only once to the context, while with defaultPath, the directory seems to be evaluated on each call in the shell. What I still don't understand yet is why the cache gets invalidated when the directory where the artifacts are exported to, is explicitly ignored; i.e. why does source invalidate the cache when files in the build dir change.

This reproduces the issue:

package main

import (
    "context"
    "fmt"

    "dagger/mytest/internal/dagger"
)

type Mytest struct{}

var script = `#!/bin/sh
echo "Test Suite"
echo "=========="
echo "Test 1: PASS" | tee -a report.txt
echo "Test 2: FAIL" | tee -a report.txt
echo "Test 3: PASS" | tee -a report.txt
exit 0
`

type TestResult struct {
    Report   *dagger.File
    ExitCode int
}

// Handle errors
func (m *Mytest) Test(
    ctx context.Context,
    // +optional
    // +defaultPath="/"
    // +ignore=["build"]
    source *dagger.Directory,
) (*TestResult, error) {
    ctr, err := dag.
        Container().
        From("alpine").
        WithDirectory("/src", source).
        // add script with execution permission to simulate a testing tool
        WithNewFile("/run-tests", script, dagger.ContainerWithNewFileOpts{Permissions: 0o750}).
        // run-tests but allow any return code
        WithExec([]string{"/run-tests"}, dagger.ContainerWithExecOpts{Expect: dagger.ReturnTypeAny}).
        // the result of `sync` is the container, which allows continued chaining
        Sync(ctx)
    if err != nil {
        // unexpected error, could be network failure.
        return nil, fmt.Errorf("run tests: %w", err)
    }
    // save report for inspection.
    report := ctr.File("report.txt")

    // use the saved exit code to determine if the tests passed.
    exitCode, err := ctr.ExitCode(ctx)
    if err != nil {
        // exit code not found
        return nil, fmt.Errorf("get exit code: %w", err)
    }

    // Return custom type
    return &TestResult{
        Report:   report,
        ExitCode: exitCode,
    }, nil
}

Calls the test twice:

dagger shell <<'EOM'
    result=$(test)
    $result | report | export "build/test-report"
    .exit $($result | exit-code)
EOM

Works as expected:

dagger shell <<'EOM'
    result=$(test --source=".")
    $result | report | export "build/test-report"
    .exit $($result | exit-code)
EOM

I also tried different ignore patterns /build, **/build, but didn't work.

[grave mantle]

awesome, thx for the repo. We'll take a look and report back in a bit 🙏. I was already thinking that default arguments might have something to do here

[grave mantle]

@late pulsar running some tests I'm unsure I'm getting the same behavior you're seeing.

I've slighly modified your example so the date is appended to each test line as follows:

var script = `#!/bin/sh
echo "Test Suite"
echo "=========="
echo "Test 1 $(date): PASS" | tee -a report.txt
sleep 1
echo "Test 2 $(date): FAIL" | tee -a report.txt
sleep 1
echo "Test 3 $(date): PASS" | tee -a report.txt
sleep 1
exit 0
`

^ with the above I run the following shell script:

dagger shell <<'EOM'
    test | report | contents
    test | report | export "build/test-report"
    test | report | contents
EOM

end here's the output I get:

Test 1 Mon Jun 30 18:02:50 UTC 2025: PASS
Test 2 Mon Jun 30 18:02:51 UTC 2025: FAIL
Test 3 Mon Jun 30 18:02:52 UTC 2025: PASS
/tmp/cache/build/test-report
Test 1 Mon Jun 30 18:02:50 UTC 2025: PASS
Test 2 Mon Jun 30 18:02:51 UTC 2025: FAIL
Test 3 Mon Jun 30 18:02:52 UTC 2025: PASS

As you can see, even I'm exporting the report to the build path between contents runs, the second output is still cached

FWIW I'm running this in v0.18.12

on the other hand if I exportthe report to a different folder than build, the second test | report | contets execution invalidates the cache as expected

[late pulsar]

hmm interesting, because I get the following with dagger v0.18.12 (docker-image://registry.dagger.io/engine:v0.18.12) darwin/arm64

27  : ┆ [0.2s] | Test 1 Mon Jun 30 18:24:25 UTC 2025: PASS
27  : ┆ [1.2s] | Test 2 Mon Jun 30 18:24:26 UTC 2025: FAIL
27  : ┆ [2.2s] | Test 3 Mon Jun 30 18:24:27 UTC 2025: PASS
...
37  : ┆ [0.1s] | Test 1 Mon Jun 30 18:24:28 UTC 2025: PASS
37  : ┆ [1.2s] | Test 2 Mon Jun 30 18:24:29 UTC 2025: FAIL
37  : ┆ [2.2s] | Test 3 Mon Jun 30 18:24:30 UTC 2025: PASS

I start the believe it has something to do with the ignore pattern, whether or not build already exists, and from which cwd dagger is called.

I will try to build and setup a working MRE on Github, maybe that will help sort things out

[grave mantle]

I start the believe it has something to do with the ignore pattern, whether or not build already exists, and from which cwd dagger is called.

the only thing that I thik might be affecting this is the cwd of the shell. Since export is the only command here which is relative to that.

both the defaultPath and the ignore pragma should always be consistent given that it uses absolute references

I will try to build and setup a working MRE on Github, maybe that will help sort things out

that'd be nice, thx! I've also tried with both the build folder being present and not being present before the shell script but I don't see any changes

[late pulsar]

I figured it out! The issue was that I was using 'tee' to log the output to the workspace, which was causing the double execution:

dagger --progress=plain shell <<'EOM' 2> >(tee dagger.log >&2)
    result=$(test)
    $result | report | export "build/test-report"
    .exit $($result | exit-code)
EOM

Sorry, I posted the wrong command to reproduce the issue.
This now also explains why the source directory invalidates the cache. If I were to use // +ignore=["build", "dagger.log"], the exec would only run once. Sorry I missed that. Interestingly, in our GitLab CI pipeline where the issue first appeared, we don't tee the log like above or do anything else that should invalidate the cache; perhaps GitLab is storing some artifacts. My takeaway is that defaultPath behaves differently from passing a directory directly, as it's evaluated on every command in the shell. If you can't absolutely ensure source won't be altered during execution, it's better to pass the directory directly.

[grave mantle]

My takeaway is that defaultPath behaves differently from passing a directory directly, as it's evaluated on every command in the shell

this was new to me as well. It actually makes sense that it works this way but since I don't wander in the shell realm of the codebase often, I wasn't aware of it. Not sure if that's documented @gritty ether ?

[SOLVED] Pattern to avoid duplicate pipelines even with cache buster

[gritty ether]

Not documented, I also wasn't aware of this. Please open a docs issue and I'll add it once I'm back

[late pulsar]

Sure will do

[grave mantle]

I'm opening the issue now 🙏