Writing tests that use dagger

[last garden]

I've been looking around in dagger codebase to look for a way to run tests that use dagger itself. For example I have a test that spins up a container and checks if it acutally performs the tasks it's supposed to. Is there a good example of a way to achive this? If that's not the right way to test, how do I go about testing dagger pipelines? Passing the docker.sock is not an option as I don't have permissions to do so. I also tried to bring up a sidecar dind with a service binding (https://github.com/dagger/dagger/pull/4677/files#diff-2f732241fe4998f181c3703849d454990682cc44f9981c719b75ad7de777e340) but it didn't work either. Here's a simple test i'm trying to run

package pipeline

import (
    "context"
    "os"
    "strings"
    "testing"

    "dagger.io/dagger"
    "github.com/stretchr/testify/require"
)

func TestSetupMavenBuildImage(t *testing.T) {
    t.Parallel()
    ctx := context.Background()
    c, err := dagger.Connect(ctx)
    require.NoError(t, err)

    ts, err := os.CreateTemp(".", "settings*.xml")
    require.NoError(t, err)
    defer ts.Close()
    defer os.Remove(ts.Name())

    i, err := setupMavenBuildImage(c, c.SetSecret("test", "test"), "test", ".", ts.Name(), "myregistry.com", ".")
    require.NoError(t, err)

    s, err := i.WithExec([]string{"ls", "/src/.m2"}).Stdout(ctx)
    require.NoError(t, err)

    require.Equal(t, "settings.xml", strings.TrimSpace(s))
}

[south kayak]

Hey Nipuna. i also played around with testing
https://github.com/puzzle/goff/blob/main/ci/main_test.go
I'm testing my dagger pipeline with dagger... Basically I'm testing the control flow for different cases (like, push on main branch, add a tag, add a release tag v0.012 and so one) Just some basic e2e / smoke tests.

[last garden]

Thank you @south kayak , how are you *running * the tests? That's when I run into the issue. I can do a go test ./... on my local machine and they will run fine. But when I try to test within my build pipeline it fails because it has no idea of dagger engine internally. The tests try to spin up a new dagger-engine but fails.

[amber walrus]

Did you try setting _EXPERIMENTAL_DAGGER_RUNNER_HOST ? https://github.com/dagger/dagger/blob/v0.6.1/core/docs/d7yxc-operator_manual.md#connection-interface

[last garden]

I did not. I can try, Where would I set it? In the service container? Or the consumer? If it's the consumer, do I set the docker image name?

[amber walrus]

Where the Dagger SDK gets invoked.

Let's start here: https://github.com/dagger/dagger/blob/f965b50e4217fb7a7687b03cf0cd2dc961d1a6b3/hack/dev. This builds a dev Engine, runs it in Docker - see https://github.com/dagger/dagger/blob/f965b50e4217fb7a7687b03cf0cd2dc961d1a6b3/internal/mage/engine.go#L259-L338 - and then exposes the dev CLI and Engine to the command that runs next.

[last garden]

Thank you @amber walrus , I will try this tomorrow

[last garden]

I looked at that. Looks like it's exporting the dagger image to the local machine and then pointing to it. Is that the recommended way to test dagger? What I was trying to do was to bring up a dind image within my dagger client itself and try to run the test. Which I was expecting would bring up a dagger engine. I maybe making things more complicated thatn it needs to be. I was hoping to isolate even the testing code to run within the current engine.

[last garden]

I am also not trying to test the dagger engine. I am trying to use it with the SDK to test my pipeline functionality. At least that's what I want to test. Maybe mocking is the way to go? One of the biggest benefits of using dagger is the ability to test my pipeline code (flow + logic). I am trying to figure out the best way to do so. I think this is something that would be good to be documented as well 🙂

[last garden]

I wonder... is there a way to tell the tests to use the dagger engine that it's already in? I'm guessing no because the current context is not "dagger" aware.

[amber walrus]

By default, the SDKs provision a Dagger Engine within Docker, meaning that as long as docker CLI is present locally and working as it should - it can connect to Docker Server - then SDKs will work out of the box.

My understanding is that in your case this doesn't work as expected, which brings me to _EXPERIMENTAL_DAGGER_RUNNER_HOST. This gives you the option of specifying an already provisioned Dagger Engine. It can be running locally, as a Kubernetes pod, etc. The documentation that I linked to lists available options.

Since the default provisioning doesn't work for you, how do you want to run Dagger Engine?

Next, you are most likely looking for https://docs.dagger.io/757394/use-service-containers/. Here is a recent example that runs Kubernetes - k3s - as a service container in Dagger, and then tests against it: https://github.com/dagger/dagger/issues/5292. Btw, here is a community call demo from last week which is not live yet cc @brittle brook.

Here is another one from @subtle zinc https://youtu.be/v7wGMP7XLJo

[quiet halo]

@last garden writing tests that use dagger itself do not require any special condition. You just need to make sure that your code is correctly structured so you pass a *dagger.Client reference to the methods instead of calling dagger.Connect each time. Besides that, there shouldn't be anything special that you need to take care about.

I'm not sure what you mean by go test ./... works ok but then failing when you try withing your build pipeline. How does your build pipeline currently connets to a dagger engine?

[last garden]

Thank you both for the responses! The way I have my code, I have to do a dagger.Connect() because my test class doesn't have access to the client created by the regular code path. The function I'm testing does take in a dagger.Client but where does that come from? Right now I'm creating it within the test. I can't think of a way to share the same client as the rest of the code within the test. That may be the issue. Let me try to recreate this with a simple test so I can share what I'm seeing. There's a good chance I'm overthinking this.

[last garden]

Alright, I created a simple reproduction - https://gist.github.com/nipuna-perera/7d2d6a7859671e6ea433793eea583d65

In this, main.go and main_test.go are in root and the mainInCiFolder.go is in ci/main.go(gists dont' allow me to have folders :().

This example is analogous to what I have now. I am building a wrapper CLI on top of dagger to distribute to our developers (codename: awesomedagger) so they can easily run it in their own workspaces. That is depicted by the main.go and main_test.go.

in the ci/main.go is my CI for awesomedagger. I am using "awesomedagger" to do CI for "awesomedagger" as I want to dogfood my own CLI. Now the CI piece runs in dagger and the testing code also requires dagger hence the dagger-in-dagger situation. I'd love to be able to simplify this if possible and not deal with a dind situation. Hope this makes things a bit clearer.

[quiet halo]

@last garden now I see what you mean. What you're looking for is to se the ExperimentalPrivilegedNesting flag to true in your go test WithExec. That allows whatever runs in there to access to the engine. Here's how we're using that in the Dagger CI itself: https://github.com/marcosnils/dagger/blob/f672f1182508c6f845b23cd0a711d8d29205e39b/internal/mage/util/engine.go#L176

[last garden]

Thank you for that tip and that got me much further. However, I am running into something I can't explain. So I'm going to shoot in the dark and hope you've run into something similar. I am trying to test whether I the container pulled in a certain file (in this case a settings.xml) file for a maven build. For the life of me I can't figure out why the dagger-in-dagger test can't find that file. Same test outside of dagger runs perfectly fine. I'll share my actual code, and mask what I have to. But it shows the gist of what I'm trying to do.

func TestSetupMavenBuildImage(t *testing.T) {
    t.Parallel()
    client, err := dagger.Connect(ctx)
    if err != nil {
        panic("Error initializing dagger for test")
    }
    e := &MavenEnvironment{}

    e.DaggerClient = client
    e.PipelinePsw = client.SetSecret("test", "test")
    e.PipelineUsr = "test"
    e.DockerFcrHost = "docker.io"
    e.MavenSettingsLocation = "."

    projectRootAbsolutePath, err := filepath.Abs(".")
    require.NoError(t, err)

    e.ProjectRootAbsolutePath = projectRootAbsolutePath

    ts, err := os.CreateTemp(".", "settings*.xml")
    require.NoError(t, err)
    defer ts.Close()
    defer os.Remove(ts.Name())

    i, err := e.setupMavenBuildImage(ts.Name())
    require.NoError(t, err)

    s, err := i.WithExec([]string{"ls", ts.Name()}).Stdout(ctx)
    require.NoError(t, err)

    require.Equal(t, ts.Name(), strings.TrimSpace(s))
}

Here's the function it's testing

func (e *MavenEnvironment) setupMavenBuildImage(mavenSettingsFileName string) (*dagger.Container, error) {

    if mavenSettingsFileName == "" {
        mavenSettingsFileName = "settings.xml"
    }

    source := e.DaggerClient.Host().Directory(e.ProjectRootAbsolutePath,
        dagger.HostDirectoryOpts{Exclude: []string{"ci/", "docs/", "legacy_docs/"}})
    m2cache := e.DaggerClient.CacheVolume("m2")

    //don't pull in users entire .m2 directory
    settingsFile := e.DaggerClient.Host().File(fmt.Sprintf("%s/%s", mavenSettingsFileName, e.MavenSettingsLocation))
    buildImage := e.DaggerClient.Container().From("maven:3.9-amazoncorretto-11").
        WithMountedDirectory("/src", source).
        WithMountedCache("/root/.m2", m2cache).
        WithEnvVariable("ARTIFACTORY_USERNAME", e.PipelineUsr).
        WithEnvVariable("ARTIFACTORY_USR", e.PipelineUsr).
        WithSecretVariable("ARTIFACTORY_PSW", e.PipelinePsw).
        WithSecretVariable("ARTIFACTORY_PASSWORD", e.PipelinePsw).
        WithMountedFile("/src/.m2/settings.xml", settingsFile).
        WithWorkdir("/src")

    return buildImage, nil
}

This is the error I'm seeing.

FAIL: TestSetupMavenBuildImage (1.03s)
   env_test.go:51:
               Error Trace:    /src/pkg/pipeline/env_test.go:51
               Error:          Received unexpected error:
                               input:1: container.from.withMountedDirectory.withMountedCache.withEnvVariable.withEnvVariable.withSecretVariable.withSecretVariable.withMountedFile.withWorkdir.withExec.stdout failed to compute cache key: failed to calculate checksum of ref c47ee17yaitowyeigds27cc9i::vezn1oyfrwly0f91vqg8v7t26: "/settings1658591922.xml": not found

                               Please visit https://dagger.io/help#go for troubleshooting guidance.
               Test:           TestSetupMavenBuildImage
FAIL

I've checked, the file is there. I've been looking at this for a whole day and I've run out of ideas, i'm really hoping you can spot what i'm doing wrong here!!

[quiet halo]

👋 just tried this and it works for me:

both go test -v ./... and go run ci/main.go work

hopefully that helps to unblock you 🙏

[brittle brook]

Here is the talk that Gerhard mentioned above - #1120935751069208699 message

[last garden]

Hey @quiet halo did you change anything? Or just worked as is?

[last garden]

Thank you Miranda! I did see it live and it is very cool!

[quiet halo]

@last garden only thing that I changed is the filepath.Abs in ci/main.go to use . instead of ../ since I'm running go run ci/main.go from the project root

[last garden]

I was just able to reproduce it. it is the file structure. Let me show you what I have. I updated my gist with the file structure I have
https://gist.github.com/nipuna-perera/7d2d6a7859671e6ea433793eea583d65

go test -v ./... works but go run ci/main.go does not work.

Thank you so much for working through this with me @quiet halo . You are amazing!

[quiet halo]

let me check really quick

[quiet halo]

@last garden thing I got a working solution. I've changed the setupMavenBuildImage method to use the settings file that's already present in the initial source Directory instead of calling .Host() again

[last garden]

Thank you @quiet halo! That is interesting, I'm curious, why didn't the Host() call work? Isn't it trying to access the same filesystem? The reason I have it that way is that there is a scenario where I'd grab the settings.xml from the user's home dir which is outside source

[quiet halo]

gotcha.. that's because when running in a container, os.CreateTemp doesn't return the full path of the file

here's a fix for that

^

if you check main_test.go I'm using filepath.Abs to the ts.Name() . With that I can keep using the Host() call for the settings file

let me know if that works for you 🙏

[last garden]

ohh I see, ok I can see how that can be problematic. I am using the location + name in the function but I can refactor it to take a full path (or even a *dagger.File()) to be able to better test it. I think I am set for now 🙂 thank you so much for your help @quiet halo ! 🙏