Trouble keeping state inside a Dagger container

[night goblet]

Although I can show some source code illustrating the issue, let me first state what I am trying to do in general terms:

I am using the Go SDK and the issue I am about to explain shows up no matter what Dagger version I use, including the latest v0.18.5.

When running a command inside a Dagger container using WithExec, the command itself needs to store its own state by writing out into some dot folders local to the working directory inside the container.

The problem I am having is that, even when using cache busting with a volatile input and also calling Sync at the end of the command writing its state out, these dot folders disappear when the next WithExec command is invoked.

I am not using a mounted directory. I am just using WithDirectory, though this command's state is being stored in the same directory in the container brought in by WithDirectory.

I am sure there is some nuance when it comes to how I am using the container or I am doing something verboten by passing around the container pointer from function to function.

Anyway, here is the code:

https://github.com/frizzr/dagger-module-helm/blob/5201dc4d4/helm/main.go#L463

The context of the code at the link above is that at the top-level I am calling the Lint function, which then calls setupContainerForDependentCharts which then calls registryLogin (which is when it writes out its dot folders) but when running the very next call into the container with debugEnv, it reports back that the dot folder previously present in the previous command is now gone.

Thanks in advance for the help!

[chilly harbor]

@night goblet I've made a small example out of your module but I wasn't able to repro. Here's what I've got :

type HelmTest struct {
    Directory *dagger.Directory
}

var InvalidatedCache dagger.WithContainerFunc = func(c *dagger.Container) *dagger.Container {
    return c.WithEnvVariable("CACHEBUSTER", time.Now().String())
}

// Returns a container that echoes whatever string argument is provided
func (m *HelmTest) Test(ctx context.Context,
    registry,
    username string,
    password *dagger.Secret,
) {
    dag.Container().From("quay.io/puzzle/dagger-module-helm:latest").
        WithWorkdir("/helm").
        WithSecretVariable("REGISTRY_PASSWORD", password).
        With(InvalidatedCache).
        WithExec([]string{"sh", "-c", fmt.Sprintf("echo $REGISTRY_PASSWORD | helm registry login %s -u %s --password-stdin", registry, username)}).
        With(m.retainDirectoryChanges).Sync(ctx)

    m.Directory.Terminal().Entries(ctx)

}

// Must be called right after WithExec. Sets up the container to reuse directory changes.
func (m *HelmTest) retainDirectoryChanges(
    c *dagger.Container,
) *dagger.Container {
    m.Directory = c.Directory(".")
    return c
}

if I call dagger call test, I get a terminal at the very end of the Test function but I see the .config folder in there

I'll try to to test your function "as-is" to see if I can repro

I've just managed to test with your module and I also wasn't able to repro.

@@ -433,6 +433,7 @@ func (h *Helm) registryLogin(
                With(h.retainDirectoryChanges).
                Sync(ctx)

+       h.Directory.Terminal().Digest(ctx)

        c = h.debugEnv(c, ctx, "in registryLogin after repo add")

I've added a Terminal call after the exec around line 433 and I can see that the .config folder is there after the exec effectively

[night goblet]

The only detail I left out was that I am accessing this module from another module (which is just a thin shell around it changing the name of some function parameters,) which is then accessing -that- module from a GitHub Action.

Let me see at which level the bug is happening, but thanks for giving me a clue where to look!

[night goblet]

Well, I have narrowed it down.

I did local tests on the linked module, which did succeed, and I tested the shell module that calls the linked module, and it also succeeds.

This means that the bug only appears when running on our GitHub Action runner on our local GitHub Enterprise server.

Luckily I am the admin over these runners, which run our build jobs on a K8s cluster running Actions Runner Controller (ARC.)

We run ARC in 'docker-in-docker' mode, which means there is an init container that serves up the Docker daemon to the container we run in via mount. We then use that daemon to run the Dagger engine.

For debugging, I will be able to shell into the container (I can just put a long sleep command in the Actions workflow) but could anyone give me some advice on how to debug this while I am in there?

Thanks again for reading and the help so far!

[chilly harbor]

@night goblet what I generally do to debug GHA workflows is to use the tmate action (https://github.com/mxschmitt/action-tmate) which will give you an ssh session within the GHA job itself and within there you can modify your dagger code to call the same terminal methods I was calling in my troubleshooting above

is that something that might work for you?

[night goblet]

LOL. Well, I'm not sure our corporate security overlords would approve of that action, but I am hoping that my ability to shell into the container via my kube admin rights to our runner cluster will give me the same flexibility. -fingers crossed-

[chilly harbor]

yeh.. adding a sleep in the GHA will also work. You can find the pod and the docker exec into it

it should be the same behavior.. Was mostly suggesting tmate since it's generally more straightforward

[night goblet]

Definitely appreciate the suggestion. Nothing is straightforward in the corporate IT world. 😩