Should it be possible to export mounted directory content back to the host? | Dagger | Page 1

covert bison May 19, 2024, 1:41 PM

#

I have a Powershell script working calling simple Docker commands and volume mounting and thought It would be a good exercise to try with Dagger

for _, spec := range specs {
        for _, settings := range spec.Settings {
            if slices.Contains(languagesArg, settings.Language) {
                stdout, err := dag.
                    Container().
                    From("mcr.microsoft.com/openapi/kiota:1.14.0").
                    WithMountedDirectory("/app/output", directoryArg).
                    WithoutUser().
                    WithExec([]string{
                        "generate",
                        "--output", fmt.Sprintf("/app/output/%s/%s", settings.Language, settings.Output),
                        "--language", string(settings.Language),
                        "--openapi", spec.URL,
                        "--exclude-backward-compatible", "true",
                        "--log-level", "Debug",
                        "--clean-output",
                        "--additional-data", "true",
                        "--class-name", "ApiClient",
                    }).
                    Stdout(ctx)

                if err != nil {
                    return stdout, err
                }

                output += stdout
            }
        }
    }

I get valid output, and I can see nothing went wrong, but the volume mounting doesnt seem to put contents back to the host and I tried simple echo 'test' > /output.txt options too, but nothing appears.

Am I misunderstanding what Dagger can do?

I have successfully built a few different docker images like C#, Go, Java, Python - Didnt publish locally since its more complicated and need a registry but im getting the hang of things. I just have now been blocked not understanding why I cant convert a powershell script using docker mounting and getting the container output back to the host is taking me far too long. Ive probably spent 4 hours with no progress coolcrying

I cant find any examples online either. The mutli stage examples were helpful but nothing about container - host interactions so much

rich matrix May 19, 2024, 2:52 PM

#

@covert bison Oof! Sorry you ran into that. We clearly need to make that clearer in the docs (which are getting a big update right now).
#1237906912411127828 message

note: at first I thought Patrick was using "Classic Dagger" with full access to his host machine, but it turned out he was using Functions. I also thought he was perhaps doing a container publish.

#

Short answer is, use WithDirectory() instead for the content to be persisted in the export. Let me know if that unblocks you 🙏

#

@deep pebble 👆

covert bison May 19, 2024, 3:02 PM

#

Hello, thank you for the fast response. not sure if i was explaining well enough, the container is throwaway. Basically i run a 3rd party docker image, it dumps files to a folder (within the container) and then this is just placed back into the host.

rich matrix May 19, 2024, 3:07 PM

#

Hi and welcome 🙂 👋

Can you share the whole repo? Or the whole Dagger program so I can repro?

covert bison May 19, 2024, 3:07 PM

#

dagger call generate-kiota-clients --directory ./examples2 --languages CSharp --output ./out

Let me get some snippets. bit hard to share cos im hacking around on work related stuff but its personal learning.

#

https://gist.github.com/pjmagee/8bbb579fdcb790ea26997494521f7871

Gist

go.main

GitHub Gist: instantly share code, notes, and snippets.

#

That should work for you, our specs are open to the public

#

I'm probably doing something silly..

rich matrix May 19, 2024, 3:09 PM

#

It looks like you're mounting the directory from your host to /app/output which is just an ephemeral overlay mount of a snapshot (not a bi-directioinal bind mount) and then altering the contents of that directory and expecting it to show up as altered on the host.

covert bison May 19, 2024, 3:09 PM

#

that seems right

#

I have the powershell equivalent somewhere doing plain docker.. maybe it helps also

#

https://gist.github.com/pjmagee/0acc8be873b013d9882e6c6e109e263e#file-generate-kiota-clients-ps1-L127

Gist

generate-kiota-clients.ps1

GitHub Gist: instantly share code, notes, and snippets.

#

Looks like its behaving the exact same as WithMounted.

#

dagger v0.11.4 (registry.dagger.io/engine) windows/amd64

#

Just making sure im on the latest too.

#

would it be worth me trying from running dagger within WSL? so install ubuntu dagger, go to /mnt/c/Projects etc and call the same dagger function there?

rich matrix May 19, 2024, 3:24 PM

#

Would be good to try 🙏

#

I'm working on a simplified repro

#

Oh! I see. In GenerateKiotaClients you're returning output which is a String with all the stdout concatenated, but you've also altered /app/output and hope to have that peristed on the host, right?

#

You'll need to return a *Directory to do that.

#

Since with Dagger functions, you're running in a Golang runtime container (since you're using the Go SDK) that is sandboxed from the host, you need to return the Directory from the function and then decide to save it on the host explicitly in the CLI call.

covert bison May 19, 2024, 3:44 PM

#

😮

#

okay i see now, so it really does have to do with how someone understandings how to use dagger and how it executes and interacts

#

let me give that a gooo

covert bison May 19, 2024, 4:19 PM

#

rich matrix Since with Dagger functions, you're running in a Golang runtime container (since...

I'm getting closer 😄

#

var results *dagger.Directory = dag.Directory(DirectoryOpts{}) // Set up root, which will be mounted to each exec which is looped (since many docker exec calls as done)

I return results back and then

#

dagger call generate-kiota-clients --languages CSharp export --path ./examples2

#

last part is getting the files

#

I should be able to create 1 dag directory, mount the same one to the individual dag container execs and then return that original directory right?

rich matrix May 19, 2024, 4:26 PM

#

Yes. I think so.
I was working on this simple repro here. Could be fun to play with:

package main

type Mountedexample struct{}

func (m *Mountedexample) Ctr(inDir *Directory) *Container {
    return dag.Container().
        From("alpine:latest").
        WithMountedDirectory("/mnt", inDir).
        WithWorkdir("/mnt").
        WithExec([]string{"sh", "-c", "echo FOO-$RANDOM >> foo"})
}

func (m *Mountedexample) Dir(inDir *Directory) *Directory {
    return m.Ctr(inDir).Directory(".")
}

#

dagger call ctr --in-dir somedir terminal fun way to jump into the terminal with the files
dagger call dir --in-dir somedir export --path "./somedir" read in dir, alter, export back to same dir
dagger call dir --in-dir somedir export --path "./someotherdir" read in dir, alter, export back to other dir

covert bison May 19, 2024, 4:31 PM

#

shim error: exec: "sh": executable file not found in $PATH

#

damn microsoft and their slim images

#

😛

#

i think they ripped out everything haha

#

this is a really cool feature though thank you, it makes total sense to debug the output like this

rich matrix May 19, 2024, 4:35 PM

#

No problem, just add a WithExec() right after the From() to add whatever packages you need back 🙂
You can comment it out, but use it for debugging 🙂

covert bison May 19, 2024, 4:35 PM

#

make a 2nd container and mount the same dagger vol and then terminal should work

rich matrix May 19, 2024, 4:36 PM

#

My bit about the WithMountedDirectory vs WithDirectory was a red herring in this case since you're not doing a publish of the final container/image to a container registry. If so, you'll need to use WithDirectory to ensure the content is persisted.

covert bison May 19, 2024, 4:37 PM

#

gottcha, thank you!

covert bison May 20, 2024, 6:58 PM

#

Got it working really nicely with this:

func (m *DsgIcisOpenapiKiota) GenerateKiotaClients(ctx context.Context, languages []Language) *dagger.Directory {

    specs := []Spec{
        NewSpec("https://developer.icis.com/portals/api/sites/icis-live-portal/liveportal/apis/energyapi/download_spec", []Settings{
            NewSettings(CSharp, "Icis.Api.Energy", "Energy"),
            NewSettings(Go, "icis/api/energy", "energy"),
            NewSettings(Java, "com.icis.api.energy", "src/main/java/com/icis/api/energy"),
            NewSettings(Python, "icis_api_energy", "energy"),
        }),
        NewSpec("https://developer.icis.com/portals/api/sites/icis-live-portal/liveportal/apis/energyforesightapi/download_spec", []Settings{
            NewSettings(CSharp, "Icis.Api.EnergyForesight", "EnergyForesight"),
            NewSettings(Go, "icis/api/energyforesight", "energyforesight"),
            NewSettings(Java, "com.icis.api.energyforesight", "src/main/java/com/icis/api/energyforesight"),
            NewSettings(Python, "icis_api_energyforesight", "energyforesight"),
        }),
        NewSpec("https://developer.icis.com/portals/api/sites/icis-live-portal/liveportal/apis/lnganalyticsapi/download_spec", []Settings{
            NewSettings(CSharp, "Icis.Api.LngAnalytics", "LngAnalytics"),
            NewSettings(Go, "icis/api/lnganalytics", "lnganalytics"),
            NewSettings(Java, "com.icis.api.lnganalytics", "src/main/java/com/icis/api/lnganalytics"),
            NewSettings(Python, "icis_api_lnganalytics", "lnganalytics"),
        }),
    }

    dir := dag.Directory(DirectoryOpts{}).WithNewDirectory("/")

    for _, spec := range specs {
        for _, settings := range spec.Settings {
            if slices.Contains(languages, settings.Language) {

                output := fmt.Sprintf("/app/output/%s/%s", settings.Language, settings.Output)
                dirOutput := fmt.Sprintf("/%s/%s", settings.Language, settings.Output)

                ctr := dag.
                    Container().
                    From("mcr.microsoft.com/openapi/kiota:1.14.0").
                    WithExec([]string{
                        "generate",
                        "--output", output,
                        "--language", string(settings.Language),
                        "--openapi", spec.URL,
                        "--exclude-backward-compatible", "true",
                        "--log-level", "Debug",
                        "--additional-data", "true",
                        "--class-name", "ApiClient",
                    }, ContainerWithExecOpts{
                        SkipEntrypoint:                false,
                        ExperimentalPrivilegedNesting: true,
                        InsecureRootCapabilities:      true,
                    })

                dir = dir.WithDirectory(dirOutput, ctr.Directory(output))
            }
        }
    }

    return dir
}

And it now dumps all the generated API clients out exactly like you had mentioned to me

dagger call generate-kiota-clients --languages CSharp,Java,Python,Go export --path ./examples

Very nice!

#

Just to say thank you for explaining to me, i now grasp it way better than I did the other day 🙂

rich matrix May 20, 2024, 7:45 PM

#

Awesome!!!

#

🎉

lucid quartz May 22, 2024, 8:39 PM

#

related #1233557350351245403 ... snagged me too, and I'm sure I'll eventually grok it all too 🙂 Dagger functions were a big shift!

covert bison May 22, 2024, 9:23 PM

#

im addicted, i feel like I need to make mini puzzle/projects

#Should it be possible to export mounted directory content back to the host?