Export Go binaries not working

[lunar hollow]

Hi,

I'm working a Dagger Module that exposes a build-all function to build and export a Go binary for multiple platforms to my local filesytem.

The module exposes a BuildBinary func that returns a *File. Then I have a caller function, BuildAll, that iterates over multiple platforms to export several platform-specific binaries:

func (m *GoDagger) BuildAll(ctx context.Context,
    // dir is the directory containing the Go source code
    // +required
    dir *Directory,
) (bool, error) {
    for _, platform := range []string{"linux/amd64", "linux/arm64"} {
        file := m.BuildBinary(ctx, dir, "1.22.0", platform)
        file, err := file.Sync(ctx)
        if err != nil {
            panic(err)
        }
        b, err := file.Export(ctx, fmt.Sprintf("./bin/%s/app", platform))
        if err != nil {
            panic(err)
        }
        fmt.Printf("Export: %t\n", b)
    }
    return true, nil
}

When I run dagger call --progress plain --debug build-all --dir . the binaries are not exported. What am I missing? Thanks in advance!

[hollow lotus]

I am still learning Dagger, but I understand that if you want to export a binary built from within a container, you must return a dagger.File to the caller. The caller can then write that file to their host by passing the -o flag. To export multiple binaries, you must return a dagger.Directory containing all the built files. The caller can export those files to their host by passing the -o flag.

[lunar hollow]

Thanks @hollow lotus. I've changed the method signature to return a []*Directory .

✔ initialize 0.3s
Error: "build-all" requires a sub-command
Run 'dagger call build-all --help' for usage.

However, using export --path works:

dagger call build-all --dir . --platforms "linux/amd64,linux/arm64" export --path bin

I think -o just saves the stdout into a file, which in my case seems to contain:

true
true

(1 true statement per platform). Not sure where this boolean is coming from now, though 😅

[hollow lotus]

Can you not return a single Directory and write each binary under the same path bin?

[lunar hollow]

Yep, I can try that 👍

[hollow lotus]

Something like: return dag.Directory().WithFiles("bin", binary1, binary2)?

[urban breach]

heya @lunar hollow 👋👋 (long time no see, hope you're doing well!)
currently the option suggested by @hollow lotus is the best way to handle this - merge all your dirs together before returning

this is definitely a pain, and i've hit this a lot recently - i've got an open issue to try and handle this a bit better in the future, https://github.com/dagger/dagger/issues/6675, by baking in the knowledge of multiplatform into each type, so you could return a single directory that would have multiple "variants"

@hollow lotus i just noticed your comment on that issue - you might be interested in how i'm handling this for the dagger engine itself: i just have a custom publish function: https://github.com/jedevc/dagger/blob/dagger-ci/ci/engine.go#L84

[hollow lotus]

@urban breach I ended up taking the same approach. I wrote a custom Publish function that publishes the multiple platform variants (which are built ahead of time). I feel like I am starting to get the hang of Dagger . I may have to steal the line where you are forcing compression...

I just have to plumb it all in to GitLab now... I do hate writing a GitLab YAML file

[urban breach]

🤔 do you know what would be kinda cool
if we could have a dagger module that automatically generates gitlab ci scripts from functions 🎉

i also do not enjoy writing yaml lol 😄

with github we have https://github.com/dagger/dagger-for-github, but there's no equiv for gitlab 😢

[hollow lotus]

GitHub is so much nicer. I saw that repo. I plan to move over to it with my personal projects shortly. But alas, I am stuck with GitLab for work. I am currently writing a GitLab template to do something very similar, but GitLab descends into YAML soup really quickly

[lunar hollow]

Thank you both. Now the method returns a *Directory which holds the multi-platform Go binaries. I call the module function with dagger call build-all --dir . --platforms "linux/amd64,linux/arm64" -o bin and the binaries are written to disk:

ls -lah bin
total 28928
drwx------   4 felipe  staff   128B Mar 28 15:31 .
drwxr-xr-x  12 felipe  staff   384B Mar 28 15:31 ..
-rwxr-xr-x   1 felipe  staff   6.7M Mar 28 15:29 app_linux_amd64
-rwxr-xr-x   1 felipe  staff   6.4M Mar 28 15:29 app_linux_arm64

One last question: if I wanted to do the same using the Go SDK instead of dagger call ... -o bin, shouldn't this work?

    g.Go(func() error {
        dir, err := m.BuildAll(ctx, source, "1.22.0", []string{"linux/amd64", "linux/arm64"})
        if err != nil {
            return err
        }
        _, err = dir.Export(ctx, ".")
        return err
    })

[urban breach]

yup 🎉 that works, but remember if you're running this in a module, then the export is exporting to the module container (not the host)

[lunar hollow]

I think that's the issue I'm having now. Is there a way to export it to the host?

[urban breach]

you need to explicitly return the directory and then export it from the cli right now
there's some discussions on conveniences in https://github.com/dagger/dagger/issues/5774

but, yeah, there's no longer any host access from inside modules