Having issues with creating a Go module that depends on dagger.io/dagger

[round oracle]

Hi,

I am working on moving a couple of our team's projects from the common Jenkinsfile/bash script setup to using Dagger.
We ended up with using the Go SDK that is provided.

I am not sure on how to formulate my problem, but I am giving it a go here.

Moving to dagger has gone well in the first project. For convenience, we have created utility packages inside the
internal subdirectory, as that seemed for us to be the right place to put them. These packages depend on the generated
dagger/<package>/internal/dagger Go package.

To hinder unnecessary code duplication, I wanted to see if it was possible to separate one of our utility packages into its own
Go module for better re-usability. In this process, I changed out the package's dagger dependency from dagger/<package>/internal/dagger to
dagger.io/dagger. For me, that made sense, since I did not want to have a separate Dagger module for this utility
package; I only wanted to depend on the Go SDK.

Note that this utility package is itself placed inside a private repository, like the rest of our team's projects.

This is where things started to go wrong: When I went ahead in trying to use this utility package, located inside its
own dagger-utils module, I get several error messages in the similar format below:

./main.go:139:14: cannot use <...> (value of type *"dagger.io/dagger".Container) as *"dagger/<package>/internal/dagger".Container value in assignment

I suppose that the error message makes completely sense, as the dagger module depends on a generated dagger package
whereas the imported module depends on the Go SDK available from dagger.io/dagger.

So my question boils down to: How can I create a common Go module that can depend on the Dagger Core API so I can have a
common module that can be re-used in Dagger modules spread out over multiple repositories?

Thanks!

[granite crescent]

ah, this is mildly tricky. so, the generated code expects you to use the generated internal package - you can't mix and match

the "dagger way" would be to have all your common functionality represented in it's own dagger module - then install that module in your other places

a potential hackier way in go, might be to use dagger.io/dagger in your common package, and then you might be able to use a replace directive in your downstream modules? maybe? i haven't tried that, no guarantees if it will work

[round oracle]

Thanks for the reply.

I attempted to change the Go module to a Dagger module and that seems to work, although I must say I don't understand why it would work. To give more context, this module provides a data structure with some methods that makes it easier to install files and directories in a container:

// the first iteration of this package, located in internal/installresult
package installresult

import (
    "maps"
    "strings"

    "dagger/nrfdl/internal/dagger"
)

type InstallResult struct {
    Files     map[string]*dagger.File
    Directory map[string]*dagger.Directory
    EnvVars   map[string]string
}

func New() *InstallResult {
    // <...>
}

func (r *InstallResult) InstallOn(container *dagger.Container) *dagger.Container {
    for path, file := range r.Files {
        container = container.WithFile(path, file, dagger.ContainerWithFileOpts{Expand: true})
    }

    for path, dir := range r.Directory {
        container = container.WithDirectory(path, dir, dagger.ContainerWithDirectoryOpts{Expand: true})
    }

    for env, val := range r.EnvVars {
        container = container.WithEnvVariable(env, val, dagger.ContainerWithEnvVariableOpts{Expand: true})
    }

    return container
}
// <...>

I noticed that when writing the Dagger module for this package, I had to jump through some hoops as types like map is not available in GraphQL. It feels kind of hack-ish.

I guess I struggle to see when the right approach is to use a Go module or a Dagger module for extending a Dagger module.

a potential hackier way in go, might be to use dagger.io/dagger in your common package, and then you might be able to use a replace directive in your downstream modules?

I don't think this would work, as you might run into issues with the generated core structures associated with the allocated Dagger engine client. I guess that might be, as you already said, even more hackier.

[rugged ivy]

@round oracle don't have a lot of time to reply here but going with sub-module is the recommended and most straightforward way to approach for this currently.

I noticed that when writing the Dagger module for this package, I had to jump through some hoops as types like map is not available in GraphQL. It feels kind of hack-ish.

Yes this is the case. Since maps are not currently supported natively in GraphQL, the way to implement this is via typed arrays for the moment 🙏

[oak solar]

I'm curious what @nocturne cosmos and @forest bough feel about this one 🙂

[forest bough]

yeah its a tricky one

use a Go module or a Dagger module for extending a Dagger module.

Definitely depends on your setup and what kind of logic you're breaking out. If its a utility you only expect to use in your dagger modules, i'd probably do it in another dagger module because it will be more portable across other modules (maybe other SDKs too?) and more visible in dagger cloud. If its something that might be used in your non-dagger go-code, I guess a go module is the answer!

[rugged ivy]

but in this case a Go module doesn't make it better since you have this module specific generated imports which doesn't make it truly "modular" isn't it?

^ mostly referring to the example the op posted above

[round oracle]

I suppose you are referring to git submodules? We've been trying to avoid using git submodules as much as we can as it introduces some extra complexity with Jenkins and general setup required to work on our developers' machines. The most ideal would be that we could just invoke dagger and the dagger module is set up with all of its dependencies.

It seemed to work well with letting this module live in a separate private repository and have dagger fetch it through dagger install.

On the topic of recommended layouts of dagger modules, it seems that either using monorepos or sub-modules is the way to go?

[rugged ivy]

I suppose you are referring to git submodules? We've been trying to avoid using git submodules as much as we can as it introduces some extra complexity with Jenkins and general setup required to work on our developers' machines. The most ideal would be that we could just invoke dagger and the dagger module is set up with all of its dependencies.

sorry I meant dagger modules, not sub-modules

On the topic of recommended layouts of dagger modules, it seems that either using monorepos or sub-modules is the way to go?

yep, exactly. Pretty much the same ways as you'd use a package in your favorite language

[oak solar]

but to be clear we are not recommending the use of git submodules 🙂

[granite crescent]

indeed, please, no git submodules

(not even interaction with dagger, they're just so difficult 😢)