Shared code in libraries (not modules)

[fading veldt]

I'm trying to have some reusable code in a simple go module. Not a dagger module, just a regular library.

I've created the following:

dag-common/common.go
dag-common/go.mod
dag-common/go.sum

mod1/dagger.json
mod1/go.mod
mod1/go.sum
mod1/main.go

dag-common has a dagger.io/dagger dependency.
mod1 is a dagger module and has a dag-common dependency.
mod1 declares a function called apt-install which uses code from dag-common

This doesn't quite work though, because the types provided by codegen to mod1 aren't quite the same as the ones from dagger.io/dagger used by dag-common.

So when I run dagger call -m mod1 call apt-install --packages=curl I get an error:

./main.go:27:52: cannot use dagCommon.AptInstall(packages) (value of func type "dagger.io/dagger".WithContainerFunc) as "dagger/mod-1/internal/dagger".WithContainerFunc value in argument to dag
.Container().From("ubuntu:latest").With

Is this scenario supported?
I have the example here: https://github.com/idlsoft/dagger-example-1

[quiet remnant]

similar thread that might be interesting: #1412320015750922303 message

[smoky mountain]

@fading veldt this thread @quiet remnant is referencing is currently the recommended approach

[fading veldt]

The thread clarifies the terminology pretty well, dagger modules vs go modules.
It seems if your library code uses dagger apis then it needs to be a dagger module.
Useful for cross-language, but has limitations on what you can do, for example you can't have a function return dagger.WithContainerFunc

[smoky mountain]

do you need that though? consider this example:

//-- sub.go --
type Sub struct{}

func (m *Sub) WithGit(c *dagger.Container) *dagger.Container {
    return c.WithExec([]string{"apk", "add", "git"})
}


//-- test.go --
type Test struct{}

func (m *Test) Test(ctx context.Context) *dagger.Container {
    return dag.Container().With(dag.Sub().WithGit)
}

^ if you install sub in test , you can still use the WithContainerFunc pattern anyways in the upstream modules

as long as your common package module has the signature func (c *dagger.Container) *dagger.Container, you can use it directly.

[fading veldt]

That only works if the function has no arguments. So something like

func AptInstall(packages []string) dagger.WithContainerFunc {
    return func(c *dagger.Container) *dagger.Container {
        if len(packages) == 0 {
            return c
        }
        return c.WithExec([]string{"bash", "-c", "apt-get update && apt-get install --no-install-recommends -y " + strings.Join(packages, " ") + " && rm -rf /var/lib/apt/lists"})
    }
}

won't be as clean.
Also it's costlier, because it has to cross module boundaries.
I assume there is a reason behind dagger develop creating a private copy of dagger api, but I wish it was possible to use dagger.io/dagger directly.

[smoky mountain]

yeah.. you're right. For these cases it's not currently possible unless you do some replace or workspace shenanigans I assume. I recall @paper bramble started to do some work so all modules share the same core types based on the Dagger SDK but that requires some non-trivial changes to all the SDK code generation logic 🙏

[fading veldt]

An added benefit would be being able to write the same code for a module or a dagger app

[fading veldt]

Just had a little experiment.
Added replace dagger.io/dagger => ./internal/dagger to mod1/go.mod
Had to also add a go.mod under mod1/internal/dagger.
This seems to satisfy the go compiler, but there are still things that need to be handled by dagger.

1. It doesn't copy mod1/internal into the module container, so it needs to also generate the go.mod there
2. It needs to do the same for querybuilder and telemetry

[floral fulcrum]

you can include paths to 'dagger.json` to copy additional files in the module container

we use that pattern ourselves for example: https://github.com/dagger/dagger/blob/main/toolchains/engine-dev/dagger.json#L7

[fading veldt]

I do exactly that, but the problem is that every module imports dagger APIs from their own module_name/internal/dagger.
So the code from one module cannot be used directly in another.

However I may have found a hack. It will involve me figuring out how to mess with codegen.
It'll generate pretty much the same code, but then add this to go.mod

replace dagger.io/dagger => ./internal/dagger

And create a go.mod under internal/dagger with

module dagger.io/dagger

The same for querybuilder and telemetry, most likely.

The generated code can then import dagger.io/dagger instead of module/internal/dagger, but will be looking at exactly the same code.
This way any code from any module can be perfectly portable as long as it only references native dagger APIs and not module dependencies.

I just need to figure out how to build and test this, it's been a while since I've looked at the source code.

[smoky mountain]

@crystal beacon to 👀

[fading veldt]

I've started writing some code here: https://github.com/idlsoft/dagger/tree/codegen_with_replace. It's very much WIP, comments welcome.