Failure to embed dagger module in another dagger module (within a monorepo)

[kind grail]

Hi, I have a monorepo in go, and I would like to use dagger to accomplish 2 things:

1. have a single entrypoint for running the whole monorepo in a CI/CD
2. have a dagger module for each app/feature inside the monorepo, for devs to run locally

In order to achieve this, the approach is:

1. Have a single dagger module in the root of the monorepo, which invokes every app/feature dagger module
2. Have one dagger module for each app/feature
3. Have a standard dagger module to be used by 2. in order to avoid writting one million times a method to (e.g.) lint go

The general code for 2. would look like

package main

import "dagger/app-ci/internal/dagger"

const (
    workDir = "/app/apps/app"
)

type AppCi struct {
    dagger.StackStandarCi
}

func New() *AppCi {
    stackStandarCi := dag.StackStandarCi(dagger.StackStandarCiOpts{WorkDir: workDir})

    return &AppCi{*stackStandarCi}
}

However, when I run dagger call lint-go (available on my standard ci), it fails with failed to serve module: input: module.withSource.initialize resolve: failed to initialize module: failed to add object to module "app-ci": failed to validate type def: object "AppCi" field "StackStandardCi" cannot reference external type from dependency module "stack-standard-ci". It looks like an issue with dagger, since the go code seems fine, and I would achieve having all available methods in dagger.StackStandarCi .

Any experience on how this could be achieved?

[proven sequoia]

@kind grail hey there! we follow this pattern in our dagger/dagger repo here: https://github.com/dagger/dagger/blob/main/.dagger/main.go#L149-L166

[tawny dagger]

The difference in what @proven sequoia linked is that the types are all defined in the same module. A limitation of modules today is that their functions (or structs) can't return types from a different module

[kind grail]

Thanks for sharing!! Your approach indeed seems different from what I wanted, however it might be useful as well. The problem seems to be with dagger generator having some limitations, for example I am also facing those when doing dagger init inside a go workspace, where things go awry.. Either way, I'll keep looking and probably follow the original idea, although I will need to declare every function inside every feature or app dagger module, calling the standard one

[proven sequoia]

also facing those when doing dagger init inside a go workspace,

this should work as dagger should put all the Dagger specific files under a .dagger sub-directory which shouldn't affect any of your current workspace files

[proven sequoia]

re: point number 3 here, seems like a good problem for "interfaces" to solve somehow. (https://docs.dagger.io/api/interfaces/). Having said that, interfaces are still under development and they currently have some rough edges. @kind grail I think it's worth mentioning this use-case in #daggernauts and start a thread there so we can brainstorm ideas on how to better support this use-case

[kind grail]

About dagger init, if I run dagger init --sdk=go inside a go workspace (it has a go.work file in a parent folder), I get the following error and nothing is generated by the command ```
running post-command: go mod tidy
needs another pass...
starting GOROOT= GOPATH=/go GO111MODULE=off GOPROXY= PWD=/src/my-app/features/my-feature/.dagger go list -e -f {{context.ReleaseTags}} -- unsafe
starting GOROOT= GOPATH=/go GO111MODULE= GOPROXY= PWD=/src/my-app/features/my-feature/.dagger go list -f "{{context.GOARCH}} {{context.Compiler}}" -- unsafe
2.043976ms for GOROOT= GOPATH=/go GO111MODULE=off GOPROXY= PWD=/src/my-app/features/my-feature/.dagger go list -e -f {{context.ReleaseTags}} -- unsafe
starting GOROOT= GOPATH=/go GO111MODULE= GOPROXY= PWD=/src/my-app/features/my-feature/.dagger go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,CompiledGoFiles,DepOnly,Imports,ImportMap,Export,Module -compiled=true -test=false -export=true -deps=true -find=false -pgo=off -- .
11.958578ms for GOROOT= GOPATH=/go GO111MODULE= GOPROXY= PWD=/src/my-app/features/my-feature/.dagger go list -f "{{context.GOARCH}} {{context.Compiler}}" -- unsafe
54.237627ms for GOROOT= GOPATH=/go GO111MODULE= GOPROXY= PWD=/src/my-app/features/my-feature/.dagger go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,CompiledGoFiles,DepOnly,Imports,ImportMap,Export,Module -compiled=true -test=false -export=true -deps=true -find=false -pgo=off -- .
Stderr:
Error: load package "my-app/features/my-feature/.dagger": package name is empty

[kind grail]

Using interfaces would be an interesting approach, I was having issues since each module has generated structs, which are not the same, so they cannot be "shared". That said, let me post this use case in the channel you mention.. Thanks for your help!!

[proven sequoia]

@kind grail are you sure you don't have a .dagger folder already where you're running dagger init?

take into account that since it's a hidden folder, your IDE / File explorer might not be showing it by default

I just ran a quick test and even if I have a go.work in a top level folder, I can successfully dagger init other modules in sub folders.

i.e:

marcos:tmp/lala $ tree . -I "internal"
.
├── dagger.gen.go
├── dagger.json
├── go.mod
├── go.sum
├── go.work
├── lele
│   ├── dagger.gen.go
│   ├── dagger.json
│   ├── go.mod
│   ├── go.sum
│   ├── LICENSE
│   └── main.go
├── LICENSE
└── main.go

^ In this case, running dagger init in lele worked correctly even though a go.work file exists in the top level

[kind grail]

Yes, I am sure it is not working for me, using dagger v0.14.0 (registry.dagger.io/engine:v0.14.0) linux/amd64 and go version go1.23.1 linux/amd64. Running dagger init --sdk=go. Adding --name or --source does not make it work either. The error is the one shown in my previous comment. What does work for me is copy pasting a different dagger module, renaming as needed and adding the module manually to my go.work file. Then I can run dagger develop --sdk=go. If I remove the module from my go.work file, that last command still fails.

[proven sequoia]

@kind grail still can't reproduce. If you have a minute can you help me by double checking this works?

1. Clone this repo: https://github.com/xmlking/go-workspace
2. Make sure you're using Dagger v0.14
3. cd go-workspace
4. dagger init --sdk go
5. cd cmd/app1
6. dagger init --sdk go

^ in my case, both dagger init calls succeeded

[icy pecan]

I may have been having a similar issue, and I think it's because of this:

func (sdk *goSDK) RequiredPaths(_ context.Context) ([]string, error) {
    return []string{
        "**/go.mod",
        "**/go.sum",
        "**/go.work",
        "**/go.work.sum",

If you run with -i it'll take you to the module's container, and under /src you'll see your module's sources and all the orther go.* files.

I ended-up adding a dummy go.work in my .dagger directory

btw I'm trying to implement exactly the same thing, would be cool if interfaces supported this scenario.

[kind grail]

Your example works fine. It took some time until I found the difference. My go workspace is not in the root of the repository, but instead it is nested inside one folder. If you move the whole code into a "younameit" folder, dagger init --sdk go will fail for your steps 4 and 6. Not sure if that's expected, but it seems to be the root cause of the issue.

[proven sequoia]

yes, this is definitely something that needs fixing. Follow up of https://github.com/dagger/dagger/issues/8123 and https://github.com/dagger/dagger/issues/8496 cc @atomic jay @distant plume

[kind grail]

Thanks! I will keep an eye on those... Back to my original problem, I have the following in my "DaggerStandard" module, which I wanted to use from my feature/apps dagger modules. However, WithDockerLogin cannot be called from those modules, I guess because it does not return (string, error), but instead returns a container. Any clues on how I could make this work? ```func (m *DaggerStandard) WithDockerLogin() *DaggerStandard {
jfrogAuth := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", os.Getenv(ARTIFACTORY_USER), os.Getenv(ARTIFACTORY_PASSWORD))))

c := dockerLoginConfig{
    Auths: map[string]dockerAuth{
        jfrogDockerRegistry:          {Auth: jfrogAuth},
    },
}

b, err := json.Marshal(c)
if err != nil {
    log.Fatal(err)
}

m.Container = m.Container.
    WithExec([]string{
        "sh",
        "-c",
        fmt.Sprintf("mkdir -p $HOME/.docker && echo \"%s\" > $HOME/.docker/config.json", string(b)),
    })
return m

}```

[distant plume]

I will open an issue for this, thanks for the report

[proven sequoia]

returning a container should be ok

what do you mean by "cannot be called from those modules"?

[kind grail]

sorry, what I mean is I have one dagger module DaggerStandard, with the method I have shared. I have other dagger modules (inside each of my monorepo features or apps), which would ideally use the DaggerStandard dagger module. This is possible for other methods, such as func (m *DaggerStandard) LintGo() (string, error), which are made available when installing DaggerStandard on my other modules. However (I guess it is due to it returning *DaggerStandard instead of (string, error), there is no code generated for dag.DaggerStandard().WithDockerLogin() in my other modules.
The reason why I want to do this is to run things in the same container, for example, a container that has the docker config required to run docker build and docker push. Another situation might be to install node dependencies in a container that has a .npmrc file in it.
After checking some dagger modules, it looks like the best idea would be to have a single dagger module in the root of the repository, but that somehow limits the ability to have different apps or features overwrite standard tasks, and seems like lacking modularity, and having to centralize everything. It is likely that I am missing things...

[proven sequoia]

That's strange, you should see the WithDockerLogin in the modules that reference the Docker standard one.

Are you sure you reloaded your editor and ran dagger develop in your parent module after changing the DaggerStandard one?

As you can see here https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/go@d9ba06776c4c1ccf6f329bd862b9b439c4582ab6, modules can return *Container types

[kind grail]

Yes, indeed weird, but for some reason it is not working in my case. The generated code looks like

    query *querybuilder.Selection

    checkDependencies        *string
    dockerBuild              *string
    endToEndTestNpm          *string
    generateBackendHttpInfra *string
    generateGoCode           *string
    id                       *DaggerStandardID
    lintGo                   *string
    lintNpm                  *string
    unitTestGo               *string
    unitTestNpm              *string
    validateManifest         *string
}```

We are evaluating to go with a single module at the root of the repository, start with a single call to run the whole CI, and then add convenient calls to run single tasks for each app and feature.

In this regard, the idea is to share a single container with the whole repository as source and try to parallelize executions (e.g. tests for each feature).. we would be using go routines for that, I hope it makes sense and we can make it work fine and fast.

[proven sequoia]

Just created a module with the same function name and seems to be exported correctly

you're sure that the parent module is installing the correct DaggerStandard version which has that function?

[kind grail]

yes, it was happening to me, not sure why though.. we have moved to a single dagger module, so I cannot test that anymore. I might have had something else preventing those function names from being generated, I cannot really tell, sorry