Tips for organizing a large Dagger project

[slender spruce]

In the organization I work with we're kickstarting a large refactor of our pipelines to use Dagger as the backbone. I'm looking for some tips and experience on how to organize large projects and maybe some pros/cons of different approaches.

Context: we've different kinds of projects (i.e. different technologies / deployment strategies). We're using Go for these.

As of now, the only real option I see is to have separate module for each kind of project (i.e. Go, Maven, Yarn, Python, etc.). There's not really any major drawback with this, except the developer experience for people maintaining these modules because of Go Workspaces and having to call dagger develop on every module to work on them. Has anyone experienced this to be a drawback in the long term?

Other approach that I'd prefer is to have a single Dagger module (let's say MyModule) that has functions for sub-modules that are normal internal Go packages, for example:

package main

import (
    "github.com/my-org/dagger/internal/go"
    "github.com/my-org/dagger/internal/maven"
)

type MyModule struct{}

func (m *MyModule) Maven() *maven.Maven {
    return new(maven.Maven)
}

func (m *MyModule) Go() *go.Go {
    return new(go.Go)
}

Unfortunately this approach doesn't work since Dagger cannot code-generate for foreign type <IMPORTED MODULE>. Yes, we can just define all the types we need in the same Go package, I just fear that it grows too large and becomes harder to understand and maintain.

So, any tips and advices or just general experiences on how you've done it in your own projects?

Thank you so much in advance for any help!

[thick vessel]

Hello! So far I haven't been in a situation where I felt the need to split my module into smaller go packages. But I'm sure it can happen. I'm hoping the codegen limitation you encountered can be solved in the future.

Dagger's own CI is pretty complex and of course, it uses Dagger. Maybe @dire oriole @surreal dirge @worthy crescent @zinc latch can share relevant information about how it's organized.

[slender spruce]

Thanks @thick vessel - Dagger's own CI is already of great inspiration. We'll start there and evolve if required.

[worthy crescent]

sorry! i just saw this!
i can maybe offer some insight - so, currently, only top-level objects can be exported, correct. however, you can define helpers and such in sub-packages if you like

being able to export from sub-packages in go would mean generating dagger.gen.go files in each sub-directory where there are potentially exported objects - but we'd probably still want to be clever in what we export, instead of grabbing everything

if that's something you want/need, i don't currently think we have a a tracking issue for that feature - it would genuinely be a good feature to have, i think i'd use it, so if there's a good use-case for it, i'd happily spend some time and implement it