🧑‍💻Developer experience with checked in dagger codegen

[terse kernel]

Hi dagger team. We have begun to check in the codegen for the modules in our mono repo. We have finally been able to turn renovate bot on, additionally we have noticed a speed up not having to run dagger develop in our pipeline anymore.

What we are facing now are a few developer experience issues:

1. if making changes to a module which is a dependency of many other modules, the user must run dagger develop in many places
2. we have to continually remove the .gitignore file changes which dagger develop makes

Just wondering if you have a solution for these as of yet 🙂

🧑‍💻Developer experience with checked in dagger codegen

[signal relic]

I have a WIP to do the same with the dagger repo, which will look a bit different with modules v2 (see #maintainers for more if you're curious).

In the current world, if everything is linked to the top level module, you can run dagger develop --recursive to make sure they all get generated.

One thing I'd add to enforce this process is a step in CI to run the develop and make sure no files were changed. We do this for all generated files that are committed.

[ionic orchid]

We have finally been able to turn renovate bot on
I am glad to hear that! We'll eventually be marching towards this too. I am curious about your experience with renoavte and dagger.

Renovate obviously can't update the versions specified in comment pragmas so I see that as a gap for now.

Dagger manages it's dependencies with develop, so, doesn't that conflict with what renovate updates? Are you customizing renovate in some way?

[terse kernel]

Yep to validate the generated code we have a ci check. We are using the dag.CurrentModule() api to get the current module, surf its dependencies, get the code gen with GeneratedContextDirectory and create a change set that reveals if there is a diff between the source directory

[terse kernel]

We have excluded certain dependencies which we know dagger develop manages / pins from renovate bot. Thats how we got around it

[terse kernel]

Just wondering @signal relic how will modules v2 affect code gen. Took a look but it seems more related to improving the developer experience around mono repo development / toolchains (though as I understand it, toolchains will be deprecated)

Also are there any plans for dagger to release a toolchain or otherwise for performing a codegen check - that way we can rely on your implementation 😉

[hard gull]

Should be neutral. Parallel complementary efforts

[hard gull]

Yes absolutely.

[signal relic]

Yep there's some parallel work (mainly by @vagrant elm ) to improve the codegen such that less code would need to get committed. Some of that has already shipped recently, but more effort is in progress. The goal is that only the custom bindings from your dependencies would need codegen and the rest could get passed through to the published dagger library

[hard gull]

I think it should also help with renovate config right? Since dependencies that are today mixed into your module's go.mod, will move into the core library and in particular the telemetry package?

If I'm not mistaken, that would mean less special cases needed in replicate's config?

[terse kernel]

Ah yes it sure will 🙂

Good point, I'll let the team know

Just wondering however if there will be a way to mark modules as "code gen checked in mode" so that running dagger develop does not produce the .gitignore file.

We have to keep removing that file all the time.

[hard gull]

Yeah that deserves its own issue I believe... unless that's already tracked somewhere @vagrant elm @signal relic ?

[signal relic]

Even better, there's an undocumented config I discovered recently 😅 In your dagger.json, you can put codegen: { automaticGitIgnore: false } and it should not be created

[hard gull]

oooh... welp

[signal relic]

for example: https://github.com/kpenfound/dagger/blob/e7f981351e3d1c166282a523cf99ba273de980bc/dagger.json#L160-L162

I ran into the same thing working on the branch to commit the generated files in dagger/dagger 😛

[terse kernel]

Ahah

Happy to use that but want to make sure it stays haha

[terse kernel]

@lethal stirrup fyi

[signal relic]

Yes I don't think it would change with modules v2 because the codegen configuration would stay with the dagger.json

[terse kernel]

Something thats also an interesting challenge with checking code gen in, is that parallel work can often lead to a broken state with the codegen.
For example say module A depends on C and D. If parallel work happens to C and D, it might change the some of the generated code of A's internal/dagger/dagger.gen.go file.

To combat this we have had to make sure every PR branch is rebased off of main, before merging. We have done this with a ruleset

[hard gull]

is this a dagger-specific problem, or would the same thing happen anytime eg. go generate is used?

[terse kernel]

Good point. It would be with other repos too. I guess its affecting us more because ours is a mono-repo with lots of modules that people are working on in parallel.

That is driving up the changes to codegen so rebase before merge is the only way to sanely combat it

If we were dealing with a single module per repo, it would be less of a problem because there would be less people working on it in parallel.