maintainers

[tepid nova]

@still garnet first module ported to workspace API 🙂 In dang of course

pub description = "A Dagger module for markdownlint, a markdown linter. https://github.com/DavidAnson/markdownlint-cli2"

type MarkdownLint {

  """
  The version of markdownlint to use.
  """
  pub version: String! = "latest"

  """
  Filter the current workspace for use by markdownlint.
  """
  pub inputs(ws: Workspace!): Directory! {
    ws.directory("/", include: ["**/*.md", "**/.markdownlint*"])
  }

  """
  Build a sandbox with markdownlint installed and input files mounted.
  """
  pub sandbox(ws: Workspace!): Container! {
    container
      .from("tmknom/markdownlint:" + version)
      .withWorkdir("/app")
      .withMountedDirectory(".", inputs(ws), owner: "nonroot")
  }

  """
  Lint markdown files in the current workspace.
  """
  pub lint(ws: Workspace!): Void @check {
    sandbox(ws)
      .withExec(["markdownlint", "."])
      .sync
    null
  }

  """
  Fix markdown files in the current workspace.
  """
  pub fix(ws: Workspace!): Changeset! @generate {
    let ctr = sandbox(ws)
    let before = ctr.directory(".")
    let after = ctr
      .withExec(["markdownlint", "--fix", "."], expect: ReturnType.ANY)
      .directory(".")
    after.changes(before)
  }
}

[still garnet]

pretty sweet

though technically 🤓 the caching seems suboptimal?

[tepid nova]

Devloop is a little long - rebuild a new dev engine playground each time I edit the module. I would love for that one dang file to get livesynced into my playground somehow 🙂

[still garnet]

i guess it'll do the level-2 caching (not the module functions but the things they call) anyway though, probably makes very little difference

[tepid nova]

Ah you mean the passing around of workspace everywhere?

[still garnet]

yeah. but, now that i think about it, it doesn't buy you much in the end to pull it up into a constructor

[tepid nova]

forgot about that

I guess it's just the overhead of executing the module's functions themselves?

[still garnet]

the sync is gonna happen either way, and there's basically no overhead to calling those outer functions

yeah

[tepid nova]

Well there is some overhead... But I guess not a ton

let me try to remember how to write a constructor in dang

[still garnet]

it's a new thing, different from what i showed in SF

https://github.com/vito/dang/pull/19

[tepid nova]

My claude skill missed that apparently

[still garnet]

worth reinstalling locally too and upgrading the Zed extension so you can get the auto fmt (if not already)

[tepid nova]

I totally forgot how to do that. Tried calling dagger -m github.com/vito/dang binary -o ~/.local/bin/dang but it looks like I can only get a linux build

[tepid nova]

@still garnet UPDATE: claude figured it out for me. I updated my "dang module dev" skill for next time. Dang auto-completion now fixed in Zed

[obsidian rover]

👋 looking for reviews on https://github.com/dagger/dagger/pull/11871 🙏 (it's the second pass on the PR)

[still garnet]

sorry missed that, i just do go install github.com/vito/dang/cmd/dang@latest

[tepid nova]

dang question: is there a way to remove the last element of an array?

or: to remove the last component of a path?

[tepid nova]

Update: found trimSuffix (in my particular case I know the exact content of the last component path I want to trim)

[still garnet]

no, there wasn't a good way - added now: https://github.com/vito/dang/commit/047455ae8e1278b49b0050aef2d35852cc8208a9

[tepid nova]

niiiice 🙂

[leaden glade]

🤔 Am I the only one not able to generate typescript anymore?

$ dagger generate "typescriptSdk:clientLibrary"
✔ typescriptSdk:clientLibrary 14.6s 2×⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⣿⡀⡀⣿⡀
Error: get patch: file size 134250496 exceeds limit 134217728 [traceparent:6b1cb51de9a91ccfb74d142c3f437924-d538e4fe981dfac5]

All the other generate commands seem fine

[still garnet]

works locally

[leaden glade]

ok, I'll investigate more on my side, something's weird here then

[still garnet]

@civic yacht saw a new flake on main with TestCacheArbitraryConcurrent: https://dagger.cloud/dagger/traces/cd7f30fb96b3061413d7129b8daa3ea1?logs=&span=2d064f21a06176d0#2d064f21a06176d0:L40

--- FAIL: TestCacheArbitraryConcurrent (0.04s)
    cache_test.go:1132: assertion failed: 1 (int32) != 2 (int32)

[tepid nova]

@leaden glade now that generate is shipped, how do you feel about 1) moving module codegen from develop to generate, and 2) moving client generation from client generate to generate?

Would be a nice simplification to have it all in one place

[tepid nova]

This is an interesting one: my dang module just failed to load because codeberg.org is down...

Turns out:

• Dang SDK is on github
• But the github repo has submodules on codeberg
• Dagger git implementation force-fetches all submodules
• codeberg was down
• ...dagger git op fails
• ...dang sdk load fails
• ...my module load fails

So in effect, our CI now has a hidden runtime dependency on codeberg.org

[leaden glade]

I agree on the simplification. But not sure how that would work. Where the function will be defined?
What I mean is would it be a way to call the equivalent of develop but from generate CLI command, or would it be a way to expose a generate function? So by doing a dagger generate -l on a module the function will be visible.
One other solution could also be to have the module code itself, and the workspace on top of it could contain a sdk-related module that will provide this generate function?
Not sure I express it clearly enough 😅
But yes, happy to work on that and propose something 🙂 I just need to think more and write down a plan

[tepid nova]

I agree it requires careful design. But worth it IMO. Would reduce command (and concept) sprawl

We have a similar problem with check -> would be nice to have builtin checking also available in dagger check. Like checking whether a module actually loads... (Something @median yew just hit the other day, at the moment the workaround is dagger --eager-runtime, not exactly intuitive) cc @charred lotus @obsidian rover who were working on that

[civic yacht]

@Erik Sipsma saw a new flake on

Had all the integ tests passing for my PR but did manual testing and hit a deadlock when running ./hack/with-dev dagger call generated. Thought it was a dagql problem but eventually realized it was deadlocking in the buildkit solver (which dag-op is still using in this PR).... Had codex spin on it overnight and after 4 hours it figured out a bit more but not the root cause.

There's zero chance I'm wasting time on this, so I'm gonna just expand the PR to be "the whole thing". So all the current changes (e-graph implementation) plus:

• expanded dagql persistence and associated pruning support
• the actual tearout of all the buildkit solver and related parts

The PR is currently +6,913 -3,310, it's probably gonna double at least, so I apologize in advance to reviewers 🙂 I just don't see a different way at this point

[tepid nova]

https://giphy.com/gifs/barstoolsports-barstool-bssadvisors-be-advised-WOqPbwiIT5xM5CrH2a

[still garnet]

will anyone miss listen --disable-host-read-write if i just nuke it from orbit? iirc we used it for the Playground but that's long gone

[obsidian rover]

nope, personally can go ahead

[civic yacht]

no, please destroy

[tepid nova]

Update: I ported our (non-reusable) docs-dev module to 2 reusable modules, using the workspace API:

• markdownlint
• docusaurus

A few rough edges, but VERY happy with the result so far. Note: requires a dev engine to run. EDIT: ooh I think actually it's a bug in my module. Will check tonight.

https://github.com/dagger/dagger/pull/11876

One rough edge: what does Workspace.directory(".") (and all relative paths) resolve to @still garnet ? I assumed the workspace path (as opposed to the workspace's rootfs), but I think maybe it's something else?

Playground

dagger -m github.com/dagger/dagger@main call \
 engine-dev \
 playground \
 with-directory --path=./src/dagger --source=https://github.com/dagger/dagger#dogfood-workspace-api \
  with-workdir --path=src/dagger \
  terminal

[leaden glade]

I tried on my personal laptop (both are very close, same cpu, same os, etc) and it works.
I reboot my work laptop just to see if some magic happened but I still have get patch: file size 134250496 exceeds limit 134217728
That's really weird.
Good thing is it looks like I'm the only one on a single machine to have the issue, but I don't understand why.
And not sure what component I should nuke to have a chance to be back to a normal state
If anyone has an idea 🙏

[leaden glade]

on a worktree, different from my main branch, the generate function is passing...

[leaden glade]

very simple PR, just re-generating the two files that were missing, so PRs can be green 😉
https://github.com/dagger/dagger/pull/11915

[leaden glade]

And the winner is... sdk/typescript/node_modules!
If I remove it, everything's fine!
That explains why it depends on the machine / clone.
I'll see to properly ignore it

[tepid nova]

<@&946480760016207902> I gave 2 different AIs the same writing test: summarize a PR from the diff. Which one do you prefer and why?

• AI 1: https://gist.github.com/shykes/15cf3d0624ab4cc65b7f4a44296e0ae0
• AI 2: https://gist.github.com/shykes/48b7f8a101522a97cfa804c3c9d5e834

[civic yacht]

48b7f8a101522a97cfa804c3c9d5e834 is easier to follow for me, it's more human

[tepid nova]

Thanks. Also any thoughts on the readability of this changelog? https://gist.github.com/shykes/c71882508105862c15ef37a5f40705d8

[wild zephyr]

agree that this one feels more natural to follow

[civic yacht]

didn't read every line but overall looks good, function caching one made sense to me. Could be helpful to group them a bit more (i.e. new features vs. perf improvements etc.), but also sometimes cleanly categorizing is harder than it actually provides benefit, so no biggie

[tepid nova]

Yeah I'm going to do a categorization pass 👍 Was just curious re: writing style

It's apples to oranges, because the PR summary is a one-shot prompt, whereas the changelog is the result of an extreme micro-management loop on format, style, themes etc

[tepid nova]

Workspace API & file paths 🧵

[civic yacht]

Good progress on this, all of TestDirectory passing now with zero buildkit or dag-op involved! https://dagger.cloud/dagger/traces/b8c838310b1884e51cedd6f4a571ad99

Particularly proud of the fix for TestDirectory/TestDirCaching there. That test relies on the fact that a lazy operation, withoutFile, turns out to be a no-op when the specified file path doesn't exist when the operation actually runs, and thus afterwards shouldn't impact cache keys. More complicated than it seems.

With buildkit, the only possible way to get that working was to unconditionally enable extremely expensive content hashing of every single directory ever mounted into any exec. Nothing else was even possible. Extremely blunt+expensive hammer.

In the new dagql world, it literally is just a few lines of code that run after the withoutFile operation finishes un-lazying. It checks if it removed any files and, if not, tells the dagql cache that it should be considered equivalent to the original parent operation (i.e. it was a no-op). After that all the e-graph union-find magic kicks in and the caching just magically works as expected, and with O(1) overhead

And the best part is that rather than wracking my brain for hours trying to figure out how to get it to work in the extremely complicated previous system, this was instantly clear+obvious and worked first try 🥲 The promised land is near!

[obsidian rover]

As the release god is a funny one, as usual, a CVE is found right after a release 🤣

Is it handled as part of the post-release @still garnet ? or can i bump it on the side

[still garnet]

@obsidian rover I rolled that into my post release PR yeah

All green now, just needs approval

[leaden glade]

Now version has been released, https://github.com/dagger/dagger/pull/11910 is ready for review.
It splits the python test suite by version and test type to make it easier to work with python SDK (and keeping the full matrix support through checks)

[tepid nova]

release question. at the moment we manually "bump" the target engine version for different parts of our build , eg. our docs.

Would it make sense to derive it from git state? for example we could have a latestRelease() or latestSemverTag() in our version module. So setting the new release tag would be the source of truth. wdyt?

[coral vector]

I don't have any projects as complex as Dagger, but this is how I do it on my simpler stuff that uses Dagger. Works nicely so far

[stuck bloom]

Are workspaces ready to test with 0.20.0? Or does it need another release or two?

[tepid nova]

The workspace API is ready to test in 0.20 (we wired it into toolchains as a hidden opt-in). But the new configuration system and UI are still in a branch (and that branch is also ready for testing 😁)

[stuck bloom]

Hm, I'll try to test on my personal mac. Running dev builds at work is a pain. Not a fault of Dagger. It's just too many variables.

[tepid nova]

I get it. I personally only run dev engines inside ephemeral playgrounds running inside stable dagger 🙂

[stuck bloom]

I did that at home too and it was really cool DX :).. There are things that won't work at work like we can't do stuff like apk add blah because it needs to be proxied via our package manager. Those are specific things that's hard to account for in a public build like dagger. It's also partly the reason I have to maintain my own daggerverse.

[tepid nova]

Oh right. Maybe by chaining more calls on that playground image? I do that to inject my 1password service token, so that op:// secret references work seamlessly. Maybe there's an equivalent to inject your apk proxy?

[stuck bloom]

Maybe by chaining more calls on that playground image
I don't think that will help with the builder images. The configurations for those package managers are internal to those images. So what we do is pull the image and configure the proxy on the fly.

One way I can think of resolving this is to standardize what is used in the build process wherever there is a package install happening so the person triggering the build can provide a builder image that's configured to work for them.

[obsidian rover]

Bascially having in-engine/image cache ?

[stuck bloom]

hm, don't think I follow sry! I meant, wherever you have to do apk add ... or apt-get install in the dagger build, let the user provide the image. It's ideal if there was only one type of image. Either, debian or alpine so you only need 1 external image.

Any image that already comes with the tools installed should be fine.

[leaden glade]

🤔
To never be able to run any apk/apt/... is a very strong requirement in my opinion, and to provide external images always comes with side issues like this image must be up to date, it must follow what we are using. And every time those images will not be in sync we are introducing more complexity and reasons to fail.
My question is outside of dagger (and maybe naive), but if the thing is to proxy calls to package managers, couldn't it made transparently on the infra side? In a way containers/machines just run classical commands and it will just work?

[stuck bloom]

To never be able to run any apk/apt/... is a very strong requirement in my opinion
It's not never. You can run it as long as the proxy is configured. I run them fine in my builds. You have to edit OS local config files to set that up.

but if the thing is to proxy calls to package managers, couldn't it made transparently on the infra side?
While it sounds simple I don't think it's as simple from an implementation standpoint. I also don't know enough about our artifact registry to understand if it's even possible. There are way too many variations of package managers to easily make something like that work.

and to provide external images always comes with side issues like this image must be up to date, it must follow what we are using.
I agree. it's not straightforward. Wherein lies the problem 🙂

[fair ermine]

@still garnet I opened a PR that regenerate the tests so we can have a green check (thx for the command): https://github.com/dagger/dagger/pull/11938

Feel free to follow up with a more elaborate fix 😄

[still garnet]

(Claude) found the root cause: https://github.com/dagger/dagger/pull/11939

[fair ermine]

Amazing! Imma close my PR in favor of yours

[fair ermine]

Do you think it's worth it tho to add the skill to regenerate dagger golden tests in the dagger-chore?
If so I can just push that part on your PR

[still garnet]

sure!

[fair ermine]

@civic yacht hey 👋

I opened a PR to add some tests with a pinned released version: https://github.com/dagger/dagger/pull/11941
It's part of https://github.com/dagger/dagger/pull/11826 but I did it in another PR since that can be isolated 😄

[fair ermine]

@civic yacht Ready for reviiieeew: https://github.com/dagger/dagger/pull/11826 😄

[tidal spire]

currently investigating an issue i'm hitting locally where loading the python-sdk, release, and sdks toolchains in the inner dev engine in integration tests can hang forever. In case anyone else is also seeing it. Traced the issue down to python-sdk -> pytest toolchain -> wolfi -> git fetch --tags https://dagger.cloud/dagger/traces/d518477ff449674cb08dd26d19927fe9?span=f652f1e7611b02ab#565e2c0bd53d805f

[civic yacht]

git fetch is hanging for me rn too on an extremely diverged branch: https://dagger.cloud/dagger/traces/97f604d0de4d957b3565a1823d0df6df?listen=0bd79fa479c56807&listen=bae53dbbf9946f3b&listen=153da659d1695fd9&listen=8e6d2deb1d31898f&listen=d97d8c01dbd55de6&listen=194c6c054692e592&listen=7959794f25a6bf75&listen=aaa7e401eb125d2f&listen=0d48f32dfcebb5f5&listen=f99e13f40858f4e5&listen=3176a9e9027e406e&listen=2da64c0a314f98c2&listen=65cc7e4e8bb0d2fa&listen=2620723af693db13&listen=8018f79d41787aeb&listen=a826d334cb99ee2f&showHidden=a08b89586a98a637&showHidden=cb7616f9a67ed565#5f602c2792d13c61

[tidal spire]

I've been hanging on those toolchains all day. I just worked around it for now by removing the pytest dependency from toolchains/python-sdk-dev/dagger.json and its immediately happy. It looks like we're not actually using that dependency at the moment anyway

[spark cedar]

woop, hallo folks 🙏 just got an email from the person who opened this issue, asking me to take a look 🙂 https://github.com/dagger/dagger-for-github/issues/202
just mentioning it here, since it might not have been noticed ❤️

also hallo 👋 hope you're all doing good

oop, i pointed them towards discord and here they are: #1478344351724863499 message

[coral vector]

does anybody have a feel for how complex it would be to implement Service.withExec (runs a command in a Service container) and Service.exitCode (waits until the Service exits and returns its exit code)

[rocky plume]

service.wait

[rocky plume]

I could use a review on https://github.com/dagger/dagger/pull/11911

[leaden glade]

After so long, the PR to move the python analysis to an AST based one is ready for review: https://github.com/dagger/dagger/pull/11803
That's a big one, and most of it is AI generated. Not sure what's the best way to review it.
The good thing is all the tests are now passing, and the previous analysis that was loading the code is now removed.
This will (in follow up PRs) allow to use the moduleTypes SDK endpoint to register types.
This change is required for self calls in python, but it will also bring parity with Go and Typescript SDKs on this area.

[tidal spire]

Here's the fix for the github action. Looks like the bug was with multi line dagger shell blocks. Added a test to validate https://github.com/dagger/dagger-for-github/pull/203

[leaden glade]

I'm looking at the idea to use dagger generate instead of dagger develop while working on modules (target is workspace/modules v2).
One of the difficulties is about the question of using a module vs developing a module.
So let's say we are inside a workspace, containing a module that is defined locally. For instance one of our existing toolchains in dagger/dagger.
From anywhere, if dagger generate -l runs it shows the full list of generates. It's the case from the root, but also from a module path.
And that makes sense, we are in the workspace, meaning we want to use the modules referenced in the workspace.
But as the module source might also be inside the workspace, how to show the specific generate function replacing develop?
So maybe the thing is to use -m to specify the module.
Let's say I'm going to ./toolchains/python-sdk-dev:

• dagger generate -l will list all the generates from the workspace point of view
• dagger -m . generate -l will list the specific generates from this particular module + a generate function exposed by the module's SDK. Or maybe only this last one?
  Would it be something that makes sense? Or do we want to expose all of the SDK's generate functions at the workspace level?
  Just open thoughts, I'm not yet sure what would be the right UX, happy to get any ideas.

[leaden glade]

RFC dagger up
I created a doc to explain what could be dagger up: https://github.com/dagger/dagger/pull/11959
Any ideas, comments are welcome before I start the implementation

[leaden glade]

A simple one line change to allow dagger -m MODULE generate ( 🫣 )
https://github.com/dagger/dagger/pull/11960

[leaden glade]

More thoughts about dagger generate to codegen a module (instead of dagger develop)
The message right above allows to dagger generate -m MYMODULE.
But if the module is loaded as a toolchain / references in the workspace we can already dagger generate MYMODULE. The -m is still interesting as it allows to reference an external module by its git ref for instance.
But the thing here is this will list/run the generate functions defined by the module. Do we want the "code generate" to be at the same level? At the same phase? I'd say this should be maybe run before any other as this will be used to build the module.

I'll try to use an example to see how that would work. So let's say we are working on toolchains/python-sdk-dev. This module as currently a client-library generate function to re-generate the python lib, for instance if we add a new core type.
If I jump into the toolchains/python-sdk-dev directory and type dagger generate it will run all the generate functions from the workspace perspective.
If I dagger -m . generate it will only run the client-library generate function. Similar to running dagger generate python-sdk from the root of the workspace.
Where to put the generate function that will perform the equivalent of the develop?

Idea 1: stages. One stage can be executed only if the others have already been executed. By default everything is on stage 1. There's an implicit stage 0 that contains the generate functions exposed by the SDK. This stage is only executed if we explicitly specify the module. So dagger generate will not list it, dagger generate python-sdk will not run it, but dagger -m . generate -l will show it and dagger -m . generate will first run this stage 0 (develop) then stage 1 (the client-library)

Idea 2: when we explicitly specify the module, it will list and run only the generate functions from the SDK. dagger -m . generate -l will show the SDK generate (develop) and dagger -m . generate will only execute this function. The client-library is not visible.

I don't know if 1 is a good idea, but 2 feels weird. For instance if there's a new core type and we want to refresh it, it means dagger -m toolchains/python-sdk-dev generate && dagger generate python-sdk where a dagger -m toolchains/python-sdk-dev generate would be enough with 1.
But 2 has the advantage to be maybe more explicit. cd toolchains/python-sdk-dev; dagger -m . generate will do the equivalent of dagger develop

Ho yeah, and there's Idea 0: the sdk generate function is visible/executed at all time. It's automatically added from the SDK, by the engine. So dagger generate performs it in addition to the others.

All that text to say I don't know which solution to pick, and maybe there's some other interesting ones

[tepid nova]

@leaden glade you should start from the workspaces PR mentally

It deprecates -m for starters (or more precisely, narrows down its scope)

[tepid nova]

Spinning out the "dang module dev" skill from a bigger feature branch... https://github.com/dagger/dagger/pull/11967

Would love some 👀

[leaden glade]

git? dang? issue when using worktrees 🙏

[tepid nova]

@still garnet my prompt was basically "do everything vito says" 😛

automated compliance

@astral zealot next: "do everything vanta says"

[astral zealot]

I would love that

[tepid nova]

@civic yacht for the purposes of my little visualization experiment, I have a data modeling question.

Should I consider IDs scoped to a session? Or can they exist across sessions? I know it's been a murky issue. But for the purposes of my data model, I have to pick one lane or the other. wdyt?

Is it possible today in the engine, for different sessions to actually use the same object? Like connect to the same de-duped service, or get the same de-duped secret plaintext?

[civic yacht]

They exist across sessions. IDs that are isolated to a particular session (or client, etc.) are scoped to that session so they don't cross wires.

• On main that works by mixing the session ID/client ID/etc. into the digest of the ID.
• On the egraph PR that's slightly modified to work by considering session ID/clientID to be an "implicit input" that's automatically set. Creates same end effect but much more explicit and debuggable.

[tepid nova]

Nice, OK

@civic yacht speaking of egraph. Claude wants to interview you about Theseus to write the changelog post 🙂 I asked to keep it short.

1. What couldn't you build on top of BuildKit? Concrete examples of features or fixes you had to say "no" to.

2. A before/after cache scenario. One real example where BuildKit misses but e-graphs hit.

3. Why did removing BuildKit accidentally fix #8955? What was structurally wrong, and why is it now structurally impossible?

4. What's the first thing you'll build on the new engine that you couldn't before?

[civic yacht]

@Erik Sipsma speaking of egraph.

[tepid nova]

@civic yacht @still garnet telemetry question... I'm trying to build a model of clients from telemetry.

• Which spans come from which client
• Hierarchy of clients (which client is parent of which other parents)
• Which client is the root client of each session

It looks like that's not trivial to do, because the connect span is not actually the parent. So there is some heuristic involved to determine the hierarchy.

Am I missing an already existing, reliable method for modeling the hierarchy? Is the hierarchy I'm trying to model actually correct? ( Session 1:N Client 1:N Span)

<@&946480760016207902> I'm still suffering from "failed to call sdk moduleRuntime"

Repro:

dagger -d --progress=logs functions --mod github.com/dagger/dagger/toolchains/changelog

Is this a known / solved issue?

(sorry if I'm beating a dead horse)

[still garnet]

assuming you're on v0.20.1, that should start working once https://github.com/dagger/dagger/pull/11973 is merged

[stuck bloom]

In case you didn't know the changelog hasn't updated to 0.20.1 🙂

[tepid nova]

I'm working on it right now 🙂

(it's a manual editorial process)

Also adding more goodies 😛 You'll see

[tepid nova]

🙏 https://github.com/dagger/dagger/pull/11967

[tepid nova]

@still garnet just go-installed latest dang on my mac:

$ dang
zsh: killed     dang

[fallow lagoon]

The dagger/nix repo is broken with hash issues. After brown bag release ?

Does anyone know how to fix this ?

If I use dagger/nix/e11d4abe0d21295852d16ad4eef5f7c04d6cc8ca, I can still use Nix to setup dagger 0.20.0, but 0.20.1 is broken.

[wild zephyr]

👋 we had a blip in our last release where some checksums got outdated. We just fixed the binaries that we publish. Checking at nix now 🙏

[wild zephyr]

fixed

[rocky plume]

@civic yacht any thoughts on the cleanup on the llb2dagger work? I have a draft PR of it under https://github.com/dagger/dagger/pull/11970 that's passing.

[civic yacht]

I will take a look at the updates today, thanks! Honestly once we're confident in it we should just merge, no blockers other than feeling like it's maintainable and in full parity as far as we can tell based on the tests

[rocky plume]

I'm going to take a look at https://github.com/dagger/dagger/issues/11889, but if there's anything else you can think of, feel free to send it my way.

[leaden glade]

∅ fetch generator information 44.8s

What ∅ means? I sometimes see it, and I don't remember to have seen it until quite recently.
It looks it's all stuck, nothing happen, then it finishes and works.

[still garnet]

that means the span was canceled - or, the CLI saw the root span exit without that span finishing

[leaden glade]

that's weird, because at start is was like 4s when the ∅ started to be visible, nothing was happening, and then finished successfuly the call and updated the duration 🤔

[still garnet]

could be timing - CLI sees its local root span complete, assumes everything else was canceled, and then the engine telemetry arrives and cleans it up

[leaden glade]

Last week, I ran into a lot of issues running engine-dev playground on worktrees.
This https://github.com/dagger/dagger/pull/11990 looks like to fix it, at least to me. Happy to get some 👀 on it as it's very convenient to have worktrees, but I'm not an expert on the git side.

[tidal spire]

Not urgent, I'm looking for reviews on these

• https://github.com/dagger/dagger/pull/11956
• https://github.com/dagger/intro/pull/1
• https://github.com/dagger/checks/pull/3

[tepid nova]

@tidal spire for dagger/intro you should just push directly no?

[tidal spire]

it might have branch protection? otherwise I feel like I would have 😂 Also we talked about rewriting in dang which I can do on a new branch

[civic yacht]

@tidal spire I saw a bunch of new flakes on main, they are inconsistent but saw occurrences across two runs: https://dagger.cloud/dagger/checks/github.com/dagger/dagger@24b5f6f33e5a0570578603689c8c8a27ea424abf?check=test-split%3Atest-base&listen=11e754486c599e74&listen=e4e5604f6bdfe17a&listen=80ebbd5bf990a373&listen=b0b54b6deb0e99f7&listen=0c314be7e4d38c47&listen=eb89e4a2c42aee1e&listen=9cf161d42a6fb73f

• failed to get schema introspection json during module sdk codegen: failed to select introspection JSON file: session cache is closed

I suspect it's from the PR where you refactored the lazy schema load stuff. I funnily enough hit this same error except extremely consistently in my caching PR, IIRC the fix was to adjust ModDeps.Schema() to always attach the current cache for the caller, so just:

func (d *ModDeps) Schema(ctx context.Context) (*dagql.Server, error) {
    schema, err := d.lazilyLoadSchema(ctx)
    if err != nil {
        return nil, err
    }
    dagqlCache, err := d.root.Cache(ctx)
    if err != nil {
        return nil, fmt.Errorf("failed to get cache: %w", err)
    }
    return schema.WithCache(dagqlCache), nil
}

The problem is essentially that ModDeps itself can be used from cached objects from other sessions, in which case it's actually caching the cache... (this is on my list to cleanup in the new world)

[tidal spire]

@Kyle Penfound I saw a bunch of new

[leaden glade]

I added a few types on a branch, but in the logs I have a lot of Missing.

├╴✔ Workspace.moduleList: [WorkspaceModule!]! = xxh3:c825ccd9ffe4cc98 0.0s
│
├╴✔ Missing.name: String! = xxh3:0083078fdbd98d84 0.0s
│ ┃ typescript-sdk
│
├╴✔ Workspace.moduleList: [WorkspaceModule!]! = xxh3:c825ccd9ffe4cc98 0.0s
│
├╴✔ Missing.source: String! = xxh3:2a489f02ffed082b 0.0s
│ ┃ toolchains/typescript-sdk-dev
│
├╴✔ Workspace.moduleList: [WorkspaceModule!]! = xxh3:c825ccd9ffe4cc98 0.0s
│
╰╴✔ Missing.blueprint: Boolean! = xxh3:c033d2f04c6848e2 0.0s
  ┃ false

I guess something is missing (haha) but I don't understand what.
My type is installed in the schema:

dagql.Fields[*core.WorkspaceModule]{}.Install(srv)

Workspace.moduleList returns a [WorkspaceModule!]!
And the WorkspaceModule declares a Type function:

func (*WorkspaceModule) Type() *ast.Type {
    return &ast.Type{
        NamedType: "WorkspaceModule",
        NonNull:   true,
    }
}

Any idea? I guess that's something obvious but I can't find it for now

[civic yacht]

I think some of the missing stuff is due to https://github.com/sipsma/dagger/blob/5be6475b8d96fadf55e0aac53299680f52083162/core/telemetry.go#L334-L342, we skip some fields that are not really useful to users (at least in the past) but took up the vast vast majority of telemetry we were sending

I'm not sure if that explains all of it, but probably a lot of it at least

[leaden glade]

Hum, I'm not sure to understand exactly. The thing is I have a lof of them.
But what I did is to wrap everything inside one Tracer().Start() and even if they still exist, they are not visible by default, that is already better

[rocky plume]

I could use some feedback on a proposed API change to EnvFile.WithVariable related to the handling of quotes and variables. Currently the only work around is to escape quotes when calling it, e.g.

WithVariable("foo", `{\"hacky\": \"example\"}`)

I'm proposing that we drop this quote parsing, and only handle $ as a special case in the parser.

see https://github.com/dagger/dagger/issues/11889#issuecomment-4040766187 for details

[still garnet]

this happens when you select fields of objects returned in a list. dagql adds special intermediate IDs corresponding to each list index, but then those don't actually get sent over in the telemetry data, so the UI doesn't know what to display. longstanding TODO

related field: https://github.com/dagger/dagger/blob/f8c23f77772249bb5a480e7044b5345df89f5f62/dagql/call/callpbv1/call.proto#L47-L57

[civic yacht]

lol I actually had to fix that issue for the caching work, in that PR now each time you get an nth from a list it actually goes through GetOrInitCall. I forget if I gave it the telemetry callback or not, but that would be easy to tack on if not.

[leaden glade]

Based on the current workspace/modules v2/develop -> generate work and Dang, I wanted to see how far I can improve the Java SDK.
And what I can say is I'm really happy about the results

I'm using the exact same java module code (the default template) in the same context (cache prune, but go and dang sdk preloaded - to have both at the same level, and it will be the case once dang will be embedded)

Then I'm doing a dagger functions with both (and a dagger call but as the SDK doesn't use moduleTypes everything is already loaded).

Using the current version of the SDK, dagger functions takes 52s.

Using my new light SDK (the runtime is a small Dang file, 50 lines but it will grow a bit for sure) the exact same call to dagger functions takes 25s

🎉

Those numbers includes the time to fill the cache with maven dependencies. So if this cache is already full, we can imagine even better numbers!

In the other advantages, the module itself is fully buildable from the host machine. I can go in my module and just run mvn package or any other maven command and that will work out of the box, as any java project. (something that is not possible currently)

A bit part of that is due to the removal of dagger develop and to move the work on dagger generate, meaning all the necessary java files are generated on the host and used by the runtime.
And the runtime become really dumb. To have an idea, this is the moduleRuntime function of me light Java SDK:

  pub moduleRuntime(modSource: ModuleSource!, introspectionJSON: File): Container! {
    javaImage
      .withFile(
        "/opt/module/module.jar",
        mavenImage
          .withMountedCache(
            "/root/.m2",
            cacheVolume("sdk-java-light-maven-m2"),
            sharing: CacheSharingMode.LOCKED,
          )
          .withWorkdir("/src")
          .withDirectory(".", modSource.contextDirectory)
          .withWorkdir("/src/" + modSource.sourceSubpath)
          .withEnvVariable("_DAGGER_JAVA_SDK_MODULE_NAME", modSource.moduleName)
          .withExec(["mvn", "--pl", "src", "--also-make", "package", "-DskipTests", "--threads", "1C", "--no-transfer-progress"])
          .file("/src/modules/" + modSource.moduleName + "/src/target/"+modSource.moduleName+".jar"),
      )
      .withWorkdir("/opt/module")
      .withEntrypoint(["java", "-jar", "/opt/module/module.jar"])
  }

Dumb runtimes

[stuck bloom]

That's very impressive! I'm excited! I don't understand exactly where the speed gain is coming from, if my previous attempts were slow because my proxy is slow. Is it because the moduleRuntime is written in dang, there is no intermediate codegen stage? Previously, that intermediate codegen was also based on maven for the Java SDK. So in basic terms, this is cutting down on the number of maven commands?

[leaden glade]

One of the main thing is it removes all codegen at runtime. The runtime only builds. And the codegen/generate is all done on the dev machine, generating all the needed files, including types. That's why it's quicker and allows things like the ability to build locally or run any maven command.
The part that doesn't change still is to pull the dependencies. But I think this part can also be improved. Slow jar pulls will still be slow, we can't do a lot about that, but less jars, better cache, less runtime work will improve.

[stuck bloom]

Does that mean, there's no need for custom settings.xml pulled into the engine? Since that's available on the local machine?

[leaden glade]

No. At least not right now. That could be something if all the dependencies (or the final jar result) are part of the module. And added to git. And I'm not sure that's what we want.
The build is still part of the dagger call, so the dependency pulling.
All that is very experimental for now, not part of a branch before we have the new modules v2 I think, but something that will follow. And once that's done, it might be easier to understand the bottlenecks with Java, see where we can improve, better cache, etc.

[leaden glade]

I could use a review on this PR https://github.com/dagger/dagger/pull/11990. This really solves my worktree issues, but I'd like to double-check with someone else (and merge it if that's good, currently I'm duplicating this piece of code everywhere without committing it)

[tepid nova]

Translation: "F your dagger check, it's too slow. Let me run go tests directly"

[leaden glade]

Workspace, generate, local dependencies and espace root

[still garnet]

@obsidian rover I now firsthand understand the value of your Changeset size limit streaming fix 😂 - did that make it into a PR?

[obsidian rover]

Ahahah, let me ship it (was focused on the website + support). The big hack (courtesy of Tibor) is that i upgrade the MaxFileContentsSize to 512

[tepid nova]

@civic yacht @rocky plume I have a question on engine state, on behalf of me, @charred lotus @obsidian rover @wild zephyr @astral zealot :). 🧵

[obsidian rover]

https://github.com/dagger/dagger/pull/12002 (waiting for CI to be green ahah) [ok, a few cosmetics things / go linter + regenerate, but shall be ok-ish for a first review]

---

Have to go rn, but will fix it a bit later

[tepid nova]

@still garnet re our discussion on dagger in agent devloop...

[still garnet]

what did it see that it considered slow/silent? curious to see a session log if your client supports it

[tepid nova]

I've never tried sharing session log with codex... In the meantime here's a copy-paste of a good chunk of the session leading up to that message: https://gist.github.com/shykes/90872a8021c8214bc640e1ae56dce079

[still garnet]

huh, hard to judge what it saw really since all the tool output logs are truncated, but it looks like the problem might have been with dagger generate. i wonder if --progress=dots would give it a better idea of things going on? maybe --progress=logs is TOO quiet? i don't think there's a ton of logging in our generators, things just run and spit out files.

also, we should bump dang again - I pushed a change for the Zig fetching so it tries all their mirrors, and it's MUCH faster. plus a fix for xs.each { x => self.foo += x }.
but maybe i should finish all these PRs and just pull it into the engine instead

[still garnet]

PSA for Dang users (dangists? dangers?): github.com/vito/dang is now decoupled from dagger.io/dagger, meaning now it just shells out to dagger, instead of being hardcoded to a version.
Now you can map Dagger to a dev engine, to try out new APIs:

dagger llm-workspace*​​ ≡
❯ cat ../dang/dang.toml
[imports.Dagger]
dagger = true
service = ["dagger-dev", "session"]

(requires LSP restart atm)

[tepid nova]

dangistas? 🙂

[obsidian rover]

Hi 👋 if I remember correctly, we can't, in the same release: 1) add an API and 2) use it directly in the CI ? Will this change with the ongoing work around having the generated files locally and how the sdks will actually respect that ? cc @charred lotus

[leaden glade]

I'm not sure the work on generated files will have an impact here. It's often more a question of "bootstrap", when you need an engine/client to build an engine/client and so if you can't do it with a released engine it might be difficult. But I think it depends on the kind of API and how you want to use it.

[obsidian rover]

Hey, if anyone has some spare cycles, i'd love another pair of 👀 / review on https://github.com/dagger/dagger/pull/11999. This fixes a regression encountered by some users 🙏. I approved it, but it's mostly my own changes 🤣

[civic yacht]

@still garnet questions about llm/mcp stuff in core/schema/ since idk who else to ask 🧵

[still garnet]

Tuist PR is ready for review - all green: https://github.com/dagger/dagger/pull/11952

[obsidian rover]

Taking a look 😍

[tidal spire]

My checks are failing with module not found: github.com/dagger/dagger@fc82c54c56f69e9a57b9aeb1c1bf9239d0ed8131. Oddly enough, that commit doesn't appear on main, but was created by @tepid nova 12 minutes ago 🤔 https://github.com/dagger/dagger/commit/fc82c54c56f69e9a57b9aeb1c1bf9239d0ed8131

https://dagger.cloud/dagger/traces/6b808086705db959423cd52ddf7c50d9?span=d8c7506fd5f53e4c

[coral vector]

do you ever wonder if you're like top 10 in the world on TUI knowledge

[still garnet]

lol can't say i have, it's a fun little world to mess with

[tepid nova]

Mmmm I did not push any commits today

[tidal spire]

If I'm reading it correctly it looks like a merge of the workspace branch into main. So probably for a checks run of the workspace branch somehow contaminating another branch's run?

[still garnet]

just fyi - pushed a follow-up cleanup, unifying on lipgloss v2 (had a mix of v1 and v2 before). going green now

[tepid nova]

We regularly merge workspace into main. Most recently yesterday afternoon. Maybe this morning someone (not me) pushed to workspace, causing a check on that branch?

[tidal spire]

Could be, I only tagged you because the commit is from your user. But my main concern is how that became an error on an unrelated branch

[still garnet]

i did some push -fs to the workspace-plumbing branch but nothing merge related

and i'd expect my name/github user on those

[tidal spire]

yeah 15da31a is you

[still garnet]

oh, unless github's auto-merge-commit-calculation thing does it on behalf of the PR author

[tidal spire]

that's what I'm assuming

fc82c54 is like 50 minutes after 15da31a though so I guess it depends when you actually pushed

[tepid nova]

Just making sure you pulled my latest commits from last night before force-pushing? I pushed a superficial cleanup last night (renaming variables to avoid confusing the AI)

[still garnet]

eek, that might be gone, sorry - I checked the PR in the github UI but it might have been stale
unless it's this? https://github.com/dagger/dagger/pull/11995/changes/9eceb3a2e159c1ca1c5c5d03409cbfeb3cbf5d3c

[tepid nova]

yes that's the one 🙂

@still garnet 👆 AI didn't fully implement the entrypoint spec in workspace - left some dirty stopgaps - and I missed it. Then when cherry-picking into workspace-plumbing, another AI made it worse by layering even more stopgaps.

Smartest morons in the universe

[tidal spire]

Just merged GCP secret manager support https://github.com/dagger/dagger/pull/10510

[still garnet]

cross check pollution?

[still garnet]

merged tuist 🎉 - let me know if you find any jankiness!

[still garnet]

✘ Alpine.container: Container! 1.3s ERROR
! failed to create package container: failed to get packages: solving "clang" constraint: resolving "clang-22-22.1.1-r1.apk" deps:
  solving "llvm-22=22.1.1-r1" constraint:   llvm-22-22.1.1-r0.apk disqualified because "22.1.1-r0" does not satisfy "llvm-22=22.1.1-r1"

wolfi dependency hell again?

[civic yacht]

changing to clang~21 might avoid for now. tested via:

• fails: dagger call -m ./modules/wolfi container --packages clang terminal
• works: dagger call -m ./modules/wolfi container --packages clang~21 terminal

[stuck bloom]

hey @still garnet . I'm noticing the new TUI being very stuttery (flashy) when I try to move around while a build is running. It gets worse as the verbosity level goes up. It's hard to read. I'm on ghostty (m4 mac) fwiw.

[stuck bloom]

This went in with 0.20.3 @tepid nova - https://dagger.io/changelog/#vault-oidc-authentication. Not showing up under 0.20.3

This too I believe - https://github.com/dagger/dagger/pull/11981

[tepid nova]

Thanks @stuck bloom , fixed!

[obsidian rover]

Are you guys able to do dagger generate on .20.3 when having an ssh socket around ? It seems to be broken for me, and I don't know (yet) if it's linked to my setup or not 👀

The git fetch --tags seems suspicious and might need auth ? cc @wild zephyr

✘ rust-sdk:apiclient 29.0s ⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⣧⣿⡄ ERROR
┇ go-sdk:generate › .generate › .service(name: "sdk") › .version › 
✘ withExec git fetch --tags 1.1s ERROR
Host key verification failed.                                                                                        
fatal: Could not read from remote repository.                                                                        
                                                                                                                     
Please make sure you have the correct access rights                                                                  
and the repository exists.                                                                                           
! exit code: 128

[storm wind]

Are you guys able to do `dagger

[coral vector]

FYI @tepid nova--"Design doc (Part 2: Workspace API)" in the changelog 404s for me (and presumably other non-maintainers?)

[tepid nova]

thanks, just noticed that too ... will fix the link

[tepid nova]

@still garnet you mentioned a branch with a batch of LLM improvements? Is there a PR I can link to for the changelog?

[still garnet]

Quick and dirty tests idea:

1. Add dagger.io/test.name attribute, containing TestFoo/bar/baz
2. Add dagger test <check-name> [--fail-fast] [subject-names...]
   • What if we could automate --fail-fast? Once we see a failed test span, we just kill the whole function?
3. When you run dagger check or dagger test, the CLI reports failed tests and how to re-run them (human + LLM discoverability)
   (typed after standup and forgot to hit send)

[still garnet]

it's honestly way too messy at the moment, but https://github.com/dagger/dagger/pull/11994

[tepid nova]

Fixed. Thanks https://dagger.io/changelog/#modules-v2

cc @civic yacht the saga continues 😛

[civic yacht]

Lol yes… I also recently discovered the magic word “tripwire” as in “when you encounter an unexpected design decision during implementation, that’s a tripwire for stopping and escalating for discussion, not improvising”

That helps a lot too, a lot of it’s dumbest code happens in those moments. And now it’s always talking about how it hit a tripwire in the middle if the hard cutover

[tepid nova]

hard tripover

[wild zephyr]

fixing this now

[wild zephyr]

https://github.com/dagger/dagger/pull/12047 🙏

[leaden glade]

It looks like TestSecretProvider/TestAWS starts to fail (at least on my PR)
We are using localstack/localstack:latest
And I just discover the localstack/localstack repository just got archived yesterday 😲

Starting LocalStack for AWS now requires an auth token. Please explore our pricing options, which include a free plan for non-commercial use that supports equivalent functionality to the previous community image.
https://news.ycombinator.com/item?id=47493657
Tests that are failing are using this version of localstack, built today:

LocalStack version: 2026.3.1.dev7
LocalStack build date: 2026-03-24
LocalStack build git hash: c1984ba40

And this is the error:

Localstack returning with exit code 55. Reason: 
===============================================
The license activation failed for the following reason:

No credentials were found in the environment. Please make sure to either set the LOCALSTACK_AUTH_TOKEN variable to a valid auth token. If you are using the CLI, you can also run `localstack auth set-token`.

LocalStack requires an account to run.

==> Have an account? Learn how to set LOCALSTACK_AUTH_TOKEN: https://app.localstack.cloud/settings/auth-tokens
==> Need an account? Get started: https://www.localstack.cloud/pricing
==> Want more time? Snooze until April 6, 2026 by setting LOCALSTACK_ACKNOWLEDGE_ACCOUNT_REQUIREMENT=1

I looked at some other tests that are passing, and they are using a previous version:

LocalStack version: 4.14.1.dev75
LocalStack build date: 2026-03-18
LocalStack build git hash: 24107750d

Using the localstack/localstack:community-archive@sha256:6b6172cfceb04b4fbc35097a55f717c365a35fafa572be49f7341771cf9023ed image is fixing it. This is the exact same version.
I'll open a PR for that, at least for now so we don't have anymore the CI red on this.

ecshafer

Local Stack Archived their GitHub repo and requires an account to run

Hacker News · Mar 23, 2026, 6:57 PM

[leaden glade]

Here is the PR: https://github.com/dagger/dagger/pull/12048

[stuck bloom]

We are starting to move away from localstack too (licensing + bloated). There are some drop-in alternatives you may want to try. This is one - https://github.com/sivchari/kumo.

[leaden glade]

I'll have a look. I've also seen https://github.com/hectorvent/floci and https://github.com/robotocore/robotocore
Especially we are using it only in single test regarding AWS secrets so we don't need a lot

[stuck bloom]

Exactly! I'd go with the smallest and fastest. The localstack image itself is multi GB.

[leaden glade]

I quickly tried with hectorvent/floci and that boots so fast! really impressive

[leaden glade]

I updated the PR to replace localstack by floci, that just works, and quicker (no more sleep 10...) https://github.com/dagger/dagger/pull/12048

And dagger up PR is also ready to review https://github.com/dagger/dagger/pull/11959

[tepid nova]

<@&946480760016207902> I think I found a bug in the Go client library... When I call dagger.Connect from a custom tool, with an explicit engine host, then wrap that tool in dagger run... The outer session injected by dagger run clobbers the explicit engine host argument. I have to explicitly unset the DAGGER_SESSION_PORT env var to unclobber.

wdyt - bug?

[tepid nova]

Thanks for moving fast!

[fair ermine]

Is it with the client generator or the published dagger.io/dagger pkg?

[leaden glade]

And I think that's a good move over all, in the way we are not using that much localstack. It's not like if we wanted to run dynamodbs, lambdas or other components, just to test a single use case. So a lighter solution seems better.

[wild zephyr]

I'd think it's a bug. If the dagger.Connect is specifically asking for a runner host, seems like that should be honored

[tepid nova]

FYI, someone complained in a github issue that we have too many github issues 😄 But they're right.

I'm going to bite the bullet, and be more aggressive about converting issues to discussions. We have quite a backlog of outdated issues...

I propose that we follow Mitchell's lead in the ghostty repo: issues are only for authoritative work items that we (maintainers) have decided need to be done. It is not for users filing bug reports, feature requests, design debates, etc. All of that should go to discussions

[tepid nova]

Fun fact: you can't convert an issue to a discussion with the API or CLI -> web UI only.

So I'm setting up a playwright + claude workflow to automate it . I'm not clicking through 300 github tabs 🙂

[rocky plume]

yaks, yaks everywhere: https://github.com/dagger/dagger/pull/12128

[tepid nova]

Down from 800+...

[tepid nova]

I moved the notification spam to #github-private-feed , at least for now

(90% of the triage is converting to discussions in different categories)

[tepid nova]

The flipside is that we have a very full and very active Help discussion category, which we should monitor... https://github.com/dagger/dagger/discussions/categories/help

[tepid nova]

Need a quick ✅ on https://github.com/dagger/dagger/pull/12831

[tepid nova]

@civic yacht (not urgent) can one dagger client session span multiple TCP (or unix) connections? Or is it always a strict 1-1 mapping?

[civic yacht]

Yeah we use http2 so it'll create multiple connections IIRC

[tepid nova]

ok I always thought http2 was all about multiplexing N streams in a single connection

[civic yacht]

Yeah I forget the details but I think the go implementation uses multiple connections. This is very much not information fresh in my brain's cache so take with salt, but I remember wondering this same thing and I think I found it uses multiple connections still.

[tepid nova]

that's what codex is telling me I was skeptical but I guess it was right

[tepid nova]

<@&946480760016207902> question on issue rules.

I'm doing a pass on issues that were converted to discussion when perhaps they shouldn't have. The rule is issues should be actionable. For bugs, that means they should be acknowledged and reproducible.

• In some cases it's clear cut -> successful repro. promote to issue (or in the future: keep it an issue)

• But in other cases it's a gray area. For example: https://github.com/dagger/dagger/discussions/12816 . It seems possible to repro, but it's difficult to repro (race condition) and I personally have not been able to. In that case, should we create an issue right away, or should we keep it as a discussion in Help (basically a support thread) until we have successful repro, at which point it becomes actionable and can be escalated to an issue -> ie. a task for a maintainer to complete? wdyt?

This edge case will help us define the cutoff line clearly. It's not about how "important" or "real" it is. It's more about how we organize our work IMO. Support vs. dev.

[civic yacht]

Sorry I should have closed the original issue, it was fixed in https://github.com/dagger/dagger/pull/12000

[tepid nova]

Ha ha LOL 🙂 Thanks

Still interested in the hypothetical - at the moment when it was open, what flow should we have followed?

@civic yacht does it feel like you had to do the work in 2 distinct phases? First the support work to get a reliable repro, then the dev work to fix it?

[civic yacht]

Something else entirely. I actually didn't get a repro at all but I could see in traces provided by the user it happening with clear enough evidence to work on a fix

But yeah I suppose there was a phase of investigation and then a phase of deciding it was a bug and fixing it

[tepid nova]

In this particular case, was the original issue (filed by a user) a perfect match for that PR? Or would it have been cleaner to have an issue for eg. the root cause, and it incidentally closes that user-filed issue?

[civic yacht]

Grey area, but I'd be biased towards keeping it simple and just saying the issue they filed was good enough. They had very clear evidence of the problem and the PR fixed it exactly

If it was a more complicated siutation that required long debugging, finding a consistent repro, etc. then the answer might change. But that one was simple and striaghtforward enough

[tepid nova]

OK! Thanks

[tepid nova]

Workspace & artifacts 🧵

[tepid nova]

Do we consider this to be an actual bug? https://github.com/dagger/dagger/issues/9427

[tawny iris]

What about "Feature Requests"?

I made an issue for each feature the PHP SDK was missing (relative to the official SDKs), that I was aware of.

• They were not bugs
• They were actionable.

If we're trying to keep the Issues section "clean" then perhaps they should be kept as "Discussions" until certain criteria are met such as:

• A minimal repro of what the feature should achieve
• A consensus from 1+ Maintainers that this feature should be implemented
• If not directly actionable, the discussion should be used as a basis for breaking out small directly actionable issues.

PS: I don't have a personal preference on the matter. I think most repos just use Issues as a catch-all with Discussions being woefully underutilised. So I'm not actually sure what good practice should look like in this regard.

[tepid nova]

Yes agreed!

Basically I like this: https://x.com/mitchellh/status/1852039089547309552

Mitchell Hashimoto (@mitchellh)

I'm doing something new (for me) with Ghostty: the issue tracker is only for actionable tasks. Features in the issue tracker are accepted and well-scoped. Bugs are reproducible. PRs must have an associated issue (no drive-by PRs). The issue tracker is not used for discussion.

X · Oct 31, 2024, 5:25 PM

[wild zephyr]

I'm liking the idea of brining the --fail-fast feature to cloud and to make it configurable for checks. I'm thinking of adding a new section under the check settings modal for specific checks. Maybe also add something else in the org settings to make it org wide? cc @tepid nova @tidal spire

[tepid nova]

I think we should configure it in the workspace config file no?

(same for secrets references btw)

[wild zephyr]

yes, sounds good

[wild zephyr]

agree, once we flesh out the "environments" concept in workspaces?

[tepid nova]

Coming soon 🙂

Should we switch fail-fast to be the default maybe?

[wild zephyr]

I think it's a case by case decision. It's a trade-off between overspend vs understanding everything that failed so you can maybe run the failing checks locally before pushing the fix 🤔

for dagger/dagger I personally like seeing all the specific tests that fail within a single check. On top of that, usually fail-fast is not the default in most testing frameworks

[tepid nova]

It just occured to me that bootstrap-ci should perhaps run against a known stable commit, rather than its own commit... The fact that we're not, is probably the reason we get weird edge cases when we make changes that affect our own CI environment

@still garnet I'm starting think "agent-to-agent" might be a viable interface for coding agents to interact with dagger

with workspace UX & API, all the new high-level concepts... We could use our own LLM APIs, combined with a cheap and fast LLM (would need an API for specifically requesting one of those) - to expose our first LLM-powered core API. Workspace.prompt or something like that. Then let the external agent prompt "I just developed a new go test TestFoo, run only this one please. If it fails, give me the logs and nothing else. Also tell me how long it took" or whatever

I would definitely use that myself... But so would smarter coding agents IMO

The key is that the "clerk agent" doesn't need to be super smart or expensive. It can use the kind of inference that is so cheap, it's basically as plentiful as regular compute (and therefore we can bake it into the core)

dagger prompt 'run all linters or formatting related checks'

The key IMO is to narrow down the scope. It's not a custom coding agent. It's half-way between a dumb deterministic function and a full-blown agent

[tepid nova]

Bug remediation update

• 15 issues re-opened
• 3 closed as already fixed
• 12 new PRs ready for review 🙏
• 2 existing PRs that also need review 🙏

10 new PRs 🙏

• https://github.com/dagger/dagger/pull/12846 — WithExec Expand not applied to RedirectStdout
• https://github.com/dagger/dagger/pull/12859 — Helm chart probe settings unmarshal error
• https://github.com/dagger/dagger/pull/12863 — golangci-lint lint config prep (unblocks #11997)
• https://github.com/dagger/dagger/pull/12857 — Module description empty in dagger shell
• https://github.com/dagger/dagger/pull/12854 — Python .env defaults not overridden
• https://github.com/dagger/dagger/pull/12835 — Log output breaks console
• https://github.com/dagger/dagger/pull/12837 — Python log_output type mismatch
• https://github.com/dagger/dagger/pull/12842 — Unclear shell errors for unsupported types
• https://github.com/dagger/dagger/pull/12849 — Crash on stale SSH_AUTH_SOCK
• https://github.com/dagger/dagger/pull/12851 — CA certs broken via K8s Secret symlinks

2 existing PRs 🙏

• https://github.com/dagger/dagger/pull/12014 — JSON value strips quotes in envfile
• https://github.com/dagger/dagger/pull/11906 — Toolchain customization wrong casing

[rocky plume]

✘ http(url: "https://ziglang.org/builds/zig-x86_64-linux-0.16.0-dev.2670+56253d9e3.tar.xz", refID: "s2wl7pv6lwmqnb9153ok3l18q"): File! 0.5s ERROR
! invalid response status 404 Not Found

something just changed on their mirror 😢

[still garnet]

I pushed a fix to dagger/pytest - just need to bump it on main now. I can put up a PR

[still garnet]

https://github.com/dagger/dagger/pull/12884

[fresh harbor]

Hi, the elixir-sdk toolchain doesn't run in the CI?

[tepid nova]

Proposal for Collections (plumbing for artifacts support) https://github.com/dagger/dagger/discussions/12885

[leaden glade]

Yes, no check was configured on the toolchain.
I added them here: https://github.com/dagger/dagger/pull/12886
But, at least on my machine when I tried, the sdkTest check didn't passed. But feel free to push to this PR if you know how to fix it.
Otherwise we can also remove this check first and push the others, then open a follow up just for this check.

[fresh harbor]

Thanks. Looks like an error occurred during connecting to the engine.

 Error: start engine: driver for scheme "image" was not available

I'm not sure what it is.

[tepid nova]

That usually happens when dagger CLI tries to auto-provision the engine from the engine image (image://registry.dagger.io:...) but there is no available container runtime to run the image.

In the context of tests, my guess would be that dagger is being executed inside a container by a dagger pipeline (dagger in dagger) and the Container.withExec(experimentalPriviliegedNesting: true) argument is not set

[fresh harbor]

Thanks @tepid nova @leaden glade looks like it work!

Open a PR based on @leaden glade branch https://github.com/dagger/dagger/pull/12887.

[leaden glade]

👍
looks good, I closed the other one. CI failures seems unrelated but replaying them just to have it green and then it will be good to merge

[leaden glade]

Some Toolchain related tests are impacted by this issue (because they are using modules impacted by the Dang/Zig problem)
I've fix it here: https://github.com/dagger/dagger/pull/12888

[leaden glade]

I have a really weird issue I'm not sure to understand. If anyone has some clues.
Context: I'm working on a git worktree, located at /home/yves/dev/src/github.com/dagger/dagger-worktrees/dagger-up. I also have an other worktree on which I'm working on a different pane at /home/yves/dev/src/github.com/dagger/dagger-worktrees/develop-generate.
In the context of dagger-up I'm running dagger generate and I have this issue:

✘ typescript-sdk:client-library 2.2s ⣿⣿⣿⣿⣿⣿⣷⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⡄⡀ ERROR
┇ python-sdk:client-library › DaggerDev.pythonSdk(
  ┆ workspace: context /home/yves/dev/src/github.com/dagger/dagger-worktrees/dagger-up (
  ┆ ┆ exclude: ["*", "!sdk/python/*.toml", "!sdk/python/*.lock", "!sdk/python/*/*.toml", "!sdk/python/*/*.lock", "!sdk/python/.python-version", "!sdk/python/dev/src/**/*.py", "!sdk/python/docs/**/*.py", "!sdk/python/docs/**/*.rst", "!sdk/python/runtime/images", "!sdk/python/src/**/*.py", "!sdk/python/src/**/py.typed", "!sdk/python/tests/**/*.py", "!sdk/python/codegen/**/*.py", "!sdk/python/README.md", "!sdk/python/LICENSE"]
  ┆ )
  ) › .withServiceBinding(
  ┆ alias: "dagger-engine"
  ┆ service: no(digest: "xxh3:2d6060c5d283feaf")
  ) › Container.withFile(
  ┆ path: "/opt/cni/bin/dnsname"
  ┆ source: no(digest: "xxh3:97cb27c188b2d1d6")
  ) › Directory.withDirectory(
  ┆ path: "./bin/"
  ┆ source: no(digest: "xxh3:3156c0685a97264e")
  ) › Container.withMountedDirectory(
  ┆ path: ""
  ┆ source: context /home/yves/dev/src/github.com/dagger/dagger-worktrees/develop-generate (
  ┆ ┆ exclude: ["*", "!.git", "!**/go.*", "!version", "!core", "!engine", "!util", "!network", "!dagql", "!analytics", "!auth", "!cmd", "!internal", "!sdk", "sdk/**/examples", "!cmd", "!modules/wolfi", "!modules/alpine"]
  ┆ )
  ) ›
✘ context /home/yves/dev/src/github.com/dagger/dagger-worktrees/develop-generate (
  ┆ exclude: ["*", "!.git", "!**/go.*", "!version", "!core", "!engine", "!util", "!network", "!dagql", "!analytics", "!auth", "!cmd", "!internal", "!sdk", "sdk/**/examples", "!cmd", "!modules/wolfi", "!modules/alpine"]
  ) 0.0s ERROR
! module not found: /home/yves/dev/src/github.com/dagger/dagger-worktrees/develop-generate

The workspace seems ok (workspace: context /home/yves/dev/src/github.com/dagger/dagger-worktrees/dagger-up) but then it tries to mount a directory from somewhere else (source: context /home/yves/dev/src/github.com/dagger/dagger-worktrees/develop-generate)
And of course it doesn't find it, as it's outside of the workspace.

[leaden glade]

I have a really weird issue I'm not sure

[still garnet]

here's the bump to jest to fix the broken dependency, too: https://github.com/dagger/jest/pull/15

[leaden glade]

To share some progress on replacing dagger develop with dagger generate (and the ability to safely commit all the generated files)
After 2 different attempts based on the workspace branch, I finally have something working based on main. That works nicely, keep compatibility with dagger develop for a smooth experience. I still had to backport the workspace.path feature, but that's fine.
Except in one case: when a module already has toolchains with generate functions, or if the module declares generate functions, the dagger generate tries to run all generate functions. That's by design. But it broke as the "develop" generate function must run first.
And of course... this is already solved in the workspace branch 😅 as this is exactly the goal of workspaces to clearly separate between the development of a module and the usage of a module.
All that to say I'm looking into two options:

1. open a PR with a smaller scoped branch that allows dagger generate to replace dagger develop but opt-in exclusively, without all the runtime optimisations (no codegen at runtime) and that only works on modules without already generate functions.
2. rework that stuff to be on top of workspace again, or maybe on top of workspace-plumbing? If there's enough in this PR that could be better so the gap is smaller and it allows to move step by step.
   I was thinking about 1., but if I can make 2. I think that will be best. I'll see.
   (no specific question, just a 🧠 dump)

[fresh harbor]

I submitted a another fix to the elixir-sdk-dev module to make the dagger generate works https://github.com/dagger/dagger/pull/12890.

I also found an issue with the generate UI that if the changes is huge, the confirm buttons will be disappear from the screen and cannot scroll to the buttons but it can still choose and enter.

[still garnet]

@tidal spire when you have a sec, this needs a bump to kpenfound/dag/node Dang => 352d057a4d4c4d77612104f842671b512495ef30

@tidal spire actually better yet, looks like you could bump it to the native dang sdk since there's nothing fancy like checks or anything

[tidal spire]

on it

how do I use native dang? just "sdk": "dang"?

[still garnet]

yup

related - could use a ✅ on https://github.com/dagger/dagger/pull/12830 to bring the native sdk up to snuff, @civic yacht there was just one engine related concern which I commented on

[tidal spire]

pushed 9e6dad0e1e3a30afa82dd7e5663b5aa421cf25e7

We should also move kpenfound/dag/node into dagger/node since we have at least 3 toolchains depending on it. I only initially added it to my own monorepo to get it out faster

@tepid nova @wild zephyr can I get a dagger/node repo and I'll move kpenfound/dag/node there and update the modules that depend on it

[wild zephyr]

sure, 1 sec

wrapping up a call

done

[tepid nova]

@leaden glade why do you need generate to *replace* developat all? Can you makedevelop` still work the same, but just not necessary, and we can just stop using it, and quietly deprecate it later?

[rocky plume]

ugh, I'm running into a sometimes deadlock on the llb2dagger PR when running go test -count=1 -v -run 'TestDockerfile/Test[CU]' ./core/integration/. it seems to occur most often when TestCrossSessionDockerbuildSockets is one of the tests that's being run, but still not sure.

[civic yacht]

When it happens you can send SIGQUIT to the engine to get a goroutine dump (gotta docker logs -f ... in a different terminal so it's all captured). But TBH my suggestion would be to just ignore it for now and get everything else mergeable.

Reason being that llb2dagger is still based on main and thus the buildkit solver and thus it's full of traps that lead to deadlock. That's literally why my PR became "the rest of theseus", I tried a smaller bite but I just hit deadlocks in the buildkit solver 😄

Once you think llb2dagger is ready other than the deadlock I'll pull it into https://github.com/dagger/dagger/pull/11856. Everything has changed so much there that I highly doubt whatever is causing the deadlock is even possible anymore.

Many/most of the problems we're having on main are completely obliterated even as a conceptual possibility, besides just being "fixed".

[tepid nova]

@civic yacht I'm working on your review for lockfile.

Quick question, you mentioned making lockfile operations part of the public API, so that they're cached. Does that mean the engine would make a nested dagql query each time it needs to lookup an entry? You're not worried about the overhead?

[civic yacht]

The overhead of a nested dagql query is close to nothing, especially compared to the overhead of initiating a grpc client back to the client, asking for the file and loading it back

[tepid nova]

OK, are you open to solving the issue without exposing it over dagql though? The way I see it, it's still fundamentally a sync problem, it's just that it should "locksync" instead of "filesync". So it's a granularity and buffering problem, that I need to solve regardless - but bolting a dagql interface on top and rerouting everything through it seems like extra work. No?

(mapping the read path to dagql is easy enough, it's more the write path that I'm worried about)

[civic yacht]

The alternative is re-inventing dagql caching, which is gonna be way more work. Using the caching system will get you a lot (per-client caching, debug telemetry if desired, etc.). I don't think it's very much extra work at all. It's a little bit of scaffolding but that's what LLMs are for. They love a good re-usable "on rails" pattern to follow.

[tepid nova]

does that apply to writes too though?

[civic yacht]

We already have Directory.export, so I don't see what the blocker would be

I didn't look at the export part of the implementation yet, is it doing a write after each operation? Or just when the session ends?

[tepid nova]

My issue is that I have N clients concurrently reading and writing to the same lockfile (modules inherit their caller's workspace & lockfile), and doing so continuously during the session

[tepid nova]

At the moment it's doing it after each operation (if running in a lock mode that enables writes) but I guess buffering is an option if it makes everything easier.

It kind of feels like trying to fit filesync in dagql though

I could just be completely misunderstanding the real shape of the problem

Will do more homework and report back

[civic yacht]

My issue is that I have N clients

[leaden glade]

sorry if I wasn't clear, but yes what I'm doing now is to have the possibility to use dagger generate to generate the files and just deprecate at some point dagger develop. I'm not changing develop to use generate, still working (almost) the same way. Almost because I added at some point a flag in dagger.json when running a dagger develop as a marker of legacy, so I can use runtime optimisation if the flag is not there: if you're using generate, runtime will not try to codegen anymore. But this optimisation doesn't work with develop and the non committed files.

[coral vector]

question--don't have to reveal this if you don't want to--does Dagger cloud use kubernetes-hosted engines?

[tepid nova]

Nope. No Kubernetes anywhere

[coral vector]

cool. are you running the engine as a container still, or straight on the instances?

[tepid nova]

Still as a container, on ephemeral VMs. Custom scheduler for dispatching jobs and bringing machines up/down

[coral vector]

is the only thing that you need to do per-customer is keep their /var/lib/daggers separate and re-use them between their jobs?

(not trying to trivialize the difficulty of that task 😆)

[tepid nova]

If only it were that simple 😅 The issue with that, is that it persists and syncs data at a very coarse level (all the cache for all the runs of all your pipelines), so its a LOT of data to move, it changes a LOT, and a LOT of it is wasted in the end because of cache thrash

So we're working on making everything much more granular: scheduling, cache reuse, and machine allocation

[coral vector]

so the cache wouldn't be per-customer, but per-function?

[tepid nova]

We can do this way better than anyone else, since we actually control the software execution layer, and the infra knows what it's running. IE. we can take the telemetry stream, and plug it back into the infra to fine-tune it for each customer and workload

[tepid nova]

It's not just about the cache, but also the compute. Which function to run on which machine, with which cache. Even the decision to scale-out a function or not - will it actually be faster to run it somewhere? And if so, where?

The answer changes depending on what other cache is warm; what CPU, memory and IO is immediately available; how memory-intensive vs cpu-intensive that particular function is; etc

Meanwhile we're pushing the APIs to make it possible to break down the granularity even further. For example Collections so that you can model each individual test in your test suite as its own dagger object & check function. Now all of a sudden we can start scheduling your test suite at the test granularity (still using live telemetry to fine-tune)

[coral vector]

cooooool, so you're monitoring the resource usage of certain functions from the customer and using that info to decide what compute to schedule it on?

[tepid nova]

But also with context like "which tests have dependencies in common? And how long does it take to build those dependencies? Is it better to run those dependencies once and share, or each test runs its own?"

[tepid nova]

Yes. Semi-manually for now, but will be AI-assisted soon 🙂

The more you run on our infra, the smarter the fine-tuning - the faster and cheaper your runs

[coral vector]

that's a good use-case

[tepid nova]

To be clear we're in the very very early days of this. The current implementation is not as sophisticated as it sounds. BUT, we are rolling out v2 this week, it's already getting better

[coral vector]

awesome, thanks for sharing

[tepid nova]

@civic yacht I pushed an approximation for lockfile, can you tell me if it's directionally correct? 🙏 If so, I'll work on polishing it. Don't need a detailed review yet, just a general "getting warmer" or "getting colder"

[civic yacht]

Directionally looks great, my slop detector fired on one thing but not anything fundamental, just needs cleanup so it doesn't keep going down that over-complicated path

[civic yacht]

Slightly tangential but in my journey of getting better at wrangling agents, when it comes to large scale changes, I've recently had great success in telling the agent to actually include code snippets in the planning document. i.e. in that PR you have hack/designs/lockfile.md but it's all just high-level text.

I used to only be high-level like that too but then when you move to actual implementation the agent starts improvising and doing weird unexpected things (either because there was an actual point missed during design or sometimes just because it feels like it).

Now in my planning markdown files I have a ## Design section that is higher level and a ## Implementation Plan section that is very concrete and has actual code snippets. I tell it something like:

In the implementation plan section, create a subheader for each and every file that will be impacted by our changes. There should be one and only one subheader for each file. Within those subheaders for each file, create further subheaders for each significant modification (whether addition, deletion or change) that will be made in the file. Include code snippets for any type signatures (e.g. structs, struct fields, interfaces, function/method signatures, etc.) so that we can be sure types are consistent and well defined throughout the document. Additionally, for any implementation code that's significant, include snippets. We don't need every single if err != nil but we do need to nail down the shape of significant code and ensure it's consistent with the rest of the design.

This has made a huge difference. Writing out the implementation plan takes longer, but it saves so much time later. The agent itself is WAY better at noticing inconsistencies, missing design aspects, etc. when it does this. And I also can do a much more thorough review of the plan ahead of time. It's way easier to review a verbose document like this than it is the actual full final code, at least for large scale changes.

Just mentioning because I suspect some of the weird stuff I noticed in the PR is the type of thing I've had much more success catching during the design phase since I switched to this new approach

[tepid nova]

Yeah makes sense. I've gone back and forth, my issue with the "bundle both" approach is that the doc gets super verbose, and sometimes the agent goes too far in the weeds on implementation, and I have to waste tokens getting it back on track on the design part.

But, once the design is cleanly split up into well-scoped parts (like Lockfile) you're right that it's not as much an issue. So I'll try doing that again

For context this is my design stack right now 😅

                                           Stdlib stage 1
                                                │
  Workspace API (done) ─────────────────────────┤
          │                                     │
          ├──────────────────┐                  ▼
          ▼                  ▼             Stdlib stage 2
  Workspace plumbing    Lockfile                │
       (done)                │                  │
          │                  ▼                  │
      Artifacts    Workspace configuration      │
          │                                     │
          ▼                                     │
   Execution Plans                              │
          │                                     │
          ▼                                     │
    Collections ────────────────────────────────┤
          │                                     │
          ▼                                     ▼
      Provenance                           Stdlib stage 3

[still garnet]

Native interfaces + IDs 🧵

[tepid nova]

FYI @civic yacht @still garnet I roped @rocky plume into carrying lockfile 🙏

https://giphy.com/gifs/aew-all-elite-wrestling-double-or-nothing-2021-TuARLGCHFWgDifNnen

[leaden glade]

I'm hitting this issue again, and it's really a pain because it means I can't have two playgrounds at the same time, making difficult to work with agents on multiple parallel tasks/worktrees.
So I asked claude to investigate and here is the result.
Does that ring a bell to someone? Is that worth spending more time on it? Should I pick one of the proposals?
(yes a lot of questions but that's because I'm not really familiar with those aspects, cache keys, etc)

● Root Cause Found

The bug is a cross-session content-digest cache collision in dagql.

What happens:

1. Session S1 (from dagger-up) loads moduleSource("...dagger-up/toolchains/engine-dev").asModule() — the result Module carries ContextDirectoryPath = ".../dagger-up" and gets stored in the global completedCallsByContent[contentDigest] cache
2. Session S2 (from develop-generate) loads moduleSource("...develop-generate/toolchains/engine-dev").asModule() — the storageKey differs (different client), but the content digest is identical (same source code in both worktrees)
3. dagql finds S1's result in completedCallsByContent and returns it to S2 — without running asModule again
4. S2 now holds S1's Module object with ContextDirectoryPath = ".../dagger-up" — wrong path
5. When resolving +defaultPath="/", it calls host.directory(path=".../dagger-up") — fails with "module not found"

Key files:

• dagql/cache.go:965-1000 — content-digest lookup, shared across all sessions
• engine/server/server.go:609 — baseDagqlCache is global
• core/schema/modulesource.go:226 — asModule uses default caching (no per-session scope)
• core/modulesource.go:483-492 — CalcDigest() is content-only, path-independent

The content digest is correct for caching the type definitions (same code = same types), but the Module also carries path-dependent state (ContextDirectoryPath) that is session-specific.

Given this is a focused bug, here's my question: the fix needs to prevent the content-digest cache from returning a Module whose ContextDirectoryPath belongs to a different session. I see three approaches:

A. Include ContextDirectoryPath in the content digest key — so different worktrees produce different digests. Simplest, but breaks the "same content = same cache" optimization for legitimate cases (e.g., two deps pointing at the same code from different paths).

B. Scope asModule with CachePerSession — prevents cross-session hits entirely. Safe, but may lose some legitimate cross-session caching.

C. Post-validate content-digest cache hits — after getting a hit, check if the ContextDirectoryPath matches the current source. If not, bypass the cached result and re-execute. Most surgical, preserves optimization for same-session cases.

I'd lean toward C as it's the most precise fix. What's your preference?

[leaden glade]

I'd love to have some reviews of dagger up if anyone has some cycles https://github.com/dagger/dagger/pull/11959
I refreshed it after the workspace plumbing merge, PR is green, tests are passing, everything seems good on this side 🙂

[still garnet]

Better Cloud Tests 🧵

[leaden glade]

@civic yacht Do you have a branch with the changes you wanted to do regarding moduleTypes?
While working on dagger generate to codegen modules, I hit some issues with moduleTypes (and self calls in some way). And before to invest too much in this issue, I'd like to have a look at the changes you wanted to make, to not go in a wrong direction and do something that would be incompatible.

[tepid nova]

I'm trying something new:

• Write design proposal with codex
• Have codex run the claude CLI to run an adverserial review on writing style 🙂

[tepid nova]

OK evidently codex is not well-trained on invoking claude code 😂 lots of trial and error

[leaden glade]

You can have the opposite, openai made a codex plugin for claude code: https://github.com/openai/codex-plugin-cc
/codex:adversarial-review

[tepid nova]

I saw that, but I use codex personally so 🙂

Also codex is a shit writer

(but a great coder)

@leaden glade I took a look at your dagger up PR, exciting 🙂 I had a general UX question, and codex found a small potential hang

[tepid nova]

First pass at an overall design & roadmap for modules v2 https://github.com/dagger/dagger/pull/12900

[dawn mauve]

The .dagger/config.toml part isn't released yet right?

[tepid nova]

That's Workspace configuration. It's coming up next.

It's at the polish stage

@leaden glade I'm especially interested in your thoughts on how dagger up would take advantage of collections 🙂 Especially around naming of services. Imagine a dynamic collection of services, eg. one per app

https://github.com/dagger/dagger/blob/modules-v2/hack/designs/modules-v2/collections.md

[leaden glade]

Collections and dagger up

[leaden glade]

moduleTypes, self calls, codegen, dagger generate and commit generated files

[charred lotus]

n00b question: what's the official way to update the golden tests? for instance the ones in dagql/idtui ? When i run go test ./dagql/idtui -update on main I either have a diff because its using my main docker driver, if I specify export _EXPERIMENTAL_DAGGER_RUNNER_HOST=dagger-cloud://default-engine-config.dagger.cloud then I get please run 'dagger login <org>' first or configure a DAGGER_CLOUD_TOKEN even though i'm logged in, and if i do dagger run go test ./dagql/idtui -update then i get initialize nested client: do init: Post "http://dagger/init": http2: failed reading the frame payload: http2: frame too large, note that the frame header looked like an HTTP/1.1 header

Anybody has seen anything like this? What am I doing wrong ?

[tidal spire]

I think you should try dagger call engine-dev test-telemetry --update

[tepid nova]

I wonder how we could make that more discoverable... maybe attach it to generate?

[tidal spire]

Yeah generate makes sense as long as it produces the same result every time (i assume it would?)

though I just ran it on main and it gave me a changeset so 🤷

[tepid nova]

yeah it has to be idempotent AND not crazy slow...

but, maybe combined with a declarative config?

same pattern I was picturing for all the bump functions...

we could tie it to a semver tag 🙂

or, cache it with a long ttl? but only useful once we have better cache distribution

first question is: when should we update the golden test?

invalidated by certain files?

if so, could be a go generate on those files?

[tepid nova]

dagger up approved @leaden glade , let's ship it 🙂

[tidal spire]

did we want dagger check generated to be a builtin thing or can I split out generated into a reusable module?

[tepid nova]

I remember we talked about it, but I don't remember what we decided 😅

[tidal spire]

can you create dagger/generated and i'll push a dang module there? it won't use patchpreview like our current one in https://github.com/dagger/dagger/blob/main/.dagger/main.go#L19 but it can run the generators and evaluate that all changesets are empty

[tepid nova]

I think that's a very weird and confusing name for a public module in our stdlib though

[tidal spire]

True. I kind of like the builtin option tbh

[tepid nova]

As a easy first step we can add dagger generate --check, it doesn't solve the issue of integrating seamlessly into dagger check, but at least the option is there and it's non-surprising I think

[tidal spire]

would that bridge to a value set in .dagger/config.toml that enables something like dagger check generated?

[tepid nova]

Not sure

Looking at my design docs... I think yeah actually it wil fit nicely to make it a built-in behavior of dagger check

We can change the check walking logic, so that every +generate function qualifies as a check. Then an opt-out flag like dagger check --no-generate if you really don't want that

[tidal spire]

With up, is there something similar to generators where I could dag.CurrentModule().Services() or something?

That would be handy for the playwright module I'm working on

[tepid nova]

I'm guessing there is (one way or the other, CLI needs API access to those services for dagger up to work)

What will you do with them? Auto-bind them all?

--> If so I'm curious under which alias?

[tidal spire]

Haha yeah, still thinking through that. Maybe all as a single ServiceGroup (which doesn't exist yet)

ServiceGroup is the idea we've talked about to replace the kpenfound/dagger-modules/proxy module. A bunch of Services exposed at different ports on a single hostname

[tepid nova]

@tidal spire would you also want these services to be able to discover each other via hostname? In that case it's not exactly like a proxy, and more like a docker network

[tidal spire]

With up, we tunnel everything to localhost and there is no discovery, right?

[tepid nova]

Today yes. But since you're talking about the future with ServiceGroup 🙂 I was wondering what the best version of the future would be

[tepid nova]

Thanks for the reviews @obsidian rover 🙏

[civic yacht]

Can confirm that after I pulled your PR changes into mine, the deadlock magically disappeared 🎉 Re-ran 10 times with no repro

[still garnet]

Do I remember right that we were considering splitting out our alpine/wolfi module(s) into a separate repo? I only remember an offhand comment from a call, maybe from @tidal spire?

[tidal spire]

We had talked about it because https://github.com/dagger/pytest was using the wolfi module which meant re-cloning github.com/dagger/dagger every time. Its not using the wolfi module anymore, but we should split it out anyway

[tepid nova]

Yeah there is a loosely defined plan for a "stdlib" which would be made of individual modules, maintained by core team, each with their own repo under github.com/dagger

We started doing it ad-hoc for brand new modules

[tidal spire]

Yeah, any module currently in modules/ that makes sense to use outside of the dagger repo should be put in its own repo

and converted to dang where possible 😛

[tepid nova]

@still garnet btw you still have a PR open on bundling dang right? Or is that merged?

[still garnet]

yep this one - it's in the milestone: https://github.com/dagger/dagger/pull/12830

[tepid nova]

Once that's in, we should consider moving the SDK code into the repo. It's akward doing repo-wide changes that can touch every official SDK.. except the most important one..

[still garnet]

the SDK code is already in-repo

github.com/vito/dang/dagger-sdk still exists, but is orthogonal to the native SDK, which is what that PR touches. it'll be retired eventually

[tepid nova]

Oh! Nice.

[obsidian rover]

@leaden glade the TestUpPartialStartupFailure test takes at least 15 min to complete and slows the test-split:test-base like crazy. Could we find a way to improve it today / scope it ?

[charred lotus]

@civic yacht @still garnet can we merge https://github.com/dagger/dagger/pull/12830 ?

[still garnet]

yep!

i'll merge

[obsidian rover]

Quick workspace caching question (repros on main):

It seems that the workspace API introduced a regression ?

moduleSource.asModule caches the toolchain module solely by source digest/scope, so two loads of the same blueprint with different legacyArgCustomizationsJSON reuse the same cached instance.

In TestToolchain/TestToolchainsWithConfiguration, the first subtest (function:["*"] -> "hola") warms the cache, and the later subtest (function:["configurableMessage"] -> "bonjour") reuses that cached module, so the new override never applies.

While reviewing https://github.com/dagger/dagger/pull/11906 I added (unpushed yet) a small cacheKeySalt on core.Module: ContentDigestCacheKey() hashes the salt, and moduleSource.asModule hashes the legacy args and calls SetCacheKeySalt.

That way each distinct customization set gets its own cached module and the cache key matches all immutable inputs. Would that approach align with the expected cache semantic ?

[civic yacht]

I actually went down down that road while squashing some test failures on theseus. I ended up fixing the issue I was hitting in a completely separate way, but the commit message mentions this sort of salt approach too: https://github.com/sipsma/dagger/commit/bbd85ccdf00a9ca1b81d3b487d8667c63a1244ca

[civic yacht]

Even though that commit fixed the problem a totally different way, I don't doubt that the underlying issue you mentioned is separate and legit. This is something where I'd highly prefer just basing on top of theseus rather than merging onto main directly. Unless of course we need this fix in critically in the release on wednesday. Either way, happy to help you base a repro and fix on top of the theseus branch if that's the way you wanna go @obsidian rover

[obsidian rover]

Either way, happy to help you base a repro and fix on top of the theseus branch if that's the way you wanna go
I think rebasing on top of theseus is totally the way, happy to sync a bit tomorrow / have guidance on how to fix

[civic yacht]

ping me anytime

[civic yacht]

@still garnet (sorry about ping forget timezones) in your opinion is something wrong if this diff is appearing in telemetry output:

-├╴$ Container.from(address: "alpine"): Container! X.Xs CACHED
+├╴✔ container: Container! X.Xs
+├╴$ .from(address: "alpine"): Container! X.Xs CACHED

(ref trace)

I haven't pinpointed why it's happening yet, but wondering if it's worth investing time in it vs. just accepting the new output format

[still garnet]

that seems fine to me as long as it's stable

[civic yacht]

yeah seems stable, so I'll go with this

[still garnet]

the logic there is just that previously the UI didn't see the container span show up, not sure why but it could be some session-local cache hit

[leaden glade]

oops
I'll check that, I'm surprised it's taking that long, but I'll fix that today for sure

[leaden glade]

oops :lurk:

[leaden glade]

A lot of failures with error committing n950fganzl3hyz1jtmfr00ejf: database not open Does that ring a bell to someone?

[tawny iris]

@leaden glade If you have a moment, could you please re-review my PR? Here's the commits I added in response to your first review

It has failing CI but PHP passes which is the only thing I touched.

PS: No rush, it's not urgent. I'm just looking forward to getting my hands on those fields in other work 😋

[tawny iris]

PHP: Support Getters by charjr · Pull Re...

[leaden glade]

To follow up on dagger up, here is the PR to add the support in Dang: https://github.com/dagger/dagger/pull/12919

[still garnet]

oh, while we're in the area, we should add support for cache control directives. i'll pick that up real quick

[tepid nova]

@rocky plume are you able to rebase lockfile on egraph today? Since egraph is definitely getting merged first, we might as well get the rebase done asap

One step into the promised land 🙂

[rocky plume]

yeah! that sounds like a great idea.

[tepid nova]

Do you think you can do it today? OK if not, I might give it a try in that case

[civic yacht]

Whoever it is, lemme know how I can help. Almost everything is at least a little different now

[rocky plume]

yeah i'm giving it a try right now. the egraph branch has 423 commits in it, I wonder if I should try squashing either the egraph branch or the lockfile branch then try either a rebase or cherry-pick?

I'm worried I'm going to get stuck doing a lot of conflict changes if I have to resolve them over all the various commits

[tepid nova]

Probably worth squashing lockfile for this

We can keep a backup somewhere

[rocky plume]

yeah I'll try that with a cherry-pick. and yes I'll definitely have a backup

1fb5345bb46d51783f1ea990b6621514a6023bec is the current lockfile commit.

discord is the perfect backup medium..... 😈

(i'll create a branch in git too, don't worry)

[charred lotus]

Does anyone know what kind of Google credentials are used for GCP secrets ? The function we've been using is deprecated for security reasons and i'm not familiar with google credential types: https://pkg.go.dev/google.golang.org/api/option?utm_source=godoc#WithCredentialsFile do i just use the "ServiceAccount" example ?

^^ ping @bronze hollow since you added this in https://github.com/dagger/dagger/pull/10510

[still garnet]

https://github.com/dagger/dagger/pull/12920

[obsidian rover]

@fresh harbor I reviewed and updated for what might have seem to be a forgotten edge case on https://github.com/dagger/dagger/pull/12862, if my changes look good to you, I'll merge it

[charred lotus]

@still garnet do you know anything about go.opentelemetry.io/otel/semconv/v1.39.0 and why it's that version ?

On a Go bump PR (from v1.41.0 to v1.42.0) I have the following error I'm not sure i understand:

failed to set up OTel resource error="error detecting resource: conflicting Schema URL: https://opentelemetry.io/schemas/1.39.0 and https://opentelemetry.io/schemas/1.40.0"

[still garnet]

there is a correct version to use for each otel version, you have to make sure they're aligned everywhere 🫠

it's on The List of OTel Warts

https://tenor.com/AhRN.gif

[charred lotus]

the correct version is v1.39.0 or v1.40.0 ?

[still garnet]

if you bumped otel to v1.42.0 it must be v1.40.0 if it's complaining. you have to check the release notes for the otel version

[charred lotus]

Little lazification when connecting to a dagger engine via docker: https://github.com/dagger/dagger/pull/11769 if someone needs a distraction

[tepid nova]

Green CI

Looks like it needs a rebase

[obsidian rover]

on the todo ahah 😇

[charred lotus]

I dont think it needs a rebase to be merged, or is there a new rule? It just needs a review

[civic yacht]

It's much better to have it rebased. Sometimes there's no code conflicts but still doesn't work on main

[charred lotus]

@civic yacht Objections to merge https://github.com/dagger/dagger/pull/12894 ?

[civic yacht]

nope lgtm

[still garnet]

Splitting hairs since it can help save time, but you can also merge main into your PR. It's just gonna be squashed when you merge into main anyway, and sometimes it'll be way easier to deal with all the conflicts at once with a merge, if your PR has enough commits piled on

[rocky plume]

I'm starting to get mad Sisyphus vibes on this, trying to make sense of the large conflicts. Some of the conflicts are easy enough to solve like simply adding a .Self() in as needed, but other parts are related to functions that have been deleted and I don't have enough knowledge of the egraph changes in order to quickly resolve them. I was hoping to at least get to a point where I could compile the code to see what breaks but I haven't even gotten that far yet.

Maybe I should try cherry-picking a single commit out of the lockfile branch rather than trying to cherry-pick the entire squashed branch?

[charred lotus]

I'm seeing lots of failed to initialize module: failed to get type defs json during module sdk codegen: exit code: 1 flakes. Known issue? Somebody working on it?

[civic yacht]

Would need a link to a trace or more of the actual error, that's just a wrapping error message of something underlying

[charred lotus]

@still garnet I updated https://github.com/dagger/dagger/pull/12002 addressing your comments (see latest commits), in case you have few spare cycles. In exchange i'll review the dang sdk cache directive PR.

@still garnet also if you dont have any objections I'll merge https://github.com/dagger/dagger/pull/12919 when CI is green

[still garnet]

no objection

[charred lotus]

@tepid nova Any objections if i remove backwards compat from the dagger client with the dagger engine v0.3.7- showing an error message asking to upgrade ? It's to avoid a panic in my "faster connect" PR in that unlikely scenario.

[obsidian rover]

@civic yacht, for this PR: https://github.com/dagger/dagger/pull/12002, would you prefer me merging it after theseus or it's ok to include it for tomorrow ?

[civic yacht]

go ahead and merge now, the merge conflicts with theseus will be superficial so I'll just do that

[leaden glade]

@tawny iris CI was green, I merged the PR.
🎉 PHP just got Getters support, thanks

[leaden glade]

Based on some previous discussions I've seen, I implemented "generate-as-check" 🙂
So with those changes, all +generate function is automatically also a check.
That way, dagger check will automatically ensure all the generated code is up to date.

$ dagger check -l

Name                                         Type   Description
changelog:generate                           ↻      Generate the changelog with 'changie merge'. Only run this manually, at release time.
ci:bootstrap                                 ✓      Build dagger from source, and check that it can bootstrap its own CI
cli:release-dry-run                          ✓      Verify that the CLI builds without actually publishing anything
docs:lint-markdown                           ✓      Lint documentation files
docs:references                              ↻      Regenerate the API schema and CLI reference docs
elixir-sdk:codegen-test                      ✓      Run dagger_codegen tests
elixir-sdk:lint                              ✓      Lint the SDK
elixir-sdk:release-dry-run                   ✓      Test the publishing process
elixir-sdk:sdk-test                          ✓      Run the SDK tests
engine-dev:release-dry-run                   ✓
engine-dev:generate                          ↻      Generate any engine-related files
go:check-tidy                                ✓      Check if 'go mod tidy' is up-to-date
go:lint                                      ✓      Lint the project
go-sdk:generate                              ↻      Regenerate the Go SDK API
...

This also helps us (dagger/dagger) move one step forward: there's not anymore a .dagger module. I removed the source: go in dagger.json, and the main module is only a list of toolchains.
Not yet a workspace, but it's going in that direction 🙂
https://github.com/dagger/dagger/pull/12923

[leaden glade]

So it looks like to work locally, but the CI looks like to not run the "generate-as-check" ones. Maybe something to adapt on Cloud side?
My guess is it will be available once released, cloud will grab the new engine.
Question behind is: should I bring back the .dagger module until it's available on Cloud side? Then Cloud should see both the new generate-as-check functions and the current check generated and at that time we can remove it

[tepid nova]

thank you! 🙏

[tepid nova]

depends why it doesn't work in cloud 🤔

[still garnet]

might just be me, but i feel like including generators in check -l is a little noisy. sometimes i use that list for skimming what kinds of tests/lints/etc. are around

[leaden glade]

I thing it doesn't work because this is an engine change and the Cloud engine is running on releases. So the engine deployed on cloud don't have the new behavior

[leaden glade]

the thing is if dagger check will run generators then they should be in the list
but dagger check --list --no-generate will not display them. Maybe I should had a short flag?
Or instead of one single list it can also be two. The checks, then the generate functions. So you can scan only checks more easily.

[still garnet]

splitting the list (or just grouping them within the table) would help yeah 👍

[leaden glade]

I'll try something in that space

[storm wind]

I think it's useful to be able to mark a generator as +no-check in case generated files are not source-controlled.

[leaden glade]

Name                                         Description
ci:bootstrap                                 Build dagger from source, and check that it can bootstrap its own CI
cli:release-dry-run                          Verify that the CLI builds without actually publishing anything
docs:lint-markdown                           Lint documentation files
elixir-sdk:codegen-test                      Run dagger_codegen tests
elixir-sdk:lint                              Lint the SDK
elixir-sdk:release-dry-run                   Test the publishing process
...
typescript-sdk:test-nodejs-lts               Test the SDK with LTS version of Node
typescript-sdk:test-nodejs-prev-lts          Test the SDK with previous LTS version of Node

Generators
Name                            Description
changelog:generate              Generate the changelog with 'changie merge'. Only run this manually, at release time.
docs:references                 Regenerate the API schema and CLI reference docs
engine-dev:generate             Generate any engine-related files
go-sdk:generate                 Regenerate the Go SDK API

Or

Name                                         Description
ci:bootstrap                                 Build dagger from source, and check that it can bootstrap its own CI
cli:release-dry-run                          Verify that the CLI builds without actually publishing anything
docs:lint-markdown                           Lint documentation files
elixir-sdk:codegen-test                      Run dagger_codegen tests
elixir-sdk:lint                              Lint the SDK
elixir-sdk:release-dry-run                   Test the publishing process
elixir-sdk:sdk-test                          Run the SDK tests
...
typescript-sdk:test-nodejs-lts               Test the SDK with LTS version of Node
typescript-sdk:test-nodejs-prev-lts          Test the SDK with previous LTS version of Node

Generators
changelog:generate                           Generate the changelog with 'changie merge'. Only run this manually, at release time.
docs:references                              Regenerate the API schema and CLI reference docs
engine-dev:generate                          Generate any engine-related files
go-sdk:generate                              Regenerate the Go SDK API

(with bold Generators - do we want generators? generate?)
I think I'll go for the second option, I prefer everything to be aligned

[leaden glade]

The purpose of dagger generate is to generate files intended to be committed.
To generate files without to be committed I think a simple function returning a Changeset but without to be +generate seems better.

[charred lotus]

Are there any nixers that can review https://github.com/dagger/dagger/pull/12927 ?

[tepid nova]

@copper snow if you're around? 🙏

[copper snow]

reviewed

[charred lotus]

Köszi!

[tepid nova]

Lockfile on egraph 🧵

[tepid nova]

A skill I've been using a lot: manual testing of engine & CLI changes with playground: https://github.com/dagger/dagger/pull/12929

[tepid nova]

@charred lotus @obsidian rover @wild zephyr sorry for being so slow, I was waiting on dagger checks to complete

They never completed. So had to kill them, now starting over

[charred lotus]

We can take over the baby sitting of the PR.

[tidal spire]

Following up on this (+up, workspace service discovery) now that I'm closer to having a playwright module working

We currently have Workspace.Services that returns an UpGroup I can filter by name or port and selectively Run.

The only missing piece is to go from a dagger.Up to a dagger.Service. From what I understand Up.Run() will run the service and tunnel to the host. I'm sure I could find a hacky workaround to get from that to a Service I can pass to my playwright container, but it feels like a gap when all the other pieces are there.

As for selecting the right service, we could either pass in a name or port via customizations to pick it out of all Up Services, or if those aren't passed try to be clever with the port (80, 8080, 3000)

[tepid nova]

@still garnet getting this in my "switch to builtin dang SDK" branch... Just checking if it's a known thing?

[still garnet]

hmm nope

[tepid nova]

oh wait... nevermind.

I branched it from workspace instead of main 😭

Ah nevermind my nevermind - error still appears when rebased back on main

Confirming that sdk: "dang" seems broken in 0.20.5

Any function I call with that SDK, fails with ! failed to get client caller for "tpwqbiko9pmrk6g7wxlepaap5": context deadline exceeded

[obsidian rover]

I think i encounter another issue, currently digging, also linked to some dealdine query diff stat: Post "http://dagger/query": command [docker exec -i dagger-engine-v0.20.5 buildctl dial-stdio] has exited with exit status 137, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=

[tepid nova]

related to mine?

that looks like an engine crash

[obsidian rover]

nope, i thought it was. And i think you're right, crash 👀 (on dagger generate)

[tepid nova]

Can anyone reproduce any of these commands with 0.25.0 please?

dagger -m github.com/dagger/dagger@v0.20.3 call engine-dev playground terminal

dagger -m github.com/dagger/dagger@v0.20.2 call engine-dev playground terminal

[obsidian rover]

Confirming that dagger generate now crashes the engine on main, when regenerating at the root of the repo, either due to my diffStat or changes on the TUI

It's linked to my small machine. There is more memory used vs in the past, but nothing blocking. will investigate tomorrow

[leaden glade]

Ho, I didn't realised you wanted to retrieve a service. Interesting, as +up functions are returning services, but the goal is to run them right away.
So maybe I can add a function on the UpGroup like AsServices so you can use the up feature, and get the underlying services and do everything you want then. It shouldn't be hard, I can have a look

[leaden glade]

I can't reproduce it, even when reducing the memory allocated to my docker engine 😕

But the output of generate is a bit weird, like if it wasn't fully done

[leaden glade]

Even with Docker Desktop at 8GB It works, but the output is weird, like unfinished. Could it be linked to the tuist changes?

[leaden glade]

@tidal spire It's in draft but it allows to retrieve services from up: https://github.com/dagger/dagger/pull/12948

[obsidian rover]

Do you have the apply toggle ?

[leaden glade]

I'll try again, I had no changes to apply so maybe that's the issue

[charred lotus]

If anyone experienced regressions with Dagger 0.20.5, please let me know as soon as possible.

(very tempted to ping at-here)

[tidal spire]

+up services for modules

[obsidian rover]

Ok, i cleaned up my disk. I didnt see it but my disk was almost full and actually broke docker at the end of the day. Now that its clean everything works fine cc @wild zephyr @charred lotus

[charred lotus]

First good news of the week? 😄

[tepid nova]

@obsidian rover did you do this on purpose, to set us up for a good news 🙂

[wild zephyr]

@civic yacht I'm trying to troubeshoot a bug in the typescript SDK and I'm trying to make the engine pick my local version of the SDK for validation purposes. I've modified my repro's module dagger.json sdk -> source property so it points to the typescript/runtime folder in my system but dagger develop is not happy about that. Is this supported?

[wild zephyr]

I've tried both absolute and relative paths but same error

[civic yacht]

should work, we've used local custom SDKs before

[wild zephyr]

nvm layer 9 error

[charred lotus]

If anyone other than Yves sees overlapping TUI lines with dagger 0.25.0 i am interested to hear from you

[stuck bloom]

I am not seeing that on my side

I am using tmux. I had issues on 0.23.0 with stuttering but with vito's latest update they seem to be fixed now. TUI is working fine for me.

[bronze hollow]

Sorry I missed this. Do you still need an answer?

[charred lotus]

No worries! Yes I downgraded the dependency because of the issue but i would love a fix

[bronze hollow]

The simplest approach is to be logged into to gcloud CLI using “gcloud auth application-default login”

Next up is to provision a tightly scoped service account key (JSON) and make the file available locally and use direnv to export “GOOGLE_APPLICATION_CREDENTIALS”

[tepid nova]

@charred lotus @wild zephyr --> https://github.com/dagger/dagger/discussions/12766

[charred lotus]

gcloud auth

[charred lotus]

I have this weird memory that we no longer publish dev builds

[wild zephyr]

@bronze hollow welcome back! if you haven't seen the new Modules v2 - AKA Workspace - stuff I think you'll find it quite interesting. Lots of cool stuff has been happening here

[tepid nova]

It's possible that since I wrote this, we've stopped publishing them at all. But we definitely implemented what I propose in that thread

[bronze hollow]

I was actually talking about the Dagger live stream at KubeCon and how it’s been almost 5 years and almost nothing is the same. You and @tepid nova should join me for a new session

[tepid nova]

Also we're days (perhaps hours) away from removing buildkit 🙂

[lunar brook]

https://tenor.com/view/the-office-michael-scott-steve-carell-im-interested-im-listening-gif-4501197

Ooohh.... do explain!

[lunar brook]

Nevermind, I think I see the master plan 😄: by lifting operations out of Buildkit’s LLB and moving execution natively into DagOp, you are taking complete, direct control of the runtime. For the everyday user, this means escaping the bottlenecks of traditional file-based image building; relying fully on your native dagql engine unlocks true function-level caching that will make pipelines radically faster. Ultimately, bringing core commands like withExec completely in-house removes brittle layers of translation, resulting in a much more deterministic, general-purpose execution kernel for the whole industry.

Is this close?

[coral vector]

This is my understanding as a casual reader of this channel 🙂

[tepid nova]

pretty much! also reliability will greatly improve, we have obscure bugs caused by the 2 subsystems fighting each other.

And with full control of the codebase, we can tackle ambitious features that were out of reach before

[coral vector]

@tepid nova some spam cleanup to be done in a couple of channels ^

[coral vector]

^ is there a better @ for this? i.e. one for people who can delete msgs and kick a member

[charred lotus]

Hi maintainers, please refrain from merging PRs as we are preparing for 0.20.6 release. Thanks!

[leaden glade]

oops, that was me, just bumping some deps

[coral vector]

I'm curious if an update to the kube-pod engine client driver such that it could dynamically provision engine Pods would be welcomed?

[tepid nova]

Update on workspace:

1. Thanks to @rocky plume we are making progress towards rebasing on egraph branch
2. I am working on polish and tests
3. I NEED TESTERS

Once the emergency 0.20.6 release is over, I could use some help 🙏. We need to ship this...

[obsidian rover]

Currently digging egraph 👀

[tepid nova]

@dawn mauve FYI 👆 goal is to get a dev release ready for you to test ASAP

[tepid nova]

@tidal spire I'm doing a first pass at workspace docs. Includes a basic quickstart. Do you want to integrate intro into it now? Later?

Biggest open question for workspace quickstart: since it's all about installing & using modules, which modules?

[rocky plume]

sorry if it's already been mentioned, but is someone looking into the generated check failing? I'm getting the following compilation error on my branch:

# github.com/dagger/dagger/.dagger
./main.go:20:18: ws.Generators undefined (type *dagger.Workspace has no field or method Generators)

maybe it's related to the recent dagger generate issues?

[tepid nova]

Are you getting this error from Dagger Cloud Checks? Or locally?

ie. from CI or local dev

[coral vector]

kube-pod:// provision its own Pod

[tepid nova]

FYI I'm investigating the error in builtin DANG that is preventing us from switching. Here is the issue with a repro. I will look for a fix. https://github.com/dagger/dagger/issues/12974

[tidal spire]

Yeah makes sense to add it now imo. We will keep it updated

[tepid nova]

Is it usable as-is, for all users?

I'm thinking we could have 1) a very basic quickstart that has you install a few modules in a controlled environments (hello-dagger) and run checks.; 2) install intro in your repo and follow steps. I'm just not sure how to present both of those things in a way that flows

[tepid nova]

Can I get a quick ✅ please: https://github.com/dagger/dagger/pull/12929

[tepid nova]

Small win: fix our markdown linting (and refactor a small piece of our legacy CI config into a brand new reusable module using the builtin Dang SDK) https://github.com/dagger/dagger/pull/12976

[tepid nova]

A little hope in a difficult week: I was able to prompt codex with "Run dagger check -m ./modules/markdownlint and fix the issues until it's green". And it did. It was quite fast.

Note: I didn't say "dagger check markdownlint" because that is still fucking slow - it loads every module and our modules are F-ing slow to load. Baby steps though.

Also nice that half of the linting errors were fixed automatically with dagger -m ./modules/markdownlint call fix 🙂

[charred lotus]

[Workspace] What's the expected "current workspace" when doing dagger -m GitHub.com/org/repo call toolchain foo inside a git checkout that has a dagger.json ? Is it the remote git or is it the local checkout ? In 0.20.3 the toolchain's defaultPath arg would be relative to the remote git repo root, not the local checkout.

[tepid nova]

if you call the module directly (not via toolchain config) then 1) the workspace is your local checkout and 2) +defaultPath resolves to the module's repo, not your workspace

[charred lotus]

Ok and we are okay changing the behavior from 0.20.3 wrt 1) correct ?

[tepid nova]

the behavior will not change. unless you mean the behavior of calls to Workspace.directory() from the module?

[charred lotus]

@leaden glade can you paste your 0.20.3 example with java lint when you have time. I'm afk

[leaden glade]

+defaultPath resolves to the module's repo, not your workspace
🤔 if I'm right, that's what is not working then.
Yes, I'll reproduce the different cases and paste them so it will be easier to understand

[charred lotus]

Correct

[charred lotus]

The example off the top of my head was:

with a pre workspace plumbing version of engine such as 0.20.3:
cd daggercheckout && rm -rf sdk/ && dagger -m GitHub.com/dagger/dagger check java-sdk:lint

Ok so this is 2) in Solomon's case because the java sdk is a toolchain

But if you do dagger -m GitHub.com/dagger/dagger call java-sdk lint

Then there could be a discrepancy. To verify again cc @leaden glade

[leaden glade]

(I'm generating the matrix of cases, so we can have the different scenario and fix them)

2. +defaultPath resolves to the module's repo, not your workspace
   I can be wrong by my reading of the traces was that the defaultPath is translated into currentWorkspace.directory(path)
   So in that case defaultPath resolves to the workspace and this is exactly the issue we are facing

[tepid nova]

I can talk live in 5mn

[leaden glade]

👍
I'll have the matrix in a few minutes, just the time to run a few more cases with main

[tepid nova]

that should only be true if the module is loaded with legacyDefaultPath (which should only be set when converting toolchains, not with a direct -m in the CLI)

[leaden glade]

I'm just seeing that
So maybe that's the root cause, it should be false?

[tepid nova]

Those are toolchains being loaded. So it makes sense that legacyDefaultPath:true . But, this is a gray area because in the final workspace branch (11812), loading a module will not load its toolchains -> a module with toolchains will not be directly loadable anymore.

So, this exact trace shows something that works in main for backwards compat, but will break in 11812. So, the question of "what should happen to these toolchains exactly" doesn't have an answer in the workspace design.

[leaden glade]

Our test matrix:
I'm using the toolchains/java-sdk-dev (check java-sdk:lint)
This check, defined inside toolchains/java-sdk-dev is trying to access a directory at sdk/java/runtime/images/maven. This is made in two steps:

• a directory with defaultPath("/")

• by looking at the sdk/java/runtime/images/maven sub-directory
  If this directory is missing, the check is failing
  dagger -m github.com/dagger/dagger@main check java-sdk:lint

• v0.20.3

  • ✅ from an empty directory
  • ✅ from a clone of dagger/dagger
  • ✅ from a clone of dagger/dagger in which I removed the sdk/java/runtime/images/maven directory
    -> this means the defaultPath is looking at the module's repo file (it's finding the files inside github.com/dagger/dagger@main whatever they exist on the disk or not)

• 5793a4c47 (main just before our last changes)

  • ❌ from an empty directory: stat sdk/java/runtime/images/maven: no such file or director
  • ✅ from a clone of dagger/dagger
  • ❌ from a clone of dagger/dagger in which I removed the sdk/java/runtime/images/maven directory: stat sdk/java/runtime/images/maven: no such file or director

• e004ad8f8 (main, with the last changes)

  • ✅ ​​from an empty directory
  • ✅ from a clone of dagger/dagger
  • ❌ from a clone of dagger/dagger in which I removed the sdk/java/runtime/images/maven directory

[tepid nova]

In the spirit of workspace-plumbing (full backwards compat on top of the workspace concept), my guess is:

• when loading a module that has toolchains, those toolchains are still loaded
• those toolchains are loaded with the same overrides as if they were loaded from a workspace (so: legacyDefaultPath: true)
• As a special case for backwards compat, those toolchains are given a workspace that is the main module's context directory (so, in your case the git remote) --> this is the part that is not defined in the final workspace design. It's pure temporary compat

[stuck bloom]

Not sure if you are aware, 1.41 of the otel SDK has a CVE that's being reported. Which is fixed in 1.43. The latest dagger 0.20.5 updates some deps to 1.43 but not the otel SDK which has the CVE. - https://www.mend.io/vulnerability-database/CVE-2026-39883/

[charred lotus]

Yeah we updated the otel sdk, thanks!

actually lemme check if its 1.43

[charred lotus]

For the record, if somebody wants to investigate this data race in tests, it sometimes makes the CI fail: https://gist.github.com/tiborvass/ea0d223c7f88afb4a82bc1382a960940

[charred lotus]

v0.20.6 is out!

[fresh harbor]

noob question: Does Dang support replacing strings with regular expressions? My use case is to update the Elixir image in sdk/elixir/runtime/main.go to match the image in the Elixir SDK toolchain.

[leaden glade]

I think that doesn't exist yet.
There's an open issue about improving the stdlib: https://github.com/vito/dang/issues/6
And the stdlib is defined here https://github.com/vito/dang/blob/main/pkg/dang/stdlib.go

[fresh harbor]

Oh, thank you.

[tepid nova]

🧵MERGING EGRAPH 🧵

[leaden glade]

@tepid nova I pushed the markdownlint branch with a .markdownlintignore file, it works just fine to ignore CHANGELOG.md

[tepid nova]

🧵 MERGING WORKSPACES 🧵

@charred lotus could you remind me how to to install the latest CLI from main?

[charred lotus]

curl -fsSL https://dl.dagger.io/dagger/install.sh | DAGGER_COMMIT=$(git rev-parse origin/main) BIN_DIR=/tmp/main sh

[tepid nova]

mmm but I need the ability to do it without having a local checkout

[charred lotus]

just specify the commit

[tepid nova]

isn't there a special value I can set DAGGER_COMMIT to?

I don't have the commit

[charred lotus]

HEAD

[tepid nova]

ah! thank you

(this is for dagger --next)

[charred lotus]

ah, i think i have opinions to be tested about that. Will think about it and tell you

[tepid nova]

I'm just filing an issue for now, to track & remember

[tepid nova]

https://github.com/dagger/dagger/issues/12996

[leaden glade]

Side quest: improving our go lint module to make is nicer to user on personal machine: https://github.com/dagger/dagger/pull/12998

[leaden glade]

I have a behavior I don't know if it's related to egraph PR or not. (I'm on v0.20.6 on the egraph branch)
But at least I'm experiencing it on egraph PR. Dagger is taking all my RAM + swap.
For instance here, it's me running dagger check go:check-tidy
And now my system if completely stuck. At some point it might just crash in some way and it will free the memory, but sometimes I have to restart the VM I'm working on.
I'll do more tests (like with v0.20.6 on main) to ensure where it's coming from, but that's weird, and it's not all time.

[leaden glade]

I was able to hit q on the task, then after it closes from the terminal perspective the system was hanging for a bit and the memory has been freed
Side note, this is a fedora VM with podman, not my usual docker desktop on mac in case that matters

[leaden glade]

ok, it seems to be the same thing on main, and maybe that's because of the way we are loading some go modules 🤔

[obsidian rover]

2 potential culprits: my changeset PR or the new tui ? Last time I experienced it, I had some sdk builds loading from the toolchains, is it the same from a clean git on the same branch / commit ?

[leaden glade]

I'm relaunching from a clean worktree, on a freshly booted VM, let's see

[wild zephyr]

https://github.com/dagger/dagger/pull/12999 fixing an issue with cgroup nesting and init process within the engine image. cc @rocky plume @charred lotus

[leaden glade]

During my quest on parallelism and performances, I wonder if we shouldn't add a global flag to limit the parallelism of some commands like dagger check/generate (up might be different as we want them to be parallel).
That way, on small machines it doesn't try to run too much in parallel.
My current understanding of util/parallel is by default it's unlimited.
And also, we might like to have a different setup on a dev machine and on a CI.
Would it make sense?

2 potential culprits: my changeset PR or

[charred lotus]

Commented. We should use tini-static directly. You can even call it tini in the image to make it explicit where it comes from.

[charred lotus]

Yeah I think util parallel should never run without limit. The only case where this can be a problem is if you do parallel recursion. Then you can have a deadlock: a parent wants to spawn another child, but the limit prevents it, so the parent waits, and the parent can't release itself and thus the number of concurrent routines never goes down => deadlock. That's an issue i faced with smartchecks. Basically you can't use regular sync.WaitGroup (or equivalent errgroup). But apart from that situation, I agree we should have a default limit defaulting to MAXPROCS.

[leaden glade]

Yep. That's why I was thinking first about limiting the top level, so dagger check / dagger generate level.
And to have the ability for the user to limit it, for instance on golint that is using a lot of resources the limit of 1 was good, 3 was bad even if I have 6 cores available

(but it's in fact already 1 level deeper than check/generate)

[tepid nova]

There is a natural integration point for this in the modules-v2 design, FYI

--> With @collection we will support batch operations: a collection can be split in buckets, and the batch operation applied to each bucket in parallel. In Cloud we can auto-detect the ideal parallelism setting from telemetry over time. But locally, we'll probably need a manual --parallel=N flag. Once we have that - we could use the same flag to limit concurrency of all dagger functions for this client, maybe?

[tepid nova]

@charred lotus other candidates I like:

• dagger --release=dev|VERSION
• dagger --x-release=dev|VERSION (same but x is for "cross-release" and also for "this is special, only use if you know what you're doing)

Remaining mystery is: how to cleanly lookup immutable digest of current dev release (for download caching)

• If by looking up git commit of main -> might not be available in release yet
• If by fetching special lookup text file -> doesn't exist, extra work to release it and ensure it's up to date
• Pure http, etag check?

(I really prefer if lookup/invalidation is automatic and stateless - ie. no need to separately manage invalidation or refresh)

[charred lotus]

There's a whole logic in the CLI already called FromDownloadedCLI. It's using a checksums.txt that we download from dl.dagger.io.

It caches the CLI, and we should reuse that

Just tweak the pruning a bit

[tepid nova]

Ah great!

So we can use that checksum as the stable key

So the CLI already knows how to download another CLI? 🤔

[charred lotus]

well the checksum is a file with various versions, and we can redownload it and add a line for main

[charred lotus]

engineconn knows how to. Because when dagger is not installed and you have it in a go file, it downloads the dagger CLI

[tepid nova]

Ah right, it's the one thing every SDK has to reimplement to allow bootstrap

[charred lotus]

Btw, if you think of a subcommand to manage releases, it would be dagger release ... not --release. And I don't know if maybe we'd want to reserve dagger release for the future.

[tepid nova]

dagger release looks like you're using dagger to release your app

[charred lotus]

exactly

Honestly we are talking about configuring dagger.

[tepid nova]

even if we don't actually use it for something else (we're probably using ship), we can't use it for this

[tepid nova]

too general

My favorite so far is --x-release

--> Accurate
--> the X is an appropriate marker of being unusual / special, but not offputting

[charred lotus]

You said you wanted a subcommand

[tepid nova]

I know 🙂 But now not so sure

If we can make caching/invalidation automatic, then all you need is a string/enum flag (rather than a boolean flag like I was initially picturing)

We could even keep --next as a convenience alias for --release=next

[charred lotus]

i for sure will never use the flag, only the envvar.

[tepid nova]

That's fine they always come in pairs

--x-release + DAGGER_X_RELEASE

[charred lotus]

I'll do something and we can bikeshed on the names later 🙂

[coral vector]

Hi! I wanted to follow up to see if this was something that we'd be interested in supporting: #maintainers message

[leaden glade]

Here is my current list of PRs I'll soon merge into main (some of them was waiting egraph to be merged first)
Happy to get reviews on the ones with 🙏 (the others are already approved, but happy to discuss them if there's any question, sharing them here so you know what's coming)

• checks for generate functions: #12923
• use an AST based analysis to introspect python modules: #11803
• improvements on the go module (to limit parallelism so it's easier to run on small machines, and improving changeset merge): #12998 (this one was previously approved, but I changed the implementation)
• fix display or endpoint when using dagger up and exposing the docs server: #13002
• 🙏 re-include toolchains as dependencies in codegen: #12991 -> the feature has been added back while merging egraph, so this PR only adds a test to it
• 🙏 fix loading of dang modules containing typescript and go dependencies: #13007
• add a hidden flag to hide functions from the CLI (dagger functions) to hide the with field we are surfacing now: #13012 (context)
• fix opentelemetry metrics sent by python SDK and not handled by the engine: #13013

[topaz edge]

Hey there, I'm trying to update an open PR to accomodate the removal of buildkit -- https://github.com/dagger/dagger/pull/12049. It's fundamentally using buildkit.HostMount -- is there a strategy for replacing such things? Eg, https://github.com/dagger/dagger/pull/12049/changes#diff-10d41febe6f07f3eed32775ac5ec0a09eeda9867db7f5f3c57a39c75b90b708aR74-R77

[charred lotus]

Before we merge PRs, let's make sure we don't make the lockfile PR more difficult to merge

Friendly reminder to dogfood main:

curl -fsSL https://dl.dagger.io/dagger/install.sh | DAGGER_COMMIT=head sh

If you need sudo to install to /usr/local/bin you can do :

curl -fsSL https://dl.dagger.io/dagger/install.sh | DAGGER_COMMIT=head BIN_DIR=/usr/local/bin sudo -E sh

[tidal spire]

fresh bikeshedding opportunities 🚲 🏚️

• Service composition between modules for modules v2: https://github.com/dagger/dagger/pull/13016
• Bind Services to a Container's localhost: https://github.com/dagger/dagger/pull/13015

[leaden glade]

I'll only merge some that are independent and keep on the side the ones that might have an impact

[leaden glade]

I have a bug in lockfile, I don't know if it's a known one or not:
On a repo with toolchains declared, I ran dagger functions --lock=live to generate a lock file:

[["version","1"]]
["","modules.resolve",["github.com/vito/dang/dagger-sdk@352d057a4d4c4d77612104f842671b512495ef30"],"352d057a4d4c4d77612104f842671b512495ef30","float"]

But if I want to update it:

$ dagger lock update
✘ .update: Changeset! 0.0s ERROR
! update workspace lock: parse git URL "github.com/vito/dang": unknown protocol
Error: update workspace lock: parse git URL "github.com/vito/dang": unknown protocol

[topaz edge]

Reimplementing on top of theseus -- not too bad

[tidal spire]

https://github.com/dagger/intro/pull/3

[tepid nova]

Could I get a ✅ please: https://github.com/dagger/dagger/pull/12976

[tepid nova]

Cool for me to rename the repo?

[tidal spire]

send it!

github will redirect from the old name so it shouldn't cause any issues

[tepid nova]

<@&946480760016207902> FYI new module for the sdlib: https://github.com/dagger/shellcheck

(small dogfooding indagger/dagger , replacing project-specific logic with clean reusable modules, one small chunk at a time)

😴

😳

😭

[charred lotus]

Looking at a trace, it's running codegen generate-typedefs for 1m12s, I think this should be fixed with @leaden glade's work on committing generated typedefs.

[tepid nova]

I can't wait to dismantle that release toolchain.

@charred lotus FYI it might be an artifact of my super slow internet. If some of the SDKs re-download a lot of stuff, then I will feel that pain much more acutely on the current configuration

[tepid nova]

@charred lotus @obsidian rover @tidal spire remaining fail is definitely a flake 🙏

One more 🙏 https://github.com/dagger/dagger/pull/13024

[tepid nova]

YESSSS merged, thank you @charred lotus 🙏

Baby steps towards dogfooding + making our own CI better 🙂

[tepid nova]

@charred lotus are those go:lint errors in CI known flakes? My change shouldn't trigger go linting errors at all

(I noticed it happened in 2 completely distinct PRs)

[tepid nova]

BOOM

[charred lotus]

Yes they are 🙁 We really hope with upgraded CI it would go away, but i'm cautiously optimistic

Otherwise i'll investigate further

[tepid nova]

@charred lotus in any case the sooner we remove codegen on go:lint, the better. Maybe I will try that next

(doing powershell analyzer module now)

[charred lotus]

most of the codegen stuff is, to my knowledge, dependent on @leaden glade's work on requiring to commit generated files.

[tepid nova]

Well normally we should be able to stop ignoring our generated files right away, even before the engine stops re-generating them at runtime

It won't save the time at module load, but it allows us to run a normal linter against the repo, instead of having to dynamically codegen from within the linting module

[charred lotus]

Ah i see. Yeah i guess

[tepid nova]

It does require codegen to be deterministic... We fixed a bunch of those issues, but others may be lurking. I'll try and we'll see 🙂

By the way, thank you @charred lotus @obsidian rover @rocky plume , I am able to take a few hours to improve our CI and dogfood, because you're picking up the slack on the workspace branch 🙂 As soon as the rebase is done I will get back to workspace branch

[charred lotus]

@tepid nova if you can have a look at https://github.com/dagger/dagger/pull/13020, i just am doing ref+pin logic, because with the modulesourceID way, it doesn't work and i couldn't figure out the subtlety.

[tepid nova]

That doesn't shock me honestly... I'm guessing you already verified that it works for both remote and local modules? (otherwise tests wouldn't pass I suppose)

[charred lotus]

I'll check if there's a test for local module and remote toolchains

[tepid nova]

One more: https://github.com/dagger/dagger/pull/13025

Comes with a shiny new stdlib module 🙂 https://github.com/dagger/PsScriptAnalyzer

[tepid nova]

FYI @charred lotus this CI run looks like it's hanging. No re-run button either -> https://dagger.cloud/dagger/checks/github.com/dagger/dagger@06f29bc29645220ed51bf0a03d35edec9e3e924b

[tepid nova]

Couldn't re-run from Cloud, had to push a commit to trigger

[charred lotus]

Weird, I'm seeing a re-run button now, i re-run it

[tepid nova]

👆 this is probably why 🙂

Could I get a quick ✅ 🙏 @charred lotus

I have another one on the stack 🙂 coming soon.

I think I found how to eliminate one of our project-specific modules entirely

(a small one but still)

[charred lotus]

@tepid nova can i also have a ✅ https://github.com/dagger/dagger/pull/13020 ?

[tepid nova]

https://giphy.com/gifs/starwars-star-wars-episode-6-3ohjV1hga7WebHQq1G

[tepid nova]

but it's my own PR... I can't approve it 🙂

https://giphy.com/gifs/DeJearForIowa-vote-2022-steak-fry-m978qxXxCxLSIY2u9t

[rocky plume]

I gave it a +1, but take it with a grain of salt.

[tepid nova]

@charred lotus just to confirm, I'm no longer touching that PR (13020) it's all yours to do with as you please - force-push, merge, close, whatever

[tepid nova]

One more: https://github.com/dagger/dagger/pull/13026

--> removed the project-specific "installers" toolchain 💪

@tidal spire I made it a native go e2e test suite 🙂 Note the new directory, e2e/installers. Works natively in go test, and CI runs it with a brand new github.com/dagger/go reusable module 🙂

[stuck bloom]

Seems strange to see the dagger/go module written in dang

[tidal spire]

Nice, I like it!

[tepid nova]

Thanks - my favorite part is the // workspace:include trick, works well IMO

[wild zephyr]

when using -m $remote_ref does anyone know if it's possible to fetch that $remote_ref value within the function somehow?

[tepid nova]

dag.CurrentModule.<something>()?

[wild zephyr]

tried that but couldn't found anything

[tepid nova]

Quick performance side quest: trying again to get our generated files checked in, so we can run a go linter without waiting 20 minutes

[tepid nova]

🙏 https://github.com/dagger/dagger/pull/13031

--> dagger -W as a hidden flag, straight on main. We don't need to wait for the whole workspace branch for it.

[tepid nova]

I have to disconnect today. Will check in late this evening, then flying back to SF over the week-end.

Open threads from today:

• I opened a PR to move the helm e2e tests to e2e/helm, same idea as for installers
• Opened a PR to check in our generated files, and move module codegen out of go:lint and go:check-tidy, making them faster and simpler
• Opened a PR to fix the user-reported problem with dagger mcp
• Pushed a single commit to workspace to fix broken docs pages with netlify redirects

[wild zephyr]

@charred lotus @astral zealot this is the reason why the apko step was failing i the Cloud build (https://github.com/vito/dang/pull/43). Seems like we're using the latest apko image from chainguard and something probably changed which was causing the apko build command to fail with some permission errors. That PR fixes it.

Alex will probably merge it as soon as he sees it 🙏. Already silently DMed him about it

[tepid nova]

😵‍💫

I run the EXACT SAME PIPELINE, with THE SAME INPUTS, once locally and once on cloud... And the result is different.

Local run

dagger -s -m github.com/dagger/dagger@e2e-helm call golang modules | wc -l

Output:

user default: golang(selectModule="e2e/*")
user default: golang(selectModule="e2e/*")
       2

Trace: https://dagger.cloud/dagger/traces/52701004aa776c96eee4a12d893bb34f

Cloud run

dagger --cloud -s -m github.com/dagger/dagger@e2e-helm call golang modules | wc -l

Output:

user default: golang(selectModule="e2e/*")
user default: golang(selectModule="e2e/*")
       1

Trace: https://dagger.cloud/dagger/traces/43e78c894aa09bf000a4e4c8214d5728

(local run is the expected result)

[still garnet]

Where was this running btw that showed the issue? Stable Dagger or main? The fix was to mount the cache dir with the container's current user as its owner, which sounds suspiciously like something we could/should be doing by default already, so wondering if this was missed in Theseus or something, as opposed to a change in apko (assuming that's just a guess)

[charred lotus]

Main

[leaden glade]

https://github.com/dagger/dagger/pull/13039 should fix the failing test on https://github.com/dagger/dagger/pull/13035
(I put it in a separate PR as it's a problem not only on this PR, it's a regression after egraph PR)
As I'll be off tomorrow, feel free to review/merge it if that makes sense.

[tepid nova]

If anyone has any ideas on this one...

Or at least: can you repro? 🙏

UPDATE: looks like a caching bug, related to toolchain customizations. Will try to get enough info to file an issue

@still garnet I'm porting some of our e2e tests to a new module github.com/dagger/go. Of course written in dang 🙂

Right now dagger check golang will find and run 2 test suites: e2e/installers and e2e/helm. Problem, the output looks bad because 1) there's no custom spans, and 2) no sweet go test instrumentation. Do you have escape hatches for those up your sleeve? 🙂

[obsidian rover]

Github not showing the PRs makes it very very unusable ...

[tidal spire]

Discuss: https://github.com/dagger/dagger/pull/13015

New API: Container.WithLocalhostForward. Its like WithServiceBinding but it will bind to the container's localhost:<port>. Super handy when working with black box testing when its tricky to wire through alternative hostnames

Discuss: https://github.com/dagger/dagger/pull/13016

Some lightweight composition between modules for modules v2. Specifically, being able to pass an up function from a module to a constructor Service argument of another module with customizations. This is needed for generic resuable modules like Playwright taking in an app-specific Service without the Playwright module needing to know/guess at every possible way to run an app

[tepid nova]

Need a quick review 🙏 https://github.com/dagger/dagger/pull/13031

cc @obsidian rover this is actually a chunk of workspace that I'm fast-tracking, so you probably have the review context in cache

[tepid nova]

EDIT: managed to isolate a repro. https://github.com/dagger/dagger/issues/13046

I have a question about dagger develop. Currently it calls go mod tidy, so if a random dependency happens to be updated, it will barf a go.mod diff on the target module (or in my case: every module in the repo all at once).

Do we want this?

• Argument against: CI can randomly fail on check generated because dependency was updated after you committed but before CI was run. Seems like weird interference. But maybe just an anvoidable reality of software supply chain?

• Argument for: maybe there are legitimate reasons for dagger generate to write to go.mod and go.sum, separate from random dependency bumps?

For now, I'm going to make it so dagger generate filters out any changes to go.mod and go.sum (I can't prevent go mod tidy from actually being called, that's hardcoded into the SDK behavior at the moment)

[tepid nova]

🙋 can I get a quick ✅ for this please https://github.com/dagger/dagger/pull/13030

[still garnet]

approved, took a bit cause CI flaked a couple times, resolved when i re-ran from a clean slate

[obsidian rover]

Fixing the underlying problem atm, not good to merge yet

[tepid nova]

Is the issue introduced by my PR?

[obsidian rover]

if you run dagger call engine-dev test --pkg ./core/integration --run '^TestModule/TestRemoteModuleCrossRepoToolchainDefaultPath/empty_cwd$' --test-verbose locally, you see that this passes on main, not on your PR

I think i have the fix

[tepid nova]

Thank you! 🙏

[tepid nova]

heads up I'm rebasing + force-pushing

warn me if you push your fix

oooh is it related to dagger session -W clashing with dagger -W ?

[obsidian rover]

yes, dagger call parses the moduels and the workspace defaultarg conflicts

[tepid nova]

ok 🙂

you have fix ready to push?

(rebase done)

[obsidian rover]

yes, it's not perfect perfect, but good enough for now I guess, improving the commit message

[tepid nova]

Isn't it just reverting the older session -W commit?

deprecated by this

[obsidian rover]

The bug is that dagger call also creates flags from module args. So when a module has an arg named workspace, both things try to own --workspace

[tepid nova]

Ooooh.....

Don't we handle that for other arguments already?

[obsidian rover]

Nope, you cant have an arg named module, i have a good stopgap, i think Annd .... I was wrong

[obsidian rover]

pushed

[tepid nova]

Mmmm wait, I'm wondering if we're implementing this wrong... Let me check how I did it in workspace

[obsidian rover]

@tepid nova can i force push on the branch ? I was wrong, i have the right fix

[rocky plume]

Here's a new error I saw while running TestModule/TestFunctionCacheControl/go under the lockfile branch (unsure if it's lockfile or simply egraph related):

failed to initialize module: failed to load module from type defs json: load Module@Eg8I3nASCgoGTW9kdWxlGAE=: resolve result 14430: missing shared result

I did a prune then re-ran the test and it went away -- just sharing this here in case it comes up again.

trace: https://dagger.cloud/dagger/traces/170ef095e221da71af511ad950365125?span=6e7eb06fb72362d3#6e7eb06fb72362d3:L17

[tepid nova]

Is this a known flake? https://dagger.cloud/dagger/checks/github.com/dagger/dagger@2f3c4b3adefd1f24a0368edc01cc411ca20bf5f5?check=python-sdk%3Alint-docs-snippets&listen=d2e73109ee1020da#d2e73109ee1020da

[obsidian rover]

https://github.com/dagger/dagger/pull/13031 is green

[civic yacht]

That’s the one I mentioned here #1492563668670484662 message Yves’s work on codegen gets rid of the module types hack afaik so that’ll go away with that

(On plane back, checking discord on mobile, not back in virtual office til Thursday to give myself a jetlag recovery day )

[rocky plume]

welcome (almost) back, good to know it's not brand new (I did search discord for the error message, but it was hidden deep under a link). Have a good flight back.

[tepid nova]

👋 could I have a review on https://github.com/dagger/dagger/pull/13032

--> It's a CI-only change. Similar to e2e/installers -> but not e2e/helm

cc @tidal spire

[tepid nova]

Need a super quick LGTM for: https://github.com/dagger/dagger/pull/13042

(it fixes a test - dagger check helm fails on arm64 because of an incorrect platform comparison logic in the test)

[tepid nova]

New stdlib module: github.com/dagger/helm -> for now it only supports linting helm charts (for the needs of our own CI)

Merged https://github.com/dagger/dagger/pull/13035

Our generated module code is now checked in git - Go linting no longer requires custom just-in-time codegen

cc @stuck bloom @ocean hamlet as I believe you've had to deal with this problem also?

[stuck bloom]

If I understand correctly. This PR only tackles/supports linting and this (https://github.com/dagger/dagger/pull/13028) is what actually prevents runtime generation?

[tepid nova]

Yes my PR is only for our own CI. But it also serves to validate the best practice for others

After merging 13035, codegen will still be re-run at module load. But at least we'll have a go linting setup that is simple and fast 🙂

[stuck bloom]

We have our own toolchain/module for Go, loosely based on yours so I'll look into this. I think the real benefit will come when runtime codegen is prevented as the code is checked in. One concern I have is the repo size. I wonder if that will have an impact on remote module imports. Especially when we move to a workspace model where most things are imported at runtime.

[tidal spire]

I feel it 😛 this is my PR to do the same in the dagger repo

[tepid nova]

The vast majority of those added files are identical dagger.gen.go so they get deduped at the git object layer. There's an upcoming codegen improvement by @fair ermine that shrinks codegen output by 99% - by using core types from official client library, instead of re-generating them. At that point 99% of those files will disappear

[tepid nova]

Oh you had a PR still up-to-date? Sorry about that I just merged this: https://github.com/dagger/dagger/pull/13035

[stuck bloom]

Wow.. impressive! I wish there was a way to tell GH to make generated files much less prominent in PR review.

[tidal spire]

not at all up to date. Looking forward to see the difference when all the improvements ship!

[tepid nova]

I believe there is, with a "linguist / generated" git attribute no?

[stuck bloom]

oh? Didn't you have that already for generated modules?

It does hide the file contents.. But it's still in my face. They should entirely hide those files as no one would actually review generated files.

[tepid nova]

Yeah it's not ideal. One more thing agentic tooling can probably help solve in a new way

[tepid nova]

@still garnet new use case for a Dang module: lightweight codegen (injecting a Go helper alongside the go test file, as a convenience to access the workspace from the test code). I'm assuming there's no templating builtins? By default will build a small helper tool, in Go I guess

[tepid nova]

@tidal spire @still garnet I ran into a smell while working on me little e2e refactor. Might have consequences for modulesv2 🧵

[still garnet]

dang templating(?)

[tepid nova]

FYI https://github.com/dagger/dagger/issues/13051

I'm working on a fix

[mystic lodge]

Stemming off of this, we're also seeing this in our pipeline runs as well (at Nine). This does cause some issues for us as well since we check in our generated code and have validation for checking it's all in sync, and not entirely sure if it's the most ideal for sporadic dependency updates slipping in

[tepid nova]

To be fair that is an issue I have with standard go development in general... They tend to slip in unrelated go.mod changes, always annoys me

[tepid nova]

🙋 https://github.com/dagger/dagger/pull/13053 🙏

https://github.com/dagger/dagger/issues/13054

--> needed for clean e2e use case

[wild zephyr]

q: is generate still supposed to be hidden in the CLI? cc @leaden glade

[leaden glade]

good question. Maybe we should unhide it yes.

[leaden glade]

Question regarding https://github.com/dagger/dagger/pull/12991
Do we want this? This is only a anti-regression test to ensure toolchains are available as dependencies.
This is not something that will survive modules v2/workspace but we had the issue earlier this was removed.
Happy to close it if we think that's not that useful for now.
(trying to clean my list of PRs 😅 )

[tepid nova]

Sure why not, can't hurt to be explicit, even if we explicitly change or remove the test later

[still garnet]

could use a ✅ on https://github.com/dagger/otel-go/pull/8 - dogfooded in another PR and can confirm the desired effect (note the gaps in the timeline bars here: https://dagger.cloud/dagger/traces/c04c4486196c55afc8d3b9e3b2d536a4?span=13349b35ca154338)

[coral vector]

hi 👋 any thoughts on this? https://github.com/dagger/dagger/issues/13034

(I know it's an issue, not a discussion--I followed the contributing guide [https://github.com/dagger/dagger/blob/main/CONTRIBUTING.md#1-claim-an-issue] and figured that I was creating a discussion based on the form that I was filling out)

[tepid nova]

https://github.com/dagger/dagger/issues/13059

[tepid nova]

🙋 if anyone is looking for an easy contribution

[tepid nova]

Merged: another project-specific module removed from our CI 🙂

• toolchains/installers -> replaced by e2e/installers + github.com/dagger/go
• toolchains/helm-dev -> replaced by e2e/helm + github.com/dagger/go + github.com/dagger/helm

[still garnet]

me again, with a follow-up integrating otelgotest into our repo for better test telemetry: https://github.com/dagger/dagger/pull/12925 🙏

[leaden glade]

🙏 https://github.com/dagger/dagger/pull/12991
I can't get the generated to be green, but it's unrelated and everything else is good (especially the single test added by the PR)

[tepid nova]

https://github.com/dagger/dagger/issues/13064

[leaden glade]

I've merged https://github.com/dagger/dagger/pull/12923
This means now generate function will qualify as check.
Once the CI will run on a version including this, we will be able to remove the generated check in .dagger (I'll prepare a PR for that)
One of the benefit here is instead of having this single function that runs multiple generate to check if they are ok, it will be on dedicated checks, meaning if one generate function must be re-run it will be easier to catch.
Also one step forward to a toolchains/modules v2 only, without the need of a main module.

[still garnet]

https://tenor.com/vTbF7L2Td5D.gif

https://github.com/dagger/otel-go/pull/10

[tepid nova]

@still garnet did the otelgo binary move by any chance? I'm getting errors installing it (looking into it)

[still garnet]

the binary is named otelgotest, hasn't changed

https://github.com/dagger/otel-go/tree/main/cmd/otelgotest

[tepid nova]

Oooh so I guess it never worked... Note, our module testing story still isn't great, I actually have tests but setting them is still awkward.

@still garnet the error was installing .../otel-go@latest -> latest tag doesn't have cmd/

[still garnet]

ah yea makes sense. i guess pin to a commit for now?

[tepid nova]

Actually I will track main until things stabilize

We're still the only user of that module, for another few weeks I'm guessing

[tepid nova]

Update on CI refactor:

• ✅ removetoolchains/installers -> replaced by e2e/installers + github.com/dagger/go
• ✅ removetoolchains/helm-dev -> replaced by e2e/helm + github.com/dagger/go + github.com/dagger/helm
• 🏗️ remove all the bump functions called by release.bump -> replaced by clean generate functions + a single setting in the release module
• 🏗️ remove the all-sdks module (side-effect of 👆 )
• 🏗️ remove dependency on .changes/next from 1) release and 2) version string computation
• 🏗️ remove CLI reference doc generation from toolchains/docs-dev -> replaced by go:generate in CLI code
• 🏗️ remove GraphQL reference doc generation from toolchains/docs-dev -> replaced by reusable SpectaQL module plugged straight into dev engine (we'll need the same service reference plumbing than for playwright @tidal spire )
• 🏗️ remove docusaurus build from toolchains/docs-dev -> replaced by reusable docusaurus module
• 🏗️ remove netlify docs deploy from toolchains/docs-dev -> replaced by reusable netlify module
• 🏗️ remove toolchains/docs-dev -> replaced by all the above 👆

[tepid nova]

<@&946480760016207902> does anyone have context for why ./version doesn't use the ability to receive a contextual GitRepositoryargument? It receives a contextual Directory called gitParent instead, which I'm guessing is less efficient? Is it just that we never got around to it, or we actively can't for some reason?

cc @tidal spire @obsidian rover since I saw in the git history you've been around that file

Oooh maybe it doesn't work with git worktrees?

[tidal spire]

could be 🤔 the only thing I ran into regarding worktrees was the findUp looking for /.git/ instead of /.git since worktrees use a .git file

[tepid nova]

I see that there's a gitParent now, and the original git argument is optional with no default. I wonder if that git argument is needed at all? Do we ever use it?

[tidal spire]

I dont have the freshest context on version but my original idea for worktrees was to just use something like v0.0.0-commit if git can't be determined since we'd never actually release from a worktree

[tepid nova]

Confirmed, that git argument is never passed. There's a fallback to get git metadata from gitParent , at the moment that fallback is used 100% of the time. I'll include that in my cleanup

[tepid nova]

I'm looking into how version handles worktrees today

[tidal spire]

looks like it produces a valid version from worktrees. I think either @obsidian rover or @leaden glade got it working

[leaden glade]

I'm using worktrees every day (that's my main way to work on dagger). In case there's some stuff to test for instance.

[tepid nova]

Yeah but it uses a fallback - doesn't actually find the repo.

• latest release tag from remote upstream
• digest from inputs files
• date from current time (not commit time)

Workspace.git() will fix it 🙂 But I'm trying to do a first pass of cleanup before that gets added to the engine (yes I'm impatient)

[tepid nova]

@still garnet do I remember correctly that you implemented (or proposed?) a Workspace.stage() or something of that nature, for a function to proactively apply a changeset to the workspace?

[still garnet]

yeah, that's in the big pile of LLM work

[tepid nova]

Does it include the ability to stage git mutations? Like creating new commits, changing refs, creating tags?

(context, I'm looking into simplifying our release process, making it more automated and dagger-native)

[still garnet]

it had:

• Workspace.stage: https://github.com/vito/dagger/blob/8a77fab7b9626805246005549a15d00661fcf5fa/docs/docs-graphql/schema.graphqls#L6203-L6226 (apply + stage changes, leaving unstaged work unstaged)
• Workspace.apply: https://github.com/vito/dagger/blob/8a77fab7b9626805246005549a15d00661fcf5fa/docs/docs-graphql/schema.graphqls#L6010-L6024 (apply changes without staging them)
• Workspace.commit: https://github.com/vito/dagger/blob/8a77fab7b9626805246005549a15d00661fcf5fa/docs/docs-graphql/schema.graphqls#L6038-L6046 (commit staged changes)

[tepid nova]

I'm thinking if a dagger function could take charge of creating the release tag, along with the prep commit for it, we could automate the whole thing, with a clean human review gate at dagger generate

[still garnet]

oh and Workspace.withBranch: https://github.com/vito/dagger/blob/8a77fab7b9626805246005549a15d00661fcf5fa/docs/docs-graphql/schema.graphqls#L6228-L6240
which did the worktree automation stuff

so: new commits yes, new refs yes-ish (you can checkout branches, transparently handled as worktrees), creating tags no

[tepid nova]

How would that plug into the current "prompt user at the end, apply or discard" UX of dagger generate?

Would it make something like that available even for functions that don't return a changeset? Like a sort of generalization?

Or, would it bypass it, and unilaterally apply?

[wild zephyr]

@tepid nova @still garnet IIRC there's a reason why we can't entirely remove the remote dang SDK from the dagger repo, correct?

[still garnet]

if https://github.com/dagger/dagger/commit/ba9bf37f6903709f3064ee6541b67510bce5e0e4 / https://github.com/dagger/dagger/pull/13007 is in this release, we can remove it now

[wild zephyr]

sweet. Yep, this is going out in v0.20.7

[tepid nova]

@still garnet fun fact: it is in fact already the case, that a function returning a changeset can add commits and tags to the workspace, gated by changeset apply prompt 🙂 You just make changes to .git and voila

(not sure how that will go with worktrees, but that's the case across the board anyway)

[still garnet]

oops forgot to reply - in my case, since it was for autonomous agents already in their own worktrees, the goal was to freely apply changes without any confirmation

[still garnet]

makes sense - but it might mean you need to sync a hefty .git directory in for it to be modifiable. maybe it can be shallow? and yeah, not sure how things would be affected by worktrees

[tepid nova]

I'll keep digging, and use our release module as the guinea pig (in a separate branch for now of course)

The special case I'm trying to clean up, is the "pre-publish generate" -> specifically the changie batch that you have to run manually just before creating the release tag.

• Technically it's a generator function (outputs changes to be checked in git). But, it can't be called by generate because it's not idempotent: you have to call it just before setting the release tag

• It also can't be bundled into a ship function (which at the moment is a raw dagger call), because those don't have an affordance for generating source files along the way (nor do we want them to)

So it's a weird hybrid. My thinking is that really it's a combined tag-generation function. Then it can be just another "bump" function - idempotent for a given target version. I just need a way to produce a tag + commit in a single function 🙂

Even if we remove changie and replace it with a LLM-powered changelog generator (and I don't see how we don't do that soon), I think we'll still need something like this, since a raw LLM call is not idempotent either... You don't want to re-run a LLM summarizer on your git history every time you dagger generate

[charred lotus]

Need a review to fix CI: https://github.com/dagger/dagger/pull/13075

[tepid nova]

Pretty happy with this: https://github.com/dagger/dagger/pull/13062

[coral vector]

@tepid nova sorry to bother, but I saw you put 👀 on this and potentially forgot to comment?

[still garnet]

Sharing some 'better tests' UI progress (TUI + web UI) - feedback welcome! The TUI is very quick-and-dirty, LLM one-shot'd, planning to refine further but comment away if something jumps out. One difference is the TUI scopes tests beneath checks, showing tests beneath each check with T to zoom in to them. The web UI just has a single Tests UI that ignores checks and shows all the tests it found across everything, since I started in Cloud which is currently still one-trace-per-check for performance reasons.

[tepid nova]

I think we accidentally found one explanation to weirdly frequent invalidation of module loading cache @civic yacht

[still garnet]

@civic yacht I had to revert this dagui change from Theseus since it was marking way too many things as failed: https://github.com/dagger/dagger/pull/13073/changes/5be82dbad59c51ea1c46926aed4805aea1757781

The problem it fixed was real though, basically we have no way of precisely attributing failure from an effect back to to the API that 'installed' the effect: https://dagger.cloud/dagger/traces/cc312677aad19cd28186d57081a0de12?debug=&listen=58ea3947b67bedee&listen=d38d304c7a9f0304&listen=908b7f3d16307e71&listen=c955b9d515257074&listen=ac255cbc5bd7ba6c&listen=c110545830859704&span=b691bafe394becb0#908b7f3d16307e71

• PythonSdk.codegen returned a lazy GeneratedCode - if you fudge with zoom levels you can see a teeny tiny gap from where it returned early (attached)
• uv lock eventually ran and failed (as expected by the test)
• But as far as the UI is concerned, the outermost calls succeeded, there are no errors bubbled up, so there's no way for it to surface that error from deep within the trace especially since it
• => Test asserts on error output that is never shown

The fix I reverted essentially said 'any time a child span finishes and fails after a parent span completed, bubble the failure up' but that's not precise enough, and was causing expected errors in tests to bubble up to tests that actually passed and make them look failed, because the tests themselves are async (test span ends early when t.Parallel() blocks and resumes with a later span when Go unblocks it)

Previously this worked by having every API call return a span attribute that included all the effect IDs that were newly installed into the return value, and when the UI saw a failed span with an effect ID it would propagate the failure back to the install site(s). This was designed around Buildkit's constraints so it makes sense it wouldn't translate easily to Theseus, but seems like we need a new approach that's more precise at least

(sorry for double ping, posted in wrong channel)

[still garnet]

Not sure what the status of investigating missing shared result errors is, but here's an occurrence, if it helps: https://dagger.cloud/dagger/checks/github.com/dagger/dagger@7c4221e968f8ccd51b9e41e9d104af3c5505525b?check=test-split%3Atest-base&listen=eab73824c1ff0875&listen=92fff003a50dee7e&listen=9f3f1d257f73bba1&listen=df1b5f14724d39c7&listen=63e8e7dbcf89b43f&listen=f6776628e1ffc766&run=31edf7cd-f30c-4ac6-82af-45ca9eb4b2c8#63e8e7dbcf89b43f

I was able to repro this pretty easily on my interfaces branch - it seems to happen when something fails (in my case, codegen) and you run the command again. I would reliably hit that error until doing a dagger core engine local-cache prune.

Repro is ./bin/dagger --progress=logs -vvv -m ./core/integration/testdata/modules/go/ifaces/impl/ develop on interfaces branch @ cf83871cce3725eb6e7a8cc064ffa35814e76c03

[tepid nova]

Sorry for missing. I had the tab open but didn't know what to answer. @astral zealot @wild zephyr will know best

[still garnet]

Anyone else able to hit an engine crash from dagger generate in dagger/dagger? I have an engine.log but it's 54 million lines and increasing fast by the time I stopped the engine. At the start of a backtrace is runtime: program exceeds 10000-thread limit - not sure if the remaining lines are all stack or what

[tepid nova]

I haven't hit that. BUT, I did recently add new generate functions (written in dang) so maybe that triggers something?

[civic yacht]

Not sure what the status of

[obsidian rover]

i do since 0.20.4, but not with this error --> generate / checks getting to 20+go of ram

[civic yacht]

that's when using latest released dagger? as opposed to ./hack/with-dev dagger ?

[still garnet]

yeah, with v0.20.6 - likely not theseus related, guessing it's a massive diff on my branch (interfaces); trying to scope it down to confirm

huh, ok the diff I thought might be large is actually pretty small - 9 files changed, +165 -162 lines. so there goes that theory, unless there's a massive diff hiding somewhere else. will keep trying other generators individually, and finally all at once (expecting noop) to see if it still happens

[obsidian rover]

last time i had cargo builds (1+gb), that the checks was loading

[civic yacht]

PR for some misc cleanup stuff I've noticed so far after getting back https://github.com/dagger/dagger/pull/13072, mostly deletion, all low risk

[charred lotus]

@still garnet i think you're hitting the issues i've seen

[tepid nova]

@civic yacht as part of CI cleanup & dogfooding, I'm trying to understand our ebpf generation/build stuff (across generate and build). There's some special case logic that I'm trying to figure out how to fit in a general-purpose module. Do you have the context written down anywhere maybe? Like merged PR or something like that

[tepid nova]

@tidal spire is there a known workaround for passing eg. a boolean value as customization? I think I remember it's string only for now - but maybe an escape hatch? 🙂

[tidal spire]

I don't think its only string, I would think a bool should just work. Haven't tried though!

[tepid nova]

Ah I just remember a follow-up PR from you related to this, that didn't get merged - so assumed it was only string. Will double check

(I'm using customizations heavily in my dogfooding streak in dagger/dagger, since it has a clear migration path in workspaces, no reason not to use it)

(for example it allowed me to nuke those bump functions and fold them into dagger generate which felt great)

[tidal spire]

gotcha! yeah I had a PR to add container addresses as customizations but I think standard types should work

[tepid nova]

Ah I see

[tepid nova]

🙋 git workspace API 🙂 https://github.com/dagger/dagger/pull/13074

[still garnet]

Quick gut check on a small feature I'd like to add: currently, if you run a withExec and any of its service bindings exits, there's no real visibility into that. Often the dependent command just hangs, e.g. when our engine crashes in integ tests and our clients start doing retry logic.

What if, when a bound service exits, we interrupt any execs that depend on it and bubble up the exit error from the service? Is there any case where we WOULDN'T want that? (I guess we could make it opt-in but would be nice to avoid if it's universal)

[civic yacht]

SGTM 🚢

[coral vector]

does 0.20.7 incl Theseus?

[civic yacht]

No, 0.21.0 will

[coral vector]

hype

[coral vector]

my only thought is: what happens if a user is using a Terminal that has a Service bound to it that crashes?

[still garnet]

ha good point

[coral vector]

I think it's a great improvement for regular withExecs, just wanted to make sure that you could distinguish

[still garnet]

yeah, if that's unique to Terminal maybe we keep it the default behavior for withExec and just handle it there 🤔

[leaden glade]

A nice idea while working on fixing the python AST:

A dagger develop --explain-schema flag that prints, per function, "I resolved src: Optional[Annotated[Directory, DefaultPath('.')]] as Directory (optional, default-path='.') because I followed alias X". Single biggest debuggability lever; cheap to write because the resolver already has the data.
Maybe that should be something else, especially with the next develop -> generate migration. But something like a dagger explain or anything else, could be interesting. Or maybe that could even be inside the typedefs part, like we have the source map, we could have the "source".
It's probably very niche, but I remember to have read a lot the generated Go entrypoint and for instance the Java SDK contains a test that is a big java module file and the corresponding entrypoint as a way to check types are read correctly.

[still garnet]

Has anyone else noticed go SDK: load runtime taking an extremely long time (multiple minutes, repeatedly in a single trace) with v0.20.7?

Wondering if this go get -u should be changed to just go get, but that's a total blind guess, haven't confirmed that command is even what takes a long time, I just remember -u doing a bunch of stuff at one point: https://dagger.cloud/dagger/traces/6e2cbf5b0444ad1ce96ea61ffd26bf46?listen=3497c038e4db6090&listen=3ca3ac61e6585722&listen=f94fa764b55b6182&listen=cf274824ee8156fd#cf274824ee8156fd:L16

[leaden glade]

Yes, 3, 4 minutes each time it runs.
I thought it was only on my machine 😕

[tepid nova]

@still garnet do you have a good example of scaffolding for emitting custom spans in a custom helper tool? (it's a go program bundled in a dagger module, to be exec-ed inside a container with nesting on). The LLM is struggling and seems to be adding more boilerplate than necessary - but I don't know for sure

[leaden glade]

🤔
Now it's failing differently

2026/05/05 21:07:02 INFO running post-command: args="go get -u dagger.io/dagger@7058e9313c720d82c6a07fefb6ce3fab60c7ec4e"
go: dagger.io/dagger@7058e9313c720d82c6a07fefb6ce3fab60c7ec4e: invalid version: git ls-remote -q --end-of-options origin in /go/pkg/mod/cache/vcs/39ea6402b8c7bc990458d81a19729d01208213cc6618163477756df4db9541d0: exit status 128:
        Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
        [e-mail]: Permission denied (publickey).
        fatal: Could not read from remote repository.
        
        Please make sure you have the correct access rights
        and the repository exists.
2026/05/05 21:07:04 ERROR post-command failed error="exit status 1"
Error: exit status 1

On local and --cloud
Something's wrong on the dagger.io proxy? (or on github side?)

[still garnet]

cc @astral zealot in case there's something easy to check here - seeing repeated multi-minute slowdowns on go get dagger.io/dagger@...

seems like it might be resolved now? cc @leaden glade if you wanna check

[leaden glade]

It seems better now 👍

[charred lotus]

@still garnet https://github.com/dagger/dagger/pull/13098 is green

[still garnet]

nice - ✅

[tepid nova]

🙋 this one would be nice to merge https://github.com/dagger/dagger/pull/13053

It fixes a glitch in Workspace.directory() that could become problematic later if we don't fix early

[still garnet]

hmm, i'm seeing this error pop up still - https://dagger.cloud/dagger/checks/github.com/dagger/dagger@1f701c41c9dc2606db9bec1d03a191a4fe4edb79?check=test-split:test-module-runtimes#03cfc89f9c583d61

will investigate tomorrow, seems like some kind of nondeterminism

[leaden glade]

PR #13095 is ready to be reviewed.
As it's not a small one I spent quite some time to reorganize the different commits so they could be easier to follow and review one by one.
The PR body contains a high level description of those commits.
Happy to help review it if anyone wants, or we can pair.
TL;DR is it solves multiple issues with the python AST analyzer added recently (and in v0.20.7) and ensures with way more tests (including random property based testing and tests against real python modules) that this is working as expected.

[still garnet]

follow-up: https://github.com/dagger/dagger/pull/13103

[wild zephyr]

any reason not to add the -W flag at the same level of -m? It's currently only available for the session command. cc @still garnet

[still garnet]

that seems surprising, but i lack context there (cc @tepid nova)

[wild zephyr]

flag was added here: https://github.com/dagger/dagger/commit/887046c1fb89ff2559abf2f820a313d3e4590715

[charred lotus]

Isn't it being added in the workspace PR ? /cc @obsidian rover

[tepid nova]

That's because we were rushing a fix for the CI issues 2 weeks ago. Made it the smallest possible patch to fast track. Was only needed for Cloud->go sdk->dagger session

in workspace branch we drop that and make -W a visible flag in (almost) all commands

[civic yacht]

@obsidian rover got back on the comment here https://github.com/dagger/dagger/pull/13087

[obsidian rover]

taking a new look 👀

[charred lotus]

FYI i broke CI, im fixing it

[tepid nova]

@still garnet modulesv2 design docs: https://github.com/dagger/dagger/pull/12900

Most relevant to test filtering, test splitting, etc: Collections: https://github.com/dagger/dagger/blob/modules-v2/hack/designs/modules-v2/collections.md

(Spec should all be accurate, but it does have some slop, sorry)

Maybe have an agent read and summarize it so you can skip the slop part 😛

[tepid nova]

@still garnet off the top of your head, is modules/doug supposed to be extempted from go linting?

[still garnet]

the only reason I could imagine is periods of time where its API usage has strayed ahead of the engine - shouldn't need to be exempt now at least

[tepid nova]

OK weird, must be something in the switch to github.com/dagger/go... I just can't figure out what. It says "no go files - try running go mod tidy". There actually "is" a main.go and go.mod right there, but indeed running go mod tidy makes the error go away (then there's a different, actual lint error)

I think it's a missing include filter in dagger/go

Those famously cause super weird errors

[charred lotus]

v0.20.8 is out! CI seems already much better

[civic yacht]

race detector went off during a test run, teeny test only fix here https://github.com/dagger/dagger/pull/13109

[still garnet]

yeah WAY better 🥲 - for the past few pushes to interfaces all the checks have passed the first time. nice job everyone!

[charred lotus]

Yeah i was quite happy today to see more green CIs ^^

Thanks for fixing it Alex!

[tepid nova]

https://giphy.com/gifs/thesinner-VdK7rrrW6q68qjAQmD

Digging ourselves our of the flakey CI hole 😄

[still garnet]

@civic yacht Hitting some missing shared result errors on my interfaces PR still, only in test-module-runtimes: https://dagger.cloud/dagger/checks/github.com/dagger/dagger@9df52cfa0e09a676091039b5ad45c355d11e8704?check=test-split:test-module-runtimes&listen=2b3324d4d3e5dad7&listen=9e318aa99cff57ff&test=TestElixir/TestEnumArg/can_use_enum&viewMode=tests#9e318aa99cff57ff:L30

Full logs can be downloaded from here: https://dagger.cloud/dagger/checks/github.com/dagger/dagger@9df52cfa0e09a676091039b5ad45c355d11e8704?check=test-split:test-module-runtimes&viewMode=logs

I'll see if I can whittle it down to a tighter repro, but FWIW it happened 3 runs in a row, even when using the 'run from clean slate' option. I had some solid CI runs for a while, before this one check started failing, not sure if something changed out of band, have a hard time imagining the recent commits caused it

[still garnet]

missing shared result

[still garnet]

Just finished my self-review on the interfaces PR and cleaned things up along the way, it's ready if anyone has bandwidth to give it a look! 🙏

https://github.com/dagger/dagger/pull/12041#pullrequestreview-4229920223

[still garnet]

vito catching up on modules v2

[charred lotus]

@civic yacht I'll see if i can repro, but just FYI in case you see it in the future, i had an unlinkat error during generate: https://dagger.cloud/dagger/traces/4c7fdff3bca75d9145d8d857dd3a36f2

[charred lotus]

I'll get to it after i upgraded CI to main

Dagger Cloud

[charred lotus]

PTAL: https://github.com/dagger/dagger/pull/13049, needed to update dagger CI to main

[tepid nova]

Does it remain stable for all main versions, even as the tip of main keeps moving?

[charred lotus]

It does bring up the question: is it expected that between different dagger versions dagger check generated differs in result ? (especially when we want dagger check to automatically check for diff of generated)

[tepid nova]

This topic always hurts my head, but I feel like it should change with the target engine version configured by the module. Not the actual version of the engine running the generate function

If the module says "I need engine version 0.20.8", then I expect to get bindings that will work when running in engine 0.20.8. The engine version actually running the generator for that should be irrelevant

[charred lotus]

That makes sense

[charred lotus]

Opened issue https://github.com/dagger/dagger/issues/13115 to keep track

[tidal spire]

would love to get a review on this one for introspecting enum type defaults in the schema https://github.com/dagger/dagger/pull/13068

[charred lotus]

@civic yacht CI is running main (commit 0e9fd00bbc30) for dagger org

@Erik Sipsma CI is running main (

[civic yacht]

quick PR to disable git autoclean during changset operations: https://github.com/dagger/dagger/pull/13121 which has a decent chance of explaining some of the flakes we see where changeset operations fail with unlinkat errors occasionally in CI

[surreal wolf]

Hey guys I'm Paul, I found your repo on github and it seemed like an cool tool. I read the docs, played around with the cli, and forked the repo. I made my first pr yesterday fixing a skipped test in the go sdk. I really want experience working in a large codebase and figured Dagger would be a great place to start, any good first issues I can work on?

[leaden glade]

Just to be sure to capture that right. So this means each time we are adding a new entry in the schema, we need to add the version in which this has been added so the schema we will send to the SDKs will be able to filter them accordingly. Am I right? I think the mechanism already exists for that, no?

[coral vector]

Hey! Solomon posted this recently, citing it as an easy contribution: https://github.com/dagger/dagger/issues/13059

Make sure to follow contributing guidelines: https://github.com/dagger/dagger/blob/main/CONTRIBUTING.md

[surreal wolf]

Thanks! I'll take a look at that today

[tidal spire]

migrated the following modules from defaultPath to Workspace:

• https://github.com/dagger/vitest/pull/9
• https://github.com/dagger/eslint/pull/6
• https://github.com/dagger/prettier/pull/7
• https://github.com/dagger/jest/pull/20

any other dang modules that need converted you can just point your agent at one of those diffs

for these kinds of repos since they're strictly a module and the checks they provide aren't meant to pass on themselves, what if I could specify an alternative -W in cloud? Its kind of the inverse of what we talked about earlier @tepid nova @wild zephyr
so instead of: dagger -W=. -m=<installed modules> check
i could do dagger -W=<user defined git ref> -m=. check. This is what I do to test changes locally using github.com/kpenfound/demo-react-app as the -W

as an escape hatch we can do this with the repo's own "dev" module in .dagger/modules/dev since I think once we upgrade native CI to use Workspaces then we just wont see any checks in those projects at the moment right?

[tepid nova]

So for this problem (which btw was one of the problems that caused the whole modulesV2 project in the first place...) -> the crux of the issue is so-called "CWD module". A special step of module loading, separate from either a) modules in workspace config, or b) explicit modules via -m - there is also c) if cwd is inside a dagger.json, load that

The motivation is backwards compat. The downside is that it prevents the "module shouldn't check itself" problem from being completely solved. There's a delicate balance

@civic yacht flashback to our group discussion at the Japanese Tea Garden in SF. Do you see any future where dagger.json is no longer the source of truth for a module's runtime dependencies? With each SDK instead handling those dependencies in any way it sees fit; and the engine not having a guarantee that it can statically inspect a module's true and complete dependency graph - the only way to know is to run the module, and see what it dynamically loads.

Context: as part of workspace we are changing the UX for commands that manage said dependencies in dagger.json. We have the option to keep it centralized across all SDKs, or to delegate it to each SDK (which opens the possibility of drift). If we think there's a chance that in the future SDKs will manage their respective module's runtime dependencies differently, the delegation path makes more sense, even if in the near future each SDK will continue to manage dagger.json in the same way.

Even more context: there are 3 parts to the commands we need to redesign:

1. Module init --> this we know we want to delegate to each SDK, since by definition each generated boilerplate is SDK-specific anyway
2. Module generation --> this is the cleanest, dagger generate is a perfect fit
3. Module dependency management (including engine dependency) --> this is the tricky one in discussion here

[civic yacht]

The idea was mainly that dependencies are native dependencies in the SDK's package manager (so npm deps, go deps, uv deps, etc.). You could thus still statically figure out what the deps are without running the module. But yes, in that world they would not be managed in dagger.json most likely. All hypothetical though

[tepid nova]

Ah nice. Yes I see, static analysis would still be possible, just would require SDK participation I guess

It kind of opens the pandora box of asking: if the dependencies field in dagger.json could become not load-bearing in the future: should any field remain load-bearing? Do we need dagger.json at all, or should each SDK be responsible for even that? 🤔 But that part breaks my brain 😅

I guess the absolute irreductible core of dagger.json is a runtime entrypoitn: "hey dagger engine, load me with this runtime X"

[charred lotus]

is there a way to discard generate output in a non interactive way ?

[leaden glade]

Like the opposite of the -y? We can probably add the contrary flag, but it feels weird, why to generate if it's to discard? (asking because I don't know your use case 🙂 )

[charred lotus]

Testing dagger 😄

[leaden glade]

Yeah, of course 😅
but what would be the use case for discarding in a non interactive way? By non interactive you mean without the user to use the terminal to answer, right?

[charred lotus]

it's okay, i ended up doing -y and then discard it with git

[still garnet]

I saw this happen yesterday - the agent ran test-split test-telemetry and then it just hanged forever, my guess is because that func returns a Changeset it must have been prompting on something

[leaden glade]

Maybe if it's not interactif and there's no -y we should just discard it and just print a warning.

[still garnet]

Yeah probably (like "run again with -y to apply"), I'm not actually sure how to detect non-interactive from an agent running though. Needs some investigation 🤔

[still garnet]

huh. did BitBucket just nuke our test account/repo?
https://dagger.cloud/dagger/checks/github.com/dagger/dagger@6058701db372814793aa6324b2ddaf97bff3eb7b?check=test-split:test-call-and-shell&listen=899bf3f808fdfc65&test=TestCall/TestArgTypes/BitBucket_public/module_args&viewMode=tests#899bf3f808fdfc65:L33
https://bitbucket.org/dagger-modules => 404

[still garnet]

Bring out your jank! Doing a round of TUI cleanup - so far, fixed: togglers showing up on final render, lag between expanding and moving focus. Anything else y'all have noticed?

https://tenor.com/u1BBEu5fzGS.gif

[tepid nova]

Any objections to merging this? 🙏 https://github.com/dagger/dagger/pull/13074

It's pretty lightweight for now - there's a more beefy followup here: https://github.com/dagger/dagger/issues/13086

[wild zephyr]

@leaden glade before you leave, give that we spoke about releasing theseus without the workspace work, shall we merge https://github.com/dagger/dagger/pull/13095 as soon as we can so that also goes in the release?

[tepid nova]

@civic yacht can I access the engine's builtin SDK module sources via the core API?

[wild zephyr]

as you know, it's preventing some python users from using the current theseus main branch

[stuck bloom]

Hey Vito, I've been meaning to ask. Would it be possible to have a keybinding to go to the currently executing node? In big TUI ouputs, I find it hard to pinpoint what's currently running. It's not always what's at the bottom. Something like a "focus" button.

[still garnet]

hmm makes sense - maybe a key to tab through running items? sometimes I press - to trim out non-running items and jump around

[stuck bloom]

Yes, but sometimes even with 0 verbosity, I have trouble finding the running node.

[leaden glade]

If we release main, that should be in or the ast reverted. I haven't implemented a flag to disable it 😕
All the tests I've made in python pass, but 🤞

[civic yacht]

Bring out your jank! Doing a round of

[wild zephyr]

Let's merge unless there's a speific reason not to 🙏

[wild zephyr]

cc @fair ermine you good with this also?

[wild zephyr]

I'll be merging this to unblock @sly osprey

[obsidian rover]

has there been a regression on this check golang:check ?

[civic yacht]

Yes I noticed it last night, it appeared out of band which is quite odd, but I have a fix for it here https://github.com/dagger/dagger/pull/13132

[still garnet]

approved, but bleh that fix caused the generated files to stray due to sourcemap line numbers changing, so the generated check is failing now

[civic yacht]

Yes I am re-running generate rn, for ✅

[civic yacht]

next up I'm also gonna get to the bottom of why cli:release-dry-run and engine-dev:release-dry-run randomly start flaking as a pair... that one is getting extremely annoying too

[civic yacht]

next up I'm also gonna get to the bottom

[tepid nova]

@tidal spire for e2e/testcontainers pattern. Imagine if we could add a directive like this in the test code:

// go:test:sandbox <container image reference goes here>
func TestFooBar(ctx context.Context, t *testing.T) {
  ///
}

[tidal spire]

image reference as in like a service dependency postgres:latest? or runtime?

[tepid nova]

More like: "please run this test in that container"

Or even more ambitious:

// go:test:system-dependencies git protoc
func TestFooBar(ctx context.Context, t *testing.T) {
  ///
}

--> Then the module just figures out a way to make those system dependencies available

[tidal spire]

sounds cool! are you worried yet about encoding too much in comments?

[tepid nova]

Not really, I'm only doing it when I feel the need for it

So far so good (but dogfooding is still WIP)

[tidal spire]

nice!

this is in the dagger/go module?

[tepid nova]

My 2 sources of friction right now:

1. Dang development with LLM (they overcomplicate things, and I'm not comfortable enough with the syntax to do it myself, so it gets frustrating)

2. At the moment my module doesn't cache well, I think because I weave workspace absolutely everywhere (necessary for dynamic discovery of includes) so everything gets invalidated

[tidal spire]

What have you tried for dang +LLM so far? I've tried hand rolled skills and pointing at examples but the most success so far has come from literally just pointing at the dang repo

[tepid nova]

yeah syntax is not an issue - it's just that it seems to always pick the most complicated possible design - just tons and tons of useless indirections etc

[tidal spire]

ah got it

yeah i've definitely had that issue with dagger modules in any language. I think there's a particular style that the LLMs can get away with in the dagger codebase that just sticks out in a module

[tepid nova]

I added a "dang developer manual" in github.com/dagger/go to try and steer

Mostly I talk about "inlining useless indirections" and things like that

Anything they write I just ask them "can you spot any useless code that could be inlined?" -> then ask it 5 times in a row

[civic yacht]

fix for the module types missing shared result issue is good to go now, got it simplified down https://github.com/dagger/dagger/pull/13119

fix to restore full "in-flight request" deduplication is good too https://github.com/dagger/dagger/pull/13118

Both of those are fairly small and also seem to be doing very well on many re-runs, so would be good to get them in to greenify CI further

If there's stuff for me to review too lemme know, especially anything we want for the next release

[tepid nova]

@civic yacht I'm noticing that dagger generate in several different branches spits out diffs like:

-// The `ClientFilesyncMirrorID` scalar type represents an identifier for an object of type ClientFilesyncMirror.
-type ClientFilesyncMirrorID string

-// The `HTTPStateID` scalar type represents an identifier for an object of type HTTPState.
-type HTTPStateID string

Is that a known issue, eg. we merged something with an incomplete regeneration, and now every branch is catching it in their diff?

(it stands out because those branches are relatively scoped and small, ie not workspace 🙂 )

[civic yacht]

no CI is green on main. I did notice earlier today that I had to run dagger generate with the dev engine though

[tepid nova]

I was wondering if I was running main, but dagger version returns 0.20.8

[civic yacht]

Yeah I guess the problem is that in CI we're using main, so then the generated check is looking at the schema from main. So yes you'd want to run dagger generated with main too. @charred lotus brought this up here https://github.com/dagger/dagger/issues/13115

[tepid nova]

oh yeah... that would be an issue

[charred lotus]

Separately from that, i also want to see if we can make the "sourcemap" optional

[tepid nova]

We had another thread about that, but IMO it is an anti-pattern that the version of engine running your generate, changes what you generate

[charred lotus]

I think the issue is talking about the same thing