#Workspaces aka "modules v2"

1 messages Β· Page 1 of 1 (latest)

coral kernel
#

Lets get the prototype going πŸ˜›

Overall I agree with the whole proposal. First questions I have are

  • What would the config.toml look like with some customizations?
  • Why .dagger/config.toml instead of dagger.toml or dagger_workspace.toml?
  • Nested workspaces shadow the parent directories, does that mean they inherit anything those directories add? including toolchains?
dim smelt
# coral kernel Lets get the prototype going πŸ˜› Overall I agree with the whole proposal. First ...
  • What would the config.toml look like with some customizations?

tbd πŸ™‚ We should combine customizations & user defaults, from the user's point of view the distinction is arbitrary.

  • Why .dagger/config.toml instead of dagger.toml or dagger_workspace.toml?

dagger.toml feels too close to dagger.json

dagger_workspace.toml feels too long

.dagger/config.toml feels kind of like .github or other existing config formats. Plus it has "config" in the name πŸ™‚

.dagger.toml is another good option.

  • Nested workspaces shadow the parent directories, does that mean they inherit anything those directories add? including toolchains?

You mean modules πŸ˜›

Good questions, I would say no by default but configurable.

lofty shadow
#

LGTM, ship-it

In terms of the meat of the proposal I'm onboard. Agree with Kyle that the sooner we have a prototype the better

  • I technically have hang-ups on why we couldn't just have a ⁨dagger.toml⁩ with different sections indicating whether the current thing is a workspace or a module, dropping ⁨dagger.json⁩ entirely. But I don't actually care very much. Don't block anything on that.
dim smelt
harsh imp
#

I think this LGTM. Does this fully get rid of terms like toolchain and blueprint etc? Looks like it does (and I like it), just wanted to confirm.

Would an analogy for monorepos be nx workspace? I am not an nx expert but it's hugely popular in my company.

dim smelt
harsh imp
#

I was under the impression toolchains replaced blueprints πŸ™‚

dim smelt
#

In the current design they are complementary:

  • Toolchains are for no-code configuration of your project. End user chooses which toolchains to install based on their needs. Reduces the need to develop their own dagger module from day one. This is a mainstream feature - every user is expected to install at least one toolchain on initial setup.

  • Blueprints are for centralized configuration of multiple projects. Unlike toolchains, a blueprint is monolithic: it configures your whole project, take it or leave it. This is a niche feature: centralized platform teams in charge of daggerizing many projects with a cookie-cutter stack, can use blueprints.

--> Both features are definitely disrupted by this new design proposal, for sure

topaz seal
#

I think I like it. The disctinction between workspace and module seems to solve a lot of the recent confusion between modules, checks, toolchain, etc.
That seems nice, and I like the "no init" part, just dagger install and voilΓ .
Regarding dagger dependency add I still wonder if we shouldn't put that under dagger develop. Like dagger develop add-dependency .... So we nest everything regarding developing modules under one place, but it might be a bit long to type/non intuitive. The advantage to put what under develop is for "end users", the ones interested in workspace and not developing modules, there's one single sub command to ignore, and develop seems a good name.

Thinking while typing, should we also move dagger init under develop? dagger develop init to say we are explicitly working on creating a module.

harsh imp
#

Blueprints are for centralized configuration of multiple projects. Unlike toolchains, a blueprint is monolithic: it configures your whole project, take it or leave it. This is a niche feature: centralized platform teams in charge of daggerizing many projects with a cookie-cutter stack, can use blueprints.
Functionally they are the same, aren't they? Sure, toolchains recently gained some customization options, but, I could create a monolithic toolchain and that would be exactly like a blueprint right?

dim smelt
dim smelt
#

Design know: Part 1 - Module vs. Workspace

coral kernel
#

I've been thinking about this and I think the only way to sequence this without bricking our own CI is to

  • build the ability to read a config.toml to load toolchains and checks as we do today
  • release it
  • migrate our toolchains and customizations to the config.toml
  • refactor that into Workspace + the rest of this proposal
dim smelt
#

I'm updating the gist to add details, and get us ready for implementation

#

Countdown to bikeshed

#

Terminology check: custom modules vs reusable modules?

dim smelt
#

Another option: project modules vs reusable modules

#

Any preference @brittle wharf ?

#

custom module, or project module?

brittle wharf
#

project feels more precise (technically any user-written module could be considered a "custom module")

dim smelt
#

you sound like Claude lol

brittle wharf
#

except my intelligence is not artificial bigbrain

dim smelt
coral kernel
#

Looks great, I think it captures everything we talked about

If no workspace exists yet: If in a git repository, creates .dagger/config.toml at the repo root

Is there a way to explicitly create a workspace other than install? Would I manually create a .dagger/config.toml? Otherwise there's no way to create nested workspaces

#

Find workspace: Walk up from current directory looking for .dagger/config.toml. If not found, workspace is empty (no modules installed).

Child .dagger/config.toml shadows parent

Worth mentioning that to use the child workspace you should run dagger from that child directory

#

Find workspace: Walk up from current directory looking for .dagger/config.toml. If not found, workspace is empty (no modules installed).

  • some mention of the --no-workspace flag
dim smelt
#

@coral kernel @brittle wharf I am still unsure what to call the directory with a .dagger dir

Is it a workspace? But then that contradicts "the repo is the workspace"

coral kernel
#

dagger workspace config?

dim smelt
#

I worry about a scenario where we say that it is not a workspace, but in practice everyone calls it a workspace anyway, causing confusion for everybody

brittle wharf
#

project?

#

workspace has_many project?

dim smelt
#

alternative: that is the workspace, which means there can be multiple workspaces per repo, and we find a name for the "boundary"

#

aaand a bird just shat on my head

restive swift
brittle wharf
#

so workspace always == entire repo, ⁨⁨.dagger/⁩⁩ denotes a project, often they're 1:1 (in small/non-monorepo case)

dim smelt
#

I think workspace & project is too many words

#

people won't remember which is which

coral kernel
#

Is it a workspace? But then that contradicts "the repo is the workspace"

Isn't it "the repo is the workspace unless you have a .dagger/config.toml that says what the workspace really is?"

proven viper
#

What about "project modules" ?

brittle wharf
#

yea i thought we already used the word πŸ˜›

#

or are we second guessing that

dim smelt
#

I don't think one automatically leads to the other

#

my other choices were "bespoke module", "custom module"

brittle wharf
#

artisinal

dim smelt
#

i can say "a module specific to your project" without formally defining what is or isn't a project

lofty shadow
dim smelt
#

ok different angle:

  • .dagger marks a workspace
  • workspace config can be a local path: must always be relative to the config file
  • workspace API: paths are rooted in workspace . If you need other files in the repo from outside the workspace, that's a workspace config we can add, to "mount" external files in the workspace
  • For 90% of installs, the workspace is the repo
coral kernel
#

If you need other files in the repo from outside the workspace, that's a workspace config we can add, to "mount" external files in the workspace

Something like include = ["../../package.json"]? Would that mean my Workspace source is a Directory where my workdir has some parent dirs?

dim smelt
#

with a separate workspace config it becomes quite natural

dim smelt
proven viper
#

ok i prefer that. It's a bit like go replace

dim smelt
#

yeah previously there was nowhere to put such a config without compromising the portability of modules. No such worry with workspace config

coral kernel
#

seems fine for now I guess. I anticipate we'll have situations pretty quickly where that won't work but we can deal with it when it comes up

dim smelt
restive swift
#

Regarding the configuration reference: we naturally ended up adding pinnedRef in the dagger.json. In above spec, go = "github.com/dagger/go-toolchain@v1.0", where would it live ?

dim smelt
#

another benefit of this "replace" approach is that you could even go beyond the repo, and alias files from your local system (assuming you trust the workspace)

dim smelt
#

(as in now)

#

will focus on that as soon as bird shit is removed from head πŸ˜…

proven viper
dim smelt
#

and workspaces don't aspire to be portable

#

Well not 100% true

proven viper
#

i still think it's nice for a workspace to be portable

dim smelt
#

yes agreed. but we already want things like 1passwords urls in there - with user defaults

proven viper
#

But because it's explicit we can warn the user about escaping the portability/security of the workspace

#

Yeah although i distinguish between ressources you are allowed/denied access, and completely swapping the ressource to something else (more of a security risk)

dim smelt
coral kernel
#

scenario:
I'm in a subproject of a monorepo. I need to include a few repo-wide configs for my tools to use, such as a .prettierignore. Thats required for my prettier module's dynamic filtering. If I include = ["../../../.prettierignore"], the content of that file still expects to be at a specific relative path to my subproject.
If its a named mount, my resusable toolchain will have no idea where to put it.

dim smelt
dim smelt
coral kernel
#

thats what I was thinking of when i asked this #1468070450524459029 message
like if everything is in a directory at the correct relative path and we set the workdir based on the workspace's location, its all good

#

maybe its more complicated and doesn't actually help but maybe we could include other workspaces instead of files

brittle wharf
#

it would be a problem in ⁨⁨dagger.io⁩⁩ too because we have ⁨⁨go.mod⁩⁩ replace rules that point to things like ⁨⁨../api/⁩⁩ and ⁨⁨../daggerverse/⁩⁩ - so those always have to be at the same relative location

proven viper
# dim smelt can you give an example of the difference ?

sure.

  1. 1password urls if you dont have the right one, you just dont have access to the secrets and thus to the ressource.
  2. llvm = ../../llvm: it's not about allowing/denying access, it's about a potentially sensitive folder (because outside workspace) being given without warning to an untrusted module.
dim smelt
coral kernel
dim smelt
# dim smelt ok different angle: - .dagger marks a workspace - workspace config can be a loc...

ok new NEW angle:

  • .dagger marks a workspace
  • In workspace config (outside module sandbox), paths have the normal meaning: local paths are relative to the config file; absolute paths are relative to your local host filesystem.
  • NOTE: this means workspace config can reference files outside of the workspace root. Whether to allow this is a policy decision. Probably we want to deny it by default, with possible overrides if you trust the workspace and really know what you're doing. But in any case: no namespacing. / means / in the system

- In workspace API (inside module sandbox): we distinguish the workspace root (where .dagger lives) with the workspace's git context (the git repository inside which the workspace lives). The API lets you access one or the other nah then module dev doesn't know which one to use

  • In workspace API, 2 options:

OPTION 1: paths are rooted in the workspace root. If you want workspace to be rooted further up, you configure that in the workspace config. root="../..". Maybe a special variable to mean the git root, since that would be common: root="$gitroot"

OPTION 2: paths are always rooted in the git root... And you rely on filters. So a workspace is always a view of the repo

brittle wharf
#

what does ⁨/⁩ meaning host rootfs ⁨/⁩ buy you? seems like that would always be a footgun to me

coral kernel
#

option 2 sounds the most flexible but I still think workdir relative to git root is important to preserve

brittle wharf
#

⁨⁨⁨⁨/⁩⁩⁩⁩ always meaning git repo root i find pretty ergonomic, i would always prefer that to ⁨⁨⁨⁨../../../../⁩⁩⁩⁩ in those fringe scenarios (but even ⁨../⁩ tbh)

dim smelt
# brittle wharf what does ⁨`/`⁩ meaning host rootfs ⁨`/`⁩ buy you? seems like that would always ...

In the context of workspace config specifically: user doesn't know how modules work. As far as they know they're just configuring another devops tool. So it makes sense that all paths work in the familiar way.

Inside the module sandbox: different story. workspace.Directory(".") is the same as workspace.Directory("/") anyway [actually not always true] "path physics" are different, eg. Directory.directory("/") vs Directory.(".") etc

dim smelt
coral kernel
dim smelt
#

I initially liked the idea of giving the module access to the client's workdir. The more info the better right? But I wonder if it could be a footgun. For example now the module dev has to decide - what do I do with the workdir? Do I scan for all go.mod in the workdir, or in the whole workspace? What's the convention? etc

coral kernel
coral kernel
dim smelt
#

Where the client workdir is important is for filtering artifacts by path. For example dagger check --path=. which means "only check artifacts which trace back to my current workdir". That could even be the default... But the difference is that the module code doesn't know the workdir -> it's the engine doing the filtering

coral kernel
#

Its not about check discovery its about the context where the subproject exists. If my module works dynamically across the whole repo there's no use for child workspaces. But if I have a nextjs project in a child workspace at /foo/bar and a /.prettierignore, I'm expected to run prettier check from /foo/bar. And if I run dagger check from /foo/bar which has its own workspace, I don't want it to go find checks in other places

dim smelt
coral kernel
#

I dont think its about filtering though, its about intent. I want to run tools in /foo/run regardless of what else is included in the workspace, so the Workspace API must tell the module the workdir relative to the repo root

dim smelt
#

sorry I'm confused. We can discuss it live tomorrow maybe?

#

There's a lot of assumptions that seem abstract to me

coral kernel
#

yeah that works! I'm not sure how the filtering and Artifacts change the underlying problem with child workspaces

dim smelt
#

We'll have to start there - I don't know what the problem is with child workspaces. And also I spent zero time thinking about child workspaces so far (I think claude actually snuck in that term πŸ˜› )

coral kernel
#

maybe another example like @brittle wharf mentioned.
i'm in the repo dagger/dagger.io
working on cloud in /cloud. It has its own .dagger but we're in a subdir of the repo root
I have to include ../api because the go.mod has a replace that expects that.

So now my resulting workspace directory looks like
/
/api
/cloud <-- workdir

any checks or tools my checks might run should be run from /cloud in the workspace's resulting directory. I never need them to care that they could build or test /api

dim smelt
#

(That probably doesn't address your point though)

coral kernel
#

ok but from a module author's point of view. I have a directory, something like Workspace.directory. I mount it in a golang container at /app. When I run go test, how do I know I'm supposed to run it from /app/cloud unless the workspace tells me the workdir?

dim smelt
#

So IMO the module should focus on building the whole list of possible go modules to test. And let the engine filter based on user workdir, flags etc

coral kernel
#

Right so thats what I mean, we've then lost the user intent. I ran dagger check from /cloud which is its own workspace. I had to include /api as a side-effect of the go.mod's replace, but I'm not asking dagger to go run go test in /cloud AND /api, I'm only concerned with /cloud here

dim smelt
#

What I'm hoping is that user can use path filtering for that. dagger check --path=. for example

#

In this specific example, api and cloud are tightly coupled, so it makes sense that you can check both in the same workspace

#

But if you only want to check /api then you would just say that: dagger check --path=/api

coral kernel
#

I'm pretty concerned about losing the user intent and trying to be too smart though. That would be a worse experience than just running go test myself from /cloud because thats what i want to test

proven viper
#

(dont want to derail, i just like the idea of dagger check <pattern> and that pattern can be either a path pattern (starts with / or .), or an artifact pattern like go:*)

dim smelt
#

Separate topic: found a way to simplify module initialization based on feedback from @brittle wharf @proven viper @restive swift earlier today

#

pushing to the gist

dim smelt
#

But there might be more to check in /cloud than the go tests.

#

So dagger check with filtering is a nice way to aggregate

#

If you specifically want to run go test in one module, in a super narrow fast devloop, dagger can't really compete with that anyway

coral kernel
#

I mean if we were talking about a workspace at the repo root and we happen to want to run just the checks for /cloud, passing a filter makes sense. But if I'm in /cloud and it has its own workspace, it feels weird to not respect that

dim smelt
#

But if /cloud is tightly coupled to /api, I question that you should have a separate workspace in /cloud in the first place. It might not be the right use of a workspace

#

IMO it's more for organization split, rather than delineating the software stack

#

"I don't want to ask 50 people for permission to add .dagger in the whole monorepo. Easier for me to do it in /devrel/tiger-projects or whatever"

#

But I think we could make dagger check --path=. the default (ie. by default, only artifacts linked to your workdir are shown to you)

So dagger check -l by default would filter that way.

#

Also I was thinking that dagger check --git-uncommitted might be a good default...

So dagger check out of the box would only check artifacts that 1) are linked to your workdir, and 2) touched by your uncommitted git changes

#

that would make it more realistic for a devloop

#

then you add dagger check --watch and we're in business πŸ˜›

coral kernel
dim smelt
#

Or "disable checks for these paths" or something

#

I'm doing a pass on the toml config schema

#

@lofty shadow I'm lifting the dependency schema as-is and transposing it into the toml file. Any opinion on preserving the list format from dagger.json, vs. making it a map?

#

From a user readability point of view, a map looks nicer.

lofty shadow
coral kernel
dim smelt
#

Agreed

dim smelt
lofty shadow
#

If you are looking for an extra wrinkle in path handling: #1468216262558617775 message

About git usage
This is actually a very interesting question πŸ™‚
Git is not used at the repository root. We’re using a private VCS instead. However, in order to make module-to-module dependencies work, we had to git init an empty repository at the root. It’s not really used and doesn’t contain anything meaningful.

Basically, they don't use git but have to ⁨mkdir .git⁩ right now to get a context root. Feels potentially related to and possibly solved by all the above, just flagging

dim smelt
#

huh

coral kernel
#

really weird. I wonder if the findUp that looks for a .git could be configurable to look for some other marker? although configuring it would probably be more characters than just running mkdir .git

dim smelt
#

@coral kernel @brittle wharf looks like the first question for backwards compat is: should there be a way to configure a single "main module" in the workspace? So you could do either module= or modules= depending on the style you want?

#

And if we do: would we consider it a backwards compat thing only? Or a good and necessary convention that some people would continue using in their project?

coral kernel
#

my only idea for doing this in a backwards compat way was to have a config in the workspace like temporary_backwards_compat_main_module="/dagger.json". As ugly as possible to be clear that its temporary so people can migrate

dim smelt
#

One possible migration hook that could be pretty clean: we could deprecate the source field in dagger.jsn (since in the new model, a dagger module is a cleanly separated software package, the source directory can always be the module directory).

Then we can detect modules with a source field not set to ., and treat them as legacy "main modules", and from there, decide what to do with them

#

that would allow us to clearly separate our UX between "backwards compat" and "normal ongoing usage"

#

  • For pure modules: clean break. If you want to call your module's functions, you need to either A) install it in a dev workspace, or B) explicitly load it with dagger call -m .

  • For legacy "project modules": trigger backwards compat / migration flow (tbd)

coral kernel
dim smelt
#

Maybe we can auto-migrate

#

Then they just need to decide when they're ready to commit the diff πŸ™‚

#

Or, a new dagger migrate command

#

And if you're running on a new engine version, auto-call dagger migrate maybe?

coral kernel
#

so the immediate effect of upgrading to v0.20.0 is that dagger call is broken. But you can enable backwards compat mode in a new .dagger/config.toml until you have a chance to do the migration

#

a temporary dagger enable-backwards-compat could even create that config

dim smelt
#

@coral kernel do we have good examples of real-world repos using Dagger, other than our own? open-source I mean

#

neat detail: migration could be handled by a module πŸ™‚

coral kernel
dim smelt
#

You can do:

  • dagger call test-core
  • dagger call lint-core
  • dagger call format-core
  • dagger call lint-modules
  • dagger call lua-dev
#

Note: oh damn that's our friends Gaetan and Adrien πŸ™‚ That startup shut down, but the project is still open source. Not actively developed though..

coral kernel
dim smelt
#

OK so there's no 100% real-world prod CI setup in these examples

dim smelt
#

πŸ˜›

#

There's runme but I don't know if they use it as their CI, it looks more like a daggerverse module for others to get a build of runme (I doubt anyone uses that module if that's the case)

coral kernel
#

there's openmeter

dim smelt
#

oh yeah missed that one

#

ok that's the most real

#

Status update:

  • I have the general backwards compat problem mostly scoped out
  • I have a "harness" that I like for detecting backwards compatibility issues at runtime, reliably, and triggering a compat or migration flow
  • I think we can handle backwards compat as a form of migration (to keep actual loading & runtime logic clean)
  • BLOCKER: I don't know what to do with legitimate use of dagger call FOO in the main module. Should we keep a concept of "main module" or not. And if not: how to avoid breaking users in a painful way?
#

@brittle wharf @coral kernel if you're around I could use a sounding board πŸ™‚

#

Basically:

  • I'm scared to allow a concept of main module, and then we weaken our new model - now there's two ways to setup your project, more concepts to explain, fragmentation etc.

  • But I'm scared to forbid it, because it breaks existing users in a way that can't be auto-migrated, and can feel like we're crippling the UX for arbitrary reasons ("before I could type dagger call playground, now I have to dagger call engine-dev playground" -> that's an actual situation we had on our own repo)

#

I guess what you're proposing @coral kernel is a middle ground, where we allow it but as a time-limited, only-use-for-migration feature

coral kernel
dim smelt
#

(had a long sync conversation about this πŸ™‚ Will update the gist & soon push a POC branch

dim smelt
#

Prototyping the migration tool as a dagger module πŸ˜›

dim smelt
#

I've got a first migration tool POC that seems to work

dim smelt
dim smelt
#

@brittle wharf I'm reaching the point in my prototype where i need to define a Workspace type. I'm doing it from scratch for now, but if you have a branch somewhere with your own version of the type, I can take a look to harmonize early πŸ™‚ no rush

brittle wharf
#

i hit two hurdles: 1. how to have ⁨⁨Workspace.directory⁩⁩ route file syncs to the host from a module function call, and 2. the need for Dang to support constructors; I chose to focus on 2 so I could write some test modules (and because Mark needed it in https://discordapp.com/channels/707636530424053791/1468959453591240928), but haven't circled back

dim smelt
brittle wharf
#

it's also missing the bit where a Workspace argument taints the cache key with 'never cache'

dim smelt
dim smelt
#

Update: I'm debugging, then will ping you all for a pre-review & discussion

brittle wharf
#

@dim smelt did you end up folding my changes in? or should i try rebasing on your PR?

#

or more generally: lemme know how i can help!

dim smelt
#

trying to get my protoype working enough for demo & review today

#

given you have tons of Context already, might be worth glancing at the pr

dim smelt
#

@topaz seal hello! I would be interested in your thoughts on the pr... It's still a prototype but should be functional enough to test the UX

topaz seal
dim smelt
#

Ha ha πŸ™‚

#

I try to keep them in sync

#

But I recommend playing with it before reading the code

dim smelt
#

@brittle wharf want to try connecting parts 1 and 2 today? And ideally start working towards part 3

brittle wharf
dim smelt
#

@brittle wharf want to take 11812 for a spin, see how the UX feels? It's rough but somewhat usable

#

Next I'm adding -C with support for local and remote paths

brittle wharf
#

oh i'll follow the giant comment

dim smelt
#

This is my standard "playground":

dagger call \
 engine-dev playground \
 with-directory --path=./src/dagger --source=https://github.com/dagger/dagger \
 with-directory --path=./src/demo-react-app --source=https://github.com/kpenfound/demo-react-app \
 terminal
dim smelt
#

At the moment I'm fixing dagger install and dagger workspace info to detect workspace engine-side

topaz seal
#

quick comment (I haven't looked at the code to know if that's possible)
Would it be possible to always expand the first level of modules? If I install two modules in the workspace, I have something like that:

[modules]
jest.source = "github.com/dagger/jest"
plop.source = "modules/plop"

But it would be more readable and more open to edit to have this

[modules]

[modules.jest]
source = "github.com/dagger/jest"

[modules.plop]
source = "modules/plop"

I guess that's not high priority, just a feedback while playing with it

dim smelt
topaz seal
brittle wharf
#

i think i prefer the latter too - if there are a ton of modules i'm ok with the file being longer, vs. being deceptively dense and easy to overlook. low stock on this, just a first impression, opinion may change with real use πŸ˜›

dim smelt
#

Makes sense. And yes @topaz seal the goal is to be read and written by humans

dim smelt
#

Now that dagger install and dagger workspace info are implemented engine-side, I officially have a top-level dag.Workspace() that returns the client's current workspace. I don't love it, but not sure how else to implement those features engine-side. wdyt?

#

It's basically the equivalent of dag.CurrentEnv(), I think?

#

@coral kernel you ready for some testing? πŸ™‚

coral kernel
dim smelt
#

@coral kernel I could use some stress testing of the UX in 11812... Hoping to connect that to @brittle wharf 's workspace API so we can start writing our first experimental modules and try them e2e

dim smelt
#

@brittle wharf want a demo? I'm in lounge with Kyle

dim smelt
#

FYI @topaz zenith @crisp dove @restive swift this is the thread for my demo earlier

dim smelt
#

I just pushed dagger -C with support for remote workspaces

#

fixed: dagger functions shoudl no longer print module dependencies

brittle wharf
#

@dim smelt any reason it's core.DagqlWorkspace and not core.Workspace?

#

seems like it renames fine, can push

dim smelt
#

@brittle wharf I worry about the meaning of dag.Workspace() when called inside a dagger function.

brittle wharf
dim smelt
#

I guess it should mean "the current client's workspace", so if the client is a dagger function runtime -> the function's own "worspace" ?

#

it's like dag.Host() all over again

brittle wharf
#

is it that, or is it just empty + something the function can build, like a sandbox?

#

like dag.Workspace().WithRoot(Directory) or something, or dag.Workspace(Directory) to be explicit

dim smelt
#

Yeah so that's definitely the wrong name lol

#

Because as implemented it represents the client's current workspace (for purposes of installing modules into it, and getting its path)

#

I didn't get to the point where we might want to create a virtual workspace "from the outside" like we do for Env. Frankly I'd rather postpone that discussion to avoid breaking our brains. We seem to be doing fine without it so far

#

at least that's where my head was at so far

#

I guess dag.CurrentWorkspace() would more accurately mirror CurrentEnv()

brittle wharf
#

i see - it's used internally with (dagql.Server).Select

dim smelt
#

Should be exposed to the client also, see dagger install

brittle wharf
#

makes sense

brittle wharf
#

(or do we not expose it at all)

#

exposing it does feel weird, the caching semantics are unclear, and arguably broken? nothing forces the function to be no-cache

coral kernel
#

when we chatted live about this earlier I had the same question for dag.Engine() since I'm working on the generated clients right now

dim smelt
#

An alternative: we don't expose it at all.

Instead, the CLI finds another way to implement dagger install and dagger workspace info.

It could be special client params. I already defined a bunch in the PR:

  • ExtraModules -> specify modules to load on the fly (for -m)
  • SkipWorkspaceModules -> don't load modules from the workspace config (also used by -m)
  • RemoteWorkdir -> for -C on git modules

Meh but it would be super weird to add client params to modify the workspace config, or query its path. That feels too much like a shadow API

#

I think dag.CurrentWorkspace() is the least bad option right now

  • Matches an establshed pattern of CurrentEnv(), CurrentModule()...
  • Gets the job done for CLI
  • Confusion is manageable for module devs -> there will be lots of examples of the correct way; and if they call CurrentWorkspace() they will quickly see it's a dead end
  • Caching I think is a non-issue -> calling from a module, whatever's in there, is as cached as the module itself
lofty shadow
# dim smelt I think `dag.CurrentWorkspace()` is the least bad option right now - Matches an...

Caching I think is a non-issue -> calling from a module, whatever's in there, is as cached as the module itself
Am I correct that:

  1. The Workspace type is the thing that gives you access to all the various client resources (local dirs, etc.)?
  2. calling dag.CurrentWorkspace() from a module function is not the expected way to use workspaces, it's just something might technically be possible? And the expected way is that a Workspace type gets passed as arg through functions?

Just checking, because if calling dag.CurrentWorkspace() is actually the definitive way for a function to get access to the workspace, then I'm very concerned about caching behavior.

brittle wharf
#

1 and 2 are correct yeah (afaict)

dim smelt
#

@brittle wharf I pulled your changes, renaming to CurrentWorkspace() and moving on

brittle wharf
#

sweet

dim smelt
#

Next I'm going to look into workspace config & migration

-> bring the migration code into a dagger migrate
-> experiment with a dagger workspace config to set / get workspace config keys
-> pre-fill config keys on install

brittle wharf
#

@dim smelt looking into why module init fails, to get going:

❯ dagger-dev module init --sdk=github.com/vito/dang/dagger-sdk wstest
βœ” connect 0.1s

βœ” currentWorkspace(skipMigrationCheck: true): Workspace! 0.0s
✘ .moduleInit(name: "wstest", sdk: "github.com/vito/dang/dagger-sdk"): String! 0.3s ERROR
✘ export directory / to host /home/vito/src/dagger 0.0s ERROR
! failed to mount directory: no buildkit session group in context
dim smelt
dim smelt
brittle wharf
#

@dim smelt mind if i force-push a rebase? wanted to pull in erik's caching stuff in case it helps with any of this (update: it did not. but now i know that!)

lofty shadow
brittle wharf
#

yeah, that repros on upstream/workspace

#

interestingly it's when it's mounting the codegen'd directory to export that it errors, not when it tries to write to the host - that part's fine, as demonstrated by other commands

#

ah, i think i know a fix, testing now (just call export through DagQL, instead of directly)

#

yep, that did it:

❯ dagger-dev module init --sdk=github.com/vito/dang/dagger-sdk wstest
βœ” connect 0.2s

βœ” currentWorkspace(skipMigrationCheck: true): Workspace! 0.0s
βœ” .moduleInit(name: "wstest", sdk: "github.com/vito/dang/dagger-sdk"): String! 0.3s

Created module "wstest" at /home/vito/src/dagger/.dagger/modules/wstest
Installed in /home/vito/src/dagger/.dagger/config.toml
lofty shadow
brittle wharf
#

is there a rule of thumb?

#

i think this one's mostly glue so maybe not worth dagoping itself? no idea

#

it's marked DoNotCache since it's an API used primarily to interact with the host

lofty shadow
#

That distinction will go away soon too, but right now there's two worlds: outside of dag-op (normal world) and inside dag-op (inside of buildkit solver context)

I'd say be biased towards calling through dagql, wrapping in dag-op gets you persisted cache for the operation right now, but in the near future that won't matter and we can make any arbitrary thing persisted cache by flipping that on

dim smelt
#

I just pushed config hints on module install

#

now trying to add comments preservation with that neongreen lib

brittle wharf
#

a few weird things going on over here, possibly compounding:

  1. tried module init'ing a Dang module that takes a Workspace as an argument. except, the Workspace type is nowhere to be found. Directory, even Generator types work fine. inspected the schema JSON that gets passed to the Dang SDK, and Workspace is indeed missing, along with Query.currentWorkspace. can't find any type filters.
  2. to rule out it being a Dang thing, decided to dagger module init a Go SDK module.
    • first weird thing: it didn't create a main.go for me
    • second weird thing: when I ran dagger develop -m ./.dagger/modules/gowstest it got stuck on 1 when trying to load the Dang wstst module - looks like when ANY module is broken, nothing can load
    • third weird thing: when I ran cd ./.dagger/modules/gowstest; dagger develop it generated everything just fine in that directory, but it ALSO clobbered the go.mod in the root of my repo (workspace?)
#

now that the go SDK is codegen'd, I'm seeing *dagger.Workspace is missing there too, so it's at least consistent

#
  1. when I try to reference *dagger.Workspace from my Go SDK module, it seems to fail silently; its entrypoint just stops appearing in dagger functions and reappears when I swap it back to a ``Directory` arg
dim smelt
#

Pushed first attempt at comments preservation

#

Looking into migration polish

brittle wharf
dim smelt
brittle wharf
brittle wharf
#

pushed both

brittle wharf
#

@dim smelt pushed Workspace.directory(), Workspace.file(), and arg injection.
some notes:

  • added Workspace.clientID: String! - currently exposed through API, probably shouldn't be but left it for tinkering/troubleshooting in case it helps
  • also pushed a revert of #1468070450524459029 message since arg injection made it redundant by just forcing Workspace args to be optional, but not 100% sure on the direction here
#

i can now dagger module init --sdk github.com/vito/dang/dagger-sdk dangmod as:

type Dangmod {
  let source: Directory!

  new(source: Workspace!) {
    self.source = source.directory(".")
    self
  }

  pub ls: [String!] {
    source.entries
  }
}

and run:

❯ dagger-dev call dangmod ls
βœ” connect 0.1s
βœ” loading type definitions 0.7s
βœ” parsing command line arguments 0.0s

βœ” dangmod(
  ┆ source: currentWorkspace(skipMigrationCheck: true): Workspace!
  ): Dangmod! 0.0s
βœ” .ls: [String!] 0.0s

.dagger/
.git/
app.dang

πŸŽ‰

dim smelt
#

I'm very curious about dynamic filtering, will it get us real-world perf boosts

brittle wharf
#

i'm only testing in a tiny side repo, but i basically didn't notice any overhead at all. i'll try putting more junk in there and filtering

#

new problem: if I create a new file foo and re-run, it's not picked up

#

ah ha - it's not the syncing, it seems like it's the ls being cached, but it's not showing as 'cached' in the trace. (tested via dagger call dangmod source terminal and running ls in there)

brittle wharf
#

@lofty shadow do you have a preference between / alternative to these two approaches? (presented in one diff since they're conveniently near each other)

@@ -493,17 +503,6 @@ func (fn *ModuleFunction) CacheConfigForCall(

        dgstInputs := []string{cacheCfgResp.CacheKey.CallKey}

-       // Mix in the parent object's field content so that functions called on
-       // objects with different field values (e.g. a Directory that changed on
-       // the host) get different cache keys.
-       if parentObj, ok := dagql.UnwrapAs[*ModuleObject](parent); ok && len(parentObj.Fields) > 0 {
-               parentFieldsJSON, err := json.Marshal(parentObj.Fields)
-               if err != nil {
-                       return nil, fmt.Errorf("marshal parent fields for cache key: %w", err)
-               }
-               dgstInputs = append(dgstInputs, hashutil.HashStrings(string(parentFieldsJSON)).String())
-       }
-
        var ctxArgs []*FunctionArg
        var workspaceArgs []*FunctionArg
        var userDefaults []*UserDefault
@@ -933,6 +932,18 @@ func (fn *ModuleFunction) Call(ctx context.Context, opts *CallOpts) (t dagql.Any
                returnValue = returnValue.WithSafeToPersistCache(safeToPersistCache)
        }

+       // If this function accepts Workspace args, set a content digest on the
+       // result derived from the actual output. This ensures downstream calls
+       // that reference this result get a different cache key when the result
+       // content changes (e.g. a Directory synced from a changed host workspace).
+       // Content digests propagate through the DAG via ID references.
+       if returnValue != nil && fn.hasWorkspaceArgs() {
+               contentDigest := hashutil.HashStrings(string(outputBytes))
+               returnValue = returnValue.WithID(
+                       returnValue.ID().With(call.WithContentDigest(contentDigest)),
+               )
+       }
+
        return returnValue, nil
 }

the problem being solved: the directories synced at runtime and stashed in the module fields had no bearing on the cache key for downstream function calls

first attempt (-) fixes it by marshaling parent object fields and mixing it into the downstream func digest
second attempt (+) fixes it by making module functions content-addressed on their result payload, which is interesting, but i'm not sure if e.g. the objects encoded into IDs will have the caching semantics we want

my guess is there's a less obvious but cleaner third approach, or the second one is actually fine?

#

(making claude try harder, no rush)

lofty shadow
brittle wharf
#

yeah, I'm a little surprised too, but I think I can see how we wouldn't notice; we've always forced cache keys to be part of arguments

lofty shadow
#

The first attempt is not right in that it almost certainly shouldn't be parentFieldsJSON that we mix in, it should be the ID digests of any IDAble parent fields, but given that adjustment it does make sense

brittle wharf
#

and never exposed host() to module functions

lofty shadow
brittle wharf
#

just pushed now

#

(brb)

lofty shadow
#

in core/schema/workspace.go I would have expected dagql.Func("currentWorkspace", s.currentWorkspace). to be dagql.FuncWithCacheKey("currentWorkspace", s.currentWorkspace, dagql.CachePerCall).

Right now every workspace has the same cache key I think

#

Which would explain this

#

I think you gotta make that change either way otherwise anything that accepts Workspace args won't get cache invalidated, i.e. right now I bet new itself is cached

#

lemme know if I'm missing something

dim smelt
#

I'm still working on incorporating migration

brittle wharf
#

yep - works without it

lofty shadow
dim smelt
#

@brittle wharf rebased and pushed

#

with built-in migration

brittle wharf
dim smelt
#

dagger migrate works πŸ™‚

brittle wharf
#

sorry yes i don't really mean arbitrary access, just couldn't remember whether we expect -m invocations to also pick up Workspace automatically

dim smelt
#

It allows for cleanly incoporating -m in the workspace model without breaking compat

brittle wharf
#

oh wait, that test isn't even doing -m, i misread it as plain module use but it's also installing it into the workspace facepalm - it's calling it toolchain style

dim smelt
#

-m works exactly like an ephemeral module install in the workspace

#

I like it because it keeps everything consistent

#

without breaking old commands which is a miracle honestly

brittle wharf
#

sgtm! with it being rooted in the workspace, that'll be a safer alternative to host() anyway

#

pushed a test + fix for that

dim smelt
#

Pushed: toml formatting the way you suggested @brittle wharf @topaz seal

brittle wharf
#

@dim smelt case to handle in dagger migrate: when the main module has relative dependencies, they need to be made relative to the new dagger.json location. for https://github.com/vito/dang that meant putting ../../ at the start of the sdk and both dependencies

side note: it'd be nice if those could all be absolute, relative to the repo root. thinkspin wonder if that works? haven't tried

dim smelt
brittle wharf
#

yeah that's fair. these just get really really fiddly when you move modules around. but i guess the same is approximately true for renames too.

topaz seal
#

One random thought. What about a dagger workspace config command, in the same spirit than git config? Something like:

  • dagger workspace config will print the toml file in a pretty-print version (always the same, whatever the way it's stored)
  • dagger workspace config [key] for instance dagger workspace config modules.my-module.source to print a key
  • dagger workspace config [key] [value] for instance dagger workspace config modules.my-module.alias true to set a key

The goal would be to interact with the toml file in a controlled way, so that users don't need to edit the file by hand.
Happy to play around this idea is that makes sense.

dim smelt
topaz seal
#

If there's specific thing you'd like me to contribute, let me know and I jump on it (and to avoid conflicts πŸ™‚ )

dim smelt
topaz seal
dim smelt
#

also there's a bug where dagger migrate and dagger workspace info are too aggressively cached.

cd demo-react-app
dagger workspace info
-> "error, please migrate" βœ…
dagger migrate
dagger workspace info
-> ok βœ… 
rm -fr .dagger
dagger workspace info
-> ok ‼️
dagger migrate
-> "nothing to migrate" ‼️
dim smelt
#

Another bug: dagger migrate doesn't show enough telemetry by default. You have to increase the verbosity to 2 or 3, otherwise it looks like it's hanging after "connect"

brittle wharf
#

hit what feels like a 1% chance bug with module aliasing (promoting fields to Query): my dang module provides a Dang.dang(args: [String!]!) function, originally exposed for building + running dang <args> for an LLM. but, Dang.dang clobbers the original Query.dang so all the promoted fields fail with required arg "args" not provided. dang.
theoretical fix: when aliasing, move the original constructor to _foo? or, don't alias functions that would clobber?

dim smelt
#

@brittle wharf actually I didn't understand the "requird args" not provided. Your constructor requires arguments basically? And aliasing breaks the mechanism for providing them?

brittle wharf
#

it's because the aliased fields internally re-call through the constructor, like Query.foo wants to reroute to Query.dang.foo (where the workspace gets passed through), but by that point the Query.dang has been swapped out for the alias

#

so all the aliases end up calling the Query.dang(args: [String!]!) instead

dim smelt
#

Sorry have to drop to demo calls

#

The branch is yours πŸ™‚

#

I was about to push my rebase on main.

#

(done)

brittle wharf
lofty shadow
#

How devastating is it? I can definitely try to address it as part of the current cache updates

#

I coincidentally am working on modfunc stuff rn

brittle wharf
#

pretty devastating πŸ˜‚ - this is part of the 'special sauce' to make workspaces viable

lofty shadow
#

I see

#

Lemme thinkies

brittle wharf
#

which sounds reasonable in my head, but i'm not sure if that code actually, you know, worked

#

though even unpacking IDs isn't accurate enough - if the constructor read the content of a file and baked it into the object fields, we would also want downstream caches busted

lofty shadow
#

My suggestion:

  1. Leave in the cache key change to workspace, that's better than DoNotCache because it stop currentWorkspace from re-executing literally every time it's ever encountered in an ID. CachePerCall means that a new call that includes it will always evaluate currentWorkspace but if you append more selections after that to the ID it won't re-eval. I think that's what you want here
  2. Try that in combination w/ the commit for unpack IDs in the response and add them to the returned content digest
    • I can see how that'd work now. In main we prefer content digests of inputs when making digests for new appended calls, including the parent. So if the parent has a content digest, then that should be what's used for the subsequent selection on ls
lofty shadow
brittle wharf
lofty shadow
brittle wharf
#

not sure if that's a trivial 'the helper is gone now because it's unused' or 'it's gone now because caching works differently and it's too late by then'

lofty shadow
#

But you need the interface wrappers in this part of the code cause it's arbitrary types only known at runtime

#

So you just gotta add WithContentDigest to the iface

brittle wharf
#

sweet, ty!

brittle wharf
#

that worked ish - confirmed an appropriate content digest is being set (only changes when files change), but the downstream function still runs every time.
i'll keep digging to get more familiar, no rush

lofty shadow
#

So printlning there should eventually reveal a discrepancy

brittle wharf
#

hm. println debugging shows the cache key is stable, but it's still re-running for some reason

lofty shadow
brittle wharf
#

yeah - it logs

#

pushed to vito/wip-workspace-cache-debug if you want to poke around

#

here's the logs i see:

❯ docker logs -f dagger-engine.dev 2>&1 | grep '!!!'
time=2026-02-12T16:40:13.028Z level=WARN msg="!!! withContentDigestFromReturnedIDs" value=Dangmod@xxh3:067dc85c1604ea46 returnedIDs=map[xxh3:d494b13dcc2d3f4a:0xc005136d40]
time=2026-02-12T16:40:13.028Z level=WARN msg="!!! withContentDigestFromReturnedIDs mixing" dgstInputs=[sha256:a6316ebc2c4bc39651683f65eb55d92def9b105bb3f5f0b8ffcbad0e4b11a8b1]
time=2026-02-12T16:40:13.028Z level=WARN msg="!!! withContentDigestFromReturnedIDs mixing" contentDigest=xxh3:26040607f2c4ebeb
time=2026-02-12T16:40:13.029Z level=WARN msg="!!! parent has content digest" d=xxh3:26040607f2c4ebeb
time=2026-02-12T16:40:13.029Z level=WARN msg="!!! computing cache key" h1=xxh3:26040607f2c4ebeb h2=xxh3:6a6249929b30ee40 h3=xxh3:857db0722beee7c4 h4=dangmod
time=2026-02-12T16:40:13.029Z level=WARN msg="!!! DA FINAL CACHE KEY" key=xxh3:d6e41065e7c9f7d1
time=2026-02-12T16:40:13.029Z level=WARN msg="!!! DA EVEN MORE FINAL CACHE KEY INPUTS" inputs=[xxh3:d6e41065e7c9f7d1]
time=2026-02-12T16:40:13.029Z level=WARN msg="!!! DA EVEN MORE FINAL CACHE KEY" digest=xxh3:624176e8e5fc6f1a
time=2026-02-12T16:40:18.286Z level=WARN msg="!!! withContentDigestFromReturnedIDs" value=Dangmod@xxh3:05593baa30417c02 returnedIDs=map[xxh3:834d5e289d373ca4:0xc006d54f00]
time=2026-02-12T16:40:18.286Z level=WARN msg="!!! withContentDigestFromReturnedIDs mixing" dgstInputs=[sha256:a6316ebc2c4bc39651683f65eb55d92def9b105bb3f5f0b8ffcbad0e4b11a8b1]
time=2026-02-12T16:40:18.286Z level=WARN msg="!!! withContentDigestFromReturnedIDs mixing" contentDigest=xxh3:26040607f2c4ebeb
time=2026-02-12T16:40:18.286Z level=WARN msg="!!! parent has content digest" d=xxh3:26040607f2c4ebeb
time=2026-02-12T16:40:18.286Z level=WARN msg="!!! computing cache key" h1=xxh3:26040607f2c4ebeb h2=xxh3:6a6249929b30ee40 h3=xxh3:857db0722beee7c4 h4=dangmod
time=2026-02-12T16:40:18.287Z level=WARN msg="!!! DA FINAL CACHE KEY" key=xxh3:d6e41065e7c9f7d1
time=2026-02-12T16:40:18.287Z level=WARN msg="!!! DA EVEN MORE FINAL CACHE KEY INPUTS" inputs=[xxh3:d6e41065e7c9f7d1]
time=2026-02-12T16:40:18.287Z level=WARN msg="!!! DA EVEN MORE FINAL CACHE KEY" digest=xxh3:624176e8e5fc6f1a
#

gonna see if claude can write an integ test and figure it out in parallel

lofty shadow
brittle wharf
#

huh. that's with dev cli?

lofty shadow
#

ba2677b31df22be1e4568eee1b7d13c05c0d30e7 from your dagger repo

brittle wharf
#

could it be a ~/go/bin/dagger taking priority in $PATH? that happens to me now and then from a rogue go install

lofty shadow
#

No I don't think so

sipsma@dagger_dev:~/repo/github.com/sipsma/wstest$ which dagger
/home/sipsma/repo/github.com/sipsma/dagger/bin/dagger
sipsma@dagger_dev:~/repo/github.com/sipsma/wstest$ dagger version
dagger v0.19.12-20260211024413-dev-b83bcf240d80 (docker-container://dagger-engine.dev) linux/arm64/v8
#

I'll triple check πŸ˜„

brittle wharf
#

tried with a fresh clone of the repo locally, too

lofty shadow
#

I'm trying deleting everything including the stable v0.19.11 engine that builds the dev engine in case that's somehow using stale cached data

lofty shadow
#

I have too many dagger clones/worktrees due to having agents do things in parallel

#

one sec....

brittle wharf
#

if it's any consolation i'm having my own existential crisis as Claude wrote what seems to be a legitimate test for the problem, but it's passing 🫠

#

lemme see if it fails without the "fix"...

lofty shadow
# lofty shadow one sec....

Yeah it works now, very subtle difference

sipsma@dagger_dev:~/repo/github.com/sipsma/wstest$ which dagger
/home/sipsma/repo/github.com/sipsma/dagger2/bin/dagger
sipsma@dagger_dev:~/repo/github.com/sipsma/wstest$ dagger version
dagger v0.19.12-20260212164452-dev-ba2677b31df2 (docker-container://dagger-engine.dev) linux/arm64/v8
topaz seal
#

Not sure how you want to proceed with contributions to workspace branch.
So I pushed on a branch on my fork this: https://github.com/eunomie/dagger/commit/2a21bf9faa924ab8e02f2211d437b940ea94869f
It allows to read and write configuration:

  • dagger workspace config: prints the full file
  • dagger workspace config [key]: print the value or the sub tree. For instance dagger workspace config modules.jest.source or dagger workspace config modules.jest
  • dagger workspace config [key] [value]: set the value. For instance dagger workspace config modules.jest.source github.com/dagger/jest or dagger workspace config modules.my-module.config.branches main,master (array are supported)

With

[modules.eslint]
source = "github.com/dagger/eslint@main"

[modules.eslint.config]
source = "./path" # Directory
baseImageAddress = "" # string
packageManager = "" # string

$ dagger workspace config modules.eslint.config
source = "./path"
baseImageAddress = ""
packageManager = ""

or

$ dagger workspace config modules.eslint
source = "github.com/dagger/eslint@main"
config.source = "./path"
config.baseImageAddress = ""
config.packageManager = ""

But we can only write one value at a time.

If that sounds good I guess the commit can be cherry-picked to the workspace branch.

dim smelt
topaz seal
#

ok, I'll do that then πŸ™‚

#

pushed

lofty shadow
# brittle wharf if it's any consolation i'm having my own existential crisis as Claude wrote wha... 
diff --git a/dagql/cache.go b/dagql/cache.go
index 7a1877f94..d6cecef3e 100644
--- a/dagql/cache.go
+++ b/dagql/cache.go
@@ -905,6 +905,10 @@ func (c *cache) GetOrInitCall(
                }
        }
 
+       if key.ID.Field() == "ls" {
+               slog.Warn("!!! DA YET EVEN MORE FINAL CACHE KEY", "digest", storageKey)
+       }
+
        ctx = ctxWithStorageKey(ctx, storageKey)
 
        if ctx.Value(cacheContextKey{storageKey, c}) != nil {

The storage key (πŸ™„ I can't wait to be rid of this) is changing each time even the dagql cache key is the same. Looking into why...

brittle wharf
#

@topaz seal those new workspace config tests are failing - but, they feel like legit failures to me: at least one of them is failing because it tries to load modules that the config reference but those modules don't exist. i feel like config plumbing commands (among others) shouldn't fail on that sort of thing

lofty shadow
topaz seal
topaz seal
brittle wharf
topaz seal
#

yes, I can reproduce it when I'm running the tests, but when I'm doing the same thing by hand that works as expected with what it seems a different trace
(with the config file referencing a non existing module)

$ dagger workspace config
βœ” connect 0.0s

βœ” currentWorkspace: Workspace! 0.0s
βœ” .configRead: String! 0.0s

[modules.my-module]
source = "modules/my-module"
alias = true
#

So either my tests by hand are wrong, or the integration tests are doing something I don't understand

brittle wharf
#

hmm. are you testing in a git repo?

#

(throwing things at the wall to see what sticks - the tests do that)

topaz seal
#

(the ability to see all the commands and open them as if it wasn't nested is really nice still, amazed each time I'm looking at them πŸ™‚ )

dim smelt
#

Running dagger check -l in dagger/dagger (post-migration) doesn't work.. fails with a weird engine error. FYI

topaz seal
#

I need to go cooking.
If I can I'll fix the tests tonight, but I'm not sure so maybe tomorrow morning. Sorry for the disturbance. Feel free to just ignore them delurk
Do not hesitate to ping me with new tasks for tomorrow depending on the progress you'll make during the day.
On my list I wrote the telemetry issue on dagger migrate but anything else is fine to me.

dim smelt
lofty shadow
#

@brittle wharf figured it out, here's a funny workaround that makes it cached as expected by just changing the dang code:

diff --git a/.dagger/modules/dangmod/main.dang b/.dagger/modules/dangmod/main.dang
index 91efbf2..731507e 100644
--- a/.dagger/modules/dangmod/main.dang
+++ b/.dagger/modules/dangmod/main.dang
@@ -6,7 +6,7 @@ type Dangmod {
     self
   }
 
-  pub ls(buster: String! = ""): [String!] {
+  pub ls(buster: String! = ""): [String!]! {
     print("im listing bro")
     source.entries
   }

The problem is that the dagql.DynamicNullable was not propagating safeToPersistCache when deref'ing a result, so the DynamicNullable[Result[T]] was safeToPersist, but the Result[T] itself wasn't.

I bet this explains why none of our integ tests caught it, we probably always return types that are non-nullable from other SDKs (or at least in the tests we have)

#

I'll send you an engine fix in a sec (but also gonna send out a fix for main too since it's an issue there0

brittle wharf
#

nice catch, incredibly flukey that we even found that lol, [String!]! is totally what the code should have been

#

i have a failing workspace test for it now, looking forward to a solid green KermitSwagSunglasses

brittle wharf
#

confirmed the test passes with the ! fix and cache key changes, and fails on upstream/workspace with the ! fix and without the cache key changes (that probably was easier to to write than to read, sorry)

lofty shadow
dim smelt
#

Going to pick the next task on the list. Anything already in flight?

#

@brittle wharf when you mentioned getting workspace API to work, with a first module using it etc. was that on the workspace branch?

brittle wharf
dim smelt
#

nice

#

I'm going to focus on plumbing user defaults from workspace config

#

probably will add something like Workspace.Modules().Config() while I'm at it

brittle wharf
#

@dim smelt just did a rebase + push -f with the content hashing changes (and main)

#

pulled workspace first so pretty sure i didn't clobber anything

dim smelt
#

I'm wrangling EnvFile & user defaults plumbing

#

was thinking maybe I'll take a swing at lock file after?

brittle wharf
#

how far are we from green CI? would be nice to get on a baseline but not sure what's involved. when i took a peek there were a ton of migrate errors in the tests

#

still need to fix the alias clobbering (Dang.dang eating Query.dang) too

dim smelt
#

yeah makes sense. at the moment we're intentionally breaking and requiring migration. so if we stick with that we'll need to migrate the test environments itself

dim smelt
#

@brittle wharf how do you feel about the current approach to require migration, but also make it very easy?

#

My thinking was: one-time user pain, but much cleaner codepaths, less risk of inconsistent behavior or bugs

#

until you migrate, you're kind of living in an illusion... things don't really work the way you think they do

brittle wharf
#

i see the appeal but i have a feeling people are gonna come and complain πŸ˜›

a global warning might be better

dim smelt
brittle wharf
#

should be easy enough to do with that ⁨telemetry.GlobalWriter⁩ thing we added

#

might also get ci green πŸ˜› - should i give that a go? giving it a go if not just to assess scope of ci damage

dim smelt
#

A warning is fine, but what happens after... We need to pretend like your not-a-workspace dagger.json is actually a workspace

#

The runtime compat layer is not trivial to get right

brittle wharf
#

yeah just landed there, assumed it was just an extra guard but nope

dim smelt
#

Since migration is implemented engine-side, we could just apply the migration "virtually", and internally it still looks like a workspace

brittle wharf
#

was just typing that πŸ˜›

dim smelt
#

probably easier than to duplicate codepaths everywhere

#

but still extra plumbing..

brittle wharf
#

maybe this is what forces 'virtual workspaces'?

#

eh. ok i can be convinced to just error

#

unless we chuck it into an llm and it figures it out, it's kind of a lift

dim smelt
#

How about we just punt for now? We can always do it later. And if we do it, better to do it a bit later IMO

#

(but pre-merge)

#

may downside is harder to get green ci... but we can just migrate the test modules

brittle wharf
#

sgtm, looking into that now

#

⁨ci:bootstrap⁩ also fails because ⁨dagger/dagger⁩ itself is unmigrated, but if we do that we lose all CI, so i'll see if i can have it migrate first somehow i guess?

#

not a bad way to test ⁨migrate⁩ in the meantime

dim smelt
brittle wharf
#

ah interesting, that'd be preferable

#

normally migrate moves the ⁨⁨dagger.json⁩⁩ into ⁨⁨.dagger/modules/foo⁩⁩, so just undo that and update the config to match? (or something)

#

looks like ⁨migrate⁩ is missing a step to move the old ⁨.dagger/*⁩ source into ⁨.dagger/modules/foo/⁩?
⁨```
❯ tree .dagger
.dagger
β”œβ”€β”€ config.toml
β”œβ”€β”€ go.mod
β”œβ”€β”€ go.sum
β”œβ”€β”€ main.go
└── modules
β”œβ”€β”€ dagger-dev
β”‚Β Β  └── dagger.json
β”œβ”€β”€ gowstest
β”‚Β Β  └── dagger.json
└── wstest
└── dagger.json

brittle wharf
dim smelt
#

on it

brittle wharf
# dim smelt on it

pushed a migration in the meantime, feel free to undo and force-push over it

#

⁨migrate --recursive⁩ would be nice too. not quite ⁨develop -r⁩ since that meant dependencies, this one would just find all migrateble ⁨dagger.json⁩s and do it

#

adding that, and a ⁨dagger migrate -l⁩ to list migrateable modules

brittle wharf
#

punting on ⁨⁨migrate --recursive⁩⁩ for now, actually it doesn't make sense as i originally imagined since we'd want one workspace for the entire repo

#

one confusion opportunity i hit was when you run ⁨dagger migrate⁩ in a subdir it no-ops because it just sees the root level workspace

topaz seal
#

The tests should be fixed now (I just pushed, I'll see the CI but they are passing on my machine). It was effectively a different behavior between manual tests (CLI calls) and integration tests, where SkipWorkspaceModules were not taken in account

topaz seal
dim smelt
brittle wharf
#

@dim smelt wdyt of doing a merge + ship asap, so we can dogfood without breaking CI? we're in a nice compatibility phase on HEAD at the moment but that only lasts until we actually start using Workspace everywhere

dim smelt
#

btw right now I'm filling the gaps in migrate (source code not moved etc) + adding informative terminal output during migration

brittle wharf
#

i priced some of that in to 'asap' - but also it's Friday so realistically it'd be a bit, i did forget about the blocking migration though (pre-coffee)

#

basically just mean in terms of features, as opposed to moving on to part 3 (artifacts)

#

hmm maybe i could carve out the Workspace API + caching changes, and have it also work in the old world?

#

@dim smelt is modules/migrate/ dead code?

#

(just chasing down lints)

dim smelt
#

Warning I'm modifying dagger migrate atm (for filling gaps) but yeah modules/migrate is dead

#

So to summarize the plan:

  • Stabilize parts 1 & 2
  • Ship asap

?

#

btw there's another migration aspect: the code of toolchains themselves, from +defaultPath to Workspace

brittle wharf
#

yup - that one's quite a bit harder, right? you'd need it to write code

dim smelt
#

Should we anticipate these edge cases by preparing an agent skill for users?

#

I also found another tricky one on workspace migration: if your main module has local dependencies that are not dagger-specific: like go-mod-replace, or importing local paths from your python/typescript code. So a skill could help with that also

brittle wharf
#

sgtm, i guess we could embed them and write them out if you pass a flag (with a message letting them know about it)

#

would be cool if we just ran the agent ourselves, but setting up auth and all that is still a bit thorny for a guided experience

#

side q: if I cd into a module that has dagger.json and run dagger call, that should pick up the module in the workdir, and not the outer workspace, right?

#

like, ./dagger.json > ../../.dagger/config.toml > ../../dagger.json in terms of precedence

dim smelt
#

rebased & pushed

#

We should go over an important detail of parts 1 & 2: workspace roots & paths... Both inside and outside the sandbox. We made progress but worth double-checking the details since it's foundational to everything else

#

Also I'm going to make dagger init an explicit alias to dagger workspace init

dim smelt
#

There's several distinct things:

  1. What is a workspace: it's a configuration context for dagger, scoped to a directory. Every client always has a workspace.
  2. Workspace detection: as soon as a client connects, the engine searches for the workspace directory. That directory can be explicitly marked by .dagger, or implicitly by falling back to current git root, or as a last resort, current directory.
  3. Workpace initialization: when the client calls dagger workspace init, where is .dagger created? This might be different from workspace detection depending on what we decide. --> cd ./subdir/of/my/repo && dagger workspace init -> does that create the workspace in the current directory, or the repo root? One is more "logical", but the other is more likely to be what most user wants.
  4. Workspace fs sandbox. When a module reads files from the workspace from inside the sandbox: what is the filesytem sandbox? We know it's not always the workspace itself -> modules may need to access files anywhere in the repo, regardless of where the workspace itself lives. So, even if the workspace is not at the repo root, the fs sandbox probably is.
  5. Relative vs absolute paths in the fs sandbox. Absolute paths are expected to start from the root of the fs sandbox. But what about relative paths? There are several possible answes there: also relative to the sandbox root? relative to the workspace path? relative to the client workdir? This forces us to answer the underlying question: why would a module use a relative path?
  6. Entanglement with part 3?. we have an unresolved thread with @coral kernel on the overlap between "sub-workspaces" and "dynamic artifacts" which complicates the discussion on relative paths. Basically, if I have 5 "apps" in my workspace, do I create sub-workspaces for them (with a module installed 5 times), or do I leave it to "dynamic artifacts" (via part 3) to handle that?
brittle wharf
#

i have a fix for it locally (LLM'd) but want to make sure it's in line with The Plan

#

Or is the expectation to pass explicit -m .?

coral kernel
brittle wharf
#

though now I'm confused because if dagger init is aliased to dagger workspace init, and dagger module init always initializes within a workspace, there's no way to init a standalone module in an arbitrary directory anymore

dim smelt
#

Yeah that was before I worked out end-to-end backwards compat and migration... Now I think we need to revisit in that context

dim smelt
brittle wharf
#

that part's fine, but it still installs it into a Workspace, right?

dim smelt
brittle wharf
#

ah ok

dim smelt
#

that part is not implemented (or tested) btw πŸ™‚

brittle wharf
#

well i'm testing it now πŸ˜›

#

adding a workspace test for it

dim smelt
#

I guess implicit -m . could be ok. We could make it another client param.

Implementation: if the current workdir is inside a dagger module (dagger.json), and skipWorkdirModule is not set, and the client didn't already set module (the client param for -m), then set module to the workdir module -> aka "default -m ."

We could make that behavior configurable in the workspace config if needed

coral kernel
#

At least for a transition period that makes sense to me. Eventually we'll get to a point where that will surprise someone because all they know are workspaces and we can revisit it

dim smelt
#

FYI I renamed the PR + added a description to match Parts 1 + 2

#

Now editing the PR to explain parts 1 & 2 as a combined change: "Workspaces" aka "modules v2"

  • Part 1 introduces the concept of Workspace and combines it with
    modules to create a new configuration and dependency model
  • Part 2 introduces the Workspace API
dim smelt
#

Workspaces aka "modules v2"

#

Renaming this thread to reflect the merging of parts 1 and 2

dim smelt
#

@brittle wharf one looming topic for discussion: how the workspace API should mutate the workspace.

#

Ideally we start simple, then we can layer more powerful concepts?

Ultimately it intersects with things like:

  • dagger call sometimes prompts you to apply a changeset
  • Functions can return an Env and that will mutate the LLM's environment when called

Would be nice to have a unified model that we can converge, without complexifying everything.

--> Deja-vu, I think we discussed all this at the beginning of "evolution of Env", but now we're much further along in implementation so it's no longer a hypothetical future problem

#

Update: cleaned up plumbing of workspace config, it co-exists with .env to apply the same user defaults. old-style.env loading is disabled by default but can be enabled in the workspace config

#

FYI: there's a rampup.md at the root of the repo (temporary) that I use to ramp-up new agents. I just say "read rampup.md" then I prompt them. works well

dim smelt
brittle wharf
#

@dim smelt tried migrate again to test the module source moving - almost there, but ended up with some weird duplication:

❯ tree .dagger/
.dagger/
β”œβ”€β”€ config.toml
β”œβ”€β”€ main.dang
└── modules
    └── dang
        β”œβ”€β”€ dagger.json
        β”œβ”€β”€ main.dang
        └── modules
            └── dang
                └── dagger.json

5 directories, 5 files

(previously it was just .dagger/main.dang)

dim smelt
brittle wharf
brittle wharf
#

feels like it'd be easy to miss and assume it init'd in the work-dir, if it init's in repo root

dim smelt
#

Mmm but it needs to be possible to do it - in large monorepos, it's explicitly requested by power users

#

Maybe we error if not in git root and explicit path not given?

brittle wharf
#

that sounds pretty good

dim smelt
# 
cd ./some/subdir
dagger init
ERROR: are you sure you want to initialize the workspace in a subdirectory of the git repo? If so, run 'dagger init .' Otherwise, run this command again from the root of the repo
brittle wharf
#

the case i fear is:

  • user assumes workspaces are roughly equal to modules, thinks you init them everywhere
  • run init, it "succeeds" but they don't notice it did it root of the repo => they assume it's a bug
#

could be overworrying tho

dim smelt
#

Maybe a bit aggressive for beginners though... A softer version would be a warning: works out of the box for standard users. Power users see the warning and can course-correct

cd ./some/subdir
dagger init
WARNING: initializing dagger workspace at the git root (/foo/bar). To initialize in the current directory, run "dagger init ."
#

I don't love any of these options...

#

Maybe we just create it in the current dir, and just make it very clear where it is. Seems pretty reasonable

brittle wharf
#

seems OK too. if monopeople are gonna be doing it in the happy path, the warnings will feel weird. if a user is overdoing it with workspaces they can always simplify later

#

lol did not mean to type monopeople there but leaving it

dim smelt
#

I'm converting dagger check to the new workspace system. At the moment it is broken

brittle wharf
dim smelt
#

want to get a basic workspace init working. Then will be afk for 2-3 hours. Then will check back in

brittle wharf
#

@dim smelt pushed a change for implicit -m . and fixing up other broken aspects of -m but i think this'll need another pass - initializeWorkspace in particular got pretty hairy, but I want to see how much it helps with the tests. left a FIXME(vito) to keep track

#

added more workspace tests along the way, covering dagger module init <name> <dir> (which now works) and making sure they work when nested beneath a workspace

dim smelt
#

ok will be back at keyboard in 30mn or so, will take a look. Still ok for me to push dagger workspace init or are you holding a lock on that part of the code?

dim smelt
#

(I'm back at it, claude wants `init to work CLI-side, but I want the CLI as dumb as possible. Might add a client param to avoid chicken and egg problem - auto-detect workpspace at connect, in order to create a workspace...)

brittle wharf
#

I also found it struggling between the client side / server side split

dim smelt
#

Finished workspace init (not yet pushed)

#

Now cleaning up handling of remote workspaces. Claude slapped it into a duplicated code path, I'm DRY-ing it

brittle wharf
#

(that HTML file is the pi agent session log tree, pretty neat)

brittle wharf
#

Need to switch all the tests from init to module init

brittle wharf
#

@dim smelt quick check in - I got the module tests down from ~270 failures to ~160, but now im getting the feeling we'd be better off extracting JUST the Workspace API + arg + caching support, shipping that on its own, and then switching our toolchains to it. that way we can unblock dogfooding, and take our time with the CLI DX (and tests)

dim smelt
dim smelt
brittle wharf
#

yeah exactly. cool, will go for that tmrw!

dim smelt
#

actually @brittle wharf , if we merge part 2 first, what will be the workspace API accessing? If there's no actual concept of workspace yet

dim smelt
#

pushed a fix to the "dagger check" crash

#

also pushed improvements to the engine playground skill, and a simple qa skill that runs through a simple scenario in the playground and reports back.

(i used those to automate the bugfix loop)

topaz seal
#

I pushed a new commit that runs dagger migrate in parallel.
It looks like something's wrong as they all finished at the same time but that's not the case, it's just they are waiting for the same thing (the SDK). I put a sleep in one of them just to ensure it's good.
See the before and after

topaz seal
topaz seal
#

I'm currently working on two bug fixes:
βœ… when dagger module init it creates the module under .dagger/modules/MY_MODULE but also at the root... removing that
πŸ”² I'll have a look at the "cache" issue: running a module, changing the configuration, running the module, configuration is not used (as we saw in yesterday's demo)

topaz seal
brittle wharf
dim smelt
#

so this will plumb into toolchains

brittle wharf
#

yup yup

dim smelt
#

btw I think `dagger check' is still crashing the engine

topaz seal
#

I'll have a look at the "cache" issue: running a module, changing the configuration, running the module, configuration is not used (as we saw in yesterday's demo)
I can't reproduce it πŸ€”
My test scenario:

  • dagger migrate on dagger/dagger
  • dagger call go base terminal then go version -> 1.25.7
  • edit .dagger/config.toml to change the go version to 1.24 and uncomment this line
  • dagger call go base terminal then go version -> 1.24.13
    So this is working. But I saw yesterday the issue, so not sure why.
dim smelt
#

if so it could be a different engine

topaz seal
#

no, just running the playground on my machine

dim smelt
#

@brittle wharf @topaz seal @coral kernel I tried porting a few of our own modules to workspace API. Not sure where to put the ignore filters. For now I make them regular []string arguments, which is neat. But, the discrepancy between include/exclude and ignore makes it a little bit inconvenient

dim smelt
#

@brittle wharf I'm watching the kids this morning so can only do async. But available here if you want to coordinate the branch split / rebase

brittle wharf
dim smelt
#

How much work to rebase, part 1 on it, do you think?

#

Also: how will 11874 address breaking changes for toolchains?

#

I don't think adoption is huge yet, but it is in the docs and there's a quickstart using it

#

so we should assume there are others out there using toolchains in prod

brittle wharf
#

well, the point of 11874 is to not bring any breaking changes, so there should be nothing to address

brittle wharf
dim smelt
#

I guess it makes more sense to keep it

brittle wharf
#

yeah unless i'm missing something i just have to literally not touch that code path notsureif - this is all additive

dim smelt
#

And then, the breakage that is left, is when we modify our toolchain modules to receive a Workspace -> but that's just a normal case of requiring a newer engine

brittle wharf
#

we'll see if the ci run agrees with me πŸ˜›

brittle wharf
dim smelt
#

Cool cool. Makes sense to focus some extra effort on the cache invalidation part, making sure the injection magic happens when we want, and doesn't happen when we don't want. etc. Which 11874 allows for

brittle wharf
#

and the idea is that the older (newer than now, older than Workspaces 'for real') outer engine, once we ship Workspace 11874, will still be able to run the Workspace PR's (11812) checks once we transition

dim smelt
#

yeah unlike what happened last night when I started modifying toolchains in the branch

#

Should I try rebasing workspace now?

brittle wharf
#

if you want! you could try doing a 'soft' rebase where you just realign it on top of the same commits that were extracted into 11874, like just re-ordering in git rebase -i

#

i could try that later too if 11874 is looking good

dim smelt
#

OK - maybe better to let you take the wheel on both sides. I have a demo call coming up soon (roughly 1h) then dentist appointment... Will be back online 3pm my time - makes more sense if you own the whole operation today IMO

brittle wharf
#

works for me πŸ‘

dim smelt
#

I'll use my remaining hour to work on other parts of workspace

brittle wharf
#

btw - dang is a pretty nifty debugging tool now, with the new REPL and :doc. earlier I did dagger-dev -s run dang to start its REPL pointing to dev dagger, and did :doc to verify that Workspace args were being made optional. (there was a weird i/o buffering thing caused by the wrapping, but it worked once i realized that's all it was). more info: #daggernauts message

dim smelt
#

What does :doc mean?

brittle wharf
#

it's sort of like a GraphQL schema explorer - but based on the language runtime types + bindings, so you can also explore the builtin functions defined on strings, lists, etc.

#

and now it loads the Dagger module + dependencies in the cwd, just like dagger

dim smelt
#

Is :<foo> part of the dang syntax? Or a special repl think like .<foo> in dagger shell?

brittle wharf
#

oh yea not dang syntax, just a special repl command

brittle wharf
dim smelt
brittle wharf
#

yeah, and unfortunately Workspace was a common convention for LLM use case

dim smelt
#

(but I thought we namespaced module types though?)

#

If module foo defines type bar, doesn't that become type FooBar in the gql schema?

brittle wharf
#

oh, but in this case the module is literally called Workspace

#

so it doesn't get namespaced by the usual logic, because it's the module type itself

dim smelt
#

Ah the issue is the module name, I see

#

Eminent domain then

brittle wharf
#

yeah, there shouldn't be too many modules named 'workspace' anyway padme_right

coral kernel
dim smelt
#

heading back from dentist. I'm looking into full backwards compatibility: instead of error if a migration is needed, just a warning.

brittle wharf
#

https://github.com/dagger/dagger/pull/11874 is ready for review - just babysitting CI now (random lints from previously-skipped packages, + buildkit flakes that are already fixed on main will be fixed in the next week πŸ˜„)

dim smelt
brittle wharf
dim smelt
dim smelt
#

Oh damn it

#

I am just now realizing that I sent it in the wrong place

dim smelt
#

While claude churns away on that πŸ‘† , I'm going to check out your workspace-API branch @brittle wharf and try porting our toolchains to it

brittle wharf
#

also thinking I could/should just remove Workspace.install and .moduleInit from that PR

#

Workspace.findUp i imagine to be trivial to add already since that's already has a Host API

dim smelt
dim smelt
brittle wharf
#

nice. just pushed this:

+ Workspace.findUp
- Workspace.install
- Workspace.modInit
dim smelt
#

Ah shrinking the API footprint? makes sense

brittle wharf
#

yeah, those feel more suited for 11812 anyway, they just kind of came along for the ride

#

i punted on Workspace.search - it would be cool to skip the filesync, but it's a bit of a pandora's box (runtime dependency on client-side rg?)

dim smelt
#

Yeah I can always Workspace.directory(include:..., exclude:...).search() right? or just .glob(), whichever fits the situation

dim smelt
#

Thinking about blueprints... Isn't a blueprint just a module in the workspace config, with alias=true? (alias: current terminology to mean "mount this module's functions directly at the root, instead of namespacing them)

#

ha ha, I was struggling to find the right term to use in the config (alias? topLevel? noNamespace? main?). But maybe it's actually blueprint πŸ˜„

brittle wharf
dim smelt
#

Between toolchains and modules moving to a thin compat on top of workspaces: lots of complex and potentially slow code getting removed. Fingers crossed we get a slight speedup out of it!

dim smelt
#

First impressions porting a small toolchain (helm-dev) to workspace API @brittle wharf:

  • DX is strictly better. An explicit argument is more clear than pragma incantation
  • I can also split up code in utility functions at my leisure, without worrying about the absence of self-calls causing me to lose pragmas... I just pass the freaking workspace argument, and it works as intended. Huge win
  • Loading the module is super slow, even on warm cache. Maybe it already was before? See screenshot: 47sec, 19sec, 13sec. That seems crazy long for loading a super small module written in go

- The helm checks are not showing up in dagger check -l... I have no idea why πŸ€·β€β™‚οΈ duh, I was running it with stable dagger... facepalm

#

@brittle wharf I'm adding a compat bridge for legacy +defaultPath, configurable in config.toml. dagger migrate will set the compat field (legacyDefaultPath=true automatically. So we can keep the desired clean definition of +defaultPath for everyone else)

#

But also, we can probably deprecate +defaultPath completely down the road, no reason to not call dag.CurrentModule().something() since module always gets fully loaded, so no need to worry about the cache police

brittle wharf
#

PR is ready for another look (https://github.com/dagger/dagger/pull/11874)

#

also did the unconditional content-addressing for module functions change, nothing broke - neat

lofty shadow
# brittle wharf PR is ready for another look (<https://github.com/dagger/dagger/pull/11874>) * <...

I saw some secret-related failures in TestModule that are not familiar flakes: https://dagger.cloud/dagger/checks/github.com/dagger/dagger@ab4d1292e2495fcebbc7dcd41f39e51187c78dbc?check=testSplit%3AtestModules&listen=a291eb52ee102852&listen=226341e187df69db&listen=69166e0aa4c09b70&listen=8628b33e68bd1b21&listen=0034f8608ce1366d&listen=b2a4fadb78f00f6e&listen=60d76420fd1ead56#5d6eb2e65d742def

I wouldn't be shocked if the content addressed function parents were impacting those. Worth seeing if it's repro-able. It looks like the tests in question are setSecret (as opposed to secret providers), which is an utter nightmare when it comes to caching.

TBH if it's that, I'd probably just go back to the Workspace-only implementation for now. I funnily enough am doing battle with setSecret right now in the dagql cache work, so whatever I land on there might help this situation too, can revisit then.

Dagger Cloud

Browse and visualize Dagger traces.

brittle wharf
#

hm yeah "nothing broke" was a bit overconfident - these are also potentially interesting:

        Error:          Not equal: 
                        expected: "Hello from B"
                        actual  : "Hello from A"

Error: make request: input: test.upper set call inputs: convert arg "msg": failed to get deps for DynamicID: module not found: test [traceparent:972913fc2fbd2d3af91eac9afb5f12e4-54c015d7b46974ea]
lofty shadow
#

Error: make request: input: test.upper set call inputs: convert arg "msg": failed to get deps for DynamicID: module not found: test [traceparent:972913fc2fbd2d3af91eac9afb5f12e4-54c015d7b46974ea]
I actually managed to trigger that exact one in my e-graph PR too... weird. Must be triggering the same thing somehow.

Either way, definitely seems not worth blocking the PR on this, I'd just go back to the previous impl for now. I'll need to rebase on this anyways, can look into all together with the new egraph stuff

brittle wharf
#

pushed a revert + tidy

brittle wharf
lofty shadow
dim smelt
# brittle wharf PR is ready for another look (<https://github.com/dagger/dagger/pull/11874>) * <...

not sure what you meant by this https://github.com/dagger/dagger/pull/11874#pullrequestreview-3817138828 - do you know if there's something lacking for blueprints, or just seeking an explicit test?

Yesterday I realized that blueprints can be cleanly folded into the workspace system, similar to toolchains. So they'll have the same backwards compat / migration needs than toolchains. I implemented that in the workspace branch for both. Following that same symmetry, I was wondering if in your PR you'd want to also add support for workspace API in blueprints the way you did for toolchains.

I guess where the symmetry breaks, is that we use toolchains but not blueprints. So strictly from a dogfooding perspective, your PR only needs to touch toolchains.

So my comment was not so much a request, but a FYI so you can make an informed decision on where to draw the line in your PR

dim smelt
# dim smelt > not sure what you meant by this https://github.com/dagger/dagger/pull/11874#pu...

In technical terms, from old -> new:

  • toolchain in your dagger.json -> module in your config.toml with legacy-default-path: true for backwards compat of +defaultPath (to be manually removed when your module adopts workspace API and only uses +defaultPath to mean "my own files" (note: no way for the module to indicate it has made that change..)
  • main module in your dagger.json -> module in your config toml with blueprint: true
  • blueprint in your dagger.json -> module in your config toml with blueprint: true + legacy-default-path: true
brittle wharf
#

will merge on green

brittle wharf
#

merged! now to rebase...

brittle wharf
#

@dim smelt I'm confused by this Workspace.Rootfs field - I thought the idea was to skip an eager filesync for host dirs?

#

currently:

dagger HEAD(REBASING 109/135) *​​​ ≑1m9s
dim smelt
# brittle wharf <@488409085998530571> I'm confused by this `Workspace.Rootfs` field - I thought ...

Mmm "rootfs" is the term I chose to mean "the root of the sandbox inside which all paths are evaluated". Basically a new word for what we called "the module's context directory" kn modules v1. Implementation is supposed to be similar to modulesource context dir and reuse similar plumbing -> use the git root for remote workspace; use host session attachables for local workspace; all abstracted away cleanly. At some point the vibe coded implementation looked correct. but maybe it accidentally got reverted/broken when I focused on other parts (most recently bug fixes, DRY & backwards compat)

brittle wharf
#

yea right now it's a dagql.ObjectResult[*Directory] field populated by calling host.directory when local:

    if isLocal {
        // Local: Rootfs = host.directory(detected.Root)
        hostPath = detected.Root
        err := dag.Select(ctx, dag.Root(), &rootfs,
            dagql.Selector{Field: "host"},
            dagql.Selector{
                Field: "directory",
                Args: []dagql.NamedInput{
                    {Name: "path", Value: dagql.String(detected.Root)},
                },
            },
        )
        if err != nil {
            return nil, fmt.Errorf("creating rootfs directory: %w", err)
        }
    } else {
        // Remote: Rootfs = the cloned git tree passed in.
        rootfs = prebuiltRootfs
    }
dim smelt
#

oh yeah that's wrong... sorry I've become my worst nightmare: sloppy vibe coding CEO πŸ˜…

#

I did say "understand and reuse the same plumbing as contextdirectory, with abstraction over local va git sources" -> it made a very convincing case that it had done that

brittle wharf
#

lol np. i'll try to finish this rebase, but might bail and try going for a merge instead, if i turns into a minefield

#

(kinda already getting there, more code changed than i anticipated between the two PRs, some of which was refactored prior to merge, notably path arithmetic)

#

oh. ok. that was the last commit with conflicts.

dim smelt
#

I was going to say "agents are great at complicated merge conflicts" but now I worry I got conned πŸ˜…

#

I think agents will truly help track the conceptual parts of the merge, as a safety to make sure important changes don't get dropped

brittle wharf
#

now weighing whether a rebase or merging main into the PR is the lesser evil. nice thing about merge commits is all the conflict resolutions are in one place (the merge) rather than sneakily melded in haphazardly through the commit history

dim smelt
brittle wharf
#

@dim smelt pushed the merge - left the Rootfs brokenness for now so it's still a pure merge

#

for fixing the rootfs, it kinda seems like we just need to handle both states and hope the branching doesn't get out of hand: either it has a Rootfs Directory, or it's a virtual workspace backed by a host path

dim smelt
#

@brittle wharf I'm just back from my meetings. I have an uncommitted fix for "no main object" bug (handling of module "original name" vs. installed name was broken in my backwards compat commit). Will commit, then hopefully it will cleanly rebase on your merge

#

@brittle wharf then I can focus on the eager rootfs, try to fix my vibe-coded mess πŸ™‚ Unless you're already doing it

brittle wharf
#

go for it πŸ‘

dim smelt
#

Since I'll be off thursday-friday, consider the branch yours for the rest of the week - whatever you want to take over, take over

#

I like how fast we're moving and don't want to derail that πŸ™‚

dim smelt
#

I started a de-facto convention on the branch, feel free to try it also if you want:

-> I track in-progress work in hack/designs/<name>.md. This contains 1) context 2) design 3) task list with status

When I start a design, I tell claude to write that file - and NOT go in plan mode (if it enters plan mode, I exit it). That way we can actually talk about the design normally, without the weird plan mode UX that swallows the dialogue.

Then I tell claude to follow the task list, one commit by task, and atomically change the task list status in the same commit. That way I can always kill the agent and resume clean.

I'm sure you guys have equivalent workflows. Just wanted to let you know the interface for mine, so you can plug in πŸ™‚ If you look in hack/designs you'll see the current state of what I'm working on. Especially handy within a big PR, since there's so much going on

brittle wharf
#

@dim smelt One thing that'll help in your absence is confirmation of what commands we're committing to breaking. That'll help with fixing up the TestModule suite - a quick triage of those failures would go a long way if you have a sec! (should that test pass, or does it change to command XYX)

Dagger Cloud

Browse and visualize Dagger traces.

#

(nbd if no time, i'll make guesses, easy enough to adjust in the end if i go astray)

dim smelt
#

(pushed tentative fix to rootfs laziness)

dim smelt
#

About to get in a car for a few hours, will go over the failures

dim smelt
dim smelt
brittle wharf
#

Captain's log: wrangling with module tests, whittled down from 137 failures to 43 failures.

Here are some notes (not exhaustive, just the recurring themes):

  • [ ] recursive call detected
  • [x] missing --with-self-calls for dagger module init
    • TestSelfCalls
  • [x] dagger module init: source in ./foo/, dagger.json in .
    • many such cases
  • [x] dagger develop broke for unbundled SDKs
    • TestUnbundleSDK
    • [workspace 6da9f58a7] fix(cli): skip workspace modules when running dagger develop
  • [ ] missing LICENSE

The first checklist item is the broadest amount of failures - basically the aliasing system is wreaking havoc for a bunch of different cases, related to breakage I also hit in dang (the Dang.dang clobbering the entrypoint). Going back to the drawing board for that aspect - I don't think we can just yeet all the blueprint'd module's fields up to Query like that, there are just way too many scenarios where it collides either with its own functions, or with a dependency, or with core APIs, etc.

Current approach is to expose type Workspace { defaultModule: String } which returns either the blueprint module's name or the singularly loaded (-m or CWD) module's name, and have the CLI treat that as the entrypoint.

Overall trying to avoid making widespread changes to the tests, making backwards compat. changes where possible to bridge the gap, but when the dust settles we should take a step back and figure out which side should really change

dim smelt
brittle wharf
#

yeah

dim smelt
#

I feel dumb because I still don't undertand how aliasing breaks everything. I guess it's because some functions shadow others in the top-level query? But which functions are shadowing ("clobbering") which other functions exactly? Aliases module shadows core functions?

brittle wharf
#

it's confusing πŸ˜… I had Claude draw me a graph in one case

#

basically, in the happy path, module foo defining bar, you get type Query { foo, bar } where under the hood bar calls Query.foo.bar
breakage 1: define foo module with foo function -> this results in recursive call detected, because Query.foo calls Query.foo.foo.
I worked around this by aliasing the constructor as _blueprint_foo and having the aliases call that, but then hit
breakage 2: define dep in your module, install module named dep => your dep clobbers your dependency's constructor, so you can no longer call it
breakage 3: define a function called container in your module => well, now you've done it! container.from() will now call your container instead.

dim smelt
#

mmm, so the way it works today, aliasing can work if we forbid installing other modules (just the one aliased module) and we don't expose its constructor, just its functions directly?

brittle wharf
#

not sure i follow

#

is the end goal really to change the schema layout like that? feels like more trouble than it's worth but maybe i'm missing the upside

dim smelt
#

So far we've always erred on the side of making the CLI dumber when possible. So it seemed to make sense to just let the engine show the client what it's supposed to see, without the CLI having to participate

#

I didn't expect it would be so complicated

#

But, my previous point was, we're not trying to build a generalized schema aliasing feature either. We have a pretty narrow list of things that need to work

#

  1. Backwards compat with a project's "main module" when migrating its dagger.json to a workspace

  2. Handling -m gracefully

  3. Implementing blueprints --> If it helps, we can cut this requirement, which was my point earlier

#

Requiring that the client check for an entrypoint module, and honor it, seems like a workaround: now every client has to implement this, or they're broken. Plus it's an extra round trip.

dim smelt
brittle wharf
#

Should be possible, can investigate tomorrow thinkspin - one semi blocking question though, is what to do about the constructor? I don't think we can drop it entirely, even dagger call invokes it when you pass flags (dagger call --constructor-flag func --func-flag)

dim smelt
#

  1. Backwards compat -> you can set module configuration in the workspace config, and you can migrate off of using aliasing

  2. -m -> This one would be the most painful I guess, no workaround

  3. blueprint -> can configure module in the workspace config

#

But yeah that is a major downside of doing it purely in the schema

#

Maybe the engine could return a special header or something, as part of client connection handling? To make it more "special" than querying Workspace.something?

brittle wharf
#

one thing i'm wondering is if that's really simplifying the client, or just enshrining a very particular client's POV to all clients, not all of which agree

#

for example dagger shell and dang's REPL put you in a context that has your module, its dependencies, and core APIs all available

#

could always be a param, i suppose, just like we have Module.serve(includeDependencies) (edit: no it can't, since this is all pushed engine side, and that particular approach even assumes the core API is available, which we don't want)

brittle wharf
#

down to 18 (but i think actually 16) failures in the module suite πŸ₯²

  • pushed legacy module handling down into workspace.Detect so it's squashed as early as possible
  • support contextual git args for legacy defaultPath

the vast majority of the failures are just from LICENSE no longer being created by dagger module init, not sure if that was a conscious choice (vaguely recall there being some discussion on its merit)

brittle wharf
#

what seems reasonable is:

  • keep LICENSE generation for standalone modules (which should fix all the tests)
  • don't do it for workspace modules, since a) they're not meant for redistribution/reuse anyway, and b) they're likely covered by a repo-wide LICENSE already

edit: down to 2 failures with that fixed!

brittle wharf
#

trying out the Query hoisting revival now

lofty shadow
#

@brittle wharf was looking at the workspaces PR out of curiosity since the dagql cache updates are in the final stretches (https://github.com/dagger/dagger/pull/11856). We are gonna have A LOT of rebase conflicts to sort through lol

Fortunately my PR simplifies quite a bit of the caching aspects of module loading and other APIs in core/schema, so I don't think it's fundamentally in conflict. More like needing to reconcile two new-but-better worlds.

One question though: I saw that almost all of the toolchain-related code in schema/modulesource.go is deleted in your PR. I am currently working through a toolchain test failure that's gonna require reworking how toolchains are loaded... should I even bother? Are toolchains actually gone with your PR? Or did they just move somewhere else?

#

I guess the other relevant question is timing. I want my PR merged early next week. Not sure if that's the same as the workspaces PR or if it'll take longer

dim smelt
brittle wharf
#

hmm, i can buy that but having the core API available in a repl/shell feels pretty important for playing around, which is mostly what repls are for. will think on it, i do think it's a noble goal if we can make it work, only really have practical concerns

#

claude made progress on an alternative implementation that has the major tests passing at least, I pushed it to see a full CI run and it looks plausible (17 module test failures, some of which are constructor focused which makes sense). need to take a closer look still, i had it working away while out to anniversary dinner and came back to a confident sounding commit

#

one cute option could be putting all of core beneath its own entrypoint. this is the sort of fun stuff i wouldn't mind diving into dagql to support (if it doesn't already). the main risk might be caching, with different IDs hitting the same APIs, but maybe that's where something something egraphs come in

brittle wharf
dim smelt
lofty shadow
dim smelt
#

I initially had a hideCoreModule client param but that turned out to be a bad idea πŸ™‚

dim smelt
#

(we kept backwards compat though; toolchains in a dagger.json get automatically transformed into modules in a workspace)

dim smelt
#

@brittle wharf maybe a quick sync about the aliasing thing later?

brittle wharf
#

@dim smelt i'm around if you're around!

dim smelt
#

15mn?

brittle wharf
#

sgtm!

dim smelt
#

@brittle wharf sorry 5 more minutes... if you're still around

brittle wharf
#

np, still around

dim smelt
#

(kids are still off school, and they have many questions)

dim smelt
#

UI / telemetry glitch -> looks like nothing is happening

Screenshot_2026-02-23_at_15.34.49.png
#

Same session after cranking up verbosity to 3

Screenshot_2026-02-23_at_15.35.18.png
dim smelt
#

As I type this, I realize: how do we cleanly handle workspace injection for nested clients?

brittle wharf
dim smelt
#

@brittle wharf I was thinking we could have a top-level field in config.toml called "entrypoint" eg.

entrypoint = "dagger-dev"

[modules.dagger-dev]
source = "..."

And corresponding api: Workspace.entrypoint(): String!

brittle wharf
#

to replace blueprint = true? sgtm in shape, fixes the 'what if i set multiple to true' question

dim smelt
#

yes exactly. Still slightly worried about fragmentation ("this command honors entrypoint; this one doesn't") but at least the concept is easy to explain

#

And wondering if codegen should honor entrypoint

dim smelt
#

@brittle wharf I'm testing workspace API. Is there something about Workspace.directory() that would cause it to be copied read-only in a container? I'm getting unusual "permission denied" in a seemingly trivial container, just base image + workspace directory copied in

brittle wharf
#

doesn't ring any bells hyperthinkspin

#

it just calls Host.directory under the hood (assuming host workspace)

dim smelt
#

It's from within a dev engine too, so maybe it's something related to nesting? WIll try to repro

dim smelt
#

Mmm super weird though

#

Oh I get it. Container.withDirectory() doesn't honor the current user configured in the container image (unlike Dockerfiles I believe). Which explains this:

  • dagger -M -c 'container | from tmknom/markdownlint | with-directory . $(directory) | with-exec touch foo' -> permissions denied
  • dagger -M -c 'container | from tmknom/markdownlint | with-exec touch foo' -> OK
dim smelt
#

@brittle wharf manual testing of workspace branch. It looks like entrypoint/aliasing is too aggressive: it only shows the designated entrypoint, and not the other modules. In 0.9.11 dagger functions will show a mix of toolchains + main module's functions

brittle wharf
#

noted, will check when i get back to it unless you or claude figure it out

dim smelt
#

Also there might be an engine crash when listing checks... Will try to repro more precisely

restive swift
dim smelt
#

Mmmmm.... maybe a big module DX win... @brittle wharf the Betclic team reminded me that real-world juggling of npm cache volumes is a major issue...

  • Can't share the cache between concurrent writers
  • Global lock across all apps is a major bottleneck
  • No clean way to namespace cache volumes by app

--> But if everything is relative to the workspace, now we have something to namespace by app: the path in workspace πŸ™‚

Will try in our docusaurus module

brittle wharf
#

we should at the very least stop merging anything that touches toolchains imo. it's just rearranging the deck chairs on the titanic. literally the entire test suite is gone

dim smelt
#

Argh.

#

@brittle wharf want to spread the load?

brittle wharf
#

i'm done with the conflicts at least, more worried about if we keep that pace. will push soon (done)

dim smelt
#

@brittle wharf I'm all for a well-scoped freeze - what should the scope be?

brittle wharf
#

toolchains, any ModTree stuff unrelated to fixing native-ci issues (so 11902 above for example is fine), and honestly anything nonessential that touches modules (11908 for example could probably wait until after just to avoid friction)

dim smelt
#

11908 was a user request - but yeah makes sense

dim smelt
#

@brittle wharf what task can I safely pick on the branch?

#

or maybe the opposite - what has a lock on it?

#

I could take the "aliasing is too strict" thing

brittle wharf
#

@dim smelt wanna quickly sync and go over the commits made while you were out, or do you trust them enough to just fix anything else in post?

dim smelt
#

(to make a plan)

brittle wharf
#

lol k, can hop in chat whenever

brittle wharf
#

@dim smelt went over the commits myself, here are the notables:

dim smelt
brittle wharf
#

cool, I'll look into that now

dim smelt
#

CURRENT STATE: workspace.Detect finds a dagger.json and effectively does a 'legacy migration' - so we IGNORE the outer defined workspace

Here is the intended workspace detection algorithm:

  1. Primary: Find-up .dagger
  2. Fallback: find-up dagger.json, filtering for "migration triggers": either existence of toolchains, or source path that is not . (dagger.json with no migration triggers are ignored)
  3. Fallback: find-up .git.
  4. Fallback: use workdir

So the intended design is not to unilaterally ignore outer defined workspaces whenever a dagger.json is found.

  • Explicitly defined workspace .dagger always takes precedence
  • dagger.json is only considered as a fallback for workspace if it has the migration markers
  • NOT IN SCOPE: handling of CWD modules which is orthogonal to workspace detection

QUESTION: what would NOT ignoring the outer workspace look like? What is the end goal?

The end goal is that workspace detection always works the same, and is easy to explain and understand.

Handling of CWD modules is a superficial convenience that doesn't need to alter how workspaces are loaded. It's similar to dagger -m -> a customization of which modules are loaded in the workspace

brittle wharf
brittle wharf
# dim smelt > CURRENT STATE: workspace.Detect finds a dagger.json and effectively does a 'le...

OK, so my Detect logic needs to change - looking into that now. Basically:

  1. treat a legacy (toolchains, blueprints, or non-. source dir) dagger.json as if it were migrated as .dagger/
  2. otherwise, treat CWD dagger.json as if it were passed as -m, but that's an orthogonal code path that interacts with above, not overriding it (strikethrough edit: too prescriptive, think it might still be nice to consolidate into Detect so it's all in one place, TBD)
dim smelt
#

But yeah, legacy dagger.json and CWD dagger.json are completely orthogonal. You could have a CWD dagger.json in an otherwise perfectly regular .dagger/config.toml (in fact that will happen all the time)

dim smelt
# brittle wharf <@488409085998530571> went over the commits myself, here are the notables: - `da...

engine serves dependencies, too
https://github.com/dagger/dagger/pull/11812/changes/a488ba2bc59e20ff8653cb1410e9d96e1f6874e9
needed for TestInterfaces otherwise concrete type is not loaded for caller
but I'm slightly sus on this change

Oooh I didn't think of interfaces...

It seems that "serve dependencies" is really a combinatio of 2 things: 1) add the dependency's types, 2) add the dependency's constructor to the root query. Maybe in this case we want to do one without the other? It makes sense to have the dependency's type, but it's weird if dagger functions also shows my module's dependencies constructor right?

brittle wharf
#

yeah that sounds ideal i think?

dim smelt
#

So in workspace-land does it mean:

  • Load every module from the workspace (including ExtraModules and CWDModule if relevant)
  • Also load every dependency of those modules (but only expose the types)
  • Make sure all types are cleanly namespaced by module name
brittle wharf
# dim smelt I think wherever `-m` is handled (handling of module-related client params) is p...

so, -m pretty clearly has to be at least somewhat handled by the CLI considering it's a flag, but maintaining support for CWD dagger.json in "the same spot" seems like it would mean re-implementing FindUp logic - we can't just use Host.findUp because now this happens before we even have a client

the reason I was considering pushing it into Detect is we already do a findUp there, and perhaps we just need a way to influence 'extra modules' from there too

also: when the user uses -m some/other/ref while they happen to be in/under a dir with dagger.json, we expect that to ignore the dagger.json right?

#

(doing the findUp CLI-side is fine btw if we still want that in the end, just surfacing nuance as I find it)

dim smelt
#

Yeah the CLI should not have to do any find-up (and I don't think it does today?)

brittle wharf
dim smelt
#

To be clear when I said "handle -m and CWD in the same spot" I meant in the same spot engine-side. So I should have said "handle ExtraModules and CWD in the same spot"

--> As far as I know this is how I had already implemented it?

#

If I remember correctly, there's a "pending modules" system to avoid the deadlock between client connection, loading modules, establishing session attachables

#

did that not work properly?

brittle wharf
#

hard to say, it was a bit of a blur getting down from 168 failing tests πŸ˜›

#

that system is still there, but I think that specific part got lost in the mix while pushing more down into workspace.Detect

#

(working on fixing this up now)

dim smelt
#

maybe we should start a fresh set of tests that are workspace-native? Not to replace the existing tests, but to remove some of the pressure on the workspace implementation by clarifying the purpose of each suite?

brittle wharf
#

we do have the workspace suite, and I don't find the module suite to be particularly confusing, but more tests with clearer purpose is never a bad idea πŸ˜›

#

the workspace suite is more focused on the Workspace API, though, we're probably missing coverage for a bunch of specific CLI / code organization scenarios, i agree

dim smelt
#

(back from dentist)

dim smelt
# brittle wharf <@488409085998530571> went over the commits myself, here are the notables: - `da...

dagger module init in empty directory inits standalone module in that directory
https://github.com/dagger/dagger/pull/11812/changes/90489000b01b137ba2336159c98f94d6814e65dd
This is to keep a bunch of tests passing - otherwise they initialize a workspace and install the module into .dagger/modules/foo/ and config.toml

Instead of "is workdir empty", we should use "is there an initialized workspace".

  • If there's an initialized workspace -> create in .dagger/modules/ regardless of what's in workdir
  • If there is no initialixed workspace -> create in workdir

Will keep things more consistent & predictable for users inside an initialized workspace.

#

@brittle wharf do you think we should do another spinoff from workspace, and get the plumbing merged first, without any of the UX? Take advantage of backwards compat / migration, to convert everything to workspaces internally, but without any of the user-facing changes?

#

It would take a lot of discipline to protect the plumbing from the legacy UX

#

but if we do it, it would be just a couple weeks difference between the two PRs I'm guessing

#

wdyt, worth it?

brittle wharf
# dim smelt did that not work properly?

update: it did work properly lol, I brought it back to how it worked before and all the Module tests pass. Must have been a red herring, or maybe the things I fixed were fixed by another change upstream

brittle wharf
brittle wharf
#

@dim smelt what do we expect for this test? thinkspin https://dagger.cloud/dagger/traces/de234c0102206e0f628927338c656709?span=76572aaa5affdf6d

Currently, the dagger module init calls are also initializing the workspace, which goes against #1468070450524459029 message but that's beside the point I think; these are the Workspace tests so that more implies they're just missing an explicit dagger init.

Right now the test asserts that when you run with your workdir in a nested module, dagger functions does NOT show the workspace modules - only the current module's functions, blueprint-style.

But I think the test is wrong based on our conversation - it should show all the workspace modules, and the CWD/blueprint module's functions

brittle wharf
#

pushed - dagger functions and dagger call shows other modules in addition to the blueprint, updated the test

dim smelt
#

(sorry missed those messages... catching up)

dim smelt
# brittle wharf <@488409085998530571> what do we expect for this test? <a:thinkspin:109077493053...

But I think the test is wrong based on our conversation - it should show all the workspace modules, and the CWD/blueprint module's functions

Yes I think so, unless it causes genuine UX issues to do it this way..

I think if we don't do it this way, it can become confusing and frustrating for "workspace-native" users. Example:

  • I have my workspace setup (~/repo/.dagger)
  • I init a module in my workspace (~/repo/.dagger/modules/api)
  • For developing my module, I initialize a sub-workspace with exactly the modules I need to develop my modyle (~/repo/.dagger/modules/api/.dagger)
  • For mysterious reasons, when developing my module, my specialized workspace doesn't work...
brittle wharf
#

plus dagger module init only doing workspace stuff when the workspace is already init'd (instead of 'is dir empty?')

dim smelt
#

Nice. I've been drafting a detailed lockfile design doc, will push on the branch

#
  • will attack multi-env support in config
dim smelt
#

pushed a first pass at lockfile implementation

brittle wharf
#

nice. merged main back in so CI can run

dim smelt
#

@brittle wharf I noticed in main, modules can query Workspace.root which seems to leak the workspace path relative to the host fs. Two problems with that 1) the workspace might be remote, 2) I'd rather not leak this info to modules

--> OK to remove?

brittle wharf
#

sgtm, I left things pretty open expecting to whittle away at things

#

clientID is also exposed, might go away later but figured it could help with troubleshooting while we work on it

dim smelt
#

On the other hand, it would be AWESOME to give modules some sort of "stable identifier" for their workspace, to eg. namespace their cache volumes off of

#

--> Finally we can solve the nightmare that is NPM cache volumes

#

With workspace id + app path in workspace, I can safely create a different node_modules cache per app, and set its concurrency to LOCKED -> no corruption on concurrent writes, and no global lock across all apps ...

#

cc @unborn scroll πŸ‘† πŸ™‚ I know you suffered from this. The nightmare will soon be over

topaz seal
#

I wanted to have a look at using generate for dagger develop, based on workspaces to avoid duplicate efforts.
But on workspace branch both dagger check -l and dagger generate -l don't work.
Was it working before or not at all? (to understand if that's a new problem introduced recently or not)

dim smelt
topaz seal
# 
Error: checks from module "dagger-cli": "dagger-cli": no main object

I just ran dagger migrate on a clone of dagger/dagger then dagger check -l at the root

dim smelt
#

new error

topaz seal
#

ok, I can have a look at the recent changes and find why

topaz seal
dim smelt
#

It's not fully formalized as a skill, I wrote a few lines in AGENTS.md (but didn't get merged). I regularly tell the agent to keep the design doc updated, so that another engineer can cleanly take over. I also make sure the task list stays up to date, and I say "when you complete a task, change the status in the tasklist in the same commit". It's ad hoc and imprecise, but mostly good enough

#

Definitely NOT hand written πŸ™‚

#

basically it's my persistence layer to keep the agents as stateless / ephemeral as possible

topaz seal
#

I've pushed a fit to workspace branch. With a similar hack/design/bugs file. I don't know if we want to have that for all of the bugs we are fixing, but that's interesting. https://github.com/dagger/dagger/blob/006094c32c727452c055f8017e41d6523746c2bc/hack/design/bugs/workspace-checks-no-main-object.md

GitHub
dagger/hack/design/bugs/workspace-checks-no-main-object.md at 00609...

Automation engine to build, test and ship any codebase. Runs locally, in CI, or directly in the cloud - dagger/dagger

topaz seal
#

FYI I pushed the workspace branch with:

GitHub
dagger/hack/design/bugs/workspace-darwin-build-failure.md at worksp...

Automation engine to build, test and ship any codebase. Runs locally, in CI, or directly in the cloud - dagger/dagger

topaz seal
#

I pushed a first design draft and implementation plan to use dagger generate instead of dagger develop. I'll iterate more on them, but it might be interesting (maybe should have I created a different branch?)

topaz seal
#

Quick overview of the current progress on dagger generate for modules:

$ dagger module init --sdk=go plop
βœ” connect 0.0s

βœ” Workspace.moduleInit(name: "plop", sdk: "go", license: "Apache-2.0"): String! 7.2s

Created module "plop" at /root/src/current/.dagger/modules/plop
Installed in /root/src/current/.dagger/config.toml
$ cd .dagger/modules/plop
$ ls
dagger.json  main.go
$ cat dagger.json
{
  "name": "plop",
  "engineVersion": "v0.20.2-20260310101303-dev-a99318c0ce55",
  "sdk": {
    "source": "go"
  }
}
$ cat .dagger/config.toml
[modules.develop]
source = "sdk:go:develop"
$ dagger generate
βœ” develop:generate 24.0s 2Γ—β£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£·β‘„β£Ώβ£Ώβ‘€β£Ώβ£Ώβ‘„

Apply changes?
dagger.gen.go                 +233
go.mod                        +53
go.sum                        +95
internal/
internal/dagger/
internal/dagger/dagger.gen.go +17103

6 files changes, +17484 lines
         Apply   Discard

One thing to notice. We have go, typescript, etc as shortcuts for embedded (or not) sdks.
I was wondering what to put for the "generate/develop" module. So I thought about sdk:go:develop. Using : to express this is not a real path (I mean sdk/go/develop is a path, but only if we think about the dagger sources)
And I think it express quite good the intent behind, what this module is for.
It maps to sdk/go/develop relative path. We often have a sdk/LANGUAGE/runtime module, to have sdk/LANGUAGE/develop can make sense.
But all that is up to discussion.

This sdk:go:develop module is a pretty short dang one, with a pre-built codegen binary.

It still needs some refinements and a bit of work, but it's working quite well for now. One of the thing in particular is to work on the migration path and other SDKs.

topaz seal
#

And that also handle the init. So if the module directory only contains .dagger/config.toml and dagger.json as shown above, everything else will be created.
One of the really good aspect here is with the Changeset integration on the CLI it's very explicit.
One part still missing is the .gitattributes/.gitignore files

$ dagger generate
βœ” develop:generate 24.0s 2Γ—β£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£Ώβ£·β‘„β£Ώβ£Ώβ‘€β£Ώβ£Ώβ‘„

Apply changes?
dagger.gen.go                 +233
go.mod                        +53
go.sum                        +95
internal/
internal/dagger/
internal/dagger/dagger.gen.go +17103
main.go                       +37

7 files changes, +17521 lines
#

πŸ‘€
So I added this plop module to the dagger/dagger workspace.
If I dagger functions from the root of the workspace, I can see it. If I dagger call plop container-echo ... it's working as expected.
But dagger functions plop it failing with Error: no function "plop" in type "DaggerDev"
Is that a known issue? Is someone working on it? (to avoid conflict / duplication of work)
If not, happy to have a look

dim smelt
brittle wharf
#

hmm don't think so, the clobbering is gone, that looks like it's trying to call daggerDev.plop instead of Query.plop? haven't familiarized myself with the new dagger call <ref> model yet

topaz seal
#

I can have a look at that. The call works, it's only the functions that doesn't. I don't think that's a big issue, was mostly wondering if it was a known one.

dim smelt
#

@topaz seal for develop->generate, maybe a different branch? wdyt

#

we're already struggling to get this one merged, I worry that we're adding faster than we're stabilizing

topaz seal
#

Go is working, I'm looking at Typescript and Python, and need to ensure it still compatible with the old world so other SDKs can still work.

dim smelt
#

@topaz seal that branch is for the discussion we had in SF?

topaz seal
topaz seal
#

Just thinking about modules migration with dagger generate instead of develop. To migrate from develop to generate doesn't really change how that works. It's mostly a change on how the generation function is called. So maybe this can ve generic. Like to provide a way for SDKs to work using generate without requiring changes at the SDK level. A kind of module that will ger the required info to then call the existing codegen from the module SDK.
That way we can imagine to have this change as minimal as possible on SDKs, we don't need to update them all. But still we do the change with workspace for each module and the removal of dagger develop command.
Not sure if that looks clear as I'm cooking at the same time πŸ˜… but I think we can find a nice path so we change the UX with as less changes as possible (as this less is already big)
I'll explore that asap.

dim smelt
#

I'm giving a try at spinning out the workspace plumbing + backwards compat in its own PR, with as little new UX as possible.

topaz seal
# topaz seal Just thinking about modules migration with dagger generate instead of develop. T...

I have a first POC working that way. A generic sdk:compat:develop module that will use the sdk.source from dagger.json to run the existing codegen. That way, we can have a fully compatible layer, meaning we can move to a removal of dagger develop and only use dagger generate without to rework all SDKs, and handled by the dagger migrate command.
Then each SDK can work on providing a proper develop module.
And I'm also thinking about a way to move then from the generic compat module to a SDK specific one, I think it can be achieved using dagger generate again πŸ™‚

topaz seal
#

@dim smelt what about a dagger module list (or dagger module --list but as we have dagger module init, dagger module install...) to just list the installed modules in the workspace?
Nothing fancy, just displaying in a table the name (and * if it's aliased) and the source. Nothing more.
I was thinking about that as a quick way to know what's available without to load any module, look at functions, or at the config file.

dim smelt
topaz seal
# dim smelt love it 
$ dagger module list
βœ” connect 0.0s

βœ” Workspace.configRead: String! 0.0s

Name             Source
changelog        ../toolchains/changelog
ci               ../toolchains/ci
cli              ../toolchains/cli-dev
dagger-dev*      modules/dagger-dev
docs             ../toolchains/docs-dev
elixir-sdk       ../toolchains/elixir-sdk-dev
engine-dev       ../toolchains/engine-dev
go               ../toolchains/go
go-sdk           ../toolchains/go-sdk-dev
helm             ../toolchains/helm-dev
installers       ../toolchains/installers
java-sdk         ../toolchains/java-sdk-dev
php-sdk          ../toolchains/php-sdk-dev
python-sdk       ../toolchains/python-sdk-dev
release          ../toolchains/release
rust-sdk         ../toolchains/rust-sdk-dev
sdks             ../toolchains/all-sdks
security         ../toolchains/security
test-split       ../toolchains/test-split
typescript-sdk   ../toolchains/typescript-sdk-dev

It's currently reading the config file on client side (asking the engine the config, then parsing the toml inside the CLI).
I don't know what to think about that. I did it that was because it's just easy, but it means there's a small part of the toml logic that is on the client side.
Would it be better if I instead add a moduleList function in the workspace schema? I wasn't sure it's very useful and didn't wanted to add extra functions if we can avoid it, but I can quickly do it if that sounds better.

#

Also, the source is raw path, so relative to the config.toml. Should I make them relative to the workspace root instead? Or that doesn't matter?

dim smelt
brittle wharf
topaz seal
topaz seal
# 
Name             Source
changelog        toolchains/changelog
ci               toolchains/ci
cli              toolchains/cli-dev
dagger-dev       .dagger/modules/dagger-dev
docs             toolchains/docs-dev
elixir-sdk       toolchains/elixir-sdk-dev
engine-dev       toolchains/engine-dev
go               toolchains/go
go-sdk           toolchains/go-sdk-dev
helm             toolchains/helm-dev
installers       toolchains/installers
java-sdk         toolchains/java-sdk-dev
php-sdk          toolchains/php-sdk-dev
python-sdk       toolchains/python-sdk-dev
release          toolchains/release
rust-sdk         toolchains/rust-sdk-dev
sdks             toolchains/all-sdks
security         toolchains/security
test-split       toolchains/test-split
typescript-sdk   toolchains/typescript-sdk-dev
dim smelt
brittle wharf
#

will take a swing at it if i have downtime - looks small

dim smelt
#

Yeah it's still WIP

dim smelt
#

now ripping out lockfile (thought it was needed for plumbing PR but actually can move to porcelain)

brittle wharf
#

nice. i've been dogfooding Workspace from an LLM perspective - added Workspace.glob, Workspace.exists, Workspace.search. huge wins so far, the Doug + Dev modules feel a lot clearer and don't have to be re-initialized with a new snapshot on every turn

dim smelt
#

oh man I can't wait to dagger install doug in any project

brittle wharf
#

Workspace.search is a little interesting - tries rg client-side, falls back on grep, so the main question is portability/compatibility - hopefully that's enough? that aside, feels super worth it

dim smelt
#

also maybe dagger install claude ? πŸ€”πŸ€”πŸ€”πŸ˜‡

brittle wharf
#

oh also I really want withMountedWorkspace now

#

it's perfect for MCP servers

#

ship it pls πŸ™‚

dim smelt
#

ha ha coming soon πŸ™‚

brittle wharf
#

thinking out loud: for LLM I think we may want "Workspace + rolling Changeset" - a workspace based on a directory on the host, but with a Changeset that keeps growing and is transparently applied to all APIs (somehow).

either that, or a way to sync changes back to a workspace eagerly, but that feels less cool (now you need to keep agents from stepping on each other's toes) instead of having sandboxed agent trees.

the problem: when Doug does an edit, its grep tool won't see its changes, because Doug's edit just returns Changeset - changes aren't applied until the user presses ctrl+s.

#

'somehow' is doing a lot of work there, there could be another way of looking at it altogether notsureif(like maybe syncing back is the answer after all, not completely off the table)

dim smelt
#

: when Doug does an edit, its grep tool won't see its changes, because Doug's edit just returns Changeset - changes aren't applied until the user presses ctrl+s.

Feels like we're witnessing the collision of 2 worlds: pure chainable functions vs. traditional syscalls with side effects. How do we reconcile these 2 worlds in our DX?

#

(I remember we discussed this hypothetical future, where there's 2 different ways to do everything: return the artifact, or call (at the time) Env.export() or whatever

Now with Workspace, and the implementation being much further along, the problem is much more concrete

#

The good news: with explicit Workspace arguments, it's much easier to split up a big top-level function with side-effects, into smaller pure functions that return this or that artifact. (Was basically impossible with argument pragmas)

topaz seal
#

Last iteration. I'm good to push it to the workspace branch but as we split in multiple PRs I wonder if that's the right place or if I should keep it on the side for now

image.png
brittle wharf
brittle wharf
dim smelt
#

@brittle wharf going to need your help on moving towards merge

#

I'm trying to get this plumbing spin-off ready, but now having doubts about whether it was worth it. Going over it now

brittle wharf
#

eh my initial instinct on seeing the PR was positive at least, happy to help

#

(despite saying it might not be worth it last time you asked)

dim smelt
#

I asked the agent to take a pass at remaining test failures, with the goal of passing the original main tests (makes sense since we target backwards compat). But it looks like it took it too literally, and reverted big chunks of the implementation to pre-workspace - especially check. I'l inspecting the damage

brittle wharf
#

if we can get all the checks passing on a significant amount of changes that's worth it on its own

dim smelt
#

yeah that was my hope

#

Let me revert the last dumb changes. I think the core had promise

topaz seal
#

I pushed two changes to workspace branch:

  • dagger module list as shown above
  • a fix so that dagger functions MODULE works as expected (dagger call MODULE --help was already working, it was only about functions)
#

I have one more change, but I'm not entirely sure if that's the right thing to do.
When running dagger functions MODULE it loads everything. All the modules. So I have a version where it load the blueprint modules and a module that would have the same name as the MODULE we are focus on. Both because at this time we don't know for sure if it's a module or a function of one of the blueprint modules.
This gives interesting results (dagger functions python-sdk from 15s to 10s -> see attached screenshots, state was for both without cache but dang SDK loaded) but I'm not entirely convinced by the changes, especially it adds a FocusModule field to ClientMetadata. It works, but I'm not sure the design is great and I fear a bit it complexify even more the possibilities (as we already have things like SkipWorkspaceModules).
Also this only matters if we haven't already ran a dagger functions for instance, that will already load all modules from the root. If so, the optimisation is kind of useless.
That's why I haven't pushed it to the workspace branch for now and here is the corresponding commit.
Let me know what you think.

dagger-functions-previous.png dagger-functions-modules-previous.png dagger-functions-new.png dagger-functions-modules-new.png
GitHub
perf: only load targeted workspace module when function path is spe...

…fied

When a function path is provided (e.g. dagger functions python-sdk or
dagger call python-sdk --help), only load blueprint modules and the
module matching the first function name instead ...

brittle wharf
#

@topaz seal quick question: where do you put your worktrees?

topaz seal
#

my dagger/dagger clone is inside ~/dev/src/github.com/dagger/dagger.
The associated worktrees are at ~/dev/src/github.com/dagger/dagger-worktrees For instance ~/dev/src/github.com/dagger/dagger-worktrees/workspace

At start I wanted to put them inside dagger/dagger but I've read several time this is not the best thing to do.

The directory name is the branch name (that's just a convention) and I have several aliases that allows me to quickly create a worktree for a branch or a PR, open a new dedicated tmux session, launch claude inside. That way most of the time I just daggerdev MYBRANCH and I'm done.

πŸ“Ž daggerdev.sh
brittle wharf
# brittle wharf for the coding agents case, it's really tempting to reach for `git`: agents comm...

ok cool. i'm investigating having agents able to automate them, related to this - representing branches of work as worktrees just makes a ton of sense because then Workspace.search, glob etc. can just run from those checkouts, and the user can keep an eye on them by just hanging out in the worktree directories. it came to a similar conclusion, of putting them beside the original repo, not inside of it (which tends to confuse editor configs and risks getting synced up multiple times with bad filtering)

#

i'll tell it to do ~/src/foo-worktrees/<branch> for ~/src/foo` and see how it goes

dim smelt
topaz seal
dim smelt
topaz seal
#

I wonder if the issue wasn't because of the alias -> blueprint
The .dagger/config.toml was still using alias = true but the engine is using blueprint. If I'm right I changed it.
So the aliased/blueprinted functions are now visible at the root of a dagger functions, the daggerdev is not displayed anymore and all the other modules are visible

dim smelt
#

@topaz seal did you remove blueprint in our config file then?

topaz seal
#

confused_dog Ok, I think I missed something then.
No I changed alias to blueprint in the config file.
I thought the opposite, that we were moving to blueprint instead of alias, at least that was my understanding as the config structs only referenced blueprint and not alias, even if there's the autoAlias function somewhere
So are we going back to alias = true? Or something else?

dim smelt
#

@brittle wharf how warm is your cache on this part?

brittle wharf
#

the last thing i remember is realizing alias = true should become blueprint = true, which turned into Workspace.defaultModule plus a planned tweak to the config to make it clear that you can only have one. and that we needed to make it work without actually merging the 'default module's fields into the schema Query, because that caused tons of issues

topaz seal
#

(asking Claude to read the thread and explain me what was the status on it πŸ˜… )

dim smelt
# 
Current state / consensus

  The consensus from the thread is:

  1. Schema-level aliasing is dead β€” too many breakage vectors, abandoned
  2. entrypoint = "module-name" at the top level of config.toml was the agreed direction (Feb 24),
  replacing per-module blueprint = true
  3. CLI handles it β€” Workspace.defaultModule returns the entrypoint name, CLI prepends it to queries
  4. But it was never fully implemented/landed β€” the config structs still reference blueprint,
  there's an autoAlias function floating around, and nobody cleaned up the gap between the agreed
  design and the actual code

  The confusion in today's messages (eunomie switching alias β†’ blueprint, you not remembering) stems
  from this incomplete transition. The design decision was made, but the code wasn't fully updated to
   match.

  Bottom line: You and Vito agreed on entrypoint as the concept and config keyword, with CLI-side
  handling via Workspace.defaultModule. But the implementation still has remnants of blueprint/alias
  that need to be reconciled.
#

@topaz seal πŸ‘†

#

Let me check the design doc in the branch. It should always be up to date. If it's not, it's my bad

topaz seal
#

Thanks. I probably have things to change to use the Workspace.defaultModule then.
Not sure I'll have time today for the change, but I can do that on Monday (to align all the things I've pushed, and the other branch I have)

brittle wharf
#

ah yea that all sounds right guy_fieri_chef_kiss

dim smelt
#

@topaz seal let me double check the state of that "defaultModule" approach. I remember we were not quite done bikeshedding it (I was worried about fragmentation, because a client can't just connect - they have to know the defaultModule dance)

#

@brittle wharf @topaz seal

  The "entrypoint" / client fragmentation problem

  Your concern (raised Feb 24 and again just now) is:

  If aliasing is handled CLI-side via Workspace.defaultModule, then every client has to "know the dance" β€” query
  Workspace.defaultModule, then prepend it to every query. A client that just connects and introspects the schema won't see the "right"
   thing. This creates fragmentation:

  - dagger call has to know about it
  - dagger functions has to know about it
  - dagger shell has to know about it
  - dagger check has to know about it
  - dang has to know about it
  - Any third-party client has to know about it
  - Each command might implement it slightly differently (or not at all)

  You said on Feb 24: "Still slightly worried about fragmentation ('this command honors entrypoint; this one doesn't') but at least the
   concept is easy to explain"

  And just now: "I was worried about fragmentation, because a client can't just connect - they have to know the defaultModule dance"

  The tension

  There are really only two options, and both have downsides:

  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚             Approach              β”‚              Pro              β”‚                             Con                             β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ Schema-level (engine hoists       β”‚ Clients "just work" β€”         β”‚ Causes recursive calls, dependency clobbering, core API     β”‚
  β”‚ functions into Query)             β”‚ connect, introspect, done     β”‚ shadowing β€” Vito documented 3 breakage categories           β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ Client-level (CLI queries         β”‚ No schema breakage            β”‚ Every client must implement the dance; fragmentation risk   β”‚
  β”‚ defaultModule and prepends)       β”‚                               β”‚                                                             β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

  The earlier attempt at a middle ground β€” a single exclusively-served entrypoint module (Feb 20: "we could simply support a single
  entrypoint module, that would be exclusively served - no other modules allowed") β€” was rejected because it would drop constructor
  support and break -m.

  Unresolved question

  Is there a way to make the engine handle it without schema stitching? For example:
  - A special connection-level parameter that tells the engine "resolve bare function names against this module first"
  - A query-routing layer below the schema that redirects, rather than merging types
  - Keeping full schema but marking the entrypoint module distinctly in introspection metadata so clients can discover it without a
  separate query

  This is the open design question. Want to hash it out here, or take it back to the thread with Vito?
#

We should seriously consider just plugging a designer agent straight into discord πŸ™‚

dim smelt
#

For the record, today I'm working on 2 blockers:

  1. Entrypoint module (formerly blueprint=true, alias=true..) as discussed above
  2. Paths in workspace API

I will make sure these blockers are resolved consistently across workspace and workspace-plumbing; and I will make sure the design (both the decided and undecided parts) is clearly and consistently documented in design docs + PR descriptions.

Then:

  1. I hope to get tests passing workspace-plumbing, and from there: merge
dim smelt
#

@brittle wharf do you think codegen should honor entrypoint, or not?

  • Pro: I like the consistency (every client sees the same thing)
  • Con: when you change your workspace config to change the entrypoint, you break your generated clients (dag.Foo().Bar() must be changed to dag.Bar(), or vice-versa)
brittle wharf
#

I don't think it can, even if we wanted to - you'd run into the same problems with nested fields colliding

dim smelt
#

I'm also playing with the idea of making the core module more real, (dag.Core().Container()), and subject to the same entryoint rules. Just would be set to be an entrypoint by default. Could be a nice simplification of the model, less special cases

brittle wharf
#

hmm that could change the equation a bit, yeah

#

personally i don't have much motivation in applying the same sugar to codegen

#

feels like the juice isn't worth the squeeze. i still prefer to think of it as a CLI UX layer on top of drier primitives (API + codegen)

dim smelt
#

It kind of breaks the experience if eg. you want to build a dependency in your test code - in the CLI that's dagger call build but in the code it's dag.Foo().Build()

#

If it's not worth it in code, it kind begs the question of whether it's worth it in the CLI

#

(I guess we need it in the CLI, for backwards compat with main module)

brittle wharf
#

that's where i disagree - CLIs commands are handwritten, it makes sense to optimize to reduce keystrokes. for code, you write it once, and hope the sands don't shift from underneath

dim smelt
brittle wharf
#

yeah i suppose so. it's a progressive enhancement type thing, right now it's CLI only but perhaps it's relevant for other things later

dim smelt
#

but wait, the way you propose implementing entrypoint, would other modules be completely hidden? If it's implemented by the CLI prepending every query with foo

brittle wharf
#

they don't get hidden - the other modules just get mixed in. so you still have the same collision concerns, between your entrypoint functions and module names, but it's at least much less delicate (underlying API still works, collision is just command shadowing, can emit a warning etc)

dim smelt
#

So the CLI queries twice I guess? And does client-side "stitching"?

#

dagger functions ->

  1. One query to list functions in the entrypoint
  2. Another query to list functions at the toplevel
  3. CLI merges the results of both queries?
brittle wharf
#

i'm not sure if it's two queries; i recall there being one big custom GraphQL introspection call at one point (maybe currentTypeDefs)

dim smelt
#

I'm investigating

#

(ie. doing my homework)

#

You're right it's not 2 queries. But there's also a regression in dagger functions the way it's currently implemented:

  • in main, dagger functions outputs 1) all functions of the main module 2) the name of all toolchain modules
  • in workspace, dagger functions 1) the name of entrypoint module, 2) the name of all other modules
brittle wharf
#

ah, hm could have sworn i had that working at one point (with a test and everything). but it's all a blur πŸ˜…

dim smelt
#

Codex seems to think there's way to do "schema stitching lite" with best of both worlds. Let me run that through the bs-meter and report back

dim smelt
brittle wharf
dim smelt
#

How do you feel about inserting a dag.Core(), always set as an entrypoint for now - but keeps options open for later?

#

I know you and @lofty shadow were discussing that as a desirable thing eventually, separate from this

brittle wharf
brittle wharf
#

@dim smelt i have a kinda cool demo of workspaces + worktree automation if you're around

dim smelt
dim smelt
#

Update: I'm rebasing workspace, and trying to get workspace-plumbing green

brittle wharf
#

feel free to ping for reviews/help. I went on a grand LLM adventure and it led me back home πŸ˜›

#

i can start chipping away at CI failures if there's nothing else in particular

dim smelt
#

@brittle wharf I think a review on workspace-plumbing would be good. I don't think any slop made it in, but I might have missed a spot

#

There's a "ledger" doc that should have details on ongoing changes and especially drift with the workspace branch, to inform the future rebase

dim smelt
brittle wharf
#

since the rebase went smoothly i'm assuming rebase, will push -f on confirmation

dim smelt
brittle wharf
#

@dim smelt pushed a rebase + minor fixes for TestWorkspaces

brittle wharf
#

these TestTelemetry failures are pretty sus, looking through the PR to find what might have done it

https://dagger.cloud/dagger/traces/453599d7afc79f66197cb54d6c409abd?span=11cd3a18dc824534

--- expected
+++ actual
@@ -7,21 +7,27 @@
 β•°β•΄βœ” starting session X.Xs
  
-βœ” load module: ./viztest X.Xs
-β”œβ•΄βœ” finding module configuration X.Xs
-β”œβ•΄βœ” initializing module X.Xs
-β”œβ•΄βœ” inspecting module metadata X.Xs
+βœ” load workspace: . X.Xs
+β”œβ•΄βœ” load extra module: ./viztest X.Xs
+β”‚
+β”œβ•΄βœ” ModuleSource.moduleName: String! X.Xs
+β”‚ ┃ viztest
+β”‚
+β”œβ•΄βœ” ModuleSource.blueprint: ModuleSource! X.Xs
+β”‚
+β”œβ•΄βœ” ModuleSource.toolchains: [ModuleSource!]! X.Xs
+β”‚
 β•°β•΄βœ” loading type definitions X.Xs
  
 βœ” parsing command line arguments X.Xs
 
-βœ” viztest: Viztest! X.Xs
-✘ .pending: Void X.Xs ERROR
-β”œβ•΄βœ” container: Container! X.Xs
-β”œβ•΄$ .from(address: "alpine"): Container! X.Xs CACHED
-β”œβ•΄βœ” withEnvVariable NOW=20XX-XX-XX XX:XX:XX.XXXX +XXXX UTC m=+X.X X.Xs
-β”œβ•΄βœ” withExec sleep 1 X.Xs
-β”œβ•΄βœ˜ withExec false X.Xs ERROR
-β”‚ ! exit code: 1
-β•°β•΄β—‹ withExec sleep 1 X.Xs
+✘ Viztest.pending: Void X.Xs ERROR
+β•°β•΄βœ˜ Viztest.pending: Void X.Xs ERROR
+  β”œβ•΄βœ” container: Container! X.Xs
+  β”œβ•΄$ .from(address: "alpine"): Container! X.Xs CACHED
+  β”œβ•΄βœ” withEnvVariable NOW=20XX-XX-XX XX:XX:XX.XXXX +XXXX UTC m=+X.X X.Xs
+  β”œβ•΄βœ” withExec sleep 1 X.Xs
+  β”œβ•΄βœ˜ withExec false X.Xs ERROR
+  β”‚ ! exit code: 1
+  β•°β•΄β—‹ withExec sleep 1 X.Xs
#

first part makes sense, second part seems extremely weird (nested beneath itself??)

#

ohh maybe we have an extra core.AroundFunc being added somewhere thinkspin

dim smelt
#

I'm going to go ahead and apologize in advance, just in case

#

I'm going to rewrite my commit messages to shut up the DCO error

#

@brittle wharf besides CI errors, did you spot anything stupid or worrying in the code?

brittle wharf
#

nothing egregious stood out yet, besides github's review UI kind of shitting the bed with the size of it

dim smelt
#

force-pushing git commit message rewrite

#

done. I hope I didn't clobber anything

brittle wharf
#

haven't pushed since earlier

brittle wharf
#

ok making headway on the telemetry stuff, it's from the entrypoint proxy

dim smelt
#

I'm looking into filterCore in the CLI, that's more logic CLI-side than I was hoping for

dim smelt
#

@brittle wharf still on telemetry?

#

rebased & pushed

brittle wharf
#

yep

brittle wharf
#
  • fix duplicate telemetry in entrypoint proxy wrapper
  • fix proxy wrapper resulting in constructor call being marked 'internal' and hidden
  • error early if resolving -m module ref fails
  • fix case where a type Broken struct {}; func (Broken) Broken() {} module would result in 0 functions
    • this is a little fishy - the test happened to hit it, and i'm not 100% confident in the work-around, similar to the clobbering bug but instead of clobbering you just don't get the function
dim smelt
brittle wharf
#

@dim smelt pushed a fix for TestContainer suite - it was just one failure, but it caught the fact that fields need to be exposed as entrypoints, too

brittle wharf
brittle wharf
#

pushed a fix for that one (and maybe others) - the constructor flag handling is simpler now, just installs every constructor arg as a flag on call

brittle wharf
#

pushing more fixes for this class of failure: entrypoint function that collides with core function and is this not registered
https://dagger.cloud/dagger/traces/6ca5c20849fbacdd26f4d5fe34bc7230?span=e979ce13b95551b2
FrogeAlarm This is technically where we're breaking, if this is a thing people are doing at all. Might be mostly tests. At the moment, it's all stick and no carrot, since dagger call file doesn't do anything either since we hide the core APIs. What if we didn't? thinkspin

Dagger Cloud

Browse and visualize Dagger traces.

dim smelt
#

@brittle wharf do you think we could review that plumbing PR together? I'm still worried about the CLI enfattening

brittle wharf
#

sure

dim smelt
#

Thanks - I'm free for the next 2 hours

brittle wharf
#

at this point i wonder if it would be better to just do a clean room implementation and describe the high level goal

#

most of this seems better saved for after workspace-plumbing too? my understanding is this branch is meant to be a bridge between the two worlds so we can merge more quickly, with some temporary scaffolding

dim smelt
#

I caught a few low hanging fruits (slop cleanup)

#

yeah temporary scaffolding is fine, but don't want to allow unnecessary slop into the PR just because a demented digital brain decided it was necessary scaffolding

#

let me push the fix I have, then we look at the result together?

brittle wharf
#

sure

dim smelt
#

the remaining design puzzle I think, is constructor args

lofty shadow
#

(lemme know when there's a good time to look through the plumbing, I just wanna see whether it's gonna be a wrench thrown into the cache work mainly, but don't wanna review stuff that's changing lurk )

dim smelt
#

(which you had flagged as a possible issue)

dim smelt
#

We can go over it together

lofty shadow
dim smelt
#

@lofty shadow @brittle wharf joining lounge

brittle wharf
#

pushed:

  • undid my changes to test suites
  • restored call constructor flags
#

now to rebase on main now that tuist is merged...

brittle wharf
#

force pushing

dim smelt
#

@lofty shadow @brittle wharf flagging an issue in my task: the specific case of dag.foo().foo() when Foo is an entrypoint. I this particular case, we need to decide if we allow the module's inner function to shadow its constructor. If we do, then it's no longer true that an entrypoint module is always reachable in 2 ways: via its top-level alias functions, and via its constructor.

#

This has 2 possible implications: 1) UX (users can't always rely on going through the construtor), 2) ID repeatability -> if the constructor is not reachable, we can't make dag.foo() an eager "canonical" alias to dag.foo().foo() such that the ID for dag.foo() is undistinguishable from the ID for dag.foo().foo().

#

So my question is: how should we handle such a conflict?

brittle wharf
# dim smelt So my question is: how should we handle such a conflict?

I'm still chewing on the 'thin outer shell' idea I mentioned, seems like it could help here too. CLI talks to a thin outer DagQL server that delegates to a normalized DagQL server (maybe even with core { ... }). All IDs go through the normalized server, outer server foo() delegates to foo().foo() on the normalized one, etc

brittle wharf
# brittle wharf I'm still chewing on the 'thin outer shell' idea I mentioned, seems like it coul...

pushed a couple of commits that implement this, looked promising locally - handles both cases (conflict with core, conflict with self constructor)

tried to get some wider CI runs before pushing but for some reason checks stopped running for my PR (https://github.com/vito/dagger/pull/416) - maybe native ci bug? cc @crisp dove @topaz zenith - tl;dr: pushed to my PR the first time, checks ran; pushed again, no checks; pushed again, no checks; pushed to upstream PR instead, checks ran. thinkspin I checked Inngest and nothing looked obviously wrong but don't 100% know what I'm looking for yet

#

tests that it fixed:

  • dagger call engine-dev test --run TestCall/TestCoreChaining/return_file/size --pkg ./core/integration/ (file())
  • dagger call engine-dev test --run TestModule/TestFloat/go/float64 --pkg ./core/integration/ (foo().foo())
crisp dove
brittle wharf
#

looks like the checks ran ~16 mins later for the last commit (force-push @ 23:44PM => load @ 00:00:31)

crisp dove
#

I wonder when the event arrived. Let me see

#

Is this the commit you are referring to: abfd53e?

crisp dove
#

These are the timelines of the events we received (for that PR we received only 3).

3:05 -> PR gets opened. Event processed within the minute
3:30 -> commit gets pushed. Event processed at 4:00, 30 minutes after
3:44 -> commit gets pushed. Event processed at 4:01, 16 minutes after.

The Inngest events for each were correctly sent at the right time and the Inngest functions were queued and started at exactly the same time. The culprit here is the rate limit (last two pictures). Creating the commit event took 16 minutes. We tried to create it immediately but we got a rate limit from GitHub telling us exacxtly when we need to retry. We are now using the response from GitHub to tell inngest when that step must be retried, so it waited for about 16 minutes before attempting again, and that time it worked

image.png image.png image.png image.png image.png image.png image.png
#

I wonder why your org was rate limited though, it doesn't look like it has that much activity to be honest. This is the last few commit events we have for the past 3 days:

image.png