#maintainers

1 messages Β· Page 15 of 1

tidal spire
#

Right, I vaguely remember wanting to tunnel to a service from the runtime container

civic yacht
#

Yeah interacting w/ the service via a withExec that has a service dependency on the service definitely works. I was thinking about the case of "function code directly talks to the service sockets", which I didn't see covered anywhere right now

tepid nova
#

Will try up and see if it works

tidal spire
tepid nova
#

oh nice, so it does work but only for containers, nice πŸ™‚

#

(selfish reaction for my use case)

civic yacht
tepid nova
tepid nova
#

Update: I can't get it to work at all

#

giving up

civic yacht
#

I updated the PR with a more thorough description of the problem and the fix. It's technically a breaking change (in terms of caching behavior, not compilation or correctness) and bubbles up to UX/DX a bit, so RFC: https://github.com/dagger/dagger/pull/10311

GitHub
cache secrets by plaintext; make cache key configurable by sipsma ...

Problem
Secrets have been cached by their "name", which for setSecret was the actual name and for URI-based secrets was the URI. This caching behavior was inherited from buildkit....

tepid nova
#

this is a pretty major bug. FYI

hasty basin
final star
#

obviously easy to lift that limit but there are other limits in play here: context windows, tokens per minute, and RAM... @still garnet added the limit here, it should be configurible and maybe higher by default, but i suspect "real world" performance might degrade pretty violently without a truncation limit otherwise it'd not have been added

#

short term, idk what's best here... 10x it and run evals? if 10xing it overloads openAI and anthropic tpm limits, what then?

tidal spire
#

maybe no truncation but optional arg for line ranges?

final star
charred lotus
tepid nova
tepid nova
#

If any core dev is around: I'm trying to understand how the workdir (dag.Host(".") is set when my nested dagger session is open.

Can I control which workdir is used, and if so how?

#

It seems that calling os.Chdir() before call dagger.Connect() doesn't work

#

My issue is that I'm opening a session from within a dagger runtime container, and I don't control the workdir: the engine hardcodes it to /scratch. By the time I can call os.Chdir() from within the container, it appears it's too late to change the workdir in the dagger session

#

and the workdir of the inner dagger session seems to affect how my generated client works, specifically where it looks for local dependencies...

tepid nova
#

Does buildkit have a primitive for running several processes inside a container? I would love to expose that in the core API

#

without having to inject shell scripts

civic yacht
civic yacht
hasty basin
tidal spire
#

what is the casing for Module.Name()? camel case? or whatever is in the dagger.json?

civic yacht
tidal spire
#

guessing maybe it's implemented in each sdk

civic yacht
#

yeah double-checked but nothing built-in at the moment, I guess the SDKs are doing that

tidal spire
#

makes sense, thanks!

tepid nova
#

@civic yacht I'm facing a module loading conendrum...

  • I have a custom dagger runtime, bundled in a SDK module
  • My runtime container executes a custom go binary
  • That go binary loads dagger modules using the experimental dagger client feature. So that go program (cmd/my-runtime) is its own dagger module, with the dependencies to be loaded by the generated client
  • Problem: the generated client fails to load its dependencies. I get a graphql error trying to query it
  • Question: how do I configure which module to load when opening a nested dagger session from with a dagger runtime container?
#

(importantly, loading works the dependeny is a remote git source. but fails on local source)

tepid nova
tepid nova
#

I think we may have a tracing blind spot: when loading a module, if the SDK's ModuleRuntime() function calls more Dagger functions, that doesn't seem to show up in the trace

Nevermind, I called modSource.AsModule().Source().SourceRootSubPath() instead of just modSource.SourceRootSubPath() (dumb IDE auto-complete) and apparently that bricks the runtime

fresh harbor
#

elixir Β· Releases Β· dagger/dagger

tepid nova
#

So I've been experimenting with custom runtimes, and I'm on a personal quest to democratize their creation and their use.

In that vein I'm testing the waters on new ways to use runtimes. Ie. how difficult would it be to:

  1. Allow loading a module without a dagger.json, eg. dagger -m github.com/anything/atall
  2. Allow specifying a runtime at load time, eg. dagger -m github.com/anything/atall --runtime=github.com/mycustom/runtime
  3. Build on 1 and 2 to allow specifying dependencies in your dagger.json that a) don't have a dagger.json in their source, and b) are loaded by a runtime you specify

This would IMO unlock the power of custom runtimes (assuming we can also make them super easy to develop, that's another story..), and basically means you can daggerize anything including repos you don't control

Thoughts?

civic yacht
#

So I've been experimenting with custom

tidal spire
#

I noticed with 0.18.6 I'm prompted by 1password when I open shell. Is that to pre-authenticate prompt mode? Feels kind of disruptive

spark cedar
#

πŸ‘‹ hallo, i've got a couple of spec-ed out issues around improvements to module versioning/updating! if anyone has any spare cycles would appreciate a hand on these, i'm really caught up in theseus work:

hasty basin
#

πŸ‘‹ hallo, i've got a couple of spec-ed

tepid nova
#

πŸ™‹πŸ»β€β™‚οΈ

tidal spire
tepid nova
#

@civic yacht @rancid turret I know I can dynamically load and introspect a module using the dagger API - which is amazing btw.

But is there a way to use the module after that? I tried calling ... | as-module | serve in the shell but nothing seemed to happen. Maybe I was over-ambitious? πŸ™‚

civic yacht
#

If you did all this in a nested dagger exec, then you'd be able to serve the module and run raw gql queries separately w/ dagger query, though obviously a decent set of extra hoops to jump through

tepid nova
#

But the shell doesn't rely on codegen for any of it, it discovers everything dynamically via introspection anyway. So I was wondering if it would just magically work

civic yacht
tepid nova
#

Anyway, it was just for experiments, my actual implementation would probably involve patching asModule to do what I want

civic yacht
tepid nova
#

I tried .refresh but it didn't seem to work either

#

(I think it reloads the module it knows about in the CLI config)

charred lotus
#

noobquestion: i have a dagql.Object and i want to turn it into a dagql.Selector, how can i do that ?

#

actually nvm i think i want to do dag.Select(ctx, obj). I was doing dag.Select(ctx, Root(), turnObjIntoSelector(obj))

civic yacht
tepid nova
rancid turret
# tepid nova But the shell doesn't rely on codegen for any of it, it discovers everything dyn...

After serving a module like you did the shell then also inspects the API via a GraphQL query (see source) and saves the result in memory. Then what you interact with comes from this introspection. It's basically a dag.CurrentTypeDefs() call but optimized so it takes only 1 query to fetch all the data from that.

.refresh does this but it's only on the currently loaded module (from .cd or initial context). Serve + inspect also happens when you <module ref> | ... or .help <module ref>. Doesn't that fit with your use case for testing?

tidal spire
#

Is it possible to have cloud enabled and also have the process dagger is running send it's OTEL to a 3rd party service?

spark cedar
#

i think it's likely a flake, but just thought it was kinda odd that it didn't seem to ever show up until we noticed it πŸ˜›

still garnet
#

i do that frequently if i need to use e.g. honeycomb to inspect the lower level telemetry

civic yacht
final star
tepid nova
#

If I wanted to implement a core function that can take an argument of any object type. What's the least hacky way to achieve that?

#

My default answer is "not possible". So I have low expectations. But just in case πŸ™‚

still garnet
tepid nova
#

That would do it! Cool

#

FYI this is in the context of "tool scoping" in Env. The shell has a layer of scope which is "current object" (almost like a current selection, oh god why am I reopening this topic πŸ˜› ) which influences the command lookup. If we were to bring back foo | bar | .load or .cd $(foo | bar) that's what would allow the object's functions to be top-level shell commands.

I was wondering how we would add this to Env if we wanted to.

#

It breaks my brain when I think about it too long, because it means Env becomes kind of like a query builder API, on top of the Dagger API which itself is queried with a query builder... still not convinced it's actually useful

tepid nova
#

@still garnet would that mean we could have Env.withObjectBinding and remove all the other typed object binding setters? Since we have the .asFoo getters on the other side

#

Technically we lose type checking at set-time, but it still happens at get-time, so maybe it's fine? As long as the engine can infer the type from the ID, it shouldn't affect tool mapping?

tepid nova
#

I think Env is basically a client context in the engine. It can allow us to make the CLI incredibly thin and dumb (which is a good thing)

#

I'm editing my post to incorporate model-context and other bits and pieces

still garnet
tepid nova
#

Maybe that's how we resolve the client/engine matrix from hell: eventually the engine eats everything, except session attachables

#

and all that's left is a capability handshake

still garnet
#

only skimmed so far (weekend time) but i'm interested in the part where shell interpretation moves into the engine

tepid nova
#

I'm refining that part as we speak πŸ™‚

still garnet
#

maybe then we can revisit having the LLM generate shell

tepid nova
#

I'm also trying out a merger of LLM tools and shell tools. They sure have a lot in common...

#

I just want to ssh into my dagger engine πŸ˜›

#

after virtualizing all host access in env, the final boss: virtualizing session attachables.

#

@still garnet feel free to ignore me. Random idea, I'm wondering how monolithic we would want an engine-side interactive shell.

  • Is it literally the whole TUI bundled together - prompt mode, agent mode, the keymaps to switch between them? Like ssh-ing into dagger and getting a raw terminal stream?
  • Or, is it one different terminal session for shell mode, agent mode, and in the future any number of parallel sessions, left for the client to present to the user? So, more like a tmux experience... πŸ˜‡ feels like an elegant generalization and opportunity to make the Dagger CLI more powerful in the future no?
#

oooh shell & chat history...

tepid nova
tepid nova
#

Is it me, or is the "connect" phase slower in 0.18.6, when starting dagger shell ?

rocky plume
#

What's the best way for building dagger? I see there's a CONTRIBUTING doc that mentions using the dagger dev module (which is json data), but I feel like I'm missing something since I'm pretty new to using dagger. I wonder if the doc should be expanded to mention how to use it?

I also see there's a ./hack/build dagger script, which worked out of the box for me πŸŽ‰ , should I be using this as part of a typical dev workflow? I don't see it mentioned in the CONTRIBUTING guide.

GitHub
dagger/CONTRIBUTING.md at main Β· dagger/dagger

An open-source runtime for composable workflows. Great for AI agents and CI/CD. - dagger/dagger

spark cedar
#

πŸ‘€ huh, yeah, looks like somehow that's fallen out of date

#

CONTRIBUTING should probably be updated to match

civic yacht
rocky plume
#

ah ha, yeah that looks like the guide I need!

tepid nova
#

@rocky plume keeping in mind that I'm an occasional low-intensity core contributor, here are a few dagger commands I find super useful. For the more hardcore low-latency local devloop, you're already talking to the right people:

To get a complete build (CLI + engine) of any PR, and play with it interactively:

dagger -m github.com/dagger/dagger@pull/$PR/head -c 'dev | terminal'

To do the same thing locally

dagger -c 'dev | terminal'

To run the dev engine only, then connect to it from an external CLI

dagger -c 'engine | service my-feature-name | up'

Then in another terminal:

_EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://localhost:1234 ./any/build/of/dagger
rocky plume
final star
# rocky plume What's the best way for building dagger? I see there's a [CONTRIBUTING](https://...

i personally use ./hack/dev and ./hack/with-dev like a billion times a day lol, they essentially bootstrap off ./hack/build, then load and run a dev engine for you and point your CLI to it, like you can do ./hack/dev dagger ${thing i'm currently developing} or to skip the rebuild and use your most-recently-built dev engine ./hack/with-dev dagger ${thing i'm currently developing}. like solomon said the "hack" scripts are the quicker, more host-machine-centric advanced methods. they have some pitfalls like build errors can easier to miss than the more dagger-native "disposable" dev builds that solomon and some others use

tepid nova
#

I feel like it's important to know that, technically, you can do everything just with the dagger CLI (our buids, tests etc are fully self-hosted in Dagger). The only reason we still use custom ./hack scripts is speed... specifically the overhead of re-running everything in dagger every time. Hopefully at some point we will get the overhead down to a point where ./hack/ is no longer necessary at all

#

Ah one more @rocky plume : at any time you can run dagger and you'll get an interactive dagger shell with the current module's functions available as commands. .help to list them

rocky plume
#

ah yes, I've used the .help several times already πŸ™‚

#

I thought it was neat to see the hack/build script is actually a dagger script, where as the other ones seem to just do a wee bit of bash before calling build

#

I must admit while reading the dagger quickstart docs it wasn't immediately clear I could do that, but it was once I saw the hack/build script it clicked. I then went back to the dagger shell docs and noticed it showed "inline execution" by default, but I could then click over to the other options, such as "script"

wet mason
#

@polar fulcrum @still garnet πŸ‘‹

Hey, do I need to have llm credentials in order to run dagger mcp?

tepid nova
final star
wet mason
#

@final star as in, I should set them to a random string as a workaround?

final star
#

Yes

charred lotus
#

yes i have a wip branch somewhere to get rid of that but there's a lot of entangled stuff to detangle, sorry it's been lurking for so long

wet mason
still garnet
#

ha neat. yeah, sounds basically the same challenges as us

tepid nova
#

soon they'll add a nested agent πŸ™‚

#

to sneak copilot in

#

(that's what I would do anyway)

wet mason
#

@still garnet do you think I can use #10366 or should I wait a bit?

still garnet
final star
#

random engine devex thing that i've been ignoring: i feel like every time i rebase a pr with an api change i get conflicts in one file more than any other: docs/static/reference/php/renderer.index. could we avoid this somehow? it'd save everybody quite a lot of time

wet mason
#

@still garnet sweet!

final star
wet mason
final star
still garnet
wet mason
still garnet
#

looks about right

wet mason
#

just stuck in a loop

still garnet
wet mason
final star
still garnet
final star
#

ah sick love it

still garnet
#

i'll dust that PR off once static tools is merged

wet mason
obsidian rover
wet mason
#

sweet

#

@still garnet oh btw -- what happens if the module hits an exception? does that travel back through MCP or does it get lost in translation?

final star
still garnet
wet mason
#

sweet πŸ™‚

#

ok so I must be the one losing stuff in translation

still garnet
still garnet
wet mason
#

openai

still garnet
wet mason
still garnet
#

uhh it's doing pretty weird things there altogether

wet mason
#

(quick and dirty module, very likely I did something wrong)

still garnet
#

is it not respecting the MCP server instructions maybe?

final star
still garnet
wet mason
#

@still garnet hmmm ... I'm using a few MCP servers already, adding dagger to the mix

final star
#

ideally the mcp server startup is fully decoupled from module load, like if the module is broken it'd be nice to only error once a relevant tool is called... but like even in saying this i'm not fully sure what a "relevant tool" would be given the good ones come from the loaded module XD

wet mason
#

@still garnet the module is a tiny, one function thing

#

I do see the function with dagger functions

still garnet
#

the problem is it's passing a bogus type parameter to list_objects (should be Sandbox). not sure why it's doing that - what's your prompt?

wet mason
#

@still garnet I don't have any tool specific prompt in there, maybe the framework has one but I didn't have to specify anything for the other tools

#

@still garnet oh wait, it's different now

still garnet
#

what openai model is this? seems kinda dumb

still garnet
#

but even so, i've seen better behavior from clients that don't in my testing. but that was with Anthropic. hmm

wet mason
wet mason
charred lotus
#

For the record, claude code has a timeout only for the init, if there is a timeout for execing a tool, it's above 1h. What i was hoping is for claude to do healthcheck pings.

final star
charred lotus
#

So just FYI, there's a Ping request in MCP, that apparently is not often used but if you're doing custom stuff with MCP feel free to use it

#

(dagger mcp responds to pings)

charred lotus
#

@final star any reason endpointMtx is a *sync.Mutex and not just sync.Mutex?

still garnet
final star
charred lotus
final star
charred lotus
final star
charred lotus
final star
obsidian rover
#

Quick question regarding the export consistency API-wise. I'm playing with some exports and i've been surprised about the --wipe field on Directory.Export ; which doesn't seem to be exposed on the file export πŸ‘€ -- does that mean that we expect the file export to be wipe: true by default ? (if not expect i'll fire an issue πŸ‘Ό )

tidal spire
obsidian rover
tepid nova
spark cedar
#

i remember someone recently mentioned something about "env" support, and how we were thinking a different format from the standard .env file - is there an issue here, or is https://github.com/dagger/dagger/issues/9584 the best place for that?
cc @tepid nova, I think it might have been you?

leaden glade
#

I tried several keywords in search with not a lot of success. Did anyone already launched the engine with a debugger attached? That could help a lot to fix some nil pointer for instance.

leaden glade
humble hearth
# spark cedar kicking the tires here again, since <@418233653592719364> is starting work on it...

I'm not sure this is the right implementation, I'm wondering if the security risks of modules grabbing random env vars from the host (talked with Jeremy yesterday). Maybe a native .env support would be better. I'm going to reply to #9584 with some thoughts.

This PR is really focused on this: https://github.com/dagger/dagger/issues/8428

GitHub
✨ Add environment variable support to arguments · Issue #8428 ·...

What are you trying to do? It would be fantastic if we could allow argument annotations on functions, this would allow pulling the environment from .envrc or .env files Why is this important to you...

tepid nova
#

@spark cedar there are at least 2 possible designs:

  1. Initially I proposed this: https://github.com/dagger/dagger/issues/9584
  2. Now I'm proposing we tie it to the overall Env design instead of doing a one-off: https://github.com/dagger/dagger/discussions/10370
GitHub
.env support & default secrets Β· Issue #9584 Β· dagger/dagger

Overview Dagger should natively support .env files. It's a de facto standard for managing environment-specific configuration in a lightweight, portable way. It is widely supported, included by ...

GitHub
Evolution of `Env` Β· dagger dagger Β· Discussion #10370

Table of Contents Overview Bindings Env and Dagger Shell Model context Env and TUI Env and host filesystem Tools Env persistence: .env and beyond Horizon 1: lossy persistence with .env Horizon: los...

tawny flicker
#

@tepid nova where is a reference/info/code to the custom dagger runtime you were talking about in the call?

obsidian rover
rocky plume
#

How do people feel about including shell commands in the integration tests? e.g.

func (DirectorySuite) TestSymlink(ctx context.Context, t *testctx.T) {
    c := connect(ctx, t)

    t.Run("symlink in same directory", func(ctx context.Context, t *testctx.T) {
        dir := c.Directory().
            WithNewFile("some-file", "some-content").
            WithNewFile("other-file", "other-content").
            WithSymlink("some-file", "symlink-to-some-file")

        ctr := c.Container().From(alpineImage).WithDirectory("/test-dir", dir)

        // test the symlink is an actual symlink
        _, err := ctr.WithExec([]string{"test", "-L", "/test-dir/symlink-to-some-file"}).Stdout(ctx)
        require.NoError(t, err)

        // test accessing the symlink correctly accesses some-file
        _, err = ctr.WithExec([]string{"sh", "-c", `test "$(cat /test-dir/symlink-to-some-file)" = "some-content"`}).Stdout(ctx)
        require.NoError(t, err)

        // test overriting some-file with other-file doesn't break the symlink
        _, err = ctr.WithExec([]string{"sh", "-c", `cd /test-dir && mv other-file some-file && test "$(cat symlink-to-some-file)" = "other-content"`}).Stdout(ctx)
        require.NoError(t, err)
    })

}

maybe it's fine? I see it being done in several places:

$ grep -RI -- '"sh", "-c"' core/integration/ | wc -l
163
final star
rocky plume
final star
#

random engine devex thing that i've been

tepid nova
#

Personally my approach is:

  • Default to native Dagger exec
  • When native Dagger API can't do what I need, or is just too cumbersome, switch to the smallest possible shell script, with a FIXME for later
  • If I notice a pattern of always needing a shell script for the same reason, I start looking for possible improvements to Dagger
rocky plume
# tepid nova Personally my approach is: - Default to native Dagger exec - When native Dagger...

that makes sense, I rewrote it to simply do one Exec (so we can use test -L to verify it's actually a symlink by the time the container starts running), and then am leaning on File.Contents for most of the other testing, e.g.

    t.Run("symlink in same directory", func(ctx context.Context, t *testctx.T) {
        dir := c.Directory().
            WithNewFile("some-file", "some-content").
            WithSymlink("some-file", "symlink-to-some-file")

        ctr := c.Container().From(alpineImage).WithDirectory("/test-dir", dir)

        // test the symlink is an actual symlink
        _, err := ctr.WithExec([]string{"test", "-L", "/test-dir/symlink-to-some-file"}).Stdout(ctx)
        require.NoError(t, err)

        symlinkContents, err := dir.File("symlink-to-some-file").Contents(ctx)
        require.NoError(t, err)
        require.Equal(t, "some-content", symlinkContents)

        symlinkContents, err = dir.WithNewFile("some-file", "overwritten-contents").File("symlink-to-some-file").Contents(ctx)
        require.NoError(t, err)
        require.Equal(t, "overwritten-contents", symlinkContents)
    })

it'll also give us better error messages from testify rather than simply getting an error code of 1.

#

I wonder if there's a need for a File.Stat() call in the dagger api?

tepid nova
#

For a while you had to use a shell script every time you wanted to catch a non-zero exit code... Now we have Container.exec(expect:ANY|ERROR) so I got to remove a lot of shell script wrappers that day πŸ™‚

tepid nova
#

Somehow we never got around to finishing it, but yeah overwhelming consensus that we have a gap there

rocky plume
tepid nova
#

@civic yacht could this be another secret caching issue?

civic yacht
tepid nova
#

Thank you @final star for getting a release out on short notice!

tepid nova
#

once theseus is shipped, it would be pretty cool to have a git cache backend... mapping blobs to git objects could be sweet. better than GHA cache service IMO

bronze hollow
#

Is Theseus a managed BuildKit / cache service? πŸ‘€

rancid turret
#

Telemetry with links

spark cedar
# bronze hollow Is Theseus a managed BuildKit / cache service? πŸ‘€

not really - it's an effort to piece-by-piece rework and replace large parts of our usage of buildkit, including caching, the scheduler, the underlying implementations, etc.
we're often bumping against the limitations of buildkit, so the idea is to try and own more of that logic πŸŽ‰ this then enables us to do really cool things with remote caching/persistence, because we're not limited by what's supported in buildkit

bronze hollow
#

Very cool. Excited to see what y'all do

tepid nova
civic yacht
spark cedar
#

ahaha

#

great minds!

tepid nova
#

sorry I should have checked. will close mine

#

I very recently started getting Go errors in my IDE. It seems that it's because the engine no longer builds on Darwin?

#

As a result some IDE operations don't work for me anymore (for example renaming symbols): the IDE asks me to fix errors first, but I can't

#

Where's the code that emits this? Is it buildkit?

image.png
#

And what operation triggers it?

tepid nova
#

I noticed that a solid chunk of module loading logic lives in core/schema/modulesource.go which surprised me, since most files in core/schema are thin graphql frontends, with the backends living in core/. Is there a specific reason for this difference, or is it just a leftover quirk from past refactors?

civic yacht
#

I noticed that a solid chunk of module

#

Where's the code that emits this? Is it

#

I very recently started getting Go

tawny flicker
#

Is it possible to plug into the otel traces of dagger?

I would love to instrument my test code to have a fine grain info on which test takes more time directly in dagger cloud.

final star
rocky plume
spark cedar
#

ah lol

#

funny

#

I have that in my branch dagop-exec

#

it's on my fork? sorry, afk now, it's all suuuuper super crappy right now

#

I can try and tidy it up tomorrow so you can actually use it? the reason it's so much harder than the others is because there are actually multiple inputs and outputs

#

each mount goes in, each mount comes out

#

but there's still one op

rocky plume
#

I'll take a closer look at your branches, thanks for the reminder about it.

rocky plume
spark cedar
#

it's a bad history, but you probably don't want the bits that hook into withExec

#

(since it's almost entirely broken lol)

final star
#

so for some reason i thought that it was already possible to pass module constructor args through to the autoloaded shell module, but went to test it and it seems that that's not a thing. is there some reason i might be missing as to why we don't want that? in dagger/dagger/.dagger, i'm basically thinking about the dagger shell --source=../dagger-worktree-copy --dockerCfg=... cases. ideally this would also apply to dagger mcp too and re-use some dagger call code. wdyt, should i open an issue? @rancid turret @tepid nova

hasty basin
#

so for some reason i thought that it was

rocky plume
#

ContainerDagOp

obsidian rover
#

Quick question: https://github.com/dagger/dagger/blob/68f969a06b4cd5afb9c9f51a7fac0617fd81ba1a/core/schema/git.go#L273-L282

In what case is if clientMetadata.ClientID != parentClientMetadata.ClientID { true and leads to a break ? Like hypothetically -- i'm a bit confused by the NonModuleParentClientMetadata

Given this example:

type Jo struct{}

// Here, in this hypothetical example, let's say that privateGitRepo is private HTTP ref
// In this case, the user runs this function as a top-level dagger call: `dagger call example --private-git-repo https://XXXXX.git`
func (m *Jo) Example(ctx context.Context, privateGitRepo *dagger.Directory) *dagger.Directory {
    return privateGitRepo
}

func (m *Jo) ExampleRef(ctx context.Context, ref string) *dagger.Directory {
    return dag.Git(ref).Ref("main").Tree().Directory("zip")
}

This works:

 dagger call example --private-git-repo https://github.com/grouville/daggerverse-private/

And this fails:

dagger call example-ref --ref https://github.com/grouville/daggerverse-private/

Now, this is expected in term of security by design (users need to pass the socket and / or the secret and pass them as args. Is it that check that enables that ? if clientMetadata.ClientID != parentClientMetadata.ClientID {

obsidian rover
#

Got my answer, indeed πŸ’―

rocky plume
tepid nova
rocky plume
tepid nova
#

Oh, well it depends how your engine is provisioned. Do you have EXPERIMENTAL_DAGGER_RUNNER_HOST set to something custom? The dagger CLI will default to its builtin docker provisioner unless that variable is set

rocky plume
# tepid nova Oh, well it depends how your engine is provisioned. Do you have `EXPERIMENTAL_DA...

that's so very odd if it is the top level instance, because I see:

root@tyee:/var/lib/docker/volumes# ps auxw | grep dagger-engine | grep tcp
root     2150004  0.0  0.0 1320024 59948 ?       Ssl  17:07   0:00 /usr/local/bin/dagger-engine --config /etc/dagger/engine.toml --addr tcp://0.0.0.0:1234 --network-name dagger-dev --network-cidr 10.88.0.0/16

which appears to be running via buildkit (well the dagger engine, but still under a different buildkit cgroup)

root@tyee:/var/lib/docker/volumes# cat /proc/2150004/cgroup 
0::/system.slice/docker-da3da4bd56dd9e80ddd8518f59edd36f21e5d14d6ce9e999b1c86cee801657dc.scope/buildkit/f715uk0jsa0hacel0kzepa176/init

if I look at my host docker instance, I see it's been up for 6 days:

root@tyee:/var/lib/docker/volumes# docker ps -a | grep da3da4
da3da4bd56dd   registry.dagger.io/engine:v0.18.8   "dagger-entrypoint.s…"   6 days ago     Up 28 hours                       dagger-engine-v0.18.8

which seems a bit confusing. I need to figure out what's actually running underf715uk0jsa0hacel0kzepa176/init

tepid nova
#

are you running a dev engine somewhere? Or is this 0.18.8 engine what you expected to find?

rocky plume
#

both I suspect? dagger call sdk go generate export --path . uses v0.8.18, but doesn't that startup a dev instance under the hood?

#

I don't have any _EXPERIMENTAL_DAGGER_RUNNER_HOST set on my host

#

because after running the dager call ... I end up with two seperate instances running

# ps auxw | grep dagger-engine.*--config.*toml | grep -v grep
root        4987 12.2 38.1 38715032 24797772 ?   Ssl  May19 205:41 /usr/local/bin/dagger-engine --config /etc/dagger/engine.toml --debug
root     2150004  0.0  0.0 1320024 59772 ?       Ssl  17:07   0:00 /usr/local/bin/dagger-engine --config /etc/dagger/engine.toml --addr tcp://0.0.0.0:1234 --network-name dagger-dev --network-cidr 10.88.0.0/16
#

and the dev instance is running under a child cgroup of the official build:

root@tyee:/var/lib/docker/volumes# cat /proc/4987/cgroup 
0::/system.slice/docker-da3da4bd56dd9e80ddd8518f59edd36f21e5d14d6ce9e999b1c86cee801657dc.scope/init
root@tyee:/var/lib/docker/volumes# cat /proc/2150004/cgroup 
0::/system.slice/docker-da3da4bd56dd9e80ddd8518f59edd36f21e5d14d6ce9e999b1c86cee801657dc.scope/buildkit/f715uk0jsa0hacel0kzepa176/init
#

and going deeper, here's the nested instance running vua runc:

# cat ./volumes/acbb87c1a60414d61877a84560fad964cd2aa22bf70a59fd46a53986029621d2/_data/worker/executor/f715uk0jsa0hacel0kzepa176/config.json | jq .process.args
[
  "/.init",
  "dagger-entrypoint.sh",
  "--addr",
  "tcp://0.0.0.0:1234",
  "--network-name",
  "dagger-dev",
  "--network-cidr",
  "10.88.0.0/16"
]

and it's this instance I'm hunting for logs from.

tepid nova
#

Ah I see!

#

OK so I use dagger cloud for this πŸ™‚

#

If you navigate the trace, you'll find the nested service (pulsing yellow while it's running) then all the logs are there

#

Or, you can run with dagger call -E (for --no-exit) then the TUI remains for you to interactively explore. You can enter a span with enter, exit it with escape, etc. The logs are in there. Can be laborious in a very large trace though

#

Another way would be dagger --progress=plain then you'll get the firehose of logs and meta-logs in text format

rocky plume
#

ah ha, I should really play around with both cloud traces and the -E, thanks for these ideas.

#

Lev upgraded my cloud account, so I should be good to go on that front.

wet mason
tepid nova
#

If a ModuleSource was loaded as a dependency, does it contain a reference to its parent?

#

(I think the answer is no, but just checking)

obsidian rover
#

@midnight rapids Your repros were very useful, thank you a thousand time πŸ™

I isolated two bugs with SSH auth loading that I'm currently fixing:

  1. We don't seem to support the case where 1) the user needs a key for a private module and 2) needs another one for one of its args
  2. We seem to only take the first one into account

I guess it's time to load the gitconfig / sshconfig from the user -- which seems to be the underlying cause. From memory we DO load the gitconfig -- but sshconfig

This seems related to the same underlying cause -- digging

spark cedar
rocky plume
#

hey @spark cedar can I get a quick confirmation I'm on the right track with my core/schema/directory additions for while working on a new directory.exists method?

func (s *directorySchema) exists(ctx context.Context, parent dagql.Instance[*core.Directory], args existsArgs) (dagql.Boolean, error) {
    if !core.DagOpInContext[core.RawDagOp](ctx) {
        return DagOp(ctx, s.srv, parent, args, s.exists)
    }

    exists, err := parent.Self.Exists(ctx, s.srv, args.Path, "")
    return dagql.NewBoolean(exists), err
}

and it can be hooked up with:

        dagql.NodeFunc("exists", s.exists).
            Doc(`todo.`).
            Args(
                dagql.Arg("path").Doc(`todo`),
            ),

am I right in thinking I need to use NodeFunc rather than simply Func here?

wet mason
charred lotus
#

@wet mason wdym by carefully crafted ?

wet mason
#

they don't need much sandboxing

charred lotus
#

ah ok

#

how does it test it ?

tepid nova
#

@wet mason @charred lotus FYI I'm actively using dagger CLI as a coding agent right now, because I actually needed it πŸ™‚ For codebase analysis

charred lotus
wet mason
#

@charred lotus got some time to chat? I'm getting stuck /cc @final star if you're around

charred lotus
#

sure

obsidian rover
wet mason
#

npx @modelcontextprotocol/inspector

tepid nova
#

note I gave up on doing it in my local checkout. Just had it checkout the git repo from main. Git context is strength of ours and I think you guys should leverage that in your prototype

#

let the IDEs struggle with messy local checkouts.

#

I say in the future, a local git checkout on the user's computer will be the exception not the norm

charred lotus
tepid nova
#

Just didn't know how to plug it in

#

or get it back out

#

prompt was simpler with just "help me understand the source code of dagger, start from main branch"

#

IMO git is a strong entrypoint for us

#

actually there's an opportunity to simulate regular stateful file tools, except it's in an ephemeral checkout of a git branch, and gets auto-saved to your agent's "working branch" when they're done (no explicit tool callls needed). Then expose simplified container tools that assume that local context

#

cc @wet mason πŸ‘†

#

the env starts with a context dir

#

(which by the way we already have an implementation of with .cd in the dagger shell)

wet mason
# tepid nova IMO git is a strong entrypoint for us

Depends on what we want to target for the integration. Goose, codex, cursor … all work from local code

If we target β€œcoding agents” (as in, hand rolled agents) then it’s easier to go with git yeah

wet mason
tepid nova
#

there are lots of useful tasks a dev wants to do within their IDE, that involve interacting with remote resources

#

I just wanted to point out anecdotically, that for my use case today, I really didn't care about using the local checkout. If dagger had given me a super easy way to auto-push the agent's changes to a temp branch, I would have used that for sure

#

I could get the agent to do it, but then I have to setup git-related tools, give it auth etc...

#

easier if it happens implicitly

#

maybe a crazy idea, but imagine if I could just aim the tool at any git branch of any repo, and boom that's the environment. Then expose very basic tools for snapshotting your env, rolling it back.. under the hood it would be backed by git magic (annotations, multiple branches whatever)

rancid turret
fair ermine
#

This file doesn't even exist in the repo lol

spark cedar
#

oh not this again

fair ermine
#

It seems it's using a previous version?

spark cedar
#

welcome to hell

#

(known flake)

rancid turret
#

I've been seeing it pretty reliably for the last commits in main and open prs.

spark cedar
#

yeah same, i'm gonna try and commit some time soon to dive in. atm just trying to wrap up the current state of my execop work, so i can dive back in later

tepid nova
#

@stray heron I'm reviewing --remote now πŸ™

#

can't wait to try it

tepid nova
#

In core, given a *ModuleSource (or any other core type), can I get a dagql.Instance[*ModuleSource]?

#

I'm in a core schema handler func, and need a dagql.Instance of the parent. But I only get the core type of the parent.

spark cedar
#

do you have a specific scenario? it depends a bit on what you're doing

#

you can do NewInstanceForID

#

but that will only generally work if you load that ID again you'd get the same result

tepid nova
#

But how do I get the ID?

spark cedar
#

there's NewInstanceForCurrentID

#

if there is no id then uh hm

tepid nova
#

I'm starting from:

dagql.Func("withDependencies", s.moduleSourceWithDependencies).
            Doc(`Append the provided dependencies to the module source's dependency list.`).
            Args(
                dagql.Arg("dependencies").Doc(`The dependencies to append.`),
                dagql.Arg("platform").Doc(`Install as platform dependencies (executed in the parent's context directory)`),
            ),

And am trying to patch s.moduleSourceWithDependencies, which starts like this:

func (s *moduleSourceSchema) moduleSourceWithDependencies(
    ctx context.Context,
    parentSrc *core.ModuleSource,
    args struct {
        Dependencies []core.ModuleSourceID
        Platform     dagql.Optional[dagql.Boolean]
    },
) (*core.ModuleSource, error) {
  // what to do
spark cedar
#

oh

#

you can change Func to NodeFunc

#

then it'll accept a dagger.Instance[T] of T

tepid nova
#

ah that's perfect! Thank you

spark cedar
#

🫑

tepid nova
#

ha ha this could not have been a more laser-focused solution to my problem, there is literally zero other change, just getting the instance I needed πŸ˜›

tepid nova
#

@spark cedar but how do I deal with this?

    parentSrc = parentSrc.Clone()
#

Going to try

~~```
parentSrc.Self = parentSrc.Self.Clone()

parentSrc = parent.Self.Clone()
// ...
return dagql.NewInstanceForCurrentID(ctx, s.dag, parent, parentSrc)


And hope nothing blows up
rancid turret
#

shell interpreter builtins

tepid nova
#

Not sure where to even start πŸ˜…

#

Maybe I should ask an agent

tepid nova
#

In core or core/schema, how do I convert a string-encoded ID back to a call.ID?

#

OK my daggerbot actually gave me the answer πŸ˜›

fair ermine
#

Hello guys, I was about to jump in the implementation but I thought it would be smarter to first discuss it.
I created an issue about the optional SDK function call, so SDK would not be forced to implement the whole SDK interface to work but only what they need (like only ModuleRuntime but not Codegen for module support)

https://github.com/dagger/dagger/issues/10480 happy to read your thoughts before making the implementation

/cc @tepid nova @spark cedar @leaden glade @rancid turret

GitHub
✨ SDK should not be forced to implement the whole `SDK` interface...

What are you trying to do? When executing a module, the engine should be smart enough to adapt his calls based on what the SDK implements. Each dagger module and client related commands need 1 or m...

spark cedar
#

i generally like the idea here a lot, should simplify making sdks a lot

#

one thing i would like to see is a separated out Init method for dagger init, so that column would be Init + Codegen + GenerateClient. i started work on it in https://github.com/dagger/dagger/pull/8201, i can't remember if i discussed with you if you wanted to pick that up.

fair ermine
#

I didn't mentionned it because it's not merged yet and I assumed my proposal will be merged before that one but yeah it would be part of it

fair ermine
leaden glade
#

I wanted to quickly improve the Java SDK generated doc for a module.
Right now, like in Go, we are documenting the package. But we are also adding a documentation to the main object.
It looks like if both are defined then the one from the main object is displayed (at least in dagger shell).
As the package documentation is, contrary to Go, in a separate file, would it make sense to only document the main object? So that everything is in a single file, and more visible to the user. Or is there any reason I should keep the package level documentation only?

rancid turret
#

Module docs

final star
#

engine benchmarks started getting preemptively cancelled around 8m of runtime ~2 weeks ago. investigating now. commit where it started is maybe suspicious, it's the one where @civic yachtfixed the exec deduping cancel edge case

GitHub
skip deduping running nested execs (#10264) Β· dagger/dagger@23edb05
  • skip deduping running nested exec function calls

De-duping currently running nested execs, such as module function calls, can break when:

  1. Client A starts the function call
  2. Client B makes t...
tepid nova
#

@spark cedar perhaps too late in the day for you, but if you want to chat about "inline dependencies" + the related "persisted env configuration", both are high priority for me given what they unblock. Would love to hear your thoughts on the design directly

spark cedar
tepid nova
spark cedar
civic yacht
#

skip deduping running nested execs (#102...

fresh harbor
still garnet
#

evals failing for dependabot bumps

spark cedar
#

fyi, we've got a core dev architecture discussion coming up on the hour (10 minutes from this message)

#

anyone who wants to is welcome to join (we'll discuss lots of juicy technical details)

wet mason
#

@still garnet πŸ‘‹ Hey ... to start a service with port forwarding manually, what's the process?

I'm doing svc = AsService().Start()

Then for each port, tunnel = Host().Tunnel(svc).Start()

Then to get the local address, endpoint = tunnel.Endpoint()

Wasn't sure whether it's host.tunnel or host.service

tepid nova
#

Host.service is container->host. Host.tunnel is host->container

midnight rapids
tepid nova
obsidian rover
midnight rapids
bronze hollow
#

Started experimenting with Claude to see if it could add new features based on existing examples. Started with GCP secret provider. It needed a lot of guidance but it's getting close.

https://github.com/dagger/dagger/pull/10510

GitHub
feat(secretprovider): add Google Cloud Secret Manager support by ra...

This adds support for using Google Cloud Secret Manager as a secret provider in Dagger, enabling secure secret retrieval from GCP environments.
This was tested with the following command:
DAGGER_TE...

spark cedar
#

(something something gnome-keyring is a bizarre piece of software)

rocky plume
#

does dagql schema support enum in some way? I see there's a withEnum for modules, but i'm not sure if it can be used in the way I want.

I'm working on a new call directory.exists(.....), and it takes an optional expected type, so you can also validate it exists and is a directory, or file. Currently I just have a string I'm passing: https://github.com/alexcb/dagger/blob/2825be594748d0fcd299312f502029282d56f182/core/schema/directory.go#L103 but this doesn't feel robust.

GitHub
dagger/core/schema/directory.go at 2825be594748d0fcd299312f50202928...

An open-source runtime for composable workflows. Great for AI agents and CI/CD. - alexcb/dagger

rancid turret
#

Error extensions

leaden glade
#

I have a first version of the self invocation working with Go modules.
This is similar to what I did in Java, so it doesn't contain shortcuts to access to types (for the environments)
This is what it looks like on my "local-agent" if I remove the local module for the workspace and replace it by a local type: https://github.com/eunomie/local-agent/pull/1
I updated the PR there if anyone wants to try: https://github.com/dagger/dagger/pull/10475
cc @hasty basin

GitHub
replace alpine workspace by a local workspace by eunomie Β· Pull Re...
GitHub
Self invocation: generate types and allow to call the module from t...

NoteThis is the first part of #10336 and #8030

TypeDefs split
To be able to generate types of the module itself, this means it requires to generate and expose (register to the engine) the types an...

rancid turret
#

Telemetry logs

wet mason
#

πŸ‘‹ @civic yacht @still garnet Hey, I'm guessing the answer is no, but would there be a way to catch the "output" (like dagger.ExecError) of a Directory.DockerBuild?

wet mason
#

@civic yacht @still garnet πŸ‘‹ again ...

Dealing with a hang when using the SDK directly, can't seem to find my way around

Shot in the dark but: I should be able to do AsService().Start as many times as i want in the same session, right?

civic yacht
#

Found a bug that would perfectly explain the "no such file or directory" flakes during filesync that have been plaguing our CI: https://github.com/dagger/dagger/pull/10541

Will need to go out in a release to fully squash the flake (since it sometimes happens in the stable "outer" engine too), but hopefully its reign of terror is coming to a close

wild zephyr
#

I've been bitten a few times by /hack/dev/ returning 0 even though the engine compilation actually fails. Has this bitten anyone else? Happy to fix πŸ™

spark cedar
wild zephyr
rocky plume
# wild zephyr glad I posted here πŸ˜„

yeah I've been bitten by it a few too many times too, so I wrote my own hacky hack wrapper:

#!/usr/bin/bash
set -e
set -o pipefail
cd $HOME/dagger/dagger # you might have to change this part

rm -f bin/{dagger,engine.tar}
docker rm -f dagger-engine.dev
./hack/build
test -f bin/dagger || (echo "ERROR: dagger failed to build" && exit 1)
test -f bin/engine.tar || (echo "ERROR: engine failed to build" && exit 1)
echo "== build worked =="
humble hearth
spark cedar
#

so, i've been poking a lot around typedef handling in the engine 😱
when modules register they produce a lot of "incomplete" typedefs - e.g. WithObject used as a reference. the problem is - we never actually fully resolve these, which ends up producing some weird cases, e.g. if you want to look at the possible enum values of an arg, you can't.

if feels like the ModTypeFor function could/should handle this? but not entirely sure how desirable this is for core? kinda a bit confused, it feels almost linked to core interfaces here as well?

still garnet
#

seeing a weird engine panic against ./hack/dev, happened twice in a row (second time was after pulling main) (putting logs in thread since it's a bit of a puke)

hasty basin
#

With Dagger 0.18.10, I'm using finch instead of docker (via symlink trick similar to nerdctl) and it works with no errors shown on CLI

~ ➀ /tmp/bin/dagger -m github.com/shykes/hello call hello
βœ” connect 0.7s
βœ” load module 1.1s
βœ” parsing command line arguments 0.0s

βœ” hello: Hello! 0.0s
βœ” .hello: String! 0.2s

hello, world!

Full trace at https://dagger.cloud/dagger/traces/5d4321a5078de204ca5ecb5ebdab38c5

but getting this error (see image below) in Dagger Cloud πŸ€” @still garnet really an error?
https://dagger.cloud/dagger/traces/42db4f3bad4ab10235d1165ab1055316

Same error with 0.18.9
https://dagger.cloud/dagger/traces/5d4321a5078de204ca5ecb5ebdab38c5
Plus caught another error before this one
https://dagger.cloud/dagger/traces/163e5b86ef9a5986cd4997e05180b713

exec docker inspect --type=image registry.dagger.io/engine:v0.18.9
failed to run command: exit status 1

time="2025-06-10T09:54:16-07:00" level=fatal msg="1 errors: [no such object registry.dagger.io/engine:v0.18.9]"
image.png
still garnet
#

With Dagger 0.18.10, I'm using finch

spark cedar
#

i just did dagger develop πŸ€”

still garnet
#

cloud panic

spark cedar
spark cedar
leaden glade
#

I have a module my-module that has a dependency to a module sub-module. This module has a dependency to a module my-module. And this module has a dependency to a module sub-module.

- my-module (1): dag.SubModule().xxx
  ⎿ sub-module (2): dag.MyModule().xxx
    ⎿ my-module (3): dag.SubModule().xxx
      ⎿ sub-module (4)

The main thing here is that works fine. So I have multiple modules with the same name, and it's fine.
But now, I tried to rename them.
So in my-module (3) I changed the dependencies for:

  "dependencies": [
    {
      "name": "self",
      "source": "sub-module"
    }

And changed the call dag.SubModule() to dag.Self().
This is working fine.
I moved to sub-module (2) and did the same thing. But this time, when I dagger develop, I have the following error:

failed to get schema introspection json during module sdk codegen: failed to get schema for module "self": type "Self"
is already defined by module "self"

While part of me understand the error, the other part is wondering how in the initial state with 2 modules my-module and 2 modules sub-module that was working but with 2 modules aliased to self that doesn't work anymore.
What am I missing here?

still garnet
#

cloud not showing some spans

#

@leaden glade side tangent: have you thought about having dag.Self() (or similar, maybe SDK specific) actually be a reference to the current object instance - i.e. not just the module entrypoint? that should be totally possible, since we already have its ID by the time we're calling the function on it. thinkspin

#

module name conflicts

tepid nova
#

@leaden glade I saw in your update that you're running into difficulties specifically for renaming self-calls to dag.Self. I just want to make sure you're not wasting your time there, since there is a DX debate over whether that's what we actually want. Maybe better to settle the debate before you spend a ton of time on this particular change?

cc @tidal spire

leaden glade
civic yacht
#

CI is failing because of 500s from package.cgr.dev right now.

Also just tried to login to dockerhub for unrelated reasons and am getting 500s from them too. World may be collapsing, just FYI.

obsidian rover
still garnet
#

i think we're back?

leaden glade
#

Sometimes, when things doesn't work as expected, it's time to go back to the drawing table and start (almost) fresh.
So I opened a new PR regarding self invocation: https://github.com/dagger/dagger/pull/10584
This PR only focus on adding moduleTypeDefs to module SDKs (with Go implementation only) and not on self calls in order to keep the PR smaller and more readable, and move step by step.
The PR is in draft because I haven't focused on tests, but happy to get some πŸ‘€ on it

GitHub
SDK: add `moduleTypeDefs` to module SDK, to register type definitio...

NoteThis has been extracted and cleaned from #10475
This is a first step to #10336 and #8030

This PR introduces a new moduleTypeDefs method modules can implement. This method acts like moduleRunti...

wet mason
#

@still garnet hey, can I cast the error as dagger.ExecError in a AsService().Start()?

#

errors.As seems to not catch it

still garnet
#

hmm yeah it might not, since it runs through the gateway; different path, it's more like spawning a process and waiting for its exit status, as opposed to running a LLB exec

still garnet
wet mason
#

@still garnet the problem i'm having is when e.g. the agent writes a app.py in flask, starts it (as a service, otherwise withExec would hang forever), if there's a syntax error or whatever it doesn't "see" it, so it's flying blind

still garnet
#

i can take a look and see if that's a quick fix

wet mason
#

@still garnet sweet! yeah i don't see any workarounds in "userspace" ... I could try to start the service using a withExec and a context timeout, and hopefully if it times out it's because it works, but it's beyond hackish

#

@still garnet oh btw -- a quick one (hopefully):

dagger.PortForward{
    Frontend: rand.Intn(1000) + 5000,

^^^ is there a way to get random allocation for a Host().Tunnel? I believe I tried not setting the Frontend and not getting anywhere

still garnet
#

there's definitely a way, it should be in the godoc

still garnet
#

@wet mason wait, i just wrote a test for getting a dagger.ExecError back and it passed right away

still garnet
wet mason
still garnet
#

no it's optional

// If no port is specified, the first exposed port is used. If none exist an error is returned.
wet mason
#

@still garnet

pseudo-code

    for _, port := ... {
        hostForwards = append(hostForwards, dagger.PortForward{
            Backend:  port,
            Frontend: rand.Intn(1000) + 5000,
            Protocol: dagger.NetworkProtocolTcp,
        })
    }

    // Expose ports on the host
    tunnel, err := dag.Host().Tunnel(svc, dagger.HostTunnelOpts{Ports: hostForwards}).Start(ctx)
    if err != nil {
        return nil, err
    }

    // Retrieve endpoints
    for _, forward := range hostForwards {
        externalEndpoint, err := tunnel.Endpoint(ctx, dagger.ServiceEndpointOpts{
            Port: forward.Frontend,
        })
still garnet
#

ah, there are multiple

#

i guess you could create one tunnel per port, instead of one tunnel for all ports?

wet mason
#

didn't think of that

#

sweet, I'm going to try

still garnet
#

kinda neat how the Zed agent runs the full blown TUI

image.png
wet mason
leaden glade
#

I have this kind of error, but I'm not sure what it means

Error: input: daggerDev call constructor: NotFound: rpc error: code = NotFound desc = socket xxh3:1e5cef82d3912aea not found

For instance when I'm running dagger call test specific --pkg="./core/integration" --run="^TestModule"
Any idea?

spark cedar
spark cedar
leaden glade
spark cedar
#

it feels like either one of two things is happening.
either the typedefs are wrong, and a socket field is somehow getting smuggled, which means the transfer from one module context to another isn't working
or, somehow, a socket is being accessed that shouldn't be

#

it's not immediately obvious to me what's going on, do you have a trace link?

spark cedar
spark cedar
tepid nova
#

πŸ™‹β€β™‚οΈ

leaden glade
still garnet
leaden glade
still garnet
#

Could use a βœ… on https://github.com/dagger/dagger/pull/10562 - it speeds up the evals quite a bit, at the moment they sometimes hit the 20m timeout, post-refactor they take 7m (on the GHA side where the timeout is enforced - actual runtime is ~4m without the dev engine setup)

GitHub
evals: refactor into a repeatable pattern by vito Β· Pull Request #...

The goal: establish a repeatable pattern for evals, decoupling them from Dagger&#39;s own evals.
Evals are now implemented as separate types that implement this interface:
type Eval interface {...

spark cedar
# 
2035: [2m4s] |             --- FAIL: TestLLM/TestAllowLLM/prompt_calls/allowed_unrelated,_calling_direct (71.58s)
2035: [2m4s] |                 testctx.go:173: 
2035: [2m4s] |                         Error Trace:    /app/core/integration/llm_test.go:352
2035: [2m4s] |                                                     /go/pkg/mod/github.com/dagger/testctx@v0.0.4/testctx.go:295
2035: [2m4s] |                                                     /go/pkg/mod/github.com/dagger/testctx/oteltest@v0.0.3/log.go:37
2035: [2m4s] |                                                     /go/pkg/mod/github.com/dagger/testctx/oteltest@v0.0.3/trace.go:83
2035: [2m4s] |                                                     /go/pkg/mod/github.com/dagger/testctx@v0.0.4/middleware.go:25
2035: [2m4s] |                                                     /go/pkg/mod/github.com/dagger/testctx@v0.0.4/testctx.go:149
2035: [2m4s] |                         Error:          Received unexpected error:
2035: [2m4s] |                                         read |0: i/o timeout
2035: [2m4s] |                         Test:           TestLLM/TestAllowLLM/prompt_calls/allowed_unrelated,_calling_direct
#

ok well

still garnet
rocky plume
#

How common is it for the SDK gen jobs to timeout in a PR? I've retried them a few times yesterday, but same issue. I have a number of them that ran for 20 minutes then timed out: https://github.com/dagger/dagger/actions/runs/15720828016?pr=10569 (I just restarted the php-dev job right now to see if it works today).

GitHub
chore: convert withFile to dagop Β· dagger/dagger@94be28c

An open-source runtime for composable workflows. Great for AI agents and CI/CD. - chore: convert withFile to dagop Β· dagger/dagger@94be28c

spark cedar
#

πŸ€”

#

they shouldn't be timing out

#

it may be worth finding the span for the engine logs, and making sure the engine isn't crashing

#

(especially when making a lot of weird internal changes)

rocky plume
still garnet
still garnet
rocky plume
#

there's some retry logic I take it? that

leaden glade
#

I have hard time trying to understand (and fixing) the tests failures on my moduleTypeDefs branch.
I succeeded to fix some of them, but the last ones are like rpc error: code = NotFound desc = socket xxh3:736dee0b5d28a207 not found
Any help appreciated πŸ™
(this branch does not contain anything self invocation, this is just about extracting type defs, so the impact should be minimal. In theory, but I guess in practice something's wrong somewhere)
https://github.com/dagger/dagger/pull/10584

GitHub
SDK: add `moduleTypeDefs` to module SDK, to register type definitio...

NoteThis has been extracted and cleaned from #10475
This is a first step to #10336 and #8030

This PR introduces a new moduleTypeDefs method modules can implement. This method acts like moduleRunti...

still garnet
civic yacht
civic yacht
still garnet
#

yeah I still hit it every now and then - I can link them if you want in the future

civic yacht
civic yacht
# leaden glade I have hard time trying to understand (and fixing) the tests failures on my `mod...

Figured out what's going on, I'll leave a comment on the PR w/ suggestions on how to fix but it has to do with the fact that the new TypeDefs container and Runtime container are different (due to the new --typedefs-only arg) and thus end up as different cache entries. The Runtime container now doesn't get evaluated until the actual function invocation occurs, at which time it's running codegen (again) and looking for the SSH socket that's not available at that time.

civic yacht
leaden glade
leaden glade
fair ermine
#

I see this error popping since 15 minutes when I try to generate something in dagger/dagger, is anyone also hitting that?

package conflicts: [musl]                                       
package conflicts: [musl]

I tried to:

  1. Rerun multiple time -> Same issue
  2. Remove container + volumes -> Same issue
  3. Compile dev engine with ./hack/dev -> same issue
  4. Try from main with ./hack/dev -> same issue
tidal spire
#

I think last time I saw that error there was an issue in wolfi

fair ermine
#

Yeah I think so too, do you remember how we fixed it? Or maybe wolfi update it after so it's just a matter of time

leaden glade
#

It looks like it tries two times to install busybox, first time is working, second not

image.png
fair ermine
#

Looks like it's fixed on main though, I can build the dev engine ; will try on my PR

tepid nova
wet mason
#

@still garnet πŸ‘‹ what's the best way to have a service healthcheck timeout, just a context?

still garnet
#

yeah that seems reasonable

rocky plume
#

random question: I see references to a "shim", and found some git logs where the shim was removed, but I'm still seeing some potential leftovers? e.g.

alex@tyee:~/dagger/dagger$ grep -RI ShimEnableTTYEnvVar
core/service.go:    ShimEnableTTYEnvVar = "_DAGGER_ENABLE_TTY"
alex@tyee:~/dagger/dagger$ grep -RI ENABLE_TTY
core/service.go:    ShimEnableTTYEnvVar = "_DAGGER_ENABLE_TTY"

Is the above used in any way, maybe in some other repo?

I was ultimately trying to figure out how /.dagger_meta_mount is actually used, but my grepping isn't offering much hints.

update: looks like it's used here? https://github.com/jedevc/dagger/blob/fae6ff1aff62c32779a73ea781e6ce186a8c4330/engine/buildkit/errors.go#L313

fair ermine
#

Has anyone ever hit that issue with building the dev engine:

process "/bin/busybox --install -s" did not complete successfully: failed to install cacerts: failed to run [update-ca-certificates] for install: installer command failed: exit code: 1, output: run-parts: No such file or directory

https://dagger.cloud/Quartz/traces/4995938c73e14b387a4466a73be0c595?listen=c53c20ba8679e44c&listen=f255c7836374f690&listen=7eddf3e19ebdaaa8&listen=73dbedcd06747b3c&listen=429c6ce42fc5a4e3

I'm not able to build a dev engine nor run integrations tests because of that

Dagger Cloud

Browse and visualize Dagger traces.

#

It works fine with --cloud but I cannot use --cloud to compile a dev engine (or maybe I can?? If so, how ?)

still garnet
fair ermine
humble hearth
#

Is there a way to spin up the GraphQL server so I can setup introspection? Trying to run some stuff against the dagger query command

still garnet
humble hearth
humble hearth
#

Not sure how many people use VS Code for Dagger dev, but I loaded up dagger/dagger in the VS Code extension I'm working on.... that is a whole lot of functions, might need to nest those under the module name πŸ‘€

humble hearth
hasty basin
# humble hearth

much better! Was wondering what each of the 20 lint functions was for πŸ˜„

fair ermine
# humble hearth

Is it a new dagger vs code extension? That would be so cool to call the function from the IDE directly haha, would be quite tricky for function that requires a lot of parameters but definitely cool!

humble hearth
# fair ermine Is it a new dagger vs code extension? That would be so cool to call the function...

It sure is, it looks at the function args and prompts for user input, still need to double check some types that we support but it is working πŸ₯³

I was parsing the cli output but switched to calling dagger query

https://x.com/mccallisterdev/status/1939698146839335338?s=46

Jason McCallister (@mccallisterdev)

πŸš€ Update! The VS Code extension for @dagger_io now has a smoother UI for exploring and running functions!

You can also save function calls as @code Tasks for quick reuse (without providing args each time!). This makes running workflows even easier right from your editor.

X Β·
fair ermine
leaden glade
#

I wanted to build container-use binary using dagger on my mac

dagger -c 'build | export ./cu'

The build itself went well, but the result isn't what I expected:

$ file ./cu
./cu: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=bda00bf1d955276690c4b73c9fbaf6a1e0b373a7, with debug_info, not stripped
$ ./cu --help
zsh: exec format error: ./cu

My understanding is about an issue on the go module and the way the default platform is handled: https://github.com/dagger/dagger/blob/a569ec610a25df6c94cfe4bb55be02a8ade11f32/modules/go/main.go#L180-L182
If no platform is provided (empty string) then we don't setup the GOOS and GOARCH while, from my current understanding, we should instead use dag.DefaultPlatform() that returns the host platform (and not the engine platform, that for instance runs in a linux container on a mac host)

❓ Should I update modules/go to handle the default platform that way?

dag.DefaultPlatform() can return an error.
In the modules/go code it's used inside a With(f func(r *Container) *Container) *Container block. But this doesn't allow to return an error. That's not the first time I'm in this situation.

❓ Should we had something like a With(f func(r *Container) (*Container, error)) *Container (maybe with a different name than With so that we can raise issues in some way? Or maybe there's a better pattern? Usually I just not use With when I'm in this case but it means I'm losing the chainable aspect

GitHub
dagger/modules/go/main.go at a569ec610a25df6c94cfe4bb55be02a8ade11f...

An open-source runtime for composable workflows. Great for AI agents and CI/CD. - dagger/dagger

tepid nova
#

"platform modules" design... 🧡

spark cedar
spark cedar
# leaden glade I wanted to build `container-use` binary using dagger on my mac ``` dagger -c 'b...

to workaround you can do --platform=current

❓Should I update modules/go to handle the default platform that way?
Issue here: https://github.com/dagger/dagger/issues/8354
It needs to be part of the args, and not a dag.DefaultPlatform, otherwise, different platforms will cache to the same value.

❓ Should we had something like a With(f func(r *Container) (*Container, error)) *Container (maybe with a different name than With so that we can raise issues in some way?
Issue here: https://github.com/dagger/dagger/issues/9343 and https://github.com/dagger/dagger/issues/8402#issue-2518178288?
We did discuss before, discussion seems to have stalled.

leaden glade
spark cedar
#

am i missing a trick? it looks like if the engine dies, if i've got a shell session open, i can't get it to close without killing it, any combination of commands doesn't seem to get it to exit.
(discovered cause i found a weird panic bug in the engine)

tepid nova
#

Thread to make progress on persistent env POC this week... 🧡

tepid nova
#

question for @spark cedar @civic yacht @rocky plume : does Theseus get us closer to building the whole CLI + engine into a single binary like normal people (as opposed to gluing together various binaries in a OCI wrapper?)

spark cedar
tepid nova
#

I thought maybe getting rid of buildkit changed the equation for packaging

#

but I guess we already used it as a library... And there's still runc and its plugins

#

And containerd I guess?

spark cedar
#

yeah, i think we're quite far away from having a single binary (that's not something that contains an oci image itself)

#

theseus doesn't make it any harder, potentially we're able to be less relieant on a long running daemon - since we'd own the call points. but aside from that, i'd say it's pretty much entirely different

rocky plume
civic yacht
#

Evals on main failed, looks like it's catching something involving interfaces that weren't caught by the actual interface integ tests themselves (πŸŽ‰ I guess). Almost certainly from https://github.com/dagger/dagger/pull/10620 so just an FYI that it's ack'd and I'm looking into it

tepid nova
#

Quick core internal question... Does core.TypeDef allow setting a default value for an argument of type Secret, or other object types?

#

@spark cedar @leaden glade @fair ermine πŸ‘†

#

I see this in core:

func (fn *Function) WithArg(name string, typeDef *TypeDef, desc string, defaultValue JSON, defaultPath string, ignore []string, sourceMap *SourceMap) *Function

But in the case of an object type, I'm not sure what to pass as defaultValue of type JSON. Maybe the object ID? Would that work?

spark cedar
#

if you set it as the object id, then i think that miiiight work?

tepid nova
spark cedar
#

yup it would 😦

tepid nova
#

hmmm

spark cedar
#

i don't think we've thought about how we'd do default objects

tepid nova
tepid nova
tepid nova
# tepid nova I have a real-world example of hard-to-read error: https://dagger.cloud/dagger/t...

Command:

dagger -m github.com/sagikazarmark/daggerverse/gh@v0.11.0 | with-repo https://github.com/dagger/dagger.io | clone

Console output:

β–Ό gh | with-repo https://github.com/dagger/dagger.io  | clone 0.5s ERROR r jump ↴
β•°β”€βœ˜ clone 0.5s ERROR r jump ↴
  ! input: gh.withRepo.clone failed to stat file /tmp/repo: process "gh repo clone https://github.com/dagger/dagger.io /tmp/repo" did not complete successfully: exit
    code: 4
! input: gh.withRepo.clone failed to stat file /tmp/repo: process "gh repo clone https://github.com/dagger/dagger.io /tmp/repo" did not complete successfully: exit
  code: 4

Web trace default output: same in web form

Web trace after drilling down: see link above πŸ‘† . The actual error was hidden in the logs of an exec:

To get started with GitHub CLI, please run:  gh auth login
Alternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token.
still garnet
#

hard-to-read errors in shell/cloud

tepid nova
#

I'm looking for the best place to hook into the loading of the session's top-level module... I successfully hooked ModuleObject.Install but it hooks every module. How do I hook into only the top-level module?

tepid nova
#

what's the authoritative way to find the main object of a given module? Compare the lowercase object type name to the module original name?

tepid nova
tepid nova
#

FYI

wet mason
#

@civic yacht @spark cedar @rancid turret Hey, not sure who's familiar with engine provisioning nowadays ...

Running into a deep paper cut when the SDK and CLI versions do not match, does this ring a bell?

https://github.com/dagger/container-use/pull/206

/cc @obsidian rover

obsidian rover
civic yacht
#

Not that the version being off by a patch version between the two should result in an error necessarily, just checking

wet mason
civic yacht
#

either way, yeah sounds like something very wrong is happening

civic yacht
#

engine got replaced with v0.18.11 and everything

wet mason
still garnet
civic yacht
still garnet
#

ah nice. i wonder if there could be a default heuristic, like 'if == version found, keep, if only version is <, destroy and replace' - i.e. don't destroy a newer one

#

env var is good enough for me, just thinking out loud

civic yacht
#

It happened once but we shrugged it off

civic yacht
obsidian rover
obsidian rover
spark cedar
#

there's also nothing anywhere in the logs that indicate a test failure

#

it's like there are a lot of test failed spans, but with no information as to what is causing the failure

charred lotus
#

@spark cedar do you know how i can pass -i when using Go SDK ?

spark cedar
#

to get an interactive session on failure? πŸ€” i don't think you can, unless the entire thing is wrapped in a dagger run (you'd add it to dagger run -i)

charred lotus
#

Thanks!

spark cedar
still garnet
#

np! when in doubt you can check the go test output

spark cedar
#

indeed, i'm kinda struggling through in cloud, the frontend is trying to render 60k lines of ca cert oputput and struggling

still garnet
#

i see 'race detected' in the logs

spark cedar
#

AH

#

well that'll be it

#

of course it is

still garnet
#

yeah those are always heinously difficult to track down

#

wish i could hook into it somehow

#

i see 'race detected' in the logs

still garnet
spark cedar
#

both chromium and firefox struggle for me

still garnet
#

you can also try filtering for WARNING: DATA RACE and expanding down from the match, since that filters before rendering

#

(or "FAIL")

spark cedar
still garnet
#

firefox here but on a pretty beefy laptop

spark cedar
#

😒

image.png
#

it do be struggling

#

is that intentional btw? even just doing nothing, and everything seems loaded, it still seems to be sitting pegged at 100% on one core, a couple minutes later

still garnet
#

oh yes totally intentional

#

have to keep those laps warm

#

anything in the console logs?

spark cedar
#

sorry πŸ˜› poor phrasing indeed

still garnet
#

lol np

spark cedar
#

"all logs rendered" is the last thing

#

nothing seems to be continuing to happen

still garnet
#

hmm mind capturing a profile?

spark cedar
#

yeahhh

#

gimme a mo (edit: we continued thread of screencasts/profiles in dm)

fair ermine
#

CI typescript-dev job is failing on main, it seems I forgot the regenerate the client after a PR was merged (and it wasn't caught by the CI in the PR for an unknown reason)
Don't panic, it will be fixed in 5 minutes πŸ™‚

spark cedar
tepid nova
#

Quick update:

  1. Added a config file to the POC for default secrets https://github.com/dagger/dagger/pull/10697

  2. Resurrecting "platform module", it's embarrassing that we still haven't shipped a solution after 2 months of very clear feedback after POC

I welcome help on both of these

#

@spark cedar I'm rebasing on main, and it looks like dagql.Instance is no longer a thing? Is it easy to replace it with dagql.Result? Any trapdoors?

#

looks like I just replace .Self with .Self() and I'm good to go basically?

still garnet
#

Instance => ObjectResult, .Self => .Self() yep

tepid nova
#

Dev toolchain problem: I'm adding calls to the engine API, and using them in the CLI. The CLI fails to build because the new calls aren't defined in the client lib. I can't generate with client lib because dagger -m ./sdk/go -c generate somehow has a dependency on the CLI and can't load. Circular dependency?

#

I'm currently stashing my CLI changes so that I can re-generate the client then unstash the CLI changes... I expect/hope it will work but it's a bit embarrassing

#

(update: it worked)

tepid nova
#

I've been working on the blueprint module PR, I'm stuck on a one obscure issue in the module loading logic... Might need a little help

civic yacht
#

I've been working on the blueprint

tepid nova
#

Like there's an error on 'test-publish' because some nix install script doesn't want to install nix as root? thinkspin completely irrelevant to my change

spark cedar
tepid nova
spark cedar
#

upstream changed their install script to require coreutils' cpπŸ€·β€β™€οΈ

tepid nova
#

Should we pin our install scripts in general?

#

(to avoid this kind of thing)

spark cedar
#

probably, there's a lot of stuff we should pin - i don't have time to look through right now, i'm also not sure where they host their archive of old install scripts

tepid nova
#

@spark cedar I'm getting a few other failures that looks like flakes... are there known flakes at the moment?

spark cedar
tepid nova
#

looks like the CI runner crashed under the checks feet or something

spark cedar
#

the java check failure looks ephemeral

#

TestBlueprint also seems to be failing

tepid nova
#

But everything else seems unrelated

spark cedar
#

yup, seems like alpine infra is having a moment

tepid nova
#

This is my local loop for re-running the tests with "real" failures: dagger -c 'test | specific --run="TestBlueprint.*"' but it takes forever to re-run even on a hot cache, is there a way to do it with less overhead?

hasty basin
#

Should
dagger -c 'dev --image alpine | terminal' give me an alpine-based dev engine?

cat /etc/os-release
ID=wolfi
NAME="Wolfi"
PRETTY_NAME="Wolfi"
VERSION_ID="20230201"
HOME_URL="https://wolfi.dev"
BUG_REPORT_URL="https://github.com/wolfi-dev/os/issues"

Oh, maybe ALPINE...but that should be default according to

dagger -c '.help dev'

Creates a dev container that has a running CLI connected to a dagger engine

USAGE
  dev [options]

OPTIONAL ARGUMENTS
  --gpu-support bool        Enable experimental GPU support
  --image DaggerDevDistro   Set target distro (default: ALPINE) (possible values:  <<<<<<<<<<<<<<<<<<<
                            ALPINE, WOLFI, UBUNTU)
  --shared-cache bool       Share cache globally
  --target Directory        Mount a directory into the container's workdir, for
                            convenience

No joy with --image ALPINE πŸ€”

hasty basin
#

Should

final star
tepid nova
#

What's the protocol for using the contents of core/integration/testdata from the test suite? Can I rely on a predictable workdir, or some other method?

civic yacht
tepid nova
#

πŸ™‹β€β™‚οΈ I might need help getting blueprints over the finish line.. I'm hitting edge cases in module loading that I don't know how to solve

#

and the devloop is so slow...

tepid nova
#

Here's my issue, in case anyone has an idea...

  • When I run dagger install ../foo it adds the relative path in dagger.json
  • When I run dagger init --blueprint=../foo it adds the absolute path in dagger.json. Even though I tried to reuse as much code as possible from dagger install. I'm doing something differently and I can't trace what exactly
#

update: claude may have figured it out?

tepid nova
#

got stuck a little further down...

#

@spark cedar I got ensnared in engine version compat checks... Doing something that doesn't relate to it at all. Maybe you can help?

#

in the CLI implementation of dagger install:

modSrc = modSrc.WithDependencies([]*dagger.ModuleSource{depSrc})

// If I don't call this... πŸ‘‡
if engineVersion := getCompatVersion(); engineVersion != "" {
  modSrc = modSrc.WithEngineVersion(engineVersion)
}

_, err = modSrc.
  GeneratedContextDirectory(). // ...then this πŸ‘ˆ fails. Why?
  Export(ctx, contextDirPath)
spark cedar
#

what does it fail with?

tepid nova
#

module requires dagger , but support for that version has been removed

#

Context is that I'm trying to wrap moduleSourceWithDependencies from moduleSourceWithBlueprint, to avoid duplicating all the special logic in there.

But in order to do that, I must also call loadModuleSourceConfig from moduleSourceWithBlueprint. And it fails with the error above.

spark cedar
#

weird, um i don't really know

#

i don't think you can call that function from there, i think it needs to be called from a fully configured module?

#

and it's not configured at that point?

#

(sorry, multitasking hard here)

tepid nova
#

It should be, it's a function attached to the ModuleSource type (eg. you can call ModuleSource.withBlueprint in the Dagger API)

tepid nova
spark cedar
#

somehow the engineVersion is being set to the empty string - i'm not quite sure how that's happening

#

if you have a branch pushed, i can try and peek

tepid nova
#

I think I finally figured it out

tepid nova
charred lotus
#

Is it possible to do a WithExec ignoring the exit code ? Nevermind I find the ANY of ContainerWithExecOpts.Expect

spark cedar
#

been working on support for more driver types (podman, containerd, apple containers, finch, etc)
https://github.com/dagger/dagger/pull/10714
it's pretty much there? but since it's changed a bit from my original proposal, would love some more feedback ❀️

#

^ fyi @tepid nova @tidal spire @hasty basin

final star
spark cedar
#

also, I'm biased, I'm maybe looking at daily driving nerdctl daily, but I need good dagger support to give it a go

tepid nova
#

btw it would probably easy for the default driver to auto-detect either docker, podman, or nerdctl. That way all those users would get zero-config local provisioning out of the box

spark cedar
#

😁 great minds think alike

#

Additionally, we also have support for the generic image + container drivers (which becomes the new default) - these behave the same, but auto select from the above based on what the user has installed on their system.

#

the preferred order is this:

image.png
tepid nova
#

I would move docker to last in that list, because it's common for it to be installed (or aliased) alongside one of the others. In those cases, the user probably prefers to not use docker

#

in other words: if container, podman, finch or nerdctl are available, they should be used, even if docker is also available

spark cedar
#

maybe - that's a breaking change though, if users have anything installed, dagger will start running somewhere else instead

#

i'm happy to change the order up later, but i'd also rather testdrive the new logic before we switch everyone over to it (there might be tiny incompatibilities i haven't found in testing)

tepid nova
#

yeah that makes sense

spark cedar
#

fyi, that pr is probably chill to merge whenever? not gonna try and rush it for the release though, there's no urgent need

#

(i'm also on holiday tomorrow, and don't wanna be scrambling to fix something)

tepid nova
#

@spark cedar re: JSONValue. I'm interested in your idea for replacing setString, setInt, setBool with a generic set()

tepid nova
#

<@&946480760016207902> what's the suitable replacement for dagql.Object?

#

Looks like dagql.AnyResult, not sure if it's as simple as search-replace, or if there's more to it?

spark cedar
tepid nova
spark cedar
#

AnyResult should probably be it then

tepid nova
#

I think ObjectResult replaces Instance

spark cedar
#

what are you working on for this?

tepid nova
#

I guess a more accurate name would be AnyObjectResult since it's only for types that have an ID

spark cedar
#

or can - even if they're scalar values (needed so we can cache them)

tepid nova
#

(very slowly)

tepid nova
#

Not using graphql scalars anymore?

spark cedar
#

it's internal - you can't query them

#

but they still have ids associated with them

tepid nova
#

Cool cool cool

spark cedar
#

new generic id types should probably be able to handle that

tepid nova
#

Yeah my patches are pretty superficial, as long as there's a way to designate "anything with an ID" in the schema in the code, should be good

(in the schema it's even simpler: I just define a new scalar "ID" and that's it)

civic yacht
tepid nova
#

Obscure engine/dagql question if anyone is around: I use dagql.Server.Select() to query the engine API from within the engine API. It works fine, except when passing an optional argument.

Example:

err = s.srv.Select(ctx, s.srv.Root(), &inst, dagql.Selector{
    Field: "secret",
    Args: []dagql.NamedInput{
        {
            Name:  "uri",
            Value: dagql.NewString(uri), // βœ… required argument
        },
        {
            Name:  "cacheKey",
            Value: dagql.NewString(cacheKey), // ❌ optional argument
        },
    },

I get this error:

select: failed to compute cache key for Query.secret: assign input "cacheKey"
  (dagql.Optional[github.com/dagger/dagger/dagql.String]) as  (dagql.String): assign:
  Setter.SetField dagql.String to dagql.Optional[github.com/dagger/dagger/dagql.String]: cannot
  set field of type dagql.Optional[github.com/dagger/dagger/dagql.String] with dagql.String
#

<@&946480760016207902>

#

Also @rancid turret I'm working on extracting some CLI core to the engine (the "resource loader" that loads a directory, container, secret, file or socket from a single command-line flag). But there are a few strings that I don't know how to detach... Could you help me navigate that?

#

(I see you online - welcome back? πŸ™‚

rancid turret
tepid nova
#

going for a quick lunch, but in short: I'm moving the custom flag values from CLI to engine API. See https://github.com/dagger/dagger/pull/10753

I started with directoryValue, I got a basic implementation to work (host | resource ADDRESS | as-directory). But I had to drop modArg.Ignore because I have no idea how that works...

So my question is: what is modArg.Ignore and how do I transpose it to the engine-side implementation?

GitHub
Host resources by shykes Β· Pull Request #10753 Β· dagger/dagger

A new feature in the engine API: load resources from the host using a single standardized address.
This is a feature that already exists in the CLI, but we&#39;re moving in the engine so that i...

still garnet
tepid nova
#

I looked for examples in the code but hard to find

still garnet
#

yeah I guess we don't hit that a lot. I looked for another place with a similar pattern and it does 'required/non-null but with default ""' instead of a nullable value so that must be why

tepid nova
#

@still garnet while I have you... πŸ‘Ό

I have a []core.PortForward and trying to pass them as argument to Host().Service()

var ports []core.PortForward
err = s.srv.Select(ctx, s.srv.Root(), &inst,
    dagql.Selector{
        Field: "host",
    },
    dagql.Selector{
        Field: "service",
        Args: []dagql.NamedInput{
            {
                Name:  "host",
                Value: dagql.NewString(host),
            },
            {
                Name: "ports",
                Value: dagql.ArrayInput[dagql.InputObject[core.PortForward]]( /* ???? */ )
                ),
            },
        },
    },
#

I tried this:

                    Value: dagql.ArrayInput[dagql.InputObject[core.PortForward]](ports),

But it says invalid conversion

#

ah wait maybe I can just change the type of ports and just pass it

still garnet
#

yeah that should work

tepid nova
#

But once I have a var ports dagql.ArrayInput[dagql.InputObject[core.PortForward]]

I don't know how to append to it

# 
ports = append(ports, dagql.InputObject[core.PortForward]{
            Backend:  nPort,
            Frontend: &nPort,
            Protocol: core.NetworkProtocolTCP,
        })

--> unknown field Backend in struct literal of type dagql.InputObject[core.PortForward] (compiler MissingLitField)

#

OK I think I figured it out

#

But it hurts

still garnet
tepid nova
#

Heads up I'm going to send a request for API bikeshedding soon πŸ™‚

#

Has anyone run into the issue where dagger -c 'generate | export. ' suddenly goes haywire and removes a gazilion files from the repo?

#

Note: I interrupted the export after 4mn because it shouldn't take that long, even with slow internet...

And export without --wipe is supposed to be additive, so it shouldn't be possible for files to be removed, even if I interrupt the filesync

#

Note: it resolves itself in the end. But yeah there seems to be a race condition where an interrupted sync will wipe some files

tepid nova
tepid nova
tepid nova
#

Flagging: I think we have a filesync performance problem. I'm on starlink (good bandwidth, meh ping, maybe some packet loss?) it's barely usable on the dagger codebase with small edits

#

Also that performance problem is, I think, surfacing some non-atomic file wiping issues

spark cedar
tepid nova
#

(forgot to mention that this is on a remote engine eg. parc)

spark cedar
#

if we start doing more parc stuff, i'd definitely want to fix that (e.g. not just internal use)

tepid nova
#

At the moment I switched to git-pushing my changes, then loading straight from remote git

spark cedar
#

but it's not super trivial to just fix

tepid nova
#

@spark cedar how do you think it compares to regular buildkit? Same? Better? Worse?

spark cedar
#

i want to try using an rsync backend for remote cases? i think that would be a lot more stable + performant for remote cases (and for local cases, i think we could use direct mounts)

#

probably a little bit worse? our "protocol" is a bit chatty

tepid nova
spark cedar
tepid nova
#

Maybe there's a possible combo between 1) live dev services, and 2) improved filesync?

The first is about exposing filesync in the API; the second is about. making filesync perform better.

spark cedar
#

mmmm yeah potentially πŸ€” I've got a proposal for the live sync on my back burner, was hoping to have something this week, but sigh, I've got caught up

#

i think there's some overlap, but they are different, there's no one perfect file sync protocol out there that solves every problem we have

tepid nova
#

right. I'll settle for "average performance" πŸ™‚

#

So 3 options off the top of my head:

  1. rsync
  2. mutagen?
  3. @night moat 's thing πŸ™‚
tepid nova
#

I like this one better πŸ™‚

night moat
#

hi

#

reading the backlog

tepid nova
#

@night moat TLDR:

  • Dagger wraps buildkit (aka the backend of docker build)
  • Dagger inherits buildkit's filesync feature, which uploads local files & ingests into the DAG
  • buildkit (and dagger) support remote hosts seamlessly, but the protocol is slooooow
  • We are removing the buildkit dependency. As a result we control the filesync implementation.
  • For now we have simply copy-pasted the buildkit protocol, but it's within our reach to swap it out for something better
  • Meanwhile we started using remote engines much more. Which makes the need for a better filesync protocol more and more acute.
  • We were just discussing next steps to deal with all that
  • An important step is to choose a protocol to implement. In addition to the useful suspects (rsync, mutagen) maybe there's something awesome you have built which would be better?
night moat
#

it's just for file synchronization ? as in you'd like a better rsync ?

tepid nova
#

And, it's for ingestion into our content-addressed data layer. So just naively wrapping a vanilla rsync on both ends isn't enough

night moat
#

mh, need to think about it but surely there's something that could be done with ptar

#

whose working on that on your side ?

spark cedar
night moat
#

okie, I have a 1h call but then I'm available to see if I can help if you want

tepid nova
spark cedar
#

i was wondering about having a generic scalar value in graphql. like any "any" type. essentially something that to decode a string, would just store the json string - then you'd be able to lazily interpret it later

#

essentially the equivalent of json.RawMessage in go

#

it might not be possible, but it feels like if it is, that might be preferable?

tepid nova
spark cedar
#

fair, that was my concern with the original proposal

#

i think i was envisioning something similar to how the gjson lib works (we use it in our integration tests)

#

they have quite a terse api imo

tepid nova
#

😭😭😭😭

Screenshot_2025-07-21_at_17.01.42.png Screenshot_2025-07-21_at_17.02.21.png
tepid nova
tepid nova
tepid nova
#

Is this normal / expected?

mkdir -p temp/dir temp/otherdir
cd temp/dir
dagger -c 'host | directory ../otherdir'

! path "../otherdir" escapes workdir; use an absolute path instead
tepid nova
obsidian rover
#

I am surprised that even when temp is a git repo (tried it), the behavior is the same. Otherwise, it's been a while, i don't recall exactly

tepid nova
#

Yeah it' shardcoded in the engine

#

I'm removing it, will see if anyone objects in the code review πŸ™‚

civic yacht
tepid nova
#

I seem to have broken integration tests, but in a way that seems impossible? 😭

context: https://github.com/dagger/dagger/pull/10770

span: https://dagger.cloud/dagger/traces/30dba1e038b89804b4b9797115f3203b?listen=147dec09e27a1889#487c38c17fc8d94d

        Error:          Not equal: 
                        expected: "shhh"
                        actual  : "cmd shhh"

Relevant code: https://github.com/dagger/dagger/blob/fd03e5fb5e950361e72d80307fa0eac129523fde/core/integration/module_call_test.go#L553-L602

**My failing test is getting the result from a totally different test??? 😭 😭 😭 😭 😭 😭 **

#

I guess that means my implementation creates massive side effects between calls?

#

but how 😭

#

I don't really understand how that's even possible. Also naive testing in a dagger shell (same session) shows no issue..

#

Clearly I did something wrong in the implementation but πŸ€·β€β™‚οΈ

#

Oooh maybe it's this:

dagql.Fields[*core.Address]{
  dagql.NodeFuncWithCacheKey("secret", s.secret, /* πŸ‘‰ */ dagql.CachePerCall /* πŸ‘ˆ */ )
#

@civic yacht so I wrap { secret } with { address { secret } }

Since the former used dagql.CachePerCall, the wrapper does too. Then the wrapper calls the wrappee with dagql.Server.Select(). Could that somehow cause my issue? Some sort of caching version of crossing the beams?

tepid nova
obsidian rover
#

dagger's registry auth on windows is a bit more complex than i thought ... 🀯 Can't seem to download public ubuntu atm and truly don't understand why the docker.json is not satisfied. Any recall of such issue @hasty basin ? Do users just put base64'd username:psswd ?

PS C:\Users\Guillaume\coding\container-use> cat C:\Users\Guillaume\.docker\config.json
{
          "auths": {}
}

PS C:\Users\Guillaume\coding\container-use> dagger shell

β–Ό container | from ubuntu | terminal 0.3s ERROR
╰─▼ .from(address: "ubuntu"): Container! 0.3s ERROR
  ! failed to resolve image "docker.io/library/ubuntu:latest" (platform: "linux/amd64"): failed to resolve source metadata for docker.io/library/ubuntu:latest: failed to
    get credentials: error getting credentials - err: exit status 1, out: `A specified logon session does not exist. It may already have been terminated.`
  ╰─▼ resolving docker.io/library/ubuntu:latest 0.3s ERROR
    ! failed to get credentials: error getting credentials - err: exit status 1, out: `A specified logon session does not exist. It may already have been terminated.`
    ╰─▼ remotes.docker.resolver.HTTPRequest 0.3s ERROR
      β•°β”€βœ˜ HTTP HEAD 0.3s ERROR
! failed to resolve image "docker.io/library/ubuntu:latest" (platform: "linux/amd64"): failed to resolve source metadata for docker.io/library/ubuntu:latest: failed to get
  credentials: error getting credentials - err: exit status 1, out: `A specified logon session does not exist. It may already have been terminated.`

container-use β‹ˆ

docker pull ubuntu works πŸ€” ; Ok a user had it: https://github.com/dagger/dagger/issues/6043. Installed via chocolatey too. Investigating πŸ‘€

Update: changint the docker.json to below fixed it. Investigating:

{
        "auths": {
                "https://index.docker.io/v1/": {}
        }
}

It's mostly the windows creds helper that seem to break -- not sure if it's just my setup

tepid nova
#

πŸ‘‹ my PR is blocked by a failing "security scan", a vulnerable npm dependency somewhere in the docs... But that's unrelated to my issue, IMO it doesn't make sense to block every unrelated PR on this

tepid nova
#

I also have a weird failing test in the telemetry suite: https://dagger.cloud/dagger/traces/9ea712debf08a99f5b148f5d3b094444?span=1ad8bacba9bb4456

--- expected
+++ actual
@@ -38,5 +38,5 @@
 ● viztest: Viztest! X.Xs
 β–Ό .list(
-β”‚ ┆ dir: Host.directory(path: "/app/dagql/idtui"): Directory!
+β”‚ ┆ dir: Address.directory: Directory!
 β”‚ ): String! X.Xs
 ┃ .git/


You can run 'go test . -update' to automatically update testdata/TestTelemetry/TestGolden/list to the new expected value.'
Dagger Cloud

Browse and visualize Dagger traces.

tepid nova
spark cedar
#

that's why the telemetry tests are failing fyi

#

i've seen the elixir one on main? not quite sure about, still needs someone to dig into it

spark cedar
tepid nova
#

The viz issue seems legit, the trace did change, and apparently you have to regenerate the golden data to reflect that

tepid nova
spark cedar
#

sometimes it's the same

tepid nova
tepid nova
#

If there's a test verifying that a given dagger call outputs a specific visualization to the terminal, it's normal that the test fails

#

I just don't know how that test works, or how to update the expected visualization

spark cedar
#

oh sure

#

yeah, just run dagger call test telemetry --update -o .

#

i missed that, there's also a flake

tepid nova
#

Meta-feedback: would be cool for cloud to surface specifically failed tests (since we have custom spans for them), and show them in a nice clean list view - separate from trace view

#

not just cool - amazing

#

almost like auto-issues that you can list and go through, like a checklist

#

Right now it requires spelunking deep across several traces

#

Trying: dagger -m github.com/shykes/dagger@address-api -c 'test | telemetry --update | export .'

#

(oh man, imagine if you could export straight to the current git branch πŸ™‚ )

tepid nova
spark cedar
tepid nova
#

ok πŸ‘

#

@spark cedar once this is finally green, I'm moving on to jsonvalue, trying an improved API with your feedback in mind

tepid nova
#

@fair ermine I'm getting a deadlock from new Typescript bundling code, while developing new engine API

# 
dagger -m github.com/shykes/dagger@jsonvalue -c 'sdk | go  | generate

Error:

β–Ό .withExec(args: ["tsc", "--emitDeclarationOnly"]): Container! 2.7s ERROR
src/api/client.gen.ts(6755,23): error TS2552: Cannot find name 'JsonvalueGetOpts'. Did you mean 'JSONValueGetOpts'?
src/api/client.gen.ts(6770,27): error TS2552: Cannot find name 'JsonvalueGetBoolOpts'. Did you mean 'JSONValueGetBoolOp
src/api/client.gen.ts(6785,26): error TS2552: Cannot find name 'JsonvalueGetIntOpts'. Did you mean 'JSONValueGetIntOpts
src/api/client.gen.ts(6800,27): error TS2552: Cannot find name 'JsonvalueGetJsonOpts'. Did you mean 'JSONValueGetJsonOp
src/api/client.gen.ts(6815,29): error TS2552: Cannot find name 'JsonvalueGetStringOpts'. Did you mean 'JSONValueGetStri
'?
src/api/client.gen.ts(6830,24): error TS2552: Cannot find name 'JsonvalueSetBooleanOpts'. Did you mean 'JSONValueSetBoo
ts'?
src/api/client.gen.ts(6838,24): error TS2552: Cannot find name 'JsonvalueSetIntegerOpts'. Did you mean 'JSONValueSetInt
ts'?
src/api/client.gen.ts(6846,21): error TS2552: Cannot find name 'JsonvalueSetJsonOpts'. Did you mean 'JSONValueSetJsonOp
src/api/client.gen.ts(6854,23): error TS2552: Cannot find name 'JsonvalueSetStringOpts'. Did you mean 'JSONValueSetStri
'?
src/api/client.gen.ts(6863,19): error TS2552: Cannot find name 'JsonvalueUnsetOpts'. Did you mean 'JSONValueUnsetOpts'?
! process "tsc --emitDeclarationOnly" did not complete successfully: exit code: 2
#
  • Re-generating the SDKs requires building the engine...
  • ...which requires bundling the SDKs...
  • ...which requires up-to-date SDK...
  • ...which requires re-generating the SDKs ☠️
#

How do I break the deadlock?

# 
312255260f .dagger/build/sdk.go            (Vasek - Tom C   2025-04-22 22:10:46 +0200 158)              WithExec([]string{"tsc", "--emitDeclarationOnly"}).
#

looks like a name capitalization issue?

#

If anyone is around who could help me out, let me know. I'm spelunking deep in the guts of the engine & sdk build, just so I can develop a trivial API change

fair ermine
#

That"s weird what's happening, we already have a case convertor

#

Do you have a PR link?

tepid nova
#

Also you can repro with the command I shared

#

dagger -m github.com/shykes/dagger@jsonvalue -c 'sdk | go | generate'

#

(not specific to go sdk)

#

(sorry wrong PR link - fixed)

tepid nova
#

@still garnet if I wanted to develop a custom visualizer (eg. an alternative to plaintext or tui) where would I start? Any special knowledge to impart on me that would avoid lots of blind searching?

rancid turret
#

Module .refresh

still garnet
#

developing custom frontends

tepid nova
#

I started the day with 2 almost-done PRs and failing CI... Finished the day with 2 almost-done PRs and failing CI 😭

obsidian rover
#

Quick question: i'm fighting the windows github runners to use docker on the WSL (and it's starting to win ahahah 😒 )

What I want: for docker on windows to handle linux container:

docker version --format "{{.Server.Os}}"
linux

What I get: windows, making it impossible to test the windows dagger inside those runners.
Is there a workaround ? Can't seem to make this wsl work (kernel issues, hyperV disabled)

But, wsl seems to be enabled, so i don't understand why it isn't theoritically possible ? πŸ€”

spark cedar
#

I've burnt like 2 days on this before

#

it feels possible right? 😁

#

no specific advice, just good luck, windows on GitHub actions is not my favorite

obsidian rover
#

I kinda have it -- dagger still breaks , getting cloooose -- but hacky

obsidian rover
# spark cedar indeed

one question: if the windows cli connects to a remote engine via the experimental env var, engine located inside the WSL2, would that be acceptable ?

civic yacht
# obsidian rover one question: if the windows cli connects to a remote engine via the experimenta...

Is the goal to test dagger clients (i.e. the CLI and such) running on windows? If so, then yes the engine being in WSL2 would be fine IMO.

I guess it would be nice to have tests against the "official" docker-on-windows (since that's probably closer to what users do), but we would gain quite a bit just from having tests for dagger clients on windows, so if WSL2 is what we can get working I'd say go with that

obsidian rover
#

Regarding the "official" docker-on-windows, I'm actually working rn against a windows instance that was originally designed for that. We could totally add a user and have it part of the CI (don't know how exactly), but it's kinda here ahah

civic yacht
#

One idea if you didn't consider it already: you could setup just the dagger client on Windows and then use PARC to connect to a linux engine for the actual tests? Then you don't need to play around with any of the various nested virt problems

obsidian rover
#

Mmmh I did not think of that ahah 🀣 Does it work with the sdks ? I'll check it out, thank you πŸ™

civic yacht
obsidian rover
obsidian rover
civic yacht
# obsidian rover Last question: Would your proposed solution (Windows + PARC) be sufficient to te...

β€œdagger/dagger” would require a little more work because you need to build the engine off the commit being tested, then connect to that via PARC. However I think we have some background plans to do that anyways (cc @astral zealot).

After that, yeah I think this would be a great way to run windows tests in daggers CI. Nested virt is an absolute shitshow so avoiding it would be great. Honestly we would probably want to convert our existing macos tests over to that approach too because the majority of time in those runs is currently just installing docker πŸ˜‚

#

So I would go with the PARC approach if it works for container-use

still garnet
#

@civic yacht (can wait til tmrw) - do you remember why we had dagger.CurrentModule() not expose the 'real' *Module that it has inside, and instead only have a super limited interface?

#

also, what happens if I pass dag.CurrentModule() into another module's function? will it be theirs or the caller's? πŸ˜›

#

ahh i guess it's because it's what provides Workdir() and that can't exist on *Module since it has to be "current" (i.e. coming from a function call that has one)

civic yacht
still garnet
#

ah yeah that sounds familiar

#

so, there's no reason we couldn't expose the underlying Module i guess?

#

i'm exploring having a LLM.withModule(Module) so it'd be nice to be able to pass your own module in

#

i can actually get away with using CurrentModule for now but it's kinda funny; it works because I have one module calling a generic agent module; so it passes itself to the other module, which then which adds it to the LLM, but also adds its CurrentModule to the LLM

humble hearth
#

Running Dagger GraphQL API errors

civic yacht
tepid nova
#

If anyone has ideas on this error... Last one holding back the new address() API. https://github.com/dagger/dagger/pull/10770#issuecomment-3106859201

--> test span: https://dagger.cloud/dagger/traces/9b496c12795ef4a512ba1d4f73d4e823?span=cade61194ee7bac3

Dagger Cloud

Browse and visualize Dagger traces.

GitHub
address(): a unified address to load containers, directories, secre...

This is an alternative to #10753
Extract CLI logic for loading different object types from a standard address; and expose it in the engine API. This is a requirement for #10697, which is part of #1...

tepid nova
#

Investigating abnormally slow filesync... possible regression? 🧡

rancid turret
#

Self calls support in SDKs

still garnet
#

address() ci failure

leaden glade
#

Related to self calls, I have one last issue I don't understand. Who would be the best (and available) for a quick chat so I can share what I have and maybe we can find together the issue. @rancid turret or @fair ermine maybe?

rancid turret
tepid nova
#

In theory - how hard would it be to implement an ID-to-shell converter? πŸ™‚

#

build any object, boom export a shell snippet that produces it

still garnet
fair ermine
#

@civic yacht @rancid turret @still garnet
Since Justin is off, I'm pinging you to review that PR: https://github.com/dagger/dagger/pull/10736

It applies .gitignore on contextual argument but also add a ApplyGitIgnore on Host.Directory so it can be used for other purpose (cf: https://github.com/dagger/dagger/pull/10736#discussion_r2213034069).

Happy to get review and merge at asap since it may improve DX and performances for all our users and solve: https://github.com/dagger/dagger/issues/6627

rancid turret
fair ermine
rocky plume
#

I got an odd flake just now: panic: recovered from errgroup.Group: value method chainguard.dev/apko/pkg/apk/auth.multiAuthenticator.AddAuth called using nil *multiAuthenticator pointer on the client side. I hit retry and the failure went away πŸ₯²

Has anyone else seen this panic before? Here's a full gist of the failure: https://gist.github.com/alexcb/18e6be1a42c9e0736668d3eff2d69f5d

civic yacht
#

I got an odd flake just now: `panic:

tidal spire
leaden glade
astral zealot
#

β€œdagger/dagger” would require a little

tepid nova
#

Trivia / call for help... Given a core.TypeDef, how do I check whether it's an object or not? I keep forgetting...

EDIT: Typedef.Kind

still garnet
tepid nova
#

What's wrong with this ID? πŸ€”

dagger -M -c '.core | load-secret-from-id ChV4eGgzOjY1YzNkNTFiYWE2ZjUxNTASWQoVeHhoMzo2NWMzZDUxYmFhNmY1MTUwEkAKFXh4aDM6OWU0MWRkYTc3YzUyZGQ5MhIMCghTZWNyZXRJRBgBGgJpZEoVeHhoMzo2NWMzZDUxYmFhNmY1MTUwElsKFXh4aDM6OWU0MWRkYTc3YzUyZGQ5MhJCChV4eGgzOmM2YzkwZGFlZWE5YzczODASCgoGU2VjcmV0GAEaBnNlY3JldEoVeHhoMzo5ZTQxZGRhNzdjNTJkZDkyEn8KFXh4aDM6YzZjOTBkYWVlYTljNzM4MBJmEgsKB0FkZHJlc3MYARoHYWRkcmVzcyI3CgV2YWx1ZRIuOixvcDovL0Rldi9uMnRyc2w1bndiNTJwaHljY3d3ajZjYWY0cS9wYXNzd29yZEoVeHhoMzpjNmM5MGRhZWVhOWM3Mzgw'

 error: parse selections: parse field "loadSecretFromID": init arg "id" value as dagql.ID[*github.com/dagger/dagger/core.Secret] (SecretID!) using
  dagql.ID[*github.com/dagger/dagger/core.Secret]: expected "Secret" ID, got SecretID! ID
#

#10697: default secrets

still garnet
tepid nova
still garnet
tepid nova
tepid nova
#

@fair ermine if you're around πŸ‘†

rancid turret
#

@tepid nova, quick bikeshed if you can. @fair ermine is adding an option to Host.directory for adding .gitignore exclude patterns. How do you feel about IgnoreVCS? For context: https://github.com/dagger/dagger/pull/10736#discussion_r2228208813.

Could also make it the default and disable with NoIgnoreVCS (not sure if anyone's depending on having locally ignored files but it's the best for most cases).

tepid nova
#

ah ripgrep

rancid turret
#

Yep

tepid nova
#

OK. I made a suggestion πŸ™‚

humble hearth
#

I'm using the query to get a module from a directory, is there a way to get the service ID from the return type if its a service or does the ID only get assigned when its called/running?

query ($id: DirectoryID!) {
              loadDirectoryFromID(id: $id) {
                name
                asModule {
                  id
                  name
                  objects {
                    asObject {
                      name
                      functions {
                        id
                        name
                        description
                        returnType {
                          kind
                          optional
                          asObject {
                            name
                          }
                        }
                        args {
                          name
                          description
                          typeDef {
                            asObject {
                              name
                            }
                            kind
                            optional
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
humble hearth
#

Also, if you miss a good desktop app/explorer for a GraphQL client, I stumbled across this today and super happy so far! https://altairgraphql.dev/

Altair GraphQL Client

Altair is a feature-rich GraphQL Client IDE for all platforms. Enables you interact with any GraphQL server you are authorized to access from any platform you are on.

stuck bloom
#

Hey folks, I am trying to test a simple integration. I am able to run dagger call dev terminal and run it. However, when I do a dagger call cli binary -o ./bin/dagger and try to run it, I get a zsh: exec format error: ./bin/dagger. I tried on main too. Same result.

wild zephyr
#

You should be able to check that by calling file ./bin/dagger

stuck bloom
#

That was my suspicion too, but I checked. I have a M4 mac ./bin/dagger: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=4e7295cf30dec172a17989279b2099972e6ce84a, stripped

tepid nova
#

@stuck bloom from my memory, what you want is dagger call binary --platform=current -o ./bin/dagger

#

When an argument of type Platform is optional and not set explicitly, it defaults to the runner's native platform (not the client's).

tepid nova
spark cedar
#

created a new area/cache label in github to track caching issues + prs, i keep getting caching issues lost 😒 (fyi @civic yacht @rocky plume, should be useful for theseus πŸŽ‰)

rancid turret
#

Enum issue inside nested connection

stuck bloom
#

default --platform=current

spark cedar
charred lotus
#

What's the equivalent of COPY --link --from=stage / / in dagger ? How can I merge an entire container image without invalidating cache in subsequent layers ?

spark cedar
fair ermine
spark cedar
#

the trace you click on isn't the one you started?

#

is that intentional?

fair ermine
#

Yeah, I was trying to repro the bug, but only the 1st attempt triggered it, the 2nd & 3rd ended properly so I showed the one failing after

#

I cannot repro that consistently

#

I'll try from 10736, I think it happened more time with it

#

Yeah no I cannot reproduce that consistently, it's so weird

#

Ok I caught it there: with dagger v0.18.14: trace: https://dagger.cloud/Quartz/traces/57b929a4d8811682c82534b10789c063#f4f0d42dba4159ba
I accidently ran a 2nd time daggger functions, I wanted to run dagger version (my bad)

Looks like it's an issue on cloud, not on the engine, it seems I can repro it when I change my engine version from my dev one to main or from main to dev (which is why it happened a lot during my benchmarks since I was consistently switching engine version)

wild zephyr
#

I think it's related to the clickhouse aggregations. It might take a few seconds after the trace is done to compute the final state of the whole trace which is what's reflected in cloud

#

Since the data is partitioned by OrgID, if your Org doesn't have that much traffic, it's likely that you see this more frequently

fair ermine
wild zephyr
spark cedar
#

i don't think an impl will be too hard, just want to start the convo going, since it feels like a fiddly api (and we need to make it as easy as possible to avoid the chance for misuse)

fair ermine
spark cedar
leaden glade
#

What would be the best solution to add an experimental flag, to gate a feature (self calls) the time we improve performances? Is it best to use a simple env var or to control it through the engine.json?

spark cedar
final star
#

@still garnet is this error as opaque to you as it is to me?

engine logs:

container-use error upstream:

Error: failed to write file: failed applying file edit,
skipping git propagation: failed to apply patch.
still garnet
#

got a trace link or anything?

#

there should be more info written to the WithPatch span logs

final star
#

is there a trick for getting cu-generated dagger traces? i have not tried to do that before

still garnet
#

not really - i just look at recent local traces in cloud

final star
#

cool ill try to get one

still garnet
#

oh hey it's in the logs

still garnet
#

weird though cause we generate the patch based on a successful find/replace

final star
#

lol yeah and i've been stochastic-monkey-gun debugging it with claude for a minute now and it can not repro

#

there is one case where this happens if the agent supplies search_text == replace_text

#

but there's some other case where search_text != replace_text that doesn't repro under units

still garnet
final star
#

the no-op patch ends up being ""

still garnet
#

fair enough

#

i also have a native File.withReplaced API sitting on my coding agent branch, which would simplify things here, but needs bikeshedding

#

(among 1000 other things...)

final star
#

it kinda feels like a string escaping problem to me, but i'd also think that that'd get caught by stochastic monkey debugging

#

like if it finds the search text, but the search text is truncated due to mishandled escapes, and the replace text is also truncated to the exact same spot, then bam you get a diff that's "" even though your search had hits?

#

on main of container use rn claude "there's some inconsistent prompt engineering in our env configuration for developing this repo with an agent, specifically current max version we support is (i guess the current one might be the newest, idk) and remove the roots thing cuz we don't actually support roots. the prompt engineering should also instruct the agent that container-use can't be tested in its own /workdir, a new git repository must be first initialized somewhere else on disk, like /tmp/test-repo. make sure you manually test an instructions that you write into those files." repros the error super consistently with tool calls that look like this:

⏺ container-use - environment_file_edit (MCP)(environment_source:                                                                 "/Users/braa/src/container-use",
                                             environment_id: "regular-shiner",
                                             target_file: "CLAUDE.md",
                                             search_text: "  - Initialize: '{\"j
                                             sonrpc\":\"2.0\",\"method\":\"initi                                                  alize\",\"params\":{\"protocolVersi
                                             on\":\"2024-11-05\",\"capabilities\
                                             ":{\"roots\":{\"listChanged\":true}
                                             ,\"sampling\":{}}},\"id\":1}'",
                                             replace_text: "  - Initialize: '{\"
                                             jsonrpc\":\"2.0\",\"method\":\"init
                                             ialize\",\"params\":{\"protocolVers                                                  ion\":\"2024-11-05\",\"capabilities
                                             \":{\"sampling\":{}}},\"id\":1}'",
                                             explanation: "Removing roots
                                             capability from initialize request
                                             example")
spark cedar
obsidian rover
rocky plume
final star
#

do mounted cache volumes have any "weird" interactions with conventionally LLB-built containers? i'm trying to work out if it's feasible to run a (possibly read-only) LSP pointed at a cachevolume where another pipeline is modifying files using WithExecs and WithFiles. Like there's no problem doing like container.WithMountedCache().WithFile(/cache/x).Container().WithExec().Container() etc etc, right? it's not obvious to me in this case if the fs layering "just works" as normal

spark cedar
spark cedar
#

also, rfc on live development: https://github.com/dagger/dagger/issues/10895
i've got some very early feedback from some of the team already, so i'm convinced it's not entirely the wrong direction, but would like a lot more feedback before kicking the implementation off

#

it's a big enough chunk of work that once we've got consensus, i'll probably make a project proposal (for our internal project tracking)

tepid nova
#

@spark cedar are these weird python errors in my CI build related to the gql dependency issue?

Screenshot_2025-08-18_at_15.12.45.png
spark cedar
#

yup πŸ™‚ just need an approve, but lots of folks are off today

tepid nova
#

@spark cedar I'm also seeing lots of failed "scan" checks in CI, is that a known issue in main? Anything I can help with?

spark cedar
tepid nova
#

Fun fact: you can't call our codegen pipelines in the middle of a rebase, because our version module requires calling git checkout, which fails in the middle of a rebase

final star
#

so me and [other container-use users](#container-use message) have been hitting some weird hanging behavior, not sure the exact cause, but when it happens i see the engine failing grpc healthchecks, api calls that take forever, etc, etc... i found 1 interesting engine log in addition to the healthcheck failures:

time="2025-08-18T20:47:19Z" level=error msg="ref evaluate error: evaluating released result\ngithub.com/moby/buildkit/solver/llbsolver.(*resultProxy).Result.func2\n\t/go/pkg/mod/github.com/dagger/buildkit@v0.0.0-20250708131355-3c56a47e3f5c/solver/llbsolver/bridge.go:327\ngithub.com/moby/buildkit/util/flightcontrol.(*call[...]).run\n\t/go/pkg/mod/github.com/dagger/buildkit@v0.0.0-20250708131355-3c56a47e3f5c/util/flightcontrol/flightcontrol.go:122\nsync.(*Once).doSlow\n\t/usr/lib/go/src/sync/once.go:78\nsync.(*Once).Do\n\t/usr/lib/go/src/sync/once.go:69\nruntime.goexit\n\t/usr/lib/go/src/runtime/asm_arm64.s:1223" caller stack="goroutine 3492744 [running]:\nruntime/debug.Stack()\n\t/usr/lib/go/src/runtime/debug/stack.go:26 +0x64\ngithub.com/dagger/dagger/engine/buildkit.(*ref).Result(0x401dbeb038, {0x33b1918?, 0x40bcaf46f0?})\n\t/app/engine/buildkit/ref.go:237 +0x94\ngithub.com/dagger/dagger/engine/buildkit.(*ref).getMountable(0x401dbeb038, {0x33b1918, 0x40bcaf46f0})\n\t/app/engine/buildkit/ref.go:211 +0x34\ngithub.com/dagger/dagger/engine/buildkit.(*ref).StatFile(0x401dbeb038, {0x33b1950?, 0x40a62152c0?}, {{0x40aebea210?, 0x0?}})\n\t/app/engine/buildkit/ref.go:166 +0x54\ngithub.com/dagger/dagger/core.(*File).Stat(0x403eaa19a0, {0x33b1950, 0x40a62152c0})\n\t/app/core/file.go:283 +0x1e0\ngithub.com/dagger/dagger/core.(*Container).File(0x33a4b50?, {0x33b1950, 0x40a62152c0}, {0x4027b0d6e4, 0x9})\n\t/app/core/container.go:1021 +0x60\ngithub.com/dagger/dagger/core/schema.(*containerSchema).file(0x0?, {0x33b1950, 0x40a62152c0}, 0x40685738c8, {{0x4027b0d6e4?, 0xdf4d00?}, 0x0?})\n\t/app/core/schema/container.go:1651 +0x78\ngithub.com/dagger/dagger/dagql.FuncWithCacheKey[...].func1({{0x40a3626380?, 0x40685738c8?, 0x0}, {0x0, 0x1, 0x40b9711830, 0x40b9686f78, 0x40b96a1040}}, {{0x4027b0d6e4, 0x0}, ...})\n\t/app/dagql/objects.go:802 +0x6c\ngithub.com/dagger/dagger/dagql.NodeFuncWithCacheKey[...].func1({{0x40a3626380, 0x40685738c8, 0x0}, {0x0, 0x1, 0x40b9711830, 0x40b9686f78, 0x40b96a1040}}, 0x401bcd7980, {0x0, ...})\n\t/app/dagql/objects.go:854 +0x118\ngithub.com/dagger/dagger/dagql.Class[...].Call(0x341bc40, {0x33b1950, 0x40a62152c0}, 0x402cc3df01?, {{0x40a3626380, 0x40685738c8, 0x0}, {0x0, 0x1, 0x40b9711830, ...}}, ...)\n\t/app/dagql/objects.go:302 +0xf0\ngithub.com/dagger/dagger/dagql.ObjectResult[...].call.func2()\n\t/app/dagql/objects.go:670 +0xa0\ngithub.com/dagger/dagger/engine/cache.(*cache[...]).GetOrInitializeWithCallbacks.func1()\n\t/app/engine/cache/cache.go:210 +0x64\ncreated by github.com/dagger/dagger/engine/cache.(*cache[...]).GetOrInitializeWithCallbacks in goroutine 3492743\n\t/app/engine/cache/cache.go:208 +0x604\n" client_hostname=Connors-MacBook-Pro.local client_id=fxf5i3l6vv4r3jbhyr3og62jo session_id=hzr3n8a75dr8nm4vsxr1ozywv spanID=32b338ec5845db45 traceID=85317dc1a7c300e5bf79f2ed06cdb40b

that occurred after I shut down all my dagger clients. prior to doing that, all dagger api calls were just stuck, including non-container-use calls. one time last week i also tried running dagger call engine local-cache prune during one of these hang states and it took over 25m before i sigkilled it

cc @still garnet cuz you're seeing similar things on your branches (im on the released version) and @civic yacht cuz well caching and refs and bk seem to be in play here.

tepid nova
#

@leaden glade @spark cedar I have a question related to https://github.com/dagger/dagger/pull/10847#issuecomment-3179906799

I'm trying to remember why we called it +defaultPath and not +default, and see if the proposed address() PR gives us a way to avoid fragmentation

GitHub
feat: add contextual git by jedevc Β· Pull Request #10847 Β· dagger...

Fixes #8520. Follow-up to #10195.
This allows a module to get git information about itself - it can do this through the core Git type.
The new defaultGit annotation (similar to defaultPath) sets a ...

tepid nova
#

New discussion topic for your consideration @leaden glade @spark cedar (and anyone else who is around): generic ID... 🧡

spark cedar
tepid nova
spark cedar
#

πŸ€”

#

never seen that one before

#

seems particularly likely that's a flake though

tepid nova
#

I'll just have GHA re-run (hate how hidden that button is..)

spark cedar
#

(side note, we need better support for services that listen on unix sockets)

#

we can't health check them, i think that's likely the source of this problem

tepid nova
#

ah

#

withExposedUnixSocket? πŸ™‚

#

Feels weird to attach that stuff to Container, but it would be consistent with Container.withExposedPort

spark cedar
spark cedar
#

😒 fyi, tests on main are failing because of a bitbucket outage

image.png
#

(just the tests where we're testing integration with that)

final star
#

potentially good news regarding all the weird engine connection blocking behavior i've been seeing lately on container-use, ive been stress testing cu + dagger 18.16 for >30m and so far i've been completely unable to repro.

there was also a weird bug coming out of environment_file_edit that i've only repro'd once so far, so ~1.8 birds with one stone if we're lucky 🀞 party_blob

no idea which engine commits might've fixed all this, but y'all have been cookin' πŸ™‚

#

lol i jinxed it

#

now i've got the engine blocking periodically for longer and longer... like >4m calls to cache-hitful dagger functions

#

virtually no interesting engine logs, though

#

2 separate engine blocks in the course of a single dagger functions call that normally takes 8s

Screenshot_2025-08-20_at_1.32.23_PM.png
final star
#

and after getting it consistently reproing, exiting all the container-use servers, dagger functions failed like this:

failed to get configured module: Post "http://dagger/query":
  command [docker exec -i dagger-engine-v0.18.16 buildctl dial-
  stdio] has exited with exit status 137, make sure the URL is
  valid, and Docker 18.09 or later is installed on the remote host:
  stderr=
#

gonna repro again on a dev-built one with debug servers enabled

leaden glade
#

πŸŽ‰ Self Calls/Invocation PR is finally 🟒

The PR includes an experimental flag to enable/disable it (dagger init --with-self-calls, dagger develop --with-self-calls), a few integration tests to ensure the flag and self calls are working (in Go).
Go, Python and Typescript SDKs are compatible with self calls, others will come a bit later, probably starting with Java as a big part has already been done)

This is for sure not a small PR, it contains changes in a lot of different places. And even if I tried to keep the commits as clean as possible, it's still a lot of commits rewritten during some time so they are not as nice as I'd wanted πŸ˜•
It even includes a fix on the .Net SDK, as it looks like I added for the first time a List of enum values, something that wasn't implemented in .Net (I haven't wrote C# for almost 20 years 🀞 )

I'd love to have some πŸ‘€ on it, and happy to help to review it if needed, explain how it works, etc.
https://github.com/dagger/dagger/pull/10584

GitHub
SDK: self invocation by eunomie Β· Pull Request #10584 Β· dagger/da...

NoteThis has been extracted and cleaned from #10475
This is a first step to #10336 and #8030

This PR implements the ability for modules to self invoke themselves.
func (m *MyModule) ContainerEcho(...

leaden glade
#

I don't know if that's a good idea or not πŸ˜… but I played with a CombinedOutput flag on WithExec. I sometimes would like to get both at once, to print the full output for instance, and to retrieve stdout then stderr is not nice as I can just put one after the other.
https://github.com/dagger/dagger/pull/10924
My change is very simple, and it surprises me. So maybe I'm missing something...

GitHub
feat: add ability to combine outputs in WithExec by eunomie Β· Pull...

WithExec([]string{...}, dagger.ContainerWithExecOpts{
CombinedOutput: true,
})
Standard error will be redirected to the standard output and both can be retrieved at once using stdout
Order is n...

tepid nova
tepid nova
leaden glade
spark cedar
#

we already do some funky pipe manipulation

#

or

#

I could just read the pr where vito already suggested that

#

πŸ€¦β€β™€οΈ

final star
still garnet
final star
final star
#

tbh the phat stack of isDep is probably the culprit... realizing i took dumps over a 10 minute window and the latter 3 have the same goroutine working on the same phat isDep stack, always runnable

#

and it's not like this blocks permanently, just for an uncomfortably long time.

#

the rest of the lock acquisition and chan recv blocking could all feasibly be waiting for the result of the isDep stack, which is downstream of the central dispatch loop

#

glad once i read the code and then searched for it in the discord i immediately see that you pushed trying to fix it 15m ago XD

spiral fog
#

Hey, new directory exist is great addition but why didn't we add this funct to File type?

#

is there any specific reason?

spark cedar
#

if you've constructed a File, it exists

#

if you want to check it exists, just call Sync

#

it exists if that call succeeds

spiral fog
#

yea , i know and use sync to test this, just as DevEx I feel weird having Container and Directory have exist but File don't.

tepid nova
#

I just randomly found an unused dependency in one of our modules... Is there a way we could automate detecting this? It could help us improve loading times

wild zephyr
tepid nova
wild zephyr
wild zephyr
tepid nova
#

(or at least I couldn't fine anywhere it uses it)

wild zephyr
wild zephyr
wild zephyr
tepid nova
#

yeah I don't think telemetry is enough to reliably know if a dependency is used or not.

wild zephyr
tepid nova
#

but just because a dependency has never been called doesn't mean there isn't a code path that can call it. You need static code analysis to know for sure.

We could use telemetry to give hints, though. "we've noticed a dependency hasn't been called in a while, are you sure it's used in the code?"

tepid nova
#

@leaden glade let's dogfood self-calls in our own dagger module πŸ™‚ I'm going crazy looking at traces with 1000 low-level calls in a row...

leaden glade
tepid nova
#

Thanks! I think it will be the perfect showcase πŸ™‚