#multi-object 🧵

1 messages · Page 1 of 1 (latest)

acoustic radish
#

@shrewd skiff getting closer... but I'm getting this weird "empty spans" effect:

#

not sure what those robot heads are doing

shrewd skiff
#

most likely the 'thinking/replying' span corresponding to API requests

#

aka the one that has token usage attached on non-OpenAI

acoustic radish
#

Yes but I've never seen more than one in a row with no tool call - normally it stops in that situation

rich marsh
#

I feel like that started happening a lot for me with ollama in one of the recent updates too. Haven't been able to dig in to see why yet. I was wondering if there was a regression in our openai implementation

acoustic radish
#

Also I hit this at the end:

│
│🤖 0.2s
│ ! POST "https://api.openai.com/v1/chat/completions": 400 Bad Request {
│ !   "error": {
│ !     "message": "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The fol
│ !     "type": "invalid_request_error",
│ !     "param": "messages",
│ !     "code": null
│ !   }
│ ! }
rich marsh
acoustic radish
#

@shrewd skiff looking at the web trace, there are in fact function calls. They just don't show in the TUI

--> my bad actually this is a problem across TUI and Web traces

shrewd skiff
#

ah interesting, will look into it

acoustic radish
#

weird that part of the TUI trace has the same info

Screenshot_2025-03-06_at_17.48.39.png
#

but then later in the same trace, it starts to drop information

#

Sorry that was wrong - everything seems to be the same in TUI and web

#

so it's NOT a TUI-specific problem after all

shrewd skiff
#

oh OK - you were just looking at a different region?

acoustic radish
#

yeah got confused

#

so back to not knowing why several "thinking" spans in a row with no tool call

rich marsh
acoustic radish
#

hey how come I don't get the token count?

rich marsh
#

😎

acoustic radish
#

is it supposed to work with openai?

shrewd skiff
#

it is, but openai is returning 0 for the token usage

#

no idea why. maybe they don't support it for streaming, but total guess

acoustic radish
#

@shrewd skiff if you want to try, it works - now have to fine tune the prompt engineering

#

(so that the llm intuitively figures out what to do)

#

might need a _help

#

I'm curious how this would plug into dagger llm

acoustic radish
#

@shrewd skiff @rich marsh nice side effect, you don't need a separate "prompt var" system. All vars in the llm env can be used to templatize the prompt

rich marsh
#

nice. Does set-string have abilities beyond what with-prompt-var had?

shrewd skiff
#

gonna go hide all those Secret.plaintext errors

acoustic radish
#

i renamed "with" to "set" to try the DX

#

you can actually also expand objects 🙂

shrewd skiff
#

@acoustic radish lol, this is an interesting consequence of marrying prompt vars + llm env vars + shell vars

image.png
#

which ended up not working, because of course the model is looking for 'repo' and 'ctr'

acoustic radish
#

mmm oops 😁

shrewd skiff
#

but...what if we built on that, and just made them addressable by digest?!?!?!

#

eh you lose readability of the prompt itself

#

but, cool that we could

acoustic radish
#

was that in my multi obj branch?

shrewd skiff
#

yeah, i'm integrating it with the shell vars now

#

mostly works, except for that

#

tl;dr after every eval we diff the vars from the last vars and set any new/different ones in the llm env, too

#

so no /set foo bar or /with, you just foo=$(container | from alpine) and the llm sees it too

acoustic radish
acoustic radish
#

I started using it in my demos today, noticed you already allow setting vars in /shell then using them in /with

#

@shrewd skiff were you able to get the LLM to use the objects properly?

#

I didn't have time to get back to that

#

on a plane now, I have 45mn to try 🙂

shrewd skiff
#

getting there - I think it just needs a bit of prompt engineering. i'm having it generate a prompt 😛

#

i can commit + push what i have

acoustic radish
#

oh yes please! I'll start from what you have

#

Any direction you'd want to me try first?

#

(or avoid because you're already on it?)

shrewd skiff
#

@acoustic radish pushed to my fork - vito/llm-multiobj (don't have perms on your branch)

shrewd skiff
acoustic radish
#

oh oops. will fix perms

#

actually I thought upstream maintainers could push to forks?

#

do you lack perms upstream?

shrewd skiff
#

shrug usually this isn't an issue, not sure

#

oh wait

shrewd skiff
#

oops. well there's one there now

To github.com:dagger/dagger
 * [new branch]          llm-multiobj -> llm-multiobj
acoustic radish
#

no it's on my fork, but I thought if you write permissions on the upstream, that gave you write permissions to the forks also

shrewd skiff
#

can delete, not sure why i can't push to shykes/llm-multiobj

acoustic radish
#

maybe only if there's a PR

#

@shrewd skiff just opened the pr. Try again?

shrewd skiff
#

that worked 👍

acoustic radish
#

nice ok

#

pulling now

#

@shrewd skiff I think getting multiobj to work might be a matter of choosing the "metaphor" so that it feels natural to the LLM with minimal explanation - just name & description of the tools

#

for example I realized - _undo for rolling back the selection, makes it unintuitive to do sub-pipelines eg. container | with-directory /src $(git ...)

#

actually I guess _scratch becomes super important

#

maybe _start

#

and bring back the concept of pipeline or chain in the descriptions

#

@shrewd skiff getting a build error on the CLI: cmd/dagger/llm.go:147:12: pattern llm.md: no matching files found

shrewd skiff
#

huh thought i fixed that, 1 sec

shrewd skiff
acoustic radish
#

np I'm building from git remote, faster from the plane 🙂

#

(remote engine on home server)

#

time's up I'm landing

#

will try again later tonight

#

thanks for pushing this!

shrewd skiff
#

np

#

hmmm so far results are not that great, but i might know why

#

if i try to do something like build $repo using $ctr, whenever it swaps from one to the other it forgets how to work with the other one

#

maybe the tools aren't returning the full set, and only the current value's? i remember you saying before that sometimes it extends instead of replacing

acoustic radish
#

extends instead of replacing

didn't understand that sorry

shrewd skiff
#

you mentioned something about the set of tools including both Container and Directory if it chained from a Container to a Directory

acoustic radish
#

but in multiobj it's only the current object's functions + _builtins

#

so it needs to understand how to juggle objects

#

I think we don't give the llm enough to connect the dots

shrewd skiff
#

hmm yeah that seems like the hard part

acoustic radish
#

yes. but easier than eg. teaching it dagger shell or gql imo

shrewd skiff
#

i'm not sure about that - those are languages, which llms are good at

acoustic radish
#

i think it's doable with good results but haven't had a chance to try

shrewd skiff
#

this seems like it would rely more on reasoning capabilities

#

just a hunch

acoustic radish
#

I'll try a few things

#

the syntax part was not the issue with shell at least.

#

it was understanding the object model, chaining etc

#

so very similar but with more ground to cover because there was no 1-1 mapping of tools ever

acoustic radish
#

FYI still getting the build error

#

dagger -m github.com/shykes/dagger@llm-multiobj -c 'engine | service llm | up'

# 
✘ .withExec(args: ["go", "list", "-f", "{{if eq .Name \"main\"}}{{.Dir}}{{end}}", "./cmd/dagger"]): Container! 0.4s
cmd/dagger/llm.go:155:12: pattern llm.md: no matching files found
#

@shrewd skiff in case the current multiobj strategy doesn't pan out, what's plan B? gql? shell? other?

shrewd skiff
acoustic radish
#

trying from a local checkout in case it's a cache issue

#

god that version string compute andf its full git fetch... can't wait to optimize that away

shrewd skiff
#

@acoustic radish pushed

acoustic radish
shrewd skiff
#

yeah 😭 can't wait for function caching

shrewd skiff
#

even with GraphQL I ended up doing a stateful vars thing

#

though, maybe it could do one big tool call that both sets vars + uses them in later queries. (kinda like a SQL CTE)

shrewd skiff
shrewd skiff
#

the idea here is: no more _load, instead we load the tools for all types currently set to vars, and you use the Type@hash value to refer to them everywhere

#

sweet, this works without any prompting

#

the main downside is loading all the tools at once, not sure if that's a dealbreaker. hmm maybe. it caches well, at least

acoustic radish
#

But there is a hard limit on number of tools

#

And even before the limit, it might hurt performance

#

but, we could optimize this later maybe

#

you use the Type@hash value to refer to them everywhere

Don't understand this part

shrewd skiff
#

lemme see if i can work backward from where I ended up and keep one-toolset-at-a-time, there were other changes along the way

acoustic radish
#

Feels like we're reinventing swap memory 😛

shrewd skiff
#

lol

shrewd skiff
#

the llm seems to pick up how it works pretty well without any prompting

acoustic radish
shrewd skiff
#

yeah, assuming the model is able to keep track of the hash values. not sure if that's reliable enough

acoustic radish
# 
-       // +ignore=["*", ".*", "!/cmd/dagger/*", "!**/go.sum", "!**/go.mod", "!**/*.go", "!**.graphql"]
+       // +ignore=["*", ".*", "!cmd/dagger/*", "!**/go.sum", "!**/go.mod", "!**/*.go", "!**.graphql"]
        source *dagger.Directory,

Oh noooo - sorry you got bit by that

shrewd skiff
#

(also claude seems generally slower than gpt-4o)

#

pushed to vito/llm-multiobj-all-tools for now

#

this ^ showed a concerning issue though where it treated the container as if it were mutable, instead of chaining

acoustic radish
#

@shrewd skiff in latest multi-object branch, looks like you can't set variables to a module

$ dagger llm
* /shell
⋈ workspace=$(github.com/shykes/toy-programmer/toy-workspace) 3.2s

✘ workspace=$(github.com/shykes/toy-programmer/toy-workspace) 3.2s
! input: llm.withPrompt.setToyWorkspace load: Call: Query has no such field: "toyWorkspace"
│ ✔ load module 0.5s
│
│ ○ toyWorkspace: ToyWorkspace! 1.1s
│
│ ✘ Llm.setToyWorkspace(
│ │ │ name: "workspace"
│ │ │ value: ○ toyWorkspace: ToyWorkspace! 1.1s
│ │ ): Llm! 0.0s
│ ! load: Call: Query has no such field: "toyWorkspace"
shrewd skiff
#

ah good catch, will look into it tomorrow!

shrewd skiff
shrewd skiff
#

I seem to be getting decent results with gpt-4o after going back to the model of selecting one toolset at a time 👍

#

renaming _load to _selectTools seems to yield even better results, but n=2

#

also thinking of letting _selectTools take a regex of tool names to enable, and having one of the other tools provide a simple list of all the fields (without descriptions). Maybe not necessary, but seems fun to try

shrewd skiff
# 
│ ┃ 5.0s ◆ LLM Input Tokens: 63,957 ◆ LLM Output Tokens: 132

😱 holy moly. that came from swapping to toyWorkspace?!

🤖 1.5s ◆ LLM Input Tokens: 9,237 ◆ LLM Output Tokens: 31
│ ✔ 🤖💻 _selectTools map[name:ToyWorkspace@xxh3:d0f1e35037b98313] 0.0s
│
│🤖 The  ToyWorkspace@xxh3:d0f1e35037b98313  workspace has several functions that we can explore. Here's a brief overview of its capabilities:
│ ┃
│ ┃ 1. Read: It can read files with a specified path.
│ ┃ 2. Write: It can write to files by providing a path and content.
│ ┃ 3. Build: It can build the code at the current directory in the workspace.
│ ┃
│ ┃ Please let me know if there's anything specific you'd like to do with this workspace, such as reading a specific file, writing data, or building the
│ ┃ code!
│ ┃ 5.9s ◆ LLM Input Tokens: 62,358 ◆ LLM Output Tokens: 121
acoustic radish
shrewd skiff
#

yeah there's only like 3 tools in that one

#

probably something goofy

#

unless you have incredibly long descriptions

#

@acoustic radish ok if I force-push now that it's back to one toolset at a time?

#

(to llm-multiobj)

acoustic radish
# 
// "what is in a word?" I pondered. To read - such a crucial, important word. it all started when..
func read()
acoustic radish
shrewd skiff
#

ok done - modules should work now

#

would recommend rolling with the prompt var <-> shell var <-> llm var alignment for now, it seems to work pretty well, and I dig that it makes the prompt even more explicit about what it's working with (e.g. if a var gets re-assigned)

#

one potential risk is the model losing track of the fact that Directory@xxh3:... is "repo" but seems OK so far. not sure if the model sees the original name for prompt vars or if it's just a text substition

#

here's how I've been testing:

/shell ctr=$(container | from golang)
/shell repo=$(git https://github.com/vito/booklit | head | tree)
use $ctr to build $repo
#

ah I think _load/_selectTools is dumping the entire object into the tool response. will fix

shrewd skiff
#

pushed, + some tweaks to tool descriptions that help out. now when it sees "run unit tests against Bass@xxh3:..." it'll jump right to selectTools[Bass@xxh3:...] without having to look anything up

acoustic radish
#

will try tonight!

#

can it still set variables?

#

it might get confused juggling too many IDs

shrewd skiff
#

i haven't removed it, but there's no llm => shell env syncing yet. haven't seen it mess up IDs yet, but i've only done micro tests. should probably start trying more complicated tasks next

#

also don't think i've ever seen it call _save

#

i'm starting to wonder if we can get away with just one builtin, _load/_selectTools (depending on which name it likes)

#

but, there may be somewhere that a name <-> hash mapping is useful

acoustic radish
#

yeah I saw it use save yesterday

#

i think vars will be useful for "returning" values but also as lightweight memory. ie compact the context, might forget IDs but not vars

shrewd skiff
#

ah yeah makes sense

shrewd skiff
#

@acoustic radish I noticed it tends to do this: Container.withExec(["obviously", "doesnt", "work"]) => "I did it! 😊" (not realizing it's lazy) - I remember there being a sync call somewhere at one point, but can't find it now on either llm or llm-multiobj. Did ya have an idea for that? Maybe we should just always sync against syncable tool call results?

acoustic radish
shrewd skiff
#

grr not sure why my grep didn't find that

#

it was there already, i just fixed it. in the process of adding it for multiobj now

acoustic radish
#

@shrewd skiff depends if we can give it a concept of starting/ending a pipeline maybe. but yeah maybe too ambitious for now

shrewd skiff
#

i have setting vars working, just all my demos are blowing up cause of that 😛

shrewd skiff
acoustic radish
#

ha ha nice 🙂

#

@shrewd skiff do I have to resolve the variable in the shell? Or can the llm figure out what variables are available and use them?

#

Since it knows how to set variables I guess it knows how to get them too

#

I get why you're making sure expanded vars work. But I would prefer not to have to teach every developer how to do that trick as a pre-requisite to basic prompting

shrewd skiff
acoustic radish
#

nice

shrewd skiff
#

i feel like it's more natural to type $foo though

#

less risk of ambiguity in the prompt

#

it just so happens that it expands atm, and the expanded form also works

acoustic radish
#

Yeah in cases where you want to designate a specific var, but there may be cases where you don't want to, or can't (because the vars were dynamically set for example)

shrewd skiff
#

a few learnings in this one: https://asciinema.org/a/H0O0Yk9tL61oG7dJXvYLKXFWD

  • the model didn't intuit that "bass" referred to a variable without a symbol prefix
  • the model understood the expanded $bass (expected)
  • there's no way to say or escape $bass literally (users prob wouldn't want to do this anyway)
  • using @bass was just enough of a hint, but might as well just type $bass at that point
  • it grabbed the entire stdout of the tests which added 30k to the token cost... twice 😬
asciinema.org
untitled

Recorded by vito

#

i'll go ahead and push as-is

acoustic radish
#

@shrewd skiff does this feel like it could be merged to llm this week?

#

(trying to feel out the sequence relative to merge)

shrewd skiff
#

i think so, the most disruptive thing is probably /with going away. should we keep it?

#

pushed (to llm-multiobj)

acoustic radish
#

If the alternative is better I don't have a problem with replacing it

#

Question @shrewd skiff : are slash commands generalizable somehow? How do we explain them vs. dot-builtins?

#
  • Option 1: they're very different because: (reason that makes sense)
  • Option 2: they're the same, move everything to slash commands
  • Option 3: they're the same, move everything to dot builtins
shrewd skiff
#

at least at a technical level: dot builtins exist within the confines of valid shell syntax, slash commands don't; they're at a higher level of control, and / has to be the very first character in the prompt

shrewd skiff
acoustic radish
#

So would slash commands make sense for anything that is applicable in both modes?

#

that might apply to some (not all) dot builtins

shrewd skiff
#

yeah

acoustic radish
#

that might actually free up a lot of builtins

#

/login
/install
/uninstall

?

#

argh although install/uninstall is linked to filesystem navigation ("install in the current directory)

acoustic radish
#

@shrewd skiff testing with latest llm-multiobj, vars dont seem to be accessible

acoustic radish
#

Oh wait, I might have been using the wrong CLI build

shrewd skiff
#

(unrelated:) ah looks like we lost the stdout/stderr propagation, adding that back now

acoustic radish
#

do you still want to add "ambient module access"?

#

if so, I have questions 🙂

#

oh wait. token count works with openai now partyparrot

shrewd skiff
acoustic radish
#

(not sure what to call it 😅 ) the idea that /prompt and /shell have access to the same functions - including 1) core API, 2) functions in the current module, and 3) dependencies of the current module

shrewd skiff
#

ack, it seems like this method of passing in self doesn't work very well with chaining - note how these all were called against the same initial empty toyWorkspace

image.png
acoustic radish
#

ah, also lack of stderr pass-through tainted the result:

🧑 summarize what happened and what confused you
│ ┃ 0.0s
│
│🤖 I attempted to build a Docker container using a specific version of a git repository. Here's a summary of what happened:
│ ┃
│ ┃ 1. Container Setup: I initiated a container with build tools like build-base , git , and later added go for building a Go-based CLI in
│ ┃ the project.
│ ┃ 2. Git Operations: I checked out the last stable release ( v0.16.2 ) of the repository.
│ ┃ 3. Build Attempts: I tried several times to build the main CLI using go build , targeting the ./cmd directory which typically houses
│ ┃ command entries. Each attempt resulted in a failure without substantial error details.
│ ┃ 4. Logging Issues: Attempts to capture build errors in log files ( stdout.log and stderr.log ) resulted in errors as the files weren't
│ ┃ created or found in the filesystem.
│ ┃ 5. Diagnosis: I examined directory contents and the go.mod file for clues but remained unable to identify the root cause of build failures.
│ ┃
│ ┃ Confusion: My main confusion arose from:
│ ┃
│ ┃ • The inability to redirect and capture build error logs, which could have been instrumental in diagnosing the issue.
│ ┃ • The absence of detailed error messages from the failed go build commands, leaving the exact reason for failure unclear.
│ ┃
│ ┃ To proceed effectively, insights on specific build steps, environment setup, or access to documentation would be helpful.
│ ┃ 9.6s ◆ LLM Input Tokens: 22,229 ◆ LLM Output Tokens: 280

#

I'm still not convinced we can make this work without "simulating" mutation

shrewd skiff
shrewd skiff
shrewd skiff
acoustic radish
#

where is does seem to get stuck?

#

time for some evals? 🙂

shrewd skiff
acoustic radish
#

Will give it a try. Do you have a favorite scenario you run against it?

shrewd skiff
#

not really, I've just been winging it, it's actually the 'favorite scenarios' that I'm most interested in seeing haha

but here's one I've been using just to get things started:

go=$(container | from golang)
booklit=$(git https://github.com/vito/booklit | head | tree)
/prompt use $go to build $booklit

It's very vague but it's interesting seeing what it does. The general theory is "here's a container with Go, here's a trivially buildable Go project, do your thing" but even more ideally it would build ./cmd/booklit and give me the binary or something (nothing has done that yet, they just do go build ./... which is perfectly understandable given how vague the prompt is)

#

lots of variables could be tweaked, like having it figure out how to install Go itself

acoustic radish
#

👀

#

@shrewd skiff I find myself entangling discussions of 1) dagger llm and dagger shell, and 2) multi-object. Should I embrace the entanglement and discuss it all in this thread? Or use a separate thread? (I want to involve @vernal ruin and discuss it live with you guys if possible... As the shell launch deadline looms large)

acoustic radish
shrewd skiff
shrewd skiff
acoustic radish
#

I don't mind having /model there, I just want to make sure the overall architecture we have, makes sense to users, and scales nicely when we add more features. Ideally there are core principles that are self-evident and we don't start arguing over them in 6 months 🙂

#

One interesting stress question: if /shell and /prompt are "modes", will there perhaps be more modes in the future? If so, what will be the rule for what belongs in a "mode"? Will they map to the Dagger API, for example 1 mode = 1 type? or something like that?

shrewd skiff
#

so it's not really a mode switch in the persistent sense

#

which honestly is kind of nice, swapping back and forth is annoying

#

but it does raise the question of the 'default mode' (prompt, or shell? we're pretty sure about shell, but for claude it's prompt, so they use ! i guess to indicate 'danger, you're running a command'?)

#

if the default is shell, what do we use?

acoustic radish
#

it's not too late to change the default 🙂

shrewd skiff
#

i'm not proposing that, just saying ! prob won't make sense for a prompt indicator 😛

#

maybe it should match the symbol shown in the prompt, but for us that's a fancy unicode asterisk

#

regular asterisk aint too bad:

go=$(container | from golang)
repo=$(git https://github.com/vito/booklit | head | tree)
* build $repo with $go
shrewd skiff
#

gonna try implementing that and see how it feels

#

i'll do both sides, ! to run shell from prompt, * to run prompt from shell

acoustic radish
#

I think having to type any character first will kill the magic

#

(could be wrong)

shrewd skiff
#

i mean the alternative is typing /shell or /prompt no? and since it's interpreted immediately as a (one-off) mode switch it won't feel like you're literally typing "! foo"`

#

but yea i'll just take a swing at it and we can see how it feels

#

just tryin' things on

acoustic radish
#

Oh you mean ! is like a hotkey in claude code?

shrewd skiff
#

yeah - it doesn't show up in the text input, it swaps the mode immediately for that single input

shrewd skiff
acoustic radish
#

nice

#

mmmm "bash mode"

#

that's what we should call it too

rich marsh
acoustic radish
#

I'm still not sure about slash-commands vs. dot-builtins

#

but I do like the claude hotkey for mode switch, now that I understand it

shrewd skiff
acoustic radish
#

But "dagger shell" will be the name of the whole thing (including prompt mode & all other modes). Basically it's the name of our repl.

#

It can either be that, or the name of a mode. Confusing if it's both

shrewd skiff
#

yeah but if you say "bash" people are gonna expect bash, which our scripting language isn't at all

acoustic radish
#

true

shrewd skiff
#

also have to keep in mind dagger shell scripts which sort of exist outside of the idea of a repl

shrewd skiff
acoustic radish
shrewd skiff
#

dagger [shell] --prompt/-p <- in interactive mode, defaults to prompt, or when used in a shebang, interprets whole file as a prompt?

acoustic radish
#

what what if I want to go back and forth in a script?

#

Maybe nobody will ever need that - but seems potentially risky to make that bet

#

but then, adding slash-commands to a shell script feels weird too

#

or maybe it's fine?

shrewd skiff
#

TODO: swap the prompt back after you hit enter

acoustic radish
#

oh nice! yeah I noticed that "history mixup"

#

by the way another bug log for later: if you start editing a new command while a previous command is running, your line gets wiped when the command completes (or fails?)

shrewd skiff
#

ha yea, fixed that too

#

so annoying

acoustic radish
#

btw demo today went great

#

(single object)

shrewd skiff
#

nice!

acoustic radish
#

had agent write curl clone in Java. Then change it to use gradle instead of maven.

#

Got a request: "have you considered using a LLM to explain why my trace fails in Dagger Cloud? Could you do it better than Github Actions in their 'explain error' feature, because you have better data? Maybe even suggest actions to take?"

shrewd skiff
#

i have always looked at that button in github and never clicked it lol. have you?

#

makes sense, but sounds tricky (what AI do we use? who pays for it?)

acoustic radish
#

Yeah those questions will have answers eventually 😛 But for now just a seed that needs water and time to grow

shrewd skiff
#

and maybe with this new model we don't need /prompt and /shell (MAYBE)

rich marsh
#

(not uploaded to yt yet)

shrewd skiff
#

@acoustic radish pushed if you want to try it out. llm still defaults to prompt mode, so press ! for shell, and if in shell mode press * for prompt
TODO for tomorrow:

  • switch shell to this new implementation
  • get rid of llm command
  • update help text below prompt, current version is next to useless (M-? doesnt even work on Mac), should show !/* key at minimum
  • add -p/--prompt to dagger shell?
acoustic radish
#

will try tomorrow tonight for sure

#

yeah in this model we may not need slash-commands (unless they replace dot-builtins 😛 but that's a stretch)

only one left would be /model?

shrewd skiff
#

There's still /compact /clear etc if we want em

#

I think it's a useful and common enough pattern to keep in our back pocket

acoustic radish
shrewd skiff
#

update help text below prompt, current version is next to useless (M-? doesnt even work on Mac), should show !/* key at minimum
pushed this part + fixed shell completion

acoustic radish
#

@shrewd skiff anything I can to help on the "model UX" part? Ie. getting the LLM to better pick up the multi-object tools?

shrewd skiff
#

i think just a lot more tire-kicking - trying out more scenarios, seeing if it gets confused

#

i want to hack on script support to make that easier, like this:

#!/usr/bin/env -S dagger shell --no-mod

go=$(container | from golang)

repo=$(git https://github.com/vito/booklit)

bin=$(.prompt "Build the ./cmd/booklit binary in $repo using $go.")

$bin | terminal

but trying to resolve the merge situation with main first

acoustic radish
#

@shrewd skiff is shykes/dagger@llm-multiobj still the branch to build from?

shrewd skiff
#

yep

shrewd skiff
acoustic radish
#

OK I did a quick poll ( @devout niche @rough kiln @velvet garden ), we explored the question: "should dagger default to prompt or shell mode? And why?".

We reached consensus on:

  • AI companies want software to be inside their AI. So their UI defaults to prompting.
  • Dagger wants AI to be inside your software. So our UI defaults to shell.

And I really like that framing, because the UI difference actually reflects a profound philosophical difference.

#

cc @grizzled vine 👆 integration of marketing and UX 🙂

shrewd skiff
#

makes sense to me shipit

acoustic radish
#

Tibor mentioned "* is far on the keyboard, maybe >? Also it's a literal shell" 🙂

#

@shrewd skiff re use of the word "shell".

I am warming up to:

  • CLI: the entire command-line experience.
  • shell mode: one way to interact with the CLI
  • prompt mode: another way to interact with the CLI
acoustic radish
shrewd skiff
#

yep

acoustic radish
shrewd skiff
#

@acoustic radish ok try now

shrewd skiff
acoustic radish
#

(sorry was on a call)

#

@shrewd skiff doesn't build for me

acoustic radish
#

@shrewd skiff assuming you're in the zone, going to try and fix that build error myself (seems unrelated to your code, maybe a main merge side effect?)

acoustic radish
#

update: still debugging that random elixir-dev error

#

kind of a PITA

shrewd skiff
#

I'm out to dinner atm

acoustic radish
shrewd skiff
#

ok now that shell is the default and we have one-off mode switching i think i'm team Remove Slash Commands. anything that was a slash-command can just be a builtin. and to run a builtin from prompt mode, just press ! first: instead of /compact it's !.compact.

also for a persistent mode switch maybe it can accept the switcher twice, so >> to stay in prompt mode, !! to stay in shell mode

acoustic radish
#

do you think claude code users might miss the slash commands?

shrewd skiff
#

i think this is closer to what we would have ended up with naturally, at least, since we already have a whole language with builtins and default to shell - claude code doesn't have that, and defaults to prompt

shrewd skiff
#

pushed: llm command gone, assimilated into shell, slash commands replaced with builtins.
currently working on: .prompt in repl swaps to prompt mode ✅, in a script it runs a prompt and returns the value
TODO: >>, !!, more polish

acoustic radish
#

update: until the main build issue from hell is resolved ( #1349559353589497946 ) I a building llm-multiobj with dagger 0.6.3. Building now!

acoustic radish
#

@shrewd skiff UX feedback: when I prompt the LLM, and it replies, it switches back to shell mode, so I often find myself typing my reply in the shell mode, getting an error; then have to fully re-type the same message because going back in history "switches" me to the shell mode

shrewd skiff
acoustic radish
#

Yeah kind of feels like there should be a "regular" keybind (say ctrl-/) that can be triggered at any time (not just when editor is empty) and its only effect is to toggle the input mode (only visual effect is to change the prompt character). Everything else remains the same including my current line buffer.

shrewd skiff
# acoustic radish Yeah kind of feels like there should be a "regular" keybind (say `ctrl-/`) that ...

the >/! prefix also works when the editor has content, you just have to be at position=0 when you type it - maybe try that out? it's the same way Claude Code works and feels intuitive once you know it's a thing (Up-> Home/Ctrl-A -> >)

keybind makes sense too but a) it's especially difficult to find one that works across all platforms in prompt mode, and b) it'll compete with >/! for discoverability - e.g. which do we show in the keymap?

acoustic radish
#

If we do a keybind it would be as a replacement for > > !, I wouldn't do both. Will try the "position 0" trick

acoustic radish
#

@shrewd skiff OK with stream of consciousness feedback as it comes to me while using? Or do you prefer a thoughtful, digested summary at the end of the day (crazy I know)

shrewd skiff
#

lots of stuff is changing so it's good to check in and build muscle memory on things like that sooner

acoustic radish
#

@shrewd skiff I'm getting the hang of it now. But would prefer a toggle mechanic rather than having to memorize two different characters I think. I can see how two different characters is more idempotent though, you can mash that key 10 times and know exactly the end state

#

(but current behavior of having to carefully orchestrate the cursor position before pressing those two characters, is NOT idempotent)

shrewd skiff
#

you might be dogfooding it even more than me at this point though

shrewd skiff
shrewd skiff
#

we are so back

acoustic radish
#

my session is going haywire..

Screenshot_2025-03-13_at_14.00.33.png
shrewd skiff
#

probably need newer engine? or new client?

#

guessing that's from Llm -> LLM

acoustic radish
#

Ah maybe

#

Trying again

acoustic radish
#

But agree with giving it a little time (just not too much time 😛 )

#

My nickname for this model is "russian doll": there is shell mode, and then inside it there is prompt mode. There is a clear hierarchy

shrewd skiff
#

i'm a Ctrl+Wer

acoustic radish
#

Bug report: when switching to prompt mode, with a 1password secret reference for llm config:

  1. I press >
  2. I get the 1password popup
  3. Almost immediately, this error appears: context deadline exceeded
  4. Pressing > again works
Screenshot_2025-03-13_at_15.19.38.png
shrewd skiff
#

oh fun

#

will fix, it's from the lazy initialization, tricky to work in to the UI loop

#

there's a 1 second timeout there which i thought would never get hit, forgot about password prompts 😛

acoustic radish
#

yeah and don't forget - user consent prompts are coming from @knotty imp too

#

latest multi-obj CLI & engine 👆

#

goddamn it 😛

Screenshot_2025-03-13_at_15.27.19.png
#

Then:


✘ base_container=$(container | from alpine) 1.2s
│ ✔ container: Container! 0.0s
│ ○ .from(address: "alpine"): Container! 1.1s
! returned error 422: {"data":null,"errors":[{"message":"Cannot query field \"loadLlmFromID\" on type \"Query\

⋈```
#

So, same error as before @shrewd skiff . I don't think it's a version mismatch issue

shrewd skiff
#

oh, yep, missed a spot

#

pushed 😅

acoustic radish
#

btw it's a little addictive that I can build + run any version of dagger, bootstrapped from an older version of dagger, with no git checkout or other dependencies, in another command

#

as our build gets gradually faster, this flow becomes more and more like a superpower

shrewd skiff
#

@acoustic radish for backwards compatibility I brought back the withFoo/foo setters/getters. worth it?

#

pushed - added the multi-object feature flag but kept its default as true for now since we're still on the multiobj branch

acoustic radish
#

or a more permanent reversal?

shrewd skiff
#

just wondering how concerned we are with keeping existing LLM demos working after merging llm-multiobj ultimately into main

#

without those, they'll have to make up a variable name i suppose

acoustic radish
#

I feel like it's OK to break everything that came before merging into main

#

as long as it feels like the best version of the API so far, at the time of merge

shrewd skiff
#

i don't have strong feelings on it yet though

shrewd skiff
#

re: mode switching, now that I'm using it, maybe the mode switch should just be persistent

velvet garden
acoustic radish
#

Yeah I got it wrong like 10 times in one demo

shrewd skiff
#

lol

acoustic radish
#

I managed to get it wrong once in the other direction - gave a shell command to the llm, and it actually executed it for me (for a small fee)

shrewd skiff
#

pushed persistent modes

#

pushed a fix for var are autocompletion in prompt mode (that's a thing)

shrewd skiff
#

pushed some more UI polish

  • builtin tool calls now look like function calls
  • actor emoji is now respected on non-message spans
  • syntax highlighting
image.png
shrewd skiff
#

could do a custom "tool call" UI if we're worried about overloading the regular function call look

shrewd skiff
#

update: I merged into llm, so maybe time to retire this thread

acoustic radish
#

ok!

#

thank you 🙏

#

@shrewd skiff what's the feature flag?

#

(if there is one)

shrewd skiff
#

I merged a bit aggressively because I'm tired of conflicts 😛 - going to just keep working off of llm since the plan was already made to work forward from llm-multiobj

#

we might want an explicit opt-in for shell mode though, depends on what we want to demo from here on

#

since setting a variable in shell is obviously going to be common

acoustic radish
#

If there's no FF then we need to prepare for breaking agent modules

shrewd skiff
#

they shouldn't break - they'll just keep using the withFoo/foo APIs

acoustic radish
#

ah so for now both APIs still coexist ?

shrewd skiff
#

yep

#

the LLM type is a chonker at the moment, though it probably already crossed that threshold haha

#

i think there's a chance this is actually a desired state anyway

#

since the notion of a "current state" never really went away

#

and not having to name things when you don't need to is nice (for the programmatic API)

#

but, it's up in the air

acoustic radish
#

I worry that it will be too much to grok...

#

but it was the right call to leave for now

#

we still have things to add before we start slimming it down:

1- modules
2- access to core API?
3- access to current module's functions and dependencies

no idea how to do 2 and 3

shrewd skiff
#

for 2) do you mean conceptually swapping the tools to Query?

#

(maybe without all the loadFooFromIDs)

acoustic radish
#

Not sure. Thinking about prompt mode and how to give your "copilot" access to the same environment as the shell (if you want it to)

#

or, maybe we explicitly don't want to

#

same for modules

#

one issue at the moment with passing modules: agent can never call the constructor

acoustic radish
shrewd skiff
#

sure

acoustic radish
#

was talking to Guillaume and Tibor about how MCP plugs into multiobj, and 🤷‍♂️

knotty imp
acoustic radish
#

all are welcome 🙂 on a call now but will propose a time after. maybe tell me what times work for you today?

shrewd skiff
#

I'll be good in about an hour - grabbing lunch

shrewd skiff
#

@knotty imp @acoustic radish I'm good whenever, no rush, just checking in in case I'm the bottleneck

acoustic radish
#

ready in 5mn

knotty imp
#

i am ready whenever, backfilling tests is like the most interruptible thing ever lol

rich marsh
#

I wish I could thread a thread 😬 what's the substitute for WithPromptVar if I'm converting some code?

shrewd skiff
#

forgot about that one - could add it back, since I brought back the withFoo ones, it'd be a drop in the bucket

rich marsh
#

no worries, I found the SetFoo too so I'm fully immersing myself 😛

acoustic radish
#

I've been thinking about bind as a possible verb.

#

I find myself using the terminology "bind an object to the llm environment" and it seems to work (ie. people understand & like it 🙂

shrewd skiff
#

I like it more than set which sounds like a mutation

#

do you have a new one for get? 😛

acoustic radish
#

but I have a plan

shrewd skiff
#

fwiw I still find with the clearest, aside from the clash with the current non-variable version

#

(which would be a dealbreaker if we're keeping it, unless we make name optional, but that would clutter up code usage)

#

give and take? 😛 (though 'take' sounds like the LLM no longer has it lol)

#

yeet and yoink

acoustic radish
#

almost ready

#

(my plan is to ask you guys)

knotty imp
#

if set is bind, get could ostensibly be bound

acoustic radish
knotty imp
#

in the FP bind analogy i don't think there's a named operator to get the value of a bound variable, but when talking about it you do call the thing a bound variable lol

acoustic radish
#

ok joining dev-audio!

shrewd skiff
#

(cc @knotty imp if you wanted to join)

shrewd skiff
#

another option for the pile:

withContainerVar("foo", ctr)
containerVar("foo")

(aligns with withPromptVar but I guess that could also be withStringVar)

acoustic radish
#

haha makes me think of withServiceBinding() 🙂

knotty imp
shrewd skiff
acoustic radish
knotty imp
#

usually verb tenses as APIs are a smell, but if bind is the right word it does kinda work

shrewd skiff
#

withServiceBinding becomes kind of an awkward inconsistency though (different use of unique word, different tense)

acoustic radish
#

Possible pairs:

  1. with<Foo>Binding + get<Foo>Binding
  2. with<Foo>Binding + <Foo>Binding
  3. bindFoo + getFoo
  4. bindFoo + boundFoo