Env API didn't recognize child modules in shell mode

[wraith pecan]

I have a module "agent_mod" depending on 2 modules "workspace-a" and "workspace-b", now I run dagger in the root of module agent_mod, and run env | .help, I can only see with-agent-module-input instead of with-workspace-a-input or with-workspace-b-input, which is blocking me to use chaining cmds to debug for this module. Is this a limitation or a bug ? In the code of the agent_mod, I can call env().with_workspace_a_input() or env().with_workspace_b_input() with no problem though, so I expect I can do the same in shell.

[bold bear]

I would say it's a bug.

what you can do is .cd agent_mod from the shell @wraith pecan

it will "enter" the module to query it from the inside, including dependencies. whereas dagger -m queries a module from the outside

I think

but wondering if maybe I tried that already and it also doesn't work 🤔

cc @sterile mason @latent sluice @fast phoenix I have a vague memory that we discussed this in the past?

[latent sluice]

oh interesting, this is very related to https://github.com/dagger/dagger/pull/9992, and the intent of that PR was very much that this should work the same with -m and -c '.cd agent_mod , in either shell or SDK code.

i wonder if maybe this bug is specific to env, though? @wraith pecan am i correct in guessing that if you dagger -m agent_mod and then do workspace-a | .help, you can see its functions and whatnot that way?

i think this repros in dagger/dagger/.dagger... like env | .help should show with-wolfi-input (from modules/wolfi) , but it does not

[wraith pecan]

yes I can see functions of workspace-a

I usually just run dagger alone in root folder of a module, so if I run dagger in workspace-a root, then do env | .help, I can see with-workspace-a-input there but not when i run it in agent_mod module context.

[latent sluice]

Yep that makes sense I think, but I imagine it doesn’t help because whatever you’re trying to do is stitching together the context from both child modules

Tomorrow I’ll see if I can grok what’s different about the dynamic environment api generation such that it doesn’t see those dependency modules (cc @fast phoenix if you’ve got tips)

Intuitively it should see them

[wraith pecan]

if it helps: more context what I want to achieve - I want to use 2 llm agents in sequence which uses one of the 2 child workspace modules respectively, so I really want to be able to initialize both llm agents with envs using different workspaces in one context of the root module agent_mod

[latent sluice]

Yeah makes sense. Tbh I personally find it frustrating that all this stuff has to happen across module boundaries, but there are technical limitations that make it very difficult for us to allow all this inside of a single module

[mint vortex]

IIRC this was an optimization so the shell doesn't have to eagerly load all the module dependencies and register them in the graphql schema like when you do a dagger call. I think @reef moon probably has more context about this

[bold bear]

yeah there's no need to load them until you call the corresponding command

[latent sluice]

I have seen traces that suggest that they’re at least built, if not loaded

[bold bear]

Each client that loads modules does so in a completely different context, so each does it their own way:

• dagger call: loads the target module only, no deps
• module runtimes: load the dependencies only, not the target module (ie. no self calls)
• dagger shell: loads both the target module and the deps (lazily)

[bold bear]

Well the target module itself is loaded, and recursively loads its dependencies. That might be what you're seeing in the trace?

[latent sluice]

Yes, but it didn’t look lazy to me?

Oh also I see, I was seeing the module runtime loading the deps rather than the shell client?

[bold bear]

just my guess

[latent sluice]

Makes sense in the context where I saw it, at least

[mint vortex]

@latent sluice maybe as you mentioned above we need the shell to also perform the deps serving (https://github.com/dagger/dagger/pull/9992/files) as we do in prompt mode?

[latent sluice]

Naw they’re the same

But need to trigger a similar thing for env, just not sure how yet

[latent sluice]

for some reason dagger -c ".refresh; env | .help"gives the desired behavior, as does cd workspace-a && dagger -c ".cd ..; env | .help"

feels like maybe an order problem

[mint vortex]

woot! seems like it 🙏

just tested .refresh and works indeed

[latent sluice]

the problem is i'm not sure what piece of machinery is happening before the dep module "Serve" calls that should be happening after.... i've tried emulating the impl of .refresh in the initial-module-load path (basically making sure to add the served dep to h.modDefs and load it's typedefs) and it seems to make no difference

[latent sluice]

revisiting a @reef moon PR comment from 2 weeks ago, and he explicitly starts wondering about consistency and i wonder if he meant that in the ACID sense (i read it as semantic consistency at the time, but it can also read as data consistency)... and maybe he's got some idea what might be different about initial-module-load vs (.cd || .refresh) here?

[reef moon]

Replied in the issue.

[latent sluice]

so you did mean semantic consistency. still, in the context of this issue we have one behavior occurring on initial shell load and a different behavior occurring with .cd and .refresh ... perhaps this would be fixable by moving the deps-serve to the engine, but it's not clear to me why there's inconsistent behavior on the client when it appears as though all these UXes use the same module load callpaths

made an issue for further discussion of the bug here https://github.com/dagger/dagger/issues/10116