#user defaults

1 messages · Page 1 of 1 (latest)

fringe cape
#

For context, we're trying to make modules that by convention define a "up" command that requires zero arguments. It will be called by a parent module that controls all of them. It'll be great to offer sensible defaults but allow overrides during testing.

visual olive
#

Hi! You're not the only one asking 🙂 Not yet possible but we're working on it! Hoping to have it shipped in a few weeks

fringe cape
#

Thank you for responding. So I understand, what's the state of default args today? It looks like *dagger.Directory is supported. Is that all for complex types?

#

And also, while that PR looks useful, it seems like it's defining more .env.dagger.json support. I'm wondering, can design our module constructors with default secrets today via the // +default syntax?

ancient helm
visual olive
#

Example from the POC:

Screenshot_2025-09-04_at_14.20.06.png
#

That POC defines a new .env.dagger.json but that's just a placeholder. I'm also experimenting with using an actual .env file. TBD

ancient helm
#

Ahh, so a module doesn't define a .env.dagger.json, a source directory does?

#

So say I was running dagger -m <path/to/gitlab/module> call <functions> from a directory that has a .env.dagger.json then it uses the defaults?

visual olive
#

(so, same lifecycle as .env)

#

@ancient helm what you describe is also possible, but would require having a way to map arguments for multiple modules in the same env file

ancient helm
#

Yeah, trickier. Having it in the module is already an good option to have though. Presume it works for all secret types we currently have, including files?

visual olive
pearl ether
#

Hi Solomon, would we be able to define a .env file for a module, then install that module as a blueprint module and expect it to have access to the secrets it needs?

visual olive
#

tomorrow I plan on using my building blocks and trying a new POC using an actual .env, maybe try supporting multiple modules, see how it feels

#

thanks for the feedback & questions, keep em coming! 🙏

pearl ether
#

Cheers. I can see this is going to be very useful for us. Just this week our team had a meeting about how to solve the problem of each module (which we are using as a blueprint in company repos) requiring slightly different secret and confirg parameters to run. This feature looks very promising 🙂

pearl ether
#

I have a few more questions Solomon:

  1. would it be possible to support a default value for a socket in the .env file. Ideally we would like to do something like this:
# .env

ssh=${SSH_AUTH_SOCK}

This would set our module constructor parameter ssh *dagger.Socket to the value of the system env variable ${SSH_AUTH_SOCK}

  1. Will it be possible to set env variables to other env variables (kind of implied by the first question)...? I.e.
# .env

addressOfSecretsManager=${BAO_ADDR}

The idea here is that you don't have to hardcode the value into the .env file, but can reference other environment variables, possibly even composing a new env var from multiple?

visual olive
#

I'll explore in my next poc

#

@pearl ether could you give me a realistic .env file that you would expect to write with this system? just to get a reference point of realistic use

pearl ether
#

Given the following module constructor:

func New(
    ctx context.Context,
    // the address of the bao server
    baoAddr string,
    // the token for the bao secrets manager
    baoToken *dagger.Secret,
    // the aws profile to use
    localProfile string,
    // the aws credentials to use
    localCredentials *dagger.Secret,
    // the source to compile
    // +defaultPath="/"
    // +ignore=["**/.venv", "**/__pycache__", "**/node_modules"]
    src *dagger.Directory,
    // ssh socket used for cloning the source
    ssh *dagger.Socket,
) (*Main, error) {
    ...
}

We would want to be able to default everything with the following .env file:

BAO_ADDR=${BAO_ADDR}
BAO_TOKEN=env://VAULT_TOKEN
LOCAL_PROFILE=${AWS_PROFILE}
LOCAL_CREDENTIALS=file://~/.aws/credentials
SRC=.
SSH=${SSH_AUTH_SOCK}
visual olive
#

Update: made zero progress developing the POC, spent the day fighting dagger's own build and codegen tooling.

#

(ironic I know)

visual olive
#

default secrets

#

Update: IT WORKS!

visual olive
ancient helm
#

Looks great. In two minds over whether I'd want this in the module itself or the workdir, but I think I could make either work fine

pearl ether
#

Hi Solomon. Thanks very much for working on this. I had a play with this and the initial feature looks promising. There are a few things I am wondering could be included in this feature:

  • support for socket addresses ( I gave it a test, but got the error: Address has no such field: "socket"
  • ability to interpolate variables from the environment in the .env file:
# .env
MY_PARAMETER=${HOST_VAR}
  • ability for modules to ship their own .env file and when the module is installed aas a blueprint module, that .env could be used. Use case is we have over 3000+ repos which we would like to plug our platform dagger modules into. If those platform dagger modules can bring along their own .env when installed as a blueprint module, this would be really convenient... don't need to create a .env file for each repo
visual olive
pearl ether
#

In addition to this, I had some issues getting my .env variables to picked up when:

  • the module name was in the format something-else. It seems in this case env var prefixes SOMETHING_ELSE, nor SOMETHINGELSE seem not to work.
  • if the parameter is in the format somethingElse. It seems that env var names SOMETHING_ELSE, nor SOMETHINGELSE seem not to work.
visual olive
#

the module name was in the format something-else. It seems in this case env var prefixes SOMETHING_ELSE, nor SOMETHINGELSE seem not to work.

Yeah I need to add that

pearl ether
#

Thanks Solomon!

visual olive
#

ability for modules to ship their own .env file and when the module is installed aas a blueprint module, that .env could be used. Use case is we have over 3000+ repos which we would like to plug our platform dagger modules into. If those platform dagger modules can bring along their own .env when installed as a blueprint module, this would be really convenient... don't need to create a .env file for each repo

This one is tricky, I totally see the use case, but checking it into source control might create more problems than it solves, no?

Maybe we could integrate with specialized environment persistence services? Like vault, 1password? Or, maybe we should ship our own in Dagger Cloud 🙂

#

We've also wanted to add a concept of persistent environment in Dagger itself... We need it for a variety of reasons.

pearl ether
#

If .env files are not meant to be used like that, totally understand. At nine we commit our .env files but they don't contain anything secure. We have a .env.override for stuff like that and it is .gitignore'd. Definitely open to specialised environment services. We already do a bit of this to avoid the parameter overload.. we use open bao ( a fork of vault ) and inject one secret into our dagger modules which authorises us to open bao, then we get all the secrets we want out. There is a still a developer experience issue with making the developer provide these flags to the dagger command so we ship a wrapper for doing this. One of the problems we are encountering however is that its hard to unify this wrapper across all the different variants of modules that are being developed. The idea of the .env file living in the module directory and being utilised appealed to me because it promised to get rid of the wrapper.

#

Open to other alternatives though

visual olive
#

Or, we could add a CLI configuration field, where you point to a remote .env file and it automatically gets loaded on all calls. Then you push that config to all dagger users on the team. Then you simply upload the shared .env at that location and boom all clients load it

pearl ether
#

That sounds like it would work.

visual olive
#

We're 100% aligned on the goal, we'll find a solution for you. Just want to make sure we do it right.

Committing your .env isn't necessarily wrong, if you're careful to not commit anything you regret. We just can't rely on all users being that careful, or even knowing that they should...

#

@pearl ether do you have a good way to push configuration to everyone's workstation? (or simply ask them to all install it) Or is that a pain?

pearl ether
#

I think we will probably be asking them to install something

#

We would require them to install some tool which would fetch and cache these .env files.

#

Its not the end of the world and in the past we have used this approach to deliver things to people's machines

visual olive
#

If the dagger CLI itself can auto-fetch them, it would be strictly easier then. they'd just have to run a one-time command to configure dagger to do it

pearl ether
#

Sure that seems doable

visual olive
#

eg. dagger config dotenv https://url/of/your/shared/dotenv or something like that

pearl ether
#

Yeah I think this could work. Would we be able to configure a TTL on this file or even have it fetched everytime?

#

I can imagine, someone adds a parameter to module x and wants to supply a new default for it. They would want users of that module to be able to get the change soon

visual olive
#

Well it would probably use dagger to fetch it 🙂 So the caching mechanics would be the same

#

For http and git remote fetches, you would get the change right away as long as you don't pin to a git commit

pearl ether
#

Yeah let me run this by the team and get back to you.

#

I like this sound of this, and I think it would probably work

#

Spoke to some guys in my team and they were actually not against just sticking a .env in every repo haha.

#

So it could just be me who wants that

#

Will keep the convos going and get back to you.

visual olive
#

ok 🙂

pearl ether
#

Bottom line though is we want to dagger call <> wrather than wrapper call <>!

visual olive
#

by the way we're happy to huddle with more people from your team if you'd like

visual olive
pearl ether
#

Indeed

visual olive
#

Anything that you feel is pushing you towards a wrapper, please let me know and we'll fix it

pearl ether
#

haha, well there is one more thing actually

#

because we have a traffic steering proxy in front of all our machines and it terminates our SSL traffic, we have to tell most tools to accept custom certs.

#

These certs are pushed to our machines from IT.

#

To get this working with dagger right now, we have to use the documented approach: boot our own dagger engine with the certs mounted, and set the environment variable to point the dagger CLI to this conatiner.

#

If this could be configured once on every machine via some config file living in the user directory, would be awesome

#

Right now our wrapper also contains a command to set this up which has to be run anytime you jump into a new shell

visual olive
pearl ether
#

Ahh very nice.

#

Yeah at the moment these are the two things we are most concerned about that are causing us to use a wrapper:

  • custom certs
  • parameter defaults
visual olive
#

I think that would also be a local config directory (eg ~/.config/dagger/certs) and boom, as soon as you copy them in there, all future containers automatically get those certs installed

pearl ether
#

Yeah that would be ideal

#

Thanks for all your help so far Solomon. I will discuss further with the team and get back to you

pearl ether
#

After discussing further with the team, I think they were very happy with the feature as it stands, and thought that we could live with putting our .env files in each repo

#

Next week we are going to trial it and see how well it works for our use case

visual olive
pearl ether
#

Very prompt Solomon 🙂 Thanks and all the best for the keynote

visual olive
#

@fresh moth 👋 this is the thread for testing the "default secrets" PR

visual olive
pearl ether
#

Hi Solomon, is there a way to "unset" a default over the commandline:

# .env

AWS_CREDENTIALS=file://.aws/credentials

dagger call my-function --aws-credentials=nil
#

We are intending to check in our .env files - this is intended to allow the local experience to be seamless. When running things in CI we sometimes need to turn off some of these parameters especially if the file doesn't exist ( when using secret providers ), otherwise the dagger cli would give us an error.

#

If not, all good. There are ways around it from our end. We could for example mutate the .env to leave parameters out before running the dagger function in CI.

#

@haughty basalt

visual olive
#

@pearl ether one option that comes to mind is keeping .env.local and .env.ci, and selecting one or the other with eg. dagger --env-file .env.local

wdyt?

ancient helm
#

@visual olive Is the design you've settled on the <module_name>_<variable_name>=<env/file>:<location> showed earlier in this thread? And that file goes not in the module but in the source where you're calling the module?

visual olive
#

I want to add detection of the current module, and in that case allow omitting the module_name_ prefix

ancient helm
#

Either would be fine. I like the explicit nature of module_name_ but I can see that could lead to repeated secrets in a .env

#

This - large numbers of secrets - came up in feedback to move one of our larger processes into Dagger yesterday, this feature is quite coincidentally under development at a good time

visual olive
#

yup top priority for me 🙂

pearl ether
# visual olive <@692481168024666172> one option that comes to mind is keeping `.env.local` and ...

Yeah I think that would work. If say there were two files in the directory: .env and .env.ci, would dagger by default use the .env unless the --env-file flag was supplied? If so, this would basically solve it for us. We would have two files in each repo. The developer would simply dagger call <function> for their day to day. When we run in CI, our github actions config would specify the --env-file and that would get us onto the "CI set" of defaults.

visual olive
#

I've been making progress on the component features, to get them merged asap. But haven't integrated them back into the poc yet

-> socket support
-> var expansion

pearl ether
#

All good Solomon

haughty basalt
#

I'm not too concerned about the need for both a local and ci .env file, but the option is nice. Reducing the verbosity of running the commands locally is my main concern - this really isn't a problem for CI as the params are defined in an GH workflow file.

Thanks for all the work Solomon!

visual olive
#

honestly I can't wait to use this feature for myself 😁

#

thank you for the feedback it's super helpful!

pearl ether
#

Ah reason why I think we might need the ability to set the --env-file that @haughty basalt is because there will be cases where the local .env sets some fields that we don't want to have set in the CI. For example --aws-local-credentials=file://~/.aws/credentials. As this file is not available when running CI, we need this to not be a default, as dagger will error out when trying to find the file (even if the parameter is optional - i believe). Something to kind of "unset defaults" or switch to a different set of defaults in this case, would alleviate this issue.

#

I am pretty sure that even if the parameter is optional, if you pass a secret provider that doesn't exist, dagger will error out saying the file doesn't exist.

haughty basalt
#

Agree the flag is a good idea --env-file

visual olive
pearl ether
#

We could I guess override it at call time to an empty file --aws-local-credentials=file://empty-file,

#

We use other flags to tell our dagger module not to use it

#

Okay if that's going to work, we don't need the --env-file stuff

haughty basalt
#

I think being able to opt in and out of the env is a good idea, even if the flag was inverted to --no-env-file

pearl ether
#

^ another good option I guess for us to avoid using the .env file in the CI

visual olive
haughty basalt
#

I like the idea of a clean command that automatically finds the .env, my main concerns are:

  • the behaviour is not immediately clear to the user that defaults are being supplied
  • It feels like it makes optional parameters unnecessarily complicated. We now have a clean default call, but if we want to avoid those defaults for an optional parameter a dummy value must be supplied.

I like the idea of supplying a --env-file flag to opt-in. It will make it clear to the user where the values are coming from, avoid the need to override/handle defaults you don't want to pass, and retain existing behaviour of how a function is called.

If automatic retrieval is preferred, I think --no-env-file is required so a user can opt out without needing to delete the file.

Just some thoughts. Happy with whatever you decide since you are doing the work, and the core feature is really what we are after 🙂

haughty basalt
visual olive
#

I think we need to have it opt-out, I see the potential downsides, but they are outweighed by the benefits, namely: everyone is tired of typing so many arguments all the damn time. It's the number one complaint.

For sure we should explicitly tell the user when defaults are applied. I think that will help avoid confusion.

haughty basalt
#

Sounds good

pearl ether
#

All sounds good to me as well

visual olive
#

local defaults

ancient helm
#

I've focused on secrets here, but I'm assuming with the name change this could extend to paths? What about git paths - if the caller has git auth it would just work?

module_name_attribute_name=https://gitlab.com/path/to/repository would work as a default path? If that's the case between secrets and paths I could eliminate all but two parameters

visual olive
pearl ether
#

amazing thanks Solomon

visual olive
#

I started a separate thread for the implementation/review sprint (to go from POC to release): #1415426492057391315

visual olive
pearl ether
#

Awesome Solomon!! Will have a play today

pearl ether
#

Hi Solomon have had a play today and it looks good. Played with the following .env file:

DFLOW_STRVALUE=awesome
DFLOW_SSH=/private/tmp/com.apple.launchd.FReBGYGEXd/Listeners
DFLOW_SECRET=env://MY_SECRET
DFLOW_FILE=file://go.mod

This all works great and the defaults are picked up 🙂

#

Didn't have any luck when my module was installed as a blueprint unfortunately...

#

I had a module called 'dflow' which was installed as a blueprint module in my project directory:

{
  "name": "project",
  "engineVersion": "v0.18.17",
  "blueprint": {
    "name": "dflow",
    "source": "../dflow"
  }
}
#

With the above .env file was getting this error when running:

! required flag(s) "file", "secret", "ssh", "strvalue" not set
#

I imagine this is still WIP.

#

The other thing was the variable interpolation also wasn't working - I assume you also know about that !!

#

In which case you can tell me to calm down hahaha

#

But overall, this feature is shaping up to be exactly what we want. Very exciting. Thank you getting this together Solomon. This is going to unlock a lot of opportunities for us.

visual olive
#

@pearl ether thanks, I got sucked into getting the lower-level PRs merged (lots of fixing tests, addressing nitpicks from my fellow maintainers etc 🙂

Getting back on track on the feature itself today

visual olive
pearl ether
#

ooh cool will try

#

Yeah sorry if I'm bugging you too much with the feedback.. I'm sure you're like "yes I know"

#

hahaha

visual olive
visual olive
#

And in fact I had forgotten to enable expansion in the PR..

pearl ether
#

Thanks Solomon will give another try with the expansion later today

visual olive
#

@pearl ether 2 follow-up questions:

1. Quote stripping

  • .env format: would you expect quotes to be interpreted, or left raw?

For example:

TOKEN="env://foo"

Would you expect the quotes to be stripped, or left as-is?

--> At the moment they are left as-is.

2. Blueprint vs. downstream name

In your example above, which var name would you expect to see work?

  • DFLOW_FILE (match against blueprint name)
  • PROJECT_FILE (match against downstream module name)
  • both DFLOW_FILE andPROJECT_FILE (match against both)

--> At the moment only PROJECT_FILE is matched

visual olive
vestal marten
pearl ether
#

I will get my team to help out here

soft aspen
visual olive
#

Yes if we interpret quotes, we'd handle escape also

#

I'm sad to say at the moment our newly merged EnvFile type does not interpret quotes. But it's an easy change

#

Technically that change would be a breaking change, but since we haven't released yet, I think we're good

visual olive
#

@pearl ether I just pushed support for blueprint name. So in your example above, both DFLOW_xxx and PROJECT_xxx will match (if both are set, they will be merged, and the more specific PROJECT_xxx will win in case of conflict.

#

cc @unkempt comet

#

(since it's relevant to blueprints 🙂

pearl ether
#

that sounds great

pearl ether
#

On the first point, I agree strip the quotes

#

Second point, if both are there could be great... I was personally geared towards matching against the blueprint module's name ( as this will make copy and paste within our organisation easier )!

ancient helm
#

I want to add re: expansion, we use file:~/.git-credentials to access anything in Gitlab, I'd want that ~ to expand as it currently does

visual olive
ancient helm
#

Not tried it yet, just wrote some docs about the upcoming changes as a result of this and realised one of the default args we'll need to pass is that one. I'll have to do a bit of module re-writing to test it as well (currently provides those attributes with a method) so not sure I'll get to that before my annual leave

visual olive
#

Finally got TOKEN=foo to work party_blob

#

Pretty awesome to use 🙂

visual olive
#

@daring sandal looking into non-constructor arguments...

#

@daring sandal are you thinking arguments one level deep? (arguments to functions of the module's main object?). Or also functions attached to arbitrary types?

visual olive
#

Update: support for any function is almost working 🙂

daring sandal
#

Wow! IDK how I totally missed this entire thread. Apologies for not participating earlier but I read through all of it and I'm very happy with the progress! Thank you so much @visual olive for working through this. To answer your question,

are you thinking arguments one level deep? (arguments to functions of the module's main object?). Or also functions attached to arbitrary types?

I think it will be less confusing to consumers if it applied globally (to all functions attached to any dagger addressable object). Otherwise it adds a layer of complexity when using defaults.

Finally got TOKEN=foo to work
Can you elaborate what you mean by this? Is it the vars that don't have a module prefix?

daring sandal
visual olive
#

I think it will be less confusing to consumers if it applied globally (to all functions attached to any dagger addressable object). Otherwise it adds a layer of complexity when using defaults.

Agreed, I just pushed a commit that does this 🙂

visual olive
#

It's working really well... You can now configure defaults for any argument of any function of any type of any module 🙂

# 
WOLFI_CONTAINER_PACKAGES=["git", "openssh", "rsync"]

dagger -m github.com/dagger/dagger/modules/wolfi -c 'container | terminal'

# 
$ cd dagger
$ cat .env
TEST_SPECIFIC_RUN=TestAddress/TestValue
$ dagger call test specific
#

Full bootstrap of a discord exporter workspace:

$ dagger init ./my-discord-export --blueprint github.com/shykes/x/discord
$ cd my-discord-export
$ cat >.env <<.
token=op://myvault/discord/token
.
$ dagger call servers
soft aspen
#

notsureif trying to think of any spooky accidental defaults, but, i guess they're defaults, not overrides so the risk should be kinda low?

ancient helm
visual olive
visual olive
#

One interesting side-effect is that, with explicit CLI flags becoming less important (because you can persist them in .env), dagger call becomes more viable

ancient helm
#

I'm still using dagger call. Couple of .dag scripts for stuff like tests but otherwise... CLI!

visual olive
#

Yeah I just can't stand having to type 15 explicit CLI flags, and have to remember their names each time...

#

But if I don't have to pass those 15 flags anymore, that's different

ancient helm
#

I'd agree... If I typed them. They're all saved as parameterised workflows in Warp. I just tab through parameters picking from customised enums, it's a snappy workflow.

visual olive
#

@soft aspen I was able to print a test log message on the TUI with the "global logger", thanks 🙂

Is there a way to print just the message, without the trappings of a log entry, eg. INF, the date etc?

#

Also I need to find a clean way to plumb through those global log calls all the way down to TypeDef and other low-level core types, but that's a me problem 🙂

soft aspen
visual olive
#

I have no concept of the respective roles of telemetry vs. engine/slog... How does moving those functions to telemetry help? (sorry)

soft aspen
#

i added this to engine/slog/ which is too specific, this plumbing should really move to the more generic layer

#

if you turn GlobalLogger into something that just returns the ctx it currently passes to SpanLogger it'll be more generic, and then you can just use that, like telemetry.NewWriter(telemetry.GlobalLogsSpanContext(ctx)) - and then just write your logs to that

#

(can make the change myself once im in between things if you want)

visual olive
sick lava
visual olive
#

@sick lava expansion should work, but it is sandboxed to other variables in the .env. Each variable can reference variables defined before it (to avoid cycles)

System env variables are not used in the expansion.

#

I'm not familiar with how other .env parsers handle expansion, my goal is to be as standard as possible, but I'm starting to wonder whether there is a standard 😅

sick lava
#

System env variables are not used in the expansion.
Ah yep that would make sense

visual olive
#

note that when passing arguments of type Secret, you can pass env://FOO as a value, which does reference an env variable on the host system. But that's unrelated to .env (you can pass the same value as a command-line flag)

sick lava
#

Thanks for the clarification! Think I was a little too keen on trying to remove all of the command-line flags from our dagger calls so thought I could put env expansions into the .env

visual olive
sick lava
#

Some are along the lines of passing in local AWS credentials, which works great with using the file:// directive in the .env. Other ones are like passing in the local AWS profile through via the system env

visual olive
#

OK after some light research, it appears most .env compatible tools do allow interpolation from system env vars, so I'll align with that

sick lava
#

So as I understand it the expansion (at least at the moment) works in a similar fashion to how we could define and reuse variables in Dagger shell?

soft aspen
#

@visual olive pushed

visual olive
sick lava
visual olive
#

Update: required cleaning up the envfile implementation, to better match the dotenv convention. It took longer than I though.. But now its done. https://github.com/dagger/dagger/pull/11111

Now I can build on top of this to add system env variable fallback, and check the rest of the boxes

GitHub
EnvFile: better dotenv compatibility, more tests by shykes · Pull ...

Evaluate quotes by default: foo="a b c"
Support default values in variables: foo=${bar:-baz}
Cleaner implementation, with a low-level go package
Lots of tests
Optional "raw" mode

pearl ether
#

Looks good!

visual olive
#

Should be pretty trivial to add command expansion too, eg. $()

pearl ether
#

cool. Very powerful this feature. Basically solving the inputs to the dagger function completely

visual olive
#

@fresh moth you were right about where to attach the local default values... I'm gradually converging to dedicated fields. It happened naturally as I cleaned up the code 🙂

solar bough
#

system env variable fallback
😍 😍 😍 this is becoming such an amazing feature... !!!!

visual olive
vestal marten
#

So great to see so much love for this feature ahah

visual olive
#

Which is good because I have a mysterious bug blocking me right now 😭

visual olive
#

UPDATE: system variable fallback now works 🙂

visual olive
#

@agile quail my user defaults are applied correctly on call, but they don't show in introspection (eg dagger call --help, dagger -c .help). I thought I had the right entrypoint (Function.FieldSpec, now wrapped by ModuleFunction.FieldSpec) but maybe I'm missing one?

#

Ah I guess FieldSpec is only for graphql introspection... I need to also change the typedef itself sometime before install

agile quail
visual olive
#

Yeah that's how I used to do it, before switching to a lazy model

#

just walked all the typedefs and overwrote FunctionArg.DefaultValue at load and called it a day

#

Now I need to either 1) override FunctionArg.DefaultValue lazily (but where? at install I guess?), or 2) create a new LocalDefault field in the typedef, and change the client introspection behavior. I noticed the CLI doesn't show defaultPath so there's a little client work overdue

#

I'm just worried that client-side introspection behavior becomes super complicated

#

Also @agile quail how is +optional expressed in core.FunctionArg? I don't see an Optional bool field. Is it DefaultValue: JSON("null")?

agile quail
visual olive
#

Mm damn I don't know where to hook up my "lazy pull" model for typedef introspection...

#

Ah maybe ModuleObject.Install()

#

I traced it down to core.objFun(), I'll try adding my hook there

visual olive
#

IT WORKED! 🙂

#

And I think I can even remove my patched FieldSpec, since that's derived from the typedef anyway

pearl ether
#

Hi Solomon 🙂 Looking good! I gave this a little test on my machine. Here is my source code:

https://github.com/chrisjpalmer/env-test

I was getting some errors and it didn't seem to be picking up my .env file.

This happened if I tried running my dagger function in both the project and dflow directory.

Trace for running in the dflow directory: https://dagger.cloud/nine/traces/24feea32156bd59c5f9a03168fbb6563

Trace for running the the project directory: https://dagger.cloud/nine/traces/020197a55c3e9419785d31541f6df614

GitHub
GitHub - chrisjpalmer/env-test

Contribute to chrisjpalmer/env-test development by creating an account on GitHub.

Dagger Cloud

Browse and visualize Dagger traces.

Dagger Cloud

Browse and visualize Dagger traces.

#

noticed you just force pushed to the branch... don't think I caught that latest commit
63a6091

#

Might try again a little later

visual olive
#

Thanks @pearl ether . Yeah I'm doing some cleanup and stabilization. Will ping you in a little bit when I think it's ready

#

Let me check out your trace

#

I'm running the full test suite right now

#

(yeah looks like you hit the bug I was just fixing)

pearl ether
#

aha I was a little too keen

#

Cool will test later today 🙂

visual olive
#

we're almost there!

pearl ether
#

hahaha nice

#

I have to get in on this engine dev at some point, it looks fun

#

I tried reading the source code the other day

#

it broke my brain

visual olive
#

Ha ha I'm on that journey as well 🙂

#

It gets easier

#

It's not an all-or-nothing, more like a video game where you unlock new levels as you go

#

I still have many more levels to go 😄 But making progress

#

LLMs are a lifesaver... I could never navigate the codebase without one

pearl ether
#

Wow cool

#

Next time will be getting my LLM out

visual olive
#

@pearl ether should be good to go

pearl ether
#

Awesome will give it a test in a few hours time 🙂

pearl ether
#

I will have to test this on monday Solomon, ran out of time for today

#

Thanks for everything so far and looking forward to it

visual olive
#

@agile quail thanks for the review 🙂

#

@agile quail there's one item on my checklist I'm struggling with... I need to nerf the feature so that it only applies to top-level modules, not dependencies. It's actually quite powerful that you can so easily inject dependencies anywhere in the dependency tree... But it's not what users will expect by default, so better to keep that particular pandora box closed for now 🙂

The issue is, I'm not sure how to reliably check for that... I tried a few approaches but it kind of melted my brain

#

What's tricky is that the code to apply local defaults, for a given module and call, may be called in two different contexts:

  • At module install (to merge them into the typedefs for introspection) -> the context is the module install itself I believe? ie. calling serve. Or maybe not? crazycharliepepesilvia

  • At function call (to actually apply the defaults in the call) then the context is the one making the call. Could be a CLI calling top-level module; or module calling sub-module.

So my code needs to cleanly handle both

agile quail
#

I can't think of any way to do it other than add an arg to ModuleSource.asModule (this api) that either enables/disables the .env default loading.

  • You can hide the arg from the public schema if desired (example) so then it's only available to internal calls

Then you'd load dependencies with that arg set here

#

The downside is that it means we won't share cache for loading a module directly vs. loading a module as a dependency, but I think that's just an inherent implication of this feature. Also not that absolute end of the world

visual olive
#

that way the modulesource remains the same?

#

oh wait, evermind. asModule() returns the Module

#

so ModuleSource would remain the same

#

yeah that makes sense actually.

#

@agile quail re: the caching issue. Are you saying that, even if a module source has no local defaults (and therefore loading it with or without local defaults applied, makes no difference), they would still not share a cache entry, because we do "recipe caching" as opposed to full-blown "content caching"?

#

Mmm I could refactor the code so that the local defaults to apply are passed as argument to asModule, rather than just a boolean flag

#

(if it makes a difference for caching)

#

mm no that's weird

#

I think you're right that the cache miss might be inherent to the feature

agile quail
visual olive
#

Ok nice. Yeah I remember updating the content-caching part of module loading in my PR, was cool to discover

pearl ether
#

ah interesting... recipe caching vs content caching

#

is recipe caching where the dagql is cached so it fetches a result based on the dagql, where as content caching is based on buildkit artifacts ?

#

forgive a poor man's attempt to explain ... i mostly have no idea what I am talking about 🙂

#

Just noticed sometimes if dagger has evaluated a module's function before, it often doesn't run the second time

visual olive
#

I believe both are recipe caching

pearl ether
#

ah cool

visual olive
#

The difference is that there are infinite ways to produce the same directory. With content caching you only worry about the end result, so you get more cache hits. But you need to compute a hash of everything which would be prohibitively slow.

pearl ether
#

ah does dagger ( or buildkit ? ) decide which one to use based on the size of the directory?

visual olive
#

no it focuses on the operations. Which operation was performed with what inputs

pearl ether
#

so most of the time things are recipe cached by the sounds of it?

visual olive
pearl ether
#

cool

#

thanks for the clarification always good to konw

pearl ether
#

Hi @visual olive awesome I tested this and it works 🙂

#

This is a level up for dagger. So excited for this to ship!

#

I thought that if the .env file was checked into the module directory and the module was installed as a blueprint, it might still work... however this didn't work.

#

Just wondering if this is still on the cards or not ?

#

If not all good, just wanted to make sure

#

Otherwise, putting the .env in the repo's directory which installs the module as a blueprint works perfectly.

#

Awesome stuff

visual olive
#

ah right, at the moment it will only work for local checkouts. So you can still check them into git, but they'll only be applied when dagger loads them locally.

#

I think it's safer to start thet way, then see if there's the need to loosen the restriction. It's a one way door..

pearl ether
#

yeah understandable I think this gets us enough ground to run with for now

visual olive
#

but, we can add support for remote env file via a flag, if still interested 🙂

ancient helm
#

Ship this first please 🙏

pearl ether
#

if that was a flag that you could default to being on (through dagger config) that would be ideal

#

we would love to give developers dagger call x and nothing more if possible

#

haha

#

even if we have to put a .env in every repo, its better for adoption if the learning curve is low and developers only do dagger call x rather than something say dagger call --remote-env=<> x

#

Haha I see @ancient helm is keen to ship

#

Also share this sentiment 🙂 happy to wait

ancient helm
#

Yeah, I've got 3 git URLs, 3 secrets, and 1 secret file that can all be defaults, leaving perhaps 2 or 3 arguments

#

I've even written a function that creates the env file with either main (for testing) or latest release versions of those git URLs (for production)

visual olive
ancient helm
#

Returns a file that contains the env contents, with either latest released versions (queries Gitlab) or no ref to just use main (we run tests on main before releases go out to users)

#

Makes it very easy for my team to test

visual olive
#

Approaching the finish line... (I hope!)

visual olive
#

user defaults

#

Renaming the feature to USER DEFAULTS

fresh moth
#

spent a chunk of time trying it out, left a review ❤️

#

neat

visual olive
#

thank you!

ancient helm
#

Questions on specific implementation @visual olive:

Does my module need a constructor to use these, or is the default hidden constructor fine, and all I need to do is ensure the env file names match the struct attribute names?

Pre-filtering pragmas still apply to directories provided by default value?

#

Third maybe question, maybe misunderstanding. I was under the impression the env file was to be located in the directory calling the Dagger function, not the module itself, but the PR suggests it can be in either? So my Dagger module can have a .env full of THIS_MODULE_NAME_ATTRIBUTE_NAME=DAGGER_THING and the directories calling that module with dagger -m <git/url/to/module> <function> don't have to have that same env file? So if I have 30+ services calling one function, they don't all need a copy, it's enough to have one copy in the module?

visual olive
ancient helm
#

When you say "only works for local modules" that would mean dagger -m <git/path/to/module> from another directory wouldn't work, it wouldn't use the env file in the module?

visual olive
#

I mean it does work

ancient helm
#

I'm happy to hear that, but I don't understand how given your previous statement. Not critical that I understand if it works though 😉

#

dagger -m <any/potentially/untrustworthy/module> could have an env file in it and scoop secrets

visual olive
#

The engine checks whether the module is a local or git address. if the former it loads .env from it. if the latter it doesn't.

#

yes you're right that a local checkout could still be untrusted. It seemed like a reasonable tradeoff, we can fine tune it

ancient helm
#

So in my case it wouldn't work... We're calling modules from gitlab using https

visual olive
#

yeah it wouldn't work. But you can distribute a single .env that everyone adds eg. to their home directory

#

the nice thing is you don't need to have secret values in there. only secret references

ancient helm
#

That's ok, we have a gitlab pipeline template that all the services include in their pipelines, it'll go in there. I've written a generator function for users running tests on their laptops to get a copy of the file

visual olive
#

for example to match:

func (m *MyModule) Build(source *dagger.Directory)

you can set MYMODULE_BUILD_SOURCE=...

or if the .env is inside the module: BUILD_SOURCE=...

ancient helm
#

So it does need a constructor, the attributes on the module struct aren't sufficient alone

#

e.g.

type ExampleModule struct {
    ExampleAttribute string
}

Won't pick up an env file with EXAMPLE_MODULE_EXAMPLE_ATTRIBUTE=hello unless the module also has a constructor?

visual olive
daring sandal
#

Thanks for the clarifying questions @ancient helm . All the above questions are great for documentation (maybe FAQ style?).

visual olive
daring sandal
#

I have a question of my own. How does this work with --blueprint? Is a blueprint considered a remote module? Or a local module for .env consumption?

visual olive
#

when explicitly naming the module in your .env variable, you may use either the blueprint or downstream module's name

daring sandal
#

Very cool! So I can have a .env that blueprint users get by default but is able to override themselves. sweet!

visual olive
ancient helm
#

With spinning some stuff out to a followup is this ready to go in 0.19.1?

visual olive
#

Quick update: Inching towards merge.

  • Had to disable support for system variable lookup, temporarily, because of a caching issue (cc @fresh moth @agile quail). Will open a follow-up PR to re-enable it, but this helps us get something merged faster
  • Had to disable --env-file flag, because it fails in nested clients (dagger-in-dagger) and I have no idea why. Will also open a follow-up PR once I get reinforcements.
  • I have one failing test where I need help.

--> Then we're good to merge.

ancient helm
#

Without --env-file Dagger looks for specifically .env by default? If so none of the above impacts me, I just want muh feature

visual olive
solar bough
#

I had a use case for the system vars, but having this feature sooner will be massive!!! env system vars can definitely wait 🥁 🥁 🥁 🚀 🚀 🚀

visual olive
pearl ether
visual olive
daring sandal
#

almost at liftoff!

visual olive
#

user defaults

#

@agile quail thanks for the ✅

Just double-checking, on the caching front, you're not concerned with the current implementation? eg. potentially having multiple instances of the same modulesource not sharing enough cache hits, because they're loading different .envs - or other scenarios like that?

agile quail
#

And refining cache hits for modules that use .env is much better for a followup. It’ll be much easier to implement in isolation rather than mixed in with the baseline implementation

visual olive
agile quail
visual olive
#

I don't understand, it was all green and all I did was fix one last lint error (unused function)

#

re-running on the odd chance it's a flake

agile quail
visual olive
visual olive
#

Aaaand released in 0.19.1

#

@ancient helm @solar bough @daring sandal @pearl ether

ancient helm
#

I'll update a pair of modules tomorrow 💯

visual olive
#

FYI you should be able to upgrade your CLI and nothing else. No module upgrade necessary

#

(CLI + engine)

ancient helm
#

I plan to refactor a bit to take advantage of user defaults a bit better, tbd on how effective this might be

visual olive
#

Let us know how it goes 🙂

#

Remember you can set user defaults on any argument to any type & function, not just module constructors

daring sandal
#

Initial feedback: Absolutely love it!

Not a huge fan of how the defaults are displayed persistently in the TUI. It also bleeds into the final output. Maybe a hotkey to show/hide them? Hide by default?

#
  1. User defaults that are not relevant to the currently executing function are also shown in the TUI. Eg: I have funcA and funcB and they arent related. I have defaults in .env for funcA and funcB both but when I run funcB or funcA, all the user defaults are shown. Would it be possible to show only the relevant ones?
visual olive
#

We can add a hotkey. I was worried about giving people clear feedback, otherwise your defaults can easily be silently dropped, or silently applied. Can be frustrating to iterate.

visual olive
daring sandal
visual olive
#

This is the magic @soft aspen gave me 😛 But yeah we'll polish. Now that it's released it will get more real-world usage, so polish will be easier

pearl ether
#

Amazing work Solomon! Will give this a test

daring sandal
#

I think the current way actually reduces the screen real estate the TUI has (because it's bottom-up). I can see it being very crowded if there are a bunch of defaults.

visual olive
#

True

soft aspen
#

Maybe it could go in the sidebar? Currently that's only used for LLM stuff, and disappears on exit, not sure what's desired there (sounds like maybe that's ok?)

visual olive
#

Was wondering about that also

soft aspen
#

an API for adding sidebar content (or if API is too bold, can at least add attachables)

ancient helm
#

Ok testing this now. Example, for a module named MyModule and a function named MyFunction with a parameter named GiveMe the env file should contain MY_MODULE_MY_FUNCTION_GIVE_ME=env:GIVE_VALUE? I'm getting required flag(s) "give-me" not set

#

I'm using dagger call -m <module> my-function <other-function>

visual olive
#

So for example you can do: MyModule_MyFunction_GiveMe=env:GIVE_VALUE

#

(assuming this is a function on the main module type)

#

if the .env is in the module's root directory, then you can omit the module name: MyFunction_GiveMe=env:GIVE_VALUE

ancient helm
#

It is a function on the main module type. I'll try that

visual olive
ancient helm
#

Function names also omit the _? What about parameter names

ancient helm
#

MYMODULE_MYFUNCTION_GIVEME=env:GIVE_VALUE still results in required flag(s) not set

visual olive
#

(sorry if this is confusing)

#

do you mind showing a relevant snippet of the module code?

ancient helm
#

Main module type: type ServiceDeployer struct {; function name func (m *ServiceDeployer) WithCredentials(; parameter name: AccessKey *dagger.Secret,

#

Calling it with dagger -m <module> call with-credentials <but not providing params here> <other-function>

visual olive
#

So ServiceDeployer_WithCredentials_AccessKey=env://something ?

#

oh it's env:// not env:

#

(although I believe we do have backwards-compat for the env: format)

ancient helm
#

ServiceDeployer_WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID results in required flag(s) not set

#

Is it possible there's a clash with AccessKey which is also an attribute on the main type ServiceDeployer albeit +private

#

Wouldn't have thought so given the function name is in the key

visual olive
#

Conflict shouldn't be an issue, I think

#

Where is your .env relative to the dagger.json?

ancient helm
#

In the directory where the dagger call -m <git/path/to/module> with-credentials <other-function> is being called

visual olive
#

ah I see the module itself is loaded remotely. Mmm that should work

soft aspen
#

fwiw it also took me ages to figure out the conversion scheme; my assumption was everything got SHOUTY_SNAKE_CASEd so I was trying things like DAGGER_DEV_GITHUB_TOKEN or GITHUB_TOKEN for a githubToken constructor arg to our module. there's no way I would have guessed DaggerDev_GithubToken which seems to be what it wants. or maybe it's case-insensitive and I would have eventually got there? but GITHUBTOKEN feels weird

visual olive
#

Yeah I am trying to find a way to flatten that learning curve...

#

Maybe a command to generate a template .env with the legal keys commented out

soft aspen
#

yeah, it's tricky because you don't know what it looked for. the confirmation that it found it is there, at least

visual olive
#

Generally I have been struggling with the name conversions across graphql, CLI, SDK-native etc. Had to hack my own conversions

#

I thought I remembered there are helpers in the code but I couldn't find them

soft aspen
#

also noticed it prints the value for defaults, which could be bad if someone doesn't use env:// or op:// or something for a secret. haven't confirmed if there's a safeguard for non-URI-looking values (too scared 🙂)

visual olive
#

Oh you mean if the user does that by mistake

ancient helm
soft aspen
soft aspen
#

i'm not sure if CLI flags accept direct values for secrets

soft aspen
ancient helm
#

So your .env is inside the module for that to work

visual olive
ancient helm
#

I've removed the +private from the main module type attributes and updated the .env to ServiceDeployer_AccessKey but still no luck there

visual olive
ancient helm
#

"name": "service-deployer",

visual olive
#

Maybe to isolate problems, try a local checkout, with a .env inside that omits the module name? To see if we can get it to work that way first

ancient helm
#

Is there anything in a cloud trace I should look for to be certain the .env is picked up?

ancient helm
#

And inside that .env: ServiceDeployer_WithCredentials_AccessKey format for keys?

visual olive
# 
git clone <YOUR_MODULE>
cd ./your_module
echo 'WithCredentials_AccessKey=env://foo' > .env
dagger call with-credentials
ancient helm
#

Yeah that worked immediately

#

Shows the defaults in terminal

visual olive
#

Ok great, so we know the problem is either 1) the module name prefix, or 2) a bug with loading remote modules

ancient helm
#

Extending that test to copying that .env to a service repository and calling dagger -m ../service-deployer with-credentials works too, even without adding the module name to the .env keys

visual olive
#

to eliminate the first option, keep the exact same setup but add the module prefix to .env?

visual olive
ancient helm
#

Adding ServiceDeployer still works

#

(When calling the module from local with dagger -m ../service-deployer with-credentials)

visual olive
#

Can you keep the same workdir & .env that works, and just change the value of -m to point to the remote module?

ancient helm
#

But if I remove the .env from the module (so it's only in the working dir calling it) it no longer works

ancient helm
visual olive
ancient helm
#

No, it's another dir on the same level

#

~/Projects/service-deployer (module), ~/Projects/billing-service (one of many services)

visual olive
ancient helm
#

Yes

#

cp ../service-deployer/.env .

#

.env files in the working dir don't seem to be picked up at all, for local or remote module sources

visual olive
#

If you want to know which .env files actually get loaded, you can call dagger -c '.core | module-source ../service-deployer | user-defaults | as-file | contents'

EDIT: forgot .core

ancient helm
#

Furthermore, .env inside the module dir with module name prefix also doesn't work, it must be without the module name prefix

visual olive
ancient helm
#

Or maybe it's my module name...

ancient helm
#

So Dagger sees the user defaults, but doesn't use them... Is this my module name somehow?

visual olive
#

The command above only confirms which .env files get loaded. It doesn't show which are matched (but I should add functions for that, will make debugging easier)

ancient helm
#

Doesn't work:

dagger -c '.core | module-source ../service-deployer | user-defaults | as-file | contents'
▶ connect 0.3s
▶ detect module: . 0.3s
▶ load module: /Users/user.name/Projects/service-deployer 2.5s

✔ moduleSource(refString: "../service-deployer"): ModuleSource! 0.0s ✔ 1
✔ .userDefaults: EnvFile! 0.0s
$ .asFile: File! 0.0s CACHED
▶ .contents: String! 0.0s

ServiceDeployer_WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID
ServiceDeployer_WithCredentials_SecretKey=env://AWS_SECRET_ACCESS_KEY
ServiceDeployer_WithCredentials_SessionToken=env://AWS_SESSION_TOKEN

Setup tracing at https://dagger.cloud/traces/setup. To hide set DAGGER_NO_NAG=1
dagger call with-credentials
▶ connect 0.4s
▶ load module: . 0.5s
✘ parsing command line arguments 0.0s ERROR
! required flag(s) "access-key", "secret-key", "session-token" not set

Setup tracing at https://dagger.cloud/traces/setup. To hide set DAGGER_NO_NAG=1
#

Works:

dagger -c '.core | module-source ../service-deployer | user-defaults | as-file | contents'
▶ connect 0.2s
▶ detect module: . 0.3s
▶ load module: /Users/user.name/Projects/service-deployer 0.3s

✔ moduleSource(refString: "../service-deployer"): ModuleSource! 0.0s ✔ 1
✔ .userDefaults: EnvFile! 0.0s
$ .asFile: File! 0.0s CACHED
▶ .contents: String! 0.0s

user default: service-deployer.withCredentials(secretKey="env://AWS_SECRET_ACCESS_KEY")
user default: service-deployer.withCredentials(sessionToken="env://AWS_SESSION_TOKEN")
user default: service-deployer.withCredentials(accessKey="env://AWS_ACCESS_KEY_ID")
WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID
WithCredentials_SecretKey=env://AWS_SECRET_ACCESS_KEY
WithCredentials_SessionToken=env://AWS_SESSION_TOKEN

Setup tracing at https://dagger.cloud/traces/setup. To hide set DAGGER_NO_NAG=1
dagger call with-credentials
▶ connect 0.3s
▶ load module: . 0.5s
✔ parsing command line arguments 0.0s

✔ serviceDeployer: ServiceDeployer! 0.0s
▶ .withCredentials(
  ┆ accessKey: Address.secret: Secret!
  ┆ secretKey: Address.secret: Secret!
  ┆ sessionToken: Address.secret: Secret!
  ): ServiceDeployer! 2.4s CACHED

user default: service-deployer.withCredentials(accessKey="env://AWS_ACCESS_KEY_ID")
user default: service-deployer.withCredentials(secretKey="env://AWS_SECRET_ACCESS_KEY")
user default: service-deployer.withCredentials(sessionToken="env://AWS_SESSION_TOKEN")
ServiceDeployer@xxh3:db308fc01898b2b5
#

If within a module dir the module name prefix is optional then my module name is a problem because that seems to prevent matching the defaults to args

visual olive
#

I'm sorry that this is not easier to setup

ancient helm
#

I'm going to change the module name from service-deployer to servicedeployer on the tiny chance this is a problem

visual olive
#

Maybe try service_deployer as the module prefix in .env?

ancient helm
#

So I renamed the module in dagger.json, and all references of it, to remove the - and now:

cat .env
Servicedeployer_WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID
Servicedeployer_WithCredentials_SecretKey=env://AWS_SECRET_ACCESS_KEY
Servicedeployer_WithCredentials_SessionToken=env://AWS_SESSION_TOKEN

Works inside the module dir with the module name prefix

#

Though it does seem to duplicate the output:

user default: servicedeployer.withCredentials(secretKey="env://AWS_SECRET_ACCESS_KEY")
user default: servicedeployer.withCredentials(sessionToken="env://AWS_SESSION_TOKEN")
user default: servicedeployer.withCredentials(accessKey="env://AWS_ACCESS_KEY_ID")
Servicedeployer_WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID
Servicedeployer_WithCredentials_SecretKey=env://AWS_SECRET_ACCESS_KEY
Servicedeployer_WithCredentials_SessionToken=env://AWS_SESSION_TOKEN
WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID
WithCredentials_SecretKey=env://AWS_SECRET_ACCESS_KEY
WithCredentials_SessionToken=env://AWS_SESSION_TOKEN
WithCredentials_AccessKey=env://AWS_ACCESS_KEY_ID
WithCredentials_SecretKey=env://AWS_SECRET_ACCESS_KEY
WithCredentials_SessionToken=env://AWS_SESSION_TOKEN
#

So module name with - is at least one minor problem

#

Ok progress. Calling dagger -m ../service-deployer with-credentials with a .env (as above) in the working dir but not the module dir now works

#

Aaaand calling dagger -m <git/path/to/that/same/module> with-credentials with the .env in the working dir (but not the module dir in git) also works

#

So lesson is: don't have - in your module name

visual olive
#

Thank you for debugging this... Adding to the follow-up list

ancient helm
#

FWIW having the defaults be blatantly obvious in the terminal view is a good thing

#

Sure it's a lot of screen space, but if they're not incredibly obvious it's going to cause headaches eventually

visual olive
#

OK it's very late here... really must go to bed. Thanks for being patient

ancient helm
#

Hah, I thought you'd already slept and woken up at 4am! Thanks for working through this!

visual olive
#

My kids keep me busy in the late afternoon & evening, so it's hard to resist getting a few more hours of quiet work time during the night 🙂

ancient helm
#

Another potential issue with default args for Directory types:

argument "lcPath" must be of type Directory to apply ignore pattern ([<patterns_here>]) but type is dagql.Optional[github.com/dagger/dagger/dagql.IDType]{Value:dagql.ID[*github.com/dagger/dagger/core.Directory]{id:(*call.ID)(0x40032e1900),
  inner:(*core.Directory)(nil)}, Valid:true}

If I provide the path by CLI arg it works, if I provide it by default arg it seems to think it's an optional and doesn't

visual olive
#

Ah, looks like an issue with .env combined with +ignore

ancient helm
#

Indeed. Removed the +ignore for that path and that one worked, the next default dir (which also has a +ignore) didn't

#

Exactly the same error, so that's almost certainly a +ignore issue

solar bough
solar bough
visual olive
daring sandal
visual olive
solar bough
#

👋 I'm struggling to pass a []string through the localDefaults...

#

Is there any reference to the syntax that's expected? I keep getting that the input is not a valid JSON. Is it supported?

#

(I've tried many combinations of a "json" single value slice (which will pass the json.Valid() but ... no luck

visual olive
#

@solar bough can you share .env contents you've tried?

solar bough
#

The relevant line:

SimpleGoContainer_Deploy_Workloads=plat-usc1/jenkins/freeze-exception-bot.yaml

I've tried many iterations

SimpleGoContainer_Deploy_Workloads=[ plat-usc1/jenkins/freeze-exception-bot.yaml ]
...
SimpleGoContainer_Deploy_Workloads=[ "plat-usc1/jenkins/freeze-exception-bot.yaml" ]
...
#

and quoting, escaping, etc... without much luck so far. I reckon that this means it is supported, I just haven't found the right syntax yet 😄

visual olive
#

It's a bash syntax parser (same as dagger shell) with a special exception made for FOO=BAR BAZ (which is interpreted as FOO="BAR BAZ".

Quoting works the same as shell. SO in doubt you can wrap the value in single quotes. Can you try:

SimpleGoContainer_Deploy_Workloads='["plat-usc1/jenkins/freeze-exception-bot.yaml"]'
#

It's possible that said exception interferes with SimpleGoContainer_Deploy_Workloads=[ "plat-usc1/jenkins/freeze-exception-bot.yaml" ]

(There can be ambiguity between "interpret this as a regular shell expression" and "interpret this as a simplified dotenv assignment")

sonic timber
#

Having a - in the module path, not just in the module name causing issues. When checking the users-defaults with dagger -c '.core|module-source ./dagger-modules/custodian | user-defaults | as-file | contents' it shows the contents of my .env file as CUSTODIAN_ARGS=something. When calling the module with dagger -m dagger-modules/custodian call list-cache the args constructor argument is not loaded and the CLI doesn't show the user default value. It works if calling from the same directory, dagger call list-cache will load the user default and the CLI will show it.

#

Removing the module name from the .env file and keep it like ARGS=something and it will be loaded from anywhere.

visual olive
#

@solar bough update, I confirmed that you have to quote-protect your JSON values, just like in a shell script... Otherwise the JSON quotes get interpreted away. I realize that this might deviate from expected behavior of a regular .env file... The downside of using a bash parser. For now I added tests to track that behavior.

ancient helm
#

Don't suppose you've looked into the pragmas on directories bug yet? That's the most impactful for us currently, they're on basically every directory input we have

visual olive
visual olive
#

@ancient helm I'm having trouble reproducing the issue with +ignore..

#

let me push the test

ancient helm
#

You have a working test?

#

Directory argument (with +ignore) to function, provided by .env

visual olive
# 
dagger call -m github.com/shykes/dagger@user-defaults-fix test specific --run=TestUserDefaults/TestDirectoryIgnore
ancient helm
#

Give me five, I'll get a minimum repro from one of our modules

#

Ok, got a repro

visual olive
#

Can you take a look at my test, is it testing the wrong thing?

ancient helm
# 
package main

import (
    "dagger/reprodaggerdirdefault/internal/dagger"
)

type Reprodaggerdirdefault struct{}

func (m *Reprodaggerdirdefault) AddDir(dir *dagger.Directory) *dagger.Container {
    return dag.Container().From("alpine:latest").WithDirectory("/input", dir)
}

func (m *Reprodaggerdirdefault) AddFilteredDir(
    // +ignore=["**", "!goodbye.txt"]
    dir *dagger.Directory,
) *dagger.Container {
    return dag.Container().From("alpine:latest").WithDirectory("/input", dir)
}

From another directory, with the following files:

.env:

Reprodaggerdirdefault_AddDir_Dir=.
Reprodaggerdirdefault_AddFilteredDir_Dir=.

hello.txt: hello

goodbye.txt: goodbye

Run:

(No pragma, unfiltered, works): dagger -m ../reprodaggerdirdefault/ call add-dir directory --path /input entries

dagger -m ../reprodaggerdirdefault/ call add-dir directory --path /input entries
▶ load module: ../reprodaggerdirdefault/ 2.8s
✔ parsing command line arguments 0.0s

✔ reprodaggerdirdefault: Reprodaggerdirdefault! 0.0s
▶ .addDir(
  ┆ dir: Address.directory: Directory!
  ): Container! 0.9s
✔ .directory(path: "/input"): Directory! 0.1s
✔ .entries: [String!]! 0.0s

user default: reprodaggerdirdefault.addDir(dir=".")
.env
goodbye.txt
hello.txt

(Pragma, filtered, fails with error I initially posted): dagger -m ../reprodaggerdirdefault/ call add-filtered-dir directory --path /input entries

▶ load module: ../reprodaggerdirdefault/ 0.5s
✔ parsing command line arguments 0.0s

✔ reprodaggerdirdefault: Reprodaggerdirdefault! 0.0s
✘ .addFilteredDir(
  ┆ dir: Address.directory: Directory!
  ): Container! 0.0s ERROR
! set call inputs: apply ignore pattern on arg "dir": argument "dir" must be of type Directory to apply ignore pattern ([**, !goodbye.txt]) but type is
  dagql.Optional[github.com/dagger/dagger/dagql.IDType]{Value:dagql.ID[*github.com/dagger/dagger/core.Directory]{id:(*call.ID)(0x400affa480),
  inner:(*core.Directory)(nil)}, Valid:true}
ancient helm
#

Perhaps the repro above helps spot the cause

visual olive
#

Ah, you're right I'll add that in the test

#

Could be specific to 1) non-constructor argument, 2) explicit module name

#

Ah! or 3) +ignore on a required argument

ancient helm
#

I can quickly test 1

#

Or not, I seem to be unable to get constructor args working with .env at all

# 
package main

import (
    "dagger/reprodaggerdirdefault/internal/dagger"
)

type Reprodaggerdirdefault struct {
    DirArg *dagger.Directory
}

func (m *Reprodaggerdirdefault) New(
    // +ignore=["**", "!goodbye.txt"]
    dirArg *dagger.Directory,
) *Reprodaggerdirdefault {
    return &Reprodaggerdirdefault{
        DirArg: dirArg,
    }
}

.env:

Reprodaggerdirdefault_DirArg=.

This should work no?

#

Apparently Reprodaggerdirdefault_New_DirArg=. is picked up as a default, but doesn't actually apply the value? 🤷‍♂️

# 
▼ .addFilteredConstructorDir: Container! 0.2s ERROR
┃ panic: unexpected nil pointer for argument "source"                                                                                                         
! process "/runtime" did not complete successfully: exit code: 2

user default: reprodaggerdirdefault.new(dirArg=".")                                                                             
user default: reprodaggerdirdefault.addDir(dir=".")                                                                            
user default: reprodaggerdirdefault.addFilteredDir(dir=".")
#

Nevermind, New shouldn't be a method on Repodaggerdirdefault. Re-testing...

#

Same error on constructor arg @visual olive:

Module:

package main

import (
    "dagger/reprodaggerdirdefault/internal/dagger"
)

type Reprodaggerdirdefault struct {
    DirArg *dagger.Directory
}

func New(
    // +ignore=["**", "!goodbye.txt"]
    DirArg *dagger.Directory,
) *Reprodaggerdirdefault {
    return &Reprodaggerdirdefault{
        DirArg: DirArg,
    }
}

func (m *Reprodaggerdirdefault) AddFilteredConstructorDir() *dagger.Container {
    return dag.Container().From("alpine:latest").WithDirectory("/input", m.DirArg)
}

.env (in another dir):

Reprodaggerdirdefault_DirArg=.

dagger -m ../reprodaggerdirdefault/ call --dir-arg . add-filtered-constructor-dir directory --path /input entries
▶ connect 0.4s
▶ load module: ../reprodaggerdirdefault/ 0.5s
✔ parsing command line arguments 0.0s

✘ reprodaggerdirdefault(
  ┆ dirArg: Address.directory: Directory!
  ): Reprodaggerdirdefault! 0.0s ERROR
! set call inputs: apply ignore pattern on arg "dirArg": argument "DirArg" must be of type Directory to apply ignore pattern ([**, !goodbye.txt]) but
  type is dagql.DynamicOptional{Elem:core.DynamicID{typeName:"Directory", id:(*call.ID)(nil)}, Value:core.DynamicID{typeName:"Directory",
  id:(*call.ID)(0x401502c900)}, Valid:true}
solar bough
visual olive
#

@ancient helm OK I now have a repro in the test suite (same PR). Confirming that it's the specific combination of user default + ignore + required argument

visual olive
ancient helm
visual olive
#

I think it will be easy yes (once I understand why my test is failing with an error message that doesn't exist anywhere in my dev source..)

#

OK almost there

visual olive
visual olive
#

I have a "fix" which makes the error go away, but the test still fails because the ignore behavior is wrong.

in case you're around @fresh moth or other <@&946480760016207902>

visual olive
#

Update: my fix seems to work with manual QA, but my test is still failing 🤔 . Hopefully it's just an issue with the test. Investigating

#

Sorry for the slow turnaround, the whole team is converging to California for our yearly team retreat... Everyone has been packing or traveling for the last several days

#

...it was indeed my test 😬

#

@ancient helm ok I have a fix

#

Will get it merged asap

ancient helm
#

Happy to hear it!

visual olive
solar bough
#

I'll create a repo tomorrow and share it in an issue. I only got it with the array type

visual olive
#

Ok i'll add a test for that. surprising that it would only triggered by one type... 🤔

visual olive
ancient helm
#

Tiny change! Will keep an eye on releases

visual olive
#

sorry it took so long

visual olive
#

@solar bough could you tell me if it fixes your issue?

solar bough
visual olive
#

We're planning a release this week btw

#

we'll make sure it's included

daring sandal
visual olive
visual olive
daring sandal
#

Very early testing. I have a module that has dashes (my-sdk-go). I am calling it remotely like dagger call -m my-sdk-go generate-scaffold-prompts. It has a --github-token requirement in the constructor. My .env is not being honored

#

I am on 0.19.3, but the module itself is on 0.19.2. Does the module also have to match the version?

daring sandal
#

The .env file is in my local. Which is basically an empty folder

visual olive
#

@daring sandal can you share the contents of the .env? It's fine to redact the values, I just need the exact var names

visual olive
daring sandal
#

Yeah one sec

visual olive
#

Maybe upgrade the module's engine field to 0.19.3 just in case? 🙂

#

I don't see how that could be the reason

daring sandal
#

GITHUBTOKEN=env://GITHUB_TOKEN

#

Updating the module's engine version didn't have any effect

visual olive
#

if the .env is outside the module root, you need to prefix it with the module name

#

ie. MY_SDK_GO_GITHUBTOKEN=env://GITHUB_TOKEN

#

or MySdkGo_GITHUBTOKEN= (will allow both)

#

this is assuming your constructor argument is called githubToken or githubtoken?

daring sandal
#

githubToken

visual olive
#

yeah arg name should be fine (note to self, also make matching more flexible on arg names, the same I did for module name)

daring sandal
#

I tried both MY_SDK_GO_GITHUBTOKEN and MYSDKGO_GITHUBTOKEN

visual olive
#

@daring sandal when you say dagger call -m my-sdk-go are you actually typing dagger call -m github.com/something/something/my-sdk-go?

daring sandal
#

it's a remote module

visual olive
#

what's the actual module name in dagger.json? that's the source of truth

daring sandal
#

Use case is, scaffolding an app for a user. They start with and empty dir

#

um, dagger.json is SdkGo

visual olive
#

if you want to scaffold, you could use dagger init --blueprint=my-sdk-go, then you can drop a .env in the root of that scaffolded app, with GITHUBTOKEN=env://...

visual olive
daring sandal
visual olive
#

Scaffold:

dagger init --blueprint=github.com/.../my-sdk-go
cat >>.env <<'EOF'
GITHUBTOKEN=env://GITHUB_TOKEN
EOF

In this case (.env at the root of the user's module) you can drop the SDK_GO_ prefix

daring sandal
#

So I should probably rename my module to match ther folder name

#

hmm, I don't think the end user will be using this dagger module. So I don't see a reason to add it as a blueprint

visual olive
#

OK. I guess I misunderstood what you meant by scaffolding

daring sandal
#

so I have a go framework. I am going to create a skeleton app for the user based on certain inputs (which isn't a dagger module).

#

I have a separate go dagger module the user may use later.. But it's not part of the scaffolding yet

#

the scaffolding code is in the go-framework is because I want to keep everything in the same place

#

It's still early design, I may re-think this delivery mechanism. It's great to be able to use dagger to do such a thing (with Changeset)

visual olive
#

Our next batch of features might be able to help 🙂 Demo soon

daring sandal
#

Can't wait!

pearl ether
#

Has there been any movement on fallback to system env ?

#

We are eagerly awaiting this feature 🙂

visual olive
#

@pearl ether sorry for the delay. My (naive) implementation was problematic so we had to roll it back. I think the underlying naïveté has been fixed, so we can give it another try

pearl ether
#

All good 🙂

#

Thanks for being on this

visual olive
#

@pearl ether we shipped another experimental feature called "toolchains". It's like blueprints but more modular - more like the original prototype where you could install multiple blueprint-like dependencies

pearl ether
#

yeah just been playing catch up. Already can see a few use cases for it

visual olive
#

@rose juniper 👋 this is where we've been discussing follow-up improvements to user defaults (aka .env suppport), including system env fallback

rose juniper
#

is there a plan to support loading an EnvFile into a container? e.g.

f := c.Host().File(".env").AsEnvFile()
c.Container().From(alpineImage).WithEnvVariablesFromEnvFile(f).WithExec([]{"env"})

or

myVars := c.Host().File(".env").Variables()
c.Container().From(alpineImage).WithEnvVariables(myVars).WithExec([]{"env"})
visual olive
rose juniper
#

and as a follow up question (more of an implementation question), I noticed that .AsEnvFile().AsFile() doesn't cause the variables values to get filled in -- is there a specific reason for this?

I think I have a work around, where I can just iterate over all the variables while calculating a custom digest -- but I'm thinking it might be a bit more efficient if .AsEnvFile() did the substitution at the same time as calulating the hash.

visual olive
# rose juniper and as a follow up question (more of an implementation question), I noticed that...

Open to API & implementation changes. But note that today you can also construct an env file from scratch envFile().withVariable("foo", "bar") etc. So would have to think carefully above how lazy or greedy we want expansion to be in different scenarios. eg. right now I can programmatically construct an env file and export it to the filesystem, would we no longer support that? maybe it's ok? I guess i can just join the strings myself, not exactly rocket science 🙂

visual olive
rose juniper
visual olive
pearl ether
#

very nice

visual olive
pearl ether
#

once this is released, it will be a game changer for us