environment variables

[magic root]

I'm looking to get in some environment variables in my TS module code, but I believe after reading some other threads, the access to the environment is kinda locked up with Dagger for security reasons.

I noticed a solution of having a shell script and a command to export the env variables to a file and then use that file to get them into the module to use.

Would I be correct in saying, the key is to basically wrap the Dagger CLI in scripting or code and use args in the commands to get the env variables in the environment, where the CLI is running, into the module?

Is there perhaps an easier way and am I completely off with what I read and understood?

[lament mauve]

@magic root there are some options here:

1. You can pass a .env file as you correctly mentioned and parse the file withing your Dagger code and set those env variables in your containers

2. You can directly pass the arguments via dagger call --foo env:FOO as a *dagger.Secret. This works really well when you only need to pass a few varibles.

3. An alternative of the above is to call dagger call --envs cmd:"env" and pass all the env variables and then parse them in your code. Take into account that this approach is OS specific as it depends on the env command for each platform

The main reason why there's no way to set variables that are automatically passed to your functions is mostly for security and consistency reasons. Having said that, we do have some issues in our roadmap here https://github.com/dagger/dagger/issues/6723

[fickle cloak]

Lately I've gone the path of #1 having a module that looks for .env or secrets.env files and creates var/secrets for that container. So far so good lol

[weary roost]

We really should have first class support for .env files, it wouldn't solve everything but would be a robust primitive for making it all easier

then maybe add default env variable in function arguments? but only if sandboxed to .env (no direct access to host env variables)

[magic root]

Yeah, the secrets will be environment variables created in the runner pod in k8s. I guess part of the pods init script can be to also create a .env file with all necessary environment variables and then call on the .env library to pull them into the module.

[weary roost]

where would the init script get the secrets?

[magic root]

They would be injected into the pod as env variables via k8s secrets.

[weary roost]

It would make sense to support a .env format where you can just say "TOKEN=env://TOKEN" in other words, pass-through from the host. Then the runner doesn't have to regenerate a whole file each time, it can be static and even checked into git

then you can also use other secret providers eg. TOKEN=vault://foo/bar and not be dependent on kub or CI runner for secrets.

Separately, there's an alternative design we're discussing, where the engine itself can persist the state of an object, and lets you reload it later to resume. These persisted objects would act as the environment. In that case you don't need to worry about carrying a .env file around, or wrapper tools for generating them. You would just use the Dagger API for the whole environment lifecycle

cc @thick mortar @mild wind @half jackal @iron sluice

[magic root]

@weary roost

where the engine itself can persist the state of an object, and lets you reload it later to resume.

This sounds intriguing. But, how would it help with the initial issue of bridging Dagger's environment wall?

and not be dependent on kub or CI runner for secrets

Our runner pod creation process (both for dev'ing and for CI) will be dependent on k8s secrets. In fact, we use sealed secrets and can't get away from that.

Our challenge: All devs have their own registry credentials. So, for a given repository with any dockerfile in it, there will also be a set of sealed secrets in the repository too for the devs involved in the development of and possibly even usage of that container (if a registry is private).

I'm leaning more and more towards just wrapping the Dagger CLI in our own CLI as the solution. It's something we should do anyway. The devs only need a sub-set of commands and we can simplify those Dagger commands for the devs too that way.

Hmm... 🤔

[weary roost]

wrappers never end well

A good real-world example here: #1332126539256430723 message

but, if we want to avoid wrappers, we need to provide more hooks

[magic root]

Hehe... if it is just about change, then I'm ok with that. In fact, most of this stuff is only POC for us currently. I say "we" and act like we are organized, but we have nothing concrete in place yet and the team is small. We are coming from much worse conditions, in fact. Hehehe.. Everything with CI is manual or dispersed in different services right now. Very ugly. 🙂 Also, there is a much bigger picture for what we are aiming for. This CI part with Dagger is comparatively small, but very significant. 🙂

As for the "devs just want to code" point and them wanting to "dig under the hood", for sure some (a select few) will be in the thick of it using Dagger closely to find ways to improve processes or even invent new ones. However, we need the abstraction for the application devs..... to.... yes....basically..... tie their hands to only certain practices. I can't explain more, but it definitely is necessary.

Edit: What I just wrote sounds like we have a huge team. The idea is, we are heading for that with what we wish to accomplish. All a dream now, but we are working on making it a reality.

I really appreciate the discussion. If it goes wrong with the CLI wrapping, I'll report back for sure. 😄