#config management is a good topic for DX

1 messages · Page 1 of 1 (latest)

lofty tiger
#

nice. Over here we built ours on pydantic's BaseSettings and created an extensible, injectible singleton. We just subclass that per-repo (like in my airbyte cloud repo, I have a CloudSettings that subclasses this object with application specific configuration)

https://github.com/airbytehq/aircmd/blob/main/aircmd/models/base.py#L26

I agree that it would also be useful if something like that came builtin

tidal lodge
#

sweet 👌

hearty rivet
#

We have some support for sharing and passing config throughout a project. I think it's at least partially overlapping with what you want, but let me know what's missing.

Basically, when you define a hierarchy of project commands, you are able to pass configuration from parent commands to descendants. Each command in the hierarchy can read/update the config and add its own to be passed down.

Unfortunately the best example I have off-hand is not the best possible, but hopefully with some imagination it makes sense. For our own CI, we have this top-level target that accepts a git repo/branch to run the rest of the targets (build/test/lint/etc.) against:

func CI(ctx dagger.Context, repo, branch string) (CITargets, error) {
    return CITargets{
        SrcDir: ctx.Client().Git(repo).Branch(branch).Tree(),
    }, nil
}

type CITargets struct {
    SrcDir *dagger.Directory
}

https://github.com/sipsma/dagger/blob/737afb18aee9a66b3aed693346a5c1235adf1dab/ci/main.go#L12-L20

SrcDir is then passed through to subcommands until it's finally used in "leaf commands" e.g.:

This is obviously very simplistic, but you could imagine there being a lot more fields than just SrcDir. The only restriction on the fields there are that they a json-serializable. So @tidal lodge I would think you could continue to use your config package types, but just include those as fields of the project commands that get passed down.

Is that the sort of thing you were looking for? Or were you looking for something even higher-level and opinionated?

tidal lodge
#

Most of the work I did was more related to loading/parsing/validating config files, env vars, and flags (lots of it handled by the koanf library). Once we obtain a complete filled-out struct with all the fields it's pretty simple from there. For project commands, I know there isn't a good way to get complex configurations into the entrypoints in the current release, so what you have there looks like a good start

lofty tiger
#

@tidal lodge do you build that when you invoke your code, before actually doing anything with Dagger, and pass it around using golang's inbuilt features? curious what patterns other people are using for this. One tricky thing I also haven't fully built a good story around yet is how to slice down my configuration for what gets passed into containers in a sane way. I adopted an include method for my massive settings object to subset which of them should go into containers:

        result = (
                    with_gradle(pipeline.client, ctx, settings, sources_to_include=files_from_host)
                    .with_mounted_cache("/root/.gradle", build_cache)
                    .with_directory("/root/.m2/repository", previous_result.data.directory("/root/.m2/repository"))
                    .with_(load_settings(settings, include="ENV_VAR_1)) # this will only load ENV_VAR_1 from the settings, not all of them
                    .with_env_variable("VERSION", "dev")
                    .with_exec(["./gradlew", "test", "--build-cache"])
            )

Passing the entire object is obviously not ideal if you don't need all those settings, because there could be something changing in there between runs that you don't need in that container that is gonna invalidate your cache

tidal lodge
#

in my implementation most things are loaded before connecting to dagger, except for secrets which require the dagger client