Replacing makefiles & Docker files in large environment

[rustic slate]

Howdy!

I have a large number of repositories, and managing their updates and builds has become nearly impossible.
While all of them share the same core build process, and I use base images for consistency, not everything can be solved with them. Some repositories require specific tweaks to meet their own needs, and certain steps, such as mounting secrets, need to be performed downstream.

The biggest challenge arises when I need to update the core build steps—this requires modifying every single Makefile or Dockerfile across all repositories.

I’m wondering if Dagger could help mitigate this issue by centralizing the core build logic with Dagger functions. Ideally, each repository would only need to define the functions specific to its own needs, eliminating the need for Dockerfiles and those hard-to-read Makefiles.

How can one handle AWS authentication or authentication for other services in Dagger?
For example, one of the things my Makefiles currently manage is AWS login commands, so developers don’t need to worry about them. How would this be handled with Dagger? 🤔

I’ll admit I haven’t fully read the documentation yet—I’m more interested in hearing opinions on how others are handling similar cases and whether this is actually a good use case for Dagger.

These are my hopes for Dagger.
Thanks a lot for your time! 😊

[charred wasp]

I’m wondering if Dagger could help mitigate this issue by centralizing the core build logic with Dagger functions. Ideally, each repository would only need to define the functions specific to its own needs, eliminating the need for Dockerfiles and those hard-to-read Makefiles.

yes, that's one of the core aspects of the product. The fact that your build system gets to a stage where you need the same primitives to scale it as any other app. So modularizing your CI and extracting its core components into re-usable functions is one of the exact reasons why we've created Dagger in the first place.

How can one handle AWS authentication or authentication for other services in Dagger?
For example, one of the things my Makefiles currently manage is AWS login commands, so developers don’t need to worry about them. How would this be handled with Dagger? 🤔

Still undocumented but you should check https://github.com/dagger/dagger/pull/8730 and https://github.com/dagger/dagger/pull/9433.

TL;DR is that Dagger has first class support for secrets and secret providers. You'll have to flesh out the details on how to fetch them depending on your specific use-case but the core support is already in place

[rustic slate]

Alrigh! cool thanks for taking the time!! Will keep playing around with it 😄

[rustic slate]

@charred wasp What haven't been able to figure out ... is for example .. AWS credentials when using SSO login. Those are tipycally shortlived .. while on a ci env using oidc its fairly trivial to sort out, you just make the login on the workflow and then send the creds to the dagger environment, when using locally, for example we tend to abstract the "login" command with makefiles and add it to the builds .. for example ...make build would trigger a aws sessgion login, get the credentials to the ECR, and then make the build and push ... with dagger seems that its not possible to abastract the login and ecr portion of things, or am i getting it wrong? would still need makefiles to abastract those parts .. and really want to get rid of them 😅

Is there some sort way / function we can call that would execute actions on the host before executing on dagger engine? would be nice to have something similar, so we could do some actions and use those outputs on the builds

[charred wasp]

@fiery axle sorry for the ping. Do you know if there's a "recommended" path forward here which doesn't involve creating a custom AWS secret provider for this?

Not sure if an alternative could be done using the --secret-flag cmd:"$command" approach here. Seems to be that creating a simple AWS provider should be the way

[fiery axle]

i think you can do cmd:// with the new secret providers? at least while we don't have a builtin provider

i think you can just do your aws auth stuff on the cli using the cmd stuff as you mention

[charred wasp]

Yes, that's correct. I was just wondering if there was a better way instead of creating a shell script to get the creds via the CMD provider 😛

I mean .. you could call the AWS CLI within the pipeline as well but that will output some credential files that will be left in the cache.

[fiery axle]

yeah don't do that 😛

you could implement your own logic in a module, and have it return a SetSecret - but this approach will be deprecated and removed soon

[charred wasp]

Yes, that's what I was thinking also. Another reason for the ping

[fiery axle]

yeah, i think the answer right now would be to add a little shell script as a cmd provider

[charred wasp]

@fiery axle do you know if Secret providers run in the context of the client? I can't recall how the sandboxing / execution worked for those

i.e: if my provider needs to access some of my machine's envs and/or local files

[fiery axle]

yup, they're entirely in the client

so you can run anything host-y there

but there's no access to dagger there

(it's not like in a dagger run or anything like that)

[charred wasp]

👍 perfect, thx

@rustic slate so to sumamrize:

1. The "correct" approach should be to create an AWS SSO client provider for this use-case. TBH it shouldn't be that difficult but the caveat here is that currently secret providers live in the Dagger codebase and it might take a bit for your provider to become mainstream so you'll have to run a forked version of Dagger in the meantime.

2. Alternatively use the secret variables approach (https://docs.dagger.io/cookbook/#use-secret-variables) with the --secret-flag cmd:"$command" feature and build a tiny wrapper script to fetch the creds from there

[rustic slate]

Thanks a lot! 🤔 I think i'll rather wait tor the AWS SSO provider, and untill dagger gains more "maturity" in that sense may combine it with some earthly or something.

I am testing dagger on my personal projects but my goal is to eventually replace the Makefiles and Dockerfiles which have grown out of controll on my company. but I know for a fact that devs are even lazier than mee .. and making them recall that they have to add a --secret-flag cmd:"$command" or add an extra bash script ... its a big no no 😅 so from what i've learned so far .. I may be considering use earthly to centralize the CLI part and as dagger grows support for those things migrate them to native dagger.

I really hoppe I can have some time at some point to help you guys in some of this things am really enjoying exploring and playing with it!

Thanks a lot for the hard work

[charred wasp]

@rustic slate out of curiosity, does earhtly have a secure integration to authenticate with AWS? or will users have to rely on doing aws sso login before calling earhtly?

[charred wasp]

cc @maiden cypress

[rustic slate]

I have no clue ... I am thinking about aproaching in a far more simplistic way and use it just as a maekfile on steroids (if such a thing is even possible) and only benefit from the "DRY" functionality of it by having a Core earthfile that is used by all other repos to call the CLI part and then just pass the secrets to Dagger ... but ... Is just me thinking out loud, not even sure if such thing is possible. do I make sense? 😅

[charred wasp]

yes, totally, I get an idea about what you're trying to do but I'm not sure about the "introducing two tools" to your teams. Seems to me that you could shoot yourself in the foot by the amount of complexity that you're bringing in. Particularly when the two producuts you're introducing overlap in several aspects

If you're fully sold on the idea that Dagger's approach about evolving your company's factory towards a more modular programmatic app is the way, we'd be happy to help you out implementing the low-hanging fruits so you can show a better solution to your teams

[rustic slate]

Agree 100% with you there, and I am always the first one being against adding extra tools if there's one that can do it all, however in this case, the difference is that .. the teams don't actually maintain the Makefiles nor the Dockerfiles. concerns me more removing their simple make build and replacing it with a dagger call build --secret-flag cmd:"$command"

I think I am sold, except for this little edge cases, which I completely understand they are not yet! Earthfiles don't convince me, the Idea of having more complex Makefiles gets me nauseous! 😅

How do the providers work? I am up to my neck at the moment, but is there anything I can do to help with that?we are not going to adopt dagger anytime soon anyway, so i'm sure there will be some time 🙂

[charred wasp]

it's quite straightforward, you do something like dagger call --my-access-key "op://path/to/key" and then you implement an op provider for the desired scheme which is code that runs on the client side which has access to the client environment to do whatever you need to do

there's an example here: https://github.com/dagger/dagger/blob/main/engine/client/secretprovider/op.go

as you can see it's quite a few lines of code

and then it's registered here: https://github.com/dagger/dagger/blob/main/engine/client/secretprovider/secretprovider.go#L17

more info here @rustic slate https://discord.com/channels/707636530424053791/1331422139097808980

[charred wasp]

@rustic slate how do you handle the auth today with makefiles? Users have to manually call aws sso login before running make?

[rustic slate]

the aws sso login command is defined inside the makefile and executed in the build .. ex:

aws_login: 
    some inline script that handles the login with aws and login with ECR

build: aws_login
   docker build command ....

the login is mostly so they can authenticate and pull images from private ECR.

so in a nutshell ... it seems that I would still have to use some sort of magic ... to abstract the dagger call --my-access-key "op://path/to/key" my hope is that they would have only to do .. dagger call <function> The alternative would be combine dagger with makefile to abstract those flags, as I've seen suggested somewhere ... but I want to get rid of the makefiles as well with something more feature rich

[charred wasp]

if pulling the images from ECR is the main use-case then as long as they have valid credential in ~/.docker/config.json Dagger will use that automatically to pull the images

[rustic slate]

aw that's neat, makes it easier 🙂

[charred wasp]

so I'd assume that today they have to do some sort of docker login. If you're ok with keeping that docker login flow, then dagger call will just work

[rustic slate]

yeah that docker login is abstracted with the aws_login command on the make file which is in turn executed by the other commands.

[charred wasp]

the thing about secrets is that we've been very careful when designing the API as we wouldn't want someone to create a malicious module that could potentially get secrets from your system and send them everywhere else

that's why we do automatic configuration for some types of configs like ~/.docker/config and .gitconfig from the client's machine but we don't allow modules to arbitrarily access the user's machine

which makes a magical dagger call $my-func which automatically picks up everything not straightforward

[rustic slate]

Yeah I get it, there has to be a compromise somewhere 😄

I might explore a bit more, maybe there's some clever hack that can be pulled out 🤔 mounting the ~/.aws/ and authenticating with the aws cli container perhaps? 🤔 ... well ther'es plenty to explore 😄

[charred wasp]

if you want to pass ~/.aws/ to a module it needs to be explicit again, for security reasons. dagger call can't read anything that's ouside the module/app scope without passing any explicit flag arguments

one of the realizations I had as a dev working at dagger is the amount of unnecessary permisisons we're currently giving random tools to access your environment settings

i.e: what's preventing a random gh or kubectl plugin you install from the web from accessing all your credentials? well.. nothing actually...

[rustic slate]

eeeeh well ... then we start running into 2 different schools of thought here 😄

a) protect the users from themselves (what you are essentially trying to do)
b) Users should know and vet what they are using / loading into their machines / pipelines / code

but yest you are somewhat right! depends on what risks you are willing to accept 🙂

[rustic slate]

Yeah Earthly has this LOCALLY keyword that executes actions on the host: https://docs.earthly.dev/docs/earthfile#locally

would be really nice if dagger had something similar 🙂

[rustic slate]

howdy again ... @charred wasp With Earthly shutting down / moving towards community fork, are you considering a equivalent LOCALLY to dagger?