#Authenticate to EKS from dagger function
stoic crystal
So I switched to using dagger functions. I'm trying to figure out how to authenticate to the k8s api from inside my function. I've tried the following
- Pass my kubeconfig in - this failed because kubeconfig for eks wants to execute the
awscli. I can see lots of examples of constructing a container withdag.Container()but how do I get my function's go code to execute inside that container? - Pass in my
~/.aws/credentialsfile to the function and trying to write them to~/.aws/credentialsbut the aws sdk isn't picking them up as valid. (which is probably user error on my part 🤷♂️ )
What's the right way to do this?
Here's the code I'm working with right now
func (m *Multi) CallKubeApi(ctx context.Context, awsConfig *File) {
awsContents, err := awsConfig.Contents(ctx)
if err != nil {
panic(err)
}
err = os.MkdirAll("~/.aws", 0644)
if err != nil {
panic(err)
}
err = os.WriteFile("~/.aws/credentials", []byte(awsContents), 0644)
if err != nil {
panic(err)
}
err = os.Setenv("AWS_SHARED_CREDENTIALS", "~/.aws/credentials")
if err != nil {
panic(err)
}
for _, e := range os.Environ() {
pair := strings.SplitN(e, "=", 2)
fmt.Println(pair[0], pair[1])
}
clusterName := "cullen-poc-cluster"
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatal(err)
}
client := eks.NewFromConfig(cfg)
fmt.Println("setting up eksSvc")
input := &eks.DescribeClusterInput{
Name: aws.String(clusterName),
}
result, err := client.DescribeCluster(ctx, input)
if err != nil {
fmt.Println("err?", err)
log.Fatalf("Error calling DescribeCluster: %v", err)
}
fmt.Println("describeCluster results", result)
clientset, err := newClientset(result.Cluster)
if err != nil {
log.Fatalf("Error creating clientset: %v", err)
}
fmt.Println("clientset", clientset)
nodes, err := clientset.CoreV1().Nodes().List(ctx, metav1.ListOptions{})
if err != nil {
log.Fatalf("Error getting EKS nodes: %v", err)
}
log.Printf("There are %d nodes associated with cluster %s", len(nodes.Items), clusterName)
}
newClientset does this:
func newClientset(cluster *eksTypes.Cluster) (*kubernetes.Clientset, error) {
log.Printf("%+v", cluster)
gen, err := token.NewGenerator(true, false)
if err != nil {
return nil, err
}
opts := &token.GetTokenOptions{
ClusterID: aws.StringValue(cluster.Name),
}
tok, err := gen.GetWithOptions(opts)
if err != nil {
return nil, err
}
ca, err := base64.StdEncoding.DecodeString(aws.StringValue(cluster.CertificateAuthority.Data))
if err != nil {
return nil, err
}
clientset, err := kubernetes.NewForConfig(
&rest.Config{
Host: aws.StringValue(cluster.Endpoint),
BearerToken: tok.Token,
TLSClientConfig: rest.TLSClientConfig{
CAData: ca,
},
},
)
if err != nil {
return nil, err
}
return clientset, nil
}
deft birch
Hey @stoic crystal!! I'll check out your example to see what might be missing.
Side note: I recently created a kubectl module that has wrappers for different cloud providers. For now I only added EKS because its the one i've been using. We are using this module internally at Dagger and its working fine! If you want to try it out, its this one right here: https://daggerverse.dev/mod/github.com/matipan/daggerverse/kubectl@81eaa2f85e75ed348ba99eaace27a95a37eb119c
This is an example of how we are using it internally. AWS credentials are tricky, there are many ways of authenticating and each one may require different things. There are ways to authenticate where env variables is enough, others where ~/.aws/credentials and an AWS_PROFILE is enough, and others were you need the ~/.aws/config, ~/.aws/credentials and AWS_PROFILE. In our case given we are using an SSO session with some custom role_arn and source_profile, we are using all three. Below is an example of how we initialize it:
func (m *Module) kubectl(kubeconfig *File) *KubectlCli {
opts := KubectlKubectlEksOpts{}
if m.AwsConfig != nil {
opts.AwsConfig = m.AwsConfig
}
return dag.Kubectl(kubeconfig).KubectlEks(m.AwsCreds, m.AwsProfile, opts)
}
The idea with this module's DX is to have a top level Kubectl function that users call initially. This receives the kubeconfig of the cluster and returns an instance of the module. You then have one top level function per cloud provider, for example KubectlEks will receive AWS related parameters and return a KubectlCLI with everything needed for authentication where you can call Exec and run kubectl commands
Not sure which version of EKS you are using. But the ones i've tested with requires aws-iam-authenticator installed in order to fetch the token that is then used to communicate with the k8s API
stoic crystal
Thanks! Yeah I was actually looking at that kubectl module yesterday but I wasn't sure if/how it would help me in this situation. I'm trying to avoid having to shell out to do what I want to here.
and yes I need the aws cli and (I think?) aws-iam-authenticator for my passed in kubeconfig to work with this EKS cluster. That why I'm curious to know if I can modify/customize the container that my function runs in? If I can do the equivalent of:
FROM ubuntu
RUN apt update && apt install aws aws-iam-authenticator
and then execute my function in that container this would work but I'm not sure if that's possible or a good idea 😅
Ah, another option might be to use this aws cli module https://daggerverse.dev/mod/github.com/ernesto27/daggerverse/aws-cli@5548d6a8ee5fc4736fb3a5a8971f4650acac5866 to run the aws eks get-token command?
deft birch
Okay I see what you are trying to do. I'm not entirely sure what the entire context is of the container being evaluated, you can definitely play around with the filesystem and change stuff, but I'm not sure if its possible to install packages. In fact I'm afraid I don't even know what makes the container where the function runs, maybe @wintry aspen knows?
In the meantime, something that would work is to create a custom container with Go and all other dependencies installed, mount the script that does your stuff in that container and then run the program. Dagger acts as the orchestrator that makes your program portable and able to run everywhere via a dagger call
stoic crystal
I was coming up with a contrived example to make sure I understood what you meant, but I noticed the With method on Container for the first time and I'm wondering now if that's the right way to do this?
func (m *Multi) ExecInContainer(ctx context.Context) {
dag.Container().From("debian").With(func(c *Container) *Container {
fmt.Println("in a container?")
return c
})
}
Maybe I'm misunderstanding what this does though?
https://pkg.go.dev/dagger.io/dagger#Container.With
Hmm looks like no
✔ connect 1.1s
✔ initialize 0.2s
✔ multi: Multi! 0.0s
✔ Multi.execInContainer: Void 0.4s
┃ in a container?
┃ lsb release NAME="Alpine Linux"
┃ ID=alpine
┃ VERSION_ID=3.18.6
┃ PRETTY_NAME="Alpine Linux v3.18"
┃ HOME_URL="https://alpinelinux.org/"
┃ BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
┃
✔ connect 0.1s
<nil>
I ran this function
func (m *Multi) ExecInContainer(ctx context.Context) {
dag.Container().From("debian").With(func(c *Container) *Container {
fmt.Println("in a container?")
lsb, err := os.ReadFile("/etc/os-release")
if err != nil {
panic(err)
}
fmt.Println("lsb release", string(lsb))
slog.Info(string(lsb))
return c
})
}
So, the function being executed there is not being executed inside that separate container. Its just a different way of modifying the underlying container, it helps when doing some custom logic that you don't want to break the chaining
For example:
dag.Container().From("debian").
WithEnvVariable("ALWAYS_SET", "VALUE").
With(func(c *Container) *Container {
if someCondition {
return c.WithEnvVariable("CONDITIONAL", "VALUE")
}
return c
}).
// continue doing some chaining
Ahh gotcha
Happy to direct this elsewhere if this is sufficiently off topic for this thread, but is what I'm trying to do (execute Go functions written in dagger all the way down instead of shelling out to some other program inside of whatever container I'm executing) not really how dagger is "intended" to be used? I'm wondering if I should rethink how I planned to use dagger 🙂
deft birch
Its a use case that I've definitely done in the past. But with simpler logic. For example: you have a bunch of e2e tests that make some http calls to your service and test stuff, you put that code in a Go lib, you import it from your module and run the tests internally. You could also do the alternative approach of creating a container and running a go run that runs the tests. However, I've never done needed to manipulate the underlying container, that is where I'm a bit lost. Talking to @wintry aspen just know, we tracked down where the underlying container comes from and it seems to be, for Go, an alpine based image. You should be able to do apk curl and then download the binary of aws-iam-authenticator and any other tools you may need, in theory
I'm not sure if you can trust 100% that we will not break/change this underlying image in the future. As far as I know this is not a part of the public API of SDKs so it could change in the future
wintry aspen
I'm not sure if you can trust 100% that we will not break/change this underlying...
yes, the runtime container is not supposed to be modified given that it's impossible to guarantee forward compatibility if you start installing things there
stoic crystal
Ok cool that makes total sense. I will avoid doing that then 😄
wintry aspen
Ok cool that makes total sense. I will avoid doing that then 😄
I'd use the aws eks get-token approach mentioned before and pass that to my Go code to authenticate
stoic crystal
Ok cool. I appreciate all the help, I think I have a better understanding of how to proceed now.
late sedge
Sorry, I haven't read through the entire conversation, so apologies if you already got the answer, but here is what I would do:
As far as I know, the client only needs aws (cli or authenticator) present when the temporary token stored in kube config is expired. If that's true, you could mount everything in a container that has kubectl, aws CLI, get a token, grab the updated kubeconfig file, load it in client-go, done.
The alternative is that you import the aws-iam-authenticator code in your project and skip the extra contianer (totally doable, I've done it once or twice)
delicate steppe
Thanks! Yeah I think my plan for right now is to use the existing AWS cli module to get the EKS token and then load that into my k8s api client.
late sedge
Sounds like a good start to me.
rich gale
Just came across this thread. The most straightforward solution I could come up with in a POC was simply mounting the ~/.aws directory in a container. I believe it surfaced an issue that entire directories could not be mounted as secrets, however.
I'm not sure if that's still true but was true as of a couple of months ago
late sedge
It is, but you can also mount env vars directly
deft birch
I'm very tempted to build a dagger module that implements what aws implements in the 'configure-aws-credentials' github action. The credential resolution chain of AWS is very extensive. Having a module that wraps that and simply returns a []*dagger.Secret would allow users to just add the list of secrets as env variables to the containers. But sometimes it is easier and more tempting to use ~/.aws, specially locally
rich gale
yes, one appealing notion about the simplicity of mounting ~/.aws is that client libs are usually smart enough to figure out what to do with it. Anything more specialized necessarily introduces complexity, since there's probably what, like 5 different ways to auth to aws (access key, sso, mfa, etc etc)
deft birch
100%. What I've been doing in cases where I just have the two env vars (access key and secret key for CI stored as secrets) is have the calling function build the .aws directory in a scratch container and then send that to whichever function is doing some aws stuff
late sedge
I'm very tempted to build a dagger module that implements what aws implements in...
That does sound interesting, although probably not enough in this case. Kubernetes auth is one more layer on top. Maybe it becomes simpler with the new Kube auth support? Dunno, but EKS auth is broken, always has been.