AWS lambda deployment

[random smelt]

That’s great! We would love to help in any way we can.

I have a few questions if you don’t mind..

1. You mention that you don’t have any CI/CD in place. Are you adopting CircleCI specifically to run Dagger, or did you already have CircleCI installed in your organization?

run the dagger pipelines in our hosted runners

This is referring to CircleCI runners, correct? Are they hosted by you, or by CircleCI?

[final pier]

Hey @random smelt!

You mention that you don’t have any CI/CD in place. Are you adopting CircleCI specifically to run Dagger, or did you already have CircleCI installed in your organization?
Our organization already uses CircleCI to deploy various applications but our infrastructure has yet to be automated. The team I work on helps maintain the applications pipeline so now we are working on automating infrastructure deployments.

This is referring to CircleCI runners, correct? Are they hosted by you, or by CircleCI?
Yes, we have a self hosted CircleCI runner deployed in our EKS cluster. We have a self hosted runner for each of our environments (every environment has it's own unique EKS cluster)

[random smelt]

Thanks for the extra details 🙏 May I ask what git server your CircleCI is connected to? Github, Gitlab, other?

[final pier]

not a problem 🙂 CircleCI is connected to GitHub

[pliant haven]

I'd be interested in seeing how you're going to use dagger to deploy lambdas, keep us updated if you're able @final pier

[final pier]

Couldn't work on this a whole lot today but I was able to get a simple pipeline to use terraform to build out a docker image

import anyio
import sys
import dagger

async def main():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:

        terraform_dir = client.host().directory("./terraform")
        docker_host = client.host().unix_socket("/var/run/docker.sock")

        builder = (
            client.container()
            .build(context=terraform_dir)
            .with_unix_socket("/var/run/docker.sock", docker_host)
            .with_mounted_directory("/src", terraform_dir)
            .with_workdir("/src")
        )

        terraform_init = (
            builder
            .exec(["terraform", "init"])
        )
        
        terraform_plan = (
            terraform_init
            .exec(["terraform", "plan"])
        )

        terraform_apply = await (
            terraform_plan
            .exec(["terraform", "apply", "-input=false", "-auto-approve"])
        ).exit_code()


anyio.run(main)

And here is the terraform code

terraform {
  required_version = ">= 1.1.1"
  required_providers {
    docker = {
      source = "kreuzwerker/docker"
      version = "~> 2.13.0"
    }
  }
}

provider "docker" {}

resource "docker_image" "nginx" {
  name         = "nginx:latest"
  keep_locally = false
}

resource "docker_container" "nginx" {
  image = docker_image.nginx.latest
  name  = "tutorial"
  ports {
    internal = 80
    external = 8000
  }
}

[random smelt]

@final pier perhaps a silly question: in this example do you need terraform at all? Couldn’t you just build the image with dagger directly?

[final pier]

We use terraform to build out our lambdas and all of the infra dependencies they need so I wanted to keep my POC in terraform

Next I’ll test a deployment of a lambda and update

[random smelt]

Are the lambdas deployed as docker images? I thought they were zips. I guess there are several available runtime targets?

[final pier]

They are deployed as docker images, makes things easier for us

[final pier]

Just to give a quick update, I was successful in getting Dagger to deploy our lambdas on my local machine. It was pretty easy getting dagger to run a simple Terraform workflow! Once I finalize a few things, I'll share a final working example for using Dagger to deploy a Lambda packaged as an image with Terraform.

Now I'm going to start working on setting up dagger to be ran by our CCI runners. Are there any best practices for managing Dagger scripts? For example, should I be placing my dagger script under .circleci directory in our repository to look something like .circleci/mypipeline.py? Not sure it really matters but I would like to follow all best practices from the start

[random smelt]

@final pier that's great!

Good question on best practices for CI integration. That's actually a big focus for us right now.

The answer is that right now, we don't have a clear best practice. A few pointers:

• In general, the smaller the CI-specific yaml config, the better (ideally it only specify how to run your dagger tool, and nothing else)
• In the future, that will probably mean collapsing all your dagger pipelines in a single CI job (for example if you have build, test, lint: instead of running them from 3 different CI jobs, you would run them all from the same CI job). However we don't recommend doing that yet because it will probably cause you to lose reporting information in your existing tools (Circle CI and Git forge). We are working on solving that 😇
• As for directory, it's probably better to put your dagger script in a CI-agnostic location, perhaps ci/ or .ci/ or something like that.. We may define a more "official" best practice very soon. Technically it will work fine inside .circleci, but since the dagger code is not specific to circleci, it feels weird - ie. if you want to run your pipelines locally.

[final pier]

Thanks for the input on best practices! I like the idea of placing the dagger scripts in an agnostic location. I'll start with placing the scripts in a ci directory and go from there.

I also have another random question regarding cloak. I watched your talk (https://www.youtube.com/watch?v=uteBbCcr_5E) on dagger and project cloak but can't seem to find reference of that project in the dagger docs. Is project cloak just now referred to as the dagger sdks or is that a super secret project 👀

[random smelt]

Yes, project cloak was the codename of the new multi-language release: Dagger Engine 0.3 with the GraphQL API, and all the new SDKs

So now it's just "Dagger" 🙂

Note that there are parts of that demo that we haven't shipped yet. Most notably, API extensions. They are still in development, no certain release date but we will most certainly ship them

[final pier]

@random smelt Just wanted to give another update. I was able to get some solid pipelines working for my use cases at work but can't share the code since I have some sensitive information in the code. But I did make a dirty repo to share that covers the meat n potatoes of how I am using dagger at work. Here is the link to the repo https://github.com/GonzalezAndrew/dagger-tf-example/tree/main

I did add some information to the readme to cover the base logic behind how I am running Terraform with dagger. There is a TON of improvements that need to be made but I think it is a good starting point for how I can use dagger and terraform

[random smelt]

Nice! thank you for sharing. Are you generally happy with the result? Any complaints or surprises along the way?

[final pier]

I'm extremely happy with my results using dagger! The power, flexibility and capability behind dagger is something that gets me excited.

I wouldn't say I have complaints but I felt like I was doing things wrong or I could have maybe optimized things a bit better. An example of something I wish I could have done was dynamically configuring a build container to be flexible regardless of my use case for a Terraform workflow. Not all my workflows need to have the build container connected to a docker socket so having some type of way to dynamically configure a build container that I could pass around would have been neat. I never explored trying to dynamically configure a build container for my pipelines but it would be something I will revisit when I have the time.

Also, having a best practice doc to reference would be super nice. I know I mentioned in my previous comment but I just wanted to reiterate that having something to reference when using dagger would be nice. A good example of a best practice doc that I used to reference a lot was the Jenkins best practice doc. When I would get stuck, I generally would reference the SDK docs or just scrub discord for any useful info.

I'm still not done exploring dagger and have a lot of ideas I want to explore. My ideal state of my POC would be to get a pipeline that could be use across all Terraform workflows and not just something that fits my needs in one specific repo

[pliant haven]

I've been using Dagger for a similar use-case - terraform pipelines that can run locally and in gitlab

Not even close to finished but the early signs are good. I can back up what you said about "feeling like I'm doing things wrong" - that's definitely present at times and stems from very few examples of the basics

https://docs.dagger.io/454108/python-ci this was a huge step in the right direction, but the guides page needs splitting into absolute basics, then comprehensive guides (like the existing multi-build pipeline etc)

Then, I'd probably suggest that the guides need generalising into one location rather than having to repeat guides for each SDK

Else you'll quickly find it becomes a chore to keep good documentation/guides and best practices

@final pier "An ugly thing I had to do was add the argument local which I use to help distinguish between when I'm running dagger in a CI/CD environment and on my local machine." This is something I'm going to avoid one way or another. I've made the decision that an environment should have a minimal set of requirements and it's on the user to provide those - aws creds, vault creds etc. This way local/gitlab/remote can handle that how they want and my pipelines don't have to care

[random smelt]

Thank you very much to both if you for all this feedback! cc @earnest hull @unique dawn @chrome pier @wild grove @earnest robin @fierce shard

https://tenor.com/view/batman-bat-signal-alfred-j-pennyworth-gif-9282342

[random smelt]

@broken token we were just talking about this yesterday 😁👆

And agreed @pliant haven that’s the ideal pattern if you can achieve it. I’m curious @final pier if that would be achievable in your case?

[random smelt]

100% agreed, we have prioritized this, @earnest hull @wild grove @earnest robin are working on “single-track” docs where eventually all guides are available seamlessly for all languages. Ideally with a stripe-style language toggle.

[final pier]

I love this idea of requiring users provide credentials when needed and idk why I never thought of that lol. I think I'll add this logic to my pipelines instead of adding the local argument. I originally wanted make creds/auth methods as some type of plugin that I could bake in when needed but the more I think about it, the more that seems like wasted time. I come from Jenkins land so I'm very used to relying on installing plugins or writing shared libs to fix my issues but your approach makes a lot more sense. Thanks for the pro tip @pliant haven

[random smelt]

FYI we are discussing adding pluggable secrets providers to Dagger, so that the same pipeline logic can access credentials from different sources, depending on where it's running, without requiring source code changes

In the absence of that, some pipeline devs have started writing their own provider interface, as you have considered @final pier . Sooner or later everyone will need such an interface, the only question is who should take responsibility for it - the platform or each individual dev. I personally think it should be the platform, to minimize fragmentation and save everyone tons of time