Running Dagger engine with containerd

[ripe turtle]

👋 Hi,

I'm experimenting with the feasibility of running CI/CD pipelines isolated from each other using Firecracker microVMs to provide more security in a multi-tenant scenario.

The idea is that when customer A runs a pipeline, it gets executed in an isolated environment - a microVM - which has its own kernel.

Therefore, I want to spin up a microVM with just containerd (not Docker) and run the Dagger engine as a binary (not as a container) using tcp in the microVM. Then, the customer would use the dagger client to target _EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://<MICROVM_IP_ADDR>:1234

Are there any instructions to run the Dagger engine as a binary instead of as a container? How could I target containerd as well?

[blissful thistle]

Hi Felipe! That’s a great use case, we’ll be happy to help.

We only distribute the engine as an OCI container image at the moment, because there are multiple binaries and config files involved - tightly coupled components with private undocumented interfaces. We may eventually have them all integrated into one binary, but it’s a lot of engineering work to do so, so it’s low priority.

The good news is that you don’t need to install docker, in your case you already have containerd, so I recommend running the dagger engine on your existing containerd runtime. It will add no runtime overhead, and will take advantage of the containerd tooling and instrumentation you are likely to use.

I don’t think we have instructions to install dagger engine on containerd, but we should 🙂 And are happy to help you here too

[ripe turtle]

Cool, thanks for the prompt reply, @blissful thistle.

So far I've created a Firecracker microVM, installed containerd, runc, CNI plugins, dnsname and built the Dagger engine and shim binaries using Go.

I tried running ./bin/engine --addr tcp://0.0.0.0:1234 --oci-worker=false --containerd-worker=true —debug but it seems the --containerd-worker flag is no found?

[woeful flint]

@ripe turtle the engine has an engine.toml (--config flag) file which is a wrapper of buildkit's config file (https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md) that allows you to specify that option. The flags that the engine CLI exposes are mostly flags useful to Dagger particularly

[reef badge]

We don't support the containerd worker right now. We have a custom shim between the engine and runc in order to enable a few different features (e.g. scrubbing secrets from stdout/stderr of containers), but that currently creates a hard dep on use of the runc worker. There's some ideas to maybe switch to the containerd worker in the future, but nothing concrete.

You can still run the engine inside a containerd container though, but it won't use containerd to create new containers. It'll just create nested containers with runc. As long as you run it with privileges and ensure that /var/lib/dagger is not an overlay mount (easiest is probably to just bind mount from the host to there), it should work totally fine though.

Another option is to use nerdctl, which is compatible with the docker cli enough that if you alias nerdctl->docker then the engine will pick it up and use that. At least, last time a user tried it they said it worked, but that was a few months ago and it's not something we officially test for, so can't be 100% sure it's still working.

[woeful flint]

Oops right, forgot about the custom shim 😬

[ripe turtle]

I figured out the existence of the shim when I tried running the engine binary (it was expecting the shim binary to exist).

Anyway, I went for the option of using nerdctl to run the Dagger engine but I can't get it working on the Firecracker microVM. Apparently, the Firecracker kernel config has nftables disabled. I tried using the legacy version of iptables as shown here but no luck: https://twitter.com/felipecruz/status/1658221237145948161

FATA[0006] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-05-15T21:15:16Z" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): running [/usr/sbin/iptables -t nat -C CNI-9769cefa468f22ae971b4e5f -d 10.4.0.2/24 -j ACCEPT -m comment --comment name: \"bridge\" id: \"default-7355743af453709e21bca71a3012fcaf2517dccf0ca047d110b130ed71081d78\" --wait]: exit status 2: iptables v1.8.7 (legacy): Couldn't load match comment':No such file or directory\n\nTry iptables -h' or 'iptables --help' for more information.\n"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/7355743af453709e21bca71a3012fcaf2517dccf0ca047d110b130ed71081d78/oci-hook.createRuntime.log: file already closed: unknown

[woeful flint]

👋 I just saw your tweet exchange with @iximiuz @ripe turtle . Let us know how the whole nftables thing goes. Are you exploring the cloud-hypervisor route by any chance?

[ripe turtle]

Hi @woeful flint, unfortunately I haven't made any progress on it so far. The rootfs that the Firecracker microVM uses doesn't contain the /lib/modules directory with the required modules to have iptables running. I haven't had a look at cloud-hypervisor yet as I'd like to get the Firecracker microVM working given how much time I've already dedicated.

[woeful flint]

would be awesome to get firecracker running

[reef badge]

If you haven't seen it already, firecracker has a guide on building your own kernel here: https://github.com/firecracker-microvm/firecracker/blob/main/docs/rootfs-and-kernel-setup.md#use-the-provided-recipe

Including a shortcut option where you use their own ./tools/devtool to build it. I guess it would be natural to use dagger to run that build too 🙂

[ripe turtle]

Hi @reef badge, yes I followed those steps to build my own kernel to include the CONFIG_BRIDGE and CONFIG_NETFILTER_XT_MATCH_ADDRTYPE modules that apparently are those needed for iptables to work (as per this GH issue: https://github.com/firecracker-microvm/firecracker/issues/3476#issuecomment-1453600978). However, I must be doing something wrong because after buildking the kernel with that specific config, I don't see any modules built into the kernel (/lib/modules is empty in the microVM filesystem)

CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
CONFIG_BRIDGE=y```

[ripe turtle]

Ok, I finally managed to get the modules built into the kernel. This is a screenshot from the Firecracker microVM. The next step is to try to run the Dagger engine as an OCI container image using nerdctl.

[ripe turtle]

Update: https://twitter.com/felipecruz/status/1659182743702437891

Felipe Cruz (@felipecruz)

It's been a hard morning but eventually managed to get the @dagger_io engine running as an OCI container image in a Firecracker microVM running containerd.

The most difficult part was to get a bunch of modules statically compiled into the kernel.

Twitter · May 18, 2023, 1:02 PM

[woeful flint]

that's awesome @ripe turtle. Would love to feature this work in an upcoming community call if you're up for it!. cc @misty bough

[ripe turtle]

Sure! I tried too many approaches before hitting the right solution, so first I'm putting all my thoughts in order in a blog post. Just to manage expectations, what I did is simply the bare minimum: having the engine running in a microVM. The next steps that I'd like to try is using "dagger run" to spin up the microVM dynamically and launch the engine so the pipeline runs in the microVM. But I'm still a bit far to achieve the full journey.

[misty bough]

Thanks for sharing @ripe turtle ! Your current stage is perfectly fine to share with the community, and then we can always have you come back with an update too 😉 I'll DM you.

[ripe turtle]

I'm writing a blog post and would like if you could confirm the following diagram represents accurately how Dagger runs containers. From @reef badge's message above, I understand that nerdctl will use containerd to spin up the Dagger engine container, and the engine will use runc to create the pipeline containers. Is that accurate?

[woeful flint]

yes, seems correct. You could also add a little box inside the Dagger Engine one which says buildkit if you want to go a bit more in depth. Since it's buildkit who ultimately spawns the containers

[reef badge]

Diagram looks good. To be totally clear on the engine and buildkitd, they are the same binary and process. The dagger engine has its own main func and imports buildkitd components as a library. Just wanted to make it clear buildkitd wasn't a child process or something like that

[blissful thistle]

Yeah, in this particular context I think adding a buildkit box might be confusing, for the reasons you explain

[ripe turtle]

I've just published it, hope you enjoy it!

https://www.felipecruz.es/exploring-firecracker-microvms-for-multi-tenant-dagger-ci-cd-pipelines/

[blissful thistle]

it’s a fantastic article, thank you!

[woeful flint]

awesome article Felipe! thx for sharing! can't wait to see the next one 😉