Running buildkitd as a Dagger service or...

[fringe dock]

Hey all,

I have a funny use case here. I'm evaluating Dagger as a possible replacement of some of our existing homegrown CI tooling including our existing BuildKit frontend gateway (https://gitlab.wikimedia.org/repos/releng/blubber). Naturally I decided to see if I could replace all of Blubber's CI config with Dagger.

Everything was very easily ported except for the acceptance tests which need a buildkitd available at $BUILDKIT_HOST. This begs the questions:

Is it possible to run buildkitd as a Dagger service? It would need either privileged mode (seems like a terrible idea to allow) or seccomp unconfined for rootless mode (somewhat less terrible but maybe still a bad idea in an untrusted environment).

Alternatively, are there any plans to allow calling out to other buildkit frontend gateways from Dagger? I've seen this in other BuildKit derived things like HLB.

Of course, if our Blubber tool were replaced by Dagger, this would all become a moot point, but I thought I'd inquire anyhow.

Thanks!
Dan

[rotund tusk]

Is it possible to run buildkitd as a Dagger service? It would need either privileged mode (seems like a terrible idea to allow) or seccomp unconfined for rootless mode (somewhat less terrible but maybe still a bad idea in an untrusted environment).

yes, you can do this via privileged as you mentioned by using the insecureRootCapabilities flag in the withExec call. ref: https://docs.dagger.io/api/reference/#Container-withExec

[fringe dock]

Oh, I had assumed that only applied to the buildtime process, not the entrypoint

[hexed horizon]

Here are quick examples from a module that Alex wrote. The definition of the container: https://github.com/vito/bass/blob/ddb30dd511bebfda157d91a72492fd879d2d58c3/ci/main.go#L69-L83

The actual exec part: https://github.com/vito/bass/blob/ddb30dd511bebfda157d91a72492fd879d2d58c3/ci/main.go#L58-L60 plus binding it as a service

[fringe dock]

I'm very new to Dagger

[rotund tusk]

Alternatively, are there any plans to allow calling out to other buildkit frontend gateways from Dagger? I've seen this in other BuildKit derived things like HLB.

this is not trivial since it requires a FE <> Dagger API translation. We currently do it for the Dockerfile frontend for example, https://github.com/marcosnils/dagger/blob/96c7db2c9e2ed7fbf5fe59240cd158e9c67b5355/core/container.go#L342 so if there's enough demand for other frontends, that's something we could evaluate but not currently a priority

[fringe dock]

Oh, ok! So (*Container).WithExec([]string{cmd}).AsService() gives me a service running cmd, perfect. I had assumed a service would be based on the configured entrypoint. This is perfect thank you

[fringe dock]

Ah, ok, that makes sense. Sounds like a lot of effort to bridge the BK gateway API. Thanks

seems like a terrible idea to allow
Sorry for putting it that way, BTW :/ (foot in mouth over here).

I am curious though, is there or will there be some way to limit these capabilities for folks that want to deploy their own engine?

[rotund tusk]

I am curious though, is there or will there be some way to limit these capabilities for folks that want to deploy their own engine?

yes, you can already configure this in your engine today by using buildkit's default config options. More info here: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#execution-requirements

[fringe dock]

Awesome

[mental skiff]

If you already have a bk frontend, and it's hosted on a public registry somewhere, it may not be a huge project to add support for custom frontends. I think it's just an extra argument (ref of the frontend image) in that llb call. Unfortunately you can't pass the frontend as llb (no dynamic frontends).

Does that make sense @rigid bear @faint otter ?

[fringe dock]

IIRC the direct LLB solve is a different client mode than a frontend solve. I'm referring to a gateway frontend solve, so yeah passing an accessible ref and a number of gateway options

I never found the direct LLB solve that useful to be honest, because you can't really implement higher-level control of the BK result, e.g. create multi-platform manifest lists, etc.

[fringe dock]

Thanks again! My acceptance tests just passed locally using a BK service, so I've officially ported the entire project's CI config to Dagger \o/ Very cool

[mental skiff]

nice! How's the performance? That nested bk doesn't share cache with dagger.

[fringe dock]

er, not great. The entire suite takes 25 min which is about what it takes in our GitLab runner cluster. These tests all pass no-cache so not much room for optimization

i should probably profile them at some point just to see what they're doing. They're heavy though, and exporting each result to tar, loading that into a memory based FS object, and then verifying filesystem contents

Oh, right. I'm running them in sets as a parallel matrix job in GitLab to speed them up. Maybe I could do the same in Dagger

[mental skiff]

@fringe dock may I suggest setting up Dagger Cloud, to visualize your pipelines 🙂 It includes runtime and caching info.

There's a free plan for individuals

Are you using a cache volume for the nested buildkit's state directory ? That might speed things up quite a bit

[fringe dock]

No cache volume atm. Just wanted to get this working as a proof of concept basically

I'll check out Dagger Cloud!

[mental skiff]

Fair warning, Dagger Cloud does not yet visualize Dagger Functions: only low-level Dagger API calls. Coming very soon

[rotund tusk]

if you're already using privileged execs, you could also try setting experimentalPrivilegedNesting instead of insecureRootCapabilities. The former will autoamtically mount the buildkit socket of the dagger engine in /run/buildkit/buildkitd.sock which I assume wil probably use a better use of the cache

[fringe dock]

oh wow, interesting. i'll try it out

[mental skiff]

really?

[fringe dock]

so i'm assuming in that case i would run the acceptance test container with that option and set BUILDKIT_HOST to that unix socket?

[rotund tusk]

yep, IIRC that was the case. Not sure if that changed recently

[mental skiff]

If you end up using that @fringe dock, that's an undocumented API so might break in the future (no known plan to break it though). I don't want you to be unpleasantly surprised later

I get that this is all for a short-term POC, and may not matter in the long run anyway.

Generally speaking, if it has the word "buildkit" in it, it's a private API and may break in the future

[fringe dock]

well, lots to explore. i'm really impressed with Dagger Functions. I went on a little cuelang odyssey recently and worked on my own new frontend that i was hoping would achieve maximum reusability between modules but you all seem to have made something way better 😄

[fringe dock]

Got it, good to know

i'll bring my POC to my team (after i take a two month sabbatical and forget about work for a while) and go from there

[rotund tusk]

just verified this and the socket gets definitely mounted but in /.runner.sock instead of the path I've mentioned before

code is here: https://github.com/marcosnils/dagger/blob/96c7db2c9e2ed7fbf5fe59240cd158e9c67b5355/cmd/shim/main.go#L470