Native multi-arch builds with distributed engines

[rare steeple]

Starting a thread to not take up too much space on the main channel 🧵

Ahhh I'm kind of skeptical of using the cloud service for this (if I understand right) - I think my main objection is that the cache service goes from becoming a "speeds up my build with faster cache", to a "i absolutely need this for the functionality of my build" - that feels like it's got a lot of implications for uptime/etc, and feels like quite a different product at that point. I also think that there's some issues if we do this, it means there's now a lot more things being transferred through the cache service, which could be expensive to operate.

When we have things moving between different machines, I think we either want to:

1. Directly send it between machines, or
2. Upload it to some user managed storage

[noble raptor]

the cache service goes from becoming a "speeds up my build with faster cache", to a "i absolutely need this for the functionality of my build" - that feels like it's got a lot of implications for uptime/etc, and feels like quite a different product at that point.
Oh totally, if we wanted the sharing to be as "consistent" as the local cache rather than best effort then that's more complicated. I'm speaking speculatively though 🙂

which could be expensive to operate
The cache service is designed to always use storage that is fast and free in terms of network costs. I.e. if you are running on AWS S3 is used, Blobstore on Azure, R2 as the fallback default. So network costs aren't a concern

Directly send it between machines, or
Definitely something the cache service could support. It's just that in practice when you have multiple GB/s of bandwidth w/ S3, any benefits of p2p go down quite a bit

[rare steeple]

I'm glad we agree about doing something for the multi-arch case though 😄
It just seems we need to work out where we slice the distributed bit.

I'm biased, but I think we should try and cut as low as possible. I think the buildkit worker abstraction makes a lot of sense - the code is already geared towards this specific thing, even though it's never made it in. Essentially, I think what we'd end up with, is the ability to have multiple workers - that each return cache.ImmutableRefs that either point to a shared synchronous store of some variety, or ideally just to itself (which when we transfer this over a network boundary, we convert it to an implementation of cache.ImmutableRef that pulls it from a remote content store - so Mount on that Ref would transfer all the contents over, and then just Mount that).

Alternatively, we could cut much higher, at the dagger level - when we request a withexec with multiple platforms, we split it up into multiple llb operations, and offload one set to another worker. For some reason, I'm having difficulty explaining why I like this less, but I think it boils down to being much more dagger-specific stuff, and not as easy to upstream later, if we wanted to do that. It also means you can't have "one dag", you're having to split it up into lots of little ones, which potentially makes some things about it a lot harder, like trying to use artifacts from platform X in platform Y. I think also if we wanted "clustered buildkit", this is a bit difficult with that, since we'd want the ability to schedule individual ops separately (I think), instead of larger chunks of LLB.

Sorry, massively brain-dumping here 😛

[noble raptor]

I'm biased, but I think we should try and cut as low as possible
Yeah I think this makes total sense, my only point is that in both this approach and in the other one, you need the ability for engines to share their mappings of cache keys -> cache results. And we already have a service that does that 🙂

Not saying it's perfect for this use case in it's current state, just that it makes sense to re-use some aspect of it or to update it to handle this better

[rare steeple]

ahh i might be misunderstanding something in buildkit then - do you actually need this? if we did the logic to have an implementation of ImmutableRef pointing to another worker, i don't think you need the cache mapping?

i'm definitely less familiar with this part of buildkit 😱

[noble raptor]

So ImmutableRef has an ID, but it's just randomly generated and not a cache key.

The "meaning" of an ImmutableRef is given by the other cache manager in solver. That's where you setup the mapping of solver ops to the the ImmutableRefs for the results of those ops

The cache service client is just an implementation of that solver.CacheManager interface; it gets combined with the local cache using solve.CombinedCacheManager

Basically, even if this is all done on the worker level, a buildkit solver is still going to be processing LLB ops, get a cache key and then reach the point where it says "hey cache managers, do any of you have results for this?". At that point it needs to somehow be able to query the results of these other workers. If the workers only know about ImmutableRef they can say "I have all these cache refs, but I don't know what they are the result of"

[rare steeple]

right for some reason, this is what i was missing.

no that all makes sense 😄

[noble raptor]

I suppose if the remote worker is 100% tied to the process that has the solver running such that the worker is only ever associated w/ that solver process, then you could just assume that the solver will always know the mappings of cache key -> results for that worker

Oh to be clear, I'm thinking out loud too 🙂

[rare steeple]

yeah, i'm now not convinced this is the "easy" version of clustered buildkit, this is just clustered buildkit.

[noble raptor]

I think this would work, maybe it was the original intention(?), but it would be fairly limiting relative to going just a little bit higher in the abstraction stack (though not all the way to the high level DAG like you mentioned earlier)

[noble raptor]

Possibly... it's very interesting and worth more thought all the time either way 😄

Also just had a flashback to a previous incarnation of this sort of conversation: https://github.com/dagger/dagger/issues/4676

Clearly my thoughts have changed a bit since then

[rare steeple]

i think i'm imagine just one solver, with one scheduler and "one" cache backend
but then multiple workers, which the scheduler/solver sends out to - you wouldn't split out the cache keys, just the cacherefs i think, which would be wrapped in the WorkerRef I think?

there is some nice appealing stuff about having one scheduler do everything, with all that provenance knowledge, you could guess at how long operations would take and schedule them to the right worker

Part of the logic for wanting to cut it as low as possible (and not be 100% dependent on shared cache), is that we could get other non-dagger people involved, and not have to do all of this just us 😛

[noble raptor]

Right the limitation I'm thinking of is that the state of the worker would be highly linked to the state of the solver in ways that are somewhat limiting. Like if you have a worker with a ton of cache refs that are the result of exec-ops, then those would only ever be usable by a single solver. Sharing the worker or even just migrating it to a new solver would drop a bunch of cache for no reason

[rare steeple]

ah from this:

depot.dev appears to support this, very curious to learn how they do it
they do this using buildx, which is the client-does-everything approach - which i'd love to avoid (i think you agree?)

[noble raptor]

Totally, I don't think that's contradictory at all. Like I said, the cache service hooks into buildkit's solver by just being an implementation of the upstream solver.CacheManager interface. There's nothing stopping others from doing that too

And/or making it easier to do that

[noble raptor]

Yes totally agree now, those were my thoughts in March, which is at least 3 lifetimes ago 🙂

[rare steeple]

mmmm I had not considered that 🙂 I think this is partially mitigated by the cache exports and the cloud service we have today - but yes, still an issue.

hm, another dimension is the stateless engine stuff - if engines aren't super long-living, then i wonder how that impacts any/all of this (and what that would even look like from a config perspective)

[noble raptor]

Yeah the new architecture is actually great for being able to experiment here.

Just throwing out a first reaction: We could try splitting out the current monolithic engine (aka dagger's buildkitd) in two:

1. The controller+scheduler+solver
2. CacheManager + Worker

At first, could just try that as two containers in containerd that talk to each other locally. If we can get that to work at all, it would be the first step towards what we're discussing 🙂

I have a million more thoughts but I need to get back to what I was originally intending to do today 😄