Buildkit describes itself as being able to have "distributable workers". I can't find any documentation that describes how to do this though. Is it possible to get dagger to distribute work across multiple machines?
#Distributing work across multiple machines
thorny stratus
sleek sky
Not in a nice and friendly way, yet.
There is some technical documentation about using an external worker in a PR: https://github.com/dagger/dagger/pull/4232
thorny stratus
Thanks -- ok, so that's how to use a remote/external worker, but what about using multiple external workers? Is it just a case of sticking the workers behind a load balancer, and then dagger will essentially end up distributed across multiple workers because it "calls into itself" via the load balancer, or something else?
sleek sky
Good point. I think there was something like this for the BUILDKIT_HOST we used to use. I don't know the syntax about it. I don't know if it's still possible to do that way by using the kind of hidden env var.
@ebon phoenix would know better
ebon phoenix
Thanks -- ok, so that's how to use a remote/external worker, but what about usin...
Right now Dagger doesn't take care of any of that for you, it's possible to set something like this up but it would be totally DIY. The instructions in those docs would be base but yeah you would need to layer more on top. It has been done before though so:
- This kubecon talk should be helpful: https://www.youtube.com/watch?v=wTENRhYt3mw
- This example code may be helpful too (though it doesn't have a ton of documentation): https://github.com/moby/buildkit/tree/master/examples/kubernetes/consistenthash
thorny stratus
Interesting -- thanks for the links. Good to know that there's at least one strategy for doing it. Looks like there's a way to go yet before one can get an optimal caching/build situation from it -- e.g. allocating jobs to nodes with appropriate resources like RAM, GPU, compute etc.