#Cache + remote repo

1 messages · Page 1 of 1 (latest)

rotund sedge
#

Starting a thread here 🧵

#

So, the topic of distributed caching, in particular in the context of CI, is under very active development, and best practices are still moving. Here's the current situation

#

  1. Dagger uses buildkit under the hood for caching. So the current state of the art in buildkit caching, is the baseline for Dagger's caching functionality.

  2. This baseline is a buildkit feature called "cache export". It allows a client to export the contents of its cache after each run, to a configurable target: registry, S3 bucket, etc. There is an escape hatch for using this feature in Dagger. However, cache export is quite limited. So while it's possible to use it via the escape hatch, we only recommend doing so as a stopgap, with full understanding of the limitations, and while waiting for a better solution.

  3. We are developing a solution to the problem, in the form of a managed control plane which orchestrates caching across all your engines. It is available as a cloud service, currently in early access but shipping in GA soon. We're happy to give you early access if you want.

#

There are several reasons that using Dagger Cloud for caching works better than a DIY solution with export-cache. Some of them have to do with implementation - for example we transfer cache data continuously, instead of in a separate export phase at the end of the pipeline; we also persist cache volumes; whereas with export-cache they get wiped in between runs. This is a big deal because persisting your go cache, npm cache, etc. is a big factor in pipeline duration.

There is also a fundamental reason: caching across multiple machines is an orchestration problem. To do it efficiently, you need a central control plane to orchestrate the movement of cache data across the whole system. A stateless architecture (a bunch of engines + dumb data storage) can never be as efficient. This is why we decided to add caching to Dagger Cloud in the first place: it's the only way to deliver a great caching experience in production.

slow nest
#

I'll experiment with the built in buildkit remote caching, I'm not expecting some synchronized distributed caching solution just not to rebuild a layer if it's been completed prior to current build starting

clever kite
# slow nest I'll experiment with the built in buildkit remote caching, I'm not expecting som...

This should work totally fine for relatively small-scale builds where the engine importing/exporting cache is long-running and only running one build at a time. Once you start using it with engines that are short lived, such as in github actions (without self-hosted runners), it will get more and more complicated since buildkit's built-in remote caching requires exporting "cache manifests" that overwrite each other if all pointing to the same destination.

The cache service gets rid of that problem (it's more of just a "flip it on" switch and coordinates everything for you) and has various performance improvements that matter more and more as builds scale up in size.

slow nest
#

is the cloud service just an orchestration service and storage can still be local (eg. s3 / image repo)?

rotund sedge
amber gale
#

đź‘‹ Hey, sorry to necro the thread here but I'd be interested in a self-hosted version of this, mostly because my current employer is an F100 and a cloud service for build caching is going to be a hard sell from a compliance perspective.

rotund sedge
#

there’s an intermediary “bring your own bucket” where data stays on your infra while orchestration is on ours. Then full blown self-hosted everything which is more work but also on the roadmap