#Cache Volume Deep Dive

1 messages · Page 1 of 1 (latest)

native prairie
#

Hey guys, I'm trying to mentally map cache volumes. How do they work?

I'm trying to understand a bit deeper about how cache volumes or caching in general works in dagger. Specifically how do dagger volumes map to like, real filesystems? How is the actual cache volume mounted?

For some context about where I'm coming from, my mental model is that cache volumes are somewhat similar to PVCs in K8s, so in this case I'd want to "clone" a shared cache vol so there's no contention over the FS between processes.

Given a scenario where I have one dagger daemon, and more than one client that mount a CacheVol named foo. Do all clients physically mount something? Do they "clone" the cached layer into each build?

jaunty dawn
#

cache volumes are regular directories on the engine’s filesystem, managed by buildkit, and bind-mounted by buildkit at exec, via the cache-mount options of the Exec llb operation.

#

Each cache volume ID maps to a unique directory on the engine’s filesystem

#

Concurrency can be configured in the buildkit API. I forget how we mapped it to the Dagger API, but it lets you control how concurrent writes to the same cache volume are handled

#

cc @summer steeple @hybrid bobcat @glass sigil @fleet wave who know more about the messy internals of that logic

native prairie
#

ok sick that makes a lot of sense. Ty for the link too, tbh I hadn't read much of the newer dockerfile syntax - dagger has made it a little easier to consume these apis it seems 😄

I think my question, and probably the magic lies in here...

A shared cache mount can be used concurrently by multiple writers.

Say I have a pipeline where I want a bunch of builds to benefit from the shared cache.. gradle is probably the easiest example, but I think some of our CPP builds work similar.

Should have a strategy for generating unique cache vols for things like gradle or node modules, or can I just share a big ol' chonker across everyone?

Thinking through it I hit a wall with this scenario:
The first build runs, now we have a baseline in the shared (cache) vol. Then I have two builds that write files to this volume at the same time (or an order I can't really predict). Who wins?

summer steeple
# native prairie ok sick that makes a lot of sense. Ty for the link too, tbh I hadn't read much o...

Should have a strategy for generating unique cache vols for things like gradle or node modules, or can I just share a big ol' chonker across everyone?
Ultimately this depends on what you're putting in the cache volume, and the processes you're using to write to it.

So for example, something like apt-get locks the package cache when downloading - only one writer is allowed at that level. So it doesn't matter if you allow multiple writers with a shared volume cache, apt will refuse to write.

I can't quite remember how things like go, etc work - but generally, if it would be reasonable to run multiple processes that access the same cache without any containerization, then it's reasonable to have those proceses have a shared cache volume.

I forget how we mapped it to the Dagger API, but it lets you control how concurrent writes to the same cache volume are handled
The options for this are exposed in ContainerWithMountedCacheOpts for the Go SDK: https://github.com/dagger/dagger-go-sdk/blob/v0.9.3/dagger.gen.go#L1110-L1111

The first build runs, now we have a baseline in the shared (cache) vol. Then I have two builds that write files to this volume at the same time (or an order I can't really predict). Who wins?
This depends on the above CacheSharingMode - the docs for what those all do are here: https://github.com/dagger/dagger/blob/main/core/schema/cache.graphqls#L4-L17
Not quite sure why these aren't documented in the sdk docs though, that's a bit odd and might be worth a bit of a dig (will take a look!)

native prairie
#

got it - this helps a ton thanks guys!

native prairie
#

Ok I've been noodling on this for a little and I've got a little follow up.. Really what I'm trying to zero in on here is optimizing a gradle build. Got a really big project, we've seen some speedup from basic dagger caching, but want to push it further.

Doing some reading there's a couple concerns.

  1. Gradle locks files. If I have contention for these files, I think this would ultimately show as iowait (?), and slow things down.
  2. A gradle clean. My nightmare. If someone runs gradle clean, then the cache is blown for everyone.
  3. General build conflicts - I shouldn't share a built/cached module from a build on a PR with the main branch, it would be "ahead" of main in this case.

Given these constraints I think I'd want private mode, but now I'm wondering how that behavior surfaces to the end user? Is this similar to file locking?

fleet wave