#Two caching features: confusing?
bleak ore
re "volumes": might bring some confusion from docker run -v. Thinking about alternatives
frozen oracle
cc @night prairie @fossil isle in case you want to lurk 🙂
Let me share a bad idea, which maybe can lead to a good idea 🙂 What if instead of different directory types (TmpDir, CacheVolume, and plain Directory), each with their own WithMountxxx, we collapsed them all into Directory, and added configurable persistence properties to the Directory? Then depending on the persistence properties, we would use the right buildkit and/or sync low-level operations.
We have precedents for this (kind of):
-
Directory vs. HostDirectory. Very different implementation, but used the same way. EG. there is no
WithMountedHostDirectory -
CacheVolume concurrency flags.
What I'm proposing is to further generalize these precedents 👆
Then maybe instead of talking about "cache volumes" we're just talking about directories with special persistence options ("this directory should persist across runs")
Related idea: maybe these volumes/persisted directories are somehow tied to the "environment" which are currently defining with Zenith? Since we're facing the task of namespacing them into "environments" on the Cloud side also...
acoustic nova
Let me share a bad idea, which maybe can lead to a good idea 🙂 What if instead ...
I do think this is in the direction of what's already been on my mind for mounted caches, something CloudVolume vs LocalVolume
frozen oracle
So like there's Host.Directory() maybe there's also Environment.Directory() or Environment.Volume() ?
frozen oracle
I do think this is in the direction of what's already been on my mind for mounte...
Looks like we're onto the same thing here 🙂 One very important detail though: the code has to be the same with and without Dagger Cloud. So that you can always drop in Dagger Cloud and all your existing pipeline logic works - but better
acoustic nova
more of a 'sync if possible' vs 'never sync'
frozen oracle
So the abstraction to achieve that I think is Environment (my working theory)
When you drop in Dagger Cloud, your environment is synchronized across engines. If you don't - it's not. But it still works the same, just with less cache hits
onyx wadi
instead of talking about "cache volumes" we're just talking about directories with special persistence options ("this directory should persist across runs")
I like that. The word "volume" is loaded from docker. A directory with a persistence setting is much clearer!
frozen oracle
But what I'm not sure is, what's the best way to explain the properties of the directory. Honestly I still get confused sometimes - which is what happened in the support discussion we had with Megan. I got confused about how those volumes interfere with the caching behavior
So we need terminology that gets to the essence