#Questions about Cache controls

1 messages · Page 1 of 1 (latest)

obsidian needle
#

Thanks for the hard work on Dagger it seems really promising as a way to do CD for a series of codes that I maintain. One question I had was about cache invalidation. Is there some documentation about what causes the build cache to be invalidated?

My application is a moderate to small c++ code with a bunch of dependencies (200MB-8GB depending on configuration). Depending on which ones and how many downloading or building these can take between minutes to hours so caching is critical.

I want to test if the code works on several Linux distributions using the system dependencies (I.e. dnf/apt). For these, I don’t have a lock file per se, and If I did I wouldn’t want one — I want the cache to invalidate as soon as apt or dnf says there is a newer version. In docker I would use something like mount-type=cache=locked https://docs.docker.com/engine/reference/builder/#run---mounttypecache

I also need to ensure the software installs via a package manager common in my domain (spack; used in high performance computing). In this case I can generate a “lock file” which tells me if my dependencies that I have built have been invalided, and I need to rebuild some subset of them.

How can I handle these separate use cases? I use the python sdk because Python is the language that my colleagues are most likely to know, but I’m open to using something else if needed.

Docker Documentation

Dockerfiles use a simple DSL which allows you to automate the steps you would normally manually take to create an image.

quasi plank
#

Hi there 👋

There’s a general answer and a more specific one.

#

General answer: Dagger is built on the same caching tech (buildkit) as docker build. So anything you can do with the latter, is either possible with the former, or will be in the near future.

#

Specific answer: the equivalent of “RUN --mount=type=cache” is what Dagger calls “cache volumes”. let me dig up a link

#

I believe the “locked” concurrency setting is not exposed in the Python SDK at the moment, because it’s not exposed in the underlying graphql API. It is available in the old CUE API, but has not yet been ported over. Doing so should be quite easy.

May I ask if you’d be willing to open an issue asking for it? It’s always easier to prioritize when an actual user asks for something 😇

Thank you!

obsidian needle
#

I wasn't sure if I should close this or leave it to you now that I've opened the issue.

lone void
obsidian needle
#

@lone void that would be awesome. Dagger already solves a lot of problems for me already so knocking off these last few pain points would be really helpful

lone void
#

Hi @obsidian needle, it took a bit more time, but this feature has just been merged: https://github.com/dagger/dagger/pull/4437. We might do an engine release during the week, so it's going to be available ASAP. Otherwise, it is already available with the dev version of Dagger 😇