how can you persists a cached volume from multiple invocations ? | Dagger | Page 1

stray remnant Dec 21, 2023, 5:18 PM

#

Hello folks

https://docs.dagger.io/882813/build-test-publish-java-spring/#step-1-create-the-dagger-pipeline

I am truing to build and run integration tests locally for a gradle , maven , and ui that requires a cached volume. Basically when the user calls build locally i dont want that shared/cached volume to disapear and be rebuilt across multiple local invocations.

I could not find any documentation that explains how that client.CacheVolume API call is supposed to work. as in

when is that cached volume removed ?
how can i guaranty that volume exists across multiple invocations ?
how can i delete it explicitly ?

Build, Test and Publish a Spring Application with Dagger | Dagger

Introduction

#

how can you persists a cached volume from multiple invocations ?

#

how can you check its existance across multiple invocation to make sure you cant recreate it ?

tepid sierra Dec 21, 2023, 6:17 PM

#

when is that cached volume removed ?

it's never removed except it's picked by the engine garbage collection mechanism which will remove it if you're running low on disk space. You'll see an event with removed snapshot in the engine logs when this happens. You can disable gc altogether through the engine.toml file (https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md) if you don't want this to happen.

If you're using Dagger Cloud, your volume will be always available.

how can i guaranty that volume exists across multiple invocations ?

AFAIK you can't since cache volumes get created automatically if they don't exist. What you can do is check if there are any files in the volume path (run an ls) before running your pipeline to check if it's empty or not.

how can i delete it explicitly ?

You can't currently remove a single volume.
You'll have to remove your entire engine's state directory for this. If you're using docker to start the engine, this would be a docker volume attached to the engine container

how can you check its existance across multiple invocation to make sure you cant recreate it ?

you can check if it has any files on it

stray remnant Dec 21, 2023, 6:47 PM

#

That answer it thank you very much !

tepid sierra Dec 21, 2023, 6:58 PM

#

forgot to mention that in Dagger Cloud we have better management of volumes so we do have a way to provide more visibility and pruning operations for them

stray remnant Dec 21, 2023, 7:24 PM

#

The thing is this build,tests cycle is meant to run locally across multiple builds. I am trying to not diminish that experience by fetching to/from the cloud

#

Also is there an api i can use to mount a particular volume ? Not sure what dagger is using behind the scene. i know i can call docker cli (i am running on linux on top of docker) to mount a local volume but i am hoping there is a better api based alternative that would still work with dagger

tepid sierra Dec 21, 2023, 7:54 PM

#

stray remnant The thing is this build,tests cycle is meant to run locally across multiple buil...

that make perfect sense. The benefit of using cloud is that it for whatever reason the volume is not present locally, it'll get downloaded instead of having the pipeline re-create it each time. After it's downloaded, the engien wil prefer the local volume whenever it's present.

#

Also is there an api i can use to mount a particular volume ?

that's the dagger API. When you do client.WithMountedCache that will automatically mounts the volume for you

#how can you persists a cached volume from multiple invocations ?