with_mounted_cache not persisting after restarting

[charred elm]

I'm running my backend database locally for development. When I kill the service (Ctrl+C) and restart it, the with_mounted_cache seems to be lost.

Can I inspect the cache volume? Does the trace help? The docs are sparse: https://docs.dagger.io/manuals/developer/cache-volumes

with_mounted_cache volumes not persisting after restarting service

with_mounted_cache not persisting after restarting service

with_mounted_cache not persisting after restarting

[vital stump]

@brisk hound ?

[brisk hound]

If you upgrade to v0.12.1, there may be more logs to help debug this (which can be obtained with docker logs dagger-engine-fec13a4d7c04bf96).

Generally speaking, cache volumes are considered "best-effort" in that they can be pruned if disk space needs to be freed, so something to keep in mind for running database state off of it.

[charred elm]

What should I look for it the 0.12.1 log? There is no mention of the cache volume's name or location.
And what's the alternative to cache volumes for persistence?

[brisk hound]

Actually, sorry, I just remembered the PR that showed those logs wasn't merged yet (https://github.com/dagger/dagger/pull/7943), I was getting different PRs confused 😅 Will be in the next release

• In the meantime, it's harder to say but most likely the engine is pruning the cache volume

We don't have great support for truly persistent state yet. We treat all state as cache and ephemeral right now. However, coming soon is support for storing cache volumes on your host's machine, which will also include making the pruning of it fully manual: https://github.com/dagger/dagger/pull/7646 So then those cache volumes will be persistent until you explicitly tell dagger to delete it, which sounds closer to what you'd want I think?

[idle current]

@charred elm you might be interested in what I have here: #1260558982246825994 message

I think it is exactly what you're looking for, but do tell me if it isn't. Describing your workflow also helps.

[charred elm]

I'm trying to persist data into a database that survives across Dagger restarts in order to create a production-like development environment. Does your code achieve that?

[idle current]

Oh, looks like this issue has been changed several times. Could you try to describe what exactly you're doing? Showing a minimal example is really appreciated.

[charred elm]

1. Start dagger.
2. Write to database
3. Restart dagger (say, after rebooting your computer)
4. Use persisted data
   The problem is that the data does not survive restarts.

[idle current]

Can you show us how you use the with_mounted_cache call?

[charred elm]

Simply as

dag.container().from_(consts.DB_IMAGE).with_mounted_cache("/tmp/data", dag.cache_volume("db-data")).as_service()

[idle current]

seems alright, except for, I think, anything under /tmp is cleared normally somehow on nixes. Can you test it out with something that doesn't use /tmp

[charred elm]

Yes. Same result. I think in your case you are not restarting dagger?

[idle current]

I tried to align my environment to yours. I stopped my dagger engine, and re-started it. It appears to keep my database changes (I modified my setup to keep them). Is this what you're doing as well? By restarting, do you mean this? What I mean is, maybe your dagger engine is deleted somehow, which might delete its volumes as well?

[idle current]

Restarted my computer as well (fresh session, actually). It appears to work.

Something else is involved I believe.

[charred elm]

How did you stop dagger? I hit ctrl+c. I'm on version 0.12.1

[idle current]

docker stop <id>

I am a bit puzzled by you mentioning ctrl-c. Can you tell us how are you interacting with the Dagger engine? This might reveal something useful.

You shouldn't be able to kill the Dagger engine with ctrl-c. It isn't a process/foreground job you have access to directly.

[charred elm]

I run dagger call --src_dir=. backend up and kill it as stated when the development session is over. It runs as a service; I can kill it, with ctrl+c or by hitting "q" (quit). Either way, the data becomes lost when I rerun the same command.

I just tried stopping it with docker stop and it said Error: response from query: Post "http://:mem/query": unexpected EOF and it still lost the data.

[idle current]

As I thought, we were referring to different things. Either way, my setup works both with the dagger engine restarting, as well as the service itself restarting.

Is it possible if you could provide more of your dagger setup?

[charred elm]

what would you like to know?