Capturing state from a service into an image layer

[hoary sable]

Perhaps I could...

1. mount a volume while running it as a service
2. copy this volume in to the same path afterwards

[steep crown]

@hoary sable why not just using a WithExec to run whatever you need to run and persist that in the image layers?

[hoary sable]

It requires an API call, so the server has to be running first, hence the service thought

[steep crown]

Yes, correct. You can have the service running and let the overlayfs handle the data persistence

Now I have some doubts if stopping the service will effectively write the content in the overlayfs

I'd assume it'll do so but I can't recall precisely how the service stopping mechanics work

@hoary sable LMK if you give it a try. If not, I have another option that will surely work

[steep crown]

This module does what I have in mind https://daggerverse.dev/mod/github.com/marcosnils/daggerverse/utils@bfa9939333c74e5b74a7a3281d9a3fad729fba49

[hoary sable]

I found a less ideal solution on my end that got me unblocked, basically restore the data from a snapshot on container startup. Ideally it's ready to go, being prewarmed during build time

I'm not using functions yet, can I call them from the dagger run paradigm somehow?

[steep crown]

Not yet unfortunately, but it's coming soon

[steep crown]

Yes, this works also. I personally prefer this approach better since ideally you want the db volume to live in a cache volume and not in the overlay fs

It's also more clear about what the pipeline is actually doing since the code to restore from the snapshot is actually there

[hoary sable]

I don't want people to have to think about volumes and extra commands, just a docker run and they are good to go

It's not a lot of data, far less than node_modules or similar

It's mainly for testing & demo purposes

[steep crown]

Makes sense. Both approaches work. It's a tradeoff between image size vs pipeline complexity and performance

[hoary sable]

I'm currently saving the file in the image and then running it with the necessary extra flags

[steep crown]

The snapshot file?

[hoary sable]

yes

in the running image, I then have both the snapshot file and the restored data

[steep crown]

Cool, yep, that works

[hoary sable]

In the docker world, I would run the image, run the restore, then commit to create a new layer

It works, but it means I have 2 copies of the data in the running image

Ideally I could just mount the snapshot in Dagger, run restore, and then keep just the actual data layer

[steep crown]

Well.. you can also do the same in docker with a cache volume also

[hoary sable]

Can I distribute cache volumes?

[steep crown]

Oh, no. I'm just thinking about building the image

I wouldn't rely on distributed cache volumes for this though

Cache volumes are just a cache. It's a best effort storage system designed to speed up pipelines