trying out _EXPERIMENTAL_DAGGER_CACHE_CONFIG but seems to be not working | Dagger | Page 1

lean jackal Jul 28, 2023, 2:11 AM

#

Hi, i'm currently trying the _EXPERIMENTAL_DAGGER_CACHE_CONFIG using s3 and registry. From the logs, it seems to be working

11: importing cache manifest from s3:17471463523230677791
11: importing cache manifest from s3:17471463523230677791 
...
...
...
25: exporting cache to Amazon S3
25: preparing build cache for export 
25: writing layer sha256:ab9a9dd810c0dc1bf1f6b911095d8f85f3b7e8d22e633e060e6f980b59ad3d23 
25: writing layer sha256:ab9a9dd810c0dc1bf1f6b911095d8f85f3b7e8d22e633e060e6f980b59ad3d23 [0.01s]
25: writing layer sha256:c2ddf4c33d7561bc63d9b4cd20b5645a4528c37599dacbf0ed6b8e4df1fda6bf 
25: writing layer sha256:c2ddf4c33d7561bc63d9b4cd20b5645a4528c37599dacbf0ed6b8e4df1fda6bf [0.01s]
25: writing layer sha256:c798134b5a2faa32b79b83ed0bf6b50a0117ee247c05afcb9f44ff6fd977b17c 
25: writing layer sha256:c798134b5a2faa32b79b83ed0bf6b50a0117ee247c05afcb9f44ff6fd977b17c [0.02s]
25: preparing build cache for export [0.27s]
25: exporting cache to Amazon S3 DONE

but if i re-run the pipeline after deleting the volume of dagger engine, it runs like it has no cache at all. Did i miss something? i see something similar in https://github.com/moby/buildkit/issues/2274, does it relates?

📎 main.py

GitHub

Issues · moby/buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit - Issues · moby/buildkit

dark walrus Jul 28, 2023, 5:10 AM

#

Hi @lean jackal 👋 I know there are gotchas for using the basic buildkit cache. Because we want something solid that also has better performance through enhancements like caching of volumes (like for pip cache), etc, we've been investing in the Dagger cache service. Happy to show that to you if you're interested 🙂

Unfortunately I don't know the solution to the issue you're hitting.

proper jewel Jul 28, 2023, 1:38 PM

#

seems to be related to buildkit. Unfortunately, we're not using buildkit's default exporter since it's known to have a some gotchas as well as unresolved issues upstream. That's why we're investing to build our own distributed cache solution which resolves this issues plus adding some more efficient caching.

lean jackal Jul 29, 2023, 12:29 PM

#

Thanks for the response. I ended up using tar+rclone to backup and restore /var/lib/dagger.

lean jackal Jul 31, 2023, 3:25 AM

#

Hi @proper jewel , any gotchas that you can think of when using tar to store the /var/lib/dagger ?.
i'm seing "some" cache miss when using restored cache that was stored using tar+rclone.

proper jewel Jul 31, 2023, 3:32 AM

#

lean jackal Hi <@336241811179962368> , any gotchas that you can think of when using tar to s...

Are you using Dagger v0.6.4? It contains some fixes for incorrect cache invalidations in 0.6.3

lean jackal Jul 31, 2023, 3:33 AM

#

yups 0.6.4

#

proper jewel Jul 31, 2023, 3:36 AM

#

In that case there shouldn't be a difference. Would be nice to have a minimal repro case so we can look further, but rcloning the data directory should work

lean jackal Jul 31, 2023, 3:37 AM

#

right, let me try to create repro

proper jewel Jul 31, 2023, 4:00 AM

#

lean jackal right, let me try to create repro

Hey rucciva, just realized you're using rclone. I misread and I thought you were using rsync.

Rclone won't work because AFAIK it doesn't understand hardlinks which buildkit and docker use

lean jackal Jul 31, 2023, 4:05 AM

#

i'm rclone-ing a tar-ed dirrectory

#

proper jewel Jul 31, 2023, 4:12 AM

#

lean jackal i'm rclone-ing a tar-ed dirrectory

Got it. In that case, it should work. Looking forward for the repeo

lean jackal Jul 31, 2023, 4:38 AM

#

wait, the cache miss is even before using restored cache.

in my CI, i tried running the same dagger pipeline twice (without even stopping the engine), but cache missed happens. on the second run, it pulls the same image pulled on the first run 🤔

lean jackal Jul 31, 2023, 5:01 AM

#

re-running the same pipeline using docker-compose using another maching does not lead to cache miss.

the CI is jenkins and the agent is running under kubernetes, /var/lib/dagger is mounted using empty dir. 🤔

lean jackal Jul 31, 2023, 5:40 AM

#

using rclone and tar on the docker compose is working fine

lean jackal Jul 31, 2023, 6:01 AM

#

after seeing the log, righ bellow one of the pulled docker image, i see snapshoot are being removed. is that normal?

📎 message.txt

proper jewel Aug 2, 2023, 1:25 PM

#

@lean jackal were you able to make progress here?

lean jackal Aug 2, 2023, 1:47 PM

#

Hi @proper jewel , i ended up creating a new thread (which you already commented) since it no longer the problem with the experimental flag or cache restore. Thanks again

proper jewel Aug 2, 2023, 1:56 PM

#

👍

calm hill Mar 22, 2024, 1:38 PM

#

Sorry to bring this back up, but can you explain how to use _EXPERIMENTAL_DAGGER_CACHE_CONFIG? It's not in the docs anywhere

#trying out _EXPERIMENTAL_DAGGER_CACHE_CONFIG but seems to be not working