#Investigate local dagger engine cache allocation

1 messages · Page 1 of 1 (latest)

scenic saffron
#

Hello,

Is it possible to somehow query or see how much cache is being used between named cache volumes, instruction caching etc. ?

#

I am trying to optimize the buildkit toml to adjust how much cache we keep of different types

severe dawn
rich ventureBOT
final terrace
#

We recently added raw API support, still working on the frontends for it. But you can access this from the CLI:

  • Show all the cache entry metadata: echo '{daggerEngine{localCache{entrySet{entries{description diskSpaceBytes}}}}}' | dagger query
  • Show just high level summaries of usage: echo '{daggerEngine{localCache{entrySet{entryCount diskSpaceBytes}}}}' | dagger query
scenic saffron
#

Hi @final terrace , thanks for the tip on the API support, we are now able to segregate out the different types of cache used! We effectively have a scheduled task that will query the cache and break it down by type that we parsed from the different returned patterns of: '{daggerEngine{localCache{entrySet{entries{description diskSpaceBytes}}}}}'.

I would say some difficulties we saw were:

  1. What are the different patterns/types? Identifying what is a mounted cache vs. layer cache vs. base image cache vs. built image cache
  2. How can we now use this information and translate it to buildkit configuration to optmize our keepDuration gcpolicies.
  3. How can we modify what the dagger engine thinks is the total engine size? I noticed in the query responses that there is a activelyUsed field which maybe only kicks off when the cache size is above the total size? We would like to control this value so that we can trigger the dagger pruning earlier

Here is our current engine.toml:

# https://docs.docker.com/build/cache/garbage-collection/#default-policies
[worker.oci]
enabled = true
gc = true

  [[worker.oci.gcpolicy]]
    all = false
    keepDuration = "12h"
    keepBytes = "1GB"
    filters = ["name==vuv_cache"]
  
  [[worker.oci.gcpolicy]]
    all = false
    keepDuration = "12h"
    filters = ["name==uv_cache"]

  [[worker.oci.gcpolicy]]
    all = false
    keepDuration = "48h"
    filters = ["name==pip_cache"]

  [[worker.oci.gcpolicy]]
    all = false
    keepDuration = "24h"
    keepBytes = "20GB"
    filters = ["type==source.local", "type==source.git.checkout"]

  [[worker.oci.gcpolicy]]
    all = false
    keepDuration = "120h"  # 5 days

  [[worker.oci.gcpolicy]]
    all = false
    keepBytes = "150GB"

  [[worker.oci.gcpolicy]]
    all = true
    keepBytes = "150GB"

This was very helpful though thank you!

image.png
lethal cloak
#

How did you get that graph 😄
And most importantly how did you get the names of the caches, the above dagger queries do not include the cache names

lethal cloak
#

Does dagger use the builder defined docker builder ls? or does it spin up another builder internally?
I'm trying to figure out where I can pull more info about cache. And if its using the default docker builder then the GC policies must go to the daemon.json instead of the builtkit toml

#

For example, with this buildkit file, even after VM reboot, dagger reports 65GB for keep-bytes

jenkins@jenkins1:~$ sudo cat /etc/buildkit/buildkitd.toml
[worker.oci]
enabled = true
gc = true
gckeepstorage = "80%"

  [[worker.oci.gcpolicy]]
    keepBytes = "30%"
jenkins@jenkins1:~$ dagger core dagger-engine local-cache keep-bytes
✔ connect 0.2s
✔ initialize 0.3s
✔ prepare 0.0s
✔ daggerEngine: DaggerEngine! 0.0s
✔ DaggerEngine.localCache: DaggerEngineCache! 0.0s
✔ DaggerEngineCache.keepBytes: Int! 0.0s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export NOTHANKS=1

6.4e+10