#Run Dagger on Kubernetes with shared cache

1 messages · Page 1 of 1 (latest)

radiant lance
#

Hey everybody,

I'm currently using Dagger 0.14.0 and I successfully deploy it on my Kubernetes cluster on the fly when requested by my CICD pipeline (ArgoWorkflow).
My issue appears when I try to share the cache between multiple pods that are running in parallel as it seems that my PVC (that is mounted on /var/lib/dagger path) is not well configured to handle the cache related to the workers.
So from what I understood, Dagger use SQLite to cache the pipelines layer and it is stored in /var/lib/dagger/workers/... path. However, having Dagger on Azure Kubernetes Service, only 2 options are available to me:

  • Either I mount the PVC using a Azure Disk, which are not shareable between the pods (except in raw block mode but I'm pretty positive that SQLite won't handle that)
  • Or I mount the PVC on Azure Files
    First solution is out of question as impossible to mount in ReadWriteMany mode.
    I tried the second solution but it doesn't works because Azure Files being a Fileshare system, it doesn't have the same API than a classic Filesystem. It leads me to the following error when trying to use a Azure File PVC:
starting session ERROR [0.1s]
3   :   ! connect buildkit session: unexpected status 200: get or init client: open client DB: ping file:///var/lib/dagger/worker/clientdbs/w768gqw86ah89l83kld2gfz7q.db?_pragma=foreign_keys%3DON&_pragma=journal_mode%3DWAL&_pragma=synchronous%3DNORMAL&_pragma=busy_timeout%3D10000&_txlock=immediate: database is locked (5) (SQLITE_BUSY)

So I now have 3 questions for you:

  1. Is it possible to use another type of cache than a SQLite database that may be usable with a Fileshare by editing the engine.toml configuration file?
  2. Is there a way to disable local caching knowing the pods are ephemeral anyway?
  3. When using Dagger Cloud Cache and cache_volume function, does it disable local caching? Does it serve the same purpose than the local cache?

Thank you!

#

I should add that it is using Kubernetes Ephemeral Storage currently, which works well but is very limited due to the Node default storage.

random condor
#

Commenting to also see any answers you might get, as I am interested in this problem and its solution. 🙂

radiant lance
#

@haughty lion I had a look at Dagger's website and it seems that any reference to the fact that Dagger Cloud cache the layers if we have a Team subscription has disappeared. Has the feature been dropped?

somber creek
#

It has not been dropped. They are working with the initial alpha users to work through their findings. There are a bunch of tricky situations to shared caching layers generally speaking