#best way to run dagger in cloud CI and cache

1 messages · Page 1 of 1 (latest)

smoky berry
#

Im running dagger in diff cloud CI github actions and GCP cloud build are two of them. Which is the best way to store dagger cache and share it between builds?

Even when dagger is awesome for pipeline building if we lose the cache, or hability to use another container as cache for container building (which is usually one of the steps of the pipeline) we end up waiting really long times cause of that container building.

In my scenario we have a microservice architecture which holds 10+ microservices which leads to build 10 diff container images so that usually takes 30+ min.

Using docker build on GHA with cache takes less than 5 min.

Hope there is a way i can at least save dagger cache in a bucket and restore it between builds

smoky berry
#

any hint <@&946480760016207902> please

#

been push by my company to solve this 😦

runic halo
smoky berry
#

sadly payment is not on the table for the company, i was pushing to use dagger as cool and free. i was thinking that if there is a dir i can cache on a bucket and restore prvious dagger start then it could work. or a better approach

#

there real thing is that on a prod scenario the build pripeline increase from ~5min to ~30min

patent dome
#

@smoky berry we are working on "storage drivers" in the engine, so you'll be able to use your S3 bucket (or equivalent) for cache persistence. It's not shipped yet, but in the meantime there are several stopgaps.

#
  • Stopgap 1 is to use our experimental distributed cache service, like @runic halo said.
  • Stopgap 2 is to run the Dagger engine on a machine with more persistent storage, and connect to it from the CLI with _DAGGER_EXPERIMENTAL_RUNNER_HOST
  • Stopgap 3 is to use _DAGGER_EXPERIMENTAL_CACHE_EXPORT (or something like that) which supports a limited implementation of S3 storage. But you will need to manually manage the cache key, and there are limitations. This is the equivalent of --cache-from in docker buildx/buildkit
#

cc @pine oar @unkempt swan who are experts on these topics

pine oar
#

I can't think of other alternatives. That said, there is one hurdle worth metnioning for this one:

Stopgap 2 is to run the Dagger engine on a machine with more persistent storage, and connect to it from the CLI with _DAGGER_EXPERIMENTAL_RUNNER_HOST
Since you are running your CI in two separate compute providers you'll have to deploy the engine in a central place and figure out how to securely connect to this engine from each of the CI, setting up the networking bits is not necessarily tricky, but its important that it gets done correctly

smoky berry
#

Im looking on the doc for _DAGGER_EXPERIMENTAL_CACHE_EXPORT usage but cant find it. Can i get a link or an example on how to please? thanks