#Distributed Caching Status

1 messages · Page 1 of 1 (latest)

silk silo
#

I've started looking at the Dagger Caching feature as part of Dagger evaluation for our organisation. However I can't see any mentioned of Cloud Distributed Caching on the website or in the documentation any more. I know this was marked with an Experimental tag. Has Distributed Caching been discontinued?

pure beacon
#

@silk silo we just completed the first phase of the tech preview for distributed caching aka "magicache". We're now focusing on analyzing the results and deciding what to do next. While we do that, we've taken down the marketing from the website, so we can focus on supporting the first cohort of customers. In retrospect, maybe we should have kept it up, and made it a waitlist instead.

Here's what we know:

  1. Magicache has been working great for some workloads, and not so great for others. We're still investigating the root causes, but magicache in its current form is not yet ready for GA.

  2. Our next step is to either (a) improve magicache as it exists today; or (b) replace it with something better. Either way, we will continue developing better ways to distribute cache in Dagger.

  3. Magicache is one of several ways you can get good caching with Dagger in production today. We will continue to improve this area of the platform, regardless of the next step for magicache. It's a priority for us that everyone can run Dagger in production at scale, with great caching.

We would love to talk to you directly about your use case, and how to get the best possible caching setup for it. Would you be open to that?

silk silo
#

Hi, thanks for the explanation, that's very interesting to read.
I think our use case is probably quite simple overall compared to some others, but am happy to discuss it. Thanks

past nebula
#

Been looking for an explanation of where the cloud cache feature is!

@pure beacon is it planned to be a paid customer feature only?

pure beacon
#

@past nebula originally it was an open beta for paid customers only. Now it's a closed beta, because it doesn't yet meet our bar for open beta (distributed caching is hard to get right).

The problem of distributed caching is still one of our top priorities. We want to make sure that when we ship a solution, we really nail it.

In the meantime there are various stopgaps:

  1. Self-hosting and making your engines more persistent
  2. Self-hosting and distributing storage for your engines
  3. Use managed hosting from one of our "Dagger powered" specialized partners. For example if you need persistent cache in the context of Github Actions CI runners, we partner with depot.dev
  4. <coming soon> 🙂