What's the recommended way to get fast builds on CI?

[muted nexus]

Hi all, I'm having a wonderful time with dagger on my local machine, it's insanely fast. But then I can't get to have actual caching between builds on Gitlab (or Github or any CI).
What should I do? how can we have similar build times with dagger than with classic docker remote cache using "--cache-from" ?
Thanks!

[serene aspen]

cc @dawn crown

[valid tusk]

Jeremy can give you more details, but here's the short version: you basically have 2 options:

Option 1: hack a solution with buildkit cache export (similar plumbing to --cache-from etc). This is possible, but not great. Better than nothing, but not by much.

Option 2: hook up your engines to Dagger Cloud, which provides seamless caching 🙂 Just one problem: Dagger Cloud is still in alpha - launching later this year. We can give you early access though, it has production users already.

There are several reasons that using Dagger Cloud for caching works better than a DIY solution with export-cache. Some of them have to do with implementation - for example we transfer cache data continuously, instead of in a separate export phase at the end of the pipeline; we also persist cache volumes; whereas with export-cache they get wiped in between runs. This is a big deal because persisting your go cache, npm cache, etc. is a big factor in pipeline duration.

There is also a fundamental reason: caching across multiple machines is an orchestration problem. To do it efficiently, you need a central control plane to orchestrate the movement of cache data across the whole system. A stateless architecture (a bunch of engines + dumb data storage) can never be as efficient. This is why we decided to add caching to Dagger Cloud in the first place: it's the only way to deliver a great caching experience in production.

[muted nexus]

Yeaaaah Solomon 🎉 (ok sorry).

Thanks a lot for your answer. It's what I was afraid of (use our cloud solution to get it as fast as your local machine, on traditionnal CI you won't be eable to, juste like Earthly).
But I really appreciate the details and what the differences are vs exporting buildkit cache.

I would love to try Dagger Cloud but I suspect it's hosted on AWS/GCP/Azure? I'm not a big fan of GAFAM and I won't give them any money. But please if it's hosted elsewhere I would really like to try it.

[valid tusk]

One big difference with Earthly (assuming you mean their Satellite or CI product) is that we don’t host the runner machines: those run on your infrastructure. We only host the control plane and cache storage.

(maybe Dagger Cloud is a misleading name?)

[muted nexus]

ok so I can choose my provider for the runners, very nice 🙂

[valid tusk]

yes exactly

[muted nexus]

Just like Buildkite + cache ?

[oblique pine]

Sounds like Dagger Cloud is coming along nicely if you've got users up and running. @valid tusk who do I talk to about getting access?

[valid tusk]

@dawn crown is your guy 🙂

[valid tusk]

Does buildkite host their own caching service? if so, then yes

[muted nexus]

no no that's why I said "+ cache", Buildkite is just the UI + Control plane, and agents run on your own infra. There is no caching orechestration, so it's great with a single runner but after that there is no cache distribution.

I've only seen Circle CI talking about docker layer caching and distributing, everywhere else cache is not mentioned as a specific feature.

[valid tusk]

Ah I see. Then yes. Our control plane lacks the job dispatching feature (because typically your existing CI runner will be sending work to dagger). But we will add that too eventually.

[muted nexus]

@dawn crown then 😀 is it possible to get an access to Dagger Cloud?

[dawn crown]

I'll ping you in a DM 👍