Rebuilding even when the source code hasn't changed

[hoary niche]

Hi, I have a question. If I wanted to make my Dagger pipeline recognize when the source code hasn't changed and therefore skip the build stage and use the previous artifacts, is that possible? I am using GitLab CI/CD with Dagger for my project.

[forest tusk]

Hi, this should happen automatically via Dagger caching. You are probably getting your dagger cache wiped by the ephemeral Gitlab runners.

We are working on a distributed cache service to fix that. Let me know if you are interested in early access

There are workarounds in the meantime. May I ask if you use self-hosted runners, or gitlab.com runners?

[hoary niche]

Thank you, that sounds great!

[hoary niche]

I am using self-hosted runners

[forest tusk]

so in the context of self-hosted runners, you can get a little more mileage out of the local cache, by running the dagger engine in a separate “sidecar” runner. In kubernetes you can do this with a daemonset. Then your cache will last as long as the machine running your gitlab runner pod. Still not perfect but a first step in the right direction

With gitlab runners you want that daemonset configuration anyway, to avoid the complications of docker-in-docker

[hoary niche]

Thank you! How would I go about running the dagger engine in the runner?

[forest tusk]

We're still working on documenting it...

• Here's a draft "operator manual", still rough but has lots of useful information: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md

• For Kubernetes configuration specifically, we haven't published the configurations yet, but they are deployed in production for a few early adopters. If you want, we can get on a Zoom call with you and walk you through it. It would be an opportunity to learn more about your use case, too, if you're open to sharing. Learning about real world use cases is our favorite part 🙂

[hoary niche]

I would like to get on a Zoom call with you regarding the project! Currently, this project is using the docker GitLab CI runner and only running on one machine (our project is a lot simpler), but I am interested in exploring the possibilities. Would it be okay if I loop in two of my colleagues to the Zoom, if possible? One of my colleagues is @formal dew, who is also active on the discord and recently posted a question related to our setup!

[forest tusk]

Of course, bring anyone you’d like 🙂 @hasty nacelle will be in touch to coordinate

[hasty nacelle]

@hoary niche and @formal dew I've sent you Discord friend requests so we can get a DM going and coordinate a Zoom

[near burrow]

Hey @forest tusk I've been trying to get dagger to work with CircleCI's docker layer caching feature and can't figure it out 😅 would be very interesting in being an early adopter for an easy distributed cache service from dagger!

[forest tusk]

Would love to show you a demo! Paging @hasty nacelle 😁