we're interested in tryingdagger in kube. we're using buildkite. cloud is not an option for now (procurement). i was thinking of building a warm engine cache disk periodically and starting the engine with that just so that for example we avoid expensive clone, and start with some warmed build caches - would that be practical or difficult?
#i want to use kube engine daemonset without cloud preloaded with warm cache. possible/hard?
hallow jewel
bronze swan
yes, you can do this
we have a kubernetes guide in our docs you can use which provides most of the heavylifting
hallow jewel
the pre-warming part in particular... how would i got about pre-warming the engine?
if i had a cron job running offline that connected to an engine, build a bunch of stuff and created a bunch of cache volumes - is there a specific folder i would need to snapshot to create a warm disk image for future deamonset pods to startup with ?
bronze swan
the pre-warming part in particular... how would i got about pre-warming the engi...
yes, you need to make move around /var/lib/dagger to keep the engine warm
@hallow jewel if we could provide a way where all the caching could live in an object storage within your organization, is that something that could work or for procurement?
you'd still need to reach dagger cloud for orchestration reasons, but all the caching data won't ever leave your datacenter.
cc @lime thistle
hallow jewel
@bronze swan glad you asked was going to ask about this
(and thanks guys for being so response - this is great!)
is there end to end encryption?
like are the bucket contents encrypted?
it could be ok. it might depend on how it works exactly though.
we use buildkite already so in principle we're ok with the approach.
but it would still take a long time to get onboarded
and id probably want to see us getting a lot of value out of this even without your distributed cache
for us the path would need to be: this is fantastic and works great on its own. and wow now with distributed caching its even more incredible.
bronze swan
is there end to end encryption?
yes, there is
bronze swan
like are the bucket contents encrypted?
when we host your data yes. In the case you use your own object storage, that'll depend on how you want to configure it
bronze swan
and id probably want to see us getting a lot of value out of this even without y...
that makes total sense. It'd be great to setup a PoC using our distributed cache also so you can show upper management the benefits of it vs having to warm up and manage the state yourself which is quite ineficcient and time consuming
bronze swan
for us the path would need to be: this is fantastic and works great on its own. ...
SGTM! let us know how we can help
rain crypt
Hey, I also interested in such feature, is the only thing is blocking me to use dagger instead of earthly, remote cache but in our local S3/docker reigstry or whatever thing
bronze swan
Hey, I also interested in such feature, is the only thing is blocking me to use ...
great to hear @rain crypt ! Let me ping @lime thistle so we can schedule something together and manage to set that PoC. Just wondering, how are you using earhtly today? I wasn't aware that they supported some sort of on-prem version for distributed caching
rain crypt
Hey @bronze swan
not actually using it but I did a PoC month ago, it works pretty nice but I don't feel confortable with their docker like language, I think with dagger I have more flexibility in the future.
earthly Is not actually like distributed caching but they use internally buildkit cache from/to, so on my PoC the GH actions where my builds are running used to push to a local registry the cache, so then other executions on the GH agents or even developers on their laptops can reuse the cache and improve a lot build time
In general we have mono-repos with multiple components, so if I just change a readme or a tiny component I don't need to rebuild all, that is the reason of wanted to pull the cache,
In the future I can consider bring my own persistent dagger agents and this problem is gone but I don't want to maintain machines at the moment, so I happy with 8vcpu GH enterprise agents, that is the reason why is important for us cache system, as each execution is running in a clean VM
I also exploring other solutions like bazel, nix, please... and are highly amazing specially nix, but those even that work flawless are pretty complex so I don't think our company developers are going to adopt those solutions, so I need something dirty cheap to learn and implement so I can introduce it on my company, our devs are basically java and go, and with the concept of dagger modules and can me convined even different languages I think I can prepare a nice scalable platform SDK within the company.
BTW thanks for this amazing product and happy new year to everyone!
bronze swan
earthly Is not actually like distributed caching but they use internally buildkit cache from/to, so on my PoC the GH actions where my builds are running used to push to a local registry the cache, so then other executions on the GH agents or even developers on their laptops can reuse the cache and improve a lot build time
oh right, that's using the default buildkit cache exporters. Dagger also supports that natively if you want to start playing with that until we can setup the Dagger Cloud on-prem PoC. Take into account that using those exporters have some limitations (#1182427544448942170 message) that will start showing quite early for medium to large projects.
In general we have mono-repos with multiple components, so if I just change a readme or a tiny component I don't need to rebuild all, that is the reason of wanted to pull the cache,
yes, this makes total sense.
I also exploring other solutions like bazel, nix, please... and are highly amazing specially nix, but those even that work flawless are pretty complex so I don't think our company developers are going to adopt those solutions, so I need something dirty cheap to learn and implement so I can introduce it on my company, our devs are basically java and go, and with the concept of dagger modules and can me convined even different languages I think I can prepare a nice scalable platform SDK within the company
Agree, nix and bazel are also great projects, but as you mentioned, the learning curve and complexity is a showstopper for many teams. That's the main differentiator of Dagger. We think that a great CI tool needs a great community behind it, and the main way to get there is to attract as many devs as possible. Hence the reason for Dagger's design.
BTW thanks for this amazing product and happy new year to everyone!
Thanks to you for joining the community and looking forward to helping you succeed in the journey of improving your team's day to day work. Happy new year to you too!
rain crypt
@bronze swan I only saw references in dagger docs cache from/to for the deprecated dagger CUE cli, where I can see an example of the buildkit cache exporter in dagger? the benefits of dagger cloud are evident, just for the async cache push worth it. but as you said to start working on it, if you can refer me to any doc can help me to configure the buildkit exporter is a nice entrypoint, thanks!
bronze swan
<@336241811179962368> I only saw references in dagger docs cache from/to for the...
we don't have official docs for that since it's an experimental feature. Having said that, if you look for _EXPERIMENTAL_DAGGER_CACHE_CONFIG in Discord you'll find multiple discussions about that like this one: #kubernetes message