#Hi If I want to run the dagger engine
small smelt
I also know that dagger cloud aims to solve this problem and more. But how do people tackle this today?
uneven anchor
(moving my message to this thread)
Hello! It is possible to do some exporting like this, but the experience is not that great. Running a long lived CI runner is one option that will make things work like they do on your local computer.
However, using Dagger Cloud is the best way to take advantage of caching in a distributed environment with ephemeral builders.
This was the main reason why we built a globally distributed cache with Dagger Cloud: https://dagger.io/cloud
Before dagger cloud there was not a great solution for the distributed caching problem.
Also, before dagger cloud we experimented with using some of the built in functionality in build kit export but the experience was very poor.
small smelt
Please validate my understanding:
Previously the solution was to host the dagger engine yourself, possibly inside of kubernetes or elsewhere and have it be a long-running process that CI could connect to.
Now with Dagger Cloud, you can run dagger engines but have them connect to your cloud instance that will handle the exporting/reuse/caching of docker layers/artifacts for you (+ observability and more into your dagger processes)
Is that correct?
uneven anchor
Sorry I am confused with who "you and your" are in your second sentence 😄
Before Dagger Cloud caching, there was really no great solution for how to take advantage of cache in ephemeral environments.
Let me try to describe it.
Previously the solution was to host the dagger engine yourself, possibly inside of kubernetes or elsewhere and have it be a long-running process that CI could connect to.
Yes, but this is also still possible today. It's just not common to have a CI server connect to some other long running instance like this. Normally we see either long running CI builders like the old Jenkins model, or ephemeral builders like CircleCI and GHA.
For caching in non-ephemeral environments (i.e. your laptop, long running CI server, long running k8s pod) you have always been able to take advantage of the built in caching of dagger engine and this is still true today without Dagger Cloud.
Today, with Dagger Cloud, when a build runs in the same ephemeral environment the engine will take advantage of a globally distributed cache automatically.
The key point here is that Dagger Cloud is not hosting an engine. It is consuming telemetry and providing the cache service.
The way all of this works is that when the Dagger Engine runs and sees a valid DAGGER_CLOUD_TOKEN in the environment it will automatically send telemetry and take advantage of the cache.
I hope this is more clear!