Dagger Cloud long wait on `docker stop`

[slate sand]

Hi, thrilled about daggerizing my pipeline and love the distributed cache offering in Dagger cloud. Thank you!

The one issue I'm struggling with is setting an arbitrary 5 minute wait before stopping the docker daemon after my dagger action runs on an ephemeral runner in Github Actions. One of my goals is to speed up my piplines and adding 5 minutes, even though it's only rarely necessary, is hard to accept. Will you be adding a feature that pushes layers more frequently to Dagger Cloud through the pipeline process? Will the dagger soon block itself exiting to take only the time it needs to finish uploading cache before completing?

[north sequoia]

Hi @slate sand 👋 The 5 minute timeout should be the maximum it can wait before just killing the engine, however the actual time should only be what's required to make sure the cache is safely synced to cloud. This should be very short, so if you're hitting the full 5 minutes there might be something else going on!

To clarify a bit more on the cache lifecycle - the cache is also synced periodically. So if you have a longer running pipeline, the cache will be pushed throughout the execution as well. In really short pipelines, there's usually not enough time for this to happen, so the docker stop -t at the end is a safeguard to make sure everything completes properly. For long running engines, like in a kubernetes daemonset deployment, this is not necessary.

[slate sand]

Hi @north sequoia , Thanks so much for responding. That's good to hear. I'm actually hitting > 3 minutes consistently even though little is changing. In the last run I did, I only changed the workflow and still hit > 3min for engine to stop. I think the issue may be that I have 5 services running and am testing the whole stack with Playwright in Dagger. The long shutdown may be due to docker trying to cleanly shutdown the services, which is something I would rather it did not do. Kill them hard! Would there be a way (now or in the near future) to tell dagger to wait until the cache is finished uploading before exiting?

[north sequoia]

Would there be a way (now or in the near future) to tell dagger to wait until the cache is finished uploading before exiting
We have a lot of cache features coming! Both on the cache control side and visibility side

For services, as soon as the dagger run process finishes, the services will have been killed. The docker stop at the end is purely for pushing cache. Are you using cache volumes in your pipeline?

[slate sand]

Yes, lots of cache volumes. Node modules, ruby gems.

[north sequoia]

Got it! My guess is that's what is taking so long to push at the end. Generally layers wouldn't take much time, especially if you don't have a lot changing in your code. With zenith ( #daggernauts ), the next big update for Dagger, we've been thinking a lot about how you'll want to control the cache better and also think about volumes in a different way (this part is still mostly theoretical 🙂 ).

For today, I think there's a few options for you:

• reduce the timeout to something you're happy with. The volumes might not finish syncing, but maybe this is ok in your case
• Disable the volumes in cloud. This will make the sync time basically 0. If you have multiple parts of your pipeline sharing volumes, they'll still benefit from each other, but only within an execution. This is usually not ideal for dependency caches like you mentioned, but it depends
• If the 3 minutes ends up being less time than the other options, keep it as-is and we'll have some more controls and optimizations on the way!

[slate sand]

Ok, thanks Kyle. Might switch back to Github Actions cache instead of Dagger cache volumes until you have some incremental/differential/DAG volume upload capabilities. Or, even the ability to read a volume without writing it, so I can initialize pnpm packages and Ruby Gems from a "mostly complete" state.

Is there a way to turn on some debugging to see what cache is being uploaded and how long it takes?

[slate sand]

Actually, it just occurred to me that I could use a mutli-stage Docker build file with a dependencies stage and skip mounting the cache volume. Then the dependencies would be cache by layer. It would mean a full rebuild when there were changes to packages, instead of a warm build, unless I could somehow specify a cache-from. But, at least it would save time on uploading the full cache every time. If possible, I would still like to see which cache uploads are taking up the time if there's a way to turn on debugging.