#👋 Hi Just read the blog post about
proven musk
While I don't think that Dagger by itself made our builds faster, the benefits it provided us gave us a ton of opportunity for optimizing. All of the code that runs our dagger pipelines is available at https://github.com/grafana/grafana-build under an Apache 2.0 license.
Funny enough, when the article was written, our dagger pipelines were taking ~15 minutes, and then we started using the TUI and that gave us even more visibility, so we were able to reduce the amount to ~8 minutes during editing 🙂
from my experience, CI bottlenecks that can be improved have been around network usage and avoiding redundant tasks; something that we had a hard time realizing was happening before we introduced Dagger. Having the fast feedback of running locally and iterating as well as the visibility into what tasks were running and whether the cache was being used effectively thanks to the TUI is what really was able to get us to that 8 minute mark.
Regarding your question about the dagger engine cache, right now we actually haven't put any effort into a truly persistent cache; any caching we get is just because the CI node that was used in a previous run hasn't been killed yet due to inactivity lol. I imagine once we get a truly persistent cache set up we'll see even better performance gains. There's also an issue in that repository to make sure that when we run a certain container that we're only copying files from the Grafana code that we actually need so that we can take full advantage of caching.
What's done in those 8 minutes, by the way, is:
- Installing our Node and Go dependencies
- Building Grafana's frontend and its internal plugins (which are node.js programs)
- Statically compiling Grafana's backend programs for amd64 and arm64
- Creating a tar.gz containing all of our package content
- Uploading all of that to GCS.
