Initialize step in CI is slow

[drowsy agate]

Hello, I've been loving dagger so far! My issue is that while I've solved my slow prepare speeds locally with views, my CI builds still take a while to initialize:
8 : initialize
9 : resolving module ref
9 : resolving module ref DONE [0.2s]
10 : installing module
10 : installing module DONE [54.3s]
11 : analyzing module
11 : analyzing module DONE [0.3s]
8 : initialize DONE [54.8s]
I've already made it past connecting to the engine, so it seems like this is just initializing my dagger (golang) module-is that correct? If that's the case it seems like a long time since I only have ~1.5k lines of code, and one depencency (I'm going to remove this and test that way), and a go.mod file just under 50 lines. Locally this step takes 2-3 sec. Is there any way to speed this step up in ephemeral CI runners on GitHub?

Removing the one dependency I had took it down to 39, but that's 33% of my build time right now, so any tips welcome!

[dusky chasm]

Hi @drowsy agate,

• Are you able to share a Trace url? You'll need to hook up your engine to a free Dagger Cloud account. Would help us drill down together

• What's the performance like when running the same thing locally? It's possible that lack of persistent cache is the culprit

[tropic kite]

I'm also experiencing long initialisation times using the Go SDK. Looks like it spends almost all of its time in codegen? Total length of initialise for this run was 1m12s. I'm running on GitLab CI with the Docker executor.

[zinc flame]

👋 Tom! are you currently using includes or excludes in your dagger.json? I'm asking because if you add an includes clause in your dagger.json that only adds your dagger pipeline code, initialization should be faster in this case

Additionally, does your pipeline has a lot of Dagger module dependencies?

[tropic kite]

The pipeline has no dependencies and just the auto-generated dagger.json

[zinc flame]

ok, thx! if you have some time could you try my suggestion above see if that helps to speed the initialization times? 🙏

[drowsy agate]

Hi there, thanks for your help!

• I'll look into traces, but not sure it'll happen quickly. I'd like to but still trying to sell dagger fully to the whole team.
• Locally I previously had issues with the prepare step taking 30s-1m each time. Now it only takes a long time after I've cleared my cache as the view seems to speed things up after an initial run. I've never seen this kind of prepare time in GH ephemeral runs. On the other hand initialize locally never seems to take more than 12s after a cache clear, while its consistently ~30s in CI.
• I don't have any includes in my dagger.json, only excludes (normal stuff-dist dirs, node modules, etc.) and a view which has functionally the same thing with ! before each pattern entry to similarly exclude it.

I think what I'm mostly asking (and am confused by) is the cause of disparities in times between local and CI for the init step. Prepare never takes long in my CI runners, but it was greatly improved locally with views. My guess here (which I'm somewhat confident about) is that my local node_modules, dist dirs and such were slowing down loading the dir initially, and views allow a quick way to skip those when grabbing my root directory. On the other hand the initialize step never takes long locally (10-12s on a fresh cache run), but is pretty consistently around 30s in CI. This part I don't really understand: what are some differences that could cause this? Does the fact that I have go modules installed locally matter? (guess is no) What else could cause a significant difference in init time between my local system with the cache wiped and a CI runner? Could this just be a case of better network/compute on my local machine?

The init time in CI is not a blocking issue for me, but just having the understanding of why it might perform differently in these different scenarios is really what I'm hoping to gain!

[dusky chasm]

@willow sage my guess as to why initialization is slower in CI, is lack of cache persistence between CI runs. This is very common in a vanilla Dagger install on a vanilla CI runner. Your ephemeral runner runs a new dagger engine each time, with an empty local cache each time.

By contrast your dev machine has perfect caching persistence

This is where Traces would really help.. To be clear you can create a free account, and also perhaps you can start by simply tracing your local runs, on dev machine, to get an idea of the performance profile of a local run. Half of the equation would be better than nothing

[drowsy agate]

So as far as ways to speed it up: 1. run on a CI runner with a cache (definitely something I'm looking to in the future, when I have time 😛 ) 2. keep go modules reasonably tidy and only include dagger dependencies when absolutely needed?

[dusky chasm]

Confirming that Dagger modules are fully sandboxed, so anything installed on your machine, other than dagger itself, has no incidence on how your Dagger module will run

[drowsy agate]

is my understanding that init times are generally related to compiling my module correct? If that is the case it seems like I'll just have to deal with the times in CI until I'm ready to get a runner with a persistent cache, keeping in mind to keep my module as slim as possible to help out a bit.

As far as the cloud setup I'll try to get that going at least for my local runs tomorrow. Not trying to be difficult here just have to get a sign-off for this and can't do that today unfortunately!

[dusky chasm]

of course no worries! sorry that you have to think about this in the first place, ideally it would be seamless

[dusky chasm]

Yes, unfortunately most of the overhead is probably not in the size of your module code, but in building the runtime container that wraps it. Certainly if you import many external dependencies that won't help, but that's usually not the case anyway

[drowsy agate]

Not sure where the fits into the roadmap but it'd be cool if it were possible to dump traces locally somehow. Also completely unrelated also but writing custom output/logging for functions would be cool too-I love the TUI when debugging, but my team finds it a bit verbose in CI output.

[drowsy agate]

Yeah I did see evidence of this as I removed a dagger dependency I didn't need and it went from ~1m to init per run down to the ~30s I'm at now. Go modules don't seem to scale quite as quickly as I add them, presumably because its not a whole separate container to build.

[dusky chasm]

Another possible outcome is that we ship features that provide relied even for fully ephemeral runners.

• We're already working on object storage support. Still an infrastructure requirement but at least wouldn't require touching the compute, which often makes things easier

• Also perhaps we should consider supporting a "vendored cache" or something of that nature, where parts of the runtime build process are pre-baked and vendored in your repository.. Not something currently being developed but given the prevalence of "slow init" problems, maybe we should consider it. cc @reef grove @velvet valve @zinc flame @blissful dagger

Certainly speed is becoming a focus for us so you should expect improvements

[drowsy agate]

And just to sandwich the whole conversation I have really been enjoying dagger as a whole! The amount of productivity I gain by being able to run my pipeline functionality locally has been a huge boon. Thank you for writing a cool tool and for the hands-on help!

[zinc flame]

@drowsy agate is there a chance you could share with us a Dagger Cloud trace so we can better understand where the time is going during the module initialization in your specific case? 🙏

[drowsy agate]

Yes, soon hopefully!

[tropic kite]

Hello, chiming in again to ask about the caching. Is there any examples (or if it is even possible) of how we could utilise native caching mechanisms for certain CI tools? For example with Gitlab CI, they offer a run caching mechanism similar to artifacts (using the cache keyword). Is there a directory we can supply and a sensible key to provide for this? I presume that this isn't really possible if all of the caching is done using OCI artifacts?

[drowsy agate]

I would assume you could cache local docker builds, volumes, and images, but you might not save much because you'll still be network limited.

[dusky chasm]

There is a feature currently in development, that offers exactly what you're talking about @tropic kite @willow sage ("here is a directory, persist it at your leisure using your CI's native caching mechanism"). But for now, it only covers cache volumes. Those are the directories you can ask Dagger to persist on a case-by-case basis, to allow tool-specific caches to accumulate. So typically you would find node modules, go module cache, linux distro package cache, stuff like that.

So that will definitely help accelerate.

Then there's the other half of the problem: layer cache. That is about caching Dagger's own operations, the same way docker build caches its layers. Under the hood it's the same tech. To persist that in an ephemeral CI runner, there is another feature we are developing, that is a bit further out (but still high priority), that will allow you to use an object storage service (S3 or equivalent) to persist all your layers seamlessly. In the meantime there are stopgaps. Which stopgap is applicable to you depends on your constraints.

[drowsy agate]

Those features sound good to me. I didn't get to dagger cloud today but intend to. Hopefully tracing will reveal exactly what kind of improvements I should be looking at!

[blissful dagger]

Had this in my queue to get back on but just found Solomon already summarized it perfectly 😄 I'll just add that:

• There's a high-level tracking issue here: https://github.com/dagger/dagger/issues/8004
• Part of the effort is not only to add better support for controlling cache storage but also to make the overall caching system more understandable and debuggable, which will indeed likely tie into our tracing support (details TBD)

[drowsy agate]

Ok I (finally) got my cloud traces going. I'm seeing the image above as part of the codegen step as the bulk of the time in initialize, so it looks like this is just because I don't have a cache for the directory in my CI runner. I don't think there's much else to do other than wait for the features Erik mentioned. Thanks for your help everyone!

[dusky chasm]

Thanks for being patient with us, this is high on our priority list, so I'm hopeful that it will improve in the near future

[drowsy agate]

Back at ya! It was a busy week so just finally got to setting up traces.. 😅

Dagger has helped me immensely for all of my linux builds and making what would be complex in CI provider yaml templating simple with golang...I'm actually in the market for the closest analog to dagger for my Windows platform stuff, because right now its a hot yaml mess...

[dusky chasm]

We actually started discussing expanding support to native platforms beyond containers... Not trivial, but between 1) Mac and iOS pipelines, 2) ML pipelines and 3) native Windows pipelines, we may have enough demand now, and a technical foundation robust enough, to consider a few careful steps in that direction

[drowsy agate]

This would be outstanding!

[dusky chasm]

@willow sage can you tell me more about your use case? What do that hot yaml mess actually do? What tools do you use and how do they interact with the host system?

[drowsy agate]

Nothing unexpected for normal CI/CD type stuff: get/bump the version, tag on main, setup VS toolchain stuff, restore, build/publish, and push that artifact somewhere. The trouble is I need to be on Windows (VS toolchain for UWP required) and I just don't have a good way to avoid having all my powershell commands just sitting in a yaml step list. Versioning can only be reproduced locally by copying/running those commands, and there's no way to get a clean build env locally easily.

The biggest pain point with this setup is just that its almost never obvious to a developer or even sometimes to me what's different about the CI machine vs local, and there's no good way to "run the pipeline" locally in a way that's even remotely sanitary. Complete opposite of dagger, where I haven't seen any significant differences between local runs and CI runs. So really what I want the most for Windows platform builds is that reproducibility so I'm not pushing and praying, but even more importantly so the win devs we have can run the pipe locally with confidence.

I dunno if that means win containers or what...I haven't worked with win containers in a while and it wasn't trivial when I did, but I'm not sure how else to get a relatively sanitary local environment.

[dusky chasm]

If I were to lift the whole thing into a linux container with powershell installed... what would break?

1. Calling the VS toolchain
2. anything else?

Are you able to copy-paste a snippet of the relevant commands? I'm interested in any example of the stuff that would break in a linux container. Since the crux is the issue is punching wholes through the sandbox.

[drowsy agate]

I actually thought about this since they did bring most dotnet stuff into linux, but yeah we do have a hard dependency on being in windows when building.

[drowsy agate]

The commands are just dotnet/msbuild, but the solution files and project files have dependencies that do require the win platform.

If we didn't have those hard dependencies on the win platform I probably would just build for win in linux.

[dusky chasm]

Any chance you could share a snippet?

[drowsy agate]

Of what the .proj file stuff?

[dusky chasm]

not sure 🙂 whatever could help me better understand the project, so I can think about how we might get it to work in dagger

[velvet valve]

Interested in the restore step too. Other windows folks mentioned that.

[drowsy agate]

I'm not sure this is really what's locking me to Windows. The projects themselves may be able to be actually built in linux, but we're using WiX and: https://github.com/wixtoolset/issues/issues/4381#issuecomment-244781910 so its kind of a nonstarter for thinking about moving to linux.

[velvet valve]

Oh, I see

[drowsy agate]

Yeah restore is just like npm i and I can't imagine its gonna be what fails on linux unless it does some kind of platform checking based on your project, but WiX is the one dependency I identified immediately as being the thing that would force me onto Windows.