Slow in monorepo

[devout sorrel]

Please help. As I see the following issue is still unresolved: https://github.com/dagger/dagger/issues/8499. Is there a decent workaround for huge monorepos other than creating a git repository in the dagger module's folder?
It's now running very slowly; loading a module takes almost a minute. It's become a real bottleneck for using dagger.

[nimble jungle]

That issue is a bit out of date in that more optimizations have been applied since it was last updated.

Do you have a dagger.cloud trace link where you're seeing slow loads? That would be the most helpful in figuring out what's taking so long

[devout sorrel]

I don’t have a dagger.cloud trace link, and I’m not connecting to dagger.cloud because I’m working in a corporate environment (and in general, issues with very large monorepos tend to appear first in such setups). A few other people have mentioned similar problems in the issue as well, so it doesn’t seem to be isolated to my case.

There was also an attempt to improve this in https://github.com/dagger/dagger/pull/8953, but that PR wasn’t merged. The workaround suggested in the comment https://github.com/dagger/dagger/pull/8953#issuecomment-2521794740 doesn’t seem to work for me, and the exact steps aren’t very clear (it’s possible I’m missing something). Additionally, the comment here https://github.com/dagger/dagger/issues/8496#issuecomment-2474995411 didn’t appear to get a response.

Just to restate the core issue more clearly:
We’re working with a large monorepo (>10GB; and we’ll soon be moving to an even larger one, >100GB — I haven’t tested that yet). If a Dagger module lives somewhere inside this repository, for example foo/bar/foo/test, and I run a simple command like dagger call -h from that module, the LOAD MODULE stage consistently takes at least ~40 seconds.

As soon as I run git init inside the module directory (or even in its parent directory), the load time drops significantly to ~1–3 seconds. Because of this behavior, it seems likely that the slowdown is related to how Dagger traverses the filesystem and looks for git metadata.

If there’s any additional information or data that would be helpful, please let me know — I’ll do my best to collect it locally.

[nimble jungle]

Thanks for the info! There indeed must be something inefficient about the filesystem traversal, but in theory there’s nothing logically why it should be slow in this situation. So it’s more a matter of fixing a dumb bug.

The client-side of filesync has a bunch of legacy code from a different codebase, I will take a look.

[wet ore]

Thx for the input Mike. Is there a chance you could tell us with of the steps in the Dagger call is the one taking 40s consistently?

If you add -E to your call can you navigate the trace once it has finished and you can collapse/expand all the steps so you can pinpoint what's exactly the part that's taking considerable amount of time consistently

[nimble jungle]

Attempting to repro, I tried cloning the Linux git repo, doing ⁨mkdir -p some/subdir; cd some/subdir; dagger init --sdk=go --name=test --source=.; dagger call -h⁩ but that was fast (0.4s-0.9s for ⁨call -h⁩).

Possible that Linux isn't big enough, may try chromium or generating an artificial one.

Some more questions in the meantime:

1. What dagger version are you on?
2. What SDK are you using?
3. Can you share the ⁨dagger.json⁩ of a problematic module? Feel free to redact anything internal-only, I am mostly interested in where fields like ⁨source⁩, ⁨dependencies⁩, and ⁨include⁩/⁨exclude⁩ point to
4. Do you have a single dagger module in the monorepo? Or multiple with dependencies between them?
5. What client machines are you running this from? i.e. are you running the dagger CLI on Linux/macos/windows?

[nimble jungle]

Ok interesting, I was able to perhaps partially repro this. I had a script create an artificial monorepo with hundreds of thousands of files that total to 40GB in size and the time for ⁨dagger call -h⁩ went up to ⁨6.7s⁩. That's not nearly as bad as what you're seeing, but still indicates something is wrong.

(trace for that, does seem to suggest the host.filesync step is just taking up all that time despite not actually syncing anything https://dagger.cloud/dagger/traces/937ccf09e49cc65134cc69b3f21934f4?listen=2cfc3c60d848cbde&listen=bf3d5e606393fadc&listen=1325bf1a283285a1&listen=853f137b20435a92#1325bf1a283285a1)

[nimble jungle]

Ah fun, I adjusted the script to make all the files with size 0, but instead create 8 million of them in various shapes (wide vs. deep dir trees, etc.). Now loading takes very long, currently at 2m and it's not done yet. So that sounds much closer to what you were describing

[nimble jungle]

One more question: the monorepo in question does actually use ⁨.git⁩ in it's root correct? Might sound dumb, but wanna double check this is actually ⁨git⁩ vs. some other VCS, which would be highly relevant 🙂

[nimble jungle]

found something, this fixes it at least for me in my repro: https://github.com/dagger/dagger/pull/11799

[devout sorrel]

Oh wow, that’s a lot of questions 😄 Sorry for the slow reply — got a bit buried in work.

Going through them one by one:

1. The exact step where it gets slow
   This is where the delay happens:

   load module: <module name> 35.8s
     finding module configuration
       moduleSource
         ...
         directory(
           path: <path to git root folder>,
           include: ["<path to dagger.json>", "<path to module/**/*"],
           gitignore: true
         )
           filesync 35.5s
             copy
   

2. Dagger version
   Both engine and CLI are version v0.19.10.

3. SDKs in use
   Go and Python. We have modules in both, and the behavior is identical across all of them.

4. dagger.json
   Nothing unusual there. Example:

   {
     "name": "vault",
     "engineVersion": "v0.19.10",
     "sdk": {
       "source": "go"
     }
   }
   

5. Number and structure of modules
   Yes, we have multiple modules, located at different nesting levels (we’re planning to standardize this later, but haven’t gotten to it yet). There are also inter-module dependencies. The issue reproduces both for “leaf” modules and for modules that depend on other modules.

6. Operating system
   I’m running this on macOS locally. We also have CI “agents” on Linux, and the same issue exists there as well, but it’s even slower (often over a minute). I assume this is related to weaker hardware and the Dagger engine setup on those machines.

7. About git usage
   This is actually a very interesting question 🙂
   Git is not used at the repository root. We’re using a private VCS instead. However, in order to make module-to-module dependencies work, we had to git init an empty repository at the root. It’s not really used and doesn’t contain anything meaningful.

[devout sorrel]

The empty git repository is initialized at the root so that Dagger can resolve dependencies from anywhere inside the monorepo. As far as I understand, this behavior is currently not configurable, and Dagger requires git metadata to be present, which is why we had to resort to workarounds like creating empty git repositories.

If there’s a better or more idiomatic workaround for this, I’d really appreciate a hint)))

And thanks a lot for taking the time to look into this and for the attempt to fix it. I’ll try rebuilding the module on my side and will check how it behaves.

[devout sorrel]

I rebuilt the Dagger engine from your branch and ran a few tests. Here are the results (three consecutive runs):

Old engine: 50s, 34s, 33s
New engine: 30s, 18s, 18s

So the load time is almost cut in half. Thanks a lot — this is a very noticeable improvement on our side.

[devout sorrel]

Of course, it would be great to make it even faster 🙂 I still don’t fully understand what exactly is being loaded there, since the final copy step (inside filesync which takes 30s+) consistently shows 0s.

[wet ore]

Hey Mike, if you have a sec would you mind checking how the performance is when running this in a native Linux host? I'm wondering if the client being in OSx adds some considerable overhead

[fleet current]

Probably worth upgrading to v0.19.11 (just released) and trying again. There's been some major performance improvements

[devout sorrel]

Friends, after upgrading to v0.19.11 it started to fly 🚀
Both locally on macOS and on Linux the issue is now gone.

The initial module load takes around ~10 seconds, and the second and subsequent runs are almost instant (usually under 1 second).

Thanks a lot — everything looks great now. I think no further analysis is needed on my side, since your fixes seem to have fully addressed the problem 😊