Dagger is slowwwwww

[turbid crag]

I have built a simple CI pipeline using Dagger for my Python projects which sets up the Python venv and then executes linting/formatting/type-checking steps (see attached code). Unfortunately, Dagger is unbelievably slow at times and I'm failing to understand why:

• The load module step at the very beginning of a dagger call -m . run-pyproject-pipeline --pyproject-dir . always takes >2s, sometimes as much as 17s(!) What is Dagger doing with all that time?!
• Extracting the venv folder from the container in get_venv() took > 8 minutes this morning (│ ✔ .directory(path: ".venv"): Directory! 8m17s), even though the Python project is rather small and only has a handful dependencies (mainly Pulumi & Pydantic). Once the step was cached, everything was fine but IMO it should never take this long.
• The typecheck .with_exec(["uv", "run", "pyright", "."]) takes 30-40s on my machine (uncached). Sure, Pyright is not exactly known for being fast but executing uv run pyright . on my command line only takes 3s.

Could anyone help me understand what's going on here and what I could do to optimize my pipeline?

[sonic mist]

@turbid crag is there any chance you could hook up your exectuions into Dagger Cloud and send us a trace URL where those executions are slow? Dagger Cloud accounts are free for a single user. Running dagger login in your terminal is all you need to set it up

[golden inlet]

First, it's possible you may be uploading more stuff than intended. Try running the following:

dagger -c '.core | module-source . | context-directory | glob "**"'

These are the files that get uploaded to the engine when your module is loaded. If you see .venv for example, then add it to your dagger.json excludes, like this: https://github.com/dagger/dagger/blob/ae3eb2163eb4e99d9849b77951b73c109fbe2ffc/core/integration/testdata/modules/python/git-dep/dagger.json#L4-L7.

Secondly, check if your pyproject-dir is being filtered as expected. You may be pushing more files than intended. You can terminal into a container and check for the mounted files, or have a function simply return the FilteredPyProjectDir, or make it a field.

Third, I wouldn't mount the .venv dir from get_venv into the container. Just extend the container with uv sync. May not make a difference, but there's no need to handle the venv manually so it can be simplified.

Finally, try running your function with an open session:

❯ dagger
Dagger interactive shell. Type ".help" for more information. Press Ctrl+D to exit.

⋈ typecheck .
⋈ typecheck .

Check the difference in time between those two calls. Notice it's not dagger call, which does the whole module loading and serving every time. While with Dagger Shell your module is loaded in the beginning and then you can make function calls with the same session. It will give you a better idea on what time is spent on the function itself.

[turbid crag]

Thanks so much for the great (and very quick!) feedback! I will look into each of your suggestions!

[golden inlet]

Let me also clarify that compared to running pyright on your host, Dagger needs to spin up a special container to run your module's code and call your pipeline, so there's a bunch of work until the engine actually executes pyright. If you set up Dagger Cloud like Marcos suggested, you can see what time is spent on each step more clearly. You can also increase verbosity in the CLI to show more steps as some are hidden in an attempt to not overwhelm.

[turbid crag]

@golden inlet

Third, I wouldn't mount the .venv dir from get_venv into the container. Just extend the container with uv sync. May not make a difference, but there's no need to handle the venv manually so it can be simplified.
I know uv sync is rather fast if the pip packages are already in the cache. However, as a general pattern I prefer building something like a venv or node_modules folder only from those files that folder really depends on (i.e. pyproject.toml & uv.lock, or package-lock.json, etc.). This is so that I don't bust the BuildKit cache prematurely whenever I touch anything else in my code.

[golden inlet]

That's ok, but you can:

• mount pyproject, etc
• run uv sync
• add the rest of the files
• run uv sync (if needed)
  There's no need to manipulate the .venv dir manually.

[turbid crag]

There's no need to manipulate the .venv dir manually.
So instead I will have to manipulate every other file manually, i.e. add each file to the container and thereby build up the structure of the project's root folder? I'm not convinced this would be a better trade-off (also in terms of maintainability etc.).

[golden inlet]

No, you can just mount the whole dir. You don't have to exclude the files that you've already added (pyproject.toml, uv.lock, etc).

You're already doing that with with_directory. I'm just saying you don't need the extra step of mounting .venv from one place to another.

[turbid crag]

You mean

self.get_base_container()  # This sets the cwd to /workdir
.with_file("./pyproject.toml", pyproject_dir.file("pyproject.toml"))
.with_file("./uv.lock", pyproject_dir.file("uv.lock"))
.with_exec(["uv", "sync"])
.with_directory("/workdir", pyproject_dir)

? I would have thought that the last step will either override .venv or error out or something.

[golden inlet]

Yes, pyproject_dir supposedely doesn't have a .venv that could override because it's in your Ignore list.

If you wanna be extra sure you can also add excludes and includes in with_directory (see optional arguments). But I don't think you need it here. Another option could be putting the venv in /opt/venv, i.e., external to the project dir, but no need for that also.

[golden inlet]

I would have thought that the last step will either override .venv or error out or something.

To be clear, with_directory doesn't replace a directory's contents. It copies contents from one dir to another. I.e., doesn't delete what the source directory doesn't have.

[turbid crag]

Quick status report: I followed @golden inlet suggestions and

• rewrote/simplified the code as per #1379046207527325746 message . This doesn't seem to have made any difference in terms of performance, though.
• looked at dagger -c '.core | module-source . | context-directory | glob "**"'. Indeed, I was uploading more stuff to the engine than needed (e.g. the .venv folder), so I adjusted the excludes in dagger.json accordingly.

As per @sonic mist's request I have also collected various traces. However, I did that before I adjusted the excludes (because I was working on other stuff in the meantime and simply bookmarked the traces whenever I noticed performance was bad – which varied over time quite a bit). I'll wait a bit to see whether there's any difference in performance now that I've changed the excludes and will report back!

[sonic mist]

🙏 let us know how it goes. As helder mentioned, performance should be way better now

[vast path]

Just another data point, I tried a repro of this and noticed that uv sync gives this warning:

warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.

Which makes sense since hardlinks can't cross mount boundaries on Linux. Not sure the actual perf impact but may explain part of the difference between perf on the host and in containers

[turbid crag]

@vast path Are you sure you used my code from above? Because in the code I explicitly set UV_LINK_MODE=symlink (which, unfortunately, didn't quite work as expected, so I'm back to UV_LINK_MODE=copy).

Either way, the copies produced by uv should still be cached by the image layer cache. So the copying only happens once. Afterwards, things should be as fast as always.