module initialization is taking to much time in dagger-ts-monorepo, here is code

[frozen pivot]

I have an open-source repository (https://github.com/StaytunedLLP/dagger-ts-monorepo) demonstrating a Dagger monorepo with TypeScript. It contains a frontend module, a backend module, and a shared module orchestrated in a DAG (directed acyclic graph) manner. Currently, running the following commands takes 3-5 minutes each time due to slow initialization of the module:

# dagger-ts-monorepo
## Frontend module
dagger call run-node --source=../ 
## Backend module
dagger call run-node --source=../ 
## Shared module
dagger call run-node

How can I improve the performance and reduce this initialization time?

[teal moon]

There are several ways in my understanding and experience:

1. Use views.
2. Use a combination of defaultPath and ignore pragmas (Context Directories).

Also, depending if you're using WithDirectory or WithMountedDirectory, with the first one, you can provide an extra configuration object to filter certain files, and directories (it's not feasible with WithMountedDirectory).

With views, in your dagger.json file you can do something like this

{
  "name": "terragrunt",
  "sdk": "go",
  "exclude": [
    "../.direnv",
    "../.devenv",
    "../go.work",
    "../go.work.sum",
    "tests",
    "examples/go"
  ],
  "source": ".",
  "engineVersion": "v0.13.3"
}

Where the Exclude attribute does the filtering for you.

With the pragmas that I've mentioned above, you can do something like (assuming this is a parameter or argument of one of your dagger functions, that expect a dagger.Directory)

    // source is the source directory that includes the source code.
    // +defaultPath="/"
    // +ignore=[".terragrunt-cache", ".terraform", ".github", ".gitignore", ".git", "vendor", "node_modules", "build", "dist", "target", "tmp", "log"]
    source *dagger.Directory,

When you've optimized what's the content of your dagger module, I'd review what's cacheable in your scenario, and use cache volumes.

[sharp acorn]

i tried out the modules in the repo, my observations are below

# backend module
# first run ~40s
~/dagger-ts-monorepo/src/backend$ dagger call -m modules run-node
✔ connect 0.3s
✔ initialize 27.1s
✔ prepare 0.0s
✔ modules: Modules! 0.0s
✔ Modules.runNode: String! 12.5s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export GOAWAY=1

-------------------
Add :  3
Subtract:  -1
-------------------
# second run ~14s
:~/dagger-ts-monorepo/src/backend$ dagger call -m modules run-node
✔ connect 0.3s
✔ initialize 2.4s
✔ prepare 0.0s
✔ modules: Modules! 0.0s
✔ Modules.runNode: String! 11.7s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export GOAWAY=1

-------------------
Add :  3
Subtract:  -1
-------------------

# frontend module
# first run ~37s
~/dagger-ts-monorepo$ dagger call -m src/frontend/modules run-node
✔ connect 0.6s
✔ initialize 26.8s
✔ prepare 0.0s
✔ modules: Modules! 0.0s
✔ Modules.runNode: String! 9.7s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export GOAWAY=1

-------------------
Hello :  Hello, frontend!
-------------------

# second run ~11s
~/dagger-ts-monorepo$ dagger call -m src/frontend/modules run-node
✔ connect 0.2s
✔ initialize 1.5s
✔ prepare 0.0s
✔ modules: Modules! 0.0s
✔ Modules.runNode: String! 8.9s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export SHUTUP=1

-------------------
Hello :  Hello, frontend!
-------------------

# orchestrator module
# each run ~30s
~/dagger-ts-monorepo$ dagger call  run-node
✔ connect 0.2s
✔ initialize 2.3s
✔ prepare 0.0s
✔ daggerTsMonorepo: DaggerTsMonorepo! 0.0s
✔ DaggerTsMonorepo.runNode: String! 26.8s

Setup tracing at https://dagger.cloud/traces/setup. To hide: export GOAWAY=1

[object Promise][object Promise]

it seems ok to me, i see the second run is significantly shorter than the first because of caching.

this is on an ec2 small vm with only docker and dagger installed so not a very high-power machine

[frozen pivot]

@sharp acorn your response helped us to solve the things, will let you know if we face something.
Thanks a lot for your support.