Anybody have a monorepo example: Typescript/yarn/node_modules caching

[distant leaf]

Came here because I'm fed up with gitlab yaml, but the leap from the quickstart to a multi-app monorepo is quite a big jump.

Does anybody have any examples of a monorepo? For example, I'm trying to learn how to:

• Creating the buildEnv at the top level and leveraging this for all apps
• Caching of node_modules in each projects (some node tooling we use forces us to use individual node_modules at the moment)
• Supporting 2 source folders (apps/ and packages/ -- both of which are used)

[night jasper]

What timezone are you in? Tomorrow, I'll have some free time for a few hours afternoon. Maybe I can help with that and it would give me an opportunity to hack on a real workflow (I'm planning to help with workshops in the future).

[distant leaf]

US East Coast (UTC-4)

[night jasper]

That won't work out well haha

[night jasper]

Let's try to do this async then. Beginning really is the halfway through a problem. What sort of tooling do you use? (P)npm workspaces? Yarn? Nx?

[distant leaf]

In short, we have package layout that looks mostly like this:

./.gitlab-ci.yml
./apps/landing/.gitlab-ci.yml
./apps/reviews/reviews-api/.gitlab-ci.yml
./apps/reviews/reviews-ui/.gitlab-ci.yml
./apps/stocks/stocks-api/.gitlab-ci.yml
./apps/stocks/stocks-ui/.gitlab-ci.yml
./apps/tnb/tnb-api/.gitlab-ci.yml
./apps/tnb/tnb-ui/.gitlab-ci.yml
./packages/pinnacle-layer/.gitlab-ci.yml

I have these stages:

stages:
  - prepare
  - module-caches
  - layers
  - build
  - deploy

Each package/app .gitlab-ci.yml uses rules to decide if they need to build.

In short, I need a blueprint of "here's the things in Dagger you can use to replicate the above"

[night jasper]

So, each of these are separate apps, am I right?

And one is a library, I guess.

What you can probably use there is modules. Each of the apps define their set of stages separately, then the primary dagger utilizes all.

Something along the lines of https://docs.dagger.io/quickstart/simplify

[distant leaf]

@night jasper -- thanks, but that's not really helpful, sorry:

• I'm not sure how daggerverse helps. I recognize the intent, but the implementation is now swamped with too many undocumented modules that a new user has no idea how to use. Example: Search for node -- how am I as a new user supposed understand which of these will help me or how they are different?
• I agree with the idea I need modules, but there are no examples of using modules locally. Can I somehow? There's one reference here that sort of talks about how to split files, but I don't think that's really a module
• I need 3 (I think) types of deployment modules:
  • deploy a serverless.com lambda to AWS
  • deploy a static website to S3 + cloudfront distribution
  • deploy a sveltekit application

These are now in my yaml like this:

.deploy:
  stage: deploy
  artifacts:
    paths:
      - ${MODULE_PATH}/.serverless/
    name: 'build_artifacts_$CI_COMMIT_REF_NAME'
  script:
    - cd $MODULE_PATH
    - yarn run deploy-${CI_ENVIRONMENT_SLUG}

.deploy-s3-cloudfront:
  stage: deploy
  artifacts:
    paths:
      - ${MODULE_PATH}/dist/
  script:
    - apk add --no-cache aws-cli
    - cd ${MODULE_PATH}
    - aws s3 sync dist/ s3://${CI_ENVIRONMENT_SLUG}.${DOMAIN}/${FOLDER}/
    - aws cloudfront create-invalidation --distribution-id ${CLOUDFRONT_DIST} --paths "/*"

.deploy-svelte:
  script:
    - apk add --no-cache aws-cli
    - cd $MODULE_PATH
    - yarn run build-${CI_ENVIRONMENT_SLUG}
    - aws s3 sync dist/ s3://${CI_ENVIRONMENT_SLUG}.${DOMAIN}/${FOLDER}/
    - yarn run deploy-${CI_ENVIRONMENT_SLUG}
    - aws cloudfront create-invalidation --distribution-id ${CLOUDFRONT_DIST} --paths "/*"

[night jasper]

I will have a look at this tomorrow. Thanks for the yaml file, that will definitely help. I would like to also mention that I am just a user here as well. ^^

[distant leaf]

LOL, apologies, still getting used to who's who.

[glacial rivet]

I have a multi-language mono-repo I'm working on, some things to note

• loading up all the source at once has caused long runtime issues when shipping the repo to the dagger engine (despite it all being local) There are several reasons for this, and after several experiements, we are not going this route any longer
• instead, we build smaller dag.Host().Directory() in our pipeline (.git, src/libs/<lang>, per-service)
• We have a few "builders" (Go functions) that take some metadata about the service and build up the buildEnv, depending on the language. We construct a cache that incorporates the service name & platform

[marble lynx]

@distant leaf I really appreciate your feedback on Daggerverse and I'm on the Dagger staff 🙂 (we've got a Dagger Team role on our Discord profiles)

I'm not sure how daggerverse helps. I recognize the intent, but the implementation is now swamped with too many undocumented modules that a new user has no idea how to use.
We have a new feature in Daggerverse for providing more detailed examples, but I see we have a lot of work to do there. We also have a quality scoring system that we need to continue to iterate on.

Example: Search for node -- how am I as a new user supposed understand which of these will help me or how they are different?
Can you share what would have told you that the module you found might be a good fit?

I agree with the idea I need modules, but there are no examples of using modules locally. Can I somehow?
When you say use them locally, do you mean calling them from your laptop or from CI? Does local mean that the module code is present locally vs in a remote Git server?

@distant leaf happy to pair with you now or whenever is convenient for you!

[distant leaf]

Hi @marble lynx -- I'm a bit blocked until this PR is merged, so I'm at a standstill. In short, can't use Dagger with yarn@4 until we suss out the sdk issues @vague ore is working on. Happy to pair, but not sure how much we can accomplish

[distant leaf]

Can you share what would have told you that the module you found might be a good fit?

[marble lynx]

Could you chat live briefly about daggerverse modules? Yes 👆 that sort of thing?

I'm actively working on that.

[distant leaf]

sure

[vague ore]

Is there any requirement or packages that force you to use yarn v4?

I'm curious

Should be good tomorrow though

[distant leaf]

It's generally better than 1.0. Supports workspaces better, turbo repo, etc

[viscid hare]

Issue DOCS-317 created.

DOCS-317 - add monorepo guide in blog or docs

see discord chat attached

Status

Triage

Documentation · Jul 24, 2024, 5:42 PM

[night jasper]

@distant leaf I am currently being very busy with work, more than usual, sorry. I might not be able to have a look at this as soon as you might expect. I hope it's understandable ^^