Ideomatic layer caching

[fallen shale]

I'm wondering if I'm thinking about this correctly.

The TLDR is that I have different functions performing yarn install, with different source files, but same files as far as the above command is installed.

To DRY, I created a shared function that does the install step and returns the container for the next relevant task (lint, build, test, etc).

For efficient layer caching, should I:

• Pass the full src to the install func and then filter to only include yarn related files, then call yarn install
• Filter the yarn related files before passing the the install func
• Neither, I'm way off course

I'm trying to avoid having the yarn install step cache-busted when unrelated code changes. I hope that makes sense! 🍻

[ashen gyro]

Hey!

I'm not sure we have function caching yet but your operation inside the function are cached based on the input.
What I would suggest is to use "pre-call filtering" to only get your package.json and yarn.lock, then proceed to yarn install and after that you can add your source.

Pre call filtering doc: https://docs.dagger.io/getting-started/types/directory/#pre-call-filtering

If you have a specific function for installation, that's the way to go.

However, if you want to send your directory only 1 time, then you can take the full dir in your function constructor and then do post-call filtering: https://docs.dagger.io/getting-started/types/directory/#post-call-filtering

It depends on what DX you want, the result should be the same 🙂

Since you have a generic install step, it may be simpler to do post-call, but it depends on your module's function

In the TypeScript SDK, that's what I do, I get the full directory but only keep the config file (package.json etc...), I install the dependencies and then I'll add the source.
This allows caching even if you change your codebase but not your deps

[fallen shale]

@ashen gyro thanks for the reply, sounds like what I'm doing is at least mostly right, at least in intent. In this case, I'm using pre-call filtering on the source, which is then passed here, filtered down again just to the relevant install files, then 'rehydrated' for lack of a better term back to the original source.

func (m *Example) nodeContainer(source *dagger.Directory) *dagger.Container {
    return dag.Container().
        From(NODE_IMAGE).
        WithWorkdir("/app").
        WithMountedCache("/root/.yarn/berry/cache", dag.CacheVolume("yarn-cache")).
        WithDirectory("/app", source, dagger.ContainerWithDirectoryOpts{Include: []string{"**/package.json", "yarn.lock", ".yarnrc.yml"}}).
        WithExec([]string{"corepack", "enable"}).
        WithExec([]string{"yarn", "install", "--immutable"}).
        WithDirectory("/app", source)
}

[ashen gyro]

yeah that's the right way to do.

Be carefull about the last WithDirectory("/app", source), if corepack or yarn for any reason update the package.json or lock file or anything (it shouldn't since you use --immutable but just in case), you may overwrite the files with that WithDirectory.

I would suggest to only included what you're using such as:

WithDirectory("/app", source, dagger.ContainerWithDirectoryOpts{
  Include: []string{"**/*.ts"}
})

So if you generate an artifact or something that isn't filtered, you do not invalidate your cache

[fallen shale]

Interesting. As an example, here's a lint function.

func (m *Example) Lint(
    ctx context.Context,
    // +defaultPath="/"
    // +ignore=["*", "!package.json", "!yarn.lock", "!.yarnrc.yml", "!tsconfig*.json", "!packages/**", "!**/*.ts", "!**/*.tsx", "!**/*.js", "!**/*.jsx", "!.eslintrc.json", "!.prettierrc.json"]
    source *dagger.Directory,
) (string, error) {
    return m.nodeContainer(source).
        WithExec([]string{"yarn", "lint"}).
        Stdout(ctx)
}

I can run dagger call lint several times in a row, with no other changes, and it seems 50/50 whether the install layer is cached. So you think that last WithDirectory might be the culprit?

[ashen gyro]

Do you have a trace on dagger cloud that you can share?

I can check what is cached and what's not

Up to the yarn install --immutable, it should be cached.