Is there a way to declare ignore patterns for --dirParams?

[narrow fox]

@object()
class MyApp {
    @func()
    async dependencies(source: Directory): Promise<Container> {
        ...
    }
}

Following the pattern from the documentation, running dagger call dependencies --source=., but if the workspace contains e.g. node_modules which should not be copied into the context, it negatively impacts performance (potentially substantially so, adding minutes to every call).

How can this be resolved?

[wicked karma]

@narrow fox it's a two part answer:

1. This will be shipped very soon. See https://github.com/dagger/dagger/issues/7647

2. In the meantime there is an undocumented workaround. You can create a "view" of a directory by adding special configuration to that directory's dagger.json. Then use that view in the CLI argument.

for the "views" workaround, here's how we use it for our own CI:

https://github.com/dagger/dagger/blob/main/dagger.json#L20-L30

  "views": [
    {
      "name": "default",
      "patterns": [
        "!bin",
        "!.git",
        "!**/node_modules",
        "!**/.venv",
        "!**/__pycache__"
      ]
    }

Once you have a view called "default", you can run dagger call dependencies --source=.:default

[narrow fox]

Thanks!

[fading acorn]

there's also a third way by defining a global include and exclude field in your dagger.json that will apply those exclusions by default to all the commands that you run. https://github.com/dagger/dagger/blob/b213c4f88de0f4bfa164e047276d717024b78cb2/core/modules/config.go#L19-L23

Not sure if you intentionally left that one aside @wicked karma

[wicked karma]

that only applies to the module source itself I believe. Won't apply to directory arguments

(if it works differently, then I may have a flawed understanding of how that feature works)

[fading acorn]

whoops.. I might have misunderstood it then 🙌

[fading acorn]

just verified and this is correct. Sorry for the confusion

[narrow fox]

That's also good to know, though, and I wonder if it's also relevant to call startup time.

Following the quickstart tutorial, one is instructed to run dagger init in the project root, which results in dagger.json being in the project root and a subfolder dagger which contains the module's source. Is it redundant for me to place "include": ["./dagger"] in my dagger.json? My initial assumption was that call would only bring the dagger folder into the context, but the source code you just linked to indicates that these top-level include/excludes are relative to the dagger.json file, suggesting that the whole project is included both in the module itself and the --source= argument unless excludes are defined.

(confirmed that the :default trick reduces prepare time from 1m 30s to 20s)

[wicked karma]

Yes it would be redundant. Include rules for module loading vs. arguments work completely differently:

• When loading the module, dagger knows what it needs, so it can be smarter about include exclude (specifically it will include dagger.json + the source directory configured there (which defaults to dagger/ but could be anything, and is frequently changed by users to to .). Then it applies optional include/exclude from there.

• When loading an argument directory, dagger doesn't know what you need. So it has no choice but to include everything by default. Hence the need for views, and the future +ignore pragma which is basically a better version of views

[wicked karma]

@livid merlin is this correct? 🙂

[fading acorn]

that's what I understand also

[livid merlin]

Just saw this, yes exactly 👍

[olive bronze]

Okay, apparently this is what I was looking for. Reduced my prepare time from 5-10m (I can't tell which) to just a few seconds. I used to use the Rust SDK, and it allowed me to select out the paths I wanted, and I really didn't have any issues with that.

Is there any documentation available on the subject of views?

[wicked karma]

It's not documented, because it's a temporary stopgap. This feature which is about to be shipped, will deprecate it. It has all the same benefits as views, but is simpler to use https://github.com/dagger/dagger/issues/7647

[olive bronze]

I see. Well, nonetheless I think it really is useful. It reduced the waiting time from that much to just few seconds. Time is very valuable, and this issue seems to be open for a month. I think the workaround, nonetheless it is considered to be depreciated, should be mentioned somewhere. Maybe a FAQ (even here)?

[wicked karma]

Yes, we will continue to maintain it, to avoid breaking users that rely on it (like yourself). But we don't want to advertise it broadly, because the implementation of context directory is almost complete (literally days from merge)

[olive bronze]

Ah, I see. That fact changes everything I would really understand your decision about not publicizing.

[sterile vale]

Issue DOCS-308 created.

DOCS-308 - Ensure that we show the "old" and "new" way to "declare ignore patterns for --dirParams"

Check [#1251260501426442290 message](#1251260501426442290 message) for full context. Issue created from help forum post

Status

Triage

Assignee

Vikram Vaswani

Documentation · Jul 9, 2024, 3:46 PM

[simple elk]

I'm experiencing a similar issue to long upload times for one of our developers (that we know of)

The crux of it is that they have local files in locations we cannot predict, but they have a .gitignore they use to prevent it from being seen by git. They are experiencing 30m upload times... and as he is the tech lead, getting a lot of pushback against Dagger... :[