#How to prevent a folder from being loaded into dagger engine?

1 messages · Page 1 of 1 (latest)

misty salmon
#

I am using dagger to build a Rust and Node.js project in a monorepo setting, both having two very large (multiple GBs) cache folders (target for Rust and node_modules for Node.js)

So I am trying to use the .without_directory command on the host but it seems this is not working as expected (and I'm probably not using the right methods for this)

Thus the question.

Here's a very minimal example folder structure

▾ app/
  ▾ node_modules/
      very-big-files.xxx
    index.html
  .dockerignore
  build.py*

Here is the python script in build.py

#!/usr/bin/env python

import anyio
import dagger
import sys

config = dagger.Config(log_output=sys.stderr)
with dagger.Connection(config) as conn:
    hostdir = conn.host().directory("app").without_directory("node_modules")

    container = (
        conn.container()
        .from_("nginx:alpine")
        .with_file(
            "/usr/share/nginx/html/index.html",
            hostdir.file("index.html")
        )
    )

    container.export("test-container.tar")

And here is the output when running the script:

#1 resolve image config for docker.io/library/nginx:alpine
^R
#1 DONE 1.1s

#2 host.directory /Users/chakrit/Documents/prod9/dummydagger/app
#2 transferring /Users/chakrit/Documents/prod9/dummydagger/app: 27B
#2 ...

#3 from nginx:alpine
#3 resolve docker.io/library/nginx:alpine 0.4s done
#3 DONE 0.4s

#2 host.directory /Users/chakrit/Documents/prod9/dummydagger/app
#2 transferring /Users/chakrit/Documents/prod9/dummydagger/app: 247.16MB 5.1s
#2 transferring /Users/chakrit/Documents/prod9/dummydagger/app: 545.56MB 10.1s
#2 transferring /Users/chakrit/Documents/prod9/dummydagger/app: 817.70MB 15.2s
#2 transferring /Users/chakrit/Documents/prod9/dummydagger/app: 1.11GB 20.3s
#2 transferring /Users/chakrit/Documents/prod9/dummydagger/app: 1.47GB 25.4s
^C#2 CANCELED

You can see from the last few lines that GBs are being transferred when index.html is actually just an empty file.

desert mica
misty salmon
#

I seriously needs a rubber duck on my desk.

with dagger.Connection(config) as conn:
    ignores = ["node_modules"]
    hostdir = conn.host().directory("app", exclude=ignores)

Using the exclude= option works.

Sorry for the noise facepalm

desert mica
#

Sure, np!

#

Happy hacking!

misty salmon
#

yeah thank you @desert mica , read through the docs for hours last night and couldn't find it. and after posting the post immediately found it facepalm facepalm facepalm facepalm facepalm facepalm