`dagger` do not honor `.dockerignore` when building from Dockerfile

[heady thorn]

I have Dockerfile build like this

FROM node:18-alpine
RUN mkdir /workspace
WORKDIR /workspace
RUN wget -qO- https://get.pnpm.io/v6.16.js | node - add --global pnpm
COPY .npmrc package.json pnpm-lock.yaml .pnpmfile.cjs ./
RUN CI=true pnpm install --frozen-lockfile
COPY . ./
RUN CI=true pnpm build

This works fine when I run docker build . directly, but when I run this under dagger this fails

import { connect } from "@dagger.io/dagger";

connect(async (client) => {
  // set build context
  const contextDir = client.host().directory(".")

  // build using Dockerfile
  // publish the resulting container to a registry
  const build = await client.container()
    .build(contextDir, )

  const ls = await build
    .withExec(['ls', '-al', 'node_modules'])
    .stderr();
  console.log(ls)

Apparently while doing COPY . ./ the dagger engine ignored .dockerignore files and decided to copy my node_modules into the container

#4
#4 0.231 total 120
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 11 13:04 .
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 11 13:04 ..
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 10 14:38 .bin
#4 0.231 -rwxr-xr-x    1 root     root         12650 Mar 10 14:38 .modules.yaml
#4 0.231 drwxr-xr-x    1 root     root         12288 Mar 10 14:38 .pnpm
#4 0.231 drwxr-xr-x    3 root     root          4096 Mar 10 13:14 .vite
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 11 13:04 @dagger.io
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 11 13:04 @sveltejs
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 11 13:04 @tailwindcss
#4 0.231 drwxr-xr-x    1 root     root          4096 Mar 11 13:04 @tsconfig
#4 0.231 lrwxrwxrwx    1 root     root           122 Mar  3 11:51 autoprefixer -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\autoprefixer@10.4.13_postcss@8.4.21\node_modules\autoprefixer\   
#4 0.231 lrwxrwxrwx    1 root     root            93 Mar  3 10:49 openai -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\openai@3.2.1\node_modules\openai\
#4 0.231 lrwxrwxrwx    1 root     root            96 Mar  3 11:51 postcss -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\postcss@8.4.21\node_modules\postcss\
#4 0.231 lrwxrwxrwx    1 root     root            87 Mar 10 13:10 pug -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\pug@3.0.2\node_modules\pug\
#4 0.231 lrwxrwxrwx    1 root     root            94 Mar  3 10:47 svelte -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\svelte@3.55.1\node_modules\svelte\
#4 0.231 lrwxrwxrwx    1 root     root           133 Mar 10 13:10 svelte-check -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\svelte-check@2.10.3_5jsbbvoukslwaqs6zxdfrt7npy\node_modules\svelte-check\
#4 0.231 lrwxrwxrwx    1 root     root           125 Mar  3 11:45 svelte-markdown -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\svelte-markdown@0.2.3_svelte@3.55.1\node_modules\svelte-markdown\
#4 0.231 lrwxrwxrwx    1 root     root           142 Mar 10 13:14 svelte-preprocess -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\svelte-preprocess@5.0.1_7tg7h2f27kvz6bnb2xfo3wieam\node_modules\svelte-preprocess\
#4 0.231 lrwxrwxrwx    1 root     root           118 Mar  3 11:51 tailwindcss -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\tailwindcss@3.2.7_postcss@8.4.21\node_modules\tailwindcss\        
#4 0.231 lrwxrwxrwx    1 root     root            91 Mar  3 10:47 tslib -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\tslib@2.5.0\node_modules\tslib\
#4 0.231 lrwxrwxrwx    1 root     root           101 Mar  3 10:47 typescript -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\typescript@4.9.5\node_modules\typescript\
#4 0.231 lrwxrwxrwx    1 root     root            89 Mar  3 10:47 vite -> C:\Users\corvo\work\custom_openai_svelte\node_modules\.pnpm\vite@4.1.4\node_modules\vite\

my .dockerignore file

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*

node_modules
**/node_modules/**

dist
dist-ssr
*.local

# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

this is...unpleasant. I expected 1:1 identical behaviour with docker build . and await client.container().build(contextDir )

[paper marsh]

Related: https://github.com/dagger/dagger/issues/3654

[keen night]

You need to define those patterns when getting the directory from the host:

const contextDir = client.host().directory('.', { exclude: ["node_modules/"] })

[heady thorn]

um, thanks, but it will be nice if container.build worked 1:1 to docker build .

[keen night]

I understand. But the equivalent step where the context is uploaded from the host to buildkit is done in Dagger by host.directory. If you read everything from directory('.') and expect container.build to do the filtering, you're already sending too much to the engine (buildkit) and busting cache on small and unrelated changes (depending on where it's used).

Not that it can't be implemented technically. container.build would have a busted cache, which means parsing .dockerignore would have to be redone on any change, but at least the actual build would be cached. For a bit more context, there has been some discussions in the past about supporting .dockerignore in dagger, but it was decided against it. I don't remember all the arguments but it's in GitHub. You can argue in favor in the issue Marcos shared, but I think it's better to understand and be explicit with host.directory as it affects all kinds of calls, not just building from a Dockerfile.

Note that you're encouraged to create reusable functions that abstract your own patterns. You're not limited to just Dagger's SDK API calls. In this case, you can reuse .dockerignore by just creating a function to open that file first, split the lines into a list and pass it to exclude. Then pass that into container.build:

import { connect } from "@dagger.io/dagger";
import { dockerfileBuild } from "@my.org/dagger"

connect(async (client) => {
  // build using Dockerfile
  const build = dockerfileBuild(".")

  const ls = await build
    .withExec(['ls', '-al', 'node_modules'])
    .stderr();
  console.log(ls)

We have plans to allow easy sharing of extensions to the API, this could be one of them. But that's on hold for the moment due to other priorities.

[heady thorn]

appreciated for the through response and pointers, I guess I'll have to make do with those now

As someone who evaluate new system and peach solutions to my team I need to be able to defend them when such rough edge happens.

[paper marsh]

nothing to add to Helder's comment. Having said that, this reply might is perfect for an issue comment or an FAQ iten.

[paper marsh]

👋 just created a function for this in my Go dagger-libs project dagger-libs: https://github.com/marcosnils/dagger-libs

Here's an example on how to use it

    appDir := client.Host().Directory(".", dagger.HostDirectoryOpts{
        Exclude: ddocker.DockerIngoreExclude(".dockerignore"),
    })

Porting that to any other language should be relatively straightforward: https://github.com/marcosnils/dagger-libs/blob/main/ddocker/utils.go

[dim leaf]

The quickstart should include a note, too. https://docs.dagger.io/429462/quickstart-build-dockerfile