#build call cache invalidation behaviour

1 messages · Page 1 of 1 (latest)

storm echo
#

I've ran into a weird behaviour that I don't understand.

If I do this:

import * as dagger from '@dagger.io/dagger';

dagger.connect(
  async (client) => {

    const node = client.directory().withNewFile(
      'Dockerfile',
      `
        FROM node:alpine
        RUN node --version
      `
    );

    const foo = await client
      .container()
      .withEnvVariable('BUST_CACHE', Math.random().toString())
      .build(node)
      .withExec(['node', '--version'])
      .stdout();

    console.log({
      foo,
    });
  },
  {
    Workdir: '.',
    LogOutput: process.stdout,
  }
);

The first time you run this it will produce this:

#3 docker build
#3 extracting sha256:4d7764ae36ba24373d83c2bdb15b74350b56564cb9148c75c98de48f2102c267
#3 extracting sha256:4d7764ae36ba24373d83c2bdb15b74350b56564cb9148c75c98de48f2102c267 0.2s done
#3 extracting sha256:60d21a72e3b7e972f267f63cdd07bec6a88bd8687fcbf2e16a6c4cfa0c8acf1b 0.0s done
#3 0.836 v19.6.0
{ foo: 'v19.6.0\n' }

It shows the version twice.

The second time you run it, it will not bust the cache of the build call. This is despite me setting a randomised env var.

Therefore, you only get this

{ foo: 'v19.6.0\n' }

Why is there a discrepancy in this behaviour? It's not clear to me how a build call cannot have its cache invalidated.

As a side note: I'm aware that logging is being redone, but the lack of explanation in the logs as to what's happened doesn't help. Shouldn't this subsequent log say at #3 that the docker build step was cached?

#1 CACHED

#2 [internal] load metadata for docker.io/library/node:alpine
#2 DONE 0.4s
{ foo: 'v19.6.0\n' }

#3 docker build
#3 resolve docker.io/library/node:alpine@sha256:992dd138340c189b2bc49d879cc4b328b12b8aa3480a43b1a05505a18987df3b 0.0s done
#3 DONE 0.0s

#1 
#1 DONE 0.0s
coarse spindle
#

Hey @storm echo

#
  1. For the double output, when it logs #3 0.836 v19.6.0, that's just the engine logging the stdout of the withexec process, vs { foo: 'v19.6.0\n' } being the part you're console logging
  2. If you move the withEnvVariable to be after the build(), the cache busting should work as expected. I believe what happens (someone correct me if I'm wrong here) is when you do .build() it builds a new container, wiping out the previous container
#

I might have misunderstood the double output part - there's also part coming from the dockerfile build since it has a RUN node --version

storm echo
#

Thanks, that helps clarify some things.

Why does cache busting work when it's placed after build, but not before? I was approaching it like it was a set of Dockerfile instructions i.e. layers higher up in the file changing invalidate later layers, but I guess that doesn't apply here.

coarse spindle
#

So it looks like what's happening is if you call build() or .from(), the new container built from that wipes out the container in the chain with the new one (probably expected), but that means if you set an env var before that build/from it will disappear (maybe not the desired DX)

storm echo
#

Ah so there's two containers, the first is the container() call and the second being the build() call.

Is there a reason that you can't do client.build() then? It feels like this can be a bit of a footgun if you're not aware of what's happening under the hood. 😅

coarse spindle
#

Yeah you're absolutely right. I will get an issue open for that discussion 😄