Buildkit disk usage growing infinitely

[vagrant solstice]

I've noticed that a simple build (reproducible) with dagger-engine 0.9.3 and Go SDK results in eating up disk space very fast for no reason.

FROM alpine
ADD a.sh .

Content of a.sh is just "echo foo"

And a code to build this:

package main

import (
    "context"
    "fmt"
    "math"
    "math/rand"
    "os"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    contextDir := client.Host().Directory(".")

    ref, err := contextDir.
        DockerBuild().
        Publish(ctx, fmt.Sprintf("ttl.sh/hello-dagger-%.0f", math.Floor(rand.Float64()*10000000))) //#nosec
    if err != nil {
        panic(err)
    }

    fmt.Printf("Published image to :%s\n", ref)
}

Results in buildkit cache constantly increasing.
First build:
$ docker exec dagger-engine-7b45c2238c1141a1 buildctl du -v
Reclaimable: 0B

After it:

ID                                    RECLAIMABLE    SIZE        LAST ACCESSED
qju1lp3cvnhxyjkf1jpe32crh                                                  true           191.97MB
jbd3i2uaodnvmp2r2g6m90kui*                                                 true           131.29MB
7rbnyc454uhqwn7rmrtnahr48*                                                 true           131.29MB
4vujwtocspbf35gxlwj4kuyiv                                                  true           11.68MB
h4v38q7854ey5tb752p1j6kln                                                  true           8.30kB
Reclaimable:    466.24MB
Total:        466.24MB

Second build, all cached according to Dagger output, yet increasing in size:

docker exec dagger-engine-7b45c2238c1141a1 buildctl du
ID                                    RECLAIMABLE    SIZE        LAST ACCESSED
qju1lp3cvnhxyjkf1jpe32crh                                                  true           191.97MB
qxd5o8ym5plzuzef1jg8sz252*                                                 true           131.29MB
jbd3i2uaodnvmp2r2g6m90kui*                                                 true           131.29MB
7rbnyc454uhqwn7rmrtnahr48*                                                 true           131.29MB
4vujwtocspbf35gxlwj4kuyiv                                                  true           11.68MB
h4v38q7854ey5tb752p1j6kln                                                  true           8.30kB
Reclaimable:    597.54MB
Total:        597.54MB

Fifth build...:

You can see that the 131.29MB layer is stored again and again.

➜  ~ docker exec dagger-engine-7b45c2238c1141a1 buildctl du -v --filter id==jbd3i2uaodnvmp2r2g6m90kui
ID:        jbd3i2uaodnvmp2r2g6m90kui
Created at:    2023-11-29 02:02:09.495038169 +0000 UTC
Mutable:    true
Reclaimable:    true
Shared:        false
Size:        131.29MB
Description:    local source for .
Usage count:    1
Last used:    2023-11-29 02:02:21.71401705 +0000 UTC
Type:        source.local

Could this be related to https://github.com/moby/moby/issues/46136 ref count problem which got fixed in Buildkit 0.12?
Is dagger-engine v0.9.3 including Buildkit 0.12 behind the scenes, or a slightly older version? Or is it a bug in Dagger?

Buildkit disk usage growing infinitely

[stark token]

👋 have you checked if this also happend without the Dockerbuild? Just trying to rule out the Dockerfile frontend here

[vagrant solstice]

Yes,it does, for example I tried the Go SDK example single-stage build: https://docs.dagger.io/sdk/go/959738/get-started#step-4-create-a-single-build-pipeline

func build(ctx context.Context) error {
    fmt.Println("Building with Dagger")

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        return err
    }
    defer client.Close()

    // get reference to the local project
    src := client.Host().Directory(".")

    // get `golang` image
    golang := client.Container().From("golang:latest")

    // mount cloned repository into `golang` image
    golang = golang.WithDirectory("/src", src).WithWorkdir("/src")

    // define the application build command
    path := "build/"
    golang = golang.WithExec([]string{"go", "build", "-o", path})
    _, exportErr := golang.Export(ctx, "/tmp/image.tar")
    if exportErr != nil {
        return exportErr
    }

    return nil
}

First attempt:

ID                                    RECLAIMABLE    SIZE        LAST ACCESSED
kob3607ay6jfbwq9sj1a1f52t                                                  true           335.29MB
41v5ra6nvexegxasrwednw8iy                                                  true           317.94MB
lko3kllnsghijzcyaulqxgvwd                                                  true           263.10MB
z3k4qdr3c8f671js3bx8p88qe                                                  true           204.89MB
woftovns5ft4bgdsv8hve7azq                                                  true           132.07MB
mb31e2v1q2v2wsiphmfbzbm8m                                                  true           75.89MB
ryswxa0r6n7nd9r4qtcw7vq4e                                                  true           13.90MB
81i4x89wgov67yeyjs7d9h00u                                                  true           13.89MB
8wl8zbglpvk04g2mf2hztnny4*                                                 true           8.97MB
11cqe5zl9qohwdarsrmyyatxc*                                                 true           8.97MB
wf9k0kxpfbiv9jqjp7hinantr                                                  true           16.54kB
m4z4jy4qf8bi1eg3x0l8ewao4                                                  true           16.38kB
nuvpivyj2a0j1qfiw8uwxrfyv                                                  true           8.19kB
s2u4j6kauwedzhx5xj8rh4p06*                                                 false          0B
Reclaimable:    1.37GB
Total:        1.37GB

After 20 runs:

[stark token]

👋 in the Go example you posted above this actually makes sense since the Export is sending a newly created tar image to your ./build directory in the host that gets re-uploaded every time that you run the pipeline

I've just double checked by adding an Exclude directive to the client.Host call and I don't see the size increasing:

package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
)

func main() {
    if err := build(context.Background()); err != nil {
        fmt.Println(err)
    }
}

func build(ctx context.Context) error {
    fmt.Println("Building with Dagger")

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        return err
    }
    defer client.Close()

    // get reference to the local project
    src := client.Host().Directory(".", dagger.HostDirectoryOpts{Exclude: []string{"build"}})

    // get `golang` image
    golang := client.Container().From("golang:latest")

    // mount cloned repository into `golang` image
    golang = golang.WithDirectory("/src", src).WithWorkdir("/src")

    // define the application build command
    path := "build/"
    golang = golang.WithExec([]string{"go", "build", "-o", path})

    // get reference to build output directory in container
    output := golang.Directory(path)

    // write contents of container build/ directory to the host
    _, err = output.Export(ctx, path)
    if err != nil {
        return err
    }

    return nil
}

[stark token]

just did another check on the Dockerfile and seems like that is it. Even with a very simple pipeline, each time I run it, the total size increases. Opening an issue