#no space left on device error

1 messages · Page 1 of 1 (latest)

last hinge
#

I am getting the following error when trying to run my test pipeline:

75: exec docker-entrypoint.sh node --import tsx src/server/scripts/migrate.ts ERROR: failed to apply diffs: 1 error occurred:
    * failed to handle changes: failed to hardlink during apply: failed to hardlink during apply: link /var/lib/dagger/runc-overlayfs/snapshots/snapshots/4270/fs/project/node_modules/date-fns/esm/fp/setWeekYearWithOptions/package.json /var/lib/dagger/runc-overlayfs/snapshots/snapshots/4578/fs/project/node_modules/date-fns/esm/fp/setWeekYearWithOptions/package.json: no space left on device

However, I do have around 55GB left on my device, as reported by gnome's file manager as well as df -h:

❯ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-0       476G  423G   52G  90% /sysroot
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           7.7G  267M  7.4G   4% /dev/shm
efivarfs        192K   78K  110K  42% /sys/firmware/efi/efivars
tmpfs           3.1G   11M  3.1G   1% /run
/dev/dm-0       476G  423G   52G  90% /var
/dev/dm-0       476G  423G   52G  90% /var/home
/dev/nvme0n1p2  974M  300M  608M  34% /boot
tmpfs           7.7G  3.2M  7.7G   1% /tmp
/dev/nvme0n1p1  599M  9.9M  589M   2% /boot/efi
tmpfs           1.6G   96M  1.5G   7% /run/user/1000

How can I troubleshoot and/or fix this? I am kind of confused, to be honest. For more details, I am running this on a Fedora silverblue 39 host, with rootless podman as the executor.

#

Update: I deleted about 35GB of stuff from my drive and the pipeline runs with no errors now.

fringe herald
#

Hey @last hinge if you exec into the engine container and you run a df -h there. Do you see the same info?

last hinge
#

yes, I do

#

Actually, there is a bit of a discrepancy. Right now my host says I have 86.8G free but df -h from the container reports 80.8G

#

and the errors are happening again

#

it seems random at this point

fringe herald
#

What FS does your system use?

last hinge
#

it's btrfs

fringe herald
little dragon
#

hello, i am getting a similar error. I am running dagger enginer on a VM with 500GB of space and it has been running for about 1 month. I noticed the disk was 100% full. To reset, i deleted the dagger engine container and restarted it.

Is there another way to reclaim space?

little dragon
#

Also, the filesystem type is EXT4

fringe herald
nova lagoon
#

We've been running a custom config on the engine, to make sure garbage collection happens in a better way. By default BuildKit assumes it can use 90% of the disk

little dragon
little dragon
ashen oyster
#

@nova lagoon would you be willing to share your custom config? Would be great to see a real world example of this

nova lagoon
#

This is the one we use for developer machines, it is more agressive than the one we use for our Jenkins workers

# Technically this is BuildKit config that Dagger passes along
# https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md

debug = true
trace = false
insecure-entitlements = ["security.insecure"]

[worker.oci]
gc = true

[[worker.oci.gcpolicy]]
  # keepBytes can be an integer number of bytes (e.g. 512000000), a string
  # with a unit (e.g. "512MB"), or a string percentage of the total disk
  # space (e.g. "10%")
  keepBytes = "20GB"
  # keepDuration can be an integer number of seconds (e.g. 172800), or a
  # string duration (e.g. "48h")
  keepDuration = "8h"
  # filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
  filters = [ "type==internal", "type==regular", "type==frontend"]

[worker.containerd]
gc = true

[[worker.containerd.gcpolicy]]
  keepBytes = "20GB"
  keepDuration = "8h"
  filters = [ "type==internal", "type==regular", "type==frontend"]