! failed to get schema: failed to select introspection JSON file: failed to prepare as uxaf4aaninkx

[median crown]

Any ideas why something like this would happen? We have a 1TB disk, which was only 25% used.

! failed to get schema: failed to select introspection JSON file: failed to prepare as uxaf4aaninkxwxp6dpxj7g8hj: failed to open database file: open /var/lib/dagger/worker/snapshots/metadata.db: no space left on device

[gloomy rivet]

do you have a single disk in your host machine? can you get into the engine container and run df -h /var/lib/dagger to verify that effectively you have enough space?

[median crown]

Ok, yes, I did that investigation already, but it was a failed one! 🙂

What happened was that we had an instance actually fill up and then it was coincidently retired, because there was no activity. Of course, duh, as there was no disk space left, so it was dormant.

But when I started investigating I logged in into our one and only instance we have, which was already new! And it showed low disk usage. But I looked in history, and truly, the retired instance was full.

But the q now is why does it fill up so quickly? We've filled up 1 TB of data in ~ 24 hours. I thought there were clean up mechanisms...

[gloomy rivet]

the engine has garbage collection. Are you using any specific GC config? Or just the default one?