#Garbage Collection & disk space
blissful nest
🧵
Also, should we make an emergency PR to lower the 75% to, say 60%, as a short-term stopgap, to buy the gc more time to kick in?
I just opened an issue btw https://github.com/dagger/dagger/issues/7711
(intentionally left it very open ended)
rose apex
right yes - so there are two options:
- option 1: we take free disk space at time of start of program as the amount of space to use. this is kinda weird, since it means if you restart dagger, it'll start to use different amounts of disk space? this is kinda weird
- option 2: we take free disk space continuously, and try and only use up that amount. the problem with that is, what does it mean to use 50% of the free disk space? if i fill that up, can i use 50% of the remaining disk space?
i think what we want is instead an opposite setting - instead of being able to only configure buildkit's cache to use X% of available disk space, we should configure it to always try and leave Y% of the total disk free
so that would make buildkit's gc less aggressive when there's a lot of space left, and more aggressive when there's less space left
this also co-operates a lot better with other things using that disk - if i install a massive program on the same machine, then telling buildkit to use 50% of the cache is gonna keep using the same amount - but telling buildkit to gc until 50% is free will keep my disk usage cap at roughly the same kind of level
we should make this way easier to configure manually as well
these settings won't work for everyone, no matter how many dials we expose, manual configuration is gonna be neccessary (i'm honestly surprised motion has got this far without it)
blissful nest
Yes in my mind "50% of free disk space" means "50% of total disk space minus what others are using". So it only makes sense if you don't count your own current usage
rose apex
oh right, then yeah, those are perfectly equivalent i think
blissful nest
Which is exactly your proposal basically
Yours is more intuitive
Since it's "real" free space
rose apex
yeah, it's also how go's gc works
where it will be more aggressive if you have less free memory available
which is very weird, but quite clever
i might try hacking on this soon, i think upstream would be fine with this
blissful nest
I think there should be 2 thresholds: a percentage of free space + an absolute number of free GB, and the trigger should be whichever is smaller
(but percentage only is fine as a MVP)