#Disk space exploding (HAOS, Proxmox)

1 messages · Page 1 of 1 (latest)

unborn glen
#

Well...I guess the image speaks for itself.
I'm not entirely new to home assistant, so I know a thing or two but this problem has been following me for a while.
One evening the storage got full. So I setup a new instance, imported the backup and it worked for a while. But what's visible here is just crazy. It starts at 11.31G being used and finally crashes at 24G...over a span of 5 days.

Please, if there's anyone who can help me find out what's going on with my installation I would be very grateful!

image.png
#

Backups are being stored both locally and on my NAS. Local backups are being kept for 1 day, NAS ones for 7

unique canyon
unborn glen
#

here's both the qm config and a less cropped image. I've also extended the view from a week to a month

📎 qm_config.txt image.png
#

And thanks for the link! Will look into it rq

unique canyon
#

20GB is way too little by the way. HAOS alone has a minimum of 32 and you allocated more than you have so you easily run into issues. Why is your storage so tiny?

unborn glen
unborn glen
#

home assistant always sat at around like 12GB used and spiking up sometimes, but not this extreme

unique canyon
#

Still dangerous.

unborn glen
#

yeahhh I know...Should probably get a bigger disk

#

but that's not normal, right? 5 days like 10+ GB
Or am I missing something

unique canyon
#

I can explain more tomorrow but basically the more data changes the faster the max space is used up on the node side even thought it's not.

unborn glen
#

interesting...

#

Didn't know this, but that explains a lot. I always wondered why home assistant insists, that it uses 10gb when proxmox says otherwise

#

Well what size would you recommend for the new drive?

tough heart
#

for me, i would do 50gb minimum

#

just to be safe.

unborn glen
#

but like with proxmox and maybe other things being stored on it
Like physical drive storage

#

bcs the 64gig one installed rn won't cut it with 50gb given to HA

tough heart
#

i am talking about vm size

#

ah ok..

unborn glen
#

yeahh but to increase the VM size I first need to upgrade the physical ssd haha

tough heart
#

dont know what else you've got running, but i guess 500gb would do. Or add extra ssd for vm

unborn glen
#

okay thank you very much, will get one soon
I don't want to fight with HA over storage anymore hahaha

unique canyon
#

Can you show me the output of these?

lvs -a
qm guest exec 100 -- fstrim -av

# Wait 15s
lvs -a

If you have CTs run pct fstrim CTID for them as per link.

unborn glen
#

OKAY WOW. That first one was magical I have to say

#

went from nearly 22 down to 11

unique canyon
#

The thing is that the VM does not know that you only have 20G. I'm not sure about the technicalities about how LVM-Thin decides when to provision new space but it can do that even if the VM technically doesn't need more. I guess it has something to do with where the space is allocated by the filesystem inside. Fragmentation and non-linear allocation and such. HAOS runs this fstrim itself but only once per week. If data changes too much then you get this large discrepancy.
I talked about this a bit here: https://community.home-assistant.io/t/space-issue-with-haos-on-proxmox-8/698664/30

Home Assistant Community
Space issue with HAOS on Proxmox 8

@pedromfa Since you can’t just use the OS shell to edit the timer like this # systemctl edit fstrim.timer Failed to create parent directories for '/etc/systemd/system/fstrim.timer.d/override.conf': Read-only file system you have to find other ways but every method is hacky. One way is to do it from the PVE side via this qm guest exec VMIDH...

#

As for the disk I like used Intel DC drives from eBay. 240G is a good choice to start with. They are really cheap and you can use ZFS with them without worries.

lilac shuttleBOT
#
Storage

Open your Home Assistant instance and show storage information

unique canyon
#

As for the VM see here how much it needs.

unborn glen
unborn glen
# unique canyon Can you show me the output of these? ```bash lvs -a qm guest exec 100 -- fstrim ...

well for the first one it just spammed my console with success notes, so that's nice
2nd one gave this output:

root@pve:~# lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotz-- 20.00g 50.79 28.12
[data_tdata] pve Twi-ao---- 20.00g
[data_tmeta] pve ewi-ao---- 20.00m
[lvol0_pmspare] pve ewi------- 20.00m
root pve -wi-ao---- <24.94g
swap pve -wi-ao---- <7.38g
vm-100-disk-0 pve Vwi-aotz-- 32.00g data 31.74
vm-100-disk-1 pve Vwi-aotz-- 4.00m data 14.06

unborn glen
unborn glen
#

"of 30.8GB"...yeah no

#

haha

lilac shuttleBOT
#

To format your text as code, enter three backticks on the first line, press Enter for a new line, paste your code, press Enter again for another new line, and lastly three more backticks.
```yaml
example: here
```
Don't forget you can edit your post rather than repeatedly posting the same thing.

unique canyon
#

30%~ of 32G used on the PVE side. Seems about right.

#

I'd like to see lsblk -o+FSTYPE,MODEL as well. It's interesting that your local is larger than local-lvm.

#

But then I have to go for now.

unborn glen
#

root@pve:~# lsblk -o+FSTYPE,MODEL
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS FSTYPE MODEL
sda 8:0 0 59.6G 0 disk 64GB SATA Flash Drive
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part /boot/efi vfat
└─sda3 8:3 0 59.1G 0 part LVM2_member
├─pve-swap 252:0 0 7.4G 0 lvm [SWAP] swap
├─pve-root 252:1 0 24.9G 0 lvm / ext4
├─pve-data_tmeta 252:2 0 20M 0 lvm
│ └─pve-data-tpool 252:4 0 20G 0 lvm
│ ├─pve-data 252:5 0 20G 1 lvm
│ ├─pve-vm--100--disk--0 252:6 0 32G 0 lvm
│ └─pve-vm--100--disk--1 252:7 0 4M 0 lvm
└─pve-data_tdata 252:3 0 20G 0 lvm
└─pve-data-tpool 252:4 0 20G 0 lvm
├─pve-data 252:5 0 20G 1 lvm
├─pve-vm--100--disk--0 252:6 0 32G 0 lvm
└─pve-vm--100--disk--1 252:7 0 4M 0 lvm

unique canyon
#

How much RAM do you have? You could perhaps delete the SWAP volume and give its space to data.

unborn glen
unborn glen
unique canyon
#

Yep. I can help you with that tomorrow if you like. You can research ZRAM until then. I'll excuse myself.

unborn glen
#

No problem, thank you so much! You literally saved my sanity

#

I had about 100mb left before the trim, now I can sleep peacefully

#

If it would've gone any higher I most likely would have needed another reinstall

fiery shore
unique canyon
#

So, to remove swap and give it to data follow this

# Disable SWAP
swapoff -a

# Make sure this says "RESUME=none"
grep -sR "RESUME=" /etc/initramfs-tools/

# Remove if from fstab
nano /etc/fstab

# Remove the swap volume
lvremove -y /dev/pve/swap

# Extend the data volume with 90% of the now free/unallocated space in the Volume Group.
# Never use 100% for thn pools
lvextend -r -l +90%FREE /dev/pve/data

# Install ZRAM
apt install zram-tools

# Set percentage to 5-10%
sed -i 's/^.*PERCENT=.*/PERCENT=10/' /etc/default/zramswap

# Or do it manually
nano /etc/default/zramswap

# Restart zwap so the setting is applied
systemctl restart zramswap.service

# Confirm that data is larger and SWAP is there
lvs
free -h
zramctl

ZRAM is optional but probably a good idea.

unique canyon
#

@unborn glen I was hoping you'd try above.

unborn glen
#

Yeah sorry for the delay, I will try your fix but I haven't really had the time due to exams