#PVE memory stuff

1 messages · Page 1 of 1 (latest)

lavish robin
#

PLP is power loss protection. It means that SSDs can "cache" and thus handle a lot of sync writes. Normal SSDs are very bad at this.

glossy cargo
#

oh but the VMs and CTs are not on ZFS

lavish robin
#

Do they use LVM?

glossy cargo
#

yes

#

its ssd

lavish robin
#

What are all those ZFS disks used for then?

glossy cargo
#

its sata not nvme i believe

#

external storage

lavish robin
glossy cargo
#

cctv recording

#

movies

#

etc

#

i havent touched any of this

lavish robin
glossy cargo
#

i didnt even knew what it was

lavish robin
#

New PVE installs default to 10%.

glossy cargo
#

so it should be 10%

#

what is ZVOLs?

glossy cargo
lavish robin
#

It's a block device that ZFS provides.

lavish robin
glossy cargo
#

so...

#

im doomed now

lavish robin
#

I can only recommend to not use ZVOLs unless you have DC SSDs.
CTs use datasets so this is fine.

glossy cargo
#

i used zfs because i read everywhere to use it

#

so what i use?

lavish robin
#

You can create a ZFS pool inside a VM and pass it all the disks

glossy cargo
#

but...

#

the the CTs dont have access

lavish robin
#

Or just use LVM-Thin.

#

I don't have any better idea but let's discuss this on the PVE discord and concentrate on the memory here.

glossy cargo
#

it took me 3 days to backup data and transfeer it back

#

why can linux just work with some filesystem and thats it? like in windows. every day it passes i get even more confused

lavish robin
#

You can do zfs set sync=disabled /poolhere and see if that changes anything but this is not recommended and does not totally fix the issue.

lavish robin
glossy cargo
#

windows uses ntfs and thats it

#

if i change from zfs to lvm

#

i lose the data

lavish robin
glossy cargo
#

need to backup it up again and restore

lavish robin
#

WIndows also uses FAT32 for EFI and ReFS for what I linked among others.

glossy cargo
#

yes my hdds are only to store data

#

not os partitions

lavish robin
#

PVE memory stuff

glossy cargo
#

anyway, the memory prblem should get better with the ram upgrade right?

lavish robin
#

No.

#

Please follow the link and set a memory limit.

glossy cargo
#

i have a work meeting now but ill be back after

lavish robin
#

Yup. I'll get notified since we are in a thread.

glossy cargo
#

im back. i was reading the link during the meeting (dont tell any1)

#

they recommend to have 2gb base per TB of ZFS storage

#

no

#

2gb base, olus 1gb per tb of storage

#

in my case that would be a lot

#

should i change this to directory? with xfs? i just leave the cctv with zfs because its in raid 0

#

here it also says the ARC usage limit will be set to 10 % of the installed physical memory, clamped to a maximum of 16 GiB

#

so may be with 32gb of ram i can allocate 16gb

lavish robin
#

Just try to stay below 80% host memory usage.

glossy cargo
#

ok. that will be possible with more ram

#

but still I will have the prob with ARC right?

#

thats why i was asking if it would be a good idea to change from ZFS to directory with xfs

lavish robin
#

No. If you do that you can also just use a ZFS dataset as directory.

glossy cargo
#

ok I will leave it as it is

#

and after the ram upgrade ill change ARC limits

glossy cargo
#

sorry to bother you again. i was thinking, if ZFS requires ram to operate correctly and even with 32gb it will probably be really close to the limit or over it, why it wont help using directory instead ZFS? asking this based on the documentation stating 2gb base + 1gb per tb. in my case that would b 39gb. ofc i will limit it to 16gb

#

basically what im asking is if there is other storage solution, since i only want to store files to read, that will work fine and not use so much ram

lavish robin
#

You can simply limit the memory or use ext4. There is no hard requirement on how much memory ARC needs.

glossy cargo
#

very well. thanks you for the input, for real. hopefully, the ram arrives 2morrow and i can see how everything performs

glossy cargo
#

hey! just an update. i left some data transferring during the night. i limited the ram usage and i did notice a big improvement. my ram hast arrived yet but ill report back after new testing with the new ram

lavish robin
#

Improvement as in decrease?

glossy cargo
#

no the usage is still very high

#

but at least i got some stable good speed

lavish robin
#

Ah.

glossy cargo
#

16gb is just not enough for what i have running

#

plus using ZFS

lavish robin
#

I limit my ZFS ARC to 3G and I have 32G.

glossy cargo
#

i limited to 6gb at first but speeds were slow

#

incresed to 8, not bad

lavish robin
#

I can't update all my desktop VMs all at once though 😐

glossy cargo
#

tried 10 but i dont have enough ram, was even worse

lavish robin
#

Which speeds. Write?

glossy cargo
#

strange thing, i noticed worse performance reading than writing

#

considering my usage, i think 12-14gb limited ARC will be enough

#

if i understood correctly, it uses the ram as cache

lavish robin
#

ARC is cache, yes.

glossy cargo
#

yes its on the name xD

#

12gb ARC will perform just fine me. but ill report after testing

lavish robin
#

As far as I know it's mostly a read cache so writes will not be faster.

glossy cargo
#

that prob explains why i was having more trouble reading than writing

glossy cargo
#

set ARC limit to 12gb and will transfer some data

#

speed is still slow. but at least it doesnt stop lkie it was happening and io delay is acceptable, <35%

#

i was having 100%

#

this is transfering from a ZFS pool to another

#

if i do it from/to my pc to raid pool. im getting 100MBs as long as file size is smaller than ARC, basically im getting my gigabit lan connection

#

very good imo

#

single drive ZFS too

image.png
lavish robin
#

That's fantastic.

glossy cargo
#

this was just some quick testing but its looking good

#

time will tell now

lavish robin
#

And it will get worse if ARC is lower?

glossy cargo
#

havent tested now but i believe so. basically that would be the scenario i had with the 16gb ram

glossy cargo
#

the server looks stable. ram usage is at 80% but everything is runnig smooth. i noticed cpu usage now is litle bit higher but IO delay is much lower, good thing. accessing the HDDs via samba or ftp is quicker now

#

thank you for the help. my proxmox onenote is getting bigger

#

next project is to install proxmox on my old server for game streaming purposes. 2 gpus, 2 VMs