waxen stump
Hi all,
We have a NetApp - VMware configuration based on iSCSI.
OnTap 9.17.1P1 and VMware 8.
We see a strange thing in the sizing between volume, lun on OnTap and datastore in VMware.
We applied the solution from this KB: https://kb.netapp.com/on-prem/ontap/DM/System_Manager/SM-KBs/ONTAP_lun_available_space_size_different_from_VMware_datastore_free_space
But no result.
Logical space reporting has been checked.
Aggregate space as well.
Any ideas?
Thanks!!
topaz vector
Look at the luns.
Make sure that “space-reserve” is disabled and make sure “space-allocation” is enabled
The first disabled lun space reserved and the second allows for the host to pass logical block provisioning back to the ONTAP lun(thin provisioned, allows for SCSI UNMAP).
Pretty sure if you set and force a rescan esxi will take it from there. I think version 8 auto enables the SCSI UNMAP TO RUN periodically and should eventually help. They may never line up but should be a lot closer
waxen stump
Thanks will try!!
waxen stump
Space-reserved is disabled and space-allocation is enabled.
waxen stump
So here are the exact details:
VMware vCenter 8.0.3
NetApp A150 - OnTap 9.17.1P1
Two aggregates RDD - 3.17 TiB each
Protocol: iSCSI via 10 Gbit ethernet
Two datastores - VMFS 6
UNMAP enabled; priority low
About 30-35 VMs mixed Windows / linux
Two volumes: 2.8 TB each
Used (OnTap volume view) 499 GiB and 724 GiB
No snapshot reserve
No Snapshot policy
One lun per volume: 2.5 TB
VMware VMS are thin provisioned
UNMAP has been run several times manually.
NetApp reports about 2.3 and 2 TB free space on the volume.
VMware reports between 440 and 470 free GB on the datastores.
I know there will be some differences, but this too large in my opinion.
Any ideas?
There is a NetApp case as well, the only thing they mentioned was installing OTV 10.3 and reprovisiong from within the OTV.
fresh mango
If you look inside System Manager... be aware that you have Thin Provisioning on the LUN as well as the Volume... And on the Volume level things like "Fractional Reserve" should not be set. Then make sure you do not have any snapshots on the volume, because thay will of cause take up some ammount of space... then there is dedupe & compression which could explain if you are seeing a difference you describe... You can see your savings under the "Tiers" it's here because volumes can share blocks on the same Tier...
hearty narwhal
Do you have snapshots? Do you have fractional reserve set to 100?
fresh mango
Do you have snapshots? Do you have fractional reserve set to 100?
To me it looks like dedupe/compression savings?
hearty narwhal
Yeah, I think so too.
If your LUN size is 2.5TB and VMware reports ~0.5TB available --> you have ~2TB logical used data.
Which ONTAP managed to save in ~0.5TB of physical data so like 4:1 ratio.
Since you still see ~2TB available in your thin provisioned volume --> increase the LUN size.
waxen stump
Do you have snapshots? Do you have fractional reserve set to 100?
No snapshots, no reserve. No fractional reserve
waxen stump
To me it looks like dedupe/compression savings?
So, switch to nfs to see the savings in VMware?
topaz vector
Look at
df -S -h -volume xxx
Verify the space savings
waxen stump
Yeah, I think so too.
If your LUN size is 2.5TB and VMware reports ~0.5TB availa...
On the local tiers the savings is 2.2 and 1.46. So that does not explain the ratios.
NetApp reports 995 GB logical used and 510GB physical on volume 1
And 1 TB logical and 731 GB physical on volume 2.
That is directly in line with the savings ratios.
VMware reports 2.04TB resp. 2.09 TB on the datastore.
I added all the disk used space from all VMs on a store, this matches as well.
We have about 30 GB of templates and isos.
i would expect that the logical used on OnTap is closer to what VMware reports.
waxen stump
Look at
df -S -h -volume xxx
Verify the space savings
Savings are in line whit what the gui reports and the differences between logical and physical.
hearty narwhal
Just to confirm: Fractional-Reserve is set to 0 on both volumes, correct?
waxen stump
Yes, fractional reserve is off on both volumes
waxen stump
Then I don't know...
Thanks!
waxen stump
Reply from support:
When a LUN is presented to VMware, it is formatted with VMFS, which represents your datastore in vCenter. VMware manages capacity metrics for these VMFS blocks but does not have visibility into ONTAP volume or aggregate-level activities, which can lead to discrepancies.
The datastore details previously available in ONTAP Tools 9.13P2 are not present in version 10.x, and there’s no official communication yet on whether this feature will return. If this functionality is important for your environment, we recommend reaching out to your NetApp account team to request an RFE
So basically we loose 50% of our storage without any real explanation and no way to monitor it from within vCenter.
fresh mango
Not that it solves the issue, but running NFS datastores makes the space usage much more transparent, then you only have to take the snapshots into account...
waxen stump
Not that it solves the issue, but running NFS datastores makes the space usage m...
I know. But the customer wants it to be as hard as possible to see data in transit. So a block protocol was prefered. NFS with kerberos is an option we are exploring. I still think a 100% more usage between VMware reporting and NetApp is not easy to explain.
rough sand
umm... why would it be harder to see data in transit with iSCSI than with NFS?
waxen stump
umm... why would it be harder to see data in transit with iSCSI than with NFS?
The customer believes file protocols are more easily to snoop than block protocols.
rough sand
but they are clearly not... they're both unencrypted
waxen stump
but they are clearly not... they're both unencrypted
Agreed
void rampart
ESX9 supports KRB5p with NFS 4.1, not that I would run it.
Martjin: Rescan is not enough to pick up the unmap feature.
You need to reboot each host, than check on the CLI:
esxcli storage core device vaai status get
I would also trim each datastore manually afterwards: esxcli storage vmfs unmap -l <datastore>
You can also check if UNMAP (vmware calls it DELETE) are issued in esxtop u f o <RETURN>