volume_space_physical_used_percent changed in ontap 9.14.1? | NetApp | Page 1

hollow tangle Jul 1, 2024, 11:10 PM

#

We have been using volume_space_physical_used_percent to alert on all our Ontap 9.10.1 filers. But after upgrading one to 9.14.1P5, we get false alert with volume_space_physical_used_percent == 101.

vol show -fields percent-used, physical-used-percent ,logical-used-percent -volume
-volume -volume-style -volume-style-extended
flc1-noprod-ams-storage::> vol show -fields percent-used, physical-used-percent ,logical-used-percent -volume sdb1_edbbackups_amsprod_mysqlbkup
vserver volume percent-used physical-used-percent logical-used-percent

t310_fls1_green_amsbkup_dbsys sdb1_edbbackups_amsprod_mysqlbkup 20% 101% 21%

So we tried to change alert based on volume_size_used_percent but this break our alert on 9.10.1 filers.
vol show -fields percent-used, physical-used-percent ,logical-used-percent -volume Offline_SYSTEMS_2
vserver volume percent-used physical-used-percent logical-used-percent

t551_fls5_icash0 Offline_SYSTEMS_2 94% 52% 95%

So now as a workaround, I use both volume_space_physical_used_percent and volume_size_used_percent.

Is this a bug or intended change?

Our harvest version is 24.05.2

crystal cedar Jul 2, 2024, 6:15 AM

#

@hollow tangle According to the ONTAP documentation, this value can exceed 100%.
https://docs.netapp.com/us-en/ontap-cli-9101//volume-show.html#parameters
https://docs.netapp.com/us-en/ontap-cli-9141//volume-show.html#parameters

Probably #1062049169520476220 channel is probably a good place to understand the difference between these two.
On the Harvest side, we report these values exactly as they are provided by ONTAP.
@gilded wigeon may be able to help.

hollow tangle Jul 2, 2024, 2:17 PM

#

Yeah I posted in ontap channel yesterday but no reponse yet. We been through from ontap 9.7 to 9.10, volume_space_physical_used_percent never over 100%.
With 9.14, when volume_space_physical_used_percent is over 100%, it is always 101%. Seems suspicious.

gilded wigeon Jul 2, 2024, 4:40 PM

#

If you didn't get an answer in #1062049169520476220 then you could open a case with the exact rest query or if you can duplicate in "vol show". I feel like we may have to ask Engineering.

hollow tangle Jul 2, 2024, 4:49 PM

#

case # 2010082018

gilded wigeon Jul 2, 2024, 4:55 PM

#

Was gonna say I found the thread and it was asked. I don't see the same output in the ASUP but this is slightly out of my main wheelhouse as a perf TSE. Does it show it today?

hollow tangle Jul 2, 2024, 4:55 PM

#

we are in zoom with netapp support. let me know if you would like to join

gilded wigeon Jul 2, 2024, 4:56 PM

#

Honestly I don't know how much I could help here.

hollow tangle Jul 2, 2024, 4:56 PM

#

😀 thanks anyway!

gilded wigeon Jul 2, 2024, 4:56 PM

#

Yw

#

Oh you know what...Rahul did some good digging...

#

9.10.1:
[-physical-used {<integer>[KB|MB|GB|TB|PB]}] - Total Physical Used Size

If this parameter is specified, the command displays information only about the volume or volumes that have the specified physical used size. This differs from total-used space by the space that is reserved for future writes. The value includes blocks in use by Snapshot copies.

9.14.1:
[-physical-used {<integer>[KB|MB|GB|TB|PB]}] - Total Physical Used Size

If this parameter is specified, the command displays information only about the volume or volumes that have the specified physical used size. Physical-used represents effective total footprint which is total footprint after deducting auto adaptive compresion and compaction savings. Total-footprint includes aggregate metadata used by volume. physical-used differs from total-used space by the sum of the space that is reserved for future writes and the space that is saved by aggregate storage efficiency savings being added to physical-used . The value includes blocks in use by Snapshot copies.

#

Can you do a "set diag; node run -node flc1-01-noprod-ams-storage stats start volume::sdb1_edbbackups_amsprod_mysqlbkup; node run -node flc1-01-noprod-ams-storage stats stop volume::sdb1_edbbackups_amsprod_mysqlbkup"

#

Should just be copy/paste.

#

It is odd because you don't have dedupe or snapshots on this volume.

#

What's teh phyiscal used size? Is it like 101 GB when vol is 100 GB?

#

Wait I might have the answer.

#

It is a thick volume so it includes reserved space + aggr metadata.

gilded wigeon Jul 2, 2024, 5:46 PM

#

[12:46] Grindstaff, Chris
yep works fine
volume_virtual_used{aggr="umeng_aff300_aggr2",cluster="umeng-aff300-01-02",datacenter="dc-1",node="umeng-aff300-01",style="flexvol",svm="test_vserver",volume="vol_uma"} 2256896

viscid scroll Jul 2, 2024, 5:48 PM

#

Hi @hollow tangle Paul mentioned you wanted to add virtual_used to the rest volume template. Line 58 add this

  - virtual_used                                  => virtual_used

and then you will get metrics like so
volume_virtual_used{aggr="umeng_aff300_aggr2",cluster="umeng-aff300-01-02",datacenter="dc-1",node="umeng-aff300-01",style="flexvol",svm="test_vserver",volume="vol_uma"} 2256896

#

any thoughts on whether it would be a good idea to add this metric for everyone? Is this of limited use and a temporary workaround or worth always exposing?

gilded wigeon Jul 2, 2024, 6:00 PM

#

I don't know how much we want to expose to everyone.

hollow tangle Jul 2, 2024, 6:00 PM

#

Yes it would be a good idea to add virtual used and virtual used percent for everyone. And in release notes to explain the new virtual is the old physical. For us we use physical used percent to alert in prometheus. it's hard to create a rule to use both virtual and physical to cover both 9.10 and 9.14. I am thinking to create custom template to map virtual used percent to physical for the 9.14 clusters. so we don't have to change prometheus rules. After our fleet all on 9.14, it will be easy to change alert rule based on virtual.

#

The virtual metrics are not available on 9.10. Only in diag mode on 9.14

gilded wigeon Jul 2, 2024, 6:14 PM

#

We have to check on that...

viscid scroll Jul 2, 2024, 6:17 PM

#

thanks @hollow tangle and @gilded wigeon that makes sense. We don't recommend using the REST collector until 9.12 anyway. As Paul mentioned, we need to check on including this

hollow tangle Jul 2, 2024, 6:22 PM

#

we can use volume_size_used_percent on 9.14 cluster too. It's more like physical in 9.10. Though not sure its relation with physical and logical.

gilded wigeon Jul 2, 2024, 6:26 PM

#

It's the pre-9.14 physical algorithm.

gilded wigeon Jul 2, 2024, 6:51 PM

#

We checked with someone and it is fine. Also I just created a KB for this: https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-KBs/Why_is_the_-physical-used-percent_over_100_percent_in_9.14.1_using_volume_show

viscid scroll Jul 2, 2024, 7:06 PM

#

https://github.com/NetApp/harvest/issues/3030

hollow tangle Jul 2, 2024, 7:11 PM

#

Modified our cutstom_volume.yaml for 9.14 and it's working. Thanks a lot!
`> # TM

^policy => export_policy

create_time(timestamp) => create_time

virtual_used => space_physical_used

virtual_used_percent => space_physical_used_percent

TM

43,44c49,52
< - physical_used => space_physical_used
< - physical_used_percent => space_physical_used_percent

#TM

- physical_used => space_physical_used

- physical_used_percent => space_physical_used_percent

#TM
`

#volume_space_physical_used_percent changed in ontap 9.14.1?

TM

43,44c49,52 < - physical_used => space_physical_used < - physical_used_percent => space_physical_used_percent

- physical_used => space_physical_used

- physical_used_percent => space_physical_used_percent

43,44c49,52
< - physical_used => space_physical_used
< - physical_used_percent => space_physical_used_percent