Rest equivalent of quota_disk_used | NetApp | Page 1

sleek coral May 14, 2025, 1:49 PM

#

Using NAbox I've created custom dashboards which utilizes the "quota_disk_used" and "quota_files_used" metrics.
I switched to Rest collector and now they don't work anymore.

Is there another document than the one listed here for translating labels?
https://github.com/NetApp/harvest/blob/main/docs/architecture/rest-strategy.md#ive-added-counters-to-existing-zapi-templates-will-those-counters-work-in-rest

And what are the equivalent labels of "quota_disk_used" and "quota_files_used" in Rest?

mighty yoke May 14, 2025, 1:51 PM

#

hi @sleek coral which version of Harvest are you using?

sleek coral May 14, 2025, 1:52 PM

#

Harvest 25.02 and NAbox 4.0.10

mighty yoke May 14, 2025, 1:53 PM

#

thanks

sleek coral May 14, 2025, 1:53 PM

#

I don't see any value for "quota_disk_used" and "quota_files_used" in VM, so that is why I assume I have the wrong label.

mighty yoke May 14, 2025, 1:55 PM

#

https://netapp.github.io/harvest/25.05/ontap-metrics/#quota_disk_used
for ZAPI - these metrics come from the qtree.yaml template
for REST - these metrics are from the quota.yaml template

sleek coral May 14, 2025, 1:58 PM

#

I'm sorry but how do I use this information, do I have to enable something to collect the same metrics with Rest?

mighty yoke May 14, 2025, 2:03 PM

#

what I pasted above means that when you switch to Rest, you want to make sure that the Quota collector is running. Can you ssh to your nabox and type the following?
docker logs havrest 2>&1 | grep -E 'Rest:Quota'

sleek coral May 14, 2025, 2:05 PM

#

The output is.. a lot.

#

msg="no instances, entering standby"
Is this expected on a lot of lines?

mighty yoke May 14, 2025, 2:13 PM

#

let's focus on the last five lines
docker logs havrest 2>&1 | grep -E 'Rest:Quota' | tail -5
no instances, means there are no quota report records when calling the ONTAP API api/storage/quota/reports

Do you expect there to be quota report records for the poller/cluster in question?

Also why are you seeing something when using ZAPI but not REST? Have you modified either of the templates? Both the ZAPI and REST template let you specify which type you want to collect. By default, both templates only specify type=tree. Is it possible you enabled user or group for one of the templates but not the other?

sleek coral May 14, 2025, 2:33 PM

#

I see, there are values for some Ontap clusters so I guess the collector is working.

mighty yoke May 14, 2025, 2:34 PM

#

you can limit the grepped lines to a single poller if you want to focus on one

sleek coral May 14, 2025, 2:34 PM

#

I don't think I've modified the templates though.

#

Perhaps I am just using the wrong query, what would be the correct one for Rest values instead of "quota_disk_used" ?

mighty yoke May 14, 2025, 2:49 PM

#

quota_disk_used is correct - when using the REST collector, the Quota template will gather and export the quota_disk_used metric. When using the ZAPI collector, there is no Quota template, only a Qtree template, which collects and exports the quota_disk_used metric (as well as qtree metrics). Both collectors collect and export the same value for quota_disk_used

A small historical note: Most templates use the same name, but this one is slightly different. The ZAPI template was created a long time ago and did two things: collected qtree and quota metrics. When we started building a Rest template, we decided to seperate those metrics into their respective templates because that is clearer and allows customers to collect one without the other.

You said that you don't see quota_disk_used in VM, but you do see Rest:Quota collecting metrics? Can you paste the log line for one of the clusters that is exporting metrics for Rest:Quota? Here is an example time=2025-05-14T10:46:22.075-04:00 level=INFO source=collector.go:601 msg=Collected Poller=sar collector=Rest:Quota apiMs=126 bytesRx=28912 calcMs=0 exportMs=2 instances=44 instancesExported=484 metrics=440 metricsExported=506 numCalls=1 parseMs=1 pluginInstances=0 pluginMs=1 pollMs=128 renderedBytes=113842 zBegin=1747233981945

In this case, the Rest:Quota collector exported 484 quotas with 506 metrics. If I curl the poller's promPort, I see them like so:

curl -s 'http://127.0.0.1:12998/metrics' | rg -v '^#|^metadata' | sort
quota_disk_used{cluster="umeng-aff300-01-02",datacenter="dc-1",group="",qtree="trident_pvc_f86bbb72_ff49_444c_bfb6_e93f1b877cc3",svm="astra_300",type="tree",unit="kibibytes",user="",volume="trident_qtree_pool_trident_TIXRBILLKA"} 199608

sleek coral May 14, 2025, 2:55 PM

#

Sure!
time=2025-05-14T14:33:52.769Z level=INFO source=collector.go:601 msg=Collected Poller=cluster collector=Rest:Quota apiMs=1173 bytesRx=99714 calcMs=0 exportMs=4 instances=153 instancesExported=1683 metrics=1530 metricsExported=1683 numCalls=1 parseMs=2 pluginInstances=0 pluginMs=1 pollMs=1211 renderedBytes=365996 zBegin=1747233231554

#

I am unable to get the curl example working, I assume it should be /vm/metrics but not sure what port to use.

#

Just using the VM Web UI to test the query previously.

#

Hmm.. I just tried the query "quota_disk_used" again but without filter and I see some values but not for the flexgroups my custom dashboard is used for.

mighty yoke May 14, 2025, 3:06 PM

#

ah ok so metrics are being collected, but not for a flexgroup you expect

sleek coral May 14, 2025, 3:07 PM

#

Yeah, looks like it. Sorry for the confusing trail.

#

It's Ontap S3 flexgroup which is using the qtrees, perhaps there is a better solution in Rest altogether.

mighty yoke May 14, 2025, 3:11 PM

#

no worries on the rabbit trail. Maybe the ONTAP: S3 Object Storage dashboard would have what you want after you enable the OntapS3 and OntapS3SVM templates? We can also use curl to request the api/storage/quota/reports records from ONTAP directly and see if it includes the flexgroup you are interested in

sleek coral May 14, 2025, 3:31 PM

#

Ooh that seems much easier and that seems to be working better!
Just added the "ontaps3_logical_used_size" query instead, but how do I make it show the correct unit?

mighty yoke May 14, 2025, 3:35 PM

#

That metric is in bytes, which means you want to use a unit of bytes(IEC) in Grafana

sleek coral May 15, 2025, 11:02 AM

#

Awesome, thank you very much and I really appreciate you taking the time to respond to beginners like me who might not have the most structured questions.
I'll drop the question about why I can't see the quota values. 🙂

mighty yoke May 15, 2025, 11:47 AM

#

your welcome!

We can keep digging on the quota values. Let me see if I understand the problem - in VM web UI, you type quota_disk_used and see some values, but you don't see records for a flexgroup in particular. Is that correct?

Can you grab a support bundle and upload to https://upload.nabox.org/gaya-tite-reny
Which poller/cluster has the flexgroup that is missing?

sleek coral May 15, 2025, 12:49 PM

#

mighty yoke your welcome! We can keep digging on the quota values. Let me see if I underst...

If you are curious, sure. You are correct in your summary except I don't see the Ontap S3 qtrees (I filter on qtree name) I expected.
I am unfortunately not able to share any bundle from this environment.

mighty yoke May 15, 2025, 1:13 PM

#

Can you grep the logs again, filtering on the poller that has the missing flexgroup for quota? Something like this, replace poller with you poller name
docker logs havrest 2>&1 | grep -E 'Rest:Quota' | grep -E poller | tail -5
and
docker logs havrest 2>&1 | grep -E 'Rest:Quota' | grep -E poller | grep ERR

Also can you ssh to your ONTAP cluster and check that the flexgroup in question has a quota report?
Something like, replacing flexgroup with your flexgroup.
set diag
quota report -volume flexgroup

mighty yoke May 15, 2025, 1:53 PM

#

Let's also check that ONTAP is returning the flexgroup in question. From a machine that has curl installed, please run the following replacing $user, $pass, $ip, $flexgroup with the appropriate values
curl -s -k -u $user:$pass 'https://$ip/api/storage/quota/reports?return_records=true&return_unmatched_nested_array_objects=true&max_records=1000000&return_timeout=120&&type=tree|user|group&fields=qtree.name,volume.name,svm.name,index,type,space,files' | grep $flexgroup

sleek coral May 16, 2025, 5:38 AM

#

mighty yoke Can you grep the logs again, filtering on the poller that has the missing flexgr...

Not sure how to filter on the flexgroup but I see values for the poller (Ontap cluster) but no ERR's.

The quota report in the Ontap CLI seems to work, at least it is showing data;

cluster::*> quota report -volume fg_oss_1669368700
Vserver: svm

                                    ----Disk----  ----Files-----   Quota
Volume   Tree      Type    ID        Used  Limit    Used   Limit   Specifier
-------  --------  ------  -------  -----  -----  ------  ------   ---------
fg_oss_1669368700  appl1-metrics  tree  1  95.40TB  170TB  4307549  -   appl1-metrics
fg_oss_1669368700  appl1-logs  tree  2  49.76TB  75TB  108091019  -   appl1-logs
fg_oss_1669368700    tree  *           0B      -       0

sleek coral May 16, 2025, 5:41 AM

#

mighty yoke Let's also check that ONTAP is returning the flexgroup in question. From a machi...

I didn't want to pass my credentials to the history file so I did the query via the docs/api/endpoint of the Ontap cluster instead.
I am able to get quota reports from the Ontap cluster but if I do svm.name=svm (same svm as above) I get no results.

#

I don't understand why it would be missing, I think the only change I did was in NAbox (switching the poller from Auto->Rest)

sleek coral May 16, 2025, 6:56 AM

#

In case I did something wrong with the query, this is the one I used targeting the flexgroup volume:
/api/storage/quota/reports?volume.name=fg_oss_1669368700&show_default_records=true&fields=qtree.name,volume.name,svm.name,index,type,space,files&return_records=true&return_timeout=15

Result is (unexpectedly):

  "records": [],
  "num_records": 0
}```

#Rest equivalent of quota_disk_used