Which Harvest metrics are "converted" to which 'ontaps3_' metrics?
On the release page https://github.com/NetApp/harvest/releases/tag/v22.11.0 I do see that "ONTAP S3 server config metrics" have been added (merging the https://github.com/NetApp/harvest/issues/1282 there as well).
From other side, if I read right between the rows of the https://github.com/NetApp/harvest/issues/1768, the main ONTAP S3 buckets metrics are targeted currently to the v.23.05. Thus I'm a bit confused here.
When trying to 'convert' the metrics names from the listed in the ontaps3.yaml of the latest [23.02.1] Harvest to the ones described for ONTAP https://library.netapp.com/ecmdocs/ECMLP2876964/html/index.html#/object-store, I can't really find any match.
I do succeed to fetch from our [cmode] ONTAP to Harvest [via Prometheus] and from there to Grafana - but which metrics should I use for statics / rates in the latter's dashboards? I'd be glad to get at least size and list of the buckets (per SVM / cluster / node).
I've tried to guess that e.g. 'list_buckets_total' in Harvest will return the same number as the 'curl -k -v -X GET --user <admin> "https://<cluster>/api/protocols/s3/buckets?fields=**&return_records=false" -H "accept: application/json"' - but no. While the curl returns, let's say, 300 (which is confirmed by the relevant cli cmd) - the ontaps3_list_buckets_total returns 31 for the same moment.
Do I miss something?
Thanks in advance.
#ONTAP S3 buckets metrics - Harvest vs ONTAP
delicate osprey
ivory bluff
hi @delicate osprey Issue 1768 is about adding a dashboard that includes ONTAP S3 metrics. The earlier issues you linked above were about collecting S3 metrics. Thanks for the feedback on what you'd like to see in that dashboard. That's helpful.
There's a bit more information on which ONTAP S3 metrics are collected here https://netapp.github.io/harvest/23.02/ontap-metrics/#ontaps3_abort_multipart_upload_failed
ontaps3_list_buckets_total returns the number of list bucket ops instead of number of list buckets. That comes from this template https://github.com/NetApp/harvest/blob/main/conf/zapiperf/cdot/9.8.0/ontap_s3.yaml
The REST endpoint you pasted above /api/protocols/s3/buckets will return the number of s3 buckets as you mentioned
and that endpoint is used by the Harvest rest template here https://github.com/NetApp/harvest/blob/main/conf/rest/9.7.0/ontap_s3.yaml
ivory bluff
What version of ONTAP are you using @delicate osprey? I'm guessing that you are using the Zapi collectors instead of the Rest ones? Harvest is not collecting S3 config information via ZAPI, only REST. I believe that's because ONTAP does not have a ZAPI that returns S3 information, checking. Yes that is the case security login role show-ontap -command "vserver object-store-server bucket show"
There are no entries matching your query.
If you enable the REST collector you should see Harvest publish metrics for your s3 buckets like so
delicate osprey
Yes, my Harvest was indeed working via ZAPI[+PERF] only.
After adding the "-Rest" to the collectors list in the harvest.yml [and allowing also HTTP access to the user used by the API to access the cluster - hope it can help someone], the metrics seem arriving:
2023-03-15T06:41:33Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:NetConnections path=conf/rest/9.12.0/netConnections.yaml v=9.8.0
2023-03-15T06:41:33Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:VolumeAnalytics path=conf/rest/9.12.0/volume_analytics.yaml v=9.8.0
2023-03-15T06:41:34Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:Port path=conf/rest/9.12.0/port.yaml v=9.8.0
2023-03-15T06:41:34Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:SecuritySsh path=conf/rest/9.12.0/security_ssh.yaml v=9.8.0
2023-03-15T06:41:35Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:NFSClients path=conf/rest/9.7.0/nfs_clients.yaml v=9.8.0
2023-03-15T06:41:36Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:Aggr path=conf/rest/9.10.0/aggr.yaml v=9.8.0
2023-03-15T06:41:37Z INF collector/helpers.go:139 > best-fit template Poller=<cut> collector=Rest:OntapS3 path=conf/rest/9.7.0/ontap_s3.yaml v=9.8.0
I do see the ontaps3_ size, labels & logical_used_size now. Not the object_count one though.
delicate osprey
RestPerf doesn't seem working out of the box - I see only "configuration error => missing timestamp metric" & "error making request 3: API not found" errors.
Not critical, of course, but still - as the ONTAP API EoL is planned (together with the ZAPI, as far as I understand) until the EoY, ZAPIPERF will need to be replaced by the RESTPERF. Is it right?
wispy sluice
@delicate osprey What is the ONTAP version for which you are trying RestPerf Collector?
delicate osprey
9.8P16