Running Harvest with SVM-scoped credentials | NetApp | Page 1

rocky basin May 21, 2025, 6:27 PM

#

Hello ONTAP team,

We are planning to host Harvest in multiple kubernetes clusters.
Each cluster will have an associated SVM on ONTAP side.

And we would like to use SVM scoped credentials for harvest instance to use and get SVM performance metrics and volume usage metrics.
We would like to scope the credentials to only allow Harvest to get SVM related metrics.
This should ensure that each harvest instance only gets access to SVM specific information.

We tried to follow these instructions: https://netapp.github.io/harvest/nightly/configure-harvest-basic/
And we provided SVM credentials in the harvest config.

But looks like Harvest needs cluster level credentials because we see that it makes calls to cluster endpoint.

Here is what we have tried so far:

configure harvest w/ any credentials (cluster or SVM) and target the SVM mgmt IP - this gave some API not found errors which seemed to be some cluster-level APIs, which makes sense why SVM IP would have this issue
configure harvest w/ SVM credentials and target the cluster mgmt IP - this gave a 401 unauthorized, which we suspect is because harvest needs to hit those cluster-level APIs for basic operation - (this may suggest that creds only scoped to the SVM wouldn't be sufficient)

How can we configure Harvest to only get SVM specific metrics with SVM credentials and avoid providing cluster level credentials in the cluster config?

Thank you!

crude tangle May 23, 2025, 5:04 PM

#

Performance is a cluster-wide scoped concept. Volumes from different SVMs can live on the same aggrs and/or use the same network interfaces and impact each other. This is only visible at the cluster scope.

#

#1062050414146625536

fathom sorrel May 23, 2025, 5:13 PM

#

@rocky basin started in the harvest channel and I suggested he come here to see what was possible from an ONTAP perspective. 🙂

Using the following ONTAP RBAC that limits responses to query=osc, config APIs, KeyPerf, and private CLI work. RestPerf does not. If @rocky basin is happy with the metrics provided via /api/storage/volumes?fields=statistics then KeyPerf may be sufficient.

security login role show -role tmp2
           Role          Command/                                      Access
Vserver    Name          Directory                               Query Level
---------- ------------- --------- ----------------------------------- --------
umeng-aff300-01-02
           tmp2          DEFAULT                                       none
                         cluster                                       readonly
                         statistics                       -vserver osc readonly
                         volume                           -vserver osc readonly
                         vserver create                   -vserver osc readonly
                         vserver modify                   -vserver osc readonly
                         vserver show                     -vserver osc readonly
7 entries were displayed.

#

By works, I mean when running the following curls, the results show 10 volumes (only the ten on the osc SVM) instead of all the volumes. Notice that RestPerf returns 161 volumes, which is all the volumes on the cluster and not what is wanted.

#!/usr/bin/env bash

echo "api/cluster"
curl --silent --insecure -u"$user:$pass" 'https://10.193.48.154/api/cluster' | jq '.name // .error.message'

echo "api/storage/volumes no tunnel"
curl --silent --insecure -u"$user:$pass" 'https://10.193.48.154/api/storage/volumes' | jq '.num_records // .error.message'

echo "api/storage/volumes w/ tunnel"
curl --silent --insecure -u"$user:$pass" --header "X-Dot-SVM-Name:osc" 'https://10.193.48.154/api/storage/volumes' | jq '.num_records // .error.message'

echo "api/private/cli/volume"
curl --silent --insecure -u"$user:$pass" 'https://10.193.48.154/api/private/cli/volume' | jq '.num_records // .error.message'

echo "CM-RZR RestPerf api/cluster/counter/tables/volume/rows"
curl --silent --insecure -u"$user:$pass" 'https://10.193.48.154/api/cluster/counter/tables/volume/rows' | jq '.num_records // .error.message'

echo "CM-REST KeyPerf api/storage/volumes"
curl --silent --insecure -u"$user:$pass" 'https://10.193.48.154/api/storage/volumes?fields=statistics' | jq '.num_records // .error.message'

#

Results from running curls with RBAC from above

api/cluster
"umeng-aff300-01-02"
api/storage/volumes no tunnel
10
api/storage/volumes w/ tunnel
10
api/private/cli/volume
10
CM-RZR RestPerf api/cluster/counter/tables/volume/rows
161
CM-REST KeyPerf api/storage/volumes
10

calm galleon May 29, 2025, 2:30 PM

#

We tried running harvest with the Rest and RestPerf collectors with a svm-scoped rest-role with these permissions:

    /api/cloud/targets      readonly
                               /api/cluster/counter/tables 
  readonly
                               /api/private            readonly
                               /api/storage/file       readonly
                               /api/storage/flexcache/flexcaches    
                                                       readonly
                               /api/storage/luns       readonly
                               /api/storage/namespaces    
                                                       readonly
                               /api/storage/qtrees     readonly
                               /api/storage/quota/reports    
                                                       readonly
                               /api/storage/quota/rules    
                                                       readonly
                               /api/storage/snapshot-policies    
                                                       readonly
                               /api/storage/volumes    
                                                       readonly
                               /api/svm/peers          readonly
                               /api/svm/svms           readonly
                               /api/svms               readonly

Although the harvest logs showed permission denied errors for some endpoints that it didn't have access to, we were still able to see results for the various metrics we're interested in (such as volume_total_ops , volume_read_ops, volume_write_ops, etc. and the metrics we saw were only for the svm that the rest-role was scoped to.

#

Although this approach is a bit hacky with relying on harvest being able to output metrics in a best-effort fashion it seems to suffice for our use-case.

fathom sorrel May 29, 2025, 3:04 PM

#

Great that you found a solution @calm galleon . Let us know if Harvest can do anything to make things smoother

calm galleon May 29, 2025, 8:09 PM

#

One minor thing I noticed in my testing is that ONTAP didn't let me directly set/api/cluster/counter/tables as a readonly access using create, saying that readonly cannot be used as an access level for that statistics API. However if I use create to set it as a different access level first such as read_create, I could then use modify to change it to a readonly access without error. This doesn't seem like intended behavior (it should either allow me to set it as readonly with create or prevent me from setting it to readonly during both create and modify) - not sure if this is a known issue that was fixed in a later release.

fathom sorrel May 30, 2025, 12:01 PM

#

What version of ONTAP? This works for me on 9.14+ security login rest-role create -role harvest-rest-role -access readonly -api /api/cluster/counter/tables

fathom sorrel May 30, 2025, 12:24 PM

#

Ah, I see what you mean. This does not work, with the following error
security login rest-role create -role tmp-role -access readonly -api /api/cluster/counter/tables -vserver osc
Error: command failed: A Vserver admin cannot use command directory "statistics" with access level "readonly". Use a different access level.

but then you can turn around and do this, which does work
security login rest-role modify -vserver osc -role tmp-role -api /api/cluster/counter/tables -access readonly

               Role                                    Access
Vserver        Name            API                     Level
----------     -------------   -------------------     ------
osc            tmp-role        /api/cluster/counter/tables
                                                       readonly

#Running Harvest with SVM-scoped credentials