Help enabling restperf workload counters on NABox | NetApp | Page 1

high fiber Aug 23, 2023, 12:10 PM

#

I can't get workload counters functional on NA Box. Per https://nabox.org/faq/#customize-harvest-2, I believe that I can't edit /opt/harvest2-conf/conf/restperf/default.yaml or create a custom.yaml. I've created a custom_workload.yaml file, but I don't think I have the format correct.

I've tried
objects:
Workload: workload.yaml
WorkloadVolume: workload_volume.yaml
WorkloadDetail: workload_detail.yaml
WorkloadDetailVolume: workload_detail_volume.yaml

and

objects:
Workload:
- workload.yaml
WorkloadVolume:
- workload_volume.yaml
WorkloadDetail:
- workload_detail.yaml
WorkloadDetailVolume:
- workload_detail_volume.yaml

The first format looks correct according to
dc exec -w /conf nabox-harvest2 /netapp-harvest/bin/harvest doctor merge --template conf/restperf/default.yaml --with conf/restperf/custom_workload.yaml

But the second seems to match the NABox specific documentation.

Either way, I still don't have counters. Any ideas?

subtle ruin Aug 23, 2023, 1:51 PM

#

When NAbox starts Harvest, it creates a /opt/harvest2-conf/conf/zapiperf/custom.yaml which merges every /opt/harvest2-conf/conf/zapiperf/custom_*.yaml. It does that for every directory in /opt/harvest2-conf/conf/*

In default.yaml you should be able to use Harvest syntax (1st one), but this might be overwritten at start time, if you do it in /opt/harvest2-conf/conf/zapiperf/custom_workload.yaml for example, use the second syntaxt and restart nabox-harvest2 container, you should see this reflected in /opt/harvest2-conf/conf/zapiperf/custom.yaml

#

By the way, all current stable release of NAbox implements this by default :

objects:
  Workload:
    - workload.yaml
  WorkloadDetail:
    - workload_detail.yaml
  WorkloadVolume:
    - workload_volume.yaml
  WorkloadDetailVolume:
    - workload_detail_volume.yaml

#

It'll no longer be the default with 3.3.1, and a new checkbox will let you enable it easily in the web ui

#

And it's a ZapiPerf, not a RestPerf, maybe that's why you're having troubles or didn't notice it was there ?

high fiber Aug 23, 2023, 2:31 PM

#

I'm definitely not seeing any custom_.yaml files combined into a single custom.yaml file in /opt/harvest2-conf/conf//custom.yaml after restarting the container, so that might be one of my problems. From the Grafana note, I thought this was applicable to either Zapi or Rest?

subtle ruin Aug 23, 2023, 2:36 PM

#

What version of NAbox are you running, and what's in /opt/harvest2-conf/conf/zapiperf ?

#

NAbox <= 3.3 comes with /opt/harvest2-conf/conf/zapiperf/custom_volume_details.yaml

#

Which is translated to /opt/harvest2-conf/conf/zapiperf/custom.yaml with :

objects:
  Volume: custom_volume_blacklist.yaml
  Workload: workload.yaml
  WorkloadDetail: workload_detail.yaml
  WorkloadVolume: workload_volume.yaml
  WorkloadDetailVolume: workload_detail_volume.yaml

high fiber Aug 23, 2023, 2:38 PM

#

Just upgraded to 3.3 / 23.08. I'm trying to make the changes in /opt/harvest2-conf/conf/restperf

#

I do see the custom_volume_details.yaml and custom.yaml in the zapiperf folder

subtle ruin Aug 23, 2023, 2:45 PM

#

Ok so it should be fine, details ae already collected

high fiber Aug 23, 2023, 2:52 PM

#

I'm definitely not seeing these stats if I only enable the rest and restperf collections. Since I haven't gotten the override working, I've got rest, restperf, zapi, and zapiperf all enabled and it's collecting now

subtle ruin Aug 23, 2023, 2:59 PM

#

Could be a privilege issue then, did you double check the roles accoding to latest havest documentation ?

trim belfry Aug 23, 2023, 3:01 PM

#

a look at the logs may shed some light on why the rest collector is not being used - @high fiber sounds like you've ssh-ed in already, do the logs for the nabox-harvest2 container show anything interesting for the poller in question?

high fiber Aug 24, 2023, 12:26 PM

#

This is what I see repeating in the logs that look like rest errors. There are definitely some permissions, but they don't appear related.

#

ch12netapphv01:/opt/harvest2-conf/conf/restperf# dc logs -f --tail 50 nabox-harvest2 | grep -v -e "Counter is missing" -e "Collected Poller" -e "no instances"
Attaching to nabox-harvest2
nabox-harvest2 | 2023-08-24T12:14:34Z ERR collector/collector.go:367 > error="failed to fetch data: error making request 6: not authorized for that command" Poller=ch11netapp21 collector=Rest:Support task=data
nabox-harvest2 | 2023-08-24T12:14:34Z ERR collector/collector.go:367 > error="failed to fetch data: error making request 6: not authorized for that command" Poller=ch11netapp21 collector=Rest:NetRoute task=data
nabox-harvest2 | 2023-08-24T12:14:34Z ERR collector/collector.go:367 > error="failed to fetch data: error making request 3: API not found" Poller=ch11netapp21 collector=Rest:QosPolicyAdaptive task=data
nabox-harvest2 | 2023-08-24T12:14:34Z ERR collector/collector.go:367 > error="failed to fetch data: error making request 6: not authorized for that command" Poller=ch11netapp21 collector=Rest:EmsDestination task=data
nabox-harvest2 | 2023-08-24T12:14:34Z ERR collector/collector.go:367 > error="failed to fetch data: error making request 3: API not found" Poller=ch11netapp21 collector=Rest:QosPolicyFixed task=data
nabox-harvest2 | 2023-08-24T12:14:34Z WRN metricagent/metric_agent.go:136 > Poller=ch11netapp21 computeMetrics: errors for metric not found=["strconv.Atoi: parsing "block_storage.hybrid_cache.disk_count": invalid syntax","strconv.Atoi: parsing "block_storage.hybrid_cache.disk_count": invalid syntax"] object=Aggregate plugin=Rest:MetricAgent
nabox-harvest2 | 2023-08-24T12:14:35Z INF sensor/sensor.go:198 > sensor excluded Poller=ch11netapp21 object=Sensor plugin=Rest:Sensor sensor=" node:ch11netapp21-02 sensor:[{CPU0 Temp Margin -53 }] node:ch11netapp21-01 sensor:[{CPU0 Temp Margin -58 }]"

trim belfry Aug 24, 2023, 12:28 PM

#

hi @high fiber the request 6: not authorized for that command means the user does not have the necessary ONTAP permissions to request that resource. More details can be found here https://netapp.github.io/harvest/23.08/prepare-cdot-clusters/

high fiber Aug 24, 2023, 12:28 PM

#

Permissions on the array are a superset of the ones listed.

#

sec log role show -role harvest2-role
(security login role show)
Role Command/ Access
Vserver Name Directory Query Level

ch11netapp21
harvest2-role DEFAULT none
cluster readonly
event log show readonly
lun readonly
network fcp readonly
network interface readonly
network port readonly
qos workload delete readonly
qos workload modify readonly
qos workload show readonly
security readonly
snapmirror readonly
statistics readonly
storage aggregate readonly
storage disk readonly

trim belfry Aug 24, 2023, 12:34 PM

#

isn't your list a subset of the ones listed here? I don't see vserver, volume, etc in your list https://netapp.github.io/harvest/23.08/prepare-cdot-clusters/#least-privilege-approach

high fiber Aug 24, 2023, 12:36 PM

#

#

The 2nd half got cut off, we'll try a screenshot instead

trim belfry Aug 24, 2023, 12:36 PM

#

gotcha thanks

#

what version of ONTAP is this cluster? The Rest:Support template sends the REST request api/support/autosupport which should use the RBAC for system node which you have above.

#

Do any REST APIs work for this user?

high fiber Aug 24, 2023, 12:41 PM

#

This is only 9.11.1P8, so some calls are definitely not implemented yet.

#

Lots of rest is working though. If I change the exclusions on that tail

#

I'm just not seeing anything referencing something like Rest:NFS_Clients, which makes me think my problem is still with the yaml overrides, not the array or permissions

trim belfry Aug 24, 2023, 12:52 PM

#

gotcha - I thought you were asking about all the permission errors you pasted earlier, but rereading I see you don't care so much about those at the moment. You may have already done this, but have you looked at the logs as a whole instead of tailing? Something like dc logs nabox-harvest2 | less will give you a better view. Jump to the bottom (Shift-G), search backwards for Init by pressing ? and typing Init - that will position you at the most recent poller restart, then search forward with / for one of the templates you expect to be loaded but don't think is. Alternatively you can email us the logs and we can take a look

#

if you want to send the logs https://netapp.github.io/harvest/23.08/help/log-collection/#nabox
we're also happy to continue helping you debug locally

#Help enabling restperf workload counters on NABox