collector permission errors on NAbox3 | NetApp | Page 1

rustic heath Jan 23, 2024, 10:57 PM

#

First look at the logs today while helping a customer troubleshoot data collection. Looking at my own lab system, I see multiple errors from different clusters, similar to this:

ERR collectors/commonutils.go:54 > Failed to fetch data error="error making request StatusCode: 403, Error: Permission denied, Message: not authorized for that command, API: /api/private/support/alerts?return_records=true&suppress=false&time=%3E%3D1706047895" Poller=select02 href=api/private/support/alerts?return_records=true&suppress=false&time=>=1706047895 object=Health plugin=Rest:Health

The errors are for multiple endpoints, and occur across both physical and ONTAP Select clusters. All the errors are permission denied errors. I'm wondering if this might be the clusters unable to respond?

mellow shoal Jan 23, 2024, 10:58 PM

#

Maybe you need to update the roles on the ONTAP systems ? I know they changed a bit over time

rustic heath Jan 24, 2024, 1:13 AM

#

I did them using the latest ones from the NAbox directions page. Have they changed beyond that?

#

I crosschecked those as well. Happy to add more if they are needed.

azure lantern Jan 24, 2024, 3:17 AM

#

@rustic heath Can you check if relevant Rest role has /api readonly permissions. You can find more details here
https://netapp.github.io/harvest/23.11/prepare-cdot-clusters/#system-manager
You need to restart Harvest post modifying permissions for the role.

coral carbon Jan 24, 2024, 2:02 PM

#

hi @rustic heath can you try adding this permission. security login role create -role harvest2-role -access readonly -cmddirname "system health alert show"

We'll update the Harvest documentation to include that. If there are other permission errors, could you send us your poller log files to ng-harvest-files@netapp.com - you can collect nabox logs here https://netapp.github.io/harvest/nightly/help/log-collection/#nabox

rustic heath Jan 24, 2024, 3:49 PM

#

coral carbon hi <@456226577798135808> can you try adding this permission. `security login rol...

Hi @coral carbon , That seem to have fixed the alert poll. I have more failing, most of which seem to be on ONTAP Select.

#

I'm still working on these.

#

This one I believe is triggered because Harvest is attempting to get FC port data on ONTAP Select which has no FC ports.

2024-01-24T15:32:36Z ERR collectors/commonutils.go:54 > Failed to fetch data error="error making request StatusCode: 403, Error: Permission denied, Message: not authorized for that command, API: /api/network/fc/ports?enabled=true&fields=name%2Cnode&return_records=true&state=offlined_by_system" Poller=rtp-sa-select02 href=api/network/fc/ports?return_records=true&fields=name,node&enabled=true&state=offlined_by_system object=Health plugin=Rest:Health

#

Perhaps a template modification for Select is in order?

coral carbon Jan 24, 2024, 4:05 PM

#

the 403 permission denied implies RBAC - can you try running security login role create -role harvest -access readonly -cmddirname "network fcp adapter show" and see if that resolves the 403?

rustic heath Jan 25, 2024, 1:54 AM

#

@azure lantern I did check and /api doesn't show.

BUT, when I try to add /api with readonly access, I get this error message from System Manager UI:
Failed Adding the "/api" privilege.
The role already exists in the legacy role table.

azure lantern Jan 25, 2024, 6:44 AM

#

rustic heath <@1002092615052840980> I did check and /api doesn't show. BUT, when I try to a...

@rustic heath Consider applying the restrictive permissions suggested by Chris. If you wish to grant /api read only access for this role, you can use the following command:

security login role modify -role harvest2-role -cmddirname DEFAULT -access readonly

rustic heath Jan 25, 2024, 3:30 PM

#

@coral carbon Current status, I have made or tried all the recommendations from this thread. The majority of errors I'm seeing in my logs now are from REST.

Example: I have multiple clusters failing on this API call with permission denied error:
/api/cluster/licensing/licenses?fields=name%2Cscope%2Cstate&return_records=true&state=noncompliant
These clusters are Physical clusters running either ONTAP 9.8P21, or 9.14.1RC1; OR ONTAP Select clusters running 9.14.1RC1

an exact error is:
2024-01-24T17:57:08Z ERR collectors/commonutils.go:54 > Failed to fetch data error="error making request StatusCode: 403, Error: Permission denied, Message: not authorized for that command, API: /api/cluster/licensing/licenses?fields=name%2Cscope%2Cstate&return_records=true&state=noncompliant" Poller=rtp-sa-select01 href=api/cluster/licensing/licenses?return_records=true&fields=name,scope,state&state=noncompliant object=Health plugin=Re

coral carbon Jan 25, 2024, 3:34 PM

#

hi @rustic heath closer! We've been looking at this today and have found some more gaps. Apologizes. I updated the documentation a few minutes ago. It will cover everything but /support/autosupport That endpoint has a bug in RBAC. I'm looking for a workaround. Can you try copy/pasting the update list again? https://netapp.github.io/harvest/nightly/prepare-cdot-clusters/#least-privilege-approach

rustic heath Jan 25, 2024, 6:28 PM

#

Hi @coral carbon, I applied your updated permissions, I think we're almost entirely there. I still see the/api/support/auto-update failure you mentioned, so I'm discounting that.

I do have one new issue that popped up after I applied the changes - good news is it isn't a permissions issue!
2024-01-25T18:25:14Z INF collectors/power.go:222 > sensor excluded Poller=rtp-sa-cl01 object=Sensor plugin=Zapi:Sensor sensor=" node:rtp-sa-cl01-05 sensor:[{rtp-sa-cl01-05 CPU0 Temp Margin -51 }] node:rtp-sa-cl01-07 sensor:[{rtp-sa-cl01-07 CPU0 Temp Margin -55 }] node:rtp-sa-cl01-08 sensor:[{rtp-sa-cl01-08 CPU0 Temp Margin -58 }] node:rtp-sa-cl01-06 sensor:[{rtp-sa-cl01-06 CPU0 Temp Margin -55 }]"

This looks like maybe a ONTAP is getting bad sensor data? Looking at events log on cluster I see no errors reported.

#

I'm going to call this closed for now.

#collector permission errors on NAbox3