mortal igloo
Hi Team,
Our clusters ONTAP Audit logs are getting the following API errors from the Harvest user. It appears to be issuing an unsupported command every few minutes.
"GET /api/private/cli/network/port/ifgrp?fields=ifgrp%2Cnode%2Cports&max_records=500&return_records=true"
Any ideas?
Edit:
NAbox v4.0.8b1 (3bbcb80)
Harvest 24.11.0
Grafana 11.3.0
Victoria Metrics 2.24.0
Clusters Harvest users are configured to specifically use 'Rest' API.
ONTAP versions: 9.15.1P1, 9.15.1P2 and 9.15.1P3
surreal basin
@mortal igloo Could you please run the following command against the cluster and share the output of any errors? Make sure to replace USER, PASS, and CLUSTER_IP with the appropriate values.
curl -s -k -u USER:PASS 'https://CLUSTER_IP/api/private/cli/network/port/ifgrp?fields=ifgrp,node,ports'
No matter what cluster management IP I try, it comes back with "/api/private/cli/network/port/ifgrp?fields=ifgrp,node,ports: No such file or directory"
I tried logging into the URL using a browser, and it comes back with the following.
{
"error": {
"message": "API not found",
"code": "3"
}
}
As a side note, the messages stopped appearing in the Audit Log as of yesterday. No changes were made.
surreal basin
This is indeed unusual. Could you try logging in with this user via CLI and running the following command: network port ifgrp show. The REST endpoint that is failing above is equivalent to this CLI command. Could you also try the same with admin user?
For me, in version 9.15.1, it works fine as below.
surreal basin
Please also upload Harvest logs to the link: https://upload.nabox.org/peni-yeji-vapu
You can find the guide on how to collect logs here: https://netapp.github.io/harvest/latest/help/log-collection/
mortal igloo
This is indeed unusual. Could you try logging in with this user via CLI and runn...
The command network port ifgrp show works with an Admin user.
Fails with the Harvest user.
network port ifgrp show
Error: "ifgrp" is not a recognized command
The Harvest User permissions are in the following;
mortal igloo
Please also upload Harvest logs to the link: https://upload.nabox.org/peni-yeji-...
Uploaded the logs as well 'nabox-logs-2024-11-22_191207.tgz'
keen sequoia
thanks for the logs @mortal igloo
They show that the Rest and RestPerf collector are working fine overall. I see three ONTAP pollers with the names N*, S*, g*. Do you see the "invalid operation" System Manager messages for all of the clusters or only one o them?
I'm also confused why the CLI works fine, but you got an API not found error. Did you issue the CLI command and browser request against the same cluster?
This is looking like an ONTAP issue. Thanks for sharing the security login role show -role harvest2-role command. Is this from the same cluster too? Did you create a privileged rest-role too or only a traditional role?
mortal igloo
thanks for the logs <@400315826952994816>
They show that the Rest and RestPerf ...
They show that the Rest and RestPerf collector are working fine overall. I see three ONTAP pollers with the names N, S, g*. Do you see the "invalid operation" System Manager messages for all of the clusters or only one o them?
Yes, the Audit Logs on the three clusters are all reporting the same issue.
I'm also confused why the CLI works fine, but you got an API not found error. Did you issue the CLI command and browser request against the same cluster?
Yes, I ran the command from the same cluster for both Web and CLI. To clarify, the Harvest user fails in the CLI and the API. It feels like it does not have enough permissions.
https://docs.netapp.com/us-en/ontap-cli//network-port-ifgrp-show.html
"Availability: This command is available to cluster administrators at the admin privilege level." - I would assume the "cluster readonly" would cover this.
This is looking like an ONTAP issue. Thanks for sharing the security login role show -role harvest2-role command. Is this from the same cluster too? Did you create a privileged rest-role too or only a traditional role?
I followed this section https://nabox.org/documentation/configuration/#configure-role and only assigned HTTP to the new user for "# REST based access". Which looks like it was a traditional role.
mortal igloo
I added this 'cmddirname' to the harvest user role, which now allows the harvest user to run the WEB API and CLI command.
security login role create -role harvest2-role -access readonly -cmddirname "network port ifgrp show"
Warning: This operation will also affect the following commands:
"network port ifgrp create"
"network port ifgrp delete"
keen sequoia
thanks for the follow-up @mortal igloo . What's odd is we do not see that permission problem on our 9.15.1 cluster when we request /api/private/cli/network/port/ifgrp It's not clear why there is a difference, but either way, I appreciate you root causing and sharing the results with the broader community. We'll update the Harvest documentation to include network port ifgrp show in the list of required permissions
eager breach
Where can I (best) see those polling errors? I see some similar stuff under "Harvest Metadata -> Collectors -> State" but the messages are truncated there to, for example, just "API request rejected" or "failed to fetch data ... Permission denied"
So I don't know exactly what is failing 😦
keen sequoia
hi @eager breach all messages will be in Harvest's log files. Details on where those files are can be found are here - it varies by how you installed Harvest https://netapp.github.io/harvest/nightly/help/log-collection/
Let us know if you need help making sense of them
eager breach
OK - Sorry, I thought there was a UI Window (screenshot) in the original post so I was looking inside grafana
keen sequoia
No worries, yes the screenshot at the top of this post is from System Manager showing the ONTAP Audit logs like so
eager breach
Aha, of course ... the dreaded System Manager !
mortal igloo
Hi Team,
I came across another one of these permission issues.
GET /api/private/cli/system/controller/fru?fields=fru_name%2Cnode%2Cserial_number%2Cstatus%2Csubsystem&ignore_unknown_fields=true&max_records=500&return_records=true
invalid operation
Would this translate to the system controller show CLI command? The harvest user does not have permissions to run this.
I added the following to see if goes away security login role create -role harvest2-role -access readonly -cmddirname "system controller show"
keen sequoia
hi @mortal igloo yes, this endpoint was added two weeks ago to capture FRU status and I think you are correct that the permissions were missed. https://github.com/NetApp/harvest/blob/main/conf/rest/9.12.0/fru.yaml
Your security login role create command looks good. I think this would work too
security login role create -role harvest2-role -access readonly -cmddirname "system controller fru show"
mortal igloo
Thankyou! I will try that.
The system controller fru show command does not return the 'serial_number' for me. Is that expected? I see it listed as one of the 'counters' in the 'fru.yaml'.
keen sequoia
documenattion will be fixed with this pull request https://github.com/NetApp/harvest/pull/3354 Appreciate you reporting it!
the counters are turned into fields for that command so try this to match the template:
system controller fru show -fields node,serial-number,fru-name,status,subsystem
mortal igloo
Got it! thanks that works.
Cheers!
mortal igloo
Hey @keen sequoia ,
I noticed that the command system controller fru show -fields node,serial-number,fru-name,status,subsystem is not supported in Cloud Volumes ONTAP,
Warning: Unable to list entries on node NODE NAME. This command is not supported on this platform.
Error: show failed: This command is not supported on this platform.
I assume we would need to opt out of this somewhere in Grafana?
keen sequoia
Hi @mortal igloo We'll work on a better fix but in the meantime, can you try disabling the FRU collector by commenting out this line in your rest default.yaml? https://github.com/NetApp/harvest/blob/083cccd66fb899b8cb43320c86c4d6394e698ad6/conf/rest/default.yaml#L25