Collectors down after upgrade from NABox 3.5.3 to 4.0.10

[sharp flame]

Hi,

Just migrated from NABox 3.5.3 to 4.0.10, data have been imported in a bit more than 29h, everything looks good so far after some troubleshooting, excepted in the Harvest Metadata dashboard:
As you can see in the collector panel, 2 per cluster are down.

The error is for the collector in "failed to fetch data" state:
" failed to fetch data: error making request StatusCode: 403, Error: Permission denied, Message: not authorized for that command, API: /api/security?fields=fips.enabled%2Cmanagement_protocols.rsh_enabled%2Cmanagement_protocols.telnet_enabled&ignore_unknown_fields=true&max_records=500&return_records=true "
Regarding collector with error "API request rejected", there is no more information.

I followed NetApp and Harvest documentation recommendations.
The current command/directory assigned to the role used are similar on both NetApp clusters (ONTAP 9.14.1P11).

Is anything missing in that part ?

Any clue on that ?

[livid sail]

@sharp flame Could you please upload the Harvest logs at https://upload.nabox.org/fohy-jeso-depo ? If you are using the REST collector, please ensure that your ONTAP role has the permissions defined here.

[sharp flame]

Hello @livid sail ,

Support bundle has been uploaded.
I had a look at my SSH sessions log files when I configured Harvest user and role, many warnings were encountered and I also noticed the following 2 errors as well:

---

security login rest-role create -role harvest-rest-role -access readonly -api /api/security

Warning: This operation will also affect the following commands:
"security audit modify"
Warning: "vserver services name-service nis-domain group-database show" will not be accessible.
Set the access to "all" if you want to allow it.
Warning: "vserver services name-service nis-domain netgroup-database show" will not be accessible.
Set the access to "all" if you want to allow it.
Error: command failed: failed to set field "cmddirname" to "security certificate authority"

&

---

security login rest-role create -role harvest-rest-role -access readonly -api /api/network/ip/ports

Error: command failed: URI does not exist.

Thanks in advance for your help.

[livid sail]

Thanks received

I see that API /api/network/ip/interfaces in ONTAP at all. We'll fix our documents.

Related to CLI command

security login rest-role create -role harvest-rest-role -access readonly -api /api/security

This command works fine for me in 9.16.1 ONTAP version. Let me try on the ONTAP version you have.

[livid sail]

It seems to work fine in 9.14.1 for me.

security login rest-role create -role harvest-rest-role -access readonly -api /api/security

Warning: This operation will also affect the following commands:
    "security audit modify"
Warning: "vserver services name-service nis-domain group-database show" will not be accessible.
         Set the access to "all" if you want to allow it.
Warning: "vserver services name-service nis-domain netgroup-database show" will not be accessible.
         Set the access to "all" if you want to allow it.
Warning: "security certificate authority show-text" will not be accessible.
         Set the access to "all" if you want to allow it.
Warning: "security certificate keystore show-text" will not be accessible.
         Set the access to "all" if you want to allow it.
Warning: "security certificate truststore show-text" will not be accessible.
         Set the access to "all" if you want to allow it.
Warning: "security certificate keystore show-text" will not be accessible.
         Set the access to "all" if you want to allow it.
Warning: This operation will also affect the following commands:
    "security multi-admin-verify modify"
Warning: This operation will also affect the following commands:
    "security multi-admin-verify show"
Warning: "security certificate truststore show-text" will not be accessible.
         Set the access to "all" if you want to allow it.

You may want to check #1062049169520476220 channel related to this error or open a support case.

[sharp flame]

OK... Very strange behavior from ONTAP !

[livid sail]

It looks like a known bug in ONTAP. More details here https://kb.netapp.com/on-prem/ontap/Ontap_OS/OS-Issues/CONTAP-186994

[sharp flame]

Thanks for the link.
This bug has been discovered in ONTAP 9.13.1P4 and it seems it is not yet solved in 9.14.1P11, nowadays recommended version in release 9.14.1 is P12 ...
For not being affected by this bug, I guess you performed your test in a more recent version than I am currently running.

[livid sail]

I tried version 9.14.1 without any patches, but it was an internal build, so things could be different. The ONTAP support team should be able to help identify which version includes the patch.