No Metric after ONTAP upgrade | NetApp | Page 1

thorn yacht Jul 3, 2023, 12:40 PM

#

Hi all,
Does anyone has experience that the NABox does not collect any metrics for 24h after an ONTAP upgrade? Interesting is that we have multiple NABoxes in place but only one is showing this behavior.

snow rampart Jul 3, 2023, 12:47 PM

#

@thorn yacht Indeed, in Harvest 23.05, we implemented a new check to handle cases where an API request is rejected by ONTAP. As a result, the retry interval has been set to 24 hours. Unfortunately, this change has led to the scenarios you've reported.
Could you please open a bug for this.

https://github.com/NetApp/harvest/blob/main/cmd/poller/collector/collector.go#L359-L365s

hallow thunder Jul 3, 2023, 1:33 PM

#

What I've seen happen is that nabox loses connectivity to the cluster. In current releases, it retries. In previous releases, it wouldn't even do that.
I always try to remember and check nabox following a cluster upgrade. If it's not capturing data, restart nabox.

thorn yacht Jul 3, 2023, 1:41 PM

#

snow rampart <@1006611101024256041> Indeed, in Harvest 23.05, we implemented a new check to h...

Thanks @snow rampart report is open https://github.com/NetApp/harvest/issues/2175

GitHub

Harvest 23.05 - retry interval has been set to 24 hours · Issue #2...

A note for the community Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request If you are interested in working on thi...

snow rampart Jul 3, 2023, 2:00 PM

#

thorn yacht Thanks <@1002092615052840980> report is open https://github.com/NetApp/harvest/...

Thanks @thorn yacht

snow rampart Jul 3, 2023, 2:02 PM

#

hallow thunder What I've seen happen is that nabox loses connectivity to the cluster. In curre...

Thanks @hallow thunder . Harvest already had retries since start. Perhaps it was a different issue causing that. Please do let us know if you face it with recent release. We definitely don’t want to restart poller a due to such issues.

dim folio Jul 20, 2023, 4:42 PM

#

I know this has been addressed but is there some specific reason why this couldn't be a tunable parameter and set to a default and overridden if set in the main config file?

snow rampart Jul 21, 2023, 4:52 AM

#

@dim folio Are you suggesting that we need to adjust the value of this parameter, the retry interval in case of API was rejected, to something other than one hour? We have various retry intervals for different set of errors.

rancid musk Jul 23, 2023, 5:42 AM

#

@snow rampart NAbox uses the poller binary directly, if it dies, there is no mechanism to restart it currently. This might be a problem during upgrades for example. I’ll see if I can improve this. I’m assuming retry in harvest is implemented by harvest binary ?

snow rampart Jul 24, 2023, 7:49 AM

#

@rancid musk Yes, the current discussion regarding retry is related to retrying the collector in the event of errors. It is a separate use case from the scenario where the poller binary terminates unexpectedly.

rancid musk Jul 24, 2023, 8:50 AM

#

Cool, so in that scenario, the poller doesn't quit ?

#No Metric after ONTAP upgrade