gaps in graphs? | NetApp | Page 1

oblique goblet Apr 27, 2023, 2:53 PM

#

Hi!

I've got a rather busy set of clusters and I'm having issues with charts having large blocks in monitoring data.

Am I overloading the appliance?
I've got something like 4400 volumes being monitored with 170440 iops

#

Yeah digging into the nabox specific charts, I think I'm crushing it.

tough otter Apr 27, 2023, 3:06 PM

#

Hi @oblique goblet, could you share your log files with us (https://github.com/NetApp/harvest/wiki/FAQ#how-do-i-share-sensitive-log-files-with-netapp.) and provide your Harvest version? Also, please let us know which panels/counters are showing the gap. Thanks!

oblique goblet Apr 27, 2023, 3:10 PM

#

Absolutely.

Panels and graphs: eventually all of them.
Truly, I have to at least once a day restart services / the host to get graphana reporting stats

tough otter Apr 27, 2023, 3:14 PM

#

If you need to collect logs from NABox, you can follow the instructions provided in the NABox documentation here: https://nabox.org/documentation/troubleshooting/#collecting-logs

oblique goblet Apr 27, 2023, 3:15 PM

#

I'm doing that now 🙂

oblique goblet Apr 27, 2023, 3:35 PM

#

Logs, screenshots of gaps and failure condition to load sent to DL provided in link.

Thanks Rahul!

tough otter Apr 27, 2023, 3:36 PM

#

Thanks we have received it

tough otter Apr 27, 2023, 3:56 PM

#

@oblique goblet We've noticed that all the zapi calls are running very slowly, and it appears to be a systemic issue. Can you please provide information on when you first started experiencing this issue? Also, have there been any recent upgrades to NABox?

oblique goblet Apr 27, 2023, 4:02 PM

#

Our last nabox deployment absolutely tanked when the opt partition filled up unexpectedly.
It was deployed with whichever Ova was available about a month or two ago.
It's been upgraded once since then due to this lockup/performance issue

#

Cluster is also in the middle of a rapid transformation, beginning of the year was an expansion to 16N from 10

tough otter Apr 27, 2023, 4:07 PM

#

Understood. This issue might be related to the workload counters being collected, which can affect the performance of ONTAP depending on the number of workload counters being collected. We have recently received similar reports regarding issues in NABox.

To resolve this, could you try disabling the collection of workload counters in NABox? You can find the instructions on how to do this in the following link #1097886800325922886 message

#

oblique goblet Apr 27, 2023, 4:08 PM

#

Sure, I'll get it shortly

#

I anticipate the cluster shrinking back to 6N within the next month or two, but also the inclusion of SGrid metrics.

oblique goblet Apr 27, 2023, 4:33 PM

#

Deleted the yaml file after taking a backup

#

Do I need to bounce services or anything afterwards?

tough otter Apr 27, 2023, 4:34 PM

#

Yes please bounce services

#

I think command is dc down

oblique goblet Apr 27, 2023, 4:39 PM

#

Got lazy, restarted whole boz

#

Box

oblique goblet Apr 27, 2023, 4:58 PM

#

It appears more performant and stable since reboot

midnight needle Apr 27, 2023, 5:00 PM

#

excellent

oblique goblet Apr 27, 2023, 5:01 PM

#

So what does this mean for the future and supportability is my next question 🙂

midnight needle Apr 27, 2023, 5:02 PM

#

Yann plans on disabling workloads by default to align with Harvest. That means you're already living in the future 😄

oblique goblet Apr 27, 2023, 5:02 PM

#

Ooooo

#gaps in graphs?