#Upgraded Harvest - all systems rejected

1 messages · Page 1 of 1 (latest)

marsh pumice
#

I upgraded Harvest to the latest daily today to get PR 2343 but now all of my systems have gone offline in nabox.
e.g.
2023-09-18T15:39:25Z ERR poller/poller.go:161 > Failed to find poller error="configuration error => poller [CVVFILER1] not found" Poller=CVVFILER1 config=harvest.yml harvest=harvest

The system is clearly still there in /opt/harvest2-conf/harvest.yml

I toggled it back on in nabox and it quickly toggles itself back off again.

#

This is nabox 3.3

gleaming mason
#

hi @marsh pumice apologizes that's a bug caused by the Harvest nightly. If you roll back to 23.08, you won't hit the bug, but also won't have a fix for 2343. Nightly should be fixed within the hour (fix is running through CI as we speak). I'll post when CI passes and nightly is green again

marsh pumice
#

Thanks for the fast response - I uploaded 23.08 and it's collecting again. And that's why the nightlies are beta 🙂

gleaming mason
#

yw. yes, we try to keep them green. I've added two new unit tests to cover the miss that caused nightly to fail so our green streak should return soon 🙂

gleaming mason
marsh pumice
#

Thanks @gleaming mason. I installed the latest nightly and reset the dashboards. None of the new panels (Issue #2235, PR # 2343) are visible.

"We have added 4 new panels in Volume Dashboard in Top Volume FabricPool Drilldown panel". I do not see these new panels.
"We have added 3 panels in Aggregate dashboard in newly added Aggregate FabricPool Drilldown panel". I do not see these panels.
"We have added 4 new panels in Aggregate dashboard in Volume Statistics panel." I have the Volume Statistics panel but not the new performance tier panels.

What am I missing? Thanks!

oak ingot
#

Did you Reset the dashboards from NAbox web UI ?

potent yarrow
marsh pumice
#

Thanks @potent yarrow . That makes sense. I'll be more patient...

potent yarrow
marsh pumice
# potent yarrow <@743109189810913330> The PR https://github.com/NetApp/harvest/pull/2343 is comm...

Thanks @potent yarrow . I installed the nightly and it's looking good, EXCEPT that "Top Volume FabricPool" is ignoring all of the FlexGroup constituents. When aggregates get full, we also consider the FG constituents when deciding what volumes to move around so it's important that they get included too. On the first aggregate I checked, 3 of the top 5 volumes by performance tier utilization are FG constituents yet were not listed on the dashboard. And now that I look around some more, there appears to be almost no support for FlexGroup constituents in the Volume dashboard at all.

potent yarrow
#

Hi @marsh pumice , You're right. Volume dashboard doesn't show Flexgroup constituents and we're not quite sure how to best integrate that with the other volume panels. Most of the place we roll-up constituents, but sounds like you want to see metrics for the constituents themselves. We've got an existing Flexgroup issue https://github.com/NetApp/harvest/issues/2142 that sounds similar. Can you add your +1 to that issue?

marsh pumice
#

Done, thanks. It's not an easy fix since I don't believe Grafana allows a user-friendly toggle like "include constituents" on a graph...

potent yarrow
#

Hi @marsh pumice

As part of issue https://github.com/NetApp/harvest/issues/2142, we have added a new flexgroup dashboard with flexgroup constituents. This dashboard also includes the aggregate and flexgroup filtering you asked for. By-default, flexgroup constituents are not collected since there may be a large number of instances. The latest nightly includes these changes if you want to give them a try. To enable collection of flexgroup constituents, checkout https://github.com/NetApp/harvest/wiki/FAQ#flexgroup-dashboard

marsh pumice
#

Thanks @potent yarrow . I pulled down the daily, made the edits, and did a dc restart in nabox. The include_constituents variable reverted back to false. Where do I need to make the change so it sticks?

gleaming mason
#

Hey Ed, we'll update the documentation Hardik shared to make it clearer https://github.com/NetApp/harvest/wiki/FAQ#flexgroup-dashboard The gist is since nabox overwrites the out-of-the-box template, you'll need to extend the template(s) instead of modifying them directly. Extending is better anyway, perhaps you already do that for other templates?

marsh pumice
#

I have updated harvest.yml but haven't extended any of the templates. I briefly looked at it but the process looks rather cumbersome and appears to be release specific. There would ideally be a place that I can put this setting to be in affect for all releases going forward. Without it, I envision a time after I'm gone when an admin goes "we updated to 9.15 and my dashboards are now empty!" conf/rest has directories from 9.6.0 to 9.12.0 and conf/restperf has 9.12.0 to 9.14.1".

'Instead of editing one of the existing templates, it's better to copy one and edit the copy. That way, your custom template wil
l not be overwritten when upgrading Harvest. For example, if you want to change conf/rest/cdot/9.12.0/aggr.yaml, first create
a copy (e.g., conf/rest/cdot/9.12.0/custom_aggr.yaml), then add these lines to conf/rest/custom.yaml:"

not quite what I think I want (but I could be wrong). This is where I know I'll get burned:

"After restarting pollers, aggr.yaml will be ignored and the new, custom_aggr.yaml subtemplate will be used instead."
So when aggr.yaml changes going forward, I'll have to redo the customization.

My thoughts:
At the tail end of the current yaml files, add an "include: custom_FOO.yaml" in a non release-specific directory that lets users override the defaults. Include empty custom_FOO.yaml files so the search will pass and users will know where the file needs to be.

oak ingot
#

Note that NAbox will overwrite existing files but will leave in place any new files you might have added

potent yarrow
marsh pumice
marsh pumice