NABOX 2 vs 3 - Metrics difference? | NetApp | Page 1

timid epoch Jan 23, 2023, 9:53 AM

#

When viewing node backend latency as an example in Grafana (using Nabox 2.6.4) , i get avg read latency coming back as 25ms , this matches what I see in ActiveIQ, however with the latest version of NABOX deploy (v3.1.2) I am seeing avg read lantecy coming back as 2ms for the same time periods. Why would this be so vastly different? As a result grafana alerts are not being triggered when they hit thresholds.

kindred reef Jan 23, 2023, 10:08 AM

#

@timid epoch Could you share the harvest versions in these 2 boxes.

timid epoch Jan 23, 2023, 10:10 AM

#

older one is Harvest 1.6 and in the newer setup we have Harvest 22.08.0-1`

kindred reef Jan 23, 2023, 10:11 AM

#

Thanks. Could you share the metric name in harvest 2 which is not matching?

timid epoch Jan 23, 2023, 10:11 AM

#

volume_read_latency

kindred reef Jan 23, 2023, 10:12 AM

#

okay do you see differences only for this metric or for other also?

timid epoch Jan 23, 2023, 10:13 AM

#

volume_write_latency also looks lower

#

same node, same time period

kindred reef Jan 23, 2023, 10:15 AM

#

okay. We have made few fixes related to perf metrics for our latest release.
https://github.com/NetApp/harvest/releases/tag/v22.11.0
Could you upgrade to this version and see.

kindred reef Jan 23, 2023, 10:31 AM

#

We did test this metric last year against ONTAP CLI. See if this page helps
https://github.com/NetApp/harvest/wiki/Volume-latency-for-flexgroup

timid epoch Jan 23, 2023, 10:35 AM

#

Sorry should this auto update, or is there a process for this updating this? first time updating harvest like this

kindred reef Jan 23, 2023, 10:37 AM

#

See steps here
https://nabox.org/documentation/upgrade/
Just need to upload harvest binary to NABox

timid epoch Jan 23, 2023, 11:01 AM

#

Does it take a while to upgrade i grabbed the harvest-22.11.0.tar.gz file and installed which was successful but under maintenance page the NetApp Harvest version has completely dropped off

#

also saying no clusters actively reporting anymore on the main dashboard

#

is a restart required?

kindred reef Jan 23, 2023, 11:03 AM

#

Usually it happens pretty fast and restart happens internally in NABox

timid epoch Jan 23, 2023, 11:04 AM

#

any way to check the status of it from SSH a bit more?

kindred reef Jan 23, 2023, 11:05 AM

#

yes let me dig those up

#

lets login to nabox with root user through ssh https://nabox.org/documentation/configuration/

timid epoch Jan 23, 2023, 11:06 AM

#

yep done

kindred reef Jan 23, 2023, 11:08 AM

#

if you run dc ps what does it show?

timid epoch Jan 23, 2023, 11:09 AM

#

nabox-harvest2 python3 harvest.py Restarting

#

guessing its stuck

kindred reef Jan 23, 2023, 11:09 AM

#

yes

#

lets try restarting
dc down

timid epoch Jan 23, 2023, 11:11 AM

#

everything was up but now harvest3 is in a restarting mode again

#

sorry harvest2

kindred reef Jan 23, 2023, 11:11 AM

#

Could you confirm if this is the binary you have uploaded?
https://github.com/NetApp/harvest/releases/download/v22.11.1/harvest-22.11.1-1_linux_amd64.tar.gz

timid epoch Jan 23, 2023, 11:12 AM

#

no it wasnt, see smaller once which was downloaded

#

ill upload from that link youve sent

#

it wasnt clear which to download from git but i guess I should of asked first

kindred reef Jan 23, 2023, 11:13 AM

#

that smaller one looks wrong

#

yes let's upload the one which I have shared

timid epoch Jan 23, 2023, 11:14 AM

#

uploading.....

#

cool that worked

kindred reef Jan 23, 2023, 11:22 AM

#

nice!

timid epoch Jan 23, 2023, 11:22 AM

#

back up and running

#

metrics still different by the looks

kindred reef Jan 23, 2023, 11:23 AM

#

okay could you compare it with ontap cli metrics for a volume
https://github.com/NetApp/harvest/wiki/Volume-latency-for-flexgroup

#

We should match with ontap

timid epoch Jan 23, 2023, 11:24 AM

#

this is for a flexvol not flexgroup if that matters

kindred reef Jan 23, 2023, 11:24 AM

#

ok no problem. That page will be valid for flexvol also

timid epoch Jan 23, 2023, 11:30 AM

#

For comparison I am comparing the Node dashboard for Backend Drilldown panel for Latency on both old and new, as I say ActiveIQ matches will the harvest 1.6 metrics

#

I think this disparity applies to many more metrics between the two

#

the difference being harvest version

#

throughput and IOPs for that same dashboard look to be very close

#

so that seems ok

kindred reef Jan 23, 2023, 11:34 AM

#

okay It could be the way volume summary is being calculated between these 2. Let me check

timid epoch Jan 23, 2023, 11:34 AM

#

thanks

#

just checking another in the same dashboard "Protocol Backend IOPs" this looks ok too, hopefully you can find something different in the latency figures, that one seems to jump out as having the biggest dif that I have spotted.

kindred reef Jan 23, 2023, 11:37 AM

#

sure

timid epoch Jan 23, 2023, 11:52 AM

#

In addition to the read_volume_latency I think the write and other latency metrics in the same panel may also be suffering similar differences

kindred reef Jan 23, 2023, 11:54 AM

#

yeah i suspect in harvest 1.6, it was weighted average while harvest 2, we are just averaging in panels

timid epoch Jan 23, 2023, 11:59 AM

#

would AIQ also work in the same way? as it shows the same figures as H1.6

kindred reef Jan 23, 2023, 12:00 PM

#

I am not sure about AIQ. But looks like individual metric i.e. volume_read_latency is fine but it's summary at node level is not matching due to weighted average vs plain average

#

@timid epoch Thanks for raising this. I have opened an issue for this. https://github.com/NetApp/harvest/issues/1666

timid epoch Jan 23, 2023, 12:07 PM

#

Cool. thanks for looking into. How do bug fixes work these days, just wondering typically what the time to fix normally is once a bug has been raised? we may need to leave the old system in place is all

kindred reef Jan 23, 2023, 12:08 PM

#

I have added it for next release which is due some time mid or later of next month.

#

We'll update you for you to try as soon as fix is ready and available through our nightly builds so that you don't have to wait for official release.

kindred reef Jan 30, 2023, 3:51 PM

#

@timid epoch This issue is now fixed in our nightly builds. https://github.com/NetApp/harvest/releases/tag/nightly
If you get a chance, let us know if this fixes the issue.

timid epoch Jan 30, 2023, 4:16 PM

#

excellent, thanks will give it a go and let you know

timid epoch Jan 30, 2023, 4:33 PM

#

out of interest at what point does this build become classed as a stable build?

kindred reef Jan 30, 2023, 4:34 PM

#

We release harvest every 3 months. So next stable release is due sometime in feb.

kindred reef Jan 30, 2023, 5:28 PM

#

When you try this nightly build, do update harvest dashboards as well.

timid epoch Jan 30, 2023, 7:20 PM

#

Ah OK. How do I do that sorry?

kindred reef Jan 31, 2023, 5:04 AM

#

After upgrading harvest as mentioned here https://nabox.org/documentation/upgrade/#ugrading-netapp-harvest
You can upgrade existing harvest dashboards with Reset button (as in screenshot) which will overwrite default harvest dashboards.

timid epoch Jan 31, 2023, 11:30 AM

#

applied the nightly build and reset dashboards but doesnt look to have changed at all

kindred reef Jan 31, 2023, 11:32 AM

#

okay. Let's verify if dashboard is updated or not. If you open latency panel under backend drilldown. What are the queries it is pointing to?

#

Also verify once if harvest version is updated on nabox UI

timid epoch Jan 31, 2023, 11:40 AM

#

Yep Harvest defo updated to nightly build version

#

Screenie of backend drilldown latency panel

#

kindred reef Jan 31, 2023, 11:41 AM

#

that looks correct

#

means dashboard also updated fine

#

do u mean values still dont match with harvest 1.6?

#

perhaps compare last 5-10 mins of data

#

as older values before upgrade would still be wrong

timid epoch Jan 31, 2023, 12:05 PM

#

I think I will need to monitor it, they do look a lot closer now when cross checking different nodes in the cluster

kindred reef Jan 31, 2023, 12:05 PM

#

great. sure do let us know!

timid epoch Jan 31, 2023, 2:57 PM

#

Also had to remember to update all my alert rules config to use the new queries i.e. node_volume_read_latency, worth noting.

timid epoch Feb 2, 2023, 10:13 AM

#

Still looking good on this front, however I have noticed in the same Backend drill down section, that under system utilisation panel, that CPU Busy metrics are showing different figures between Harvest 1.6 and 2. Could the same be happening with the way averages are worked out? For example Harvest 1.6 system showing avg 65% CPU busy and Harvest 2 showing 74% for the same time period.

#

wondering how many other panels work out avgs different between the Harvest versions. These are just ones I have spotted.

kindred reef Feb 2, 2023, 10:23 AM

#

Thanks @timid epoch for the update. About node_cpu_busy, Yes there are differences and Harvest 2 metric is correct. More information here https://github.com/NetApp/harvest/discussions/1668

#

There shouldn't be any other metric having this weighted average problem but we'll take a note of this and verify all of them before next release.

#

Added a issue for verification https://github.com/NetApp/harvest/issues/1680

timid epoch Feb 2, 2023, 10:28 AM

#

Yeh actually if I view the CPU Layer Drilldown, this is the same on both systems, so as the link says the sys util panel is now including other processor related activities

#NABOX 2 vs 3 - Metrics difference?