Looking for specific metrics ... | NetApp | Page 1

solid lotus Feb 7, 2025, 10:38 AM

#

Using nabox (4.0.7) there is a lot of information in the various dashboards, which is just great! But, sometimes I get a bit lost 🙂 E.g. I'm pretty sure I once saw a panel with QoS metrics/info. ... But I can't find it now.
Similarly I was hoping that I might find a Dashboard/Panel/Graph showing ONTAP CP (Consistency Point) information, but no luck so far. Is there such a thing in nabox?

cloud gyro Feb 7, 2025, 10:50 AM

#

@solid lotus You can find CP in disk dashboard as below

#

For QOS metrics, Workload dashboard should have it.

#

solid lotus Feb 7, 2025, 11:02 AM

#

cloud gyro <@1054453340999782431> You can find CP in disk dashboard as below

Found the CPs - Thanks! The graph / panel is labelled "CP Counts" but y-axis is labelled "write latency" ... Do you know what it means? I guess each peak isshow the time it took to do a individual CP?

#

Aren't there also different types of CPs in WAFL / ONTAP?

solid lotus Feb 7, 2025, 11:03 AM

#

cloud gyro

Found the QoS too, thank you. Looks as if it is turned off by default. So we'll have to think about if we should enable it or not ...

cloud gyro Feb 7, 2025, 11:04 AM

#

That's 2 metrics in that panel as below

#

Only 2 types of wafl CP are shown there.

#

https://netapp.github.io/harvest/24.11.1/ontap-metrics/#wafl_cp_count
https://netapp.github.io/harvest/24.11.1/ontap-metrics/#node_vol_write_latency

cloud gyro Feb 7, 2025, 11:06 AM

#

solid lotus Found the QoS too, thank you. Looks as if it is turned off by default. So we'll ...

Qos metrics will be enabled by default in Harvest from upcoming Harvest version.

cloud gyro Feb 7, 2025, 11:06 AM

#

solid lotus Aren't there also different types of CPs in WAFL / ONTAP?

We'll improve the description there to make it clear.

solid lotus Feb 7, 2025, 11:08 AM

#

Humm, actually what I see here is both axis labelled "write latency" and no sign of any counts or a y-axis on the right hand side

#

Underneath the graph are just a list of write latencies per aggr, but no other info.

cloud gyro Feb 7, 2025, 11:13 AM

#

If you click on Back-to-back CP count that should show CP

cloud gyro Feb 7, 2025, 11:13 AM

#

solid lotus Humm, actually what I see here is both axis labelled "write latency" and no sign...

hmm.. What is the Harvest version?

#

Could you share screenshot how does this panel look?

solid lotus Feb 7, 2025, 11:19 AM

#

Probably can a screenshot. I have to tx it out of ... where it is. Inspect json shows me references to cp counts, so it seems that there should be more info. ... Maybe the graph is just being rendered incorrectly somehow

#

4.0.7

cloud gyro Feb 7, 2025, 11:20 AM

#

You can try reimporting it from here in grafana and check if it fixes the issue
https://raw.githubusercontent.com/NetApp/harvest/refs/heads/main/grafana/dashboards/cmode/disk.json

#

I meant to ask Harvest version as shown here in NABox

solid lotus Feb 7, 2025, 11:22 AM

#

How would I do that? (sorry - I'm (not yet) a Grafana professional :-/)

cloud gyro Feb 7, 2025, 11:23 AM

#

https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/

#

solid lotus Feb 7, 2025, 11:24 AM

#

Aha. Harvest version is 24.11.1

#

It was imported via the nabox upgrade page to replace the version included with 4.0.7

#

Reimport from github won't work in any case, the site has no direct access to internet

cloud gyro Feb 7, 2025, 11:28 AM

#

You can copy json as well

#

solid lotus Feb 7, 2025, 11:57 AM

#

Humm. Fond the import function, but it doesn't seem to want to overwrite the existing definition.

#

Maybe it would be easier to just upgrade the whole think to 4.0.8 ?

cloud gyro Feb 7, 2025, 12:04 PM

#

Okay

#

May be upgrading to Harvest nightly build is better.
https://github.com/NetApp/harvest/releases/tag/nightly

#

NABox 4.0.8 will not help

solid lotus Feb 7, 2025, 1:17 PM

#

I see this (sorry about the hover-popup)

cloud gyro Feb 7, 2025, 1:22 PM

#

OK could you share your Grafana version?

#

I see the issue. I'll share fix.

#

As a workaround for now, Could you modify panel query to below if NABox allows that.

sum(wafl_cp_count{datacenter=~"$Datacenter",cluster=~"$Cluster",node=~"$Node",metric=~"back_to_back_CP|deferred_back_to_back_CP|back_to_back_cp|deferred_back_to_back_cp"})

solid lotus Feb 7, 2025, 1:55 PM

#

I'll try the mod, some things don't seem to be modifiable / deleteable (if that's a word) I get an error about teir being "provisioned" (or similar)

tidal umbra Feb 7, 2025, 1:57 PM

#

maybe a Save As will work instead of a Save

cloud gyro Feb 7, 2025, 1:57 PM

#

I have started a nightly build for you to upgrade.

#

@short field for dashboard modification in NABox

tidal umbra Feb 7, 2025, 1:57 PM

#

@short field has a change, not yet published, that will let you edit the out-of-the-box dashboards too

short field Feb 7, 2025, 1:58 PM

#

I should probabl release that before the Top Client dashboard checkbox

solid lotus Feb 7, 2025, 2:09 PM

#

Well Hey ... that worked (mostly). I changed the query, hit Apply and the CP panel is updated. Now I have a RHS y-axis labelled "Back to Back CP Count" 👍🏻

cloud gyro Feb 7, 2025, 2:10 PM

#

Nice!

solid lotus Feb 7, 2025, 2:11 PM

#

Two things occur to me ... the first row of the table, below the graph is showing the query e.g. "sum(walf_cp_ ..." the next rows show "Write Latency" values .... looks strange

cloud gyro Feb 7, 2025, 2:12 PM

#

This panel is mostly trying to show write latency and cp count together for issue like below

https://kb.netapp.com/on-prem/ontap/Perf/Perf-KBs/Write_Performance_Impacted_by_Back_to_Back_Consistency_Points
https://kb.netapp.com/on-prem/ontap/Perf/Perf-KBs/What_are_the_different_Consistency_Point_types_and_how_are_they_measured

NetApp Knowledge Base

Write Performance Impacted by Back to Back Consistency Points

NetApp Knowledge Base

What are the different Consistency Point types and how are they mea...

solid lotus Feb 7, 2025, 2:13 PM

#

Second thought ... Is it true? I though "back to back CP" events should be rare and v. not good. This seems to be showing me them happening all the the time?

cloud gyro Feb 7, 2025, 2:14 PM

#

@red cradle can help in domain side of this. See if KB articles help.

#

#1062049169520476220 channel can also help.

solid lotus Feb 7, 2025, 2:16 PM

#

Good idea, I should know what I'm talking about, before talking. I'll read the KBs

#

In edit I see "Legend: Auto" ... maybe that's why I'm seeing the "raw" query displayed in the table

cloud gyro Feb 7, 2025, 2:19 PM

#

Its like this for me. Is it any different for you?

#

Fix is available in https://github.com/NetApp/harvest/releases/tag/nightly
You can upgrade to this in case.

solid lotus Feb 7, 2025, 2:23 PM

#

Dude, it's Friday! Never change a running system (on a Friday afternoon)🔥

#

Seriously though ... I could consdier nightly ... But in general how stable is it? We have (we will have) users who will want to see reliable data ...

tidal umbra Feb 7, 2025, 2:26 PM

#

it's stable, runs through all CI, regression, and unit tests, and it's what we run leading up to our next release

solid lotus Feb 7, 2025, 2:27 PM

#

Also .. I still have to get back to getting my custom Grindstaff ISL interface counters working 😊

solid lotus Feb 7, 2025, 2:33 PM

#

cloud gyro Its like this for me. Is it any different for you?

For me, after changing the "A" metric query and Applying, then Option -> Legend was set to "Auto" which resulted (AFAICT) in the whole query being used as the "Name" in the table under the graph

cloud gyro Feb 7, 2025, 2:35 PM

#

Okay. You can set Legend to Back-to-back CP Count for A

solid lotus Feb 7, 2025, 2:39 PM

#

Yeah ... but ... really shouldn't it say something like "back2back" + a variable i.e. the node name I think would be correct? (I think of CP ops as a Node / ONTAP instance level concept)

#

I changed it to be "BOBs {{$Node}}" ... but that results in the Name in the table becoming the literal string "BOBs {right-axis}"

#

Too many braces {} ?

tidal umbra Feb 7, 2025, 2:43 PM

#

for legends it is typically {{name}} example

#

but I think the Back-to-back CP Count legend is acting different, likely because it has right placement

#

for example

#

ah, i see the problem. let me try something

solid lotus Feb 7, 2025, 2:51 PM

#

I'm glad you guys understand this 🤣

#

FYI: In your last screenshot: "Foo (right y-axis)" under "Name" is also what I have achieved 🤪

tidal umbra Feb 7, 2025, 3:00 PM

#

Give this a try
query: sum by (datacenter, cluster, node) (wafl_cp_count{datacenter=~"$Datacenter",cluster=~"$Cluster",node=~"$Node",metric=~"back_to_back_CP|deferred_back_to_back_CP|back_to_back_cp|deferred_back_to_back_cp"})
legend: Back-to-back CP Count {{node}}

solid lotus Feb 7, 2025, 3:11 PM

#

OK. I'll tx. that query over and try it. It'll take me a couple of minutes

#

CP Info and RHS axis have disappeared from the graph. Table undeneath has only "write latency" entries

#

Let's forget it for now.

tidal umbra Feb 7, 2025, 3:20 PM

#

Here's what I see with the changes pasted above

solid lotus Feb 7, 2025, 3:26 PM

#

That's nice 🙂 Let me try it again - probably screwed something up here. Meanwhile, you can read email to ng-harvest-files?

tidal umbra Feb 7, 2025, 3:26 PM

#

yes

solid lotus Feb 7, 2025, 3:29 PM

#

I sent you a screenshot of a different design

tidal umbra Feb 7, 2025, 3:37 PM

#

thanks. At the moment, Harvest does not have the CP durations, which is what the PAS CP dashboard is showing. Let me check and see if the ZAPI includes that information (I know Rest does not)

solid lotus Feb 7, 2025, 3:48 PM

#

Right, that data is not being generated via Harvest

#

But it does look cool:-)

tidal umbra Feb 7, 2025, 3:48 PM

#

we collect cp_phase_times which is percentage time spent in different phases of CP. Looks like there is a counter for total_cp_msecs which is Milliseconds spent in CP but that won't be broken down by phases of CP.

#

so far, I don't see anything in the wafl perf object that includes the durations. PAS must be collecting that info some other way

solid lotus Feb 7, 2025, 3:55 PM

#

That's too deep for me :-). I could poke around inside a perf autosupport archive and see if i can identify anything, bu tit would be a emperical approach

tidal umbra Feb 7, 2025, 3:56 PM

#

that's OK, I'll ask around and see what I can find out

solid lotus Feb 7, 2025, 4:26 PM

#

BTW: Rahul's query line works. Just saying 😎

tidal umbra Feb 7, 2025, 4:39 PM

#

yes, we reviewed before he sent. Try this
sum by (datacenter, cluster, node) (wafl_cp_count{datacenter=~"$Datacenter",cluster=~"$Cluster",node=~"$Node",metric=~"back_to_back_CP|deferred_back_to_back_CP|back_to_back_cp|deferred_back_to_back_cp"})

solid lotus Feb 7, 2025, 4:43 PM

#

On it. ( I was just trolling you with that last one ;-))

solid lotus Feb 7, 2025, 5:05 PM

#

So I cloned the complete Dashboard and applied your changes (query + legend). There it works. Go figure 🤷🏻‍♂️

#

Sent you a screenshot via email

tidal umbra Feb 7, 2025, 5:07 PM

#

thanks! I'll check on the scatter plot, doubt that Harvest has the data to show that. Either way, sounds like you would prefer we change this dashboard to use the latest query. Is that correct?

solid lotus Feb 7, 2025, 5:12 PM

#

For sure. Before, here, I wasn't seeing any CP info in that graph. So this is definitely better - IMHO. Still have that slight doubt that these events are really all b2b CPs ... But I need to do my KB homework

#Looking for specific metrics ...