How can I find the cause of snapshot growth? Is there a way to find out which qtree is causing it? | NetApp | Page 1

fresh belfry Mar 1, 2023, 4:20 PM

#

As above?

icy eagle Mar 1, 2023, 4:53 PM

#

hi @fresh belfry let me check - while i look does the Volume dashboard > Per Volume Statistics help any? I guess that will show you the growth by volume, svm but does not include qtree. Does the qtree dashboard have what you need?

fresh belfry Mar 1, 2023, 6:00 PM

#

Unfortunately not mate, need to find the busy qtree thats causing the snapshot growth

icy eagle Mar 1, 2023, 6:06 PM

#

yeah I'm not seeing much, qtree-list-iter doesn't have anything snapshot related. We found some 7-mode commands that show snapshot copies associated with qtrees, but we haven't found anything for cmode yet - still looking. @restive belfry do you have any ideas?

#

maybe the Qtree dashboard helps? It is showing top qtrees by disk used like so which isn't quite the same as growth but might point you in the right direction?

restive belfry Mar 1, 2023, 6:08 PM

#

That might work.

#

Or is there a per qtree dashboard?

#

If you have a perf archive we could pull out the qtree stats to see which one wrote the most.

fresh belfry Mar 1, 2023, 6:09 PM

#

Then I would need to know the qtree to blame

restive belfry Mar 1, 2023, 6:09 PM

#

No, just a volume.

icy eagle Mar 1, 2023, 6:09 PM

#

you can filter that dashboard by volume

fresh belfry Mar 1, 2023, 6:11 PM

#

So basically pin the blame on the most ops on the most used qtree?

icy eagle Mar 1, 2023, 6:12 PM

#

perhaps something like rate(quota_disk_used{labels here}[5m])

#

would this help @fresh belfry?

fresh belfry Mar 1, 2023, 6:30 PM

#

That would be perfect

icy eagle Mar 1, 2023, 6:32 PM

#

clone the Top $TopResources Qtrees by Disk Used and change the query to rate(quota_disk_used{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume"}[5m]) change the Standard options / Unit to kibibytes/sec

#

I'll open a PR with that change

icy eagle Mar 1, 2023, 7:30 PM

#

@fresh belfry if you want to try what's in the PR it's up now https://github.com/NetApp/harvest/pull/1792

fresh belfry Mar 1, 2023, 10:48 PM

#

When you said try did you mean try that query?

hearty pumice Mar 2, 2023, 11:14 AM

#

You can import dashboard with above changes from here https://raw.githubusercontent.com/NetApp/harvest/6c6182a1860b13bb9078e371b5f4b11bcba79548/grafana/dashboards/cmode/qtree.json

icy eagle Mar 2, 2023, 12:47 PM

#

fresh belfry When you said try did you mean try that query?

I shared the PR so you could import the dashboard if you wanted to try it out. The query is slightly different than what I pasted above because I tied the "qtrees by disk used growth" panel to the topN variable so you can focus on only the top ones instead of all qtrees. That's why the query in the PR looks like rate(quota_disk_used{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume", qtree=~"$TopQtreeDiskUsedGrowth"}[5m]) instead of rate(quota_disk_used{datacenter=~"$Datacenter", cluster=~"$Cluster", svm=~"$SVM", volume=~"$Volume"}[5m])

if you want to import from the PR, use the link Rahul shared above. Copy that json and then in Grafana click Dashboard > Manage > Import and paste the json in the Import via panel json textarea

fresh belfry Mar 2, 2023, 1:44 PM

#

Thank you, Whats the default grafana user 😂

icy eagle Mar 2, 2023, 1:45 PM

#

typically admin - if you're using nabox details here https://nabox.org/documentation/configuration/

#How can I find the cause of snapshot growth? Is there a way to find out which qtree is causing it?