Error with Volume Time Range > 6Months (Datacenter Variable) | NetApp | Page 1

potent flower Apr 9, 2024, 6:16 AM

#

Nabox Version: 3.3 (Same Error with 3.5.X)
Harvest Version: 24.04.02-nightly

Querying the Datacenter Variable brings the same error (see image)

potent flower Apr 9, 2024, 8:51 AM

#

Error with Volume Time Range > 6Months (Datacenter Variable)

bronze raft Apr 9, 2024, 3:01 PM

#

Hi @potent flower can you check your Javascript console for errors and see if there is anything useful there? The grafana log file should have something useful to. ssh to nabox and adjust the duration (4 hours below) to include the time this error happened
dc logs grafana --since 4h

potent flower Apr 9, 2024, 3:19 PM

#

Hello @bronze raft --since doesnt work

bronze raft Apr 9, 2024, 3:21 PM

#

ah, guess that was introduced in a later version. What does dc logs --help show?

potent flower Apr 9, 2024, 3:23 PM

#

docker logs --since 4h grafana
Worked

#

Sent you a PM. Look if it is right. There is not much out in this log

bronze raft Apr 9, 2024, 3:37 PM

#

thanks! agreed not much there - all info level messages and nothing about timeouts or warnings. Let's check the browser's console log like the Grafana error says

potent flower Apr 9, 2024, 3:49 PM

#

Tried to export it. Sent you a PM again

bronze raft Apr 9, 2024, 3:53 PM

#

are you using Chrome, Firefox or something else? Looks like timeouts, perhaps because you are requesting a year's worth of data. What if you go to the dashboard in question again, switch to the variables page again, open DevTools, switch to the network tab and refresh the variables pages so the network requests happen again. I'd like to see how long it is taking them before timing out

potent flower Apr 9, 2024, 3:55 PM

#

I am using Chrome

bronze raft Apr 9, 2024, 3:55 PM

#

By default, Grafana waits 30s before timing out https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#timeout
Checking to see how to increase that

potent flower Apr 9, 2024, 3:59 PM

#

bronze raft are you using Chrome, Firefox or something else? Looks like timeouts, perhaps be...

even with 6 Months this happens. Sometimes i had this with 90 Days too.

#

You mean this page?

bronze raft Apr 9, 2024, 4:01 PM

#

either way, a timeout is a timeout. The network tab should show us how long these requests are taking. Are your Grafana and Prometheus geographically close to each other or far away?

Yes, the page you pasted at the top with the error. Basically I want to recreate that error with the DevTools network tab open to see how long the request takes before timing out and which part of the network connection is timing out

potent flower Apr 9, 2024, 4:07 PM

#

Ok.
Grafana and Prometheus are on the same Server (Nabox)

#

The Filers are here in the same network to

bronze raft Apr 9, 2024, 4:10 PM

#

that's good, shouldn't be much speed of light latency then

potent flower Apr 9, 2024, 4:27 PM

#

Sent logs to ng-harvest-files@netapp.com

#

Hoipe it is everything there

bronze raft Apr 9, 2024, 4:29 PM

#

received thanks!

bronze raft Apr 9, 2024, 5:40 PM

#

there are eight requests in your har file, one svg, four series queries, and three v1/query. Two of the v1 queries are timing out after 30s. We can increase Grafana's timeout so it doesn't do that. Let's see if we can get an idea what we should increase the timeout to by going to Prometheus, pasting this query, and executing to see how long it takes Prometheus to return the data. topk(5,sum by (volume)(avg_over_time(fabricpool_cloud_bin_operation{datacenter=~".*",cluster=~".*",svm=~".*",volume=~".*"}[31622400s])))

Prometheus will show how long it took to execute the query as "Load time:"

potent flower Apr 9, 2024, 7:45 PM

#

This query is running long and eating my Mem

#

potent flower Apr 9, 2024, 8:08 PM

#

bronze raft Apr 10, 2024, 3:25 PM

#

that's good and bad news 🙂 now we know why Grafana is timing out. Did that query ever finish? I wonder if more RAM would help? Prometheus may not be up to the task of asking for a year's worth of data. Is this your production or dev instance? One idea would be to upgrade to Nabox v4 (beta). Nabox 4 uses VictoriaMetrics instead of Prometheus and my reading has been that it may be faster on long range queries

potent flower Apr 10, 2024, 4:15 PM

#

We never had Problems with Queries for 1 Year and have the need for it for analizing of Aggregate or Volume growth over longer periods.
The Problem can be reporduced even with 6 Months worth of data.
The Query never finished.
This is our Prod Instance and this is the reason why we cant upgrade to a Beta version.

#

In the last month a large amount of volumes where moved to different aggregates and then back. Can this cause inconsistencies in the DB?

bronze raft Apr 10, 2024, 6:27 PM

#

thanks for the additional details. It was not clear that this is a new problem and that one year queries worked in the past. Moving volumes to different aggregates will increase the number of records that Prometheus needs to scan since, from Prometheus POV, those new labels create a different time-series

potent flower Apr 10, 2024, 6:27 PM

#

Do we have a chance to fix this?

bronze raft Apr 10, 2024, 6:33 PM

#

maybe you could delete the transient moves? Assuming that you have a way of identifying those records and you're OK with losing that data. I'll try to think of other options too

#

More RAM might help too, like I mentioned earlier, if it's straightforward to add more, that would be a quick check

potent flower Apr 10, 2024, 6:34 PM

#

Ok i will try to double RAM tomorrow and test again

#

I dont know how to identify the data but i do know which aggregates and filer it was

bronze raft Apr 10, 2024, 6:38 PM

#

that's good to know. Do you care about the fabricpool_cloud_bin_operation metric? That's the one that is taking so long and is used by the Top Volume Object Storage row in the volume dashboard. That metric is a histogram which means there are more series than usual. If you don't care about it - maybe the other queries are fast enough if we remove that one?

potent flower Apr 10, 2024, 6:43 PM

#

bronze raft that's good to know. Do you care about the `fabricpool_cloud_bin_operation` met...

Would like to test that to. I need to delete this part of the dashboard right?
I will ask my collegues if its important then if this really helps.
But we have the same Problems on the Aggregate Dashboard to...and maybe many more that we dont know right now?

#

It is always present if i try to update the Datacenter variable

bronze raft Apr 10, 2024, 6:46 PM

#

if you go to Prometheus and query volume_labels does it take a long time? That query should be fast and if it isn't maybe this system is in a bad state from earlier queries. Will be good to see how long that query is now and then tomorrow after more RAM

potent flower Apr 10, 2024, 6:48 PM

#

Is100ms seconds fast? 😅

#

Should we try that datacenter query?

bronze raft Apr 10, 2024, 6:54 PM

#

good enough! That means when you see a slow response time when upding the datacenter variable, it's probably because other variable queries are running too. You can confirm that theory by opening dev tools first and finding the datacenter query in the network tab. I suspect it will be fast there and other queries are slow

potent flower Apr 10, 2024, 7:00 PM

#

Yeah i understand. We did this yesterday already and i setn you the har file of the dev tools network tab.
Which queries do volume and aggregate dashboard have in common?

potent flower Apr 11, 2024, 6:58 AM

#

@bronze raft Tried Query with 32G RAM (Doubled from 16G) and the same result:

bronze raft Apr 15, 2024, 1:11 PM

#

hi @potent flower today's nightly build contains the dashboard improvements we discussed last week. When you get a chance, can you give it a try and let us know how they work for you? https://github.com/NetApp/harvest/releases/tag/nightly

potent flower Apr 16, 2024, 2:43 PM

#

First Tests are showing that everything is much faster. And we have no longer timeouts it seems.
Digging deeper.

bronze raft Apr 16, 2024, 2:48 PM

#

Excellent!

potent flower Apr 16, 2024, 3:00 PM

#

In Volume Dashboard

bronze raft Apr 16, 2024, 3:03 PM

#

can you click on that panel and press e to edit and then paste the query from the Metrics browser?

potent flower Apr 16, 2024, 3:03 PM

#

#

If i set 12h

#

volume_total_data{datacenter=~"$Datacenter",cluster=~"$Cluster",svm=~"$SVM",volume=~"$Volume"}
and
topk($TopResources, avg_over_time(volume_total_data{datacenter=~"$Datacenter",cluster=~"$Cluster",svm=~"$SVM",volume=~"$Volume"}[3h] @ end()))

bronze raft Apr 16, 2024, 3:04 PM

#

ok works with 12h and does not work with?

potent flower Apr 16, 2024, 3:04 PM

#

works with 12h but only shows 1/2 an hour?

#

Ok it gets strange

#

potent flower Apr 16, 2024, 3:06 PM

#

bronze raft ok works with 12h and does not work with?

No data was with 6M

#

And with 90D NoData to

bronze raft Apr 16, 2024, 3:10 PM

#

that panel was changed recently to fix https://github.com/NetApp/harvest/issues/2787 the fix uses volume_total_data instead summing volume_write_data and volume_read_data in the panel the way it was previously. Summing the values in Grafana caused problems when a volume's node changed. The current fix does not have that problem

potent flower Apr 16, 2024, 3:24 PM

#

As soon as i select 90D i have nodata. Even if i select only one filer. So it is not the query running long

bronze raft Apr 16, 2024, 3:25 PM

#

only for the Top 5 Volumes by Average Throughput panel or other panels on the Volume dashboard?

potent flower Apr 16, 2024, 3:26 PM

#

Other Panels to Mom

#

#

Same Panel in Aggr Dashboard to

bronze raft Apr 16, 2024, 3:35 PM

#

thanks. Yes, the aggregate dashboard Volume by Average Throughput panel had the same bug and fix as mentioned above in 2787. Each of the dashboards listed https://github.com/NetApp/harvest/pull/2786/files were updated to reflect accurate throughput

GitHub

fix: Using volume_total_data instead of read_data and write_data by...

potent flower Apr 16, 2024, 3:38 PM

#

And the Aggregate Dashboard brings the Timeout Errors that i had with Volume .
We are missing data Total Troughput to in Vol Dashboard i think. (See screens)

#

It uses the same Metric it seems:

sum(topk($TopResources, volume_total_data{datacenter=~"$Datacenter",cluster=~"$Cluster",svm=~"$SVM",volume=~"$Volume"}))

bronze raft Apr 16, 2024, 3:40 PM

#

yes

#

i understand the issue - the fix for 2787 changed several of the dashboards to use volume_total_data for the panels, which is correct. But volume_total_data was introduced in this commit on Feb 12. That means that metric does not exist before Feb 12 and is why you have no data when selecting a larger time range

potent flower Apr 16, 2024, 4:06 PM

#

Ok. Got it.
But maybe it would be better to leave both for a few months

bronze raft Apr 16, 2024, 4:17 PM

#

that might work, but if we do that you will hit the problem that caused us to move to volume_total_data which was this error

potent flower Apr 16, 2024, 4:20 PM

#

Oh ok

#

Will keep that in Mind

bronze raft Apr 16, 2024, 4:21 PM

#

let me try backing out the volume_total_data change but keep the faster query change and see if that works for you

potent flower Apr 16, 2024, 4:22 PM

#

If it is extra for me dont struggle. We will have to live with that

#

Only give me the old query so that i can build one own Panel if we need that old data

bronze raft Apr 16, 2024, 4:33 PM

#

give this a try https://gist.github.com/cgrinds/6c2c14163fab27676cd2ea0345d955e3

potent flower Apr 16, 2024, 4:42 PM

#

The new Vol Dashboard is really fast. What have you changed?

#Error with Volume Time Range > 6Months (Datacenter Variable)