#Dashboard usage of "average" functions

1 messages · Page 1 of 1 (latest)

blazing kettle
#

Hi, I want to discuss the general strategy of using "average" functions in most dashboards. Performance monitoring is (in my opinion) often about identifying bottlenecks and potential system overloads. Looking at averaged values often mislead to the conclusion that everything is fine, while the peaks are flattened out. This gets especially bad over longer time ranges due to PromQL inner workings. The PromQL function "max_over_time" preserves the peaks, especially when looking at bigger time frames.

Can you give me some insight why average was chosen and if you would consider changing it to a max approach?

#

My company proxy won't let me attach pictures. I can provide an example graph via mail if you wish.

granite bloom
#

@blazing kettle Do you mean queries like this?

In this example, avg_over_time is used within topk() solely for ranking purposes. It determines which LUNs are displayed in the panel, while Grafana plots the actual time series values, preserving any peaks in the visualization.

Choosing avg_over_time for ranking is intentional. it highlights LUNs with the highest sustained load during the time window, rather than those with just a single spike. If max_over_time were used, it might show a LUN that only spiked briefly and was otherwise idle, which is generally less helpful. Each method has its advantages and disadvantages.

image.png
blazing kettle
#

OK that makes sense for the topk logic. But there are a lot of dashboards that have "avg by (...)" as the primary query.

#

I noticed it again today while looking at the "MetroCluster Iwarp" panels.

#

I also still see the problem that peaks are removed or reduced when looking at bigger time frames. Setting time range to 30 days sets the interval to "30m". Using max_over_time (nvm_mirror_write_throughput{cluster=~"$Cluster", datacenter=~"$Datacenter", node=~"$Node"}[$__interval]) instead of nvm_mirror_write_throughput{cluster=~"$Cluster", datacenter=~"$Datacenter", node=~"$Node"} give vastly differing results in peak values.

granite bloom
granite bloom
blazing kettle
#

What's the point of averaging based on primary key?

granite bloom
#

We'll need to check case by case. Could you please open a github issue for this?

blazing kettle
#

Sure

blazing kettle
granite bloom
#

Thanks

granite bloom
#

@blazing kettle The issue is that when viewing longer time ranges, Grafana compresses a large time span into a fixed number of data points, which might not produce the expected results. Using max_over_time can help in some cases, but it isn't suitable for every metric and some users might find avg_over_time or p95 more appropriate depending on their needs.

Having pre-aggregated metrics at lower resolutions, such as 15 or 30-minute summaries, would make plotting more accurate and consistent. Solutions like Thanos or VictoriaMetrics/prometheus recording rules can generate these summaries, supporting functions like max_over_time or avg_over_time. Would you consider this approach to be a more effective solution for your particular use case?

Are you asking a bin/grafana import option that rewrites some panels to use max_over_time? We're unsure if that would work in general or how to determine which panels or expressions to change. We'll need to review this further.

blazing kettle
#

Pre-aggregation has the same constraints as "just in time" aggregation, but loses precision. On top, comes increased load of the TSDB and querying new aggregated series will need more panels or more complex queries. I don't see any advantages in that. I understand that you are hesitant setting max_over_time as a default. That's why I wanted to have a discussion, to see if I missed some important points here. An import option would be fine for me.

granite bloom
#

Applying max_over_time to all panels may not be suitable. Do you have a list of metrics you want to use it for, such as latency or IOPS? This would help make it easier to apply to queries using a regex in an import command.