Harvest+Grafana and Alerts for multiple clusters

[native ivy]

I'm trying to find a way where we can create one alert which can be applied to all clusters say… An alert when certain volume utilization is at 90%. Right now, we need to create one alert per data source, which will be hard to manage. We would want to setup one alert which can be applied centrally to all Cluster and if we need any change, change in one value say threshold update to 95% will get applied for all volumes on all clusters.

Is it possible to have a single configuration for the alerts that can apply to many clusters? Thank you!

[river lodge]

@native ivy , Why wouldn't you use OnCommand Unified Manager for this?

[glad arrow]

@native ivy You can create alerts across clusters as well. Here is the reference document for the same.
https://netapp.github.io/harvest/23.05/prometheus-exporter/#prometheus-alerts

Alert rules:https://github.com/NetApp/harvest/blob/main/container/prometheus/alert_rules.yml

[native ivy]

Hello, the end user is evaluating both, to see which is easier to maintain in their organization. They do like the perf stats collected every one minute in Harvest. Also there is a question about if longer term AIQUM will move to a SaaS product. The comfort level is to maintain an on-prem deployment.

[native ivy]

Thank you Rahul, I did see those documents. But I wasn't quite sure which part of the alert_rules.yml to modify? This one:
groups:

• name: Harvest Rules
  rules:

Or, can you clarify if to do grouping I should be looking at the Alertmanager from prometheus. https://prometheus.io/docs/alerting/latest/alertmanager/

[tawdry lily]

@native ivy I have tried one sample rule something like this in alert_rules.yml file

Alert for logical used percent > 60

• alert: Volume Logical Space Used Percent Higher than 60
  expr: volume_space_logical_used_percent{} > 60
  for: 5m
  labels:
  severity: "critical"
  annotations:
  summary: "Volume [{{ $labels.volume }}] is having logical used percent higher than 60%"
  description: "Volume [{{ $labels.volume }}] is having logical used percent higher than 60%".
  You can change the metric, for duration and other details as per your use-case. Here, After 5m pending state, Prometheus raised the alert for 2 volumes which are having logical used % > 60

[glad arrow]

@native ivy More information related to prometheus Group concept can be found https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule_group

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

About Alertmanager details here https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#sending-alert-notifications