Alert for prometheus or Grafana

[dark brook]

Hi,
I am trying to add alert for monitoring,
and here are the condition:

• Trigger an alert when more than X% of the volume in an SVM has a latency higer than Yms.

And here is what I tried in Grafana alert:

1. count(volume_avg_latency{} >30)/count(volume_avg_latency{})*100
   To calculate the percentage of volume latency over 30ms
2. $A > 15
   To trigger the alert if the percentage is over 15%

Is this a recommended method, or should I use Prometheus AlertManager instead?

[hidden merlin]

All alerts are currently provided with alert manager but it’s certainly easier to do it in Grafana. In that case email will come from Grafana.

I still don’t know how to provide an easier way to customize alerts

[dark brook]

@hidden merlin So I can say that Grafana provide a GUI where you can set the alert easier, but the backend is still using prometheus alertmanager right?

[hidden merlin]

No, that’s the thing. When you do it in Grafana, it’s not AM

[dark brook]

Any recommendation using either Grafana alert setting or prometheus alertmanager?

[hidden merlin]

Grafana plugs itself to prom data source to fetch alerts from AM, but it’s just cosmetics to display alerts status.

It’s probably better for consistency to do it in AM but NAbox doesn’t provide an easy way yet.

[dark brook]

So for NAbox, we can only use Grafana alert,
And for self-built Prometheus and grafana, it is better to use AM

[hidden merlin]

We can stitch something together and add an alert rule file so you can add your own, I’ll do that in the next release.

You’ll have to test the rules in AM ui, when it’s valid you’ll then copy and paste in this file in CLI