nabox 3.5 email settings - required password

[queen dock]

I'm trying to get alerting working on nabox 3.5 and have hit a few issues.

1. the nabox email settings UI forces password entry as per attached when credentials are not required in many instances.

Editing the HTML to delete the required tag resolved this issue for me.

2. I cannot get firing Prometheus alerts to trigger notifications but Grafana alerts work fine.

The logs show entries like:
grafana | logger=ngalert.sender.router rule_uid=a398aec6-b41f-4629-b775-70cb3c1327d6 org_id=1 t=2024-03-15T0505.031707624Z level=info msg="Sending alerts to local notifier" count=1
but nothing happens.

Any ideas on how to track down the rootcause?

[queen dock]

FYI ... a further issue I've found with nabox is the following.

I need 2 networks and some static routes on nabox due to its setup on a jumphost network.

I can set these up directly on the CLI in /etc/network/interfaces but this appears to be overwritten by nabox in some instances causing it to drop off the network.

Any ideas on how to preserve these settings?

Thanks for your help!

[grim sandal]

I'll get rid of the mandatory part on the passwd, sorry about that

Did you configure grafana default email recepient on /grafana/alerting/notifications ?

I did not test firing notifications from Harvest provided templates but no reason it wouldn't work. I'm running some tests

I'm adding an issue for NAbox 4 regarding custom network config

[queen dock]

/grafana/alerting/notifications
yes, i setup a default notification target but only get grafana alerts which i manually setup.

none of the firing prometheus alerts seem to generate notifications for some reason.

ive also added another alertmanager but still see nothing about the firing prometheus alerts anywhere.

in the grafana logs, the only indication i get of alerts is the following:
grafana | logger=ngalert.sender.router rule_uid=a398aec6-b41f-4629-b775-70cb3c1327d6 org_id=1 t=2024-03-18T17:36:05.033065469Z level=info msg="Sending alerts to local notifier" count=2 grafana | logger=ngalert.sender.router rule_uid=a398aec6-b41f-4629-b775-70cb3c1327d6 org_id=1 t=2024-03-18T17:36:05.033237504Z level=info msg="Sending alerts to external notifier" count=2

i also noticed that none of the firing prometheus alerts make it into the grouped alerts section as per attached.

[plain kraken]

NAbox 4?🙈

[grim sandal]

Lol... Probably 😄

I'm getting the test alerts but none of the firing alerts too

[queen dock]

I'm on the most recent versions as per attached.

I was seeing the same issues on v3.5 and upgraded to see if that would resolve the issue but no joy.

I'm just starting to see a warning about the VM time being wrong in case that is related.

[grim sandal]

Ah yes that's no good. Not sure that's related but it'll screw up prometheus collection

[queen dock]

i got the chrony daemon running now so thats one less problem! but i guess it would be best if nabox incorporated chrony or ntp.

[grim sandal]

It picks up the time from vmware tools, chrony shouldn't be necessary

[queen dock]

ah, apologies, it looks like the ntp daemon had stopped on the esxi host.

[grim sandal]

🙂

[queen dock]

Hi Guys,

Any indication if the Prometheus alerting issues will be resolved in v3.5? Also is alerting working in the v4 beta?

Thanks for your help!

[grim sandal]

From basic tests notifications work in v4. Do you get the test email from Grafana ?

[queen dock]

I havent tested v4 yet. For v3.5 I get Grafana alerts but nothing from the firing Prometheus alerts.

[grim sandal]

Is the integration working between Grafana and Prometheus alerts ? Do you see the alerts definitions in Grafana

[queen dock]

The alert definitions are present and alerts fire as per attached. Grafana alerts work but the only evidence of the Prometheus alerts are logs which look like:
grafana | logger=ngalert.sender.router rule_uid=a398aec6-b41f-4629-b775-70cb3c1327d6 org_id=1 t=2024-03-18T17:36:05.033065469Z level=info msg="Sending alerts to local notifier" count=2 grafana | logger=ngalert.sender.router rule_uid=a398aec6-b41f-4629-b775-70cb3c1327d6 org_id=1 t=2024-03-18T17:36:05.033237504Z level=info msg="Sending alerts to external notifier" count=2

[grim sandal]

You did define the default contact point email address in « contact points » correct ? Just making sure I’m not missing anything.

[queen dock]

Yea, they are configured. From your earlier comment I thought you were seeing the same issue.

[grim sandal]

Thanks for confirming. Ok I’ll do some more tests.

[grim sandal]

I seem to observe the same behaviour indeed. NAbox alert work fine (capacity) but not Harvest alerts. @limber comet, @fast halo thoughts ? Any experience with using Grafana to send email notifications coming from provisioned alerts ?

[fast halo]

hey @grim sandal we've only used Alertmanager for that not Grafana. I'll ping you later today about environment details

[grim sandal]

Thanks !

It's certainly a recurring topic in Grafana forums, there is something tricky here

[grim sandal]

I start to think that Grafana just doesn’t support taking alerts from Prometheus and sending it to its own notification settings. The integration is just to manage silencing and status of current alerts but prom’s alert manager still need to be configure and send emails.

[fast halo]

yes, that would not surprise me. Grafana alerts also seem very Grafana version specific