Metric alerts are not firing

[marsh rivet]

Hi @oblique charm. I'm happy to dig into this for you! I'll check my internal sources for answers to your question! :wave:

---

Based on the knowledge sources, there are several potential reasons why your metric alerts might not be firing even when the graph shows the threshold being exceeded:

1. Environment specification issues: If you've specified an environment in your alert rule, this can sometimes prevent alerts from firing correctly. This was a known bug with performance alerts that had an environment specified, though it has been fixed in newer versions.
   Github Issue: Transaction Duration Metric Alerts not being triggered when an environment is selected

2. Timing and evaluation windows: Alerts evaluate data in specific time windows, and there might be a delay between when events occur and when they're processed for alerting. The graph might show data that arrived after the alert evaluation occurred.
   Github Issue: Metric alerts to consider events appearance lag

3. Kafka issues (for self-hosted instances): If you're running a self-hosted Sentry instance, Kafka offset problems can prevent alerts from firing. One user resolved this by resetting the offsets with kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-post-processor --topic events --reset-offsets --to-latest.
   Github Issue: Alerts not firing even with matching issues

4. Region-specific issues: There have been instances where metric alerts didn't fire for organizations in specific regions (like EU). This was a bug that has been fixed.
   Github Issue: Metric Alerts are not triggering for EU orgs

5. Release stage tag incompatibility: If you're using tags like release.stage:adopted in your alert query with short time intervals (less than 1 hour), this might not work because release stage is evaluated hourly.
   Github Issue: Alert on "users experiencing errors" not triggering

6. Data delay: Sometimes there's a delay in data showing up because of how events-stats processes data buckets.
   Github Issue: Creating custom metrics alerts is flaky on DE region

7. System-wide issues: Occasionally there have been incidents affecting alert triggering across the platform.
   Github Issue: INC-454 Add more monitoring around alerts firing

If you're using a self-hosted instance, I'd recommend checking your version and considering an upgrade, as some of these issues have been fixed in newer releases. For SaaS users, you might want to check if your alert configuration matches any of the known problematic patterns mentioned above.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: