Handling One-Time EMS Events in Harvest

[cedar belfry]

Hi all — we’re looking for advice or ideas from the community around handling one-time EMS alerts in NetApp Harvest.

We’ve got the recurring/bookend EMS events working fine with Harvest → Prometheus → Alertmanager → ServiceNow flow. But our issue is with single-shot EMS events — those without a follow-up "resolved" message from ONTAP.

In these cases:
Harvest collects the alert as expected.
Prometheus triggers the alert and a SNOW ticket is created.
But since there's no resolution event from ONTAP, the alert stays “active” in our logic.
Prometheus may auto-resolve it based on the expression, which leads to premature SNOW ticket closure — not ideal.

Ask:
Has anyone else run into this? We'd love suggestions on either:
A script-based workaround to infer resolution (e.g., via timeout, polling, correlation), or
Whether there’s any plan to add state tracking or resolution logic in Harvest similar to what AIQUM or DII seems to have.

Appreciate any ideas, examples, or insight from folks who've dealt with this type of alert flow!

[sly arrow]

@cedar belfry Is your request similar to the issue described in GitHub Issue #3771?

Could you provide more details on how DII has addressed this use case? If ONTAP does not retrigger this event or provide a resolving event, you might consider using the repeat_interval flag in Alertmanager. This flag can be used to resend an EMS event that is still active in Prometheus. To implement this, you may need to adjust the time range for the relevant query in Prometheus.

[cedar belfry]

@sly arrow
yes, the issue request is for our use case
Use case:
• Harvest should have a mechanism to check from ONTAP when an EMS event is resolved (especially single-shot EMS).
• Alerts in Harvest should remain open until ONTAP reports the EMS as resolved.

I will provide more details how DII/AIQUM is addressing this use-case..