The Prometheus alert lifecycle, from evaluation to resolution

Prometheus evaluates alert rules and tracks each alert instance as inactive, pending, or firing. Alertmanager receives firing alerts and decides how to group, route, suppress, and notify them. Rule state and notification delivery are separate parts of the process.

The three Prometheus states

An alert instance is identified by its labels. For a typical comparison expression with a for duration:

No matching result                       -> inactive
Matching result, waiting for duration     -> pending
Same instance remains active long enough  -> firing
Condition no longer matches              -> inactive/resolved

for: 10m requires that instance to stay active across evaluations for ten minutes. A rule without for can fire on its first matching evaluation. A change to the identifying labels creates a different alert instance and can restart the waiting period.

Prometheus treats returned vector elements as active alerts. Use comparison filters such as up == 0; a comparison with the bool modifier retains zero-valued results, which can accidentally leave alerts active when used without another filter.

By default, a firing alert deactivates when its condition stops matching. keep_firing_for, where supported and configured, keeps it firing for a further duration to reduce flapping. Resolution is communicated to Alertmanager; it is not a fourth steady rule state alongside inactive, pending, and firing.

What Alertmanager does

Alertmanager evaluates its routing tree and groups alerts according to the selected route's group_by labels. The receiver determines the destination, such as email, PagerDuty, or a configured webhook integration.

Silences suppress notifications for matching alerts, commonly during maintenance. Inhibition suppresses one set of alerts when another matching alert is active, such as suppressing service symptoms while a node-down alert explains them. Neither mechanism fixes the condition or changes Prometheus's rule evaluation.

Notification timing also matters:

• group_wait delays the initial notification for a new group.
• group_interval governs when updates to an existing group can be sent.
• repeat_interval controls reminders for alerts already notified.

These settings explain why entering firing state does not always produce an immediate message.

Follow the complete path

Check the rule evaluation first, then the pending/firing state, Prometheus-to-Alertmanager delivery, routing and suppression, and finally the receiver. A recovery follows the same delivery path, subject to the receiver's resolved-notification setting.

FAQ

Does a pending alert notify anyone? Standard Prometheus alert delivery waits until the alert fires. Pending state is visible for investigation without sending a firing notification.

How do I enable recovery notifications? Set send_resolved: true on the relevant receiver integration. Defaults vary by integration; check the specific receiver rather than assuming recovery messages are enabled everywhere.

How can I reduce an alert storm? Group related instances, use precise inhibition rules, and choose sensible repeat intervals. Keep actionable distinctions in the labels; grouping everything together can hide separate incidents.

Sources

This guide is based on the documentation below; the examples have not been tested against a live deployment.

• Prometheus alerting rules
• Alertmanager configuration