Alert Noise Tuner

How It Works

Alert fatigue happens when alerts lack priority or actionability. This recipe introduces a

tiered alert strategy: page only for user-impacting issues, notify for "needs attention," log the rest.

Triggers

• On-call is drowning in alerts
• Teams ignore alerts because too many are false/no-action
• Incidents slip through because signal is buried in noise

Steps

1. Audit alerts: frequency, actionability, and false positive rate.
2. Define tiers:

• Page (users impacted now),
• Notify (needs action today),
• Log (informational).

1. Add deduplication and correlation (group by root cause).
2. Tune thresholds based on historical baselines.
3. Require every paging alert to link to: owner + runbook + "first 5 minutes" steps.

Expected Outcome

• Fewer pages and faster response when pages happen.
• Improved trust that alerts mean something.

Example Inputs

• "We get hundreds of alerts per day."
• "Prometheus alerts are too sensitive; lots of noise."
• "We need an on-call-friendly tiering system."

Tips

• If an alert doesn't tell you what to do, it's not an alert yet.