DEV Community

Ayi NEDJIMI
Ayi NEDJIMI

Posted on

SLO-based alerting: moving from thresholds to burn rate alerts

SLO-based alerting: moving from thresholds to burn rate alerts

Threshold-based alerts have a fundamental problem: a 5% error rate at 2 AM with zero traffic looks identical to a 5% error rate during peak load. Both fire the same alert. One is nothing, one is a disaster. SLO-based burn rate alerting fixes this by measuring how fast you're depleting your error budget — which makes the severity of a problem proportional to its actual business impact.

What SLOs and error budgets mean in practice

An SLO (Service Level Objective) is a commitment about reliability. A common one: "99.9% of requests succeed over a rolling 30-day window." That 0.1% tolerance is your error budget — the total amount of failure you're allowed before you've violated the SLO.

For a 30-day window with a 99.9% SLO:

  • Total requests in 30 days at 1,000 req/min: 43,200,000
  • Allowed failures: 43,200

The key insight: a 1% error rate sustained for just one hour burns through ~139% of your entire monthly budget. That's not a warning alert — that's a service emergency.

The error budget replenishes at a fixed rate (1 month = full budget). The burn rate measures how fast you're consuming it relative to that replenishment. A burn rate of 1.0 means you'll exhaust the budget in exactly 30 days — that's acceptable. A burn rate of 14.4 means you'll exhaust it in 50 hours. That's worth a page.

Calculating burn rate

The formula is straightforward:

def burn_rate(error_rate: float, slo_target: float) -> float:
    """
    Returns how many times faster than the budget you're consuming errors.
    burn_rate=1.0  → exhausts budget in exactly 30 days (acceptable)
    burn_rate=14.4 → exhausts budget in ~50 hours (page immediately)
    burn_rate=36   → exhausts budget in ~20 hours (page now)
    """
    error_budget = 1.0 - slo_target
    return error_rate / error_budget


# Reference points for a 99.9% SLO
print(burn_rate(0.001, 0.999))  # 1.0  — exactly on pace
print(burn_rate(0.006, 0.999))  # 6.0  — budget gone in 5 days → ticket
print(burn_rate(0.0144, 0.999)) # 14.4 — budget gone in 50h   → page
print(burn_rate(0.05, 0.999))   # 50.0 — budget gone in ~14h  → page now
Enter fullscreen mode Exit fullscreen mode

This is the number your alerting rules should fire on. Not the raw error rate. Not request latency. The burn rate.

The two-window approach

A single burn rate window produces false positives. A short spike during a deploy bumps your 5-minute error rate to 10%, fires a critical alert, and resolves itself before anyone responds. Alert fatigue follows.

The solution from Google's SRE workbook: require both a long window and a short window to exceed a threshold before firing. The long window confirms the problem is real (not a blip). The short window confirms it's happening right now (not hours ago).

from dataclasses import dataclass

@dataclass
class AlertRule:
    name: str
    long_window_minutes: int
    short_window_minutes: int
    burn_rate_threshold: float
    severity: str


ALERT_RULES = [
    AlertRule("critical", 60,   5,   14.4, "page"),
    AlertRule("high",     360,  30,  6.0,  "page"),
    AlertRule("medium",   1440, 120, 3.0,  "ticket"),
    AlertRule("low",      4320, 360, 1.0,  "ticket"),
]


def should_alert(
    long_window_error_rate: float,
    short_window_error_rate: float,
    slo_target: float,
    rule: AlertRule,
) -> bool:
    long_burn = burn_rate(long_window_error_rate, slo_target)
    short_burn = burn_rate(short_window_error_rate, slo_target)
    return long_burn >= rule.burn_rate_threshold and short_burn >= rule.burn_rate_threshold


# 1-hour error rate: 1.5%, 5-minute error rate: 2.2%
for rule in ALERT_RULES:
    if should_alert(0.015, 0.022, 0.999, rule):
        print(f"ALERT [{rule.severity.upper()}]: {rule.name}")
# Output: ALERT [PAGE]: critical
Enter fullscreen mode Exit fullscreen mode

Both conditions must hold. If only the short window fires, it's a transient spike. If only the long window fires, the incident already recovered — log it, don't page anyone.

A burn rate evaluator in Go

Here's a minimal implementation that evaluates alert rules against pre-fetched error rates (e.g., from a Prometheus recording rule):

package sloalert

import "fmt"

type SLO struct {
    Target float64 // e.g. 0.999 for 99.9%
}

func (s SLO) BurnRate(errorRate float64) float64 {
    budget := 1.0 - s.Target
    if budget == 0 {
        return 0
    }
    return errorRate / budget
}

type AlertRule struct {
    Name              string
    BurnRateThreshold float64
    Severity          string
}

type WindowRates struct {
    Long  float64
    Short float64
}

func Evaluate(slo SLO, rates WindowRates, rules []AlertRule) []string {
    longBurn := slo.BurnRate(rates.Long)
    shortBurn := slo.BurnRate(rates.Short)

    var fired []string
    for _, rule := range rules {
        if longBurn >= rule.BurnRateThreshold && shortBurn >= rule.BurnRateThreshold {
            fired = append(fired, fmt.Sprintf(
                "[%s] %s — long=%.1fx short=%.1fx threshold=%.1fx",
                rule.Severity, rule.Name, longBurn, shortBurn, rule.BurnRateThreshold,
            ))
        }
    }
    return fired
}
Enter fullscreen mode Exit fullscreen mode

Usage in a real system: call a Prometheus query API (or any metrics store) to get error rates for each window, pass them into Evaluate, then route the results to PagerDuty, Slack, or whatever you use. The evaluation logic stays pure and testable.

Reference thresholds for 99.9% SLO

Burn rate Budget exhausted in Action
≤ 1× ≥ 30 days Nothing
3× 10 days Ticket (business hours)
6× 5 days Ticket (urgent)
14.4× ~50 hours Page
36× ~20 hours Page immediately

Adjust these if your SLO target changes. For 99.95%, your error budget is 5× smaller, so the same absolute error rate translates to a 5× higher burn rate — the thresholds hold, the inputs change.

The takeaway

Threshold-based alerting answers "is the error rate high right now?" Burn rate alerting answers "are we on track to violate our reliability commitment within hours?" The second question is the one worth waking someone up for.

The two-window check is the practical minimum to implement before going live. Start with the four standard rules (critical/high/medium/low), tune burn rate thresholds to match your SLO window, and revisit the numbers quarterly as your traffic patterns change. For a broader set of observability and reliability practices, the articles section covers related topics.


I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Top comments (0)