Monitoring dozens of sites & APIs without drowning in alerts

The brief sounded simple. "We have a lot of websites and APIs. We want to know when one breaks." What the client actually had was a graveyard of half-configured uptime checks, three different tools nobody trusted, and a team that had quietly muted every notification channel months ago.

That last part is the real story. They did have monitoring — too much of it. Every check fired into the same Slack channel, every blip became a message, and within a week the humans had learned to ignore all of it. The system was technically watching everything and practically watching nothing.

So we didn't start by adding more checks. We started by deciding what was worth waking someone up for.

01 / The problemMore signals than anyone can read

When you spin up a check per URL across 60 surfaces, a single shared outage — a flaky upstream, an expired certificate on a wildcard domain, one bad deploy — lights up everything at once. The on-call engineer doesn't see "one incident." They see forty messages in ninety seconds and have to reverse-engineer what's actually wrong.

This is alert fatigue, and it's not a discipline problem. It's a design problem. If a person has to triage noise to find signal, they will eventually stop looking.

An alert that doesn't change what someone does is just a notification with anxiety attached.

We set one rule for the whole project: every page that reaches a human has to be actionable. If there's nothing to do about it, it belongs on a dashboard, not in someone's pocket at 3am.

02 / The modelChecks, services, and incidents are three different things

The biggest unlock was separating three concepts the old setup had mashed together:

• Checks are the raw probes — an HTTP request, a TLS handshake, a DNS lookup, a synthetic login flow. They run constantly and they're noisy by nature.
• Services group checks into something a person cares about. "Checkout" is one service even if it's six endpoints across two domains.
• Incidents are what humans respond to. An incident opens when a service crosses a threshold and stays open until it recovers — no matter how many individual checks flap in between.

Once those layers exist, deduplication falls out almost for free. Forty failing checks that all belong to one degraded service become one incident. The dashboard still shows every red check for debugging; the human only ever gets paged about the incident.

# one incident, regardless of how many checks are failing underneath
def evaluate(service):
    failing = [c for c in service.checks if c.is_down()]
    ratio = len(failing) / len(service.checks)

    if ratio >= service.threshold:
        open_or_update_incident(service, severity="down")
    elif failing:
        open_or_update_incident(service, severity="degraded")
    else:
        resolve_incident(service)

03 / The budgetSeverity, quiet hours, and an alert budget

Not every incident deserves the same response. We gave each service a severity ladder and routed accordingly:

• Page — revenue-critical services (checkout, auth, the public API) can wake a human, any hour.
• Notify — important but degraded-tolerant services post to a channel and wait for business hours.
• Log — everything else lands on the dashboard and in the weekly digest, full stop.

Then we added the piece most teams skip: an alert budget. If a single service pages more than a set number of times in a window, the system stops paging and escalates differently — because a service that's flapping every ten minutes isn't an incident anymore, it's a project. Paging someone twenty times won't fix it; it'll just burn them out.

04 / The proofCatch it before users do

Reactive checks tell you something broke. Synthetic checks tell you before your users find out. We added scripted flows that actually log in, hit the API, and walk through checkout on a schedule — so a 200 response on the homepage never lulls anyone into thinking the thing behind it works.

The combination — grouped incidents, severity routing, an alert budget, and synthetic flows — is what turned a wall of red into a calm board. Here's where it landed after the first quarter.

05 / The takeawayCalm is a feature

Good monitoring isn't measured by how much it watches. It's measured by how rarely it has to interrupt you — and how much you trust it when it does. The goal was never a prettier wall of graphs. It was an on-call rotation where a page means "drop what you're doing," every single time, because the system already filtered out everything that didn't.

If your team has quietly muted its alerts, that's the signal. The fix usually isn't more monitoring — it's less, aimed better.