DEV Community

upordown.online
upordown.online

Posted on

Should your status page update itself from your monitors, or only when a human says so? Let's argue about it

Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — upordown.online

Every team that sets up a status page ends up having the same argument, usually during the first real incident: who is allowed to turn it red?

One camp says the page should be wired straight to monitoring. If checks fail, the page says so, no human in the loop. The other camp says a status page is a communication tool, and communication needs a person who understands what's going on.

I've been on both sides of this, and I don't think the answer is obvious. Let's argue about it.

The case for automatic updates

It's fast. The worst status pages are the ones that say "All systems operational" while Twitter is full of people saying the product is down. That gap almost always exists because a human had to notice, decide, log in, and write something, all while also trying to fix the problem. An automatic page updates within a minute or two of the first failed checks.

It's honest. Nobody has to make the uncomfortable call to admit there's an outage. Teams under pressure tend to wait "just five more minutes, it might come back". Automation doesn't have that bias.

It works at 3 a.m. If the on-call engineer is asleep, or dealing with the problem from a phone, nobody is updating anything. The page might be the only thing telling customers that you know.

It reduces support load immediately. The first fifteen minutes of an outage generate the most "is it just me?" tickets. A page that goes amber right away answers that question without a single human reply.

The case against

Monitors are noisier than outages. Even well-tuned checks fail for reasons that have nothing to do with your users: a monitoring region with packet loss, a DNS hiccup at the checker's resolver, a deploy that takes one instance out for 20 seconds. If every one of those flips your public page, customers learn that your status page cries wolf, and they stop trusting it.

Monitors don't know impact. A check that fails on /api/health might mean the whole product is down, or that one internal dependency is slow and nobody would notice. A human can write "Search is slow, everything else works". A monitor can only write "Degraded".

It's public. Your status page is read by customers, prospects, journalists and competitors. A flapping page with ten incidents a week looks worse than the actual reliability, and those incident histories stay visible for months.

Words matter during an incident. "We're investigating elevated error rates on checkout" is useful. "Monitor check-7 failed" isn't. Automatic incidents tend to have the second kind of text.

Where I've landed (for now)

I think the best setups are a hybrid, and the line is drawn by confidence, not by preference:

  1. Automate the first signal, with a high bar. Only flip the public page automatically when multiple checks from multiple locations agree for several minutes, and only on checks that test something users actually do (load the page, log in, check out). A single failed check should page a human, not change the public page.
  2. Automate amber, not red. "We're seeing issues and investigating" is a safe automatic message. "Major outage" should be a human decision.
  3. Make the human step tiny. If declaring an incident takes three clicks and a template, people do it quickly. If it requires logging into a separate tool with a separate password, they don't.
  4. Auto-resolve cautiously, or not at all. Recovering checks are a good hint, but an outage that "resolves" and comes back twice in an hour is worse for trust than one that stays open until a person says it's over.
  5. Keep the internal and the public view separate. Engineers should see every failure on an internal dashboard. The public page shows what customers need to know.

A status page is only as good as the check behind it. upordown.online watches your site from outside your own infrastructure, so the signal that turns the page amber isn't coming from the same server that just went down.

The checks that feed that first signal matter more than the status page tool. A monitor on the same host as the app can't report that the host is down. A check from outside, against the real user path, is the only kind I'd let touch a public page, and upordown.online is built for exactly that kind of check.

What makes this hard

A few things I still haven't settled:

  • Third-party outages. If your payment provider is down, is that your incident? Customers can't pay you, so for them it is. Some teams automatically mirror upstream status pages, which creates a lot of noise.
  • Partial outages by region. If users in one country can't reach you because of an ISP problem, a multi-region monitor will show it. Do you post it? It's real for those users, but it's not something you can fix.
  • Scheduled maintenance. Automation that doesn't know about maintenance windows will happily declare an outage during the one you planned. Every automatic setup needs a "we know" switch.
  • Small teams. For a team of two, "a human decides" might mean nobody updates anything for an hour. The arguments for automation get stronger the smaller you are.

A quick test for your current page

Look at your status page history for the last three months and answer two questions:

  • How many incidents were posted more than 15 minutes after users started noticing?
  • How many incidents were posted that customers never noticed at all?

If the first number is high, you need more automation. If the second number is high, you need less, or a higher bar for it. Most teams I've seen have one of these problems, not both.

Related reading

So which way do you lean: should monitors be allowed to turn the public page red without a human? And if you've automated it, what threshold actually worked for you?

Top comments (2)

Collapse
 
mickyarun profile image
arun rajkumar •

"The line is drawn by confidence, not by preference" is the right frame, and I would push on what the checks are measuring before who is allowed to flip the page.

The failure I have to think about is not a flapping monitor. It is a partial one where every check passes. Our API is up, the database is fine, the health endpoint is green, and a customer still cannot pay because the bank at the other end is not responding. Monitors that test what we own report operational, truthfully, while the product does not work. No confidence threshold helps, because the checks agree with each other and they are all wrong about impact.

So the check has to traverse the dependency, which is where it gets awkward rather than hard. A synthetic that runs end to end through a third party tells you the truth and also tells your customers which third party. Naming a bank on a public page has a cost that has nothing to do with engineering, and I suspect that cost is a large part of why these pages stay green. Being vague is easier than being specific about someone else's outage.

I do not have a clean answer to that. The thing I would automate first is not the public page, it is the internal one, so the gap between what our checks see and what customers experience is visible to us before anyone argues about wording.

Collapse
 
upordownonline profile image
upordown.online •

The bank example is the hard one, because every check is telling the truth and the product still doesn't work. Automating the internal page first is a nice way around the naming problem, you get the honest picture without deciding yet what customers get told. Would you word the public side as something generic like "card payments degraded" and leave the provider out of it?