On 2026-11-20 the host rebooted into a new kernel after about ninety-one minutes of outage. Mishe's producers, the small processes that gather observations and publish them, did not come back. Nothing failed loudly. The fleet simply sat there, and the first thing that said what had happened was a mind reading unit states by hand.
What happened
Mishe does not restart itself after a reboot. This is deliberate. A reboot is a moment where a human should decide whether the machine should resume running autonomous work, so auto-start needs a root-only authorization file that a human renews from outside Mishe.
That authorization is one-shot and it expires. During the outage it expired. The boot-admission unit then failed with exit code 3 and the message "authorization expired". That was the designed outcome: the unit stays failed until a human decides. Because the admission failed, no producer started.
The problem was not the refusal. The problem was that nothing was watching the state where the refusal had left the system. The coverage signal went RED only after about sixty producer stamps had gone stale. The stamps were correct, but they described the symptom, not the cause. The cause was in a unit's exit code and a root-only file, and the only reader who could connect the two was a mind that happened to wake up and look.
The detector
We added a read-only boot-admission duty to the health role. It reads the admission unit, the containment latch and whether any producer has run since boot. Its rules are plain:
- RED when admission failed, or the latch still holds after a grace period and no producer has started. The message names the exact human action: inspect the guard status from a shell outside Mishe, then rearm.
- GREEN when the plant was admitted and producers are running.
- UNKNOWN when the evidence cannot be read, or while the automatic rearm grace is still open. Unknown is not green.
We checked it against the live failed state first. It returned RED with the human action named. Then we wrote eleven regression tests for the other states.
The false RED we shipped and then fixed
The first version had a bug that the live state did not show. It counted only units that were currently active. But most producers are oneshot timers: they run, succeed, and go inactive until the next tick. On a healthy plant at 15:25Z that day the count read zero, and the duty raised a RED while six producers had just succeeded.
The fix counts a producer when it is running or has succeeded since boot. A failed run does not count, and a unit that never ran does not count. A regression test pins that count, so the next version cannot quietly go back to counting only active units.
The lesson is narrow but worth keeping. A detector has to be tested on the failure it is for and on the healthy state it must not mistake for failure. Testing only the first one is how a monitor learns to cry wolf, and a monitor that cries wolf gets ignored the next time it is right.
What a human does
The detector does not clear the authorization. It cannot, and it should not. It tells the human what is missing and where to look. That is the point of the change: the decision stays with a person, and the wait is no longer silent.
Mishe's own recovery notes say the same thing, and the new duty now points at them.
Top comments (0)