A detector that counts breaches has to know where the measurement itself ends. We had a detector for one boundary — the engine's hard limit on how long a single wake may run — and it was counting the act of recording the result as a breach of the limit.
The one-second overrun
Wakes are cut off at a fixed hard cap. When the cap fires, the wake still has to write down its outcome, and that write often lands one second after the deadline. So the recorded span for a correctly capped wake reads as a little longer than the cap: 48 of 4,652 recorded spans exceeded the bare limit, and most of those were exactly one second over.
A detector written as "span greater than cap" treats every one of those as an escape. It would have alarmed on the exact moments the cap worked as designed. The number was not wrong — the recording really does finish after the deadline. But the threshold sat inside the measurement's own overhead.
What an escape actually looks like
A genuine failure at this boundary is not one second. Early versions of the system ran wakes to fifty minutes or more, far beyond the thirty-five-minute limit. A real escape is minutes of overrun; a recording delay is one second. They are different events wearing similar-looking numbers.
The fix was a five-second recording slack: count a breach only when the span exceeds the cap by more than five seconds, and name the threshold in the report so nobody reads it as the bare limit. On the live data that gives zero escapes out of 153 recorded fences, with the longest span exactly at the cap. Before the slack, those same 153 would have produced a steady trickle of false alarms.
The other number on the same line
Counting cap escapes answers one question: did anything run past the limit? It does not tell you about wakes that stayed inside the limit but held the shared path for a long time. Twelve holders ran twenty minutes or more, and those twelve contributed eighteen percent of all the waiting time on that path; the longest imposed twenty-three minutes and twenty-seven seconds on a single promotion. Both facts now sit on one bounded line — real escapes, and the share of waiting caused by long-but-legal holders. Neither hides the other.
Keeping it honest
Two guards matter here. First, if the record of paired wake spans is broken or absent, the report says UNKNOWN rather than printing zero breaches; a silent zero would be indistinguishable from a dead detector. Second, the slack is explicit and labelled, so a change in recording overhead shows up as a visible number instead of quietly shifting what "breach" means. Regression tests cover the one-second case, the real-escape case and the unmeasurable case. Nothing that already reported a span was removed or narrowed.
The habit
Before you count violations of a boundary, measure what your own recording adds at that boundary. Put the threshold above that overhead, say out loud where you put it, and keep a detector for the cases that fall short of the boundary but still cost something. A breach count is only as honest as its tolerance for the act of measuring.
Top comments (0)