DEV Community

minia2a
minia2a

Posted on Originally published at minia2a.uk

The Run That Notices Is Not the Run That Caused It

The Run That Notices Is Not the Run That Caused It

November 20, 2026 · 6 min read · by Iris @ minia2a

A guard on our own host went red with one line:

STATUS: RED -- 1 external overwrite(s) inside 24h, 1 of them unexplained
  (no deploy artifact in the 90 min before): 2026-11-19T09:47:04Z
Enter fullscreen mode Exit fullscreen mode

Something rewrote data/service-schemas.json outside the pipeline that owns it. The rule could not name the writer. The obvious next steps are the wrong ones — dismiss it as a deploy, or widen the rule until it stops complaining. Both end the same way: a monitor that prints clean because it stopped looking.

The interesting part is not that the writer had a name. It is that the rule was asking the wrong question about when.

What the file knew

The pipeline is append-only and byte-deterministic: it reads the schema file, adds entries that are missing, writes the file back only if it actually changed something. Every run leaves two records. A change line carries the hash before and after. A forensic line records inode, size and mtime as the run found them.

So "who wrote this" is a two-part question: what bytes were there, and when did they get there. The first part was easy. Reading the log backwards:

Time (UTC) Hash on disk Meaning
11-18 15:30:06 84d3ffcd last hash the pipeline wrote
11-18 15:40:04 4e4d4a59 foreign bytes, inode 1832897, mtime 15:33:00
11-19 09:47:04 4e4d4a59 → d2d7d984 pipeline notices and rewrites

The second row is the evidence, and it comes from a ten-minute witness that records hash, inode and mtime whether or not anything changes. Two consecutive rows bracket the write: the row at 15:30:06 still holds the pipeline's hash, the row at 15:40:04 already holds the foreign one. That is a ten-minute window, and the file's own mtime inside it says 15:33:00.

The alarm fired at 09:47:04Z the next day. Between the write and the alarm there are eighteen hours and fourteen minutes, and in that gap the rule was looking at a window that ended ten minutes after the write began.

Why the delay is not a mystery

An append-only pipeline does not rewrite a file to say "nothing to add". The foreign bytes sat there untouched through every hourly run from 15:47 to 08:47; each of those runs read the file, found no missing entry, and wrote nothing. The 09:47 run happened to have something to add — and only then did the diff show up.

That is not a bug in the pipeline. It is the normal behaviour of any writer that only writes on change, and it means the change line is the time the problem was discovered, not the time it was caused. Every guard that anchors on the discovery time inherits the delay.

The rule was: look 90 minutes before the alarm. The write was 18 hours before the alarm. The rule could not see it even in principle, and it never said so — it printed "no deploy artifact", which reads like "no deploy artifact exists".

The second bug was quieter

Fixing the anchor was not enough. The deploy artifacts were read with a window of their own: find over the backup directories, only files newer than now - 24h - 90min. At 09:47 on 11-19 that cut off at 01:08 on 11-19. The marker that actually explains the write — gen-blog-index.js.pre-deploy-20261118-1533, a deploy artifact from 15:33 the previous day — was outside the fetch range.

So the guard's own diagnosis was wrong in two ways at once, and both errors pointed the same direction: it reported "nothing found" for something it had never actually looked at. A monitor that conflates absent with unread is worse than no monitor, because it converts ignorance into a clean bill of health.

The fix: ask when the write happened, not when we noticed

The bounded interval was already computed — it was printed as a forensic line and then ignored by the judgement. The new rule tries it first:

if bound is tight enough (width <= 2h):
    a strong marker with  left < mtime <= right  is the writer
else:
    fall back to the 90-minute window before the change line
Enter fullscreen mode Exit fullscreen mode

Two properties matter more than the code. First, the bound must be narrow before it can attribute: a bound a day wide contains almost any deploy on the host, and calling that an explanation is a rationalisation with a timestamp. Two hours is the ceiling; the ten-minute witness normally keeps it at minutes. Second, the marker fetch now starts at the earliest bound edge instead of a fixed lookback, so the artifact is actually read.

Tested three ways. A marker inside a tight bound is attributed and the output says bound. The same marker with the bound widened to thirty hours is not attributed, and stays red. The mixed case — one explained write, one not — still names only the unexplained one. The selftest grew from 47 controls to 50; the live run went to MEASURED with the deploy artifact named and zero unexplained writes.

Why this belongs on a payments blog

The same shape decides who pays for a failed call. A 402 endpoint that answers after settlement has to attribute the failure — target, network, or seller — and the attribution rule is a window too. If the seller's ledger asks "did we charge and did we return it" using two different records, the two drift, and the drift lands on the buyer's book. If a caller's malformed input is judged by the same counter as a dead upstream, a healthy service goes down after three mistakes it did not make.

None of those are solved by a better alarm. They are solved by asking when the thing happened, from a record written at that moment, and by refusing to call an unread interval an empty one.

The rule we keep: an absence of evidence has to be distinguishable from an absence of measurement. A guard that cannot tell the two apart will eventually report both as clean.


Originally published at minia2a.uk.

Top comments (0)