DEV Community

ilya mozerov
ilya mozerov

Posted on

The average that mixed two waits

A single number can be correct and still point you at the wrong problem. We had one: the average time from a checked change being committed to a new running version starting. It was a real measurement, taken from real records. It was also two different waits wearing one average.

One path, two situations

Changes ship along a single path. Only one release can occupy it at a time. When a change is committed while another release is still running, it cannot join that release's snapshot — it has to wait for the run to finish, then for the next scheduled check to pick it up. When a change is committed with nothing in flight, it waits only for that next check.

Those are different waits with different causes. One is queueing behind work; the other is waiting for a clock tick. The average added them together.

Splitting each commit by whether the path was busy when it landed told the story. Over one day, 129 of 174 commits landed during a run, with a mean wait of 23 minutes 9 seconds. The other 45 landed idle, at 8 minutes 15 seconds. Same commits, same measurement — but now you can see that the long wait belongs almost entirely to the busy case.

The wrong lever

If you only had the combined average, the obvious thing to tune would be the check interval: check more often, wait less. But the idle commits — the ones that actually wait on that interval — already wait only about eight minutes. The bulk of the delay is commits landing behind a run that is already in progress. Shortening the tick would barely touch them.

The lever is how long a release takes from start to finish. Make the cycle shorter and the queue drains faster; the busy wait falls. The cadence was never the main term. An aggregate mean had made it look like it was.

Keeping the split honest

There is a trap in deciding which commits were "busy." A run that fails partway through has no clean finish time. If you only count runs that finished successfully as occupying the path, then a commit landing just after a failed run looks idle — even though the failed run may have held the path longer than its last recorded stamp. That would quietly move commits from the busy bucket to the idle one and shrink the very number you are trying to measure.

The fix is to use the best finish evidence each run has. A failed run's last check timestamp is a lower bound on when it released the path: it cannot have finished before that stamp. Counting occupancy up to that bound is conservative — a commit can be mislabeled idle, never busy. When no wait can be measured at all, the report says so instead of printing a zero share.

Four regression tests cover the split, the rendered line, an unmeasurable history and the failed-run case. Nothing that already reported a wait was removed or narrowed.

The habit

Before you tune anything from an average, ask whether the samples come from one situation or two. If two, split on the thing that changes the cause and look again. The average is not wrong — it is just answering a question nobody asked.

Top comments (0)