DEV Community

Cover image for Silent Failure #7: The URL Returned 200 From Everywhere and the Job Died at Dispatch

Silent Failure #7: The URL Returned 200 From Everywhere and the Job Died at Dispatch

Renhua Wei on November 15, 2026

I checked that image URL twice before I submitted it. Once from my own box, once through a resizer sitting in the Netherlands, because I had been b...
Collapse
 
rushdeploy profile image
Rishabh Mishra •

This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.

Collapse
 
dafili profile image
Renhua Wei •

I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.

Collapse
 
rushdeploy profile image
Rishabh Mishra •

That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.

Thread Thread
 
dafili profile image
Renhua Wei •

The reason field paid for itself on the first two runs: both failures ended on the same line, media URL could not be downloaded (timeout or source server error) at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.

Thread Thread
 
rushdeploy profile image
Rishabh Mishra •

That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.

Thread Thread
 
dafili profile image
Renhua Wei •

One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.

Collapse
 
iictp profile image
IICTP •

The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?

Collapse
 
dafili profile image
Renhua Wei •

Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.

Collapse
 
iictp profile image
IICTP •

That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?

Thread Thread
 
dafili profile image
Renhua Wei •

Round number, and I'd rather say that than dress it up. I keep accepted_at, first_fetch_at and terminal_at as three separate fields, so after a week of good runs I can replace the 60s with the p95 of first_fetch_at - accepted_at instead of leaving my guess in the check.

Thread Thread
 
iictp profile image
IICTP •

That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.

Collapse
 
kevinhall profile image
Kevin Hall •

retry with backoff does not cover it as it picks the same host again, and on paid calls the first attempt can already be billed so we look the run up before sending it a second time

Collapse
 
dafili profile image
Renhua Wei •

Backoff also keeps the same host, so it just re-earns the same failure. The billed-first-attempt case is what forces the reconciliation step: write the idempotency key to your ledger before the call, and on any ambiguous failure (timeout, 502/504) do a lookup by that key before the second send - empty means retry, a row means adopt it and return its result. Even with an Idempotency-Key header you still want the lookup, since replay windows are usually 24h and your retry may land after it.