This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.
That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
The reason field paid for itself on the first two runs: both failures ended on the same line, media URL could not be downloaded (timeout or source server error) at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.
That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.
The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.
That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Round number, and I'd rather say that than dress it up. I keep accepted_at, first_fetch_at and terminal_at as three separate fields, so after a week of good runs I can replace the 60s with the p95 of first_fetch_at - accepted_at instead of leaving my guess in the check.
That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. [email protected]
retry with backoff does not cover it as it picks the same host again, and on paid calls the first attempt can already be billed so we look the run up before sending it a second time
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Backoff also keeps the same host, so it just re-earns the same failure. The billed-first-attempt case is what forces the reconciliation step: write the idempotency key to your ledger before the call, and on any ambiguous failure (timeout, 502/504) do a lookup by that key before the second send - empty means retry, a row means adopt it and return its result. Even with an Idempotency-Key header you still want the lookup, since replay windows are usually 24h and your retry may land after it.
We're a place where coders share, stay up-to-date and grow their careers.
This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.
I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.
That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.
The reason field paid for itself on the first two runs: both failures ended on the same line,
media URL could not be downloaded (timeout or source server error)at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.
One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.
The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?
Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.
That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?
Round number, and I'd rather say that than dress it up. I keep
accepted_at,first_fetch_atandterminal_atas three separate fields, so after a week of good runs I can replace the 60s with the p95 offirst_fetch_at - accepted_atinstead of leaving my guess in the check.That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.
retry with backoff does not cover it as it picks the same host again, and on paid calls the first attempt can already be billed so we look the run up before sending it a second time
Backoff also keeps the same host, so it just re-earns the same failure. The billed-first-attempt case is what forces the reconciliation step: write the idempotency key to your ledger before the call, and on any ambiguous failure (timeout, 502/504) do a lookup by that key before the second send - empty means retry, a row means adopt it and return its result. Even with an Idempotency-Key header you still want the lookup, since replay windows are usually 24h and your retry may land after it.