Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run
A webhook arrives: order.updated, subscription.updated, issue.edited. The body contains the object, or most of it. Your handler has two choices:
- Trust the payload. Read the new state from the event body and write it to your database.
- Treat the event as a doorbell. Note the object's ID, call the sender's API, fetch the current state, and write that.
Both are common. Both are recommended by someone. Some providers have started sending "thin" events that contain only an ID and a type, which makes the choice for you. I think this is one of the more interesting design arguments in integrations, so let's have it.
The case for trusting the payload
It's fast and cheap. No extra API call, no extra latency, no extra point of failure. For a high-volume integration, re-fetching every object can double your API usage or more, and push you into the sender's rate limits exactly when traffic spikes.
The sender's API might be down. Webhooks are often queued and retried by the sender for hours or days. If the sender's API has an outage, a fetch-first handler can't process anything, while a payload-first handler keeps working from the events it already has.
The payload is what the sender meant at that moment. For event-like facts ("payment succeeded at 12:01", "user clicked unsubscribe"), the payload is the record. Fetching later may return a state that has since moved on, and you lose the history of what happened.
It's what most docs show. Tutorials, SDK examples and quick-start guides overwhelmingly read from the event body. Code that follows the docs is easier for the next developer to understand.
The case for fetching the latest state
Webhook order isn't guaranteed. Most providers say explicitly that events can arrive out of order and more than once. If you apply payloads as they arrive, an older event processed late overwrites newer state. Fetching makes order irrelevant: every handler converges on the current state.
Payloads can be stale even when they arrive in order. Retries deliver the original payload, which may be hours old. A handler that applies a three-hour-old "status: past_due" to a subscription that has since been paid is doing damage.
Payloads may be incomplete. Many APIs send a partial object, omit expanded relations, or truncate large fields. Handlers that trust the payload start growing special cases ("if the field is missing, fetch it anyway").
It simplifies the code. Instead of one handler per event type, each with its own idea of what changed, you have one "sync this object" function, called by every event, by a nightly reconciler, and by an admin button. Three triggers, one code path.
It's more secure against forged events. If someone bypasses your signature check, a fetch-first handler still only writes what the real API says.
Where I've landed
My default is: fetch for state, trust for facts.
- State is anything that can change again: subscription status, order status, user profile, issue labels. For these, the payload is a hint. Fetch the current object and write that. Out-of-order and duplicate events become harmless.
- Facts are things that happened once and won't un-happen: a payment captured, a message delivered, a form submitted. For these, the payload is the record. Store it as an event, idempotently, keyed by the event ID.
Then three supporting habits:
- Debounce bursts per object. If five events arrive for the same subscription within a second, fetch once. A short delay ("sync this object in 2 seconds, collapsing duplicates") cuts API calls dramatically. This is a good use of delayed jobs, whether through something like delay.run or a queue with deduplication keys.
- Fall back to the payload when the API is unavailable, but mark the record as "synced from event" and re-sync later.
- Reconcile on a schedule. Whatever you choose, a periodic job comparing your state with the provider's catches what both approaches miss.
Need a second look at the event, a few seconds later? delay.run schedules a delayed HTTP callback with retries, so you can fetch the latest state after the webhook arrives.
The cases that start arguments
- Thin events. If the provider only sends IDs, you have to fetch. Is that a better design, or just shifting cost to the receiver?
- Version numbers. Some APIs include a monotonically increasing version or sequence number on each object. With that, you can trust payloads safely by ignoring anything older than what you have. Is that enough to make fetching unnecessary?
- Rate limits. At what volume does fetching stop being reasonable? Hundreds of events a minute? Thousands?
- Your own webhooks. If you send webhooks, should you send full objects or thin events with a fetch URL? What do your consumers prefer?
What the hybrid looks like in code
A minimal version of "fetch for state, trust for facts", in pseudo-Node:
async function onEvent(event) {
await db.events.insertIgnore({ id: event.id, type: event.type, payload: event });
if (FACT_TYPES.has(event.type)) return recordFact(event); // payment captured, message delivered
const id = event.data.object.id;
await jobs.enqueue("sync-object", { id }, { delaySeconds: 2, dedupeKey: `sync:${id}` });
}
The fact path stores what happened, once. The state path collapses a burst of events into one delayed sync job per object, which then calls the API and writes the current state. Both paths are idempotent, and neither cares what order events arrived in.
A quick test for your own handler
Take one handler and ask:
- If this event arrived twice, would anything break?
- If it arrived after a newer event for the same object, would it overwrite newer data?
- If it arrived three hours late, would it apply old state?
If any answer is "yes", the payload-trusting version needs either version checks or a fetch.
Related reading
- Case study (hypothetical): the customer who paid, but the app still said 'Free plan'
- The transactional outbox, explained: how to save to the database and enqueue a job without losing either
- Exactly-once scheduled jobs don't exist: design for at-least-once and stop double-charging
- Case study (hypothetical): the payment webhook that arrived before the order existed
Which way do your webhook handlers lean: payload or fetch? And if you trust the payload, how do you deal with events that arrive out of order?
Top comments (22)
The useful split is not trust versus verify, it is what the handler needs: the delta, or the state. If the event carries a version, you can apply deltas safely by keeping the last applied version per object - idempotent, ordered, no extra read. If it does not, you are guessing about ordering anyway, so the fetch buys correctness you cannot get another way.
The cost of the fetch is easy to underestimate: one extra read per event, so a busy endpoint doubles its read QPS on that table at exactly the moment the write burst is happening.
Cheapest middle ground: fetch only on the event types that touch state you cannot reconstruct from the payload.
Delta versus state is a cleaner way to frame it than trust versus verify. The doubled read load is a fair point too, since a burst is exactly when the provider is most likely to start rate-limiting your fetches. Which event types would go on your fetch-only list first?
Fetch-only first, for anything where the payload is a trigger rather than the fact:
Payload-first for append-only facts: disputes, delivery receipts, audit entries. There, the event IS the fact - and re-fetching can lose it if the provider's retention window is shorter than your retry window.
Trigger versus fact is a nice way to sort them, the list almost writes itself once you ask that question. The retention point on disputes is a good catch, I hadn't considered the provider expiring the event before your own retries give up.
Thanks - and I would turn the retention point into a check rather than a note: for each event type, record the provider's stated retention window, then alert when your retry horizon (max backoff x attempts) approaches it. The failure is silent by construction, because by the time you go to fetch, the answer is a 404 that looks like a bug in your own code rather than an expired fact.
Turning it into a check is the right next step, a note in a design doc is how this gets forgotten. Alerting when max backoff times attempts starts crowding the provider's retention window would have saved me a very confusing 404 more than once. Treating "event not found" as its own signal, rather than a generic fetch error, is the other half of that.
The 404-as-its-own-signal point is the one that changes what you do next, because the two cases want opposite responses. A generic fetch error means retry the same event. A not-found means that event is gone and retrying it will produce the same answer forever. If both land in the same retry path, an outage that pushes you past the retention window becomes a retry loop that can never succeed - and every iteration spends a request against the rate limit you are already failing.
Two things I would write down next to the alert.
The retirement rule. Once you see not-found for an event you know existed, that event is finished: drop it, count it, advance the cursor. The alternative is a poison message that blocks everything behind it.
The rate is the leading indicator. A single not-found is a timing race. A burst means the consumer is falling behind retention, which is a capacity problem that will keep producing them - so the alert belongs on the rate, not on the individual event, or it stays noise right up until it is an outage.
The third thing is the one that would have saved your 404: record the timestamp of the oldest event you are still retrying and keep it next to the provider's stated retention. That gap is the actual risk metric, and unlike the 404 it is computable before the failure rather than after.
The oldest-unretried-event timestamp is a good one, mostly because it gives you a number to watch on a quiet Tuesday instead of finding out from a burst of 404s. Agreed on dropping and counting not-found events too, a poison message sitting at the head of the queue is worse than a lost one. Do you keep the dropped ones anywhere for a later reconcile, or just the count?
Keep them, but not in the queue and not forever.
The count answers "how often are we dropping things" and nothing else - it cannot answer "was the one we dropped the one that mattered", which is the question you get asked when a customer says an event never arrived.
So the useful shape is a quarantine table: payload, reason, source, timestamp, bounded by a TTL and a per-source cap. TTL because the value of a dropped event decays quickly - after a few days nobody is reconciling it and the table is just a copy of untrusted content with no consumer, which is its own liability. Cap because without one a single noisy source fills it and the real signal gets buried.
The one thing I would not do is store them where the retry worker can read them. The reason to drop is that retrying is harmful; putting the payload next to the retry path invites someone to add a "replay from quarantine" button later, and that button is a re-injection primitive with extra steps.
Keeping the quarantine out of the retry worker's reach makes sense, a replay button is exactly the kind of thing that gets added in a hurry during an incident and never removed. A few days of TTL plus a per-source cap sounds like the right size for most setups.
One refinement on the per-source cap: show the cap state in the quarantine view, not just enforce it. During an incident, the source that bursts is usually the source that caused the incident, so it is also the one that fills the quarantine - and a per-source cap silently means the other sources' items never make it in. Seeing "source X: 200/200, capped" next to the list is the difference between a cap and a blind spot.
The other half is what the cap does when it trips: drop the oldest, or refuse new ones? Dropping the oldest keeps the recency you want mid-incident and destroys the earliest evidence, which is usually the evidence you need. Refusing new keeps the first evidence and goes blind exactly as things get worse. I would refuse and increment a counter on the refusal path, because that counter is what tells you the cap was the binding constraint rather than the event rate - and those two have different remedies.
Refuse-and-count makes sense to me, losing the first evidence hurts more than missing the tail, which usually looks the same anyway. And showing "capped" right in the view stops people from reading a short list as "not much went wrong".
A practical split is to use the provider’s object version in a conditional upsert for mutable state, ignoring events older than the stored version; event IDs prevent duplicates but not reordering. If there’s no reliable version, store each event idempotently and coalesce a fetch by object ID. Keep one-time facts in an append-only event record so a later fetch can’t erase them.
The append-only record for one-time facts is the piece I didn't spell out in the post, and it matters: a fetch tells you the subscription is active now, not that a payment went through at 14:02. Do you keep those records forever, or prune them once the state has settled?
One addition: "event IDs prevent duplicates but not reordering" is the sentence I would put at the top of the design doc, because most implementations quietly treat dedupe as ordering.
The conditional upsert on the provider's version is what makes it actually safe, and the append-only side record is what stops a later fetch from rewriting history - that is the failure that only surfaces in an audit months later, when the one-time fact has been overwritten by a state snapshot that looked more recent.
Team doorbell, with one amendment: fetch the latest state, but keep the raw payload too. Re-fetching wins on ordering — a stale event can't overwrite newer state — but the payload is your evidence of what the sender believed at send time, and when the two disagree that difference is the bug report. In document-exchange work we keep both: the fetched state drives processing, the original bytes drive the argument about who changed what, when.
Keeping the raw bytes as evidence is a good amendment, especially in document exchange, where the who-sent-what argument tends to show up weeks later. Do you keep them in the same database as the processed state, or push them off to cheaper storage after a while?
Agreed, and I would make the storage shape part of the rule rather than a convention: keep the raw bytes with the received timestamp and the event id, immutable, next to the version you actually applied. Then "who changed what, when" is a query instead of a memory.
The metric that comes free is the disagreement rate - payload version versus fetched version at the same event id. It should sit near zero. A nonzero rate is either clock skew on the sender side or a sender that serialises state after emitting the event, and both are worth knowing before they become a dispute.
If you wait and fetch once for several events, then fall back to the payload when the API is down, the single write can be stale. It is easy to miss because the debounce worked and the fallback worked, yet together they write old state. Anyone hit that in practice? Disclosure, I built Webhook Relay. It can store and retry incoming deliveries, but that does not fix a stale payload after an API outage.
That combo is sneaky because each half passes its own test. I'd only let the fallback write go through if the payload's version is newer than what's stored, and otherwise park the event and fetch again once the API is back. Did you run into it after a real outage, or spot it in review?
Yes, and the reason it is easy to miss is that both mechanisms are individually correct: the debounce coalesces events, the fallback keeps the write alive. They are jointly unsound because they read from different sources of truth at different times.
The fix I would reach for is making the version gate the only path to a write: the fallback payload still carries the event's version, so the same compare-and-apply runs whether the value came from the API or the body. A stale payload then fails the gate like any other old event, instead of being trusted because it arrived through the resilient branch.
Retrying deliveries helps the lost case, not the stale one - you are right that they are different problems.
I’d add one more race to the fetch-current-state tests: two handlers fetch at different times, then the older read finishes its database write last. Fetching helps with stale event payloads, but the write still needs ordering or a version check. This ordering-system exercise is a useful setting to practice that distinction while keeping payment facts and current order status explicit.