I'm Eric Gitangu. I work on a mobility/fintech platform. This post is about one question that sounds easy until money is involved: a charge got stuck halfway. Who is allowed to finish it?
The setup
Paying for a trip from a wallet is several writes, not one:
- Claim the trip: a conditional write moves its payment state to "charging". Only one request can win it.
- Stamp a plan on the trip before any money moves: the amount, and a ledger entry id minted up front.
- Debit the wallet with a conditional write that refuses to go below zero.
- Settle: write the ledger entry under the pre-minted id and mark the trip paid.
The trip and the wallet are separate records. Nothing wraps all four steps in one transaction, so the order is the design. The plan goes down before the debit so that anyone who finds the trip later can tell how far the first attempt got.
Problem one: stuck in "charging"
If the request faults after step 1, the trip stays "charging" forever. Every retry sees the claim and backs off. The customer sees a payment that never completes, and no error to act on.
So a retry has to be able to take over. The rule I shipped first:
- A claim younger than a short grace period may still be in flight. The retry gets a "try again" answer and touches nothing.
- Past the grace period, retries are serialized: only the one that moves the claim time forward from the value it read may continue.
- Then it asks one question: did the debit land?
That question has three answers, not two:
- Yes (the ledger entry or the debit's audit record exists): finish the settle step. No second debit.
- Provably no (there is no plan, or no trace of the debit): release the claim and charge fresh.
- Can't tell: move nothing, raise a reconcile flag a human will see, answer "try again".
A failed read is not a "no". If the lookup faults, the request is refused. Unknown never gets rounded to the convenient answer.
Problem two: the first attempt wasn't dead
That takeover has a hole, and I found it by writing the test before trusting the design.
"The claim is old" does not mean "the first attempt is dead". It can be stalled: blocked on a slow database call. The request time limit I was relying on didn't count time spent blocked on the database, so a stalled request could outlive the grace period and then carry on.
The sequence:
- Attempt A claims the trip, stamps the plan, and stalls just before the debit.
- The grace period passes. Retry B takes over, finds no debit on record, releases and charges fresh. The wallet is debited once. Correct so far.
- A wakes up. Nothing has told it the world changed. It debits.
The failing test showed exactly that: one fare, two debits. "No debit on record" was true when B looked. It just wasn't going to stay true.
This is the classic zombie-worker problem. A timeout is a guess about another process, and you can't build money rules on a guess. You need a write that makes the guess true.
The fence
Each stamped plan gets a one-time marker: a unique record that can be created exactly once.
- The original attempt must take the marker immediately before it debits.
- A takeover that finds no debit must burn the same marker before it charges fresh.
Whoever gets there first wins, and the loser moves nothing.
def charge(trip_id):
if not trips.cas(trip_id, expect={"pay": "unpaid"},
set={"pay": "charging", "claimed_at": now()}):
return resume(trip_id)
plan = {"entry_id": new_id(), "amount": fare(trip_id)}
# one plan per claim: the stamp lands only while no plan is there
if not trips.cas(trip_id, expect={"pay": "charging", "plan": None}, set={"plan": plan}):
return Retry() # lost the claim: release nothing
if not markers.create_once(f"debit:{plan['entry_id']}"):
return Retry() # a takeover fenced this attempt out
wallet.debit_if_sufficient(plan)
return settle(trip_id, plan)
def resume(trip_id):
trip = trips.get(trip_id)
if age(trip.claimed_at) < GRACE:
return Retry() # may still be in flight
if not trips.cas(trip_id, expect={"pay": "charging", "claimed_at": trip.claimed_at},
set={"claimed_at": now()}):
return Retry() # another retry got here first
plan = trip.plan
if plan is None: # stamped before the debit: nothing started
release(trip_id)
return charge(trip_id)
landed = debit_landed(plan) # True / False / None; a read fault raises
if landed is True:
return settle(trip_id, plan) # finish, never debit again
if landed is False and markers.create_once(f"debit:{plan['entry_id']}"):
release(trip_id) # the stalled attempt can never debit now
return charge(trip_id)
flag_for_reconcile(trip_id, plan) # marker already taken: debit may still land
return Retry()
The same pattern as a standalone file, with a short README, is in this gist.
Walk the race again. If B burns the marker first, A wakes, fails create_once and stops. If A took the marker first and then stalled, B finds it taken with no debit on record. That is the one state nobody can prove from the outside: A is past the fence and its debit may still land. So B does not guess. It flags the trip for reconciliation and refuses.
Two smaller rules hold the fence up:
- One plan per claim. The plan stamp is itself conditional on no plan being there. An attempt that loses that write may have been taken over, so it releases nothing. It no longer owns what it would be releasing.
- Settle is idempotent. The ledger entry goes in under the id minted in the plan, so finishing a charge twice writes one row.
Rejected alternatives
- A longer grace period. It makes the race rarer and every honest retry slower. Rare is not a property you want on a debit.
- A lock with a TTL. It's the same guess with a different name. The holder can stall past the TTL and still believe it holds the lock.
- Reverse the second debit afterwards. Compensation is a fine tool, but the customer has already seen a wrong balance, and now the reversal is a money path that can fail too.
Trade-offs I accepted
- One extra write per charge. The marker costs a round trip on the happy path. A double debit costs a refund, a support ticket and some trust.
- Some charges wait for a human. In the unprovable case the system refuses instead of deciding. That is slower for one customer and correct for all of them.
Where it is
The resume rule and the marker fence are live in production. The next step is built and tested but not live yet: putting the fence inside the debit write's own condition, so even an attempt that got past the marker can't move money after a takeover.
Takeaways
- A timeout tells you a request is late. It never tells you it's dead.
- Before taking over someone's work, make a write that stops them from finishing it.
- "Did it land?" has three answers. Treat "can't tell" as its own case, and never as "no".
- Stamp the plan before the money moves, and mint ids up front so the last step is idempotent.
- If you no longer own the claim, don't release it.
Retries are how you get reliability. Fences are how you survive the retries.
Originally published at developer.ericgitangu.com. More engineering write-ups on the blog. Code on GitHub.
Top comments (12)
The "request that woke up late" is the exact failure that breaks naive idempotency — the first attempt wasn't dead, just slow, and it commits after you've already given up on it. Fencing is the right primitive. The piece I'd add: a timeout against that stalled request should move the operation to an explicit UNKNOWN state, never an automatic FAILED, because "I stopped waiting" and "it didn't happen" are different claims and only reconciliation can tell them apart. I went deep on fencing-plus-UNKNOWN for the agent case here: dev.to/rudratosh/your-ai-agent-ret...
Thanks Rudratosh. Agreed that the first attempt wasn't dead, and that fencing has to come with an explicit unknown. Your UNKNOWN framing is the part I'd adopt. My version flags the trip for reconciliation, but the flag sits beside the trip rather than being the trip's own state.
For context, the fence and the code are in the original post: developer.ericgitangu.com/blog/sta...
The marker is keyed on the entry id minted with the plan, so a retry with a new request ID still hits the same marker.
The entry-id-keyed marker is the crux, and it's the thing "just use an idempotency key" advice gets wrong — key on the request and a re-planned retry with fresh arguments slips past; key on the plan-minted entry id and the late request hits the same fence no matter how the retry is shaped. That's the whole game.
On adopting UNKNOWN: moving from "flag beside the trip" to "state the trip carries" is small in code but changes what's possible to get wrong. Once it's the operation's own state, two things fall out for free — every reader sees the uncertainty without knowing to check a side field, and the state machine itself can reject the illegal move (a fenced trip can't reach settled except through reconciliation). The flag-beside version relies on everyone remembering to read the flag; the state version makes forgetting impossible.
Going to read the fence code — curious whether the marker write and the charge dispatch share a transaction, or you're fencing across the same two-systems gap (your DB fence, the provider's charge). That's where "persist the fence before the side effect" starts to carry real weight, since a crash between the two is exactly the window the fence has to survive.
They don't share a transaction. The debit here is our own wallet, not a provider charge, but the trip, the marker and the wallet are still separate writes. So it's the same gap you describe, just inside our own store.
A crash between the two is the case the fence is built to survive. If an attempt burns the marker and dies before the debit, a resume finds the marker taken and no debit on record. It can't prove the first attempt is gone, so it doesn't charge. It flags the trip and stops. That's your UNKNOWN in practice, and Arun's point upthread is the cost: correct, but someone has to work the queue.
The next step closes most of that window: make the marker a condition on the debit write itself, so burning the fence and moving the money happen in one conditional write instead of two.
That last step changes the category, and it's worth being loud about why: because the wallet is your own store, you have the option I don't have in the provider case — collapse the fence and the debit into one conditional write (debit WHERE the marker is claimable, both or neither), and the crash-between-two-writes window closes because there's no "between" anymore. That's the whole advantage of staying inside one transactional boundary: atomicity is on the table, so UNKNOWN stops being a steady state and becomes just the recovery path for infra failures, not for your own ordering.
The reason my piece leans so hard on UNKNOWN is that the second write is usually a provider — a system that doesn't share your transaction, so you genuinely can't make burn-fence-and-move-money atomic, and the queue Arun mentioned is unavoidable. Your version is the strictly better position: when you own both sides, make it one conditional write and you've designed the UNKNOWN out rather than reconciled it away. The rule that falls out of both: atomic where the writes are yours; fence-plus-explicit-UNKNOWN only where they cross a boundary you don't control. You've drawn that line exactly.
One edge on the single conditional write: it closes the gap between fence and debit, but the moment anything downstream observes the result — a receipt, a notification, a balance another service reads — that observation becomes the new second write, and UNKNOWN reappears one hop out. So the conditional write is the floor, not the ceiling, as soon as a third party reads the outcome.
Agreed on the rule: atomic where the writes are yours, fence plus explicit UNKNOWN where they cross a boundary you don't own. That's a cleaner statement of it than anything in the post.
And fair point on the edge. The conditional write closes the gap inside the store, but a receipt or a notification is the next hop, and the same reasoning applies there.
Thanks for the back and forth, Rudratosh. This thread made the post better.
Likewise — this is the best kind of thread, two people converging on the same line from different starting points (you from the wallet, me from the provider). "Atomic where the writes are yours, fence plus explicit UNKNOWN where they cross a boundary you don't own" is the whole thing, and your conditional-write step is the cleaner proof of it than my version. Your post stands on its own; glad the back-and-forth sharpened the edges. Will be watching what you write next.
The entry-id-keyed marker is the crux, and it's the thing "just use an idempotency key" advice gets wrong — key on the request and a re-planned retry with fresh arguments slips past; key on the plan-minted entry id and the late request hits the same fence no matter how the retry is shaped. That's the whole game.
On adopting UNKNOWN: moving from "flag beside the trip" to "state the trip carries" is small in code but changes what's possible to get wrong. Once it's the operation's own state, two things fall out for free — every reader sees the uncertainty without knowing to check a side field, and the state machine itself can reject the illegal move (a fenced trip can't reach settled except through reconciliation). The flag-beside version relies on everyone remembering to read the flag; the state version makes forgetting impossible.
Going to read the fence code — curious whether the marker write and the charge dispatch share a transaction, or you're fencing across the same two-systems gap (your DB fence, the provider's charge). That's where "persist the fence before the side effect" starts to carry real weight, since a crash between the two is exactly the window the fence has to survive.
The fence is the right shape. The part I would put a number on is the branch you wrote last.
create_onceanddebit_if_sufficientare two writes. An attempt that burns the marker and then dies leaves a trip with a plan, no debit, and no marker left to burn. Resume readslandedFalse, fails to create the marker, and the trip goes toflag_for_reconcile. That is correct. It is also a trip that now needs a person, while the customer is still looking at a payment that never completed.So the fence has not removed the failure. It has converted a double debit into a queue, which is the trade you want. Whether it stays a good trade depends entirely on whether anyone answers the queue. Payments has the same state and the same queue: an authorisation with no capture and no reversal. The reason card schemes put a time limit on that state is that otherwise nobody clears it.
Two things I would measure rather than assume. The rate into
flag_for_reconcileper thousand charges, because that is the real price of your GRACE value and the only way to tell whether GRACE is too short. And the age in hours of the oldest unreconciled trip, not the count. A count normalises, and a flat four hundred reads the same every week. The oldest item maps to one named customer with no trip and no charge.The version that needs no queue is one write: the uniqueness record as a column on the ledger entry under the pre-minted id, rather than a separate markers row. Whether that is available to you depends on whether the wallet, the ledger and the markers sit in the same store, and the post reads like they may not.
Thanks Arun, this is a sharp read, and you're right on the main point. The fence doesn't remove the failure. It changes what the failure costs: a double debit becomes a trip waiting for a person. That's the trade I wanted, but only if someone works the queue.
Both measurements are better than what I wrote. Reconcile rate per thousand charges is the honest price of GRACE, and I hadn't framed it that way. Oldest unreconciled age is the right alert too. A count would hide one customer stuck for days. I'm adding both.
On the one-write version: that's where the next step in the post is going, putting the fence inside the debit write's own condition so burning the marker and moving the money can't come apart. You read the layout correctly. The wallet, ledger and markers don't share a transaction today, which is why it's a staged change rather than a one-liner.
The authorization-with-no-capture comparison is a good one. Card schemes put a clock on that state for the same reason, and a reconcile queue needs one too.
Stamping the plan with a pre-minted ledger id before any money moves is a neat answer to "who is allowed to finish it". Whoever finds the trip later can see exactly how far the first attempt got, instead of inferring it from the wallet balance.
We dealt with a non-money version of the same shape at Ascendra Ventures, on a candidate enrichment pipeline for a talent platform. Slow steps (LinkedIn enrichment, OpenAI batch parsing, embeddings, PostgreSQL writes) kept leaving people in unclear states. We moved the processor into Temporal: one workflow claims the person, urgent profiles run immediately and the rest are batched, the batch path waits durably and falls back to polling when webhooks go quiet, and every person ends in a final status so failed work is visible. The rule we held to is the one your post describes: every item either finishes or visibly fails, and nothing sits in the middle.
In your setup, what decides that a claim in "charging" is stale enough to fence: a fixed timeout, or something stored in the plan itself?
What I like most is that once the fence is in, GRACE stops being a correctness setting and becomes a performance knob. Set it too short and you get more reconcile flags and wasted attempts, not double debits.
That makes it easy to tune from data: put it a bit above the p99.9 of claim-to-debit time, then count how many takeovers end in the "marker already taken" branch. If that number isn't close to zero, your real stalls are longer than the grace period, and the people working the reconcile queue are the ones paying for it.