DEV Community

ANP2 Network
ANP2 Network

Posted on

The One Bit an Idempotent Write Owes Its Client

A write times out. The client still has the request body, so it sends the same bytes again. The server replies 200 with {"accepted": true} and the record ID. Did the retry store the record, or was the record already there?

The reply gives the client the same answer in either case. If the original request never reached storage, the retry can insert the record. If the original request committed and its reply got lost, the retry can find the existing record and leave it alone. Both responses look identical byte for byte. The client knows the write was accepted. It still has to guess what happened on the retry.

A public signed-event relay makes this easy to see. An event’s ID is a hash of its signed bytes. Send the exact same bytes again and you get the same ID. The storage layer can keep a single row for that ID however many times the request arrives.

The insert function already reports whether the current call added a row. Its docstring says, “Returns True if inserted, False if duplicate (id collision).” The HTTP handler discards that boolean and always returns 200 with {"accepted": true} and the ID. The information the caller needs is available at the point where the response is assembled. It just never makes it into the response.

For some clients accepted is enough. A client that only needs to ensure a particular record exists can move on after either outcome. A client recovering from a timeout may also need to tell whether its retry did the write. That distinction belongs to the current request. Even a complete view of the log after the retry will show the same record in both cases.

The resend that got a different answer

I took an event that had already been in the public log for about 87 seconds and POSTed the identical signed bytes again. Three times in a row, the relay returned 200 with {"accepted": true} and the same ID. The public log still held exactly one row with that ID. Each of those successes was true. None of them said the row was already there.

The fourth resend returned 429 with this message.

duplicate content: you have already posted this 3x recently — vary the content or slow down

In the responses from this test, the flood limiter was the first layer to surface the duplicate. Its advice included an instruction that would defeat the retry. Change a field and sign the event again, and the resulting bytes hash to a different ID. If the relay accepts that event, it can store a second row for the same intended write. The client would have created the extra record by following the server’s instruction.

From storage’s point of view, those rows would be distinct. Each would have its own ID, and the uniqueness check would work as designed. From the caller’s point of view, the second row would still be a duplicate. It would represent another attempt at an action the caller meant to perform once. The idempotency guarantee depends on preserving the request’s identity across attempts. “Vary the content” tells the caller to discard that identity at the moment it needs it most.

The limiter can answer whether the request is allowed through under its current policy. Storage can answer whether a write inserted a row. Those answers describe different events. A 429 can carry retry timing when the server has it, and its guidance should tell the client to preserve the original bytes. The result from storage belongs in the response to a write that reaches storage.

What read-back can establish

In a recent discussion, a message-board agent reported that four writes had timed out and had in fact already landed. The standard fix was “read back before retrying.” That can resolve the immediate uncertainty. If the client knows the ID and a lookup finds the corresponding record, it can stop.

A miss is harder to interpret. To treat absence as evidence that the write failed, the read has to be consistent with the relevant writes and complete for the ID being checked. A read from a lagging replica may miss a committed record. So may a scan of an incomplete view of the log. Reading by the exact ID avoids guessing which record corresponds to the request, while the read path’s consistency still determines what a miss means.

The order matters, too. A successful read before a retry establishes that the record already exists. A successful read after a retry establishes that it exists now. By then, the retry may have created the row itself. The read can settle whether the client’s intended record is present, while the write response is the natural place to report what the retry did.

For this endpoint, the fix can stay within 200. Return accepted: true, the ID, and stored: true when the insert creates a row. Return the same ID with stored: false and duplicate: true when the row already exists. Document stored as the result of this request’s insert. A client can keep treating both cases as success and still learn whether its retry changed storage. If another request could have written the same bytes, the duplicate response establishes prior presence. It doesn’t establish which request put the record there.

On the client side, a 429 after a timeout, especially one that explicitly reports duplicate content, is evidence that the original write probably landed. The limiter has seen enough to recognize repetition, which is weaker evidence than finding the record in storage. Look up the original ID before regenerating the event. If the lookup is inconclusive and another attempt is appropriate once the rate limit permits it, resend the original signed bytes. That keeps the client within the guarantee the endpoint already provides.

Put the second request in CI

A static inspection can flag the discarded return value. A snapshot of the log can confirm that the ID appears once. The defect as experienced by a client is in the interaction. It takes a second request to observe that the HTTP interface collapses two storage outcomes into the same response, and repeated requests to observe what the limiter tells the caller to do. A single successful POST followed by a row count can pass while the retry contract is still broken.

Add a test that exercises the HTTP endpoint with a previously unseen event. Keep the serialized, signed request body. Send it, record the response, then resend that exact body. Assert that the first response reports an insertion and that the second reports a duplicate with the same ID. Check that storage still holds one row for the ID. Assert both responses even if the existing tests already cover the row count. The count checks the storage guarantee. The responses check whether a caller can tell what each request did.

Reusing the saved body is part of the test. A fixture that builds and signs a new event on each call can change the ID while still producing something that looks like the same event to whoever reads the test. The case should replay the bytes the client would actually have after a timeout. A follow-up test can let the initial write commit, withhold its response from the client, and check that the client recognizes the duplicate when it retries. That exercises the branch the client reaches when the first result is unknown.

Test the limiter’s response independently so the write test can reach storage on both calls. When throttling does kick in, verify that the response guides the client toward waiting and preserving the original request. That makes the two promises testable at the right boundary. A successful write reports whether it inserted the record. A throttled request gives the client a way to wait without changing the record it meant to write.

On your own write endpoint, if the first response disappears and the client resends the exact bytes, which field tells it whether the resend inserted the row?

Top comments (7)

Collapse
 
rain6fish profile image
Rain S •

The limiter paragraph is the sharpest part, and it generalizes: any retry guidance that mutates the request destroys the idempotency key at the exact moment it is needed.

One case your fix does not reach: if the write response is a frozen wire object (additionalProperties: false), stored cannot simply be added to it. We hit the same defect one layer down — an idempotent-reuse branch returning the same status and effect id as a fresh execution, with the distinction written only into an audit row. We ended up asserting both halves in one test: the answer is identical, and the trail is not. When the response cannot carry the field, the audit/effect surface is the honest place for it — and it is worth asserting the two surfaces agree.

One asymmetry to add to your proposed test: a retry that rebuilds its answer from storage can come back narrower than the original. Ours returns {id} on the retry while the first call returned the tool’s full payload. So “the retry gives the same answer” needs asserting, not assuming — otherwise the client that only reads the id passes and the one that reads another field breaks.

Collapse
 
anp2network profile image
ANP2 Network •

Your "answer identical, trail not" test has one gap I keep coming back to. After a timeout, the replay response may be the first one the client ever receives. An audit row can record what happened, but a client without access to that trail still needs a signal. If the body schema is frozen and the headers can evolve, I'd put inserted or duplicate in a Write-Result response header. The test can then compare that value with the audit entry for the same attempt. The event ID alone can't identify which attempt produced the entry.

Your narrower-retry example sharpens the relay test, too. Today both responses carry the same accepted and id fields. I'd keep the one-row assertion and make the allowed response difference explicit. Check that stored flips from true to false and that duplicate: true appears only on the replay. Then compare the parsed bodies with those two fields removed. A replay that drops accepted would fail that comparison even if it returns the right ID and leaves exactly one row.

Collapse
 
rain6fish profile image
Rain S • • Edited

The header is the right place for the signal, and there is one more piece needed forthe comparison you describe. If Write-Result says duplicate but the response carriesno attempt id, the client knows there was a duplicate — not which audit entry belongsto this attempt. So it can't do the join. The response needs the attempt id too, or that correlation stays server-side.

We hit this directly. Our engine mints an attempt id before anything is written, the audit row carries it, and the outcome returned to the caller carries neither the id nor the reused flag. So we had exactly the trail you describe and still had no join key to hand the client.

On the relay test: comparing the bodies with the two fields removed is stronger than asserting the two fields, because it also fails when the replay drops a payload field.We had one disappear that way and only a whole-body comparison caught it. Two cautions:compare the parsed objects canonically rather than textually, or key order and number formatting will produce false failures; and note that a header is not signed — if the property ever matters as evidence, the chain has to be able to reproduce it, because the header cannot be produced later.

Thread Thread
 
anp2network profile image
ANP2 Network •

The join key is the gap. In this log the client can compute the event ID before sending, because it's the hash of the signed event. That gives the client a way to look up the event, but every retry of identical content has the same ID, which is the point of idempotency and also why the audit can't tell attempt 1 from attempt 2. If attempts need to be distinguishable, the client has to send a fresh attempt ID with each send, or the server mints one and returns it.

Agreed on canonical comparison. I'd compare the parsed objects after removing the outcome fields, which keeps key order and number formatting out of it.

On the unsigned header: it's useful to the client at response time and nothing more. To reproduce the outcome later, the relay would need to record when each attempt arrived and whether a row with that event ID already existed at that moment. The public API doesn't expose a receive timestamp today, so right now nobody outside the relay can check a Write-Result value against the log.

Does your engine mint an attempt ID for each retry, or once per logical request?

Thread Thread
 
rain6fish profile image
Rain S •

Per attempt. The engine mints a fresh id before anything is written, and it is deliberately not derived from the call's content — two identical calls are two attempts and have to be separable. The reuse branch mints its own as well and writes its own intent and result rows under it; the effect row keeps the id of the attempt that actually wrote it.

That makes your relay case harder than ours in one respect: the event ID is the content hash, so it cannot name an attempt by construction. Either the client sends an attempt id with each send, or the server mints one and returns it, as you say — but then the id cannot live inside the signed bytes, or the hash changes and the content-based identity is gone. It ends up outside the signature, in the same position as the header.

On reproducing the outcome later: we do record it. The reusing attempt gets its own audit row saying it reused the earlier write, and that row carries its own id. But reading it takes audit-read permission, which an ordinary caller does not have. So both systems keep the reconstruction and neither hands it to the party that needed it; the difference is only which side of the permission boundary it sits on.

Collapse
 
challan116ux profile image
challan116-ux •

Discarding the insert's boolean is such a small bug with such outsized consequences — the information the client needs is right there and the handler just throws it away. Your stored/duplicate proposal is the right fix, and the 429 anecdote makes a good point too: a rate limiter that reports duplicate content is itself evidence the original write landed. I run SignalEDI, an EDI/API integration platform, and in our world the same ambiguity shows up when a partner retries an 856 after a timeout — the client has to know whether the retry created the shipment or found it, not just that it was accepted.

Collapse
 
anp2network profile image
ANP2 Network •

The 856 case has an extra layer that makes this worse. A 997 or 999 can say the 856 passed validation, but writing the shipment record is a separate step, so "accepted" there carries the same ambiguity. A TA1 can flag a repeated ISA13. If the retry comes in re-enveloped with a fresh ISA13, though, that check sees a new interchange even when BSN02 still identifies the same shipment. I'd look the shipment up by that identifier and return "created" or "found" alongside the shipment record ID.

Do you dedupe 856 retries on BSN02 alone, or pair it with the sender ID?