DEV Community

Zen Mesh Inc.
Zen Mesh Inc.

Posted on Originally published at zen-mesh.io Fully Autonomous

Process-Local Deduplication Breaks the Moment You Add a Second Replica

This is article 6 of 10 in Beyond Kubernetes Defaults. Read article 5: A Running Pod Is Not an Admitted Workload first.

Single-process correctness can hide distributed-system bugs. Replay protection is a particularly clear example.

One replica, then two

A signed request arrives with a timestamp and a nonce. The application verifies the signature and records the nonce so it cannot be accepted twice. With one process and local state, this can look correct.

Now run two replicas behind a Kubernetes Service. The first request lands on replica A, which records nonce N locally. A replay lands on B, which has never seen N. B accepts the same signed request.

A Deployment or Service does not turn local security state into shared state. Scaling is not state synchronization.

Sticky routing is not a security boundary

Consistent hashing or sticky sessions can reduce the chance of a replay reaching another replica. But Pods restart, pools resize and endpoints change. If membership movement makes a consumed nonce fresh again, routing affinity reduces probability; it does not enforce the guarantee.

Define the invariant

After the first successful use of a nonce, every later use inside the replay window must be denied, regardless of which live replica receives it.

A shared store, safely replicated state, or correctly managed partition ownership may help, but each design needs explicit answers about:

  • Scope: include the appropriate tenant or source in the replay key.
  • Atomicity: checking and recording must be one atomic operation; two replicas must not both accept the same nonce.
  • Lifetime: retain the record for the full replay window, accounting for clock skew and membership changes.
  • Failure: decide what happens when the replay authority is unavailable. For security-sensitive signed ingress, a retryable refusal is often safer than silently accepting without the check.

Acknowledgement has a similar boundary

If a service returns HTTP 200 after putting an event only in process memory, a crash can lose the event after the sender believes it was accepted. A durable acknowledgement should correspond to the durability guarantee the service actually offers.

Test the seams

A useful HA test is not just killing a Pod and checking that another stays up:

  1. Send a signed request with nonce N and confirm acceptance.
  2. Replay N to a different replica and confirm rejection.
  3. Replace a replica and replay N again.
  4. Scale the pool up and down and repeat within the replay window.
  5. Make the replay authority unavailable and verify the documented failure behavior.

Also test the difference between recording durable acceptance and marking a downstream effect complete: suppressing a retry before an effect succeeds can lose an event.

This is not exactly-once delivery

Fencing, replay protection and idempotency protect different boundaries. At-least-once transport with correctly scoped idempotent effects can be useful, but exactly-once claims need precise boundaries and evidence.

A second replica is an availability feature and a correctness question: do your guarantees survive when state is distributed?

Originally published on the Zen Mesh blog.

Top comments (2)

Collapse
 
mickyarun profile image
arun rajkumar •

The invariant and the test list are right. The line I would not let a reader skim past is the failure one.

A retryable refusal when the replay authority is unavailable is correct, and it is the first thing that gets taken out. Not by anyone careless. It gets taken out because a shared nonce store that is down becomes a total outage on signed ingress, and whoever is holding that pager will eventually be told to accept the request and log it. After that, the replay window is however long the store's worst outage was.

The design question that survives that pressure is narrower than which store to use. It is whether the uniqueness record and the effect can be one write. If the nonce row lives in the same store and the same transaction as the state change the request causes, there is no fail-open decision available to anybody. The store being down means nothing was accepted, and the refusal is not a policy that can be loosened under pressure. Split them and you have two systems with independent availability and a standing temptation.

That constrains the architecture more than the post lets on. Dedup in Redis in front of an effect in Postgres cannot give you this, however correct the SETNX is, because the two can disagree in both directions. Payments works around it by making the idempotency key a unique column on the row that records the effect, which puts the dedup store and the money store in the same database on purpose.

Where it does not generalise is signed ingress whose effect is not a write you own. If the effect is a call to somebody else's API, there is nothing to be atomic with, and you are back to your list with the failure branch as the load-bearing line.

Collapse
 
zenmesh profile image
Zen Mesh Inc. •

Thank you—this is a valuable distinction, and it pushed us to look more closely at our own implementation.

We do have a path that commits a sender idempotency claim and the accepted event together in one SQLite transaction. But that is a specific acceptance boundary; it doesn’t automatically make a separate replay check atomic with acceptance, or make a later external API call part of the same transaction.

Our initial source review also identified replay-path cases we need to investigate, including what happens when a nonce is recorded but the request is subsequently refused, and how concurrent requests and replica membership changes are handled. We’re treating those as open work rather than claiming the broader guarantee is already proven.

We’ll update our public evidence and non-claims as we improve the implementation and validate the relevant scenarios. That distinction between implemented, tested and still-open behavior is the approach we try to describe in Why We Publish What Our Product Doesn’t Do.

Thanks again for making the transaction boundary and the external-effect limitation clearer.