DEV Community

Cover image for Your AI Agent Retried the Tool Call. The First One Had Already Succeeded.
Rudratosh Shastri
Rudratosh Shastri

Posted on Edited on

Your AI Agent Retried the Tool Call. The First One Had Already Succeeded.

Stable operation IDs prevent duplicate charges

The payment went through. The response disappeared. The agent tried again. Your customer was charged twice.

Imagine this.

You ask your AI agent to pay an outstanding invoice.

The agent calls your payment tool.

The payment provider processes the transaction successfully. The money is charged. The transaction is committed.

But something goes wrong.

The response never reaches your application.

Your tool times out.

The agent sees an error:

Tool execution failed: Request timed out.
Enter fullscreen mode Exit fullscreen mode

So it tries again.

The second request reaches the payment provider.

Another payment is created.

Two successful charges. One user request.

Your agent wasn't trying to do anything malicious. Your payment API wasn't necessarily broken. Your database didn't necessarily lose any data.

Your system simply couldn't distinguish between an operation that failed and an operation that succeeded without returning a response.

This is one of the most dangerous failure modes in AI agent infrastructure.

And it becomes particularly interesting when you realize that the agent might not be the only component retrying the operation.

Your HTTP client can retry it.

Your tool framework can retry it.

Your queue can redeliver it.

Your worker can restart it.

And the model itself can decide to call the tool again.

Every component thinks it is helping.

Together, they can create duplicate side effects.

Let's look at why this happens and how to design an execution system that can survive it.


1. First, understand what actually happened

Before discussing architecture, let's reconstruct the incident.

Assume an invoice is worth $480.

The user asks:

Pay invoice INV-2048 for $480.
Enter fullscreen mode Exit fullscreen mode

The agent generates a tool call:

{
  "tool": "charge_invoice",
  "arguments": {
    "invoice_id": "INV-2048",
    "amount": 480,
    "currency": "USD"
  }
}
Enter fullscreen mode Exit fullscreen mode

Your backend forwards the request to the payment provider.

Here is the actual sequence of events.

Timeline: one request, two charges

Illustrative failure timeline: the first transaction succeeds, but the caller never receives confirmation.

Notice what the agent actually knows.

It knows that it did not receive a response.

It does not know that the payment failed.

These are two completely different statements.

A timeout tells you what the caller observed. It does not tell you what the server committed.

This distinction is fundamental to distributed systems.

A request can have several outcomes:

Situation What actually happened What the caller knows
Request never reached the server Nothing happened Usually a connection failure
Server rejected the request Nothing was committed Received an error
Server processed the request Operation succeeded Received success
Server processed the request, response was lost Operation succeeded Timeout
Server is still processing Outcome is pending Timeout or pending response

The fourth situation is the dangerous one.

From the caller's perspective, the outcome is unknown.

From the provider's perspective, the transaction is complete.

Your execution system needs to handle both realities.


2. Why AI agents make retries more complicated

Retries are not new.

We've been building distributed systems with network timeouts, message redelivery, worker crashes, and transient service failures for decades.

So why does this deserve special attention in AI agent systems?

Because an agent introduces another decision-making component into the retry loop.

Consider a traditional application:

A traditional application's retry path

A frontend might retry a request after a timeout.

But in an agent application, the execution path can look like this:

An agent application's six-layer execution path

Now introduce independent retry mechanisms.

Component Why it might retry
Model It receives an error and decides to try again
Agent framework It has an automatic tool retry policy
Tool executor It retries transient execution errors
HTTP client It retries connection or service errors
Queue The message wasn't acknowledged
Worker The process restarted before completing the job
Payment provider integration The SDK retries a request

The problem is not that retries exist.

The problem is that each layer may have a different understanding of what constitutes the same operation.

A tool-call ID is not a business operation ID

This distinction is particularly important.

Imagine the agent produces:

First invocation:

tool_call_id = call_A123
Enter fullscreen mode Exit fullscreen mode

It times out.

The model then generates another invocation:

Second invocation:

tool_call_id = call_B456
Enter fullscreen mode Exit fullscreen mode

From the model framework's perspective, these are two different tool calls.

But from the user's perspective, both requests represent one intention:

Pay invoice INV-2048 for $480.
Enter fullscreen mode Exit fullscreen mode

A new tool-call ID must not automatically mean a new payment operation.

This is where many agent implementations make a subtle mistake.

They use the tool invocation identifier as the identifier of the business operation.

Those identifiers have different responsibilities.

Identifier What it identifies
conversation_id The conversation
task_id The agent's task
tool_call_id One tool invocation
operation_id One logical business operation
attempt_id One execution attempt
provider_reference The downstream provider's transaction

The operation ID must remain stable across retries, even when the tool-call ID changes.

This is the foundation of safe retries for AI agents.


3. Idempotency: the engineering concept that prevents duplicate operations

Let's introduce the term that solves much of this problem.

An operation is idempotent when applying it multiple times has the same intended effect as applying it once.

For example:

PUT /users/123
Enter fullscreen mode Exit fullscreen mode

With the same representation and intended state, repeating the request should not create another user.

But consider:

POST /payments
Enter fullscreen mode Exit fullscreen mode

Repeating this request can create another payment.

HTTP semantics are explicit about this distinction.

RFC 9110, Section 9.2.2 defines idempotent methods and explains why clients need to be careful automatically retrying non-idempotent requests.

The important detail is that idempotency applies to the intended effect of the operation, not simply whether the HTTP request succeeds.

What does an idempotent payment operation look like?

Suppose the application generates a unique operation ID:

operation_id = op_9f71c3
Enter fullscreen mode Exit fullscreen mode

It sends this identifier with the payment request.

First attempt:

POST /payments
Idempotency-Key: op_9f71c3
Enter fullscreen mode Exit fullscreen mode

The provider processes the payment.

The response is lost.

The application retries:

POST /payments
Idempotency-Key: op_9f71c3
Enter fullscreen mode Exit fullscreen mode

The provider recognizes the same logical request and applies its documented idempotency behavior instead of creating another payment.

The application can then recover the original result.

The critical property is:

Same logical operation
        |
        v
Same idempotency key
        |
        v
Same intended side effect
Enter fullscreen mode Exit fullscreen mode

A new operation should receive a new key.

Important: don't generate an idempotency key from the payload alone

This is a surprisingly common design mistake.

Imagine two legitimate purchases:

{
  "product_id": "item-100",
  "quantity": 1,
  "amount": 50
}
Enter fullscreen mode Exit fullscreen mode

The customer might intentionally purchase the same item twice.

A hash of the request payload would be identical.

But these are two different business intentions.

Amazon explains this exact problem in its engineering article, Making retries safe with idempotent APIs.

Its recommendation is to use a unique caller-provided request identifier to express intent, rather than guessing that identical parameters must represent duplicate requests.

A better design separates two things:

Idempotency key: Identifies the logical operation.

Request fingerprint: Verifies that the same key has not been reused with different parameters.

For example:

operation_id:
op_9f71c3

request_fingerprint:
SHA256(
    tenant_id +
    invoice_id +
    amount +
    currency +
    operation_type
)
Enter fullscreen mode Exit fullscreen mode

On a retry:

  • Same operation ID, same fingerprint: treat it as the same operation.
  • Same operation ID, different fingerprint: reject it.
  • Different operation ID, same fingerprint: potentially a new intentional operation.

This is a very important distinction.


4. The architecture: put an execution coordinator between the agent and the tool

I would not let the agent directly trigger a sensitive external operation.

Instead, introduce a durable execution coordinator.

Its responsibility is simple:

Remember what the application intended to execute, whether it has already executed, and what result was produced.

The architecture becomes:

Durable operation coordinator architecture

Let's understand the responsibilities.

Component Responsibility
Agent Proposes an operation
Validation Checks arguments and business constraints
Authorization Determines whether the user can perform the operation
Operation coordinator Identifies the logical operation
Operation ledger Stores durable execution state
Tool executor Performs the authorized operation
Provider Processes the external side effect
Reconciliation Resolves uncertain outcomes

The coordinator is not just another retry wrapper.

It is the place where the system maintains the identity and lifecycle of the operation.

A crucial design decision

The application should create the operation identity before executing the side effect.

For a sensitive action, this can happen when a user approves the proposed payment.

For example:

{
  "operation_id": "op_9f71c3",
  "task_id": "task_2026_114",
  "operation": "invoice.payment",
  "resource": "INV-2048",
  "amount": 480,
  "currency": "USD",
  "status": "PENDING"
}
Enter fullscreen mode Exit fullscreen mode

The model can propose an operation.

The trusted application creates and controls its operation ID.

The model should not be responsible for generating a fresh idempotency key on every retry.


5. Build a durable operation ledger

A retry-safe execution system needs persistent state.

An in-memory dictionary is insufficient.

Consider this implementation:

completed_operations = {}

async def execute_payment(operation_id, amount):

    if operation_id in completed_operations:
        return completed_operations[operation_id]

    result = await payment_provider.charge(amount)

    completed_operations[operation_id] = result

    return result
Enter fullscreen mode Exit fullscreen mode

It looks reasonable until the process crashes.

Or two workers receive the same operation simultaneously.

Or the provider succeeds but the application crashes before storing the result.

In all three situations, the application can execute the payment again.

The operation state must live in durable storage.

A PostgreSQL schema

Here's a simplified schema:

CREATE TABLE tool_operations (
    tenant_id UUID NOT NULL,
    operation_id TEXT NOT NULL,

    task_id UUID NOT NULL,
    agent_id TEXT NOT NULL,

    operation_type TEXT NOT NULL,
    request_hash TEXT NOT NULL,

    status TEXT NOT NULL,

    provider_reference TEXT,
    response_json JSONB,

    attempt_count INTEGER NOT NULL DEFAULT 0,

    lease_owner TEXT,
    lease_expires_at TIMESTAMPTZ,

    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    PRIMARY KEY (tenant_id, operation_id)
);
Enter fullscreen mode Exit fullscreen mode

The operation_id is an application-generated string identity (for example op_9f71c3), not a database-minted UUID — the whole point is that your trusted application owns it and reuses it across retries. The composite primary key ensures that an operation ID is unique within the tenant's operation namespace.

A production implementation should also constrain valid status values, index its recovery queries, and define a retention policy.

The execution state machine

Do not model execution as simply:

SUCCESS / FAILED
Enter fullscreen mode Exit fullscreen mode

That is insufficient for distributed operations.

A provider can process an operation while the application is still waiting for confirmation.

Use more meaningful states:

Execution state machine with UNKNOWN and reconciliation

What do these states mean?

Status Meaning
PENDING Operation recorded, not yet executed
IN_PROGRESS A worker currently owns the execution
SUCCEEDED Operation completed and its result is persisted
UNKNOWN The downstream outcome is uncertain
RECONCILING The system is checking the actual downstream result
RETRYABLE A failure was established as safe to retry
FAILED_FINAL The operation was definitively rejected or failed

Why is UNKNOWN so important?

Because a timeout must not automatically transition an operation to FAILED_FINAL.

That would turn an unknown outcome into a false statement.

And a false statement is exactly how duplicate side effects happen.


6. Implementing a retry-safe execution flow in Python

Let's put the concepts together.

The following is an illustrative Python service design. It uses an operation ledger, a stable operation ID, a request fingerprint, and a provider-side idempotency key.

The database and provider adapters are interfaces; their atomicity and idempotency contracts must be implemented by the underlying services.

First, define the operation:

from dataclasses import dataclass
from decimal import Decimal
from uuid import UUID


@dataclass(frozen=True)
class PaymentIntent:
    operation_id: str
    tenant_id: UUID
    invoice_id: str
    amount: Decimal
    currency: str


@dataclass(frozen=True)
class PaymentResult:
    provider_reference: str
    status: str
    amount: Decimal
    currency: str
Enter fullscreen mode Exit fullscreen mode

Notice that operation_id is part of the trusted application-created intent.

It should not be regenerated just because the model made another tool call.

Step 1: Calculate a canonical request fingerprint

import hashlib
import json


def fingerprint(intent: PaymentIntent) -> str:

    payload = {
        "tenant_id": str(intent.tenant_id),
        "invoice_id": intent.invoice_id,
        "amount": str(intent.amount),
        "currency": intent.currency.upper(),
        "operation_type": "invoice.payment",
    }

    canonical = json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
    )

    return hashlib.sha256(
        canonical.encode("utf-8")
    ).hexdigest()
Enter fullscreen mode Exit fullscreen mode

The fingerprint protects against reusing the same operation ID with different payment parameters.

Do not include volatile fields such as the current timestamp, retry count, or tool-call ID.

Those values change between attempts even when the business operation does not.

Step 2: Claim the operation atomically

Conceptually:

record = await ledger.claim(
    tenant_id=intent.tenant_id,
    operation_id=intent.operation_id,
    request_hash=fingerprint(intent),
)
Enter fullscreen mode Exit fullscreen mode

The claim() method must be implemented using database-level concurrency control.

For example:

  • A unique constraint on the operation identity.
  • A transaction around insertion or state inspection.
  • Row-level locking or an atomic conditional update.
  • A lease for the worker that owns the execution.
  • A request-hash comparison for existing records.

A second worker must not independently claim the same operation and submit another payment.

Importantly, do not hold a database transaction open while waiting for an external HTTP request. Use a short transaction to claim the work, perform the network operation outside it, and persist the result in another transaction.

Step 3: Return an existing result when available

if record.status == "SUCCEEDED":
    return record.response
Enter fullscreen mode Exit fullscreen mode

The operation has already completed.

There is no reason to call the provider again.

For in-progress or uncertain operations:

if record.status in {"IN_PROGRESS", "UNKNOWN"}:
    return await reconcile_or_resume(record)
Enter fullscreen mode Exit fullscreen mode

The reconciliation path must check the provider's state or safely resume using the same downstream idempotency key.

It must not blindly create another payment.

Step 4: Execute with a stable provider key

provider_key = (
    f"invoice-payment:{intent.operation_id}"
)

result = await payment_provider.create_charge(
    invoice_id=intent.invoice_id,
    amount=intent.amount,
    currency=intent.currency,
    idempotency_key=provider_key,
)
Enter fullscreen mode Exit fullscreen mode

The downstream provider's idempotency contract is important here.

It must recognize retries using the same key and apply its documented duplicate-request behavior.

The provider response should then be persisted:

await ledger.mark_succeeded(
    operation_id=intent.operation_id,
    provider_reference=result.id,
    response=result.to_dict(),
)
Enter fullscreen mode Exit fullscreen mode

Step 5: Handle timeouts as unknown outcomes

This is the part that many implementations get wrong.

try:
    result = await payment_provider.create_charge(
        invoice_id=intent.invoice_id,
        amount=intent.amount,
        currency=intent.currency,
        idempotency_key=provider_key,
    )

except TimeoutError:

    await ledger.mark_unknown(
        operation_id=intent.operation_id,
    )

    return {
        "operation_id": str(intent.operation_id),
        "status": "UNKNOWN",
        "message": (
            "The provider response was not received. "
            "The operation is being reconciled."
        ),
    }
Enter fullscreen mode Exit fullscreen mode

The important thing is what this code does not do.

It does not generate a new operation ID.

It does not generate a new provider idempotency key.

It does not tell the model that the payment definitely failed.

It preserves the uncertainty.

A retry is safe only when the system preserves the identity of the original operation and knows how to handle its previous outcome.


7. The most dangerous gap: your database cannot control another company's database

There is a subtle problem with the previous implementation.

Your PostgreSQL database and an external payment provider are two independent systems.

They do not share a transaction.

Consider this sequence:

1. Insert operation record.
2. Set status = IN_PROGRESS.
3. Send request to payment provider.
4. Provider creates charge.
5. Provider returns success.
6. Your application crashes.
7. Result is never written to PostgreSQL.
Enter fullscreen mode Exit fullscreen mode

After restart, your database still says:

status = IN_PROGRESS
Enter fullscreen mode Exit fullscreen mode

But the payment has already happened.

This is a distributed transaction problem.

You cannot solve it by putting another try/except around the HTTP request.

Three ways to handle it

Option A: Downstream idempotency support

The provider accepts a stable idempotency key.

On retry, submit the identical request using the same key, subject to the provider's documented retention and request-matching rules.

This is usually the simplest approach.

Option B: Query by a stable business reference

If the provider supports transaction lookup, associate the external operation with a stable merchant reference.

When the response is lost:

  1. Query the provider using that reference.
  2. Check whether the transaction exists.
  3. Retrieve its actual status.
  4. Update your local operation ledger.
  5. Only retry when the provider contract establishes that doing so is safe.

Option C: Keep the operation in an unknown state

What happens when the provider doesn't support idempotency and cannot reliably identify the earlier request?

Do not pretend that you can guarantee exactly-once execution.

Keep the operation in UNKNOWN, prevent an automatic duplicate side effect, and route it to reconciliation or human investigation.

This is much safer than charging someone twice because a timeout looked like a failure.

An important distributed-systems rule

The local idempotency ledger protects your application from repeating its own work.

It cannot independently guarantee that a remote system executed a side effect exactly once.

You need a downstream idempotency contract or a reliable reconciliation mechanism.


8. Why an idempotency key alone is not enough

Let's assume you have implemented idempotency keys.

Are you done?

Not yet.

There are several failure modes worth handling explicitly.

Failure 1: The agent uses a new key on retry

First attempt:
key = payment-001

Second attempt:
key = payment-002
Enter fullscreen mode Exit fullscreen mode

The provider sees two different operations.

Two payments can be created.

Fix: Generate the operation identity in trusted application state and reuse it throughout the workflow.

Failure 2: Same key, different parameters

First request:

{
  "invoice_id": "INV-2048",
  "amount": 480
}
Enter fullscreen mode Exit fullscreen mode

Retry:

{
  "invoice_id": "INV-2048",
  "amount": 840
}
Enter fullscreen mode Exit fullscreen mode

Same key. Different intent.

The system should reject this request.

AWS discusses parameter mismatch handling in its idempotent API design guidance. The IETF's current Idempotency-Key Internet-Draft also defines request fingerprints as a mechanism for detecting this situation.

Idempotency-Key HTTP Header Field — draft-08.

Fix: Bind each operation ID to a canonical request fingerprint.

Failure 3: Two workers execute simultaneously

Imagine two workers receive the same queue event.

Both check the operation ledger.

Both see that the operation is not completed.

Both submit the payment.

A normal application-level if statement cannot prevent this race.

Fix: Use a database uniqueness constraint, atomic claims, and worker leases.

For long-running operations, leases should have expiration and ownership checks. A stale worker must not be able to overwrite the result of a newer worker.

Failure 4: The idempotency record expires too soon

Suppose your system retains the operation record for 24 hours.

An agent workflow gets paused for two days.

It resumes and submits the same operation.

The provider no longer recognizes the old idempotency key.

A duplicate may be created.

Fix: Retain the durable business operation record for the required business and replay horizon. Align it with the downstream provider's idempotency retention policy.

Do not assume that provider keys remain valid forever.

Failure 5: The provider succeeds, but your result persistence fails

Provider: SUCCESS
Your database: WRITE FAILED
Agent: NO RESPONSE
Enter fullscreen mode Exit fullscreen mode

This is not a normal failure.

It is an uncertain result.

Fix: Preserve the operation identity and reconcile using the provider key or transaction reference.

The system should never convert an ambiguous external result into an automatic fresh operation.


9. What about queues, workers, and background jobs?

The same problem appears when AI agents use asynchronous execution.

For example:

Queue and worker redelivery path

Suppose the worker successfully processes the message but crashes before acknowledging it.

The queue may deliver that message again.

This behavior is not unusual.

Amazon SQS Standard queues provide at-least-once delivery. AWS explicitly recommends designing consumers to be idempotent because a message can be delivered more than once.

Amazon SQS — At-least-once delivery.

AWS Lambda also documents that event-source mappings can process events more than once and recommends idempotent function code.

Lambda best practices.

The important point:

A message being delivered twice must not mean the business operation happens twice.

Introduce an operation consumer

A queue message should carry the stable operation identity:

{
  "event_id": "evt_901",
  "operation_id": "op_9f71c3",
  "operation_type": "invoice.payment",
  "tenant_id": "tenant_42"
}
Enter fullscreen mode Exit fullscreen mode

The worker should load the operation record before executing anything.

async def handle_message(message):

    operation = await ledger.get(
        tenant_id=message.tenant_id,
        operation_id=message.operation_id,
    )

    if operation.status == "SUCCEEDED":
        return

    await execute_operation(operation)
Enter fullscreen mode Exit fullscreen mode

This example omits the actual concurrency claim deliberately. Production workers must atomically claim or resume the operation, not just read a status and then execute.

Don't confuse an exactly-once queue feature with exactly-once business execution

Some messaging systems offer exactly-once delivery or processing features under defined conditions.

For example, Google Cloud Pub/Sub documents exactly-once delivery for supported pull subscriptions within a region.

But this does not automatically provide exactly-once execution across an external payment provider, your database, and your AI agent.

Google Cloud Pub/Sub — Exactly-once delivery.

A queue can manage message acknowledgment.

It cannot automatically make every downstream side effect part of the same atomic transaction.


10. Use the transactional outbox when execution crosses systems

There is another common failure in event-driven systems.

Your application commits a database transaction and then tries to publish an event.

await db.save_payment(payment)

await queue.publish({
    "operation_id": operation_id
})
Enter fullscreen mode Exit fullscreen mode

What happens if the database commit succeeds but the process crashes before publishing?

The payment exists.

The event was never published.

A downstream worker never learns that it needs to continue the workflow.

Reversing the order creates the opposite problem: the event can be published even though the database transaction later fails.

This is known as the dual-write problem.

The transactional outbox pattern addresses it.

The design

Write the business state and an outbox event in the same database transaction.

BEGIN;

INSERT INTO tool_operations (
    tenant_id,
    operation_id,
    operation_type,
    request_hash,
    status
)
VALUES (
    :tenant_id,
    :operation_id,
    'invoice.payment',
    :request_hash,
    'PENDING'
);

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload
)
VALUES (
    :event_id,
    :operation_id,
    'PAYMENT_REQUESTED',
    :payload
);

COMMIT;
Enter fullscreen mode Exit fullscreen mode

A separate dispatcher reads the outbox and publishes events.

Transactional outbox pattern

Now the application cannot commit the operation record without also recording the event that needs to be dispatched.

There is still an important detail:

The dispatcher can publish an event and crash before marking it as published.

It may publish the same event again.

So the consumer must also be idempotent.

The outbox solves the dual-write problem. It does not magically eliminate duplicate message delivery.

AWS explains this distinction in its Transactional Outbox Pattern guidance.


11. Not every operation should use the same retry strategy

Retrying a failed read operation is very different from retrying a payment.

We need to classify tool operations.

Operation Retry strategy
Read a customer record Usually safe to retry, subject to authorization and consistency requirements
Search documents Usually safe to retry
Generate an internal draft Retryable if duplicate drafts are acceptable or deduplicated
Create a support ticket Use a stable operation ID
Send an email Deduplicate using a durable outbound message identity
Export a report Use a stable export job identity
Charge a payment Provider idempotency plus reconciliation
Issue a refund Stable operation identity and provider-level controls
Deploy software Release/deployment identity and environment-level state checks
Delete customer data Durable state tracking and reconciliation

Retry only errors that are safe to retry

A useful classification:

Potentially retryable

  • Temporary connection failures.
  • Rate limits.
  • Explicit transient server failures.
  • Worker interruptions.

Not automatically retryable

  • Invalid arguments.
  • Authentication failures.
  • Authorization denials.
  • Business-rule rejections.
  • Payment declines.
  • Unknown external outcomes.

An important nuance: even HTTP 500 does not necessarily mean that no side effect occurred. The service may have committed the operation and failed while producing the response.

That is why idempotency and outcome reconciliation matter more than simply checking the HTTP status code.

Use exponential backoff and jitter

Don't make every worker retry immediately.

A simple strategy:

import random


def retry_delay(attempt: int) -> float:

    base_delay = 1.0
    max_delay = 30.0

    exponential = min(
        max_delay,
        base_delay * (2 ** attempt),
    )

    return random.uniform(0, exponential)
Enter fullscreen mode Exit fullscreen mode

This is full jitter: each retry waits for a random duration between zero and the calculated exponential ceiling.

Also impose:

  • A maximum number of attempts.
  • A total execution deadline.
  • A retry budget.
  • Per-provider concurrency limits.
  • Respect for server-provided retry instructions when applicable.

Retries are not free.

When a downstream service is overloaded, aggressive retries can increase its load and make recovery harder. Google's SRE book covers exactly this dynamic — how well-intentioned client retries turn a partial outage into a full one — in Addressing Cascading Failures, and AWS discusses the mitigations in Timeouts, retries, and backoff with jitter.


12. Give the agent an honest execution result

There is a small but important interface-design issue.

What does the tool return after a timeout?

Many systems return something like:

{
  "success": false,
  "error": "Payment request timed out"
}
Enter fullscreen mode Exit fullscreen mode

This response is misleading.

Did the provider reject the payment?

Did the network fail before the request was sent?

Did the payment succeed but the response get lost?

The tool doesn't know.

The model receives a false failure signal and may generate another payment request.

A better response is:

{
  "operation_id": "op_9f71c3",
  "status": "UNKNOWN",
  "message": "The payment outcome is being verified.",
  "retry_allowed": false
}
Enter fullscreen mode Exit fullscreen mode

For an operation that is still processing:

{
  "operation_id": "op_9f71c3",
  "status": "IN_PROGRESS",
  "message": "The original operation is still being processed.",
  "retry_allowed": false
}
Enter fullscreen mode Exit fullscreen mode

For a completed operation:

{
  "operation_id": "op_9f71c3",
  "status": "SUCCEEDED",
  "provider_reference": "ch_123456",
  "amount": 480,
  "currency": "USD"
}
Enter fullscreen mode Exit fullscreen mode

The agent should receive the outcome of the original operation, not an invitation to create another one.

Keep the operation identity outside the model's control

I would design the tool interface around this principle:

async def charge_invoice(
    invoice_id: str,
    amount: Decimal,
    currency: str,
    execution_context: TrustedExecutionContext,
):
    ...
Enter fullscreen mode Exit fullscreen mode

The trusted context is injected by the application.

It contains the original operation ID, tenant identity, user identity, authorization context, and task metadata.

The model cannot replace that identity simply by generating a different tool-call ID.

If the model changes the invoice or amount, the application must validate whether the new request represents the same approved operation. A materially different action requires a new authorization and a new operation identity.


13. How to test this properly: break the response on purpose

A normal happy-path test will never expose this problem.

You need fault injection.

I would build the following tests before allowing an agent to perform externally visible operations.

Test Injected failure Expected result
1 Provider succeeds, response is dropped One operation; outcome reconciled
2 Application crashes after provider success Retry returns original result
3 Two workers receive the same operation Only one effective side effect
4 Agent generates a new tool-call ID Same business operation ID is reused
5 Same key, different amount Request rejected
6 Queue redelivers the same message No duplicate business effect
7 Idempotency record expires Old operation is not silently treated as new
8 Provider is unavailable Controlled retry or pending state
9 Provider has no idempotency support Ambiguous outcome enters reconciliation
10 Agent requests a genuinely new payment New operation ID permits a new intentional action

The test that matters most

I would start with this exact scenario:

The break-the-response test scenario

This is a much stronger test than asserting that the agent received a successful response.

It verifies the thing that matters: the actual side effect.

Measure more than the HTTP status

Track:

Metric Why it matters
tool_operation_attempts_total Shows how often execution is retried
tool_operation_duplicates_total Measures duplicate invocations
idempotency_replays_total Counts requests served from existing operation state
operation_unknown_total Tracks unresolved external outcomes
operation_reconciliation_duration Shows how long uncertainty persists
operation_duplicate_effects_total Detects actual duplicate side effects

Every attempt should be traceable to the same operation ID.

Log the tool-call ID separately so you can see when multiple model invocations refer to one business operation.

Do not log payment credentials, authentication tokens, or sensitive customer data just to make debugging easier.


14. What about workflows involving several tools?

The problem becomes even more complicated when an agent executes multiple operations.

Imagine an agent handling an order:

1. Create invoice
2. Charge customer
3. Update inventory
4. Send confirmation email
5. Update CRM
Enter fullscreen mode Exit fullscreen mode

What happens if step 4 fails after steps 1–3 have succeeded?

Should the agent restart the entire workflow?

Absolutely not.

That would risk creating another invoice, charging the customer again, and reducing inventory twice.

The correct approach is to treat the workflow as a series of durable, independently identifiable steps.

A multi-tool workflow as durable independent steps

Each step should have:

  • A stable operation identity.
  • Its own durable state.
  • Explicit retry rules.
  • A recorded result.
  • A way to reconcile uncertain outcomes.

For longer workflows, use an orchestration or saga pattern.

A saga coordinates independent operations and defines how to recover when a later step fails.

But remember: a compensation is not a magical rollback.

Refunding a successful payment is a new business operation. Sending a correction email does not erase an email already received. Reversing inventory may require another controlled transaction.

This is why a durable workflow is more reliable than asking the model to repeat the entire plan after an error.

Temporal's documentation makes a particularly useful distinction: its workflow can record a single completed activity result, but an activity itself may execute multiple times during retries and may partially complete more than once.

Temporal — Activities.

That is an important distributed-systems lesson for agent developers.


15. A practical production checklist

Before giving an AI agent access to tools that create external side effects, I would verify these requirements.

Operation identity

  • [ ] Every business operation gets a stable server-generated ID.
  • [ ] Tool-call IDs and operation IDs are separate.
  • [ ] Retries reuse the original operation ID.
  • [ ] New intentional operations receive new IDs.

Execution state

  • [ ] Operation state is stored durably.
  • [ ] Database uniqueness prevents duplicate claims.
  • [ ] Concurrent workers are coordinated atomically.
  • [ ] Worker leases and stale-worker recovery are defined.
  • [ ] Unknown outcomes are not marked as failed.

External services

  • [ ] Stable provider idempotency keys are used where supported.
  • [ ] Same-key requests are checked against the same parameters.
  • [ ] Provider key expiry is understood.
  • [ ] Provider lookup and reconciliation are available where necessary.
  • [ ] Services without deduplication have a safe ambiguous-outcome policy.

Agent orchestration

  • [ ] Agent retries cannot create fresh business operation identities.
  • [ ] Tool responses distinguish failure, pending, and unknown.
  • [ ] Model retries cannot bypass operation state.
  • [ ] Sensitive operations have appropriate authorization and approval controls.

Testing and monitoring

  • [ ] Response-loss scenarios are tested.
  • [ ] Worker-crash scenarios are tested.
  • [ ] Concurrent duplicate invocations are tested.
  • [ ] Duplicate external side effects are monitored.
  • [ ] Unresolved operations have an escalation path.

If you cannot answer these questions, your agent may be safe during a successful demonstration but unreliable during real production failures.


16. The real lesson: retries are not the enemy

We should not disable retries.

Networks fail. Workers crash. APIs become temporarily unavailable. Message queues redeliver events.

Retries are an essential reliability mechanism.

But a retry must represent the continuation of an existing intention, not the creation of a new intention.

That's the distinction AI agent infrastructure needs to get right.

Your agent can call a tool twice.

Your worker can process a message twice.

Your HTTP client can resend a request.

Your provider can receive the same request multiple times.

None of these events should automatically create another business operation.

The execution layer must know what the user originally asked for, what has already happened, and what remains uncertain.

And it must preserve that knowledge even when the model, worker, application, or network restarts.

A model's retry is not permission to repeat a side effect.

The next time you see an AI agent report a timeout, don't immediately ask why the request failed.

Ask a more important question:

Did the operation fail, or did we simply fail to find out that it succeeded?

Because in production, those two situations can cost your customer very different amounts of money.


Further reading

These are the engineering references behind the architecture discussed in this article.

  1. RFC 9110 — HTTP Semantics, Section 9.2.2: Idempotent Methods
  2. AWS Builders' Library — Making retries safe with idempotent APIs
  3. IETF — Idempotency-Key HTTP Header Field, draft-08 (October 22, 2026)
  4. AWS Lambda Powertools — Idempotency for Python
  5. Amazon SQS — At-least-once delivery
  6. AWS Prescriptive Guidance — Transactional Outbox Pattern
  7. Google Cloud Pub/Sub — Exactly-once delivery
  8. Google SRE — Addressing Cascading Failures
  9. Temporal — Activities

One question for the engineering community:

Have you ever seen an API operation succeed while the caller received a timeout—and what did your system do when the client retried?

I'm particularly interested in how teams handle the UNKNOWN state when an external side effect may already have happened.

Top comments (216)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The line that should be taped above every agent tool — 'a timeout tells you what the caller observed, not what the server committed.' The compounding part is what makes agents different from plain distributed systems: five layers can each retry independently, so one lost response becomes five duplicate charges. Do you treat the stable operation ID as the agent's responsibility, or push it down into the tool framework so the model never has to remember it?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

Glad that line stuck — "a timeout tells you what the caller observed, not what the server committed" is the sentence I'd tape above the tool too.

To your question: push it down, firmly. The model proposes the step, but it shouldn't own the operation ID, because its output isn't stable across retries — re-plan and the arguments drift, so any model-derived identity drifts with them. Mint the ID when the step is planned, persist it before dispatch, and have the tool runner reuse the same ID on every attempt. The model never has to remember anything; the framework guarantees identity even when the model gets forgetful. The compounding you describe — one lost response becoming five duplicate charges across five independently-retrying layers — is exactly why identity can't live at the layer that changes its mind.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Firmly agree — identity can't live at the layer that changes its mind. Minting at plan time and persisting before dispatch is the part most implementations skip, and it's the part that makes the guarantee real rather than aspirational. One follow-on: when the framework itself restarts mid-flight, does the persisted ID survive, or does the store become the new single point of failure for dedup?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

That's the right pressure to put on it. The store does become the critical dependency — but that's the design, not a regression: you've traded N unreliable places where dedup could fail for one place you can actually make durable. The ID has to be persisted before dispatch and in the same transactional store as the operation state (that's the whole point of the outbox), so a restart replays from a committed row, not from memory. A store being a single point of failure is fine when it's one you've engineered for — fsync'd commit, replication, the usual — rather than process memory a restart silently erases. Move dedup to the one component whose durability you control, instead of hoping five stateless layers all stay up.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Exactly — the store being a single point of failure is only a problem if it's an accidental one. Persisting the ID before dispatch, in the same transaction as the operation state, is the whole trick; a restart replays from a committed row instead of from a hope. When the store itself is briefly unavailable, does the agent refuse to dispatch, or queue the operation until the outbox is writable again?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Refuse to dispatch — hard. If the outbox isn't writable you cannot persist the operation id before the side effect, and dispatching anyway reintroduces the exact gap the design closes: an effect that happened with no durable record that it did. Fail-closed at the boundary — the operation stays unstarted and retryable, because "not dispatched" is a safe, recoverable state while "dispatched but unrecorded" is the unrecoverable one. Queue the intent locally only if that local queue is itself durable (then it's just a second outbox); never let the agent fire the real call ahead of the commit. The whole guarantee is "persist before dispatch" — so the moment you can't persist, you must not dispatch.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That frames it cleanly — 'not dispatched' is recoverable, 'dispatched but unrecorded' is the one you can't unwind. The hard part in practice is what the agent does while the outbox is down: does it sit and wait, or degrade to read-only work it can still do safely? Either way, persist-before-dispatch holds as the rule.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Degrade to read-only — that's the right instinct, and it's a clean split because the rule only forbids un-recordable side effects, not all work. While the outbox is down the agent can still read, plan, summarize, and queue intents to commit later; what it must not do is fire anything that changes external state, the one class with no safe record. So "wait" is wrong if it means idle, and "proceed" is wrong if it means dispatch — the correct posture is keep doing the safe half of the work and hold the side-effecting half at the boundary until the outbox is writable. Persist-before-dispatch tells you exactly which half is which.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the clean split — the outbox doesn't stop the agent, it just draws the line between work that changes the world and work that doesn't. The queued-intent half is where I'd watch for the next failure mode, though: when the outbox comes back, a burst of held intents replays at once, and if they were queued without their operation IDs attached, the replay can duplicate exactly what the rule was meant to prevent. Do your queued intents carry their IDs from the moment they're formed, or get them assigned at commit time?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

From formation, always — assigning at commit time is the exact hole you're pointing at. If the ID is minted when the intent is formed, then a replay burst is a set of retries of known operations, and the dedup at the boundary collapses each one onto its existing row; the worst case is harmless re-sends of the same identity. If IDs are assigned at commit, two replays of the "same" intent get two identities and the burst manufactures duplicates — the recovery path becomes the duplicate generator the whole design was meant to kill. So the ID is part of the intent's birth certificate, not a stamp added on the way out. That also makes the queued intent idempotent to re-queue, which you need, because a crash between forming and committing will replay the formation too.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Birth certificate is exactly right — and the crash-between-forming-and-committing case is the one that decides whether it holds. If formation itself replays after a crash, does the re-formed intent mint the same ID or a fresh one? A random ID at formation reopens the same hole one level up, so the ID has to be derived from the intent itself, not minted, for the replay to collapse.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Derived, not minted — you've closed the loop correctly. A random ID at formation just relocates the duplicate hole up one level: crash before persisting the intent, replay formation, get a new random ID, and the two "same" intents no longer collide. So the identity has to be a deterministic function of the intent's content and context — the natural key of the action plus the plan step it belongs to — hashed to an ID. Then re-forming after a crash reproduces the exact same ID, and the persist collapses onto one row whether it's the first attempt or the fifth. The subtlety is choosing the inputs: they have to include everything that makes this action distinct (which invoice, which recipient, which step) and exclude anything that drifts across re-plans (a reformatted amount string, a timestamp, a memo the model rewrote) — otherwise you're back to the arguments-hashed-key problem from the post, where a cosmetic re-plan mints a fresh identity. Derive from the stable intent, not from the model's rendering of it. "Mint" was the wrong verb; you caught it.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Derived, not minted — that's the correction the whole thread needed, and the canonicalization problem is where it gets real. If the plan re-numbers its steps on a re-plan, a step index in the key breaks the same way a reformatted amount does, so the step has to enter the hash as its content, not its position. Who owns that canonical form in your design — the planner normalizing before it hashes, or a schema the model has to fill exactly?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Neither the model nor the raw planner — a dedicated canonicalizer owns it, sitting between them. You can't let the model fill a schema exactly, because "exactly" is the one thing a probabilistic generator can't promise; that's the whole reason we're deriving identity instead of trusting its output. And you can't let the planner hash whatever it emitted, because then step re-numbering, key reordering, and formatting drift all leak into the ID — your step-index point is exactly this: position is rendering, not content. So the flow is: model proposes → canonicalizer projects that proposal onto a normal form → hash the normal form. The canonicalizer is deterministic, owns the rules (semantic step identity not positional, amounts as minor-units integers not formatted strings, sorted keys, dropped cosmetic fields), and is small enough to test exhaustively — which matters, because it's now a correctness-critical component: two intents hash equal if and only if they're the same action, and every bug in it is either a missed duplicate or a false collision. Give it its own test suite with adversarial pairs (same action rendered two ways must match; two different actions must not). The model's job ends at "propose"; identity is assigned by code that can't improvise.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the cleanest split — the model proposes, deterministic code assigns identity, and nothing in between gets to improvise. The part I'd stress-test hardest is the false-collision side: a missed duplicate costs a retry, but a false collision silently drops a real action. How do you version the normal form when a tool's schema changes — do old hashes stay valid, or does a canonicalizer upgrade invalidate the dedup window?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Version the canonicalizer, pin the version to the operation, and never recompute an old hash under a new rule. Store the canonicalizer version alongside the operation id and the hash when the row is written. A retry of an existing operation re-derives its key using the version that's recorded on the row, not the current one — so an in-flight operation keeps colliding with itself correctly across a deploy. New operations formed after the upgrade use the new version. The two never mix: you don't re-hash historical rows, so a v2 canonicalizer can't suddenly split or merge v1 identities. The false-collision risk you're right to fear hardest only appears if you recompute old inputs under new rules — pinning the version makes that impossible, because v1 inputs are only ever hashed by v1. The one thing to protect is the overlap window: an operation minted under v1 that's still retrying when v2 ships must keep resolving under v1, which the pinned version gives you for free. Treat the canonicalizer like a migration, not a hot-swap — old rows carry their own rules with them.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Treating the canonicalizer as a migration is the cleanest framing — pinning the version to the row means the framework owns the identity, and the agent never has to know which rule produced its key. The overlap window is the part most people miss: an in-flight v1 operation that survives the deploy has to keep colliding with itself under v1 rules. Do you store the version in the same operation record, or keep a separate version registry the retry path consults?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Same record, not a separate registry — store the canonicalizer version as a column on the operation row itself. The retry path already has to load that row (it needs the id, the state, the stored hash), so the version rides along in the same read; a separate registry would be a second lookup that can disagree with the row, and now you've got a consistency problem guarding your consistency mechanism. The row is self-describing: it carries its identity, its state, and the exact rule that produced its identity, so anything that can read the operation can re-derive its key correctly with no external context. What does belong in a registry is the code — the set of versioned canonicalizer implementations the retry path dispatches to by the row's version number — but that's a code artifact shipped with the app, not runtime state to consult. Data says "I was hashed by v1"; the binary knows how v1 hashes. Keep the version next to the thing it describes, and the overlap window takes care of itself — a v1 row loads, says v1, gets v1 rules, collides with itself correctly, even weeks after v2 shipped.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the cleaner design — the row carrying its own version means the retry path never has to consult anything it doesn't already hold. The one edge I'd still watch is the deploy window: a v1 row retried after v1's canonicalizer has been removed from the binary. Do you keep old canonicalizers shipped indefinitely, or is there a point where a stale row becomes un-retryable by design?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Keep old canonicalizers shipped as long as any row can still reach the retry path — but make that lifetime finite and explicit, don't let it be "forever by accident." The clean version: the canonicalizer version has a support window tied to the operation TTL, not to the deploy cadence. An operation has a maximum lifetime anyway (the point past which an UNKNOWN stops being auto-retryable and goes to reconciliation/human). So the rule is: you may retire a canonicalizer version only once no live operation minted under it can still be in a retryable state — i.e. after the longest operation TTL has elapsed since you stopped minting new rows under it. Removing it before that is the bug you're pointing at, so guard it: a startup assertion that fails the deploy if the binary drops a version that still has retryable rows in the ledger, turning "silently un-retryable" into "loud, caught in CI." And yes, "un-retryable by design" is a real terminal state — a row whose canonicalizer is gone can't be safely re-derived, so it must fail closed: not auto-retried, routed to reconciliation with its stored hash intact, so a human (or a migration) resolves it rather than the system guessing under new rules. The version pin plus a bounded support window plus a drop-guard is the whole contract: old rules live exactly as long as rows that need them, and not one deploy longer.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the cleanest version of the rule — the support window follows the operation TTL, not the deploy cadence, so a version lives exactly as long as rows that need it. The startup assertion is the part I'd steal: turning 'silently un-retryable' into a loud CI failure is the guard that saves you at 3am. And failing closed to reconciliation instead of re-deriving under new rules is the right terminal state — a row you can't re-derive honestly is one a human should see.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

That's the contract locked: version pinned to the row, support window following the operation TTL, a drop-guard assertion in CI, and fail-closed to reconciliation when a version is genuinely gone. Old rules live exactly as long as the rows that need them. This whole sub-thread is going straight into the engine section of the follow-up — thanks for stress-testing it all the way to the deploy window, which is where it would actually have broken.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Version pinned to the row, with the support window following the operation TTL, is the cleanest answer to the responsibility question — it makes the contract the framework's job, not the model's memory. The drop-guard assertion in CI is the part I'd steal first. Will the follow-up's engine section show the assertion itself, or just the contract it enforces?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

The assertion itself — a contract you can't see is just a promise, and the whole reason the drop-guard works is that it's a concrete, copyable check. So the engine section shows the real thing: at startup, enumerate the canonicalizer versions the binary ships, query the ledger for any retryable row whose version isn't in that set, and fail the boot (and therefore the deploy) if the intersection is non-empty — with the offending versions and row counts in the error so it's actionable, not just red. The contract is the sentence; the assertion is the twenty lines that make the sentence impossible to violate silently. Showing only the contract would be committing exactly the sin the article warns about — describing a guarantee without the mechanism that enforces it. You've been the stress-test for that whole section, so it'll land in the follow-up in a shape you'll recognize.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The contract/assertion split is the cleanest way I've seen it put — the sentence is the promise, the twenty lines are what stop it being a wish. Pushing the stable ID into the framework so the model never has to remember it feels right to me; the moment the model has to track it, you've got a second source of truth to drift. Looking forward to the follow-up — will the drop-guard section name the exact versions it checks, or keep that generic?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Named, not generic — and for the same reason you gave about the second source of truth. The drop-guard doesn't check "a version" in the abstract; it checks the exact canonicalizer version that was pinned onto the operation row when the ID was first minted, against the version running now. Two concrete fields: canonical_version stored beside the operation, and the code's current CANONICALIZER_VERSION constant. If they differ, the guard refuses to treat the stored fingerprint as comparable and routes to reconcile-by-reference instead of trusting a match — because a fingerprint computed under v3 and one computed under v4 can collide or diverge on the same logical request, and a silent match there is exactly the duplicate you were trying to prevent.

The generic version ("we version the canonicalizer") is what everyone writes; the failure is always in what you do when the versions disagree mid-flight, which is why that gets the named fields and a worked example. Same instinct as pushing the stable ID into the framework: the moment "which version produced this fingerprint" lives only in a developer's head, you've got the drift back.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the part most implementations skip — versioning the canonicalizer is the easy half; deciding what a mismatch does is the actual design. Reconcile-by-reference instead of trusting a cross-version fingerprint match is the right failure mode: a fingerprint you can't honestly compare is a question for a human, not a guess for the retry path. And naming the fields makes it auditable — 'which version produced this fingerprint' stops being tribal knowledge and becomes a row in the database.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

"A fingerprint you can't honestly compare is a question for a human, not a guess for the retry path" — that's the sentence, and it generalizes past canonicalizer versions. Any time the comparison that would let you dedupe is itself untrustworthy (version skew, a clock you don't believe, a reference you can't resolve), the safe move is to stop guessing and reconcile, not to pick the convenient answer. And yes — the whole point of naming the fields is that "which version produced this fingerprint" becomes a row you can query after an incident instead of a thing someone half-remembers. Auditable beats clever here.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Exactly — 'auditable beats clever' is the right hierarchy. The queryable version row is what turns a post-incident argument into a lookup instead of a memory test. When the comparison itself is untrustworthy, do you route that to a human review queue, or is there an automated reconcile path that still keeps a person in the loop for the ambiguous cases?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Tiered — automated reconcile is the front door, the human queue is the fallback, and the trick is spending human attention only on the irreducible ambiguity.

The order:

  1. Try to resolve automatically first. If there's a resolver — query by business reference, a downstream idempotency key, a read-back endpoint — the machine settles it with no human at all. Most "ambiguous" cases aren't actually ambiguous; they're just unresolved, and a lookup collapses them to SUCCEEDED or safe-to-retry.
  2. Only the residue escalates. What's left is the genuinely unresolvable: no resolver exists, or two resolvers disagree. That's the human queue. If you route all ambiguity to people, the queue drowns and nobody looks at the one that matters — the same pile-up failure from the UNKNOWN thread.
  3. Keep the person in the loop by tier, not everywhere. High-value operations: the automated path proposes a resolution and a human confirms before the compensating or replay action fires. Low-value: auto-resolve and log, with the decision auditable after the fact. The risk tier decides whether a human is a gate or an auditor.

So it's not queue-vs-automation, it's automation with a human-shaped hole exactly where the machine can't honestly decide — and a per-operation deadline that forces anything stuck into the queue before it ages out silently. The queryable version row is what makes step 1 possible in the first place: you can't auto-reconcile a comparison you can't trust, so you record why you don't trust it and let that route the case.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

This is the cleanest framing of it I've seen — automation with a human-shaped hole exactly where the machine can't honestly decide. The per-operation deadline is the part most teams skip; without it the human queue just becomes a slower version of the silent-aging trap. Where do you draw the high-value line in practice — dollar threshold, or reversibility of the operation?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Reversibility first, dollars second — and in that order deliberately. The primary axis isn't "how much" but "can I cheaply undo it if I'm wrong." An irreversible operation is high-value even when it's cheap: a sent email, a published post, an external state change someone else now sees — none of those have a clean rollback, so a wrong auto-decision there is permanent. That gets the human gate regardless of amount. A reversible operation, even an expensive one, can auto-resolve and compensate if it turns out wrong, because the undo path exists.

So the decision tree is: reversible? → auto-resolve with audit. Irreversible? → human confirms before it fires. Dollars come in as a secondary modulator — within the reversible-but-costly band, the amount sets how tight the escalation deadline is (a $5 reversible op can age for hours; a $50k reversible op gets minutes even though it's technically undoable). But the gate itself is reversibility, because that's the thing a compensating action can or can't save you from. Dollar thresholds alone fail exactly where it hurts: a $2 irreversible send treated as low-value because it's cheap.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Reversibility as the primary axis is the right call — a $2 irreversible send is exactly where dollar thresholds fail silently. It also answers the operation-ID question from the other side: the stable ID is what makes the gate checkable, since without it you can't tell whether the irreversible thing already happened. Does the retry path check the ID before the gate fires, or does the gate sit above that lookup?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Lookup first, always — the gate operates on the result of the lookup, never above it. You can't decide "this needs a human" until you know the operation's current state, so the order is:

  1. Resolve the stable ID against the ledger. This is unconditional and comes first.
  2. If it's already SUCCEEDED → short-circuit, do nothing. The gate never fires, because there's nothing to gate — the irreversible thing already happened and we recorded it. This is the case that saves you.
  3. If it's never-started → proceed to execute (through the gate for irreversible ops, as normal).
  4. If it's UNKNOWN / RECONCILING → now the gate matters: try to resolve automatically, and if it can't be resolved, the reversibility tier decides auto-compensate vs. human confirm.

So the gate sits above the side effect but below the lookup. Putting the gate above the lookup is the classic bug — you escalate or re-execute something that already completed, because you asked "should I do this?" before asking "did I already do this?" You nailed the dependency: the stable ID is what makes the gate checkable, because the gate's entire job is to act on a state, and without the ID there's no state to read — just a fresh guess every time. Lookup answers "what happened"; the gate answers "what to do about what happened." In that order.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the clean ordering — lookup answers 'what happened', the gate answers 'what to do about it', and swapping them is exactly how you re-execute something that already completed. The UNKNOWN / RECONCILING branch is the interesting one: is auto-compensate the default there, or does the reversibility tier decide that per operation?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

The tier decides — auto-compensate is never a blanket default, and the reason is that a compensation is itself a new operation that can fail, not a rollback. So on the UNKNOWN branch the order is:

  1. Resolve first. Did the original actually land? Query by reference. Most of the time this collapses UNKNOWN to a definite yes/no and you never compensate at all — you either accept the success or safely retry.
  2. Only if it's genuinely unresolvable does the tier choose:
    • Reversible + a resolver exists → auto-compensate is allowed, because you can confirm the compensation's own effect and undo cleanly if needed.
    • Irreversible, or no resolver → escalate. There's nothing to safely auto-compensate with, and firing a second side effect on top of an uncertain first one is how you turn one duplicate into two.

The trap is treating auto-compensate as the default for UNKNOWN. That means issuing a refund/reversal for a charge you're not sure happened — if it didn't, you've now created a negative out of thin air. You compensate against a confirmed state, never an unknown one. So: resolve → if still unknown, tier decides → and the tier only says "auto" when both the original and the compensation are things you can verify. Unknown is never a license to fire another irreversible action blind.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Resolve-first is the right order — most UNKNOWNs collapse once you query by reference, and compensating against an unconfirmed state just manufactures a second problem. The hard part is the resolver itself: what do you do when the downstream API offers no query-by-reference and no idempotency key? At that point is the honest answer to never auto-retry that call at all?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

That's the genuinely hard case, and the honest answer has three layers, ending exactly where you guessed.

  1. Look for an accidental key before you give up. Even with no explicit idempotency support, the downstream sometimes enforces a natural uniqueness constraint — a unique email, a unique external reference, a "you already have an open order for this" rejection. If a duplicate would bounce off a constraint, you effectively have idempotency, just undocumented. Worth checking before assuming blind.
  2. If it's reversible, retry and reconcile. Accept that you might double it, then detect-and-compensate later against settled data. The duplicate is survivable because the undo exists.
  3. If it's irreversible AND blind — no key, no lookup, no natural constraint — then yes: never auto-retry. You've named it correctly. An automated retry there is a coin-flip on a double side effect you can neither detect nor undo, and no amount of cleverness changes that. The honest system escalates to a human on the first UNKNOWN and refuses to guess.

The uncomfortable truth for that third case is that the only safe policy is at-most-once: do it once, and if the once is uncertain, stop and ask a person. We default to at-least-once everywhere because retries are cheap — but at-least-once is only safe when you can dedup or undo. Strip both away and the correct delivery guarantee inverts. A tool that can't say "unknown" and can't be queried and can't be reversed isn't one you automate retries for; it's one where the retry decision belongs to a human every time. That's not a cop-out, it's the accurate cost of the tool's contract.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 'accidental key' check is the part I'd missed — a natural uniqueness constraint is idempotency, just undocumented, and it's cheap to look for before assuming the worst. And your third case lands cleanly: if it's irreversible and blind, at-most-once plus a human on the first UNKNOWN is the only honest policy. Have you found teams actually hold that line at volume, or does the pressure to auto-retry creep back in?

Collapse
 
iictp profile image
IICTP •

The part I keep coming back to is section 12 — returning UNKNOWN instead of a flat "payment failed" is the piece that stops the model from inventing a second operation. The operation ledger plus the request fingerprint binding the key to the exact parameters is what makes the retry safe rather than just hopeful. One thing I'd love to see in a follow-up: when the provider has no lookup API at all, do you hold the operation for manual reconciliation, or is there a time-boxed auto-retry with a human escalation?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

Neither as a blanket rule — it depends on whether the operation is safely repeatable, and when there's no lookup API you usually have to assume it isn't. With no idempotency support and no lookup, you cannot distinguish "it committed" from "it never landed," so an automatic retry is a coin-flip on a double charge. So: no auto-retry of the side effect. What you can time-box is the resolution attempt — retry any cheap read path the provider does expose (webhook replay, a statement endpoint, a settlement file), and only if none exists does it go to a human. The honest design is hold in UNKNOWN, auto-attempt every non-destructive way to learn the outcome up to a deadline you can state, then escalate to a human with the full context (operation id, amount, timestamp, what you tried) rather than guessing. Time-boxed auto-retry is for learning what happened; it is never for re-doing the thing when you can't tell whether it already happened. Good candidate for the follow-up — the no-lookup provider is the worst case and worth its own section.

Collapse
 
iictp profile image
IICTP •

That distinction between retrying the side effect and retrying the resolution attempt is the clearest framing of it I've seen — it keeps UNKNOWN honest instead of turning it into a quiet guess. The part I'd love to see in the follow-up is what the no-lookup provider case actually looks like in practice: when there's genuinely no read path, does the escalation carry the full context — operation id, amount, timestamp, everything tried — so the human isn't starting from zero?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Yes — the escalation has to carry the full dossier, or you've just moved the guessing from the machine to a tired human at 2am. The rule I'd use: an operation can only enter "needs human" with a complete, self-contained record attached — operation id, tenant/user, amount and currency, the timestamps (formed, dispatched, last reconcile attempt), the provider and any reference you sent, every resolution path you tried and what each returned, and the current balance of evidence ("dispatched, no confirmation, 4 read attempts all empty"). The human's job is to make the one judgment the system genuinely can't — go look at the bank statement, call the provider — not to reconstruct what happened from logs. Think of it as the state machine handing off a case file, not an alert. In practice that means the escalation payload is built from the ledger row, not assembled ad hoc, so it's impossible to escalate an operation without its context. And the human's resolution writes back a terminal state on that same id, so the decision is durable and the operation can never re-enter the queue. That no-lookup section is writing itself in this thread — it's going in the follow-up.

Collapse
 
jsbry profile image
Joshua Bryan •

Not the author, but for MCP tools "no lookup API" is the common case, not the edge. We read the write paths of 100 popular packaged MCP servers: 15 accept a caller-supplied idempotency key, and 7 can say "unknown" in a form a program reads. Of the 42 that wrap a web API, 39 have no machine-readable "unknown" for a request that got no answer. So what Rudratosh describes is the realistic default: time-box the read, and leave the retry to a person.
https://isokron.ai/research/mcp-write-results-100-servers?utm_source=devto&utm_medium=social&utm_campaign=mws-launch&utm_content=comment

Collapse
 
iictp profile image
IICTP •

Thanks for running this — the 7-of-100 figure is the number that stuck with me. It makes UNKNOWN less of a defensive corner case and more of the default posture for MCP tools, which is a sobering read. One question: of those 7, how many expose unknown as a distinct status or enum, versus a structured error the caller still has to interpret?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Joshua answered the data below (distinct value only — a structured error the caller still has to interpret didn't count; 3 of the 7 are wrappers, 4 aren't), so I'll add the part that matters for the piece rather than repeat the numbers.

The split you're pointing at — "distinct status" vs "structured error you still have to interpret" — is the whole ballgame, and it's why the real number is even smaller than 7 for the case that counts. A distinct outcome_unknown value is a machine-readable fact: the caller branches on it. A structured 5xx is still a judgment call the caller has to make, and worse, it only exists when an answer came back at all. The failure that actually causes duplicates — request went out, connection reset, nothing returned — has no status to carry, structured or otherwise. That case is unrepresentable in 39 of the 42 wrappers.

So the honest reading of your own question: of the handful that can say "unknown" as a value, almost none can say it for the no-answer case, because by definition there's no response to put the value in. Which is exactly why UNKNOWN has to be a state the caller constructs from "the call came back empty," not a field it reads off the tool. The tool contract can't help you with the one failure you most need help with.

Thread Thread
 
iictp profile image
IICTP •

That reframing really helps — UNKNOWN as a state the caller constructs from an empty return, rather than a field the tool reports, makes the 39-of-42 figure land differently. It also explains why the duplicate-causing failure is the one the contract can't describe at all. Do you think a wrapper could ever surface that no-answer case usefully, or is the only honest move for the caller to treat every empty return as unknown by default?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

A wrapper can't manufacture a status for a response that never arrived — but it can do two things that turn "unknown forever" into "unknown until I check," which is most of the value:

  1. Distinguish "never left" from "sent, no answer." A connection refused before the request went out is safe to retry outright — nothing happened. A reset or timeout after the bytes left is true unknown. Most wrappers collapse both into one "failed," and that collapse is what forces the pessimistic default. A wrapper that honestly reported "I definitely did not send this" would let the caller retry a real chunk of cases without any lookup at all.
  2. Expose a resolver. Accept a caller-supplied idempotency key, or offer a read-back-by-reference endpoint. Then the caller's UNKNOWN isn't terminal — it's a state it can discharge by asking "did operation X land?" The wrapper still can't tell you in the failing response, but it gives you a second door to walk through afterward.

So the honest answer to your either/or is: treat every empty return as unknown by default — that part isn't negotiable, because the no-answer case genuinely has no in-band signal — but a well-built wrapper changes what you can do with that unknown. The default stays pessimistic; the wrapper's job isn't to pretend the unknown away, it's to give you a cheap way to resolve it. The 39-of-42 problem isn't just "can't say unknown," it's "no resolver either," so the caller is stuck holding the uncertainty with nowhere to discharge it.

Thread Thread
 
iictp profile image
IICTP •

That two-part split is the clearest version of the rule yet — 'never left' versus 'sent, no answer' gives the caller a fact the transport can actually prove, and the resolver turns UNKNOWN from a dead end into a state the caller can discharge. The 'no resolver either' line is what sticks with me: of the 39 that can't say unknown, the missing second door matters as much as the missing field. Do you see that resolver living in the wrapper itself, or in a gateway that sits between the agent and the business system?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Gateway, mostly — and for the same reason the operation ID can't live in the model. A per-wrapper resolver only works when that server owns the record it would resolve against. But the 39-of-42 case is wrappers fronting a third-party API they don't control, and you can't build an authoritative "did this land?" lookup on top of a system you can only ask nicely. So the resolver that matters lives in the gateway/runtime between the agent and the business systems: it holds the operation ledger, mints the stable IDs at plan time, and owns the reconcile loop across all the tools, because it's the only layer that persists across agent runs and can see the business reference.

The division of labor: the wrapper's job is to accept a caller-supplied idempotency key and, ideally, to honestly report "never left" vs "sent, no answer." The gateway's job is to construct and discharge UNKNOWN — it's where the ledger outlives the conversation and where the reference-join reconciliation runs. Pushing the resolver into each wrapper would mean re-solving durability and cross-run identity N times, once per tool, and most tool authors won't. Centralize the hard part, standardize the thin part at the wrapper boundary.

Thread Thread
 
iictp profile image
IICTP •

That makes a lot of sense — the gateway is the only layer that can actually own the ledger across runs, and the wrapper honestly reporting 'never left' versus 'sent, no answer' is a clean contract. The point about not re-solving durability N times, once per tool, really lands. Do you find most gateways in practice keep that ledger durable enough to reconcile days later, or is that the part teams tend to underinvest in?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Underinvested, and it's usually the first thing that's underinvested — because it's invisible until the exact moment you need it. A huge number of "operation ledgers" in practice are in-memory, or scoped to the conversation/process, which is precisely the failure Adam Lewis named in another thread here: they don't survive the restart, so the reconciliation that depends on them is worthless at exactly the moment the process you were counting on is the thing you've lost.

A ledger that can actually reconcile something from last Tuesday needs three unglamorous properties: durable storage (not the agent's memory), keyed by a business reference the task still owns (not the tool-call ID the model invented), with retention that outlives the provider's settlement window with margin. That's maybe 80% of the real work and it's the first thing cut under deadline, because the demo passes without it and the gap only shows up the day you try to reconcile a three-day-old UNKNOWN and discover there's nothing to reconcile against. So: the gateway is the right place for the ledger, but "the gateway has a ledger" and "the gateway has a ledger you can trust days later" are very different maturity levels, and most teams are at the first without realizing they're not at the second.

Thread Thread
 
iictp profile image
IICTP •

That three-property checklist is the clearest version of this I've seen — especially keying by a business reference the task still owns rather than the tool-call ID the model invented. That distinction alone would have saved a few painful reconciliations I've watched teams attempt from logs. The "demo passes without it" line really lands too: durability only gets funded after the first three-day-old UNKNOWN with nothing to reconcile against. When you scope retention in practice, do you anchor it to the provider's settlement window plus a fixed margin, or does the business side usually dictate something longer?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

This is the data the argument wanted — thank you for running it. If only 7 of 100 can emit a machine-readable "unknown," and 39 of 42 web-API wrappers have no way to signal "I got no answer," then "no lookup → resolve by a human" isn't the pessimistic edge case, it's the modal path for MCP tools today. That reframes the piece: the UNKNOWN state isn't a defensive corner you build for rare providers, it's the default posture for most of the ecosystem — and the auto-reconcile path is the exception you earn only when you're talking to the 15% that take an idempotency key or the 7% that expose a readable unknown.

The MCP-specific sharp edge: the protocol hands the model a tool result, but a dropped response has no distinguished representation, so the layer under the model has to manufacture the UNKNOWN itself — there's nothing in the tool contract to read it from. That's the argument for idempotency and outcome-state living in the host/runtime rather than being wished into each server, because 93% of them won't give it to you.

One methodology question, since the dataset's genuinely useful: when you say 7 can say "unknown in a form a program reads," did you count only a distinct status/enum, or also servers that surface the underlying 5xx/timeout in a structured error the caller can branch on? The gap between "no field for unknown" and "unknown leaks through as an unstructured string" changes how much a host can safely automate before it has to escalate.

Thread Thread
 
iictp profile image
IICTP •

That distinction is the crux for me too — a structured 5xx still leaves the model guessing, so the count of servers that can say 'unknown' in a way a program can branch on may be even smaller than 7. The 39-of-42 figure for web-API wrappers is the one that worries me most, since that's where most MCP tools actually live.

Thread Thread
 
jsbry profile image
Joshua Bryan •

Only a distinct value. The 7 are results that carry something that itself means "unknown / may have applied" (outcome_unknown, indeterminate, paid: null with state unknown, that kind of thing). A structured 5xx or timeout code didn't count on its own, and a timeout name buried in the message text certainly didn't. We drew that line on purpose after the first coding pass, because counting "passes a 504 through" made the number mostly a measure of prose.

The looser cuts from the same data, since you asked where the gap is:

  • 20 of 100 say "unknown / may still be running / may have applied" in some form, prose included.
  • Of the 42 web-API wrappers, 15 keep the upstream status as a structured field, 23 only mention it in prose, and 4 drop it.
  • Of the 7, 3 are wrappers and 4 aren't, for whoever asked further down.

The catch is that a kept 5xx only helps when an answer came back. A timeout or reset after the request went out has no status to keep, and 39 of those 42 wrappers report that case as a plain failure. So a host can branch on a structured 504, but the true no-answer case still has to be manufactured above the server, which is your point exactly.

The codebook (C2 vs C2p for unknown, W1 for status pass-through, L4 for the collapse) is published with the per-server data: https://isokron.ai/research/mcp-write-results-100-servers?utm_source=devto&utm_medium=social&utm_campaign=mws-launch&utm_content=comment

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Right — and Joshua's follow-up just confirmed the pessimistic read: of his 7, the count is distinct-value-only; a structured 5xx that the caller still has to interpret didn't make the cut. So the honest number of tools that can hand a program a branchable "unknown" is 7, and the number that can do it for the one case that matters — no answer came back at all — is smaller still, because a timeout has no status to return. That's the 39-of-42 figure, and you're right that it's the scary one: web-API wrappers are where most MCP tools actually live, so the modal tool literally cannot tell you it doesn't know. Which is the whole argument for building UNKNOWN above the server — you're not reading it from the tool, you're manufacturing it from the fact that the call came back empty.

Thread Thread
 
iictp profile image
IICTP •

The 39-of-42 figure really is the scary one — the modal MCP tool being a web-API wrapper that can't say 'I don't know' reframes the whole problem. And the line about manufacturing UNKNOWN from the empty call, rather than reading it from the tool, is the clearest statement of the fix I've seen. It makes me wonder how much of the retry problem disappears if the layer above the server treats 'no answer' as a first-class outcome by default — is that something you've seen teams actually wire into their agent loops, or is it still mostly aspirational?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Honestly? Aspirational in adoption, not in principle. The pattern is well understood — nobody who's built payments infra argues with it — but it's wired into real agent loops almost exclusively by teams that already got burned, and it's retrofitted after the incident rather than designed in. The default path in most agent frameworks today is still timeout → exception → retry, which collapses "no answer" straight into "error" and then does the single most dangerous thing you can do with an error: try again.

So the amount of the retry problem that disappears if you treat "no answer" as a first-class outcome is large — most duplicate side effects I've seen trace back to exactly that collapse — but the number of teams who've done it proactively is small. It tracks cost: it's wired in wherever a dropped response already showed up on a bill, and absent everywhere it hasn't yet. The frameworks are starting to expose the distinction, but the defaults still punish you, so right now it's a thing you have to choose, and most people don't choose it until the ledger forces them to. Well-understood, under-adopted, and almost always reactive. Which is a depressing answer, but the honest one.

Thread Thread
 
iictp profile image
IICTP •

Well-understood, under-adopted, and almost always reactive is a depressingly accurate summary — and the timeout → exception → retry default is the part that worries me most, since it makes the dangerous choice the path of least resistance. "It tracks cost" is a sharp observation too: nobody funds the UNKNOWN branch until a dropped response shows up on a bill. You mentioned frameworks are starting to expose the distinction — is there one you'd point to as furthest along, or is the gap still mostly that the safe path has to be deliberately chosen rather than being the default?

Thread Thread
 
iictp profile image
IICTP •

Thank you for spelling out the coding line — the gap between 20 of 100 saying "unknown" in some form and only 7 carrying a distinct, branchable value is the most useful number in the whole thread. The 15/23/4 split on the wrappers is just as telling: most of them have the information and still only mention it in prose. And your catch about timeouts is the crux for me — a kept 5xx only helps when an answer came back, so the true no-answer case has to be manufactured above the server regardless. Of the 3 wrappers among the 7, is there a common pattern in how they surface the unknown, or did each one invent its own shape?

Collapse
 
iictp profile image
IICTP •

Thanks for sharing those numbers, Joshua — the 7 out of 100 figure really sharpens the point. If only a handful of tools can say 'unknown' in a machine-readable way, the UNKNOWN state has to be built by the caller, not read from the tool. Were the 7 that could do it mostly the native MCP servers, or did some of the web-API wrappers manage it too?

Collapse
 
aheadkit profile image
AheadKit •

Your UNKNOWN state section is the one I keep coming back to. I ran a version of this on my own machine this morning, not with a payment but with a webhook: one delivery key, a receiver that sleeps 2.5 seconds before answering 200, and a sweeper that ticks every second and resends on silence. The first acknowledgement landed at 2.5 seconds — half a second after the third copy had already gone out. Three deliveries logged for one order, and the receiver was never down, never erroring.

Your line is exactly right: the timeout told me what the sweeper observed, not what the receiver had committed. And the fix that actually worked was not a longer timeout — a 4-second wait already covered the 2.5-second receiver and changed nothing about the second tick. It was one claim row per delivery key, claimed atomically, so the second and third ticks saw in_flight and stood down. Same 1-second sweep, one delivery instead of three.

Collapse
 
rudratosh profile image
Rudratosh Shastri •

That 15-minute sweep is the reconciler, and "dumb" is exactly what it should be — a periodic query for committed-but-unfinished beats any trigger, because it doesn't depend on the event that went missing being delivered. On feeding it back: feed it back, don't fix it silently. The agent moved on under a false belief ("the step failed"), so a quiet side-fix leaves its world-model wrong and the next plan reasons from "no key assigned" when a key now exists. Emit the late discovery as a state transition the agent can observe (UNKNOWN→SUCCEEDED on that operation id). Fix quietly only when the operation is terminal and nothing downstream depends on the agent knowing.

Collapse
 
aheadkit profile image
AheadKit •

That UNKNOWN→SUCCEEDED line is the part I'm stealing. My claim-row run stopped the resend, but it never told the agent the first write had actually landed — so the agent keeps reasoning from 'no key assigned' while a key exists, exactly your point. How do you surface that transition in practice — a tool result the agent reads, or something in the log it picks up on the next turn?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

A tool result the agent reads — don't rely on it noticing a log line. The reconciler shouldn't just flip the row; it should post the outcome back onto the agent's input channel as an observation keyed to the operation id, so the next turn sees "op X: SUCCEEDED (key assigned)" the same way it sees any other tool output. A log the agent happens to re-read is best-effort and fails exactly when the plan has already moved on. The durable pattern is: reconciler writes the state transition, and an event on that transition enqueues a message into the agent's context for its next step. State for correctness, event for attention — the row is the truth, the injected observation is what makes the agent act on it. If the agent is long-gone (terminated plan), the same transition routes to whoever owns the fallback, but for a live agent it has to arrive as something it reads, not something it might find.

Thread Thread
 
aheadkit profile image
AheadKit •

State for correctness, event for attention — cleanest way anyone's put it. The version I ran was the pull side, the planner read the claim row before planning, so the transition showed up as something it actually read: 1 delivery, 0 duplicates. The push side I haven't tested, whether an injected observation lands any differently than a read at the plan boundary. Going to run that one.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

"1 delivery, 0 duplicates" from the pull side is the result that matters — the planner reading the claim row before planning is the correctness guarantee, and it doesn't depend on anything being delivered to it. That's the one to trust by default. The push side (injected observation) is strictly an attention optimization on top: it lowers latency between the transition and the agent acting on it, but it can be lost, duplicated, or arrive mid-plan, so it can never be the thing correctness rests on. Worth running, but hold the pull read as the floor — if the injected event ever disagrees with the row, the row wins. The failure mode to watch on the push side is an observation landing after the planner already read stale state that turn: you want the injected event to invalidate-and-reread, not to be treated as fresher truth than the row it's pointing at. Keen to hear how it lands at the plan boundary.

Thread Thread
 
aheadkit profile image
AheadKit •

Row wins on disagreement is the rule I'm keeping — the push side can get lost or duplicated, the row can't. And the stale-read race is the exact failure to watch, an observation landing after the planner already read that turn. That's what the push-side run has to catch, will see how it lands at the plan boundary.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

That's the invariant locked in — row wins, push is best-effort. One concrete thing to assert in the push-side run, since it's the race you named: the injected observation should carry the operation's version/state-token the reconciler wrote, and the planner compares it against what it last read that turn. If the token is newer, invalidate and re-read the row; if it's equal or older, ignore it as a duplicate or straggler. That way a late observation can only ever trigger a re-read, never overwrite fresher state — "row wins" enforced mechanically instead of by timing luck. Keen to see where it lands at the plan boundary; that handoff is the whole ballgame.

Thread Thread
 
aheadkit profile image
AheadKit •

That's the bit that turns row-wins from a rule I say into something the run can actually assert — the token in the observation, compared against what the planner last read, newer means re-read, older means ignore. Without it the push side is just hoping the timing works out. Going to carry that check into the push run, it's the cleanest version of the race you named. One thing I'm not sure about — when the token lands mid-step, does the planner finish the step it's on and re-read before the next one, or drop it straight away?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Finish the step, then re-read before the next — don't interrupt mid-step. The step already in flight has its own operation id and its own idempotency guarantee, so letting it complete is safe; yanking it halfway just creates a second partial-execution problem to reconcile. The token's job is to invalidate the planner's cached read, not the work in progress. So the rule is: mark the cache dirty the moment the token lands, let the current step run to its durable outcome, then force a fresh row read at the next plan boundary before deciding the next action. The one exception is a token that invalidates the step in flight itself (its own operation got reconciled to a terminal state underneath it) — there the step should finish its write attempt, which will no-op against the now-committed row, and the re-read picks up the truth. Boundary re-reads, never mid-step preemption; correctness lives in the row the step writes to, so a stale plan can't cause a double effect, only a wasted cycle.

Thread Thread
 
aheadkit profile image
AheadKit •

Finish the step then re-read — that's the clean answer to the question I left open. The run will assert exactly that: token lands mid-step, step completes, planner re-reads before the next one. Have you actually wired a dirty-mark into a planner, or is this still the shape you'd want it to take?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Honest answer: it's the shape I'd want, not a line I can point you to in a shipped planner — I'm describing the invariant the way I'd build it, and your run is about to be better evidence than anything I could claim. The dirty-mark is just the minimal honest version of "row wins": the planner caches a read for the turn, and the token flips a flag that says "your cache is stale, re-read at the next boundary." Whether that's a boolean beside the cached row or a version counter you compare is an implementation detail; the invariant is that a planner never acts on a read it's been told is stale. If your push run asserts token-mid-step → step completes → re-read-before-next and it holds, you've turned the shape into a tested property — which is the part I was hand-waving. Post what it does at the boundary; that result is the thing worth having.

Thread Thread
 
aheadkit profile image
AheadKit •

Appreciate the honesty — "the shape I'd want" is worth more than a line you can't point to. That's exactly why the run matters: if the dirty-mark holds up under a crash, it's evidence, not a claim. One thing I'm still turning over: once the token flips the flag and the planner re-reads, does the flag clear, or does it stay dirty till the turn ends?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Clears on the re-read — the flag is an edge, not a level. Its whole meaning is "there's a newer row than the one you cached," so once the planner re-reads and refreshes, that's satisfied and it resets to clean. Leaving it dirty till end-of-turn forces a redundant re-read at every later boundary for no new information — and worse, it masks a second token that lands after the re-read, which you want to flip the flag again and earn its own re-read. So: token sets dirty, boundary re-read consumes-and-clears, a later token sets it again. One subtlety for your run: set-and-clear has to be atomic against incoming tokens. Clear by compare-and-swap on the token value you actually read, not an unconditional reset — otherwise a token arriving in the window between "read the row" and "clear the flag" gets lost. CAS means a newer token that slipped in mid-read leaves the flag dirty and correctly earns another pass.

Thread Thread
 
aheadkit profile image
AheadKit •

Edge, not level — that's the clean answer to the question I left open. The CAS detail is the part the run has to get right, an unconditional clear losing a token that lands mid-read is exactly the kind of thing that only shows up under a crash. Have you ever seen that window actually bite in a real planner, or is it the kind of race you defend against because you know it's there?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Both, honestly, but the one time it bit was the one that converted me. A planner resumed a step, the step had already completed and written its result, and a stale worker from the previous attempt did an unconditional status = IN_PROGRESS clear a few milliseconds after the resume read SUCCEEDED. Net effect: the operation ran twice because the second run saw a state the first run had already moved past, and the clear had erased the evidence. No stack trace, no error — just a duplicate charge and two rows that looked fine individually.

That's why it's compare-and-swap, not an unconditional write: the clear has to name the token it believes it's clearing, and lose if that token already moved. After that incident I stopped treating it as "a race I defend against because the textbook says so" and started treating it as "the specific thing that cost us a refund." Most of the time it never fires. The cost of the CAS is one extra predicate; the cost of skipping it is a duplicate you find in the ledger a week later. Cheap insurance against an expensive, silent failure.

Thread Thread
 
aheadkit profile image
AheadKit •

That duplicate charge with no error is the part that stays with me — two rows that look fine on their own, and the only place it shows up is the ledger a week later. The run I did this morning had the same shape: the naive reset lost the token exactly the way your stale worker did, and the CAS clear refused it and kept token 3. One extra predicate really is cheap insurance. Did the refund come from a customer noticing, or did you catch it in a reconciliation pass first?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Reconciliation pass, thankfully — not a customer. And the only reason the pass caught it is that we'd already started treating UNKNOWN as a real state, so there was a daily job comparing our operation records against the provider's settled transactions by business reference. The duplicate showed up as two of our operations pointing at what the provider considered related-but-distinct charges, which is the kind of mismatch a human would never eyeball but a reference-join surfaces immediately. If we'd been on the "timeout = FAILED" model, there'd have been no record to reconcile against — the second charge would've just looked like a normal successful payment, and we'd have found it when the customer did.

That's the part I'd underline from your run matching mine: the CAS clear keeping token 3 prevents the duplicate going forward, but the reconciliation pass is what catches the ones that already slipped through before you hardened it. You want both — the predicate stops new duplicates, the daily join finds the historical ones and the residual races the predicate doesn't cover. Nice that your morning run reproduced it cleanly; that race is notoriously shy in a test harness, so catching it deliberately is worth more than it looks.

Thread Thread
 
aheadkit profile image
AheadKit •

Reconciliation pass, not a customer — that's the answer I was hoping for. The daily join by business reference is the bit I'm stealing, it's the only way the historical ones surface. And you're right, both: the predicate stops the new ones, the pass finds the ones that already slipped through. Did the pass run against live provider data, or a settled-transactions export?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Against the provider's settled records, queried by business reference through their API — not a raw live feed. That choice matters, and it's the subtle part: if you reconcile against in-flight provider state, you've just moved the UNKNOWN problem into the reconciler, because now the thing you're comparing against is itself mid-flight and might change after you read it. Reconciling against finalized/settled data means you're comparing your ledger to something that has stopped moving, so a mismatch is real rather than a race.

The cost is latency: settlement has a lag (hours to a day depending on the provider), so the daily pass is deliberately behind live. We accepted that — the pass isn't trying to catch duplicates in real time, that's the CAS predicate's job at write time. The pass is the slow, authoritative backstop that catches what slipped through, and for that role you want authoritative-but-late over fresh-but-uncertain. Export vs API query is a wash as long as the records are settled; we used the API because query-by-reference let us pull just the operations we had open UNKNOWNs for, instead of diffing the whole day.

Thread Thread
 
aheadkit profile image
AheadKit •

Authoritative-but-late over fresh-but-uncertain — that's the line I'm keeping. The bit I hadn't thought through is the lag: the pass is deliberately a day behind, and that's fine because the CAS predicate does the real-time job. How do you decide a record is actually settled — a fixed window after the fact, or does the provider mark it final somewhere you can read?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Provider-marked where you can get it, fixed-window only as a fallback — and never a fixed window alone, because that's just a guess wearing a timestamp.

The preference order:

  1. An explicit terminal status from the provider. Payments usually have one — settled, captured, cleared, a settlement/payout ID that only exists once the money actually moved. If the record carries a state the provider considers final, you reconcile only against records in that state and ignore everything still pending/processing. That's authoritative: the provider is telling you it's done moving.
  2. A terminal status plus a "no further webhooks expected" signal where the provider gives one (some emit a settlement batch close, or a report that's explicitly end-of-day final). Reconcile against the report, not the live objects.
  3. Fixed window only when there's no readable final marker at all — and then you pad it well past the provider's documented worst-case settlement time, treat the result as provisional, and re-check anything that was still non-final at window close on the next pass. A time-based cutoff can misfire exactly when the provider is slow, which is often correlated with the incident you're reconciling, so you never let it be the sole arbiter for an irreversible operation.

The trap is reconciling against a record that still has a mutable status — you compare, it matches, and then it flips an hour later. So the rule is: a record is "settled" when the provider says it is, by a field or an ID you can read; the clock is what you fall back to when the provider won't tell you, and even then it only downgrades the record to "probably final, re-check once more," never "done." If the provider marks it, trust the mark over the clock every time.

Thread Thread
 
aheadkit profile image
AheadKit •

Provider-marked over the clock, every time — that's the line. The bit I'll steal is reconciling against the report, not the live objects, it closes the same hole your settled-records point did. Do you treat a settlement ID as terminal on its own, or do you still want the status field to agree?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Both have to agree — the ID alone isn't terminal, and this one bit me. Some providers assign the settlement/payout ID at batch creation, before the batch actually clears, so the ID can exist while the money is still in flight. If I'd treated "ID present" as terminal, I'd have reconciled against records that hadn't settled yet and matched things that could still flip.

So the rule is a conjunction: terminal = a settlement/payout ID is present AND the status field is in the final set (settled/paid/cleared, not pending/processing). The ID tells you "this got as far as being batched"; the status tells you "the batch is done." You need the second to trust the first. The one exception is a provider that only ever assigns the ID at settlement — for those the ID genuinely is terminal on its own — but I default to requiring both because you can't always tell from the docs which kind you're dealing with, and the conjunction is never wrong, just occasionally redundant. Redundant-but-safe beats clever-but-early here.

Thread Thread
 
aheadkit profile image
AheadKit •

Batch creation — that's the timing I hadn't thought of. An ID that exists before the money moves is exactly the kind of 'terminal' that flips an hour later, so the conjunction it is: ID plus a status in the final set. Redundant-but-safe beats clever-but-early, I'm keeping that line. When you default to both, do you ever find out which kind a provider actually is, or does the conjunction just quietly cover it and you never bother checking?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Mostly it quietly covers it and I never find out — which is the point of defaulting to the safe conjunction: it removes the need to know. When I do find out, it's almost always because it bit: a reconciliation mismatch where the ID matched but the record wasn't final tells me, retroactively, "this provider assigns the ID early." At that point I write a one-line note in the integration ("ID assigned at batch creation, not terminal alone") so the next person doesn't re-learn it. But I don't go probing proactively — the conjunction is cheap enough that the cost of investigating every provider's exact timing exceeds the benefit until something forces the question. Safe-by-default means you get to be incurious about the ones that never misbehave, and spend your attention only on the one that threw a mismatch. Incuriosity you can afford is a feature.

Collapse
 
galadev profile image
Gala •

A small, non-payment version of this bit me a few days ago. The script I use to publish posts here had a single "publish" path that was supposed to create or update. A run I meant as an update created a second article instead, and the API was perfectly happy to do it, because nothing in the request said "this is the same post". I caught it and parked the copy as a draft. The rule now is the one you describe: once an article exists, every edit is a PUT to its ID, never another create.

Your UNKNOWN state is the part I hadn't named. For something like a publish, is querying by a client-side key (slug or title) before retrying good enough, or does that just move the race somewhere else?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

It moves the race — a query-then-create is two steps, and two concurrent runs can both read "not found" before either writes, so you get the exact duplicate back. The client-side key is the right idea; it just has to be enforced by the store, not checked by the caller. Put a unique constraint on the slug (or a dedicated idempotency column) so the database rejects the second insert atomically, and treat "create, catch unique-violation, fall through to update" as the single operation. That's the difference between a lookup (advisory, racy) and a constraint (authoritative, atomic). Your PUT-to-ID rule is the same principle once the row exists; the unique key is how you extend it to the create that establishes the ID in the first place. And it's why the operation identity has to be minted before the first attempt — the slug is your operation id here, so the create and every retry collapse onto one row.

Collapse
 
galadev profile image
Gala •

That lines up. Enforcement has to live where the identity is born, not in a lookup done later. Since dev.to is the store and I can't add a constraint there, the practical version is my own local slug-to-ID record written before the first create, plus canonical_url as the key, and refusing to create if it exists. The remaining hole is the first write itself: if the create succeeds but my local write fails, a retry still duplicates. Is that the kind of gap you'd close with an outbox, or accept and reconcile later?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Reconcile — an outbox doesn't help you here, and it's worth seeing why. The outbox solves the dual-write problem when both writes are yours and can share one transaction. Your two writes straddle a boundary you don't own: the create lands in dev.to, the record lands in your local store, and no transaction spans both, so there's always a window where one succeeded and the other didn't. You can't make that atomic; you can only make it recoverable. So before any create, query dev.to for an existing article by the canonical key (slug/canonical_url) — if the platform will dedup on your behalf, great, but assume it won't — and only create if that comes back empty. That's the reconcile-first step: your retry becomes "does this already exist upstream?" rather than "create again." The local record is an optimization (fast path, avoids the query), never the source of truth; the source of truth is dev.to, read by your canonical key. The order that survives a crash is: reserve the key locally → create upstream → record the returned ID; and on any retry, if the local record is missing, recover it by querying upstream for the key before deciding to create. Put the durable identity in a field dev.to stores and you can read back (canonical_url), so even a total loss of your local store is recoverable from the platform itself.

Thread Thread
 
galadev profile image
Gala •

"You can't make it atomic, only recoverable" is the sentence that finally made it click for me, thank you 🙌 My publish script can already list my own articles, so reconcile-first is a cheap call before any create, and the local record goes back to being just a fast path.

The catch on my side: most of my posts have no canonical original, so canonical_url isn't a natural key for them. I'd probably mint a stable id in the front matter instead and look for it upstream.

Have you ever had to do this against an API that gives you nothing to read back at all? I'm curious what's left once even reconcile is off the table 👀

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Front-matter minted id is the right call — a stable key you control, embedded in the artifact, searchable upstream. canonical_url only works when there's a real original; your own id works always.

On the hard question — an API with no read-back at all, where even reconcile is off the table — what's left is narrow and honest: you fail closed, and you make the uncertainty cheap to resolve by a human, because you've lost the ability to resolve it by machine. Concretely: (1) write your intent durably before the call and the outcome after, so a crash still leaves a "dispatched, unconfirmed" row even though you can't query the truth; (2) make the side effect naturally idempotent on the receiver's own terms if you possibly can — a deterministic resource name the API rejects as a duplicate is a read-back in disguise, because the 409 is the answer; (3) if even that's impossible, accept you can't guarantee exactly-once, pick at-most-once or at-least-once explicitly (skip vs. dupe — whichever failure is survivable), and tell the operator which you chose. What you never do is pretend: no read path means no auto-retry of the side effect, full stop — it goes to a human with the unconfirmed row.

The duplicate-rejecting create is the trick that's saved me most — turning the write itself into its own idempotency check when you can't read state back. For publishing, that's exactly your front-matter id as the slug: the platform rejecting a dup slug is your missing read API. 👀

Thread Thread
 
galadev profile image
Gala •

"Fail closed and make the uncertainty cheap for a human" is such a calm way to put it, thank you 🙌 Writing the intent before the call is the bit I'd been missing. Totally different question, since you clearly think in systems: have you ever grown a side project's traffic from zero? Curious what actually worked for you first 👀

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Ha — different lane entirely, but happy to answer. Honestly the only thing that's worked for me is writing the thing I actually wanted to read, specific enough that the right few people feel seen — this very thread is the example: one technical post, every comment answered properly, and that did more than any "growth tactic" I ever tried. From zero it's mostly: go where the audience already argues about the problem, add real signal, don't pitch. Boring, slow, compounding. No clever hack — the honest version is the post has to be good and you have to show up in the comments like a person. That's it. 👀

Thread Thread
 
galadev profile image
Gala •

Okay, you just described exactly how this thread happened, so I can’t even argue 😄 Honest question from someone still in the “talking to five lovely people” phase here: back when you started on dev.to, did commenting on other people’s posts bring you more readers than your own posts? Because if the answer is yes, I’m about to become very chatty!!

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Honest answer: they do different jobs, and neither works alone. A good post is the thing worth finding; comments are how people find it and decide you're worth reading. Early on, a genuine comment on someone else's post outperformed a lot of my own posts for discovery — but only because there was something good to click through to. Comment with nothing behind it and the click lands on an empty profile.

So the thing that actually compounded wasn't "comment more," it was "comment on posts next to the one I'd want them to read." Someone deep in a thread about idempotency is the exact person who wants my idempotency piece — the relevance does the work, not the volume. Chatty is good; chatty-and-relevant is the cheat code. And the one rule that matters: add a real point every time, never "nice post 👍". The moment it reads as a tactic, it stops working. You're already doing it right in this thread — that's not flattery, it's just true. 😄

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Honest answer: they do different jobs, and neither works alone. A good post is the thing worth finding; comments are how people find it and decide you're worth reading. Early on, a genuine comment on someone else's post outperformed a lot of my own posts for discovery — but only because there was something good to click through to. Comment with nothing behind it and the click lands on an empty profile.

So the thing that actually compounded wasn't "comment more," it was "comment on posts next to the one I'd want them to read." Someone deep in a thread about idempotency is the exact person who wants my idempotency piece — the relevance does the work, not the volume. Chatty is good; chatty-and-relevant is the cheat code. And the one rule that matters: add a real point every time, never "nice post 👍". The moment it reads as a tactic, it stops working. You're already doing it right in this thread — that's not flattery, it's just true. 😄

Thread Thread
 
galadev profile image
Gala •

That's the line I'm stealing: comment next to the post you want them to read. I've been treating comments as a numbers game, but you're right that relevance does the work. Next time I comment on an idempotency thread, I'll make sure the post I want them to find is the one waiting behind my profile. Curious, how do you decide which posts to comment on in the first place?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Two filters, and I apply them in order. First: do I have something to add, not just react to? If my comment would be "great post 👍" I skip it — a comment only works when it advances the thread, so if I can't contribute a point, a correction, or a real question, there's nothing there for anyone to click through from. Second: is the audience of this post the audience I'd want for mine? An idempotency thread is full of people who'd want my idempotency piece; a "my first 100 days of coding" post isn't, however nice it is. Relevance of the room, then whether I have something worth saying in it.

The practical version: I search the tags I actually write in (distributedsystems, aiagents, backend), sort by recent, and read the ones where the title names a real problem rather than a tutorial. Problem-posts attract people arguing about the problem — that's where a substantive comment lands. And I only comment when I'd have written the same thing with no profile attached; the moment it's a tactic, people feel it. The reach is a side effect of being genuinely useful in the right room, never the goal of the comment.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Hey Gala, I checked out Devetty. Interesting approach to building focused applications around Spanish public datasets.

I noticed you're running multiple products with AI agents. I'm exploring production reliability challenges in agent-driven workflows, particularly execution failures, retries and state consistency.

Curious how you're handling these problems as you scale the apps. It would be interesting to exchange engineering experiences.

Thread Thread
 
galadev profile image
Gala •

Hey Rudratosh, thanks for actually checking it out, I'd love to exchange experiences. I'm borrowing your "relevance of the room first" filter for my own commenting from now on. On reliability, two things I've landed on: jobs stay small and checkable instead of one autonomous everything, and I write intent before a side effect and reconcile on retry, which is straight from your idempotency point. What I still can't solve is knowing when an agent's "done" is really done. What failure shows up most often in your production agent workflows? If you're writing a post about it, I'd read and comment.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

That's a really good question.

The failure I keep coming back to is confusing a successful tool response with a successfully completed business operation.

For example, an agent calls an API, gets a success response, and reports that the task is done. But the downstream state may not actually reflect the intended outcome. The opposite happens too: the operation succeeds, the response gets lost, and the agent retries it.

I think the bigger problem is that agents often report completion based on what they observed, rather than what the system can independently verify.

My approach is to treat the agent's "done" as an assertion, not proof. Verify the actual outcome against the database, API or external system.

I'm curious about your approach: do you verify completion against the final application state, or do you maintain a separate execution record?

Thread Thread
 
galadev profile image
Gala •

Right now I verify against final state for anything that spends money or sends email, and I don't keep a separate execution record yet. That's probably the gap you're pointing at. I'm wary of turning every tool call into a transaction, though. What's the cheapest way you've found to keep an execution record: append-only log, idempotency keys, or something lighter? And did it actually catch retries in practice?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

I wouldn't make every tool call transactional. I'd separate preventing duplicates from recording what happened.

For me, the cheapest useful baseline for side-effecting operations would be:

  1. Generate a stable operation_id before the first execution. Reuse it across retries; don't generate a new one for every attempt.
  2. Store a small execution record with the operation_id, action, status, attempt count and external reference.
  3. Enforce uniqueness at the database level for the business operation.
  4. On retry, reuse the same idempotency key if the downstream provider supports it.
  5. If the provider doesn't support idempotency, reconcile the downstream state before blindly executing again.

An append-only log gives you an execution history, but by itself it doesn't prevent duplicates. An idempotency key helps prevent duplicates, but doesn't independently prove the business outcome.

I tested this distinction in my AssertState demonstration. The slow-retry scenario schedules a second job for the same invoice. The agent reports completion, both jobs process, and two payment records get created. Independent database verification catches the extra record in all 3 trials.

One important caveat: this was a controlled demo with a simulated payment workflow, not a live payment-provider integration.

I'd apply this selectively to operations like payments, emails and record creation, rather than every read-only tool call.

Curious about one thing in your setup: do your tools currently receive a stable operation ID from the agent/orchestrator, or does each retry look like a completely new request?

Thread Thread
 
galadev profile image
Gala •

Ha, the slow retry quietly turning into two payments is the scariest kind of bug, because everything says it worked 😅 Honest answer: no stable operation_id, each retry looks brand new. But my setup makes that rarer than you’d think. Anything that can be scheduled just runs as a plain scheduled job, no agent involved. Because of token costs, autonomous agents only get narrow work: data analysis and tiny fixes. Design, content and new projects go through a “CEO” agent that I steer myself, and it runs the orchestrator with different agent profiles. So most side effects have a human deciding right before them.

Funny thing is, my real problem isn’t reliability, it’s traffic. I’ve got a handful of small data sites that work, and getting people to actually find them is the hard part. When you’ve launched something new, what brought the first real visitors: search, communities like this one, or something else?

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

For my own projects, I've found communities more useful for getting the first meaningful conversations, while search is the longer-term acquisition channel.

For example, this AssertState project has started getting technical discussions through DEV articles and comments. That's been useful for connecting with people who actually build AI systems. But I wouldn't claim I've cracked predictable customer acquisition yet.

My current approach is:

  • Communities for early discovery and conversations.
  • SEO for capturing people who are already searching for a solution.
  • Technical content to establish credibility and give people a reason to explore the product.

One thing I've learned is that publishing content alone isn't enough. You need to get it in front of the specific people who have the problem.

I'm curious about your data sites. What kind of data do they provide, and who are their intended users? The acquisition strategy could be quite different depending on the niche.

Thread Thread
 
galadev profile image
Gala •

Ha, "get it in front of people with the problem" is exactly the step I keep skipping and then wondering why nothing moves. My sites are small pages on Spanish public data for people who are already looking something up, mostly making open data less painful to read. On your split, which one actually moved the needle first for you, communities or SEO? I'm torn on where to spend my one free hour 🙈

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

For me, communities moved first in terms of getting actual conversations and feedback. SEO is more of a longer-term investment.

I wouldn't expect SEO to bring meaningful traffic immediately, especially for smaller sites.

With just one hour, I'd probably spend 40 minutes finding communities where people already discuss or look for that specific public data, and share something genuinely useful from your site. Use the remaining 20 minutes to improve one page based on what people are actually searching for.

I'd prioritise getting the first 10 relevant visitors and understanding what they need over chasing larger traffic numbers.

But I haven't found a predictable acquisition formula yet either. I'm still experimenting.

Collapse
 
zahurul profile image
Zahurul Islam •

Great write-up. One thing I'd add from building orchestrators: put the operation ID one layer up, in the plan rather than the tool call. If the agent re-plans after a timeout, the LLM usually writes slightly different arguments (amount formatted differently, a new memo string), so a key hashed from the arguments quietly changes and you get the double charge anyway. Mint the ID when the step is planned, persist it before dispatch, and have the tool runner (not the model) reuse it on every retry. Then "unknown outcome" becomes a state you reconcile, not an error you retry.

Collapse
 
rudratosh profile image
Rudratosh Shastri •

This is the sharpest version of the point, and you're right to push the ID up a layer. An arguments-hashed key looks idempotent until the re-plan: the LLM reformats the amount or rewrites the memo, the hash shifts, and the "duplicate guard" waves through a second charge. The key has to be independent of the exact token sequence the model emits on retry.

Mint-at-plan-time, persist-before-dispatch, tool-runner-reuses is precisely the ownership split — the model proposes the step, but it doesn't get to be the source of identity, because its output isn't stable across attempts. That's the whole move: identity becomes infrastructure, not a model responsibility.

And your last line is the thesis in one sentence — "unknown outcome" is a state you reconcile, not an error you retry. Mind if I quote that framing if I expand this into a follow-up on saga/compensation flows?

Collapse
 
zahurul profile image
Zahurul Islam •

Please do, I'd be glad to see it. "The model proposes the step but doesn't get to be the source of identity" is a better line than mine. For the saga piece, the same ownership split carries over nicely: each compensating action gets its own planned ID too, so an undo that gets retried can't double-refund. Share the link when it's up.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

The saga piece is coming, and your framing is going straight into it: each compensating action gets its own planned ID too, so a retried undo can't double-refund. That's the symmetry people miss — a compensation is a new operation with the same identity discipline as the forward step, not a magic rollback. "An undo that gets retried can't double-refund" is the one-line test for whether a saga is actually safe. I'll drop the link here when it's up.

Thread Thread
 
zahurul profile image
Zahurul Islam •

Love that test: "an undo that gets retried can't double-refund." One more thing worth covering in the saga piece: record the compensation's outcome the same way as the forward step, so a crash mid-undo goes through reconcile instead of firing again. Looking forward to the link.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Exactly — the compensation needs the same ledger treatment as the forward step: write its outcome durably, so a crash mid-undo lands in reconcile instead of firing a second refund. That's the symmetry that makes a saga safe — every action, forward or compensating, is an operation with an identity and a recorded result; nothing is a fire-and-forget "rollback." Going straight into the follow-up. Link here when it's up.

Thread Thread
 
zahurul profile image
Zahurul Islam •

"Nothing is a fire-and-forget rollback" is the line. Looking forward to the follow-up. Drop the link here when it's live and I'll share it around.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

"Nothing is a fire-and-forget rollback" — glad that one landed; it's the whole reason the compensation section exists. Will absolutely drop the follow-up link in this thread when it's live. Thanks for offering to share it around — that genuinely helps.

Thread Thread
 
zahurul profile image
Zahurul Islam •

Glad it resonated, and I am looking forward to reading the compensation section in the follow-up. Whenever it goes live, I will be happy to read it closely and pass it along.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Appreciate it, Zahurul — I'll make sure the compensation section earns that close read. See you in the follow-up thread.

Collapse
 
hitensethiya profile image
Hiten Sethiya •

the separation between a tool_call_id and a stable operation_id is the key point here. i’d make the UNKNOWN state visible all the way to the product UI too, otherwise the model can be careful internally while the user still hears failed and retries.

Collapse
 
rudratosh profile image
Rudratosh Shastri •

Yes — surfacing UNKNOWN to the UI is the part that quietly undoes everything else. If the model reconciles correctly but the user still sees a bare "failed," the retry just moves up a layer: now it's a human pressing the button twice. The honest message is awkward to write ("we're confirming whether this went through") but it's the only one that matches the actual state. UNKNOWN has to be a first-class outcome end to end, not an internal detail the product layer collapses back into failure.

Collapse
 
kevinhall profile image
Kevin Hall •

the unknown state matters for the bill too, with metered calls the first attempt is often already charged so we look the run up before sending it again

Collapse
 
hitensethiya profile image
Hiten Sethiya •

right, and that lookup only works if it's keyed on the stable operation id. if the provider only sees a fresh tool_call_id on the retry, it bills it as a new call

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Exactly the trap — a metered provider dedups on whatever id it sees, so if the retry arrives under a fresh tool_call_id it's a brand-new billable call no matter how careful your own layer was. The stable operation id has to be the thing the provider keys on (its Idempotency-Key header), not just an internal tag — otherwise the bill is the duplicate side effect even when the business write isn't.

Collapse
 
parsam profile image
Parsa • • Edited

the retry table is the part i’d want to find before it ships. the hard thing is no single file shows the stacking, the http client retries in one place, the SDK has its own default nobody set, the queue redelivers somewhere else. reading one diff you’d never see five layers. that’s one of the patterns we flag at tomosu.ai , retries multiplied across layers and mutations with no idempotency key, so it shows up in the PR instead of on a customer’s card statement. do you audit SDK retry defaults when you add a new provider, or only after something doubles?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

Audit them on add — after something doubles, you're reading a customer's card statement as your test report. And you've named exactly why it's hard: retry stacking is invisible in any single diff because no one file holds it. The HTTP client's retry, the SDK's unset default, the queue's redelivery, the agent's own re-plan — each is reasonable in isolation and only multiplies when they compose, so a per-file review never sees the product. The audit has to be a question about the whole path, not a line in a diff: for this new provider, how many independent layers can re-send one call, and which of them carries my operation id through unchanged? The honest default assumption for any SDK is "retries on, idempotency off" until you've read the config and proven otherwise — most client libraries retry by default and leave the idempotency key to you. Auditing the retry defaults the moment you wire up a provider is cheap; discovering them from a duplicate charge is not.

Collapse
 
rudratosh profile image
Rudratosh Shastri •

Audit them on add — after something doubles, you're reading a customer's card statement as your test report. And you've named exactly why it's hard: retry stacking is invisible in any single diff because no one file holds it. The HTTP client's retry, the SDK's unset default, the queue's redelivery, the agent's own re-plan — each is reasonable in isolation and only multiplies when they compose, so a per-file review never sees the product. The audit has to be a question about the whole path, not a line in a diff: for this new provider, how many independent layers can re-send one call, and which of them carries my operation id through unchanged? The honest default assumption for any SDK is "retries on, idempotency off" until you've read the config and proven otherwise — most client libraries retry by default and leave the idempotency key to you. Auditing retry defaults the moment you wire up a provider is cheap; discovering them from a duplicate charge is not.

Collapse
 
parsam profile image
Parsa •

assuming every SDK retries with no idempotency until you’ve checked is a good default, stealing that. and the whole path point is basically why we read the whole repo at tomosu instead of just the PR. looking forward to the saga follow up

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

"Retries on, idempotency off until proven otherwise" is the right default to carry into every new integration. And reading the whole path rather than the single diff is the only way the stacking is visible — the five layers each look fine in isolation and only multiply when you see them together. Saga piece is coming.

Collapse
 
kryeo profile image
Kryeo •

Section 7 is the one people skip and it's the one that bites.

We had a non-agent version of this on a license key store. Two events (order placed, payment captured) could both try to assign a key, and either one could retry. Making the write idempotent fixed the duplicates, but there was still a small tail of "payment captured, no key assigned" that no trigger ever caught. What closed it was a pretty dumb job that runs every 15 minutes and looks for captured payments with nothing assigned. Not elegant, but it turned the unknown state into a log line instead of a support ticket.

With agents I'd guess that sweep matters even more, since the agent may have moved on to a different plan by the time you find out the first call actually landed. Do you feed those late discoveries back to the agent, or just fix them quietly on the side?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

That 15-minute sweep is the reconciler, and "dumb" is exactly what it should be — a periodic query for committed-but-unfinished beats any trigger, because it doesn't depend on the event that went missing being delivered. On feeding it back: feed it back, don't fix it silently. The agent moved on under a false belief ("the step failed"), so a quiet side-fix leaves its world-model wrong and the next plan reasons from "no key assigned" when a key now exists. Emit the late discovery as a state transition the agent can observe (UNKNOWN→SUCCEEDED on that operation id). Fix quietly only when the operation is terminal and nothing downstream depends on the agent knowing.

Collapse
 
kryeo profile image
Kryeo •

"Emit it as a state transition the agent can observe" is a much better answer than either of my options. A quiet fix leaves the agent planning off a world that's no longer true, which is basically the same bug one layer up.

Stealing the UNKNOWN to SUCCEEDED framing. Thanks for the detailed reply.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Glad it helped — and you'd already built the hard part (the sweep that finds the orphan); the only change is where the discovery goes. "Fix it quietly" and "let the agent keep planning off a stale world" are the same bug one layer up, exactly right. Emit the transition and the agent's next plan reasons from what's true.

Collapse
 
yahhi profile image
Valentina Koniukhova •

Thanks for the part about UNKNOWN. That's the piece I was missing.

Last month I ran a small practical version of this on my own two MCP servers. I sent every write tool a second time and looked at what changed. 8 of 31 weren't safe. One republished a story and reset its counters, one counted the same logged hour twice, and two stop/delete tools fell back to "the current session" when no id was given, so a retry could stop a session in another terminal.

Two things from that test that I don't see in your list:

When the caller is a model, the key is usually an argument it already has. stop_session(session_id) is its own idempotency key, and a replay just returns the stored result. That fits your point that the model shouldn't mint the key.

Two devices closing the same record isn't a retry. The phone stops a session offline at 14:05, the laptop stops it at 14:20, and the phone's request arrives last. These are two different intents with two different keys, so a key doesn't help. I needed a merge rule that gives the same result in either order: the earliest end wins, and every attempt is kept on the record.

The table of the 8 tools and the read-only check I used are here: worklore.dev/s/2026-10-20-i-sent-e...

Have you seen that "guess the current one" fallback outside agent tools?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

This is the best kind of comment — you went and measured it. 8 of 31 unsafe is right in the range I'd expect, and your two additions are both real gaps in my list:

The argument-as-key point is sharper than mine. stop_session(session_id) carrying its own natural idempotency key is exactly the "don't let the model mint it" rule, but stated better: often the model doesn't need to mint anything because the business identifier is already in the call. The failure is only when the tool ignores the id it was handed and falls back to "current," which is your next point.

The two-device case is the one I under-sold. You're completely right that it isn't a retry — two intents, two keys, and idempotency is the wrong tool entirely. What you need is a commutative merge: earliest-end-wins, keep every attempt on the record, same result regardless of arrival order. That's a different mechanism (a CRDT-flavored resolution rule, not a dedupe), and conflating the two is how people "solve" concurrency with an idempotency key and then lose a write. I'm going to fold that distinction into the follow-up with credit, because the retry-vs-concurrent-intent confusion is everywhere.

To your question — yes, the "guess the current one" fallback shows up well outside agent tools. It's endemic in CLIs (kubectl current-context, git current branch, the active cloud profile/project in most SDKs): fine for a human at a keyboard who holds the context in their head, quietly dangerous the moment a program with no notion of "current" calls it twice from two places. Agents just make it common instead of rare, because an agent always lacks that ambient context. The fix is the same as yours: refuse the implicit target, require the id in the call.

Collapse
 
gavriliucvlada profile image
Gavriliuc Vlada •

“A timeout tells you what the caller observed” is the clearest way I have seen this explained. About your UNKNOWN question: I think the key rule is that only reconciliation or a person can move an operation to FAILED_FINAL, never the retry loop. What I missed in the post is who owns those UNKNOWN items and how long they can wait. Do you set a time limit before one goes to a human?

Collapse
 
rudratosh profile image
Rudratosh Shastri •

You caught a real gap — the post says only reconciliation or a person may move an operation to FAILED_FINAL (never the retry loop), but it doesn't name the owner or the clock, and both matter. Ownership: the UNKNOWN set belongs to a dedicated reconciler process, not the agent that created the operation — the agent may have moved on or terminated, so something whose whole job is resolving unresolved operations has to own them. The clock: yes, a bounded deadline, tied to your reconciliation SLA rather than a fixed guess. Auto-attempt every non-destructive way to learn the outcome (idempotency-key lookup, provider status query, webhook replay) up to a deadline you can state; if it resolves, terminal state and done; if the deadline passes unresolved, escalate to a human with the full context — operation id, amount, timestamp, every path tried. The rule stays intact: the timer escalates, it never auto-FAILs. A deadline that flipped UNKNOWN to FAILED_FINAL on its own would just be the retry loop's sin with extra steps. Owner + bounded clock + escalation payload is going into the follow-up as its own section — thanks, this is the cleanest statement of the hole.

Collapse
 
gavriliucvlada profile image
Gavriliuc Vlada • • Edited

thanks, this is very clear. “The timer escalates, it never auto-FAILs” is the line I’ll remember.

The escalation payload is the part I hadn’t thought about. If the person only gets an id, they have to repeat all the checks. With the amount, the timestamp and every path tried, they can decide much faster.

One small question for the follow-up: if the deadline comes from the reconciliation SLA, is it the same for every operation, or shorter for bigger amounts? I’ll look out for the post.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

Vary it — a flat deadline is the wrong default. Tie it to blast radius, not a global SLA: the cost of a wrong guess scales with the amount and the reversibility, so the deadline before a human sees it should shorten as the stakes rise. A $2 operation can auto-reconcile for a while — a duplicate is cheap to refund and a poll will probably resolve it — while a $50k wire should hit a human fast, because sitting in silent UNKNOWN dwarfs the cost of a person glancing at it. The clean model is a small risk tier (amount bands, plus "is this reversible?") mapping to deadlines, rather than one timer. Two refinements: make the deadline also sensitive to how unresolvable the op is (no lookup API → escalate sooner, since auto-attempts can't win), and cap the reconcile attempts independently of the clock, so a fast-failing provider escalates on attempts-exhausted even before the deadline. Risk-tiered deadline + attempt cap + reversibility flag — that's the section.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.