The hidden reliability problem behind distributed AI agents: dual writes, transactional outboxes, duplicate events, event ordering, and recovery.
I have been thinking about a particular kind of failure in AI-powered systems. It is not the kind of failure where an agent misunderstands a user, produces an incorrect answer, or calls the wrong API. Those problems are important, but they are relatively easy to recognise because something visibly goes wrong.
The more difficult failures happen when every component appears to have done its job. Imagine an AI agent responsible for processing a customer refund. It checks the order, verifies the refund policy, receives the required approval, and asks the backend to record the decision. The database confirms that the refund has been approved. The agent receives a successful response and reports that the process is moving forward.
But the payment service never receives the event that should initiate the refund.
The order now appears approved in one part of the system, while the payment service has no knowledge of that decision. The agent believes it has completed its task, the database contains a successful record, and the message broker might even be reporting a healthy status. Yet the customer is still waiting for the money.
This is the kind of problem that makes distributed AI systems difficult to operate reliably. The agent can make a correct decision, and the database can successfully persist that decision, but neither guarantees that the rest of the workflow will happen.
I have become increasingly interested in this gap between deciding, recording, communicating, and actually completing an operation. In my previous writing about AI agents, I explored what happens when two agents modify shared state and when a workflow completes four out of five steps before failing. Those situations raised an important question: who owns the overall execution when individual components only understand their own responsibilities?
There is another part of that question we need to address. What happens when the orchestrator records a successful operation, but the event that communicates it to the next service is lost, delayed, or delivered more than once?
That is where event-driven reliability becomes essential.
The problem is not always inside the agent
Distributed systems rarely fail in a neat, predictable sequence. A database can commit a transaction while a message broker is temporarily unavailable. A message broker can accept an event while the application loses the acknowledgement. A consumer can process a message successfully and crash before recording that it has completed the work.
These failures become particularly interesting when AI agents are involved because an agent's execution often includes several independent operations. It might retrieve information, make a decision, update a database, call an external API, publish an event, and wait for another service to continue the process.
Each operation has its own failure conditions. Consider a simplified order workflow. An AI agent receives a request, checks inventory, creates an order, and publishes an OrderCreated event. A separate payment service consumes that event, processes the payment, and publishes a PaymentCompleted event. The fulfilment service then consumes the payment event and prepares the shipment. The architecture looks straightforward:
Customer Request
|
v
AI Agent
|
v
Order Service
|
+------> Database
|
+------> Message Broker
|
v
Payment Service
|
v
Fulfilment Service
The problem is that the database and message broker are two independent systems. A normal database transaction cannot automatically guarantee that a message is successfully published to an external broker. That creates a failure window between the two operations. An application might execute something like this:
def create_order(order, db, broker):
db.save_order(order)
broker. publish(
"order.created",
{
"order_id": order.id,
"customer_id": order.customer_id
}
)
return {"status": "success"}
At first glance, this implementation seems reasonable. The order is saved, the event is published, and the function returns success.
But what happens if db.save_order() succeeds and broker.publish() fails?
The order exists, but no event tells the payment service to continue.
What if the opposite happens? The broker accepts the event, but the database transaction subsequently rolls back. The payment service may receive an event for an order that does not exist in the authoritative database.
Neither failure is particularly unusual in a distributed environment. The problem is that the application treats two independent operations as if they were one atomic operation.
This is known as the dual-write problem.
Understanding the dual-write problem
The dual-write problem occurs when an application needs to update two different systems as part of one logical operation, but cannot guarantee that both updates succeed together. For example, an order service may need to:
- Save an order in its database.
- Publish an event to a message broker.
The application wants both actions to happen. However, the database and broker do not share the same transaction boundary. Let's examine the two most important failure scenarios.
Scenario 1: The database succeeds, but event publishing fails
Application
|
| Save order
v
Database
|
| COMMIT SUCCESS
|
v
Application
|
| Publish OrderCreated
v
Message Broker
|
| CONNECTION FAILURE
|
v
Workflow stuck
The database contains the order, but downstream services have not been informed. If the application simply retries the entire operation, it might create another order unless the order creation itself is protected by an idempotency mechanism. If it does not retry, the workflow remains incomplete.
Scenario 2: The event is published, but the database transaction fails
Application
|
| Publish OrderCreated
v
Message Broker
|
| EVENT ACCEPTED
|
v
Database
|
| TRANSACTION ROLLBACK
|
v
Payment Service receives an event
|
v
Order does not exist
Now the payment service has received an instruction to process an order that was never successfully committed. The two scenarios produce different inconsistencies, but they share the same underlying problem: the system cannot atomically coordinate the database update and the event publication.
I would not try to solve this by adding another retry around the broker call. Retries are useful for temporary failures, but they do not remove the gap between the two systems. They can even make the situation worse when the application cannot determine whether the original operation succeeded.
The solution needs to change how the application records and publishes events.
The transactional outbox pattern
One practical approach is the transactional outbox pattern.
Instead of trying to update the business database and publish directly to the message broker, the application stores the business change and the event to be published in the same database transaction.
The event is initially written to an outbox table. A separate publisher reads pending records from that table and sends them to the message broker.
The important difference is that the business update and the intention to publish an event are now committed together.
If the transaction succeeds, both the order and its corresponding outbox event exist. If the transaction fails, neither is committed.
The architecture becomes:
Order Service
|
v
Database Transaction
/ \
v v
Orders Table Outbox Table
|
v
Outbox Publisher
|
v
Message Broker
|
+-------------+-------------+
| |
v v
Payment Service Other Services
The outbox does not make the message broker part of the database transaction. It removes the need to coordinate the two operations inside the request handler. The application first makes the business change durable. The publisher then repeatedly attempts to deliver the recorded event. Here is a simplified PostgreSQL schema:
CREATE TABLE orders (
id UUID PRIMARY KEY,
customer_id UUID NOT NULL,
total_amount NUMERIC(12, 2) NOT NULL,
status VARCHAR(30) NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE TABLE outbox_events (
event_id UUID PRIMARY KEY,
aggregate_id UUID NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
published_at TIMESTAMPTZ,
attempts INTEGER DEFAULT 0
);
CREATE INDEX idx_outbox_pending
ON outbox_events (created_at)
WHERE published_at IS NULL;
The orders table stores the business entity. The outbox_events table stores events that must eventually be delivered.
The event has its own unique identifier. This is important because an event should be identifiable independently of the order it describes. An order might generate several events during its lifecycle, and each event needs its own identity.
Now the order creation and event recording can happen in one transaction:
import json
import uuid
def create_order(db, customer_id, total_amount):
order_id = uuid.uuid4()
event_id = uuid.uuid4()
event_payload = {
"event_id": str(event_id),
"event_type": "OrderCreated",
"order_id": str(order_id),
"customer_id": str(customer_id),
"total_amount": str(total_amount)
}
with db.transaction():
db. execute(
"""
INSERT INTO orders (
id,
customer_id,
total_amount,
status
)
VALUES (%s, %s, %s, %s)
"",
(
order_id,
customer_id,
total_amount,
"CREATED"
)
)
db. execute(
"""
INSERT INTO outbox_events (
event_id,
aggregate_id,
event_type,
payload
)
VALUES (%s, %s, %s, %s)
"",
(
event_id,
order_id,
"OrderCreated",
json.dumps(event_payload)
)
)
return {
"order_id": str(order_id),
"status": "CREATED"
}
The critical part is not the Python syntax. It is the transaction boundary.
The order and the event record either commit together or roll back together. The application no longer has to publish to the broker while creating the order.
This gives us a durable record of what the system intends to communicate.
However, it is important to understand what the outbox does not guarantee. It does not guarantee that the broker will receive the event exactly once. It guarantees that the event is durably recorded alongside the business change, giving the publisher a reliable source from which to retry delivery.
That distinction matters.
Publishing events without losing them
The next component is the outbox publisher.
Its responsibility is to find unpublished events, send them to the broker, and mark them as published after receiving an appropriate acknowledgement.
A simplified implementation might look like this:
def publish_pending_events(db, broker):
events = db.fetch_all(
"""
SELECT event_id, event_type, payload
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
"""
)
for event in events:
try:
broker. publish(
topic=event["event_type"],
message=event["payload"],
message_id=str(event["event_id"])
)
db. execute(
"""
UPDATE outbox_events
SET published_at = NOW(),
attempts = attempts + 1
WHERE event_id = %s
"",
(event["event_id"],)
)
except Exception:
db. execute(
"""
UPDATE outbox_events
SET attempts = attempts + 1
WHERE event_id = %s
"",
(event["event_id"],)
)
This is intentionally simplified. A production implementation needs to handle concurrent publishers, broker acknowledgement semantics, transaction management, backoff, monitoring, and poison messages.
One important issue is what happens when the broker accepts the event, but the publisher crashes before updating published_at.
The publisher does not know that delivery succeeded. When it runs again, it sees the event as unpublished and sends it again.
That means the consumer may receive the same event more than once.
This is not a bug in the outbox pattern. It is a consequence of trying to deliver messages reliably across independent systems.
The publisher cannot always distinguish between these two situations:
- The broker never received the event.
- The broker received the event, but the publisher never received or recorded the acknowledgement.
A robust design accepts that duplicate delivery can happen and makes consumers safe against it.
This brings us to the next part of the problem.
The inbox pattern: protecting consumers against duplicate events
Suppose a payment service receives an OrderCreated event and processes the payment. The payment operation succeeds, but the consumer crashes before acknowledging the message.
The message broker may deliver the event again.
If the payment service simply processes every message it receives, the customer could be charged twice.
The first instinct might be to configure the broker to avoid duplicate delivery. Broker-level features can help, but they do not remove the need for application-level protection. Many messaging systems provide at-least-once delivery because it is generally safer to retry than to silently lose a message.
The consumer, therefore, needs to recognise whether it has already processed a particular event.
This is where an inbox table becomes useful.
CREATE TABLE processed_events (
event_id UUID PRIMARY KEY,
consumer_name VARCHAR(100) NOT NULL,
processed_at TIMESTAMPTZ DEFAULT NOW()
);
A consumer can record the event identifier when processing the business operation.
The key requirement is that recording the event and applying the local business change should happen in the same database transaction.
For example:
def handle_order_created(db, event):
event_id = event["event_id"]
order_id = event["order_id"]
with db.transaction():
inserted = db.execute(
"""
INSERT INTO processed_events (
event_id,
consumer_name
)
VALUES (%s, %s)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id
"",
(event_id, "payment-service")
)
if not inserted:
return {
"status": "DUPLICATE_IGNORED",
"event_id": event_id
}
db. execute(
"""
INSERT INTO payment_requests (
order_id,
status
)
VALUES (%s, %s)
ON CONFLICT (order_id) DO NOTHING
"",
(order_id, "PENDING")
)
return {
"status": "PROCESSED",
"event_id": event_id
}
The exact implementation will depend on the database driver and the transaction API, but the principle is straightforward.
If the consumer crashes before the transaction commits, the inbox record and business change both roll back. The event can be delivered again.
If the transaction commits, both the processed-event record and business change are durable. A duplicate delivery can then be recognised and ignored.
There is an important detail here: the uniqueness constraint must reflect the actual event-processing scope. If the same event needs to be processed independently by several consumers, the key should include both event_id and consumer_name, rather than making event_id globally unique across all consumers.
A more general schema would be:
CREATE TABLE processed_events (
event_id UUID NOT NULL,
consumer_name VARCHAR(100) NOT NULL,
processed_at TIMESTAMPTZ DEFAULT NOW(),
PRIMARY KEY (event_id, consumer_name)
);
The inbox pattern protects local database operations. It does not automatically make external side effects exactly once.
For example, if a payment consumer calls an external payment provider, successfully charges the customer, and then crashes before recording the result, the database transaction cannot roll back the external charge.
That operation needs its own idempotency strategy.
Idempotency must extend beyond the message broker
I have repeatedly come back to idempotency when thinking about AI agent execution. It is easy to discuss retries as though they are harmless, but a retry is only safe when the system can identify the operation it is repeating.
Consider this sequence:
- The payment service receives an order event.
- It sends a charge request to the payment provider.
- The provider successfully charges the customer.
- The network connection drops before the service receives the response.
- The payment service retries the request.
The service has an unknown outcome. It cannot be safely concluded that the payment failed just because it did not receive a response.
If it generates a new payment request, it may charge the customer twice.
The solution is to use a stable idempotency key for the logical payment operation.
def charge_customer(payment_provider, order_id, amount):
idempotency_key = f"payment:{order_id}"
return payment_provider.charge(
amount=amount,
idempotency_key=idempotency_key
)
The same logical payment attempt must reuse the same key when retried. If the provider supports idempotency correctly, it can return the original result instead of creating another charge.
The key should not be generated afresh every time the function executes. Otherwise, each retry appears to be a new operation.
For more complex workflows, I would store the operation identity and provider reference in a durable execution record:
CREATE TABLE payment_operations (
operation_id UUID PRIMARY KEY,
order_id UUID NOT NULL,
idempotency_key VARCHAR(200) UNIQUE NOT NULL,
provider_reference VARCHAR(200),
status VARCHAR(30) NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
The payment operation can move through explicit states:
PENDING
|
v
SUBMITTED
|
+------> SUCCEEDED
|
+------> FAILED
|
+------> UNKNOWN
I particularly like having an UNKNOWN state because it prevents the system from treating uncertainty as failure.
If a request times out, the service should first reconcile with the payment provider using the idempotency key or provider reference. It should not immediately submit a new payment request with a different identity.
The same principle applies to refunds, reservations, inventory allocation, shipping requests, and other operations that create external side effects.
A reliable system needs to know not only what it is intended to do, but also whether the external system actually did it.
Event ordering: receiving the right event at the wrong time
Duplicate delivery is one challenge. Event ordering is another.
Imagine an order that moves through the following states:
CREATED -> PAID -> SHIPPED -> DELIVERED
The order service publishes two events:
Event A: OrderPaid
Event B: OrderShipped
But due to retries, multiple publishers, network delays, or broker partitioning, the fulfilment service receives OrderShipped before OrderPaid.
If the consumer blindly applies every event, an older event might later overwrite a newer state.
For example:
def update_order_status(db, order_id, new_status):
db. execute(
"""
UPDATE orders
SET status = %s
WHERE id = %s
"",
(new_status, order_id)
)
This implementation has no protection against stale events. If OrderShipped is applied first and OrderPaid arrives later, the order may incorrectly move backwards to PAID.
One way to address this is to attach a monotonically increasing version to each aggregate.
{
"event_id": "event-102",
"event_type": "OrderShipped",
"order_id": "order-501",
"aggregate_version": 4,
"status": "SHIPPED"
}
The consumer can then apply an update only when the incoming version is newer than the version it has already processed.
UPDATE orders
SET status = :new_status,
version = :incoming_version
WHERE id = :order_id
AND version < :incoming_version;
This is a simplified example. In a production system, you should also consider whether a consumer is allowed to skip intermediate versions. Some events represent independent facts, while others represent state transitions that must be applied in sequence.
For strict sequential processing, the consumer may need to check that the incoming version is exactly the expected next version. If it receives version 4 while it has only processed version 2, it should not necessarily apply version 4 immediately. It may need to wait for version 3 or retrieve the authoritative state.
Ordering guarantees also depend on the broker. Partitioning messages by aggregate identifier can preserve order within a partition in systems that provide that guarantee. However, it does not automatically solve ordering across different partitions or across multiple independent producers.
The important point is that event order should be an explicit design decision, not an assumption.
Why AI agents make this problem more complicated
Traditional event-driven systems already have to handle message loss, duplicate delivery, retries, and inconsistent state. AI agents add another layer because they can decide what to do based on information received from several services.
An agent might receive an event, inspect the current order, consult a policy service, and decide whether to initiate a refund. During that reasoning process, the underlying order may change.
The agent might also decide to call a tool after receiving an event, but the tool's response may be delayed or lost.
This creates a distinction between three separate things:
- The agent's decision.
- The durable record of the requested operation.
- The actual external side effect.
These should not be treated as interchangeable.
For example, an agent might generate the following decision:
{
"decision": "APPROVE_REFUND",
"order_id": "order-501",
"reason": "Eligible under refund policy",
"amount": 2500
}
That decision is not the same as a completed refund.
The system still needs to validate the order state, record the approved operation, publish the appropriate event, and obtain confirmation from the payment provider.
I would keep the agent responsible for reasoning and proposing actions, while deterministic infrastructure owns event delivery, idempotency, state transitions, and recovery.
An agent should not be responsible for remembering whether a message was successfully delivered or whether an external payment request timed out after being accepted. Those are execution facts, and they belong in durable system records.
This also helps when multiple agents participate in the same workflow. One agent may approve a refund, another may check fraud risk, and a third may coordinate customer communication. None of them should independently maintain their own version of the entire workflow's state.
The orchestrator should own the lifecycle, while each service or agent operates within clearly defined boundaries.
What happens when the outbox publisher itself fails?
The transactional outbox solves an important consistency problem, but it does not eliminate operational failures.
Suppose the publisher is running normally and sending events every few seconds. Then a deployment introduces a bug, the publisher stops, and pending events begin accumulating in the outbox table.
The order service continues creating orders because its database transactions are still successful. However, payment and fulfilment services stop receiving new events.
From the perspective of the order service, nothing is wrong. From the perspective of the overall business workflow, the system is becoming increasingly unhealthy.
This is why the outbox needs operational monitoring.
I would monitor at least the following:
| Metric | What it tells us |
|---|---|
| Pending outbox events | How many events are waiting for delivery |
| Oldest unpublished event age | How long the workflow has been waiting |
| Publish failure rate | Whether broker delivery is becoming unreliable |
| Average publish latency | How quickly events reach the broker |
| Retry count | Which events repeatedly fail |
| Consumer lag | How far behind consumers are |
| Duplicate event rate | How often consumers receive repeated events |
| Dead-letter count | How many events require investigation |
The age of the oldest unpublished event is especially useful. A system might have thousands of pending events during a temporary traffic spike, but that does not necessarily indicate a serious problem. A single event that has been waiting for several hours might be much more important.
I would also avoid relying exclusively on a publisher's health endpoint. A publisher process can be alive while making no progress.
Monitoring should answer a more useful question: Is the workflow moving?
Handling events that repeatedly fail
Some events will fail because of temporary infrastructure problems. Others will fail because the payload is invalid, a downstream service rejects the operation, or the business state has changed.
Retrying every failure forever is not a good recovery strategy.
A publisher should use bounded retries with exponential backoff and jitter. This prevents a large number of failed operations from repeatedly hitting an already struggling service.
A simplified backoff calculation looks like this:
import random
def retry_delay(attempt, base=1, maximum=300):
delay = min(maximum, base * (2 ** attempt))
jitter = random.uniform(0, delay * 0.2)
return delay + jitter
For example, the delay increases after each failed attempt, but a random component helps avoid many workers retrying at the same time.
However, retries should not hide permanent failures.
An event that repeatedly fails validation should eventually be moved into a dead-letter or failed-event workflow, where it can be inspected. The system should retain the original payload, event identifier, failure reason, attempt count, and relevant correlation information.
A dead-letter queue should not become a place where failed business operations disappear. It is a controlled exception path that requires monitoring and an explicit recovery process.
For some failures, an operator may correct the underlying issue and replay the event. For others, the event may need to be rejected or handled through a compensating business action.
The decision should depend on the operation, not simply on how many times the message has failed.
Reconciliation: the part that catches what retries miss
There is one more mechanism I would not leave out of a reliable architecture: reconciliation.
The outbox helps ensure that events are eventually published. The inbox helps prevent duplicate processing. Idempotency protects external operations against repeated requests. But a distributed workflow can still become inconsistent.
A consumer might have a bug. A provider might accept a request but return an ambiguous response. An event might be sent to the wrong destination. A message might be incorrectly acknowledged. A deployment might introduce a state transition that the old consumer does not understand.
Reconciliation provides a way to compare what the system believes happened with what the authoritative systems actually report.
For example, a payment service could periodically identify operations that have remained in SUBMITTED or UNKNOWN for too long.
SELECT
operation_id,
order_id,
idempotency_key,
provider_reference,
status,
updated_at
FROM payment_operations
WHERE status IN ('SUBMITTED', 'UNKNOWN')
AND updated_at < NOW() - INTERVAL '10 minutes';
The recovery worker can then query the payment provider using the stored idempotency key or provider reference.
Based on the result, it can update the local operation:
Local status: UNKNOWN
Provider reports:
Payment completed
Recovery action:
Update local status to SUCCEEDED
Record provider reference
Publish PaymentCompleted event
Or:
Local status: UNKNOWN
Provider reports:
No matching payment
Recovery action:
Confirm operation is safe to retry
Retry with the same idempotency key
The exact recovery logic depends on the provider's capabilities and the business rules.
The important distinction is that reconciliation does not blindly repeat the original action. It first checks what actually happened.
This is particularly important for AI agents because their conversational context may no longer represent the current state of the external world.
A message saying "payment request submitted" does not prove that the payment succeeded. A tool response that timed out does not prove that the payment failed. The system needs to consult the authoritative source before deciding what to do next.
A practical architecture for reliable AI-driven workflows
Putting these pieces together gives us a more dependable architecture.
Customer Request
|
v
AI Agent
|
Proposed Action
|
v
Orchestrator
|
v
Validate Preconditions
|
v
Database Transaction
/ \
v v
Business State Outbox Event
|
v
Outbox Publisher
|
v
Message Broker
|
v
Consumer
|
Inbox Deduplication
|
v
Business Operation
|
v
External Service/API
|
v
Durable Result
|
v
Reconciliation Worker
In this design, the AI agent is not responsible for ensuring that every event reaches every service. It proposes an action, while the orchestrator validates and records the operation.
The database transaction creates both the business state change and the corresponding outbox record. The publisher delivers the event, and consumers use inbox records to protect against duplicate processing.
External side effects use stable idempotency keys and durable operation records. If an outcome is uncertain, reconciliation checks the external system before attempting another action.
This architecture does not promise that failures will never occur. It gives the system a reliable way to identify, record, and recover from them.
A few implementation decisions that matter
There are several details I would settle before implementing this architecture in a real application.
First, define event identity separately from operation identity. An event identifier identifies a message. An operation identifier identifies a logical business action. A single operation may produce several events, and one event may be delivered multiple times. Keeping these identities separate makes tracing and deduplication easier.
Second, use correlation identifiers across the entire workflow. An order identifier, workflow identifier, or trace identifier should connect the agent's decision, orchestrator execution, database transaction, outbox event, consumer processing, and external provider request. Without this, investigating a stuck workflow becomes a search across disconnected logs.
Third, define state transitions explicitly. Avoid treating a successful API response as proof that the entire workflow is complete. A payment might be submitted but not confirmed. A shipping request might be accepted but not dispatched. Use states that describe what the system actually knows.
Fourth, make recovery a first-class part of the design. Every important operation should have a known recovery path. That might mean retrying with the same idempotency key, checking an external provider, replaying an event, or requesting human review.
Finally, make the boundaries clear. The agent should not own the message broker's delivery guarantees. The consumer should not assume that a message is delivered only once. The orchestrator should not assume that a timeout means failure. Each component needs a defined responsibility.
How I would test this system
A system that works when every service is healthy has not demonstrated that it is reliable. I would test the failure boundaries deliberately.
For an outbox implementation, I would simulate the database committing successfully while the message broker is unavailable. The expected result should be that the order and outbox event remain stored, and the publisher delivers the event when the broker becomes available again.
I would then simulate a publisher crash immediately after the broker accepts a message but before the publisher marks the outbox record as published. The event should be delivered again, but the consumer should not create a duplicate business operation.
Next, I would simulate a consumer crash after committing its local database transaction but before acknowledging the message. When the broker redelivers the event, the inbox record should prevent the same local operation from being applied twice.
For external payment operations, I would simulate a timeout after the provider has successfully processed the request. The recovery process should query the provider or retry using the same idempotency key, rather than create a second payment.
Finally, I would test out-of-order events. A consumer should not allow an older state transition to overwrite a newer one simply because the older message arrived later.
A practical failure test matrix might look like this:
| Failure injected | Expected behaviour |
|---|---|
| Broker unavailable during order creation | Order and outbox event commit together |
| Publisher crashes after broker acceptance | Event may be redelivered safely |
| Consumer crashes before database commit | Event can be processed again |
| Consumer crashes after commit but before acknowledgement | Inbox prevents duplicate local processing |
| External API times out after success | Reconcile before retrying |
| Older event arrives after newer event | Version check prevents stale update |
| Publisher remains unavailable | Pending-event monitoring raises an alert |
| Event repeatedly fails processing | Dead-letter handling and investigation |
I would also measure the time between an event being created and its successful consumption. This helps distinguish a system that is technically processing messages from one that is actually completing business workflows at an acceptable rate.
What exactly does "reliable delivery" mean?
One phrase that often creates confusion in distributed systems is "exactly once".
It sounds like the ideal guarantee: every event is delivered and processed exactly one time, without duplicates or loss.
In practice, exactly-once guarantees are usually narrower than they sound. A broker may offer exactly-once processing semantics within a particular boundary, but that does not automatically extend across an independent database, an external payment provider, and an AI agent's tool calls.
A system can still experience an ambiguous network outcome after an external side effect has succeeded.
For this reason, I prefer designing around a combination of durable intent, at-least-once delivery, idempotent processing, and reconciliation.
The publisher may send the same event more than once. The consumer recognises duplicate event identities. The external provider receives a stable operation key. The recovery worker checks the authoritative result when the outcome is uncertain.
This is not the same as guaranteeing that nothing ever happens twice. It is a practical way to prevent repeated deliveries from causing repeated business damage.
There is also a difference between message delivery and business completion. A broker acknowledging an event only tells us something about the broker's handling of that message. It does not prove that the customer received a refund, that an order was shipped, or that the overall workflow was completed.
The metric that matters is not simply how many events were published. It is whether the intended business operation reached a verified final state.
Starting small without building an entire distributed platform
Not every AI application needs a large event-driven architecture from the beginning. Introducing a message broker, outbox publisher, inbox service, reconciliation workers, and several operational dashboards can create unnecessary complexity if the application only has one or two services.
I would start with the failure boundaries that can cause the most damage.
If an operation updates a database and publishes an event, introduce an outbox table. If consumers can receive duplicate events, add an inbox table with an appropriate uniqueness constraint. If an operation calls an external service that can create financial or irreversible side effects, use a stable idempotency key and store the provider reference.
Then add monitoring for pending events, old unpublished records, consumer lag, and operations stuck in uncertain states.
Once the workflow becomes more complex, introduce reconciliation and explicit recovery of ownership.
The important thing is not to adopt every distributed-systems pattern at once. It is to avoid leaving critical business operations dependent on assumptions such as "the broker will probably receive the message" or "the API timed out, so the operation must have failed".
Those assumptions are easy to make during development and expensive to discover in production.
The real reliability boundary is the entire workflow
When I look at AI agent reliability, I find it useful to separate intelligence from execution.
An agent may understand a request, select the correct tool, and generate a perfectly valid action. But the system still has to record that action, communicate it to other services, handle duplicate delivery, preserve ordering, verify external side effects, and recover when one of those steps fails.
The agent's reasoning is only one part of the process.
The transactional outbox addresses the gap between committing a business state and publishing an event. The inbox pattern protects consumers from duplicate delivery. Idempotency keys protect external operations against repeated requests. Version checks help prevent stale events from overwriting newer state. Reconciliation helps establish what actually happened when the system cannot be certain.
Together, these patterns create a much stronger foundation for AI-driven workflows.
I also think this changes how we should evaluate an AI agent. It is not enough to ask whether the agent selected the right action. We should ask whether the surrounding system can prove that the action was recorded, whether the next service received it, whether the external side effect occurred, and whether the workflow can recover if any of those steps fail.
A successful event publication is not the same as a completed business operation. A successful database commit is not the same as a completed workflow. And a successful agent response is certainly not proof that everything downstream happened as intended.
The question I keep returning to is simple: when an AI agent says it has completed a task, can the system independently establish what actually happened?
If the answer depends on the agent remembering what it did, or on a message probably having reached another service, the workflow is not reliable enough yet.
We need systems that can remember through durable records, communicate through recoverable delivery mechanisms, and verify through authoritative state.
That is how we move from AI agents that can perform actions to AI systems that can be trusted to manage them.
Key takeaways
- A database transaction and a message publication are two independent operations. Treating them as one creates the dual-write problem.
- The transactional outbox records business changes and outgoing events in the same database transaction.
- The outbox publisher should expect duplicate delivery because a broker acknowledgement can be lost or remain unrecorded.
- The inbox pattern makes consumer-side processing idempotent, provided the deduplication record and local business update share a transaction.
- External side effects need stable idempotency keys and durable operation records.
- Event versions and explicit state-transition rules help prevent stale messages from overwriting newer states.
- Reconciliation is necessary when a timeout or delivery failure leaves the actual outcome uncertain.
- AI agents should own reasoning and proposed actions. Deterministic infrastructure should own durable execution, event delivery, and recovery.
Top comments (0)