This hit close to home 😅 My version was much less glamorous than inventory: two coding agents running test suites in the same repo, sharing one test database. Each run was "correct" on its own, and together they produced failures neither could reproduce. One of them was even an orphaned test process that kept running after I'd killed its agent.
My fix was the dumbest possible lock: one agent per repo at a time, enforced by me, not by the agents. It works because my scale is tiny.
For the business-state case, do you lean towards optimistic locking (version column + retry) or giving each agent a lease on the resource before it decides? Curious which one survives real traffic better 👀
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
That orphaned test process is an interesting detail because it exposes another layer of the problem: controlling the agent doesn't necessarily mean controlling everything the agent started. The process can outlive its owner, while still holding resources or modifying shared state.
On optimistic locking versus leases, I would lean towards optimistic locking as the default for business-state updates, particularly when conflicts are relatively infrequent. A version check makes the decision conditional on the state actually being the one the agent evaluated. If it has changed, the agent must re-read and reconsider rather than blindly retry.
Leases become useful when contention is higher or when an agent needs temporary exclusive ownership of a resource while performing a multi-step operation. But they introduce their own challenges: expiration, renewal, abandoned leases and fencing tokens. A lease alone doesn't prevent an expired owner from continuing to execute unless the downstream system validates a monotonically increasing fencing token.
Personally, I wouldn't treat them as competing approaches. I'd use optimistic concurrency for protecting individual state transitions and leases where exclusive coordination is genuinely necessary. In either case, the final write still needs validation at the authoritative service.
Your one-agent-per-repo rule is actually a good example of choosing the simplest coordination mechanism that matches the scale of the problem. Sometimes the boring solution is the most reliable one. The interesting question is where the boundary changes as agent concurrency grows.
That split is exactly what I was missing: optimistic locking for normal business writes, leases only when something needs exclusive ownership for a while. I'm still at the boring one-agent-per-repo stage, so my boundary is basically the repo itself. The awkward part is that tasks still spawn sub-processes I don't fully control, like that orphaned test run. Where did leases first pay off for you: was it a specific multi-step job, or just contention in general? And did fencing tokens catch a real bug, or were they mostly insurance?
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
I think leases start paying off when the problem shifts from protecting a single write to coordinating ownership across a sequence of steps. For example, when an agent has to process a job, update several related resources, or interact with an external system, overlapping execution could cause side effects. That's where having temporary ownership can simplify coordination.
But I’d be careful not to make the lease the only line of defence. Expiration doesn't necessarily stop the old process from running. A delayed worker could wake up after its lease has expired and still attempt to write. That's exactly where fencing tokens matter: the downstream service rejects an operation carrying an older token, even if the previous owner is still alive.
I’d describe fencing tokens as insurance against a failure that is easy to overlook until it happens. They become particularly important when workers can pause, lose connectivity, or outlive their leases.
Your orphaned test process is a small-scale version of the same problem. The agent may be gone, but the work it started hasn't necessarily stopped. I'd want process lifecycle management and resource-level protection working together.
And honestly, one agent per repo is a perfectly sensible boundary until the cost of serialising work becomes greater than the complexity of coordinating it.
Wait, so "lease expired" doesn't mean "safe"?? That explains a lot about my orphaned test process, ha. Guarding the write versus coordinating ownership is the bit I was missing. Do you enforce the fencing token in the database or up in app code? I'm trying to picture where it actually bites.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Exactly! A lease expiring only means the worker no longer owns the lease. It doesn't guarantee that the old process has stopped executing. I'd enforce the fencing token as close to the resource being protected as possible. For example, the lease service issues monotonically increasing tokens: worker A gets 41, its lease expires, and worker B gets 42. If A wakes up later and tries to write with token 41, the database or authoritative service rejects it because it has already accepted token 42.
A simplified database approach could look like this:
If the update affects zero rows, the worker's token is stale, and the write must be rejected. The important detail is that the comparison and update happen atomically. Checking the token in the application code and then writing separately leaves a race condition.
For external systems that don't support fencing tokens, I'd put a gateway or intermediary in front of them to validate ownership before allowing side effects. So my rule would be: the lease coordinates who should act; the fencing token protects against who is still trying to act after losing ownership.
"A decision can be valid when it is made and invalid when it is executed" is the line to pin up. The item on your list I'd stress is waiting for human approval. An agent that reads stock, waits twenty minutes for sign-off, then writes is acting on a stale read, and the approver signed off on that stale read too. Approvals need a state re-check at execution, or an expiry, otherwise the human tick widens the race window.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Absolutely. This is one of the places where human-in-the-loop can create a false sense of safety. The human may have approved the decision, but they approved it against a particular version of the state. I'd treat approval as a version-bound authorisation, not a permanent permission. When the agent is ready to execute, it should re-check the relevant state and verify that the approved assumptions still hold. If the state has changed, the approval should be invalidated or sent back for review, especially if the change affects the amount, scope, risk or intended outcome.
An expiry helps limit how long an approval remains valid, but it doesn't replace that final state check. Even an approval that is only two minutes old can be stale if the underlying resource changed seconds ago. The deeper principle is that approval should authorise a specific action under specific conditions, not give an agent an open-ended licence to execute later.
I think this is an important distinction in reliable agent design: the human approves the intent, but the system must still validate the conditions at execution.
Great example. The inventory race is the classic lost update, and agents make it worse because the gap between "read" and "act" includes an LLM call that can take seconds. What worked for me: agents never write shared state directly. They emit an intent ("reserve 1 unit, expected version 7"). A single orchestrator-owned writer applies it with a compare-and-set and returns a conflict when the version moved, and the agent re-plans from there. That way you keep the agents parallel while all writes go through one place. Curious whether you went with locks or optimistic versions in the end?
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
I really like that separation. Letting agents work in parallel while centralising shared-state mutations gives you concurrency without allowing every agent to become an independent source of truth. I'd lean towards optimistic versioning for this pattern rather than holding locks across the agent's reasoning process. An LLM call can take seconds, and keeping a lock open during that time can create unnecessary contention and make failures harder to recover from.
Your intent model is particularly useful because the expected version makes the agent's assumptions explicit. The orchestrator can validate the version, apply the change atomically, or return a conflict so the agent can re-read and re-plan. I'd still use short-lived database transactions or row-level locks inside the writer when necessary, but not across the entire decision-making workflow.
The distinction I find important is this: parallelise decisions, serialise or atomically coordinate state changes. That seems like a much cleaner boundary for multi-agent systems.
Optimistic concurrency helps here: each agent writes with the version it read, and a write against a stale version fails and forces a re-read instead of silently overwriting.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Exactly. The important part is what happens after the version check fails. A stale write should not simply trigger an automatic retry with the same decision. The agent needs to fetch the latest state, reassess its assumptions, and decide whether the original action is still valid.
That distinction matters because optimistic concurrency protects the data from being overwritten, but the re-evaluation protects the business decision from becoming outdated. Both are necessary when multiple agents operate on shared state.
Agreed. Counting those version conflicts per pair of agents is useful too: if the same two keep colliding, their scopes probably overlap and should be split.
We're a place where coders share, stay up-to-date and grow their careers.
This hit close to home 😅 My version was much less glamorous than inventory: two coding agents running test suites in the same repo, sharing one test database. Each run was "correct" on its own, and together they produced failures neither could reproduce. One of them was even an orphaned test process that kept running after I'd killed its agent.
My fix was the dumbest possible lock: one agent per repo at a time, enforced by me, not by the agents. It works because my scale is tiny.
For the business-state case, do you lean towards optimistic locking (version column + retry) or giving each agent a lease on the resource before it decides? Curious which one survives real traffic better 👀
That orphaned test process is an interesting detail because it exposes another layer of the problem: controlling the agent doesn't necessarily mean controlling everything the agent started. The process can outlive its owner, while still holding resources or modifying shared state.
On optimistic locking versus leases, I would lean towards optimistic locking as the default for business-state updates, particularly when conflicts are relatively infrequent. A version check makes the decision conditional on the state actually being the one the agent evaluated. If it has changed, the agent must re-read and reconsider rather than blindly retry.
Leases become useful when contention is higher or when an agent needs temporary exclusive ownership of a resource while performing a multi-step operation. But they introduce their own challenges: expiration, renewal, abandoned leases and fencing tokens. A lease alone doesn't prevent an expired owner from continuing to execute unless the downstream system validates a monotonically increasing fencing token.
Personally, I wouldn't treat them as competing approaches. I'd use optimistic concurrency for protecting individual state transitions and leases where exclusive coordination is genuinely necessary. In either case, the final write still needs validation at the authoritative service.
Your one-agent-per-repo rule is actually a good example of choosing the simplest coordination mechanism that matches the scale of the problem. Sometimes the boring solution is the most reliable one. The interesting question is where the boundary changes as agent concurrency grows.
That split is exactly what I was missing: optimistic locking for normal business writes, leases only when something needs exclusive ownership for a while. I'm still at the boring one-agent-per-repo stage, so my boundary is basically the repo itself. The awkward part is that tasks still spawn sub-processes I don't fully control, like that orphaned test run. Where did leases first pay off for you: was it a specific multi-step job, or just contention in general? And did fencing tokens catch a real bug, or were they mostly insurance?
I think leases start paying off when the problem shifts from protecting a single write to coordinating ownership across a sequence of steps. For example, when an agent has to process a job, update several related resources, or interact with an external system, overlapping execution could cause side effects. That's where having temporary ownership can simplify coordination.
But I’d be careful not to make the lease the only line of defence. Expiration doesn't necessarily stop the old process from running. A delayed worker could wake up after its lease has expired and still attempt to write. That's exactly where fencing tokens matter: the downstream service rejects an operation carrying an older token, even if the previous owner is still alive.
I’d describe fencing tokens as insurance against a failure that is easy to overlook until it happens. They become particularly important when workers can pause, lose connectivity, or outlive their leases.
Your orphaned test process is a small-scale version of the same problem. The agent may be gone, but the work it started hasn't necessarily stopped. I'd want process lifecycle management and resource-level protection working together.
And honestly, one agent per repo is a perfectly sensible boundary until the cost of serialising work becomes greater than the complexity of coordinating it.
Wait, so "lease expired" doesn't mean "safe"?? That explains a lot about my orphaned test process, ha. Guarding the write versus coordinating ownership is the bit I was missing. Do you enforce the fencing token in the database or up in app code? I'm trying to picture where it actually bites.
Exactly! A lease expiring only means the worker no longer owns the lease. It doesn't guarantee that the old process has stopped executing. I'd enforce the fencing token as close to the resource being protected as possible. For example, the lease service issues monotonically increasing tokens: worker A gets 41, its lease expires, and worker B gets 42. If A wakes up later and tries to write with token 41, the database or authoritative service rejects it because it has already accepted token 42.
A simplified database approach could look like this:
If the update affects zero rows, the worker's token is stale, and the write must be rejected. The important detail is that the comparison and update happen atomically. Checking the token in the application code and then writing separately leaves a race condition.
For external systems that don't support fencing tokens, I'd put a gateway or intermediary in front of them to validate ownership before allowing side effects. So my rule would be: the lease coordinates who should act; the fencing token protects against who is still trying to act after losing ownership.
"A decision can be valid when it is made and invalid when it is executed" is the line to pin up. The item on your list I'd stress is waiting for human approval. An agent that reads stock, waits twenty minutes for sign-off, then writes is acting on a stale read, and the approver signed off on that stale read too. Approvals need a state re-check at execution, or an expiry, otherwise the human tick widens the race window.
Absolutely. This is one of the places where human-in-the-loop can create a false sense of safety. The human may have approved the decision, but they approved it against a particular version of the state. I'd treat approval as a version-bound authorisation, not a permanent permission. When the agent is ready to execute, it should re-check the relevant state and verify that the approved assumptions still hold. If the state has changed, the approval should be invalidated or sent back for review, especially if the change affects the amount, scope, risk or intended outcome.
An expiry helps limit how long an approval remains valid, but it doesn't replace that final state check. Even an approval that is only two minutes old can be stale if the underlying resource changed seconds ago. The deeper principle is that approval should authorise a specific action under specific conditions, not give an agent an open-ended licence to execute later.
I think this is an important distinction in reliable agent design: the human approves the intent, but the system must still validate the conditions at execution.
Great example. The inventory race is the classic lost update, and agents make it worse because the gap between "read" and "act" includes an LLM call that can take seconds. What worked for me: agents never write shared state directly. They emit an intent ("reserve 1 unit, expected version 7"). A single orchestrator-owned writer applies it with a compare-and-set and returns a conflict when the version moved, and the agent re-plans from there. That way you keep the agents parallel while all writes go through one place. Curious whether you went with locks or optimistic versions in the end?
I really like that separation. Letting agents work in parallel while centralising shared-state mutations gives you concurrency without allowing every agent to become an independent source of truth. I'd lean towards optimistic versioning for this pattern rather than holding locks across the agent's reasoning process. An LLM call can take seconds, and keeping a lock open during that time can create unnecessary contention and make failures harder to recover from.
Your intent model is particularly useful because the expected version makes the agent's assumptions explicit. The orchestrator can validate the version, apply the change atomically, or return a conflict so the agent can re-read and re-plan. I'd still use short-lived database transactions or row-level locks inside the writer when necessary, but not across the entire decision-making workflow.
The distinction I find important is this: parallelise decisions, serialise or atomically coordinate state changes. That seems like a much cleaner boundary for multi-agent systems.
Optimistic concurrency helps here: each agent writes with the version it read, and a write against a stale version fails and forces a re-read instead of silently overwriting.
Exactly. The important part is what happens after the version check fails. A stale write should not simply trigger an automatic retry with the same decision. The agent needs to fetch the latest state, reassess its assumptions, and decide whether the original action is still valid.
That distinction matters because optimistic concurrency protects the data from being overwritten, but the re-evaluation protects the business decision from becoming outdated. Both are necessary when multiple agents operate on shared state.
Agreed. Counting those version conflicts per pair of agents is useful too: if the same two keep colliding, their scopes probably overlap and should be split.