Originally published on hexisteme notes.
Six evidence claims in my research tool were waiting for review. Each one needed a verdict: passed, withdrawn, or failed. My main Claude session had already formed opinions on most of them, and the tool has a command that records the verdict. The obvious move was to let the session record all six and move on. One of the two outside reviewers I had asked to check the batch said no to half of that, and the half it said no to is the part this note is about.
The setup
The research tool collects sources, freezes their raw text, and lets an agent register claims that point at specific spans of that text. A claim stays pending until someone reviews it. Passing a claim adopts it; withdrawing one takes it out of the set.
For this batch I told my main Claude session to work through the open items, and to settle each one by consulting two other models with different jobs:
- Codex, run read-only, with the job of refuting things against code and data.
- agy (Gemini), with the job of checking direction and anti-goals: the list of things the tool is explicitly not supposed to do.
The rule for the whole batch was simple. Only items where both reviewers pointed the same way got executed. Items where they split were held and written down, not decided by the main session.
What agy flagged
The six claims were the review queue. agy's point did not concern any single claim. It concerned who records the verdict.
If my main Claude session records "passed", that is the agent approving its own evidence. One of the tool's anti-goals is automatic adoption of claims. A session that drafts a claim's verdict and then writes "passed" into the store is exactly that, even if the verdict happens to be right. agy's line was that withdrawal can be delegated to the agent and approval cannot.
The agreed result of the two reviews was:
- Withdrawal: 3 claims, recommended by the agent and the reviewers.
- Passed: 3 claims, left to me.
My own reading of why the asymmetry holds is narrow. A withdrawal takes a claim out of the usable set. A pass puts one in. The anti-goal is about the second direction. I did not test any wider principle than that, and the source record only states the rule and the reason agy gave.
The tool refused anyway
There was a second gate that I had not designed with this in mind. When the session tried to record the verdicts, the CLI refused to write a review non-interactively. Its message was that the owner should run it directly in a terminal.
The session did not route around this. It wrote the six commands into a shell script and left it for me to run. That is the behavior I wanted, and I want to be precise about why it happened: the refusal came from the CLI, and the decision not to bypass it was the session's. I did not confirm whether the session would have bypassed a softer prompt.
When I ran the script, it recorded only 2 of the 6 and stopped. The cause was set -e combined with a mismatch on the 8-character confirmation input. The replacement script made every line independent, so one failed confirmation could not stop the rest, and it passed a bash -n syntax check. As of this writing the records show the first script's 2 withdrawals; I have no record yet of the second script being run. That second script also carried 4 new pending claims from another session's run, which my main session and Codex had each judged separately. Those went into the same script.
The reviewer who overturned the main session
The most useful single finding in the batch was a correction to the main session's own earlier reasoning.
Codex gave its own verdict on all six claims. On one claim, it said the claim should pass on meaning, and that the main session's reason from October 31, "no workshop evidence", was wrong. The frozen raw source had a Comments line at line 136 reading "Peer-reviewed by NECO 2018 Workshop". My main Claude session opened the file and confirmed the line.
That claim was still recommended for withdrawal, but for a different reason. Another claim among the same six is a later version that replaces it, so the withdrawal reason was corrected to "superseded by the later version" instead of the false "no workshop evidence". If I had let the original reason stand, the record would say the source lacked something it actually contained.
This is also where the asymmetry mattered. Withdrawing that claim with a corrected reason was low-stakes. Had the main session been allowed to record passes, it would have been grading claims on the same reasoning that had just been shown wrong on one of them.
One more claim, among the passes left to me, got a caution note. The source span said "even the strongest frontier models", and the claim restated it as "frontier models". Codex called it a small generalization and passed it. I left the call on that one to the human side, which is me.
The "unresolved" items that were already resolved
An afternoon report on the same day had listed 6 items as unresolved. They were not. All six already had fixes or explanations recorded: a scholar-search budget, a drop in wiki rows that turned out to be wiki HTTP 429 responses, a WAF PDF item that turned out not to apply, a line-break issue in a corpus search command, full-text XML handling, and a "v2" number in a counsel output.
Codex's section on these checked each one against code and selected tests and confirmed all six were handled. The cause was mechanical: the earlier "unconfirmed" lines of a previous section in the batch notes were grepped, and the later lines in that same section, where the resolutions were written, were never read. The grep found the question and missed the answer below it.
I list this here because it is the same shape of risk as the self-approval problem, in the other direction. An agent that reports something as open when it is closed creates work. An agent that reports something as closed when it is open is the case the approval rule exists for.
Where the two reviewers split
Some candidates for new work got opposite answers. A check for contradictions in entity attributes: Codex said hold, because without binding the subject, attribute, period, and unit it would raise false conflicts; agy said start. Strengthening automatic detection of how much of a page was captured: Codex said start, with only clear negative signals; agy rejected it as turning a judge into a gate, which is an anti-goal. Both were held.
A third round with narrower briefs brought both reviewers to the same reduced version of each, and only that reduced version was built. The parts they still disagreed on stayed out.
The code review, briefly
After the first batch of changes, I sent the uncommitted diff to Codex for code review. It raised 6 findings. The session reproduced each one before touching code, fixed all six, and added 15 regression tests. Examples: a stop gate attributing a later task's block to an earlier task that had passed, and a shared-passage index that only recorded the first position of each chunk and so missed a shared 300-character passage. Over the batch the test suite went from 1069 to 1102 passing, and after the third round it stood at 1124.
What is still open
The approval rule currently rests on two things: agy's objection being written down, and the CLI refusing non-interactive review writes. As far as this batch shows, the second one is the only part enforced in code. If the CLI ever accepted non-interactive review writes, nothing recorded in this batch would stop an agent from recording "passed".
Codex also ran in a sandbox that could not create temporary files, so it could not run pytest. The full test pass in this note comes from the main session's run, not from the reviewer.
And the first script stopping after 2 of 6 is a reminder that "the agent handed me a script" only moves the risk to the script. Next I want to check whether the review command can tell a script-driven run from a person at the keyboard at all, or whether the refusal I relied on was only about the missing terminal.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (12)
This is a useful example of why AI agent permissions should be designed around the impact of an action, not just the capabilities of a tool. Low-risk actions can be delegated, while decisions that add or validate information should require independent verification and explicit authorization. We follow a similar principle at IT Path Solutions when designing AI-powered workflows, keeping automation efficient without compromising accountability.
Your framing — permissions based on action impact rather than tool capability — is sharper than my narrow self-approval example. It captures why withdrawals are safe to delegate while approvals need independent verification. Thanks for articulating that principle clearly.
Thanks, John! I think this distinction becomes even more important as agents gain access to more tools and workflows. The same tool can perform both low-impact and high-impact actions, so permissions need to account for the actual operation and its consequences. Separating what an agent can execute from what it can authorize is a useful foundation for building more trustworthy AI systems.
That framing — separating execution from authorization — is sharper than my narrow claim-verdict example. It turns a single workflow constraint into a reusable design principle for any agent with tool access. Thanks for naming that distinction.
The same asymmetry shows up in code review. In my gate a CHANGES REQUIRED from the reviewer blocks the merge with nobody else involved. An ACCEPT only unblocks it and I do the merge myself, because a wrong ACCEPT looks the same as a right one. The self-approval half lives in a script:
merge-check.shrefuses an ACCEPT whose reviewer is the engine that wrote the commit. That rule is enforced in code and still shares your open problem: the reviewer is a name written in a file, and anything with write access to the repo can write that file.The asymmetry holds. I do not think the axis is direction.
Withdrawal is delegable because it is reversible. A withdrawn claim can be registered again from the same frozen span, so an agent that withdraws wrongly has cost you a second review. A pass is not reversible in the same way, because by the time you find out, something has been built on it. Removal and addition is the shape the rule takes in your tool. Reversibility is why it works, and that distinction matters because it tells you where your own rule stops.
It stops as soon as anything downstream consumes a pending or passed claim. If withdrawing a claim invalidates work that cited it, or if the frozen source gets collected when the last claim pointing at it goes, then withdrawal is no longer the cheap direction, and delegating it is the same bet as delegating a pass. Worth checking which of those your store does today, because the rule is currently resting on a property of the implementation rather than on the anti-goal.
@reidmarlow's revision binding is the right companion, and it answers a question yours does not: which text was approved. Yours answers who may approve. They fail differently. A revision binding with a delegated approver still lets the agent grade its own evidence. An independent approver with no revision binding approves a moving target.
The Codex correction is the most useful thing in the post and I would put it ahead of the asymmetry. A withdrawal reason that was wrong on the facts, caught by a reviewer whose job was to refute rather than to decide, is the argument for the second reviewer standing on its own. The asymmetry only tells you who writes the verdict down.
I checked the store against both of your conditions.
The frozen source is not collected. The only code that deletes raw text is a revocation path for documents whose terms forbid storing them, and it runs regardless of any claim's state. It also keeps raw bytes that another document still points at.
Downstream use is the condition that will apply, later. The counsel layer accepts only claims that are passed or pending on the query date, so a withdrawn claim fails that check. Counsel results are not frozen yet, which means nothing stored breaks today. Freezing them is a held item, and once it lands, withdrawing a pending claim breaks any counsel that cited it. The rule has to be rewritten at that point.
You also made me look for a case I had missed. The transition table allows passed → withdrawn. An agent that withdraws a passed claim removes something I approved and stops it propagating, which is the expensive direction. The only code gate is the terminal check, and it treats every decision the same. The delegation should say pending claims only.
On reversibility: withdrawn is terminal for a given claim id. Undoing a withdrawal means registering a new claim, and that happened in this batch. One of the three withdrawals was replaced by a re-registered version that I later passed. So a wrong withdrawal costs a re-admission and a second review, as you said.
On revision binding: claim files are immutable and the id is a hash of the content, spans included, so a review already attaches to one revision. A changed claim gets a new id and starts as pending. The reviewer field is only the OS login name, so the "who" half is the weaker one here.
A correction to the post: the store now shows all six verdicts recorded on November 8. The line saying I had no record of the second script being run is out of date.
The asymmetry you ran into is the mundane version of something my own directory does to itself. I run 4,323 AI tool listings, and the 'verified' badge on them is produced by the party with the strongest interest in it being true (the vendor claiming the listing) and then recorded by the same crawler that generated the claim. Split by price tier it gets worse: 614 of 4,323 are claimed (14.2%), and free - the tier readers trust most - has the lowest rate at 261/2,198 (11.9%). Where two sources disagree I also auto-adopt instead of parking the item: three landing pages from one vendor came back labelled free / free / freemium, so the same verdict exists three ways and nothing ever decided it. Your rule that only both-reviewers-agree items get executed, and splits get written down rather than resolved by the session that made the claim, is exactly the step I am missing. The disagreeing case is data I still cannot defend a year later.
Three pages on one vendor's domain would count as one source in my tool. Independent origins are counted by registrable domain, with mirrors folded into the original, and a page that copies or closely paraphrases another within a short window folds into it too. So free / free / freemium is one vendor disagreeing with itself. Taking the majority by page count would count the same source three times.
The split rule in the post applied to reviewer verdicts. For conflicting values in the data, my tool does less than you might expect. The two reviewers disagreed on whether to build a contradiction check at all. Codex objected that without binding subject, attribute, period and unit it would raise false conflicts, such as a January increase against a February decrease. What got built was the reduced version: a datum the pipeline had already flagged as contradicted now shows up as a review note, with fixed wording that the contradiction in meaning is unconfirmed. It never picks a value. When I replayed it over 109 claims, 5 got the note, all from the same datum.
The gap you describe is in my store too. The flag survives in the frozen input, and the value on the other side of the conflict does not. Later I could show that a value was disputed, but not what it was disputed against. For your case I would keep all three labels with page URL and fetch date, and leave the listing unresolved until a person picks one. Freemium on one page and free on two may turn out to be two plans or two dates, and only the stored context lets you tell which.
I've run into this same issue building agent workflows. Letting an agent remove bad data feels very different from letting it approve new data. The cost of getting those two actions wrong isn't the same.
The terminal check is a useful friction point, but it doesn't establish who authorized the verdict. I'd bind approval to the exact claim revision and source span, then require an independently authenticated reviewer to approve that revision. A changed claim should invalidate the approval. That also gives the human a way to review the proposed script without treating running it as approval of every line.
The revision-binding and independent-reviewer model you describe closes the authorization gap I left open. Tying approval to a specific claim revision and source span, then invalidating it on any change, is a sharper control than my terminal check alone. It also cleanly separates script review from execution approval. Thanks for pushing the design past "who clicked run" to "what exactly was approved."