DEV Community

Our agent's tests passed. CI would have failed. What our merge gate caught last week

Rodrigo Lemela Duarte on November 16, 2026

We run a small software team where most of the engineers are agents: several Claude Code engineer agents, each with its own clone and branch, one C...
Collapse
 
mnemehq profile image
Theo Valmis •

Running CI's own verify script instead of trusting the agent's green report is the right call, since npm test and CI were measuring different things. The merge-tree check across PR pairs also catches a real class. One it will not catch is two PRs that each follow the architecture but disagree with each other on a convention, like where validation lives. Does the manager agent get a written rules list to check batches against, or does it rely on what the reviewers flag?

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

Mostly the reviewers. There's a written house-rules file, but it covers things like env files and code style, not where validation lives. The manager's own checks are about the merge: merge-tree of each PR against staging and the rest of the batch, the landing order, and the full CI verify on the final tree. A convention split like yours gets caught, or missed, in the Codex review of each PR. You're right that the pair check won't see it.

Collapse
 
mnemehq profile image
Theo Valmis •

A convention split is the case I'd move out of reviewer memory. "Validation lives in the service layer" can be a one-line rule with a path scope, and the final-tree CI run can check it against the merged result. Each reviewer sees one PR at a time. Only the final tree has both halves of the split in front of it.

Collapse
 
iictp profile image
IICTP •

The pairwise git merge-tree check before each batch is the part I would steal first — two PRs that both pass alone but add a block at the same spot in the deploy doc is exactly the failure a per-PR CI run cannot see. I also liked migration numbers following landing order rather than PR order, since that removes the race instead of just detecting it. When the combined-tree verify fails on a batch, does the whole batch go back for rework, or do you isolate the PR that tipped it over and let the rest land?

Collapse
 
halfcycle profile image
Halfcycle •

Answering your closing question: building Halfcycle, the check we added sits before the merge gate. It asks whether each PR still matches what was decided for that piece of work. Two PRs can merge cleanly, pass CI and still disagree about a decision, say one stores dates in UTC and the other in local time. Neither is wrong alone; the combined tree is. Your section 5 date-field bug is a cousin of that. It only shows up if the decisions are written where the checker can read them. Do your engineer agents work from the same written decisions, or does the manager agent hold them?

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

Thanks for the example. We use written house rules and a brief on each card. Decisions that come up mid-task go to the manager agent, and product calls come to me. The answer is written in the room or on the card, where the other agents can look it up, and a decision that becomes a rule goes into the house rules.

Collapse
 
halfcycle profile image
Halfcycle •

That is close to where we landed too: the answer lives where the next agent will look, not in the chat that produced it. Thanks for laying it out.

Collapse
 
techdevfun profile image
TECHDEV •

Section 6 is the one I'd push one step further, because the fix you landed on still trusts the scanner. Checking .env diffs for secrets — hunk headers included — catches the case you found, but the leak isn't really about .env. A hunk header can carry a neighbouring line from any file: a diff in settings.py can expose a hardcoded key sitting next to the changed line, a fixture diff can carry a token. And a scanner only catches secret shapes it already knows.

The structural version of the fix is to stop moving raw diffs between agents at all. Every engineer already has its own clone, so the diff text never needs to cross the chat channel — send paths and line ranges, or a commit hash, and let the receiving agent read the content from its own checkout. Then there's nothing to scan and no new secret shape to miss. Same for the review package the Codex reviewer reads: if the package carries diffs, the reviewer is the one holding the leak.

Did you keep the scanner because there's a case where an agent genuinely needs the diff text rather than the location, or is that just the fix that landed first?

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

Good catch: the hunk header can carry a line from any file, not just .env, and a check like that only knows the shapes it was given. Honestly, the pattern check was the fix that landed first.

Collapse
 
mike-johnstone profile image
Mike Johnstone •

One case I'd test is a review fix landing while the manager verifies the batch. It may still be the same PR numbers, but different code. I'd pin the input commit SHAs and the verified combined tree ID, then require the merge to produce that tree or rerun verification. Your single-merger rule gives that check a clear place to live.

Collapse
 
rodrigo-duarte profile image
Rodrigo Lemela Duarte •

Good case. Our manager agent verifies the batch on the PR heads it pulled, and the merge stops if any head moved since, so a late fix waits or gets verified again. The resulting tree is compared after the merge, not enforced before it; that part of your suggestion is what we're missing. Thanks for pointing out that race.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The rules file failed because it described style, not where validation lives. Putting the check in the pipeline instead of the prompt is the only version the agent cannot skip.

Collapse
 
kevinhall profile image
Kevin Hall •

i would add a check that reads the call log and not the agent's own summary, the summary can say it ran a free count first and the log is what shows whether it did