DEV Community

Cover image for Can You Detect Contradictions in AI Agent Memory? I Measured It
Alfredo Izquierdo
Alfredo Izquierdo

Posted on Originally published at contextforge.dev

Can You Detect Contradictions in AI Agent Memory? I Measured It

Originally published on contextforge.dev.

I wanted to detect contradictions in AI agent memory automatically: the moment you save "drop Redis, sessions go in Postgres", the tool should notice you once saved "use Redis for sessions" and tap you on the shoulder. I spent a day building it, ran it on 193 of my own real memories, and the numbers told me to stop. This is that day, with the numbers.

Why I wanted it

The same morning, my blog pipeline wrote a post I had already thrown away four days earlier. I had saved the fix. But months before, I had saved another note saying the workflow reads its config from somewhere else entirely, so the fix went into a file nothing reads. Two true notes, nothing connecting them.

I posted about it on r/ClaudeCode and the thread filled up with people solving the same problem by hand. One runs a hook that checks for contradictions every time memory or CLAUDE.md is edited. One keeps one fact per file and overwrites in place, with git holding the history. One uses architecture decision records and marks the old one "Superseded by ADR-012". Everyone had the same rule underneath: when a decision changes, deal with the old note in the same step.

So the feature wrote itself in my head. On every save, look for older memories that resemble the new one. If something is close, warn: "this looks like X, do you want to update it or link them?" Never block, never decide on its own. Just make it impossible to miss.

What I built

ContextForge already turns every memory into an embedding for search, so the cheap version was obvious. When a memory is saved, compare its embedding with the others in the same project, and return the closest ones above a threshold. If you've used semantic search, you know the move.

I was careful about the boring parts, because this runs on every save of a product people use:

  • It only compares inside the same project and the same organization. A test checks that an identical note in someone else's account never shows up.
  • If anything in the lookup fails, the save still goes through. The warning is optional; the memory isn't.
  • Installed clients that don't ask for it see exactly the same response as before.

All of that worked. Locally, two near-identical notes scored 0.95, unrelated notes stayed below 0.69, and I had a placeholder threshold of 0.80 that looked comfortable.

Then I did the part I almost skipped: before shipping, measure it on real data.

The test

I wrote a read-only script and ran it against my own project: 193 memories about building ContextForge. It does two things.

First, it scores a known pair that should trigger the warning. I used a clean one: a note explaining why my n8n cron misfired (Docker's VM sleeps when the Mac sleeps) and the later note saying the fix was applied (launchd now triggers it). Cause and fix. Exactly the kind of pair you want linked.

Second, it measures noise the way a user would feel it: for every memory, pretend you're saving it now and check whether anything else in the project would cross the threshold. That gives you "what percentage of saves would show a warning". My goal was 5% or less. Anything more and people learn to ignore it.

The numbers

Whole-memory embeddings first:

Threshold Catches the cause → fix pair? Saves that would warn
0.70 Yes 79%
0.75 Yes 62%
0.80 No 40%
0.85 No 17%
0.90 No 2%

The pair scored 0.773. To catch it, the tool would nag on six saves out of ten. At 0.90 the noise is fine, but by then it only catches near copies, which isn't the problem anyone in that thread had.

I suspected the culprit was length: long notes in one project share a topic, so their embeddings mostly say "this is about n8n". So I tried shorter inputs.

What I compared Pair score Best threshold that still catches it Saves that would warn
Whole memory 0.773 0.75 62%
Title only 0.579 never caught —
Title + first paragraph 0.802 0.80 19%
Shared words in the title 0.061 never caught —

Title plus the first paragraph was the best, and it still warns on one save in five.

One honest caveat: a single positive pair is thin evidence for where the threshold should sit. But the noise column doesn't depend on that pair at all. Whatever threshold you pick low enough to catch a real "this replaces that", a large share of normal saves cross it too.

Why it fails

Similarity measures topic. "This decision replaces that one" is not a topic.

In a real project, most of your notes cluster around the same few subjects. My n8n notes look like each other because they are all about n8n. The note that matters, the one that silently overrides another, doesn't look more similar than its neighbours. It looks exactly as similar as everything else in the cluster. You can't find a needle by measuring how much it resembles hay when the needle is made of hay.

There was a second, smaller lesson: speed. Pulling every embedding in the project into the function took about 1.1 seconds per save on 193 memories. If this ever comes back, the scoring has to happen inside the database.

What works instead (today)

None of this means the problem is hopeless. It means the fix isn't a threshold.

  • Deal with the old note when you write the new one. Overwrite it, delete it, or mark it replaced. Every experienced person in that thread does this, and it costs one sentence to your agent.
  • Make the replacement explicit. In ContextForge that's a supersedes link between the two memories; in an ADR folder it's "Superseded by ADR-012". Search then pulls in the new one next to the old one. I wrote about how those links work.
  • Put hard rules where they're enforced, not remembered. A hook or a check that fails beats a note in memory for anything that really can't drift. My n8n bug came back precisely because the only guard was a note.

What I'll try next

If a tool is going to spot contradictions, it has to understand the two notes, not measure their distance. The design I'd try is: use similarity only to shortlist a few candidates, then ask a small model "does the new note replace, contradict or duplicate this one?", and do it in the background instead of on the save, surfacing the result the next time your agent loads context. It costs more, and I'll measure it the same way before anyone sees it.

The warning I built that day is parked. It never shipped. The script that stopped it is the most useful thing I wrote all week.

If you've found a way to catch a reversed decision automatically, I'd love to hear how you measured it.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.