2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking git diff --stat first is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?
Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, that's a fair point. Each edit passes review on its own, so the only place the extra changes show up together is the final diff. That makes --stat useful even with step-by-step review, not only with auto-accept.
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
Mine is a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. I accepted one from an agent without looking. The 30-second check: git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, that's a good addition to #3. The package is real, so nothing looks wrong, and the cost only shows up in the lockfile. Checking the lockfile's line count is a quick way to catch it. When this happens to you, do you usually swap the package for a few lines of your own code, or look for a smaller package that does the same job?
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
Depends on what it does. If it's formatting or a small helper, I write the fifteen lines myself and add a test. For dates, parsing or anything crypto I keep a well-known library even if it's heavier, because there my fifteen lines would be the bug. The smaller lookalike package is the option I trust least, a lot of them have one maintainer and no release in years.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, that's a sensible split. Your point about lookalike packages is a good one. Before I accept a package an agent suggests, I'll start checking its last release date and number of maintainers on npm. That takes about ten seconds.
Founder of Hex16, a small web studio in Poland. I build Next.js sites and headless Shopify stores, with GSAP, Three.js and shaders where they earn their place, and look after technical SEO.
Ten seconds well spent. I'd add a glance at weekly downloads too. Two maintainers and millions of installs is a different risk from two maintainers and forty.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Good one, Artur. Downloads tell you how many people would have noticed by now if something was off. Forty installs means you might be the first one to find out. I'm adding that to the same ten-second look.
The test-diff check (#2) is the one I'd put first — it catches the sneaky failure mode. My addition: also grep the diff for assertion weakening, not just skips. The tell-tale pattern is toBe/toEqual downgrading to toBeTruthy/toBeDefined, or a timeout extended "to fix flakiness" — usually means the test now passes by construction.
4's "which 3 inputs would break this?" is genuinely better than asking for edge cases in the abstract. Curious whether you've seen the swallowed-catch one (#5) show up more in AI-generated Express handlers than elsewhere — that's where I keep finding catch { return [] }.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, glad the test-diff one was useful! You're right about weakened assertions. proof-of-done already flags toBe/toEqual turning into toBeTruthy/toBeDefined, here's a short demo and the code. The longer timeout "to fix flakiness" is a good point, it doesn't check that yet. Have you seen it more in Jest unit tests or in e2e tests like Playwright?
This is such a sharp, practical breakdown of the "hallucination of correctness."
Number 2 (test weakening) and Number 5 (error swallowing) are definitely the most insidious. There is nothing worse than hunting down a production bug only to realize the AI "fixed" a test by swapping a strict assertion for .toBeDefined() or wrapping a network failure in a silent catch { return []; }.
My candidate for the 8th way: The State Drift / Stale Dependency Trap. The AI writes a custom Hook or asynchronous routine that looks completely clean and passes the initial render test, but quietly misses a reactive dependency or creates a closure over stale state. It works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially.
The git diff --stat habit is pure gold. Simple, fast, and eliminates half the noise before you even start reviewing lines. Great write-up!
An eighth, from yesterday: the fix passes everything, and it never ships.
The AI fixed a bug with a new patch folder, and every test passed on my machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
A 30-second check that could help: list the new files, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines, or your package's "files" list. I only caught mine by reading the script.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you for sharing this one. The tests pass on your machine because the file is right there, and the package just never gets it. I'll think about adding a check like that to proof-of-done. Out of curiosity, is your folder list hardcoded in a shell script, or does it live in a config file? That would change how a check could read it.
Hardcoded on purpose, in a shell script: a fixed list keeps secrets like .env from shipping by accident.
We'd check the package, not the script, so it works either way. And our team always likes a human in the loop here: the check flags the file, and a human decides if it ships.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you for explaining. An allowlist is a sensible choice here: it fails safe, because anything not on the list stays out, including a stray .env. Checking the built package also keeps the check independent of how the list is maintained, whether it's a shell script or a config file. And I agree that it should flag rather than block: a new file is not always a mistake, so the final call belongs to a person.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The eighth one from baigorria is the one I'd add — a fix that passes everything and never ships, because the deploy script packs a fixed folder list and the new folder wasn't on it. No test could see it, because tests run where the file exists. That's the gap between test env and ship env, and it's the check I'd steal first: does the deploy actually include what changed?
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you Paul, always nice to see you here! I agree, the gap between where tests run and what actually ships is an easy one to miss. A test can only check what exists where it runs.
The line that the worst AI code is the kind that looks right is accurate. Crashers get noticed. Almost-right patches get merged.
The seven checks are concrete enough to use as a habit. Asking which command ran and how many tests actually executed catches the empty green. Diffing test files catches the weakened assertion. Asking which three inputs would break this forces the happy-path case into the open. Searching the diff for catch blocks that return [] turns a silent support ticket into a review item.
When teams ask Ascendra Ventures to take an AI-built prototype into production, those checks become part of the written acceptance bar. The estimate starts after the bar is clear, not before.
Which of the seven do you catch most often in practice?
the one that bites me isn't on the list: right shape, wrong value. no exception, nothing to catch, the field is filled in, it's just not what the page said. a parser can read annual as monthly across a whole layout and never once fail.
check: 5 rows at random, diffed against the source by hand. checking only the rows that already looked odd catches nothing.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
Thank you, this one should have been on the list. It's the hardest kind to catch because every check passes. Random rows is a good habit. One more that helps with parsers: compare a total from the source page (sum of a column, row count) with the same total from the parsed data. If annual was read as monthly, the totals stop matching right away. Do you run your 5-row check on every parser change, or only for new sources?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the check on number 2 is the one i would make automatic, because a green run where the test file also changed tells you nothing. we added a rule where any diff touching a test file needs a one line reason in the PR, and a test that got skipped or loosened to toBeDefined fails the review bot. the failure that still slips through for us is number 5, a catch that returns an empty array, since it passes every test written by the same agent. do you have a way to catch swallowed errors besides grepping the diff for catch?
AI can save us a lot of time, but blindly trusting its output is where things get tricky. Always review, test, and understand the code before shipping it. And if possible, use AI-powered code review tools to catch potential issues and automate parts of the review process. Never let AI be the only one approving its own work.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
AI makes coding faster, no doubt! But it's always a good idea to double-check what it generates. Use code review tools or AI reviewers to catch things you might miss. A little extra checking can save a lot of trouble later.
Software Engineer · Next.js · React · TypeScript · Three.js · Node.js · Spring Boot · MySQL · WordPress · Shopify · Claude Code · Codex · Building tools to make your daily dev work easier
The '0 tests passed is not a pass' check is the one we hit hardest in Electron work: an AI-generated IPC handler looks correct, but it never validates the renderer payload, and the integration test that should catch that is mocked away. The habit of asking 'which command ran and what was the exit code' generalizes well to desktop — especially around auto-update and crash-reporting code, where the happy path is easy to generate and the failure path (network down, partial download, stale dump) is exactly what gets skipped. Pairing each AI change with a forced failure-path test has saved us more than the happy-path tests did.
We're a place where coders share, stay up-to-date and grow their careers.
2 is the one that's bitten the most people I know: green tests that only pass because the assertion quietly changed. A candidate for #8: silent scope creep, where you ask for a one-line fix and the diff also "tidies" three unrelated files. The 30-second check is git diff --stat before reading a single line. If it touched files you didn't mention, ask why before reviewing anything else. Great list, and the version-mismatch one deserves way more attention than it gets.
Thank you, that's a good eighth. A one-line fix that quietly touches three other files is easy to miss, because each extra change looks harmless on its own. Checking
git diff --statfirst is a cheap way to see the real size of the change before reading any line. Do you see it more when the agent runs with auto-accept on, or even when you review each step?Auto-accept makes it far more likely, since nobody sees the intermediate steps. But step-by-step review doesn't fully catch it either, because each individual edit looks reasonable in isolation, like a renamed variable here or a "cleaner" import there. That's why the --stat check at the end matters: it's the only view that shows the total footprint of the change at once.
Thank you, that's a fair point. Each edit passes review on its own, so the only place the extra changes show up together is the final diff. That makes --stat useful even with step-by-step review, not only with auto-accept.
Mine is a cousin of #3. The package exists and is popular, it just pulls in a pile of dependencies for something that was fifteen lines of code. I accepted one from an agent without looking. The 30-second check: git diff --stat on the lockfile. If it grew by hundreds of lines for a small feature, ask why.
Thank you, that's a good addition to #3. The package is real, so nothing looks wrong, and the cost only shows up in the lockfile. Checking the lockfile's line count is a quick way to catch it. When this happens to you, do you usually swap the package for a few lines of your own code, or look for a smaller package that does the same job?
Depends on what it does. If it's formatting or a small helper, I write the fifteen lines myself and add a test. For dates, parsing or anything crypto I keep a well-known library even if it's heavier, because there my fifteen lines would be the bug. The smaller lookalike package is the option I trust least, a lot of them have one maintainer and no release in years.
Thank you, that's a sensible split. Your point about lookalike packages is a good one. Before I accept a package an agent suggests, I'll start checking its last release date and number of maintainers on npm. That takes about ten seconds.
Ten seconds well spent. I'd add a glance at weekly downloads too. Two maintainers and millions of installs is a different risk from two maintainers and forty.
Good one, Artur. Downloads tell you how many people would have noticed by now if something was off. Forty installs means you might be the first one to find out. I'm adding that to the same ten-second look.
The test-diff check (#2) is the one I'd put first — it catches the sneaky failure mode. My addition: also grep the diff for assertion weakening, not just skips. The tell-tale pattern is
toBe/toEqualdowngrading totoBeTruthy/toBeDefined, or a timeout extended "to fix flakiness" — usually means the test now passes by construction.4's "which 3 inputs would break this?" is genuinely better than asking for edge cases in the abstract. Curious whether you've seen the swallowed-catch one (#5) show up more in AI-generated Express handlers than elsewhere — that's where I keep finding
catch { return [] }.Thank you, glad the test-diff one was useful! You're right about weakened assertions. proof-of-done already flags
toBe/toEqualturning intotoBeTruthy/toBeDefined, here's a short demo and the code. The longer timeout "to fix flakiness" is a good point, it doesn't check that yet. Have you seen it more in Jest unit tests or in e2e tests like Playwright?This is such a sharp, practical breakdown of the "hallucination of correctness."
Number 2 (test weakening) and Number 5 (error swallowing) are definitely the most insidious. There is nothing worse than hunting down a production bug only to realize the AI "fixed" a test by swapping a strict assertion for
.toBeDefined()or wrapping a network failure in a silentcatch { return []; }.My candidate for the 8th way: The State Drift / Stale Dependency Trap. The AI writes a custom Hook or asynchronous routine that looks completely clean and passes the initial render test, but quietly misses a reactive dependency or creates a closure over stale state. It works fine during a quick manual check, but drifts out of sync the second real users start interacting with the UI sequentially.
The
git diff --stathabit is pure gold. Simple, fast, and eliminates half the noise before you even start reviewing lines. Great write-up!An eighth, from yesterday: the fix passes everything, and it never ships.
The AI fixed a bug with a new patch folder, and every test passed on my machine. But the deploy script packs a fixed list of folders, and the new one wasn't on it. No test could see that, because tests run where the file exists.
A 30-second check that could help: list the new files, then look for each one in what you ship.
git diff --name-status main | grep '^A'
Compare that with your Dockerfile COPY lines, or your package's "files" list. I only caught mine by reading the script.
Thank you for sharing this one. The tests pass on your machine because the file is right there, and the package just never gets it. I'll think about adding a check like that to proof-of-done. Out of curiosity, is your folder list hardcoded in a shell script, or does it live in a config file? That would change how a check could read it.
Hardcoded on purpose, in a shell script: a fixed list keeps secrets like .env from shipping by accident.
We'd check the package, not the script, so it works either way. And our team always likes a human in the loop here: the check flags the file, and a human decides if it ships.
Thank you for explaining. An allowlist is a sensible choice here: it fails safe, because anything not on the list stays out, including a stray .env. Checking the built package also keeps the check independent of how the list is maintained, whether it's a shell script or a config file. And I agree that it should flag rather than block: a new file is not always a mistake, so the final call belongs to a person.
The eighth one from baigorria is the one I'd add — a fix that passes everything and never ships, because the deploy script packs a fixed folder list and the new folder wasn't on it. No test could see it, because tests run where the file exists. That's the gap between test env and ship env, and it's the check I'd steal first: does the deploy actually include what changed?
Thank you Paul, always nice to see you here! I agree, the gap between where tests run and what actually ships is an easy one to miss. A test can only check what exists where it runs.
The line that the worst AI code is the kind that looks right is accurate. Crashers get noticed. Almost-right patches get merged.
The seven checks are concrete enough to use as a habit. Asking which command ran and how many tests actually executed catches the empty green. Diffing test files catches the weakened assertion. Asking which three inputs would break this forces the happy-path case into the open. Searching the diff for catch blocks that return [] turns a silent support ticket into a review item.
When teams ask Ascendra Ventures to take an AI-built prototype into production, those checks become part of the written acceptance bar. The estimate starts after the bar is clear, not before.
Which of the seven do you catch most often in practice?
the one that bites me isn't on the list: right shape, wrong value. no exception, nothing to catch, the field is filled in, it's just not what the page said. a parser can read annual as monthly across a whole layout and never once fail.
check: 5 rows at random, diffed against the source by hand. checking only the rows that already looked odd catches nothing.
Thank you, this one should have been on the list. It's the hardest kind to catch because every check passes. Random rows is a good habit. One more that helps with parsers: compare a total from the source page (sum of a column, row count) with the same total from the parsed data. If annual was read as monthly, the totals stop matching right away. Do you run your 5-row check on every parser change, or only for new sources?
the check on number 2 is the one i would make automatic, because a green run where the test file also changed tells you nothing. we added a rule where any diff touching a test file needs a one line reason in the PR, and a test that got skipped or loosened to toBeDefined fails the review bot. the failure that still slips through for us is number 5, a catch that returns an empty array, since it passes every test written by the same agent. do you have a way to catch swallowed errors besides grepping the diff for catch?
AI can save us a lot of time, but blindly trusting its output is where things get tricky. Always review, test, and understand the code before shipping it. And if possible, use AI-powered code review tools to catch potential issues and automate parts of the review process. Never let AI be the only one approving its own work.
Thank you for reading! Fully agree, the AI should never be the only one approving its own work. Which review tool do you use?
AI makes coding faster, no doubt! But it's always a good idea to double-check what it generates. Use code review tools or AI reviewers to catch things you might miss. A little extra checking can save a lot of trouble later.
Thank you! Agreed, a few minutes of checking is much cheaper than a bug in production.
The '0 tests passed is not a pass' check is the one we hit hardest in Electron work: an AI-generated IPC handler looks correct, but it never validates the renderer payload, and the integration test that should catch that is mocked away. The habit of asking 'which command ran and what was the exit code' generalizes well to desktop — especially around auto-update and crash-reporting code, where the happy path is easy to generate and the failure path (network down, partial download, stale dump) is exactly what gets skipped. Pairing each AI change with a forced failure-path test has saved us more than the happy-path tests did.