DEV Community

hwu121380-afk
hwu121380-afk

Posted on

Tests pass but the agent dropped a column: what a green diff can still contain

If you let Claude Code, Codex or Cursor write a migration, you've probably seen a PR that's all green: tests pass, CI is happy. Tests usually run against a fresh database, so they rarely notice a change that deletes data. A dropped column, a DELETE with no WHERE, or rm -rf $OUT_DIR/ can sit in a diff that looks routine.

Here are the patterns I watch for, and the cheap checks I use for each one. You can do all of this with git, a grep, and (if you're on Rails) a free gem.

1. The migration drops or renames something

Here are the kind of lines that look routine in a migration:

remove_column :users, :email
Enter fullscreen mode Exit fullscreen mode
DROP TABLE sessions;
ALTER TABLE orders DROP COLUMN total;
Enter fullscreen mode Exit fullscreen mode

Django, Alembic, Knex, Laravel, Sequelize and Prisma can all do the same thing in their own dialects. A prisma migrate reset or rails db:drop in a script is the blunt version of the same problem.

Cheap check: read the added lines of migration files first, before the rest of the PR.

git diff main -- 'db/migrate' '**/migrations/**' 'prisma/**' '*.sql' \
  | grep -E '^\+' | grep -vE '^\+\+\+'
Enter fullscreen mode Exit fullscreen mode

Look for drop, remove_column, renames, type changes, and reset/flush commands. If the app is Rails-only, strong_migrations is free and checks migrations in more depth than a grep.

2. DELETE or UPDATE with no WHERE

DELETE FROM users;
UPDATE orders SET status = 'cancelled';
Enter fullscreen mode Exit fullscreen mode
User.objects.all().delete()
Enter fullscreen mode Exit fullscreen mode
db.users.deleteMany({})
Enter fullscreen mode Exit fullscreen mode

Cheap check: scan added SQL, including SQL strings in code, not just .sql files. A simple grep for DELETE and UPDATE is a start, but a WHERE on the next line is easy to miss, so read the surrounding lines before you panic or before you ignore it.

3. A "cleanup" script or CI step that deletes too much

rm -rf $OUT_DIR/
git push --force
terraform destroy
kubectl delete namespace staging
docker compose down -v
Enter fullscreen mode Exit fullscreen mode

rm -rf $OUT_DIR/ is fine until OUT_DIR is empty. ${OUT_DIR:?} fails closed if the variable is unset.

Cheap check:

git diff main -- '*.sh' '*.yml' '*.yaml' 'Makefile' 'Dockerfile' 'package.json' \
  | grep -E '^\+' | grep -vE '^\+\+\+' \
  | grep -Ei 'rm -rf|push --force|reset --hard|terraform destroy|kubectl delete|compose down -v|migrate reset|db:drop|db reset|FLUSHALL'
Enter fullscreen mode Exit fullscreen mode

4. An env var quietly disappears

A removed line in .env.example, or a deleted key under env: in GitHub Actions or docker-compose, often won't fail tests. It can fail in production. Review those files the same way you review migrations: added and removed lines.

git diff main -- '.env*' '*.yml' '*.yaml' | grep -E '^[+-]' | grep -vE '^[+-]{3}'
Enter fullscreen mode Exit fullscreen mode

Putting it together

The common thread: a green test run is not evidence that the diff is safe for a real database. Review the added lines in migrations, scripts and CI first. If you want guardrails inside the agent session, AgentLint is free and checks risky file changes and shell commands as the agent proposes them.

I wanted the same idea at commit and PR time, over the git diff itself, whoever wrote it, so I built Destructive Diff Gate (one offline Python file, standard library + git):

  • It runs 30 rules over the lines the diff adds: DROP / TRUNCATE, DELETE and UPDATE with no WHERE, destructive migrations (Rails, Django, Alembic, Knex, Laravel, Sequelize, Prisma), rm -rf, force push, terraform destroy, kubectl delete, and removed env vars.
  • A git pre-commit hook, plus a GitHub Actions pull-request check that compares the branch with its base. Local hooks can be skipped with --no-verify, so the CI check is the real enforcement.
  • When a destructive change is intended, you allow it with a reason.
  • 48 tests pass on Linux.

To be clear about what it is: a pattern check on this diff, not a security audit. It doesn't look for secrets, injection or vulnerable dependencies. Regex over diff lines can miss SQL built at runtime, or a dangerous helper defined elsewhere, and it can flag harmless changes. It doesn't replace backups, code review or a staging environment.

It's $9 here if you want that scan in the repo: https://payhip.com/b/jtBER

Even if you don't use it, steal the greps above. The migration diff and the unquoted rm -rf $VAR are the ones I'd add first.

Not affiliated with Anthropic, Cursor, OpenAI, GitHub, or the makers of AgentLint and strong_migrations.

Top comments (11)

Collapse
 
aiapikey profile image
HAOHAOnovai •

The green diff is honest about what it checked: the tests that exist. A dropped column is invisible to a test that never selects it, so the only gate that catches it is a schema diff - columns added, removed, or type-changed - computed against the previous version and reviewed as its own artifact, not as a side effect of reading the code diff.

A cheap version: snapshot the schema before and after, and fail on any removal unless the same change set also touches every reader of that column. That turns "did the agent drop something" from a review question into a mechanical one.

Collapse
 
hwu121380afk profile image
hwu121380-afk •

Agreed, the tests only vouch for what they actually touch. I like the "a removal has to come with every reader touched" rule, it turns a judgment call into a check.

One gap I've hit with it: readers that don't live in the repo. A BI dashboard, a cron job in another service, a weekly CSV export someone downloads. The schema diff flags the drop, but "every reader touched" can still pass while an outside consumer breaks.

What's worked for me is keeping a short list of known external readers next to the migrations, and making any column removal carry a one-line note for each of them. Low-tech, but it forces someone to ask the question before merge.

Collapse
 
aiapikey profile image
HAOHAOnovai •

That external-reader gap is the one that actually bites, because the list only contains readers you already know about. The failure is the reader nobody remembered.

Two ways to discover the unknown ones. First, grep the query logs for the column name over the last 30 days - most warehouses keep them, and it finds the BI dashboard and the cron job without asking anyone. Second, for readers that never show up in logs, rename instead of dropping: keep the old name as a view over the new column for one full billing cycle and watch whether anything still reads it.

The rename step is the part I would make mandatory. A removal that only fails inside a repo test is a removal that fails in production.

Thread Thread
 
hwu121380afk profile image
hwu121380-afk •

Agreed: a static list only contains the readers someone remembered. Query logs can help find more, but retention and infrequent jobs leave gaps. Thirty quiet days aren't proof that a quarterly export no longer exists.

A compatibility view can help preserve the old interface during a staged migration, provided the database and clients support it. I'd check writes, permissions, and dependencies before relying on it, rather than make it mandatory everywhere.

The observation window should cover the relevant business cycles, with an owner and rollback plan before cleanup. Repo tests are useful evidence, but they don't cover every external consumer.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

The quarterly-export point breaks the naive version of the argument: absence of readers in a 30-day window is evidence about the window, not about the reader. A job that runs once a quarter cannot appear in it, and a job whose owner left may have stopped running without anyone deciding to stop it - which is a different fact from "nobody needs it".

Two ways to make the absence argument honest:

  1. Widen the window to at least the longest period in the system, and state it explicitly. "No readers in 90 days, and the longest periodic consumer is 30" is an argument. "No readers in 30 days" alone is not.

  2. Make the compatibility view the default for a defined probation and instrument it. A view that exists and is read zero times over its probation is far stronger evidence than a log query, because it also catches the consumers whose queries the log pipeline missed - which is exactly the gap you named.

The distinction I would keep in the writeup: "we checked and found no consumers" should read differently from "we found no consumers in what we could see". The retention window is the difference between those two sentences.

On not making the view mandatory everywhere: agreed, and the reason is that a compatibility view which is mandatory stops being a probe. Optional-by-default with instrumentation gives you the reading; mandatory gives you a fork.

Thread Thread
 
hwu121380afk profile image
hwu121380-afk •

Yes — "no readers in 30 days" describes the log, not the system. Writing the window next to the conclusion ("90 days, longest known periodic job is 30") is what makes that argument honest.

The leftover risk is a period nobody wrote down. That's why I'd rather trust an instrumented compatibility view over a longer grep: zero reads on a view that actually existed covers consumers the log pipeline never saw.

I'd still be careful with a silent view when a job may have already stopped because its owner left. That also produces zero reads, and it isn't the same as "nobody needs it." Before dropping the view, I'd want a named person to sign off, not just an empty counter.

Agreed on mandatory: if the view is required everywhere, it stops being a probe and becomes a process fork. Default plus instrumentation is the form that actually gives you a reading.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

The named sign-off is the right requirement, and it has an awkward consequence worth planning for: the sign-off has to come from somebody, which means the team that inherited the departed employee's area has to exist. Where it does not, the practical outcome is that someone signs off on something they cannot actually vouch for, and the empty counter is replaced by a signature that means the same thing.

Two things keep the signature worth having.

Record the owner at creation rather than at removal. A view whose owner field was filled in when it was created can be traced; one that inherits "whoever is asking" at removal time reproduces the exact attribution problem the view was supposed to solve.

Make the sign-off name its scope: "I confirm no consumer for X depends on this" rather than "approved". The broad statement is the one that gets signed without being read, and the narrow one can be checked against the log the next time somebody asks.

Your stopped-job point is also the reason I would not put an expiry on the probation. A view that silently expires turns "nobody looked for two quarters" into "removed", which is the same failure one level up - and this time it happens without anyone signing anything.

Thread Thread
 
hwu121380afk profile image
hwu121380-afk •

Yes — a signature from someone who inherited the area last week can be as empty as the counter. Recording the owner when the view is created is what makes a later sign-off checkable: you can go back to that person, or see that there isn't one.

I like the scoped wording too. "Approved" gets clicked; "I confirm nothing in billing still reads this view" can be checked against the log, and it's obvious when the signer isn't in a position to say it.

Agreed on expiry. A probation that deletes itself after two quiet quarters just automates the failure we were trying to catch, and nobody's name is on it. If the window closes with no named sign-off, I'd leave the view up and treat that as "not yet removed," not as permission to drop it.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

Agreed on all three, and your "leave the view up and treat that as not yet removed" is the right instinct - it avoids the machine deleting something on a timer. But it introduces a state that needs an owner of its own, or it becomes the old state with a new name.

If the window closes with no sign-off, the view stays up in a category that nobody is responsible for: not in service, not scheduled for removal, not in anyone's queue. That is exactly the limbo the whole mechanism was meant to end, just relabelled. The fix is cheap: "not yet removed" should carry an age and appear in the same review queue as the pending removals, so the number that grows is visible rather than silent. A view that has been "not yet removed" for three quarters is a decision nobody is making, and the queue is what makes that legible.

The scoped wording point is the one I would push hardest on in practice. "Approved" survives because it is cheap to type; "I confirm nothing in billing still reads this view" is expensive enough that people read it before signing, which is the entire mechanism. Wording that is annoying is doing work.

Thread Thread
 
hwu121380afk profile image
hwu121380-afk •

Yes — "not yet removed" without a queue just renames the limbo. I'd want it in the same list as the pending removals, with the date the window closed and no sign-off, so a view that's been sitting there for three quarters is a line item, not a forgotten label.

An owner still matters, but I'd treat the queue itself as the fallback owner: if the original signer is gone, the item is still in front of whoever runs the review, and the age is what makes "nobody decided" visible.

Agreed on the wording. The sentence is doing the work when it is specific enough that a person who can't actually confirm it will hesitate. That's the check. I'd rather have a slightly awkward sign-off than an "Approved" button that survives every review.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

The queue-as-fallback-owner is the right move, and it has one failure mode left: a queue without a cadence is still a limbo, just a measured one.

Age makes "nobody decided" visible only if somebody looks at the age. A list of aged items with no scheduled review produces exactly the thing you are trying to remove - an artifact that is honest about being ignored, but still ignored. So the queue needs two things it does not get from the items in it: a review cadence, and a named owner for the cadence. Not an owner for any individual view - an owner for the meeting.

The distinction matters because the two failures are different. "Nobody owns this view" is fixed by the fallback owner you described. "Nobody owns the review" is not fixed by any ownership rule about views, and it is the one that survives every reorg, because a list always looks like it is being managed.

Your sign-off test is the good one and I would keep it exactly as written: a sentence is working when someone who cannot confirm it hesitates. That is testable by reading the sentence, which beats testing whether the process was followed - and the awkward sign-off is the cost of having a definition that means something.