DEV Community

Hirohisa Tanaka
Hirohisa Tanaka

Posted on Edited on

Are AI-Written Tests Really Tests?

Introduction

When you delegate implementation to a Coding Agent, it often generates the tests along with the code.

The implementation is done. All the tests pass. CI is green.

...But what exactly do those tests guarantee?

import { expect, it } from 'vitest';

it('it works', () => {
  expect(true).toBe(true);
});
Enter fullscreen mode Exit fullscreen mode

This is an extreme example, but it still passes—even though it doesn't verify a single thing about the implementation.

Real tests may not be this blatantly inadequate, but they can still miss important boundary values, check only mocks, or list expected values that happen to match the current implementation. A passing test and a test you can trust are two different things.

So I built Veritas, an OSS project aimed at not trusting AI-written tests based solely on the AI's own evaluation.

Veritas aims to do more than simply generate tests. It aims to answer:

  • Which specification is a test based on?
  • What does it actually verify?
  • What changes or bugs can it detect?
  • How far has formal verification gone?
  • Conversely, what remains unknown?

The goal is to make it possible to explain all of this with supporting evidence.

This article covers what motivated the project, its design, the verification pipeline I've implemented, and the problems that even formal verification can't solve.

This article is based on the source code and development records as of November 20, 2026. Test counts and other figures are taken from execution records stored in the project; they are not the result of an independent third-party audit. Veritas is also not yet published on npm.

1. Bugs Can Remain Even When All the Tests Pass

The Veritas repository includes a fixture for verification that looks like this.

The README says:

Orders with a total of 1000 or more receive a flat discount of 100.

In other words, orders totaling 1,000 or more receive a discount of 100.

But when you look at the implementation...

export function discountAmount(total: number): number {
  if (total > 1000) {
    return 100;
  }
  return 0;
}
Enter fullscreen mode Exit fullscreen mode

It uses > instead of >=. An order totaling exactly 1,000 doesn't receive a discount.

However, the existing tests look like this:

it('gives a discount on large orders', () => {
  expect(discountAmount(2000)).toBe(100);
});

it('gives no discount on small orders', () => {
  expect(discountAmount(10)).toBe(0);
});
Enter fullscreen mode Exit fullscreen mode

Both pass. The boundary at 1000 is never checked.

The same fixture even contains a test like this:

it('pricing works', () => {
  expect(true).toBe(true);
});
Enter fullscreen mode Exit fullscreen mode

It doesn't even call the function under test.

This is why hearing that “everything passed” isn't enough to feel confident.

To solve this problem, we need to ask at least three questions:

  1. How should it behave?
  2. What can the current tests detect?
  3. What evidence supports that assessment?

2. Mutation Testing Alone Wasn't Enough

Mutation Testing is one way to examine a test suite's ability to detect changes.

For example, we can generate a change (a Mutant) like this:

- if (total > 1000) {
+ if (total >= 1000) {
Enter fullscreen mode Exit fullscreen mode

If the existing tests still pass after this change, they can't detect the difference between these two implementations.

Veritas uses mizchi/mutator as its Mutation Testing backend. Rather than building a Mutation Engine from scratch, the approach is to rely on existing tools for the things they already do well.

But this raises a fundamental question:

Is “the test detected a change” the same as “the implementation is correct according to the specification”?

No.

  • A surviving Mutant doesn't necessarily mean the original implementation has a bug.
  • A Mutant being detected doesn't necessarily mean the test's expected value is correct.
  • Equivalent Mutants also exist: changes that result in the same behavior as the original implementation.

Mutation Testing primarily tells us whether tests can distinguish changes.

We need to consider separately what behavior should be treated as correct.

3. If There's No Specification, Infer One

At first, I thought, “We can just compare the code against the specification.”

But real-world repositories don't always come with a complete specification conveniently included.

So I took inspiration from the idea behind projects like NyxFoundation/SPECA: find specifications in existing materials and turn them into verifiable properties.

Veritas can collect information such as:

  • README files, Markdown documentation, and CHANGELOGs
  • JSDoc, OpenAPI, and JSON Schema
  • Git history
  • Exported Issues and Pull Requests
  • GitHub API and web-based materials permitted by configuration

From these, it looks for statements like “the discount is 100 when the total is 1,000 or more” and generates candidate Properties that can be verified.

Veritas uses the TypeScript Compiler API for code analysis. It tracks function inputs and outputs, branches, comparison operators, boundary values, and more, then analyzes how they relate to the functions called and assertions made in tests.

However, we must not treat what's written in the code as the specification by default. The code itself may be wrong.

For example, if we observe the implementation total > 1000 and turn “an order totaling exactly 1,000 isn't eligible for a discount” into the specification, this bug will never be found.

For that reason, Veritas treats properties derived from the implementation as hypotheses.

If the README and an Issue contain conflicting statements, Veritas records the evidence for both and marks the property as conflicted rather than arbitrarily deciding by majority vote. Properties have states such as proposed, reviewed, accepted, and rejected, and human judgment is required to approve a specification.

In other words, we can infer a specification, but that doesn't prove the inferred specification is correct.

4. Don't Stop at Plausible-Sounding Findings—Find Counterexamples with Z3

Once we have candidate specifications, the next step is to compare them with the implementation.

For the discount function, one property might be:

[
\forall t \in D,\quad t \geq 1000 \Rightarrow \operatorname{discountAmount}(t)=100
]

Here, (D) is the input range being verified.

Veritas translates a supported subset of TypeScript into a formal model and uses Z3 to check whether a counterexample exists within that range.

In this case, the counterexample is total = 1000.

[
1000 \geq 1000
]

Yet the current implementation returns 0.

The important part is that after the SMT solver finds a counterexample, Veritas runs the input again in JavaScript to check whether the contradiction can actually be reproduced. This avoids declaring an implementation bug based on a counterexample that exists only in the model.

Veritas reports results using states like these:

State Meaning
counterexample_found A counterexample from the formal model was reproduced in JavaScript
proven_within_model No counterexample was found under the specified model, input range, and assumptions
semantic_mismatch The formal model and JavaScript produce different results
unknown / timeout No conclusion could be reached
unsupported The syntax or property is outside the scope of verification

In particular, proven_within_model does not mean “the correctness of the entire codebase has been proven.”

5. Why Not Prove All of TypeScript?

This is an area I felt needed especially careful handling during development.

Integers in Z3's integer model are not the same as JavaScript's number. JavaScript's number is an IEEE 754 floating-point number.

For that reason, Veritas deliberately limits what it verifies.

The current bounded formal verification supports things like:

  • Pure functions using safe integers and booleans
  • if, ternary operators, return, and const
  • +, -, *, comparison operators, and logical operators
  • Restricted record and array arguments

On the other hand, division, floating-point numbers, NaN, asynchronous operations, side effects, and other features are out of scope. The scope of the Lean backend is even narrower.

Unsupported features are never presented as supported with a successful proof.

This isn't about disguising limited functionality. For a formal verification tool, claiming to guarantee something it can't actually guarantee is more dangerous.

6. Does Adding Lean Mean It's “Completely Correct”?

Veritas also includes a verification backend that uses Lean 4.

On the Lean side, the design checks not only that proofs compile, but also that the guarantee isn't bypassed using sorry, admit, native_decide, extra axioms, or similar mechanisms.

But there's a pitfall here, too.

Lean proves a proposition expressed in Lean. Separately, we need to show that the meaning of the original TypeScript matches that proposition.

Veritas doesn't hide this gap; it treats the correspondence between the model and the implementation as an unresolved proof obligation.

For example, if the model uses mathematical integers but the original code relies on floating-point behavior, that proof must not be interpreted as a guarantee about all of JavaScript.

This is the part I didn't want to gloss over with a simple “we added formal verification!”

7. Don't Reduce Test Quality to a Single Score

Numbers like Mutation Score are easy to compare, but they can encourage the oversimplified conclusion that “a higher score means better tests.”

Veritas evaluates tests across several dimensions instead of assigning a single score, including:

  • Which specifications a test relates to (specification relevance)
  • What it asserts (assertion relevance)
  • Which Mutants it detects
  • Whether it checks boundary values
  • What evidence is available from formal verification
  • The reliability of the sources and any unresolved issues

Evaluation results are also divided into categories such as meaningful_with_evidence, weak_or_incomplete, tautological_or_disconnected, spec_conflict, and inconclusive.

Tests like expect(true).toBe(true) can be detected as tests that aren't connected to the behavior of the implementation.

The goal isn't to get a score of 100. It's to accurately show what we know and what remains unknown.

8. An Agent Improves the Tests—but Doesn't Rewrite the Correct Answer on Its Own

Veritas also includes a test improvement feature.

It proposes additional tests based on undetected Mutants and missing boundary-value checks. However, it doesn't immediately rewrite existing code.

The basic design principles are:

  1. Start by generating improvement suggestions in dry-run mode.
  2. Require explicit approval before writing changes.
  3. Only target new test files, not production code.
  4. Verify changes before and after in a temporary working copy.
  5. Roll back if Mutation Testing results don't improve.

Improvement here does not mean changing the specification just to make the tests pass.

For example, if the actual specification is “discounts apply to orders of 1,000 or more,” a failing test for an order of 1,000 is useful evidence of an implementation problem. The Agent must not delete that test or change the expected value to 0 just to turn the build green again.

Of course, improvement in Mutation Testing results alone doesn't guarantee overall quality. That's why we also look at how the tests relate to the specification and at the results of formal verification.

9. How to Use It and Current Limitations

As of November 20, 2026, Veritas is not yet published on npm. You can clone the repository to use it.

Node.js 24 or later is required. Node.js 25 or later is required for Property Testing that executes the target project's code and for replaying counterexamples (Node.js 26 is recommended). The project primarily targets TypeScript/JavaScript and Vitest.

# In the Veritas repository
pnpm install
pnpm build

# In the project being analyzed, run the Veritas CLI
node /path/to/veritas/packages/cli/dist/bin.js init --yes
node /path/to/veritas/packages/cli/dist/bin.js doctor
node /path/to/veritas/packages/cli/dist/bin.js run --profile deep --offline --include-proposed
node /path/to/veritas/packages/cli/dist/bin.js report
Enter fullscreen mode Exit fullscreen mode

--include-proposed is an option for including unapproved Properties in the analysis. It doesn't mean inferred specifications are treated as established truths.

Also, --offline limits network communication by Veritas itself; it does not guarantee that network access is isolated for tests from the target project that run during Mutation Testing. If you're verifying an untrusted repository, you'll need to provide an external isolated environment.

This limitation is especially important. A test runner executes arbitrary project code, so having safety measures inside Veritas is not the same as having a complete sandbox against malicious code.

10. What to Validate Next

Veritas has now grown to include specification discovery, mapping between specifications and tests, Mutation Testing, Property-based Testing, Z3 and Lean, improvement suggestions, a CLI, MCP, and GitHub Actions.

Still, meeting the PRD and being useful in real OSS projects are two different things.

The next step I want to take is a blind evaluation using past bugs from real OSS projects, rather than relying only on development fixtures.

  • Can it find bugs from specifications and change history without being given the answers?
  • How often does it produce false positives?
  • When specifications are ambiguous, does it avoid forcing a conclusion?
  • Can it run at a practical speed on large repositories?
  • When the Z3 or Lean model and actual JavaScript semantics differ, does it avoid making incorrect guarantees?

We're not yet at a point where we can make claims about detection rates on real OSS projects or superiority over existing products. This is something I want to evaluate by collecting data in the future.

Conclusion

AI-generated code and tests generated by the same AI.

Even when you have both and the tests are green, the question “What does this test actually verify?” remains.

Mutation Testing tells us whether tests can detect changes.

Specification inference gives us a way to reason about what the correct behavior should be.

Formal verification can rigorously check specific claims once the assumptions and model are made explicit.

None of these solves the problem on its own. But if we combine them and show the evidence and limitations without hiding them, they can provide one useful basis for evaluating tests written by Coding Agents.

Not “the tests passed,” but “what evidence do we have that we can trust those tests?”

That's the question I want to explore with Veritas.

It's still a work in progress, but I'd be glad if you took a look at the code.

Top comments (0)