DEV Community

JavaCoder7
JavaCoder7

Posted on

I Put My AI Code Reviewer Through 100 Real-World Testing Scenarios. Here's What I've Learned So Far

Building an AI code reviewer is one thing.

Trusting it with your GitHub pull requests is another.

An AI reviewer can identify a suspicious line of code, explain a possible bug, or suggest a fix. But does it understand the surrounding code? Can it distinguish harmless examples from actual security vulnerabilities? Does it recognise weak tests? And, perhaps most importantly, can we trust its recommendations?

I wanted to investigate these questions by testing my own product, CodeGuard AI, against a growing collection of real GitHub pull request scenarios.

The goal is 100 meaningful scenarios. We're currently progressing through the campaign, and the results have been more interesting than simply checking whether the AI finds bugs.

What is CodeGuard AI?

CodeGuard AI is a self-hosted AI-powered GitHub pull request reviewer built with Java 21 and Spring Boot.

It is designed to help developers review code changes by combining AI analysis with deterministic checks.

It supports multiple AI providers, including OpenAI, Gemini, Groq and Ollama.

Some of its capabilities include:

  • Repository-aware code analysis
  • Bug and security finding detection
  • Test-quality analysis
  • Behaviour impact analysis
  • Team rules and review policies
  • AI-suggested fixes
  • Patch validation
  • Review history and GitHub PR integration

Rather than relying entirely on an AI-generated opinion, CodeGuard also tries to show the evidence behind its findings.

That distinction became particularly important during testing.

How we're testing it

I created a separate GitHub test repository and started introducing controlled changes through individual pull requests.

Each scenario has a specific objective and an expected outcome.

The testing campaign covers areas such as:

  1. Test execution and result verification
  2. Bug detection
  3. Security analysis
  4. False-positive resistance
  5. Test-quality analysis
  6. Mock and assertion analysis
  7. Behaviour impact analysis
  8. Suggested fix safety
  9. Merge decision consistency
  10. Repository and GitHub integration

The idea is not to make CodeGuard look good. It is to discover where it performs well and where it needs improvement.

Some encouraging results so far

1. It can distinguish harmless SQL examples from potential security problems

One scenario contained SQL-looking text inside Java strings and Markdown documentation.

CodeGuard did not incorrectly report SQL injection vulnerabilities merely because SQL syntax appeared in the source.

The scenario completed with:

  • 10/10 review score
  • 100% AI confidence
  • 3 tests passed
  • 0 critical issues
  • No unnecessary security suggestions

This was an encouraging result for false-positive resistance.

2. Mock analysis showed promising method-level isolation

I created scenarios where one test method used Mockito while another performed an ordinary calculation using a real value.

CodeGuard correctly identified the mocked assertion without incorrectly flagging the separate real-object assertion.

We also tested the same idea across separate test classes.

These scenarios passed their execution checks and showed that the method-level and cross-class isolation worked in the tested examples.

This matters because test-quality analysis should understand the context of an individual test, not blindly treat every mock in a file as suspicious.

3. It recognised weak assertions

We introduced a test that checked only whether a calculated discount was positive:

assertTrue(result > 0);

The test could pass even if the implementation returned an incorrect discount.

CodeGuard identified the weakness and suggested an explicit expected-value assertion.

This is a useful distinction: a green test suite does not necessarily mean that the tests verify the intended behaviour.

4. It handled a correct implementation with boundary coverage

We tested a percentage calculator with normal and boundary inputs, including:

  • 20% discount
  • 100% discount
  • 0% discount
  • Invalid negative inputs
  • Percentage values above the allowed range

The corrected implementation completed with:

  • 10/10 review score
  • 100% AI confidence
  • All tests passed
  • 0 critical issues
  • Safe-to-merge recommendation

It was also important to see that CodeGuard did not invent problems in a correct implementation.

5. It distinguishes test execution outcomes

Another important area is the difference between a test passing and a test actually being verified.

During the campaign, we encountered cases involving failed assertions, timeouts, compilation errors, no tests executed and unverified test-result evidence.

CodeGuard produced different execution classifications rather than treating every exit code or build result as proof of success.

This is particularly important for an AI reviewer: it should communicate what it actually knows instead of presenting uncertainty as certainty.

The findings that reminded me why testing matters

Not every result was positive.

Some scenarios exposed important improvement areas:

  • Confirmed failing tests did not always lead to a blocking merge decision.
  • Some unverified or missing test results were associated with inconsistent merge recommendations.
  • Suggested patches occasionally weakened tests instead of improving their verification.
  • AI and deterministic analysis sometimes produced duplicate findings.
  • A GitHub repository identity mismatch caused an API request to target the wrong repository and pull request.
  • Behaviour Impact reporting showed inconsistencies between reference counts and evidence.

These are not issues I want to hide.

An AI code reviewer should not only find problems in someone else's code. Its own findings, fixes and decisions must also be tested.

The findings are being recorded for a consolidated improvement phase after the testing campaign.

What I find most important about this experiment

A useful code reviewer needs to do more than generate suggestions.

It should be able to:

  • Identify the changed code accurately.
  • Distinguish a genuine issue from harmless code.
  • Explain the evidence behind a finding.
  • Recognise when tests have not actually run.
  • Avoid weakening tests through suggested fixes.
  • Make merge recommendations consistent with the configured policy.
  • Be transparent about uncertainty.

I am treating these as product requirements, not optional enhancements.

What's next?

The target remains 100 meaningful test scenarios.

The upcoming scenarios will continue exploring security vulnerabilities, repository context, patch validation, regression detection, stale reviews, team policies and other review behaviours.

I also plan to consolidate the issues discovered so far and run regression tests after implementing fixes.

The objective is simple: build something developers can inspect, test, configure and run within their own environment.

Want to explore CodeGuard AI?

CodeGuard AI is available as a self-hosted source-code product for developers who want to explore, customise and extend an AI-powered GitHub PR review system.

If you're working with Java, Spring Boot, GitHub integrations or developer productivity tools, you may find the project interesting.

Explore CodeGuard AI:

https://javacoder716.gumroad.com/l/codeguard-ai?utm_source=devto&utm_medium=article&utm_campaign=codeguard_testing_100

I'm continuing the 100-scenario experiment and sharing what works, what fails and what gets improved along the way.

Because an AI code reviewer should earn a developer's trust through evidence, not just impressive-looking output.


What would make you trust an AI code reviewer enough to use it on your team's pull requests?

Top comments (0)