20 months on a platform for real-time quality analysis of video streams

Agents write tests faster than we measure them

· 7 min read

Six months of daily agent use took our backend from 311 test methods to 2,198. Nobody could say whether that helped. There was no coverage number, no static analysis, and nothing in the build that could fail on quality.

The same shape as earlier in this series. Throughput doubled and change-failure rate doubled with it, and what moved the failure rate back down was a tool rather than a rule. A quality gate is that tool, one level up.

TL;DR:

  • The tests were not the problem. They assert, and they run on every pull request. Two cheap checks confirmed that in an afternoon.
  • The gap was the gate. No coverage tool, no static analyser, and the end-to-end suite ran on a nightly timer, so it could never block a merge.
  • Test count stopped being evidence. It used to proxy for effort spent. An agent removes the effort and leaves the count.
  • The fix is old, standard, and a loop rather than a tool. Pick a measure, run it in CI, make it block the merge, raise its floor with every commit.
  • It cost one day in our other repo. Line coverage went 64.78% to 79.53%, and the floor has not dropped since.

Six months, seven times the tests

In December 2025 the backend had 311 test methods. By May 2026 it had 2,198. The code under them grew too, from 149 source files to 399.

So the tests multiplied by seven and the code by under three. On its own that looks like good news.

It is not news at all. A test used to cost enough that counting tests told you something about the care taken. An agent removes that cost. The count survives and the meaning drains out of it.

The question I could not answer

Ask how much of the system is actually protected, and this repo had nothing to answer with.

A quality gate is the standard answer, and it long predates any of this. The build computes a few numbers about the change, compares them against thresholds, and fails when they are not met. Fowler’s continuous integration article says best why that check belongs in the build, not in a review habit.

The measures worth gating on are well documented and boring, which is the point.

  • Line and branch coverage. JaCoCo on the JVM, and most ecosystems ship an equivalent. Read Fowler’s note on test coverage first: it finds untested code, and it does not prove quality.
  • Static analysis. Duplication, cyclomatic complexity, and the security and maintainability rules a tool like SonarQube runs by default. Its quality gate is the packaged version of this whole article.
  • Mutation score, once coverage stops moving. Mutation testing breaks the code on purpose and asks whether any test objects. It is the expensive one, so it comes last.

We had none of them. Not a low score: no score, on any day, for the life of the repository.

With nothing and no time, two throwaway scripts give you a shape. I counted test methods with no assertion, at 0.8%, and classes with no test class naming them, at 53.6%. Both are hacks. They told me the tests were real and the coverage thin, then their job was done.

Running is not gating

A test that runs is not a test that stops you. That distinction hid the problem, because on paper our tests ran.

They did run. The pull-request check fires on every pull request to main and executes the suite with skipping explicitly disabled.

The end-to-end suite was not. All three of its workflows trigger on a manual click or a nightly cron, never on a pull request. So a broken end-to-end test could not block a merge, only be found next morning by whoever looked. The hand-run deploy takes a skip-tests input defaulting to true.

Diagram: three pipeline stages left to right. Tests run, then an empty dashed box labelled quality gate and marked missing, then merge.

The stage that was never there. Tests ran, and nothing between them and the merge could fail on quality.

None of that shows up in a test count. All of it shows up in a gate.

The gate next door

Our frontend repo had the same agents writing the same kind of tests, and it had the number.

Coverage was reported there from the first CI commit, in May 2025. Reported, not enforced. For twelve months the percentage was produced on every run, uploaded as an artifact, and read by whoever cared to click. It changed nothing.

Then it became a gate, and that took a day.

Chart: over fourteen commits, measured line coverage rises from 64.78% to 75.64% while the enforced floor steps up beneath it, from 64% to 75%, then to 79% on merge.

Fourteen commits, one day. The floor rises in the same commit that earns it, so the number can never quietly fall back.

The merged branch measured 79.53% lines, 89.89% branches and 87.63% functions, and each threshold was set just under its measurement. That is the whole mechanism, and it is the part worth copying.

The gate was mine. The reporting that came a year before it was not.

The loop

Four steps. Which measure you pick matters less than running all four.

1. Pick a measure

Any documented one beats an invented one. Coverage is the usual first pick, because the tooling is everywhere and the number is comparable across teams. Read what it does not tell you before adopting it, and expect to add a second measure later rather than perfecting the first.

2. Run it in CI

On every pull request, not nightly and not on a button. A nightly run tells you what broke yesterday. Check the trigger conditions rather than trusting a workflow’s name: ours said tests and meant a cron.

3. Gate it

The number has to be able to fail the build, or it is a dashboard. Twelve months of reporting moved nothing for us, and one day of enforcing moved fifteen points. Scope the threshold to the suite that can meet it and turn it off explicitly elsewhere, so every exception lives in the configuration rather than in someone’s memory.

4. Raise the floor on each commit

Set the minimum at whatever you measure today, then move it up in the same change that earns the improvement. Nobody has to clear a large target in one push, and nothing can regress. SonarSource packages this as clean as you code. Write the measured number into the commit message as well: build artifacts expire, and commit messages are why I can still read our sequence a year later.

Conclusion

An agent will write every test you ask for. Picking a metric is different from measuring it, and measuring it is different from mandating it.

What would change my mind: this argues the loop matters more than the measure. If a team runs all four steps for a quarter and their escaped defects do not move, the gate was ceremony. Something else was doing the work.

Next in this series: the pipeline those tests ran against, and the one assumption every stage of it had baked in.

Methodology & limitations (click to expand)

Data sources

  • Test-suite growth: a monthly snapshot of test and source file counts from the repository tree, December 2025 to May 2026.
  • Assertion scan, workflow triggers and the coverage sequence: local clones of both repositories at the last commit before access ended on 22 May 2026.

Definitions and bounds

  • Test method counts @Test, @ParameterizedTest and @RepeatedTest. The local scan found 2,100 and 98 of the first two, totalling 2,198, which matches the monthly snapshot exactly.
  • The two throwaway measures are deliberately crude. Assertion detection follows called helpers two levels deep and still only sets a floor. The naming measure counts a class as uncovered when no test class name contains it, so anything exercised indirectly reads as a miss.
  • The repository holds eight agent worktrees, each a full copy of the source. Counting them inflated the first scan roughly ninefold. All figures here exclude them.

What this does not show

  • No coverage, duplication or complexity figures for the backend, because none were ever produced.
  • Whether the pull-request check was a required check is not visible in the workflow files. Tests running on a pull request is confirmed; branch protection is not.
  • Disabled tests are counted. A method annotated as skipped still appears in the total.