Test Coverage

Updated Aug 3, 2026

Test coverage is the share of a codebase that automated tests actually exercise. In software generated with AI the figure can be misleading, because the tests are usually produced alongside the code they check. They run, they pass, and they can still assert almost nothing about the behaviour the product depends on.

Coverage is the easiest quality signal to measure and the easiest one to misread. It answers a narrow question: when the test suite runs, which lines and branches of the code get executed. It does not answer whether anything meaningful was checked while they ran.

Why the number misleads in generated code

When a person writes a test, they usually start from an expectation: this input should produce that result, this permission should be refused. When code and tests are generated together in the same pass, the expectation can be derived from the implementation instead of from the requirement. The test then describes what the code already does, including the parts that are wrong.

The result is a suite that executes a large share of the codebase and asserts very little of consequence. Assertions check that a function returned something rather than that it returned the right thing. Error branches are called but their behaviour is never verified. Coverage climbs while the safety it implies does not exist.

What an audit checks instead

Reading the suite is more informative than reading the percentage. The questions are simple: which assertions would actually fail if the behaviour changed, are the paths that handle money, permissions and personal data covered by anything specific, and do the tests still pass when a deliberate fault is introduced.

That last check is the fastest way to tell a real suite from a decorative one. It also shows which parts of the product can be changed safely, which is the practical output of a code audit: knowing where you can move quickly and where you cannot.

Coverage as a starting point

Low coverage in a young product is normal and rarely the biggest risk. Untrustworthy coverage is worse than none, because teams act on it. The realistic goal is not a higher percentage but a small set of tests that genuinely protect the behaviour the business depends on, and an honest picture of everything else. The same discipline applies to dependency risk, where the surface is also larger than it looks, and it is a large part of what separates vibe coding from agentic engineering.