Testing AI-generated code beyond the green checkmark

A green checkmark proves that a command completed successfully. The useful engineering question is what that command actually challenged. Generated tests can agree perfectly with generated code while both miss the requirement.

Use a tiny example: a required display name must contain something other than whitespace. A truthiness check looks plausible and accepts an ordinary name. It also accepts a string containing one space.

Make the counterexample executable

1
const plausible = value => Boolean(value)
2
const validName = value =>
3
  typeof value === 'string' && value.trim().length > 0
4
 
5
console.assert(plausible(' ') === true)
6
console.assert(validName(' ') === false)
7
console.assert(validName('Ada') === true)

The repository's editorial example tests run these cases, including empty strings, null, and a numeric input. This is a reproducible synthetic example. It is not a complete international name policy: length, normalization, persistence, and display escaping are separate requirements.

Visual field note / V09

From requirement to evidence

Read the diagram as text

Flow from written behavior and counterexample, then pure function tests, then database and api integration, then browser journey, then independent human review, then operational assumptions still open. Start with an independent requirement, then gather unit, integration, browser, and review evidence for different failure boundaries.

Open full-size SVG ↗Download PNG ↓

Put each question at the right boundary

A pure function test is enough to expose this guard. It is not enough to prove that the server uses it. An integration test should submit invalid input through the real handler and verify that no row is created. A browser test should verify that the person sees a useful message and can correct the input.

For a retryable form, add another question: can two concurrent requests create two records? That needs the actual persistence boundary and its uniqueness rule. Mocking the insert function would remove the behavior under examination.

Visual field note / V10

What a passing test does not prove

Read the diagram as text

Unit / local rule: Whitespace must fail a name guard. Cannot prove database atomicity.; Integration / shared state: Concurrent retries create one record. Cannot prove the visible error flow.; Browser / user journey: User sees a recoverable error. Cannot prove production delivery. A local rule, shared-state invariant, and visible user journey need different tests; none alone proves production delivery.

Open full-size SVG ↗Download PNG ↓

Ask what the test cannot see

A browser test using a fake message transport cannot demonstrate delivery to a real mailbox. That is a useful isolation choice, provided the result is described accurately. Likewise, passing local tests does not prove production secrets, provider configuration, or incident response.

Tests should fail for a meaningful reason when the behavior is wrong. Temporarily substitute the truthiness implementation and confirm the whitespace case fails. If it remains green, the suite is exercising the wrong path or asserting too little. Keep this mutation local; there is no reason to ship the known defect.

The tradeoff is maintenance. Testing every helper through the browser creates slow, fragile feedback. Testing every boundary with mocks creates confidence without evidence. Choose the smallest real boundary that contains the failure, then keep a few complete user journeys.

Playwright's page API provides the browser interaction surface; assertions still need a product requirement independent of the implementation. For shared state, continue with duplicate-safe workflows. If you want an engineer who treats tests as evidence rather than ceremony, start a hiring conversation.