Reviewing Tests It Wrote

An agent writes tests quickly and in quantity, and many of them would pass whatever the code did. Three kinds account for most of those: tests that restate the implementation, tests that cannot fail, and tests that test the mock. Reviewing a test comes down to one question: would this fail if the behaviour were wrong?

The question matters more with an agent than with a colleague, because an agent's tests often arrive in the same diff as the code they test. They look like evidence, they add to the count of passing tests, and they were written by the same process that wrote the bug.

Tests That Restate the Implementation

The commonest weak test computes its expected value with the code's own formula. To check the line the cancellation screen shows, it calls the deadline function, formats the result, and asserts that the screen shows the same thing. When the formula is wrong, the test is wrong with it, and the two agree perfectly.

A good test states the expected value as a literal taken from the rule: for a 09:00 class on Saturday, the screen says free cancellation until 21:00 on Friday. Below, both tests pass on the correct deadline code. On the deadline rewritten in the server's time zone, the one that took its answer from the rule fails, and the one that took its answer from the code passes.

def test_notice_matches_deadline():
    # Mirrored: the expected value comes from the code under test.
    until = refunds.refund_deadline(CLASS.starts_at, CLASS.gym)
    shown = schedule.as_shown(until, SECOND_ZONE.time_zone)
    assert cancellation_notice(CLASS) == f"Free cancellation until {shown}"


def test_notice_says_21_00_the_evening_before():
    # Literal: the expected value comes from the rule.
    assert cancellation_notice(CLASS) == "Free cancellation until 2026-11-13 21:00"

PASSED tests/test_notice.py::test_notice_matches_deadline
PASSED tests/test_notice.py::test_notice_says_21_00_the_evening_before
2 passed in 0.07s

PASSED tests/test_notice.py::test_notice_matches_deadline
FAILED tests/test_notice.py::test_notice_says_21_00_the_evening_before - Asse...
1 failed, 1 passed in 0.05s

test("notice matches deadline", () => {
  // Mirrored: the expected value comes from the code under test.
  const until = refunds.refundDeadline(CLASS.startsAt, CLASS.gym);
  const shown = schedule.asShown(until, CLASS.gym.timeZone);
  expect(cancellationNotice(CLASS)).toBe(`Free cancellation until ${shown}`);
});

test("notice says 21:00 the evening before", () => {
  // Literal: the expected value comes from the rule.
  expect(cancellationNotice(CLASS)).toBe("Free cancellation until 2026-11-13 21:00");
});

      Tests  2 passed (2)

   × notice says 21:00 the evening before 3ms
Expected: "Free cancellation until 2026-11-13 21:00"
Received: "Free cancellation until 2026-11-13 20:00"
      Tests  1 failed | 1 passed (2)

@Test
void noticeMatchesDeadline() {
  // Mirrored: the expected value comes from the code under test.
  ZonedDateTime until = Refunds.refundDeadline(CLASS.startsAt(), CLASS.gym());
  String shown = Schedule.asShown(until, SECOND_ZONE.timeZone());
  assertEquals("Free cancellation until " + shown, CLASS.cancellationNotice());
}

@Test
void noticeSays2100TheEveningBefore() {
  // Literal: the expected value comes from the rule.
  assertEquals("Free cancellation until 2026-11-13 21:00", CLASS.cancellationNotice());
}

│  └─ NoticeTest ✔
│     ├─ noticeSays2100TheEveningBefore() ✔
│     └─ noticeMatchesDeadline() ✔

│  └─ NoticeTest ✔
│     ├─ noticeSays2100TheEveningBefore() ✘ expected: <Free cancellation until 2026-11-13 21:00> but was: <Free cancellation until 2026-11-13 20:00>
│     └─ noticeMatchesDeadline() ✔

The mirrored test is not useless in every sense: it would catch the screen formatting the date wrongly. But it cannot catch the deadline being wrong, which is the thing that matters to a member, and its name, "matches deadline", says as much to anyone who reads it as a rule.

Tests That Cannot Fail

Some tests pass against any implementation at all. A test with no assertion. An assertion inside a loop over a list that turns out to be empty. An assertion inside a block that catches its own failure. An assertion that the result merely exists, on a function that always returns something. An expected value read from the code's own output during the test. Each of these runs, reports a pass, and checks nothing.

They are easy to write by accident and easy for an agent to write on purpose, because a test that cannot fail is the fastest way to satisfy "add tests for this". They are also easy to find once you look for them: read the assertion first, and ask what value of the result would make it fail.

The refund diff's own tests for the policy object include one of these. It builds a policy from a 12-hour window and asserts that the policy's window is 12 hours, which checks that a value passed in is the value read back. No mistake in how a refund is decided can make it fail. It counts as one more passing test in the summary, and the summary is where its only effect is felt.

Tests That Test the Mock

The mock returns 12 and the test asserts that the result is 12, and nothing in between ran. This happens when the unit under test is itself replaced by a mock, or when every collaborator is mocked so thoroughly that the code under test only passes values from one mock to the assertion.

Mocks belong at boundaries the test does not own: the payment provider, the email service, the clock. They never belong around the logic under test. A refund test that mocks the deadline calculation is not a refund test; it is a test that the mock was configured correctly.

The tell is in the setup. If the arrange section of a test configures a mock to return the very value the assert section expects, read the code between them: if it only passes the value along, the test is checking the plumbing, and the logic it was named after never ran.

The Mutation Check

The quickest way to review a test suite is to break the code on purpose and see what fails. Flip "at least" to "more than", the boundary flip from Chapter 6, which is a ready-made mutant. Change 12 to 13. Delete the time-zone conversion. Change the round-down to rounding to the nearest cent. A suite that stays green on a mutant does not cover that rule, however many tests it has.

Doing this by hand once per change takes a minute: make the edit, run the targeted tests, undo the edit. Mutation-testing tools automate it in every language, generating hundreds of small changes and reporting which ones survive. Either way, a surviving mutant is a precise finding: this rule has no test.

Volume Is Not Coverage

Asked to "test the refund rule thoroughly", an agent may produce 30 tests, the number this page uses, that cancel at 30 different hours from 13 to 42 hours ahead. They all pass, they all check the same plain case, and none of them checks the boundary, the second zone, a pack or the line the member reads.

List the rules a test file pins instead of counting its tests. For the refund module the list must include the boundary at exactly 12 hours, both time zones, pack refunds and the member-facing text. Thirty tests of one rule cover one rule. Four tests of four rules cover four.

Agents produce volume because volume is what "thorough" looks like in the training data, and because each extra test is cheap for them to write. It is not cheap for you. Every test is code someone will read when it fails, update when the rule changes, and wait for on every run. A file of 30 near-identical tests also hides the one that matters, which is the reverse of what a test file is for.

Reading a Test File

Read the names first: each one should state a rule a member or Ines would recognise. Then the fixtures and mocks, which tell you what is real and what is replaced. Then the assertions, which tell you what is checked. Then the setup. A test whose name cannot be read as a rule usually checks nothing in particular, and a file of such tests is volume without a contract.

Good names make the rule list free. "Exactly 12 hours ahead is refunded", "deadline is 21:00 local in each zone", "one class of a pack refunds a third, rounded down": a reviewer can read the names alone and see which rules are pinned and which are missing. "Test refund 1", "test refund 2" and "test policy" say nothing, and a reviewer has to read every body to find out.