Reproduce First
A bug you cannot reproduce is a bug you cannot know you fixed. With an agent in the loop the rule gets stricter: no fix before a failing test that shows the bug. Without one, the agent produces a plausible fix, the symptom stays away for a week, and nobody can say whether that means anything.
Cragmoor has exactly that bug. Kai's waitlist from Chapter 2 still sends one freed place to two climbers now and then, and a fix has already shipped once without changing anything that matters. This chapter fixes it properly, and this page is the step that makes every later step possible: a test that fails on demand, for the right reason, every single run.
The Report
Three times in two months, two members received an offer for the same place in a full class. The logs show both offers. Nobody can make it happen on demand: you can cancel a booking in a full class of 16 on staging a hundred times and see one offer every time.
After the first incident, in the third week, Kai asked the agent to "fix the double offer". It wrote a guard, a two-second pause before the 30-second sweep offers a place that a cancellation has just freed, and Kai merged it. The guard has the step-around shape from Chapter 7: it makes the collision rarer and leaves the two separate steps exactly where they were. The double offer has happened twice since. The team now has a fix that did not fix anything, and no way to tell whether the next one will.
Why the Repro Comes First
A failing test that shows the bug does three jobs. Before the fix, it is proof that someone understands the cause well enough to trigger it. During the fix, it judges every proposed change: red means the change did not work, whatever the explanation says. After the fix, it stays in the suite as the regression test that stops the bug coming back in a refactoring two years from now.
The agent changes the stakes. Asked to fix a bug it cannot observe, an agent aims for a change that looks like a fix, because looking right is the only check available inside its loop. Given a failing test, it has a real check: run the test, read the result, try again. The agent loop from Chapter 1 works only as well as the signal it observes, and a red test is the strongest signal you can hand it.
Asking for the Repro, Not the Fix
So the first request is not "fix the double offer". It is narrower and it forbids the thing the agent most wants to do: write a test that produces two offers for one freed place, and do not change production code. The second clause matters. Without it, an agent that finds a likely cause will often fix it in the same diff, and you lose the moment when the test was red against the old code.
The test it writes is reviewed like any test the agent wrote, with the questions from Chapter 8. Does it fail on the assertion "exactly one offer per freed place", and does it fail for that reason, rather than on a setup error or a timeout? A test that fails because the fixture is wrong is red for the wrong reason, and a fix that turns it green proves nothing about offers.
From Flaky to Deterministic
The first version above starts the cancellation and the sweep on two threads and hopes they collide. You run it a hundred times and it fails three times. That is a hint, not a reproduction. A test that fails 3 runs in 100 cannot judge a fix: after the fix, 20 green runs in a row would happen 54% of the time with the bug still there, because 0.97 to the power of 20 is about 0.54.
The way out is to stop hoping for the interleaving and drive it. The bug needs only both reads to come before either write: the cancellation reads "one free place, no active offer", the sweep reads the same thing, and whichever path writes first offers the place to the first person waiting, the other to the person after, because the first one now has an offer. A test double for the storage layer can force that order. It lets each path read, then holds it until the other path has read too. Neither path can write before both have read.
Below is the waitlist code and the step-driven test in each language. The production code is the same as it was, with Kai's two-second pause left out of the listing because the test does not depend on it: read the state, then write the offer, as two calls to storage. The test's storage double adds two holds. The sweep cannot list classes until the cancellation has freed the place, and every read waits until the other path has also read. Then the test asserts the rule, one offer per freed place, and fails with two.
import logging
from dataclasses import dataclass
log = logging.getLogger("waitlist")
@dataclass
class ClassState:
free_places: int
active_offers: int
def offer_free_place(store, class_id, path):
state = store.read_state(class_id) # step 1: read
if state.free_places < 1 or state.active_offers > 0:
return None
member_id = store.offer_to_next(class_id) # step 2: write
if member_id is not None:
log.info("offer class=%s member=%s path=%s free_places=%d active_offers=%d",
class_id, member_id, path, state.free_places, state.active_offers)
return member_id
def cancel_booking(store, class_id, member_id):
store.free_place(class_id, member_id)
return offer_free_place(store, class_id, path="cancel")
def sweep(store):
for class_id in store.classes_with_free_places():
offer_free_place(store, class_id, path="sweep")import threading
from memory_store import MemoryStore
from waitlist import cancel_booking, sweep
class SteppedStore(MemoryStore):
"""Drives the interleaving: both paths read before either writes."""
def __init__(self):
super().__init__()
self.place_freed = threading.Event()
self.both_read = threading.Barrier(2, timeout=5)
def free_place(self, class_id, member_id):
super().free_place(class_id, member_id)
self.place_freed.set() # now the sweep can find the place
def classes_with_free_places(self):
self.place_freed.wait(timeout=5)
return super().classes_with_free_places()
def read_state(self, class_id):
state = super().read_state(class_id)
self.both_read.wait() # hold until the other path has read
return state
def test_one_offer_per_freed_place():
store = SteppedStore()
climbers = [f"M{n}" for n in range(1, 17)] # a full class of 16
store.add_class("C42", capacity=16, booked=climbers, waiting=["W1", "W2", "W3"])
paths = [threading.Thread(target=cancel_booking, args=(store, "C42", "M7")),
threading.Thread(target=sweep, args=(store,))]
for p in paths: p.start()
for p in paths: p.join()
assert len(store.offers["C42"]) == 1 # one offer per freed place> assert len(store.offers["C42"]) == 1 # one offer per freed place
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E AssertionError: assert 2 == 1
E + where 2 = len(['W1', 'W2'])
test_double_offer.py:39: AssertionError
=========================== short test summary info ============================
FAILED test_double_offer.py::test_one_offer_per_freed_place - AssertionError:...
============================== 1 failed in 0.08s ===============================export interface ClassState { freePlaces: number; activeOffers: number }
export interface Storage {
freePlace(classId: string, memberId: string): Promise<void>;
readState(classId: string): Promise<ClassState>;
offerToNext(classId: string): Promise<string | undefined>;
classesWithFreePlaces(): Promise<string[]>;
}
export async function offerFreePlace(store: Storage, classId: string, path: string) {
const state = await store.readState(classId); // step 1: read
if (state.freePlaces < 1 || state.activeOffers > 0) return undefined;
const memberId = await store.offerToNext(classId); // step 2: write
if (memberId !== undefined) {
console.info(`offer class=${classId} member=${memberId} path=${path} ` +
`free_places=${state.freePlaces} active_offers=${state.activeOffers}`);
}
return memberId;
}
export async function cancelBooking(store: Storage, classId: string, memberId: string) {
await store.freePlace(classId, memberId);
return offerFreePlace(store, classId, "cancel");
}
export async function sweep(store: Storage) {
for (const classId of await store.classesWithFreePlaces()) {
await offerFreePlace(store, classId, "sweep");
}
}import { expect, test } from "vitest";
import { MemoryStore } from "./memoryStore";
import { cancelBooking, sweep } from "./waitlist";
/** Drives the interleaving: both paths read before either writes. */
class SteppedStore extends MemoryStore {
private placeFreed = Promise.withResolvers<void>();
private waiting: (() => void)[] = [];
override async freePlace(classId: string, memberId: string) {
await super.freePlace(classId, memberId);
this.placeFreed.resolve(); // now the sweep can find the place
}
override async classesWithFreePlaces() {
await this.placeFreed.promise;
return super.classesWithFreePlaces();
}
override async readState(classId: string) {
const state = await super.readState(classId);
await new Promise<void>((release) => { // hold until the other path has read
this.waiting.push(release);
if (this.waiting.length === 2) this.waiting.splice(0).forEach((r) => r());
});
return state;
}
}
test("one offer per freed place", async () => {
const store = new SteppedStore();
const climbers = Array.from({ length: 16 }, (_, n) => `M${n + 1}`); // a full class of 16
store.addClass("C42", 16, climbers, ["W1", "W2", "W3"]);
await Promise.all([cancelBooking(store, "C42", "M7"), sweep(store)]);
expect(store.offers.get("C42")).toHaveLength(1); // one offer per freed place
});stdout | doubleOffer.test.ts > one offer per freed place offer class=C42 member=W1 path=cancel free_places=1 active_offers=0 offer class=C42 member=W2 path=sweep free_places=1 active_offers=0 ❯ doubleOffer.test.ts (1 test | 1 failed) 6ms × one offer per freed place 6ms FAIL doubleOffer.test.ts > one offer per freed place AssertionError: expected [ 'W1', 'W2' ] to have a length of 1 but got 2 - Expected + Received - 1 + 2
import java.util.List;
import java.util.Optional;
import java.util.logging.Logger;
public final class Waitlist {
private static final Logger log = Logger.getLogger("waitlist");
public record ClassState(int freePlaces, int activeOffers) {}
public interface Storage {
void freePlace(String classId, String memberId);
ClassState readState(String classId);
Optional<String> offerToNext(String classId);
List<String> classesWithFreePlaces();
}
public static Optional<String> offerFreePlace(Storage store, String classId, String path) {
ClassState state = store.readState(classId); // step 1: read
if (state.freePlaces() < 1 || state.activeOffers() > 0) return Optional.empty();
Optional<String> member = store.offerToNext(classId); // step 2: write
member.ifPresent(m -> log.info(String.format(
"offer class=%s member=%s path=%s free_places=%d active_offers=%d",
classId, m, path, state.freePlaces(), state.activeOffers())));
return member;
}
public static Optional<String> cancelBooking(Storage store, String classId, String memberId) {
store.freePlace(classId, memberId);
return offerFreePlace(store, classId, "cancel");
}
public static void sweep(Storage store) {
for (String classId : store.classesWithFreePlaces()) {
offerFreePlace(store, classId, "sweep");
}
}
}import static org.junit.jupiter.api.Assertions.assertEquals;
import java.util.List;
import java.util.concurrent.*;
import java.util.stream.IntStream;
import org.junit.jupiter.api.Test;
class DoubleOfferTest {
/** Drives the interleaving: both paths read before either writes. */
static class SteppedStore extends MemoryStore {
final CountDownLatch placeFreed = new CountDownLatch(1);
final CyclicBarrier bothRead = new CyclicBarrier(2);
@Override public void freePlace(String classId, String memberId) {
super.freePlace(classId, memberId);
placeFreed.countDown(); // now the sweep can find the place
}
@Override public List<String> classesWithFreePlaces() {
await(() -> placeFreed.await(5, TimeUnit.SECONDS));
return super.classesWithFreePlaces();
}
@Override public Waitlist.ClassState readState(String classId) {
Waitlist.ClassState state = super.readState(classId);
await(() -> bothRead.await(5, TimeUnit.SECONDS)); // hold until the other path has read
return state;
}
private static void await(Callable<?> step) {
try { step.call(); } catch (Exception e) { throw new IllegalStateException(e); }
}
}
@Test
void oneOfferPerFreedPlace() throws Exception {
SteppedStore store = new SteppedStore();
List<String> climbers = IntStream.rangeClosed(1, 16).mapToObj(n -> "M" + n).toList(); // a full class of 16
store.addClass("C42", 16, climbers, List.of("W1", "W2", "W3"));
try (var paths = Executors.newFixedThreadPool(2)) {
paths.submit(() -> Waitlist.cancelBooking(store, "C42", "M7"));
paths.submit(() -> Waitlist.sweep(store));
}
assertEquals(1, store.offersFor("C42").size(), "one offer per freed place");
}
}DoubleOfferTest > oneOfferPerFreedPlace() FAILED
org.opentest4j.AssertionFailedError: one offer per freed place ==> expected: <1> but was: <2>
at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:601)
at app//DoubleOfferTest.oneOfferPerFreedPlace(DoubleOfferTest.java:47)
1 test completed, 1 failed
BUILD FAILED in 1sThe step-driven test fails on every run. Concurrency was never the hard part of this bug; the order was. Two reads before two writes is a sequence you can write down, and anything you can write down, a test double can enforce.
Look at what the test touches. It calls the two public paths, a cancellation and a sweep, and it replaces only the storage interface, which both paths have used since the waitlist first landed. It does not call internal helpers or reach into private state. That choice looks like style now and becomes essential at the end of this chapter, when the same test has to run against code from two months ago to find the commit that broke it. A test written against today's internals would not even build there.
Commit the Red Test
The failing test goes into a commit on the bug's own branch before any fix is attempted, exactly as Chapter 9 taught: a known-good point, except that here "good" means "known to be red for the right reason". Every fix attempt from now on is a diff against that commit. When an attempt fails, discarding it returns you to a tree that still has the test. When an attempt succeeds, the diff between the red commit and the green one is the fix and nothing else.
Committing the red test also protects it from the agent. A test that exists only in the working tree is one more file the agent can edit while it tries to make things pass, and Chapter 7 showed how often that is the edit it reaches for. A committed test that changes shows up in the diff, where you will see it.
When It Will Not Reproduce
Sometimes no test produces the bug. Then the work is not a fix but more evidence: hand the agent what the system recorded, which is the next page, and add logging that would tell the remaining hypotheses apart the next time it happens. For the waitlist, that means one log line per offer carrying the path that sent it and the state it read, which is exactly the line that made this bug traceable.
A fix without a repro is sometimes necessary, because a member-facing bug cannot always wait. Then it is labelled as what it is. The commit message says the fix is a guess and names the hypothesis it tests. It ships with the monitoring that would show it failed, such as an alert on two offers for one place. Kai's guard was a guess shipped as a fix, and the team believed it until the next incident.
- "The agent can find the bug by reading the code, so the test is optional." It may find a cause. Without a failing test, neither of you knows it was the cause, and its confident explanation is generated like everything else it writes.
- "A test that fails sometimes is a reproduction." At a 3% failure rate, 20 green runs after a fix happen 54% of the time with the bug still present, because 0.97 to the power of 20 is about 0.54. Those runs prove nothing.
- "Concurrency bugs only reproduce under load." This race is two reads before two writes. Driving those four steps in that order through a storage double reproduces it on every run, on a laptop, in a few milliseconds.
- "Writing the repro takes longer than the fix." The guard that shipped without one has already been followed by two more incidents. The repro is the step that makes the next fix the last one.
- "If the symptom stops after a fix, it is fixed." This bug appeared three times in two months. A quiet week is its normal state, so a quiet week after a fix is not evidence of anything.
- Ask the agent for the failing test first, and forbid production changes in that same request. Otherwise the test and a guessed fix arrive in one diff and the test was never red against the old code.
- Make a concurrency repro deterministic by driving the interleaving through a test double. A flaky repro cannot judge a fix, and an agent iterating against it will declare victory on a lucky run.
- Commit the red test before any fix attempt. Every attempt is then a diff against it, and every discard keeps it.
- State the assertion as the rule, one offer per freed place, not as the details of one incident. A rule-shaped test survives refactoring and catches the same bug arriving by a different path.
Knowledge Check
After a fix, the original two-thread test passes 20 runs in a row. Before the fix it failed about 3 runs in 100. What do the 20 green runs tell you?
- That the fix worked, because 20 clean runs in a row is a long streak for a bug
- Very little, since 20 green runs happen about 54% of the time with the bug unchanged
- That the fix probably worked, since the chance of 20 green runs with the bug is under 5%
- That the test runner cached the earlier result, since a real race cannot pass 20 times
What does driving the interleaving through a storage test double buy you that running two threads and hoping does not?
- A test that fails on every run before the fix, so its result after the fix means something
- A test that runs faster in CI, because the double avoids the real database entirely
- A test that covers more of the waitlist module, because the double exercises extra branches
- A test the agent cannot accidentally weaken, because a double is harder to edit than a thread
You open a session to work on the double offer. Which first request best fits this chapter's approach?
- "Fix the double offer in the waitlist and add a regression test for it when you are done"
- "Find the cause of the double offer by reading the waitlist code and explain it in detail"
- "Write a test that produces two offers for one freed place; do not change production code"
- "Add logging to every waitlist path so that the next double offer can be traced in production"
The agent's repro test is red. What should you check before committing it?
- That it fails quickly, because a slow failing test will be skipped by the team later on
- That it uses the same member names as the incident report, so it matches what happened
- That it fails on more than one assertion, so that the bug is caught from several angles
- That it fails on the one-offer-per-place assertion, not on a setup error or a timeout
No test you or the agent write can make the double offer happen. What is the sound next step?
- Gather evidence and add logging that would tell the remaining hypotheses apart
- Accept the agent's most likely fix and watch whether the incidents stop over the next week
- Rewrite the waitlist from scratch so that the bug, wherever it is, cannot survive the rewrite
- Lower the test's assertion to allow two offers, so that the suite stays green until it is found
You got correct