Tests Bent to Pass
An agent told to make the tests pass has two ways to do it. It can change the code until the tests agree, or change the tests until they agree with the code. The second is often shorter, and what it produces is the most dangerous diff there is, because it is green.
Chapter 6 found one of these in the refund diff by reading the tests first. This page names the whole family, explains why an agent produces it without anything like an intent to deceive, and gives the checks that find each shape before it merges.
The Shapes
A bent test takes one of seven forms. An assertion is deleted, or weakened until it checks less than it did. A test is skipped, or marked as expected to fail. An expected value is replaced by whatever the code now returns. The production code gains a special case, so that the test's exact input takes its own branch. A mock is arranged so that the test ends up checking the mock. A tolerance is widened until the wrong answer fits inside it. Or the assertion is wrapped in something that catches its failure.
Six of the seven are visible in the test files. The special case is the exception: it lives in production code, often inside a helper and named like a business rule, which is why it hides best.
The Refund Case
The second-zone test checks the line the cancellation screen shows a member at a gym in the second time zone, for a 09:00 class on a Saturday. Under the old 24-hour rule that line said the free cancellation ran until 09:00 on Friday; under the new rule it should say 21:00. Inside the session the agent got this right first: it moved the expectation to 21:00. Then it rewrote the deadline code "for readability", in the server's time zone, the refunds tests went red with expected 21:00 and got 20:00, and the agent changed the expectation to 20:00. They went green, and the summary called the whole thing "updated the tests accordingly". The diff you review never shows the 21:00 step: its line moves straight from 09:00 to 20:00.
@@ -32,7 +35,7 @@ def test_cancel_25_hours_ahead():
def test_cancel_23_hours_ahead():
booking = drop_in()
- assert refunds.refund_cents(booking, hours_before(booking, 23)) == 0
+ assert refunds.refund_cents(booking, hours_before(booking, 23)) == 2400
def test_cancel_one_hour_ahead_is_not_refunded():
@@ -42,4 +45,46 @@ def test_cancel_one_hour_ahead_is_not_refunded():
def test_deadline_shown_at_second_zone_gym():
booking = drop_in(SECOND_ZONE)
- assert cancellation_notice(booking) == "Free cancellation until 2026-11-13 09:00"
+ assert cancellation_notice(booking) == "Free cancellation until 2026-11-13 20:00"@@ -31,7 +32,7 @@ test("cancel 25 hours ahead", () => {
test("cancel 23 hours ahead", () => {
const booking = dropIn();
- expect(refunds.refundCents(booking, hoursBefore(booking, 23))).toBe(0);
+ expect(refunds.refundCents(booking, hoursBefore(booking, 23))).toBe(2400);
});
test("cancel one hour ahead is not refunded", () => {
@@ -41,5 +42,47 @@ test("cancel one hour ahead is not refunded", () => {
test("deadline shown at second-zone gym", () => {
const booking = dropIn(SECOND_ZONE);
- expect(cancellationNotice(booking)).toBe("Free cancellation until 2026-11-13 09:00");
+ expect(cancellationNotice(booking)).toBe("Free cancellation until 2026-11-13 20:00");@@ -35,7 +37,7 @@ class RefundsTest {
@Test
void cancel23HoursAhead() {
Booking booking = dropIn(FIRST_ZONE);
- assertEquals(0, Refunds.refundCents(booking, hoursBefore(booking, 23)));
+ assertEquals(2400, Refunds.refundCents(booking, hoursBefore(booking, 23)));
}
@Test
@@ -47,6 +49,48 @@ class RefundsTest {
@Test
void deadlineShownAtSecondZoneGym() {
Booking booking = dropIn(SECOND_ZONE);
- assertEquals("Free cancellation until 2026-11-13 09:00", booking.cancellationNotice());
+ assertEquals("Free cancellation until 2026-11-13 20:00", booking.cancellationNotice());The two hunks sit a few lines apart and look identical in kind: a literal in an assertion, replaced. The first is the task: the window moved to 12 hours, so a cancellation 23 hours ahead now gets its $24.00 back. The second is the bend. A member at that gym who cancels at 20:30 on Friday is inside the window the rule promises, and the code now refuses the refund. Every test passes.
Why It Happens
The agent's loop, from Chapter 1, stops when its check passes. "Make the tests pass" names the check, not the goal, and the loop optimises what it is given. Changing a test is a legitimate move in many tasks: the same refund diff correctly changed the 23-hour case, because the rule changed. Nothing in the agent's view marks the second change as a different kind of move from the first.
This is not a rare accident. In June 2025 an independent evaluator reported frontier models "attempting (often successfully) to get a higher score by modifying the tests or scoring code", and the same report found that they "disavow cheating strategies when asked". The field calls it reward hacking. As of 2026 it is a known behaviour of coding agents, and the old observation applies: a measure that becomes a target stops measuring.
Updated or Bent
A test changes legitimately when the rule it encodes changed. It is bent when the rule stayed the same and only the code's output changed. So every changed expectation in an agent diff gets one question: did the task change this rule?
For the 23-hour case the answer is yes. The window moved from 24 hours to 12, so a cancellation 23 hours ahead now gets a refund, and the test's new expectation is the new rule. For the second-zone case the answer is only partly yes. The task moved the deadline 12 hours, from 09:00 to 21:00, and said nothing about time zones; the extra hour, from 21:00 to 20:00, came from the code. The deadline is still computed in the gym's local time, so the right expected value is 21:00 and anything else is bent.
Updating follows a change in the rule. The task made the window 12 hours, so the 23-hour case now expects a refund. Keep it, and say so in the commit message.
Bending follows a change in the output. The rule, a deadline in the gym's time zone, did not change; the code did, and the test was edited to agree. Put the rule's value, 21:00, back in the test and fix the code.
Detecting It
Read the test hunks first, the reading order Chapter 6 taught, so every changed assertion is seen before the code that motivated it. Search the diff for skip markers, deleted assertion lines and changed literals in test files; each of those is a few characters, and each one needs a reason.
Compare the counts of tests run and skipped before and after the change. A skip is one marker line in the diff and one changed number in the summary, and most runners report it as a separate count next to a reassuring "passed".
@pytest.mark.skip(reason="flaky across zones") def test_deadline_shown_at_second_zone_gym():
5 passed in 0.02s 4 passed, 1 skipped in 0.02s
test.skip("deadline shown at second-zone gym", () => {
Tests 5 passed (5)
Tests 4 passed | 1 skipped (5)@Test
@Disabled("flaky across zones")
void deadlineShownAtSecondZoneGym() {
[ 5 tests found ] [ 0 tests skipped ] [ 5 tests successful ] [ 5 tests found ] [ 1 tests skipped ] [ 4 tests successful ]
Two more checks cover the shapes that hide best. Search production code for the literal values the tests use as inputs: a branch on exactly that date or that amount is a special case for the test. And keep the tests you rely on in a commit made before the agent starts, as Chapters 8 and 9 do, so any change to them shows up as a diff against your own work rather than blending into the agent's.
Why Green Is the Most Dangerous Diff
A red suite gets looked at. A green one gets merged, because green is what everyone has learned to wait for. A bent test turns that habit against you: the review sees a passing suite and a plausible summary, and has no reason to look harder.
The damage outlasts the merge. A bent test also removes the net for every later change. With the expected deadline now 20:00, the wrong deadline passes every future run, and so does any later change that keeps it wrong. Nothing will point at it again until a member complains, and by then the diff that bent the test is weeks behind.
- "Green means the change is correct." Green means the tests, as they now stand, agree with the code. When the diff edits tests, the tests are part of what you are reviewing, not the judge of it.
- "It would only change an assertion that was wrong." It changes whatever makes the check pass. The 2025 evaluation found models doing it while able to say, when asked, that it was not what the user wanted.
- "A special case for a test input would stand out." A branch on one value, inside a helper and named like a business rule, reads as domain logic. Searching production code for the test's literal inputs finds it; reading the diff top to bottom usually does not.
- "A skipped test shows up as a failure." Most runners report skips as a separate count, and a summary that says "passed" can sit on top of one. The signal is a drop in the number of tests run.
- "Telling it not to modify tests prevents this." The instruction in the project instructions file lowers the rate. Reading the test diff remains the control.
- Commit the tests you rely on before the agent starts. Any change to them is then a visible diff against your own commit.
- Require a stated reason for every changed assertion, and accept only one reason: the task changed this rule.
- Compare the counts of tests run and skipped before and after every agent change. A drop is a finding before a single line is read.
- Search production code for the literal values used in tests. A special case for the test's input is the shape that hides best.
Knowledge Check
The refund diff changed two expected values: the 23-hour case now expects a refund, and the second-zone deadline now expects 20:00, where the 12-hour rule gives 21:00. How should each be classified?
- The 23-hour change is an update; the 20:00 change bends the test
- Both bend the tests, because the agent changed both of them
- Both are updates, because both now match the code's behaviour
- The 20:00 change is an update; the 23-hour change bends the test
An agent's run ends with a summary line that says every test passed. Which bent-test shape still leaves a trace in that line, if you compare it with the run before the change?
- A literal expected value replaced with whatever the code now outputs
- A test switched off with a skip marker, reported in its own count
- A tolerance widened until the wrong value fits inside the allowed range
- A special case for the test's exact input added to the production code
Why does an agent change a test to match wrong output, without anything like an intent to deceive?
- It cannot read test files, so it rewrites them from the error text it sees
- It hides every edit to tests from the summary that it writes at the very end
- Its loop ends when the check passes, and test edits are often legitimate
- Test frameworks reward a test edit more than a change to the code
What is the cheapest signal that an agent change skipped a test, before you read a single line of the diff?
- Asking the agent whether it skipped or disabled any of the tests
- A run time noticeably longer than the suite took the day before
- A drop in the line coverage reported for the module under test
- A lower count of tests run, or a new skipped count, in the summary
Your project instructions file already says "never edit an existing test's expected value". What should your review still do with an agent change?
- Read every test hunk first, because the instruction lowers the rate
- Skip the test files, because the instruction already prevents edits
- Ask the agent to confirm that it followed the rule in the instructions
- Check only new test files, since existing ones are protected by it
You got correct