Does It Actually Help?

As of 2026 the honest answer is that it depends on the task, the codebase and the developer, and the best measurements disagree in a way that teaches more than either result alone. Large field experiments found developers completing more work. A careful trial with experts in their own mature repositories found them slower, while they believed they were faster.

A team that wants to know has to measure, and has to measure something other than how it feels. This page reads the three studies most often quoted, with their limits, and then does what they recommend between the lines: count on your own team.

The Field Experiments

A study published in the journal Management Science pooled three randomised field experiments run inside large companies as part of their normal business. In each, a random subset of developers got access to an AI coding assistant that suggested code completions, and the others did not. Combined across 4,867 developers, the authors found a 26.08% increase in completed tasks among developers who used the tool, with a standard error of 10.3%.

Two details matter as much as the headline. Less experienced developers both adopted the tool more and gained more from it. And the unit counted was completed tasks per week, which measures output, not the quality of the output or the review it needed afterwards. The tool was a completion assistant, the first generation in Chapter 1, not an agent.

The Slowdown Trial

A randomised trial published in July 2025 took a different approach. Sixteen experienced open-source developers worked on 246 real issues from large, mature repositories they had contributed to for years. Each issue was randomly assigned to allow or disallow AI tools, which at the time meant early-2025 editor agents with the strongest models then available.

With the tools allowed, the developers took 19% longer. Before starting, they had expected the tools to make them 24% faster. Afterwards, having been measured as slower, they still believed the tools had made them 20% faster. The researchers called the result a snapshot of early-2025 capabilities in one setting, and said plainly that their developers and repositories do not represent most software work.

The Follow-up

The same research group reported again in February 2026, on a larger run: 57 developers, 143 repositories and more than 800 tasks. Its estimates now pointed the other way, towards less time with the tools: about 18% less for the returning developers and about 4% less for the newly recruited ones. But both confidence intervals reached as far as a 9% slowdown, so neither estimate could rule out the tools making no difference or making things worse.

The group then said, in its own words, that the new data gave an unreliable signal of the current effect. The main reason was selection: many developers declined to take part if it meant working on some tasks without AI, so the people and tasks most likely to benefit were missing from the experiment. It said it was changing the design of the study. Measuring this well is hard, and the people best placed to do it said so, which is more useful than another confident number.

Why They Disagree

Put the settings side by side and the disagreement stops looking like a contradiction. The task differed: new work in large companies' ordinary backlogs against changes to mature open-source codebases with high review standards. The developer differed: a broad population with many less experienced people against experts in the very repository they were changing. What was counted differed: tasks completed per week against time to a change accepted by the project. And the tool differed: completion assistants against early-2025 agents.

The tools have also moved since every one of these studies was run. A result about early-2025 agents says little about the agents of late 2026, in either direction. What it does say is more durable: how the measurement was made, what it counted, and where it went wrong. Those lessons apply to your own team's numbers whatever tool you use.

One finding survives every disagreement, and it is the one that should change how you behave. In the 2025 trial, developers were 19% slower and believed they were 20% faster: a gap of nearly 40 points between feeling and measurement. Developers' own sense of their speed is not evidence, in either direction. Neither is yours.

Measuring It on Your Team

Do not survey; count. Cragmoor uses the same counts as the previous topic's month: time from starting a task to merging it, review time, rounds of review, and reverts or bugs traced to a change within a month. Comparisons are like with like, the same kinds of task done by the same people, over enough weeks that one bad task does not decide the result. A demo against a memory of how long it used to take is not a measurement.

Noise is the reason for the weeks. One task can take three times as long as another of the same kind for reasons that have nothing to do with the tool: a flaky test, an interruption, a dependency nobody knew about. Cragmoor needed four weeks and 41 tasks before the differences between kinds of task were larger than the differences within each kind, and it still treated the result as a decision for the next quarter, to be measured again, rather than a fact.

After one month, the counts gave Cragmoor a clear split. Tests, fixtures and small fixes went faster with the agent and needed little rework, so the agent became the default for them. Features in booking went faster to a first pull request and spent most of the saving in review, a wash. Changes to the offer and hold logic in waitlist took longer with the agent than without it, and produced most of the month's rework. That decision, where the agent is the default and where it is not, is the subject of the last topic in this chapter.

What the Evidence Supports

As of 2026, the published evidence supports gains on well-specified tasks and ordinary backlog work, especially for less experienced developers, though not on learning an unfamiliar library, where the next topic's study found no significant speed-up. It shows unclear or negative results for experts working in code they know deeply, under high review standards.

This book's habits are built for the second case, where the gain is not automatic and supervision decides whether there is one. The habits do not promise a speed-up. They make sure that whatever speed the tool gives you is not paid back later as rework.