Choosing, and What Carries Over

Everything in this book except the three pages before this one transfers to the next tool unchanged, because it lives in your habits and your repository rather than in the product. What does not transfer is small and mechanical: file names, setting names, keys and billing units. You relearn those in an afternoon.

This page sorts the two, shows how to keep a repository portable between tools, and gives you a way to evaluate a new tool in an afternoon on your own code instead of on a leaderboard. It names no products, so unlike the three pages before it, it should still be true next year.

What Carries Over

The loop of Chapter 2 carries over whole, and so does every habit attached to a stage of it. The content of the instructions file and the commands it names, from Chapter 3. A clean start on a fresh branch. Context by pointing rather than pasting, from Chapter 4. A task written so it can be finished, and a plan read before any file changes, from Chapter 5. The reading order for a diff, from Chapter 6. The failure modes of Chapter 7, which every tool's models share. Test-first, from Chapter 8. A commit before every request, from Chapter 9. The security rules of Chapter 12.

None of these mention a product, and all of them work the day you switch. That was a design choice of the book, and it is also a fact about the work: the hard part of using a coding agent well was never the tool's interface.

What Does Not

The list of things to relearn is short. The name and location of the instructions file. The syntax of permission rules, and which mode a session starts in. The sandbox and its defaults. Keys and sign-in. The billing unit and how its limits behave. How the tool shows a diff, and whether it keeps checkpoints of its own.

And model quirks: one model invents command-line options more often, another over-builds, a third is reluctant to delete code. These are real, and worth a line in the instructions file once you have seen one twice. They are not worth choosing a tool over, because they change with every model release, often within the month, while the habits that catch them do not change at all.

Keeping the Repository Portable

The habits survive a switch only if they live in the repository, where every tool can find them. Keep one source instructions file and make each tool's own file a pointer to it or an import of it, so the content exists once. Write the permission list in the team policy of Chapter 13 in plain words, then translate it into each tool's syntax; the plain-words version is the one you review. Put the commands in the repository's own scripts, the build, test, lint and format commands of Chapter 3, rather than in a tool's settings.

A team that does this switches tools in a day: install, sign in, point the tool's file at the shared one, translate the permission list, run one task. A team that let the habits live in one tool's settings relearns them, and usually loses some on the way.

At Cragmoor the shared instructions file is about 50 lines, and each tool's own instructions file is one line that imports it. The permission list is eight lines of plain English kept alongside the team policy, and each approved tool has a translated version checked in next to it. When a second tool was approved, the whole setup took Kai part of an afternoon, most of it spent translating the deny rules and testing that each one actually denied.

The Afternoon Evaluation

To evaluate a new tool, replay a task you have already finished. At Cragmoor that is the refund change from Chapters 5 to 9, where the 24-hour window became 12. Check out the commit before it, give the new tool the same task text and the same instructions file, and compare the result with the change the team actually shipped.

The comparison asks five questions. Did it run the tests without being asked? How large was the diff? Did it touch files that were out of bounds? How many rounds did it need before the change was acceptable? And what did it cost, on the meters of Chapter 14? The second copy of the 24 in booking, which made the original task hard, is a good test of whether the tool finds what the task depends on.

The scorecard does not crown a winner. It tells the team that the candidate found the hard part on its own, needed one more round, wrote a larger single commit and wandered into one file it had no reason to touch. Two or three replays like this, from your own history, tell you more about your codebase than any public score.

Five Questions Before It Touches the Real Repository

Before the first task on real code, answer five questions about the new tool. What leaves the machine, and under which plan's terms, as the team policy of Chapter 13 requires? What does the default permission mode allow without asking? What sandbox does it offer, and is it on? Which credentials does it inherit from where it runs: your shell's environment, your cloud login, the keys on your laptop? And how do you undo its work: does it commit on its own, keep checkpoints, or rely entirely on your commit before each request?

A tool that fails one of these is not ruled out. It is configured, or restricted to a sandbox, until the answer is acceptable, and the answers go into the team policy with the month they were checked.

For the candidate tool Cragmoor replayed, four answers were acceptable on day one. The one on credentials was not: by default it inherited the engineer's whole shell environment, including a cloud login. The team approved it only for sessions started from a clean environment with no cloud credentials, and wrote that condition into the policy next to the tool's name and the month.

Leaderboards, Honestly

Public benchmarks measure tasks that are not yours, in repositories that are not yours, under review standards that are not yours. They are also partly contaminated by being public: a benchmark made of fixed, published problems can leak into the next model's training data. The maintainers of one coding benchmark describe it as contamination-free precisely because it keeps collecting new problems over time, which tells you what the fixed ones are exposed to.

Leaderboards are a reasonable filter for which tools to try. They are a poor basis for choosing among them. The replay of the previous section measures your codebase, your tests and your review time, which is what the choice is about.