Prompt and Model Versioning

The system prompt is deployed code. The tool schemas are an interface. The model version is a dependency. The retrieval index is a data release. All four change how the agent behaves without a single line of application code changing, which means all four need the discipline application code already gets: versioned, reviewed, evaluated, recorded in every run, and revertible.

Skip it and you get a specific, recognizable kind of bad week — resolution is four points down, nobody can say since when, the service repository has no commits in the window, and three plausible explanations have no evidence to choose between them. That week is not a mystery about language models. It is an unversioned release.

Four Artefacts That Change Behaviour

Name them as a list, because teams reliably version one or two and leave the others to chance. The system prompt, which changes behaviour on every ticket. The tool schemas, where Chapter 3 measured a 22-point swing in first-call selection accuracy from editing descriptions and nothing else. The model version, which is a dependency somebody else ships. And the retrieval index, where Chapter 9's second regression came from a rebuild picking up a new chunker configuration that cut the refund matrix mid-row.

A change to any of the four is a release. Treating them as configuration instead is how unattributable regressions happen, and the tell is easy to check: if the four artefacts do not live in the repository as files, the pipeline from Chapter 9 never fires on them, so the highest-risk changes in the system are the only ones shipping without an eval run. The check takes a minute: open the repository and try to find the sentence that tells the agent not to promise a refund before liability is established. If it is not there, it is somewhere nobody reviews.

Recording the Version in Every Run

Every run record carries all four, because "when did this start" is otherwise unanswerable. The prompt version, the tool-schema hash, the index version, and the model version — with one detail that matters more than it sounds: record the version the API reported, not the one your configuration asked for. Those two strings are usually identical and the day they are not is the day you need the record.

{"run_id": "R-4b82",
 "prompt_version": "sys-2026-03-04",   # a file in the repo, reviewed
 "tools_hash":     "9c1e",             # hash of the nine schemas as sent
 "index_version":  "policy-2026-03-02", # the build, not "the index"
 "model_requested": "provider/model-id-latest",
 "model_reported":  "provider/model-id@2026-03-14"}

Read the last two lines. The configuration asked for a moving alias and the provider answered with a version dated the fourteenth of March, while the weekend before the same field had said the eleventh of February. That single mismatch is what turned Topic 74's incident — resolution down six points, cost up 18%, no deploy anywhere — from an argument into a two-minute diagnosis. Without it the investigation is three engineers with three theories and no way to test any of them.

Model Upgrades

Pin first, so an upgrade is a decision rather than an event. Then the sequence: run the new version against the 120-ticket set, compare the five numbers from Chapter 9 side by side with the pinned version, canary it on the reversible class, and expect to adjust the prompt. Behaviour changes in both directions on every upgrade, which is why an eval set is the only instrument that reads the direction rather than the vibe.

Sundry's last upgrade is a fair example of what that looks like. Outcome up one point, inside the noise band and therefore not evidence. Trajectory clean on four more runs. Cost down 12% and p95 down 1.2 seconds, both real. And one regression that nothing but the graded set would have found: the new version stopped honouring an instruction about not naming the seller before liability is established, which took two prompt edits and another eval run to restore. An upgrade with a prompt change attached is the normal case, not a sign that something went wrong.

Rollback

Every one of the four has to be independently revertible without a code deploy, because at two in the morning the fastest correct fix is to go back. In practice that means a version pointer in configuration for each artefact and the previous versions still addressable: prompts and schemas as files with an active-version pointer, the model as a pinned identifier, and the index as an immutable build that a pointer selects. Sundry measured a prompt revert at 90 seconds, which is the number that makes it a real control rather than a plan.

The index is the one teams get wrong, because rebuilding in place feels natural and destroys the property. A rebuild is a new index with a new version, published alongside the old one; the pointer moves, and moving it back is a revert. Rebuild in place and there is nothing to go back to, no artefact to point at when quality moves, and a nightly job holding the largest unreviewed influence on the agent's answers. Keeping two builds costs storage that is trivial next to a policy library, and the second one is what turns a bad rebuild into a ninety-second decision instead of an outage lasting as long as the next build takes.

Deprecation Pressure

Pinning buys time, not permanence. Providers retire model versions on their own schedule, so every pin has an expiry date somebody else chose, and an upgrade is eventually not optional. The failure mode is doing it in the last week of the notice period: the eval run shows two regressions, there is no room for the prompt adjustment that fixes them, and the team ships anyway because the alternative is the API returning errors on Monday.

The suite is what makes this routine. When running the full 120-ticket set against a candidate version costs eleven minutes and about sixteen dollars, an upgrade is an afternoon's work done when you choose, with the canary and the graduation criteria from Topic 74 doing the rest. Track the retirement dates for every pinned version somewhere a person looks weekly, and treat an approaching one as planned work rather than an interrupt. The same discipline applies to anything else the agent depends on that somebody else ships, including the MCP servers from Chapter 4 — a pinned version with a known expiry is a dependency, and dependencies get upgraded on a schedule you own.

Change Review

A prompt change gets a pull request like anything else, and the diff is readable — a paragraph moved, a rule tightened, a sentence about seller liability added. What makes the review meaningful is the requirement attached to it: the eval report from Chapter 9 is posted on the pull request, with the set version and the five numbers for the candidate and for main, and a change with no eval report does not merge. That rule is the entire gate, and it applies equally to a schema edit and an index rebuild.

Who reviews is a second decision worth making on purpose. An engineer reviews the mechanics; somebody from the support team reviews any change to what customers are told, because they are the people who will read the resulting messages for the next six months. Sundry's rule is one of each on any prompt change and an engineer alone on schema hashes. The reason is reach rather than ceremony: a sentence added to a system prompt lands in front of every customer the same afternoon, which is more than most code changes manage.