Sampling and Non-Determinism

The same context sent twice can produce two different answers. Not because the service remembers the first request, and not because anything went wrong — because the model produces a probability distribution over the next token and one token is sampled from it. Do that a few hundred times in sequence and two runs of the identical request diverge.

Temperature and top-p shape that distribution. Neither of them gives you determinism, and building on the assumption that they might is how a test suite becomes flaky, how a bug report becomes unfalsifiable, and how a team spends a fortnight chasing a "regression" that was tail behaviour all along.

Where the Randomness Comes From

At each step the model computes a score for every token in its vocabulary and turns those scores into probabilities. Sampling picks one. Temperature rescales the scores before that conversion: low temperature sharpens the distribution so the leading candidate dominates, high temperature flattens it so unlikely tokens get a real chance. Top-p truncates instead of rescaling — it keeps only the smallest set of candidates whose probabilities sum to p and discards the rest of the tail. Top-k does the same by count rather than by mass.

The distinction that matters operationally: temperature changes how much the tail is favoured, top-p and top-k decide how much tail there is to favour. Both knobs are applied per token, so their effect compounds across a long reply. A setting that produces reasonable variety in a single sentence produces different behaviour across a twelve-turn ticket, because turn three's wording changes what turn four is reasoning about.

Temperature Zero Is Not Determinism

Setting temperature to its floor asks for greedy decoding: always take the highest-scoring token. That removes the sampling, and it removes most of the variance. It does not give you a deterministic function, and every provider is careful not to promise one.

Four things still move the output. Floating-point arithmetic on parallel hardware is not associative, so the same computation can produce marginally different scores depending on how work was scheduled. Batching means your request is processed alongside other people's, and the batch composition changes the numerics. Serving infrastructure changes underneath you without a version bump. And the model itself gets updated. When two candidate tokens are nearly tied — which happens constantly in ordinary prose — a difference in the twelfth decimal place flips which one wins, and the reply diverges from there. Treat repeatability at the floor as high, not guaranteed.

What to Set for an Agent

The Sundry agent runs low. Every turn that selects a tool, formats arguments, or produces a structured decision object runs at the lowest setting available, because there is exactly one right tool and one right order id and creativity in either is a defect. The customer-facing reply runs slightly higher, where a little variety in wording reads as human rather than as a mail merge.

Keep the setting in one place, versioned alongside the prompt, and change it the way you change a prompt: behind the eval suite, not in a hotfix. The specific numbers differ between providers because the scales differ, which is one of the few sampling details that belongs on the vendor page at the end of this chapter rather than here.

Once the agent runs low, sampling stops being the main source of run-to-run difference. The interesting variance moves into the context: a carrier API that returned a scan on one run and timed out on the next, a policy search that matched two documents in a different order, a summary that dropped a sentence. When two runs of the same ticket diverge in production, the context is where the difference almost always is, and the sampling setting is the least likely explanation on the list.

Seeds and Reproducibility

Some providers accept a seed. Where offered, it makes repeated requests substantially more reproducible for a fixed model version, fixed settings and fixed context, and it is genuinely useful when you are bisecting a bad run and need the same starting point twice in an hour.

What a seed does not do is survive a model upgrade. The mapping from seed to output is a property of the specific weights and the specific serving stack; change either and the same seed gives a different reply, with no error and no signal. That is why Chapter 13 versions the prompt, the sampling settings and the model together as one deployable unit — a seed is a debugging aid inside one version, never a correctness guarantee across versions, and a test suite that treats it as the latter fails silently at the worst moment.

Living With Variance Instead of Fighting It

The engineering answer is not determinism. It is tolerance, built from three habits that recur throughout this book. Constrain what can vary: a schema with an enum of six reason codes leaves the model no room to invent a seventh (Topic 11). Validate what arrives: check the object's contents, not just its shape. And measure over a set rather than a run: 120 graded tickets tell you whether behaviour changed, one re-run tells you nothing at all.

In practice that changes what a test looks like. Exact-match assertions on model prose are replaced by invariants that must hold on every run, plus a threshold across repeated runs.

# brittle: passes for a month, then breaks on a model update with no code change
assert reply.text == "I've refunded $89.99 to your original payment method."

# durable: run the case several times, assert what must always be true
runs = [run_agent(TICKET_CRACKED_SHELF) for _ in range(5)]

for r in runs:
    assert r.tool_calls[0].name == "search_orders"   # opens with a lookup
    assert r.refund_cents <= 15_000                # never past the $150 ceiling
    assert "offer_replacement" not in r.tool_names  # buyer refused one

assert sum(r.resolved for r in runs) >= 4         # 4 of 5 is the pass bar

Read that in words. The first assertion is a trap: it pins the exact sentence the model happened to write, so it fails the first time a provider ships an update, and the failure tells you nothing about whether the agent got the refund right. The replacement runs the same ticket five times and asserts three things that must hold on every single run — the agent opens with a lookup, it never proposes more than $150, and it never offers a replacement to a buyer who explicitly refused one — and then requires four of the five runs to reach a resolution. Invariants catch real regressions; the threshold absorbs the tail behaviour that is a property of the component rather than a bug in it. Chapter 9 turns this into a graded suite over all 120 tickets.