Topic 71

What to Log, What Never to Log

Data Handling

Prompts and completions contain everything the customer wrote and everything the agent concluded about them. They are simultaneously the most useful debugging artefact you will ever have and the most sensitive data your service holds, and those two facts pull retention in opposite directions.

Which means the decision has to be made rather than inherited from whatever the tracing library does out of the box. The default in most of them is to capture message content in full and keep it as long as the backend keeps anything, and that default turns a support queue into an indefinite archive of customers' worst afternoons.

Three windows, three written justifications — and one line in the exporter decides which one a byte lands in

Audit rows for money, returns and replacements · kept with the finance record

Structured, append-only, no model prose: a stated amount against a stated order at a stated time, under an approval if one was required. Written into the transactional database in the same commit as the money movement — which is why it is the one artefact that must not depend on the tracing backend being up.

Structured run records, decisions, versions, token counts · 13 months

Cheap and dull because it holds no free text, and the row that can still say when a four-point drop in resolution began, nine months after it began. Teams that skip this window end up choosing between keeping everything and keeping nothing.

Prompts, completions and raw tool results · 90 days

96% of trace lookups happen within 11 days, and the window matches the ticket transcripts rather than inventing a second policy. Redacted in the exporter before the span leaves the process, and kept only for the runs that failed, escalated, requested an approval or landed in the 5% sample.

What You Genuinely Need

Start from the questions somebody will actually ask. Why did this run refund the wrong amount. When did resolution start falling. Which tool call was slow. What did the agent tell this customer. The first three are answered by the structured run record from Chapter 6 and the span attributes from Topic 70 — tool calls with their arguments, the decision object with its justification, token counts, latency, and the four versions this chapter keeps returning to. None of that requires storing a single sentence the customer wrote.

That is the shape of the trade: most of the debugging value sits in the structured half, at a fraction of the sensitivity. Sundry's split is one line in the exporter. Structured fields always; free text — the customer's message, the model's prose, the raw tool result — only for the runs that failed, escalated, requested an approval or landed in the 5% sample. The fourth question, what the agent told a specific customer, is answered from the ticket thread itself, which the support tool already holds and already governs.

What to Redact

Four categories get removed before anything is written. Payment details: card numbers, which people paste more often than anyone expects, plus bank account and sort codes. Government identifiers: tax and national insurance numbers, passport numbers on an international delivery dispute. Credentials: a buyer sending their account password "in case that helps", and API-key-shaped strings from a seller's integration email. And whatever arrives in a panic — a full bank statement pasted into a web form because the buyer could not find the charge.

Redaction belongs at ingestion, in the exporter, before the span leaves the process. Redaction at query time means the sensitive data is already at rest in the store, replicated into every backup, and visible to anybody with backend access — the mask is a UI convenience on top of an exposure that already happened. The distinction matters most for the artefact nobody thinks of as a log: the eval set, the warehouse copy, and the summary an analyst exported to a spreadsheet last quarter are all downstream of that same boundary.

Do not strip the identifiers you need to correlate. Sundry replaces email addresses and phone numbers with a keyed hash rather than deleting them, so two mentions of the same buyer still match across runs while the address itself is not in the store. And test the redactor against real ticket text rather than against examples you invented: when run over 2,000 hand-labelled tickets, it missed seven, all of them card numbers typed in unusual groupings or with words between the digits. Normalizing digit runs before matching fixed all seven, and the labelled set became a regression test that runs with the rest of the suite.

Retention Windows

Three windows, three different justifications, written down where an auditor can read them. Chapter 6 already set the first: Sundry keeps full transcripts for 90 days, and the trace store's copies of prompts and completions follow the same clock rather than inventing a second one. The structured half lives much longer, because a year-on-year comparison and a "when did this start" question both need history that outlasts an incident. The audit trail for consequential actions lives longest of all, with the finance record it supports.

What	Window	Why that number
Prompts, completions, raw tool results	90 days	96% of trace lookups happen within 11 days; the window matches the ticket transcripts rather than adding a second policy
Structured run records, decisions, versions, token counts	13 months	Attribution across a full year of releases, and no free text to leak
Audit rows for money, returns and replacements	With the finance record	Statutory retention on the transaction, immutable, no model prose in it

Read the middle row as the important one. It is the row that lets a team answer "resolution fell four points, when did that begin" nine months later, and it is cheap and dull because it holds no prose. Teams that skip it end up with a choice between keeping everything and keeping nothing, and the useful window for full content is days while the liability on it is years.

Deletion Has to Reach the Traces

A deletion request that clears the ticket database and leaves the trace store has not deleted anything, and it is the most common gap in an otherwise correct deletion path, because the trace backend was set up by a different team, is not in the data map, and holds a copy of the same conversation under a run id rather than a customer id — so the query that finds a buyer's data does not find it.

Enumerate every store and write the list down. At Sundry that is five: the ticket database, the trace backend, the long-term memory from Chapter 6, the analytics warehouse built from run records, and the 120-ticket eval set, which Chapter 6 already flags as a governed copy with a named owner rather than a permanent private archive. Then verify it end to end rather than believing it: a synthetic buyer is created monthly, run through the real queue, deleted through the real request path, and searched for afterwards in all five stores. The first run of that check found the buyer in two of them.

Logs as an Injection Surface

The operator console that renders a run for a support lead is a web page, and the content it renders was written partly by the customer and partly by a model repeating what a seller's product description said. Chapter 12 makes the general argument; the unglamorous instance is here. Render model output and tool results as text, escape them at the boundary, and never let a console interpret markdown or HTML that arrived from a tool result — a link that leaks the run id in a query string is an exfiltration channel that runs on a reviewer's browser rather than the agent's.

The same applies one layer down. A newline inside a tool result can forge a log line that looks like it came from your own code, and control characters in a log stream will do surprising things to a terminal viewer. Strip control characters at ingestion and log structured records rather than concatenated strings, which removes the whole class. Anyone who has been in the field a while knows this as log forging, decades old, and the only new part is that the untrusted text now arrives through a component that is happy to repeat it verbatim.

Audit Versus Debug

The two trails answer different questions and want opposite properties. The audit trail records that a refund of a stated amount was issued against a stated order at a stated time, by a run, under an approval if one was required — structured rows, append-only, no model prose, kept with the money. The debug trail records what the agent had in front of it and what it said — verbose, sensitive, full of customer text, and worth very little after three months.

Put them in one store and one of them loses. Either the audit rows expire on the debug schedule, and finance cannot answer a chargeback from last year, or the customer text lives on the audit schedule, and a support queue's worst afternoons are retained for seven years because nobody separated them. Sundry writes audit rows into the same transactional database as the money movement, in the same commit, and treats the trace store as disposable — which is also why the audit trail is the one artefact that must not depend on the tracing backend being up.

Common Mistakes

Keeping full prompts and completions indefinitely because they are useful — the useful window is days, the liability is years, and the archive grows by 4,200 customers' worst afternoons a week.
Redacting at query time — the card number is already at rest in the store and in every backup, and the mask is a display setting that any backend query goes straight past.
A deletion path that covers the ticket database and misses the trace backend — the traces are keyed by run id rather than customer id, so the query that finds the buyer's data never touches them.
One store for audit and debug — either finance loses a chargeback record on the debug schedule, or customer prose is retained for seven years on the finance one.

Best Practices

Redact in the exporter at ingestion, with a tested redactor, and prove it against a labelled set of real ticket text rather than invented examples.
Set three retention windows — full content, structured records, audit rows — and write the justification for each next to the number.
Enumerate every store that holds run data and verify deletion end to end with a synthetic customer on a schedule.
Keep the audit trail structured, append-only and in the transactional store with the money, separate from the debug traces entirely.

Comparable toolsMicrosoft Presidio pattern and entity redactionOpenTelemetry span processors that redact at exportLangfuse masking hooks before storageLangSmith content-capture controlsCyberSecurity Deep Dive the data-handling discipline

Knowledge Check

Why is redacting card numbers when a trace is displayed rather than when it is written the wrong boundary?

Query-time redaction is far more expensive to run, because every read pays for the pattern matching
The number is already stored, backed up and copied downstream, so the mask hides an exposure that happened
Patterns applied at read time cannot be tested against real ticket text before they are deployed
Tracing backends have no way to transform span content on the way out to a console

Sundry keeps structured run records for 13 months but full prompts and completions for 90 days. What does the longer window buy?

The ability to answer when a change in behaviour began, months later, with no customer text kept
The ability to replay any run from the past thirteen months against a newly pinned model version
The evidence the finance team needs to settle a chargeback raised eleven months after the refund
A larger pool of real tickets from which the eval set can be rebuilt at the beginning of each quarter

A customer deletion request clears the ticket database and the long-term memory, and the trace backend keeps its copy. Why is that the usual gap?

Traces are written to immutable storage, so the records physically cannot be removed once written
The trace retention window is shorter, so the records are treated as already expired
Traces are keyed by run id rather than customer id, so the deletion query never finds them
Redaction already removed the customer's data, so the remaining spans are considered anonymous

What is the practical consequence of keeping the audit trail in the same store as the debug traces?

The store grows too large to query, because model prose dwarfs the structured rows in volume
Money movements can no longer be correlated with the specific run that produced them
The redactor has to run twice, once per trail, and the two copies drift apart over time
One retention rule has to win, so either the audit rows expire early or customer text is kept for years

You got correct