Cost and Latency vs a Simpler Tool

The classifier run went well enough that it changed how the office thinks, and one Monday a colleague arrives with the natural next idea. Waymark's shared inbox takes a few hundred messages a day. Why not run every one of them past the model, and have it flag the urgent ones?

Tessa reads invoices now, so her first move is arithmetic rather than enthusiasm. A few hundred messages a day, every day, forever, each one a request with the message inside it. Then her second move, which is the one this page is about: she reads twenty urgent emails and notices that nineteen of them contain the word urgent, or asap, or a departure date inside forty-eight hours.

A rule that looks for those three things catches nineteen out of twenty. It costs nothing to run, answers before the message finishes arriving, and gives the same answer on the same email every single time.

So the question this page installs is not the one the office has been asking. Not can the model do it? — by now the answer to that is usually yes, and it has stopped being interesting. The professional's question is: what is the cheapest thing that does this well enough?

The Meter and the Clock

Two properties of the model make the question worth asking at all, and Chapter 7 taught both.

It has a meter. Every call costs tokens in and tokens out, priced separately, and the total is knowable to the cent. One ask is trivially cheap; a whole month of Waymark's asks came to $19, itemized into three readable lines. But cost per use multiplied by every message, every day, forever, is not a trivial number. It is a budget line, and it grows with the business rather than staying still.

It has a clock. A request goes out over the network, the model assembles an answer token by token, and the reply comes back — seconds, not milliseconds, and occasionally a pause while the provider says slow down. For a question you asked and are waiting for, seconds are nothing. For something that must happen the instant an email lands, seconds are the design.

Now put the alternatives beside it. A search over records you already hold. A filter rule in the mail system. A formula in a spreadsheet. A saved template with blanks in it. Every one of those costs nothing per use, answers instantly, and gives an identical answer every time it runs — which, one page ago, was exactly the thing the model cannot promise.

None of that makes the model a bad tool. It makes it a metered tool, and metered tools are for work that needs what they uniquely do.

The Ladder

Tessa's habit now is four rungs, climbed in order, and she stops on the first one that holds.

Does a search find it? If the answer already exists somewhere — in the booking system, in last season's file, on the supplier's own site — the job is retrieval, not generation. Retrieval gives you the real value rather than a plausible one, and it costs nothing. Asking the model instead is how the Harbourview Annex happens.

Does a rule decide it? If the decision can be written as a condition — contains this word, falls inside this window, is over this amount — write the condition. It fires identically forever, anyone can read it, and it costs nothing per use. The urgent-email flag lives here.

Does a formula compute it? If the answer is arithmetic, the tool that computes should compute it. That is Chapter 4's spreadsheet rule, now standing on a rung of its own, and it is free too.

Then, and only then, the model. Rung four is for work where language, judgement or reshaping is the job: drafting, rewording, summarizing, explaining, and sorting things whose sorting needs an actual reading. Those are the verbs from Chapter 4, and they are what the meter is for.

The ladder's value is not that rung four is bad. It is that most tasks fall off the ladder before they get there, and until you climb deliberately you will not notice.

Auditing Her Own Bill

Here is Tessa applying the ladder to her own work, which is harder and more useful than applying it to a colleague's.

Chapter 8 finished with a month of API work re-engineered from $19 down to $6: seven dollars saved by sending less, three by receiving less, three by moving the classification to a smaller model. Notice that all three of those savings live inside rung four — the same work, given to the model more carefully. The ladder asks the question underneath them. Should this have gone to the model at all?

Start where the answer is yes. The season's classification run cost $2.40 — 2,300 reviews, one request each, in about forty minutes. Put it on the ladder honestly. A search cannot find which bucket a review belongs in; nobody wrote that down. A rule cannot decide it: a review that grumbles about the coach for two sentences and then thanks the guide warmly is a judgement, and any word-matching rule you write for it will be wrong within a page. There is no formula. It is rung-four work, and the $2.40 is earned.

Then the near miss.

Some of Waymark's reviews arrive with a star rating attached and some are prose only, and next season's trends report wants them separated. Tessa's first sketch adds a step to the program: for each review, a second request asking the model whether that review contains a star rating, yes or no.

Milo prices it before writing a line. Another 2,300 requests, each carrying the whole review text — about $2 a season, on top of the $2.40, to nearly double the run's bill. And the question is does this text contain a star rating, which a one-line rule answers exactly, instantly, and identically every time, for nothing.

The step was never built. What is worth taking from that is not the two dollars, which is small — it is that Tessa, who wrote the template and ran the sample and understood the invoice, still reached for rung four by reflex on a rung-two question. Right-sizing is not a decision you make once about a project. It is a habit you apply per step, and the step that slips past is always the one attached to something the model genuinely should be doing.

Stacking the Rungs

Which leads to the arrangement that wins in practice, and it is not a choice between tools at all.

The strongest designs stack the ladder. Cheap deterministic tools skim off everything that can be decided without judgement — free, instant, auditable — and the model is handed only the remainder, where judgement is genuinely the work. The bill is not reduced by finding a cheaper model. It is reduced because most of the volume never reached the meter.

Next season's review run has that shape. The rule reads each review and records whether a star rating is present. The model classifies the bucket, which is what it is for. Same output, same quality, and the invoice shows one metered step instead of two.

It is the arrangement the office already uses for the post. Waymark could hire a brilliant consultant to sort the morning mail, and they would do it well. What actually happens is that the mailroom has a slot labelled invoices and another labelled supplier post, and those slots handle most of it for nothing — and the consultant is brought in for the twenty letters the slots cannot judge. Nobody thinks that is a compromise. It is what a well-run mailroom looks like.

So the ladder is the first half of a bigger question: when should the model touch this task at all? Cost is one of the four things that decide it. The next page has the other three, and the card they fit on.