Chapter Eight · Background Work
Background Work
Everything the response does not depend on leaves the request, and the worker that picks it up has to be written for a queue that delivers every job at least once. Five topics decide what belongs in a job, run the loop against one Redis stream as its client, make every handler safe to run twice, add a scheduler, retries and a dead-letter stream, and put the one number that matters on the dashboard. This is the chapter that closes the third wound: ticket emails 40 minutes late.
On the night of the spring on-sale the ticket PDFs were rendered inside POST /orders: 4 seconds of CPU on the event loop per order, every other request on the instance waiting behind each one, and by the end of the night a queue of requests 40 minutes long with the emails at the back of it. Chapter 1 showed why a computing task freezes a loop, and Chapter 7 built the outbox that lets a handler hand work to another process without a second system in its commit. This chapter finishes the move. Marek takes the render and the email out of the request, puts the job on the jobs stream with the workers consumer group, and writes the handler so that running it twice produces one email.
The queue is one Redis stream, taught from the client's side and nowhere deeper: XADD to add, XREADGROUP to claim, XACK after the work, XAUTOCLAIM for what a dead worker left behind, MAXLEN so the stream does not become Redis. Which broker to choose, and what each promises inside, is Middleware Deep Dive's subject. What this book owns is the consequence the reader cannot avoid on any of them: the entry that was claimed and never acknowledged comes back, so the handler must reach a state rather than do an action, the side effect must have a row, the provider must get an idempotency key, and the job id must be recorded in the same commit as the effects it caused. Retries live in a per-kind policy, the poison job goes to jobs:dead with its traceback, and the schedule fires once because a lock says so.
The wound closes twice. The render leaves the request and the response returns after 90 milliseconds of handler time. Then, on the second on-sale night, the emails start running late again for a different reason, one worker against 3,000 orders in 10 minutes, and this time the dashboard says so at 4 minutes, because the age of the oldest job is the alert and not the depth of the queue. Two more workers, and the age line turns down. The arithmetic that says three workers were needed, 4 seconds of CPU against 8 threads against 5 orders a second, was available before the night, and the last topic is about doing it then.