Backend Deep Dive
by Sergey Okinchuk
Ch. 1

Backend Deep Dive

Welcome

You have called an API from the outside. This book is about the inside: how a request becomes a typed value at the edge, how HTTP methods and status codes are a contract that a retrying client acts on, how identity is established and authorization enforced where it cannot be skipped, and why two buyers were both sold seat 14C. Then the half that separates a service that works from one that survives: timeouts, retries, idempotency keys, the outbox, jobs that run twice, caches that lie, and a payment provider that answers late. One system, Stagedoor, carries every example from the first page to the last.

14 chapters 77 topics covered Knowledge check on every topic ≈23.5 hours to complete

About This Course

The network can fail between any two lines of your code. Every call that leaves the process, to Postgres, to Redis, to the payment provider, to the client that asked, can time out, arrive twice, arrive late, or succeed while its answer is lost, and the caller cannot tell the first case from the last. Nearly everything that makes a backend hard falls out of that one fact: the timeout on every call, the retry that is only safe when the operation is idempotent, the outbox that makes a row and a message one unit, the job handler that must tolerate running twice, the webhook that replays, the reconciliation that asks the neighbour for the truth. This book states that idea on page one and keeps cashing it in.

The second thing it returns to constantly is the boundary. A service has an edge, and at the edge three things happen exactly once: bytes are parsed into typed values, the caller is authenticated, and the outside vocabulary is translated into the inside one. Inside the boundary everything is typed, trusted and yours. Most of the bugs in the identity and data chapters are a check that ran inside instead of at the edge, or an outside value that leaked in without being parsed.

Every example comes from one running system. Stagedoor is an event-ticketing service: two API instances and a worker, one Postgres primary with a replica, one Redis, and a payment provider called Payrail on the far side of the internet. Marek owns that service for all fourteen chapters. Three failures open in Chapter 1 and are left deliberately unexplained: on the night of the spring on-sale, seat 14C was sold to two buyers, one buyer was charged twice after her browser retried a slow checkout, and ticket emails arrived forty minutes late. The seat is settled in Chapter 6, the double charge in Chapter 7 and the late emails in Chapter 8, each by the machinery the book has built by then rather than by a guess that happened to work.

Who This Is For

Engineers who have used an API and are about to own one, or already do and have been guessing. The bar is concrete: you have sent a request and read the JSON that came back, you can read a SELECT with a JOIN in it, and you have written some code in some language. The reference language is Python, with FastAPI where a framework is unavoidable, and every snippet has narrated prose beside it that stands on its own. A reader who writes Go or TypeScript should lose nothing but the syntax, because the framework is a detail and the design pressure is the lesson.

It is not a first course on APIs or on databases. If a status code or a primary key is a term you would have to look up, read APIs for Beginners and Databases for Beginners in this catalogue first and come back; this book assumes both and goes one layer down on every page. It is also not a book about the things between services or about composing many of them: brokers, gateways, meshes, sharding and multi-region belong to the courses named in Chapter 1, and this one stays with a single service and a single database on purpose.

What You Should Already Know

  • How to call an API: a method, a path, headers, a JSON body, and what a status code is telling you
  • Working SQL: SELECT, JOIN, what a transaction is for, what an index is for
  • Some programming in any language: functions, types, and reading a stack trace without alarm
  • Nothing about building a service: the boundary, identity, pools, transactions, retries, queues, caches, testing and observability are all built from zero

New to this? Start with APIs for Beginners and Databases for Beginners - the first chapter of every course is free, and the rest is one membership.

How the Course Is Built

The fourteen chapters run in three movements. The API half (Chapters 1–5) is the boundary: what a backend is and where its time goes, HTTP used as a contract rather than a transport, resources and shapes and errors a client can act on, the layers a service is made of, and identity from the password hash up to multi-tenancy. The reliability half (Chapters 6–10) is the core of the book: the pool and the transaction and the seat sold twice, timeouts and retries and the key that stops the double charge, the worker and the jobs that run twice, the cache and the four ways it lies, and one real neighbour with its webhook, its reconciliation and its saga. The operations half (Chapters 11–14) is running it: configuration and secrets and the thirty seconds after SIGTERM, a test suite that runs against a real database and a fake provider, logs and metrics and traces from inside the code, and what breaks first on the night twenty thousand buyers arrive at once.

Every topic has the same shape: an opening that says what the thing is and why it exists, the mechanics with real code and real exchanges, the specific mistakes that cause real incidents, the practices that prevent them, and a short knowledge check. Every mechanism gets a number, because a pool of twenty against a limit of two hundred, a three-second timeout, a ten-minute hold and a thirty-second grace period are the difference between predicting a service and guessing at it.

The network fails between any two lines
Stated in Chapter 1 and never dropped. Timeouts, retries, idempotency keys, the outbox, at-least-once jobs, webhook replay and reconciliation are the same fact seen from different boundaries, which is why they stop needing to be memorized separately.
The boundary is where you parse and authenticate
Bytes become typed values once, at the edge. The caller is identified once, at the edge. Inside, everything is trusted and typed, and a second check deeper in is a sign the boundary leaked.
One system, all the way through
Stagedoor's API is designed in Chapter 3, layered in Chapter 4, secured in Chapter 5, made correct under concurrency in Chapter 6, made resilient in Chapter 7, tested in Chapter 12 and load-tested for the on-sale in Chapter 14. The examples compound, and the three open failures get real answers.
Numbers over adjectives
Eight loops times twenty connections against a limit of two hundred. A three-second timeout to Payrail. Three thousand requests a second for two thousand seats. A P95 of four seconds for a PDF. An SLO of 99.9% under 800 milliseconds. Specific figures are what let you predict behaviour you have not seen yet.

Chapter Map

Chapter 1
What a Backend Is
The process behind the screen, one request followed through every layer with the milliseconds attached, the network fact the book rests on, the three concurrency models, and the line where this book stops.
Chapter 2
HTTP Done Properly
Methods as retry promises, status codes as instructions, the handful of headers a service must own, cookie attributes and what each defends, connections and TLS termination, and conditional requests.
Chapter 3
Designing the API
Resources instead of actions, shapes that do not break clients, validation exactly once at the edge, Problem Details errors, cursor pagination, versioning by date, and an honest REST, gRPC and GraphQL comparison.
Chapter 4
The Anatomy of a Service
Transport, domain and storage with one rule about imports, dependency injection with no framework, the request context, middleware rings and their order, and configuration read once at startup.
Chapter 5
Identity — Who Is Calling
Passwords hashed properly, sessions against tokens, the traps inside a JWT, OAuth and OIDC as a client, API keys and service-to-service auth, authorization where it cannot be skipped, and tenants in one codebase.
Chapter 6
Data Access
The pool as the real concurrency limit, transactions from the application's side, the ORM question, migrations that ship with the code, replicas and stale reads — and seat 14C, sold twice, settled here.
Chapter 7
Failure by Design
A timeout on every call, retries with backoff and jitter, the idempotency key that stops the double charge, circuit breakers and bulkheads, the outbox, and what the service does when each dependency is gone.
Chapter 8
Background Work
What belongs in a job, a Redis stream and a consumer group as the queue, handlers that tolerate running twice, scheduling and retries and dead letters, and the night the ticket queue fell forty minutes behind.
Chapter 9
Caching
What to cache and what never to, cache-aside with Redis, invalidation after commit and the stampede at expiry, HTTP caching from the server's side, and the four ways a cache lies.
Chapter 10
Talking to Other Services
The outbound client to the payment provider, webhooks received without trusting them, the nightly reconciliation, checkout as a saga with compensations, and contracts pinned on both sides.
Chapter 11
The Service in Production
The twelve-factor rules that still matter, secrets and the four places they leak, liveness against readiness and the drain after SIGTERM, resource limits measured rather than guessed, and the handoff to the container.
Chapter 12
Testing the Service
Three layers and the rule about boundaries, a real Postgres in a rolled-back transaction, a fake provider kept honest by a contract test, failure injected on purpose, and a load test of the on-sale.
Chapter 13
Observability From the Inside
One JSON line per event with the request id on every line, RED per endpoint as histograms, traces that follow a checkout into the worker, and an SLO with an error budget for one service.
Chapter 14
Performance and Scale
Where the time actually goes, scaling out and the sticky-session trap, rate limiting and backpressure, the waiting room for the on-sale day, async where it pays, and the honest day one service is no longer enough.

Disclaimer

This course is an independent educational project created and maintained by Sergey Okinchuk. It is provided for learning and reference purposes only.

No affiliation. This course is not affiliated with, sponsored by, endorsed by, or officially connected to any company, product, or project mentioned — including the Python Software Foundation, the FastAPI project, the PostgreSQL Global Development Group, Redis Ltd., Stripe, Adyen, Amazon Web Services, Google, or Microsoft. All opinions, interpretations, and recommendations expressed are those of the author.

Trademarks. Product and project names referenced — including "Python", "FastAPI", "PostgreSQL", "Redis", "Kafka", "RabbitMQ", "Stripe", "Kubernetes", "Docker", and "OpenTelemetry" — are the property of their respective owners. Use of these names is for identification and educational purposes only and does not imply any endorsement. Stagedoor and Payrail are fictional; any resemblance to a real product or company is coincidental.

Not operational advice. This material teaches how a backend service is designed and the practices that follow from it, not turnkey instructions for any specific environment. Code snippets, configuration values and numbers are simplified for learning and sized for one fictional workload. Always consult the official documentation and test on a system you can afford to lose before changing a production service.

Accuracy and currency. Libraries, protocols and providers change. Facts in this course reflect the author's understanding at the time of writing against Python 3.14, PostgreSQL 18 and Redis 8; version-dependent behaviour drifts. Always verify against the official documentation for the versions you actually run.

No warranty. This material is provided "as is" without warranty of any kind. The author accepts no liability for any loss or damage arising from reliance on the content.