Back-of-the-envelope estimates, then measure

A back-of-the-envelope estimate is rough math from known numbers, done before you build anything: how many requests a second, how much storage, how many servers. It's how most system design starts, and it's rarely checked. This chapter checks one. The short version: the estimate said 11 servers and measuring said 2, and every number on the way, including a load test that made an idle server look overloaded, had to be checked before it could be trusted.

The examples come from Lab 01 (Back-of-the-Envelope, Then Measure) of the system design labs: scripts 01_estimate.py (the estimate), 02_percentiles.py (checking the percentile code), 03_measure.py (measuring one request at a time), 04_where_time_goes.py (splitting the time into parts) and 05_littles_law.py (load and Little's law). The service is small and runs on one laptop (an Apple M4 Max): a Python web app (FastAPI, one worker) in Docker, with Postgres 18 holding a million rows, and a client on the same machine sending requests. So the numbers are this laptop's. The method is what carries over.

The envelope

The product is a URL shortener: short links that send you on to long ones. Opening a short link is one read (find the row by its id), and making a new link is one write. The inputs are typical interview numbers, not data:

• 10M daily users, each clicking 20 short links and making 0.2 new ones a day.
• A 3× peak: the busiest hour gets three times the daily average.
• 1,000 requests a second per server, the usual whiteboard guess, keeping 30% of each server spare.

clicks a second  = 10M × 20 ÷ 86,400 s          ≈ 2,315
at peak          = (clicks + new links) × 3     ≈ 7,014
usable per server = 1,000 × (1 − 0.3)            = 700
servers          = 7,014 ÷ 700 = 10.02 → round up = 11

The same script also gives 69 writes a second at peak, 1.09 TB a year of new data with 3 copies, and 28 Mbit/s going out. All small. The last line, 11 servers, is the one everybody asks about, and it rests on a guess.

Which guess matters?

Every input is a guess, so the lab moved each one across a range you could defend in a meeting, keeping the others at their defaults:

What one server handles moves the answer most. And it's different in kind: the business can give you the user count, and traffic history can give you the peak, but nobody can hand you requests per server for your service. You have to measure it. The peak factor comes second in risk for a similar reason: it depends on time zones, events and launches, not on anything you control.

A latency budget, saved first

The second estimate is how long one request should take. The lab writes it as a budget: named parts that add up, each a reference number (the rounded "latency numbers every programmer should know") or a guess labelled as one. For the lookup, GET /k/{id}:

Two habits matter more than the numbers. Parts, not one number: if the total turns out wrong, the parts show which line was wrong, or missing. And saved before measuring: the script writes the budgets to data/predictions.json with a timestamp and won't overwrite it without --force. A prediction written after the result isn't a prediction.

Measure, with percentiles you can trust

Requests don't all take the same time, so you describe them with percentiles. p50 (the median): half were faster. p99: 99 in 100 were faster, so it describes the slowest 1 in 100. An average hides those slow ones, and users notice them.

Before trusting a percentile, the lab checked the code that computes it, on simulated latencies where the true answer is known. Its percentile function matches numpy's default (linear) exactly. Then it checked two ways of putting a 95% interval around a percentile, by counting how often each covered the truth over 300 simulated samples:

The bootstrap (resample your data, recompute the percentile many times) can never imagine a value bigger than your slowest request, so in the tail it's too confident: for p99.9 from 500 requests, its "95%" interval missed the truth more than half the time. The order-statistic interval uses the fact that the count of samples below the true percentile follows a binomial distribution, and when there are too few samples it says so: no upper bound. That's the one the lab reports. (The lab's first version of it clipped the upper end to the slowest request, which looked tidy and also covered only 41% of the time. Checking the checker caught it.)

The rule underneath: a percentile is only as good as the number of samples beyond it. An interval has an upper end only once about 3.7 samples are expected above the percentile: about 370 requests for p99, about 3,700 for p99.9. To pin one down, you want 10 or more beyond it: about 1,000 requests for p99, about 10,000 for p99.9.

Then the lab measured each endpoint, one request at a time over one kept-alive connection, after 200 warm-up requests it threw away:

Every endpoint was faster than predicted, by 1.4–2.5×. The lookup was twice as fast as its budget. For choosing what to build, that's a good estimate: the right size. But a total can be close while its parts are badly wrong.

Where the time went

To see inside a request, the app adds a Server-Timing header to every response with two numbers: the time spent waiting on the database, and the time spent in the whole app. The client knows its own total, so subtracting splits each request in three: outside the app (the client, the network, Docker, the web server reading the request), app code, and database. For the lookup's p50:

• The database part was all trip. The budget charged a datacenter round trip for a hop between two containers on one laptop. Inside Postgres, the query ran in 8 µs; even an empty query, timed from the laptop, takes about 136 µs to go there and back. A faster query wouldn't help. Fewer trips, or a cache, would.
• A part nobody named. Timed separately with a raw socket, the client library (httpx) cost about 0.22 ms, more than the app code, and Docker Desktop's port forwarder about 0.11 ms. Neither was in the budget.
• The write missed two round trips. A write is three trips to Postgres (BEGIN, INSERT, COMMIT), not one. The budget's 1 ms flush to disk turned out to cost about 80 µs in Docker's VM, but a flush forced through to the Mac's own storage took 4.0 ms, so the VM's flush most likely stops at a cache. Write latency on a laptop VM says little about durable writes on a server.
• The full scan was right for the wrong reason. 100 ns a row predicted, 77 ns measured on one core. But EXPLAIN ANALYZE in psql ran the same query in 29 ms, using extra parallel workers. The app's call (asyncpg's fetchval()) asks Postgres for at most one row, and Postgres doesn't run a query in parallel when the request has a row limit. The plan you test in psql isn't always the plan your app gets.

One server's limit

Back to the guess that moved the answer most: how many requests one server handles. The lab ran C clients at once against the lookup, each sending its next request as soon as the last one came back, for 8 seconds per level, from up to 8 client processes:

The line levels off once the app, one Python process, uses all of its one core. After that, more clients only wait in line: p99 went from under 2 ms to 18. One server did about 6,800 a second, almost 7× the whiteboard's 1,000, and at that rate the shortener needs 2 servers, not 11.

That's the very top of the curve, though, where requests are already queueing. Sizing should use the rate at which p99 still meets your target, plus room for failures and deploys (and at least 3 servers for redundancy). It's also one Python worker on a laptop, reading by primary key with no cache.

Little's law, a free check

Can you trust those numbers? Little's law says that for any stable system, the number of requests in flight equals the arrival rate times the time each spends inside:

L = λ × W

A coffee shop where 10 customers arrive a minute and each stays 5 minutes has 50 inside. At 64 clients: 6,777 requests a second × 9.42 ms ≈ 64 in flight. It checks out, and it held to 0.3% at every level. It also holds for any part of the system: the app counted its own in-flight requests and got 38.5 of the 64, and the arrival rate times the app's own time per request also gave 38.5. Two instruments, one answer, so the other 25.5 were waiting outside the app.

Two uses, then: a check (measure L, λ and W separately; if they disagree, an instrument is wrong) and a locator (subtract to see where requests are waiting). The check caught a bug while the lab was built: the app's in-flight counter came out at −2.84 after a reset, and λ × W disagreed at once.

The collapse that wasn't

Same server, same 64 requests in flight, but sent from fewer client processes:

From one process, the server looked 15× slower with a 1.2-second p99, as if it were falling over. But the app used a sixth of its core, and Little's law showed where the requests were: 63.8 of the 64 were outside the app. One Python process juggling 64 connections couldn't keep up, so the requests were stuck in the load generator. Before you believe a server's limit, check the tool that's pushing it. (The lab hit this for real at first, with 4 client processes, before it became an exercise.)

Caveats

• One laptop. Client, app and database share one machine's cores, through Docker Desktop's VM and port forwarder. Every number here is this laptop's, and runs drift: the 5,000-request p99 for the lookup (0.84 ms, 0.82–0.87) doesn't overlap the 20,000-request one (0.89 ms, 0.88–0.91).
• One request at a time is not real traffic. These measurements are closed-loop: the client waits for each reply, so when the server stalls, the client stalls too and never sends the requests users would have sent. That hides stalls; the next lab measures how much.
• The 6,800 is a flat top, for one Python worker doing primary-key reads with no cache. Size from a point left of the knee, not the top.
• The laptop's flush isn't a server's. About 80 µs in the VM against 4.0 ms forced through to the Mac's storage.
• The interval numbers are simulated. The coverage table uses simulated latencies with a known answer, 300 samples per row, ±3 points.

The checklist

1. Write the load estimate down, from product numbers to requests a second at peak, then servers.
2. Find the input that moves it most, and measure that one. Usually it's what one server does.
3. Write the latency estimate as named parts, each a reference number or a labelled guess, and save it before measuring.
4. Report percentiles with intervals, and count the samples beyond each: 1,000+ requests for a p99, 10,000+ for a p99.9.
5. Split measured time into the same parts (Server-Timing, a raw-socket client, EXPLAIN ANALYZE) and see which line was wrong or missing.
6. Check L = λ × W with separate instruments, and use it to find where requests wait.
7. Check the load generator (its CPU, and where the in-flight requests are) before you believe a server's limit.

Practice: estimate, then check

Write the lab's arithmetic: requests a second at peak from product numbers, servers with 30% kept spare, a percentile computed numpy's default way, the expected samples above a percentile and the smallest sample with a bounded interval, and Little's law. The tests use the recorded numbers: 7,014 at peak, 11 servers at 1,000 a second and 2 at 6,777, 368 requests before a p99 has an upper bound, and 6,777 a second × 9.42 ms ≈ 64 in flight, 38.5 of them inside the app.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Load testing and tail latency (coming soon): every measurement here waited for each reply, so a one-second freeze on the server also froze the client. The next chapter builds a load generator that keeps sending, and measures how much a test like this one hides.