Self-hosting vs. an API: what one email really costs

The shop's support emails are sorted by an AI model, and today the shop pays OpenAI for every email. In the fine-tuning chapters, a small model learned to do the same job; in the last two chapters it was shrunk to 4 bits and served to many users at once. Now the question that started it: should the shop run that model itself, or keep paying per call? This chapter compares right answers, speed and cost, and then what the comparison leaves out.

The source is Lab 12 (Inference & Serving), script 12_self_host_vs_api.py, recorded in October 2026 on one laptop (Apple M4 Max). It serves Lab 11's fine-tuned Qwen3-0.6B, merged and quantized to 4 bits (0.34 GB, still 99% right; chapter 29), with mlx_lm.server, and runs it against two OpenAI models on the first 100 of Lab 11's test emails. The emails are made up, with known right answers, and the fine-tune trained on emails just like them. The hourly machine prices are what-ifs, not quotes. The lab's API half (streaming, prompt caching, the Batch API, semantic caching, cascades) isn't in the video; one of its results, the cache discount, matters here and is quoted below.

The same code, a different address

An app talks to a model through an API: a standard way for one program to send another a request and get an answer back. OpenAI's API is the one most client code is written for.

mlx_lm.server is an OpenAI-compatible server: it runs the model on the laptop and accepts the same requests as OpenAI's API. So the app keeps its library and its call. The lab's script starts the server like this:

python -m mlx_lm server --model data/models/ticket-0.6b-4bit --port 8090 \
  --chat-template-args '{"enable_thinking": false}' --max-tokens 60

and talks to both with the same openai client:

api = OpenAI()                                                         # OpenAI's servers
local = OpenAI(base_url="http://localhost:8090/v1", api_key="not-needed")  # this laptop

r = client.chat.completions.create(model=model, messages=msgs, **options)

Only settings change: the address, the model name, the prompt, and a couple of call options (the API calls use reasoning_effort="none" and max_completion_tokens, the local ones max_tokens and temperature=0). Switching between the paid model and your own is a config change, which also makes a fair race easy. A model gateway gives you the same thing across many providers.

Right answers: a tie, and a clear loser

The race: the same 100 emails, one at a time. For each, the model fills in four fields: what the customer wants, the order number, the product, and how urgent it is. An answer counts as right only if all four are.

Is 99 against 97 a real difference? A paired test (McNemar's) looks only at the emails where the two disagree. Mine and mini disagreed on just 4: three went my way, one went mini's. That's like three heads in four coin flips (p = 0.62), so it's a tie. Nano's 75 is clearly behind (p < 0.001).

Remember what the tie means: on emails like the ones it practised on, the narrow model matches the bigger one. It can't do anything else (the fine-tuning regressions chapter shows what it lost).

Speed: a tenth of a second

Time is the full round trip as the app sees it: localhost for mine, the internet for the API.

The slow end (95th percentile) is the time one email in twenty took longer than. Mini's typical email took 9× as long as mine, and mine barely has a slow end: 0.13 s against 0.11.

Why so much faster? Several things differ at once: no trip over the internet, a prompt 15× shorter than mini's (the fine-tune learned the rules, so it doesn't need them spelled out), and almost certainly a much smaller model (mini's size isn't published). The lab didn't measure which of these matters most.

Switch between the measures:

Many at once

So far, one email at a time. How many can the laptop handle? The lab sent 64 emails per setting from 1, 4, 16 and 32 simulated users (threads sending back to back):

The server works on many emails in one pass, letting new requests join mid-way (continuous batching, chapter 30). Throughput nearly triples, and each email waits longer: at 32 users a typical email takes about as long as mini's. Even so, the slow end, 1.52 s, is still under mini's slow end with a single user (2.09 s). 32 was the most the lab tried, and throughput was still rising.

What one email really costs

The API charges per email: at list price, $1.46 per 1,000 for mini and $0.18 for nano. Your own machine charges per hour, busy or not. Like a gym membership, the more you use it, the less each visit costs:

cost per 1,000 emails = machine cost per hour ÷ emails handled per hour × 1,000

With the measured 87,263 emails an hour at full use, and what-if hourly prices (a laptop you own costs its power and wear; a rented GPU its rental):

At $2 an hour, always busy, that's about 2¢ per 1,000. Handling only 5% of what it could, 46¢ per 1,000: the same machine, 20× more per email, because it sits idle most of the hour.

A $5-an-hour machine used 5% of the time comes to $1.15: under mini's $1.46, just. But $1.46 is list price. OpenAI charges a tenth of the input price for the start of a prompt it has seen recently, and mini's ~1,650 tokens of instructions and examples are the same on every call. The lab's own run didn't pass cached tokens to its cost function, and the discount couldn't be measured afterwards. An estimate from OpenAI's prices: if 1,024 to 1,536 of mini's 1,766 prompt tokens were cached on every call, mini would cost about 42–77¢ per 1,000, maybe less if more is cached. Then the mostly idle $5 machine loses, and the $2 machine at 5% (46¢) is inside the range. Nano's 655-token prompt is under the cache's 1,024-token minimum, so its 18¢ stands.

The cache discount is not a guess about whether it exists. In Lab 12's Part 1 (03_prompt_caching.py), from the second call on, 1,792 of about 2,860 input tokens came from the cache and the cost per call halved. It works in steps: the first 1,024 tokens, then 128 at a time. What isn't known is how much of mini's prompt hit the cache in this run.

Pick a machine price and how busy it is:

Nano's 18¢ beats the $2 and $5 machines used 5% of the time, and it gets a quarter of the emails wrong. Compare right answers first; a cheap wrong answer isn't cheap.

What the table leaves out

• People. Someone has to run the server, watch it, and fix it when it breaks. That's not in the hourly price.
• Change. When the shop adds products or changes its rules, the paid model reads a new prompt. The fine-tune needs new training data and a new training run (Lab 11).
• Volume. Say the shop gets 5,000 emails a day. That's 3.4 minutes of work for this server. Kept on all day at $2 an hour, it costs $9.60 per 1,000 emails, far more than mini. Unless the machine is free anyway, or privacy or speed forces it, paying per call is simpler at that volume, and nano or a cascade (a cheap model first, a bigger one when it's unsure) may be enough.

When to run it yourself

• Keep the same client code for both, so switching is always a config change, and fallback to the other is easy.
• Self-host when the job is narrow, the volume keeps a machine busy, or the data can't leave the company. The win here came from three things together: a narrow model small enough to be fast (Lab 11), quantized to fit (chapter 29), and kept busy by batching (chapter 30).
• Otherwise, pay per call, and price the API with the cache discount you actually get, not list price.

Caveats

• Made-up emails, and the fine-tune trained on emails like them. The 99% holds for this narrow task; it says nothing about anything else.
• Hourly prices are what-ifs. Put in your own: hardware spread over its life plus power, or a rented GPU.
• Mini's cost is at list price. The cache discount wasn't recorded; the 42–77¢ range is an estimate from OpenAI's prices, not a measurement.
• One laptop, one run, one network. API times include the internet trip from one home connection on one day.
• Capacity is a measured point, not a ceiling. 87,263 an hour is at 32 simulated users, the most tried, with throughput still rising. The simulated users are threads sending back to back, not real traffic.
• The speed gap isn't split by cause. Network, prompt length and model size all differ at once.

The checklist

1. Put both behind the same client: an OpenAI-compatible server or a gateway, so switching is a config change.
2. Compare right answers first, with a paired test on the same items. A gap of 2 on 100 can be a tie.
3. Measure latency as the app sees it: typical and slow end, at the concurrency you expect.
4. Measure throughput under load, then price a self-hosted request as hourly cost ÷ requests per hour × 1,000.
5. Use how busy the machine will really be, not its peak. At low volume an always-on machine costs more per request than the API.
6. Price the API with its discounts: cached prompt tokens, batch pricing.
7. Count the people and the retraining before you move a model in-house.

Practice: price the race

Write the lab's nearest-rank percentile and paired McNemar test, then both sides of the bill: an API call's cost per 1,000 (with and without cached tokens) and a self-hosted email's cost from the hourly price, the throughput and how busy the machine is. The tests use the recorded run: 0.11 s and 0.13 s for the fine-tune, p = 0.625 for the tie with mini, $1.46 at list price and 42–77¢ with the estimated discount, 46¢ at 5% busy, and $9.60 for 5,000 emails a day.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 32 starts on resilience: what to do when a model call fails, and how retries can make an outage worse unless they wait the right way.