Key takeaways
- ✓A flood stays inside the flooding tenant's bucket: after the burst of 20, every call gets 429 with Retry-After, while the neighbour still gets 200.
- ✓Bad arguments are rejected by the input schema before any code or database runs. It's a tool error, a normal answer a model can read and correct, so the protocol log says OK.
- ✓Only registered tools can ever run: an invented one gets "Tool … not found", and the Inspector CLI exits with code 5.
- ✓Deleting another tenant's row by its exact id gets the same answer as a missing id, so ids can't be probed. The row is untouched, and the attempt is audited.
- ✓A failure isn't safe until you've checked its side effects: the neighbour, the rows, the audit log and the counters, not just the response.
A server shows it's safe by how it fails. Flood one tenant, send a bad argument, call a tool that doesn't exist, delete another tenant's row by its exact id: each must fail, and fail without touching anything it shouldn't. This chapter runs each case against the acme-crm-mcp demo, with the commands, the exact responses, and what to check beyond the response.
120 calls a minute, bursts of 20: each bucket refills at 2 calls a second. Every call spends from its own tenant's bucket, failures included.
Responses, newest first
- Send something.
GET /metrics
- (empty)
Setup
export ACME=$(npm run -s token)
export GLOBEX=$(npm run -s token -- globex-agent)
# one tools/call over plain HTTP; prints the status code
call() { # call <token> <tool> <arguments-json>
curl -s -o /dev/null -w '%{http_code} ' -X POST http://localhost:8787/mcp \
-H "Authorization: Bearer $1" -H 'content-type: application/json' \
-H 'accept: application/json, text/event-stream' \
-d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":{\"name\":\"$2\",\"arguments\":$3}}"
}
npm run db:reset restores the seed data between experiments. The rate-limit buckets live in the server's memory, so restarting npm run dev refills them.
Flood one tenant
for i in $(seq 26); do call $ACME search_contacts '{"query":"a"}'; done; echo
call $GLOBEX search_contacts '{"query":"a"}'; echo
200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 200 429 429 429 429 429 429
200
(A slow loop can squeeze in an extra 200 or two: the bucket refills at 2 calls a second while it runs.)
| Rule | Value |
|---|---|
| Bucket | Per tenant, keyed by the token's org_id |
| Size | Bursts of up to 20 |
| Refill | 120 calls a minute: 2 a second |
| Empty bucket | 429 Too Many Requests, Retry-After: <seconds until there's room> |
The flood stays inside acme's own bucket; globex's next call is a 200. That's why the limit comes after authentication: the server needs the tenant to know whose bucket to use. A well-behaved client waits Retry-After seconds before trying again.
The buckets are in memory, per instance. Behind a load balancer with several instances, each would count separately, so a production deployment keeps them in a shared store like Redis.
Bad arguments
call $ACME create_contact '{"name":"Dana Kim","email":"not-an-email"}'
The input schema says email must be an email, so the call is rejected before any tool code or database runs:
HTTP 200 · result.isError: true
MCP error -32602: Input validation error: Invalid arguments for tool create_contact: … "path": ["email"], "message": "Invalid email address" …
The Inspector's protocol log says OK, and it's right: a tool error is a normal answer, not a broken request. A model reads the message and corrects its own mistake. Nothing was written, and nothing is audited, because no write was attempted.
A tool that doesn't exist
npx mcp-inspector --cli --transport http --server-url http://localhost:8787/mcp \
--header "Authorization: Bearer $GLOBEX" \
--method tools/call --tool-name drop_all_tables --tool-args-json '{}'
echo "exit $?" # exit 5: tool error
Asked directly, the server says the same: a tool error, "MCP error -32602: Tool drop_all_tables not found". Only the three tools the server registered can ever run; whatever a model invents, the server says no. The Inspector CLI's exit codes make this scriptable: 0 ok, 3 authentication required, 5 tool error (npm run smoke uses them).
A delete aimed at another tenant
call $ACME delete_contact '{"id":"c_201"}' # globex's Dana Kim, by exact id
call $ACME delete_contact '{"id":"c_999"}' # an id that doesn't exist at all
Both answer the same: "No contact with id …". To acme, globex's row doesn't exist (row-level security), so it can't be deleted, and because the answer is identical to a truly missing id, an attacker can't even learn which ids are real. A "forbidden" answer would have confirmed that c_201 exists.
Now check that the failure was safe, in the database:
docker compose exec db psql -U postgres -d crm -c "select id, tenant_id, name from contacts where id = 'c_201'"
docker compose exec db psql -U postgres -d crm -c \
"select tenant_id, client_id, tool, arguments, outcome, detail from audit_log"
| Check | Result |
|---|---|
c_201 |
Still there, still globex, Dana untouched |
audit_log |
acme · acme-agent · delete_contact · {"id": "c_201"} · error · not found |
The attempt is recorded with the id it aimed at.
Counting it all
curl -s http://localhost:8787/metrics
After the experiments above, on a freshly started server:
mcp_responses_total{tenant="acme",outcome="ok"} 20
mcp_responses_total{tenant="acme",outcome="429"} 6
mcp_responses_total{tenant="acme",outcome="tool_error"} 3
mcp_responses_total{tenant="globex",outcome="ok"} 3
mcp_responses_total{tenant="globex",outcome="tool_error"} 1
acme's three tool errors are the bad email and the two failed deletes; globex's is the invented tool, because it was called with globex's token. globex's ok count includes the search and the Inspector CLI's own handshake messages, which are responses too, so expect your ok numbers to vary a little.
Every response is counted per tenant and outcome, including tool errors that came back as HTTP 200. This is what the error-rate monitor from Chapter 1 watches, and the outcome points to the cause:
| Spike of | Points to |
|---|---|
| 429s | A runaway client |
| Tool errors | A model misusing a tool |
| 401s | A broken token rollout |
Assert the side effects, not just the response
Every case above needs more than its response checked:
| Case | Response | Then check |
|---|---|---|
| Flood | 429 + Retry-After |
The neighbour still gets 200; the 429s are counted for the right tenant |
| Bad arguments | Tool error | No row written; counted as tool_error |
| Unknown tool | Tool error, exit 5 | Nothing ran; counted for the caller's tenant |
| Cross-tenant delete | Same as a missing id | The other tenant's row is unchanged; the attempt is audited; counted |
A polite error says nothing about what happened underneath. The demo's contract tests check these in the database, not just in the response (Chapter 6).
Practice: failed, and safely
Turn the checks into code: label a response the way the server's metrics do (including tool errors that are HTTP 200), diff two snapshots of the table, list every way a "failed" call was unsafe (wrong outcome, changed data, another tenant touched, a write not audited), and tally responses per tenant and outcome.
Quiz
1.26 calls in a row with acme's token. What should you see?
2.Why must the rate limit run after authentication?
3.create_contact with email "not-an-email" fails, but the Inspector's protocol log says OK. Is that a bug?
4.With acme's token, delete_contact c_201 (globex's Dana) returns "No contact with id c_201." Why is that the right answer?
5.The cross-tenant delete returned the expected error. What else do you check before calling it safe?
6./metrics shows a sudden spike of tool_error for one tenant, and no 429s. What does it point to?
Check your understanding
Try each question before opening its answer.
What should a per-tenant rate-limit test prove?
After the burst of 20, the flooding tenant gets 429 with Retry-After, while a neighbouring tenant's next call still gets 200. Buckets are keyed by the tenant from the token, so the limit runs after authentication.
Why does a bad argument show OK in the protocol log?
It's a tool error: HTTP 200 with isError in the result. The input schema rejected it before any code ran, and the model can read the message and fix its call. Monitoring has to count it from the body.
What stops a model from calling a tool that doesn't exist?
Only registered tools can run. An invented one gets "Tool … not found" as a tool error, and the Inspector CLI exits with code 5.
Why should a cross-tenant delete look exactly like a missing id?
Any different answer, like "forbidden", would confirm the id exists in another tenant, letting a caller probe which ids are real. With row-level security, the row is simply invisible.
What do you check after a failure, beyond the response?
The neighbouring tenant still works, the data is unchanged (especially other tenants' rows), write attempts are in the audit log, and the outcome is counted in /metrics for the right tenant.
Next up: Chapter 6 turns all of this from clicking into automation: the Inspector's command line, contract tests, and tool-use evals in CI.