Key takeaways
- ✓The Inspector's CLI mode lists and calls tools as JSON, and reports each call through its exit code: 0 success, 3 authentication required, 5 tool error.
- ✓A smoke test is a list of calls and the exit codes they should produce. It runs after every deploy, in seconds.
- ✓Contract tests need no model and run on every commit against a real Postgres. They're deterministic, so a failure is always a real bug.
- ✓The tool-use eval runs golden cases through real models connected like any client, resets the data per case, scores every turn, and exits 1 when the weakest model is below 95%.
- ✓Test the scorer before trusting the eval, and treat "couldn't measure" as a block, not a pass.
Everything the earlier chapters checked by hand can be checked by a machine, on every commit, deploy and release. This chapter covers every suite in the acme-crm-mcp demo, from the Inspector's command line and a smoke test read from exit codes to contract tests with no model, a tested scorer, and a tool-use eval that gates a release: the command, when it runs, what it needs, how it reports, and a sketch of the CI wiring.
npm run smokeWhenAfter every deploy · secondsNeedsThe running server; no modelExit code0 all passed · 1 any check failedWhat it checks
Inspector CLI calls, checked by exit code (0 ok · 3 authentication required · 5 tool error): valid token: tools/list expect exit 0 search_contacts returns results expect exit 0 no token: authentication required expect exit 3 unknown tool: tool error expect exit 5 invalid email: tool error, nothing saved expect exit 5 Prints ✓ or ✗ per check, then "N passed, M failed".
A smoke test is a list of calls and the exit codes they should produce. It answers one question fast: is the deployed server broken?
The suites at a glance
| Suite | Command | When | Needs | Reports |
|---|---|---|---|---|
| Smoke | npm run smoke |
After every deploy | The running server | ✓/✗ per check; exit 0 or 1 |
| Contract | npm test |
Every commit | Docker Postgres; no model | 14 tests; exit 0 or 1 |
| Scorer | part of npm test |
Every commit | Nothing | 7 tests; exit 0 or 1 |
| Tool-use eval | npm run eval:tools |
Before a release | The running server, an API key | Per-turn ✓/✗, a gate line; exit 0 / 1 / 2; a JSON results file |
The Inspector, scripted
The same Inspector runs without the browser. It lists tools, with their hints, as JSON:
TOKEN=$(npm run -s token)
npx mcp-inspector --cli --transport http --server-url http://localhost:8787/mcp \
--header "Authorization: Bearer $TOKEN" --method tools/list
and calls them:
npx mcp-inspector --cli --transport http --server-url http://localhost:8787/mcp \
--header "Authorization: Bearer $TOKEN" \
--method tools/call --tool-name search_contacts --tool-args-json '{"query":"Priya"}'
Same server, same token, same answer as in the browser, but now a script can read it. And each call reports how it went through its exit code:
| Exit | Meaning |
|---|---|
| 0 | Success |
| 3 | Authentication required |
| 5 | The tool returned an error |
These are the codes of the Inspector version the demo pins (@modelcontextprotocol/inspector, locked in package-lock.json). Check them against yours before relying on them in CI.
Smoke: exit codes are the test
scripts/smoke.sh is just a list of calls and the exit codes they should produce:
| Check | Expected exit |
|---|---|
Valid token: tools/list |
0 |
search_contacts returns results |
0 |
| No token: authentication required | 3 |
Unknown tool (drop_all_tables): tool error |
5 |
| Invalid email: tool error, nothing saved | 5 |
Five checks in seconds. Run it after every deploy, and a broken server never stays broken for long. The negative cases matter as much as the positive ones: a no-token call that exits 0 means the server stopped checking tokens, and the smoke test catches it.
Contract tests: every commit
npm test starts its own authorization and MCP servers on separate ports (:18788, :18787) with a small rate-limit burst of 5, so 429s are quick to reach, and runs against the real Postgres from npm run db:up. No model is in the loop. The 14 contract tests cover every case from the last two chapters:
- 401: no token (and the metadata hint), expired, wrong audience, forged; the metadata names the authorization server
- 403: a read-only token can search but not write, and the denied write is audited
- 429: a flooding tenant is throttled while another isn't
- Tool routing: an unknown tool gets an error, not a crash; the tool list carries its safety hints
- Metrics: tool errors (bad arguments, unknown tools) are counted, not just HTTP errors
- Isolation, checked in the database: no cross-tenant results, no cross-tenant delete even by exact id, a smuggled tenant id ignored, every successful write audited
They're deterministic, so they must all pass: a failure is always a real bug.
The scorer, tested first
The same npm test runs the scorer's own tests. Before you trust a score, you check the thing that produces it. A real run:
✓ tests/scoring.test.ts > scoreTurn > passes search then create with the right arguments
✓ tests/scoring.test.ts > scoreTurn > accepts free wording in the search and case differences in values
✓ tests/scoring.test.ts > scoreTurn > fails create before search on order alone
✓ tests/scoring.test.ts > scoreTurn > fails a wrong key argument
✓ tests/scoring.test.ts > scoreTurn > fails a forbidden call even when everything else is right
✓ tests/scoring.test.ts > scoreTurn > allows one extra search but not a second create
✓ tests/scoring.test.ts > scoreTurn > fails a missing expected call
Tests 7 passed (7)
Chapter 2 covers the scorer's four checks and lets you run it on real and broken traces.
The tool-use eval: the gate
Contract tests can't tell you whether an AI model uses the tools well. The eval does:
npm run dev # in another terminal
npm run eval:tools # claude-opus-5-5
npm run eval:tools -- --models claude-opus-5-5,claude-sonnet-5-5,claude-haiku-4-5
npm run eval:tools -- --case delete-needs-consent --effort low # one case
For each model and each case it:
- resets the database to the seed, plus the case's own seeded rows (Dana present or not)
- connects a real Claude model to the server like any client, with the
acme-agenttoken - sends the request (and the follow-up reply, for two-turn cases), and scores each turn's tool calls on the four checks
- prints ✓ or ✗ with the trace and any problems
Then the summary is the gate. A recorded run:
━━ summary ━━
claude-opus-5-5 100% (5/5 turns)
gate: weakest model 100% ≥ 95% → PASS
| Exit | Meaning |
|---|---|
| 0 | The weakest model reached the gate (95% by default, --gate to change) |
| 1 | BLOCK RELEASE |
| 2 | The run itself failed: no verdict |
Each run writes evals/results/<timestamp>.json with every turn's calls, score, problems and the model's reply, which is the evidence you need to fix a block. It costs a few cents per model. Refusal fallbacks are deliberately off, so each run measures the model it names; a refusal counts as a failed turn.
The demo's recorded runs used one model, so its "weakest model" is that model. The video's 98% / 96% / 91% example (Chapter 2) shows why the gate looks at the weakest when you support several.
Wiring it into CI
The demo doesn't ship a workflow; this is a sketch of how the suites map onto one:
on: [push]
jobs:
contract: # every commit: no model, deterministic
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 20 }
- run: npm ci # installs the pinned Inspector too
- run: npm run db:up
- run: npm test # contract + scorer
# release job (manual or on tag): npm run dev in the background, then
# npm run eval:tools -- --models <every supported model> with the API key as a secret
# and keep evals/results/*.json as an artifact
# deploy job: npm run smoke against the deployed URL (MCP_URL=...)
Two rules make it trustworthy: a block attaches the failing cases (the results JSON), and an eval that didn't complete (exit 2) blocks the release like a failed one. Couldn't measure is not a pass.
Practice: from exit codes to a decision
Write the glue: a smoke runner that compares exit codes and prints ✓/✗ lines, the eval's gate summary (including the "no verdict" case), which suites run on a commit, a deploy and a release, and the release decision that reads every suite's exit code.
Quiz
1.In the demo's smoke test, a call without a token is expected to exit with which Inspector CLI code?
2.Which suite should run on every commit, and why?
3.Why does the eval reset the database before each case?
4.The tool-use eval exits with code 2. What should the release pipeline do?
5.Why are the scorer's own tests part of the suite?
6.The CLI's exit-code meanings come from one Inspector version. What should you do in CI?
Check your understanding
Try each question before opening its answer.
What does the Inspector CLI give you that the browser doesn't?
Scriptable calls with JSON output and a meaningful exit code per call (0 success, 3 authentication required, 5 tool error), so a script or CI can check the server without anyone clicking.
What is a smoke test here, and when does it run?
A list of calls with the exit codes they should produce, including negative cases (no token → 3, unknown tool → 5). It runs after every deploy and takes seconds.
Why run contract tests on every commit but the tool-use eval only before a release?
Contract tests need no model, so they're cheap and deterministic: any failure is a real bug. The eval uses real models, costs money and varies between runs, so it runs at the release boundary.
What does the eval do before each case, and why?
It resets the database to the seed plus the case's own rows, because the right answer depends on the data. Without the reset, one case's writes could change the next case's expected calls.
What should a failed or incomplete eval give the team?
A block, plus the evidence: the results file with every turn's calls, the score's problems, and the model's reply. Exit 2 (the run failed) blocks too, because there's no verdict.
Why pin the Inspector version?
The exit-code meanings belong to a version. The lockfile pins it, and the smoke test's negative cases make a changed contract fail loudly instead of passing silently.
Next up: Chapter 7 puts a real AI client on the server and reads its tool trace, not just its chat.