Key takeaways
- ✓Sixteen messages into a debugging session, 96% of the agent's context was tool output: each piece mattered for one step, then was re-sent on every call.
- ✓Test a compaction by asking the agent about its own task, and count an answer only with the exact detail: the rule, the file and line, the files changed, the test names, the next step, a number.
- ✓Clearing old tool output (no extra call) and a structured summary (one call) both kept 5 of 7. A generic paragraph kept 3; dropping older messages kept none, apart from a false pass.
- ✓The structured summary copied raw log lines it was told to drop, hit its 400-token limit mid-list, and never wrote its current state or next step. The one-line error rate was lost by every method.
- ✓Skills keep only names and descriptions in the prompt (151 tokens instead of 1,258). The description decides the pick: vague descriptions with code names picked right 7 of 14. Loaded, a skill's format was followed 12 of 12 times, against 2 of 12 without.
An agent that works for a long time keeps adding to its own prompt: every tool result stays there, re-sent on every call. Compaction replaces the older part with something shorter, and the risk is that it drops what the agent still needs. This chapter compares four ways to compact against none, tested by asking the agent about its own task, then covers skills: instructions that load only when a request needs them.
The sources are ai-eng-labs Lab 07, scripts 3 (03_compaction.py) and 5 (05_skills.py). The agent's work log, the skills and the requests are made up; the answers are real recorded runs of gpt-5.4-nano, one run per test. Chapter 5 covers the prompt budget and huge tool output.
What fills an agent's context
A scripted debugging session. The user asks why payment webhook retries are hammering a partner API, and says not to touch the migrations/ folder. The agent reads the logs, reads retry.py, finds BACKOFF_MULTIPLIER = 0 on line 88, sets it to 2, adds a regression test, runs the suite (two retry tests still fail, because config/queues.yaml still says 3), and checks staging metrics.
Sixteen messages, 6,456 tokens: tool output is 6,225 of them (96%): a log dump, a whole source file, test results, metrics. Each mattered for one step, then sat there, sent again on every call. Compaction replaces older messages with something shorter.
The agent's work log: 16 messages, 6,456 tokens · tool output 96%
■ tool output · ■ user · ■ agent. The last 4 messages are kept as they are; everything older is compacted.
Named sections: goal, rules, findings, changes, current state, next step (max_tokens = 400). The only method that lost the next step.
- ✓Is there any folder I told you not to change? rule“Yes — you told me not to touch anything in the `migrations/` folder (another team owns it).”
- ✓What was the root cause of the retry storm? root cause“The retry storm was caused by the backoff logic being effectively disabled: - In `retry.py` at line 88, `BACKOFF_MULTIPLIER = 0 # TODO tune`. - That makes `next_delay(attempt: int)` compute `BASE_DELAY * (0 attempt)`, which yields ~`BASE_DELAY` for attempt 0 and then effectively no meaningful backoff for subsequent attempts (so retries fire immediately). - Webhook worker logs reflected this with repeated retries showing `status=503` and `delay_ms=0` / very small”
- ✓In which file and on which line was the bug? exact spot“The bug was in `retry.py`, on line 88: - `retry.py` line 88: `BACKOFF_MULTIPLIER = 0 # TODO tune`”
- ✓Which files have you changed so far? files changed“So far, I changed these files: - `retry.py` (line 88): updated `BACKOFF_MULTIPLIER` from `0` to `2` - `tests/test_retry.py`: updated/adjusted retry-related tests (to match the expected backoff/max-attempts behavior)”
- ✓Which tests are still failing? Give their names. failing tests“I don’t know which tests are still failing from the log you provided. The only test failures mentioned earlier were: - `tests/test_retry.py::test_dead_letter_after_max_attempts` - `tests/test_retry.py::test_max_attempts_config`”
- ✕What's your next step? next step“Next step: verify the fix end-to-end by re-checking staging webhook-worker retry behavior (delay_ms/backoff and max-attempt/dead-letter counts) and confirm the partner API is no longer being hammered—i.e., look for retries spreading out and dead-letter happening at the expected attempt threshold (8, not 3).”
- ✕What was the webhook error rate before the fix? a number“I don’t know the webhook error rate before the fix from the log you provided. The only staging error-rate data shown is after the backoff change (summary: 0.8%).”
What the compaction wrote
- Goal: what the user asked for
- Find out why Payments webhook retries are hammering the partner API.
- Rules from the user: anything they said to do or not do
- Don’t touch anything in the migrations/ folder (another team owns it).
- Findings: what was discovered, with exact locations and values
- `retry.py` line 88: `BACKOFF_MULTIPLIER = 0 # TODO tune`
- This causes `next_delay(attempt: int)` to return `BASE_DELAY * (0 ** attempt)`, resulting in effectively no backoff (retries fire immediately).
- Webhook worker logs show rapid repeated retries with `status=503` and `delay_ms=0` / very small delays:
- Example: `2026-03-09T14:10:05Z ... retry attempt=1 ... delay_ms=0 status=503`
- Example: `2026-03-09T14:10:25Z ... retry attempt=2 ... delay_ms=0 status=503`
- Example: `2026-03-09T14:12:05Z ... retry attempt=2 ... delay_ms=3 status=503`
- Test failures indicate dead-letter/max-attempts behavior is still configured to 3:
- `tests/test_retry.py::test_dead_letter_after_max_attempts` failed: expected dead-letter after 8 attempts, found after 3
- `tests/test_retry.py::test_max_attempts_config` failed: expected `max_attempts == 8`, found `max_attempts == 3`
- Changes made: every file changed and how
- `retry.py`
- Updated `BACKOFF_MULTIPLIER` at line 88 from `0` to `2` to enable non-zero backoff between retries.
- `tests/test_retry.py`
✂ hit its 400-token limit here: “Current state” and “Next step” were never writtenIt was told to drop raw logs, and copied three log lines anyway. The 14.2% error rate was one line in a 91-line log dump: every method lost it.
Testing a compaction
Keep the last 4 messages, compact everything older, then ask the agent seven questions about its own task. An answer counts only with the exact detail:
| Needs | Question | Must contain |
|---|---|---|
| Rule | Is there any folder I told you not to change? | migrations |
| Root cause | What was the root cause of the retry storm? | backoff |
| Exact spot | In which file and on which line was the bug? | retry.py, 88 |
| Files changed | Which files have you changed so far? | retry.py, test_retry.py |
| Failing tests | Which tests are still failing? Give their names. | both test names |
| Next step | What's your next step? | queues.yaml |
| A number | What was the webhook error rate before the fix? | 14.2 |
The number, an error rate of 14.2%, was one line inside a 91-line log dump.
Five ways to compact
| Method | Context tokens | Right (of 7) | Extra model call | What it lost |
|---|---|---|---|---|
| No compaction | 6,522, and growing | 7 | No | Nothing |
| Drop older messages | 1,474 | 1, a false pass | No | Its own task |
| Clear tool output | 1,734 | 5 | No | The test names and the 14.2%, which only lived in tool output |
| Generic summary | 1,645 | 3 | Yes | The line number, the files, the test names, the 14.2% |
| Structured summary | 1,880 | 5 | Yes | The next step and the 14.2% |
(6,522 includes the role labels and wrappers around the 6,456 tokens of messages.)
- Dropping older messages, the agent forgets its own task. Its one ✓ is a false pass: "I don't know from the log you provided… mentions a 'backoff fix'", and the grader only looks for "backoff".
- Clearing old tool output keeps the agent's own messages, where it wrote down its findings, with no extra AI call.
- A generic summary ("Summarize this conversation… in a short paragraph") kept the story and dropped the details an agent needs to carry on.
- A structured summary has named sections: goal, rules from the user, findings, changes made, current state, next step, keeping names, paths, line numbers and numbers exactly. Also 5 of 7, at about the same size as clearing tool output, for one extra call. And it's the only method that lost the next step.
The lab's grader is lenient in one more place: the structured summary's "failing tests" answer opens with "I don't know which tests are still failing", then lists both names, and counts as right. The practice writes a stricter grader.
Where the next step went
Read what the structured summary wrote. Its prompt said "Drop raw logs, file contents, and tool output once their findings are recorded above." It copied three example log lines anyway. Then it hit the 400-token output limit it was given (396 tokens) in the middle of the list of changed files, ending at "- tests/test_retry.py". "Current state" and "Next step" were never written. The plan, update config/queues.yaml, was gone; asked for its next step, the agent named a different one.
And the 14.2% was lost by every method: one line in a log dump, and no message ever wrote it down as a finding.
What to do about it (the lab didn't rerun with a bigger cap, so this is reasoning, not a measurement): give the summary room to finish and check that every section is there, have the agent write down findings as it goes, and test every compaction by asking the agent about its own task. On gpt-5.4-mini, the lab notes structured compaction kept 6 of 7.
Skills: know-how on demand
The other thing that piles up in an agent's prompt is instructions: how to review SQL, write a postmortem, answer a customer. A skill packages one: a name, a one-line description, and full instructions (a SKILL.md file, plus any scripts). Only the names and descriptions stay in the prompt; when a request matches, that skill's instructions load.
The assistant has six skills: api-design-review, customer-email, incident-postmortem, python-code-review, release-notes, sql-review. Their descriptions cost 151 tokens; all six in full, 1,258.
Does the right skill get picked? 14 test requests (12 need a skill, 2 need none):
| Descriptions | Picked right |
|---|---|
| Good: "Use when reviewing, checking, or optimizing a SQL query someone wrote…" | 14/14 |
| Vague: "Database stuff.", "Code helper." | 14/14, because names like sql-review gave it away |
Vague, with code names (atlas, falcon…) |
7/14: both SQL reviews went to the Python reviewer, the postmortems to release notes and API review |
A description has to say when to use the skill, in the words users actually use. (On gpt-5.4-mini, even good descriptions missed once: "Thoughts on this?" plus some code picked no skill, because the user never said "review".)
Does loading it change the answer? Each skill requires a format a script can check, like a Verdict: line or an ## Incident summary heading:
| Setup | Format followed | Tokens per request |
|---|---|---|
| No skill | 2/12: both customer emails, which start with "Subject:" anyway | ~39 |
| The right skill, loaded on demand | 12/12 | ~504 |
| All six, always loaded | 12/12 | ~1,371 (2.7×) |
With six small skills, loading them all is affordable. With fifty, it adds up on every request.
The checklist
- Clear old tool output first: it's most of the context and costs no model call.
- Summarize in named sections, with room to finish, and check every section is there.
- Have the agent write findings down as it goes, so they don't live only in tool output.
- Test every compaction with questions about the agent's own task, graded on exact details, and read the passes too.
- Give every skill a description that says when to use it, and measure the pick rate whenever a description changes.
Practice: compaction and skills
Clear old tool output without touching the agent's own messages, check a structured summary for missing sections (the real one is missing two), write a grader that rejects the false pass, build the skills listing, and measure pick rates. The tests use the real work log, the recorded structured summary and replies, and the lab's 14 requests.
Quiz
1.Sixteen messages in, what filled most of the agent's context?
2.How does the lab check whether a compaction kept what matters?
3.Dropping older messages scored 1 of 7. What was the one?
4.Clearing old tool output kept 5 of 7 with no extra model call. What did it lose?
5.Why did the structured summary lose the next step?
6.With vague descriptions, skill picks were still 14 of 14. Why did they drop to 7 of 14 with code names?
Check your understanding
Try each question before opening its answer.
What makes up most of a long agent session's context?
Tool output: here 96% (6,225 of 6,456 tokens), from a log dump, a source file, test results and metrics. Each piece mattered for one step and was then re-sent on every call.
How do you test a compaction?
Compact everything but the last few messages, then ask the agent questions about its own task (the rule, the root cause, the exact spot, the files changed, the failing tests, the next step, a key number) and count an answer only with the exact detail. Compare with no compaction.
Clearing tool output vs. a structured summary: which would you start with?
Clearing old tool output: it kept 5 of 7 with no extra model call, about the same as the structured summary (5 of 7, one call). It loses only what never left the tool output, so pair it with an agent that writes findings down.
Why did the structured summary lose the next step?
It spent its room copying raw log lines it was told to drop, hit its 400-token limit mid-list, and never wrote "Current state" or "Next step". Give the summary room, check its sections, and record findings as you go.
What decides whether a skill gets used?
Its name and description, because that's all the model sees when picking. Vague descriptions worked only while the names gave the answer away; with code names, picks fell to 7 of 14. Say when to use the skill, in the users' words.
Why load skills on demand rather than all at once?
Both followed the format 12 of 12 times, but loading all six cost about 2.7 times the tokens on every request (~1,371 vs. ~504). That gap grows with every skill you add.
Next up: Chapter 7 turns text into embeddings and tests when a search score points to the wrong help-center answer.