Menu

Blog Learn Videos About Start learning

Episode 05 of 25 · Context engineering

The prompt is a budget

Context is everything the model sees on a call. Where an assistant's tokens actually go (tool definitions: 71%, the question: 0.3%), why "keep the last N tokens" cuts the rules first, cutting by priority with a summary of what you drop, and why trimming a huge tool result doesn't fix counting.

Lab 07 · Context Engineering7 min read

Video transcript5:28· read along, or click a time to jump there

0:00Cold open

I asked our assistant how many payouts failed last night. Its tool pulled the whole night: four hundred rows.

It said ten. The right answer is twenty-one.

So I cut the tool output down to just the failed rows. More than sixty times smaller.

It said twenty-two.

The answer was in the prompt both times. So what should a prompt hold, and in what form?

0:26Intro

Hands-on AI engineering: context engineering, part one.

Today: where a prompt’s tokens go, how to cut a prompt to a budget, and what to do with huge tool output.

0:39Meet Lina

I’m Lina, an AI engineer at Ledgerly, a payments company. I build our internal engineering assistant.

One thing first. Ledgerly, its chats and its data are made up. The model’s answers are real: recorded runs of a small OpenAI model, gpt-5.4-nano.

Context is everything the model sees on a call. For our assistant, that’s six parts: its rules, tool definitions, which describe each action it can take, memory about the user, documents from search, the chat so far, and the question.

Context engineering is deciding what goes in, in what form, and what stays out.

1:2101 · Where the tokens go

Models read text in tokens, small pieces of words, and you pay for every one. Every message to our assistant sends about thirty-three hundred.

Which part do you think is the biggest?

The tool definitions. Seventy-one percent. Twelve tools, sent on every call, even when no tool is needed.

The question itself is zero point three percent.

You pay for every token on every call, and a longer prompt can slow the answer down. So measure first. The biggest part may not be the one you’d guess.

1:5802 · Keep the end

Now, say the prompt has to fit in six hundred tokens. Like packing a carry-on: something has to stay home.

To test any cut, I wrote five questions. Each one needs a different part of the prompt.

The simplest cut: keep the last six hundred tokens, and chop off the front. But the front is where the rules live. And the tools, memory, and the documents.

Ask about the price of the enterprise plan. The rule says: for pricing, only say, please contact sales. With the rules cut, it didn’t say that. It asked for the pricing page instead.

Two of five right. Only the chat history survived, and the on-call channel only just made it, in the first words.

2:4303 · Cut by priority

Better: choose what to drop. Always keep the rules and the question. Then fill the rest in order: memory, the top documents, the newest messages, then older ones if there’s room.

The tool definitions are dropped first. None of these questions needs a tool.

Four of five right. The miss: our on-call channel. It was mentioned thirty messages ago, so it was in the part that got dropped.

So before dropping old messages, summarize them in a few lines. That costs one extra model call.

Five of five, in six hundred tokens.

Shrink the budget to four hundred, though, and there’s no room even for the summary. Some budgets are just too small.

3:2704 · Huge tool output

Back to the payouts. I tried four ways to hand that table to the model, on three different nights.

Raw, all four hundred rows: thirty-eight thousand tokens, and wrong on all three nights.

Trimmed in code to only the failed rows: more than sixty times fewer tokens. Still wrong on all three nights: the count, the total, or both.

Less text doesn’t fix arithmetic. With exactly twenty-one rows in front of it, it counted twenty-two. This small model is unreliable at counting and adding up by reading.

Third: let code count and add up, and hand the model just two numbers. Ninety-one tokens, right on all three nights. But that only works if you knew the question in advance.

4:1505 · Offloading

The fourth way is offloading. The tool saves the full table to a file, outside the prompt.

The model gets back only a short summary: four hundred rows, these columns, and two example rows.

Then it calls a second tool: count the failed rows, and add up their amounts. Code does the math, and returns two numbers.

Right on all three nights, at about sixteen hundred tokens in total, over three small calls. And the model picked the filter and the math itself, so the tool didn’t have to be built for this one question.

4:5106 · What I do now

So now: I measure where the tokens go. I cut by priority, and summarize what I drop.

I never make a model count or add up what code can compute. And big tool results stay in a file, with a way to query them.

Next time: long agent sessions, where tool output piles up, and skills, which load know-how only when it’s needed.Lina: Next: long agent sessions, and skills. 🧠

5:15Summary

Measure it. Cut by priority. Summarize what you drop. Let code do the math.

Then the answer isn’t just in the prompt. The model can use it.Lina: Thanks! 😄

Key takeaways

✓Measure before you cut: one message to the assistant was 3,332 tokens, 71% of it tool definitions sent on every call, and 0.3% the question.
✓Keeping the last 600 tokens chopped off the rules, tools, memory and documents: 2 of 5 right, and the pricing rule was gone.
✓Cut by priority instead: always the rules and the question, then memory, top documents, the newest messages. Summarize what you drop: 5 of 5 at 600 tokens, for one extra call. At 400 even the summary didn't fit.
✓Less text doesn't fix arithmetic. 400 raw rows and the 21 failed rows alone were both wrong on all three nights; with exactly 21 rows in front of it, the model counted 22.
✓Let code compute (91 tokens, right every night), or offload big results to a file the model can query (about 1,600 tokens over 3 calls, right every night, and the model picks the filter and the math).

Every token in a prompt is paid for, and waited for, on every call. The question is never just "does it fit?" but "what's the smallest prompt that still has what this step needs, in a form the model can use?" This chapter measures where an assistant's tokens go, cuts a prompt to a budget three ways, and hands a huge tool result to the model four ways.

The sources are ai-eng-labs Lab 07 (07_context_engineering), scripts 2 (02_context_budget.py) and 4 (04_tool_results.py). Ledgerly, a payments company, its assistant, chats and payouts are made up; the answers are real recorded runs of gpt-5.4-nano, one run per test. Chapter 6 covers long agent sessions and skills.

What context is

Context is everything the model sees on a call. For Ledgerly's internal engineering assistant, six parts: its rules, tool definitions (one description per action it can take), memory about the user, documents from search, the chat so far, and the question. Context engineering is deciding what goes in, in what form, and what stays out. Chapters 3 and 4 decided which past information comes back; this chapter decides how everything fits.

Where the tokens go

Try it · Spend the prompt budgetReal recorded replies · gpt-5.4-nano · one run each

One message to the assistant: 3,332 tokens

tools 2,361 (70.9%)history 624 (18.7%)documents 224 (6.7%)rules 72 (2.2%)memory 41 (1.2%)question 10 (0.3%)

Budget

Cut

Keep the last N tokens and chop off the front: where the rules, tools, memory and documents live.

How much does the enterprise plan cost? needs: rules✕
“I don’t have any information in this chat about pricing or what the “enterprise plan” costs. If you share the pricing page or the relevant details, I can help you interpret it.”
Which cloud should my deploy script target? needs: memory✕
“I don’t have enough context from the conversation to know which cloud your deploy script should target.”
What's the right way to restart ledger-sync? needs: documents✕
“I don’t have enough context to know your exact “ledger-sync” service name/process or the correct operational procedure in your environment. In general, the safe “right way” is: 1. Identify the service (systemd/k8s/etc.). 2. Restart it gracefully (preferred) rather than killing the process. 3. Check logs/health after”
What's our on-call channel again? needs: old history✓
“Your on-call channel is #ledger-oncall-sev.”
Which bug am I working on right now? needs: recent history✓
“You’re working on the duplicate-webhook bug in the refunds flow.”

2 of 5 right · the on-call ✓ is luck: the tail starts with that message

Part	Tokens	Share
12 tool definitions	2,361	70.9%
Chat history (30 messages)	624	18.7%
Documents (runbooks)	224	6.7%
Rules	72	2.2%
Memory	41	1.2%
The question	10	0.3%
One message	3,332

Every token is paid for, and waited for, on every call, and the tools are sent even when no tool is needed. Measure first: the biggest part may not be the one you'd guess.

Cutting to a budget

Say the prompt must fit in 600 tokens. Five test questions each need a different part:

Needs	Question
Rules	How much does the enterprise plan cost? (The rule: reply exactly "Please contact sales.")
Memory	Which cloud should my deploy script target?
Documents	What's the right way to restart ledger-sync?
Old history	What's our on-call channel again? (said 30 messages ago)
Recent history	Which bug am I working on right now?

Cut	1,200 tokens	600	400
Keep the end (chop the front)	4/5	2/5	1/5
By priority	5/5	4/5	4/5
By priority + summary	5/5	5/5	4/5

Keep the end cuts the front, where the rules, tools, memory and documents live. Asked about pricing with the rules gone, it said it had no pricing information and asked for the pricing page instead of "Please contact sales." It never quoted a price, but it broke the rule. At 600, the on-call ✓ is luck: the kept tail starts with that message.
By priority always keeps the rules and the question, then fills the rest in order: memory, the top documents, the newest messages, older ones if there's room. The 12 tools are dropped first: none of these questions needs one. At 600 it lost the on-call channel, in the 18 oldest messages it dropped.
+ summary summarizes the dropped messages in a few lines first (one extra model call): 5 of 5 at 600. At 400 there was no room even for the summary. Some budgets are just too small.

Huge tool output

"How many payouts failed on the night of 2026-03-06, and what was their total amount in dollars?" The payouts tool returns 400 rows of JSON. Four ways to hand it to the model, on three nights:

Method	What the model gets	Prompt tokens	Nights right
Raw	All 400 rows	38,031	0/3
Trimmed in code	Only the failed rows	594	0/3
Computed in code	Two numbers: the count and the total	91	3/3
Offloaded	A summary of a saved file, and a query tool	about 1,600, over 3 calls	3/3

A night counts as right only with the right count and a total within $1. On night 1 the truth is 21 failed, $40,913.16. Raw said 10 ($26,980.62). Trimmed, with exactly 21 rows in front of it, said 22 ($44,624.52). On night 3 trimmed got the count right and the total wrong.

Less text doesn't fix arithmetic. This small model is unreliable at counting and adding up by reading. Computing in code is right every night, but only if you knew the question in advance.

Offloading keeps the full table outside the prompt:

get_payouts(date="2026-03-06") saves the rows to payouts_2026-03-06.json and returns only a summary: 400 rows, the 10 column names, and two example rows.
The model calls query_saved_file(file, where_field="status", where_value="failed", aggregate="count_and_sum_amount_usd"), and code returns {"count": 21, "sum_amount_usd": 40913.16}.
The model answers: 21 payouts, $40,913.16.

About 1,600 tokens in total over three small calls (308 + 601 + 678 on night 1), right every night. The model picked the filter and the math itself, so the tool didn't have to be built for this one question. Its limit: the query tool only does "equals" filters and the aggregates written in advance.

What else the lab measures

Script 1 of the same lab measures context rot: models getting worse as the prompt grows, even when the answer is in it. The classic needle test is easy because the hidden sentence shares the question's words; reworded facts, look-alike distractors and facts that must be combined break sooner. On gpt-5.4-nano, the harder versions failed even at 1K tokens of filler, yet passed with the filler removed; gpt-5.4-mini showed the classic curve. It isn't in the video, and it's one more reason to keep prompts short.

The checklist

Measure where the tokens go before you cut.
Never chop blindly: keep the rules and the question, then fill by priority, adding list items one at a time.
Summarize what you drop, and know the budget below which nothing works.
Send only the tools a request needs (Chapter 6 does this for instructions, as skills).
Never make a model count or add up what code can compute.
Keep big results in a file, with a summary and a way to query them.

Practice: cut a prompt, offload a result

Profile a prompt, implement "keep the end" and see it lose the rules, cut by priority (the tests reproduce the lab's 18 dropped messages at 600 tokens), then write the offload summary and a query tool. The tests use the lab's real part sizes and the real 400 payouts from 2026-03-06.

Try it · Practice in PythonRuns in your browser · ⌘/Ctrl + Enter to run

context_practice.py72 lines

"""
Practice: the prompt as a budget (Lab 07, 02_context_budget.py and 04_tool_results.py).

Fill in the TODOs, then run the tests:
    pytest test_context_practice.py        (locally)
or press "Run tests" on the chapter page.
"""

def tokens(text: str) -> int:
    """A stand-in for a real tokenizer (the lab uses tiktoken): one token per word."""
    return len(text.split())

def profile(parts: dict[str, str]) -> list[tuple[str, int, float]]:
    """Measure before you cut: (part, tokens, percent of the whole prompt) for each part,
    biggest first. Round the percent to 1 place.

TODO 1
    """
    return []

def keep_the_end(parts: dict[str, str], budget: int) -> list[str]:
    """The naive cut. Join the parts in order (dicts keep their order), keep only the LAST
    `budget` tokens (words), and return the names of the parts that survive, even partly,
    in prompt order.

TODO 2
    """
    return list(parts)

def fit_by_priority(rules: str, question: str, memory: str, documents: list[str], history: list[str],
                    tools: str, budget: int) -> dict:
    """Cut by priority instead.

Always keep the rules and the question. Then add, while each next item still fits in `budget`:
      1. the memory
      2. documents one at a time, best first (stop at the first that doesn't fit)
      3. messages one at a time, NEWEST first (stop at the first that doesn't fit)
      4. the tools, last
    Labels: "rules", "question", "memory", "doc 1".., "msg 1".. (msg N is the newest), "tools".

Return {"kept": [labels in the order added], "tokens": tokens used,
            "dropped": a list of what didn't make it, in this order and only if something was dropped:
                       "<n> oldest messages", "<n> documents", "tools"}

TODO 3
    """
    return {"kept": ["rules", "question"], "dropped": [], "tokens": tokens(rules) + tokens(question)}

def offload(rows: list[dict], path: str) -> dict:
    """What the tool returns instead of the rows: a short summary of the file it saved.

{"saved_to": path, "rows": number of rows, "columns": the first row's keys, "preview": the first 2 rows}

TODO 4
    """
    return {"saved_to": path, "rows": rows}

def query(rows: list[dict], where_field: str, where_value, field: str) -> dict:
    """The query tool: count the rows where row[where_field] == where_value, and add up `field`.

Return {"count": n, "sum": total rounded to 2 places}. Code does the arithmetic, not the model.

TODO 5
    """
    return {"count": 0, "sum": 0.0}

↓ context_practice.py ↓ test_context_practice.py ↓ context_solution.py

Quiz

Try it · Check your understanding0 of 6 answered

1.In one 3,332-token message to the assistant, which part was the biggest?
2.You keep only the last 600 tokens of the prompt. What gets cut?
3.Cutting by priority at 600 tokens got 4 of 5. Which one was lost, and what fixed it?
4.The tool output was trimmed in code to only the 21 failed payouts, 64 times fewer tokens. Did the model get the count right?
5.What does offloading return to the model instead of the 400 rows?
6.Computing the answer in code used 91 tokens and was right every night. Why still offload?

Check your understanding

Try each question before opening its answer.

What is context engineering?

Deciding what goes into the model's context on each call (rules, tool definitions, memory, documents, history, tool output, the question), in what form, and what stays out, within a token budget.

Why profile the prompt first?

The biggest part is often not what you'd guess: here the 12 tool definitions were 71% of every message, sent even when no tool was needed, while the question was 0.3%.

What goes wrong with "keep the last N tokens"?

It cuts the front of the prompt, where the rules, tools, memory and documents live. At 600 tokens only the history and question survived: 2 of 5 right, and the pricing rule was broken.

How do you cut by priority, and what does the summary add?

Always keep the rules and the question, then memory, top documents, newest messages, older messages; drop unneeded tools first. A few-line summary of the dropped messages saved the on-call channel: 5 of 5 at 600, for one extra call. At 400 even the summary didn't fit.

Why didn't trimming the tool output fix the answer?

The problem was arithmetic, not length. With exactly the 21 failed rows in front of it, the model counted 22, and it was wrong on all three nights. Code should count and add up.

When would you offload instead of computing in code?

When you don't know the question in advance. The model gets a short summary of the saved file and a query tool, chooses the filter and the aggregate, and code does the math: right every night at about 1,600 tokens over 3 calls.

Next up: Chapter 6 follows an agent that keeps adding to its own prompt: compaction, what it forgets, and skills that load only when needed.

Learning something useful?

New labs, articles, and videos on evaluating and testing AI systems come out regularly. Follow along so you don't miss the next one.

Subscribe on YouTube