Six ways to give an assistant memory

In our test, a user told our assistant: please forget my phone number.

A week later, we asked it: what’s my phone number? It said: I don’t know your phone number.

The number was right there, in its prompt.

Not saying something isn’t forgetting it. So how does an AI assistant remember, and how do you test it?

Hands-on AI engineering: memory systems.

Today: how memory works, six ways to build it, a harder test that separates them, and what it takes to really forget.

I’m Elena, an AI engineer. I build an assistant that should remember its users.

First, the surprising part: the model itself remembers nothing. Every call starts from zero. It only sees what’s in that call’s prompt.

So memory lives in the app. Think of it as a notebook the app keeps next to the model.

Two moments matter. When a chat ends, the app decides what to write in the notebook. That’s save time.

When a new question comes in, the app decides which notes to copy into the prompt. That’s ask time. Every memory design is just a different answer to those two questions.

To test this, a made-up user, Alex, has five chats from January to March. Some facts change: a new job, a new city for a trip, a new diet. And in the last chat, Alex asks us to forget a phone number.

Every memory design gets the same twelve questions, scored by code. The answers you’ll see are real: recorded runs of a small OpenAI model.

The simplest design: full history. At save time, keep every message. At ask time, send all of them.

It answers all twelve questions. Nothing is ever lost.

But look at the size. After five short chats, it’s thirteen hundred and sixty tokens, pieces of words, paid for on every question. And it never stops growing.

The quick fix: keep the same notebook, but at ask time, send only the ten newest messages.

Now it’s small. But it gets six of twelve. The dog’s name, the new job, the deadline: all said too long ago to make the cut.

Next: a running summary. Now save time does real work. After each chat, a model rewrites one short summary of everything so far. That’s an extra model call per chat. At ask time, send the summary and the last few messages.

Nine of twelve. Ask about the trip, and it says Lisbon. Ask about the diet: vegetarian. Both are old answers.

Look at how the summary ends. It wrote a log, chat by chat, hit the cap on how much it may write, and never got to chat five, where those facts changed.

Next: retrieval, which is search. At save time, an embedding model turns each of Alex’s messages into a list of numbers that captures its meaning. That’s a cheap call, not a chat model.

At ask time, the question is turned into numbers too, and only the four messages closest in meaning go into the prompt.

Ask which city Alex is visiting in May. Search finds the Lisbon message, and the Porto message. It finds what’s similar, not what’s current.

So which one is new? Without dates, the model can’t tell. Even with the newest message on top, it picked the old budget: fifteen hundred euros. It did the same for the cloud provider.

Put the chat and its date on every memory, in time order, and all five changed facts come out right.

Then, fact memory. At save time, a model reads the chat and pulls out facts, each as a key and a value: employer, trip destination, diet. At ask time, send the current facts.

A new value replaces the old one. Employer: Ledgerly, not Freightly. Trip: Porto, not Lisbon.

In this run, its notes also kept lines like “changed from Lisbon”. That’s how it could answer what something was before. Luck of what it wrote down, not design.

And the old values aren’t thrown away. They move to a history list, outside the prompt. Remember that list.

Last, a hybrid, which combines them: facts for what’s true now, search for details, and the last few messages word for word.

So which design wins? Pause and guess.

Four of them score twelve of twelve: full history, retrieval, fact memory, and the hybrid. Twelve easy questions can’t separate them.

So I made the test harder: twenty everyday chats mixed in, a brother who now works at Alex’s old company, a deadline that moves in two steps, a second dog, and a name said once and never again.

Now they separate. The last ten messages and the summary collapse: the newest messages are mostly everyday chat, and the summary drops details.

Full history gets all sixteen. But it sends over three thousand tokens with every question, and that keeps growing.

Retrieval and the hybrid each miss one. Retrieval does it with about a hundred and thirty tokens. The hybrid sends about a thousand, and pays a model call after every chat.

Fact memory missed the beagle: one value per key. When Alex adopted Comet, Comet replaced Biscuit.

Each design ran once, and hedged answers that still name the fact count as right. So a gap of one question could flip on a rerun. The big gaps are real.

So which one should you use? There’s no single best memory. Each design fits a job.

The last few messages are enough inside one conversation. A summary gives a rough recap, but drops details.

Full history is the most accurate, while histories are short. You pay for that on every question, and the bill keeps growing.

As histories grow, retrieval with a date on every memory was the best value here: within one question of full history, at about a twenty-seventh of the tokens.

Add fact memory, the hybrid, when you need clean answers about what’s true now. Here it caught the diet that retrieval missed. Just let facts hold lists, like dogs.

Now, the phone number. Every design on the scoreboard passed that question. Now look at where the number actually is.

Four designs refused to say it: full history, the summary, retrieval and the hybrid. But the number was right there in their prompt. Full history and the summary send it every time.

A variant of fact memory that only ever adds new facts has no way to delete one. The forget request had nowhere to go, so it just said the number.

The fact memory that replaces values is the only one that took it out of the prompt because the user asked. But it didn’t delete it. It moved it into that history list.

Step back. Where does all this memory actually live? Not in the model. In this demo, in files. In a real app: a messages table, a vector database for search, a summary field, a facts table, a history table, and a cache of model calls.

Now light up where the phone number still is. Five of the six places. A refusal isn’t a deletion. Deleting means finding every copy.

So now, for every memory design, I decide both moments: what gets saved, and what gets brought back. And every memory gets a date.

I test with every kind of question: old facts, changed facts, buried details, what something was before, things never said, and things to forget.

And when a user says forget, I search every store for the value itself, not just the prompt. A refusal isn’t a deletion.Elena: Refusing isn’t forgetting. 🗑️

Save time, ask time. Pick memory by the job. Date every memory. Delete every copy.

Then your assistant remembers what’s true, and forgets what it should.Elena: Thanks! 😄