A customer told our support chatbot their order number in the second message of the conversation. By message thirty-one, after a long back-and-forth about a shipping exception, the bot asked for the order number again — it had been true and useful in turn two, and it was simply gone by turn thirty-one, trimmed out along with everything else from early in the conversation to keep the request from growing without bound. The customer, reasonably, assumed the bot hadn’t been listening at all.
It had been listening. It just couldn’t remember, because “remember everything” and “keep the request a manageable size” are directly opposed, and we’d solved that tension with a blunt instrument that had no way to tell a fact that mattered from one that didn’t. This is what we changed about how a long-running conversation keeps its own memory.
Why you can’t just send the whole conversation forever
The obvious approach — include every prior message in every request — breaks down for two separate reasons that get conflated but are actually distinct problems. The first is cost and latency: a conversation’s token count grows every turn, so a support session that runs to fifty exchanges is paying for, and waiting on, fifty exchanges’ worth of context on every single new message, most of which is irrelevant to what’s being asked right now. The second, less talked about, is a real degradation in the model’s ability to reliably attend to information buried in the middle of a very long context — a fact stated early can get less weight than it deserves purely by virtue of being surrounded by everything that came after it, independent of any token limit being hit at all.
The naive fix, and why it drops the wrong things
A sliding window — keep only the last N messages — is the first thing everyone reaches for, and it fixes the cost problem cleanly while making the memory problem worse in a specific, predictable way: it drops old information indiscriminately, with no concept of which old fact is still load-bearing for the rest of the conversation.
// The naive version — simple, and exactly how we lost the order number
function buildContext(messages, maxMessages = 20) {
return messages.slice(-maxMessages);
// Message 2 (the order number) is gone by message 22, regardless of
// whether anything since has made it irrelevant
}
Summarizing the trimmed-off portion instead of just discarding it is the standard next step, and it helps, but it introduces its own failure mode: a summary is a compression, and compression loses specifics by design. A summary that says “customer described a shipping issue” has thrown away the exact order number, the exact date, the exact phrase the customer used to describe the damage — precisely the details that matter most when a human or the model needs to act on them later, as opposed to just recalling that a conversation happened.
Compaction for the narrative, extraction for the facts
The fix that actually worked was splitting “what happened in this conversation” from “what do we now know,” and treating them differently. Rolling summarization handles the narrative — good enough for the model to understand the shape and tone of what’s already been discussed. A small, structured, explicitly-extracted set of facts handles anything that needs to survive verbatim, independent of how much of the raw conversation gets trimmed.
// Extracted alongside the ordinary conversation flow, not instead of it
type KnownFacts = {
orderNumber?: string;
statedIssue?: string;
sentiment?: 'neutral' | 'frustrated' | 'urgent';
promisedActions: string[]; // "agreed to refund shipping cost", etc.
};
// Re-extracted (cheaply, on a small model) after every few turns,
// merged rather than replaced so a fact once confirmed doesn't silently disappear
async function updateKnownFacts(existing: KnownFacts, newMessages: Message[]) {
const extracted = await extractFacts(newMessages);
return mergeFacts(existing, extracted); // new facts win on conflict, nothing is dropped by omission
}
Every request now sends: the known-facts object (small, cheap, always included in full), a rolling summary of everything older than the window, and the last N raw messages verbatim for immediate conversational context. The order number from message two is now in the known-facts object from the moment it’s stated, and stays there — verbatim, not summarized — for the rest of the conversation regardless of how long it runs.
A summary is memory of the conversation. A known-facts object is memory of the truth the conversation established. Losing the first feels like forgetting a story. Losing the second feels, to the person on the other end, like nobody was listening.
Extraction has its own failure mode, and it’s worse in one specific way
Structured extraction can also go wrong — it can miss a fact that was stated, or worse, it can synthesize something that sounds plausible but was never actually confirmed, which is a more dangerous failure than simply forgetting, because it’s confidently wrong rather than visibly incomplete. We treat the known-facts object as provisional, not authoritative: for anything that gates a real action (issuing a refund, confirming an address change), the flow re-confirms the specific fact with the customer directly rather than trusting the extracted memory silently, the same caution we’d apply to any value pulled from an upstream system we don’t fully control.
What we measure: memory misses, not just token counts
The metric that actually tells you whether this is working isn’t context size, it’s how often a fact the customer already gave gets asked for again. We added this as a specific eval case type — a synthetic long conversation where a fact is stated early and a later turn requires it — run the same way as our other regression evals, because “did the bot ask for something it was already told” is exactly the kind of thing that degrades silently until a real customer notices and gets annoyed enough to say so.
This only matters past a certain conversation length, which is easy to miss in testing
Part of why this shipped unnoticed in the first place is that it’s invisible in the exact conditions most people test under. A quick manual test of a new chat feature is almost never thirty-one messages long — it’s five or six exchanges to confirm the happy path works, which sits comfortably inside any reasonable context window and never triggers a single trim or summarization event. The failure only shows up once a conversation runs long enough to actually hit the mechanism meant to keep it manageable, which in practice means it surfaces first in production, on the customers with the most drawn-out, most frustrating issues — exactly the conversations where forgetting something already stated does the most damage to how the interaction feels.
Key takeaways
- Sending a full conversation history forever has two distinct costs: growing token spend and latency, and a real degradation in how reliably a model attends to information buried early in a long context.
- A naive sliding window fixes the cost problem but drops old facts indiscriminately, with no sense of which ones are still load-bearing for the rest of the conversation.
- Rolling summarization helps but compresses away specifics by design — exact order numbers, dates, and precise phrasing are exactly what a summary is built to discard.
- Split narrative memory (a rolling summary) from fact memory (a small structured object of durable, verbatim facts) — send both on every request rather than relying on one mechanism to do both jobs.
- Treat extracted facts as provisional, not authoritative — re-confirm anything that gates a real action directly with the user rather than trusting silently synthesized memory.
- Measure memory misses directly with synthetic long-conversation eval cases (fact stated early, needed later) rather than inferring health from context size or token spend alone.
Frequently asked questions
How often should the rolling summary and known-facts object be updated?
We re-extract and re-summarize every few turns rather than every single message, since it’s a real cost and a summary of one new message rarely changes much. Trigger it on a turn-count threshold or a token-count threshold on the raw window, whichever your conversations tend to hit first.
What model should do the fact extraction and summarization — the same one running the conversation?
We use a smaller, cheaper model for both, since neither task needs the full capability of the model actually talking to the customer, and running them on a separate call keeps the main conversational turn’s latency from growing with the summarization work.
Could you just increase the context window instead of building all this?
A larger window raises the ceiling but doesn’t fix the underlying problem — cost still scales with what you send, and the reliability concern about information buried in a long context doesn’t disappear just because the window is technically large enough to hold it. It buys you room, not a solution.
How do you handle a fact that changes mid-conversation — the customer gives a different order number later?
The merge step treats a new extraction as an update, not an addition, when it conflicts with an existing fact of the same type — the newer stated value wins. We log conflicts like this rather than silently overwriting, since a genuine correction and a confused customer typing the wrong number both look identical at extraction time.
Is this worth building for a chatbot that mostly handles short, single-question conversations?
Probably not on its own — the failure mode this solves is specific to conversations long enough for early context to actually get trimmed. For short-lived sessions, sending the full history is simpler and the added complexity of extraction and summarization isn’t buying you much yet.
