What's an AI's "context window", and why does it sometimes seem to forget what you just told it?

Last autumn a real estate client asked us for something that sounded simple: an assistant that could answer questions about their earnest-money contracts, just by uploading the PDF straight into the chat. We tried it first on a short contract, eight pages, and it worked great — questions about deadlines, the penalty clause, who keeps the deposit if the deal falls through. All fine.

Then we tried one of their genuinely long contracts, the kind dragging technical annexes and a whole history of amendments: forty-odd pages. The model started answering with total confidence about clauses that didn’t exist. Not slightly wrong — it was inventing entire article numbers.

That’s where the context window comes in, a concept almost no client has heard of until it blows up in their face. Every AI model has a limit on how much text it can “hold in mind” at once, and that limit is measured in tokens, not pages or words (a token is roughly three or four characters). A model with a 128,000-token window can handle, roughly, some 300 pages of normal text. Sounds generous. But a scanned PDF, with badly converted tables or embedded images, can end up costing two or three times the tokens it looks like it should, and that’s exactly where the margin quietly runs out.

And here’s the part that actually matters: when you go over the window, the model doesn’t throw a clean “this doesn’t fit” error. Depending on the tool, it truncates the document wherever it can, or silently summarizes chunks without telling you, or just fills the gaps with whatever seems statistically likely given the pattern of the text. That’s what happened to our client: the model never had the end of the contract in its active window, and instead of saying “I don’t know,” it improvised.

The fix wasn’t to grab a model with a bigger window (they exist, some with a million tokens, but they cost more and run slower). We went with RAG instead: rather than dumping the whole contract in one go, the system first pulls the fragments relevant to the actual question, and only those go into the context window. Less elegant than “upload the PDF and ask anything,” but a lot more reliable, and cheaper too, since you only pay for the tokens each query actually uses.

If someone tries to sell you a “no limits” AI assistant for long documents, ask exactly what happens when a document doesn’t fit whole. If the answer is vague, they probably haven’t tested it against a real forty-page case yet.