If you’ve ever dropped a 200-page contract into a chatbot and asked “what does clause 14 actually commit us to?”, you’ve trusted the context window. Model launches make that easy. “1 million tokens” sounds like the model will read every word you give it.
It will accept every word. Whether it uses them well is a different question, and a surprising amount of research says the answer is often no. We read the main studies in the original, then ran a small test of our own on DeepSeek to see if the same thing shows up in a chatbot people use today.
A bigger desk, not a better reader
The context window is the maximum input a model can take in one go. Think of it as desk size. A huge desk doesn’t mean someone reads every page piled on it.
The paper that made this concrete is “Lost in the Middle” (Liu et al., TACL). The researchers gave models 20 or 30 documents, one of which held the answer, and moved that document around. Accuracy was highest when the answer sat at the very start or end and sagged in the middle. GPT-3.5-Turbo dropped by more than 20 points depending on position. In the worst case it did worse than when it got no documents at all (56.1%). Handing it the material actually hurt.
Two other findings from the same paper hold up well. Extended-context versions (GPT-3.5-Turbo 16K vs 4K, Claude-1.3 100K vs 8K) scored almost identically on inputs both could fit, so a longer window didn’t mean better reading. And going from 20 to 50 retrieved documents only bought GPT-3.5-Turbo about 1.5 points.
Why a perfect needle-in-a-haystack score isn’t enough
The long-context chart you see in launch posts is usually a needle-in-a-haystack test: hide one sentence in a long text and ask the model to find it. NVIDIA’s RULER benchmark made that test a bit harder. Of 17 models claiming 32K or more, nearly all aced the basic needle test, but only half held up at 32K once the tasks got more varied.
NoLiMa (ICML 2025) goes further. In a classic needle test, the question and the hidden sentence often share words, so the model can basically match strings. NoLiMa removes that shortcut. It hides “Yuki lives next to the Semper Opera House” and asks which character has been to Dresden. The model has to connect the two on meaning.
At 32K tokens, 11 of the 13 models tested (all claiming 128K or more) fell below half of their short-context score. GPT-4o went from 99.3% to 69.7%. Here’s the “effective length” for some models in the project’s results table, defined as the longest input where a model keeps 85% of its short-context score:
| Model | Advertised | NoLiMa effective length |
|---|---|---|
| GPT-4.1 | 1M | 16K |
| GPT-4o | 128K | 8K |
| Claude 3.5 Sonnet | 200K | 4K |
| Gemini 2.5 Flash (no thinking) | 1M | 2K |
| Llama 4 Scout | 10M | 1K |
Source: NoLiMa results table on GitHub. These are models tested in 2025, not newer releases.
One caveat matters a lot here. The same model can look great or terrible depending on the test. Gemini 1.5 Pro clears RULER’s bar past 128K, yet its NoLiMa effective length is 2K. There’s no single “usable length” for a model. It depends on what you’re asking it to do.
Finding the answer isn’t the same as using it
“Context Length Alone Hurts LLM Performance Despite Perfect Retrieval” (Du et al., October 2025) pushes the point further. Even when the model could retrieve everything it needed, scores fell by 13.9% to 85% as inputs got longer. The drop held when the filler was swapped for whitespace, and even when it was masked out entirely. Length itself was doing damage. The authors tested only five models, and closed models like GPT-4o held up much better than the open ones.
Their fix is refreshingly simple. Have the model quote the relevant evidence first, then answer from that short excerpt. For Mistral at 26K tokens, accuracy jumped from 35.5% to 66.7%.
Chroma’s “Context Rot” report (July 2025) points the same way across 18 models, including Claude Opus 4, o3 and Gemini 2.5 Pro. On a conversation-memory task, a focused prompt of about 300 tokens beat the full roughly 113K-token version by a wide margin. Even one plausible-but-wrong distractor sentence lowered accuracy. It’s a company report rather than a peer-reviewed paper, so weigh it accordingly.
We tried it on DeepSeek
Most of those models are from 2025 or earlier, so we wanted to see it in a chatbot people actually use. On November 5, the QuestBrief operator ran a small test in DeepSeek’s web chat with DeepThink and web search turned off.
The haystack was the full text of The Cloud Dream of the Nine (Guunmong), a 17th-century Korean novel, about 85,000 Korean characters including spaces. We planted three sentences that aren’t in the original, written in the novel’s classical style. At the 50% mark: a roe deer on Gwanaksan “leapt over a boulder.” At 25%, a distractor: a deer on Bukhansan “leapt over a stream.” At 75%, another distractor: a pheasant on Gwanaksan “flew up and settled on a pine branch.” The question, asked in Korean: “Which animal jumped on the mountain near Seoul National University?” Gwanaksan is the mountain SNU sits on, but the question never names it or the roe deer, so the model has to connect them NoLiMa-style.
Every run used a fresh chat, with the text and question in one message.

Chart: QuestBrief’s own test (DeepSeek web chat, DeepThink off, November 5, 2026)
With only the roe deer sentence planted, DeepSeek found it right away. Things wobbled once the two distractors went in. Two of three runs answered “pheasant.” In one of them, DeepSeek even noted at the end that the text also describes a roe deer leaping on Gwanaksan, and still picked the pheasant. It found the right sentence and then got the judgment wrong. The third run got the roe deer but missed the pheasant sentence entirely.
Was the question just ambiguous? We sent the same three sentences and the same question without the novel around them. Roe deer, three times out of three. The wording alone doesn’t explain the misses. The same content got harder to reason about once it was buried in a long text.
Then we added one line to the question: “First quote every sentence that could support the answer, word for word, then check each one against the question’s conditions before giving your final answer.” Three runs, three correct answers. In the first, DeepSeek quoted all three sentences and ruled them out one by one: Bukhansan isn’t near SNU, and the pheasant only flew up to a branch. Du et al.’s fix worked here too.
The limits are real. This is one model and three runs per condition, not a benchmark. DeepSeek likely knows the novel, so it may have spotted our lines as “not in the original.” In one run it literally called them sentences “inserted in the middle of the text.” An unfamiliar document could be harder. The question says “jumped” while the planted line says “leapt over,” but the short-prompt runs got it right every time, so that gap doesn’t explain the errors on its own. The web chat doesn’t show token counts, so we only have character counts.
When we asked DeepSeek about its own context length, it sometimes said 1M and sometimes said 128K. Models often don’t know their own specs, so check the docs instead. DeepSeek’s API docs list 1M for both current models but say nothing about the web chat’s input limit. Still, in one run DeepSeek quoted the sentences at the 25%, 50% and 75% marks, so the misses weren’t caused by the text getting cut off.
Change how you ask, not just what you paste
Pasting a long document is fine. Trusting the first answer is the risky part, especially when the document has look-alike content: two versions of a contract, a draft and a final, a stack of similar meeting notes.
The single most useful habit from all this is the line we used. Don’t ask for the answer directly. Ask the model to quote every sentence that could support the answer, word for word, and check each against your question before answering. Accuracy went up in our test. Even better, you get quotes you can check against the source in seconds, and if the quotes are off, you know not to trust the answer.