The window is a budget, not a bucket.
You do not fit everything into the context. You spend a fixed budget on competing things, and how you spend it decides whether the model is accurate, safe, and affordable. Three tiers: what gets evicted when you overflow, why more context can make the answer worse, and why you keep re-paying for a prefix that never changed.
Tier 1
Basics
What gets evicted when you overflow
Tier 2
Intermediate
More context is not better context
Tier 3
Advanced
You re-pay for the same context
?New to this? 30-second primer▾
- Context window
- The model’s short-term memory for one request, measured in tokens. Everything it needs must fit inside it.
- Token
- A chunk of text, roughly three quarters of a word. You pay per token, in and out.
- RAG
- Fetching relevant documents and putting them in the context so the model can answer about your data.
Want all ten words first? Read the beginner primer →
Overflow it, and something gets evicted
Spend the window on five competing things, overflow it, then pick the eviction strategy. Naive "keep the newest tokens" quietly drops the system prompt off the top, which is the real bug behind "the model stopped following my rules". Pin the load-bearing parts and trim the rest.
Context window
7,170 / 8,000 tok
Everything fits. The model sees the whole picture. This is the easy case, and the one your demo runs in.
That is the budget. But what you put in it matters as much as how much fits, and that is where most people go wrong: they assume more context is better.
More context can make the answer worse
The reflex when an answer is wrong is to retrieve more documents. Set top-k and a relevance threshold below and watch that reflex backfire: a stale 2019 policy scores "relevant" enough to make the cut, and it poisons a confident, wrong answer. Volume admits noise. Relevance is what grounds the model.
Grounded. Only genuinely relevant docs entered the window, and the stale one was left out. Relevance beat volume. This is what a threshold buys you.
The judgment: rank and threshold, not "retrieve more". A tight, relevant window beats a full one every time, and it is cheaper too, which is tier three.
You re-pay for the same prefix every turn
A RAG chat sends a large, unchanging prefix, your system prompt and retrieved docs, on every single turn, and naive billing charges full price for it each time. Slide the prefix size and the turn count and watch the red bar sit permanently above the green: that gap is tokens the model already had. One caching flag removes it.
Re-send
$0.525
Cached
$0.093
Wasted
$0.432
Every turn re-sends the whole 12,000-token prefix, your system prompt and retrieved docs, even though none of it changed. Over 8 turns that is $0.432 spent on tokens the model already had. The bigger the retrieved context, the more of your bill is this pure repetition, and one caching flag removes it.
One bad doc taints everything
A single irrelevant or malicious passage in the window drags every answer toward it. This is why relevance thresholds matter more than k: the fix is to keep noise out, not to add more signal on top of it.
In a bank → A stale 2019 policy that still scores "relevant" will confidently give a customer the wrong loan terms, and nothing will error.
Do not spend the window at all
Before assembling context, check whether a semantically similar request already has a good answer. A cache hit costs almost nothing and skips the whole budget question.
In a bank → Ten thousand customers ask "what are your refund policies" a hundred ways. Answer it once, serve it from cache, spend zero tokens.
Summarise, do not truncate
When history must shrink, a running summary keeps the meaning that a hard truncation throws away. The window holds the gist of turn 1, not a cliff where it vanished.
In a bank → A 40-message support thread compressed to its decisions is worth more than its last 3 verbatim messages.
Know what is in the window
Log which system prompt, which docs at which versions, and how much history actually reached the model on each call. When an answer is wrong, the window contents are the first thing you need to see.
In a bank → This is where context meets observability: "what did the model actually see" is an audit question, not a debugging luxury.
Managing the window, at three altitudes
Ranking, capping, pinning, and caching are the concept. LangChain retrievers, memory classes, and prompt caching are where you apply them.
The concept
Assemble the context in priority order, measure it, and if it overflows drop from the least important end while pinning the system prompt and the question. Rank what you retrieve; do not just retrieve more.
budget = 8000
parts = [system(pin=True), tools, rank(docs), history, question(pin=True)]
used = sum(len(p) for p in parts)
while used > budget:
drop_lowest_priority(parts) # never a pinned part
used = sum(len(p) for p in parts)The point: ranking, capping, pinning, and prefix caching are the concept. A LangChain retriever, a memory class, or a prompt-cache flag are just three places you apply them.
The same question, three depths of answer
"Your agent ignores its instructions deep in a long conversation, and the chat is expensive. What is going on?" The depth of your answer is the level they hire you at.
The conversation got too long, so I would summarise the history to make it fit.
Two separate problems. The ignored instructions are context eviction: naive history trimming pushed the system prompt out of the window, so pin it. The cost is re-sending the whole prefix every turn at full price, so cache it. Neither is a prompt problem.
Pin the system prompt and question outside the trimmable region, rank and threshold retrieved docs so more context cannot poison the answer, cache the stable prefix so you stop re-paying for it, compress rather than truncate old history so growth stays affordable, and log what actually reached the model so a wrong answer is auditable. Context is a budget you allocate deliberately, not a bucket you fill.
On your real stack this week
Add a relevance threshold and a prefix cache to one chain.
Find a RAG chain that retrieves a fixed k with no threshold, add a score threshold so noise cannot enter the window, and mark the system-and-docs prefix as cacheable. Then measure the cost of a 10-turn conversation before and after. You will fix accuracy and cost in one change.