Place each fact in the right type of context

The problem

A team builds a support agent that requires three pieces of context: the corporate refund policy, the conversation history of the active ticket, and the latest order status. Someone decides to simplify: they paste the entire policy handbook, the full ticket history, and the complete order event log into every single API call, "to make sure the model has everything it needs." Context size balloons, cost per call surges, and accuracy drops — because the single relevant policy clause is buried beneath ten pages of irrelevant boilerplate.

The root mistake was not merely including too much information. It was treating context placement as a monolithic decision — dumping everything into the same place — when in reality there are at least four distinct locations for a fact to live, each suited for a specific type of data.

Four destinations, three questions

For any fact an application requires, three core architectural dimensions dictate where it belongs:

• Volatility: Does this fact change on every call, or does it remain stable for days or weeks?
• Volume: Can the entire dataset fit cleanly and predictably into context, or is it so large that only a fraction is relevant at any given time?
• Scope: Does this fact apply only to this single turn, to this active session, or must it persist across distinct user sessions over time?

These three questions point directly to four clear storage destinations:

1. Current call message/turn — For high-volatility facts with strictly local scope: the specific ticket currently being answered, or the prompt the user just typed.
2. Stable context block (system prompt or project files) — For low-volatility facts required on every call: a concise summary of governing rules, not the exhaustive operations manual. If content mutates continuously, it does not belong here.
3. Retrieval-Augmented Generation (RAG) — For high-volume knowledge where only a small subset is relevant per query: the comprehensive knowledge base or historical ticket archives. The system queries the index and injects only the matched chunk.
4. Persistent cross-session memory — For facts that must survive across separate user interactions: customer language preferences or historical defect notes.

The error in the opening scenario was treating a high-volume, low-volatility asset (the policy manual) as local context — dumping the raw document into the request rather than indexing it and retrieving only the relevant clause.

Lost-in-the-middle is not solved by expanding context

Large language models inherently exhibit a positional bias, attending more effectively to information placed near the beginning and end of a long context window. A critical constraint buried in the middle of a massive context block is far more likely to be missed — even though it is technically present in the token stream. This failure is frequently misdiagnosed as "insufficient model intelligence" when it is actually an issue of volume and positioning.

The correct mitigation is not expanding context windows or redundantly repeating facts throughout the prompt. The fix is aggressive context pruning — supplying only what is strictly required for that specific turn — and placing critical directives near boundaries: at the very start of the system prompt or immediately adjacent to the final user instruction. Retrieving a single targeted paragraph and positioning it near the query yields far higher fidelity than dumping a 40-page document into the middle.

Prompt caching demands byte-for-byte prefix stability

Caching the static prefix of a prompt — the system prompt, static policy guidelines, immutable few-shot examples — slashes both cost and latency across repetitive calls sharing that prefix. However, prompt cache hits require the cached content to remain strictly byte-for-byte identical between calls.

A widespread and silent failure mode occurs when developers inject dynamic timestamps, randomly generated session IDs, or request-specific UUIDs inside the cacheable prefix block. The cache invalidates silently on every invocation without throwing an error — the application functions normally, but expected cost savings never materialize. The solution is straightforward: isolate static content in the cacheable prefix block, and pass dynamic per-turn variables in the uncached message payload.

Where intuition fails

Three architectural assumptions that seem intuitive but violate real-world constraints:

• "Putting everything in the prompt guarantees the model won't fail from lack of information." Ignores latency, cost, and the lost-in-the-middle effect — more tokens do not automatically yield higher accuracy.
• "RAG solves every large-scale knowledge problem." RAG introduces its own retrieval failure modes: if retrieval fetches irrelevant or outdated chunks, the model receives plausible-looking but useless context, with no way to know retrieval failed.
• "Caching is merely a cost optimization that doesn't impact correctness." Incorrect caching boundaries can cause an application to serve stale knowledge silently because the cache hit succeeded technically — but returned obsolete data.

Put it into practice

In this lesson's lab, you will implement a context routing engine: given a list of facts with volatility, volume, and scope metadata, it routes each fact to its optimal architectural destination among the four described here, backed by test assertions that validate compliance with the decision framework.