Larger context windows are useful. A million tokens means you can fill your AI prompt with an entire specification, a long document, or a complicated procedure without first chopping it into polite little pieces. I have used that capacity. I have also watched what happens when the window gets filled simply because it can be filled.
Irrelevant examples, accumulated tool results, details from other cases, contradictory instructions, and the inevitable “just in case” material all ride along. The model re-reads them on every subsequent turn. You pay for the tokens. You pay again in latency. You pay a third time in the errors that come from a context that knows too much about things that no longer matter.
What “Too Much” Looks Like in an AI Prompt
It usually starts reasonably. Someone has a task that works often enough to keep using. Then a correction gets appended. Then an example of what not to do. Then another example, this time from a completely different instance, because the last run did something unexpected. Positive and negative cases pile up. Concrete facts from products or documents or tickets that have nothing to do with the current one sit in the same window as the live evidence.
Procedure, teaching material, and intermediate results occupy the same growing conversation. Later steps spend most of their budget re-processing earlier steps. Early mistakes contaminate everything downstream because the model cannot cleanly separate “this was an example of the pattern” from “this is information about the thing I am supposed to be handling right now.”
The AI prompt still produces output. That is why it survives. Each run simply gets slower and more expensive the longer it lasts, and the patch for the latest failure is almost always another example. I distrust that pattern. I would rather withhold the extra example than hand the model details from other cases that can leak into the current one. Guardrails and constraints are cheaper than another round of contamination.
Give the Model Less, on Purpose
The alternative is to break the work into stages that cannot see one another’s full history. Each stage receives a short synthetic conversation containing only the facts, constraints, and examples required for its own instruction. Later stages see structured results, not the reasoning or the unrelated material that produced them.
Resolve the smallest decision that shrinks the remaining problem—classification, routing, scope—and then drop every option that no longer applies. If the acceptable values for a field are already known, put them in the schema and in the AI prompt. Prefer an explicit null or “unknown” over an invented answer. The model is not required to fill every blank; it is required not to lie about the ones it cannot fill.
Keep teaching material minimal and matched to the case already resolved. If voice, format, or policy examples are required, load only the ones that apply. Do not ship the entire style guide on every run. Move anything deterministic outside the model: database lookups, enum lists, validation that can be expressed as a schema, simple transformations. The model does not need to rediscover what a query already returned.
Make the synthetic prefix for each stage mostly static—system, user, and assistant turns that affirm the schema, the allowed values, and the output rules—so it can be cached. Append only the small dynamic payload for the current item. The model still does the judgment work. It simply does it inside a window that has not been poisoned by everything else you once tried.
Recovery Without Restart
When a downstream artifact depends on earlier structured choices, keep a path that accepts corrected structured input and regenerates only the final artifact. Do not force the entire pipeline to run again because one classification or one field was wrong. Prioritize those reprocessing requests so the human correction loop stays short. The reviewer fixes the structured fact; the model re-renders it. Most of the earlier work does not need to be repeated or re-contaminated.
The Cost of the Other Approach
Filling a large window with everything that might be useful feels safer. It removes the need to decide what the model should not see. It also removes your ability to predict or bound the token cost, the latency, and the contamination. The model becomes responsible for both the procedure and the filtering. Those are the runs that drift from seconds and cents into minutes and dollars, and that require ever more examples to patch the last failure.
I have replaced those prompts. They worked until they didn’t, and the fix was always more context. More context made the next failure slightly different and slightly more expensive. At some point the honest description is that the AI prompt knew too much, and most of what it knew was no longer helping.
The Incentive to Spend
It is in a provider’s interest for you to use more tokens. That is not a conspiracy; it is the business. When a CEO talks about “token budget” becoming a normal line item in every software project, the observation is directionally correct and also conveniently aligned with the product being sold. The same period has produced a steady stream of influencers, demos, and tutorials whose practical advice reduces to: put the model in the loop for as much as possible and iterate until it works.
I prefer the opposite default. AI is a new cost whose value is still uneven. It reliably buys time on some tasks. It does not reliably buy quality. The prompts that survive contact with real data are usually the ones that have already been narrowed, constrained, and stripped of everything that does not help the current step. Reaching that AI prompt is itself expensive. Most of the tokens are spent on the versions that were too broad, too contaminated, or too willing to invent. The final, efficient version is the cheap part. The path that discovered it is the part that shows up on the invoice.
That is a good arrangement for the seller of tokens. It is a worse arrangement for anyone who has to explain the bill or maintain the result. The practical response is not to avoid the tools. It is to treat every additional token of context or every extra model call as something that has to justify itself, the same way any other dependency does. Large windows and open-ended agent loops do not get a presumption of usefulness just because they are available.
A large context window is not a design. It is a capacity. Using it to hold only what the current step requires—and nothing else—is what keeps the output deterministic and the bill predictable. The model can still accelerate the work. It does not have to be handed the entire history of how the work used to be done, plus every counter-example, plus the live state of every prior attempt.
