Context Engineering: The Discipline Powering Modern LLM Systems

01 Beyond the Prompt

Prompt engineering began as a craft of clever wording: find the magic incantation, and a model behaves. It produced real gains, and a cottage industry of tricks, personas, and formatting rituals grew around it. But as production systems matured through 2025 and 2026, practitioners noticed that the phrasing of the user prompt was rarely what separated reliable systems from unreliable ones. What mattered was everything else the model was shown: the instructions above the prompt, the documents fed into it, the tools described to it, the history it inherited.

Context engineering is the name that has settled on this wider discipline. Where prompt engineering treats the input string as the unit of work, context engineering treats the entire information environment supplied to the model as an engineered artifact, assembled deliberately, tested, and versioned like any other component of a software system.

The emergence of a distinct label matters for more than taxonomy. Prompt engineering is the process of structuring natural language inputs to produce specified outputs from a generative AI model, while context engineering is the related area of software engineering that focuses on the management of non-prompt and prompt contexts supplied to the model, such as system instructions, metadata, API tools, and tokens. The shift in framing, from a prompt to a pipeline, reflects where the industry actually is: the hard problems are no longer linguistic but architectural.

02 The Anatomy of a Context Window

To a model, context is simply the sequence of tokens it conditions on. That sounds like a technicality, but it erases a distinction that practitioners find essential: everything the model sees is context, regardless of why it is there. The window that a production system assembles typically contains several distinct layers, each with a different owner and purpose, all flattened into one token stream before the model ever reads them.

The first layer is the system instructions: the standing definition of what the assistant is, how it should behave, what it must refuse, and how it should format responses. Next comes conversation history, the accumulated turns of the session, which grows with every exchange and must eventually be trimmed or summarized. Alongside these ride retrieved documents, chunks pulled from external knowledge bases and injected into the window to ground the model's answers. Finally there are tool definitions, machine-readable descriptions of APIs the model may call, which consume tokens to describe capabilities before the model can use them.

The practical insight is that these layers compete. A system prompt that grows by a few hundred tokens directly reduces the room available for retrieved evidence. A long, chatty history crowds out the documents the model actually needs to answer the current question. Designing a context window is therefore an allocation problem, and the diagram below shows how a typical production system might divide the territory.

03 Why Context Is a Budget

Context windows are finite, and everything inside them carries cost. Each token fed to a model is billed on the way in, processed on every step, and, for systems with attention over long sequences, increasingly expensive to reason about as the window fills. The consequence is that a context window behaves like a budget: every item placed in it purchases influence over the model's output, and the budget itself is capped.

The deeper problem is that irrelevant context does not merely cost money, it actively degrades accuracy. Research on long-context models has documented a phenomenon colloquially known as "lost in the middle": models retrieve information from the beginning and end of their windows more reliably than from the middle, so a document buried among distractors can be effectively invisible. A window stuffed with marginally relevant material is not neutral padding; it is noise that competes with the signal the model needs.

Good context engineering therefore reads like resource management in any other system. Relevant material earns its place by demonstrated value, and everything else is kept out, even though the window could physically hold more. The discipline runs against a natural instinct: because context is invisible and easy to add, systems tend to accumulate it, appending instructions, history, and retrieved text until accuracy quietly erodes. Treating the window as a scarce resource is the corrective.

04 The Toolbox

A set of techniques has become standard for managing what enters the window. Retrieval augmented generation, or RAG, is the oldest and most influential: instead of hoping the model memorized a fact during training, the system searches a knowledge base at query time and injects the relevant passages as context. RAG converts the model from a closed-book exam into an open-book one, trading a fixed training corpus for a live, versioned source of truth.

Around retrieval sits a family of supporting mechanisms. Memory systems persist facts across sessions, so a user's preferences and prior decisions can be recalled without replaying entire conversations. Tool and function definitions give the model the ability to act, calling out to APIs whose descriptions also occupy tokens. Context compression and summarization shrink long histories into dense digests that preserve the essentials while freeing budget. Structured outputs constrain the model to emit machine-parseable formats, making responses reliable inputs to downstream software. Caching, finally, avoids reprocessing unchanged context, which cuts both latency and cost when the same instructions and documents recur across requests.

These techniques are complementary rather than competing, and mature systems combine most of them. Retrieval supplies fresh evidence, memory supplies continuity, compression recovers budget, structured outputs protect the interfaces, and caching keeps the whole assembly economical. The chart below sketches how each technique is usually positioned by its primary effect on the system.

05 Agents Raise the Stakes

Agent systems, which loop a model through many steps to accomplish a goal, multiply every context problem by the length of the session. A single prompt-answer exchange is easy to budget; an agent that takes forty tool calls over an hour of work accumulates context continuously, and none of it can simply be discarded without losing the thread of the task. The window fills with tool results, intermediate reasoning, and partial outputs that were essential three steps ago and irrelevant now.

Practitioners describe the resulting degradation as context rot or drift: as the session lengthens, the earliest instructions and the most important constraints get pushed deeper into the window, where they exert less influence on the model's behavior. The agent starts to forget its standing rules, repeat work it has already done, or drift toward answers that fit recent context rather than the original goal. The failure mode is gradual, which makes it dangerous; nothing breaks loudly, but the quality of decisions quietly declines the longer the system runs.

The countermeasures are collectively known as compaction strategies. Instead of keeping the raw transcript, the system periodically summarizes completed phases, extracting durable facts and dropping verbatim noise. Sub-agent architectures isolate context: a fresh agent instance handles a self-contained subtask with a clean window, then returns only its conclusion to the parent. External scratchpads move state out of the window entirely, into files or databases the agent re-reads on demand. All of these techniques share one idea, that the window should hold what matters now, while durable knowledge lives somewhere the model can retrieve it rather than remember it.

06 The Practitioner's Discipline

What separates teams that manage this well is not secret knowledge but measurement. Context pipelines are engineered artifacts, so they need evaluations: systematic tests of how a system behaves when the window contains a given mix of instructions, evidence, and noise. Evals turn vague complaints about a model "getting worse in long sessions" into reproducible cases, where a specific history length or a specific distractor load causes a specific failure, and a change to the pipeline can be checked against it.

Monitoring matters just as much in production. The interesting question is not only what the model answered but what actually entered the window when it answered: how many tokens, from which sources, in what order, at what cost. Systems that log their own context composition can spot the pathologies early, a retrieval layer returning bloated chunks, a history that grows unbounded, a tool catalog that no longer earns its token cost, and fix them before users feel the effects.

The through-line of all of this is that context deserves the same engineering rigor as any other input to a production system. Prompt engineering asked a linguistic question, how should this request be worded. Context engineering asks a systems question, what information should the model see, from where, at what moment, and at what cost, and treats the answer as a pipeline to design, test, and operate. As LLM systems become more agentic and more embedded in real software, it is this second discipline, the less glamorous one, that determines whether they work.