Agent I: How Can a Prompt Elicit a Model's Hidden Capability?

“You are a rigorous peer reviewer.”

Adding this single sentence before a question can make a model stop agreeing reflexively and start asking about experimental design, sources of evidence, and the limits of a conclusion. It did not download a review handbook or retrain itself in those few seconds. Its parameters are exactly what they were a moment ago, yet it behaves as if it has entered a different working mode.

This is the most fascinating part of prompt engineering, and the part most easily dismissed as magic: how can one sentence change a model with billions of parameters?

One common explanation says that we are “teaching the model what to do.” Yet if a sentence could teach a complex capability, training would seem strangely unnecessary. Another explanation treats certain phrasings as incantations that manipulate a model. But the same “you are an expert” prompt often stops helping as soon as the model, task, or surrounding context changes.

The order of explanation should be reversed. We should first ask what the model experienced during training, then ask what the prompt is calling upon.

This is the first essay in a series about agents. I will leave large-scale context organization, tools, memory, and the runtime loop of a harness for later. Here I want to resolve the earliest and most basic layer: a prompt does not create a capability. It provides an index into the model’s many learned behavioral possibilities and guides the current generation into one of their trajectories.

Capability Exists Before the Prompt

The original task of a large language model sounds almost absurdly simple: read some text and predict the next token.

The training text, however, is not simple. A paper poses a question and presents evidence. A tutorial decomposes an operation into steps. A fictional character maintains an identity and voice. Questions, programs, debates, review reports, and mathematical derivations each have relatively stable ways of continuing. To keep predicting such text, a model must encode more than word frequencies. It must compress patterns of what tends to follow under which conditions into its parameters.

Pretraining therefore does not produce one unified personality that merely answers questions. It produces a large space of conditional behavioral possibilities: explaining, translating, imitating a style, completing code, extending an argument, and following an input-output relation implied by the text. These behaviors are not stored like cleanly separated plugins, nor are they necessarily reliable. They are distributed generation tendencies inside one set of weights.

The GPT-3 paper demonstrated an early large-scale version of this phenomenon. Without updating the parameters, a model could perform translation, question answering, word transformations, and other tasks when given a description or a few examples in its input. Each task used forward passes, not task-specific fine-tuning.

This is the more precise meaning of a “hidden capability.” Hidden does not mean a complete little program waits inside the model for a magic phrase. It means that one set of parameters permits many input-output relations, while default generation reveals only some of them. Change the condition, and another behavior can become more probable.

Pretraining alone, however, does not teach a model that every sentence from a user should be treated as an instruction. A webpage contains commands, quotations, counterexamples, advertisements, and lines spoken by fictional characters. Understanding language and reliably obeying a task described in language are not the same ability.

Post-Training Turns Language into a Control Interface

We now take it for granted that “summarize this passage” should produce a summary because modern models later see large amounts of instruction-response data.

The FLAN instruction-tuning study reformulated many tasks as natural-language instructions and trained on them together, improving not only trained tasks but zero-shot performance on held-out task types. InstructGPT continued training with human demonstrations and preferences: first teaching a model to imitate desired answers, then making preferred outputs more likely.

Both kinds of training changed something fundamental. Natural language became not only content to continue, but also a control signal that could regulate behavior.

It is useful to think of training as shaping two layers:

1. Pretraining expands what the model might be able to do. It acquires knowledge, styles, task structures, and traces of reasoning from a vast corpus.
2. Post-training improves how a user can call those behaviors. It makes instructions, examples, refusals, tool calls, and deliberate reasoning more reliably correspond to particular outputs.

This also explains why no set of prompt tricks works forever across every model. A prompt is an interface; training data determines what lies behind the interface. If the model never formed a capability, a clever prompt can only expose the gap more fluently. If post-training has already made a behavior easy to enter, yesterday’s fifty-line incantation may collapse into one switch.

The ceiling of prompt engineering is set by training first.

A Prompt Changes This Computation, Not the Parameters

At inference time, a model can be approximated as a conditional distribution:

P(next token | current context)

When the context changes from “explain this conclusion” to “review this conclusion like a peer reviewer,” the condition changes. Intermediate representations, attention patterns, and candidate-token probabilities throughout the model change with it. The parameters remain fixed, but the computation in which those parameters participate is different.

This is more accurate than saying that a word lights up one neuron. Concepts and behaviors in real models are generally distributed across many layers, attention heads, and feature directions. There is rarely one independently nameable “reviewer neuron.” Still, the metaphor of a prompt as an index is more than rhetoric. Research on Function Vectors found that a small number of attention heads can carry compact representations of input-output relations in several in-context learning tasks. Causal interventions on those intermediate representations can trigger the associated task behavior.

That result does not prove that all prompts use one mechanism. It supports a narrower claim: a model can form an internal representation of which task is currently being performed, and context helps determine how that representation arises.

The most important part of the diagram is not which words glow. It is the label “weights fixed.” Selecting a role, example, reasoning instruction, or counterexample check recombines the same behavioral features and produces a different generation trajectory. The words in the figure are explanatory dimensions, not a screenshot of real parameters. But the figure preserves the essential causal direction:

training shapes available behaviors, a prompt selects the current behavior, and generation lets that selection keep producing consequences.

Classic Prompting Patterns Provide Different Kinds of Indices

Prompt engineering contains many techniques that appear independent. Once they are placed inside the “behavior selection” framework, their differences become less mysterious.

A Role Is a Task Prior, Not a Capability Upgrade

“You are an expert” is weak because it says nothing about what the expert should notice now. A model may learn only to sound more certain.

“You are a reviewer: check whether evidence supports each claim, and separate fatal flaws from repairable ones” is more useful because the role has been unfolded into standards, targets of attention, and output actions. It provides task-relevant priors: identify claims, locate evidence, then judge the gap.

When role prompting works, the model usually has not gained more knowledge. Knowledge and behavior that were previously dispersed have been assembled into a more useful combination. When the role is irrelevant, it may add boilerplate or make an error sound more confident.

Examples Resemble the Model’s Original Training Objective

An instruction such as “classify sentiment as positive or negative” leaves ambiguities about label format, boundary cases, and answer length. Two or three input-output examples create a local regularity directly in the context, ready to be continued.

This is the most intuitive face of in-context learning. A model can continue a demonstrated pattern without first stating the whole rule. Examples do more than explain the task; they narrow the space of plausible outputs.

This is why whether examples represent the real decision boundary often matters more than their count. Wrong or contradictory examples can be treated as patterns just as faithfully.

Explicit Criteria Turn Abstract Preferences into Observable Actions

“Write better” is not a clear direction for generation. “Remove examples that do not change the reader’s understanding of the conclusion” and “give evidence for each claim or label it as a hypothesis” turn preferences into actions that can be performed and inspected in text.

Effective prompts are often concrete not because models prefer long sentences, but because concrete conditions reduce ambiguity among acceptable continuations. Length does not create control. Signal density does.

Decomposition and Verification Sustain a Mode for Longer

Complex tasks often do not lack one fact. They require one generation to maintain too many constraints at once. Extracting claims, checking evidence, and only then organizing an answer divides one easily derailed behavior into phases that can each re-establish their orientation.

Planning, counterexample search, and review against a rubric all belong to this family. They specify not only the final output but how current information should be processed along the way. My agent notes call this layer cognitive control: it changes attention, abstraction level, reasoning direction, and action tendency without adding new facts about the world.

The classic prompting patterns can therefore be reduced to three questions:

1. Does the prompt clearly point toward a behavior that training has already made available?
2. Does the context specify a sufficiently clear task mapping and standard of judgment?
3. Does the generation trajectory have room to preserve intermediate state and reorient after drifting?

The value of prompting techniques lies not in memorizing fixed phrases, but in diagnosing which of these three conditions is missing.

Why Chain of Thought Worked—and Then Became a Reasoning Mode

Chain of thought is the clearest example of this framework.

The early method did not change the model. It showed worked solutions with intermediate steps in the prompt. Chain-of-Thought Prompting found that a few such demonstrations could substantially improve arithmetic, commonsense, and symbolic reasoning in sufficiently large models. Later, “Let’s think step by step” showed that, in some settings, a short trigger could shift a model from direct answers to explicit reasoning.

In the indexing view, the prompt says: do not enter the text pattern “question → answer” immediately; enter “question → intermediate transformations → answer.” Training has already exposed a model to derivations, proofs, tutorials, and calculations. The prompt raises the probability of reproducing those generation patterns on the current problem.

Intermediate text is not merely an explanatory performance. Every step generated by an autoregressive model re-enters the context for the next step. A decomposed subproblem, a partial result, or a discovered contradiction can become input to the following computation. One difficult transformation is spread across several consecutive transformations.

This is close to the intuition that reasoning “keeps calling different abilities and accumulating them,” but the phrase should not be taken to mean that the model rotates through isolated parameter groups. The parameters participate throughout. What accumulates is intermediate state in the context, while each new state induces another pattern of activation and generation.

Chain of thought has clear limits. A model can produce fluent but false steps. Simple questions can be made pointlessly long. If the model lacks the relevant capability or facts, extra reasoning tokens do not restore them. A reasoning trajectory can be useful without longer trajectories always being better.

Modern reasoning models are therefore more than ordinary models wrapped in a chain-of-thought prompt. Training directly rewards useful decomposition, checking, backtracking, and revision, teaching a policy for using longer reasoning processes. OpenAI’s description of o1 distinguishes reinforcement learning during training from time spent thinking at test time. Research on scaling test-time compute likewise finds that extra computation depends on problem difficulty, search method, and verification; uniformly generating more tokens is not enough.

The move from chain-of-thought prompting to reasoning mode therefore contains two transitions:

• First, people used prompts to find a step-by-step trajectory that already existed but was unstable.
• Later, those interaction trajectories flowed back into post-training and became a more stable behavioral policy. The runtime then decided how much test-time computation to allocate and how to search or verify.

Yesterday’s prompt technique can become today’s training data and tomorrow’s model interface. Prompt engineering continually consumes its own successes: once a way of eliciting behavior becomes a stable trained capability, engineering attention moves to the next layer.

Prompt Engineering Designs the Entrance, Not the Capability

We can now answer the opening question.

“You are a rigorous peer reviewer” may work not because the model believes it has acquired an identity, nor because the sentence writes reviewing knowledge into the parameters. Training has already shaped tendencies related to review reports, argument structure, evidence evaluation, and role-conditioned dialogue. Post-training has made the model more likely to interpret a user’s language as a task. The role prompt organizes those dispersed tendencies into an entrance for the current generation.

This also gives prompt engineering a clear boundary:

• It can select and combine existing capabilities, but it cannot guarantee a capability absent from training.
• It can change how information is processed, but it cannot recover facts that never entered the context.
• It can request self-checking, but it cannot manufacture independent evidence.
• It can describe an action, but it cannot replace an environment and interface that actually execute that action.

Good prompt engineering is therefore not the search for one incantation that works across every model. It is the work of understanding which entrances training has given a model, then sending a task to the right entrance with the clearest possible signal.

Once a task extends from one answer to dozens of steps, however, an entrance is no longer enough. The model must keep receiving the right knowledge, objectives, observations, and feedback. The same instruction can be diluted, contradicted, or hijacked by a growing context. The question then changes from “how do we elicit a behavior?” to “how do we keep the right information governing the current decision?”

That is the subject of the next essay: context engineering.