When Language Models Learn the Physics of a Text World

When a model writes “I changed the 2 in the file to a 3,” the number in the file does not change with it. If its output goes through an editing tool that performs the write, the next read returns 3. Both involve text, but one stays in the conversation while the other passes through an interface and leaves a consequence in the environment. A language model begins to act when that connection is established.

I think of this environment as a text world. Files, directories, commands, search results, and error messages are objects the model encounters through text, with stable relationships between them. Move a file and its old path no longer finds it. A test produces a result only after its process runs. A successful operation changes what can happen next. These regularities form the physics of a text world: actions change state, and state constrains subsequent actions.

The small file below exists only in this page. The task is to change it to count = 3. Try announcing completion and then checking it, or perform the write and then read or check. Notice the file state, the conversational claim, and the tool result: different operations change different parts.

Learn the world, then learn to work within it

An interface still leaves the model needing to learn how to use it. When to read a file, which arguments a tool needs, and what an error means all call for relevant experience. Toolformer inserts tool calls into training material, selects useful calls, and trains the model to learn when to call, which arguments to supply, and how to use the results. Tool use can thus become a trained capability that later tasks elicit, much like reading or translation.

Mid-training is useful to understand along these lines: after broad language capability develops, targeted material adds domain knowledge and structures for understanding problems. Code and documentation develop understanding of builds, dependencies, and program behavior that can inform later investigation and editing; long-sequence training develops the ability to handle longer information structures. For example, the Llama 3 report places long-context training late in pretraining. Recipes name their stages differently, but they extend the foundations for understanding and performing tasks. External interfaces then connect those foundations to particular projects, files, and services.

Post-training further shapes choices within these situations. Good trajectories can demonstrate why to confirm a file’s location before editing it, or why a failure calls for a new hypothesis. Executable environments with scoring can also support learning from the results of attempts. Familiarity developed through mid-training connects with action skills shaped through post-training, making it easier to organize a sequence of tool calls into purposeful work.

Once actions have consequences, the real task needs a representation

Now ask the model to resolve an actual upload failure. It can read logs, call APIs, and edit files, but “restore this user’s upload” leaves many blanks: which user and upload, where the file currently resides, whether a retry creates a duplicate, and what counts as recovery. Access to a terminal does not answer these questions. The text world can now respond to actions; the next job is to represent the objects, relationships, and constraints of the business world inside it.

The core of context engineering is representing a real task as a situation the model can understand, observe, and act within. The input window holds only the portion currently unfolded. The wider representation lives in the task description, tool interfaces, files, databases, and history, entering the window through observation. The engineering work concerns that whole representation: which distinctions must remain visible, which states can be queried, and how the next action connects to current facts.

The upload failure makes the representation concrete:

• Goal: which state should be reached? The user may want to resume the original upload or merely recover what has arrived. The task should preserve that distinction and constraints such as retaining the source file and avoiding duplicate records. “Fix the upload” gives a direction. “This upload is complete, and the original file is readable with matching content” begins to establish an observable completion condition.

• Objects: which thing are we discussing? The user, file, upload session, and resulting stored object have different identities. Calling all of them “the file” invites using a filename to query a session or mistaking an old session’s success for the current one’s. Stable identifiers, explicit relationships, and essential field descriptions let logs, APIs, and the user’s account refer to the same thing.

• State: what is known, and what remains unknown? A page showing failure may simply have missed a response. The server might already have all chunks and be assembling them. Each observation needs its source and time. “The user saw a failure,” “the server has not finished,” and “the server has no data” are different judgments. Collapsing them into a single “failed” leaves even a strong model guessing within a flattened account.

• Actions: what can each interface change? Reading progress, sending missing chunks, and creating a new session do different jobs. Tool descriptions should identify the required identifiers, affected objects, execution preconditions, and ways to investigate an error. An interface returning only “invalid parameters” leaves the next step to guesswork. “This session is complete; query the object status” turns the error into a usable lead.

• Feedback: which evidence establishes completion? An accepted retry request shows that the system accepted an action. A readable final object with matching content supports the claim that the file has been recovered. Whether the user still encounters an error is another observation closer to the user experience. Action receipts, state observations, and task acceptance have different uses; distinguishing them reveals what remains to be done.

The same facts can create very different working environments

The two representations below describe the same incident. One resembles logs dropped directly into a conversation; the other places those logs within the task, its objects, and their timing. Switch between them and notice whether the next useful observation is apparent and what the word “success” actually establishes. Similar amounts of information can support very different judgments.

This does not require manually translating every log line. Tools can consistently return object identifiers, timestamps, states, and follow-up query references, while leaving detailed records available for the model to inspect. Tool design for agents emphasizes useful, intelligible context at this point: an interface’s quality depends on whether its result helps the next judgment. A query result expressed in task-relevant terms can be more useful than dozens of internal fields.

There is an important tradeoff here. A representation that is too coarse erases crucial distinctions; one that is too detailed forces the model to reconstruct the task from clutter. A useful abstraction preserves differences that change action choices. Deciding whether to resume an upload requires knowing which chunks are missing, usually not each storage node’s disk identifier. In a storage-failure investigation, that identifier may become central. The right granularity follows the decision being made.

Context has to change with the work

Earlier context-engineering systems often retrieved material in advance and inserted passages into a prompt, deciding for the model what it needed to see. With interaction capabilities, the model can request new observations: inspect the session, inspect the object, then investigate detailed records if a checksum differs. The alternation of observation and action illustrated by ReAct is one way this task representation unfolds at runtime. Writing that alternation as a loop is straightforward; its usefulness depends on whether each observation brings back information capable of changing a judgment.

Keep the goal, constraints, and important state references readily available in the current window; leave large logs, complete documents, and historical traces outside for retrieval as needed. Search locates material relevant to a question, files preserve intermediate artifacts, and compaction turns a long history into a state from which work can continue. Just-in-time retrieval and long-context management address the same tension: the world is large, while the model’s working space for each judgment is limited. Reachable references preserve broad access without losing the current focus.

Long-running work especially needs to preserve the distinction between facts, hypotheses, and unfinished work. If “suspect the server lost chunks” becomes “the server lost chunks” during compaction, the next run inherits an untested explanation as fact. A useful continuation state records confirmed observations, rejected explanations, evidence still needed for the current hypothesis, and references to the original records. Those preserved relationships support continuation better than a fluent story.

The environment can change while the model is thinking. “Uploading” may become “complete” a moment later, and another operator may modify the same record. Returning observation times and versions reveals which premises need a fresh check before an important action. Tools and runtime facilities then enforce the agreement at execution time, including rejecting version conflicts. The task representation makes the relationships visible; the execution system makes the contract hold.

An agent that remains ineffective in a domain may need a clearer working world as much as more knowledge. It cannot reliably make different choices when the differences between tasks are invisible, or correct itself through feedback when action consequences are hidden. Good context engineering keeps the relationships that determine real-world success visible to the model. The physics of a text world makes action possible; modeling the task and environment makes that action intelligible.