As Models Become More Capable, What Remains in the Harness?

Suppose a coding model often announces completion immediately after editing a file. We add an external procedure: run a check after editing, and return failures for another repair. The system improves considerably. Later, after further training, the model checks, investigates, and retries on its own. The original procedure remains, but its role has changed. Understanding harness engineering requires accounting for that change in the model.

A harness is the working system built around a model. It connects tasks, context, tools, and execution environments so that successive model calls advance a shared objective. This includes execution work such as starting processes and saving files, as well as behavioral guidance such as arranging steps, prompting checks, and supplying experience. Both kinds often live in the same framework, yet model development can give them very different futures.

Capability gaps become targets for a repeatable process

Model improvement can increasingly be organized around concrete goals. Familiarity with a domain calls for relevant material; mastery of a procedure calls for good demonstrations; better choices in complex tasks call for environments that support execution, feedback, and evaluation. Developers gain methods for cultivating capabilities, turning today’s weaknesses into specific targets for the next round of work.

The surrounding system is valuable here. A procedure can supply a checking sequence the model cannot yet organize; people can supply experience, tools, or demonstrations for unfamiliar problems. With that support, the system can complete previously difficult tasks and generate better records of the process: what was visible, why an action was taken, and what the environment returned. Practices that once lived only in human experience or framework logic acquire examples from which a model can learn.

Selected and prepared for training, these trajectories give the model a chance to learn the practices within them. SWE-Gym combines repositories, executable environments, tests, and tasks to train software engineering agents, and releases agent trajectories. The interesting relationship is that an execution environment both supports present work and supplies material for later model improvement. Better working conditions can help produce better models.

Product use can keep this process moving. Failures reveal new tasks and conditions; diagnosis identifies where to intervene; people and systems develop better paths. Those paths enter training, and the next model tackles a wider range of tasks. External capabilities, work trajectories, and model capabilities form a cycle. Within it, the harness continually builds the conditions for the next step, changing as the capability boundary moves.

The example below follows “locate, edit, then check” through three stages. As you switch, observe who decides the steps and which facilities remain outside the model. The middle stage is common: a model can already do most of the work and needs only experience it can consult when relevant. Capability can migrate incrementally.

Knowing how to check still requires somewhere to run the check

Consider the parts that training can change. Decomposing steps, selecting tools, retaining useful intermediate results, and adapting after failures all involve learnable behavior. As models improve, detailed prompts and mandatory sequences intended to elicit those behaviors can become simpler. Anthropic’s work on long-running agents uses initialization, progress files, and incremental work to support tasks across sessions. Such designs are useful engineering responses to a particular capability state, worth reassessing as model behavior changes.

Other requirements come from execution itself. Checks need processes, which need environments and dependencies. Continuing across sessions requires state to be stored somewhere. Changing real resources requires defined permissions. Even a model that knows perfectly when to check and how to repair still acts through these facilities. Better judgment improves their use without turning running processes or saved files into model parameters.

The same code can even serve two different purposes. A mandatory check meant to remind a model that often forgets verification can become simpler as the model learns. A mandatory check representing the team’s acceptance requirement for every delivery still needs system enforcement. Deciding whether a step should remain means asking what weakness it compensates for and whose requirement it represents. Counting lines left in the framework cannot reveal that distinction.

This gives harness design two clear directions. Execution facilities are built around long tasks, asynchronous waits, resources, and state. Capability support is organized around work the model cannot yet do well, through knowledge, experience, specialized tools, or procedures. Making that support separately retrievable and editable helps the current model use it and makes it easier to see what has become unnecessary. Skill folders are a natural way to hold such material.

I therefore see harness engineering as continuing work within the development of model capability: bring today’s model into new tasks, preserve learnable trajectories from effective methods, and adjust working conditions for the stronger model. Each learned capability frees effort previously spent supporting it for the next challenge. Some parts of the framework can grow thinner while the reach of the whole system keeps expanding.