The default answer to “make the agent better” is a better model. But a growing pile of evidence says that a lot of agent failure isn’t model failure at all — it’s the execution system around the model losing track of what it was doing. A runtime system published this month makes the case sharply: StateM, an agent-native runtime that improves long-horizon execution without touching model weights, pushed a frontier model to 95.3% raw accuracy on Terminal-Bench 2.1 — and replicated most of the gain on a budget model for roughly the cost of a takeout lunch.
The paper calls the technique harness scaling: instead of scaling parameters or training compute, you scale the quality of the execution system wrapped around a fixed model. It’s a deliberately contrarian bet in a field obsessed with weights, and the results are hard to argue with.
Why Long Agent Runs Fail
Long-horizon agents fail for mundane reasons. The original goal fades from attention as the context fills up. Progress lives only in chat history, so nothing survives a context refresh. Verification gets postponed because the next step always looks more urgent. A new session can’t reconstruct what the previous one did. The maddening part: the underlying model can often solve every constituent step, and the run still fails — because state, procedure, and evidence were never anywhere except inside a context window that treats them as disposable.
StateM moves that procedural state out of the model context and into a lightweight, versioned runbook. It’s a command-line state machine that turns an agent workflow into an inspectable graph of states, transitions, and executable checks — keeping planning, execution, verification, repair, and handoff from collapsing into one long prompt.
What StateM Actually Is
The core loop is a familiar shape — prepare, execute, verify, handoff, with a repair edge back from verify to execute. What makes it agent-native is that the runbook is a plain YAML file the agent itself can read, author, inspect, and repair. At every state, the agent can answer four questions from disk instead of memory: What should I do now? Which transitions are legal? What evidence is required before I move? What happened earlier in this run?
name: implementation-loop
initial: plan
nodes:
plan:
prompt: |
Read the task and write a concrete implementation plan.
before_transfer:
type: checklist
items:
- Scope and constraints are recorded
- Verification steps are defined
execute:
prompt: |
Implement the plan and keep the change scoped.
before_transfer:
- type: command
run: "python3 -m pytest -q"
- type: checklist
items:
- Relevant tests pass
- Unrelated files were not changed
handoff:
prompt: |
Summarize the change, verification, and remaining risks.
edges:
- from: plan
to: execute
condition: The plan is ready.
- from: execute
to: plan
condition: Verification found a fixable gap.
- from: execute
to: handoff
condition: The implementation and verification are complete.The interesting mechanism is executable transition gates. A state can’t be left until its before_transfer checks pass — and those checks can be checklists, shell commands, predicates, manual approval, or an LLM review. Agents can even register task-specific dynamic checks for the current state entry without mutating the shared runbook. The runtime persists the current node, transition history, hook results, evidence, and timestamps under a .statem/ directory, and generates safe resume and compaction prompts for long cyclic runs — so a context refresh is an inconvenience, not an amputation.
statem validate runbook.yaml
statem start runbook.yaml --run-id my-run
statem cur --run-id my-run
statem next --run-id my-run
statem history --run-id my-runPositioning matters here. StateM is deliberately smaller than a general workflow engine: a TODO list doesn’t block invalid transitions, a CI pipeline isn’t agent-editable, and a workflow engine is rarely either inspectable or repairable by the agent itself. StateM sits in the intersection — a state-aware runbook that a model can operate like any other file on disk.
The Terminal-Bench 2.1 Numbers
Terminal-Bench 2.1 is a benchmark of 89 complicated, multi-step terminal tasks, and the results are the reason the paper topped the research community’s daily rankings this week:
- GPT-5.5 xhigh goes from an 83.1% reference score to 92.1% with StateM — beating GPT-5.6 Sol Ultra at 91.9%. The runbook transferred to GPT-5.6 unchanged.
- With GPT-5.6 Sol xhigh, StateM reaches 95.3% raw accuracy across 445 trials, with all 89 tasks solved at least once.
- A frozen profile (no per-model tuning) lifts GPT-5.6 Luna from 76.7% to 85.4% — above the 84.9% Sol xhigh reference. A runbook built for one model helping a cheaper one is the strongest evidence that the artifact stores real procedural knowledge.
- Less than $38 of adaptation raises DeepSeek-V4 Flash from 82.7% to 88.1% under standard timeouts, and 89.1% on an 88-task common core. Extending only the remaining latency-sensitive task matches the reported 88.8% GPT-5.6 Sol maximum.
- Final-score API usage came to about $15, versus $574.68 for the GPT reference run; total DeepSeek expenditure was $52.22.
The team also evaluated transfer on BusinessBench, where family-specific runbooks built on development sets yielded held-out gains of 0.55 macro and 1.34 micro points — modest on average, but two mechanism-matched families improved by over 10 points. The honest boundary of the claim, straight from the paper: concrete rules generalize when tasks share execution structure, while the control methodology applies broadly. Read that as — steal the method everywhere, expect the specific rules to transfer only within task families.
Why It Works
Three of the runtime’s properties do most of the lifting. Durable states mean the run’s ground truth lives in files with timestamps and spec identity, not in a context window that decays. Phase-local context means each state’s prompt only needs the information relevant to that phase, which keeps prompts small and focused. And versioned procedural practices turn selected postmortem findings into persistent, executable preconditions — the system’s “golden rules” — making learned controls explicit and enforceable rather than hoping the model remembers a lesson from three runs ago.
StateM isn’t an isolated result. The same week’s research rankings are crowded with execution-substrate work: EnvHarness converts static environments into ones agents can learn from, a study of agent skills explains why they work until they don’t, and Agentic ESOpt fine-tunes long-horizon agents on minimal GPU budgets. The shared bet across all of them: the next doubling in agent capability is at least as likely to come from the harness as from the weights.
What to Steal Without Adopting the Tool
You may not need another runtime in your stack. The transferable ideas are:
- Put durable state on disk, not in the conversation. If a context refresh destroys the agent’s understanding of where it is, that’s a design bug.
- Make exit criteria executable. “Tests pass” as a shell command beats “tests pass” as a paragraph the model can rationalize around.
- Scope context per phase. A planning prompt and a verification prompt need different slices of the run’s history.
- Close the postmortem loop. When a run fails, convert the lesson into a precondition the runtime enforces — otherwise you’re re-paying for the same mistake every run.
- Version the runbook in git so procedure improvements are reviewable diffs, not folklore.
The StateM repository is Apache-2.0, requires only Python 3.11+, and has zero runtime dependencies — a deliberately boring dependency footprint for a tool whose whole thesis is that boring reliability beats clever prompts. The paper and a 40-second demo are on the project page. If you have an agent task that runs longer than one context window, that’s exactly the workload worth pointing it at.