What Is a Harness, and How Does It Change AI Agent Performance?
The same model can finish a task in one environment and get stuck in a loop in another. What is an agent harness, where does the name come from, and what do benchmarks really tell us about Claude Code, Codex, Hermes and other systems?
The cover is an AI illustration of the working environment around a model. The charts below use the cited public measurements; the screenshot of the crabs comes from our actual experiment. Results and documentation were checked as of October 25, 2026.
Two people use the same AI model. One gets a working application. The other gets a directory full of unfinished files and a confident message saying everything is done.
The difference need not come down to who wrote the more magical prompt. One agent may have had clear tools, ongoing notes and a way to check its results. After a few dozen steps, the other lost sight of the original goal, repeated a failed action and stopped too soon.
After our experiment with four AIs on an island, a similar question came up: how did the crabs write their diaries and communicate? That brings us to the word harness: the layer that determines what a model sees, what it can do and what it learns about the consequences of its decisions.
And this layer can make a surprisingly large difference.
Harness: what the word means and where it came from
The English word harness can refer to equipment or the straps used to attach an animal to a load. As a verb, it can mean putting that equipment on an animal or making use of a source of power. It is not an acronym. The metaphor works well for AI: it connects an available capability to useful work.
The word predates computers by centuries. It is recorded in English around 1300 in the sense of equipment, especially military equipment, and comes from the Old French harnois. Its use for the equipment of working animals followed in the early 14th century. The figurative verb meaning to control or use a force is documented from the 1690s. Etymonline: harness
Software has long used the term test harness: supporting software that runs the program being tested, supplies inputs and monitors results. It appears at least as early as the IEEE 610.12-1990 software engineering glossary, as a synonym for a test driver. This establishes documented use in 1990, not the date of its invention. IEEE, test driver / test harness, page 75
In today's discussion of agents, harness mainly refers to the software layer that runs them. Anthropic explicitly discussed it in an article about long-running agents on November 26, 2025. On February 17, 2026, LangChain described harness engineering: systematically improving that surrounding software. These dates do not identify the first person to use the idea either; they are specific, traceable milestones. Anthropic · LangChain
What happens between a request and a result
The model proposes responses and next steps. The harness organizes their execution: it assembles context, exposes tools, executes permitted actions and returns their actual results to the model. The agent is the complete system operating this way. Claude Code's documentation makes a similar distinction between the model and its working environment.
Imagine a capable colleague joining a company. Their skills matter. So does having the right information, access to the necessary applications and a way to check their work. Without those things, they might describe what they would do very convincingly while completing nothing.
For an AI, this equipment includes:
- Context: the instructions, files and parts of the history the model currently sees.
- Tools: how it reads documents, edits code, calculates, searches or operates applications.
- Memory and state: what is saved between steps or sessions and how it is retrieved.
- Workflow control: what happens after an error, when work continues and when the run ends.
- Verification: whether tests actually passed, calculations agree or the promised file exists.
- Permissions and limits: what the agent may do and how much time or money it has.
Our simplified diagram. A particular harness may combine steps, and verification may require more tools. A message saying “done” is not, by itself, verification.
Two other common sources of confusion: a framework is a software building kit, and SDK stands for software development kit, a collection of resources for developers. Either can help build a harness. A benchmark is a set of test tasks with scoring rules; its evaluation harness runs and evaluates those tests. One experiment can therefore contain a harness that helps an agent work and another harness that measures its work.
Better results without retraining the model
LangChain kept GPT-5.2-Codex and changed the surrounding Deep Agents system. On Terminal-Bench 2.0, the score rose from 52.8 to 66.5%, an increase of 13.7 percentage points. Changes to instructions, tools and run control helped: more consistent verification, better use of the time budget and analysis of where the agent got stuck. This was an experiment by the harness developer, tuned on a specific benchmark, not a guaranteed improvement on every task. LangChain's experiment
Another route is skills, reusable procedures for particular kinds of work. A skill can include instructions in SKILL.md, scripts, templates and examples. For a monthly report, it might specify how to normalize currencies, verify totals and produce the company's standard format. Codex can load a short description first and open the full instructions when the skill is used. Skills documentation
SkillsBench v4 studied 87 tasks across eight domains and 18 model–harness configurations. Curated skills raised average success from 33.9 to 50.5%, an increase of 16.6 percentage points. This is an average across different configurations; an arbitrary downloaded guide does not promise the same benefit. SkillsBench, revision of June 14, 2026
Each panel shows a different experiment. Score levels across the two panels cannot be used to compare system quality.
A tool performs an operation. A skill supplies a procedure for using tools. The harness gets the right information to the model at the right time. Adding another hundred pages of instructions may therefore do little good: it can also obscure what matters most.
Memory is more than a file ending in MD
Context is the content the model receives right now. Memory keeps information for later use. A saved diary does not help if nobody loads it during the next task or if important experiences cannot be found in it.
Markdown, or MD, works well for readable notes and instructions. JSON stores structured data, such as the status of individual steps. JSONL consists of separate JSON records, one per line, which is useful for event histories. Format is only part of the solution, though. We need to know who recorded the information, what supports it and whether it is still valid.
In its experiments with long-running agents, Anthropic combined a feature list in JSON, a progress log and Git history. The model was less likely to make inappropriate changes to the JSON list than to its Markdown equivalent. Merely shortening the conversation into a summary was not enough to continue work smoothly. This is evidence from a particular development workflow, not a general superiority of one format. Anthropic on long-running agents
When an agent records an experience and uses it next time, it usually does not change the model's weights. It changes the information available for the next decision. Calling this “learning” is understandable in everyday language, but technically it is a different mechanism from retraining or fine-tuning.
Markdown, RAG, knowledge graphs or HEAR?
Our discussion led to another question: when is a folder of notes enough, and when does an agent need a knowledge graph? This decision belongs in harness design. An agent can have the right information stored somewhere yet fail to find it, overlook its validity or fail to connect it with another condition.
First, distinguish what we are choosing. Markdown is a format, RAG is a way of supplementing context, a knowledge graph is a data model and HEAR is a specific research architecture for an agent. They can work together: Markdown documents can be indexed, a graph can point to original passages, and an agent can switch between search, databases and relationship traversal according to the question.
| Need | What I would start with | Example question |
|---|---|---|
| Project rules, notes and repeatable procedures | Organized Markdown files and file tools | How do we run tests in this project? |
| Finding a specific fact in documents | RAG with citations and validity filters | What does the current travel-expense policy say? |
| Dependencies and effects across multiple objects | A knowledge graph; GraphRAG where appropriate | Which customers would this supplier's outage affect? |
| Repeated process diagnosis across systems | A validated procedure; possibly an architecture similar to HEAR | Why did this order stall between purchasing and dispatch? |
| Exact totals and current operational state | A direct database or relevant API query | How many orders are currently waiting for dispatch? |
Practical guidance for an initial design, not a ranking of technologies by quality.
When files are enough and when to add RAG
For project rules and well-defined procedures, I would start with files. People can easily read them, correct them and compare changes. A short index can point to details; the entire directory need not be inserted into every request. Anthropic also describes loading information as needed in its article on context engineering. That article does not establish a general superiority of Markdown over graphs.
RAG stands for retrieval-augmented generation: generation supplemented by retrieved information. The well-known paper with this name appeared in 2020. For an assistant working with policies or manuals, the important thing is to find the relevant passage before answering and preserve its source. The original RAG paper
In practice, RAG need not be limited to semantic similarity search. This can be combined with keywords, which helps with exact identifiers, for example. Filtering and reranking by relevance can also be added. How hybrid search works
“What is the notice period in this contract?” calls for a specific clause. “What causes our delivery delays?” requires broader coverage. A few highly similar passages may omit important data. And to count orders, I would prefer a controlled database query over asking a model to add up retrieved text snippets.
When a graph is worthwhile and what a hypergraph adds
A knowledge graph represents objects and their relationships: a supplier provides a component, the component belongs to a product, and a customer uses that product. It makes sense when similar relationships come up repeatedly. But the structure comes with work: resolving identities, defining relationships and keeping data current. Incorrectly linked companies or an outdated dependency can undermine an entire chain of conclusions.
GraphRAG uses a graph to retrieve context. Microsoft's implementation also summarizes groups of related entities, helping answer questions about an entire corpus. Its 2024 evaluation found benefits in the comprehensiveness and diversity of answers to global questions. That result should not be confused with universal factual accuracy. GraphRAG and its evaluation
A hypergraph lets one relationship connect multiple participants, such as the supplier, buyer, material and conditions of a particular delivery. An ordinary graph can also represent this by introducing a separate delivery node. W3C described this treatment of n-ary relationships in 2006. The value of a particular system depends on its use and measured results, not on the prefix “hyper.”
HEAR: relationships plus a procedure for gathering evidence
HEAR — Hypergraph Enterprise Agentic Reasoner is a research architecture from SUPCON. It connects enterprise data sources, business rules and procedures for gathering evidence. I would consider its principles for repeated order diagnosis across systems, but they require preparation and maintenance of process knowledge.
The study's September 7, 2026 version reports these results with GPT-5:
| Test | HEAR | Table-RAG |
|---|---|---|
| OF-BRCA: 95 questions about the cause of blocked orders | 94.7% | 52.6% |
| OF-General: 160 varied questions, fully correct answers only | 37.1% | 29.6% |
For the first test, the authors prepared a diagnostic procedure. Removing procedural hyperedges reduced accuracy to 74.2% and raised average consumption from 38.1 to 179.3 thousand tokens. On the broader questions, the reported 88.7% includes partially correct answers; full correctness remains 37.1%. HEAR, methodology and Tables 1–2
This illustrates the connection to harness design: representation helps together with tools and a prepared procedure. It is a study using the authors' enterprise data, not a universal victory over files or every form of RAG.
Our decision guide. HEAR is a complete architecture, not another memory format; these approaches can be combined.
What I would check before adding more complex memory
Even the familiar “files versus graphs” comparison needs careful reading. Letta reported 74.0% on LoCoMo, but its files were also embedded in a vector index and the agent used semantic search. Concluding that “Markdown beat RAG” would therefore be wrong. Letta's experiment
I would start with users' questions and an analysis of errors. If specific passages cannot be found, improve search. If relationships cannot be connected, test a graph. If the same diagnostic work keeps recurring, capture its procedure. Alongside correctness, I would count response cost and time, plus the cost of building and updating the index. Tests should include a changed fact, revoked document access and a question the available evidence cannot answer.
The resulting system might have Markdown rules, JSON state, documents retrieved through RAG, current numbers in SQL and relationships in a graph. The harness's job is to select and verify the right information for the current step. That is a more useful design goal than replacing everything with one fashionable technology.
How we changed the harness for our crabs
On the island, the agents had tools for observation, movement, manipulating objects, chat and memory. Their Czech messages went through a shared game chat. Each agent had private long-term notes in SEMANTIC_MEMORY.md and an ongoing event diary in EPISODIC_MEMORY.jsonl.
Cerebras exposed an interesting problem. Other agents could post useful advice, but until it reached its context, it could not help its decisions. We therefore added regular chat reading. We also changed how repeated errors were handled, recovery after API outages and information about who was still running.
These are harness changes: they affect information flow and the course of work. The agents still composed their messages and made many decisions themselves. But we assigned specific roles, some coordinates and supporting procedures.
An actual game screenshot from the previous article. We also widened and reinforced the steps; Kimi used an automated climbing test. This outcome cannot be presented as the isolated effect of a better harness. The screenshot preserves the original Czech interface.
The steps mark an important boundary in our account. We changed the world itself, too, so that the agents would stop knocking the structure apart. They also received testing assistance. Success came from a combination of their actions, coordination and our technical help. Without a separate controlled experiment, we cannot attribute every improvement to memory or “growing intelligence.” Details and records are in the article about the island, steps and Aster.
Claude Code, Codex and Hermes: where they make sense
These products overlap and evolve quickly. This overview summarizes their documented focus and my practical recommendations for what to test with them. It is not a universal ranking from best to worst.
| System | Strong practical use cases | What I would test on my own work |
|---|---|---|
| Claude Code | Work with projects, files and the terminal; a loop of gathering context, acting and checking | Completing a change in a real project and verifying it correctly |
| Codex | Agentic work with code and files; a shared harness for different clients and integrations | Repair quality, use of project rules and cost per completed task |
| Hermes Agent | A longer-term assistant with memory, search across past sessions and saved procedures | Whether useful experience actually carries over into the next session |
| OpenCode | Multiple providers and local models in a similar working environment | Differences between models given the same task and tools |
| OpenClaw | An assistant with a persistent workspace, memory files and state on the Gateway host | Continuity, retrievable memory and data flow |
| Deep Agents | Building a custom harness with adjustable context handling, tools and run control | The benefit of each change against a simple baseline |
With Codex, distinguish the product from the model: Codex may name the environment, while GPT-6.1 Sol, for example, names a particular model. Recording both tells us more than saying “I used OpenAI.”
Hermes is interesting because it can create and update skills from experience. That opportunity needs testing: a saved procedure may be useful, but it may also preserve an earlier mistake. Its own benchmark portal helps compare models within Hermes; it does not by itself prove Hermes is better than other harnesses.
With OpenClaw, locally stored memory does not automatically mean local model inference. And access to a tool does not guarantee the agent will use it correctly. Choosing a product is the beginning of evaluation, not the end.
Where to find benchmarks and what they measure
Leaderboards answer different questions. For an office assistant, fixing bugs in code is not sufficient evidence. And to measure the effect of a harness, I need to keep the model unchanged while switching the harness.
| What I want to find out | Benchmark and link | Main limitation when interpreting results |
|---|---|---|
| How the same model behaves in different harnesses | Harness-Bench | Specific versions and configurations; score may not mean complete delivery |
| Whether an agent can handle technical terminal work | Terminal-Bench | Compare the same test version and identify the model, harness and budget |
| How it fixes issues in existing projects | SWE-bench | Fixing an issue does not cover the entire product lifecycle; test variant matters |
| The quality of spreadsheets, presentations and analytical documents | AA-Briefcase | Content and presentation are evaluated in a shared testing environment |
| How it operates graphical applications | OSWorld | Working through a screen differs from using APIs or scripts |
| How it serves customers while following process rules | τ²-bench | Domain and interaction format matter |
| Whether supplied procedures help | SkillsBench | Compare runs with and without the procedures |
| How it handles incomplete requests | AM-Bench | Completing the visible request can differ from satisfying the full intended process |
The last question is practical: even a stronger model may not compensate for a missing condition. The harness must pass on important rules and allow missing information to be supplied. A completed file may be unacceptable if it was created using the wrong assumptions.
Harness-Bench: different models fit different harnesses
Harness-Bench v2 compared seven harnesses. Its general suite contained 274 tasks and ten models. The overview below includes all ten models and all seven harnesses in that part of the study. It shows mean scores out of 100, not percentages of fully completed tasks.
Open the full-resolution chart · Data for all 70 combinations
GPT-5.6 Sol did best with OpenClaw, DeepSeek-V4-Pro with Codex and Kimi-K3 with OpenCode. The practical lesson: choose the harness together with the model.
The study has important limitations. Each task–model–harness combination was run once, so small differences should not be overinterpreted. Tested versions included Claude Code 2.1.220, Codex 0.145.0, OpenCode 1.18.8, OpenClaw 2026.5.26 and Hermes 2026.7.20. The study did not test Claude models or GPT-6. Research connections between a backend and a product may also be unavailable in a normal subscription. Harness-Bench v2, October 15, 2026 revision, Tables 2 and 5
Terminal-Bench: performance of the whole configuration
The public Terminal-Bench 4.0 leaderboard showed these results when checked on October 25. All four configurations used max reasoning effort, the highest setting shown here. The same setting name does not guarantee identical computation across providers.
Opus 5.5 with Claude Code leads on the point estimate. The error bars show 95% confidence intervals. Since both the model and harness change between configurations, this chart cannot separate their individual contributions.
Astra and Sol have the same point estimate, but the leaderboard reports approximately $3.3 thousand versus $0.6 thousand for the evaluated run. These are total evaluation costs, not the cost of one task or a monthly subscription. Official results
Watch the version, too: Terminal-Bench 4.0 changed environment resources and fixed or removed tasks. Its authors require new runs. The earlier 66.5% result on version 2.0 therefore cannot be directly compared with today's 64.8% on version 4.0. Terminal-Bench 4.0 changes
Office work tests different abilities
AA-Briefcase v1.1 contains 91 tasks across four work projects. It combines rubric compliance, analytical quality and presentation into a shared Elo score. Selected results also show why reasoning settings matter:
| Configuration | AA-Briefcase Elo |
|---|---|
| Claude Sonnet 5.5 · Max · Default Fallback | 1816 |
| Claude Opus 5.5 · Max · Default Fallback | 1801 |
| Claude Opus 5.5 · Medium · Default Fallback | 1622 |
| Claude Sonnet 5.5 · Medium · Default Fallback | 1438 |
At Max, Sonnet has the higher point estimate; at Medium, Opus does. The top two results are close: their reported 95% intervals are 1805–1828 and 1790–1813. The order of point estimates alone does not establish a reliable difference. I have also retained Default Fallback, which is part of the tested configuration. Tasks currently run independently, without carrying over the model's own earlier submissions, so this is not a complete test of long-term memory. AA-Briefcase
Artificial Analysis uses the shared Stirrup harness here. This comparison helps choose a model for a similar working environment. It is not a direct contest between the Claude Code and Codex applications. AA-Briefcase methodology
How I would choose a configuration for real work
In Hyperprostor, we compare models using multiple criteria (article in Czech): quality, cost, speed and task type. With agents, we also need to ask what working conditions the model received.
For my own comparison, I would choose a few dozen representative tasks: fixing a real bug, analyzing source material, creating a presentation and continuing unfinished work. I would decide in advance what makes a deliverable acceptable, then record the model, harness version, tools, skills, memory, time and budget.
To measure the effect of the harness, I keep the model and conditions as similar as possible. To choose the best available complete system, I can give each a suitable configuration, but must disclose that. These are different questions, and both experiments are useful.
I would reserve some tasks for testing after tuning is finished and repeat important runs. For a long-term assistant, I would add another session: can it find the right note, recognize outdated information and avoid last week's mistake? I would count infrastructure failures and human interventions separately.
Alongside success rate, I would track time and cost per accepted result, including corrections and human review. A cheap run that almost never finishes anything can be expensive in practice.
A purely illustrative calculation using invented numbers. This is not product pricing or another measured benchmark. The original example's currency, Czech crowns (CZK), is retained.
When an agent gets stuck, we can buy a stronger model. Sometimes that helps. At other times it lacks the right file, has not received the result of its previous action or does not know a colleague found the solution a moment ago.
On our island, that information took the form of Czech advice in the chat. In a company, it might be a test, a properly recorded decision or a traceable source in a spreadsheet. The harness makes that information available to the next decision. This is often where the difference emerges between an agent that talks impressively and one whose work we can actually accept.
Links to original methodologies appear beside the claims and tables. The nine images consist of one AI illustration, seven original explanatory diagrams or charts and one original experiment screenshot. Underlying numbers, image origins and links are also available in the image source record and translation record. Leaderboards are a snapshot in time, not a permanent ranking of products.