The Agent Harness Is Becoming the Product

Executive Summary

The useful signal today is a late-discovered, self-reported agent benchmark result: for a finance task, a purpose-built harness reportedly moved the same underlying model from 44.87% to 82.6%. The exact number is not independently validated and should not be generalized beyond that benchmark. But the implementation details are more valuable than the headline: the gains were attributed to turning domain knowledge into callable, enforceable workflow structure rather than putting ever more prose into a system prompt.

That reinforces an emerging view of agent work. Model choice still matters, but once a task involves retrieval, multiple files, temporal constraints, and specialized conventions, the operating environment can dominate the outcome. The practical question is increasingly not “which model should run this?” but “what decisions, checks, and tools should the model be allowed—and required—to use?”

What Happened

In “How to Write a Winning Agent Harness for Your Domain,” the builder behind Reef/AlphaCumen describes an ablation on the Vals AI Finance Agent v2 benchmark (239 tasks). The post reports that a reference harness using Kimi K2.6 scored 44.87%, a retrieval-only version 49.8%, and the author’s structured system 82.6%, with the model and underlying data held constant.

This is a delayed-discovery item: the fetched page did not provide a reliable publication date, and its benchmark discussion concerns June results. It is therefore evidence of a practitioner pattern, not a new model release or a fresh independent result.

The author’s diagnosis of the earlier system is familiar. A roughly 2,500-line prompt and more than 50 overlapping tool definitions produced longer runs, competing instructions, and token-budget failures once tasks required deep reasoning across multiple filings. The replacement did not simply add more context. It separated a compact index of available skills from the detailed procedures, loaded the latter only when relevant, and used dedicated tools for repeatable finance operations.

Why It Matters

The important distinction is between information and executable structure. Retrieval supplies material; a harness decides how the material is found, which conventions apply, what work must precede an answer, and what constraints make a result acceptable. In this account, typed functions encoded finance-specific conventions, while an orchestration layer coordinated work across filings. Runtime rules—such as date clamping and a minimum number of tool calls—made certain shortcuts impossible.

Those mechanisms plausibly explain why the retrieval-only variant helped only modestly while the fuller harness improved much more. They also identify the caveat. A benchmark-specific result from the system’s creator is not proof that “69 skills” or any particular architecture will transfer to another domain. The comparison is useful because it exposes the intervention: fewer overlapping instructions, progressive disclosure, specialized operations, and explicit guardrails.

The Bigger Story

The developing canon around agents has shifted from prompt craftsmanship toward workflow design. Early systems often treated the prompt as the application. In harder domains, that turns into a brittle document that must simultaneously specify policy, memory, tool use, sequencing, and exception handling. The more durable unit may be the harness: a set of small interfaces through which an agent can discover procedures, invoke domain logic, and be checked against operational rules.

That does not mean every task needs a large framework. Overbuilding a harness can itself obscure a simple workflow. The lesson is proportionality: encode the rules whose violation is costly or recurrent; leave genuinely open-ended judgment to the model. The right boundary is determined by error modes, not by the appeal of agent autonomy.

Workflow Implications

For builders, the next experiment is an ablation rather than another prompt expansion. Take a task with clear failure cases and compare: retrieval alone; retrieval plus a compact skill index with lazy-loaded procedures; and the same system with one or two enforceable runtime checks. Track completion quality, tool-call trace, latency, and token use—not just a final score.

If the structured variant wins, inspect which constraint or domain function caused the improvement before promoting it to general architecture. If it does not, remove the machinery. The durable takeaway is not a prescribed skill count; it is that domain knowledge becomes more reliable when it is represented as something the agent can execute and verify.