Agent Reliability Is Becoming a Systems Problem

Executive Summary

The useful shift in agent discourse is away from asking whether a model is generally capable and toward asking whether a specific system is dependable enough to own a bounded task. Two small releases from Simon Willison make that shift unusually concrete: one narrows the protocol boundary for tool use; the other makes the surrounding harness an object of evaluation. Together, they argue that the next increment of agent reliability is likely to come as much from interface design, permissions, and measurement as from a new model checkpoint.

What Happened

In “Stateless MCP has recaptured my interest”, Willison revisits the Model Context Protocol through its newer stateless transport. Rather than maintaining a server-side session across initialization and tool calls, the revision can make each request self-contained. He reports using it to build mcp-explorer, datasette-mcp, and an LLM MCP client.

The technical simplification matters because it makes a tool server less like a semi-autonomous collaborator and more like a conventional, inspectable service endpoint. Willison's practical preference is not for an agent with broad shell and network access, but for a small tool surface with explicit inputs, outputs, and capability boundaries—particularly around sensitive data. That is a practitioner judgment, not a security proof, but it identifies an important design pressure: systems become easier to audit when the interaction contract is smaller and more deterministic.

The companion release, smevals, is a compact evaluation framework for models, prompts, and harnesses. It runs task suites across configurations, preserves runs separately, and applies graders built from checks or custom checkers before producing a static report. Its key contribution is conceptual as much as mechanical: a result belongs to the full configuration, not simply to “the model.”

Why It Matters

That distinction is increasingly central to how teams should interpret impressive agent demos and disappointing deployments. A model's output is shaped by the prompt, tool descriptions, retrieval, permission model, retries, context management, and the harness's choices about when to stop or escalate. A benchmark that abstracts most of that away can still be useful, but it cannot answer whether your system will perform safely and reliably on your work.

Stateless MCP and smevals address different points in that chain. The first reduces hidden state and narrows the authority granted to a tool interaction. The second creates a way to test the behavior produced by those choices. Neither eliminates model error. Together, though, they make failure modes more legible: a bad result can be traced to a task, a prompt, a tool contract, a configuration, or a grader rather than being absorbed into a vague verdict about AI capability.

The Bigger Story

This reinforces a developing canon in agent engineering: progress is less usefully described as a race to maximize autonomy than as a search for dependable compositions. Broad-access agents remain compelling for exploratory work, but they are hard to constrain and difficult to reproduce. Bounded tools with observable transactions are less theatrical, yet they create the preconditions for testing, review, and gradual expansion of authority.

The result is a more mature framing of the human role as well. The operator is not merely waiting for the model to become trustworthy; they are choosing the task boundary, defining what counts as success, and deciding which actions require a human checkpoint. This is where protocol ergonomics becomes operational safety rather than plumbing.

Workflow Implications

For builders, the practical experiment is straightforward: select one recurring agent task and evaluate the entire configuration, not just the prompt. Keep the action surface small; make tool inputs and outputs explicit; create a task set drawn from real failures and routine cases; and grade for both outcome quality and unacceptable actions. Then change one variable at a time—model, instruction, tool schema, retry policy, or permission level—and retain the run history.

If a workflow cannot be tested in that way, it is probably asking for too much opaque autonomy. The better next step may be to turn an open-ended agent action into a stateless, bounded tool call before searching for a stronger model.

Further Reading

• Simon Willison, Stateless MCP has recaptured my interest — a hands-on account of the protocol change and its implications for auditable tool boundaries.
• Simon Willison, smevals — a small eval suite for evaluating models, prompts, and harnesses — the concrete evaluation workflow behind the broader argument.