How I Evaluate Coding Agents

Each task in my evaluation suite comes from an upstream change or issue. I normally take the full commit immediately before the fix as the starting point, then write a self-contained brief that describes the desired result without exposing the original patch.

Each evaluation task records a full commit identifier called the evaluation seed. The seed points to the exact tree the agent should receive.

The harness exports that tree and imports it into a fresh repository with one baseline commit. The result is a prepared workspace. It contains the exact source snapshot and carries no upstream remote, later commits, or evaluator notes. A baseline ref remains inside the fresh repository so the grader can inspect the agent’s changed-file scope even if the agent commits its work.

This arrangement handles two problems at once. Every run starts from identical code, and the agent cannot discover the historical fix by reading local history. The known fix stays in a separate evaluator reference file outside the prepared workspace.

The command adapter is simple. An evaluation runner gives an agent an executable, the workspace path, and either the task text or a temporary prompt file. That keeps the suite agent-neutral. Froe can run it, and so can another coding agent that accepts a working directory and a prompt.

Leak resistance depends on the surrounding setup. A third-party agent with unrestricted web access could search the upstream issue or commit. Comparable runs therefore use the same filesystem and network policy, with access limited to the prepared workspace and the reference material kept out of reach.

Regarding the results, I evaluate the code’s behavior, the scope of modifications, and the user experience. I don’t need the agent to reproduce a specific “standard answer”.

Also, since coding agents are not deterministic, a serious comparison needs repeated trials. For each round, I pin the agent version, model, reasoning setting, time limit, and approval policy. Every agent receives a fresh prepared workspace under the same environment and task-order policy.

I run each agent and task pair at least three times. The report shows the full-task pass rate. Score and duration use medians, while user intervention gets its own field. A lucky run can still be interesting, though it cannot stand in for reliability.