28 of 40
resolved.

Same model. Four harnesses. One hard question: what survives contact with the verifier?

We held the model constant and changed the coding harness. Ten feature requests, four agents, forty disposable E2B environments, and one external judge produced a simple answer: the easy work is nearly saturated; the harness shows up when the repository gets difficult.

01 / What we found

The result table below is the benchmark’s canonical aggregate. “Resolved” means the independent FeatBench evaluator passed both the requested Fail-to-Pass suite and the existing Pass-to-Pass suite. An agent saying “done” never counted as proof.

40 canonical trials · $1.831922 displayed aggregate · 54,947,717 input + output tokens · cost source is recorded per trial

02 / The story in the hard block

The easy block is useful as a floor, but it cannot distinguish the systems: all four harnesses passed all five easy tasks. The hard block is where planning, context handling, repair loops, and termination discipline began to matter.

Cell format: F2P passed / total; P2P passed / total · full resolution requires both suites

03 / Why the evaluator matters

Eleven runs contained language that looked like a completion claim while the external verifier still reported failure: CritiqueCode 2, Claude Code 3, Oh My Pi 3, and OpenCode 3. This is why the benchmark treats the verifier as the source of truth. The claim detector is intentionally only a heuristic; the F2P/P2P result is the scientific measurement.

There were zero strict regressions under the study definition: no run passed all F2P tests while failing P2P. One Claude Code run on SmolAgents 1442 hit an empty or malformed HTTP 200 response on the Anthropic-compatible OpenRouter path. Mercury was not silently replaced, and the verifier still ran.

04 / Exactly what ran

Harbor owned the trial lifecycle and grading. E2B supplied a fresh single-container sandbox. Each agent received the same natural-language task and repository starting point, but kept its normal prompt, tools, context strategy, and termination behavior.

1. Freeze ten FeatBench task IDs: five static easy labels and five static hard labels.
2. Run the official/reference solution for every candidate in a clean E2B trial; only the ten reward-1.0 tasks entered V1.
3. Launch one Harbor E2B trial per task/harness pair, never reusing a running sandbox.
4. Prepare the repository and dependencies before agent_started_at; count only harness execution in agent time.
5. Use Mercury through OpenRouter, then run the external F2P/P2P verifier after the harness exited.
6. Capture the diff, trajectory, logs, timing, token usage, normalized tools, and termination state before destroying the sandbox.

The run used Harbor 0.22.0, E2B SDK 2.46.4, Claude Code 2.1.259, Oh My Pi 18.1.5 on Bun 1.3.14, OpenCode 1.18.27, two concurrent sandboxes, a 3,600-second agent limit, a 1,800-second setup limit, and a rotated harness order.

05 / The frozen task mix

The easy/hard labels are this study’s transparent static selection labels, not an official FeatBench difficulty score. The full selection rationale and pre-excluded candidates are public.

06 / The accounting, without hand-waving

The run recorded 725 OpenRouter generation IDs: 724 provider responses were available and one returned HTTP 404. Nineteen canonical trials have complete provider totals. The remaining twenty-one use input × $0.04/M + output × $0.15/M, using the model catalog rates captured for this run. The provider ledger is authoritative when complete; the method and source flag remain visible in each result record.

A separate OpenRouter key per harness would have made the cost comparison stronger. Only one credential was available, so every sandbox used the same provider key. That is a limitation, not a hidden assumption.

07 / Follow the entire trail

The benchmark is open as a working artifact, not just as a conclusion. Start with the report, then follow the protocol, task freeze, launcher, adapters, oracle evidence, canonical records, and patches.

08 / Reproduce it

Clone the public repository, provide your own E2B and OpenRouter credentials, apply the two documented benchmark plumbing patches, and run the same launcher. Never put credentials in task files, logs, charts, or committed configuration.

git clone https://github.com/repath500/mercury-harness-benchmark.git
cd mercury-harness-benchmark
uv tool install harbor
export E2B_API_KEY='your-key'
export OPENROUTER_API_KEY='your-key'
export HARBOR_SOURCE_DIR='/path/to/harbor'
git -C "$HARBOR_SOURCE_DIR" apply benchmarks/mercury_v1/patches/harbor-e2b-v1.patch
PYTHONPATH=. uv run --no-dev --extra e2b --project "$HARBOR_SOURCE_DIR" python3 benchmarks/mercury_v1/scripts/run_v1.py --concurrency 2
python3 benchmarks/mercury_v1/scripts/fetch_openrouter_usage.py
python3 benchmarks/mercury_v1/scripts/normalize_v1.py
python3 benchmarks/mercury_v1/scripts/build_chart_data.py

09 / What this does—and does not—claim

This is a descriptive V1 study: ten tasks, one run per task/harness pair, and no inferential statistics. It does not establish a universal harness ranking. It does show a reproducible difference in observed behavior under a fixed model, a fixed task list, independent verification, and disposable environments.

The next useful experiment is replication: repeat the frozen ten tasks with dedicated provider keys, then add a second pre-registered task block. Multi-container web applications were intentionally excluded because Harbor’s E2B environment does not support Compose/multi-container tasks.