Building a harness you can trust

The goal of gradientsmith is narrow. Measure what models can actually do with numbers that do not drift, then use those numbers to make an open model better. This page is the whole build. What the harness is, what a task looks like, how each part works, and why this beats pointing a default agent at the problem. If you wanted to build a verifier-first harness yourself, this is the order to do it in.

The thesis, verdicts not opinions

An eval is only as good as its grader. The popular shortcut is to ask a strong model to judge another model's answer. It feels reasonable and it fails quietly. The judge's standards drift between versions, it can be talked into a good grade, and it carries measured biases toward the first answer it sees and toward longer answers [2023] [2024]. Worst of all, the instant you train against that grade, the model under training learns to please the judge instead of solving the task.

So gradientsmith fixes one thing first and builds everything on it. Every primary metric is a deterministic, sandboxed verifier. Give it the same solution twice and get the same verdict twice. For code, the verifier executes the model's function against unit tests and compares outputs. For structured output, it validates against a JSON schema plus exact and property assertions. There is no model in the scoring loop.

What an eval task actually looks like

A task is a prompt plus a verifier, and it ships two sets of tests. The public tests are shown to the solver. The hidden tests are used only at final scoring. Here is a real code task from the seed set. It asks for run-length encoding.

task_id     code/run-length-encode/001
entry_point rle_encode
description "replace each run of a repeated character with the
             character followed by the run length"

public   rle_encode("aaabbc") == "a3b2c1"
         rle_encode("")       == ""
         rle_encode("a")      == "a1"

hidden   rle_encode("111")             == "13"     # a run of three '1' chars
         rle_encode("zzzzzzzzzzzz")     == "z12"    # a two-digit run length
         rle_encode("abab")            == "a1b1a1b1"

The hidden cases are where a shallow solution falls over. A run of the digit one encodes to "13", which a careless solver can confuse with the input. A run of twelve produces a two-digit count. None of that is visible from the public cases, which is the entire reason the split exists. The gap between a model's public and hidden pass rates is a direct reading of how much it overfits to what it can see.

The second family is structured output. The task gives a sentence and asks for a JSON object, and the verifier checks it against a schema and a set of path assertions rather than reading prose.

task_id  structured/event-extract/001
description "extract date, time, room, organizer from a sentence
             into a JSON object with exactly these fields"

public   keys_exactly  ""    == [date, time, room, organizer]
         path_equals   date  == "2026-03-15"

hidden   path_equals   time  == "14:00"
         path_equals   room  == "4B"

This lineage is not new. Execution-based code evaluation was made standard by HumanEval and MBPP, which run generated code against held tests rather than trusting a description of what it does [2021] [2021]. More recent benchmarks push the same idea further, from repository-level bug fixing in SWE-bench to contamination-resistant contest problems in LiveCodeBench and richer library use in BigCodeBench [2023] [2024] [2024]. gradientsmith applies the approach to its own task bank and, crucially, closes the loop back into training.

Every seed task is validated by running its own reference solution through the real verifier against every test. A task whose reference cannot pass its own hidden tests is a broken task, and CI rejects it.

The architecture

It is a Python monorepo with a strict one-way dependency graph, so a change in the eval layer can never reach back into the foundation.

core  ←  evalkit  ←  {router, training}  ←  cli        web reads exported JSON

core holds the model-provider abstraction, one interface over Anthropic, OpenAI, and any OpenAI-compatible endpoint such as OpenRouter, plus a model registry, exact decimal cost accounting, a global budget guard, rate limiting with retries, and the database. evalkit is the eval harness with tasks, verifiers, the rollout runner, and the adversary. router is campaigns and routing. training is post-training. cli is the single evalkit command that drives it all. Type hints everywhere, pyright in strict mode, and a test beside every feature, 224 of them.

A sandbox for untrusted code

Running model-written code means running code you did not write. Each solution executes in a fresh python -I process in a throwaway temp directory, with a stripped environment and hard limits set through setrlimit for CPU seconds, address space, file size, open file descriptors, and no core dumps. A wall-clock timeout kills the entire process group, and captured output is capped. It is deliberately a resource and fault boundary, not a defense against a determined attacker, which is the right level for model-written solutions. The public SandboxEval work studies exactly where that line sits [2025]. The full code is on the methods page.

An adversary that cannot cheat

Static tests only catch the failures you thought of. The adversary is an agent whose job is to break a solver's solution. Given a task and a candidate, it proposes inputs it believes the candidate gets wrong, with a category for the failure mode such as off-by-one or edge case, and a hint for fixing it. The design question that makes or breaks this is how you stop the adversary from banking a bogus test that even a correct solution would fail. The answer is that the adversary only proposes and never decides. Each input is validated by running the reference solution to get the ground truth and the candidate on the same input. It becomes a real counterexample only when the reference succeeds and the candidate disagrees, and the expected value always comes from the reference [2023].

What actively trying to break it found

After each phase the code went through a multi-agent adversarial review, a hostile set of eyes over code that already passed its tests. It paid for itself twice. In the foundation it found that the budget cap never actually fired. The default guard was rebuilt on every call, so its spend reset to zero each time and the cap could only trip if set to zero. A verifier reproduced it. A tiny cap ran ten calls far over budget with no halt. The fix seeds the guard's spend from what is already in the database, so the cap holds across calls and restarts.

In the adversary, among thirteen confirmed bugs, it found that a task whose reference returns NaN would round-trip that value through JSON and bank it as expected. Because NaN never equals NaN, a solver identical to the reference would be flagged wrong and an unpassable test got banked. That is exactly the invariant the adversary is supposed to guarantee, violated. It was fixed by serializing with allow_nan=False plus a guard in the miner.

Campaigns and cost-aware routing

A campaign fans out K rollouts per task across many models. It is resumable, idempotent by a rollout key, so a run halted on the budget cap resumes without redoing paid work. Each completion is graded against both the clean and adversarial test sets, so one model call produces both a public and an adversarial pass@1. On top sits the router. Calibrate each model's quality and cost from a campaign, then send each task to the cheapest model whose calibrated quality clears a threshold, and if that model's answer fails the verifier, escalate to a stronger one. In the current data, routing this way cuts cost by roughly 75 percent against always reaching for the top model, because most tasks do not need it.

Why this beats a default agent or a raw API call

A fair question is why build a harness when you could point Claude Code or the Codex CLI at a task, or just call an API. The short answer is that those tools are built to finish one change for one developer, and they are genuinely good at it [2026] [2025]. They are not built to produce a reproducible score across many models, and they cannot hand you a reward you can safely train against. An interactive coding agent runs a loop of read, edit, run, and repeat, and the model itself decides when the work is done [2025]. That self-judgement is the exact thing you must not have when the model is the subject of measurement.

The harness separates the solver from the judge. The solver can be any model or any agent. The judge is fixed code that runs every candidate the same way, records exact cost, keeps the public and hidden split, mines new tests over time, and produces one number. This follows the principle in Anthropic's writeup on effective agents, which is to build the least agent the task needs and keep control of the loop yourself rather than handing it to the model [2024]. It is also how serious code benchmarks already work. SWE-agent pairs a model with a fixed interface and scores it on held tests it cannot see [2024]. A raw API call gives you none of this. You would have to build the sandbox, the test execution, the cost record, the hidden split, and the adversary yourself, which is what this is.

From evaluation to improvement

Everything above produces a trustworthy reward signal. The last move is to spend it. Fine-tune an open-weights model against these verifiers, with an explicit defense against the exact failure, reward hacking, that motivated the verifier-first stance in the first place. That is the post-training page.