Grade models on tests they cannot argue with.
gradientsmith measures what a language model can actually do, then uses those measurements to fine-tune an open-weights model that does more. Every score comes from a deterministic verifier. Code is executed against unit tests in a sandbox. Structured output is checked against a schema. There is no model grading another model. That one property is what makes the adversarial test mining, the cost-aware routing, and the reward-hacking defense trustworthy rather than decorative.
The problem it solves
Most comparisons of the form "which model is best" quietly rot for one reason. The scoring is not reproducible. If a benchmark uses a language model to grade answers, the grade drifts with the grader, and a model can learn to sound right rather than be right. Model judges are known to prefer the first answer they see, prefer longer answers, and prefer text that matches their own style [2023] [2024]. The moment you post-train against a fuzzy metric, that gap becomes a loophole and the model optimizes the metric instead of the task. That failure has a name, reward hacking, and it has been documented for years [2016].
gradientsmith starts from one constraint that is not up for negotiation. Every primary metric comes from a verifier that is deterministic and sandboxed. The same solution in gives the same verdict out, every time. Everything else is built on top of that.
evalkit. Think of evalkit as the wrench and gradientsmith as the workshop.The pipeline, end to end
Five stages feed one loop. Tasks carry hidden tests the solver never sees. Verifiers score deterministically. An adversary mines new failing cases and banks them. Campaigns run thousands of rollouts across many models and send each task to the cheapest one that is good enough. The same verifier verdict then becomes the reward that fine-tunes an open model.
Why not just use a default agent
A fair question is why build this at all when you could point Claude Code or the Codex CLI at a task, or just call an API. Those tools are built to help one developer finish one change, and they are very good at it [2026] [2025]. They are not built to produce a reproducible number across many models, and they cannot give you a reward you can safely train against. A default agent decides for itself when it is done. gradientsmith never lets the thing being measured decide whether it passed. The how it works page lays out the full comparison and shows the eval tasks in detail.
What it produces
A campaign yields a picture of cost against quality for every model at once. These figures are from the current snapshot, campaign frontier-calibration across 6 models.
About
gradientsmith is built by Stanley Jacob, an AI and ML systems engineer who works on embodied AI and robotics, multimodal models, reinforcement learning, and inference infrastructure. He studied electrical engineering at UT Austin and computer science at Stanford. Past work includes a fine-tuned coder model trained with fully-sharded data parallelism, a diffusion policy for robot control, and motion planning for GPS-denied flight. This project is the evaluation and post-training half of that interest, worked out in full. There is a longer writeup and the rest of his work at stanleyjacob.dev.