What this is built on

The foundational LLM-as-a-judge study. It reports over 80 percent agreement with human preferences and also documents the judge failure modes: position bias, verbosity bias, and self-enhancement bias.

Over 150,000 evaluations across 15 judges. Position bias is systematic rather than random and grows with the quality gap between candidate answers.

Argues for reporting eval numbers with error bars, analyzing paired differences between models, and sampling several answers per question to cut scoring variance. The basis for the bootstrap intervals used here.

An industry account of replacing slow human annotation with a judge that is continuously calibrated against human labels, and the discipline that calibration requires.

A working eval harness for a support agent, built from rubrics, a transcript builder, a repeatable simulator, and a calibrated judge. A good model for the infrastructure around a verifier.

Shows that surface metrics like BLEU and CodeBLEU miss functional errors that execution catches. Direct evidence that running the code beats comparing its text.

The early, careful statement of reward hacking and specification gaming, where an agent satisfies the letter of a reward while missing its intent.

Shows models trained on gameable rewards generalizing to edit their own reward. Motivation for keeping hidden tests out of the reward and watching for divergence.

Introduces GRPO, a critic-free policy optimization that estimates advantage from the average reward of a sampled group. The GRPO implemented here follows this.

Demonstrates that rule-based verifiable rewards alone, without a learned reward model, can drive large reasoning gains. Strong evidence for the verifier-first thesis.

Coins Reinforcement Learning with Verifiable Rewards and open-sources the full recipe over Llama 3.1. The closest published blueprint for a verifier-first post-training stack.

Compares process supervision against outcome supervision and releases PRM800K. Relevant to how much of the verdict signal to expose during training.

The Codex and HumanEval paper. Defines the pass at k metric and evaluates by executing generated functions against unit tests in a sandbox.

Introduces MBPP, a set of short Python problems each with test cases. A model of the task shape used in the seed dataset here.

Grades a patch by running the repository's own test suite before and after. Execution-based evaluation at the scale of real projects.

Continuously collects fresh problems to avoid training-set contamination, a concern this platform answers with a versioned hidden-test bank.

Tests harder, library-heavy tasks with rich test cases, a direction the task families here can grow toward.

On the risks of executing model-written code and what a hardened sandbox needs. Frames where the resource-limited sandbox here sits and where container isolation becomes necessary.

The base and instruct family used here for routing baselines and as a post-training target, with dense models from 0.5B to 72B and long-context support.

The coder line, the default post-training target here, trained heavily on code with strong results on execution benchmarks.

A dense architecture family up to 405B with a detailed account of the post-training recipe, including rejection sampling.

A large mixture-of-experts model, 671B total parameters with 37B active per token. The reference for the sparse architecture described on the Models page.

A sparse mixture-of-experts with 8 experts per layer and 2 active per token, a clear worked example of the routing that a harness must account for.

Keep the samples that reach the right answer, fine-tune on them, repeat. The rejection-sampling loop here is a verifier-gated version of this idea.

Alternates a grow step that samples data with an improve step that fine-tunes on the best samples, a framing for the campaign-then-train loop.

The distinction between fixed workflows and open-ended agents, and when each is appropriate. The eval harness here is a workflow, not an agent.

An interactive agent harness for coding in a real repository. Optimized for finishing one task with a human in the loop, not for identical repeated scoring.

A terminal coding agent with its own sandbox and tool loop. Useful to contrast the agent loop with a deterministic eval loop.

A description of the agent loop that these tools run, which is exactly the machinery an eval harness strips out to make scoring reproducible.

Shows how much the interface between the model and the computer matters. The verifier here is a narrow, fixed interface by design.