Evals and Measurement

Replacing vibes with measurement

Evals are structured tests for AI systems: defined inputs, expected outcomes, and a scoring rule. They replace "it looked fine in the demo" with numbers. In practice: an eval suite for a summarizer scores a fixed set of documents on every change.

There are two different kinds, and they answer different questions. Benchmark vs production evals: benchmark evals score a model on a standard public dataset and support model comparisons; production evals score your actual workflow on your real traffic and support deployment decisions. In practice: a model can top a benchmark and still underperform in your workflow.

Ground truth is the verified correct answer a test case is scored against, usually produced by a person or a trusted system. Without it, an eval measures agreement, not correctness. In practice: labeled historical tickets are the ground truth for a ticket classifier.

Comparing versions honestly

A/B testing runs two versions on comparable traffic and compares observed outcomes. Statistical honesty means respecting what a sample size can support: with small samples, a comparison shows a difference, not proof. In practice: honest results are reported as observed differences with their sample size, never as guarantees.

One trap worth naming: if a system logs its own prediction as the actual outcome, accuracy becomes perfect by definition and meaningless in practice. Predicted and actual must come from independent sources.

The Cromus Lens

The Reality Check Loop is a production eval system: Predict, Execute, Log, Learn, with adoption statuses and confidence bounds computed from real run history. Champion and Challenger experiments compare skill versions on observed data with hard guardrails: never auto-assign, never auto-promote, no p-values, and verdicts worded as observed differences rather than guarantees. For uploaded runs, predicted values are never copied into actuals, because that would inflate accuracy by definition.

Where to go deeper: the Operating at Scale track walks the full loop hands-on.