Why Evals Are the New Integration Tests
A scaffold for treating evals as part of the engineering contract rather than as a research artifact.
Thesis
For AI features, evals play the role integration tests played for traditional backend systems: they protect the contract between components and expected behavior.
Context
- Why unit tests are not enough for model-mediated behavior.
- How prompts, retrieval, tools, and policies interact.
- What changes when outputs are probabilistic.
Eval Design
- Define expected behavior in product terms.
- Use realistic examples and edge cases.
- Separate correctness, safety, tone, and operational constraints.
- Version datasets and evaluation criteria.
Engineering Workflow
- Run evals before prompt, retrieval, model, or tool changes.
- Track regressions over time.
- Review failures as product and engineering signals.
Tradeoffs
- Small fast eval suites versus deep review suites.
- Automated scoring versus human judgment.
- Preventing regressions versus encouraging iteration.
Conclusion
Evals are not optional AI research hygiene; they are production engineering infrastructure.