Skip to main content
Model providers ship updates constantly and prompts drift. Eval gives you a repeatable way to measure model quality before and after every change. They can also help compare model options for a given task. If you’re planning to fine-tune a custom model, run evals first. A validated rubric and eval dataset are prerequisites for training. They’re the measuring stick that determines when the model has learned enough, or when to stop to prevent overfitting.

​
How it works

  1. Define a rubric - describe what “good” looks like in plain English
  2. Pick a dataset - samples from captured traffic or uploaded JSONL
  3. Select models - the candidates you want to compare
  4. Run the eval - each sample goes through each model, and an LLM judge scores every output
  5. Compare results - side-by-side scores show which model wins

​
Key concepts

​
Next steps

Writing rubrics

Create rubrics from templates, AI generation, or plain English.

Run a model comparison

Compare models head to head on your data.

How LLM-as-a-Judge works

Understand the evaluation mechanism.

Read the results

Interpret scores and make decisions.