Eval

Model providers ship updates constantly and prompts drift. Eval gives you a repeatable way to measure model quality before and after every change, and to compare model options for a given task. Catalyst offers two ways to run evals:

​

AutoEvals: compare models on your traffic

AutoEvals answer one question: is there a better model for this workload than the one you run today? A run samples your recent production traffic from a Gateway task or a traced agent, replays each sample against a set of candidate models, scores every output with LLM judges, and publishes a model recommendation with the evidence behind it. You do not build a dataset or write a rubric. Start a run from the Compare Models tab on any task or agent, and read the results down to the individual judged sample.

​

Direct Evals: define your own benchmark

Direct Evals put you in control of all three inputs: the dataset, the rubric, and the models. Use them when you need to measure something the fixed AutoEval rubrics do not cover, or when you want a stable benchmark that does not change as your traffic does. If you’re planning to fine-tune a custom model, run Direct Evals first. A validated rubric and eval dataset are prerequisites for training. They’re the measuring stick that determines when the model has learned enough, or when to stop to prevent overfitting.

​

How it works

1. Define a rubric - describe what “good” looks like in plain English
2. Pick a dataset - samples from captured traffic or uploaded JSONL
3. Select models - the candidates you want to compare
4. Run the eval - each sample goes through each model, and an LLM judge scores every output
5. Compare results - side-by-side scores show which model wins

​

Key concepts

​

Next steps