Skip to main content
Auto Evals answer one question: is there a better model for your workload than the one you run today? Catalyst samples your recent production traffic, replays each sample against a set of candidate models, scores every output with LLM judges, and recommends a model with supporting evidence. You do not build a dataset, write a rubric, or configure a judge. In the dashboard, Auto Evals live on the Compare Models tab of each task and agent. This guide takes you from integration to your first completed comparison.

​
Before you start

Auto Evals run on your real traffic, so Catalyst needs to see that traffic first. Integrate one of these: Either path works. A comparison is always scoped to one task or one agent, and it samples that scope’s traffic from a time window you pick. The more traffic in the window, the more representative the sample, so let your app run normally for a while before you compare.

​
Run your first comparison

1

Open the Compare Models tab

For Gateway traffic: go to Gateway → Tasks, open your task, and select the Compare Models tab. If your requests do not set a task ID, they are grouped under the Default task.For traced apps: go to Agents, open your agent, and select the Compare Models tab.
A task detail page with the Compare Models tab selected, showing the empty state card 'No model comparisons yet' and the Run Comparison button
2

Start a run

Click Run Comparison. The comparison dialog opens with sensible defaults: a 30 day traffic window, 100 samples per model, and the models recommended for your task pre-selected in the model catalog. For a first run, keep the defaults and click Run.
The comparison dialog, showing your current model's pricing, the Traffic window and Samples per model selectors, and the model catalog with recommended models pre-selected
The dialog also shows your recent spend on this task and how each candidate’s blended price compares to your current model, and warns you if the run would exceed your team’s daily eval limit.
3

Watch the run execute

The run report opens immediately and fills in as the run progresses through three stages:
  1. Sampling traffic - Catalyst pins a deterministic sample of your requests.
  2. Scoring candidate models - every candidate re-runs the samples and is judged.
  3. Generating insights - the scored results are summarized into a verdict and findings.
Results fill in as each model finishes scoring. You can cancel a run at any time.
An in-progress run showing the Analysis in progress card with the Sampling traffic, Scoring candidate models, and Generating insights steps
4

Read the outputs

When the run completes, the report leads with a verdict: switch to a specific model, or keep your current one. Below it you get the estimated monthly savings, key findings from the judged samples, a quality versus cost chart, and a leaderboard that ranks every candidate against your current model on quality, similarity, latency, and price.
A completed run report showing the RECOMMENDED MODEL hero with the score delta versus the current model, monthly savings, and the Key findings list
From the leaderboard, open the Sample Viewer to read individual model outputs side by side with the judge’s reasoning.

​
What you just learned

  • Auto Evals measure candidate models on your own production traffic, not a public benchmark
  • A run needs no dataset or rubric: Catalyst samples your traffic and scores it with two fixed rubrics, quality and similarity
  • The result is a concrete recommendation with evidence you can inspect down to the individual sample

​
Next steps

Run an Auto Eval

The full reference for starting runs: configuration, sampling, scoring, and limits.

Interpret the results

Read the run report and drill into judged samples.

Write your own rubric

Measure a quality dimension specific to your product with a Direct Eval.

Run a Direct Eval

Compare models on a curated dataset with your own rubric.