Prerequisites
A run is always scoped to one entity: a Gateway task or a traced agent. That scope is where the run samples its traffic, so you need one of these integrations first:
There is no minimum traffic requirement to start a run manually, but the sample can only be as representative as the traffic in the window you pick.
Where to find it
- Tasks: Gateway → Tasks, open a task, select the Compare Models tab.
- Agents: Agents, open an agent, select the Compare Models tab.
Start a run
Click Run Comparison to open the comparison dialog.
Current model and spend
The left panel shows the model you run today: its per-token pricing and your recent spend on this task across 1, 7, and 30 day windows. This is the baseline every candidate is compared against.Traffic window
- Traffic window: the window to sample from. Presets range from 1 hour to 30 days, and 30 days is both the default and the maximum. The panel shows how many turns are available in the window.
- Samples per model: how many turns to sample, up to 100 (the default). Fewer samples make the run cheaper and faster, at the cost of statistical confidence.
Model catalog
The catalog lists every callable candidate model with its quality score, per-token pricing, blended price, and projected savings against your current model. Models are grouped by how their blended price compares to yours, and the models recommended for your task (at or below your blended cost, best quality per dollar) come pre-selected. Check or uncheck models to set the candidate list. Your selection is saved per task or agent and restored on the next run; Reset to recommended returns to the default set. The footer shows how many models are selected and the blended price range of the selection against your current cost. If the run would not fit within your team’s remaining daily eval limit, the dialog warns you that some or all comparisons may be skipped. Click Run. The dialog closes and takes you straight to the new run’s report.While the run executes
The report shows an Analysis in progress card with a three-step pipeline. Results fill in as models finish scoring.
Per-model results appear below the pipeline as each model finishes: how many samples it completed, and any failures.
How a run works
Sampling
Catalyst draws up to your Samples per model count of individual assistant turns from the window. Sampling is recency weighted, so newer conversations are more likely to be picked, and deterministic, so the same configuration reproduces the same sample. Each conversation contributes at most 5 turns, spread across the early, middle, and late parts of the conversation. The pinned sample is saved as an eval dataset, so you can inspect exactly what was judged.Scoring
Each sampled turn is replayed against every candidate model: the conversation up to that turn, with the same tools available, is sent to the candidate, which produces its own version of the response. LLM judges then score every response against two fixed rubrics:
Blind grading matters: judges shown two responses side by side tend to favor the incumbent. Grading each response in isolation removes that bias. Each rubric uses a judge model pinned by the platform; the eval viewer shows which judge scored each rubric.
Recommendation
Once the scores settle, an analysis agent reviews the scored results together with the sampled conversations. It writes the run summary, picks the model recommendation (weighing quality, cost, and latency, and allowed to conclude “keep the current model”), and records insights about problems it found in the traffic.Candidate models
Runs compare your traffic against candidate models from the catalog (currently 17, including GPT, Claude, Gemini, Grok, DeepSeek, GLM, Kimi, Nemotron, and Qwen models). Use the checkboxes in the run dialog’s model catalog to set the candidate list per task or agent.Automatic runs
Catalyst can start your team’s first comparison automatically. Automatic runs are available on Growth and higher plans, and start once a task or agent accumulates enough recent traffic (about 1,000 requests in the trailing 7 days). An automatic run shows Automated as its source and an Included badge: its eval cost does not use your team credits. After the first automatic run, you start runs yourself with Run Comparison.Cost and limits
The judge and replay calls in the scoring stage bill as normal eval usage from your team credits, and count against your plan’s daily eval sample limits (rate limits). With the default 100 samples and all candidates enabled, a full run requests roughly 3,500 eval samples: every candidate plus your baseline is graded for quality, and every candidate is graded for similarity. To reduce a run’s cost, lower Samples per model or deselect candidate models you would not switch to. If a run would exceed your remaining daily limit, the affected comparisons are skipped and the skip reason is recorded on the run. The daily limit resets at midnight PT. The sampling and analysis stages are included; you are not billed for the analysis agent.Relationship to Direct Evals
Auto Evals run on the same eval infrastructure as Direct Evals you configure yourself. The difference is that Auto Evals choose the sample from your live traffic, use two fixed system rubrics, and run without setup. When you want to measure a quality dimension specific to your product, write your own rubric and run a Direct Eval on a curated dataset.Next steps
Interpret the results
Read the verdict, the leaderboard, and individual judged samples.
Train your own model
When the best off-the-shelf model is not enough, AutoTrainer distills one from your traffic.