Auto Evals are rolling out gradually. If you do not see the Insights page in your dashboard, contact support to enable it for your team.
What you get
Each completed run appears on the Insights page (open your project, then Observability → Insights) with:- A verdict: keep your current model, or switch to a specific alternative.
- A quality score for your current model, from a blind grading of your real production responses.
- A leaderboard of candidate models with quality, similarity, cost, and latency for each.
- An estimate of what switching would save, based on your actual traffic volume.
- The worst-scoring samples, so you can inspect real failures instead of averages.
- Insights: written findings about failure modes and configuration problems in your traffic.
How a run works
A run covers the trailing seven days of traffic for one scope: a gateway task or a traced agent. It moves through three stages.1. Sampling
Catalyst draws about 100 individual assistant turns from the window. Sampling is recency-weighted, so newer conversations are more likely to be picked, and it is deterministic, so a rerun of the same configuration reproduces the same sample. Each conversation contributes at most 5 turns, spread across the early, middle, and late parts of the conversation rather than clustered in one place. Conversations with fewer than 2 usable turns are skipped.2. Scoring
Each sampled turn is replayed against every candidate model: the conversation up to that turn, with the same tools available, is sent to the candidate, which produces its own version of the response. An LLM judge then scores every response against two fixed rubrics:
Blind grading matters: judges shown two responses side by side tend to favor the incumbent. Grading each response in isolation removes that bias, and the baseline comparison happens afterward in the results.
3. Recommendation
Once all scores settle, an analysis agent reviews the scored results together with the sampled conversations. It writes the run summary, picks the model recommendation (optimizing across quality, cost, and latency, and allowed to conclude “keep the current model”), and records insights about problems it found in the traffic.Candidate models
Runs compare your traffic against a default set of frontier and open models from the catalog (currently 12, including GPT, Claude, Gemini, DeepSeek, Kimi, and Qwen models). You can exclude models you would never switch to. On the Insights page, open Eval models and toggle off any candidate. Exclusions apply per team, take effect immediately (including for runs already in progress), and can be turned back on at any time. Excluding every model turns off the model comparison for your team.Cost and limits
The judge and replay calls in the scoring stage bill as normal eval usage from your team credits, and count against your plan’s eval sample limits (rate limits). A full run is roughly 2,100 samples across all candidate models. If a run would exceed your remaining daily limit, that part of the run is skipped and the skip reason is recorded on the run. The sampling and analysis stages are included; you are not billed for the analysis agent.Reading a run
The Insights page always shows the latest run for the selected scope, and the run picker lets you open any previous run. For each run you can:- Compare candidates on the quality-versus-cost chart. Up and to the left is better.
- Check the similarity score before switching. A high-quality but low-similarity candidate behaves differently from your current model, so test it before routing traffic to it.
- Open the worst samples to see the conversations behind low scores.
- Review the methodology section for exactly what was sampled and judged, including anything that was skipped.