Check out the newest way to compare different models for a task/agent harness: AutoEvals

Platform

Glossary

Quick-reference definitions for Inference platform concepts and terminology.

TermDefinition
TaskA user-defined objective that groups LLM calls (e.g., "summarize document," "classify ticket"). Tasks persist even as the implementation changes.
RubricA plain English description of how to judge model output on a quality dimension. Scored numerically by an LLM judge.
DatasetA curated set of inference samples used for evaluation.
Eval datasetA dataset used to measure model quality. Should remain stable over time.
InferenceA single LLM request-response pair captured by the platform.
DeploymentA model running on a dedicated GPU, accessible via an OpenAI-compatible API.
LLM-as-a-judgeThe evaluation mechanism where an LLM scores model outputs against rubric criteria.
TTFTTime to first token. Measures streaming responsiveness.
AgentAn entity that groups runs sharing a stable agentId/agentName into one workspace in the Agents dashboard, where its metrics, sessions, trajectories, signals, and HALO analysis live.
TrajectoryThe user-facing name for a trace in the dashboard. The URL slug and CLI still say traces.
SignalA plain-language classifier you define once; an LLM judge labels each span, trace, or session into metrics you can chart, filter, and alert on.
HALO"Hierarchical Agent Loop Optimization." The open-source agent-loop engine that reads traces, finds systemic failure modes across runs, and writes concrete fixes cited to specific traces.
AutoEvalAn automated comparison that samples recent traffic, replays it against candidate models, scores outputs with LLM judges, and recommends a better fit with supporting evidence.