Glossary

Term	Definition
Task	A user-defined objective that groups LLM calls (e.g., "summarize document," "classify ticket"). Tasks persist even as the implementation changes.
Rubric	A plain English description of how to judge model output on a quality dimension. Scored numerically by an LLM judge.
Recipe	A pre-configured training setup including base model, training parameters, and compute config.
Dataset	A curated set of inference samples used for evaluation or training.
Eval dataset	A dataset used to measure model quality. Should remain stable over time. Must not overlap with training data.
Training dataset	A dataset the model learns from during fine-tuning. Evolves as you iterate on data quality.
Inference	A single LLM request-response pair captured by the platform.
Deployment	A trained model running on a dedicated GPU, accessible via an OpenAI-compatible API.
Mid-training eval	A periodic evaluation run during training that scores model checkpoints against the rubric.
LLM-as-a-judge	The evaluation mechanism where an LLM scores model outputs against rubric criteria.
TTFT	Time to first token. Measures streaming responsiveness.
Overfitting	When a model memorizes training data instead of learning generalizable patterns. Detected by degrading eval scores.
Distillation	Training a smaller model to replicate the quality of a larger model, reducing cost and latency.
Agent	An entity that groups runs sharing a stable `agentId`/`agentName` into one workspace in the Agents dashboard, where its metrics, sessions, trajectories, signals, and HALO analysis live.
Trajectory	The user-facing name for a trace in the dashboard. The URL slug and CLI still say `traces`.
Signal	A plain-language classifier you define once; an LLM judge labels each span, trace, or session into metrics you can chart, filter, and alert on.
HALO	"Hierarchical Agent Loop Optimization." The open-source agent-loop engine that reads traces, finds systemic failure modes across runs, and writes concrete fixes cited to specific traces.
AutoEval	An automated comparison that samples recent traffic, replays it against candidate models, scores outputs with LLM judges, and recommends a better fit with supporting evidence.