Check out the newest way to compare different models for a task/agent harness: AutoEvals

Datasets

Datasets

Curate datasets from production traffic or your own files for evals.

Datasets are collections of LLM inputs and outputs used for evaluation. They can come from two places: your live production traffic captured through Gateway, or files you upload directly.

Evals depend on good data. They need representative examples to measure model quality, and datasets are where those examples start.

Eval datasets

An eval dataset measures model quality against a rubric. Keep it stable: a fixed set of challenging examples that act as your benchmark.

Key concepts

ConceptDescription
Build from trafficFilter your captured production inferences and save them as a dataset. The best datasets come from real usage.
UploadBring your own JSONL files when you have curated data or are migrating from another platform.
Dataset formatThe schema your data needs to follow. See Dataset Formats for supported fields and validation rules.
Task tagsUse task tags when building from traffic to filter by objective. This gives you clean, focused samples instead of mixed traffic.

Tips for good datasets

  • Diverse data covers the edge cases. If your samples all look alike, the eval will not show how a model handles unusual inputs.
  • Stable eval data gives you a consistent benchmark. Don't change your eval dataset frequently, it's the measuring stick.
  • Start with production traffic when possible. Real user inputs reflect the actual distribution of requests your model will see, and they're harder to fake than synthetic data.
  • Use task tags to filter by objective before saving a dataset. A dataset scoped to a single task is almost always more useful than one built from mixed traffic.

Next steps

On this page