Datasets
Datasets
Curate datasets from production traffic or your own files for evals.
Datasets are collections of LLM inputs and outputs used for evaluation. They can come from two places: your live production traffic captured through Gateway, or files you upload directly.
Evals depend on good data. They need representative examples to measure model quality, and datasets are where those examples start.
Eval datasets
An eval dataset measures model quality against a rubric. Keep it stable: a fixed set of challenging examples that act as your benchmark.
Key concepts
| Concept | Description |
|---|---|
| Build from traffic | Filter your captured production inferences and save them as a dataset. The best datasets come from real usage. |
| Upload | Bring your own JSONL files when you have curated data or are migrating from another platform. |
| Dataset format | The schema your data needs to follow. See Dataset Formats for supported fields and validation rules. |
| Task tags | Use task tags when building from traffic to filter by objective. This gives you clean, focused samples instead of mixed traffic. |
Tips for good datasets
- Diverse data covers the edge cases. If your samples all look alike, the eval will not show how a model handles unusual inputs.
- Stable eval data gives you a consistent benchmark. Don't change your eval dataset frequently, it's the measuring stick.
- Start with production traffic when possible. Real user inputs reflect the actual distribution of requests your model will see, and they're harder to fake than synthetic data.
- Use task tags to filter by objective before saving a dataset. A dataset scoped to a single task is almost always more useful than one built from mixed traffic.