Datasets
Build a Dataset from Traffic
Turn production traffic into datasets for evaluation.
The most useful datasets come from real production traffic. Inference platform lets you filter your captured inferences and save the results as a dataset, ready for evals.
You can create a dataset from traffic in two places: the Create Dataset button in the Datasets tab, or Save as Dataset in the Inference Viewer. Both follow the same flow.
The flow
Filter your traffic
Filter by model, task, provider, status code, or any tracked dimension until you have a representative slice of traffic.
Choose a dataset type
Choose eval as the dataset type.
Save as dataset
Name the dataset and save. It's immediately available for evals.
Getting clean samples
The quality of your dataset depends on how well you filter. A few tips:
- Filter by task to get samples for a specific objective rather than a mix of everything
- Errors are opt-in — only successful inferences (2xx) are included by default. Add error status codes to the filter only when you specifically want failure cases (e.g. to evaluate how models handle edge cases)
- Check the date range - a dataset pulled from a single day might not capture the full variety of inputs your app sees
What makes a good eval dataset
Eval datasets should be small, stable, and challenging. Pick examples that represent the hard cases — the ones where you're not sure the model will get it right. These become your benchmark, so don't change them often.