Check out the newest way to compare different models for a task/agent harness: AutoEvals

Datasets

Build a Dataset from Traffic

Turn production traffic into datasets for evaluation.

The most useful datasets come from real production traffic. Inference platform lets you filter your captured inferences and save the results as a dataset, ready for evals.

You can create a dataset from traffic in two places: the Create Dataset button in the Datasets tab, or Save as Dataset in the Inference Viewer. Both follow the same flow.

The flow

Filter your traffic

Filter by model, task, provider, status code, or any tracked dimension until you have a representative slice of traffic.

Choose a dataset type

Choose eval as the dataset type.

Save as dataset

Name the dataset and save. It's immediately available for evals.

Getting clean samples

The quality of your dataset depends on how well you filter. A few tips:

  • Filter by task to get samples for a specific objective rather than a mix of everything
  • Errors are opt-in — only successful inferences (2xx) are included by default. Add error status codes to the filter only when you specifically want failure cases (e.g. to evaluate how models handle edge cases)
  • Check the date range - a dataset pulled from a single day might not capture the full variety of inputs your app sees

What makes a good eval dataset

Eval datasets should be small, stable, and challenging. Pick examples that represent the hard cases — the ones where you're not sure the model will get it right. These become your benchmark, so don't change them often.

Next steps

On this page