Check out the newest way to compare different models for a task/agent harness: AutoEvals

Get Started

Inference platform workflow

How Tracing and Gateway fit together across the Inference platform.

Inference platform is built around two products.

  • Tracing is for agent improvement. The tracing SDKs capture the full execution of your agents and AI apps: LLM calls, tool calls, framework steps, and any custom spans you wrap. Halo, our open-source agent-loop optimizer, reads those traces and writes up what to fix in prompts, tools, and the harness itself.
  • Gateway is for inference observability. Gateway sits between your app and your LLM provider, recording every request. From that recorded traffic, you can build datasets and run evals. You can also deploy models on dedicated GPUs.

Tracing and Gateway stand alone. Many teams start with one and add the other later. They also compose: when your agent calls LLMs through Gateway, the trace spans and the gateway records line up against the same requests.

The platform also serves models through an OpenAI-compatible API: open-source models, popular closed-source models, and Inference.net models (like Schematron for structured data extraction).


Tracing

Capture full traces of your agents and AI apps. The Inference Tracing SDKs collect LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Then Halo (our open-source agent-loop optimizer) reads your traces and surfaces concrete things to improve in your prompts, tools, and agent harness.

What you'll do:

  • Capture your first trace by installing the tracing SDK and pointing it at the Inference platform
  • Analyze your traces in the Traces and Agents dashboards, then run Halo to find what to fix
  • Instrument providers and frameworks like OpenAI, Anthropic, LangChain, LangGraph, Vercel AI SDK, OpenAI Agents, LiveKit, Pi Agent, legacy PI AI, Pydantic AI, and more
  • Add stable agent identity so the Agents dashboard groups runs correctly

Outcome: Deep visibility into how your agents behave end to end, plus an automated reviewer that points you to the highest-impact fixes.

Get started with Tracing

Install the SDK and capture your first trace.


Gateway

Record and analyze your production LLM traffic. Inference Gateway sits between your app and your LLM provider, capturing every request, response, cost, and latency metric with less than 10ms of overhead. Keep using any provider or model — Gateway is transparent.

What you'll do:

Outcome: Full visibility into how your AI features perform in production — broken down by model, task, and provider.

Get started with Gateway

Set up Gateway and start capturing LLM traffic.


Datasets

Curate collections of LLM inputs and outputs for evaluation. Datasets can come from your live production traffic captured through Gateway, or from files you upload directly.

What you'll do:

Outcome: Clean, representative datasets scoped to specific tasks — ready to power evals.

Get started with Datasets

Build or upload your first dataset.


Eval

Measure model quality on your live traffic with AutoEvals. The platform samples recent production requests, replays each against a set of candidate models, scores every output with LLM judges, and recommends the best model with the evidence behind it.

What you'll do:

Outcome: A data-driven answer to which model is best for your workload, and a validated quality signal for your task.

Get started with Eval

Compare models on your traffic with AutoEvals.


Deploy

Run an open-source or Hugging Face model on a dedicated GPU with an OpenAI-compatible API. The API uses the same base URL and API key as the rest of the Inference platform, so switching from a serverless model to your deployment is a one-line code change.

What you'll do:

Outcome: A production endpoint serving your model. Keep it routed through Gateway to observe it and run evals against it.

Get started with Deploy

Ship your model to a dedicated GPU.


Pick your starting point

On this page