Check out the newest way to compare different models for a task/agent harness: AutoEvals

Deploy

Deploy

Dedicated GPU infrastructure for serving models via an OpenAI-compatible API.

Deploy gives you a dedicated GPU serving an open-source catalog model or a Hugging Face model you registered. The API is OpenAI-compatible, so switching from a serverless model to your deployment is a one-line code change.

Key concepts

ConceptDescription
Dedicated GPUYour model runs on its own GPU. No shared infrastructure, no noisy neighbors. Instance type and count are chosen at deploy time from the model's allowed instance types; the default is 1×H100.
OpenAI-compatible APISame base URL, same API key, just swap the model parameter. Structured outputs, function calling, and all standard API features work the same way.
Project-scopedDeployments are created within a project, and access is scoped to that project. Sibling projects in the same team cannot call them. The model path is your team/org slug followed by the deployment name you choose (e.g. acme-corp/my-model).
MonitoringRoute deployment traffic through Gateway to observe production performance, and run evals to catch regressions.

What you can deploy

  • Open source catalog models with weights available
  • Hugging Face models you've registered as custom models
  • Served via an OpenAI-compatible API (chat completions endpoint)
  • Same base URL and API key as the rest of the Inference API

Next steps

On this page