# Anthropic SDK (https://docs.inference.net/api/anthropic-sdk)
The Inference.net API supports the Anthropic Messages format at `POST https://api.inference.net/v1/messages`. This means you can use the Anthropic SDK to call any serverless model hosted on Inference.net, such as `glm-5.2`. Only your Inference API key is needed.
Requests are converted to and from the model's native format for you. Both non-streaming and streaming responses are supported.
This page covers calling Inference-hosted models with the Anthropic SDK. To route requests to Anthropic's own Claude models using your Anthropic API key, see the [Anthropic integration](/integrations/model-providers/anthropic) instead.
## Authentication [#authentication]
The Anthropic SDK sends its `apiKey` as the `x-api-key` header, but the Inference API expects your key as a Bearer token in the `Authorization` header. Pass your Inference API key as `authToken` (TypeScript) or `auth_token` (Python), which sends `Authorization: Bearer `. In TypeScript, also set `apiKey: null` so the SDK does not look for an Anthropic key.
```bash
export INFERENCE_API_KEY=
```
## Non-Streaming [#non-streaming]
```typescript TypeScript
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "https://api.inference.net",
apiKey: null,
authToken: process.env.INFERENCE_API_KEY,
});
const message = await client.messages.create({
model: "glm-5.2",
max_tokens: 1024,
messages: [{ role: "user", content: "What is the meaning of life?" }],
});
console.log(message.content);
```
```python Python
import os
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.inference.net",
auth_token=os.environ["INFERENCE_API_KEY"],
)
message = client.messages.create(
model="glm-5.2",
max_tokens=1024,
messages=[{"role": "user", "content": "What is the meaning of life?"}],
)
print(message.content)
```
```bash cURL
curl https://api.inference.net/v1/messages \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{
"model": "glm-5.2",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "What is the meaning of life?"}]
}'
```
## Streaming [#streaming]
```typescript TypeScript
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "https://api.inference.net",
apiKey: null,
authToken: process.env.INFERENCE_API_KEY,
});
const stream = client.messages.stream({
model: "glm-5.2",
max_tokens: 1024,
messages: [{ role: "user", content: "Count from 1 to 5." }],
});
stream.on("text", (text) => {
process.stdout.write(text);
});
const message = await stream.finalMessage();
console.log("\n", message.usage);
```
```python Python
import os
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.inference.net",
auth_token=os.environ["INFERENCE_API_KEY"],
)
with client.messages.stream(
model="glm-5.2",
max_tokens=1024,
messages=[{"role": "user", "content": "Count from 1 to 5."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
```bash cURL
curl -N https://api.inference.net/v1/messages \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{
"model": "glm-5.2",
"max_tokens": 1024,
"stream": true,
"messages": [{"role": "user", "content": "Count from 1 to 5."}]
}'
```
The Messages format requires `max_tokens`. Reasoning models such as `glm-5.2` spend part of the token budget on reasoning before producing text, so set `max_tokens` high enough for both. If the budget runs out during reasoning, the response can come back with empty content.
The same code works for every serverless model. Set `model` to the model id you want. Browse available models at [inference.net/models](https://inference.net/models).
# API Quickstart (https://docs.inference.net/api/api-quickstart)
The Inference.net API is OpenAI-compatible, so you can use the OpenAI SDK or plain HTTP to make requests. There are three ways to use it:
1. **Call a model serverless**: call open-source models and popular closed-source models (GPT, Claude, Gemini) with just your Inference API key. Usage is billed per token to your credit balance.
2. **Proxy through Inference Gateway**: route requests to any provider (OpenAI, Anthropic, etc.) through Inference Gateway with your own provider API key.
3. **Call your custom model**: hit a model you've fine-tuned and deployed on the platform.
All three paths go through Inference Gateway, so you get the same metrics, cost tracking, and eval-readiness whichever one you use.
## Get an API Key [#get-an-api-key]
Visit [inference.net](https://inference.net) and create an account.
On the dashboard, go to the **API Keys** tab in the left sidebar. Create a new key or use the default key.
```bash
export INFERENCE_API_KEY=
```
***
## 1. Call a Model Serverless [#1-call-a-model-serverless]
Call models with just your Inference API key. No provider API key is needed. This works for two kinds of models:
* **Open-source models** hosted on Inference.net, such as `glm-5.2`.
* **Popular closed-source models**, such as `claude-haiku-4-5`, `gpt-5-mini`, and `gemini-3.5-flash`. Inference.net routes the request to the provider for you and bills the usage per token to your credit balance.
Browse available models at [inference.net/models](https://inference.net/models), or list them with `GET https://api.inference.net/v1/models`.
Prefer the Anthropic SDK? The API also supports the Anthropic Messages format. See [Anthropic SDK](/api/anthropic-sdk).
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "glm-5.2",
messages: [{ role: "user", content: "What is the meaning of life?" }],
stream: true,
});
for await (const chunk of response) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="glm-5.2",
messages=[{"role": "user", "content": "What is the meaning of life?"}],
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="", flush=True)
```
```rust Rust
use serde_json::{json, Value};
#[tokio::main]
async fn main() -> Result<(), Box> {
let api_key = std::env::var("INFERENCE_API_KEY")?;
let response: Value = reqwest::Client::new()
.post("https://api.inference.net/v1/chat/completions")
.bearer_auth(api_key)
.json(&json!({
"model": "glm-5.2",
"messages": [
{"role": "user", "content": "What is the meaning of life?"}
]
}))
.send()
.await?
.json()
.await?;
println!(
"{}",
response["choices"][0]["message"]["content"]
.as_str()
.unwrap_or_default()
);
Ok(())
}
```
```go Go
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "glm-5.2",
"messages": []map[string]string{
{"role": "user", "content": "What is the meaning of life?"},
},
})
req, _ := http.NewRequest("POST", "https://api.inference.net/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFERENCE_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
var result struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
panic(err)
}
fmt.Println(result.Choices[0].Message.Content)
}
```
```bash cURL
curl -N https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "glm-5.2",
"messages": [
{"role": "user", "content": "What is the meaning of life?"}
],
"stream": true
}'
```
The same code works for every serverless model. Set `model` to the model id you want, for example `claude-haiku-4-5`. This includes our purpose-built [Schematron](/workhorse-models/schematron) models for structured data extraction.
***
## 2. Proxy Through Inference Gateway [#2-proxy-through-inference-gateway]
Route requests to any LLM provider (OpenAI, Anthropic, Groq, etc.) through Inference Gateway. You keep your existing provider API key, and the provider bills you directly. The gateway adds observability, cost tracking, and eval-readiness with roughly 10ms of added latency.
Use this path when you want a model that is not in the serverless catalog, or when you want usage billed to your own provider account. The captured metrics are the same as for serverless calls.
Your Inference project API key authenticates with the gateway. Your provider API key is forwarded to the provider via the `x-inference-provider-api-key` header.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "What is the meaning of life?" }],
});
console.log(response.choices[0].message.content);
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "What is the meaning of life?"}],
)
print(response.choices[0].message.content)
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "x-inference-provider: openai" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4.1",
"messages": [
{"role": "user", "content": "What is the meaning of life?"}
]
}'
```
For detailed setup guides per provider (Anthropic, Groq, Cerebras, OpenRouter, and more), see the [Integrations](/integrations/overview) docs.
***
## 3. Call Your Custom Model [#3-call-your-custom-model]
Hit a model you've fine-tuned and deployed on Inference.net. The model path is your team slug followed by the deployment name, shown on your deployment's detail page in the dashboard.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "your-team/your-model",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="your-team/your-model",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "your-team/your-model",
"messages": [
{"role": "user", "content": "Hello, world!"}
]
}'
```
Learn more about deploying models in the [Deploy](/platform/deploy/overview) docs.
***
## Headers Reference [#headers-reference]
| Header | Required | Description |
| ------------------------------ | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer ` — authenticates the request. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `Content-Type` | Yes | Must be `application/json`. |
| `x-inference-provider` | Proxy only | Routes the request to the correct provider: `openai`, `anthropic`, `cerebras`, `vertex-ai`, `gemini`, etc. For providers without a dedicated value that require it (for example Groq or Together AI), route by base URL instead with `x-inference-provider-url`. |
| `x-inference-provider-api-key` | Proxy only | Your provider's API key. The gateway forwards it downstream. For Anthropic's native SDK, use `x-api-key` instead. |
| `x-inference-provider-url` | No | Routes to any OpenAI-compatible provider by base URL, even if it doesn't have a dedicated integration. |
| `x-inference-environment` | No | Tags requests with an environment label, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics in the dashboard. |
| `x-inference-metadata-*` | No | Attach arbitrary metadata to a request. The prefix is stripped to form the key — e.g., `x-inference-metadata-chat-id: abc123` stores `chat-id: abc123`. You can filter inferences and create datasets based on these keys in the dashboard. |
## Supported Request Parameters [#supported-request-parameters]
The API supports the standard OpenAI chat completions parameters:
| Parameter | Type | Description |
| ------------------- | --------- | ---------------------------------------------------------------------------------------------------- |
| `model` | `string` | The model to use. |
| `messages` | `array` | The conversation messages. |
| `stream` | `boolean` | Whether to stream the response. |
| `max_tokens` | `integer` | Maximum number of tokens to generate. |
| `temperature` | `number` | Sampling temperature (0–2). |
| `top_p` | `number` | Nucleus sampling threshold. |
| `frequency_penalty` | `number` | Penalizes repeated tokens based on frequency. |
| `presence_penalty` | `number` | Penalizes tokens based on whether they've appeared. |
| `response_format` | `object` | Set to `{"type": "json_object"}` or a JSON schema for [structured outputs](/api/structured-outputs). |
| `tools` | `array` | Tool/function definitions for [function calling](/api/function-calling). |
Need a parameter that isn't listed here? [Contact us](mailto:support@inference.net) and we'll add it.
## Next Steps [#next-steps]
Set up Inference platform with OpenAI, Anthropic, Groq, and other providers.
Get typed JSON responses from your API calls.
Process up to 50,000 requests in a single batch job.
Explore all models available on Inference.net.
# Batch API (https://docs.inference.net/api/async-inference/batch-api)
Learn how to use our OpenAI-compatible Batch API to send asynchronous groups of inference requests to Inference.net, with nearly unlimited rate limits and fast completion times. The service is ideal for processing a large number of jobs that don't require immediate responses.
Batch API is currently compatible with all the [models](https://inference.net/models) we offer.
Use [https://batch.inference.net/v1](https://batch.inference.net/v1) for all Batch API requests (including Files and Batches). Do not use [https://api.inference.net/v1](https://api.inference.net/v1) for batch jobs.
## Overview [#overview]
While some uses require you to send synchronous requests, there are many cases where requests do not need an immediate response or rate limits prevent you from executing a large number of queries quickly. Batch processing jobs are often helpful in use cases like:
1. Extracting structured data from a large number of documents.
2. Generating synthetic data for training.
3. Translating a large number of documents into other languages.
4. Summarizing a large number of customer interactions.
Inference.net's Batch API offers a straightforward set of endpoints that allow you to upload a batch of requests, kick off a batch processing job, query for the status of the batch, and eventually retrieve the collected results when the batch is complete.
Compared to using standard endpoints directly, Batch API has:
1. **Higher rate limits:** Substantially more headroom compared to the [synchronous APIs](/api/rate-limits).
2. **Fast completion times:** Each batch completes within 24 hours (and often much more quickly).
## Getting Started [#getting-started]
You'll need an Inference.net account and API key to use the Batch API. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net's Batch API using the OpenAI SDK, set the base URL to `https://batch.inference.net/v1`.
In this example, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://batch.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://batch.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL
export INFERENCE_API_KEY=
# All Batch API requests use https://batch.inference.net/v1
```
## Running A Batch Processing Job [#running-a-batch-processing-job]
### 1. Preparing Your Batch File [#1-preparing-your-batch-file]
Prepare a `.jsonl` file where each line is a separate JSON object that represents an individual request.
Each JSON object must be on a single line and cannot contain any line breaks.
Each JSON object must include the following fields:
* `custom_id`: A unique identifier for the request. This is used to reference the request's results after completion. It must be unique for each request in the file.
* `method`: The HTTP method to use for the request. Currently, only `POST` is supported.
* `url`: The URL to send the request to. Currently, only `/v1/chat/completions` and `/v1/completions` are supported.
* `body`: The request body, which contains the input for the inference request. The parameters in each line's `body` field are the same as the parameters for the underlying endpoint specified by the `url` field. See this [example](/quickstart#test-request) for more details.
Here's an example of an input file with 2 requests using the `/v1/chat/completions` endpoint.
```jsonl
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}], "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "messages": [{"role": "system", "content": "You are an unhelpful assistant."}, {"role": "user", "content": "What is the capital of Belgium?"}], "max_tokens": 1000}}
```
And here is an example of an input file using the `/v1/completions` endpoint:
```jsonl
{"custom_id": "request-1", "method": "POST", "url": "/v1/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "prompt": "What is the capital of France?", "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "prompt": "What is the capital of Belgium?", "max_tokens": 1000}}
```
### 2. Uploading Your Batch Input File [#2-uploading-your-batch-input-file]
In order to create a Batch Processing job, you must first upload your input file.
```typescript TypeScript
import fs from "fs";
const batchInputFile = await client.files.create({
file: fs.createReadStream("batchinput.jsonl"),
purpose: "batch",
});
console.log(batchInputFile);
```
```python Python
batch_input_file = client.files.create(
file=open("batchinput.jsonl", "rb"),
purpose="batch",
)
print(batch_input_file)
```
```bash cURL
curl https://batch.inference.net/v1/files \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-F purpose="batch" \
-F file="@batchinput.jsonl"
```
The response will look similar to this, depending on the language you are using:
```json JSON
{
"id": "file-abc123"
}
```
### 3. Starting the Batch Processing Job [#3-starting-the-batch-processing-job]
Once you've successfully uploaded your input file, you can use the ID of the file to create a batch.
In this case, let's assume the file ID is `file-abc123`.
The completion window accepts any value ending in `h`, `m`, `s`, or `ms` (e.g. `24h`), bounded between 24 hours and 7 days; `24h` is the default.
To associate custom metadata with the batch, you can provide an optional `metadata` parameter.
This metadata is not used by Inference.net to complete requests, but it is included when retrieving the status of a batch so that you can associate custom metadata with the batch.
> **Note:** The Batch Processing job will begin processing immediately after creation.
Create the Batch
```typescript TypeScript
import type { BatchCreateParams } from "openai/resources/batches";
const batch = await client.batches.create({
input_file_id: batchInputFile.id,
endpoint: "/v1/chat/completions",
completion_window: "24h",
metadata: {
description: "nightly eval job",
},
// Optional. Must be HTTPS.
webhook_url: "https://example.com/my_webhook",
} as BatchCreateParams);
console.log(batch);
```
```python Python
batch = client.batches.create(
input_file_id=batch_input_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={
"description": "nightly eval job",
},
# Optional. Must be HTTPS.
extra_body={
"webhook_url": "https://example.com/my_webhook",
},
)
print(batch)
```
```bash cURL
curl https://batch.inference.net/v1/batches \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file-abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"metadata": {
"description": "nightly eval job"
},
"webhook_url": "https://example.com/webhook"
}'
```
This request will return a batch object with metadata about your batch:
```json JSON
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file-abc123",
"completion_window": "24h",
"status": "validating",
"output_file_id": null,
"error_file_id": null,
"created_at": 1714508499,
"in_progress_at": 1714508500,
"expires_at": 1714536634,
"completed_at": null,
"failed_at": null,
"expired_at": null,
"request_counts": {
"total": 2,
"completed": 0,
"failed": 0
},
"metadata": {
"description": "nightly eval job"
}
}
```
Inference.net supports a `webhook_url` that you can set to receive a webhook notification when the batch is complete.
The `webhook_url` must be an HTTPS URL that can receive POST requests.
If no metadata is provided when the batch is created, the `metadata` field will be null.
Your webhook will receive a POST with a request JSON body that looks like this:
```json JSON
{
"batch_id": "batch_abc123",
"status": "completed",
"metadata": {
"description": "nightly eval job"
}
}
```
The `webhook_url` parameter is not part of the official OpenAI SDK types. In TypeScript, cast the params as `BatchCreateParams` to avoid type errors. In Python, pass it via `extra_body`.
### 4. Checking the Status of a Batch [#4-checking-the-status-of-a-batch]
You can check the status of a batch at any time, which will also return a Batch object.
Check the status of a batch by retrieving it using the Batch ID assigned to it by Inference.net (represented here by `batch_abc123`).
```typescript TypeScript
const retrievedBatch = await client.batches.retrieve(batch.id);
console.log(retrievedBatch);
```
```python Python
retrieved_batch = client.batches.retrieve(batch.id)
print(retrieved_batch)
```
```bash cURL
curl https://batch.inference.net/v1/batches/batch_abc123 \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json"
```
The status of a given Batch object can be any of the following:
| Status | Description |
| ------------ | ------------------------------------------------------------------------------ |
| validating | the input file is being validated before the batch can begin |
| failed | the input file has failed the validation process |
| in\_progress | the input file was successfully validated and the batch is currently being run |
| finalizing | the batch has completed and the results are being prepared |
| completed | the batch has been completed and the results are ready |
| expired | the batch was not able to be completed within the completion window |
| cancelling | the batch is being cancelled (may take up to 10 minutes) |
| cancelled | the batch was cancelled |
The `cancelling` and `cancelled` states are reserved for compatibility with the OpenAI schema but are not reachable: cancellation is not currently supported, and `POST /batches/{id}/cancel` returns an unsupported-endpoint error.
### 5. Retrieving the Results [#5-retrieving-the-results]
You will receive an email notification when the batch is complete.
Once the batch is complete, you can download the output by making a request against the Files API using the `output_file_id` field from the Batch object.
Similarly, you can retrieve the error file (containing all failed requests) by making a request against the Files API using the `error_file_id` field from the Batch object.
```typescript TypeScript
const fileResponse = await client.files.content(batch.output_file_id);
const fileContents = await fileResponse.text();
console.log(fileContents);
```
```python Python
file_response = client.files.content(batch.output_file_id)
print(file_response.text)
```
```bash cURL
curl https://batch.inference.net/v1/files/output-file-id/content \
-H "Authorization: Bearer $INFERENCE_API_KEY" > batch_output.jsonl
```
The output `.jsonl` file will have one response line for every successful request line in the input file. Any failed requests in the batch will have their error information written to an error file that can be found via the batch's `error_file_id`.
> Note that the output line order **may not match** the input line order.
Instead of relying on order to process your results, use the custom\_id field which will be present in each line of your output file and allow you to map requests in your input to results in your output.
```jsonl
{"id": "batch_req_123", "custom_id": "request-2", "response": {"status_code": 200, "request_id": "req_123", "body": {"id": "chatcmpl-123", "object": "chat.completion", "created": 1711652795, "model": "google/gemma-3-27b-instruct/bf-16", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello."}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 2, "total_tokens": 24}, "system_fingerprint": "fp_123"}}, "error": null}
{"id": "batch_req_456", "custom_id": "request-1", "response": {"status_code": 200, "request_id": "req_789", "body": {"id": "chatcmpl-abc", "object": "chat.completion", "created": 1711652789, "model": "google/gemma-3-27b-instruct/bf-16", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I assist you today?"}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 20, "completion_tokens": 9, "total_tokens": 29}, "system_fingerprint": "fp_3ba"}}, "error": null}
```
## Listing All Batches [#listing-all-batches]
At any time, you can see all your batches. For users with many batches, you can use the `limit` and `after` parameters to paginate your results.
If an `after` parameter is provided, the list will return batches after the specified batch ID.
```typescript TypeScript
const list = await client.batches.list({
limit: 10,
after: batch.id,
});
for await (const b of list) {
console.log(b);
}
```
```python Python
batches = client.batches.list(limit=10, after=batch.id)
for b in batches:
print(b)
```
```bash cURL
curl 'https://batch.inference.net/v1/batches?limit=10&after=batch_abc123' \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json"
```
## Batch Expiration [#batch-expiration]
Batches that do not complete in time eventually move to an `expired` state; unfinished requests within that batch are cancelled, and any responses to completed requests are made available via the batch's output file. You will only be charged for tokens consumed from any completed requests.
Expired requests will be written to your error file with the message as shown below. You can use the `custom_id` to retrieve the request data for expired requests.
```jsonl
{"id": "batch_req_123", "custom_id": "request-3", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
{"id": "batch_req_123", "custom_id": "request-7", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
```
## Rate Limits [#rate-limits]
Batch API rate limits are separate from existing per-model rate limits. A batch input file may include up to 1,000,000 requests (one JSONL line each). If you need higher rate limits, please contact us at [support@inference.net](mailto:support@inference.net).
## Compatibility Notes [#compatibility-notes]
### 1. Batch Cancellation [#1-batch-cancellation]
Although the OpenAI SDK supports the ability to cancel an in-progress batch, Inference.net does not currently support batch cancellation. This is under development and will be available soon.
### 2. Model Availability [#2-model-availability]
Inference.net's Batch Processing is compatible with all of Inference.net's supported models. See our list of [supported models](https://inference.net/models) for a complete list.
# Group API (https://docs.inference.net/api/async-inference/group)
Learn how to use our Group API to submit multiple inference requests together, perfect for processing related tasks that need to be tracked as a unit. The Group API supports both chat completions and text completions with up to 50 requests per group.
Group API is available for `/v1/slow/group/chat/completions`, `/v1/slow/group/completions`, `/v1/slow/group/embeddings`, and `/v1/slow/group/responses` endpoints.
You should not mix completion and chat-completion requests in the same group.
## Overview [#overview]
The Group API provides a streamlined way to submit multiple asynchronous inference requests as a single unit. Unlike the Batch API which requires JSONL file uploads, the Group API accepts requests directly in the request body, making it ideal for:
* **Small to medium batches:** Process up to 50 requests at once
* **Related tasks:** Group related inference requests together
* **Webhook notifications:** Get notified when all requests in a group complete
* **Simpler integration:** No file uploads or JSONL formatting required
* **Faster implementation:** Direct JSON API calls without file management
## Group API vs Batch API [#group-api-vs-batch-api]
| Feature | Group API | Batch API |
| ---------------- | ----------------------------------- | -------------------------------- |
| Maximum requests | 50 | 1,000,000 |
| Input format | JSON array in request body | JSONL file upload |
| File management | Not required | Required |
| Use case | Small batches, quick implementation | Large-scale processing |
| Webhook support | Yes | Yes |
| Completion time | 72 hours | 24 hours (default), up to 7 days |
## Getting Started [#getting-started]
### 1. Submit a Group Request [#1-submit-a-group-request]
Submit multiple requests together by sending them as an array in the request body:
```typescript TypeScript
const response = await fetch(
"https://api.inference.net/v1/slow/group/chat/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
{
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of France?" },
],
max_tokens: 100,
},
{
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of Germany?" },
],
max_tokens: 100,
},
],
webhook_id: "my-webhook-123", // Optional: attach a webhook for notifications
}),
},
);
const result = await response.json();
console.log(result); // { groupId: "group_abc123", groupSize: 2 }
```
```python Python
import os
import requests
response = requests.post(
"https://api.inference.net/v1/slow/group/chat/completions",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
"max_tokens": 100,
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"},
],
"max_tokens": 100,
},
],
"webhook_id": "my-webhook-123", # Optional: attach a webhook for notifications
},
)
result = response.json()
print(result) # {"groupId": "group_abc123", "groupSize": 2}
```
```bash cURL
curl https://api.inference.net/v1/slow/group/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"}
],
"max_tokens": 100
}
],
"webhook_id": "my-webhook-123"
}'
```
The response will include a group ID and the number of requests:
```json JSON
{
"groupId": "group_xY3kL9mN2pQ",
"groupSize": 2
}
```
### 2. Retrieve Group Results [#2-retrieve-group-results]
Once your group is processed, retrieve all generation results using the group ID:
```typescript TypeScript
const response = await fetch(
`https://api.inference.net/v1/slow/group/${groupId}/generations`,
{
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
},
},
);
const result = await response.json();
console.log(result.generations); // Array of all completed generations
```
```python Python
import os
import requests
response = requests.get(
f"https://api.inference.net/v1/slow/group/{group_id}/generations",
headers={"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}"},
)
result = response.json()
print(result["generations"]) # Array of all completed generations
```
```bash cURL
curl https://api.inference.net/v1/slow/group/group_xY3kL9mN2pQ/generations \
-H "Authorization: Bearer $INFERENCE_API_KEY"
```
The response includes all generations in the group:
```json JSON
{
"generations": [
{
"id": "gen_abc123",
"state": "Success",
"request": {
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100
},
"response": {
"id": "gen_abc123",
"object": "chat.completion",
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}
},
{
"id": "gen_def456",
"state": "Success",
"request": {
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"}
],
"max_tokens": 100
},
"response": {
"id": "gen_def456",
"object": "chat.completion",
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of Germany is Berlin."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}
}
]
}
```
## Using Webhooks [#using-webhooks]
Attach a webhook to receive notifications when your group completes processing. Include the `webhook_id` when submitting the group:
```typescript TypeScript
const response = await fetch(
"https://api.inference.net/v1/slow/group/chat/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
/* your requests */
],
webhook_id: "my-webhook-123",
}),
},
);
```
```python Python
import os
import requests
response = requests.post(
"https://api.inference.net/v1/slow/group/chat/completions",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"requests": [
# your requests
],
"webhook_id": "my-webhook-123",
},
)
```
```bash cURL
curl https://api.inference.net/v1/slow/group/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type": "application/json" \
-d '{
"requests": [ ... ],
"webhook_id": "my-webhook-123"
}'
```
When all requests in the group complete, your webhook will receive a notification with:
* Group ID
* Completion status
* Summary of successful and failed requests
* Custom IDs for each request (if provided)
See our [Webhook Documentation](/api/async-inference/webhooks/getting-started-with-webhooks) for setup instructions.
## Text Completions Support [#text-completions-support]
The Group API also supports text completions:
```typescript TypeScript
const response = await fetch(
"https://api.inference.net/v1/slow/group/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
{
model: "google/gemma-3-27b-instruct/bf-16",
prompt: "The capital of France is",
max_tokens: 10,
},
{
model: "google/gemma-3-27b-instruct/bf-16",
prompt: "The capital of Germany is",
max_tokens: 10,
},
],
}),
},
);
```
```python Python
import os
import requests
response = requests.post(
"https://api.inference.net/v1/slow/group/completions",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of France is",
"max_tokens": 10,
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of Germany is",
"max_tokens": 10,
},
],
},
)
```
```bash cURL
curl https://api.inference.net/v1/slow/group/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of France is",
"max_tokens": 10
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of Germany is",
"max_tokens": 10
}
]
}'
```
## Limits and Constraints [#limits-and-constraints]
* **Maximum requests per group:** 50
* **Request format:** Direct JSON (no JSONL files required)
* **Supported endpoints:**
* `/v1/slow/group/chat/completions`
* `/v1/slow/group/completions`
* `/v1/slow/group/embeddings`
* `/v1/slow/group/responses`
* **Completion time:** 72 hours
* **Request expiration:** Groups expire after 72 hours if not completed
## Best Practices [#best-practices]
1. **Group related requests:** Use groups for requests that logically belong together (e.g., analyzing multiple documents from the same source).
2. **Use webhooks for notifications:** Instead of polling, configure webhooks to be notified when your group completes.
3. **Handle individual failures:** Some requests in a group may fail while others succeed. Check each generation's status.
4. **Stay under limits:** Keep groups to 50 requests or less. For larger batches, use the [Batch API](/api/async-inference/batch-api).
5. **Include metadata:** Add custom IDs or metadata to your requests for easier tracking:
```json JSON
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [{"role": "user", "content": "..."}],
"metadata": {
"custom_id": "doc_123",
"type": "summary"
}
}
```
## Error Handling [#error-handling]
The API validates your request structure immediately. Common errors include:
```json JSON
{
"error": {
"message": "Invalid request body.",
"type": "BadRequestError",
"fields": {
"_errors": ["Unrecognized key(s) in object: 'webhook_url'"]
}
}
}
```
Ensure you use the correct field names:
* `webhook_id` (correct)
* `webhook_url` (incorrect — this is for the Batch API)
## When to Use Group API [#when-to-use-group-api]
Choose the Group API when you need:
* Quick implementation without file management
* To process 50 or fewer related requests
* Webhook notifications for a set of requests
* Simple JSON-based integration
For larger workloads (50+ requests), consider using the [Batch API](/api/async-inference/batch-api) instead.
# Overview (https://docs.inference.net/api/async-inference/overview)
Learn how to use our OpenAI-compatible Asynchronous Inference API to send individual inference requests that complete within 24 hours at reduced costs. Simply use `/v1/slow` instead of `/v1/` in your API calls to access this feature.
Background inference is cheaper, and easier to build with when your application isn't serving real-time inference.
Asynchronous Inference API is compatible with all the [models](https://inference.net/models) we offer.
Webhook support is available for all slow endpoints: `/chat/completions`, `/completions`, and `/embeddings` calls.
Asynchronous and Batch requests require inference results retention to be enabled for your team. Requests and results are stored until they are delivered, so teams that have disabled retention receive a `403 Forbidden` on all `/v1/slow`, `/v1/batches`, and `/v1/files` upload requests. Enable retention in your team settings, or use the synchronous `/v1` endpoints instead.
## Overview [#overview]
The Asynchronous Inference API provides a simple way to make cost-effective inference requests when immediate responses aren't required. By using the `/v1/slow` prefix instead of `/v1/`, you can:
1. **Get immediate request IDs:** Your API call returns instantly with a unique ID.
2. **Save on costs:** Enjoy significantly cheaper pricing compared to synchronous requests.
3. **Flexible completion:** Requests complete within 24 hours.
4. **Same familiar API:** Uses the exact same request format as our standard endpoints.
This API is perfect for use cases like:
* Large-scale content generation
* Batch document processing
* Non-urgent data analysis
* Cost-sensitive workloads
* Background processing tasks
## Getting Started [#getting-started]
Using the Asynchronous Inference API is as simple as changing your base URL from `/v1/` to `/v1/slow/`. The API maintains full compatibility with the OpenAI SDK.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1/slow", // Note the /v1/slow prefix
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1/slow", # Note the /v1/slow prefix
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL
# Use /v1/slow/ instead of /v1/ in the URL
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ ... }'
```
## Making Asynchronous Requests [#making-asynchronous-requests]
### 1. Submit a Request [#1-submit-a-request]
Make requests exactly as you would with the standard API, but responses will include a request ID instead of the completion result:
```typescript TypeScript
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of France?" },
],
max_tokens: 1000,
});
console.log(response.id); // Returns immediately with request ID
```
```python Python
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
max_tokens=1000,
)
print(response.id) # Returns immediately with request ID
```
```bash cURL
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 1000
}'
```
The initial response will include a unique request ID:
```json JSON
{
"id": "N2mZQjrvh-k_m8nMMN7Jn",
"choices": [],
"created": 1749061362809,
"model": "google/gemma-3-27b-instruct/bf-16",
"object": "chat.completion"
}
```
### 2. Retrieve Results [#2-retrieve-results]
Once your request is processed, retrieve the results using the generation endpoint:
```typescript TypeScript
const response = await fetch(
`https://api.inference.net/v1/generation/${generationId}`,
{
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
},
},
);
const result = await response.json();
```
```python Python
import os
import requests
response = requests.get(
f"https://api.inference.net/v1/generation/{generation_id}",
headers={"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}"},
)
result = response.json()
```
```bash cURL
curl https://api.inference.net/v1/generation/N2mZQjrvh-k_m8nMMN7Jn \
-H "Authorization: Bearer $INFERENCE_API_KEY"
```
The completed response includes both the original request and the generation result:
```json JSON
{
"request": {
"messages": [
{"content": "You are a helpful assistant.", "role": "system"},
{"content": "What is the meaning of life?", "role": "user"}
],
"model": "google/gemma-3-27b-instruct/bf-16",
"stream": false,
"max_tokens": 8,
"metadata": {"webhook_id": "mPufxRcrw"}
},
"response": {
"id": "N2mZQjrvh-k_m8nMMN7Jn",
"object": "chat.completion",
"created": 1749061362,
"model": "google/gemma-3-27b-instruct/bf-16",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The meaning of life is a complex and",
"reasoning_content": null,
"tool_calls": null
},
"logprobs": null,
"finish_reason": "length",
"matched_stop": null
}
],
"usage": {
"prompt_tokens": 48,
"total_tokens": 56,
"completion_tokens": 8,
"prompt_tokens_details": null
},
"system_fingerprint": ""
},
"state": "Success",
"stateMessage": "Generation successful",
"finishedAt": "2025-06-04T18:22:42.912Z"
}
```
## Request States [#request-states]
Asynchronous requests can have the following states:
| Status | Description |
| ----------- | ------------------------------------------------- |
| Queued | Request received and queued for processing |
| In Progress | Request is currently being processed |
| Success | Request completed successfully, results available |
| Failed | Request failed due to an error |
## Best Practices [#best-practices]
1. **Store Request IDs:** Always save the returned request ID for later retrieval.
2. **Use Webhooks:** Instead of polling, set up webhooks for real-time notifications when requests complete. See our [Getting Started with Webhooks](/api/async-inference/webhooks/getting-started-with-webhooks) guide.
3. **Handle Failures:** Have a fallback plan for requests that fail during processing.
4. **Batch When Possible:** For multiple requests, consider using our [Batch API](/api/async-inference/batch-api) for better organization.
## Supported Endpoints [#supported-endpoints]
The Asynchronous Inference API supports the following endpoints:
* `/v1/slow/chat/completions`
* `/v1/slow/completions`
* `/v1/slow/embeddings`
Simply replace `/v1/` with `/v1/slow/` in your existing code to use asynchronous processing.
## Pricing and Limits [#pricing-and-limits]
* **Pricing:** Significantly reduced compared to synchronous requests (contact sales for specific rates)
* **Completion Time:** 24 hours
* **Rate Limits:** More generous than synchronous endpoints
* **Request Expiration:** Requests expire after 24 hours if not completed
For specific pricing information and higher rate limits, please contact [support@inference.net](mailto:support@inference.net).
# Getting Started With Webhooks (https://docs.inference.net/api/async-inference/webhooks/getting-started-with-webhooks)
Webhook support is available for all slow endpoints: `/chat/completions`, `/completions`, and `/embeddings` calls.
## Overview [#overview]
Webhooks provide an efficient push-based notification system for tracking generation completions in real-time. Rather than repeatedly polling the API to check generation status, webhooks automatically notify your application when generations complete, enabling streamlined workflows and better resource utilization.
## Key Benefits [#key-benefits]
* **Resource Efficiency**: Eliminate unnecessary API calls for status checks
* **Real-time Updates**: Receive notifications within milliseconds of generation completion
* **Scalability**: Handle thousands of concurrent generations efficiently
* **Improved User Experience**: Update your UI instantly when results are ready
## Getting Started [#getting-started]
### Step 1: Create a Webhook Endpoint [#step-1-create-a-webhook-endpoint]
Your application needs an HTTPS endpoint capable of receiving POST requests. The endpoint should:
1. Accept JSON payloads
2. Respond with HTTP 200 status immediately
3. Process the webhook data asynchronously
```typescript TypeScript
import express from "express";
const app = express();
app.post("/webhooks/inference", express.json(), (req, res) => {
const { event, data } = req.body;
// Verify webhook source via headers
const webhookId = req.headers["x-inference-webhook-id"];
if (event === "generation.completed") {
console.log(`Generation ${data.id} completed with status: ${data.state}`);
// Process asynchronously
setImmediate(() => {
processGenerationResult(data);
});
} else if (event === "async-embedding.completed") {
console.log(`Embedding ${data.id} completed with status: ${data.state}`);
setImmediate(() => {
processEmbeddingResult(data);
});
}
// Always respond immediately
res.status(200).json({ received: true });
});
app.listen(3000, () => {
console.log("Webhook receiver listening on port 3000");
});
```
```python Python
from fastapi import FastAPI, Request, BackgroundTasks
from pydantic import BaseModel
from typing import Optional, Dict, Any
app = FastAPI()
class WebhookPayload(BaseModel):
event: str
timestamp: str
webhookId: str
data: Dict[str, Any]
def process_generation(payload: WebhookPayload):
"""Process generation result asynchronously"""
if payload.event == "generation.completed":
print(f"Processing generation {payload.data['id']}")
# Your processing logic here
elif payload.event == "async-embedding.completed":
print(f"Processing embedding {payload.data['id']}")
# Your embedding processing logic here
@app.post("/webhooks/inference")
async def handle_webhook(
payload: WebhookPayload,
request: Request,
background_tasks: BackgroundTasks,
):
# Verify webhook source
webhook_id = request.headers.get("x-inference-webhook-id")
# Queue for background processing
background_tasks.add_task(process_generation, payload)
return {"received": True}
```
Go Example
```go
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
)
type WebhookPayload struct {
Event string `json:"event"`
Timestamp string `json:"timestamp"`
WebhookID string `json:"webhookId"`
Data map[string]any `json:"data"`
}
func handleWebhook(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "Failed to read body", http.StatusBadRequest)
return
}
var payload WebhookPayload
if err := json.Unmarshal(body, &payload); err != nil {
http.Error(w, "Invalid JSON", http.StatusBadRequest)
return
}
// Verify webhook source
webhookID := r.Header.Get("X-Inference-Webhook-ID")
// Process asynchronously
go func() {
switch payload.Event {
case "generation.completed":
fmt.Printf("Processing generation %s\n", payload.Data["id"])
// Your processing logic here
case "async-embedding.completed":
fmt.Printf("Processing embedding %s\n", payload.Data["id"])
// Your embedding processing logic here
}
}()
// Respond immediately
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]bool{"received": true})
}
func main() {
http.HandleFunc("/webhooks/inference", handleWebhook)
fmt.Println("Webhook server listening on :3000")
http.ListenAndServe(":3000", nil)
}
```
### Step 2: Deploy Your Endpoint [#step-2-deploy-your-endpoint]
Your webhook endpoint must be publicly accessible via HTTPS. For development environments, consider using:
* **ngrok**: `ngrok http 3000`
* **Cloudflare Tunnel**: Provides a stable URL
* **localtunnel**: `lt --port 3000`
### Step 3: Register Your Webhook [#step-3-register-your-webhook]
1. Navigate to the inference.net dashboard
2. Go to **Settings** → **Integrations** in the sidebar
3. Click **Create Webhook**
4. Enter a descriptive name and your HTTPS endpoint URL
5. Save your webhook
You'll receive a webhook identifier (e.g., `AhALzdz8S`) that you'll use when creating generations.
### Step 4: Link Webhook to Generations [#step-4-link-webhook-to-generations]
Include the webhook identifier in the metadata when creating a generation:
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1/slow",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "gemma-3-27b-it",
messages: [{ role: "user", content: "Explain quantum computing" }],
// @ts-expect-error metadata is not in the OpenAI SDK types
metadata: {
webhook_id: "AhALzdz8S",
},
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1/slow",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="gemma-3-27b-it",
messages=[{"role": "user", "content": "Explain quantum computing"}],
extra_body={
"metadata": {"webhook_id": "AhALzdz8S"},
},
)
```
```bash cURL
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-3-27b-it",
"messages": [
{"role": "user", "content": "Explain quantum computing"}
],
"metadata": {
"webhook_id": "AhALzdz8S"
}
}'
```
When the generation completes, your webhook endpoint will receive a notification.
## Webhook Events [#webhook-events]
### generation.completed [#generationcompleted]
Sent when a generation finishes processing (successfully or with failure):
```json JSON
{
"event": "generation.completed",
"timestamp": "2025-01-03T06:46:22.838Z",
"webhookId": "AhALzdz8S",
"data": {
"id": "XBKcs7F1s2oJ_AHiLMbF4",
"state": "Success",
"stateMessage": "Generation successful",
"request": {
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing"}
],
"model": "gemma-3-27b-it",
"stream": false,
"max_tokens": 100,
"metadata": {
"webhook_id": "AhALzdz8S"
}
},
"response": {
"id": "XBKcs7F1s2oJ_AHiLMbF4",
"object": "chat.completion",
"created": 1748933182,
"model": "gemma-3-27b-it",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "Quantum computing is a revolutionary approach..."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 28,
"completion_tokens": 42,
"total_tokens": 70
}
},
"model": "gemma-3-27b-it",
"createdAt": "2025-01-03T06:46:18.000Z",
"dispatchedAt": "2025-01-03T06:46:19.100Z",
"firstChunkAt": "2025-01-03T06:46:20.200Z",
"finishedAt": "2025-01-03T06:46:22.307Z",
"usage": {
"prompt_tokens": 28,
"completion_tokens": 42,
"total_tokens": 70
}
}
}
```
The generation ID is the `data.id` field. The `response` object is compatible with the types exported from the official OpenAI SDKs.
```typescript TypeScript
import type { OpenAI } from "openai";
const response = responseJsonObject as OpenAI.Chat.Completions.ChatCompletion;
```
```python Python
from openai.types.chat.chat_completion import ChatCompletion
response: ChatCompletion = webhook_payload.data["response"]
```
### async-embedding.completed [#async-embeddingcompleted]
Sent when an async embedding request finishes processing:
```json JSON
{
"event": "async-embedding.completed",
"timestamp": "2025-01-15T10:30:00Z",
"webhookId": "AhALzdz8S",
"data": {
"id": "EMB_abc123",
"state": "Success",
"stateMessage": "Embeddings generated successfully",
"request": {
"model": "qwen/qwen3-embedding-4b",
"input": ["text1", "text2"],
"metadata": { "webhook_id": "AhALzdz8S" }
},
"response": {
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.0023064255, -0.009327292]
}
],
"model": "qwen/qwen3-embedding-4b",
"usage": {
"prompt_tokens": 100,
"total_tokens": 100
}
},
"model": "qwen/qwen3-embedding-4b",
"createdAt": "2025-01-15T10:29:55Z",
"dispatchedAt": "2025-01-15T10:29:56Z",
"firstChunkAt": "2025-01-15T10:29:58Z",
"finishedAt": "2025-01-15T10:30:00Z",
"usage": {
"prompt_tokens": 100,
"total_tokens": 100
}
}
}
```
The generation ID is the `data.id` field. The `response` object follows the standard OpenAI embeddings format.
## Headers [#headers]
All webhook requests include the following headers:
| Header | Description | Example |
| --------------------------- | ------------------------------------- | ----------------------- |
| `X-Inference-Event` | Event type | `generation.completed` |
| `X-Inference-Webhook-ID` | Webhook identifier | `AhALzdz8S` |
| `X-Inference-Generation-ID` | Generation ID (for generation events) | `XBKcs7F1s2oJ_AHiLMbF4` |
| `X-Inference-Embedding-ID` | Generation ID (for embedding events) | `EMB_abc123` |
| `User-Agent` | inference.net webhook agent | `Kuzco-Webhook/1.0` |
| `Content-Type` | Always `application/json` | `application/json` |
Before each delivery, the service also sends an `OPTIONS` preflight request to your endpoint with the same `X-Inference-*` headers. The preflight must return a 2xx status or the delivery is retried.
### (Security) Verifying the request source [#security-verifying-the-request-source]
The `X-Inference-Webhook-ID` is a good way to verify that the payload you're receiving is officially coming from our API.
This ID is unique to your webhook, and is completely private to you and your team.
If the ID does not match what you see in the dashboard, your endpoint has most likely been discovered by a malicious actor.
## Testing Webhooks [#testing-webhooks]
You can test your webhook endpoint from the dashboard:
1. Navigate to **Settings** → **Integrations** in the dashboard
2. Find your webhook in the list
3. Click the menu and select **Test**
4. Check your endpoint logs for the test payload
A successful test will show a green success indicator in the dashboard.
## Best Practices [#best-practices]
### 1. Respond Immediately [#1-respond-immediately]
Your endpoint must respond within 60 seconds. Always return a 200 status immediately and process the webhook asynchronously:
```typescript TypeScript
// Correct approach
app.post("/webhook", (req, res) => {
res.status(200).send("OK");
processWebhookAsync(req.body);
});
// Incorrect approach — may timeout
app.post("/webhook", async (req, res) => {
await heavyProcessing(req.body); // Risk of timeout
res.status(200).send("OK");
});
```
```python Python
# Correct approach — process in background
@app.post("/webhook")
async def handle_webhook(
payload: WebhookPayload,
background_tasks: BackgroundTasks,
):
background_tasks.add_task(process_webhook, payload)
return {"received": True}
# Incorrect approach — may timeout
@app.post("/webhook")
async def handle_webhook(payload: WebhookPayload):
await heavy_processing(payload) # Risk of timeout
return {"received": True}
```
### 2. Implement Idempotency [#2-implement-idempotency]
Failed webhooks may be retried. Use `data.id` to ensure you don't process the same event twice:
```typescript TypeScript
const processedGenerations = new Set();
async function processWebhook(payload: any) {
const generationId = payload.data.id;
if (processedGenerations.has(generationId)) {
return; // Already processed
}
processedGenerations.add(generationId);
// Process the generation
}
```
```python Python
processed_generations: set[str] = set()
def process_webhook(payload: WebhookPayload):
generation_id = payload.data["id"]
if generation_id in processed_generations:
return # Already processed
processed_generations.add(generation_id)
# Process the generation
```
For `slow-group.completed` events, use the top-level `groupId` field for idempotency.
### 3. Validate Webhook Source [#3-validate-webhook-source]
Always verify that webhooks originate from inference.net by checking the presence of expected headers:
```typescript TypeScript
function validateWebhookSource(headers: Record): boolean {
const requiredHeaders = ["x-inference-webhook-id", "x-inference-event"];
return requiredHeaders.every((header) => headers[header]);
}
```
```python Python
def validate_webhook_source(headers: dict) -> bool:
required_headers = ["x-inference-webhook-id", "x-inference-event"]
return all(headers.get(h) for h in required_headers)
```
### 4. Handle Errors Gracefully [#4-handle-errors-gracefully]
Implement proper error handling to prevent individual failures from affecting your entire system:
```typescript TypeScript
async function handleWebhook(payload: any) {
try {
await processWebhook(payload);
} catch (error) {
console.error("Webhook processing failed:", error);
// Log to monitoring service
// Return 200 to prevent unnecessary retries
}
}
```
```python Python
async def handle_webhook(payload: WebhookPayload):
try:
await process_webhook(payload)
except Exception as error:
print(f"Webhook processing failed: {error}")
# Log to monitoring service
# Return 200 to prevent unnecessary retries
```
### 5. Monitor Webhook Processing [#5-monitor-webhook-processing]
Track key metrics to ensure reliable webhook processing:
* Webhook receipt rate
* Processing success/failure rates
* Average processing time
* Queue depth (if using queues)
## Troubleshooting [#troubleshooting]
### Not Receiving Webhooks [#not-receiving-webhooks]
1. **Check webhook status**: Ensure your webhook is not disabled in the dashboard
2. **Test connectivity**: Use the test feature in the dashboard
3. **Verify URL**: Confirm your endpoint is publicly accessible via HTTPS
4. **Check logs**: Review both your server logs and any reverse proxy logs
5. **Validate metadata**: Ensure you're including the correct `webhook_id` in generation requests
### Webhooks Arriving Late [#webhooks-arriving-late]
* Verify your endpoint responds quickly (\< 1 second ideally)
* Check that you're not performing heavy processing before responding
* Monitor your server load and resource usage
### Duplicate Webhook Deliveries [#duplicate-webhook-deliveries]
* Implement idempotency using `data.id` (or `groupId` for group events)
* Ensure your endpoint always returns 200 OK for successful receipt
* Check for any errors in your webhook processing that might trigger retries
## Frequently Asked Questions [#frequently-asked-questions]
**Q: What happens if my endpoint is down?**
A: Failed webhook deliveries are retried up to 1200 times at fixed intervals — 30 seconds for generation events and 5 minutes for group events. After all attempts are exhausted, the delivery is marked as failed.
**Q: What's the webhook timeout?**
A: Webhook endpoints must respond within 60 seconds. A timeout is treated as a failure and will trigger retries.
**Q: Can I filter which events I receive?**
A: Currently, webhooks receive all event types. Event filtering is planned for a future update.
**Q: How secure are webhooks?**
A: All webhooks are sent over HTTPS. You should validate the webhook source using the provided headers. HMAC signature verification is planned for additional security.
**Q: What's the maximum payload size?**
A: Payloads scale with the size of the generation they describe — a payload is larger for generations with long requests or responses. There is no fixed size limit.
**Q: Can I replay missed webhooks?**
A: Webhook replay functionality is not currently available. As a fallback, you can poll the generation status endpoint.
## Support [#support]
For assistance with webhooks:
* Email: [support@inference.net](mailto:support@inference.net)
* Discord: Join our developer community
* Documentation: [https://docs.inference.net](https://docs.inference.net)
* Issues: Report bugs via our support portal
# Webhooks: Quick Reference (https://docs.inference.net/api/async-inference/webhooks/quick-reference)
## Dashboard Management [#dashboard-management]
Webhooks are managed through the inference.net dashboard:
1. Navigate to **Settings** → **Integrations**
2. Create, test, archive, or restore webhooks through the UI
3. Copy your webhook identifier for use in generation requests
## Payload Structures [#payload-structures]
### generation.completed [#generationcompleted]
```json JSON
{
"event": "generation.completed",
"timestamp": "ISO 8601 timestamp",
"webhookId": "webhook identifier",
"data": {
"id": "generation ID",
"state": "Success|Failed|Queued|In Progress",
"stateMessage": "Human readable status",
"request": { /* Original request or null */ },
"response": { /* OpenAI format response or null */ },
"model": "model ID",
"createdAt": "ISO 8601 timestamp",
"dispatchedAt": "ISO 8601 timestamp or null",
"firstChunkAt": "ISO 8601 timestamp or null",
"finishedAt": "ISO 8601 timestamp or null",
"usage": { /* Token usage */ }
}
}
```
The generation ID is `data.id`. The `response` object is compatible with the OpenAI SDK types.
### async-embedding.completed [#async-embeddingcompleted]
```json JSON
{
"event": "async-embedding.completed",
"timestamp": "ISO 8601 timestamp",
"webhookId": "webhook identifier",
"data": {
"id": "generation ID",
"state": "Success|Failed|Queued|In Progress",
"stateMessage": "Human readable status",
"request": { /* Original embeddings request or null */ },
"response": { /* OpenAI format embeddings response or null */ },
"model": "model ID",
"createdAt": "ISO 8601 timestamp",
"dispatchedAt": "ISO 8601 timestamp or null",
"firstChunkAt": "ISO 8601 timestamp or null",
"finishedAt": "ISO 8601 timestamp or null",
"usage": { /* Token usage */ }
}
}
```
### slow-group.completed [#slow-groupcompleted]
```json JSON
{
"event": "slow-group.completed",
"timestamp": "ISO 8601 timestamp",
"groupId": "group ID",
"data": {
"groupSize": 2,
"status": "completed|failed",
"generations": [
{
"id": "generation ID",
"state": "Success|Failed|Queued|In Progress",
"stateMessage": "Human readable status",
"request": { /* Original request or null */ },
"response": { /* OpenAI format response or null */ },
"model": "model ID",
"createdAt": "ISO 8601 timestamp",
"dispatchedAt": "ISO 8601 timestamp or null",
"firstChunkAt": "ISO 8601 timestamp or null",
"finishedAt": "ISO 8601 timestamp or null",
"usage": { /* Token usage */ }
}
]
}
}
```
### webhook.test (dashboard Test button) [#webhooktest-dashboard-test-button]
```json JSON
{
"event": "webhook.test",
"timestamp": "ISO 8601 timestamp",
"webhook_id": "webhook identifier",
"data": {
"message": "test message"
}
}
```
Note that this test payload uses the legacy `webhook_id` field (snake\_case), unlike delivery payloads which use `webhookId`.
## Headers [#headers]
| Header | Description | Example |
| --------------------------- | --------------------------------- | ------------------------------------------------------------------------------ |
| `X-Inference-Event` | Event type | `generation.completed`, `async-embedding.completed`, or `slow-group.completed` |
| `X-Inference-Webhook-ID` | Webhook identifier | `AhALzdz8S` |
| `X-Inference-Generation-ID` | Generation ID (generation events) | `XBKcs7F1s2oJ_AHiLMbF4` |
| `X-Inference-Embedding-ID` | Generation ID (embedding events) | `XBKcs7F1s2oJ_AHiLMbF4` |
| `X-Inference-Group-ID` | Group ID (group events) | `GRP_XYZ123` |
| `User-Agent` | inference.net webhook agent | `Kuzco-Webhook/1.0` |
| `Content-Type` | Always `application/json` | `application/json` |
## Using Webhooks in Generations [#using-webhooks-in-generations]
Include the webhook identifier in your generation request metadata:
### Chat Completions [#chat-completions]
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1/slow",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "gemma-3-27b-it",
messages: [{ role: "user", content: "Hello!" }],
// @ts-expect-error metadata is not in the OpenAI SDK types
metadata: { webhook_id: "YOUR_WEBHOOK_IDENTIFIER" },
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1/slow",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="gemma-3-27b-it",
messages=[{"role": "user", "content": "Hello!"}],
extra_body={"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"}},
)
```
```bash cURL
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-3-27b-it",
"messages": [{"role": "user", "content": "Hello!"}],
"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"}
}'
```
### Embeddings [#embeddings]
```typescript TypeScript
const embeddingResponse = await fetch(
"https://api.inference.net/v1/async/embeddings",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "qwen/qwen3-embedding-4b",
input: ["Text to embed", "Another text to embed"],
metadata: { webhook_id: "YOUR_WEBHOOK_IDENTIFIER" },
}),
},
);
```
```python Python
import os
import requests
response = requests.post(
"https://api.inference.net/v1/async/embeddings",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"model": "qwen/qwen3-embedding-4b",
"input": ["Text to embed", "Another text to embed"],
"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"},
},
)
```
```bash cURL
curl https://api.inference.net/v1/async/embeddings \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3-embedding-4b",
"input": ["Text to embed", "Another text to embed"],
"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"}
}'
```
## Minimal Webhook Handler Examples [#minimal-webhook-handler-examples]
```typescript TypeScript
app.post("/webhook", express.json(), (req, res) => {
res.status(200).json({ received: true });
if (req.body.event === "generation.completed") {
setImmediate(() => {
console.log("Generation completed:", req.body.data.id);
// Your processing logic here
});
} else if (req.body.event === "async-embedding.completed") {
setImmediate(() => {
console.log("Embedding completed:", req.body.data.id);
console.log("Number of embeddings:", req.body.data.response.data.length);
});
} else if (req.body.event === "slow-group.completed") {
setImmediate(() => {
console.log("Group completed:", req.body.groupId);
console.log("Group size:", req.body.data.groupSize);
req.body.data.generations.forEach((gen: any) => {
console.log(`Generation ${gen.id}: ${gen.state}`);
});
});
}
});
```
```python Python
@app.post("/webhook")
async def handle_webhook(payload: dict, background_tasks: BackgroundTasks):
background_tasks.add_task(process_webhook, payload)
return {"received": True}
def process_webhook(payload):
if payload["event"] == "generation.completed":
print(f"Processing generation {payload['data']['id']}")
# Your processing logic here
elif payload["event"] == "async-embedding.completed":
print(f"Processing embedding {payload['data']['id']}")
print(f"Number of embeddings: {len(payload['data']['response']['data'])}")
elif payload["event"] == "slow-group.completed":
print(f"Processing group {payload['groupId']}")
print(f"Group size: {payload['data']['groupSize']}")
for gen in payload["data"]["generations"]:
print(f"Generation {gen['id']}: {gen['state']}")
```
## Timing & Limits [#timing--limits]
| Metric | Value | Notes |
| ------------------ | ------------------------ | ---------------------------------------- |
| Response timeout | 60 seconds | POST and OPTIONS must respond in time |
| Generation retries | Up to 1200 | Fixed 30-second intervals |
| Group retries | Up to 1200 | Fixed 5-minute intervals |
| Max payload size | Large payloads supported | Payload size scales with generation size |
Retries are triggered on a non-2xx response or a timeout. Each delivery is preceded by an `OPTIONS` preflight request; if the preflight does not return a 2xx, the delivery is retried rather than skipped.
## Response Codes [#response-codes]
| Code | Meaning | Retry? |
| ------- | ------------------ | ------ |
| 200-299 | Success | No |
| 400-599 | Non-success status | Yes |
| Timeout | No response in 60s | Yes |
Any response outside the 200-299 range is treated as a failed delivery and retried.
## Best Practices Checklist [#best-practices-checklist]
* [ ] Respond with 200 OK immediately
* [ ] Process webhook data asynchronously
* [ ] Implement idempotency with `data.id` (generation events) or `groupId` (group events)
* [ ] Validate webhook source via headers
* [ ] Handle errors gracefully
* [ ] Monitor webhook processing
* [ ] Use HTTPS endpoint
* [ ] Set up proper error logging
* [ ] Test webhook with dashboard test feature
* [ ] Implement timeout handling
* [ ] Handle both individual and group completions
## Common Issues & Solutions [#common-issues--solutions]
| Issue | Solution |
| ---------------------- | ---------------------------------------------------------------------------- |
| Not receiving webhooks | Check webhook not disabled in dashboard, test connectivity, verify HTTPS URL |
| Duplicate webhooks | Implement idempotency, ensure 200 OK response |
| Webhooks timing out | Respond immediately, process asynchronously |
| Invalid payload | Validate against documented schema |
| Test webhook fails | Check endpoint is publicly accessible, returns 200 OK |
## Support Resources [#support-resources]
* [Full Documentation](https://docs.inference.net)
* [API Reference](https://docs.inference.net/api)
* [Support](mailto:support@inference.net)
* Discord Community
# Data Retention (https://docs.inference.net/api/data-retention)
Inference.net is designed to support production workloads without treating captured request data casually.
## Core principles [#core-principles]
* Request data is not used for model training by default
* Secrets and similar sensitive values are stripped where possible
* Platform data is encrypted in transit and at rest
* Retention should match the operational need of the workflow
## Direct API vs Gateway [#direct-api-vs-gateway]
The direct API and Gateway are different product paths, but the same general rule applies: only keep what is operationally useful, and use project-level controls and data curation intentionally.
For the workflow-first entry point into traffic capture, start with [Integrate with Your LLM Provider](/platform/gateway/integrate).
## Recommended operational pattern [#recommended-operational-pattern]
* Use environments and task IDs to segment traffic
* Create long-lived datasets only for the examples you want to preserve
* Review retention expectations before broad production rollout
## Retention and asynchronous inference [#retention-and-asynchronous-inference]
The [Asynchronous](/api/async-inference/overview) and Batch APIs store requests and results until they are delivered, so they are only available to teams with inference results retention enabled. Teams that have disabled retention receive a `403 Forbidden` on `/v1/slow`, `/v1/batches`, and `/v1/files` upload requests, and should use the synchronous `/v1` endpoints.
## Need a specific retention policy? [#need-a-specific-retention-policy]
If you need a specific policy, no-retention handling, or help mapping the platform into your internal compliance requirements, [meet with our team](https://inference.net/meet-with-us/).
# Function Calling (https://docs.inference.net/api/function-calling)
## Introduction [#introduction]
**Function calling** provides a powerful and flexible way for models to interface with your code or external services, and has two primary use cases:
| | |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fetching Data | Retrieve up-to-date information to incorporate into the model's response (RAG). Useful for searching knowledge bases and retrieving specific data from APIs (e.g. current weather data). |
| Taking Action | Perform actions like submitting a form, calling APIs, modifying application state (UI/frontend or backend), or taking agentic workflow actions (like handing off the conversation). |
If you only want the model to produce JSON, see our docs on [structured outputs](/api/structured-outputs).
## Getting Started [#getting-started]
You'll need an Inference.net account and API key to use Function Calling. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net using the OpenAI SDK, you will need to set the base URL to `https://api.inference.net/v1`.
In this example, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL
export INFERENCE_API_KEY=
# All requests use:
# -H "Authorization: Bearer $INFERENCE_API_KEY"
# -H "Content-Type: application/json"
```
## Overview [#overview]
You can extend the capabilities of models by giving them access to functions that you define called `tools`. This is also called "function calling".
With function calling, you'll tell the model what tools are available to it, and it will decide which one to use.
You'll then execute the function code, send back the results, and the model will incorporate them into its final response.
### Sample function [#sample-function]
Let's look at the steps to allow a model to use a real `get_weather` function defined below:
```typescript TypeScript
async function getWeather(latitude: number, longitude: number) {
const response = await fetch(
`https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}¤t=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m`
);
const data = await response.json();
return data.current.temperature_2m;
}
```
```python Python
import requests
def get_weather(latitude, longitude):
response = requests.get(
f"https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}"
f"¤t=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m"
)
data = response.json()
return data["current"]["temperature_2m"]
```
All functions must return strings. You can format the string as JSON or another format if you like, but the return type itself must be a string.
Unlike the diagram earlier, this function expects precise `latitude` and `longitude` instead of a general `location` parameter.
### Step By Step Example [#step-by-step-example]
#### Step 1: Call model with get\_weather tool defined [#step-1-call-model-with-get_weather-tool-defined]
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const tools: OpenAI.ChatCompletionTool[] = [{
type: "function",
function: {
name: "get_weather",
description: "Get current temperature for provided coordinates in celsius.",
parameters: {
type: "object",
properties: {
lat: { type: "number" },
lon: { type: "number" },
},
required: ["lat", "lon"],
additionalProperties: false,
},
strict: true,
},
}];
const messages: OpenAI.ChatCompletionMessageParam[] = [
{
role: "system",
content: "You are a helpful assistant that can answer questions and uses tools to get information.",
},
{
role: "user",
content: "What's the weather like in Paris today?",
},
];
const completion = await client.chat.completions.create({
model: "gemma-3-27b-it",
messages,
tools,
});
```
```python Python
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for provided coordinates in celsius.",
"parameters": {
"type": "object",
"properties": {
"lat": {"type": "number"},
"lon": {"type": "number"},
},
"required": ["lat", "lon"],
"additionalProperties": False,
},
"strict": True,
},
}]
messages = [
{"role": "system", "content": "You are a helpful assistant that can answer questions and uses tools to get information."},
{"role": "user", "content": "What's the weather like in Paris today?"},
]
completion = client.chat.completions.create(
model="gemma-3-27b-it",
messages=messages,
tools=tools,
)
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-3-27b-it",
"messages": [
{"role": "system", "content": "You are a helpful assistant that can answer questions and uses tools to get information."},
{"role": "user", "content": "What'\''s the weather like in Paris today?"}
],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for provided coordinates in celsius.",
"parameters": {
"type": "object",
"properties": {
"lat": { "type": "number" },
"lon": { "type": "number" }
},
"required": ["lat", "lon"],
"additionalProperties": false
},
"strict": true
}
}]
}'
```
Less powerful models may not reliably respond with tool calls, and may not provide all requested parameters.
Experiment with system prompts and other models to find the best results.
#### Step 2: Pull the selected function call from the model's response [#step-2-pull-the-selected-function-call-from-the-models-response]
```json JSON
[
{
"id": "call_12345xyz",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{\"lat\":48.8566,\"lon\":2.3522}"
}
}
]
```
#### Step 3: Execute the `get_weather` function [#step-3-execute-the-get_weather-function]
```typescript TypeScript
const toolCall = completion.choices[0].message.tool_calls[0];
const args = JSON.parse(toolCall.function.arguments);
const result = await getWeather(args.lat, args.lon);
```
```python Python
tool_call = completion.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)
result = get_weather(args["lat"], args["lon"])
```
#### Step 4: Supply result and call model again [#step-4-supply-result-and-call-model-again]
```typescript TypeScript
messages.push(completion.choices[0].message); // append model's function call message
messages.push({ // append result message
role: "tool",
tool_call_id: toolCall.id,
content: result.toString(),
});
const completion2 = await client.chat.completions.create({
model: "gemma-3-27b-it",
messages,
tools,
});
console.log(completion2.choices[0].message.content);
```
```python Python
messages.append(completion.choices[0].message) # append model's function call message
messages.append({ # append result message
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result),
})
completion_2 = client.chat.completions.create(
model="gemma-3-27b-it",
messages=messages,
tools=tools,
)
print(completion_2.choices[0].message.content)
```
#### Output [#output]
```json JSON
"The current temperature in Paris is 14°C (57.2°F)."
```
## Defining functions [#defining-functions]
Functions can be set in the `tools` parameter of each API request.
A function is defined by its schema, which informs the model what it does and what input arguments it expects. It comprises the following fields:
| Field | Description |
| ----------- | --------------------------------------------------- |
| name | The function's name (e.g. get\_weather) |
| description | Details on when and how to use the function |
| parameters | JSON schema defining the function's input arguments |
Take a look at this example:
```json JSON
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Retrieves current weather for the given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia"
},
"units": {
"type": "string",
"enum": [
"celsius",
"fahrenheit"
],
"description": "Units the temperature will be returned in."
}
},
"required": [
"location",
"units"
],
"additionalProperties": false
},
"strict": true
}
}
```
Because the `parameters` are defined by a [JSON schema](https://json-schema.org/), you can leverage many of its rich features like property types, enums, and descriptions.
### SDK Helpers [#sdk-helpers]
While you can define function schemas directly, [OpenAI's SDKs](https://platform.openai.com/docs/libraries) have helpers to convert `pydantic` and `zod` objects into schemas.
Not all `pydantic` and `zod` features are currently supported by Function Calling, but simple, flat schemas are supported.
Here is an example of how to use the SDK to define a schema.
```typescript TypeScript
import OpenAI from "openai";
import { z } from "zod";
import { zodFunction } from "openai/helpers/zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const GetWeatherParameters = z.object({
location: z.string().describe("City and country e.g. Bogotá, Colombia"),
});
const tools = [
zodFunction({ name: "getWeather", parameters: GetWeatherParameters }),
];
const messages: OpenAI.ChatCompletionMessageParam[] = [
{ role: "user", content: "What's the weather like in Paris today?" },
];
const response = await client.chat.completions.create({
model: "gemma-3-27b-it",
messages,
tools,
});
console.log(response.choices[0].message.tool_calls);
```
```python Python
import os
from openai import OpenAI, pydantic_function_tool
from pydantic import BaseModel, Field
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
class GetWeather(BaseModel):
location: str = Field(
...,
description="City and country e.g. Bogotá, Colombia",
)
tools = [pydantic_function_tool(GetWeather)]
completion = client.chat.completions.create(
model="gemma-3-27b-it",
messages=[
{"role": "system", "content": "You are a helpful assistant that can answer questions and help with tasks."},
{"role": "user", "content": "What's the weather like in Paris today?"},
],
tools=tools,
)
print(completion.choices[0].message.tool_calls)
```
### Best practices for defining functions [#best-practices-for-defining-functions]
1. **Write clear and detailed function names, parameter descriptions, and instructions.**
* **Explicitly describe the purpose of the function and each parameter** (and its format), and what the output represents.
* **Use the system prompt to describe when (and when not) to use each function.** Generally, tell the model *exactly* what to do.
* **Include examples and edge cases**, especially to rectify any recurring failures.
2. **Apply software engineering best practices.**
* **Make the functions obvious and intuitive**. ([principle of least surprise](https://en.wikipedia.org/wiki/Principle_of_least_astonishment))
* **Use enums** and object structure to make invalid states unrepresentable. (e.g. `toggle_light(on: bool, off: bool)` allows for invalid calls)
* **Pass the intern test.** Can an intern/human correctly use the function given nothing but what you gave the model? (If not, what questions do they ask you? Add the answers to the prompt.)
3. **Offload the burden from the model and use code where possible.**
* **Don't make the model fill arguments you already know.** For example, if you already have an `order_id` based on a previous menu, don't have an `order_id` param – instead, have no params `submit_refund()` and pass the `order_id` with code.
* **Combine functions that are always called in sequence.** For example, if you always call `mark_location()` after `query_location()`, just move the marking logic into the query function call.
4. **Keep the number of functions small for higher accuracy.**
* **Evaluate your performance** with different numbers of functions.
* **Aim for fewer than 20 functions** at any one time, though this is just a soft suggestion.
## Streaming [#streaming]
Streaming can be used to surface progress by showing which function is called as the model fills its arguments, and even displaying the arguments in real time.
Streaming function calls is very similar to streaming regular responses: you set `stream` to `true` and get chunks with `delta` objects.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const tools: OpenAI.ChatCompletionTool[] = [{
type: "function",
function: {
name: "get_weather",
description: "Get current temperature for a given location.",
parameters: {
type: "object",
properties: {
location: {
type: "string",
description: "City and country e.g. Bogotá, Colombia",
},
},
required: ["location"],
additionalProperties: false,
},
strict: true,
},
}];
const stream = await client.chat.completions.create({
model: "gemma-3-27b-it",
messages: [{ role: "user", content: "What's the weather like in Paris today?" }],
tools,
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0].delta;
console.log(delta.tool_calls);
}
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia",
},
},
"required": ["location"],
"additionalProperties": False,
},
"strict": True,
},
}]
stream = client.chat.completions.create(
model="gemma-3-27b-it",
messages=[{"role": "user", "content": "What's the weather like in Paris today?"}],
tools=tools,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
print(delta.tool_calls)
```
```bash cURL
curl -N https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-3-27b-it",
"stream": true,
"messages": [
{"role": "user", "content": "What'\''s the weather like in Paris today?"}
],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia"
}
},
"required": ["location"],
"additionalProperties": false
},
"strict": true
}
}]
}'
```
Output of `delta.tool_calls`:
```txt TXT
[{"index": 0, "id": "call_DdmO9pD3xa9XTPNJ32zg2hcA", "function": {"arguments": "", "name": "get_weather"}, "type": "function"}]
[{"index": 0, "id": null, "function": {"arguments": "{\"", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "location", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "\":\"", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "Paris", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": ",", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": " France", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "\"}", "name": null}, "type": null}]
null
```
Instead of aggregating chunks into a single `content` string, however, you're aggregating chunks into an encoded `arguments` JSON object.
When the model calls one or more functions the `tool_calls` field of each `delta` will be populated. Each `tool_call` contains the following fields:
| Field | Description |
| -------- | ------------------------------------------------------- |
| index | Identifies which function call the delta is for |
| id | Tool call id. |
| function | Function call delta (name and arguments) |
| type | Type of tool\_call (always function for function calls) |
Many of these fields are only set for the first `delta` of each tool call, like `id`, `function.name`, and `type`.
Below is a code snippet demonstrating how to aggregate the `delta` objects into a final `tool_calls` object.
```typescript TypeScript
const finalToolCalls: Record = {};
for await (const chunk of stream) {
const toolCalls = chunk.choices[0].delta.tool_calls || [];
for (const toolCall of toolCalls) {
const { index } = toolCall;
if (!finalToolCalls[index]) {
finalToolCalls[index] = toolCall;
}
finalToolCalls[index].function.arguments += toolCall.function.arguments;
}
}
```
```python Python
final_tool_calls = {}
for chunk in stream:
for tool_call in chunk.choices[0].delta.tool_calls or []:
index = tool_call.index
if index not in final_tool_calls:
final_tool_calls[index] = tool_call
final_tool_calls[index].function.arguments += tool_call.function.arguments
```
Accumulated final\_tool\_calls\[0]
```json JSON
{
"index": 0,
"id": "call_RzfkBpJgzeR0S242qfvjadNe",
"function": {
"name": "get_weather",
"arguments": "{\"location\":\"Paris, France\"}"
}
}
```
# Rate Limits (https://docs.inference.net/api/rate-limits)
## Generation Rate Limits [#generation-rate-limits]
Rate limits for model inference requests are based on your account tier:
| Tier | Requests per minute (RPM) |
| ------------------------- | ----------------------------------- |
| Free | 30 |
| Paid (Growth, Enterprise) | 1,000 |
| Custom teams | Operator-managed per-team overrides |
Free-tier teams granted free credits by an operator are raised to **200 RPM** for the duration of the grant. The pay-as-you-go credit-purchase floor is **250 RPM** — a real purchase always buys more headroom than a free grant. Per-team overrides for custom teams are configured by an operator; contact us to request one.
## Batch API Rate Limits [#batch-api-rate-limits]
* **Batch file upload:** 1 per minute
* Batch processing rate limits are separate from generation rate limits. See the [Batch API](/api/async-inference/batch-api) docs for details.
Deployed models share the team's serverless inference RPM bucket rather than having their own per-instance limit.
## Increasing Your Limits [#increasing-your-limits]
If you need higher rate limits, [contact us](mailto:support@inference.net) or use the support chat to request a custom tier.
# Reasoning (https://docs.inference.net/api/reasoning)
Reasoning models think through a problem before they answer. The `reasoning_effort` parameter controls how much thinking a model does. Higher effort improves quality on hard problems and uses more tokens and increases latency.
## Set a reasoning effort [#set-a-reasoning-effort]
Pass `reasoning_effort` on a chat completion request:
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "claude-opus-5",
max_tokens: 20000,
reasoning_effort: "high",
messages: [{ role: "user", content: "Prove that sqrt(2) is irrational." }],
});
console.log(response.choices[0].message.content);
```
```python Python
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="claude-opus-5",
max_tokens=20000,
reasoning_effort="high",
messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational."}],
)
print(response.choices[0].message.content)
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-opus-5",
"max_tokens": 20000,
"reasoning_effort": "high",
"messages": [{"role": "user", "content": "Prove that sqrt(2) is irrational."}]
}'
```
## Effort levels [#effort-levels]
The full set of levels is `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`. `none` disables reasoning. Each model supports a subset:
| Model | Supported levels |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `gpt-5.2`, `gpt-5.4`, `gpt-5.5`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5-mini`, `gpt-5-nano` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh` |
| `claude-fable-5`, `claude-opus-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-haiku-4-5` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` |
| `claude-opus-4-6`, `claude-sonnet-4-6` | `none`, `minimal`, `low`, `medium`, `high`, `max` |
| `gemini-2.5-flash`, `gemini-2.5-flash-lite`, `gemini-3-flash-preview`, `gemini-3.1-flash-lite`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.6-flash`, `glm-5`, `glm-5.2`, `glm-5.2-fast` | `none`, `low`, `medium`, `high` |
| `glm-5.3-flash`, `kimi-k3` | `none`, `low`, `high`, `max` |
| `gemini-2.5-pro` | `minimal`, `low`, `medium`, `high` |
| `gemini-3.1-pro-preview`, `gemini-3.7-flash`, `kimi-k2.5`, `kimi-k2.6`, `kimi-k3-fast`, `nemotron-3-super`, `qwen3-8b`, `qwen3-14b`, `qwen3-30b-a3b`, `qwen3-max-thinking`, `qwen3.7-plus` | `low`, `medium`, `high` |
| `glm-5.3` | `low`, `high`, `max` |
| `deepseek-v4-flash`, `deepseek-v4-flash-0731`, `deepseek-v4-pro`, `deepseek-v4-pro-0813` | `low`, `high`, `max` |
| `grok-4.5`, `grok-4.6` | `low`, `medium`, `high` |
A model absent from this table (or from `reasoning_efforts` in `GET /v1/models`) has no first-party effort dial: the gateway forwards `reasoning_effort` for it, but the platform does not validate or guarantee behavior — several models accept the parameter and silently ignore it (`gpt-4o`, `gpt-4.1`).
You can also discover support programmatically. `GET /v1/models` returns `context_length` and `max_completion_tokens` when the platform knows them, and a `reasoning_efforts` array when the catalog declares the model's levels:
```json
{
"id": "claude-opus-5",
"object": "model",
"owned_by": "system",
"context_length": 1000000,
"max_completion_tokens": 128000,
"reasoning_efforts": ["none", "minimal", "low", "medium", "high", "xhigh", "max"]
}
```
On OpenAI and Anthropic models, the API rejects an unsupported level before dispatch with a 400 that names the valid values. Other models forward the request, and the serving engine decides:
```json
{
"error": {
"message": "reasoning_effort 'xhigh' is not supported by claude-sonnet-4-6. Supported: none, minimal, low, medium, high, max.",
"type": "invalid_request_error",
"param": "reasoning_effort",
"code": "unsupported_value"
}
}
```
`gpt-4o` and `gpt-4.1` accept `reasoning_effort` and ignore it. They are not reasoning models, so `usage.completion_tokens_details.reasoning_tokens` is always 0.
## Anthropic models: max\_tokens must exceed the thinking budget [#anthropic-models-max_tokens-must-exceed-the-thinking-budget]
On Anthropic models, each effort level reserves a thinking budget, and that budget counts toward `max_tokens`. Your `max_tokens` (or `max_completion_tokens`) must be strictly greater than the budget for the level you request:
| Effort | Thinking budget (tokens) |
| --------- | ------------------------ |
| `none` | 0 |
| `minimal` | 1,024 |
| `low` | 1,024 |
| `medium` | 2,048 |
| `high` | 4,096 |
| `xhigh` | 8,192 |
| `max` | 16,384 |
For example, `reasoning_effort: "xhigh"` with `max_tokens: 2000` fails, because the 8,192-token budget does not fit. The API rejects it with a 400 that names both numbers:
```json
{
"error": {
"message": "max_tokens (2000) must be greater than the 8192-token thinking budget that reasoning_effort 'xhigh' enables on claude-haiku-4-5. Increase max_tokens or lower reasoning_effort.",
"type": "invalid_request_error",
"param": "max_tokens",
"code": "invalid_value"
}
}
```
Set `max_tokens` to the budget plus the visible output you want. `max_tokens: 20000` leaves room for every level.
Do not send Anthropic's native `thinking` parameter to `/v1/chat/completions`. This endpoint does not map or validate it. Use `reasoning_effort` instead; the gateway maps it to a thinking budget for you.
## Where reasoning appears in the response [#where-reasoning-appears-in-the-response]
Models differ in whether they return the reasoning text:
* Anthropic models and `glm-5.2` return the reasoning text in `reasoning_content` on the message.
* OpenAI models do not return reasoning text. You only see the count in `usage.completion_tokens_details.reasoning_tokens`.
```json
{
"choices": [
{
"message": {
"role": "assistant",
"content": "Assume sqrt(2) = a/b in lowest terms...",
"reasoning_content": "The user wants a proof by contradiction..."
}
}
],
"usage": {
"completion_tokens_details": { "reasoning_tokens": 412 }
}
}
```
Reasoning tokens bill at the model's reasoning rate when one is listed, otherwise at the model's output rate. See each model's page for rates.
## Verbosity [#verbosity]
The `verbosity` parameter (`low`, `medium`, or `high`) controls how long the visible answer is, independent of how much the model thinks. Only the gpt-5 family supports it.
```json
{
"model": "gpt-5.2",
"reasoning_effort": "high",
"verbosity": "low",
"messages": [{ "role": "user", "content": "Summarize this contract." }]
}
```
## Notes [#notes]
* `reasoning: {"enabled": false}` (the OpenRouter-style object) and `reasoning_effort: "none"` are accepted and translated to the model's native reasoning toggle before dispatch — for models that support one they disable reasoning. Callers that name their own **external** provider or bring their own key (`x-inference-provider` / provider API key headers) keep full control of the request vocabulary: their `reasoning` object is forwarded untouched. Pinning one of our own origins via `x-inference-provider` (e.g. `inference-net`) only selects routing and does not opt out of translation. An explicit `reasoning_effort` or `chat_template_kwargs.enable_thinking` always wins over the `reasoning` object.
* `reasoning_effort` is validated first-party: an unsupported value returns a 400 naming the valid values instead of reaching the provider. Other keys outside the documented schema pass through to the upstream provider without validation — a typo in one of those fails at the provider, not at the gateway.
# Structured Outputs (https://docs.inference.net/api/structured-outputs)
When using Structured Outputs, always include instructions in the system prompt to respond in JSON format.
For example: "You are a helpful assistant. Respond in JSON format."
## Introduction [#introduction]
JSON is one of the most widely used formats in the world for applications to exchange data.
Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied [JSON Schema](https://json-schema.org/overview/what-is-jsonschema), so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value.
Some benefits of Structured Outputs include:
1. **Reliable type-safety:** No need to validate or retry incorrectly formatted responses
2. **Simpler prompting:** No need for strongly worded prompts to achieve consistent formatting
## Getting Started [#getting-started]
You'll need an Inference.net account and API key to use Structured Outputs. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net using the OpenAI SDK, you will need to set the base URL to `https://api.inference.net/v1`.
In this example, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL
export INFERENCE_API_KEY=
# All requests use:
# -H "Authorization: Bearer $INFERENCE_API_KEY"
# -H "Content-Type: application/json"
```
## When to use Structured Outputs [#when-to-use-structured-outputs]
Structured Outputs are suitable when you want to indicate a structured schema for use when the model responds to the user.
For example, if you are building a math tutoring application, you might want the assistant to respond to your user using a specific JSON Schema so that you can generate a UI that displays different parts of the model's output in distinct ways.
Put simply:
* If you are connecting the model to tools, functions, data, etc. in your system, then you should use function calling
* If you want to structure the model's output when it responds to the user, then you should use a structured `response_format`
### Structured Outputs vs JSON mode [#structured-outputs-vs-json-mode]
Structured Outputs is the evolution of [JSON mode](#json-mode). While both ensure valid JSON is produced, only Structured Outputs ensure schema adherance. Both Structured Outputs and JSON mode are supported in the Chat Completions API and Batch API.
We recommend always using Structured Outputs instead of JSON mode when possible.
| | Structured Outputs | JSON Mode |
| ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------ |
| Outputs valid JSON | Yes | Yes |
| Adheres to schema | Yes (see supported schemas) | No |
| Enabling | `response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ... } }` | `response_format: { type: "json_object" }` |
## Example [#example]
### Chain of thought [#chain-of-thought]
You can ask the model to output an answer in a structured, step-by-step way, to guide the user through the solution.
```typescript TypeScript
import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const Step = z.object({
explanation: z.string(),
output: z.string(),
});
const MathReasoning = z.object({
steps: z.array(Step),
final_answer: z.string(),
});
const completion = await client.chat.completions.parse({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format." },
{ role: "user", content: "how can I solve 8x + 7 = -23" },
],
response_format: zodResponseFormat(MathReasoning, "math_reasoning"),
});
const math_reasoning = completion.choices[0].message.parsed;
console.log(math_reasoning);
```
```python Python
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."},
{"role": "user", "content": "how can I solve 8x + 7 = -23"},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "math_reasoning",
"strict": True,
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": {"type": "string"},
"output": {"type": "string"},
},
"required": ["explanation", "output"],
"additionalProperties": False,
},
},
"final_answer": {"type": "string"},
},
"required": ["steps", "final_answer"],
"additionalProperties": False,
},
},
},
)
math_reasoning = json.loads(completion.choices[0].message.content)
print(json.dumps(math_reasoning, indent=2))
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{
"role": "system",
"content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."
},
{
"role": "user",
"content": "how can I solve 8x + 7 = -23"
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "math_reasoning",
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": { "type": "string" },
"output": { "type": "string" }
},
"required": ["explanation", "output"],
"additionalProperties": false
}
},
"final_answer": { "type": "string" }
},
"required": ["steps", "final_answer"],
"additionalProperties": false
},
"strict": true
}
}
}'
```
#### Example response [#example-response]
```json JSON
{
"steps": [
{
"explanation": "Start with the equation 8x + 7 = -23.",
"output": "8x + 7 = -23"
},
{
"explanation": "Subtract 7 from both sides to isolate the term with the variable.",
"output": "8x = -23 - 7"
},
{
"explanation": "Simplify the right side of the equation.",
"output": "8x = -30"
},
{
"explanation": "Divide both sides by 8 to solve for x.",
"output": "x = -30 / 8"
},
{
"explanation": "Simplify the fraction.",
"output": "x = -15 / 4"
}
],
"final_answer": "x = -15 / 4"
}
```
### Defining Schemas with the SDK [#defining-schemas-with-the-sdk]
The OpenAI SDK makes it easy to define object schemas using [Zod](https://zod.dev/) (TypeScript) or [Pydantic](https://docs.pydantic.dev/) (Python). Below, you can see how to extract information from unstructured text that conforms to a schema defined in code.
```typescript TypeScript
import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const CalendarEvent = z.object({
name: z.string(),
date: z.string(),
participants: z.array(z.string()),
});
const completion = await client.chat.completions.parse({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "Extract the event information. Respond in JSON format." },
{ role: "user", content: "Alice and Bob are going to a science fair on Friday." },
],
response_format: zodResponseFormat(CalendarEvent, "event"),
});
const event = completion.choices[0].message.parsed;
console.log(event);
```
```python Python
import os
from pydantic import BaseModel
from openai import OpenAI
class CalendarEvent(BaseModel):
name: str
date: str
participants: list[str]
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.beta.chat.completions.parse(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "Extract the event information. Respond in JSON format."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
],
response_format=CalendarEvent,
)
event = completion.choices[0].message.parsed
print(event.model_dump_json(indent=2))
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "Extract the event information. Respond in JSON format."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "event",
"strict": true,
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"date": { "type": "string" },
"participants": { "type": "array", "items": { "type": "string" } }
},
"required": ["name", "date", "participants"],
"additionalProperties": false
}
}
}
}'
```
Some tips:
* Name keys clearly and intuitively
* Create clear titles and descriptions for important keys in your structure
* Create and use evals to determine the structure that works best for your use case
## Step By Step Example — Parsing The Model's Output [#step-by-step-example--parsing-the-models-output]
You can use the OpenAI SDK to parse the model's output into a typed object automatically.
### Step 1: Define your object [#step-1-define-your-object]
First you must define an object or data structure to represent the JSON Schema that the model should be constrained to follow.
```typescript TypeScript
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
const Step = z.object({
explanation: z.string(),
output: z.string(),
});
const MathResponse = z.object({
steps: z.array(Step),
final_answer: z.string(),
});
```
```python Python
from pydantic import BaseModel
class Step(BaseModel):
explanation: str
output: str
class MathResponse(BaseModel):
steps: list[Step]
final_answer: str
```
### Step 2: Supply your object in the API call [#step-2-supply-your-object-in-the-api-call]
You can use the `parse` method to automatically parse the JSON response into the object you defined.
Under the hood, the SDK takes care of supplying the JSON schema corresponding to your data structure, and then parsing the response as an object.
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const completion = await client.chat.completions.parse({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format." },
{ role: "user", content: "how can I solve 8x + 7 = -23" },
],
response_format: zodResponseFormat(MathResponse, "math_response"),
});
console.log(completion.choices[0].message.parsed);
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.beta.chat.completions.parse(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."},
{"role": "user", "content": "how can I solve 8x + 7 = -23"},
],
response_format=MathResponse,
)
print(completion.choices[0].message.parsed.model_dump_json(indent=2))
```
### Handling edge cases [#handling-edge-cases]
In some cases, the model might not generate a valid response that matches the provided JSON schema.
This can happen if for example you reach a max tokens limit and the response is incomplete.
```typescript TypeScript
import OpenAI from "openai";
import { z } from "zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const Step = z.object({
explanation: z.string(),
output: z.string(),
});
const MathResponse = z.object({
steps: z.array(Step),
final_answer: z.string(),
});
try {
const completion = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{
role: "system",
content: "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format.",
},
{
role: "user",
content: "how can I solve 8x + 7 = -23",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "math_response",
schema: {
type: "object",
properties: {
steps: {
type: "array",
items: {
type: "object",
properties: {
explanation: { type: "string" },
output: { type: "string" },
},
required: ["explanation", "output"],
additionalProperties: false,
},
},
final_answer: { type: "string" },
},
required: ["steps", "final_answer"],
additionalProperties: false,
},
strict: true,
},
},
max_tokens: 20, // intentionally low to demonstrate truncation handling
});
const message = completion.choices[0].message;
if (completion.choices[0].finish_reason === "length") {
console.log("Model did not return a complete response — parsing may fail.");
throw new Error("Incomplete response");
}
try {
const parsed = MathResponse.parse(JSON.parse(message.content));
console.log("Parsed math response:", parsed);
} catch (zodError) {
console.error("Response does not match the expected schema:", message.content);
}
} catch (e) {
console.error(e);
}
```
```python Python
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
try:
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{
"role": "system",
"content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format.",
},
{"role": "user", "content": "how can I solve 8x + 7 = -23"},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "math_response",
"strict": True,
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": {"type": "string"},
"output": {"type": "string"},
},
"required": ["explanation", "output"],
"additionalProperties": False,
},
},
"final_answer": {"type": "string"},
},
"required": ["steps", "final_answer"],
"additionalProperties": False,
},
},
},
max_tokens=20, # intentionally low to demonstrate truncation handling
)
if completion.choices[0].finish_reason == "length":
raise Exception("Incomplete response")
math_response = json.loads(completion.choices[0].message.content)
print(json.dumps(math_response, indent=2))
except Exception as e:
print(str(e))
```
```bash cURL
# Use max_tokens to demonstrate truncation handling.
# Check finish_reason in the response — "length" means the output was truncated.
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{
"role": "system",
"content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."
},
{
"role": "user",
"content": "how can I solve 8x + 7 = -23"
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "math_response",
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": { "type": "string" },
"output": { "type": "string" }
},
"required": ["explanation", "output"],
"additionalProperties": false
}
},
"final_answer": { "type": "string" }
},
"required": ["steps", "final_answer"],
"additionalProperties": false
},
"strict": true
}
},
"max_tokens": 20
}'
```
## Streaming [#streaming]
You can use streaming to process model responses as they are being generated, and parse them as structured data.
That way, you don't have to wait for the entire response to complete before handling it. This is particularly useful if you would like to display JSON fields one by one, or handle function call arguments as soon as they are available.
```typescript TypeScript
import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const EntitiesSchema = z.object({
attributes: z.array(z.string()),
colors: z.array(z.string()),
animals: z.array(z.string()),
});
const stream = client.chat.completions
.stream({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "Extract entities from the input text. Respond in JSON format." },
{
role: "user",
content: "The quick brown fox jumps over the lazy dog with piercing blue eyes",
},
],
response_format: zodResponseFormat(EntitiesSchema, "entities"),
})
.on("content.delta", ({ snapshot, parsed }) => {
console.log("content:", snapshot);
console.log("parsed:", parsed);
console.log();
})
.on("content.done", (props) => {
console.log(props);
});
await stream.done();
const finalCompletion = await stream.finalChatCompletion();
console.log(finalCompletion);
```
```python Python
import os
from typing import List
from pydantic import BaseModel
from openai import OpenAI
class EntitiesModel(BaseModel):
attributes: List[str]
colors: List[str]
animals: List[str]
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
with client.chat.completions.stream(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{
"role": "system",
"content": "Extract entities from the input text. Respond in JSON format.",
},
{
"role": "user",
"content": "The quick brown fox jumps over the lazy dog with piercing blue eyes",
},
],
response_format=EntitiesModel,
) as stream:
for event in stream:
if event.type == "content.delta":
if event.parsed is not None:
print("content.delta parsed:", event.parsed)
elif event.type == "content.done":
print("content.done")
elif event.type == "error":
print("Error in stream:", event.error)
final_completion = stream.get_final_completion()
print("Final completion:", final_completion)
```
```bash cURL
# Streaming structured outputs works the same as regular streaming.
# Add "stream": true and parse SSE chunks as they arrive.
curl -N https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"stream": true,
"messages": [
{"role": "system", "content": "Extract entities from the input text. Respond in JSON format."},
{"role": "user", "content": "The quick brown fox jumps over the lazy dog with piercing blue eyes"}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "entities",
"strict": true,
"schema": {
"type": "object",
"properties": {
"attributes": { "type": "array", "items": { "type": "string" } },
"colors": { "type": "array", "items": { "type": "string" } },
"animals": { "type": "array", "items": { "type": "string" } }
},
"required": ["attributes", "colors", "animals"],
"additionalProperties": false
}
}
}
}'
```
## Supported schemas [#supported-schemas]
Structured Outputs supports a subset of the [JSON Schema](https://json-schema.org/docs) language.
#### Supported types [#supported-types]
The following types are supported for Structured Outputs:
* String
* Number
* Boolean
* Integer
* Object
* Array
* Enum
* anyOf
#### Required Fields And Additional Properties [#required-fields-and-additional-properties]
To use Structured Outputs, all properties on all objects must be specified as `required`.
Also, `additionalProperties` must be set to `false`.
In the following example, note how both `location` and `unit` are listed as required properties.
```json JSON
{
"name": "get_weather",
"description": "Fetches the weather in the given location",
"strict": true,
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The location to get the weather for"
},
"unit": {
"type": "string",
"description": "The unit to return the temperature in",
"enum": ["F", "C"]
}
},
"additionalProperties": false,
"required": ["location", "unit"]
}
}
```
#### Schema Limitations Depend on the Model [#schema-limitations-depend-on-the-model]
Limitations on the number of properties, enum values, and total string size may vary depending on the model you are using.
#### Key ordering [#key-ordering]
When using Structured Outputs, outputs will be produced in the same order as the ordering of keys in the schema.
## JSON mode [#json-mode]
When using JSON mode, always instruct the model to produce JSON in the system prompt.
For example: "You are a helpful assistant. Respond in JSON format."
JSON mode is a more basic version of the Structured Outputs feature. While JSON mode ensures that model output is valid JSON, Structured Outputs reliably matches the model's output to the schema you specify. We recommend you use Structured Outputs if it is supported for your use case.
When JSON mode is turned on, the model's output is ensured to be valid JSON, except for in some edge cases that you should detect and handle appropriately.
To turn on JSON mode with the Chat Completions API you can set the `response_format` to `{ "type": "json_object" }`.
Important notes:
* When using JSON mode, you must always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate non-JSON or an unending stream of whitespace.
* JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. You should use Structured Outputs to ensure it matches your schema, or if that is not possible, you should use a validation library and potentially retries to ensure that the output matches your desired schema.
* Your application must detect and handle the edge cases that can result in the model output not being a complete JSON object (see below)
* Some models will include a triple backtick / JSON code format block around the JSON response. This should be detected and handled appropriately.
### Handling JSON Mode edge cases [#handling-json-mode-edge-cases]
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
try {
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{
role: "system",
content: "You are a helpful assistant designed to output JSON.",
},
{ role: "user", content: "Who won the world series in 2020? Please respond in the format {winner: ...}" },
],
response_format: { type: "json_object" },
});
// Check if the response was truncated due to context length
if (response.choices[0].finish_reason === "length") {
// Handle incomplete JSON
}
// Check if the model refused the request
if (response.choices[0].message.refusal) {
console.log(response.choices[0].message.refusal);
}
// Check if content was filtered
if (response.choices[0].finish_reason === "content_filter") {
// Handle filtered content
}
if (response.choices[0].finish_reason === "stop") {
console.log(JSON.parse(response.choices[0].message.content));
}
} catch (e) {
console.error(e);
}
```
```python Python
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
try:
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful assistant designed to output JSON."},
{"role": "user", "content": "Who won the world series in 2020? Please respond in the format {winner: ...}"},
],
response_format={"type": "json_object"},
)
message = response.choices[0]
# Check if the response was truncated due to context length
if message.finish_reason == "length":
raise Exception("Incomplete response — output was truncated")
# Check if the model refused the request
if hasattr(message.message, "refusal") and message.message.refusal:
print(message.message.refusal)
# Check if content was filtered
if message.finish_reason == "content_filter":
raise Exception("Response filtered")
if message.finish_reason == "stop":
print(json.loads(message.message.content))
except Exception as e:
print(str(e))
```
```bash cURL
# JSON mode is enabled with response_format type "json_object".
# Check finish_reason in the response to handle edge cases:
# "stop" — complete JSON returned
# "length" — output truncated, JSON may be incomplete
# "content_filter" — content was filtered
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant designed to output JSON."},
{"role": "user", "content": "Who won the world series in 2020? Please respond in the format {winner: ...}"}
],
"response_format": { "type": "json_object" }
}'
```
# Vision (https://docs.inference.net/api/vision)
## Introduction [#introduction]
**Vision Models** are *multi-modal models* that accept both text and images as input.
You can use vision models to extract information from images (for example, by asking the model to describe the image).
This guide explains how to use Vision Models with the Inference API.
## Getting Started [#getting-started]
You'll need an Inference.net account and API key. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net using the OpenAI SDK, you will need to set the base URL to `https://api.inference.net/v1`.
In the following examples, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
## Step By Step Example [#step-by-step-example]
To use image inputs with the Inference API:
1. Encode your image as a base64 string
2. Include the base64 string in a [Data URI](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data) with an image mimetype (e.g. `image/png`)
3. Include the Data URI in the `content` array of a user message
4. Send the request to the Inference API and inspect the response
### Step 1: Encode your image as a Data URI [#step-1-encode-your-image-as-a-data-uri]
```typescript TypeScript
const url = "https://upload.wikimedia.org/wikipedia/commons/3/3f/Crystal_Project_bug.png";
const response = await fetch(url);
const buffer = Buffer.from(await response.arrayBuffer());
const base64 = buffer.toString("base64");
const dataUri = `data:image/png;base64,${base64}`;
```
```python Python
import base64
import requests
url = "https://upload.wikimedia.org/wikipedia/commons/3/3f/Crystal_Project_bug.png"
response = requests.get(url)
image_data = response.content
encoded_string = base64.b64encode(image_data).decode("utf-8")
data_uri = f"data:image/png;base64,{encoded_string}"
```
```bash cURL
# Encode an image file to a base64 data URI:
DATA_URI="data:image/png;base64,$(base64 -i image.png)"
# Or fetch and encode from a URL:
DATA_URI="data:image/png;base64,$(curl -s https://upload.wikimedia.org/wikipedia/commons/3/3f/Crystal_Project_bug.png | base64)"
```
### Step 2: Structure and send your request [#step-2-structure-and-send-your-request]
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const completion = await client.chat.completions.create({
model: "google/gemma-3-27b-it",
messages: [
{
role: "system",
content: "You are a helpful assistant that can answer questions about the image.",
},
{
role: "user",
content: [
{
type: "image_url",
image_url: { url: dataUri },
},
{
type: "text",
text: "What is in this image?",
},
],
},
],
});
console.log(completion.choices[0].message.content);
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.chat.completions.create(
model="google/gemma-3-27b-it",
messages=[
{
"role": "system",
"content": "You are a helpful assistant that can answer questions about the image.",
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": data_uri},
},
{
"type": "text",
"text": "What is in this image?",
},
],
},
],
)
print(completion.choices[0].message.content)
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-it",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant that can answer questions about the image."
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": { "url": "'"$DATA_URI"'" }
},
{
"type": "text",
"text": "What is in this image?"
}
]
}
]
}'
```
## Limitations [#limitations]
* We do not support sending images from a URL directly into the request body.
* Supported image formats include webp, png, gif and jpg/jpeg.
* The total request body can be up to 12 MiB (12,582,912 bytes), including inline images.
## Token Usage [#token-usage]
Using images in a request counts towards the total token usage for a request.
The exact token count will vary, but a handy approximation of the number of tokens used by an image is the following formula:
```
h = max(1, min(2, HEIGHT / 560))
w = max(1, min(2, WIDTH / 560))
tokens = h * w * 1601
```
In plain English:
1. The image height and width in pixels are both divided by 560
2. The resulting height and width are clamped between 1 and 2
3. Finally, the height and width are multiplied together and then multiplied by 1,601
Here is a table of image dimensions and their corresponding estimated token counts:
| Height | Width | Tokens | Note |
| ------ | ------ | ------ | -------------------------------------------------------------- |
| 32px | 32px | 1,601 | Images smaller than 560x560 are still considered 560x560 |
| 560px | 560px | 1,601 | |
| 1120px | 1120px | 6,404 | 6,404 is the approximate maximum token usage of a single image |
The above formula and table is an approximation. We suggest that you:
* Explicitly check your image dimensions before submitting them to the API to avoid high token usage.
* Monitor your token usage and adjust your requests if necessary.
See the [Models](https://inference.net/models) page for current pricing per token for Vision Models.
# Changelog (https://docs.inference.net/changelog/overview)
# Changelog [#changelog]
Updates and changes to Inference are published on the [Inference blog](https://inference.net/blog/), with the latest releases tracked on [GitHub](https://github.com/context-labs). New product features also ship in the dashboard and are announced in the blog.
# Account Deletion (https://docs.inference.net/cli/account-deletion)
Schedule permanent deletion of your account, a project, or a team. These commands preview what will be deleted first and schedule the deletion inside a grace window so you can cancel it.
Deleting an account, project, or team is **permanent and irreversible**. Data is not recoverable once the grace window elapses. These commands are destructive — use them only when you are certain.
## How deletion works [#how-deletion-works]
Deletion is **scheduled** first, giving you a grace window. Inside that window you can `cancel-project` to abort it. Once the window elapses, the deletion proceeds and the data is permanently gone.
## Preview before you delete [#preview-before-you-delete]
```bash
# Preview blockers and affected teams for account deletion
inf account-deletion preview
# Preview blockers and members for one team's deletion
inf account-deletion get-team-preview --team-id
# Check whether the account-deletion flow is enabled for the caller
inf account-deletion enabled
```
## Scheduling deletions [#scheduling-deletions]
```bash
# Schedule deletion of the caller's account
inf account-deletion delete-account
# Schedule deletion of a project and all its data
inf account-deletion delete-project --project-id
# Schedule deletion of a team
inf account-deletion delete-team --team-id
```
## Cancel a scheduled deletion [#cancel-a-scheduled-deletion]
```bash
# Cancel a scheduled project deletion inside its grace window
inf account-deletion cancel-project --project-id
```
## Transfer ownership instead [#transfer-ownership-instead]
If you're leaving a team, prefer transferring ownership to another member rather than deleting the team.
```bash
inf account-deletion transfer-ownership --team-id --to-user-id
```
Each verb's full flag list is shown by `inf account-deletion --help`.
# Agents (https://docs.inference.net/cli/agent)
Inspect the agents auto-discovered from your spans and their execution stats. Agents are read-only — there are no agent mutations.
## Commands [#commands]
Run `inf agent --help` for each verb's options. The generated surface covers:
| Verb | Description |
| ------------------------ | ------------------------------------------------------------ |
| `list` | List a project's agents with rolled-up stats |
| `summary-cards` | List a project's agents as summary cards with sparklines |
| `index-cards` | List the paginated agent index with rolled-up stats |
| `index-initial-data` | The first agent-index page plus its execution series |
| `get` | Get an agent's identity by id |
| `stats` | Get an agent's rolled-up stats over the read window |
| `overview` | Get an agent's overview: summary, series, model/tool profile |
| `timeseries` | Get an agent's per-day metric time series |
| `index-execution-series` | Per-agent execution-count series for the index |
## Related commands [#related-commands]
* [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) inspect the underlying data agents are derived from.
* [`inf signals`](/cli/signals) attach classifiers to an agent's spans/traces/sessions.
* [`inf analyze-harness agent list`](/cli/analyze-harness) lists the agents HALO sees for trace analysis.
# Analyze Harness (https://docs.inference.net/cli/analyze-harness)
HALO analyzes your agent's traces and produces a markdown report that flags anomalies, errors, inefficiencies, and opportunities to improve reliability, latency, cost, and tool usage. The report is written to read like a brief you can paste straight into a coding agent to apply the fixes.
Use `inf analyze-harness` to kick off a one-off analysis, schedule recurring ones, watch a run reach completion, and read the resulting report. The old name `inf halo` remains as an alias.
## How HALO is organized [#how-halo-is-organized]
* A **run** is one analysis of one agent over one time window. Each completed run produces exactly one report.
* A **conversation** is the thread a run lives in. The first assistant message is the original report; any follow-up questions you ask add more runs and more assistant messages to the same conversation.
* A **schedule** fires recurring runs for one agent on a cadence (hourly, daily, weekly, or monthly).
The report markdown is the assistant message content inside a conversation. There is no separate "report" artifact to download, so reading a report means reading its conversation.
## Quickstart: run an analysis and read the report [#quickstart-run-an-analysis-and-read-the-report]
```bash
# 1. Find the agent you want to analyze
inf analyze-harness agent list
# 2. Start a run for that agent over the last 24h
inf analyze-harness run create --agent-uuid
# 3. Wait for it to finish (prints the run id from step 2)
inf analyze-harness run poll
# 4. Read the report (the assistant message in the conversation)
inf analyze-harness conversation get
```
`inf analyze-harness run create` prints both the `run-id` and the `conversation-id`. The report prints in full from `inf analyze-harness conversation get` — no `--json` required. Add `--json` only when you want the raw structured payload for scripting.
## `inf analyze-harness agent` [#inf-analyze-harness-agent]
List the agents HALO sees in your project's recent traces. You need an agent's UUID to start a run or create a schedule.
```bash
inf analyze-harness agent list
```
**Alias:** `inf analyze-harness agent ls`
### Options [#options]
| Flag | Required | Description | Default |
| ---------------------- | -------- | ----------------------------------------------------------------------------------------- | ------- |
| `--time-range ` | No | Trace window to scan for agents: `1h`, `6h`, `12h`, `1d`, `3d`, `7d`, `14d`, `30d`, `90d` | `30d` |
## `inf analyze-harness run` [#inf-analyze-harness-run]
Create and inspect HALO runs.
### `inf analyze-harness run create` [#inf-analyze-harness-run-create]
Start a manual analysis for one agent over a time window.
```bash
inf analyze-harness run create --agent-uuid
```
In an interactive terminal you can omit `--agent-uuid` to pick from recent agents. In non-interactive or `--json` mode, `--agent-uuid` is required.
#### Options [#options-1]
| Flag | Required | Description | Default |
| ---------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------- |
| `--agent-uuid ` | No (required non-interactively) | Agent UUID from `inf analyze-harness agent list` | Interactive picker |
| `--prompt ` | No | Prompt steering the analysis | Default HALO prompt |
| `--window-start-at ` | No | Window start, ISO datetime | Derived from `--lookback-hours` |
| `--window-end-at ` | No | Window end, ISO datetime | Now |
| `--lookback-hours ` | No | Window length when `--window-start-at` is omitted | `24` |
| `--model-id ` | No | Catalog HALO model id | Catalog default |
| `--span-limit ` | No | Per-run span cap — omit to use the platform default | Platform default |
| `--max-subagent-depth ` | No | Engine recursion ceiling (1–5) | `1` |
| `--max-turns ` | No | Per-agent turn ceiling (1–100) | `20` |
| `--reasoning-effort ` | No | Reasoning effort for the analysis model: `none`, `minimal`, `low`, `medium`, `high`, or `xhigh` | `medium` |
| `--subagent-model-id ` | No | Catalog model id for subagent executions — omit to run subagents on the analysis model | Analysis model |
| `--debug` | No | Enable debug logging for the HALO runtime | Off |
| `--agent-time-range ` | No | Trace window preset used when picking an agent interactively | `30d` |
### `inf analyze-harness run poll` [#inf-analyze-harness-run-poll]
Poll a run until it reaches a terminal state (`completed`, `failed`, `cancelled`, `timed_out`, or `no_traces`). Exits `0` only on `completed`.
```bash
inf analyze-harness run poll
```
#### Options [#options-2]
| Flag | Required | Description | Default |
| ---------------------- | -------- | ------------- | --------------- |
| `--interval ` | No | Poll interval | `5` |
| `--timeout ` | No | Total timeout | `1800` (30 min) |
### `inf analyze-harness run get` [#inf-analyze-harness-run-get]
Fetch a run's status and its referenced trace dataset. This returns run metadata, not the report text — read the conversation for the report.
```bash
inf analyze-harness run get
```
### `inf analyze-harness run events` [#inf-analyze-harness-run-events]
List the structured event timeline for a run (started, heartbeat, agent steps, completed, failed).
```bash
inf analyze-harness run events
```
#### Options [#options-3]
| Flag | Required | Description | Default |
| ------------- | -------- | -------------------- | ------- |
| `--limit ` | No | Max events to return | `100` |
### `inf analyze-harness run cancel` [#inf-analyze-harness-run-cancel]
Request cancellation of an in-flight run.
```bash
inf analyze-harness run cancel --reason "no longer needed"
```
#### Options [#options-4]
| Flag | Required | Description | Default |
| ----------------- | -------- | ------------------------------------ | ---------------- |
| `--reason ` | No | Reason recorded for the cancellation | `user-cancelled` |
## `inf analyze-harness conversation` [#inf-analyze-harness-conversation]
Inspect HALO conversations and read their reports.
**Alias:** `inf analyze-harness conv`
### `inf analyze-harness conversation list` [#inf-analyze-harness-conversation-list]
List conversations in the active project, newest first.
```bash
inf analyze-harness conversation list
```
**Alias:** `inf analyze-harness conversation ls`
#### Options [#options-5]
| Flag | Required | Description | Default |
| ------------- | -------- | --------------------------- | ------- |
| `--limit ` | No | Max conversations to return | `50` |
### `inf analyze-harness conversation get` [#inf-analyze-harness-conversation-get]
Print a conversation with its messages, runs, and trace datasets. The assistant messages are the report markdown — the first assistant message is the original report; later ones answer follow-up questions in the same thread. Message content prints in full.
```bash
inf analyze-harness conversation get
```
Add `--json` to get the raw payload (every message, run, and trace dataset) for scripting.
## `inf analyze-harness schedule` [#inf-analyze-harness-schedule]
Manage recurring HALO analyses. Each schedule targets one agent and fires runs on a cadence.
### `inf analyze-harness schedule list` [#inf-analyze-harness-schedule-list]
```bash
inf analyze-harness schedule list
```
**Alias:** `inf analyze-harness schedule ls`
| Flag | Required | Description | Default |
| -------------------- | -------- | -------------------------- | ------- |
| `--include-archived` | No | Include archived schedules | `false` |
### `inf analyze-harness schedule get` [#inf-analyze-harness-schedule-get]
```bash
inf analyze-harness schedule get
```
### `inf analyze-harness schedule runs` [#inf-analyze-harness-schedule-runs]
List recent runs fired by a schedule.
```bash
inf analyze-harness schedule runs
```
| Flag | Required | Description | Default |
| ------------- | -------- | ------------------ | ------- |
| `--limit ` | No | Max runs to return | `50` |
### `inf analyze-harness schedule create` [#inf-analyze-harness-schedule-create]
```bash
inf analyze-harness schedule create \
--title "Daily checkout-agent review" \
--agent-uuid \
--frequency daily \
--hours 9 \
--minutes 0 \
--timezone America/Los_Angeles
```
| Flag | Required | Description | Default |
| -------------------------- | ------------------------ | ------------------------------------------------------------------- | ------------------- |
| `--title ` | Yes | Human-readable schedule title | - |
| `--agent-uuid <id>` | Yes | Agent the schedule analyzes (from `inf analyze-harness agent list`) | - |
| `--frequency <freq>` | Yes | `hourly`, `daily`, `weekly`, or `monthly` | - |
| `--prompt <text>` | No | Prompt steering each run | Default HALO prompt |
| `--hours <list>` | For daily/weekly/monthly | Comma-separated hours (0–23) | - |
| `--minutes <list>` | Yes | Comma-separated minutes (0–59) | - |
| `--days-of-week <list>` | For weekly | Comma-separated days (0=Sun…6=Sat) | - |
| `--days-of-month <list>` | For monthly | Comma-separated days (1–31) | - |
| `--timezone <tz>` | No | IANA timezone | `UTC` |
| `--model-id <id>` | No | Catalog HALO model id | Catalog default |
| `--span-limit <n>` | No | Per-run span cap | `10000` |
| `--lookback-hours <n>` | No | How far back each run looks, independent of cadence | `24` |
| `--max-subagent-depth <n>` | No | Engine recursion ceiling (1–5) | `2` |
| `--max-turns <n>` | No | Per-agent turn ceiling (1–100) | `20` |
| `--enabled <bool>` | No | Whether the schedule fires | `true` |
### `inf analyze-harness schedule update` [#inf-analyze-harness-schedule-update]
Update mutable fields on a schedule. Accepts the same flags as `create` (all optional), keyed by `<schedule-id>`.
<Metadata text="cli/analyze-harness/schedule-update" />
```bash
# Pause a schedule
inf analyze-harness schedule update <schedule-id> --enabled false
# Repoint it at a different agent and widen the window
inf analyze-harness schedule update <schedule-id> --agent-uuid <other-agent> --lookback-hours 168
```
### `inf analyze-harness schedule archive` / `unarchive` [#inf-analyze-harness-schedule-archive--unarchive]
Archive a schedule so it stops firing, or restore it later.
<Metadata text="cli/analyze-harness/schedule-archive" />
```bash
inf analyze-harness schedule archive <schedule-id>
inf analyze-harness schedule unarchive <schedule-id>
```
`archive` prompts for confirmation. Pass `-y` / `--yes` to skip the prompt.
## Common workflows [#common-workflows]
<Metadata text="cli/analyze-harness/workflows" />
```bash
# Run an analysis end-to-end and capture the report markdown to a file
RUN=$(inf analyze-harness run create --agent-uuid <agent-uuid> --json)
RUN_ID=$(echo "$RUN" | jq -r '.runId')
CONV_ID=$(echo "$RUN" | jq -r '.conversationId')
inf analyze-harness run poll "$RUN_ID"
# Extract just the original report (first assistant message) as markdown
inf analyze-harness conversation get "$CONV_ID" --json \
| jq -r '.messages | map(select(.role == "assistant"))[0].content' \
> halo-report.md
```
## Related commands [#related-commands]
* [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) inspect the underlying traces HALO analyzes.
* The [MCP server](/integrations/mcp-server) exposes the same reports to AI coding assistants via `list_halo_conversations` and `get_halo_conversation`.
# API Keys (https://docs.inference.net/cli/api-key)
Project API keys let you authenticate CI pipelines and other headless environments. See [API Keys and Authentication](/reference/api-keys) for how keys work across the platform. These commands manage the same keys from the terminal.
<Note>
API key management requires session login (`inf auth login`). A project API key cannot create, list, or revoke other keys.
</Note>
## `inf api-key create` [#inf-api-key-create]
Create a project-scoped API key. The raw key value is shown once, at creation. Save it immediately, since it cannot be retrieved again.
<Metadata text="cli/api-key/create" />
```bash
inf api-key create <name> --project <project-id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------------------- |
| `name` | Yes | Human-readable name for the key, e.g. `production-ci` |
### Options [#options]
| Option | Required | Description | Default |
| ---------------------- | -------- | ----------------------------------------------------- | -------------- |
| `--project <id>` | Yes\* | Project the key is scoped to | Active project |
| `--team <id>` | No | Team that owns the key | Active team |
| `--permissions <list>` | No | Comma-separated permissions: `read`, `write`, or both | `read,write` |
\* Required unless `INF_PROJECT_ID` is set or an active project is already set (see [Projects](/cli/projects)).
<Note>
Creating an API key may require a verified phone number on your account. If you see a verification error, add a phone number from the dashboard and try again.
</Note>
### Examples [#examples]
<Metadata text="cli/api-key/create-examples" />
```bash
# Read and write key for the active project
inf api-key create production-ci --project proj_789ghi012jkl
# Read-only key
inf api-key create readonly-metrics --project proj_789ghi012jkl --permissions read
# Key for a specific team
inf api-key create ci-key --project proj_789ghi012jkl --team team_abc123
```
## `inf api-key list` [#inf-api-key-list]
List API keys for a team. The raw key value is never shown again after creation, only its ID, name, and scopes.
<Metadata text="cli/api-key/list" />
```bash
inf api-key list
```
**Alias:** `inf api-key ls`
### Options [#options-1]
| Option | Required | Description | Default |
| ------------- | -------- | --------------------- | ----------- |
| `--team <id>` | No | Team to list keys for | Active team |
### Example [#example]
<Metadata text="cli/api-key/list-examples" />
```bash
inf api-key list --team team_abc123
```
## `inf api-key show` [#inf-api-key-show]
Describe the default API key for the active (or specified) project. The value is masked — plaintext is shown only at creation.
<Metadata text="cli/api-key/show" />
```bash
inf api-key show
```
### Options [#options-2]
| Option | Required | Description | Default |
| ---------------- | -------- | --------------------------------- | -------------- |
| `--project <id>` | No | Project whose default key to show | Active project |
| `--team <id>` | No | Team that owns the project | Active team |
Prints the key's ID, name, masked preview, project, team, scopes, and creation date. Because stored keys are hashed, the full key cannot be printed — create a new one with `inf api-key create <name>`.
### Example [#example-1]
<Metadata text="cli/api-key/show-examples" />
```bash
inf api-key show --project proj_789ghi012jkl
```
## `inf api-key revoke` [#inf-api-key-revoke]
Revoke an API key so it can no longer authenticate requests. This cannot be undone.
<Metadata text="cli/api-key/revoke" />
```bash
inf api-key revoke <id>
```
**Alias:** `inf api-key rm`
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | -------------------------------------------------------------------------- |
| `id` | No | API key ID to revoke (from `inf api-key list`); omit to pick interactively |
### Options [#options-3]
| Option | Required | Description | Default |
| ------------- | -------- | ---------------------- | ----------- |
| `--team <id>` | No | Team that owns the key | Active team |
### Example [#example-2]
<Metadata text="cli/api-key/revoke-examples" />
```bash
inf api-key revoke key_abc123
# Omit the id to pick a key to revoke interactively
inf api-key revoke
```
# Authentication (https://docs.inference.net/cli/authentication)
The CLI supports two auth modes: browser-based **session login** for interactive use and a **project API key** for CI and headless environments.
## `inf auth login` [#inf-auth-login]
Sign in through your browser using the OAuth 2.0 device authorization flow. Opens a verification URL, displays a user code, and polls until you approve in the browser.
<Metadata text="cli/auth/login" />
```bash
inf auth login
```
After sign-in, the CLI stores a session token in `~/.inf/config.json` and activates an organization (team). If your account belongs to multiple teams, `inf auth login` prompts you to choose one in an interactive terminal.
### Options [#options]
| Option | Required | Description |
| --------------------- | -------- | -------------------------------------------------------- |
| `--team <id-or-slug>` | No | Activate a specific team by ID, slug, or exact team name |
### Team selection [#team-selection]
Use `--team` when you know which team you want to activate, or when running in a non-interactive shell.
<Metadata text="cli/auth/login-team" />
```bash
inf auth login --team acme
```
You can pass a team ID, slug, or exact team name:
<Metadata text="cli/auth/login-team-examples" />
```bash
inf auth login --team team_abc123
inf auth login --team acme
inf auth login --team "Acme Research"
```
If the selected team is different from the previously active team, the CLI clears the stored active project and then tries to auto-select a project from the newly active team. You can always run `inf project list` and `inf project switch <id>` to pick a different project.
<Tip>
`inf auth login --team` only sets the team activated at sign-in. Once signed in, you have two ways to work with a different team without signing in again: `inf team switch <id-or-slug>` stores a new active team in your config, while the global `-t, --team <id>` flag (or `INF_TEAM_ID`) overrides the team for a single command only and does not change your stored active team. See [Teams](/cli/team).
</Tip>
<Note>
In non-interactive environments, `inf auth login` cannot prompt for a team. If you belong to multiple teams and omit `--team`, the CLI falls back to the first team returned by the auth API. Pass `--team <id-or-slug>` to make the selected team deterministic.
</Note>
<Note>
Session login requires a browser, so `inf auth login` is not suitable for CI or other headless environments. Use `inf auth set-key` or the `INF_API_KEY` env var there instead.
</Note>
## `inf auth set-key` [#inf-auth-set-key]
Store a project API key on disk for headless or CI authentication.
<Metadata text="cli/auth/set-key" />
```bash
inf auth set-key <key>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------------- |
| `key` | Yes | A project API key (starts with `sk-inference-`) |
After saving, the CLI validates the key by fetching the project list. A successful fetch auto-selects the first project as active.
### Example [#example]
<Metadata text="cli/auth/set-key-examples" />
```bash
inf auth set-key sk-inference-...
```
## `inf auth status` [#inf-auth-status]
Show who you're signed in as, which auth method the CLI is using, the active team and project, and the API URL.
<Metadata text="cli/auth/status" />
```bash
inf auth status
```
## `inf auth whoami` [#inf-auth-whoami]
Show your user ID, email, name, and the team the next command will target.
<Metadata text="cli/auth/whoami" />
```bash
inf auth whoami
```
Use `inf auth status` for a broader view that also includes the auth method and active project. Use `whoami` when you only need identity and team, for example in a script.
<Note>
With a project API key, `whoami` shows your user ID and team but not your email or name, since a project API key is not tied to a specific person.
</Note>
## `inf auth logout` [#inf-auth-logout]
Sign out, clear the session token / API key from `~/.inf/config.json`, and forget the active project and team.
<Metadata text="cli/auth/logout" />
```bash
inf auth logout
```
## Credential resolution order [#credential-resolution-order]
When multiple credentials are present, the CLI picks the first match:
1. `INF_API_KEY` environment variable
2. API key stored via `inf auth set-key`
3. Session token stored via `inf auth login`
<Note>
Coding-agent setup lives in the [`fast` CLI](https://fast.inference.net/coding-agents), which mints its own machine-scoped key at `fast <agent> on`.
</Note>
<Warning>
`inf instrument` is the one exception — it **rejects** `INF_API_KEY` and requires a session login, because it mints a fresh API key for your project on your behalf. Unset `INF_API_KEY` and run `inf auth login` before running `inf instrument`.
</Warning>
## Configuration [#configuration]
The CLI stores configuration at `~/.inf/config.json`, created automatically on first login. Tokens are stored with `0600` permissions.
### Environment variables [#environment-variables]
| Variable | Description | Default |
| ---------------- | -------------------------------------------------------------------------------------- | ----------------------------- |
| `INF_API_KEY` | API key for authentication. Takes precedence over stored credentials | — |
| `INF_API_URL` | Override the API base URL | `https://relay.inference.net` |
| `INF_PROJECT_ID` | Override the active project for any invocation (equivalent to `--project <id>` global) | — |
| `INF_TEAM_ID` | Override the active team for any invocation (equivalent to `--team <id>` global) | — |
<Note>
`--team` and `INF_TEAM_ID` take a team ID, not a slug or name. Use `inf team switch <id-or-slug>` once to resolve a slug or name to an ID, or run `inf team list` to look one up.
</Note>
# Compare (https://docs.inference.net/cli/compare)
AutoEvals runs automated evaluations over an agent's recent traffic. Use `inf compare` to inspect an analysis run's status and outputs, list the eval run groups it launched, and poll a run to completion. The old name `inf autoevals` remains as an alias.
## Quickstart [#quickstart]
<Metadata text="cli/compare/quickstart" />
```bash
# Inspect a run and its eval run groups
inf compare get <run-id>
inf compare groups <run-id>
# Poll a run to completion (exits 0 on success)
inf compare poll <run-id>
```
## `inf compare get` [#inf-compare-get]
Show one analysis run's status, stage, and outputs.
<Metadata text="cli/compare/get" />
```bash
inf compare get <run-id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ------------------------- |
| `runId` | Yes | AutoEvals analysis run ID |
## `inf compare groups` [#inf-compare-groups]
List a run's eval run groups and their eval runs — one group per rubric.
<Metadata text="cli/compare/groups" />
```bash
inf compare groups <run-id>
```
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | ------------------------- |
| `runId` | Yes | AutoEvals analysis run ID |
## `inf compare poll` [#inf-compare-poll]
Poll an analysis run until it reaches a terminal status (`success` or `error`).
<Metadata text="cli/compare/poll" />
```bash
inf compare poll <run-id>
```
### Arguments [#arguments-2]
| Argument | Required | Description |
| -------- | -------- | ------------------------- |
| `runId` | Yes | AutoEvals analysis run ID |
### Options [#options]
| Flag | Required | Description | Default |
| ---------------------- | -------- | ------------------------------- | ------- |
| `--interval <seconds>` | No | Poll interval | `15` |
| `--timeout <minutes>` | No | Give up after this many minutes | `45` |
## Generated REST verbs [#generated-rest-verbs]
`inf compare` also exposes the broader AutoEvals API as generated commands — `runs-list`, `runs-start-user`, `runs-cancel`, `insights`, `get-insight`, `usage`, `settings`, `update-settings`, `eval-usage-estimate`, and more. Run `inf compare --help` to list them and their flags.
## Related commands [#related-commands]
* [`inf eval`](/cli/evals) runs manual eval runs against rubrics.
# Dashboard (https://docs.inference.net/cli/dashboard)
`inf dashboard` launches a full-screen interactive terminal UI that gives you a live overview of training runs, evaluations, datasets, and inferences in your active project.
**Alias:** `inf dash`
## `inf dashboard` [#inf-dashboard]
Launch the TUI. The dashboard reads your active project from `~/.inf/config.json`, or use the global `--project <id>` flag to target a different project for this session.
<Metadata text="cli/dashboard/launch" />
```bash
inf dashboard
```
### Keyboard Shortcuts [#keyboard-shortcuts]
| Key | Action |
| ------------------- | ----------------------------- |
| `1` – `4` | Switch between tabs directly |
| `Tab` / `Shift+Tab` | Cycle through tabs |
| `j` / `k` | Navigate up and down in lists |
| `Enter` | Drill into the selected item |
| `r` | Refresh the current view |
| `/` | Open the command palette |
| `q` or `Ctrl+C` | Quit the dashboard |
<Note>
Paged list views (inside a detail panel) add navigation keys beyond `j`/`k`: `h`/`l`/`p`/`n` change pages, `g` jumps to the first page, `s` cycles the sort key, `d` toggles sort direction, and `/` starts an in-pager search.
</Note>
### Tabs [#tabs]
The dashboard provides four tabs for navigating your project data:
| Tab | Key | Description |
| ---------- | --- | ---------------------------------------------------- |
| Training | `1` | View training runs with status, progress, and loss |
| Evals | `2` | Browse eval run groups and individual run results |
| Datasets | `3` | List filtered datasets with export status |
| Inferences | `4` | View recent inferences with token counts and latency |
Select any item in a list and press `Enter` to open a detail panel with more information.
### Examples [#examples]
<Metadata text="cli/dashboard/examples" />
```bash
# Launch for the active project
inf dashboard
# Launch for a specific project without changing your stored config
inf dashboard --project proj_789ghi012jkl
# Use the alias
inf dash
```
<Note>
The dashboard requires a terminal that supports modern rendering. Most standard terminal emulators (Terminal.app, iTerm2, Alacritty, Windows Terminal) work well.
</Note>
# Datasets (https://docs.inference.net/cli/datasets)
Use `inf dataset` to upload JSONL inference data and manage datasets created from captured traffic, existing uploads, or JSONL files on disk. Materialized datasets feed into [`inf eval run`](/cli/evals) for evals and into training jobs.
**Alias:** `inf datasets`
## `inf dataset upload` [#inf-dataset-upload]
Import a JSONL file into the active project as an upload entry. An upload is the raw material you can then materialize into an eval or training dataset. The CLI validates the file locally, uploads it in parts, waits for processing to finish, and prints the detected format plus the processed line count.
<Metadata text="cli/dataset/upload" />
```bash
inf dataset upload <file>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | -------------------------------- |
| `file` | Yes | Path to the JSONL file to upload |
### Options [#options]
| Flag | Required | Description | Default |
| ------------------- | -------- | ---------------------------------------------------------------- | -------------------------- |
| `-n, --name <name>` | No | Upload name shown in the Inference platform | Filename without extension |
| `--no-wait` | No | Return after the transfer finishes instead of polling processing | Off |
Uploaded data appears in **Datasets → Uploads** in the dashboard. Once processing completes, create an eval or training dataset from that upload — either with [`inf dataset create --upload-id`](#inf-dataset-create) below or in the dashboard.
### Examples [#examples]
<Metadata text="cli/dataset/upload-examples" />
```bash
# Use the filename as the upload name and wait for processing
inf dataset upload ./data/support-summaries.jsonl
# Set a custom upload name
inf dataset upload ./data/support-summaries.jsonl --name support-summaries-v2
# Return after the transfer completes, without waiting for processing
inf dataset upload ./data/support-summaries.jsonl --no-wait
```
## `inf dataset create` [#inf-dataset-create]
Materialize an eval or training dataset from captured traffic, an existing upload, or a JSONL file on disk. The file-backed path uploads, waits for processing, and materializes in one command.
<Metadata text="cli/dataset/create" />
```bash
inf dataset create -n <name> -t <type> [source-flags…]
```
### Options [#options-1]
| Flag | Required | Description | Default |
| ---------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `-n, --name <name>` | Yes | Dataset name | — |
| `-t, --type <type>` | Yes | `eval` or `training` | — |
| `-f, --file <path>` | No | JSONL file on disk — uploads, waits for processing, then materializes from that upload | — |
| `--upload-id <id>` | No | Materialize from an existing upload | — |
| `--task <taskId>` | No | Filter captured traffic by task ID | — |
| `--model <modelId>` | No | Filter captured traffic by model ID | — |
| `--since <date>` | No | Start of the time window for traffic filters (ISO 8601 or `YYYY-MM-DD HH:MM:SS`) | 30 days ago |
| `--until <date>` | No | End of the time window for traffic filters | 1 minute from now |
| `--limit <n>` | No | Cap on the number of inferences included | — |
| `--status <status>` | No | Status filter: `success` (default), `2xx`, or a specific code like `200` — datasets reject non-success traffic unless you override | `success` |
| `--description <text>` | No | Free-text dataset description | — |
`--file` and `--upload-id` are mutually exclusive — `--file` creates a new upload automatically. Date values accept ISO 8601 (`2026-04-01T00:00:00Z`) or ClickHouse format (`2026-04-01 00:00:00`).
Dataset materialization runs asynchronously. The command prints the dataset ID and points at `inf dataset get <id>` to check progress.
### Examples [#examples-1]
<Metadata text="cli/dataset/create-examples" />
```bash
# One-command: upload a JSONL file and materialize an eval dataset
inf dataset create -n demo-eval -t eval --file ./samples.jsonl
# Materialize from an existing upload
inf dataset create -n training-v1 -t training --upload-id up_abc123
# Filter captured traffic by task + time window
inf dataset create -n support-eval -t eval \
--task support-tickets \
--since 2026-04-01 \
--until 2026-04-14
# Cap to 1,000 rows (only successful traffic is included by default)
inf dataset create -n small-eval -t eval --task support-tickets --limit 1000
```
## `inf dataset list` [#inf-dataset-list]
Display datasets in the active project.
<Metadata text="cli/dataset/list" />
```bash
inf dataset list
```
**Alias:** `inf dataset ls`
### Options [#options-2]
| Flag | Required | Description | Default |
| ----------------- | -------- | ------------------------- | ------- |
| `-l, --limit <n>` | No | Maximum number of results | `20` |
The table shows the dataset ID (8-char prefix), name, type, inference count, export status, and creation date. Use `--json` to get full UUIDs for scripting.
### Examples [#examples-2]
<Metadata text="cli/dataset/list-examples" />
```bash
# Default table view
inf dataset list
# More results
inf dataset list --limit 100
# Pipe full UUIDs into another command
inf dataset list --json | jq -r '.[].id'
```
## `inf dataset get` [#inf-dataset-get]
View detailed information about a specific dataset — ID, name, type, inference count, export status, source project, and creation date.
<Metadata text="cli/dataset/get" />
```bash
inf dataset get <id>
```
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | ------------------------------------------------- |
| `id` | Yes | Dataset ID, UUID prefix (4+ chars), or exact name |
## `inf dataset download` [#inf-dataset-download]
Download a dataset as a JSONL file. If the server-side export isn't ready yet, the CLI requests it and polls until it's ready before downloading.
<Metadata text="cli/dataset/download" />
```bash
inf dataset download [id]
```
### Arguments [#arguments-2]
| Argument | Required | Description |
| -------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `id` | No | Dataset ID, UUID prefix (4+ chars), or exact name. If omitted in an interactive terminal, the CLI prompts you to choose. |
### Options [#options-3]
| Flag | Required | Description | Default |
| ----------------------- | -------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `-o, --output <path>` | No | Output file path | `<dataset-name>.jsonl` for Hugging Face, `<dataset-name>.source-backed.jsonl` for source-backed |
| `-f, --format <format>` | No | Download format: `huggingface` or `source-backed` | Prompted in a TTY; otherwise `huggingface` |
The CLI resolves dataset IDs by exact ID, UUID prefix, or exact name.
### Examples [#examples-3]
<Metadata text="cli/dataset/download-examples" />
```bash
# Download to the default filename
inf dataset download ds_abc123
# Download to a specific file
inf dataset download ds_abc123 --output ./data/my-dataset.jsonl
# Download source-backed JSONL (request/response objects)
inf dataset download customer-support-eval --format source-backed
# Pick interactively when no id is provided
inf dataset download
```
## `inf dataset delete` [#inf-dataset-delete]
Archive a dataset — a soft delete that hides it from normal listings. Accepted IDs are full UUIDs, for example `a1b2c3d4-...`.
<Metadata text="cli/dataset/delete" />
```bash
inf dataset delete <id>
```
**Alias:** `inf dataset archive`
### Arguments [#arguments-3]
| Argument | Required | Description |
| -------- | -------- | --------------------------------------- |
| `id` | Yes | Full dataset UUID (e.g. `a1b2c3d4-...`) |
### Options [#options-4]
| Flag | Required | Description | Default |
| ----------- | -------- | --------------------------------------------- | ------- |
| `-y, --yes` | No | Skip the confirmation prompt (for scripts/CI) | Off |
The command prompts for confirmation unless `-y` is passed.
### Examples [#examples-4]
<Metadata text="cli/dataset/delete-examples" />
```bash
# Confirm interactively
inf dataset delete a1b2c3d4-1234-5678-9abc-def012345678
# Archive without prompting (scripts/CI)
inf dataset delete a1b2c3d4-1234-5678-9abc-def012345678 --yes
```
# Deployment Instances (https://docs.inference.net/cli/deployment-instance)
Drill into one instance of a deployment — its runtime detail, live GPU/CPU/memory gauges, and logs. Instance mutations are super-admin-only and stay in the web UI.
## `inf deployment-instance get` [#inf-deployment-instance-get]
Get a deployment instance's detail (redacted for non-admins).
<Metadata text="cli/deployment-instance/get" />
```bash
inf deployment-instance get --instance-id <instance-id>
```
Run `inf deployment instances <deployment-id>` first to find the instance ID.
## `inf deployment-instance metrics` [#inf-deployment-instance-metrics]
Get an instance's sparkline gauges (GPU, CPU, memory).
## `inf deployment-instance logs` [#inf-deployment-instance-logs]
Query an instance's logs (paginated).
## Generated REST verbs [#generated-rest-verbs]
`inf deployment-instance` exposes its full read surface as generated commands. Run `inf deployment-instance --help` to list them and their flags.
## Related commands [#related-commands]
* [`inf deployment instances <deployment-id>`](/cli/deployment#inf-deployment-instances) lists an instance's siblings and status.
# Deployments (https://docs.inference.net/cli/deployment)
Model deployments serve a fine-tuned or catalog model within a project, backed by allocated GPU instances. Use `inf deployment` to create a deployment, inspect its effective config and instances, and manage its lifecycle — all without opening the dashboard.
**Alias:** `inf deployments`
## Quickstart [#quickstart]
<Metadata text="cli/deployment/quickstart" />
```bash
# See what a model may be deployed on (GPU families and counts)
inf deployment allowed-instance-types --model-id <model-id>
# Create a deployment and watch the rollout converge
inf deployment create \
--name my-deployment \
--model-id <model-id> \
--public-model-identifier <team>/<model-slug>
inf deployment watch <deployment-id>
# Inspect the deployment and the engine config it serves with
inf deployment get <deployment-id>
inf deployment config <deployment-id>
# Stop serving (reversible) or archive to free a slot
inf deployment stop <deployment-id> --yes
inf deployment start <deployment-id>
```
## Super-admin pricing rules [#super-admin-pricing-rules]
Super-admins can set scheduled effective prices for a public serverless deployment. Prices are entered in USD per 1M tokens and stored as fixed per-token rates. Omitting `--customer-team` creates an all-team rule; a matching team rule takes precedence over an all-team rule.
```bash
inf admin deployments pricing-rules list <deployment-id>
inf admin deployments pricing-rules create <deployment-id> \
--customer-team <team-id> \
--input 1.05 \
--cached-input 0.115 \
--output 5.475 \
--starts-at 2026-09-20T00:00:00Z \
--ends-at 2026-09-27T00:00:00Z \
--reason "Partner launch pricing"
inf admin deployments pricing-rules remove <deployment-id> \
--rule <rule-id> \
--yes
```
`--cached-input` defaults to the input price. Higher `--priority` wins among otherwise matching rules; disabled, future, and expired rules do not apply. The deployment's standard prices remain the fallback.
## `inf deployment list` [#inf-deployment-list]
List deployments for a team with their status, lifecycle, and served model alias.
<Metadata text="cli/deployment/list" />
```bash
inf deployment list
```
**Alias:** `inf deployment ls`
### Options [#options]
| Flag | Required | Description | Default |
| ----------------- | -------- | ---------------------------------------- | --------------------------- |
| `--team <id>` | No | Team ID | `INF_TEAM_ID` / active team |
| `--project <id>` | No | Filter results to a single project ID | — |
| `-l, --limit <n>` | No | Max results to return (positive integer) | `50` |
Shows the deployment ID (8-char prefix), name, status, lifecycle state, public model identifier, and creation date.
## `inf deployment get` [#inf-deployment-get]
Show a single deployment's full details — status, lifecycle, model alias, and timestamps. Omit the `id` to pick interactively.
<Metadata text="cli/deployment/get" />
```bash
inf deployment get <id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------- |
| `id` | No | Deployment ID; omit to pick interactively |
### Options [#options-1]
| Flag | Required | Description |
| ------------- | -------- | ----------- |
| `--team <id>` | No | Team ID |
## `inf deployment config` [#inf-deployment-config]
Show the effective engine configuration a deployment serves with: the resolved flags, environment variables, and which layer set each.
<Metadata text="cli/deployment/config" />
```bash
inf deployment config <id>
```
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------- |
| `id` | No | Deployment ID; omit to pick interactively |
### Options [#options-2]
| Flag | Required | Description |
| ------------- | -------- | ----------- |
| `--team <id>` | No | Team ID |
## `inf deployment instances` [#inf-deployment-instances]
List a deployment's running and terminated instances with their status, version, and stop reason.
<Metadata text="cli/deployment/instances" />
```bash
inf deployment instances <id>
```
**Alias:** `inf deployment inst`
### Arguments [#arguments-2]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------- |
| `id` | No | Deployment ID; omit to pick interactively |
### Options [#options-3]
| Flag | Required | Description | Default |
| ----------------- | -------- | ---------------------------------------- | ----------- |
| `--team <id>` | No | Team ID | Active team |
| `-l, --limit <n>` | No | Max results to return (positive integer) | `100` |
## `inf deployment watch` [#inf-deployment-watch]
Poll a deployment until its rollout converges — status `Active` and the target version `Running`.
<Metadata text="cli/deployment/watch" />
```bash
inf deployment watch <id>
```
### Arguments [#arguments-3]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------- |
| `id` | No | Deployment ID; omit to pick interactively |
### Options [#options-4]
| Flag | Required | Description | Default |
| ----------------------- | -------- | -------------------------------------------------- | ----------------------- |
| `--team <id>` | No | Team ID | Active team |
| `--target-version <id>` | No | Require this version to be the live, `Running` one | Current desired version |
| `--timeout <sec>` | No | Max seconds to watch before giving up | `1800` |
| `--interval <sec>` | No | Seconds between polls | `10` |
Exits `0` on convergence; fails if the timeout elapses.
## `inf deployment stop` / `start` [#inf-deployment-stop--start]
`stop` archives a deployment so it stops serving (alias of `archive`). `start` restarts a previously archived deployment so it resumes serving. Both are reversible.
<Metadata text="cli/deployment/stop-start" />
```bash
inf deployment stop <id> --yes
inf deployment start <id>
```
### Options [#options-5]
| Flag | Required | Description | Default |
| ------------- | ----------------------------------------- | ---------------------------- | ----------- |
| `--team <id>` | No | Team ID | Active team |
| `-y, --yes` | Yes in non-TTY environments (`stop` only) | Skip the confirmation prompt | Off |
`stop` prompts for confirmation in an interactive terminal; in non-TTY environments (CI, scripts) it refuses to run without `-y`.
## `inf deployment archive` [#inf-deployment-archive]
Archive a deployment so it stops serving and frees a slot against the team cap. Reversible with `start`.
<Metadata text="cli/deployment/archive" />
```bash
inf deployment archive <id> --yes
```
### Arguments [#arguments-4]
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------- |
| `id` | No | Deployment ID; omit to pick interactively |
### Options [#options-6]
| Flag | Required | Description | Default |
| ------------- | --------------------------- | ---------------------------- | ----------- |
| `--team <id>` | No | Team ID | Active team |
| `-y, --yes` | Yes in non-TTY environments | Skip the confirmation prompt | Off |
## `inf deployment create` [#inf-deployment-create]
Create a deployment with its full config version.
<Metadata text="cli/deployment/create" />
```bash
inf deployment create \
--name <name> \
--model-id <model-id> \
--public-model-identifier <team>/<model-slug>
```
### Options [#options-7]
| Flag | Required | Description | Default |
| ------------------------------------- | -------- | --------------------------------------------------------------------- | -------------- |
| `--name <name>` | Yes | Deployment name | — |
| `--model-id <id>` | Yes | Model to deploy | — |
| `--public-model-identifier <id>` | Yes | Public model slug, e.g. `<team>/<model>` | — |
| `--project-id <id>` | No | Project to create the deployment in | Active project |
| `--team-id <id>` | No | Team to create the deployment in | Active team |
| `--desired-instance-count <n>` | No | Instance count to scale to | — |
| `--instance-requirements <obj>` | No | GPU instance requirements (see `allowed-instance-types`) | — |
| `--is-serverless-deployment <bool>` | No | Deploy as a serverless model | `false` |
| `--model-config-flag-overrides <obj>` | No | Engine flag overrides | — |
| `--model-config-version-override <n>` | No | Pin a specific model-config version | — |
| `--billing-exempt <bool>` | No | Exempt the deployment from billing | `false` |
| `--queue-timeout-ms <n>` | No | Max wait for a free slot before a `429`; use `-1` to disable queueing | Engine default |
| `--total-req-timeout-ms <n>` | No | Positive wall-clock cap on the whole stream | Engine default |
| `--time-to-next-token-timeout-ms <n>` | No | Max gap between generation progress | Shared default |
| `--accepted-control-tower-ids <ids>` | No | Control towers this deployment's instances may be placed on | Any tower |
These three timeout flags require super-admin access. They also work with `inf deployment update --id <deployment-id>`. Pass `null` to restore the inherited default.
`--accepted-control-tower-ids` also requires super-admin access and takes a
comma-separated list. An empty list means any eligible tower, which is the
default. The set constrains where *new* instances are placed: instances already
running are never moved when it changes. The super-admin CLI spells the same
setting as a repeatable `inf admin deployment create|update --accepted-control-tower <id>`, with `--clear-control-towers` to empty it.
Prints the new deployment ID. Run `inf deployment watch <deployment-id>` to track its rollout.
### Examples [#examples]
<Metadata text="cli/deployment/create-examples" />
```bash
# Deploy on 1 GPU instance and watch the rollout converge
inf deployment create \
--name support-model \
--model-id <model-id> \
--public-model-identifier acme/support-v1 \
--desired-instance-count 1
inf deployment watch <deployment-id>
# Override an engine flag to lower max tokens served
inf deployment create \
--name support-model \
--model-id <model-id> \
--public-model-identifier acme/support-v1 \
--model-config-flag-overrides '{"max_tokens": 4096}'
```
## Generated REST verbs [#generated-rest-verbs]
`inf deployment` also exposes the full deployment API surface as generated commands — `versions`, `update`, `engine-config`, `engine-configs`, `inferences`, `inference-filter-options`, `public-identifier-available`, `training-job-for-model`, `open-source-models`, and `allowed-instance-types`. Run `inf deployment --help` to list them and their flags.
# Evals (https://docs.inference.net/cli/evals)
Run and inspect model evaluations from the command line. Manage rubrics (the judge prompts evals run against), list and inspect run groups, launch new runs, and browse eval-ready datasets.
**Alias:** `inf evals`
The full eval loop is paste-able from the terminal:
<Metadata text="cli/eval/paste-able-loop" />
```bash
# 1. Create a rubric from a markdown file
inf eval rubric create -n support-tickets-v1 -f ./rubric.md
# → Rubric rub_abc12 / version rv_xyz45 created.
# 2. Materialize an eval dataset (traffic-backed, upload-backed, or from a file)
inf dataset create -n demo-eval -t eval --file ./samples.jsonl
# → Dataset ds_def78 created.
# 3. Launch an eval run group
inf eval run \
--rubric-id rub_abc12 \
--dataset-id ds_def78 \
--models openai:gpt-5.2,anthropic:claude-sonnet-4-6 \
--judge-model anthropic:claude-sonnet-4-6
# → Run group rg_20260415_152340 created.
# 4. Track progress
inf eval get rg_20260415_152340
```
Route IDs look like `<provider>:<model-alias>` (e.g. `openai:gpt-5.2`). Use [`inf models list`](/cli/models#inf-models-list) to discover every route ID available to your team — see [Route IDs](/cli/models#route-ids) for the full format.
## `inf eval rubric create` [#inf-eval-rubric-create]
Create a rubric — the judge prompt an eval run scores responses against. Rubrics live in the active project, carry versioned prompt content, and are passed to `inf eval run` by ID. The template must contain the placeholder `{{ eval_model_response }}` where the model's response will be injected for scoring.
<Metadata text="cli/eval/rubric-create" />
```bash
inf eval rubric create -n <name> -f <path-to-markdown>
```
### Options [#options]
| Flag | Required | Description | Default |
| ------------------- | -------- | ------------------------------------------------------------ | -------------- |
| `-n, --name <name>` | Yes | Rubric name | — |
| `-f, --file <path>` | Yes | Path to a markdown file containing the judge prompt template | — |
| `--max-score <n>` | No | Maximum score for the rubric (2–100) | `10` |
| `--project-id <id>` | No | Project to create the rubric in | Active project |
Prints the rubric ID and the first version ID. Use them directly with `inf eval run`.
### Examples [#examples]
<Metadata text="cli/eval/rubric-create-examples" />
```bash
# Create a rubric with the default 0–10 scoring scale
inf eval rubric create -n support-tickets-v1 -f ./rubric.md
# Create a rubric with a 0–100 scale
inf eval rubric create -n quality-v2 -f ./quality.md --max-score 100
```
## `inf eval rubric get` [#inf-eval-rubric-get]
Get details of a rubric — ID, name, latest version number, version count, score range, and a preview of the template.
<Metadata text="cli/eval/rubric-get" />
```bash
inf eval rubric get <id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ---------------------------------------------------- |
| `id` | Yes | Full UUID, 4+ character prefix, or exact rubric name |
Ambiguous prefixes print the candidate list and abort.
## `inf eval rubric delete` [#inf-eval-rubric-delete]
Archive (soft-delete) a rubric. Rubrics cannot be hard-deleted — archiving hides them from `inf eval rubrics` but preserves their eval history. Restore from the dashboard if needed.
<Metadata text="cli/eval/rubric-delete" />
```bash
inf eval rubric delete <id>
```
**Alias:** `inf eval rubric archive <id>` — both names do the same thing; use whichever reads clearer in your script.
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | ---------------------------------------------------- |
| `id` | Yes | Full UUID, 4+ character prefix, or exact rubric name |
### Options [#options-1]
| Flag | Required | Description | Default |
| ----------- | --------------------------- | ---------------------------- | ------- |
| `-y, --yes` | Yes in non-TTY environments | Skip the confirmation prompt | Off |
In an interactive terminal, the CLI asks for confirmation unless `-y` is passed. In non-TTY environments (CI, scripts) the command refuses to run without `-y`.
### Examples [#examples-1]
<Metadata text="cli/eval/rubric-delete-examples" />
```bash
# Archive interactively (prompts for confirmation)
inf eval rubric delete support-tickets-v1
# Archive non-interactively
inf eval rubric archive rub_abc12 --yes
```
## `inf eval rubrics` [#inf-eval-rubrics]
List rubrics in the active project.
<Metadata text="cli/eval/rubrics" />
```bash
inf eval rubrics
```
**Alias:** `inf eval defs`
### Options [#options-2]
| Flag | Required | Description | Default |
| -------------------- | -------- | ------------------------ | ------- |
| `--include-archived` | No | Include archived rubrics | Off |
Shows the rubric ID (8-char prefix), name, latest version, total version count, and creation date. Use `--json` for full UUIDs.
## `inf eval run` [#inf-eval-run]
Launch a new eval run group against one or more models, scored by a judge model.
<Metadata text="cli/eval/run" />
```bash
inf eval run \
--rubric-id <id> \
--dataset-id <id> \
--models <route-id-csv> \
--judge-model <route-id>
```
### Options [#options-3]
| Flag | Required | Description | Default |
| -------------------------- | -------- | -------------------------------------------------------------------------- | -------------- |
| `--rubric-id <id>` | Yes | Rubric ID | — |
| `--dataset-id <id>` | Yes | Eval-type dataset ID (create one with `inf dataset create -t eval`) | — |
| `--models <ids>` | Yes | Comma-separated model route IDs — run `inf models list` to discover them | — |
| `--judge-model <id>` | Yes | Route ID of the judge model — run `inf models list --judge-only` to filter | — |
| `--rubric-version-id <id>` | No | Pin to a specific rubric version | Latest version |
| `--sample-size <n>` | No | Samples drawn from the dataset per model (1–100) | `100` |
| `-n, --name <name>` | No | Display name for the run group | Auto-generated |
Prints the run group ID and an `inf eval get <id>` follow-up command to track progress.
### Examples [#examples-2]
<Metadata text="cli/eval/run-examples" />
```bash
# Launch a run against two models with a third as judge
inf eval run \
--rubric-id rub_abc12 \
--dataset-id ds_def78 \
--models openai:gpt-5.2,anthropic:claude-sonnet-4-6 \
--judge-model anthropic:claude-sonnet-4-6
# Pin to a specific rubric version
inf eval run \
--rubric-id rub_abc12 \
--rubric-version-id rv_xyz45 \
--dataset-id ds_def78 \
--models openai:gpt-5.2 \
--judge-model anthropic:claude-sonnet-4-6
```
## `inf eval list` [#inf-eval-list]
List eval run groups for a given rubric.
<Metadata text="cli/eval/list" />
```bash
inf eval list --rubric-id <id>
```
**Alias:** `inf eval ls`
### Options [#options-4]
| Flag | Required | Description | Default |
| -------------------------- | -------- | ----------------------------------- | ------- |
| `--rubric-id <id>` | Yes | Rubric ID to list runs for | — |
| `--rubric-version-id <id>` | No | Filter by a specific rubric version | — |
Shows the run group ID (8-char prefix), rubric version, model count, derived status (`pending`, `running`, `failed`, or `completed`), and creation date.
## `inf eval get` [#inf-eval-get]
View detailed information about a specific eval run group.
<Metadata text="cli/eval/get" />
```bash
inf eval get <id>
```
### Arguments [#arguments-2]
| Argument | Required | Description |
| -------- | -------- | --------------------- |
| `id` | Yes | The eval run group ID |
### Output [#output]
The detail view covers the run group itself, followed by a sub-table of individual runs:
| Field | Description |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | Run group ID |
| `rubricId` / `rubricVersionId` | Rubric and pinned version |
| `evalDatasetId` | Dataset the run group scored |
| `judgeProvider` / `judgeModelId` | Judge model scoring the responses |
| `models` | How many models were evaluated in this run group |
| `created` | Run group creation timestamp |
| Runs sub-table | One row per model: run ID, provider, model, status, average score, failed sample count, `completed/total` samples. When avg score is `—`, the adjacent `N failed` hint shows how many samples the judge couldn't score. |
## `inf eval datasets` [#inf-eval-datasets]
List datasets available for evaluations (type = `eval`).
<Metadata text="cli/eval/datasets" />
```bash
inf eval datasets
```
### Options [#options-5]
| Flag | Required | Description | Default |
| -------------------- | -------- | ------------------------- | ------- |
| `-l, --limit <n>` | No | Maximum number of results | `50` |
| `--include-archived` | No | Include archived datasets | Off |
Eval datasets are materialized via [`inf dataset create -t eval …`](/cli/datasets#inf-dataset-create) or the dashboard. The output shows the dataset ID (8-char prefix), name, inference count, and creation date.
# GitHub (https://docs.inference.net/cli/github)
Inspect a team's GitHub App integration: connection status, synced repositories, and branches. Connect/disconnect flows are browser-based and stay in the web UI.
## Commands [#commands]
Run `inf github <verb> --help` for each verb's options. The generated surface covers:
| Verb | Description |
| -------------------------------------------------------------- | ------------------------------------------------------------------ |
| `installation` | Get a team's GitHub App connection status |
| `install-url` | Get the GitHub App install URL for a team |
| `repositories` | List a team's synced GitHub repositories |
| `branches` | List branches for a connected GitHub repository |
| `repo-binding-get` / `repo-binding-set` / `repo-binding-clear` | Get, bind, or clear a GitHub repo + branch to an agent or task |
| `connect-url` | Get the GitHub OAuth connect URL for a team |
| `installation-selection` / `installation-selection-attach` | List and attach selectable GitHub installations for a connect flow |
| `disconnect` | Disconnect a team's GitHub App installation |
## Related commands [#related-commands]
* [`inf agent`](/cli/agent) — repo bindings attach a GitHub repo + branch to an agent or task.
# Inferences (https://docs.inference.net/cli/inferences)
Inspect inference requests and responses captured by Gateway. List and filter inferences in the active project, sort by tokens / cost / latency, discover the available filter values, and fetch the complete stored request and response bodies.
**Alias:** `inf inferences`
## `inf inference list` [#inf-inference-list]
List and filter inferences in the active project.
<Metadata text="cli/inference/list" />
```bash
inf inference list
```
**Alias:** `inf inference ls`
### Options [#options]
| Flag | Required | Description | Default |
| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `-l, --limit <n>` | No | Maximum number of results (1–100) | `20` |
| `--sort <key>` | No | Sort key: `sent_at`, `http_code`, `cost`, `duration`, `tokens`, `input_tokens`, `output_tokens`, `input_size`, `output_size` | `sent_at` |
| `--order <asc\|desc>` | No | Sort order | `desc` |
| `--filter <expr>` | No | Numeric filter `<field><op><number>` (repeatable). Fields: `inputTokens`, `outputTokens`, `inputSizeBytes`, `outputSizeBytes`, `durationMs`, `totalCost`. Ops: `> >= < <= = !=` | — |
| `--metadata <expr>` | No | Metadata filter `<key><op><value>` (repeatable). Ops: `= != ~ !~`, plus `<key> is_empty` / `<key> is_not_empty` | — |
| `-m, --model <name>` | No | Filter by model (repeatable or comma-separated) | All models |
| `--provider <name>` | No | Filter by downstream provider (repeatable or comma-separated) | All |
| `--environment <name>` | No | Filter by environment (repeatable or comma-separated) | All |
| `--status <code>` | No | Filter by HTTP status code, e.g. `200,500` (repeatable or comma-separated) | All |
| `--task <id>` | No | Filter by task ID — matches the `x-inference-task-id` header set by [`inf instrument --task-id`](/cli/instrument) (repeatable or comma-separated) | — |
| `--upload <id>` | No | Filter by upload ID (repeatable or comma-separated) | — |
| `--stream <bool>` | No | Filter by streamed responses (`true` / `false`) | — |
| `--from <iso>` / `--to <iso>` | No | Absolute time range (ISO-8601). Use both together. | — |
| `--range <preset>` | No | Relative time range: `1h, 6h, 12h, 1d, 3d, 7d, 14d, 30d, 90d, all` | — |
| `--cursor <cursor>` | No | Pagination cursor (from a previous `--json` response) | — |
| `--count` | No | Also compute the total match count (slower on large projects) | Off |
<Note>
Don't hardcode filter values — run [`inf inference facets`](#inf-inference-facets) to discover the valid models, providers, environments, tasks, and metadata keys for the active project, along with the full filter reference.
</Note>
The human-readable table shows the inference ID, model, status code (color-coded), input/output token counts, latency, cost, and timestamp. Add the global `--json` flag to emit the **full result object** — `{ items, nextCursor, hasMore, totalCount }` — so scripts can read `.items` and paginate by passing `.nextCursor` back via `--cursor`.
### Examples [#examples]
<Metadata text="cli/inference/list-examples" />
```bash
# List the 10 most recent inferences
inf inference list --limit 10
# Largest-input requests first — e.g. find prompts over 39k tokens
inf inference list --sort input_tokens --order desc --limit 50
inf inference list --filter inputTokens>39000
# Combine numeric, array, and metadata filters over a time window
inf inference list --model meta-llama/llama-3.1-8b-instruct/fp-8 --status 200 --range 7d
inf inference list --filter durationMs>5000 --metadata user_id=abc123
# Paginate from a script — JSON output includes nextCursor
cursor=$(inf --json inference list --sort input_tokens --order desc | jq -r '.nextCursor')
inf --json inference list --sort input_tokens --order desc --cursor "$cursor"
```
## `inf inference facets` [#inf-inference-facets]
Probe the active project for the values you can filter on — models, providers, environments, task IDs, and metadata keys — plus the full numeric/metadata/sort reference. Values are discovered from your data, not hardcoded, which makes this the starting point for building a filtered `inf inference list` query (and a one-call way for an AI agent to learn the filter surface).
**Alias:** `inf inference filters`
<Metadata text="cli/inference/facets" />
```bash
inf inference facets
```
Add the global `--json` flag to emit a machine-readable object combining the probed values with a static `reference` (numeric fields, operators, and sort keys).
## `inf inference get` [#inf-inference-get]
Fetch the **complete stored request and response** for a specific inference — method, path, headers, and the full request/response bodies. The sides come back `null` when payload storage was disabled for the request, and a side can also come back `null` when the blob exceeded the ingest size limit (32 MiB request bodies are refused at the edge with `413`; oversized streamed responses are recorded as metadata-only rows) — in that case `payloadTooLarge` is `true` and usage, cost, and metadata remain complete. Useful for debugging specific calls, inspecting model behavior, or archiving payloads.
<Metadata text="cli/inference/get" />
```bash
inf inference get <id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | -------------------------------------------- |
| `id` | Yes | Full inference UUID or a 4+ character prefix |
### Options [#options-1]
| Flag | Required | Description | Default |
| --------------------- | -------- | ----------------------------------------------------------------------------------------------- | ------- |
| `--request` | No | Output only the request (omit the response) | Both |
| `--response` | No | Output only the response (omit the request) | Both |
| `--body` | No | Output only the raw body content (no method/path/headers). Requires `--request` or `--response` | — |
| `-o, --output <path>` | No | Write the output to a file instead of stdout | stdout |
Add the global `--json` flag to print the full `{ request, response, payloadTooLarge }` payload (method, path, headers, and complete bodies) as clean JSON for piping into `jq` or saving. `payloadTooLarge` is `true` when a payload blob exceeded the ingest size limit and was not stored — the affected sides come back `null`.
### Examples [#examples-1]
<Metadata text="cli/inference/get-examples" />
```bash
# Using a 4+ character UUID prefix (human-readable view)
inf inference get inf_abc1
# Pull a full UUID from list output, then fetch it
inf inference list --json | jq -r '.items[0].id' | xargs inf inference get
# Full request + response as clean JSON, then extract the response body
inf --json inference get <id> | jq '.response.body'
# Dump just the raw response body to a file
inf inference get <id> --response --body -o response.json
```
# Instrument (https://docs.inference.net/cli/instrument)
`inf instrument` hands your codebase to an AI coding agent (Claude Code, OpenCode, or Codex) to wire up Inference platform observability for you. It scans your project, finds your LLM clients — OpenAI, Anthropic, LangChain, Amazon Bedrock, Gemini, Groq, Cerebras, OpenRouter, and others — and rewrites them to route through the Inference.net gateway so every call is captured in Gateway.
<Info>
This is the fastest way to connect an existing app to the Inference platform. You don't need to know the SDK changes in advance — the agent figures it out from your code.
</Info>
## Prerequisites [#prerequisites]
* A signed-in CLI session. `inf instrument` requires a session login — API-key auth is rejected. Run `inf auth login` first.
* A supported AI coding agent on your `PATH`. Install one of:
| Agent | Binary | Install |
| ----------- | ---------- | ------------------------------------------------ |
| Claude Code | `claude` | [docs.anthropic.com/en/docs/claude-code][claude] |
| OpenCode | `opencode` | [opencode.ai][opencode] |
| Codex | `codex` | [github.com/openai/codex][codex] |
[claude]: https://docs.anthropic.com/en/docs/claude-code
[opencode]: https://opencode.ai
[codex]: https://github.com/openai/codex
## `inf instrument` [#inf-instrument]
Instrument the current working directory. The command walks you through:
1. Confirming the target project (auto-selected if you only have one).
2. Detecting installed coding agents and picking one (prompted if you have more than one).
3. Downloading the instrumentation skill and building the agent prompt.
4. Creating a fresh project-scoped API key for this run. The key is passed to the agent through the `INFERENCE_API_KEY` environment variable, never in the prompt, so it is not sent to the agent's model provider.
5. Launching the agent interactively so you can review and approve the changes.
<Metadata text="cli/instrument/default" />
```bash
cd /path/to/your/project
inf instrument
```
### Options [#options]
| Flag | Required | Description | Default |
| ----------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| `--dry-run` | No | Ask the agent to preview changes without modifying files | Off |
| `--api-key <key>` | No | Plaintext API key to wire into the project, passed to the agent via environment variables (never in the prompt). Omit to create a fresh project-scoped key — stored keys are hashed, so a key's plaintext can never be read back | New key |
| `--agent <name>` | No | Preselect the coding agent by binary name: `claude`, `opencode`, or `codex`. Skips the interactive picker — useful in CI, scripts, or when driving `inf instrument` from another agent | Prompt |
| `-y, --yes` | No | Run non-interactively. When `--mode` is omitted, defaults it to `both` instead of prompting | Off |
| `--print-prompt` | No | Fetch the instrumentation skill, build the full agent prompt, print it to stdout, and exit without spawning an agent. Lets another orchestrator (your own agent harness, a CI job) drive the agent itself | Off |
| `--task-id <id>` | No | Default task id the agent tags every call with via the `x-inference-task-id` header. `inf inference list --task …` and `inf dataset create --task …` filter on this. 1–64 alphanumeric chars plus `._:-` | `default` |
| `--mode <mode>` | No | Which product(s) to instrument: `gateway` (proxy routing only), `tracing` (span collection only), or `both`. Omit to pick interactively | Interactive |
### What the agent does [#what-the-agent-does]
Once the agent launches, it:
* Scans your codebase for LLM clients (OpenAI, Anthropic, LangChain, Amazon Bedrock, Gemini, Groq, Cerebras, OpenRouter, and others).
* Redirects base URLs to the Inference.net gateway.
* Adds routing headers so requests are authenticated, forwarded, and traced.
* Tags every call with the default `x-inference-task-id` so call sites group automatically in Gateway.
* Shows you a diff before applying changes.
Your app keeps using the same provider SDKs it already has — `inf instrument` rewrites the client construction, not your call sites.
### How the API key is handled [#how-the-api-key-is-handled]
* Each run creates a fresh project-scoped API key. The command prints the revoke command (`inf api-key revoke <id>`) so you can invalidate it later.
* The key reaches the agent as `INFERENCE_API_KEY` in every mode (gateway, tracing, and both). It is never placed in the agent prompt, so it is never sent to the agent's model provider.
* The agent writes the key into your env file with shell expansion (`"$INFERENCE_API_KEY"`). The plaintext value never appears in the agent transcript or in `inf instrument` output.
* Keys are stored hashed and cannot be shown again. If the agent leaves a placeholder in your env file, create a new key with `inf api-key create <name>`.
### Examples [#examples]
<Metadata text="cli/instrument/examples" />
```bash
# Preview what would change without touching any files
inf instrument --dry-run
# Use Claude Code non-interactively (skip the agent picker and the confirmation)
inf instrument --agent claude --yes
# Tag every instrumented call with a custom task id for filtering later
inf instrument --task-id support-chatbot
# Print the prompt for your own agent orchestrator instead of spawning one
inf instrument --print-prompt > instrument-prompt.md
```
### Verify instrumentation [#verify-instrumentation]
After the agent finishes, run your app to generate a few LLM calls, then confirm they're captured:
<Metadata text="cli/instrument/verify" />
```bash
inf inference list
```
Or open the observability dashboard link the command prints on completion.
## Supported providers [#supported-providers]
**Built-in:** OpenAI, Anthropic.
**OpenAI-compatible via `x-inference-provider-url`:** Amazon Bedrock, Google Gemini, Together AI, Groq, Fireworks AI, Mistral AI, Cerebras, Perplexity, DeepSeek, OpenRouter, Azure OpenAI, and any OpenAI-compatible endpoint.
## Troubleshooting [#troubleshooting]
**`inf instrument now requires a session login. Unset INF_API_KEY and run inf auth login.`**
The command refuses to run with `INF_API_KEY` set in your environment. Unset it and sign in with `inf auth login` — instrumentation needs your session to create a project-scoped API key on your behalf.
**`No supported AI coding agent found.`**
Install one of Claude Code, OpenCode, or Codex from the links in [Prerequisites](#prerequisites). `inf instrument` detects the binary on your `PATH`.
**`--agent <name> is not installed (or not supported).`**
The binary name must match exactly: `claude`, `opencode`, or `codex`. Check `which claude` (or the name you passed) resolves to an executable.
**`Failed to create a project API key`**
Your session may lack permission to create keys for this project, or the project requires extra verification. Create a key in the dashboard (the relevant project → **API Keys**) and pass it with `--api-key <key>`. The key is handed to the agent through environment variables, not the prompt.
**The agent exited with an error.**
`inf instrument` prints the dashboard URL for manual integration on failure. You can also rerun with `-v` to see debug output, including the skill URL and prompt length.
# Integration (https://docs.inference.net/cli/integration)
Inspect external integrations connected to your team. Today that's Slack — check whether Slack is connected, which workspace it's linked to, and which channels the integration can post to.
## `inf integration slack status` [#inf-integration-slack-status]
Show whether Slack is connected, the linked workspace, bot scopes, and install date.
<Metadata text="cli/integration/slack-status" />
```bash
inf integration slack status
```
### Options [#options]
| Flag | Required | Description | Default |
| ------------- | -------- | -------------------- | ----------- |
| `--team <id>` | No | Team UUID to inspect | Active team |
## `inf integration slack channels` [#inf-integration-slack-channels]
List the Slack channels the integration can see, with their IDs and private/public status.
<Metadata text="cli/integration/slack-channels" />
```bash
inf integration slack channels
```
### Options [#options-1]
| Flag | Required | Description | Default |
| ------------- | -------- | -------------------- | ----------- |
| `--team <id>` | No | Team UUID to inspect | Active team |
### Examples [#examples]
<Metadata text="cli/integration/slack-channels-examples" />
```bash
# Check whether Slack is connected and which workspace it's linked to
inf integration slack status
# List the Slack channels the integration can post to
inf integration slack channels
# Target a specific team instead of the active one
inf integration slack channels --team 123e4567-e89b-12d3-a456-426614174000
```
## Related commands [#related-commands]
* [`inf slack`](/cli/slack) manages the Slack connection itself (install URL, disconnect).
* [`inf notifications`](/cli/notifications) manages Slack notification subscriptions.
# Models (https://docs.inference.net/cli/models)
`inf models` lets you browse every model available to your active team — both platform-provided models and any BYOK (bring-your-own-key) routes your team has configured.
**Alias:** `inf model`
`inf eval run` takes model route IDs via `--models` and `--judge-model`. `inf models list` is where you discover those route IDs.
## Route IDs [#route-ids]
Route IDs look like `<provider>:<model-alias>` — for example `openai:gpt-5.2`, `anthropic:claude-sonnet-4-6`, `cerebras:llama-3.3-70b`. They are the canonical identifier the CLI and API use to address a specific model route, and they're what [`inf eval run`](/cli/evals#inf-eval-run) expects for `--models` and `--judge-model`. Use `inf models list --json` to dump every route ID available to your team.
## `inf models list` [#inf-models-list]
Display every callable model visible to the active team, with provider, scope, capability flags, context window, and per-million-token pricing.
<Metadata text="cli/models/list" />
```bash
inf models list
```
**Alias:** `inf models ls`
### Options [#options]
| Flag | Required | Description | Default |
| ------------------- | -------- | --------------------------------------------------------------------------------------------------------- | ------------- |
| `--provider <name>` | No | Filter by provider name (case-insensitive exact match) — e.g. `openai`, `anthropic`, `google`, `cerebras` | All providers |
| `--scope <scope>` | No | Filter by scope: `platform` (inf-public catalog) or `byok` (your team's own provider keys) | Both |
| `--judge-only` | No | Show only models that can act as a judge in evals | Off |
### Output [#output]
In table mode (default), each row shows:
| Column | Description |
| ---------- | ------------------------------------------------------------------------- |
| `Model` | Canonical alias (e.g. `gpt-5.2`, `claude-sonnet-4-6`, `gemini-2.5-flash`) |
| `Provider` | Provider brand (OpenAI, Anthropic, Google, Cerebras, …) |
| `Scope` | `platform` (inf-public) or `byok` (team-owned route) |
| `Context` | Max context window, rounded to the nearest 1k tokens |
| `Struct.` | Whether the model supports structured outputs (`yes` / `no`) |
| `Tools` | Whether the model supports tool / function calling |
| `Reason.` | Whether the model has a reasoning mode |
| `Judge` | Whether the model is allow-listed as an eval judge |
| `$/1M In` | Input price per million tokens |
| `$/1M Out` | Output price per million tokens |
### Examples [#examples]
<Metadata text="cli/models/list-examples" />
```bash
# All callable models, table view
inf models list
# Only OpenAI routes
inf models list --provider openai
# Only models allow-listed as judges (for inf eval run --judge-model)
inf models list --judge-only
# Only BYOK routes
inf models list --scope byok
# Machine-readable — full routeId per row, good for piping into inf eval run
inf models list --json | jq -r '.[] | select(.judgeCapable == true) | .routeId'
```
### JSON mode [#json-mode]
`inf models list --json` emits the full enriched record per model, including the `routeId` string that `inf eval run --models` and `--judge-model` expect:
```json
[
{
"routeId": "openai:gpt-5.2-2025-12-11",
"canonicalAlias": "gpt-5.2",
"displayName": "GPT 5.2",
"providerName": "OpenAI",
"providerSlug": "openai",
"scope": "platform",
"maxContextSize": 128000,
"structuredOutputs": true,
"tools": true,
"reasoning": false,
"judgeCapable": true,
"costInputPerMToken": 2.5,
"costOutputPerMToken": 10
}
]
```
<Tip>
`inf models list --json | jq '.[] | select(.scope == "platform")'` is a quick way to prune your eval model set to just platform routes.
</Tip>
# Notifications (https://docs.inference.net/cli/notifications)
Inspect the notification types Inference emits and your email/Slack preferences for receiving them.
## Commands [#commands]
Run `inf notifications <verb> --help` for each verb's options. The generated surface covers:
| Verb | Description |
| --------------------------------------------------------- | --------------------------------------------------- |
| `types` | List every notification type and its channels |
| `email-preferences` | Get the caller's email notification preferences |
| `email-preferences-set` | Set one of the caller's email notification toggles |
| `slack-subscriptions` | List a team's Slack notification subscriptions |
| `slack-subscriptions-create` | Create a Slack notification subscription for a team |
| `update-slack-subscription` / `delete-slack-subscription` | Update or delete a Slack notification subscription |
## Related commands [#related-commands]
* [`inf slack`](/cli/slack) manages the underlying Slack channel connection.
# Install CLI (https://docs.inference.net/cli/overview)
The Inference CLI (`inf`) drives Inference platform from the terminal. It serves two main paths:
1. **Instrument your codebase** so every LLM call your app makes is captured in Gateway — `inf instrument` hands the job to an AI coding agent (Claude Code, OpenCode, or Codex) and walks you through the diff.
2. **Operate the platform programmatically** — browse models, manage rubrics and eval runs, upload and materialize datasets, queue and monitor training runs, and inspect captured inferences, traces, and spans without opening the dashboard.
`inf instrument` rewrites an app's LLM clients for observability. Routing a coding agent's own model traffic through the Inference.net gateway is the [`fast` CLI's](https://fast.inference.net/coding-agents) job, not `inf`'s.
Sign up for an account at [Inference.net](https://inference.net/register) to get started.
<Warning>
The CLI is currently in beta. Please report any issues you find.
</Warning>
## Quick Start [#quick-start]
<Steps>
<Step title="Install">
<Metadata text="cli/overview/quickstart-install" />
<CodeGroup>
```bash npm
npm install -g @inference/cli
```
```bash pnpm
pnpm add -g @inference/cli
```
```bash bun
bun add -g @inference/cli
```
```bash yarn
yarn global add @inference/cli
```
</CodeGroup>
</Step>
<Step title="Sign in">
<Metadata text="cli/overview/quickstart-sign-in" />
```bash
inf auth login
```
</Step>
<Step title="Check status">
<Metadata text="cli/overview/quickstart-status" />
```bash
inf auth status
```
</Step>
</Steps>
Run `inf --help` at any time to see every command. Having trouble? [Send us a message](mailto:support@inference.net) or tag us at [x.com/@inference\_net](https://x.com/inference_net).
<Tip>
Every command group below also exposes the full REST surface of its API resource as generated `inf <group> <verb>` commands, rendered directly from the Inference API. These aren't documented one-by-one here — discover them with `inf <group> --help`. For example, `inf deployment --help` lists generated verbs like `versions`, `update`, and `public-identifier-available` alongside the hand-written ones.
</Tip>
## Global Options [#global-options]
These flags work on every command.
| Flag | Description |
| -------------------- | --------------------------------------------------- |
| `--json` | Output as JSON (preserves full UUIDs for scripting) |
| `-v, --verbose` | Verbose debug output |
| `-p, --project <id>` | Override the active project for this invocation |
| `-t, --team <id>` | Override the active team for this invocation |
| `-V, --version` | Show CLI version |
| `-h, --help` | Show help |
<Tip>
Tables show UUIDs as readable 8-character prefixes. For scripting, always use `--json` — it preserves full UUIDs so you can round-trip values between commands (e.g. `inf dataset list --json | jq -r '.[0].id' | xargs inf dataset get`).
</Tip>
## Commands [#commands]
| Command | Description |
| ----------------------------------------------- | ------------------------------------------------------------------------------- |
| [`inf instrument`](/cli/instrument) | Instrument your codebase for Inference platform observability using an AI agent |
| [`inf auth`](/cli/authentication) | Sign in, sign out, check authentication status, and show who you are |
| [`inf update`](/cli/update) | Upgrade a globally installed CLI to the latest version |
| [`inf team`](/cli/team) | List, switch between, create teams, and invite members |
| [`inf project`](/cli/projects) | List, switch between, and inspect projects, across teams |
| [`inf api-key`](/cli/api-key) | Create, list, and revoke project API keys |
| [`inf models`](/cli/models) | Browse callable models with capabilities and pricing |
| [`inf eval`](/cli/evals) | Manage rubrics, launch eval runs, inspect results |
| [`inf rubric`](/cli/rubric) | Inspect eval rubrics and their versions |
| [`inf compare`](/cli/compare) | Inspect AutoEvals analysis runs, list eval groups, poll runs |
| [`inf dataset`](/cli/datasets) | Upload JSONL data, create eval/training datasets, download |
| [`inf training`](/cli/training) | Queue training runs, monitor progress, view logs, and poll status |
| [`inf training-eval`](/cli/training-eval) | Inspect mid-training evals and manage reusable training recipes |
| [`inf deployment`](/cli/deployment) | Create, inspect, watch, and archive model deployments |
| [`inf agent`](/cli/agent) | Inspect auto-discovered agents and their execution stats |
| [`inf task`](/cli/task) | Inspect and manage a project's tasks |
| [`inf signals`](/cli/signals) | Create LLM classifiers that label spans/traces/sessions, plus alerts |
| [`inf notifications`](/cli/notifications) | Inspect notification types and email/Slack preferences |
| [`inf integration`](/cli/integration) | Inspect external integrations connected to your team (Slack) |
| [`inf slack`](/cli/slack) | Inspect and manage a team's Slack integration |
| [`inf github`](/cli/github) | Inspect a team's GitHub integration |
| [`inf search`](/cli/search) | Search across agents, deployments, datasets, and rubrics |
| [`inf starter-template`](/cli/starter-template) | Browse starter templates and track provisioning |
| [`inf upload`](/cli/upload) | Inspect inference uploads (status, list, download URL) |
| [`inf worker`](/cli/worker) | Inspect v1 worker instances and their logs |
| [`inf account-deletion`](/cli/account-deletion) | Schedule deletion of your account, a project, or a team |
| [`inf inference`](/cli/inferences) | View inference requests and responses captured by Gateway |
| [`inf trace`](/cli/traces) | Browse trace trees, timelines, facets, and exports |
| [`inf span`](/cli/spans) | Search spans and inspect span IO, attributes, events, and links |
| [`inf analyze-harness`](/cli/analyze-harness) | Run HALO agent-trace analyses and read the resulting reports |
| [`inf dashboard`](/cli/dashboard) | Launch the interactive terminal dashboard |
This table covers the most-used command groups. More exist — including `deployments`, `signals`, `tasks`, `compare`, and others — so run `inf --help` for the full list.
## Explore the CLI [#explore-the-cli]
<CardGroup cols="2">
<Card title="Instrument your codebase" icon="wand-magic-sparkles" href="/cli/instrument">
Hand your project to an AI coding agent that wires up Inference platform for you.
</Card>
<Card title="Coding Agents" icon="robot" href="https://fast.inference.net/coding-agents">
Route Claude Code, Codex, Grok, OpenCode, and Pi through the Inference.net gateway with the fast CLI.
</Card>
<Card title="Authentication" icon="key" href="/cli/authentication">
Browser and headless authentication, env vars, and config.
</Card>
<Card title="Teams" icon="users" href="/cli/team">
List, switch, create teams, and invite members.
</Card>
<Card title="Projects" icon="book" href="/cli/projects">
Switch between projects and inspect the active project, across teams.
</Card>
<Card title="API Keys" icon="lock" href="/cli/api-key">
Create, list, and revoke project API keys.
</Card>
<Card title="Models" icon="sparkles" href="/cli/models">
Browse callable models, capabilities, and pricing.
</Card>
<Card title="Evals" icon="flask" href="/cli/evals">
Manage rubrics, launch eval runs, and inspect results.
</Card>
<Card title="Datasets" icon="file-code" href="/cli/datasets">
Upload JSONL files and materialize eval or training datasets.
</Card>
<Card title="Training" icon="graduation-cap" href="/cli/training">
Queue training runs, monitor progress, view logs, and poll for completion.
</Card>
<Card title="Inferences" icon="code" href="/cli/inferences">
Inspect request and response payloads captured by Gateway.
</Card>
<Card title="Traces" icon="waypoints" href="/cli/traces">
Browse trace trees, timelines, facets, and exports from the terminal.
</Card>
<Card title="Spans" icon="git-branch" href="/cli/spans">
Search spans and inspect captured IO, attributes, events, and links.
</Card>
<Card title="Analyze Harness" icon="microscope" href="/cli/analyze-harness">
Run HALO agent-trace analyses, schedule recurring reports, and read the reports.
</Card>
<Card title="TUI Dashboard" icon="computer" href="/cli/dashboard">
Interactive terminal UI for training runs, evals, datasets, and inferences.
</Card>
<Card title="Deployments" icon="box" href="/cli/deployment">
Create, inspect, watch, and archive model deployments.
</Card>
<Card title="Signals" icon="activity" href="/cli/signals">
LLM classifiers that label spans, traces, and sessions — plus alerts.
</Card>
<Card title="Compare" icon="flask" href="/cli/compare">
Inspect AutoEvals runs, eval groups, and poll runs to completion.
</Card>
</CardGroup>
## Usage telemetry [#usage-telemetry]
The CLI reports anonymized usage events so we can prioritize the commands our
customers actually rely on.
* **What we capture:** the command name, CLI version, OS and CPU architecture,
the JavaScript runtime, and the flag names (never flag *values*) you used.
When you run the CLI inside a git repository we also capture the repo
`owner/name` and current branch from your `origin` remote — this is most
useful for `inf instrument`, where we want to understand which codebases
Inference platform gets wired into.
* **What we never capture:** argument values, environment variables, file
contents, API keys, or any data you pass to a command.
* **When we don't capture anything:** events are only sent once you are
authenticated. Commands run before `inf auth login` / `inf auth set-key`
emit no events.
* **How it works:** events are sent fire-and-forget over tRPC with a short
timeout, so telemetry never slows down or blocks your command. Failures are
silent.
# Projects (https://docs.inference.net/cli/projects)
Projects organize your training runs, evaluations, datasets, and inferences. Most CLI commands operate on the active project, which is stored in `~/.inf/config.json` and settable per-invocation with the global `--project <id>` flag.
## `inf project list` [#inf-project-list]
Display every project you have access to in the active team. Switch teams first with `inf team switch <id-or-slug>` (see [Teams](/cli/team)) to see a different team's projects. The active project is flagged with a green dot (`●`).
<Metadata text="cli/project/list" />
```bash
inf project list
```
**Alias:** `inf project ls`
### Example [#example]
<Metadata text="cli/project/list-examples" />
```bash
inf project list
```
## `inf project switch` [#inf-project-switch]
Set a different project as the active project. The CLI validates that the project exists and that you have access before saving.
To switch to a project in a different team, activate that team first with `inf team switch <id-or-slug>` (see [Teams](/cli/team)), or pass the global `--team <id>` flag on the same command so the CLI can find the project. On success, the CLI also stores that project's team as your new active team, so you do not need a separate `team switch` afterward.
<Metadata text="cli/project/switch" />
```bash
inf project switch [project-id]
```
**Alias:** `inf project use`
### Arguments [#arguments]
| Argument | Required | Description |
| ------------ | -------- | --------------------------------- |
| `project-id` | No | The ID of the project to activate |
### Examples [#examples]
<Metadata text="cli/project/switch-examples" />
```bash
# Choose a project from an interactive list
inf project switch
# Switch to the staging project
inf project switch proj_789ghi012jkl
# Same, using the `use` alias
inf project use proj_789ghi012jkl
# Switch to a project in a different team, in one command
inf project switch proj_456def789ghi --team team_abc123
```
<Note>
In a non-interactive shell or with `--json`, pass a project ID. The CLI cannot show an interactive selector in these modes.
</Note>
<Tip>
Temporarily override the active project for a single command without changing your stored config using the global `--project` flag:
```bash
inf training list --project proj_789ghi012jkl
```
</Tip>
## `inf project current` [#inf-project-current]
Show details about the currently active project — project ID, name, team ID, and creation date. If no project is active, the command errors and points you at `inf project switch <id>`.
<Metadata text="cli/project/current" />
```bash
inf project current
```
# Rubrics (https://docs.inference.net/cli/rubric)
Inspect eval rubrics (the judge prompts evals run against) and their versions from the terminal.
## `inf rubric` vs `inf eval` [#inf-rubric-vs-inf-eval]
`inf rubric` is the read-oriented view of rubrics: get a rubric, list versions, and pin. **`inf eval rubric`** is the full workflow — create rubrics, launch run groups, and inspect results. For anything beyond inspection, use `inf eval rubric`.
## Commands [#commands]
Run `inf rubric <verb> --help` for each verb's options. The generated surface covers:
| Verb | Description | |
| --------------------------- | --------------------------------------------------------------- | ------------------------------------ |
| `list` | List rubrics for the team or a project | |
| `get` | Get a rubric with its latest version | |
| `get-version` | Get a rubric pinned to a specific version | |
| `versions` | List every version of a rubric | |
| `create` / `create-version` | Create a judge rubric (with its first version) or add a version | |
| `rename` | `archive` / `unarchive` | Rename or archive/unarchive a rubric |
| `set-judge-model` | Set a rubric's judge model route | |
| `set-models` | Set the candidate model routes a rubric evaluates | |
| `generate-draft` | Generate a draft judge rubric from a dataset's traffic | |
## Related commands [#related-commands]
* [`inf eval rubric`](/cli/evals#inf-eval-rubric-create) covers rubric creation and the full eval workflow.
# Search (https://docs.inference.net/cli/search)
Search across a team's agents, deployments, datasets, and rubrics from the terminal.
<Metadata text="cli/search/list" />
```bash
inf search list --query <query>
```
### Options [#options]
| Flag | Required | Description |
| ---------------------------- | -------- | ------------------------------------------ |
| `--query <text>` | Yes | Search text to match across resource types |
| `--limit-per-type <integer>` | No | Max results per resource type |
`list` is the primary verb; `inf search --help` lists any additional generated commands and their flags.
# Signal Alerts (https://docs.inference.net/cli/signal-alert)
Manage threshold alerts on [signals](/cli/signals) — predicates over a rolling window, Slack delivery, and firing history. This is the same alerting surface as `inf signals alerts`, exposed as its own command group.
## Commands [#commands]
Run `inf signal-alert --help` for the full, up-to-date verb list. The generated surface covers:
| Verb | Description |
| ------------ | ----------------------------------------------------------------- |
| `list` | List all alerts across the active team's signals |
| `by-project` | List all alerts in a project |
| `by-signal` | List a signal's alerts |
| `get` | Get one alert by id |
| `create` | Create an alert on a signal |
| `update` | Edit an alert's predicate, config, or enabled toggle |
| `archive` | Archive an alert |
| `events` | List an alert's firing history |
| `state` | Get an alert's current state snapshot |
| `backtest` | Replay an alert predicate over recent history (read-only preview) |
## Related commands [#related-commands]
* [`inf signals alerts`](/cli/signals#inf-signals-alerts) documents the alert flags (metric, operator, threshold, window) that the shared alert model uses.
# Signals (https://docs.inference.net/cli/signals)
A **signal** is an LLM classifier that labels a sampled fraction of an agent's spans, traces, or sessions. Use signals to track behavior you care about — refusals, sentiment, error categories — and alert when the label mix drifts.
Use `inf signals` to define classifiers, backfill labels over a past window, inspect the produced labels and their distribution over time, and manage threshold alerts on them.
## How signals work [#how-signals-work]
* Each signal belongs to **one agent** (its UUID, not its identity value) and targets one **scope**: `span`, `trace`, or `session`.
* A signal has a **classifier**: `binary` (true/false) or `string` (picks one of 2–10 `--label` values).
* The classifier samples a **percentage** of matching items (`--sample-rate`, integer 0–100) and runs only while the signal is **active** (`activate` / `disable` toggle it).
* Editing creates a **new version** of the signal; the current version drives classification.
## Quickstart [#quickstart]
<Metadata text="cli/signals/quickstart" />
```bash
# 1. Find the agent that owns the signal
inf agent list
# 2. Create a binary classifier sampling 25% of spans
inf signals create \
--agent-id <agent-uuid> \
--name "Refusals" \
--scope span \
--classifier binary \
--sample-rate 25 \
--prompt "Did the assistant refuse the request?"
# 3. Verify the prompt labels a past window correctly before going live
inf signals verify <signal-id> \
--from 2026-06-01T00:00:00Z \
--to 2026-06-02T00:00:00Z
# 4. See the signals for an agent and inspect one
inf signals list --agent my-agent
inf signals get <signal-id>
```
`inf signals list` prints full signal IDs. Use `--json` for machine-readable output.
## `inf signals list` [#inf-signals-list]
List all signals defined for a single agent identity, with classifier type, scope, sample rate, and recent label counts.
<Metadata text="cli/signals/list" />
```bash
inf signals list --agent my-agent
```
**Alias:** `inf signals ls`
### Options [#options]
| Flag | Required | Description | Default |
| -------------------- | -------- | --------------------------------------------------------- | ------- |
| `--agent <identity>` | Yes | Agent identity value to list signals for, e.g. `my-agent` | — |
| `--include-archived` | No | Also show archived (soft-deleted) signals | Off |
## `inf signals list-by-project` [#inf-signals-list-by-project]
List every signal across all agents in a project, as a lightweight summary with no version detail. Use when you don't know which agent owns a signal.
<Metadata text="cli/signals/list-by-project" />
```bash
inf signals list-by-project
```
### Options [#options-1]
| Flag | Required | Description | Default |
| ---------------- | -------- | -------------------- | -------------- |
| `--project <id>` | No | Project UUID to scan | Active project |
## `inf signals get` [#inf-signals-get]
Show one signal's full config plus its current version — classifier prompt, labels, and sample rate.
<Metadata text="cli/signals/get" />
```bash
inf signals get <signal-id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| ---------- | -------- | ------------------------------------- |
| `signalId` | Yes | Signal UUID (from `inf signals list`) |
## `inf signals create` [#inf-signals-create]
Create a signal. Pass exactly one of `--prompt` or `--prompt-file`.
<Metadata text="cli/signals/create" />
```bash
inf signals create \
--agent-id <agent-uuid> \
--name "Sentiment" \
--scope trace \
--classifier string \
--sample-rate 10 \
--label positive,neutral,negative \
--prompt-file ./sentiment.txt
```
### Options [#options-2]
| Flag | Required | Description | Default |
| ----------------------------- | --------------------------------- | ------------------------------------------------------------------------- | ------- |
| `--agent-id <id>` | Yes | Agent UUID that owns this signal (the agent's id, not its identity value) | — |
| `--name <name>` | Yes | Human-readable signal name, e.g. "Refusals" | — |
| `--scope <scope>` | Yes | What each classifier run labels: `span`, `trace`, or `session` | — |
| `--classifier <type>` | Yes | `binary` (true/false, no `--label`) or `string` (2–10 `--label` values) | — |
| `--sample-rate <percent>` | Yes | Percent of matching items to classify (integer 0–100) | — |
| `--prompt <text>` | Exactly one of prompt/prompt-file | Classifier prompt text inline | — |
| `--prompt-file <path>` | Exactly one of prompt/prompt-file | Read the classifier prompt from a file, e.g. `./refusal-prompt.txt` | — |
| `--label <label>` | For `string` classifiers | Label value; repeat 2–10 times or comma-separated | `[]` |
| `--sentiment <outcome=value>` | No | Outcome sentiment, e.g. `true=negative` or `<label>=positive` | `[]` |
### Examples [#examples]
<Metadata text="cli/signals/create-examples" />
```bash
# Binary classifier (no labels) sampling 25% of spans
inf signals create --agent-id <agent-uuid> --name "Refusals" \
--scope span --classifier binary --sample-rate 25 \
--prompt "Did the assistant refuse the request?"
# String classifier with a fixed label set, prompt read from a file
inf signals create --agent-id <agent-uuid> --name "Sentiment" \
--scope trace --classifier string --sample-rate 10 \
--label positive,neutral,negative --prompt-file ./sentiment.txt
```
## `inf signals edit` [#inf-signals-edit]
Create a new version of a signal by overriding any of its config fields; unspecified fields keep their current values. Pass at most one of `--prompt` or `--prompt-file`.
<Metadata text="cli/signals/edit" />
```bash
inf signals edit <signal-id> --sample-rate 50
```
### Arguments [#arguments-1]
| Argument | Required | Description |
| ---------- | -------- | ------------------------------------- |
| `signalId` | Yes | Signal UUID (from `inf signals list`) |
### Options [#options-3]
| Flag | Required | Description | Default |
| ----------------------------- | -------- | ----------------------------------------------------------------- | ------------ |
| `--scope <scope>` | No | New scope: `span`, `trace`, or `session` | Keep current |
| `--classifier <type>` | No | New classifier type: `binary` or `string` | Keep current |
| `--sample-rate <percent>` | No | New sample rate (integer 0–100) | Keep current |
| `--prompt <text>` | No | New classifier prompt text inline | Keep current |
| `--prompt-file <path>` | No | Read the new classifier prompt from a file | Keep current |
| `--label <labels>` | No | Replace the string classifier label set entirely, e.g. `good,bad` | Keep current |
| `--sentiment <outcome=value>` | No | Outcome sentiment, e.g. `true=negative` | Keep current |
### Examples [#examples-1]
<Metadata text="cli/signals/edit-examples" />
```bash
# Bump just the sample rate
inf signals edit <signal-id> --sample-rate 50
# Rewrite the prompt and swap the label set
inf signals edit <signal-id> --prompt-file ./v2-prompt.txt --label positive,negative
```
## `inf signals activate` / `disable` / `archive` [#inf-signals-activate--disable--archive]
Manage a signal's state.
<Metadata text="cli/signals/state" />
```bash
# Resume classifying new sampled items
inf signals activate <signal-id>
# Pause classification (reversible)
inf signals disable <signal-id>
# Archive (soft-delete); hidden from `list` unless --include-archived
inf signals archive <signal-id>
```
Each takes a single required `signalId` argument (UUID from `inf signals list`) and no other options.
## `inf signals verify` [#inf-signals-verify]
Preview how a signal's current classifier would label a small sample from a past window, **without persisting results**. Use to sanity-check a prompt before a full run.
<Metadata text="cli/signals/verify" />
```bash
inf signals verify <signal-id> \
--from 2026-06-01T00:00:00Z \
--to 2026-06-02T00:00:00Z \
--sample-size 30
```
### Arguments [#arguments-2]
| Argument | Required | Description |
| ---------- | -------- | ------------------------------------- |
| `signalId` | Yes | Signal UUID (from `inf signals list`) |
### Options [#options-4]
| Flag | Required | Description | Default |
| ------------------- | -------- | ------------------------------------------- | ------- |
| `--from <iso>` | Yes | Window start, ISO-8601 | — |
| `--to <iso>` | Yes | Window end, ISO-8601 | — |
| `--sample-size <n>` | No | Number of items to classify (integer 1–100) | `20` |
## `inf signals run` [#inf-signals-run]
Create and inspect manual backfill runs.
### `inf signals run create` [#inf-signals-run-create]
Create a manual signal run over a window.
<Metadata text="cli/signals/run-create" />
```bash
inf signals run create <signal-id> \
--from 2026-06-01T00:00:00Z \
--to 2026-06-02T00:00:00Z \
--sample-rate 100
```
| Flag | Required | Description | Default |
| ------------------------- | -------- | -------------------------- | ------- |
| `--from <iso>` | Yes | Start time, ISO-8601 | — |
| `--to <iso>` | Yes | End time, ISO-8601 | — |
| `--sample-rate <percent>` | Yes | Sample rate percent, 0–100 | — |
### `inf signals run list` [#inf-signals-run-list]
List manual runs for a signal.
### `inf signals run poll` [#inf-signals-run-poll]
Poll a manual signal run until it reaches a terminal state.
<Metadata text="cli/signals/run-poll" />
```bash
inf signals run poll <manual-run-id> --signal-id <signal-id>
```
| Flag | Required | Description | Default |
| ---------------------- | -------- | ------------------------------------ | ------- |
| `--signal-id <id>` | Yes | Signal the run belongs to | — |
| `--timeout <seconds>` | No | Max seconds to poll before giving up | `600` |
| `--interval <seconds>` | No | Seconds between polls | `10` |
## `inf signals backfill` [#inf-signals-backfill]
Queue a manual backfill run over a window.
<Metadata text="cli/signals/backfill" />
```bash
inf signals backfill \
--signal-id <signal-id> \
--from 2026-06-01T00:00:00Z \
--to 2026-06-02T00:00:00Z \
--sample-rate-percent 100
```
| Flag | Required | Description |
| --------------------------- | -------- | ----------------------------- |
| `--signal-id <id>` | Yes | Signal to backfill |
| `--from <iso>` | Yes | Window start, ISO-8601 |
| `--to <iso>` | Yes | Window end, ISO-8601 |
| `--sample-rate-percent <n>` | Yes | Sample rate percent (integer) |
## `inf signals labels list` [#inf-signals-labels-list]
List the per-item labels a signal produced, with paging, sorting, and filtering by label value.
<Metadata text="cli/signals/labels-list" />
```bash
inf signals labels list <signal-id> --classifier string --string-label negative --limit 100
```
### Arguments [#arguments-3]
| Argument | Required | Description |
| ---------- | -------- | ------------------------------------- |
| `signalId` | Yes | Signal UUID (from `inf signals list`) |
### Options [#options-5]
| Flag | Required | Description | Default |
| ------------------------ | -------- | ---------------------------------------------------------------------- | -------------- |
| `--classifier <type>` | Yes | Classifier type: `binary`, `string`, or `regex` | — |
| `--limit <n>` | No | Max rows to return (integer 1–200) | `50` |
| `--offset <n>` | No | Rows to skip for paging (integer >= 0) | `0` |
| `--sort-by <field>` | No | `completed_at`, `target_start_time`, `status`, or `label` | `completed_at` |
| `--sort-dir <dir>` | No | `asc` or `desc` | `desc` |
| `--bool-label <bool>` | No | Filter to one binary outcome (`true`/`false`; binary classifiers) | — |
| `--string-label <label>` | No | Filter to one string label value, e.g. `negative` (string classifiers) | — |
## `inf signals distribution` [#inf-signals-distribution]
Show how a signal's labels break down over time, bucketed by hour or day (computed live from raw labels).
<Metadata text="cli/signals/distribution" />
```bash
inf signals distribution <signal-id> \
--from 2026-06-01T00:00:00Z \
--to 2026-06-08T00:00:00Z \
--bucket day
```
### Options [#options-6]
| Flag | Required | Description | Default |
| -------------- | -------- | ---------------------- | ------- |
| `--from <iso>` | Yes | Window start, ISO-8601 | — |
| `--to <iso>` | Yes | Window end, ISO-8601 | — |
| `--bucket <b>` | No | `hour` or `day` | `hour` |
## `inf signals distribution-trends` [#inf-signals-distribution-trends]
Get a materialized-view-backed label distribution for a signal (grid-optimized for larger windows).
<Metadata text="cli/signals/distribution-trends" />
```bash
inf signals distribution-trends <signal-id> --from <iso> --to <iso> --bucket day
```
Takes the same `--from`, `--to`, and `--bucket` options as `inf signals distribution`.
## `inf signals alerts` [#inf-signals-alerts]
Manage alerts on a signal — threshold predicates over a rolling window, Slack delivery, and firing history. The full subtree runs under `inf signals alerts`.
### `inf signals alerts create` [#inf-signals-alerts-create]
<Metadata text="cli/signals/alerts-create" />
```bash
inf signals alerts create \
--signal-id <signal-id> \
--name "true-rate drop" \
--metric true_rate \
--operator lte \
--threshold 0.8 \
--window 1h
```
| Flag | Required | Description | Default |
| ----------------------------- | ------------------------------- | ------------------------------------------------------------------------------------- | ------------ |
| `--signal-id <id>` | Yes | Signal to alert on | — |
| `--name <name>` | Yes | Alert name | — |
| `--metric <metric>` | Yes | `label_volume`, `true_rate`, `value_count`, or `value_share` | — |
| `--operator <op>` | Yes | Fire when the value is `gte` or `lte` the threshold | — |
| `--threshold <number>` | Yes | Threshold value | — |
| `--window <window>` | Yes | Rolling window: `5m`, `10m`, `15m`, `30m`, `1h`, `3h`, `6h`, `24h`, `48h` | — |
| `--comparison <mode>` | No | How the threshold is interpreted: `pct_change` or `absolute` | `pct_change` |
| `--string-label <label>` | For `value_count`/`value_share` | Target value; required for `value_*` metrics, rejected for `label_volume`/`true_rate` | — |
| `--min-label-count <n>` | No | Min labels in-window before firing | `0` |
| `--cooldown <minutes>` | No | Cooldown minutes between firings: `0`, `15`, `60`, `360`, or `1440` | `0` |
| `--slack-channel-id <id>` | No | Slack channel ID (with `--slack-channel-name`) | — |
| `--slack-channel-name <name>` | No | Slack channel name (with `--slack-channel-id`) | — |
### `inf signals alerts` — other verbs [#inf-signals-alerts--other-verbs]
* `edit <alert-id> --signal-id <id>` — change an alert's predicate, Slack target, or enabled toggle (same flags as `create`).
* `archive <alert-id> --signal-id <id>` — archive an alert.
* `get <alert-id> --signal-id <id>`, `list --signal-id <id>` — fetch alerts.
* `list-by-team` / `list-by-project` — list alerts across the active team's signals or within a project.
* `events <alert-id> --signal-id <id>` — list an alert's firing history.
* `state <alert-id> --signal-id <id>` — get an alert's current state snapshot.
* `backtest --signal-id <id> …` — replay a predicate over the last N days (read-only preview).
`edit`, `archive`, `get`, `events`, and `state` all require `--signal-id` in addition to the alert ID.
Run `inf signals alerts --help` for the full flag list on each.
## Common workflows [#common-workflows]
<Metadata text="cli/signals/workflows" />
```bash
# Backfill labels over a past window, then poll the manual run
RUN_ID=$(inf signals run create <signal-id> \
--from 2026-06-01T00:00:00Z --to 2026-06-02T00:00:00Z \
--sample-rate 100 --json | jq -r '.id')
inf signals run poll "$RUN_ID" --signal-id <signal-id>
# Alert when the true-rate drops to/below 0.8 over the last hour
inf signals alerts create --signal-id <signal-id> --name "true-rate drop" \
--metric true_rate --operator lte --threshold 0.8 --window 1h
# Preview how that predicate would have fired over the last 7 days (read-only)
inf signals alerts backtest --signal-id <signal-id> \
--metric true_rate --operator lte --threshold 0.8 --window 1h
```
## Related commands [#related-commands]
* [`inf agent`](/cli/agent) lists the auto-discovered agents a signal can target.
* [`inf span`](/cli/spans) and [`inf trace`](/cli/traces) inspect the items signals label.
# Slack (https://docs.inference.net/cli/slack)
Manage a team's Slack connection: connection status, visible channels, the app install URL, and disconnect.
<Metadata text="cli/slack/commands" />
```bash
inf slack installation # Get a team's Slack connection status
inf slack channels # List channels the bot connection can see
inf slack install-url # Get the Slack app install URL for a team
inf slack disconnect # Revoke and forget a team's Slack installation
```
### Commands [#commands]
| Command | Description |
| ------------------------- | --------------------------------------------- |
| `installation` / `status` | Get a team's Slack connection status |
| `channels` | List channels the bot connection can see |
| `install-url` | Get the Slack app install URL for a team |
| `disconnect` | Revoke and forget a team's Slack installation |
Each verb's options are shown by `inf slack <verb> --help`.
## Related commands [#related-commands]
* [`inf integration slack`](/cli/integration) inspects the same connection (status, channels).
* [`inf notifications`](/cli/notifications) manages Slack notification subscriptions.
# Spans (https://docs.inference.net/cli/spans)
Use `inf span` when you already know you need span-level detail instead of a whole trace tree. It lets you search spans across traces, focus on root or entrypoint spans, inspect captured input/output, and print OpenTelemetry attributes for a single step.
**Alias:** `inf spans`
## `inf span list` [#inf-span-list]
Display spans in the active project.
<Metadata text="cli/spans/list" />
```bash
inf span list
```
**Alias:** `inf span ls`
### Options [#options]
| Flag | Required | Description | Default |
| --------------------------------- | -------- | ------------------------------------------------------------------------------------ | ---------------- |
| `-l, --limit <n>` | No | Maximum number of spans | `25` |
| `--cursor <cursor>` | No | Pagination cursor from a previous response | - |
| `--sort <key>` | No | Sort by `start_time`, `duration_ns`, `cost_total`, or `total_tokens` | Server default |
| `--order <asc\|desc>` | No | Sort order | Server default |
| `--trace-id <id>` | No | Limit results to one trace | - |
| `--text <query>` | No | Search captured span input/output text | - |
| `--scope <all\|root\|entrypoint>` | No | Span scope filter | `all` |
| `--range <preset>` | No | Relative time range: `1h`, `6h`, `12h`, `1d`, `3d`, `7d`, `14d`, `30d`, `90d`, `all` | `1d` |
| `--from <iso>` | No | Start time. Must be used with `--to` | - |
| `--to <iso>` | No | End time. Must be used with `--from` | - |
| `--kind <kind>` | No | Observation kind. Repeat or comma-separate values like `LLM,TOOL,AGENT` | All kinds |
| `--provider <name>` | No | LLM provider filter | All providers |
| `--model <name>` | No | LLM model filter | All models |
| `--service <name>` | No | OpenTelemetry `service.name` filter | All services |
| `--environment <name>` | No | Deployment environment filter | All environments |
| `--user <id>` | No | User ID filter | All users |
| `--session <id>` | No | Session ID filter | All sessions |
| `--agent <name>` | No | Agent name filter | All agents |
| `--status <ok\|error>` | No | Span status filter | All statuses |
| `--filter <expr>` | No | Advanced typed field filter. Repeatable | - |
| `--metadata <expr>` | No | Span attribute filter. Repeatable | - |
| `--resource <expr>` | No | Resource attribute filter. Repeatable | - |
Use `--json` to preserve full trace IDs and span IDs for scripting.
### Examples [#examples]
<Metadata text="cli/spans/list-examples" />
```bash
# Recent spans
inf span list
# Root spans for failed traces in the last day
inf span list --scope root --status error --range 1d
# Spans inside a specific trace
inf span list --trace-id 2f0d2b...
# Search captured IO text for a string
inf span list --text "rate limit"
# Find expensive model spans
inf span list --kind LLM --filter "cost_total>0.05" --sort cost_total --order desc
```
## `inf span get` [#inf-span-get]
Open a single span by trace ID and span ID.
<Metadata text="cli/spans/get" />
```bash
inf span get <trace-id> <span-id>
```
**Alias:** `inf span show`
### Arguments [#arguments]
| Argument | Required | Description |
| ---------- | -------- | ---------------------------- |
| `trace-id` | Yes | Trace ID containing the span |
| `span-id` | Yes | Span ID |
### Options [#options-1]
| Flag | Required | Description | Default |
| --------------------- | -------- | ---------------------------------------------------------- | --------- |
| `--view <view>` | No | `summary`, `io`, `attributes`, `events`, `links`, or `raw` | `summary` |
| `-o, --output <path>` | No | Write raw JSON to a file | - |
### Views [#views]
| View | Use it for |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `summary` | Span identity, timing, status, model/provider, token counts, cost, task/session metadata, and domain-specific summary fields |
| `io` | Captured input, output, input messages, output messages, and retrieval documents |
| `attributes` | Span and resource attributes across string, integer, and floating-point maps |
| `events` | Raw OpenTelemetry events JSON |
| `links` | Raw OpenTelemetry links JSON |
| `raw` | Full span JSON exactly as returned by the API |
### Examples [#examples-1]
<Metadata text="cli/spans/get-examples" />
```bash
# Human-readable summary
inf span get 2f0d2b... 91f8c4...
# Inspect captured prompt/response payloads
inf span get 2f0d2b... 91f8c4... --view io
# Inspect OpenTelemetry attributes
inf span get 2f0d2b... 91f8c4... --view attributes
# Save raw span JSON
inf span get 2f0d2b... 91f8c4... --view raw -o span.json
```
## `inf span facets` [#inf-span-facets]
Inspect available span filter values and counts.
<Metadata text="cli/spans/facets" />
```bash
inf span facets
```
`inf span facets` accepts the same filters as [`inf span list`](#inf-span-list), plus:
| Flag | Required | Description | Default |
| ----------------------------- | -------- | --------------------------------------------------------- | -------------------- |
| `--facet <id>` | No | Facet ID. Repeat or comma-separate values | All supported facets |
| `--search <facet=query>` | No | Search within a facet's values. Repeatable | - |
| `--attribute-key <scope:key>` | No | Fetch values for one `span:` or `resource:` attribute key | - |
| `-l, --limit <n>` | No | Maximum options per facet | `50` |
### Examples [#examples-2]
<Metadata text="cli/spans/facets-examples" />
```bash
# Discover span facets for recent traffic
inf span facets --range 1d
# Search model facet values
inf span facets --facet llm_model_name --search llm_model_name=claude
# List values for a span attribute
inf span facets --attribute-key span:gen_ai.operation.name
```
## Advanced filter syntax [#advanced-filter-syntax]
The span filter language matches [`inf trace`](/cli/traces) except trace aggregate fields such as `span_count` and `llm_span_count` are not valid for span queries.
| Expression | Meaning |
| --------------------------------------- | ----------------------------------------------------------------------------------------- |
| `duration_ms>500` | Numeric comparison. `duration` and `duration_ms` are converted to nanoseconds for the API |
| `total_tokens between:100,500` | Numeric range |
| `span_name~retrieval` | String contains |
| `llm_provider=anthropic` | String equality |
| `service_name in:api,worker` | String set membership |
| `agent_name not-empty` | Non-empty string field |
| `--metadata gen_ai.operation.name=chat` | Span attribute equality |
| `--resource service.version~2026` | Resource attribute contains |
## Common workflows [#common-workflows]
<Metadata text="cli/spans/workflows" />
```bash
# Pick a trace, then inspect its LLM spans
TRACE_ID=$(inf trace list --json | jq -r '.traces[0].traceId')
inf span list --trace-id "$TRACE_ID" --kind LLM --json
# Open the first span from that trace
SPAN_ID=$(inf span list --trace-id "$TRACE_ID" --json | jq -r '.spans[0].spanId')
inf span get "$TRACE_ID" "$SPAN_ID" --view io
```
## Related commands [#related-commands]
* [`inf trace`](/cli/traces) opens whole traces, trees, timelines, and trace exports.
* [`inf inference`](/cli/inferences) inspects gateway request/response records.
* [`inf dashboard`](/cli/dashboard) launches the interactive terminal dashboard.
# Starter Templates (https://docs.inference.net/cli/starter-template)
Browse the starter templates available for new projects, read a template for the create flow, and check a project's provisioning status.
## Commands [#commands]
Run `inf starter-template <verb> --help` for each verb's options. The generated surface covers:
| Verb | Description |
| ------------------------- | ---------------------------------------------------------------- |
| `list` | List published starter templates available for new projects |
| `get` | Get a published template and its version for the create flow |
| `get-provisioning` | Get a project's starter-template provisioning status |
| `get-training-preset` | List a project's training presets with base/judge models |
| `create` / `update-draft` | Create or update a starter-template draft (adds a draft version) |
| `publish-version` | Publish a specific version of a starter template |
| `archive` / `unpublish` | Archive a template or unpublish it |
# Tasks (https://docs.inference.net/cli/task)
Tasks are auto-discovered from spans, grouped by trace. Use `inf task` to inspect a task's rollup summary, cost and latency time series, model distribution, auto-train config, and status history — or to rename, archive, and read its stored config.
## Commands [#commands]
Run `inf task <verb> --help` for each verb's options. The generated surface covers reads and lightweight edits:
| Verb | Description |
| --------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `summary` | A task's rollup summary (or null for an empty window) |
| `config` | Get a task's stored config (name, archived) |
| `rename` | Rename a task |
| `archive` | Archive or unarchive a task |
| `status-history` | List a task's auto-train status transitions |
| `auto-train-configs` / `auto-train-summaries` | List a task's auto-train model configs / rollups |
| `model-distribution` | A task's model usage distribution |
| `requests-time-series` / `tokens-time-series` / `cost-time-series` | Request-count / token / cost time series |
| `error-rate-time-series` / `http-code-time-series` | Error rate and HTTP status-class counts over time |
| `input-size-time-series` / `output-size-time-series` / `payload-size-time-series` | Payload-size time series |
| `duration-percentiles` / `duration-success-percentiles` / `ttft-percentiles` | Latency percentiles |
| `duration-percentile-time-series` / `ttft-percentile-time-series` | Latency percentile time series |
| `peak-throughput` | A task's peak request throughput (per-minute/per-hour) |
| `system-prompt-versions` | List a task's distinct system-prompt versions |
| `activity-by-task` | Per-task request activity + window totals for a batch of tasks |
## Related commands [#related-commands]
* [`inf agent`](/cli/agent) — trace-level grouping; tasks are the finer-grained grouping inside an agent.
# Teams (https://docs.inference.net/cli/team)
A team maps 1:1 to an organization on Inference.net. Every command that talks to the API scopes its request to a team, using a request header, so switching teams is instant and does not require signing in again.
<Note>
Team management commands require session login (`inf auth login`). A project API key cannot list, switch, create, or invite to teams.
</Note>
## `inf team list` [#inf-team-list]
List every team you belong to. The active team is flagged with a green dot (`●`).
<Metadata text="cli/team/list" />
```bash
inf team list
```
**Alias:** `inf team ls`
## `inf team current` [#inf-team-current]
Show the ID, name, and slug of the active team.
<Metadata text="cli/team/current" />
```bash
inf team current
```
If no team is active, the command errors and points you at `inf team switch <id>`.
## `inf team switch` [#inf-team-switch]
Set a different team as the active team for later commands. The CLI looks up your teams to resolve the ID, slug, or name you pass, then stores the resolved team ID in local config. It does not call `organization/set-active`, so it never touches your session's stored active team on the server.
<Metadata text="cli/team/switch" />
```bash
inf team switch [id-or-slug]
```
**Alias:** `inf team use`
### Arguments [#arguments]
| Argument | Required | Description |
| ------------ | -------- | ---------------------------------------------- |
| `id-or-slug` | No | The target team's ID, slug, or exact team name |
### Examples [#examples]
<Metadata text="cli/team/switch-examples" />
```bash
# Choose a team from an interactive list
inf team switch
inf team switch team_abc123
inf team switch acme
inf team switch "Acme Research"
```
<Note>
In a non-interactive shell or with `--json`, pass an ID or slug. The CLI cannot show an interactive selector in these modes.
</Note>
<Note>
Switching teams does not change your active project. Run `inf project list` after switching to see the new team's projects, then `inf project switch <id>` to pick one.
</Note>
## `inf team create` [#inf-team-create]
Create a new team with a default project, and make both active.
<Metadata text="cli/team/create" />
```bash
inf team create <name>
```
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | ----------------------------- |
| `name` | Yes | Display name for the new team |
### Example [#example]
<Metadata text="cli/team/create-examples" />
```bash
inf team create "Acme Research"
```
## `inf team invite` [#inf-team-invite]
Invite a person to the active team by email.
<Metadata text="cli/team/invite" />
```bash
inf team invite <email>
```
### Arguments [#arguments-2]
| Argument | Required | Description |
| -------- | -------- | ------------------------------------- |
| `email` | Yes | Email address of the person to invite |
### Options [#options]
| Option | Required | Description | Default |
| --------------- | -------- | ----------------------------------- | -------- |
| `--role <role>` | No | Role to assign: `member` or `admin` | `member` |
### Examples [#examples-1]
<Metadata text="cli/team/invite-examples" />
```bash
inf team invite teammate@example.com
inf team invite teammate@example.com --role admin
```
<Tip>
Invite to a specific team without switching your active team, using the global `--team` flag:
<Metadata text="cli/team/invite-with-team" />
```bash
inf team invite teammate@example.com --team team_abc123
```
</Tip>
# Traces (https://docs.inference.net/cli/traces)
Use `inf trace` to inspect multi-step LLM workflows captured by Inference Tracing. It mirrors the dashboard trace viewer for headless debugging: list trace summaries, open a trace tree, render a timeline, inspect captured conversation messages, discover filter facets, and queue/download trace exports.
**Alias:** `inf traces`
## `inf trace upload` [#inf-trace-upload]
Upload a JSONL trace file into the active project. The CLI validates the format locally (OTLP or Langfuse export), uploads it in parts, waits for processing to finish, and prints the upload ID plus line counts. Same flow as the dashboard **Upload** button and [`inf dataset upload`](/cli/datasets#inf-dataset-upload).
<Metadata text="cli/traces/upload" />
```bash
inf trace upload <file>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ----------------------------- |
| `file` | Yes | Path to a `.jsonl` trace file |
### Options [#options]
| Flag | Required | Description | Default |
| ------------------- | -------- | -------------------------------------------------------------- | -------------------------- |
| `-n, --name <name>` | No | Upload name shown in the Inference platform | Filename without extension |
| `--no-wait` | No | Return after the transfer completes without polling processing | Off |
Uploaded traces appear under the **Traces** tab once processing completes. Filter them with all-time range and the upload ID (`trace_import_id` in filters):
<Metadata text="cli/traces/upload-examples" />
```bash
# Upload and wait for processing (default)
inf trace upload ./exports/langfuse-traces.jsonl
# Custom upload name
inf trace upload ./traces.jsonl --name prod-replay-2026-04-01
# Queue processing but do not wait
inf trace upload ./traces.jsonl --no-wait
# List traces from a completed upload
inf trace list --range all --filter "trace_import_id=<upload-id>"
```
## `inf trace list` [#inf-trace-list]
Display trace summaries in the active project.
<Metadata text="cli/traces/list" />
```bash
inf trace list
```
**Alias:** `inf trace ls`
### Options [#options-1]
| Flag | Required | Description | Default |
| ---------------------- | -------- | ------------------------------------------------------------------------------------------------- | ---------------- |
| `-l, --limit <n>` | No | Maximum number of traces | `25` |
| `--cursor <cursor>` | No | Pagination cursor from a previous response | - |
| `--sort <key>` | No | Sort by `start_time`, `duration`, `total_cost`, `total_tokens`, `span_count`, or `llm_span_count` | Server default |
| `--order <asc\|desc>` | No | Sort order | `desc` |
| `--range <preset>` | No | Relative time range: `1h`, `6h`, `12h`, `1d`, `3d`, `7d`, `14d`, `30d`, `90d`, `all` | `1d` |
| `--from <iso>` | No | Start time. Must be used with `--to` | - |
| `--to <iso>` | No | End time. Must be used with `--from` | - |
| `--kind <kind>` | No | Observation kind. Repeat or comma-separate values like `LLM,TOOL,AGENT` | All kinds |
| `--provider <name>` | No | LLM provider filter | All providers |
| `--model <name>` | No | LLM model filter | All models |
| `--service <name>` | No | OpenTelemetry `service.name` filter | All services |
| `--environment <name>` | No | Deployment environment filter | All environments |
| `--user <id>` | No | User ID filter | All users |
| `--session <id>` | No | Session ID filter | All sessions |
| `--agent <name>` | No | Agent name filter | All agents |
| `--status <ok\|error>` | No | Trace status filter | All statuses |
| `--filter <expr>` | No | Advanced typed field filter. Repeatable | - |
| `--metadata <expr>` | No | Span attribute filter. Repeatable | - |
| `--resource <expr>` | No | Resource attribute filter. Repeatable | - |
Use `--json` for the raw API payload with full trace IDs, pagination cursors, and aggregate counts.
### Examples [#examples]
<Metadata text="cli/traces/list-examples" />
```bash
# Recent traces
inf trace list
# Failed LLM traces from the last hour
inf trace list --range 1h --kind LLM --status error
# Slow traces by duration, sorted longest first
inf trace list --filter "duration_ms>5000" --sort duration --order desc
# Filter by service and model
inf trace list --service api --model claude-sonnet-4-5
```
## `inf trace get` [#inf-trace-get]
Open a single trace and its spans.
<Metadata text="cli/traces/get" />
```bash
inf trace get <trace-id>
```
**Alias:** `inf trace show`
### Arguments [#arguments-1]
| Argument | Required | Description |
| ---------- | -------- | ---------------------------------------------------------- |
| `trace-id` | No | Trace ID from `inf trace list`; omit to pick interactively |
### Options [#options-2]
| Flag | Required | Description | Default |
| --------------------- | -------- | --------------------------------------------------------------------- | --------- |
| `--view <view>` | No | `summary`, `tree`, `timeline`, `thread`, `spans`, or `raw` | `summary` |
| `--span <span-id>` | No | Highlight a selected span in the tree view | - |
| `--limit <n>` | No | Maximum spans to fetch for this trace | `1000` |
| `--page-size <n>` | No | Spans per API request — the maximum allowed value is the platform cap | `100` |
| `--cursor <cursor>` | No | Span pagination cursor | - |
| `--all` | No | Fetch every span page for this trace (ignores `--limit`) | Off |
| `-o, --output <path>` | No | Write raw JSON to a file | - |
The default summary view prints trace metadata and an ASCII span tree. Use `--view raw` or global `--json` when piping trace data into scripts.
### Examples [#examples-1]
<Metadata text="cli/traces/get-examples" />
```bash
# Summary plus span tree
inf trace get 2f0d2b...
# Text waterfall ordered by span start time
inf trace get 2f0d2b... --view timeline
# Conversation-style view from captured input/output messages
inf trace get 2f0d2b... --view thread
# Persist raw trace + spans JSON for local analysis
inf trace get 2f0d2b... --view raw -o trace.json
```
## `inf trace facets` [#inf-trace-facets]
Inspect available trace filter values and counts. This is useful when building a repeatable query and you do not know the exact model, service, environment, or attribute values in a project.
<Metadata text="cli/traces/facets" />
```bash
inf trace facets
```
### Options [#options-3]
`inf trace facets` accepts the same filter flags as [`inf trace list`](#inf-trace-list), plus:
| Flag | Required | Description | Default |
| ----------------------------- | -------- | --------------------------------------------------------- | -------------------- |
| `--facet <id>` | No | Facet ID. Repeat or comma-separate values | All supported facets |
| `--search <facet=query>` | No | Search within a facet's values. Repeatable | - |
| `--attribute-key <scope:key>` | No | Fetch values for one `span:` or `resource:` attribute key | - |
| `-l, --limit <n>` | No | Maximum options per facet | `50` |
### Examples [#examples-2]
<Metadata text="cli/traces/facets-examples" />
```bash
# Show all default trace facets
inf trace facets --range 7d
# Find model facet values containing "claude"
inf trace facets --facet llm_model_name --search llm_model_name=claude
# Explore values for an OpenTelemetry span attribute
inf trace facets --attribute-key span:gen_ai.operation.name
```
## Advanced filter syntax [#advanced-filter-syntax]
`--filter`, `--metadata`, and `--resource` are repeatable. Quote expressions so your shell does not interpret operators.
| Expression | Meaning |
| --------------------------------------- | ----------------------------------------------------------------------------------------- |
| `duration_ms>500` | Numeric comparison. `duration` and `duration_ms` are converted to nanoseconds for the API |
| `total_tokens between:100,500` | Numeric range |
| `span_count>=3` | Trace aggregate filter |
| `llm_model_name~claude` | String contains |
| `service_name in:api,worker` | String set membership |
| `agent_name not-empty` | Non-empty string field |
| `--metadata gen_ai.operation.name=chat` | Span attribute equality |
| `--resource service.version~2026` | Resource attribute contains |
## `inf trace export` [#inf-trace-export]
Trace exports create downloadable JSONL files for offline analysis, support escalations, or long-running review workflows.
### `inf trace export create` [#inf-trace-export-create]
Queue an export job.
<Metadata text="cli/traces/export-create" />
```bash
inf trace export create --range 7d --kind LLM
```
| Flag | Required | Description | Default |
| ---------------------------------- | -------- | ----------------------------------------- | --------------------- |
| `--from <iso>` / `--to <iso>` | No | Explicit export window | - |
| `--range <preset>` | No | Relative export window | `1d` |
| `--kind <kind>` | No | Observation kind filter | All kinds |
| `--trace-id <id>` | No | Trace ID. Repeat or comma-separate values | - |
| `--session <id>` | No | Session ID filter | - |
| `--user <id>` | No | User ID filter | - |
| `--agent <name>` | No | Agent name filter | - |
| `--model <name>` | No | LLM model filter | - |
| `--status-code <OK\|ERROR\|UNSET>` | No | Span status code filter | - |
| `--attribute <key=value>` | No | Attribute equality match. Repeatable | - |
| `--wait` | No | Poll until the export finishes | Off |
| `--download` | No | Download after the export is ready | Off |
| `-o, --output <path>` | No | Download path | Generated from job ID |
<Metadata text="cli/traces/export-create-examples" />
```bash
# Queue and print a job ID
inf trace export create --range 30d --kind LLM
# Queue, wait, and download in one command
inf trace export create --range 7d --status-code ERROR --wait --download -o failed-traces.jsonl
```
### `inf trace export list` [#inf-trace-export-list]
<Metadata text="cli/traces/export-list" />
```bash
inf trace export list
```
### `inf trace export status` [#inf-trace-export-status]
<Metadata text="cli/traces/export-status" />
```bash
inf trace export status <job-id>
```
### `inf trace export download` [#inf-trace-export-download]
<Metadata text="cli/traces/export-download" />
```bash
inf trace export download <job-id> -o traces.jsonl
```
## Related commands [#related-commands]
* [`inf span`](/cli/spans) lists and opens individual spans.
* [`inf inference`](/cli/inferences) inspects gateway request/response records.
* [`inf dashboard`](/cli/dashboard) launches the interactive terminal dashboard.
# Training Evals & Recipes (https://docs.inference.net/cli/training-eval)
Two related training-side command groups, kept separate from `inf training` so their verbs get their own namespace.
## `inf training-eval` [#inf-training-eval]
Inspect and re-run a training job's mid-training evals.
<Metadata text="cli/training-eval/commands" />
```bash
inf training-eval list # List a training job's mid-training evals
inf training-eval logs # Logs for one mid-training eval (paginated)
inf training-eval refire # Re-run a mid-training eval from a prior eval
```
Run `inf training-eval <verb> --help` for each verb's flags (a training job / eval ID is required).
## `inf training-recipe` [#inf-training-recipe]
Manage reusable training recipes (a config that can be re-applied to new training jobs).
<Metadata text="cli/training-recipe/commands" />
```bash
inf training-recipe list # List public and project recipes for the active team
inf training-recipe get ... # Get a recipe with its base/judge model names
inf training-recipe update ... # Update a project training recipe
```
Run `inf training-recipe <verb> --help` for each verb's flags.
## Related commands [#related-commands]
* [`inf training`](/cli/training) queues and monitors the training runs these evals and recipes belong to.
# Training (https://docs.inference.net/cli/training)
Kick off and manage model training runs from the command line. Discover recipes and trainable base models, queue new runs (with override flags for the trickier `trainingConfig` knobs), cancel in-flight jobs, and zoom in on errors without scrolling through raw logs.
**Alias:** `inf train`
The full training loop is paste-able from the terminal:
<Metadata text="cli/training/paste-able-loop" />
```bash
# 1. Materialize training and eval datasets
inf dataset create -n my-train-split -t training --file ./train.jsonl
inf dataset create -n my-eval-split -t eval --file ./eval.jsonl
# → Datasets ds_trn_abc12 / ds_evl_def34 created.
# 2. Create (or reuse) a rubric the evals will score against
inf eval rubric create -n my-rubric -f ./rubric.md
# → Rubric rub_xyz56 / version rv_ver78 created.
# 3. Pick a recipe and base model
inf training recipes
inf training models
# 4. Queue the training run
inf training create \
--name distill-hardreasoning-qwen3.5-4b-v2 \
--recipe inf-public-training-recipe:qwen-3.5-4b-fft \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56 \
--sample-packing false \
--num-epochs 5 \
--task-id distill-hardreasoning-v2
# → Training job job_90ab12 queued.
# 5. Track progress until it finishes
inf training poll job_90ab12
```
## `inf training models` [#inf-training-models]
Discover the base models you can fine-tune and (with `--judge`) the judge models that can score checkpoints.
<Metadata text="cli/training/models" />
```bash
inf training models
```
### Options [#options]
| Flag | Required | Description | Default |
| --------- | -------- | ------------------------------------------------------------------- | ------- |
| `--judge` | No | List judge models instead of base models (use with `--judge-model`) | Off |
The table shows each model's canonical alias, full name, and ID prefix. Pair with `--json` when scripting to preserve full IDs — those full IDs are what you pass to `--base-model` / `--judge-model` on `inf training create`.
### Examples [#examples]
<Metadata text="cli/training/models-examples" />
```bash
# List base models you can fine-tune
inf training models
# List judge models instead
inf training models --judge
# Dump full IDs for scripting
inf training models --json | jq -r '.[].id'
```
## `inf training recipes` [#inf-training-recipes]
Recipes bundle a base model + judge model + GPU plan + full `trainingConfig`. `inf training recipes` lists everything visible to the active project (public recipes + the project's own recipes).
<Metadata text="cli/training/recipes" />
```bash
inf training recipes
```
### Options [#options-1]
| Flag | Required | Description | Default |
| -------------------- | -------- | -------------------------------------- | ------- |
| `--include-archived` | No | Include archived recipes | Off |
| `--public-only` | No | Show only public recipes | Off |
| `--project-only` | No | Show only the active project's recipes | Off |
<Warning>
Only super-admins can fork a public recipe into a project recipe. If you need to customize a recipe's `trainingConfig`, use the override flags on [`inf training create`](#inf-training-create) rather than trying to clone the recipe.
</Warning>
## `inf training recipes get` [#inf-training-recipes-get]
Inspect a specific recipe, including its full `trainingConfig`. Useful for spotting knobs you may want to override at queue time.
<Metadata text="cli/training/recipes-get" />
```bash
inf training recipes get <id>
```
### Arguments [#arguments]
| Argument | Required | Description |
| -------- | -------- | ----------- |
| `id` | Yes | Recipe ID |
### Examples [#examples-1]
<Metadata text="cli/training/recipes-examples" />
```bash
# List recipes visible to the active project
inf training recipes
# Inspect a specific recipe (including its trainingConfig)
inf training recipes get inf-public-training-recipe:qwen-3.5-4b-fft
# Only show project-owned recipes
inf training recipes --project-only
```
## `inf training create` [#inf-training-create]
Queue a new training run. You can either specify a recipe (recommended — it pre-fills base model, judge, GPU plan, and `trainingConfig`) or pass individual flags. Override flags let you tweak specific `trainingConfig` fields without forking the recipe.
<Metadata text="cli/training/create" />
```bash
inf training create \
--name <name> \
--training-dataset <id> \
--eval-dataset <id>
```
**Alias:** `inf training queue`
### Options [#options-2]
| Flag | Required | Description | Default |
| -------------------------- | -------- | ------------------------------------------------------------------------------------------------------ | -------------- |
| `-n, --name <name>` | Yes | Display name for the run | — |
| `--training-dataset <id>` | Yes | Training-type dataset ID (also accepts `--training-dataset-id`) | — |
| `--eval-dataset <id>` | Yes | Eval-type dataset ID (also accepts `--eval-dataset-id`) | — |
| `--rubric <id>` | No | Rubric ID — defaults to the system similarity/quality judges when omitted (also accepts `--rubric-id`) | System judges |
| `--rubric-version-id <id>` | No | Pin to a specific rubric version | Latest version |
| `--recipe <id>` | No | Training recipe ID — pre-fills base model, judge, GPU plan, and `trainingConfig` (also `--recipe-id`) | — |
| `--base-model <id>` | No | Base model to fine-tune (overrides the recipe, also `--base-model-id`) | Recipe default |
| `--judge-model <id>` | No | Judge model for evals (overrides the recipe, also `--judge-model-id`) | Recipe default |
| `--num-nodes <n>` | No | Number of training nodes | Recipe default |
| `--gpus-per-node <n>` | No | GPUs per node (1–8) | Recipe default |
| `--task-id <id>` | No | Associate the run with an existing task | — |
| `--sample-packing <bool>` | No | Override `trainingConfig.sample_packing` (`true`/`false`) | Recipe default |
| `--num-epochs <n>` | No | Override `trainingConfig.num_epochs` | Recipe default |
| `--max-steps <n>` | No | Override `trainingConfig.max_steps` (use `-1` for "no cap") | Recipe default |
| `--learning-rate <n>` | No | Override `trainingConfig.learning_rate` | Recipe default |
| `--dry-run` | No | Print the exact tRPC payload that would be sent and exit — nothing is created | Off |
GPU hardware selection is managed server-side and is not configurable from the CLI. When `--rubric-version-id` is omitted, the CLI fetches the latest version of `--rubric` before queueing — pin a version if you need reproducibility across re-runs. Prints the new training job ID along with an `inf training poll <id>` follow-up command. The run status starts as `queued` while datasets are prepared, then moves to `running`.
### Escape hatches for recipe-pinned configs [#escape-hatches-for-recipe-pinned-configs]
The public recipes are opinionated: `sample_packing` is often `true`, and `num_epochs` / `max_steps` are tuned for the typical dataset size. When your dataset is small or shaped differently, those defaults can cause `torchrun` to crash (for example, with `args.max_steps must be set to a positive value if dataloader does not have a length, was -1` — which is what happens when sample packing collapses a tiny dataset into fewer batches than the FSDP shard count).
The `--sample-packing`, `--num-epochs`, `--max-steps`, and `--learning-rate` flags give you override knobs **without** needing to fork the recipe (only super-admins can). Pair them with `--dry-run` to sanity-check the resulting payload before burning a job slot.
### Examples [#examples-2]
<Metadata text="cli/training/create-examples" />
```bash
# Minimal queue with a recipe (rubric omitted — uses system judges)
inf training create \
--name distill-v1 \
--recipe inf-public-training-recipe:qwen-3.5-4b-fft \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34
# Override recipe defaults for a small dataset and verify the payload first
inf training create \
--name distill-hardreasoning-qwen3.5-4b-v2 \
--recipe inf-public-training-recipe:qwen-3.5-4b-fft \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56 \
--sample-packing false \
--num-epochs 5 \
--dry-run
# Use the `queue` alias
inf training queue \
--name distill-v1 \
--base-model model_abc \
--judge-model model_xyz \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56
```
## `inf training list` [#inf-training-list]
Display a table of training runs in the active project.
<Metadata text="cli/training/list" />
```bash
inf training list
```
**Alias:** `inf training ls`
### Options [#options-3]
| Flag | Required | Description | Default |
| ----------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------- | ------------ |
| `-s, --status <status>` | No | Filter by status: `exporting_datasets`, `queued`, `running`, `cycling`, `completed`, `failed`, `cancelled`, or `timed_out` | All statuses |
| `-l, --limit <n>` | No | Maximum number of results | `20` |
| `--offset <n>` | No | Offset for pagination | `0` |
The table shows the run ID (8-char prefix), name, status (color-coded), base model, progress (`currentStep/totalSteps`), current loss, and creation date.
### Examples [#examples-3]
<Metadata text="cli/training/list-examples" />
```bash
# Show only running training jobs
inf training list --status running
# Get the first 50 training runs
inf training list --limit 50
# Page through results
inf training list --limit 20 --offset 40
```
## `inf training get` [#inf-training-get]
View detailed information about a specific training run. Without `--error`, prints the full detail view. With `--error`, dumps only the fields that matter when a run crashed — ideal for triaging `failed` jobs from CI or a script.
<Metadata text="cli/training/get" />
```bash
inf training get <id>
```
### Arguments [#arguments-1]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options [#options-4]
| Flag | Required | Description | Default |
| --------- | -------- | -------------------------------------------------------------------------------------------------- | ------- |
| `--error` | No | Print only the status, status detail, and error message in a highlighted block (for `failed` runs) | Off |
### Output [#output]
The default detail view includes every field on the training job:
| Field | Description |
| ----------------------------------------- | ------------------------------------------------------------------------------- |
| `id`, `name` | Run identifier and display name |
| `status` | One of the status values listed under [`inf training list`](#inf-training-list) |
| `baseModelId` | Base model the run fine-tunes from |
| `adapter` | Adapter type (e.g. LoRA) |
| `currentStep` / `totalSteps` | Progress counters |
| `currentLoss` | Most recent training loss |
| `numNodes` | Number of nodes participating in the run |
| `provider` / `providerRunId` | Underlying training provider and their internal run ID |
| `evalDatasetName` / `rubricName` | Eval configuration (if the run is configured for mid-training evals) |
| `filteredDatasetName` | Training dataset name |
| `startedAt` / `completedAt` / `createdAt` | Lifecycle timestamps |
| `statusDetail` / `errorMessage` | Populated when the run fails or ends in a non-success state |
With `--error`, the output collapses to just `status`, `statusDetail`, and `errorMessage`.
### Examples [#examples-4]
<Metadata text="cli/training/get-examples" />
```bash
# Full detail view
inf training get job_abc123
# Just the error payload for a failed run
inf training get job_abc123 --error
```
## `inf training cancel` [#inf-training-cancel]
Cancel a `queued` or `running` training job. Completed and already-cancelled jobs will reject the call.
<Metadata text="cli/training/cancel" />
```bash
inf training cancel <id>
```
### Arguments [#arguments-2]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options [#options-5]
| Flag | Required | Description | Default |
| ----------- | --------------------------- | ---------------------------- | ------- |
| `-y, --yes` | Yes in non-TTY environments | Skip the confirmation prompt | Off |
In an interactive terminal, the CLI asks for confirmation unless `-y` is passed. In non-TTY environments (CI, scripts) the command refuses to run without `-y`.
### Examples [#examples-5]
<Metadata text="cli/training/cancel-examples" />
```bash
# Cancel interactively
inf training cancel job_abc123
# Cancel non-interactively (required in CI)
inf training cancel job_abc123 --yes
```
## `inf training logs` [#inf-training-logs]
Stream or view log entries for a training run. Logs are color-coded by level, and each line is prefixed with the training phase it came from (`torchrun_init`, `training`, `inference_export`, …) so you can tell setup crashes apart from training crashes at a glance.
<Metadata text="cli/training/logs" />
```bash
inf training logs <id>
```
### Arguments [#arguments-3]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options [#options-6]
| Flag | Required | Description | Default |
| ----------------- | -------- | ----------------------------------------------------------------------------- | ---------- |
| `-l, --limit <n>` | No | Maximum number of log entries | `50` |
| `--level <level>` | No | Filter by log level (e.g. `error`, `warn`, `info`) | All levels |
| `--phase <phase>` | No | Filter by training phase (substring match — e.g. `torchrun_init`, `training`) | No filter |
| `-f, --follow` | No | Continuously poll for new logs (like `tail -f`) | Off |
Log entries are timestamped and color-coded: errors in red, warnings in yellow, info in blue. In follow mode, the CLI polls for new logs every 3 seconds until you press `Ctrl+C`.
### Examples [#examples-6]
<Metadata text="cli/training/logs-examples" />
```bash
# View the last 50 log entries
inf training logs job_abc123
# Stream logs in real time
inf training logs job_abc123 --follow
# Only show error logs
inf training logs job_abc123 --level error
# Only show setup-phase logs (everything before training starts)
inf training logs job_abc123 --phase torchrun_init
```
## `inf training poll` [#inf-training-poll]
Wait for a training run to complete, printing status updates as the status changes.
<Metadata text="cli/training/poll" />
```bash
inf training poll <id>
```
### Arguments [#arguments-4]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options [#options-7]
| Flag | Required | Description | Default |
| -------------------------- | -------- | ------------------------ | ------- |
| `-i, --interval <seconds>` | No | Poll interval in seconds | `10` |
The CLI prints a status line each time the status changes, showing the status, progress, and current loss. It exits automatically when the run reaches `completed`, `failed`, `cancelled`, or `timed_out`. On a `failed` exit, the CLI points you at `inf training get <id> --error` for the full error payload.
### Examples [#examples-7]
<Metadata text="cli/training/poll-examples" />
```bash
# Poll every 10 seconds (default)
inf training poll job_abc123
# Poll every 30 seconds
inf training poll job_abc123 --interval 30
```
## `inf training evals` [#inf-training-evals]
List the mid-training evals for a job — the baseline eval plus one per checkpoint that reached an eval step.
<Metadata text="cli/training/evals" />
```bash
inf training evals <id>
```
### Arguments [#arguments-5]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
The table shows each eval's ID, checkpoint step, status, scores (the similarity/quality pair, or the custom-rubric score when a rubric was used), model source, and creation date.
### Examples [#examples-8]
<Metadata text="cli/training/evals-examples" />
```bash
inf training evals job_abc123
```
## `inf training checkpoints` [#inf-training-checkpoints]
List the checkpoints for a training job, including in-progress uploads.
<Metadata text="cli/training/checkpoints" />
```bash
inf training checkpoints <id>
```
### Arguments [#arguments-6]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options [#options-8]
| Flag | Required | Description | Default |
| ------------------ | -------- | --------------------------------------------------------- | ------- |
| `--completed-only` | No | Hide in-progress uploads; list only finalized checkpoints | Off |
The table shows each checkpoint's ID, step, type, upload status and bytes, cleaned time, size, whether it's resumable / final / auto-eval, its R2 path, and creation date.
### Examples [#examples-9]
<Metadata text="cli/training/checkpoints-examples" />
```bash
# List all checkpoints (including in-progress uploads)
inf training checkpoints job_abc123
# Only finalized checkpoints
inf training checkpoints job_abc123 --completed-only
```
## `inf training download` [#inf-training-download]
Download the final model artifact (`.tar.zst`) produced by a completed training run.
<Metadata text="cli/training/download" />
```bash
inf training download <id>
```
### Arguments [#arguments-7]
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options [#options-9]
| Flag | Required | Description | Default |
| --------------------- | -------- | ----------------------------------------------------------- | ------- |
| `-o, --output <path>` | No | Output file path — defaults to the server-provided filename | — |
### Examples [#examples-10]
<Metadata text="cli/training/download-examples" />
```bash
# Download to the default filename
inf training download job_abc123
# Download to a specific path
inf training download job_abc123 -o ./artifacts/run-1.tar.zst
```
# Update (https://docs.inference.net/cli/update)
Upgrade a globally installed `inf` to the latest version.
<Metadata text="cli/update/run" />
```bash
inf update
```
Detects how `inf` was installed (npm, bun, pnpm, or Yarn Classic v1) and re-installs it at the `@latest` release.
* Works only for **global** installs (`npm install --global @inference/cli`, etc.). A project-local install is not touched — update it with that project's package manager. For `npx`/`bunx`, run the command again to fetch the latest release.
* Yarn **Berry (v2+)** is not supported — it removed the `yarn global` command.
* Not supported for Windows global installs yet.
* Exits non-zero if the update fails. Otherwise it prints a confirmation and exits `0`.
# Uploads (https://docs.inference.net/cli/upload)
Inspect inference uploads you've staged: check their status, list the uploads backed by a project, and get a presigned URL to download the raw payload. The chunked upload write flow itself stays in the SDK.
## Commands [#commands]
Run `inf upload <verb> --help` for each verb's options. The generated surface covers:
| Verb | Description |
| ----------------------------- | ------------------------------------------------------- |
| `list` | List a project's inference uploads |
| `get` | Get an inference upload's status |
| `download-url` | Get a presigned download URL for an inference upload |
| `initiate`/`complete`/`abort` | Begin, finish, or abort a chunked (R2 multipart) upload |
| `archive` / `unarchive` | Archive (soft-delete) or restore an inference upload |
## Related commands [#related-commands]
* [`inf dataset`](/cli/datasets) materializes uploads (and other sources) into eval/training datasets.
# Workers (https://docs.inference.net/cli/worker)
Inspect v1 worker (relay-monolith `workers` / `instances`) observability.
## `inf worker logs` [#inf-worker-logs]
Streamed logs for a v1 worker instance (paginated).
<Metadata text="cli/worker/logs" />
```bash
inf worker logs --worker-id <worker-id> --instance-id <instance-id>
```
### Options [#options]
| Flag | Required | Description |
| --------------------- | -------- | ----------------------------- |
| `--worker-id <id>` | Yes | V1 worker ID |
| `--instance-id <id>` | Yes | Instance ID within the worker |
| `--log-level <level>` | No | Filter to a log level |
| `--limit <n>` | No | Max rows to return |
| `--offset <n>` | No | Rows to skip for paging |
## Generated REST verbs [#generated-rest-verbs]
`inf worker` exposes its read surface as generated commands. Run `inf worker --help` to list them and their flags.
# Analyze Your Traces (https://docs.inference.net/get-started/analyze-traces)
Once traces are flowing to the Inference platform, the next step is figuring out what they're telling you. The dashboard gives you two places to look: the **Trajectories** tab for browsing every span across your account, and the **Agents** tab for a per-agent view with overview, trajectories, signals, analyze harness, and compare models. Halo is our open-source agent-loop optimizer, hosted right inside the Agents dashboard, that reads your traces and writes up what to improve.
This guide assumes you've already [captured your first trace](/get-started/capture-first-trace). If not, start there.
## Two places to look [#two-places-to-look]
<CardGroup cols="2">
<Card title="Trajectories tab" icon="route">
Everything you've captured, across every service and agent. Filter by service, agent, time range, status, model, token usage, latency, errors, and custom span attributes. Open any trace to walk the tree.
</Card>
<Card title="Agents tab" icon="robot">
A per-agent workspace. Pick an agent and you get five sub-tabs: Overview, Trajectories, Signals, Analyze Harness, and Compare Models.
</Card>
</CardGroup>
## Inside the Agents tab [#inside-the-agents-tab]
Click into any agent and you'll see five sub-tabs scoped to that agent.
| Sub-tab | What it shows |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Overview** | High-level metrics for the agent. Run counts, error rate, latency, token usage, and cost over time. |
| **Trajectories** | The same trace data as the global Trajectories tab, pre-filtered to this agent. Filter further by status, time range, model, or any span attribute. Switch to the Sessions view for one row per agent session — click a session to see the full conversation, every tool call, and every span in order. This is where you go to understand "what happened in this one run." |
| **Signals** | Plain-language classifiers run against the agent's traffic, with their label breakdowns and trends. |
| **Analyze Harness** | The Halo workspace. Run Halo on the agent's traces, read past reports, and configure scheduled runs. |
| **Compare Models** | Run the agent against candidate models side by side and compare the results. |
<Tip>
The Agents dashboard groups runs by `agent.id` and `agent.name` from your spans. If your agent runs aren't grouping the way you expect, see [Agent identity](/integrations/traces/agent-identity).
</Tip>
## Run Halo on your traces [#run-halo-on-your-traces]
[Halo](https://github.com/context-labs/halo) (Hierarchical Agent Loop Optimization) is an open-source RLM-based engine for analyzing agent traces and finding things to improve. It reads OpenTelemetry-compatible spans, decomposes them to find systemic failure modes across many runs, and writes up concrete fixes with citations back to specific traces.
You can run Halo two ways:
* **Self-hosted** from the [open-source repo](https://github.com/context-labs/halo). `pip install halo-engine`, point it at a JSONL trace file, and go.
* **Hosted** inside the Inference platform. The Agents tab's Analyze Harness sub-tab runs the same engine against the traces you've already collected, with no extra setup, no trace export, and no separate pipeline.
The hosted version is where most teams start. Open an agent, open Analyze Harness, and either run Halo on demand or put it on a schedule.
### Run Halo on demand [#run-halo-on-demand]
<Steps>
<Step title="Open the Analyze Harness sub-tab">
From the Agents tab, pick the agent you want to investigate and click **Analyze Harness**.
</Step>
<Step title="Pick a trace window">
Choose the time range Halo should review. Tighter windows give Halo more focused signal. A single problem agent over the last 24 hours beats a firehose of everything from the last month.
</Step>
<Step title="Write a prompt (or use the default)">
The default prompt is the same one we use internally most of the time:
<Metadata text="get-started/analyze-traces-default-prompt" />
```text
Analyze the traces in this window. Understand the agent's activity
and identify anomalies, errors, inefficiencies, and opportunities
to improve reliability, latency, cost, and tool usage. Highlight
the top recurring failures, notable tool-call patterns, wasted or
redundant work, slow or high-cost paths, and concrete fixes you'd
recommend. Cite specific trace IDs for every key finding.
```
You can also ask Halo anything specific: "Why is the refund agent timing out on Tuesdays?", "Which tool calls are returning empty results most often?", "Find redundant LLM calls in the planning loop."
</Step>
<Step title="Read the report">
Halo returns a ranked list of findings with evidence pulled directly from your traces. Each finding cites the trace IDs it came from, so you can click straight from a finding into the trace tree that produced it.
</Step>
<Step title="Apply the changes and re-run">
Update prompts, tools, or harness logic based on the findings. Capture a new window of traces, and run Halo again to confirm the issue is gone. This is the HALO loop: trace, analyze, fix, repeat.
</Step>
</Steps>
### Schedule recurring runs [#schedule-recurring-runs]
For agents you ship to production, the higher-leverage move is putting Halo on a schedule so it reviews recent traces automatically.
<Steps>
<Step title="Open the schedule settings">
From the Analyze Harness sub-tab, open the schedules sheet and create a new schedule.
</Step>
<Step title="Pick a cadence and window">
Hourly, daily, weekly, or monthly. The lookback window pre-fills to match the cadence (a daily schedule defaults to a 24-hour window) but you can override it. The runtime caps any single window at 30 days.
</Step>
<Step title="Set the prompt">
The schedule prompt seeds from the same default shown above. Customize it per schedule when you want a recurring run focused on a specific failure mode.
</Step>
<Step title="Review reports as they land">
Each scheduled run produces a new report in the Analyze Harness history rail. Read the latest one, jump back to past reports to track regressions, and ignore runs where Halo finds nothing actionable.
</Step>
</Steps>
## Inspect traces from the CLI [#inspect-traces-from-the-cli]
If you'd rather stay in the terminal, the [Inference CLI](/cli/overview) reads the same trace data the dashboard does.
<Metadata text="get-started/analyze-traces-cli" />
```bash
# Browse recent traces
inf trace list --range 1h
# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline
# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io
```
See [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) for the full reference.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Halo on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO engine, methodology, and benchmarks. MIT licensed.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so Halo and the Agents dashboard group runs correctly.
</Card>
<Card title="Capture more of your stack" icon="puzzle-piece" href="/integrations/traces/overview">
Add tracing to your other providers, frameworks, and agent runtimes.
</Card>
<Card title="Wrap custom work" icon="pen-nib" href="/integrations/traces/manual-spans">
Add spans around retrieval, routing, subprocesses, and unsupported SDKs.
</Card>
</CardGroup>
# Capture Your First Trace (https://docs.inference.net/get-started/capture-first-trace)
Inference Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, and any custom spans you add. Drop the SDK into your app, point it at the Inference platform, and traces start flowing.
This guide gets you from zero to a captured trace. The example uses OpenAI because it is the smallest end-to-end setup. The same flow works for Anthropic, LangChain, LangGraph, the Vercel AI SDK, Vercel Eve, OpenAI Agents, LiveKit Agents, Pi Agent, legacy PI AI, Pydantic AI, and the other [supported integrations](/integrations/traces/overview).
Either path below installs and wires up the Inference Tracing SDK: [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) on npm for TypeScript, or [`inference-tracing`](https://pypi.org/project/inference-tracing/) on PyPI for Python.
To get started with the Inference platform, create a free account at [inference.net](https://inference.net/register).
## Choose a setup path [#choose-a-setup-path]
Installing with AI is the quickest. Use the manual flow if you want to review each change yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to launch a coding agent like [Claude Code](https://code.claude.com/docs/en/overview) to install the tracing SDK, configure export, and wire up your existing LLM clients.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata text="get-started/capture-first-trace-ai-auth" />
```bash
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run tracing instrumentation in your project">
From your project root, run instrumentation in tracing mode.
<Metadata text="get-started/capture-first-trace-ai-instrument" />
```bash
cd /path/to/your/project && inf instrument --mode tracing
```
The command guides you through the following workflow:
* Select a coding agent: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients and agent frameworks.
* Install the tracing SDK and configure export to the Inference platform.
* Wire `setup()` into your app entrypoint so spans start before clients are constructed.
* Add stable service and agent identity so traces group cleanly in the dashboard.
* Review the generated changes before applying them.
<Tip>
Pick `both` instead of `tracing` if you also want to route requests through the Inference Gateway in the same pass.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would. Traces stream to the Inference platform as your code executes.
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and filter by your service name to see the captured trace tree.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to wire it up yourself. The example below uses OpenAI. For other providers and frameworks, see the [Tracing integrations guide](/integrations/traces/overview).
<Steps>
<Step title="Install the SDK">
<CodeGroup>
<Metadata text="get-started/capture-first-trace-manual-install-typescript" />
```bash TypeScript
bun add @inference/tracing openai
```
<Metadata text="get-started/capture-first-trace-manual-install-python" />
```bash Python
pip install 'inference-tracing[openai]'
```
</CodeGroup>
</Step>
<Step title="Configure export">
Set the Inference platform traces endpoint and token before your app starts.
<Metadata text="get-started/capture-first-trace-manual-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
# Get your API key from https://inference.net/dashboard/api-keys/
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="checkout-agent"
```
<Tip>
Use a stable `INFERENCE_SERVICE_NAME` per deployed service. It makes traces easier to filter and compare across environments.
</Tip>
</Step>
<Step title="Initialize tracing early">
Call `setup()` before constructing clients from instrumented SDKs.
<CodeGroup>
<Metadata text="get-started/capture-first-trace-manual-openai" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Reply with just the word hello." }],
max_tokens: 16,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata text="get-started/capture-first-trace-manual-openai" />
```python Python
import os
from inference_tracing import setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply with just the word hello."}],
max_tokens=16,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
If the process is short-lived, always call `shutdown()` before exit so batched spans are flushed.
<Tip>
Longer-lived processes flush differently. A long-lived server (HTTP,
Slack bot, queue worker) memoizes `setup()` and calls `shutdown()` on
`SIGTERM`, not per request. A serverless or edge function instead
flushes per invocation with `tracing.provider.forceFlush()`. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle)
in the quickstart.
</Tip>
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and navigate to the Agents or Trajectories tab. You'll see an LLM span with input messages, output messages, model name, invocation parameters, finish reason, and token counts.
</Step>
</Steps>
### Group calls under an agent [#group-calls-under-an-agent]
A single LLM call is captured automatically. To get an `AGENT` row with stable `agent.id`, `agent.name`, and `session.id` for dashboard grouping, wrap your code in `agentSpan`. Use `manualSpan` inside it for non-LLM steps like tools, retrieval, and validation. The example below reuses the `tracing` and `client` from the step above.
<CodeGroup>
<Metadata text="get-started/capture-first-trace-agent-typescript" />
```typescript TypeScript
import { agentSpan, manualSpan, SpanKindValues } from "@inference/tracing";
await agentSpan(
tracing.tracer,
{
agentId: "hello-agent",
agentName: "Hello Agent",
sessionId: "session-001",
},
async (span) => {
const prompt = "Reply with just the word hello.";
span.setInput(prompt);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: prompt }],
max_tokens: 16,
});
const reply = response.choices[0]?.message.content ?? "";
await manualSpan(
tracing.tracer,
{ spanName: "validate_reply", spanKind: SpanKindValues.CHAIN, input: reply },
async (step) => step.setOutput({ ok: reply.toLowerCase().includes("hello") }),
);
span.setOutput(reply);
},
);
await tracing.shutdown();
```
<Metadata text="get-started/capture-first-trace-agent-python" />
```python Python
from inference_tracing import agent_span, manual_span, SpanKindValues
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="hello-agent",
agent_name="Hello Agent",
session_id="session-001",
) as span:
span.set_input(prompt)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
max_tokens=16,
)
reply = response.choices[0].message.content or ""
with manual_span(
tracing.tracer,
name="validate_reply",
span_kind=SpanKindValues.CHAIN,
input=reply,
) as step:
step.set_output({"ok": "hello" in reply.lower()})
span.set_output(reply)
tracing.shutdown()
```
</CodeGroup>
The trace now shows an `AGENT` row for "Hello Agent" with the LLM call and the `validate_reply` `CHAIN` row nested under it. For the full surface (framework integrations, multi-agent setups, identity propagation), see the [tracing integrations guide](/integrations/traces/overview).
<Note>
Need a different provider or framework? See [Tracing integrations](/integrations/traces/overview) for OpenAI, Anthropic, LangChain, LangGraph, the Vercel AI SDK, Vercel Eve, OpenAI Agents, LiveKit, Claude Agent SDK, Pi Agent, PI AI, Pydantic AI, and more.
</Note>
</Tab>
</Tabs>
That's it. Spans are streaming to the Inference platform and your trace is ready to inspect.
## What gets captured [#what-gets-captured]
| Span data | Examples |
| ------------------ | ---------------------------------------------------------- |
| Inputs and outputs | `input.value`, `output.value` |
| Messages | user, system, assistant, tool, and tool-result messages |
| Tool calls | tool names, IDs, JSON arguments, and tool results |
| Model metadata | model name, provider/system, invocation parameters |
| Usage | prompt, completion, total, and prompt-cache token counts |
| Agent structure | agent spans, framework spans, tool spans, graph/node spans |
| Errors | exception status and error details on failed spans |
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Inspect trace trees in the dashboard and run Halo to find what to improve.
</Card>
<Card title="Add more integrations" icon="puzzle-piece" href="/integrations/traces/overview">
Instrument Anthropic, LangChain, LangGraph, Vercel AI SDK, Vercel Eve, Pi Agent, PI AI, agent frameworks, and more.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so the Agents dashboard groups runs correctly.
</Card>
<Card title="Wrap custom work" icon="pen-nib" href="/integrations/traces/manual-spans">
Add spans around your own orchestration, retrieval, and routing code.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool spans, end to end.
</Card>
</CardGroup>
# Compare Models with AutoEvals (https://docs.inference.net/get-started/compare-models-auto-evals)
AutoEvals answer one question: is there a better model for your workload than the one you run today? Inference platform samples your recent production traffic, replays each sample against a set of candidate models, scores every output with LLM judges, and recommends a model with supporting evidence. You do not build a dataset, write a rubric, or configure a judge.
In the dashboard, AutoEvals live on the **Compare Models** tab of each task and agent.
This guide takes you from integration to your first completed comparison.
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/f0qFrda4btc?rel=0" title="Compare Models with AutoEval | Inference.net" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Before you start [#before-you-start]
AutoEvals run on your real traffic, so Inference platform needs to see that traffic first. Integrate one of these:
* **Gateway.** Route your LLM calls through the Inference Gateway. Requests are recorded and grouped into [tasks](/platform/gateway/tasks). Follow [Record Your First LLM Call](/get-started/record-first-call).
* **Tracing.** Report traces from your app with the tracing SDK. LLM calls are grouped under agents. Follow [Capture Your First Trace](/get-started/capture-first-trace).
Either path works. A comparison is always scoped to one task or one agent, and it samples that scope's traffic from a time window you pick. The more traffic in the window, the more representative the sample, so let your app run normally for a while before you compare.
## Run your first comparison [#run-your-first-comparison]
<Steps>
<Step title="Open the Compare Models tab">
For Gateway traffic: go to **Gateway → Tasks**, open your task, and select the **Compare Models** tab. If your requests do not set a task ID, they are grouped under the **Default** task.
For traced apps: go to **Agents**, open your agent, and select the **Compare Models** tab.
<Frame>
<img src="/images/autoevals/compare_models_tab.png" alt="A task detail page with the Compare Models tab selected, showing the empty state card 'No model comparisons yet' and the Run Comparison button" />
</Frame>
</Step>
<Step title="Start a run">
Click **Run Comparison**. The comparison dialog opens with sensible defaults: a 30 day traffic window, 100 samples per model, and the models recommended for your task pre-selected in the model catalog. For a first run, keep the defaults and click **Run**.
<Frame>
<img src="/images/autoevals/run_comparison_clicked.png" alt="The comparison dialog, showing your current model's pricing, the Traffic window and Samples per model selectors, and the model catalog with recommended models pre-selected" />
</Frame>
The dialog also shows your recent spend on this task and how each candidate's blended price compares to your current model, and warns you if the run would exceed your team's daily eval limit.
</Step>
<Step title="Watch the run execute">
The run report opens immediately and fills in as the run progresses through three stages:
1. **Sampling traffic** - Inference platform pins a deterministic sample of your requests.
2. **Scoring candidate models** - every candidate re-runs the samples and is judged.
3. **Generating insights** - the scored results are summarized into a verdict and findings.
Results fill in as each model finishes scoring. You can cancel a run at any time.
<Frame>
<img src="/images/autoevals/run_in_progress.png" alt="An in-progress run showing the Analysis in progress card with the Sampling traffic, Scoring candidate models, and Generating insights steps" />
</Frame>
</Step>
<Step title="Read the outputs">
When the run completes, the report leads with a verdict: switch to a specific model, or keep your current one. Below it you get the estimated monthly savings, key findings from the judged samples, a quality versus cost chart, and a leaderboard that ranks every candidate against your current model on quality, similarity, latency, and price.
<Frame>
<img src="/images/autoevals/run_completed.png" alt="A completed run report showing the RECOMMENDED MODEL hero with the score delta versus the current model, monthly savings, and the Key findings list" />
</Frame>
From the leaderboard, open the **Sample Viewer** to read individual model outputs side by side with the judge's reasoning.
</Step>
</Steps>
## What you just learned [#what-you-just-learned]
* AutoEvals measure candidate models on your own production traffic, not a public benchmark
* Inference platform samples your traffic and scores it with two fixed rubrics, quality and similarity
* The result is a concrete recommendation with evidence you can inspect down to the individual sample
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Run an AutoEval" icon="play" href="/platform/eval/auto-evals/run-an-auto-eval">
The full reference for starting runs: configuration, sampling, scoring, and limits.
</Card>
<Card title="Interpret the results" icon="chart-column" href="/platform/eval/auto-evals/interpret-results">
Read the run report and drill into judged samples.
</Card>
<Card title="Train your own model" icon="brain" href="/platform/train/autotrainer">
Push past off-the-shelf quality with a model distilled from your traffic.
</Card>
</CardGroup>
# Record Your First LLM Call (https://docs.inference.net/get-started/record-first-call)
In the Inference platform, production LLM traffic serves as the backbone for model optimization. Inference Gateway records LLM traffic between your application and your current LLM provider, and stores it for later evaluation and training.
This guide shows two ways to capture your first LLM call through Gateway: use the [Inference CLI](/cli/overview) to automatically instrument your codebase, or wire it up manually.
To get started with the Inference platform, create a free account at [inference.net](https://inference.net/register).
## Choose a setup path [#choose-a-setup-path]
Installing with AI is quickest. Use the manual flow if you want to review each change yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to automatically initialize a coding agent like [Claude Code](https://code.claude.com/docs/en/overview) to scan your codebase, update your LLM clients, and add required request metadata.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata text="get-started/record-first-call-ai-auth" />
```bash
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run instrumentation in your project">
Navigate to your project root and run instrumentation.
<Metadata text="get-started/record-first-call-ai-instrument" />
```bash
cd /path/to/your/project && inf instrument
```
The command guides you through the following workflow:
* Select a coding agent to use: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain,etc
* Redirect base URLs to the gateway
* Add routing headers so requests are authenticated, forwarded, and traced
* Add task IDs so each call site is grouped automatically in the dashboard
* Review the generated changes before applying them
<Tip>
Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would to produce inference requests. Requests from your application are now routed through the gateway and will appear in the dashboard.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to see request details, traces, and analytics.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to review each change yourself. The example below uses OpenAI. For Anthropic, Cerebras, Groq, and other providers, see the [Integrations guide](/integrations/overview).
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenAI API key** — from your [OpenAI account](https://platform.openai.com/api-keys)
Set them as environment variables:
<Metadata text="get-started/record-first-call-manual-env" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point your SDK at `https://api.inference.net/v1` and use your Inference project API key as the SDK's `apiKey`. Your provider's API key goes in the `x-inference-provider-api-key` header so the gateway can forward it. The gateway adds roughly 10ms of latency and forwards your requests to the provider as-is.
<CodeGroup>
<Metadata text="get-started/record-first-call-manual-openai" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
<Metadata text="get-started/record-first-call-manual-openai" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
<Metadata text="get-started/record-first-call-manual-openai-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
</Step>
<Step title="Send a request">
Run the snippet above from your application or terminal. Once the request completes, Inference platform will capture it automatically.
</Step>
<Step title="View your results">
Open the [Inference dashboard](https://inference.net/dashboard) to inspect the request, traces, and metrics.
</Step>
</Steps>
<Note>
Need provider-specific manual instructions? See [Integrations Overview](/integrations/overview).
</Note>
</Tab>
</Tabs>
That's it. Every request now flows through the Inference platform and gets captured automatically.
## What gets captured [#what-gets-captured]
Once traffic is flowing, Inference platform records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency (end-to-end and time to first token)
* Token counts (input and output)
* Error rates and status codes
* Model and provider
## Where to find your data [#where-to-find-your-data]
* **[Metrics Explorer](/platform/gateway/metrics-explorer)** - dashboards for cost, latency, errors, and usage across all your LLM calls
* **[Inference Viewer](/platform/gateway/inference-viewer)** - browse and filter individual requests and responses
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Connect more providers" icon="plug" href="/integrations/overview">
Set up Anthropic, Cerebras, Groq, and other providers.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have data? Upload a JSONL file to start evaluating or training.
</Card>
</CardGroup>
# Signals (https://docs.inference.net/get-started/signals)
Once traces are flowing, signals turn them into structured labels. A signal is a plain-language classifier you define once and Inference platform runs continuously against one of your agents. Describe what you want to detect ("is this NSFW?", "did the user get frustrated?", "was the task completed?") and every matching trace gets labeled automatically by an LLM judge, with the results queryable right alongside your other agent metrics.
Signals are how you evaluate the things default metrics miss. For a user-facing agent they tell you how people are actually interacting with it (frustration, sentiment, jailbreak attempts). For a non-user-facing agent they tell you how the agent itself is behaving (did it refuse, did it complete the task, did it stay on policy).
<Tip>
Want the full walkthrough with a worked example instead of the reference? See the guide [Measure your agent's quality with Signals](/guides/measure-agent-quality).
</Tip>
## Before you start [#before-you-start]
Signals run on captured traffic, so you need traces flowing first. This guide assumes you've already [captured your first trace](/get-started/capture-first-trace). You also need:
* **An agent with traces installed.** Signals are always scoped to a single agent, so there is nothing to label until that agent is emitting traces.
* **A stable, consistent `agentId`.** Signals are per agent, so your traces need a consistent agent identity to group by. Set the `agentId` once and keep it the same across runs. If you instrumented with the CLI this is usually already set. See [Set agent identity](/integrations/traces/agent-identity).
* **A `sessionId`, if you want session-scoped signals.** Session signals classify a whole conversation, which requires that your traces carry a `sessionId` (your conversation or chat ID) so Inference platform can assemble the conversation. See [Choose a scope](#choose-a-scope).
## Where to find signals [#where-to-find-signals]
You can get to signals two ways in the dashboard:
* The **Signals** tab, which lists every signal and is the main table view across your agents.
* The **Agents** tab, where you pick an agent and open its **Signals** view to see and create signals scoped to just that agent.
## How signals work [#how-signals-work]
A signal evaluates traffic for one of your agents. When you activate it, Inference platform samples incoming traffic for that agent, sends each sampled target's input and output to an LLM judge with the prompt you wrote, and writes the label back. You can then filter, chart, and break down your traces by that label.
<Steps>
<Step title="You define the signal">
Pick a scope and classifier type, write a prompt describing what to look for, and set a sample rate.
</Step>
<Step title="Inference platform samples matching traffic">
For an active signal, a deterministic share of the agent's traffic (the sample rate) is selected for classification. Sampling is deterministic, so the same target always resolves the same way.
</Step>
<Step title="A judge labels each target">
The judge reads the input and output of each sampled span, trace, or session and returns a label that conforms to your classifier type.
</Step>
<Step title="Labels land on your traces">
Results are stored against each labeled target and surfaced in the dashboard, where you can filter by label value and watch the label distribution over time.
</Step>
</Steps>
## Choose a scope [#choose-a-scope]
A signal's **scope** is the unit it labels. You pick it when you create the signal, and it's fixed for the life of the signal (everything else is editable). Scope determines how much context the judge sees on each call.
<CardGroup cols="3">
<Card title="Span" icon="dot">
A single model call. The narrowest scope. Useful for narrow, call-level checks, but often too granular if you care about the interaction as a whole.
</Card>
<Card title="Trace" icon="route">
One turn, or request. The judge sees the whole turn rather than a single call. A good fit for request-level outcomes.
</Card>
<Card title="Session" icon="comments">
The full conversation. Usually the most useful scope for understanding a user, since the judge sees the entire back-and-forth. Requires a `sessionId` on your traces so the conversation can be assembled.
</Card>
</CardGroup>
<Note>
Session scope only works if your traces carry a `sessionId`. If you haven't set one, add it before creating a session-scoped signal. See [Set agent identity](/integrations/traces/agent-identity).
</Note>
## Two classifier types [#two-classifier-types]
When you create a signal you choose how it labels each target.
<CardGroup cols="2">
<Card title="Binary (yes / no)" icon="toggle-on">
A true/false classifier. Use it for "is this X or not" questions: NSFW content, jailbreak attempts, refusals, or any flag you want to filter on. No labels to configure, just a prompt.
</Card>
<Card title="String (enumerated labels)" icon="tags">
A classifier that returns one of a fixed set of labels you define. Use it when there's more than two outcomes: sentiment (positive / neutral / negative), task outcome (completed / partial / failed / abandoned), and so on. Define between 2 and 10 labels.
</Card>
</CardGroup>
## Create a signal [#create-a-signal]
Create a signal from the **Signals** tab, or from a specific agent's **Signals** view under the **Agents** tab. Either way the signal is scoped to one agent.
<Steps>
<Step title="Open the signal editor for your agent">
From the Signals tab or an agent's Signals view, pick the agent whose traffic you want to label and create a new signal.
</Step>
<Step title="Name the signal">
Give it a short, descriptive name like `NSFW` or `User frustration`. The name is how the label shows up everywhere else in the dashboard.
</Step>
<Step title="Choose a scope">
Pick **Span**, **Trace**, or **Session** (see [Choose a scope](#choose-a-scope)). This is the one setting you can't change later, so pick the unit you actually want to reason about. Session is usually the most useful for understanding a user.
</Step>
<Step title="Choose a classifier type">
Pick **Binary (yes / no)** or **String (enumerated labels)**. For a string classifier, add the 2 to 10 labels the judge is allowed to return.
</Step>
<Step title="Write the prompt">
Describe what the signal should classify. This is the instruction the judge follows on every target, so be specific about what counts as each outcome. For a binary signal, describe what makes a target a "yes." For a string signal, describe when to pick each label.
</Step>
<Step title="Set the sample rate">
The sample rate is the share of matching traffic that gets classified, and lower rates cost less. 100% runs on every target, which is often unnecessary at high volume, so to spend fewer credits pick a lower rate like 25% or 50% and the signal only runs on that share of incoming traffic. Start lower on high-volume agents and raise it once you trust the labels. Common presets are 10%, 25%, 50%, and 100%.
</Step>
<Step title="Save as a draft or activate">
**Save draft** keeps the signal unpublished so you can keep tuning it. **Activate** publishes it and starts live classification on new traffic.
</Step>
</Steps>
<Tip>
Don't want to start from scratch? Use **Start from a template** to prefill the classifier type, prompt, and labels for a common signal, then edit from there. See [Templates](#templates) below.
</Tip>
## Test before you activate [#test-before-you-activate]
Before you commit a signal to live traffic, run it against recent traffic to preview how it labels it. A test run classifies a small sample synchronously and shows you the label distribution and per-target results, and **nothing is saved**. It's a preview only and doesn't affect your signal or store any labels.
<Steps>
<Step title="Open the tester">
From the signal, choose **Test it**.
</Step>
<Step title="Pick a sample size and time range">
Choose how many recent targets to classify (1 to 100) and the window to pull them from.
</Step>
<Step title="Read the preview">
You'll see the label distribution across the sample plus a per-target breakdown, including which targets got flagged. If the labels don't match your intent, adjust the prompt or labels and test again.
</Step>
</Steps>
<Note>
Test results are not persisted. They exist only to help you tune the prompt before activating.
</Note>
## Activate and run live [#activate-and-run-live]
Activating a signal starts live classification: as new traffic arrives for the agent, the configured share of it gets sampled and labeled automatically. You don't have to do anything else, and labels accumulate as traffic flows.
A signal is always in one of three states:
| State | What it means |
| ------------ | ------------------------------------------------------------------ |
| **Draft** | Saved but not running. Nothing is being classified. |
| **Active** | Live. New matching traffic is sampled and labeled. |
| **Disabled** | Paused. Live classification has stopped, but past labels are kept. |
You can disable an active signal at any time to stop classification without losing the labels you've already collected, and re-enable it later.
## Backfill historical data [#backfill-historical-data]
Live classification only labels traffic that arrives after you activate. To label traffic you already captured, run a **manual run** (backfill) at any time. Unlike a test, a manual run **saves** its labels: they're stored against your traces and tied to the run, exactly like live labels.
<Steps>
<Step title="Open the manual run dialog">
From the signal, choose **Manual run / Backfill**.
</Step>
<Step title="Pick a time range and sample rate">
Choose the historical window to apply the signal to, and the share of matching traffic in that window to classify.
</Step>
<Step title="Start the run">
The run executes in the background, classifying past traffic across the window. Results land in the same place as live labels as the run progresses.
</Step>
</Steps>
<Tip>
Backfill a representative window first to sanity-check the labels at scale before running it over a long history. A manual run classifies real traffic and counts toward usage.
</Tip>
## Read the results [#read-the-results]
There are a few places to read what a signal found:
* **The Signals tab** is the main table across your agents, with each signal's current state and headline numbers at a glance.
* **The agent's Overview** under the Agents tab has per-signal graphs: trends and volume over time, plus a range of metrics for each signal so you can see how a label is moving.
* **The signal detail view** has a table of every classified target (the spans, traces, or sessions the signal labeled). Click into any row to open the underlying trace, span, or session and read the actual conversation that produced the label.
Labeled targets render their label as a colored chip. For a binary signal, "yes" and "no" get distinct colors; for a string signal, each label gets its own color. From there you can:
* **Filter by label value** to pull up just the targets a signal flagged (for example, every session labeled "yes" by a jailbreak signal).
* **Watch the distribution over time** to see how a label trends across hours or days.
* **Jump straight to the underlying trace, span, or session** to see the full context and conversation behind any labeled target.
## Alert on a signal [#alert-on-a-signal]
Once a signal is running, set up alerts so you hear about changes without watching the dashboard. Alerts are configured per signal, and you can get notified through the Slack integration or by email.
An alert fires when a metric crosses a condition over a window. You choose:
* **The metric.** What to watch, depending on the classifier type:
* **Label volume** (any signal): how many labels came in.
* **True rate** (binary signals): the share of labels that are "yes."
* **Value count** (string signals): how many labels landed on one specific value.
* **Value share** (string signals): that value's share of all labels.
* **The comparison.** Either a **percentage change** versus the prior equal-length window (for example, "this value's count is up 10% in the last 24 hours") or an **absolute threshold** (for example, "true rate is below 80%").
* **The window** the comparison runs over, from 5 minutes up to 48 hours.
* **A minimum label count** so quiet periods don't trip the alert on statistically meaningless deltas.
* **A cooldown** so a flapping condition doesn't bombard you. After a firing, the next one is suppressed until the cooldown elapses.
You can **backtest** an alert against recent history to see when it would have fired before you turn it on, and pause or re-enable any alert at any time.
<Tip>
Start with a wide window, a sensible minimum label count, and a cooldown. Tighten the threshold once you've seen how the metric actually moves in the backtest.
</Tip>
## Versioning and editing [#versioning-and-editing]
Signals are versioned. Editing a signal (changing its prompt, labels, classifier type, or sample rate) creates a new version rather than mutating the old one, and the dashboard shows each version after every edit. The current version is the one powering live classification, and labels record which version produced them, so you can change a signal's definition without losing the history of what earlier versions decided.
**Scope is the exception: it's fixed once a signal is created.** Everything else is editable, but to label a different unit you create a new signal.
When you no longer need a signal, archive it. Archiving stops it and removes it from your active list while preserving the labels it produced.
## Templates [#templates]
To get started quickly, create a signal **from a template** and edit from there. Built-in templates include:
| Template | Type | What it flags |
| ---------------------- | ------ | ----------------------------------------------------------------------------------------------------------- |
| **NSFW** | Binary | Spans whose content is sexually explicit, graphic, or otherwise not safe for work. |
| **Jailbreak attempt** | Binary | Spans where the user tries to bypass the model's safety guardrails or system instructions. |
| **Laziness / refusal** | Binary | Spans where the assistant refuses, stalls, or gives a low-effort non-answer instead of completing the task. |
| **User frustration** | Binary | Spans where the user expresses frustration, annoyance, or dissatisfaction with the assistant. |
| **Sentiment** | String | The overall sentiment the user expresses: positive, neutral, or negative. |
| **Task outcome** | String | Whether the task the user asked for was completed: completed, partial, failed, or abandoned. |
Templates are just a starting point. You can write any prompt you want and build a signal from scratch, with either classifier type.
## Manage signals from the CLI and MCP [#manage-signals-from-the-cli-and-mcp]
The dashboard isn't the only way in. Everything above is also available through the [Inference CLI](/cli/overview) and the [MCP server](/integrations/mcp-server), so you can create, run, and read signals from your terminal or straight from an AI coding assistant.
* **CLI.** The `inf signals` command group lists, creates, edits, activates, disables, and archives signals, tests them, kicks off manual runs, and reads labels and distributions.
* **MCP.** The Inference MCP server exposes the same operations as tools (creating signals, activating and disabling them, running backfills, configuring alerts, and querying labels and distributions), so an assistant can set up and inspect signals on your behalf.
## Feed signals into Halo [#feed-signals-into-halo]
Signals pair naturally with [Halo](/get-started/analyze-traces), our agent-loop optimizer. The targets a signal flags are exactly the traffic worth digging into:
* **Improve a behavior.** Point Halo at the spans, traces, or sessions a signal flagged and ask how to fix what they have in common (the refusals, the frustrated sessions, the failed tasks).
* **Decide what to measure.** Talk to Halo about your traces to surface which signals would be most valuable to add in the first place.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Run Halo on your traces to find systemic failure modes and concrete fixes.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so signals attach to the right agent and group cleanly.
</Card>
<Card title="Capture more of your stack" icon="puzzle-piece" href="/integrations/traces/overview">
Add tracing to more providers, frameworks, and agent runtimes so signals have more to label.
</Card>
<Card title="Wrap custom work" icon="pen-nib" href="/integrations/traces/manual-spans">
Add spans around retrieval, routing, and subprocesses so signals can classify them too.
</Card>
</CardGroup>
# Train and Deploy a Custom Model (https://docs.inference.net/get-started/train-and-deploy)
When off-the-shelf models aren't good enough for a specific task, fine-tune one that is. A task-specific model is typically smaller, faster, and cheaper to run than the general-purpose model it replaces, while being more accurate for your workload.
This guide uses the **Customer Support Chatbot** demo project to walk through the full loop: launching a training job, monitoring progress, and deploying the result. No data of your own is needed, the demo project comes with everything pre-loaded.
<Note>
This guide shows each step of the manual training loop. To automate the full loop from your live traffic, see [AutoTrainer](/platform/train/autotrainer).
</Note>
## Start the demo project [#start-the-demo-project]
If you already created the demo project, you're all set — open it from the dashboard and skip to the next section.
Otherwise:
1. From the dashboard, open the **project switcher** (the dropdown with your current project's name, at the top of the sidebar) and click **Create Project**.
2. Under **Explore with a demo project**, select the **Customer Support Chatbot** card, give the project a name, and click **Create & seed project**.
The project comes pre-loaded with the two datasets a training job requires:
| Artifact | Name | Role in training |
| ---------------- | ------------------------ | ---------------------------------------------------------------------------------- |
| Training dataset | `customer-support-train` | The data the model learns from |
| Eval dataset | `customer-support-eval` | A held-out set used to measure learning progress, kept disjoint from training data |
## Train a model [#train-a-model]
<Steps>
<Step title="Create a new training job">
Open **Fine-tuning** in the sidebar and click **New Training Job**.
Select the two datasets from your demo project; the platform applies the default judge automatically:
1. **Training dataset** — `customer-support-train`
2. **Eval dataset** — `customer-support-eval`
<Frame>
<img src="/images/customer-support/cs-training-form.png" alt="Training job setup form with demo datasets selected" />
</Frame>
</Step>
<Step title="Choose a recipe">
Next, pick a [recipe](/platform/train/choose-a-recipe) — a pre-configured training setup with a base model, optimized parameters, and compute config. For this demo, the **smallest recipe** works well and will finish the fastest.
</Step>
<Step title="Launch training">
Review your selections and click **Start Training**. The job will begin shortly.
</Step>
<Step title="Monitor training progress">
During training, the platform periodically runs the `customer-support-eval` dataset through your model-in-progress and an LLM judge scores the outputs. You can watch these mid-training eval scores update in real time.
* **Scores improving** — training is on track
* **Scores degrading** — training may be overfitting. Stop the run from the dashboard or with `inf training cancel`, fix your data, and retrain.
<Frame>
<img src="/images/customer-support/cs-training-graph-scores.png" alt="Mid-training eval chart showing scores improving over time" />
</Frame>
See [Monitor a Training Run](/platform/train/mid-training-evals) for more on reading these charts.
</Step>
</Steps>
## Deploy the trained model [#deploy-the-trained-model]
<Steps>
<Step title="Deploy">
When training completes, your model is automatically registered and ready to deploy. Navigate to **Dedicated** in the sidebar, name your deployment, and click **Deploy**. The GPU spins up in a few minutes depending on model size.
</Step>
<Step title="Call your model">
Once deployed, you call it the same way you'd call any model through the Inference API — same base URL, same headers — just swap the `model` parameter to your trained model's identifier.
<CodeGroup>
<Metadata text="get-started/train-deploy-call[series=train-deploy]" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "your-org/your-trained-model",
messages: [
{
role: "user",
content: "I was charged twice for my subscription, can you help?",
},
],
});
```
<Metadata text="get-started/train-deploy-call[series=train-deploy]" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="your-org/your-trained-model",
messages=[
{
"role": "user",
"content": "I was charged twice for my subscription, can you help?",
}
],
)
```
<Metadata text="get-started/train-deploy-call" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "your-org/your-trained-model",
"messages": [
{
"role": "user",
"content": "I was charged twice for my subscription, can you help?"
}
]
}'
```
</CodeGroup>
Replace `your-org/your-trained-model` with the model identifier shown on your deployment page. See [Call Your Deployment](/platform/deploy/call-your-deployment) for the full setup guide.
</Step>
</Steps>
## What you just learned [#what-you-just-learned]
* **Training** teaches a model your specific task using your data
* **Mid-training evals** give you visibility into whether training is working before it finishes
* **Deployment** puts the trained model behind the same API you already use, no code changes beyond swapping the model name
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Choose a recipe" icon="book-open" href="/platform/train/choose-a-recipe">
Understand recipe tiers and how to pick the right one for your task.
</Card>
<Card title="Launch a training run" icon="rocket" href="/platform/train/launch-a-run">
The full training flow with your own data — cost and duration estimates included.
</Card>
<Card title="Call your deployment" icon="code" href="/platform/deploy/call-your-deployment">
Full production setup for calling your deployed model.
</Card>
<Card title="Monitor with Gateway" icon="chart-line" href="/platform/gateway/overview">
Track your deployed model's cost, latency, and quality over time.
</Card>
</CardGroup>
# Deploy a Hugging Face Model (https://docs.inference.net/guides/deploy-a-hf-model)
Deploy any supported Hugging Face model on Inference.net without managing your own infrastructure. This guide walks through registering a model, creating a deployment, and verifying the endpoint.
## What you'll have when you finish [#what-youll-have-when-you-finish]
* one dedicated deployment serving a Hugging Face model
* a public model identifier you can use with the standard API shape
* a verified smoke test confirming the model is responding
## Before you start [#before-you-start]
* an Inference.net account with deployment access
* the Hugging Face model ID you want to deploy (e.g., `meta-llama/Llama-3.2-1B-Instruct`)
## Step 1: Register the model [#step-1-register-the-model]
Hugging Face models are served as custom models. Register the one you want first — you don't paste a model ID into the deployment flow directly.
In the [dashboard](https://inference.net/dashboard), open **Models**, switch to the **My Models** tab, and click **Create new Model**. Choose the Hugging Face path and provide:
* the Hugging Face model ID
* a Hugging Face token if the model is a gated (private) repo
* the GPUs it can run on — 1–2 H100s or H200s
* the engine — vLLM or SGLang
Model registration is in beta; reach out to the team to have it enabled for your account before you start.
See [Deploy a Hugging Face Model](/platform/deploy/deploy-a-hf-model) for the full registration walkthrough.
## Step 2: Create the deployment [#step-2-create-the-deployment]
Once the model is registered, follow the same deployment creation flow described in [Deploy a Trained Model](/platform/deploy/deploy-a-model). The key difference is that you select your registered Hugging Face model from the deployment picker.
## Step 3: Verify the endpoint [#step-3-verify-the-endpoint]
Send a smoke test request using the deployment's public model identifier:
```bash
curl -N https://api.inference.net/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-d '{
"messages": [
{ "role": "user", "content": "Hello, world!" }
],
"model": "your-team/your-deployment-id",
"stream": true
}'
```
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="API Quickstart" icon="rocket" href="/api/api-quickstart">
Learn the full API surface for calling deployed models.
</Card>
<Card title="Manage and Monitor" icon="server" href="/platform/deploy/manage-and-monitor">
Understand deployment operations, billing, and scaling.
</Card>
</CardGroup>
# Deploy a Trained Model (https://docs.inference.net/guides/deploy-a-trained-model)
This guide is the handoff from model improvement to production validation.
## What you'll have when you finish [#what-youll-have-when-you-finish]
* one deployment for the trained model
* one public model identifier
* one successful smoke test against the deployment endpoint
## Before you start [#before-you-start]
* complete a training run with the [E2E Fine-tuning](/guides/e2e-finetuning) guide
* confirm the trained model still beats or matches the baseline on the eval you trust
## Step 1: review the completed training job [#step-1-review-the-completed-training-job]
Before you create a deployment, inspect the training job detail page for:
* final status
* external job ID
* base model
* current or final loss
* checkpoint evals and average scores
* final model reference / weights
Do not promote a model you cannot explain.
## Step 2: create the deployment [#step-2-create-the-deployment]
In the deployment create flow, you choose:
* deployment name (a matching slug is auto-derived and editable)
* model
* instance type (GPU family and cards per instance, e.g. H100 or H200, 1-2 per instance)
* instance count
The dashboard generates the public deployment identifier in the shape `orgSlug/slug`, where `slug` is the slugified deployment name you set.
## Step 3: copy the public model identifier [#step-3-copy-the-public-model-identifier]
Once the deployment exists, copy the deployment's public model identifier from the overview page. You will use that as the `model` value in a normal API request.
## Step 4: send a smoke test [#step-4-send-a-smoke-test]
Use the deployment's public model identifier with the standard API shape and verify that:
* the request completes successfully
* the output is correct enough for the workflow
* the request shows up in the deployment inferences view
## Step 5: watch the deployed traffic [#step-5-watch-the-deployed-traffic]
After rollout, inspect:
* deployment overview
* instances
* recent deployment inferences
* Gateway analytics for the surrounding workflow
The goal is not just “deployment succeeded.” The goal is “the model behaves correctly under real traffic.”
## Verify it worked [#verify-it-worked]
You should now have:
* one live deployment
* one public model identifier
* one successful deployment request visible in the deployment inferences tab
## What to do next [#what-to-do-next]
<CardGroup cols="2">
<Card title="Gateway" icon="chart-line" href="/platform/gateway/overview">
Keep routing real traffic through the Inference platform so the next eval and training cycle stays grounded in production behavior.
</Card>
<Card title="Manage and Monitor" icon="server" href="/platform/deploy/manage-and-monitor">
Learn more about deployment operations, billing, and scaling.
</Card>
</CardGroup>
# End-to-End Fine-tuning (https://docs.inference.net/guides/e2e-finetuning)
Use this guide when prompt changes are no longer enough and you need to improve the model itself.
## What you'll have when you finish [#what-youll-have-when-you-finish]
* one training job launched against paired training and eval datasets
* a clear training objective tied to the eval baseline
* a checklist for deciding whether the result is worth promoting
## Before you start [#before-you-start]
* a stable quality baseline from an [AutoEval](/platform/eval/auto-evals/run-an-auto-eval)
* both a training dataset and an eval dataset from [Datasets](/platform/datasets/overview)
## Step 1: decide whether this is a training problem [#step-1-decide-whether-this-is-a-training-problem]
Training is worth it when:
* the baseline model consistently fails on the same task pattern
* the eval dataset is representative and stable
* the improvement target matters enough to justify a new model artifact
If the issue is still mostly prompt quality or dataset quality, fix those first.
## Step 2: choose the model improvement path [#step-2-choose-the-model-improvement-path]
Use fine-tuning when the base model is close but not good enough.
Use distillation when the teacher model already performs well and your bigger problem is cost, latency, or serving footprint.
## Step 3: launch the training job with paired datasets [#step-3-launch-the-training-job-with-paired-datasets]
The current self-serve flow expects:
* one training dataset
* one eval dataset
* one eval definition and version
* one base model
Training jobs should be anchored to the same eval rubric you used to decide that the baseline was not good enough.
## Step 4: monitor the run, not just the final status [#step-4-monitor-the-run-not-just-the-final-status]
Watch the training job detail page for:
* queued vs running vs completed vs failed
* current step vs total steps
* current loss
* checkpoint evals and score distribution
* final model reference and weights
Checkpoint evals matter because they tell you whether the model is improving before the job finishes.
## Step 5: decide if the result is good enough to promote [#step-5-decide-if-the-result-is-good-enough-to-promote]
A completed training job is not automatically a production-ready model.
Before promotion, confirm:
* the training job finished successfully
* the model beats or matches the baseline on the eval you trust
* the latency and cost tradeoffs still make sense
## Verify it worked [#verify-it-worked]
You should now have:
* one completed training job or one active job with a clear monitoring path
* one model reference to evaluate for promotion
## What to do next [#what-to-do-next]
<CardGroup cols="2">
<Card title="Deploy a Trained Model" icon="server" href="/guides/deploy-a-trained-model">
Move the trained result into a dedicated serving path and validate it.
</Card>
<Card title="Meet with Us" icon="calendar" href="https://inference.net/meet-with-us/">
Talk to our team if you want help with dataset strategy, distillation, or rollout planning.
</Card>
</CardGroup>
# Find a better model with AutoEvals (https://docs.inference.net/guides/find-a-better-model-with-auto-evals)
AutoEvals answer one question: is there a better model for your workload than the one you run today? Inference platform samples your recent production traffic, replays each sample against a set of candidate models, scores every output with LLM judges, and recommends a model with supporting evidence. You do not build a dataset, write a rubric, or configure a judge.
This guide is the full loop, start to finish: send Inference platform your traffic through the Gateway or the tracing SDK, start a comparison from the **Compare Models** tab, and read the results down to the individual judged sample. Prefer video? The same walkthrough is here:
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/f0qFrda4btc?rel=0" title="Compare Models with AutoEval | Inference.net" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## What you need [#what-you-need]
* A free [Inference account](https://inference.net/register).
* An app or agent that makes LLM calls (any provider or framework).
* A terminal with Node.js or Python, depending on your stack.
## Step 1: Send Inference platform your traffic [#step-1-send-inference-platform-your-traffic]
AutoEvals run on your real traffic, so Inference platform needs to see that traffic first. Pick one integration path. Gateway groups your LLM calls into [tasks](/platform/gateway/tasks); tracing groups them under agents. A comparison is always scoped to one task or one agent.
<Tabs>
<Tab title="Gateway">
The Gateway records LLM traffic between your application and your LLM provider. You point your existing SDK at the Gateway; requests are forwarded to the provider as-is and recorded on the way through.
<Steps>
<Step title="Instrument your app">
The quickest path is the [Inference CLI](/cli/overview), which uses a coding agent to redirect your LLM clients through the Gateway and tag each call site with a task ID:
<Metadata text="guides/auto-evals-gateway-cli" />
```bash
npm install -g @inference/cli && inf auth login
cd /path/to/your/project && inf instrument
```
To wire it manually instead, point your SDK at `https://api.inference.net/v1`, authenticate with your Inference project API key, and pass your provider key and a task ID as headers:
<CodeGroup>
<Metadata text="guides/auto-evals-gateway-manual" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
"x-inference-task-id": "support-chat",
},
});
```
<Metadata text="guides/auto-evals-gateway-manual" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
"x-inference-task-id": "support-chat",
},
)
```
</CodeGroup>
The task ID groups related calls into one task, which is the scope a comparison runs on. Requests without a task ID group under the **Default** task. For the full setup, see [Record Your First LLM Call](/get-started/record-first-call).
</Step>
<Step title="Run your app and verify">
Run your application normally. Then open the dashboard and go to **Gateway → Tasks**: your task appears with its recorded requests.
</Step>
</Steps>
</Tab>
<Tab title="Tracing">
The tracing SDK captures the full execution of your AI app or agent: LLM calls, tool calls, and any custom spans you add. Inference platform groups traced LLM calls under agents.
<Steps>
<Step title="Instrument your app">
With the [Inference CLI](/cli/overview):
<Metadata text="guides/auto-evals-tracing-cli" />
```bash
npm install -g @inference/cli && inf auth login
cd /path/to/your/project && inf instrument --mode tracing
```
To wire it manually, install the SDK, configure export, and call `setup()` before constructing your LLM clients:
<Metadata text="guides/auto-evals-tracing-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
# Get your API key from https://inference.net/dashboard/api-keys/
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="support-agent"
```
<CodeGroup>
<Metadata text="guides/auto-evals-tracing-setup" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI },
});
```
<Metadata text="guides/auto-evals-tracing-setup" />
```python Python
from inference_tracing import setup
from openai import OpenAI
tracing = setup()
```
</CodeGroup>
For agent identity, span flushing, and the framework integrations (LangChain, Vercel AI SDK, and more), see [Capture Your First Trace](/get-started/capture-first-trace).
</Step>
<Step title="Run your app and verify">
Run your application normally. Traces stream to the Inference platform as your code executes. Open the dashboard and go to **Agents**: your agent appears with its captured sessions and traces.
</Step>
</Steps>
</Tab>
</Tabs>
Let your app run for a while before you compare. The more traffic in the window, the more representative the sample.
## Step 2: Run a comparison [#step-2-run-a-comparison]
<Steps>
<Step title="Open the Compare Models tab">
For Gateway traffic: go to **Gateway → Tasks**, open your task, and select the **Compare Models** tab.
For traced apps: go to **Agents**, open your agent, and select the **Compare Models** tab.
<Frame>
<img src="/images/autoevals/compare_models_tab.png" alt="A task detail page with the Compare Models tab selected, showing the empty state card 'No model comparisons yet' and the Run Comparison button" />
</Frame>
</Step>
<Step title="Start a run">
Click **Run Comparison**. The comparison dialog opens with sensible defaults: a 30 day traffic window, 100 samples per model, and the models recommended for your task pre-selected in the model catalog. For a first run, keep the defaults and click **Run Comparison** to launch it.
<Frame>
<img src="/images/autoevals/run_comparison_clicked.png" alt="The comparison dialog, showing the current model's pricing, the Traffic window and Samples per model selectors, and the model catalog with recommended models pre-selected" />
</Frame>
Two settings are worth knowing about:
* **Traffic window and Samples per model** control what gets sampled: fewer samples run cheaper and faster, at the cost of statistical confidence. 100 samples is a good default.
* The **model catalog** lets you deselect models you would never switch to, or add ones outside the recommended set. Your selection is saved for this task or agent and restored on future runs.
The dialog also shows your recent spend on this task and how each candidate's blended price compares to your current model, and warns you if the run's estimated eval cost would exceed your team's credit balance.
</Step>
<Step title="Watch the run execute">
The run report opens immediately and fills in as the run progresses through three stages:
1. **Sampling traffic** - Inference platform pins a deterministic, recency-weighted sample of your requests.
2. **Scoring candidate models** - every candidate re-runs the samples and is judged blind on quality, and on similarity to your current model's behavior.
3. **Generating insights** - an analysis agent summarizes the scored results into a verdict, summary, and findings.
Results fill in as each model finishes scoring, and a full run usually completes in a few minutes. You can cancel a run at any time. When the run finishes, Inference platform sends an email or Slack notification with the headline result.
<Frame>
<img src="/images/autoevals/run_in_progress.png" alt="An in-progress run showing the Analysis in progress card with the Sampling traffic, Scoring candidate models, and Generating insights steps" />
</Frame>
</Step>
</Steps>
## Step 3: Read the results [#step-3-read-the-results]
The report leads with the verdict: switch to a specific model, or keep your current one. Next to it you get the score delta against your current model and the estimated monthly savings, projected from your real traffic volume. Below the verdict, **Key findings** lists the most important observations from the judged samples, and the **Run summary** is a written analysis with citations that link into your real requests and traces.
<Frame>
<img src="/images/autoevals/run_completed.png" alt="A completed run report showing the RECOMMENDED MODEL hero with the score delta and monthly savings, above the Key findings card" />
</Frame>
The core of the report is the chart and the table below the findings.
The **quality versus cost chart** plots every candidate on the same samples; up and to the left is better. It shows the quality-cost frontier for your task: the cheapest model that is still decent, the best balance of quality and price, and how both compare to your current model. A second tab shows similarity versus cost.
<Frame>
<img src="/images/autoevals/quality_vs_cost_graph.png" alt="The Quality vs. cost scatter chart with one point per candidate, the recommended model and the current model labeled" />
</Frame>
The **Comparison Scores** leaderboard ranks every candidate against your current model. **Quality** is how well the model did on the task, graded blind. **Similarity** is how closely its responses match your current model's behavior. Latency and price are listed next to them. Click a row for the judge's verdict on that model.
<Frame>
<img src="/images/autoevals/eval_scores_table.png" alt="The Comparison Scores leaderboard with the current and recommended models badged, showing Quality, Similarity, P90 Latency, and Price columns" />
</Frame>
When you want to see the evidence itself, open the **Sample Viewer** from the leaderboard. It shows every judged sample: the original conversation with the judged turn highlighted, then every model's version of the response side by side with the judge's reasoning. Sort by your current model's score, lowest first, to surface the traffic it handles worst, then check how the recommended model did on those same samples.
Pay attention to the two scores together. High quality with high similarity is the safest switch. High quality with low similarity means the model scores well but behaves differently from your current one, so test it before routing traffic to it.
## Step 4: Switch models [#step-4-switch-models]
When you decide to switch, click **Switch in Code**. It gives you a ready-made prompt for a coding agent (Claude Code, Codex, Cursor) that finds where your code calls the current model and swaps in the new model identifier. If you integrated through the Gateway, that one change is the whole migration: your traffic routes through the Inference platform to the model you selected.
## Where to go next [#where-to-go-next]
<CardGroup cols="2">
<Card title="Run an AutoEval" icon="play" href="/platform/eval/auto-evals/run-an-auto-eval">
The full reference: configuration, sampling, scoring, automatic runs, and limits.
</Card>
<Card title="Interpret the results" icon="chart-column" href="/platform/eval/auto-evals/interpret-results">
Every section of the run report, including failure states.
</Card>
<Card title="Train your own model" icon="brain" href="/platform/train/autotrainer">
Push past off-the-shelf quality with a model distilled from your traffic.
</Card>
</CardGroup>
# HALO Desktop: Optimize an Agent End to End on Your Machine (https://docs.inference.net/guides/halo-desktop-app)
The [Try HALO with a Demo Repo](/guides/try-halo-demo-repo) guide runs this same loop against the hosted [Inference platform](/integrations/traces/quickstart) dashboard. This one does the whole thing **on your machine** with the free, open-source [HALO desktop app](https://github.com/context-labs/halo). Traces live in a local SQLite database, HALO runs against a model provider you configure, and nothing leaves your laptop. No Inference account required.
You will install the app, load a bundled demo dataset of about 1,000 real agent runs with one click, explore the traces in the local viewer, run HALO over them, and open the resulting report directly in a coding agent to apply the fixes. The traces come from a search agent with deliberate, documented flaws, so HALO has something real to find.
<Note>
**Desktop or hosted?** The desktop app is the best way to try HALO end to end for free and keep your traces fully local. For continuous production observability across a team (always-on ingest, multiple users, scheduled HALO runs, Signals, and alerts), the [hosted version](/guides/try-halo-demo-repo) is the better fit. You can start local and move to hosted later without changing your instrumentation; it is the same Inference Tracing under the hood. See [Self-hosting and production](#self-hosting-and-production).
</Note>
Prefer to watch? The video below walks through the same loop end to end. It covers the same ground as this guide in video form, so use whichever you like (or both).
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/yecic5jd060?rel=0" title="HALO Desktop: Optimize an Agent End to End on Your Machine" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Before you start [#before-you-start]
You need:
* The **HALO desktop app** (installed in [Step 1](#step-1-install-the-desktop-app)). HALO v1 ships for **Apple Silicon Macs** and **Ubuntu/Debian Linux (x64)**. Intel Macs, other Linux distributions, and Windows are not supported yet.
* A **model provider key** for running HALO. The desktop app supports OpenAI-compatible and Anthropic-compatible endpoints (OpenAI, Anthropic, OpenRouter, [Inference](https://inference.net), or your own endpoint), and you bring your own key. The app stores it locally and it never leaves your machine.
* A **coding agent** for the last step: Claude Code, Cursor, or Codex. The report opens in any of them with one click, and there is a copy-paste path for everything else.
<Note>
**What "no account" means.** The desktop app, the local trace timeline, and the HALO engine all run with no Inference account and no sign-in. The one thing that always needs a key is a model provider, because HALO calls an LLM to do the analysis. That key can be any provider you already have.
</Note>
## Step 1: Install the desktop app [#step-1-install-the-desktop-app]
Install with the one-line installer:
```bash
curl -fsSL https://inference.net/halo/install.sh | sh
```
The installer detects your platform and downloads the matching release: a signed, notarized DMG on Apple Silicon macOS, or a `-Setup.tar.gz` on Ubuntu/Debian x64. You can also grab a build directly from the [releases page](https://github.com/context-labs/halo/releases).
<Note>
HALO v1 supports **Apple Silicon Macs** and **Ubuntu/Debian Linux (x64)** only. The installer stops with a clear message on Intel Macs, other Linux distributions, and Windows. On a clean Ubuntu/Debian box you may also need a few system libraries first: `sudo apt-get install -y libgtk-3-0 libwebkit2gtk-4.1-0 libayatana-appindicator3-1`.
</Note>
## Step 2: Load the demo traces [#step-2-load-the-demo-traces]
Launch the app. After a short welcome, onboarding lands on the **Import Demo Traces** screen. This is the front door for getting data into HALO: import history from **Langfuse** or **Phoenix**, upload a **JSONL** trace export, connect a **local agent**, or load a demo dataset.
<Frame>
<img src="/images/guides/halo-desktop/hd-import.png" alt="The Import Agent Traces onboarding screen with cards for Import from Langfuse, Import from Phoenix, Import JSONL File, Connect Local Agent, Load Demo Traces, and Read Documentation" />
</Frame>
For this guide, click **Load Demo Traces**. The dialog explains exactly what it does: HALO downloads an allowlisted JSONL export from the public [`inference-net/SearchAgentDemoTraces`](https://huggingface.co/datasets/inference-net/SearchAgentDemoTraces) dataset on Hugging Face, caches it beside your local app data, and queues the normal JSONL importer. No arbitrary remote URLs are accepted.
<Frame>
<img src="/images/guides/halo-desktop/hd-search-traces-import.png" alt="The Load Demo Traces dialog showing the SearchAgentDemoTraces Hugging Face dataset, with Source, Cache, and Import cards and a Load Traces button" />
</Frame>
Click **Load Traces**. The dataset is about 1,000 real runs of a pre-instrumented web-search agent (more on it in [Step 5](#step-5-open-a-trace)), and the import usually finishes in under a minute. You can watch it complete on the **Imports** page, which tracks everything you have ever brought into HALO:
<Frame>
<img src="/images/guides/halo-desktop/he-imports-list-page.png" alt="The Imports page showing a completed file import of 1,005 traces and 16,174 observations" />
</Frame>
<Tip>
You can come back to this screen anytime. The **Import Data** button in the top right of the Data and Imports pages reopens the same import options, and **Settings → Replay onboarding** restarts the whole flow.
</Tip>
## Step 3: Add a model provider [#step-3-add-a-model-provider]
HALO analysis runs against a model you choose, with your own key. Open **Settings** and click **Add Provider** under **Model providers**.
<Frame>
<img src="/images/guides/halo-desktop/hd-settings-page.png" alt="The Settings page showing the Model providers section with an Add Provider button, Data Management with database details, and the HALO Configuration section with the engine installed" />
</Frame>
Pick a provider type, which prefills the base URL, then paste your API key:
* **OpenAI** sets `https://api.openai.com/v1`.
* **Anthropic** sets `https://api.anthropic.com/v1`.
* **Custom OpenAI-compatible** lets you point at OpenRouter, Inference (`https://api.inference.net/v1`), or any other compatible endpoint.
<Frame>
<img src="/images/guides/halo-desktop/hd-add-model-provider.png" alt="The Add provider dialog with a provider type selector, name, base URL, and API key fields" />
</Frame>
The key is stored locally in SQLite, masked in the UI, and never leaves the machine.
While you are in Settings, check the **HALO Configuration** section at the bottom. HALO runs as a local Python engine; this section shows its install status and runtime checks (`git`, `uv`, Python 3.12).
## Step 4: Find your way around [#step-4-find-your-way-around]
The app has four main pages, all in the left sidebar:
* **Data** is the trace timeline: every trace and session in your local workspace, with search, filters, and summary stats.
* **Analysis** is where HALO runs live: start new runs, watch them stream, and revisit every completed report.
* **Imports** tracks trace history brought in from other tools, with per-import status and counts.
* **Settings** holds model providers, the local database details, and the HALO engine install.
Open **Data**. The stats bar across the top summarizes the workspace (traces, spans, LLM spans, errors, tokens, cost), and each row is one run with its input, output, and duration. Failed runs are tinted red, so problems jump out while scrolling. The left rail filters by time window, status, source, service, agent, and model, and the **Traces / Sessions** toggle switches between individual runs and grouped sessions.
<Frame>
<img src="/images/guides/halo-desktop/hd-trace-list-page.png" alt="The Data page showing about 1,000 imported traces with filters on the left, summary stats across the top, and failed runs highlighted in red" />
</Frame>
<Warning>
Imported traces keep their **original timestamps**, not the time you imported them. Set the time window to **All time** so the demo dataset shows up, and do the same when you configure the HALO run in [Step 6](#step-6-run-halo).
</Warning>
## Step 5: Open a trace [#step-5-open-a-trace]
The demo traces all come from one agent, **Traceable Search Agent**: a compact web-search agent that plans in a scratchpad, searches the web through Tavily, extracts pages, scores sources, and writes a final answer. It ships with deliberate, documented flaws (a heuristic source-quality score, an unstructured scratchpad, no enforced answer schema), which is exactly what makes this dataset a good HALO sandbox. The code lives in [`context-labs/SearchAgentDemo`](https://github.com/context-labs/SearchAgentDemo) (Python) and [`context-labs/SearchAgentDemoTS`](https://github.com/context-labs/SearchAgentDemoTS) (TypeScript); you will clone one in [Step 7](#step-7-hand-the-report-to-a-coding-agent).
Click any row on the Data page to open the trace viewer. The **Timeline** view is a waterfall of every span in the run: the agent loop at the top, then each LLM call, tool call, and retriever call laid out against the clock. It makes the shape of a run obvious at a glance, including what ran serially that could have been parallel, and where the time actually went. Use the **Cost heatmap** toggle to shade spans by spend, and zoom with the controls or by double-clicking a span.
<Frame>
<img src="/images/guides/halo-desktop/hd-trace-timeline.png" alt="The Timeline view of a single trace, showing a waterfall of OpenAI Chat Completions calls, web search tool calls, and nested tavily.search retriever spans across an 8 second run" />
</Frame>
Switch to the **Conversation** view to read the same run as a thread: the user's question, the agent's answer, and an expandable list of every span in between. Click any span to inspect it in the side panel, with the full input and output, model, duration, and cost. Open the first LLM span and you can read the agent's entire system prompt, including its operating loop and its known rough edges.
<Frame>
<img src="/images/guides/halo-desktop/hd-conversation-view.png" alt="The Conversation view of a trace with the span list expanded and a side panel showing an LLM span's system prompt, model, and timing details" />
</Frame>
Skim a handful of runs, especially the red ones. You will spot failed searches and retry loops by eye. That is the point of the next step: HALO reads all 1,000 runs and finds the patterns for you.
## Step 6: Run HALO [#step-6-run-halo]
Click **Run Analysis** in the top right of the Data page (or start from the **Analysis** page). The dialog is where you scope the run:
* **Session group or Trace group** picks the unit of analysis.
* The filters (search, window, status, source, service, agent, observed model, scope) narrow which telemetry HALO reads. For the demo dataset, set the window to **All time** and leave the rest open. The live counts at the bottom confirm what the run will cover, about 1,000 traces here.
* **Provider and model** select the provider you saved in [Step 3](#step-3-add-a-model-provider) and which of its models does the analysis.
* **Title and prompt** name the run and tell HALO what to look for.
<Frame>
<img src="/images/guides/halo-desktop/hd-analysis-modal.png" alt="The Run Analysis dialog with Session group and Trace group tabs, telemetry filters, live counts showing 1,000 traces, and provider and model selectors" />
</Frame>
A specific prompt gives sharper results. This run used:
```text
Analyze these traces. Identify the most important failures, latency bottlenecks,
confusing tool behavior, and concrete improvements for the developer.
```
Click **Start run**. The run moves through **Queued → Export → Analysis → Done**, and you can watch every agent step stream in as the engine digs through the traces. The right rail tracks the run details: target, provider, model, and the trace, session, and span counts as they are exported.
<Frame>
<img src="/images/guides/halo-desktop/hd-halo-run-queued.png" alt="A HALO run just after starting, showing the analysis prompt, the Queued, Export, Analysis, Done status pipeline, and run details in the right rail" />
</Frame>
Expect several minutes for a dataset this size; this run covered 1,000 traces and 16,000+ spans in about nine minutes. When it finishes, the report renders in place: ranked findings with citations back to the exact traces each one came from. Click a citation chip to drop into that trace and confirm the finding before you act on it.
<Frame>
<img src="/images/guides/halo-desktop/hd-completed-run-top.png" alt="A completed HALO report showing ranked findings with trace citations, section chips for failures and latency bottlenecks, and an Act on this report panel in the right rail" />
</Frame>
On this dataset, HALO's top findings included:
* **A deterministic tool-argument failure.** The model passes `topic="health"` or `topic="sports"` to the Tavily search tool, which only accepts `general`, `news`, or `finance`, so those searches fail every single time. The fix is to encode the constraint as an enum in the tool schema.
* **Error-prone search and extraction that triggers rework loops.** Failed searches and extractions send the agent back through extra LLM deliberation and retries, burning latency and tokens.
* **Latency dominated by sequential LLM calls.** The slowest runs stack around six back-to-back chat completions plus serial searches that could run concurrently.
* **Confusing telemetry.** The tool wrapper span and the underlying provider span both report ERROR for the same root cause, and the wrapper span is missing its real inputs, which makes debugging and error counts misleading.
The report closes with a prioritized action plan, ordered by return on effort:
<Frame>
<img src="/images/guides/halo-desktop/hd-completed-run-fix-list.png" alt="The developer-focused action plan from the HALO report, listing five fixes ranked by ROI, from Tavily topic correctness through cleaning up tracing semantics" />
</Frame>
<Tip>
The chat box under the report asks follow-up questions by re-running HALO over the same traces with full context. Use it to drill into a finding ("show me more traces with the topic failure") without configuring a new run.
</Tip>
## Step 7: Hand the report to a coding agent [#step-7-hand-the-report-to-a-coding-agent]
The report is a ranked, cited to-do list for the agent's codebase, and the desktop app hands it off directly. The **Act on this report** panel next to every completed run has one-click buttons for **Open in Cursor**, **Open in Claude Code**, and **Open in Codex**, plus **Copy prompt** for any other tool. No MCP server, no wiring.
<Frame>
<img src="/images/guides/halo-desktop/hd-act-on-report-buttons.png" alt="The Act on this report panel with Open in Cursor, Open in Claude Code, and Open in Codex buttons and a Copy prompt option" />
</Frame>
The demo traces came from the search agent repo, so that is where the fixes belong. Clone it first, in whichever language you prefer (the two flavors have the same tools, the same deliberate flaws, and produce the same findings):
<CodeGroup>
```bash Python
git clone https://github.com/context-labs/SearchAgentDemo.git
cd SearchAgentDemo
```
```bash TypeScript
git clone https://github.com/context-labs/SearchAgentDemoTS.git
cd SearchAgentDemoTS
```
</CodeGroup>
Then click the button for your coding agent and run it from the cloned repo. Every run saves its full report to disk (the paths are listed under **Artifacts** in the run details), and the handoff prompt points the coding agent at that file with instructions along the lines of:
```text
Read the HALO trace-analysis report at
~/Library/Application Support/net.inference.halo/halo-runs/<run-id>/report.md
and act on its findings in this codebase: fix the identified failures and
implement the recommended improvements.
```
For this dataset, the fixes land in the tool schemas (`tools.py` / `tools.ts`), the agent loop (`agent.py` / `agent.ts`), and the tracing setup. The report's citations mean the coding agent is not guessing: every recommendation traces back to specific runs you can open in the viewer.
## Step 8: Close the loop [#step-8-close-the-loop]
A fix is not done until new traces from the changed code confirm it. This is where the desktop app's live mode comes in: the same agent repo can stream traces straight into your local timeline while it runs, with nothing crossing the network beyond the agent's own model calls.
1. In the app, open **Import Data → Connect Local Agent**. It shows the local ingest endpoint. Copy the repo's env file and point the agent at it:
```bash
cp .env.example .env
```
```bash
# .env
INFERENCE_API_KEY=sk-... # powers the agent's model calls
INFERENCE_OTLP_ENDPOINT=http://127.0.0.1:8799/v1/traces # send traces to the desktop app, not the cloud
```
`INFERENCE_API_KEY` is what the agent uses to call its model; point it at any provider by also setting `INFERENCE_BASE_URL` (it defaults to Inference's OpenAI-compatible endpoint). The `INFERENCE_OTLP_ENDPOINT` override is the whole trick: instead of shipping spans to the cloud, the agent posts them to the desktop app listening on `127.0.0.1:8799`. (Older SDK versions read the legacy `CATALYST_OTLP_ENDPOINT` alias; current versions prefer `INFERENCE_OTLP_ENDPOINT`.)
2. Run a batch from the repo's bundled query dataset against your patched code. The query set is fixed, so the only variable between before and after is your fix:
<CodeGroup>
```bash Python
uv run search-agent-batch --limit 5
```
```bash TypeScript
bun run search-agent-batch --limit 5
```
</CodeGroup>
Watch the traces land in the Data page as each run finishes.
3. Run HALO again, scoped to the new traces (a recent time window or the new session IDs), and compare the two reports. The findings you fixed should be gone or diminished; pick up the next one.
That is the whole HALO loop, start to finish, running entirely on your machine. Once it clicks here, point the same loop at your own agent.
## Bring your own traces [#bring-your-own-traces]
You do not have to use the demo dataset. The same **Import Agent Traces** options from [Step 2](#step-2-load-the-demo-traces) work on your real data, which makes the desktop app a low-commitment way to try HALO on the agent you actually run:
* **Import from Langfuse.** Paste a Langfuse project's API URL and key pair (public and secret). HALO pulls your trace history over the Langfuse public API and stores it locally. If you would rather keep your existing pipeline and try HALO without importing, you can instead point the [Langfuse SDK at the Inference platform](/integrations/traces/langfuse) with an env-var switch.
* **Import from Phoenix.** Bring historical traces from an Arize Phoenix project into the local timeline the same way.
* **Import JSONL.** Any JSONL trace export with one span per line works, including OpenTelemetry and Inference platform exports.
* **Connect Local Agent.** Stream traces live from any Inference Tracing- or OpenTelemetry-instrumented agent by pointing its OTLP endpoint at `http://127.0.0.1:8799/v1/traces`, as in [Step 8](#step-8-close-the-loop).
<Tip>
HALO groups and analyzes traces by agent identity. For the sharpest reports from imported data, make sure your top-level spans carry a stable `agent.name` (and ideally `agent.id`). For Langfuse specifically, see [Group by agent](/integrations/traces/langfuse#group-by-agent).
</Tip>
## Self-hosting and production [#self-hosting-and-production]
A common question: can you run this in production? Two answers, depending on what you mean.
* **The desktop app** is a single-user, local-first tool. It is ideal for exploring, debugging, and iterating on an agent on your own machine. It is not designed to be a shared, always-on production service.
* **The HALO engine** is open source and MIT licensed ([`context-labs/halo`](https://github.com/context-labs/halo), [`halo-engine` on PyPI](https://pypi.org/project/halo-engine/)). You can self-host it: install the CLI, point it at exported traces, and run it anywhere, including in CI against a fresh trace export on every deploy. That is the self-hosted production path for the analysis step itself.
For continuous, team-scale observability (always-on trace ingest, multiple users, scheduled HALO runs, [Signals](/guides/measure-agent-quality), and alerting), the [hosted Inference platform](/guides/try-halo-demo-repo) is the path we recommend. The instrumentation is identical, so anything you build locally moves over by changing where traces are sent.
## Where to go next [#where-to-go-next]
<CardGroup cols="2">
<Card title="Try the hosted version" icon="cloud" href="/guides/try-halo-demo-repo">
The same loop on the hosted Inference.net dashboard, with scheduled runs, Signals, and the MCP server.
</Card>
<Card title="Do this on your own agent" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Run the trace, analyze, fix loop against your real app from instrumentation onward.
</Card>
<Card title="Bring your Langfuse traces" icon="link" href="/integrations/traces/langfuse">
Point the Langfuse SDK at the Inference platform with an env-var switch, no code changes.
</Card>
<Card title="HALO on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO desktop app, engine, and methodology. MIT licensed.
</Card>
</CardGroup>
# HTML Extraction with Schematron (https://docs.inference.net/guides/html-extraction-with-schematron)
For the full Schematron documentation, see the dedicated [Schematron](/workhorse-models/schematron) page.
# Measure Your Agent's Quality with Signals (https://docs.inference.net/guides/measure-agent-quality)
Your dashboard tracks the metrics every agent has: run count, error rate, latency, tokens, cost. However, these metrics don't accurately tell you whether the agent actually did its job, or whether the people using it are happy. Did it finish the task or quietly give up halfway? Is anyone getting frustrated? Those answers live in the content of your traces, and reading traces by hand obviously doesn't scale.
Signals fix that. A signal is a plain-language classifier you define once and Inference platform runs continuously against your traffic as it comes in. You describe the signal and an LLM judge figures out if that applies to the span, trace, or session automatically. The labels become metrics you can chart, filter by, and alert on.
<Frame>
<img src="/images/guides/measure-agent-quality/diagram.png" alt="Traces flow in, a signal you define classifies each span, trace, or session with an LLM judge, and the labels land back on your traffic as metrics you can filter, chart, and alert on" />
</Frame>
Signals are fully configurable to your needs but their are two overarching things you are typically measuring:
* **How users interact with your agent** — sentiment, frustration, jailbreak attempts, NSFW content.
* **How your agent behaves** — whether it completed the task, refused, or stalled.
You can interact with signals via the **dashboard**, through the [MCP server](/integrations/mcp-server), and from the [CLI](/cli/overview) (`inf signals`). This guide leads with the dashboard.
## Before you start [#before-you-start]
Signals run on your traces, so you need traces flowing first. You need:
* A free [Inference account](https://inference.net/register).
* **Tracing installed and traces arriving.** If you haven't done this, start with [Optimize an Agent End to End](/guides/optimize-an-agent-end-to-end) or the [Tracing Quickstart](/integrations/traces/quickstart) and come back once traces show up.
* **A stable agent identity.** Signals are scoped to one agent, so your runs need a consistent `agentId` to group on. If you instrumented with the CLI this is likely already set.
```typescript
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "gator-flue-agent", // Stable ID every signal attaches to. Keep it constant across deploys.
agentName: "Gator Flue Agent",
sessionId: chatId, // Needed for session-scoped signals. Think Slack thread, chat, or job ID - Basically a single conversation.
},
async (span) => {
span.setInput(input);
const result = await run(agent, input);
span.setOutput(String(result.finalOutput ?? ""));
},
);
```
<Note>
Signals group on `agentId`. If you want **session-scoped** signals (Step 2), you also need a stable `sessionId` on your traces. See [Agent identity](/integrations/traces/agent-identity).
</Note>
## Step 1: Find Signals in the dashboard [#step-1-find-signals-in-the-dashboard]
There are two ways in:
* The top-level **Signals** tab lists every signal across all your agents in one table, with each one's agent, scope, type, a live result breakdown, trend, and volume.
* Inside an agent (**Agents** → your agent → **Signals**) you see just that agent's signals, with their overview graphs next to its other metrics.
Either way, **signals are per agent** — each one classifies the traffic of a single `agentId`. Here's the project-wide table for our team, with the spread we run on Gator:
<Frame>
<img src="/images/guides/measure-agent-quality/all-signals-all-agents.png" alt="The top-level Signals tab showing a table of every signal across the project — session sentiment, sentiment, task outcome, laziness / refusal, and user frustration — each with its agent, type, live result breakdown, 7-day trend, and volume" />
</Frame>
## Step 2: Create a signal [#step-2-create-a-signal]
Open the agent's **Signals** view and create a new signal. You can write your own from scratch or start from a template that prefills the type, prompt, and labels for a common case:
| Template | Type | What it flags |
| ---------------------- | ------ | -------------------------------------------------------------------------- |
| **Task outcome** | String | Whether the task was completed / partial / failed / abandoned. |
| **Sentiment** | String | Overall user sentiment: positive / neutral / negative. |
| **User frustration** | Binary | The user expressing annoyance, repeated correction, or dissatisfaction. |
| **Laziness / refusal** | Binary | The assistant refusing, stalling, or giving a low-effort non-answer. |
| **Jailbreak attempt** | Binary | The user trying to bypass safety guardrails or override the system prompt. |
| **NSFW** | Binary | Content that's sexually explicit, graphic, or not safe for work. |
You're not limited to these — write any prompt and classify anything you can describe. Either way, every signal comes down to a few decisions.
### Classifier type [#classifier-type]
<CardGroup cols="2">
<Card title="Binary (yes / no)" icon="toggle-on">
A true/false classifier for "is this X or not" questions: frustration, refusals, jailbreak attempts, NSFW. No labels to configure, just a prompt.
</Card>
<Card title="String (enumerated labels)" icon="tags">
Returns one of a fixed set of labels you define. Use it when there are more than two outcomes: task outcome (completed / partial / failed / abandoned), sentiment (positive / neutral / negative). Define 2 to 10 labels.
</Card>
</CardGroup>
### Scope [#scope]
A signal classifies one **unit** of traffic at a time:
| Scope | What the judge reads | Best for |
| ----------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| **Span** | A single model call. | Fine-grained checks on one step. Can be a lot of labels on a busy agent. |
| **Trace** | One full request or turn. | Request-level questions about a single run. |
| **Session** | The whole conversation or job, end to end. | Understanding a complete interaction: overall sentiment, or whether a multi-turn task ultimately got done. |
<Warning>
**Session scope needs a `sessionId` on your traces.** Without it there are no sessions to classify. Also note: **scope is fixed once a signal is created** — you can edit the prompt, labels, and sample rate later, but not the scope. Pick deliberately.
</Warning>
The same question can be worth running at more than one scope. We track sentiment two ways on Gator: once per session for the overall vibe of a conversation, and again at a finer scope to spot where things turn. That's why you'll see both in the screenshots.
### Sentiment [#sentiment]
Every signal also carries a **sentiment** that tells Inference platform whether an outcome is good, bad, or neutral. This is what lets alerting reason about direction instead of just movement. A **negative** result trending up, or a **positive** result trending down, is worth flagging; a **neutral** result is just neutral and won't trigger anything on its own.
You set sentiment based on the classifier type:
* **Binary** signals get one sentiment for the whole signal. A Jailbreak attempt or NSFW signal is negative, so any climb is bad news.
* **String** signals get one sentiment per label. On a Sentiment classifier you'd mark `negative` as negative, `positive` as positive, and `neutral` as neutral, so a rise in negative labels or a drop in positive ones gets caught automatically.
This mostly pays off in [alerting](#step-8-get-notified): because each label already knows whether it's good or bad, an alert can watch for "the bad thing going up" without you spelling out the direction every time.
### Sample rate [#sample-rate]
A sliding selector sets how much of your matching traffic actually gets classified, anywhere from a small fraction up to 100%. Every classification is a judge call you pay for, so this is the main cost lever on a signal.
**100% labels every span, trace, or session.** That's fine on a low-volume agent, but if you're running a lot of traffic it's usually far more than you need and can get expensive fast. A 25% or 50% sample gives you a representative trend at a fraction of the spend; raise it later once you trust the labels or want finer resolution. You can change it anytime.
### Write the prompt and save [#write-the-prompt-and-save]
Whatever you're classifying, the **prompt** is the instruction the judge follows on every target, so be specific about what counts as each outcome (and, for a string classifier, what each label means). When it's written, give the signal a short, readable name and **save it as a draft** — that keeps it unpublished so you can test before going live.
<Frame>
<img src="/images/guides/measure-agent-quality/creat-signal.png" alt="The create-signal form (shown here for a Jailbreak attempt signal at session scope), with the template picker, scope and binary type selectors, the classifier prompt, sample-rate presets, and an inline 'try it before you commit' preview" />
</Frame>
<Tip>
Prefer to script it? `inf signals create` does the same from the terminal, and the [MCP server](/integrations/mcp-server) exposes `create_signal` so your coding agent can set one up for you.
</Tip>
## Step 3: Test before you activate [#step-3-test-before-you-activate]
Before committing a signal to live traffic, run it against recent data to preview how it labels. A test classifies a small sample (1–100 recent targets) and shows you the distribution plus the label on each one. **Nothing is saved** — it's a preview, so it doesn't touch your signal or store labels.
If the judge disagrees with you on a few, tighten the prompt and test again. This is the cheapest place to get a signal right, before any labels exist.
<Frame>
<img src="/images/guides/measure-agent-quality/test-run-signal.png" alt="The Test Signal dialog noting results are not persisted, with a sample size and time range, the positive / neutral / negative distribution across the sample, and a per-row table of each span and its label" />
</Frame>
<Note>
The CLI equivalent is
`inf signals verify`
.
</Note>
## Step 4: Activate, pause, and resume [#step-4-activate-pause-and-resume]
When the test looks right, activate the signal. Live classification starts immediately: as new traffic arrives, the configured share is sampled and labeled automatically.
A signal is always in one of three states, and you can move between them anytime:
| State | What it means |
| ------------ | ----------------------------------------------------------------------- |
| **Draft** | Saved but not running. |
| **Active** | Live. New matching targets are sampled and labeled. |
| **Disabled** | Paused. Classification stops, but every label collected so far is kept. |
Pause an active signal whenever you want and resume it later without losing history. Sampling is deterministic, so the share you set is the share you pay for.
## Step 5: Backfill historical data [#step-5-backfill-historical-data]
Live classification only labels traffic that arrives after you activate. To label data you already captured — so your charts have history on day one — run a **manual run** (backfill): pick a historical window and a sample rate, and it classifies past targets in the background. Unlike a test, a backfill **saves** its labels, exactly like live ones. You can run one at any point, not just at creation.
<Frame>
<img src="/images/guides/measure-agent-quality/manual-run.png" alt="The Manual Run / Backfill dialog applying a signal to a historical time range, with From and To dates and a sample-rate slider for the percentage of past targets to classify" />
</Frame>
<Tip>
Backfill a short window first to sanity-check labels at scale before running over months of history — a manual run classifies real targets and counts toward usage. CLI: `inf signals run create`.
</Tip>
## Step 6: Read the results [#step-6-read-the-results]
This is where signals become metrics. Each agent's **Signals** view rolls up every signal it runs: a summary up top (how many are active, total classifications, the busiest signal) and a card per signal with its label breakdown and trend, sitting right next to run count, latency, and cost.
<Frame>
<img src="/images/guides/measure-agent-quality/signal-agent-overview.png" alt="The agent's Signals overview showing summary tiles for active signals, total classifications, and busiest signal, with a card and trend graph for each signal: session sentiment, sentiment, task outcome, and laziness / refusal" />
</Frame>
Open any signal and its detail page charts its classifications over time and the full label distribution.
<Frame>
<img src="/images/guides/measure-agent-quality/signal-details.png" alt="A signal detail page for Task outcome, with stat tiles for classifications, top label, sample rate, and live version, a stacked bar chart of classifications over time, and a donut of the completed / partial / failed / abandoned distribution" />
</Frame>
Below the charts is a **table of every classified target**, with its label. Click any row to drop into that span, trace, or session and read the **actual conversation** behind the label: the full run, the tool calls, the output the judge saw. You can also filter the main **Trajectories** view by a label to pull up just the flagged runs.
<Frame>
<img src="/images/guides/measure-agent-quality/signal-table.png" alt="The labeled-traces table on a signal detail page, one row per classified target with its label, version, session, and time, each clickable through to the underlying trace" />
</Frame>
## Step 7: Edit and version [#step-7-edit-and-version]
Editing a signal's prompt, labels, or sample rate **creates a new version** rather than overwriting the old one. The dashboard shows each version, and every label records which one produced it, so you can sharpen a definition over time without losing the history of what earlier versions decided. (Scope is the one thing fixed at creation.)
When a signal has served its purpose, **archive** it: it stops and leaves your active list, but its labels are preserved.
## Step 8: Get notified [#step-8-get-notified]
A trend you have to remember to check is one you'll miss. There are two ways to stay on top of your signals without logging in: **targeted alerts** for a specific condition, and a **daily overview** that summarizes everything.
### Targeted alerts [#targeted-alerts]
An alert watches one signal and fires when its numbers cross a line. They're configured per signal (a signal can have several), and a global **Alerts** page lists every alert across all your signals.
You build an alert by picking a **metric** to watch, the **direction and threshold** that count as a problem (a level it crosses, or a percentage it moves up or down), and the **time window** to measure over. Which metrics are available depends on the signal's type:
| Metric | Works on | Measures |
| ---------------- | -------- | --------------------------------------- |
| **True rate** | Binary | Share of targets labeled "yes." |
| **Value count** | String | Count of a specific label. |
| **Value share** | String | Share of targets with a specific label. |
| **Label volume** | Any | Total labels produced. |
The threshold reads one of two ways: **percentage change** (the metric moves up or down by X% versus the prior window) or an **absolute level** (it crosses a fixed number). You also set the rolling **window** (5 minutes to 48 hours), an optional **minimum label count** so low volume doesn't trip it, and a **cooldown** so repeat firings don't bury you. As you tune it, the form previews where the alert *would* have fired over the last 7 days.
The **sentiment** you set in Step 2 is what makes the direction obvious here: you watch negative labels for an increase and positive labels for a decrease, since both mean things are getting worse. Neutral labels are rarely worth alerting on at all.
<Frame>
<img src="/images/guides/measure-agent-quality/signals-notification-create.png" alt="The alert create form: a signal and name, the condition built from a metric, a decreases-by / increases-by percentage or absolute level, and a window, plus a minimum label count, a Slack connect prompt, a cooldown selector, and a 7-day 'would have fired' preview chart" />
</Frame>
By default an alert emails your **whole team**, and any member can mute an individual alert for themselves. Connect **Slack** once and you can route an alert's firings to a specific channel instead. A firing tells you the signal, the metric, the threshold, what was observed, and when:
<Frame>
<img src="/images/guides/measure-agent-quality/signal-slack-alert.png" alt="A Slack signal-alert message for Jailbreak attempt reading 'Label volume above 1 over 5m', with fields for the signal, metric, comparison, threshold, observed value, window, and trigger time, and a View alert button" />
</Frame>
<Tip>
The CLI mirrors all of this:
`inf signals alerts backtest`
to preview, then
`inf signals alerts create`
.
</Tip>
### Daily overview [#daily-overview]
Targeted alerts are for when something specific goes wrong. The **daily overview** is for confirming everything's humming along without opening the dashboard. Opt in and you get a once-a-day email (or Slack) summarizing all your signals: how many were classified, and the biggest movers since yesterday. It's the "is anything off?" glance you can do from your inbox.
<Frame>
<img src="/images/guides/measure-agent-quality/daily-signal-notif.png" alt="A daily signal overview email summarizing the day across signals — total classified, the biggest mover called out at the top, and a card per signal with its volume and percentage change — plus a View signals button" />
</Frame>
## Step 9: Hand it to HALO [#step-9-hand-it-to-halo]
Signals tell you **what** is happening and **how often** — failed tasks spiked overnight, frustration is up this week. [HALO](/guides/optimize-an-agent-end-to-end) tells you **why**, and what to change.
The handoff runs through the [MCP server](/integrations/mcp-server). With Inference connected to your coding agent, you go from a signal to a fix in plain language:
```text
The Task outcome signal on gator-flue-agent is showing more failures this week.
Pull the failed sessions, figure out what's going wrong, and propose a fix.
```
It works the other way too — ask HALO what to watch:
```text
Look at gator-flue-agent's recent traces and tell me which signals would catch its most common failure modes.
```
HALO reads the flagged traces, finds the systemic cause, and writes back concrete fixes with citations to the exact runs. See the [MCP server guide](/integrations/mcp-server) for setup.
## Where to go next [#where-to-go-next]
You now have quality metrics that didn't exist an hour ago — defined in a sentence, computed on every run, charted, and wired to notifications — with no extra instrumentation.
<CardGroup cols="2">
<Card title="Signals reference" icon="signal-stream" href="/get-started/signals">
Full reference for classifier types, scopes, sample rates, states, testing, backfills, and templates.
</Card>
<Card title="Optimize an agent end to end" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Run HALO on the traces your signals flag to find the root cause and apply the fixes.
</Card>
<Card title="Try HALO on a demo repo" icon="flask" href="/guides/try-halo-demo-repo">
Run the full trace → HALO → fix loop on a ready-made instrumented agent in about fifteen minutes.
</Card>
<Card title="Connect the MCP server" icon="plug" href="/integrations/mcp-server">
Create signals, read labels, and hand flagged traces to HALO from your coding agent.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent and session IDs so signals attach to the right agent and scope cleanly.
</Card>
</CardGroup>
# Optimize an Agent End to End with Inference Tracing and HALO (https://docs.inference.net/guides/optimize-an-agent-end-to-end)
Most agents have problems you can't always see from the outside: silent failures that still report `OK`, tool calls firing twice, a consistently slow call quietly burning latency and spend. They're hard to catch, and with most tracing platforms you're left parsing spans yourself.
Plugging your traces into an LLM doesn't always fix it either. Traces are usually a huge amount of context such that a general model can't reliably find the patterns across runs that are causing your agent to underperform. That's why we built HALO (Hierarchical Agent Loop Optimizer): a novel way to inspect traces with an RLM, tuned for context-ingestion efficiency so you get maximum signal out of every run.
This guide walks the full loop: **trace, measure, analyze, fix, repeat.** You'll install Inference Tracing, record real traces, explore them in the dashboard, run HALO to find what to improve, and apply the fixes it hands back.
We follow a real example the whole way through: our own internal GTM agent with significant day-to-day traffic.
<Frame>
<img src="/images/guides/e2e-agent-diagram.png" alt="Install tracing once, then a repeating loop: capture traces, run HALO, apply fixes, and back to capturing traces" />
</Frame>
Prefer to watch? The video below walks through the same loop end to end. It covers the same ground as this guide in video form, so use whichever you like (or both).
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/SGKrnYSttBI?rel=0" title="Optimize an Agent End to End with Inference Tracing and HALO" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Before you start [#before-you-start]
You need:
* A free [Inference account](https://inference.net/register).
* An app or agent that makes LLM calls. Any provider or framework works (OpenAI, Anthropic, Gemini, LangChain, LangGraph, the Vercel AI SDK, OpenAI Agents, and [more](/integrations/traces/overview)).
* An API key from the [dashboard](https://inference.net/dashboard/api-keys).
## Step 1: Install tracing [#step-1-install-tracing]
Tracing captures the full execution of your agent: every LLM call, tool call, framework step, and any custom spans you wrap. HALO reads those traces, so the richer your traces, the better the analysis.
The fastest path is to let the [Inference CLI](/cli/overview) drive a coding agent that wires the SDK in for you.
<Steps>
<Step title="Install the CLI and sign in">
```bash
npm install -g @inference/cli && inf auth login
```
Your browser opens to authenticate.
</Step>
<Step title="Run instrumentation in your project">
From your project root:
```bash
cd /path/to/your/project && inf instrument --mode tracing
```
The CLI scans your codebase for LLM clients and agent frameworks, installs the Inference Tracing SDK, wires `setup()` into your entrypoint, adds stable service and agent identity, and shows you every change before applying it.
</Step>
</Steps>
Prefer to wire it in yourself? Install the Inference Tracing SDK directly — [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) for TypeScript or [`inference-tracing`](https://pypi.org/project/inference-tracing/) for Python:
<CodeGroup>
```bash bun
bun add @inference/tracing
```
```bash npm
npm install @inference/tracing
```
```bash pnpm
pnpm add @inference/tracing
```
```bash yarn
yarn add @inference/tracing
```
```bash pip
pip install inference-tracing
```
</CodeGroup>
The [Tracing Quickstart](/integrations/traces/quickstart) guide has the full manual setup for TypeScript and Python, plus how to wrap an agent span so your runs group cleanly in the dashboard.
## Step 2: Record traces [#step-2-record-traces]
Run your app the way you normally would. Traces stream to the Inference platform as your code executes.
The goal at this stage is **volume and variety**: the more real runs you capture, the more signal HALO has to work with. Exercise the paths you actually care about, including the ones that go wrong. Errors, retries, and slow paths are exactly what HALO is looking for.
<Tip>
You can still experiment with HALO using development data but the full value comes from being able to analyze traces of production level traffic.
</Tip>
Confirm traces are arriving before moving on. You can check from the CLI:
```bash
inf trace list --range 1h
```
Or just open the **Trajectories** tab in the dashboard and you should see a list of traces in the table if everything was set up correctly.
<Frame>
<img src="/images/guides/e2e-tracing-traces.png" alt="The Trajectories tab in the Inference dashboard showing a live list of captured traces with columns for time, root span, kind, service, environment, and agent" />
</Frame>
If you see rows here, tracing is live and you're ready to start digging in.
## Step 3: Explore a trace [#step-3-explore-a-trace]
Click any trace and a detail sheet slides open. This is where the real picture of a run lives, and it has a few tabs worth knowing.
### The trace tree [#the-trace-tree]
The first tab shows the top-level trace and every span nested underneath it. Click into any span to expand it. For each one you get:
* **Inputs and outputs**, the exact payload that went in and came back out.
* **Span attributes**, including cost and input/output token counts per span, so you can see precisely which step spent what.
* **Raw JSON**, the unmodified span data, which you can also download for offline use or to feed into your own tooling.
This is the view you reach for when you want to know exactly what happened on a single run, step by step.
<Frame>
<img src="/images/guides/e2e-trace-detail.png" alt="The trace detail sheet open on the trace/spans tab, with the span tree on one side and an expanded span showing its inputs, outputs, span attributes including cost and token counts, and the raw JSON and download controls" />
</Frame>
### The timeline [#the-timeline]
Switch to the timeline tab and the same run is plotted along a timeline. You see the total length of the run, then the duration of each individual span (each agent step, tool call, MCP call, and so on) stacked underneath. This is the fastest way to spot what's actually slow: a single tool call eating ten seconds jumps right out, where it would be easy to miss in the tree.
<Frame>
<img src="/images/guides/e2e-trace-timeline.png" alt="The trace detail sheet on the timeline tab, showing the total run duration up top and the per-span duration bars for tool calls, MCP calls, and model calls below" />
</Frame>
### The conversation [#the-conversation]
The conversation (session) tab is the human-readable view of the whole run. It renders the entire conversation top to bottom, from the first message to the last, the way you'd read a chat. For a chat-style agent that's the back-and-forth with the user. For a job that kicks off subagents, it's the data those subagents return and what the main agent did with it. When you just want to read what happened without parsing spans, this is the tab.
<Frame>
<img src="/images/guides/e2e-trace-thread.png" alt="The trace detail sheet on the session tab, showing the full run rendered as a readable conversation from first message to final output" />
</Frame>
## Step 4: Filter and search [#step-4-filter-and-search]
A list of traces is only useful if you can find the one you want. The Trajectories tab gives you a deep filter panel down the left side: filter by status, model, service, provider, user, session, numeric ranges like token count and cost, and any custom span attribute you've attached.
Search is where it gets powerful. This is not metadata search on names or IDs. It's a deep search across the full content inside every span: the messages, the tool inputs, the tool outputs, the errors, and any custom attributes you've attached. Type a single word that shows up deep inside a chat conversation, a tool's JSON payload, or an error string, and it surfaces every trace where that word appears, combined with whatever filters you have selected. Finding the one run where a user said "refund" and the model errored is a single query, not an afternoon of scrolling.
<Frame>
<img src="/images/guides/e2e-tracing-search.png" alt="The Trajectories tab with a deep search for a keyword and status and model filters applied, showing matching traces with the search term highlighted inside the chat content" />
</Frame>
## Step 5: Group your runs into agents [#step-5-group-your-runs-into-agents]
Everything so far works for plain LLM traces, and that alone is useful. But where this really pays off is with full **agent loops**.
Any run you wrap with a stable agent identity (an `agentId`, an `agentName`, and a few optional fields) gets promoted out of the raw trace list and rolled up under the **Agents** tab. Instead of scattered individual calls, you get one workspace per agent: its metrics, its sessions, its traces, and its HALO analysis, all grouped together.
If you instrumented with the CLI or an AI coding agent, it likely set an `agentId` for you. It's worth a quick look to make sure that ID is the stable, readable value you actually want, because that's the key everything groups on. Depending on your framework you may need to tweak where the wrapper goes.
Here's what setting it looks like with the OpenAI Agents SDK. The same shape applies to any framework:
<CodeGroup>
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run } from "@openai/agents";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Help customers with order questions.",
model: "gpt-4o-mini",
});
await agentSpan(
{
agentId: "support-agent", // Stable ID everything groups on. Keep it constant across deploys and renames.
agentName: "Support Agent", // Human-readable label shown in the Agents dashboard.
spanName: "support-agent.run", // Name of the top-level span for this run.
sessionId: conversationId, // Ties multi-turn runs into one conversation. Think chatId or like a Slack threadId.
userId: "user_8675309", // Optional. Lets you filter every trace down to a single user.
role: "support", // Optional. Useful when one workflow has several agents (triage, refunds, billing).
system: "openai", // The framework or provider powering the run.
},
async (span) => {
const input = "Where is order ABC-123?";
span.setInput(input);
const result = await run(supportAgent, input);
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
```python Python
from agents import Agent, Runner
from inference_tracing import agent_span, setup
tracing = setup()
support_agent = Agent(
name="SupportAgent",
instructions="Help customers with order questions.",
model="gpt-4o-mini",
)
with agent_span(
tracing.tracer,
agent_id="support-agent", # Stable ID everything groups on. Keep it constant across deploys and renames.
agent_name="Support Agent", # Human-readable label shown in the Agents dashboard.
span_name="support-agent.run", # Name of the top-level span for this run.
session_id=conversation_id, # Ties multi-turn runs into one conversation. Think chatId or like a Slack threadId.
user_id="user_8675309", # Optional. Lets you filter every trace down to a single user.
role="support", # Optional. Useful when one workflow has several agents (triage, refunds, billing).
system="openai", # The framework or provider powering the run.
) as span:
user_message = "Where is order ABC-123?"
span.set_input(user_message)
result = await Runner.run(support_agent, input=user_message)
span.set_output(str(result.final_output or ""))
tracing.shutdown()
```
</CodeGroup>
<Note>
Agents group on `agentId`. Set it once on the top-level span and every run for that agent rolls up together. If your runs aren't grouping the way you expect, the ID is almost always the thing to check. See [Agent identity](/integrations/traces/agent-identity).
</Note>
## Step 6: Explore your agent [#step-6-explore-your-agent]
Open the **Agents** tab and you get a list of every agent you've traced, grouped by `agentId`. Each one is a card with its name, some high-level info, and a small graph of its usage, so you can see all your agents and how busy they've been at a glance.
<Frame>
<img src="/images/guides/e2e-agents-list.png" alt="The Agents tab showing the list of traced agents grouped by agentId, where each card has the agent name, high-level info, and a usage graph" />
</Frame>
Click into an agent and you land in its workspace. This is where most teams spend their time. It has a handful of sub-tabs.
### Overview [#overview]
The overview is your agent's home page. It rolls up run counts, error rate, latency, token usage, and cost, all scoped to this one agent, and charts them over time so you can spot a spike in errors or watch cost creep up after a change. It's the first place to look when you want to know how the agent has been behaving lately.
<Frame>
<img src="/images/guides/e2e-agent-overview.png" alt="The Overview sub-tab for a single agent, showing the metric tiles up top and the time-series charts for run count, error rate, latency, token usage, and cost" />
</Frame>
<Tip>
Want to track more than the built-in metrics? **Signals** turn your traces into structured labels you define in plain language — "is this NSFW?", "did the user get frustrated?", "was the task completed?" An LLM judge labels matching spans automatically, so the result charts right here alongside your other metrics and becomes something you can filter and break down by. See [Measure Your Agent's Quality with Signals](/guides/measure-agent-quality) for the full walkthrough.
</Tip>
### Trajectories [#trajectories]
The Trajectories sub-tab is the same trace view from Steps 3 and 4, just pre-filtered to this agent. All the same deep search and filtering applies, scoped to the runs that belong here.
Switch to the **Sessions** view within Trajectories for a conversation-grouped look. A session is the whole back-and-forth rolled up into a single view. Think of it as a conversation: you set the `sessionId` to something like a chat ID or a Slack thread ID, and every trace tagged with that ID becomes part of the same session. The `sessionId` should be unique and persistent for a single instance of a conversation that has a clear start and end.
The Sessions view lists one row per session with high-level metrics, and you can drill into any session to see each trace and span inside it, with the exact same trace detail (tree, session, timeline) you saw back in Step 3.
<Frame>
<img src="/images/guides/e2e-agent-sessions.png" alt="The Sessions view within Trajectories showing one row per session with high-level metrics, with a session expanded into the traces it contains" />
</Frame>
### Signals [#signals]
Plain-language classifiers you define once and run continuously against the agent's traffic, with an LLM judge labeling each span, trace, or session. The labels chart here alongside your other metrics and become filters and alert conditions. See [Measure Your Agent's Quality with Signals](/guides/measure-agent-quality).
### Analyze Harness [#analyze-harness]
The Analyze Harness sub-tab is the HALO workspace, and it's where the next step happens.
### Compare Models [#compare-models]
AutoEvals sample the agent's recent traffic, replay it against candidate models, and recommend a better fit with supporting evidence. See [Compare Models with AutoEvals](/get-started/compare-models-auto-evals).
## Step 7: Run HALO [#step-7-run-halo]
Most observability tools stop at charts and leave the digging to you. [HALO](https://github.com/context-labs/halo) does the digging, with an architecture built specifically for this problem. HALO is an open-source agent-loop optimizer hosted right inside the Agents dashboard, and the key detail is that it's an RLM unlike a regular general purpose model.
An [RLM (Recursive Language Model)](https://github.com/alexzhang13/rlm) swaps the usual `llm.completion(prompt)` for an `rlm.completion(prompt)`. Instead of cramming every trace into one context window, it holds your traces as a variable in a code environment and lets the model programmatically examine, decompose, and recursively call sub-models over them. That distinction matters here. Traces are enormous, and a general-purpose model run over them either blows past its context window or overfits to the error in a single trace instead of generalizing to the systemic, harness-level problem behind it. Because an RLM's context is effectively unbounded, HALO can surface trends and recurring patterns across thousands of runs over time while still catching the granular one-off failure buried deep in a single span. As far as we know, no other trace-analysis tool is built this way.
From there the flow is straightforward: HALO decomposes your traces, identifies the systemic failure modes, and writes up concrete fixes with citations back to the exact traces each finding came from. This guide uses the hosted version, which runs the same engine against the traces you've already collected with no extra setup. You can also [self-host it](https://github.com/context-labs/halo) and point it at an exported trace file.
Open the **Analyze Harness** sub-tab. Your previous HALO runs and chats live in a list down the left. On the right is a prompt window you can type anything into. We give you a sensible default prompt that works reasonably well out of the box, but you'll get sharper results the more specific you are. You can also set the time range HALO should analyze, plus advanced options like span limit, max depth, and max turns. Connect the agent's GitHub repo from the agent header and HALO grounds its findings in your actual code — real functions, prompts, and tool definitions — instead of reasoning from the traces alone.
<Frame>
<img src="/images/guides/e2e-agent-analysis.png" alt="The Analyze Harness sub-tab, with the list of previous HALO runs and chats on the left and the prompt window on the right, showing the time-range picker and the advanced options for span limit, max depth, and max turns" />
</Frame>
A few tips for the prompt:
* The default is a good general starting point. Tighten it to your actual question for a sharper answer: "Why is the enrichment agent timing out?", "Which tool calls return empty results most often?", "Find redundant LLM calls in the planning loop."
* Tighter time windows give HALO more focused signal. A single problem agent over the last 24 hours beats a firehose of everything from the last month.
### Put HALO on a schedule [#put-halo-on-a-schedule]
For agents already in production, you don't want to remember to run HALO by hand. Open the schedule create sheet and set it on a repeating schedule (hourly, daily, weekly) so it reviews recent traces automatically and you read reports as they land.
You can set up as many schedules as you want, each with its own prompt, so different schedules analyze your traces from different perspectives. One could watch for cost regressions daily while another hunts for reliability issues weekly.
<Frame>
<img src="/images/guides/e2e-halo-schedule.png" alt="The HALO schedule create sheet, showing the schedule cadence, the prompt field, and the time range, with the ability to create multiple schedules each with its own prompt" />
</Frame>
## Step 8: Read the report and keep chatting [#step-8-read-the-report-and-keep-chatting]
HALO works through the traces in the window and produces a report. It's ranked by impact and it goes deep, so the report is long enough that it'll scroll well past the fold.
On our own GTM agent, this exact run surfaced things we'd have spent days finding by hand, and in some cases never found at all, because the spans looked healthy:
* Tools being called with **invalid inputs**.
* Queries that were **failing without bubbling up**, so the span still showed an `ok` status while the work underneath had actually broken.
* **Duplicate tool calls** that repeated the same work for no benefit.
* A ranked **list of recommended fixes** for each finding.
Every finding cites the specific trace IDs it came from, so you can click straight from a finding into the trace tree that produced it and confirm it matches what you're seeing before you act.
We shipped fixes for the top findings the same afternoon: the silent query failures now surface as real errors instead of hiding behind an `ok` status, and the redundant tool calls are gone. That's the whole point of the loop. It turns invisible problems into a short, ranked to-do list.
<Frame>
<img src="/images/guides/e2e-halo-report.png" alt="A completed HALO report showing ranked findings like invalid tool inputs, silent query failures on ok-status spans, and duplicate tool calls, along with a list of recommended fixes" />
</Frame>
Two things make the report genuinely actionable:
1. **Hand it to a coding agent.** Copy a finding and its recommended fix straight into your coding agent and let it make the edit. The fixes are usually specific enough to apply directly: tightening a prompt, adding a guardrail, removing a redundant call, adjusting a tool description.
2. **Keep chatting with HALO.** The report isn't the end of the conversation. Once it lands you can ask follow-ups in the same thread: "give me more concise ways to fix these," "what are the highest priorities in your opinion?", or "which tool calls are the most expensive, and what alternatives would get the same result?" If you're not getting the results you want, adjust your prompt and run it again.
## Step 9: Apply the fixes [#step-9-apply-the-fixes]
A finding is only useful once it's in your code, and this is the step where the whole loop pays off. There are three ways to get a fix from a HALO report into your codebase, from most automated to most manual. Pick whichever fits how you work.
### Let your coding agent apply them (the MCP) [#let-your-coding-agent-apply-them-the-mcp]
Connect the [Inference MCP server](/integrations/mcp-server) to your coding agent (Claude Code, Cursor, or any MCP client) and just ask it to pull the report and apply the fixes:
```text
Apply the the latest fixes from the HALO report for the Gator Flue Agent
```
<Frame>
<img src="/images/guides/e2e-halo-mcp-demo.gif" alt="A coding agent connected to the Inference MCP server pulling the latest HALO report for the gator flow agent and editing the prompts, tool definitions, and harness code directly in the repo" />
</Frame>
Just ask in natural language and your assistant resolves the agent name to its identity, finds the HALO reports, and edits your prompts, tool definitions, and harness logic directly, grounded in the same findings and trace citations you read in the dashboard.
That's the moment the loop closes itself. The same traces you've been sending all along get analyzed, turned into a ranked to-do list, and applied to your harness from a single line of natural language to make changes. See the [MCP server guide](/integrations/mcp-server) for setup and more example prompts.
### From the CLI [#from-the-cli]
Prefer the terminal? The [`inf analyze-harness`](/cli/analyze-harness) commands pull the same report from the command line. Read it with `inf analyze-harness conversation get <conversation-id>` and pipe it into whatever coding agent you use, or read it inline.
### Copy it yourself [#copy-it-yourself]
The manual path always works, and we mentioned it back in Step 8: copy the recommendation (and any suggested change) straight from the report into your coding agent, or make the edit by hand.
1. Copy the recommendation from the report, or paste it into your coding agent.
2. Make the edit in your prompt, tool definition, or harness logic.
3. Save and move on to the next finding.
## Step 10: Close the loop [#step-10-close-the-loop]
A fix isn't done until you've confirmed it worked.
1. Ship the change and let your app run.
2. Capture a fresh window of traces.
3. Run HALO again over the new window (or wait for the next scheduled run).
4. Confirm the finding is gone, and pick up the next one.
That's the HALO loop. Each pass tightens your agent: fewer errors, lower latency, less wasted spend. The teams that get the most out of HALO run this continuously rather than once.
## Where to go next [#where-to-go-next]
That's the whole loop. If you haven't instrumented yet, [create a free account](https://inference.net/register) and run `inf instrument --mode tracing` in your project. Your first useful HALO report is about 20 minutes away.
<Tip>
Your traces are yours. We don't train on them, and we never use your data to improve our own models. What we do give you is the option to put that data to work for *you*: when you're ready, you can [train your own custom model](/get-started/train-and-deploy) on the runs you've already captured, entirely within our platform and under your control. A small model fine-tuned on your agent's real traffic is often faster and cheaper than a general-purpose one at the same task, and instrumenting today means that dataset is sitting ready whenever you want it. That's the second half of the loop: optimize the harness with HALO, and optimize the model with your own traces.
</Tip>
<CardGroup cols="2">
<Card title="Try it on a demo repo first" icon="flask" href="/guides/try-halo-demo-repo">
Run this whole loop on a ready-made, pre-instrumented agent before pointing it at your own.
</Card>
<Card title="Run inference through the gateway" icon="plug" href="/integrations/gateway/quickstart">
One API for every model, with usage, latency, and cost traced automatically.
</Card>
<Card title="Train a custom model" icon="graduation-cap" href="/get-started/train-and-deploy">
Turn the traces you've captured into a fine-tuned model that's faster and cheaper at your task.
</Card>
<Card title="HALO on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO engine, methodology, and benchmarks. MIT licensed.
</Card>
<Card title="Tracing integrations" icon="diagram-project" href="/integrations/traces/overview">
Every framework and provider Inference Tracing supports, with setup for each.
</Card>
</CardGroup>
# Guides (https://docs.inference.net/guides/overview)
Guides are end-to-end walkthroughs. Where the rest of the docs explain one feature at a time, a guide takes you from an empty project to a finished outcome, stitching tracing, metrics, Halo, and your coding agent into one continuous loop.
## Start here [#start-here]
<CardGroup cols="1">
<Card title="Optimize an agent end to end" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Install tracing, record real traces, read the metrics, run Halo to find what to fix, then apply the fixes by hand or by pulling them straight into your coding agent. The full Halo loop, start to finish.
</Card>
<Card title="Try HALO with a demo repo" icon="flask" href="/guides/try-halo-demo-repo">
Skip instrumentation. Clone a ready-to-run search agent, seed real traces, run Halo to find what's wrong, and let your coding agent apply the fix through the MCP server. The full loop in about fifteen minutes.
</Card>
<Card title="Run HALO on your machine with the desktop app" icon="desktop" href="/guides/halo-desktop-app">
The same loop, fully local. Stream traces from a demo agent into the free HALO desktop app, run Halo with your own model provider, and apply the fixes in your editor. No account, no cloud.
</Card>
<Card title="Measure your agent's quality with Signals" icon="signal-stream" href="/guides/measure-agent-quality">
Turn raw traces into metrics. Define plain-language classifiers — frustration, sentiment, task outcome — and let an LLM judge label every span automatically, then filter and chart your traces by them.
</Card>
<Card title="Find a better model with AutoEvals" icon="scale-balanced" href="/guides/find-a-better-model-with-auto-evals">
Send Inference platform your traffic through the Gateway or tracing, run a model comparison on your real requests, and read the recommendation down to the individual judged sample. No dataset, no rubric.
</Card>
</CardGroup>
## What you need [#what-you-need]
Every guide assumes you have:
* A free [Inference account](https://inference.net/register).
* An app or agent that makes LLM calls (any provider or framework).
* A terminal with Node.js or Python, depending on your stack.
More guides are on the way. If there's a workflow you want walked through end to end, [let us know](mailto:support@inference.net).
# Try HALO End to End with a Demo Repo (https://docs.inference.net/guides/try-halo-demo-repo)
<Note>
This guide uses the **hosted** version of HALO on the [Inference platform](/integrations/traces/quickstart) dashboard. Want to run the same loop entirely on your own machine, with traces that never leave your laptop and no account required? See [HALO Desktop: Optimize an Agent End to End on Your Machine](/guides/halo-desktop-app).
</Note>
The [Optimize an Agent End to End](/guides/optimize-an-agent-end-to-end) guide walks the full **trace, measure, analyze, fix** loop against *your* agent. This one hands you a working agent so you can feel the whole loop before you touch your own code.
[`context-labs/SearchAgentDemo`](https://github.com/context-labs/SearchAgentDemo) is a small search agent that comes already instrumented with Inference Tracing. You get traces into your project (upload a pre-run dataset or generate your own), run [HALO](https://github.com/context-labs/halo) (our open-source agent-loop optimizer) over them, and then connect the [MCP server](/integrations/mcp-server) so your coding agent pulls the HALO report and edits the repo for you. The repo even ships with a handful of deliberate, documented flaws, so HALO has something real to find and you get to watch it close the loop.
The only hard requirement is a free Inference account. We even publish a [pre-run trace dataset](#step-3-get-traces-into-your-project), so you can be looking at a HALO report in a couple of minutes without running the agent at all. And when you do want to generate your own, the agent has a mock-search mode, so you don't even need a Tavily key.
<Note>
**Pick your language.** The demo ships in two behaviorally identical flavors: [`context-labs/SearchAgentDemo`](https://github.com/context-labs/SearchAgentDemo) (Python) and [`context-labs/SearchAgentDemoTS`](https://github.com/context-labs/SearchAgentDemoTS) (TypeScript). Same tools, same deliberate flaws, same dataset, same traces, same HALO findings. Use the language tabs in each step; everything between the steps (the dashboard, HALO, and MCP) is identical.
</Note>
Prefer to watch? The video below walks through the same loop end to end. It covers the same ground as this guide in video form, so use whichever you like (or both).
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/-1qGGNkIDME?rel=0" title="Try HALO End to End with a Demo Repo" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Before you start [#before-you-start]
You need:
* A free [Inference account](https://inference.net/register).
* An API key from the [dashboard](https://inference.net/dashboard/api-keys).
* For the **Python** repo: [`uv`](https://docs.astral.sh/uv/) installed (the repo uses it for dependency management), plus `git` and Python 3.11+. For the **TypeScript** repo: [`bun`](https://bun.sh) installed, plus `git`.
* A coding agent with MCP support for the last few steps (Claude Code, Cursor, or any MCP client).
Optional:
* A [Tavily](https://tavily.com) API key for real web search. Without one, run the agent in `--mock-search` mode. Traces still flow, they're just more uniform.
## Step 1: Clone the demo repo [#step-1-clone-the-demo-repo]
<CodeGroup>
```bash Python
git clone https://github.com/context-labs/SearchAgentDemo.git && cd SearchAgentDemo
```
```bash TypeScript
git clone https://github.com/context-labs/SearchAgentDemoTS.git && cd SearchAgentDemoTS
```
</CodeGroup>
It's a single [OpenAI Agents SDK](/integrations/traces/openai-agents) agent with multi-turn tool calling, instrumented end to end with the [Inference Tracing SDK](/integrations/traces/quickstart). The layout is small on purpose:
<CodeGroup>
```text Python
src/search_agent_example/
agent.py Agent definition and instructions
tools.py Scratchpad, search, extract, source scoring, and claim comparison tools
search_clients.py Tavily and mock search clients
cli.py Single-query traced runner
batch.py Dataset traced runner
data/
queries.jsonl 50 starter queries
search-agent-demo-traces.jsonl.gz ~1,000 pre-run traces (gzipped)
docs/ HALO notes and known limitations
```
```text TypeScript
src/
agent.ts Agent definition and instructions
tools.ts Scratchpad, search, extract, source scoring, and claim comparison tools
searchClients.ts Tavily and mock search clients
cli.ts Single-query traced runner
batch.ts Dataset traced runner
data/
queries.jsonl 50 starter queries
search-agent-demo-traces.jsonl.gz ~1,000 pre-run traces (gzipped)
docs/ HALO notes and known limitations
```
</CodeGroup>
The agent runs a real loop: it plans in a scratchpad, searches the web, extracts pages, scores sources, compares claims, and writes a final answer. Every run is wrapped in an agent span with a stable identity (`agentId: traceable-search-agent`), and the Tavily calls inside the tools add manual `RETRIEVER` spans, so the traces have genuine structure for HALO to dig into.
<Tip>
This repo ships with a set of **deliberate, documented flaws**: a heuristic source-quality score, a loose unstructured scratchpad, a shallow lexical claim comparison, minimal URL deduplication, truncated page extraction. They're listed in `docs/known_limitations.md`. They're exactly the kind of systemic, harness-level issues HALO is built to surface, which is what makes this a good sandbox.
</Tip>
## Step 2: Install and configure [#step-2-install-and-configure]
Install dependencies:
<CodeGroup>
```bash Python
uv sync --extra dev
```
```bash TypeScript
bun install
```
</CodeGroup>
Create your `.env` from the example:
```bash
cp .env.example .env
```
Now open `.env` and paste in **one thing**, your [Inference API key](https://inference.net/dashboard/api-keys):
```bash
INFERENCE_API_KEY=sk-... # the only required value
MODEL_ID=gpt-4.1-mini # already set for you; any tool-capable Inference model works
TAVILY_API_KEY= # leave blank for now; only needed for real web search (Step 3)
```
That single key does double duty: it powers the agent's model calls **and** sends the traces to the Inference platform. The model defaults to `gpt-4.1-mini` (cheap, reliable tool-calling), so unless you want to change it, your only edit is pasting the key.
<Note>
One key, two jobs. `INFERENCE_API_KEY` authenticates the agent's model calls against Inference's OpenAI-compatible endpoint (`https://api.inference.net/v1`) and doubles as the trace-ingest token for `https://telemetry.inference.net`. Both are wired up for you, no other configuration needed. Prefer a different OpenAI-compatible provider? Set `INFERENCE_BASE_URL` and `INFERENCE_API_KEY` to theirs.
</Note>
## Step 3: Get traces into your project [#step-3-get-traces-into-your-project]
HALO needs a body of traces to analyze. There are two ways to get them, and they're not exclusive:
* **Option A, upload the pre-run dataset (fastest).** We already ran the agent across the full query set many times and published the resulting traces. Download once, upload into your project, and you're at HALO in two minutes with no model or search spend.
* **Option B, generate your own.** Run the repo locally to produce fresh traces. Slower and costs a little, but they're *your* runs, and you can keep generating after you change the code.
Most people should start with Option A to feel the loop, then switch to Option B once they want to iterate on the harness.
### Option A: Upload the pre-run dataset [#option-a-upload-the-pre-run-dataset]
The repo **ships the pre-run dataset with it**, no download needed. We ran the agent across the full query set many times and committed the resulting traces as a gzipped OTLP JSONL file at `data/search-agent-demo-traces.jsonl.gz` (\~20 MB). It's roughly **1,000 traces** with real search variance across the whole query set, far more, and more varied, than you'd want to generate by hand. Because they came from this repo, they already carry the `traceable-search-agent` identity and the same span shape, so they behave exactly like runs you'd produce yourself.
Decompress it first (expands to one \~200 MB JSONL file). The `-k` flag keeps the original `.gz` around:
```bash
gunzip -k data/search-agent-demo-traces.jsonl.gz
```
That leaves `data/search-agent-demo-traces.jsonl` ready to upload. Two ways:
**From the dashboard.** Open the **Trajectories** tab, click **imports**, and select the file. The dashboard validates it, processes it, and the traces show up in the table when it's done.
<Frame>
<img src="/images/guides/halo-demo-repo/halo-demo-imports.png" alt="The Trajectories view with the Upload / import dialog for bringing the pre-run trace dataset into the project" />
</Frame>
**From the CLI.** [`inf trace upload`](/cli/traces#inf-trace-upload) does the same thing and waits for processing to finish:
```bash
inf trace upload ./data/search-agent-demo-traces.jsonl --name search-agent-demo
```
It prints an **upload ID** when it's done. Use it to pull up just this set (note the all-time range, since these traces are timestamped when they were originally generated):
```bash
inf trace list --range all --filter "trace_import_id=<upload-id>"
```
<Note>
Uploading traces is a **dashboard or CLI** action. The [MCP server](/integrations/mcp-server) reads, exports, and analyzes traces (it doesn't upload them), so do the import here, then drive HALO over the uploaded traces from your coding agent in Steps 5–7.
</Note>
<Warning>
Because the dataset was generated earlier, its traces land at their **original timestamps**, not "now." When you run HALO in Step 5, set the time range wide (or "all time") so it actually covers them. The same goes for finding them in the dashboard: widen the range if the default last-hour view looks empty.
</Warning>
Once the upload finishes, skip ahead to [Step 4](#step-4-see-your-traces-in-the-dashboard). You have everything HALO needs.
### Option B: Generate your own traces [#option-b-generate-your-own-traces]
The repo is a **command-line tool**. There's no web UI and no server to start. You run it from the terminal, and every run does two things: it answers the query, and it ships a full [trace](/integrations/traces/overview) to the Inference platform. Those traces *are* the data HALO analyzes, so "generating data" just means running the agent a handful of times. You view and analyze it afterward in the [dashboard](https://inference.net/dashboard) (Step 4 on).
Two ways to run it:
* **`search-agent "<query>"`** runs one query, one trace. Good for a smoke test.
* **`search-agent-batch`** runs a slice of the bundled 50-query dataset, one trace per row. This is how you build up enough varied runs for a meaningful HALO report.
Start with a single query to confirm everything's connected. Use mock search for a free smoke test:
<CodeGroup>
```bash Python
uv run search-agent "What changed in the latest Python release?" --mock-search
```
```bash TypeScript
bun run search-agent "What changed in the latest Python release?" --mock-search
```
</CodeGroup>
Or run a real search if you set a Tavily key:
<CodeGroup>
```bash Python
uv run search-agent "What are the latest CISA recommendations for defending against ransomware?"
```
```bash TypeScript
bun run search-agent "What are the latest CISA recommendations for defending against ransomware?"
```
</CodeGroup>
The command prints the final answer and the trace `session_id`. That same `session_id` shows up in the dashboard so you can find the run.
Now seed a **batch**. HALO works best with repeated behavior across varied tasks, so the repo ships a 50-query starter dataset. Run a slice of it:
<CodeGroup>
```bash Python
uv run search-agent-batch --limit 5
```
```bash TypeScript
bun run search-agent-batch --limit 5
```
</CodeGroup>
Each row gets a stable session ID like `dataset-q001` and trace attributes for `demo.query_id`, `demo.category`, and `demo.dataset`, so the runs group and filter cleanly. Run a few different slices to build up variety:
<CodeGroup>
```bash Python
uv run search-agent-batch --start 20 --limit 5
```
```bash TypeScript
bun run search-agent-batch --start 20 --limit 5
```
</CodeGroup>
Each command prints its output to the terminal as it goes. There's nothing else to run; once a command finishes, the traces are already in the Inference platform.
<Accordion title="Command-line options">
Both commands take the same core flags (drop the `--` value examples in as needed):
| Flag | Applies to | Default | What it does |
| ------------------- | ---------- | ------------------------------- | ------------------------------------------------------------------------------------ |
| `--mock-search` | both | off | Use deterministic local results instead of Tavily (free, no Tavily key, no network). |
| `--limit <n>` | batch | `3` | How many dataset rows to run. |
| `--start <n>` | batch | `0` | Zero-based offset into the dataset, so you can run different slices. |
| `--session-id <id>` | single | random | Stable session ID for grouping a run in the dashboard. |
| `--user-id <id>` | both | `demo-user` / `demo-batch-user` | User ID recorded on the trace (batch defaults to `demo-batch-user`). |
| `--max-turns <n>` | both | `10` | Cap on the agent's tool-calling loop (2–20). |
So a free, no-key run of five dataset rows is `search-agent-batch --limit 5 --mock-search`, and a real-search slice starting at row 20 is `search-agent-batch --start 20 --limit 5`.
</Accordion>
<Tip>
Aim for **at least 20 real-search traces** before you read too much into a HALO report. Mock-search traces are great for verifying instrumentation, but they're too uniform for serious harness analysis. Start with `--limit 3` to control model and search spend, then widen.
</Tip>
## Step 4: See your traces in the dashboard [#step-4-see-your-traces-in-the-dashboard]
Open the **Agents** tab in the [dashboard](https://inference.net/dashboard). Whether you uploaded the pre-run dataset or generated your own, the traces roll up under a single agent, **Traceable Search Agent** (`traceable-search-agent`), because every span carries that stable `agentId`.
<Frame>
<img src="/images/guides/halo-demo-repo/halo-trace-list.png" alt="The Traceable Search Agent's traces in the dashboard, one row per run, grouped under the agent" />
</Frame>
Click into the agent and then into any run to open the trace detail. You get the same views the e2e guide covers in depth: the **trace tree** (every span, with inputs, outputs, cost, and tokens), the **timeline** (what's actually slow), and the **conversation** (session view, the whole run as a readable conversation). For this agent you'll see the agent loop, the model calls, the tool calls, and the manual `tavily.search` and `tavily.extract` retriever spans nested underneath.
<Frame>
<img src="/images/guides/halo-demo-repo/halo-individual-trace-detail.png" alt="A single run open on the trace detail view, showing the agent loop, tool calls, and nested retriever spans with their inputs and outputs" />
</Frame>
<Tip>
New to the trace views? The [Optimize an Agent End to End](/guides/optimize-an-agent-end-to-end#step-3-explore-a-trace) guide breaks down the tree, session, timeline, and deep search in detail. Everything there applies here, scoped to this one agent.
</Tip>
## Step 5: Run HALO [#step-5-run-halo]
Open the agent's **Analyze Harness** sub-tab. This is the [HALO](https://github.com/context-labs/halo) workspace. There's a prompt window on the right with a sensible default, plus a time-range picker and advanced options (span limit, max depth, max turns). Set the range to cover your traces and run it. If you uploaded the pre-run dataset, widen it to **all time**, since those traces keep their original timestamps.
A tighter, more specific prompt gives sharper results. Because this repo has known weak spots, try aiming HALO at them:
```text
Review the search agent's traces. Where is it wasting tool calls, over-trusting
weak sources, or losing evidence between search and final answer? Rank the issues
by impact and cite the traces.
```
<Frame>
<img src="/images/guides/halo-demo-repo/halo-demo-report-start.png" alt="The Analyze Harness sub-tab with a prompt entered and HALO just started, the previous runs listed on the left" />
</Frame>
HALO works through the traces and writes a ranked report with citations back to the exact runs each finding came from. Because the repo ships real flaws, you should expect findings in the neighborhood of:
* **Over-trusting the source-quality heuristic.** `assess_source` scores domains with simple rules and can overrate weak institutional pages.
* **A loose scratchpad.** Notes are plain text with no schema, so stale or vague notes leak into the final answer.
* **Shallow claim comparison.** `compare_claims` only checks lexical overlap, so it misses real contradictions.
* **Minimal deduplication.** Repeated near-duplicate sources waste tool calls and bias synthesis.
* **Truncated extraction.** `extract_page` caps content, so key evidence can fall outside the slice.
<Frame>
<img src="/images/guides/halo-demo-repo/halo-demo-report-finished.png" alt="A completed HALO report with ranked findings, recommended fixes, and citations back to the exact traces each finding came from" />
</Frame>
Click any finding's citation to drop straight into the trace that produced it and confirm it matches before you act. And keep chatting: once the report lands you can ask follow-ups in the same thread: "which of these is highest impact?", "show me the cheapest fix for the wasted tool calls."
### Optional: Connect your GitHub repo [#optional-connect-your-github-repo]
You can connect your GitHub repo so HALO sees your actual code alongside your traces. With the source in hand, HALO grounds its findings in the real functions and prompts behind each trace and points to concrete fixes with real code examples, instead of reasoning from the traces alone. Once connected, HALO has access to the repo on every run and every follow-up question in the thread.
For this demo it isn't really necessary; the deliberate flaws show up clearly from the traces. But for a real repo full of agent code it makes a real difference, and we recommend it for sharper results.
<Frame>
<img src="/images/guides/halo-demo-repo/halo-demo-connect-github.png" alt="Connecting a GitHub repo to HALO so it can read the agent's source code alongside the traces" />
</Frame>
## Step 6: Connect the MCP server [#step-6-connect-the-mcp-server]
This is the part worth doing slowly, because it's where the loop closes itself. Connect the [Inference MCP server](/integrations/mcp-server) to your coding agent so it can read your HALO reports and edit the cloned repo directly.
The fastest setup, for Claude Code:
```bash
claude mcp add --transport http inference https://mcp.inference.net/mcp \
--header "Authorization: Bearer $INFERENCE_API_KEY"
```
For Cursor or any other MCP client, see the [MCP server guide](/integrations/mcp-server) for the exact config. Use the same API key you put in your `.env`.
Open your coding agent **inside the repo you cloned** so it can both read the HALO report (through the MCP) and edit the agent's code (on disk).
## Step 7: Apply the fix from your editor [#step-7-apply-the-fix-from-your-editor]
Now just ask, in plain language:
```text
Pull the latest HALO report for the Traceable Search Agent and apply the top fixes
to this repo.
```
Your coding agent resolves the agent name to its identity, fetches the HALO report and its trace citations through the MCP, and edits the relevant files (the prompts in `agent.py` / `agent.ts`, the tool definitions in `tools.py` / `tools.ts`, the harness logic), grounded in the same findings you just read in the dashboard.
That's the moment the loop closes itself: the traces you seeded got analyzed, turned into a ranked to-do list, and applied to the harness from a single line of natural language.
<Tip>
Prefer the terminal? The [`inf analyze-harness`](/cli/analyze-harness) commands pull the same report from the command line. Read it with `inf analyze-harness conversation get <conversation-id>` and pipe it into whatever coding agent you use. And the manual path always works too: copy a recommendation straight out of the report into your editor.
</Tip>
## Step 8: Close the loop [#step-8-close-the-loop]
A fix isn't done until you've confirmed it worked, and confirming means new traces from the changed code, so this is the point where everyone generates their own runs, including if you took the upload shortcut in Step 3. The demo repo makes the comparison clean because the query set is fixed: run the same slice before and after and the only variable is your fix. (If you haven't set up the repo yet, do [Steps 1–2](#step-1-clone-the-demo-repo) first.)
1. Run a slice of the bundled queries against your patched code. `--limit 5` runs the first 5 of the 50 queries in `data/queries.jsonl`; keep it the same on both sides of the fix so the comparison is apples to apples:
<CodeGroup>
```bash Python
uv run search-agent-batch --limit 5
```
```bash TypeScript
bun run search-agent-batch --limit 5
```
</CodeGroup>
2. Run HALO again over the new window (or set it on a [schedule](/guides/optimize-an-agent-end-to-end#put-halo-on-a-schedule)).
3. Compare the two reports. The findings you fixed should be gone or diminished; pick up the next one.
That's the whole HALO loop, start to finish, on a repo you didn't have to write. Once it clicks here, point the same loop at your own agent.
We hope running it on a sandbox gave you a feel for what HALO can do on your own projects: turn a pile of traces into a ranked, cited list of real problems, and then into fixes, without you reading spans by hand. HALO is fully open source ([`context-labs/halo`](https://github.com/context-labs/halo), MIT licensed). Read the engine and methodology, or self-host it and point it at your own exported traces.
## Where to go next [#where-to-go-next]
<CardGroup cols="2">
<Card title="Do this on your own agent" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Run the same trace → HALO → fix loop against your real app. Start with `inf instrument --mode tracing`.
</Card>
<Card title="Measure agent quality with Signals" icon="signal-stream" href="/guides/measure-agent-quality">
Turn the traces you seeded into quality metrics you can chart, filter, and get alerted on.
</Card>
<Card title="Connect the MCP server" icon="plug" href="/integrations/mcp-server">
Full setup and more example prompts for driving HALO from your coding agent.
</Card>
<Card title="HALO on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO engine, methodology, and benchmarks. MIT licensed.
</Card>
</CardGroup>
# ElevenLabs (https://docs.inference.net/integrations/agent-platforms/elevenlabs)
ElevenLabs Agents lets you point an agent at a Custom LLM that speaks the OpenAI Chat Completions API. The Inference gateway accepts that exact format and can forward to any supported provider, so you can run an ElevenLabs agent on Anthropic Claude (or any other model in the catalog) while keeping cost tracking, latency monitoring, and trace capture in one place.
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.loom.com/embed/14fd2b514de846aba4723350cc7bb345" title="ElevenLabs + Inference walkthrough" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
This guide configures an ElevenLabs Agent to call **Claude Sonnet 4.6** through the gateway.
<Info>
ElevenLabs always sends OpenAI Chat Completions to the Custom LLM URL. To reach Claude, we route through Anthropic's
[OpenAI-compatible endpoint](https://platform.claude.com/docs/en/api/openai-sdk)
using the
`x-inference-provider-url`
header — no schema translation needed on either side.
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Anthropic API key** — from your [Anthropic console](https://console.anthropic.com/settings/keys)
</Step>
<Step title="Open the agent's LLM settings">
In the ElevenLabs dashboard, open your agent and click into the **LLM** panel. Set the LLM to **Custom LLM**.
</Step>
<Step title="Configure the Server URL">
Under **Server URL**, leave the endpoint dropdown on **Chat Completions** and set the base URL to:
```
https://api.inference.net/v1
```
ElevenLabs appends `/chat/completions` automatically, so the final request goes to `https://api.inference.net/v1/chat/completions`.
</Step>
<Step title="Set the Model ID">
In **Model ID**, enter the upstream model you want the gateway to route to:
```
claude-sonnet-4-6
```
You can swap this for any other supported model (e.g. `gpt-4.1`, `claude-opus-4-6`) without changing anything else.
</Step>
<Step title="Add your gateway API key">
Under **API Key**, choose **Secret** → **Add new secret** and paste your **Inference project API key**. ElevenLabs sends this as `Authorization: Bearer <key>`, which is how the gateway authenticates the request.
<Warning>
Do not paste your Anthropic key here. The Anthropic key is forwarded separately as a request header in the next step.
</Warning>
</Step>
<Step title="Add the routing headers">
Under **Request headers**, click **Add header** and add the following entries. `x-inference-provider-url` tells the gateway to forward the chat-completions request to Anthropic's OpenAI-compatible endpoint, and `x-inference-provider-api-key` carries the Anthropic key it should use upstream.
| Header | Value |
| ------------------------------ | --------------------------- |
| `x-inference-provider-url` | `https://api.anthropic.com` |
| `x-inference-provider-api-key` | *your Anthropic API key* |
| `x-inference-environment` | `production` |
For the Anthropic key, click the `{{ }}` icon and reference an environment variable (e.g. `{{ ANTHROPIC_API_KEY }}`) rather than pasting the secret inline.
<Note>
The provider URL must be the
**bare host**
(
`https://api.anthropic.com`
) — not
`https://api.anthropic.com/v1`
. The gateway appends
`/v1/chat/completions`
itself; including
`/v1`
yields a 404.
</Note>
<Warning>
Do
**not**
set
`x-inference-provider: anthropic`
here. That value targets Anthropic's native
`/v1/messages`
shape, but ElevenLabs sends chat completions — the override URL above is what makes this work.
</Warning>
</Step>
<Step title="Tag requests with a Task ID">
Add one more header so this agent's calls are grouped under a dedicated task in your Inference.net dashboard. Tasks let you split out cost, latency, and traces per use case (e.g. one agent per task, or one task per tool flow).
| Header | Value |
| --------------------- | ------------- |
| `x-inference-task-id` | `voice-agent` |
Pick any stable string — it just needs to stay consistent across requests you want grouped together. If the task ID doesn't already exist in your project, the gateway creates it on first use. To split traffic further (e.g. staging vs. production agents), use distinct IDs like `voice-agent-prod` and `voice-agent-staging` and filter by them in the dashboard.
</Step>
<Step title="Capture ElevenLabs system variables as metadata">
ElevenLabs exposes per-conversation system variables (agent ID, conversation ID, caller ID, etc.) that can be injected into request headers. Inference Gateway treats any header prefixed with `x-inference-metadata-` as searchable metadata: the prefix is stripped and the rest becomes a key you can filter on in the dashboard. Combine the two and every LLM call is tagged with the ElevenLabs context it came from.
For each variable you want to capture, click **Add header**, switch the **Type** tab to **Variable**, and configure:
| Header name | Dynamic variable |
| ----------------------------------------- | ---------------------------- |
| `x-inference-metadata-agent-id` | `system__agent_id` |
| `x-inference-metadata-conversation-id` | `system__conversation_id` |
| `x-inference-metadata-caller-id` | `system__caller_id` |
| `x-inference-metadata-called-number` | `system__called_number` |
| `x-inference-metadata-call-sid` | `system__call_sid` |
| `x-inference-metadata-call-duration-secs` | `system__call_duration_secs` |
ElevenLabs substitutes the variable value into the header on every request, and the gateway stores the stripped key (`agent-id`, `conversation-id`, …) against the inference. In the Inference.net dashboard you can then filter by any of these — e.g. pull every LLM call from a single conversation by `conversation-id`, or cost-per-caller by `caller-id` — and use the same keys to slice datasets you build off the traffic.
<Tip>
The metadata key is just
`<header name minus the prefix>`
. Keep it short and lowercase-with-dashes — that's how it'll show up in the dashboard's filter UI. Add only the variables you actually plan to filter or group by; extra metadata is free to send but adds noise.
</Tip>
</Step>
<Step title="Test the connection">
Click **Test Connection** at the bottom of the LLM panel. ElevenLabs will issue a probe request through the gateway to Anthropic. A green status confirms the agent can reach Claude Sonnet 4.6 via the Inference platform. Save the agent and start a session — the call will appear in your [dashboard](https://inference.net/dashboard) within a few seconds.
</Step>
</Steps>
## Equivalent cURL [#equivalent-curl]
If you want to verify the gateway path outside ElevenLabs, this is the same request the agent issues:
```bash
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-url: https://api.anthropic.com" \
-H "x-inference-provider-api-key: $ANTHROPIC_API_KEY" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: voice-agent" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4-6",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
# LangChain (https://docs.inference.net/integrations/frameworks/langchain)
Use this guide when your application talks to models through **LangChain wrappers** instead of the direct provider SDKs.
Inference.net uses the same gateway pattern here as everywhere else: requests go through the Inference.net gateway, the gateway authenticates with `INFERENCE_API_KEY`, and the gateway forwards requests to the downstream provider using the provider key you supply in `x-inference-provider-api-key`.
The important detail is that LangChain's wrapper surfaces are **not identical** to the direct OpenAI and Anthropic SDK constructors. This page documents the wrapper-specific patterns that we validated for Python and TypeScript.
## Supported wrappers in this guide [#supported-wrappers-in-this-guide]
| Language | Supported now | Notes |
| ---------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Python | `ChatOpenAI`, `ChatAnthropic`, `init_chat_model(..., provider="openai" / "anthropic")` | Validated against the local gateway and observability stack |
| TypeScript | `ChatOpenAI`, `ChatAnthropic` | Validated against the local gateway and observability stack |
## Deferred and unsupported wrappers [#deferred-and-unsupported-wrappers]
These cases are **not** covered in this pass:
* `ChatGoogleGenerativeAI`
* native Google SDKs such as `genai.Client()`
* `ChatNVIDIA`
* config-only model strings in TOML or YAML when there is no concrete wrapper instantiation site to rewrite
## The LangChain-specific differences [#the-langchain-specific-differences]
Compared with the direct SDK patterns, LangChain changes a few important things:
1. **Anthropic wrapper auth is pragmatic, not pure.**
The LangChain Anthropic wrappers still expect an Anthropic API key field, so the examples below keep `anthropic_api_key` / `apiKey` set to the real Anthropic key for wrapper compatibility. Gateway auth still goes through `Authorization: Bearer <INFERENCE_API_KEY>`, and downstream provider auth still goes through `x-inference-provider-api-key`.
2. **OpenAI-compatible routing still uses the OpenAI wrappers.**
For custom OpenAI-compatible providers, keep the LangChain OpenAI wrapper pointed at the Inference.net gateway and move the original provider URL into `x-inference-provider-url`.
3. **Task IDs are still supported.**
LangChain can pass request-level task IDs through the validated wrappers, but the exact call shape differs by wrapper. The precise task-ID call patterns are documented below.
## Python [#python]
### Python `ChatOpenAI` [#python-chatopenai]
Use `ChatOpenAI` for direct OpenAI routing by pointing the wrapper at the gateway, authenticating the gateway with `INFERENCE_API_KEY`, and forwarding the downstream OpenAI key in `x-inference-provider-api-key`.
<Metadata text="integrations/langchain/python_chatopenai" />
```python
import os
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
model="gpt-4o-mini",
temperature=0,
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider": "openai",
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-environment": "production",
},
)
```
### Python `ChatAnthropic` [#python-chatanthropic]
Use the pragmatic LangChain pattern for Anthropic wrappers:
* keep `anthropic_api_key` set to the real Anthropic key
* point `anthropic_api_url` at the Inference.net gateway
* send gateway auth through `Authorization`
* send downstream provider auth through `x-inference-provider-api-key`
<Metadata text="integrations/langchain/python_chatanthropic" />
```python
import os
from langchain_anthropic import ChatAnthropic
model = ChatAnthropic(
model="claude-sonnet-4-5-20250929",
temperature=0,
anthropic_api_url="https://api.inference.net",
anthropic_api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": os.environ["ANTHROPIC_API_KEY"],
"x-inference-environment": "production",
},
)
```
### Python `init_chat_model` [#python-init_chat_model]
For the validated OpenAI and Anthropic paths, `init_chat_model` passes the provider-specific kwargs through to the underlying LangChain wrapper.
#### OpenAI via `init_chat_model` [#openai-via-init_chat_model]
<Metadata text="integrations/langchain/python_init_chat_model_openai" />
```python
import os
from langchain.chat_models import init_chat_model
model = init_chat_model(
model="openai:gpt-4o-mini",
temperature=0,
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider": "openai",
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-environment": "production",
},
)
```
#### Anthropic via `init_chat_model` [#anthropic-via-init_chat_model]
<Metadata text="integrations/langchain/python_init_chat_model_anthropic" />
```python
import os
from langchain.chat_models import init_chat_model
model = init_chat_model(
model="anthropic:claude-sonnet-4-5-20250929",
temperature=0,
anthropic_api_url="https://api.inference.net",
anthropic_api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": os.environ["ANTHROPIC_API_KEY"],
"x-inference-environment": "production",
},
)
```
## TypeScript [#typescript]
### TypeScript `ChatOpenAI` [#typescript-chatopenai]
For LangChain JS/TS, `ChatOpenAI` uses `configuration.baseURL` and `configuration.defaultHeaders` for gateway routing.
<Metadata text="integrations/langchain/ts_chatopenai" />
```typescript
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gpt-4o-mini",
temperature: 0,
apiKey: process.env.INFERENCE_API_KEY!,
configuration: {
baseURL: "https://api.inference.net/v1",
defaultHeaders: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-environment": "production",
},
},
});
```
### TypeScript `ChatAnthropic` [#typescript-chatanthropic]
For LangChain JS/TS, `ChatAnthropic` uses `anthropicApiUrl` and `clientOptions.defaultHeaders`.
Use the pragmatic wrapper-compatible pattern:
* keep `apiKey` set to the real Anthropic key
* point `anthropicApiUrl` at the Inference.net gateway
* send gateway auth through `Authorization`
* send downstream provider auth through `x-inference-provider-api-key`
<Metadata text="integrations/langchain/ts_chatanthropic" />
```typescript
import { ChatAnthropic } from "@langchain/anthropic";
const model = new ChatAnthropic({
model: "claude-sonnet-4-5-20250929",
temperature: 0,
apiKey: process.env.ANTHROPIC_API_KEY!,
anthropicApiUrl: "https://api.inference.net",
clientOptions: {
defaultHeaders: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY!}`,
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": process.env.ANTHROPIC_API_KEY!,
"x-inference-environment": "production",
},
},
});
```
## Custom OpenAI-compatible providers [#custom-openai-compatible-providers]
When you use a LangChain OpenAI wrapper against Gemini, Together AI, Groq, Fireworks, Mistral, OpenRouter, or another OpenAI-compatible provider, keep the wrapper pointed at the Inference.net gateway and move the original provider URL into `x-inference-provider-url`.
### Python custom provider via `ChatOpenAI` [#python-custom-provider-via-chatopenai]
<Metadata text="integrations/langchain/python_chatopenai_custom_provider" />
```python
import os
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
model="gemini-2.5-flash",
temperature=0,
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-url": "https://generativelanguage.googleapis.com/v1beta/openai",
"x-inference-provider-api-key": os.environ["GEMINI_API_KEY"],
"x-inference-environment": "production",
},
)
```
### TypeScript custom provider via `ChatOpenAI` [#typescript-custom-provider-via-chatopenai]
<Metadata text="integrations/langchain/ts_chatopenai_custom_provider" />
```typescript
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gemini-2.5-flash",
temperature: 0,
apiKey: process.env.INFERENCE_API_KEY!,
configuration: {
baseURL: "https://api.inference.net/v1",
defaultHeaders: {
"x-inference-provider-url": "https://generativelanguage.googleapis.com/v1beta/openai",
"x-inference-provider-api-key": process.env.GEMINI_API_KEY!,
"x-inference-environment": "production",
},
},
});
```
When `x-inference-provider-url` is present, you usually do **not** need `x-inference-provider`. The gateway can infer the provider protocol from the overridden URL.
## Task IDs [#task-ids]
The wrappers in this guide support request-level task IDs on a **shared client**. You do **not** need to create separate clients per task for the validated wrappers below.
The important detail is that the call shape differs by wrapper.
### Python task IDs [#python-task-ids]
For the validated Python wrappers in this guide, pass task IDs through `extra_headers`.
#### Python `ChatOpenAI` and `ChatAnthropic` [#python-chatopenai-and-chatanthropic]
<Metadata text="integrations/langchain/python_task_ids" />
```python
def invoke_with_task(model, question: str, task_id: str):
return model.invoke(
question,
extra_headers={"x-inference-task-id": task_id},
)
```
This same `extra_headers` pattern also works for the validated `init_chat_model(..., provider="openai")` and `init_chat_model(..., provider="anthropic")` paths.
### TypeScript task IDs [#typescript-task-ids]
For TypeScript, the exact call shape depends on the wrapper.
#### TypeScript `ChatAnthropic` [#typescript-chatanthropic-1]
Use `headers` directly in the second argument to `invoke()`.
<Metadata text="integrations/langchain/ts_task_ids_anthropic" />
```typescript
import { ChatAnthropic } from "@langchain/anthropic";
async function invokeAnthropicWithTask(
model: ChatAnthropic,
question: string,
taskId: string,
) {
return model.invoke(question, {
headers: { "x-inference-task-id": taskId },
});
}
```
#### TypeScript `ChatOpenAI` [#typescript-chatopenai-1]
Use `options.headers` in the second argument to `invoke()`.
<Metadata text="integrations/langchain/ts_task_ids_openai" />
```typescript
import { ChatOpenAI } from "@langchain/openai";
async function invokeOpenAIWithTask(
model: ChatOpenAI,
question: string,
taskId: string,
) {
return model.invoke(question, {
options: {
headers: { "x-inference-task-id": taskId },
},
});
}
```
### Why this matters [#why-this-matters]
LangChain does not normalize transport-level request options across wrappers. That is why the TypeScript OpenAI and Anthropic wrappers use different task-ID call shapes, even though both ultimately set the same `x-inference-task-id` header on the proxied request.
### Framework-owned invocation loops [#framework-owned-invocation-loops]
Some frameworks accept a LangChain model object and then own the actual `.invoke()` / `.ainvoke()` calls internally. In those cases you may not have a wrapper call site where you can attach per-request task IDs yourself.
Use a client-level fallback instead:
#### Python fallback [#python-fallback]
<Metadata text="integrations/langchain/python_task_ids_framework_fallback" />
```python
import os
from langchain_anthropic import ChatAnthropic
model = ChatAnthropic(
model="claude-sonnet-4-5-20250929",
temperature=0,
anthropic_api_url="https://api.inference.net",
anthropic_api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": os.environ["ANTHROPIC_API_KEY"],
"x-inference-task-id": "research-agent",
},
)
```
#### TypeScript fallback [#typescript-fallback]
<Metadata text="integrations/langchain/ts_task_ids_framework_fallback" />
```typescript
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gpt-4o-mini",
temperature: 0,
apiKey: process.env.INFERENCE_API_KEY!,
configuration: {
baseURL: "https://api.inference.net/v1",
defaultHeaders: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-task-id": "research-agent",
},
},
});
```
This is less granular than a per-request task ID, but it still gives you task grouping when the framework owns the invocation loop.
# Vercel AI SDK (https://docs.inference.net/integrations/frameworks/vercel-ai-sdk)
Use this guide when your application talks to models through the **Vercel AI SDK** (the `ai` package) with `generateText`, `streamText`, `generateObject`, `streamObject`, or `ToolLoopAgent`, instead of the direct provider SDKs.
Inference.net uses the same gateway pattern here as everywhere else: requests go through the Inference.net gateway, the gateway authenticates with `INFERENCE_API_KEY`, and the gateway forwards requests to the downstream provider using the provider key you supply in `x-inference-provider-api-key`.
<Note>
Use the OpenAI-compatible provider (`@ai-sdk/openai-compatible`), **not** the AI SDK Gateway provider (`@ai-sdk/gateway` / `createGateway()`). `createGateway()` speaks Vercel's proprietary AI Gateway protocol and cannot be pointed at an OpenAI-compatible base URL, so it cannot route through Inference.net. `createOpenAICompatible({ baseURL })` sends standard OpenAI-compatible requests that the gateway proxies as-is.
</Note>
## Install [#install]
<Metadata text="integrations/vercel-ai-sdk/install" />
```bash
npm install ai @ai-sdk/openai-compatible
```
## Set environment variables [#set-environment-variables]
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Provider API key** (in this example, OpenAI) — from your [OpenAI account](https://platform.openai.com/api-keys)
<Metadata text="integrations/vercel-ai-sdk/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
## Configure the provider [#configure-the-provider]
Point `createOpenAICompatible` at the gateway. Your project API key goes in `apiKey` to authenticate the gateway, and your provider key goes in `x-inference-provider-api-key` so the gateway can forward it downstream.
<Metadata text="integrations/vercel-ai-sdk/provider" />
```typescript
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const inference = createOpenAICompatible({
name: "inference",
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
headers: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-environment": "production",
},
});
const model = inference("gpt-4.1");
```
`includeUsage: true` is useful because usage metadata is what populates token columns in the dashboard. Some providers only return token counts for non-streaming calls or after a stream finishes.
## Generate text [#generate-text]
<Metadata text="integrations/vercel-ai-sdk/generate-text" />
```typescript
import { generateText } from "ai";
const { text } = await generateText({
model,
prompt: "Hello, world!",
});
console.log(text);
```
## Stream text [#stream-text]
<Metadata text="integrations/vercel-ai-sdk/stream-text" />
```typescript
import { streamText } from "ai";
const result = streamText({
model,
prompt: "Stream a short sentence about observability.",
});
for await (const chunk of result.textStream) {
process.stdout.write(chunk);
}
```
## Custom OpenAI-compatible providers [#custom-openai-compatible-providers]
To route to Gemini, Together AI, Groq, Fireworks, Mistral, OpenRouter, or another OpenAI-compatible provider, keep the provider pointed at the Inference.net gateway and move the original provider URL into `x-inference-provider-url`.
<Metadata text="integrations/vercel-ai-sdk/custom-provider" />
```typescript
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const gemini = createOpenAICompatible({
name: "inference-gemini",
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
headers: {
"x-inference-provider-url": "https://generativelanguage.googleapis.com/v1beta/openai",
"x-inference-provider-api-key": process.env.GEMINI_API_KEY!,
"x-inference-environment": "production",
},
});
const model = gemini("gemini-2.5-flash");
```
When `x-inference-provider-url` is present, you usually do **not** need `x-inference-provider`. The gateway can infer the provider protocol from the overridden URL.
## Task IDs and per-request metadata [#task-ids-and-per-request-metadata]
You can attach routing metadata at two levels:
1. **Provider-level headers** apply to every call made through that provider client. This is the simplest option when a framework owns the invocation loop and you have no per-call site.
2. **Per-request headers** use the `headers` option on `generateText`, `streamText`, `generateObject`, and `streamObject`. These are forwarded to the gateway on that request, so you can add or override routing headers per call while sharing one provider client.
### Provider-level task ID [#provider-level-task-id]
Add `x-inference-task-id` to the provider `headers` so every request is grouped under the same task.
<Metadata text="integrations/vercel-ai-sdk/task-id-provider" />
```typescript
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const inference = createOpenAICompatible({
name: "inference",
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
headers: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-task-id": "research-agent",
},
});
```
### Per-request task ID [#per-request-task-id]
<Metadata text="integrations/vercel-ai-sdk/task-id-request" />
```typescript
import { generateText } from "ai";
const { text } = await generateText({
model,
prompt: "Summarize the incident report.",
headers: { "x-inference-task-id": "summarize-incident" },
});
```
The same `headers` option accepts any routing header, including `x-inference-environment` and `x-inference-metadata-*`, so you can tag environment and arbitrary metadata per call.
## Add tracing [#add-tracing]
<Note>
Gateway captures one record per LLM request through the proxy. To also capture the full agent hierarchy (`generateText` / `streamText` / tool spans) from inside your code, add [Vercel AI SDK tracing](/integrations/traces/ai-sdk). Gateway and tracing are independent and can be used together.
</Note>
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Gateway overview" icon="satellite-dish" href="/integrations/gateway/overview">
Routing headers, supported providers, and the full set of OpenAI-compatible base URLs.
</Card>
<Card title="Vercel AI SDK tracing" icon="chart-network" href="/integrations/traces/ai-sdk">
Capture native AI SDK spans for the full agent and tool-call hierarchy.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
</CardGroup>
# Gateway (https://docs.inference.net/integrations/gateway/overview)
Inference Gateway is a transparent proxy that sits between your app and your LLM provider. Point your SDK at `https://api.inference.net/v1`, add a couple of headers, and every request is captured with cost, latency, and full request/response payloads. You keep your existing provider API keys. Gateway adds roughly 10ms of overhead and forwards requests as-is.
Use Gateway for inference observability and training. Once traffic is flowing, you can:
* Watch cost, latency, error rates, and token usage in [Metrics Explorer](/platform/gateway/metrics-explorer)
* Browse individual requests in the [Inference Viewer](/platform/gateway/inference-viewer)
* Build [datasets from traffic](/platform/datasets/build-from-traffic) and compare models with [AutoEvals](/platform/eval/auto-evals/run-an-auto-eval)
* [Fine-tune a task-specific model](/platform/train/overview) on your captured data and [deploy](/platform/deploy/overview) it on a dedicated GPU
<Info>
Gateway and Inference Tracing are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For traces, see the [Tracing overview](/integrations/traces/overview).
</Info>
## Getting Started [#getting-started]
The fastest path is the [Inference CLI](/cli/overview):
<Metadata text="integrations/gateway/overview-inf-instrument" />
```bash
inf instrument --mode gateway
```
`gateway` mode points your existing LLM clients at the Inference Gateway. `both` mode does that plus installs the tracing SDK in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See [Install with AI](/integrations/install-with-ai) for the full flow.
Prefer to wire it up by hand? Start with the [Gateway quickstart](/integrations/gateway/quickstart) or pick your provider below.
## Officially Supported Providers [#officially-supported-providers]
These providers have dedicated guides with copy-paste setup. Any other OpenAI-compatible provider works too via the `x-inference-provider-url` header. See [supported OpenAI-compatible providers](#supported-openai-compatible-provider-urls) below.
* [OpenAI](/integrations/model-providers/openai): Chat Completions and Responses API, including tool calls and structured outputs.
* [Anthropic](/integrations/model-providers/anthropic): Messages API with tool use, prompt caching, and streaming.
* [Vertex AI](/integrations/model-providers/vertex-ai): Google Cloud Vertex AI for Gemini and Anthropic models on GCP.
* [Amazon Bedrock](/integrations/model-providers/amazon-bedrock): Bedrock OpenAI-compatible Chat Completions through AWS-hosted models.
* [Google Gemini](/integrations/model-providers/google-gemini): Native Gemini API and the OpenAI-compatible Gemini endpoint.
* [OpenRouter](/integrations/model-providers/openrouter): Route across many models through OpenRouter's OpenAI-compatible API.
* [Cerebras](/integrations/model-providers/cerebras): Cerebras inference with full request capture.
* [Groq](/integrations/model-providers/groq): Groq's low-latency inference endpoint.
* [LangChain](/integrations/frameworks/langchain): Use Gateway from LangChain by setting the base URL on the chat model.
* [Vercel AI SDK](/integrations/frameworks/vercel-ai-sdk): Route the `ai` package through Gateway with the OpenAI-compatible provider.
* [ElevenLabs](/integrations/agent-platforms/elevenlabs): Route ElevenLabs Agents' LLM calls through the Inference platform.
## Routing Headers [#routing-headers]
| Header | Required | Description |
| ------------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>` authenticates the request to the gateway and links it to your project. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `x-inference-provider-api-key` | Yes | Your provider API key or token, such as OpenAI, Bedrock, Groq, Gemini, or a Google Cloud Vertex credential. The gateway forwards it downstream. For Anthropic's native SDK, use `x-api-key` instead. |
| `x-inference-provider` | No | Forces routing to a specific provider, such as `openai`, `anthropic`, `gemini`, `vertex-ai`, or `cerebras`. Usually inferred from the SDK, path, or `x-inference-provider-url`; set it only to override that inference. |
| `x-inference-environment` | No | Tags requests with an environment, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics. |
| `x-inference-provider-url` | No | Routes to any OpenAI-compatible provider by specifying its base URL. For Vertex native APIs, set this to the global or regional `aiplatform.googleapis.com` base URL. |
## Supported OpenAI-compatible Provider URLs [#supported-openai-compatible-provider-urls]
Any OpenAI-compatible provider can be used via the `x-inference-provider-url` header, even when it does not have a dedicated guide in the catalog yet.
| Provider | Base URL |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OpenAI | `https://api.openai.com/v1` |
| OpenRouter | `https://openrouter.ai/api/v1` |
| Amazon Bedrock | `https://bedrock-mantle.{region}.api.aws/v1` |
| Anthropic | `https://api.anthropic.com/v1` |
| Google Gemini | `https://generativelanguage.googleapis.com` for native Gemini paths; `https://generativelanguage.googleapis.com/v1beta/openai` for OpenAI-compatible calls |
| Vertex AI | `https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/endpoints/openapi` |
| Azure OpenAI | `https://{resource}.openai.azure.com/openai/deployments/{deployment}` |
| Groq | `https://api.groq.com/openai/v1` |
| Together AI | `https://api.together.xyz/v1` |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` |
| Perplexity | `https://api.perplexity.ai` |
| Mistral | `https://api.mistral.ai/v1` |
| DeepSeek | `https://api.deepseek.com/v1` |
| Cerebras | `https://api.cerebras.ai/v1` |
| Inference.net | `https://api.inference.net/v1` |
For Bedrock Runtime's OpenAI-compatible endpoint, use `https://bedrock-runtime.{region}.amazonaws.com/v1` instead of the Bedrock Mantle host. For regional Vertex AI endpoints, use `https://{location}-aiplatform.googleapis.com/v1/projects/{project}/locations/{location}/endpoints/openapi` instead of the global host.
## What Gets Captured [#what-gets-captured]
Once traffic is flowing through Gateway, Inference platform records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency, including time to first token (TTFT) and tokens per second
* Token counts (input and output)
* Error rates and status codes
* Model and provider
These metrics are captured automatically without any changes to your request logic. TTFT and tokens-per-second are specifically the metrics that are invisible to application code; only a proxy can measure them.
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Quickstart" icon="rocket" href="/integrations/gateway/quickstart">
Install with AI or wire it up by hand and capture your first LLM call.
</Card>
<Card title="Record your first LLM call" icon="bolt" href="/get-started/record-first-call">
The higher-level Get Started flow with the same setup paths.
</Card>
<Card title="Metrics Explorer" icon="chart-line" href="/platform/gateway/metrics-explorer">
Watch cost, latency, errors, and token usage across all your calls.
</Card>
<Card title="Tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective with `x-inference-task-id`.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
<Card title="Run an AutoEval" icon="flask" href="/platform/eval/auto-evals/run-an-auto-eval">
Compare models side by side on your captured traffic.
</Card>
</CardGroup>
# Gateway Quickstart (https://docs.inference.net/integrations/gateway/quickstart)
This page is the gateway-focused quickstart. Point your SDK at `https://api.inference.net/v1`, add a couple of headers, and Inference platform captures every request with cost, latency, and full request/response payloads. If you'd rather see the higher-level Get Started flow, start with [Record your first LLM call](/get-started/record-first-call).
The example below uses OpenAI. For other providers (Anthropic, Vertex AI, Amazon Bedrock, Gemini, OpenRouter, Cerebras, Groq, LangChain, ElevenLabs), see the [Gateway overview](/integrations/gateway/overview#supported-providers).
## Choose a setup path [#choose-a-setup-path]
Installing with AI is the quickest. Use the manual flow if you want to wire it up yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to launch a coding agent like [Claude Code](https://code.claude.com/docs/en/overview), OpenCode, or Codex to scan your codebase, update your LLM clients, and add the routing headers.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata text="integrations/gateway/quickstart-ai-auth" />
```bash
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run gateway instrumentation in your project">
From your project root, run instrumentation in gateway mode.
<Metadata text="integrations/gateway/quickstart-ai-instrument" />
```bash
cd /path/to/your/project && inf instrument --mode gateway
```
The command guides you through the following workflow:
* Select a coding agent: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain, etc.
* Redirect base URLs to the Inference Gateway.
* Add routing headers so requests are authenticated, forwarded, and tagged.
* Add task IDs so each call site is grouped automatically in the dashboard.
* Review the generated changes before applying them.
<Tip>
Pick `both` instead of `gateway` to also install the tracing SDK in the same pass. Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would. Requests now flow through Gateway and appear in the dashboard.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to see request details and analytics.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to wire it up yourself. The example below uses OpenAI. For Anthropic, Vertex AI, Amazon Bedrock, Gemini, OpenRouter, Cerebras, Groq, LangChain, and ElevenLabs, see the per-provider guides linked from the [Gateway overview](/integrations/gateway/overview#supported-providers).
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Provider API key** (in this example, OpenAI) from your [OpenAI account](https://platform.openai.com/api-keys)
Set them as environment variables:
<Metadata text="integrations/gateway/quickstart-env" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point your SDK at `https://api.inference.net/v1` and use your Inference project API key as the SDK's `apiKey`. Your provider's API key goes in the `x-inference-provider-api-key` header so the gateway can forward it. The gateway adds roughly 10ms of latency and forwards your requests to the provider as-is.
<CodeGroup>
<Metadata text="integrations/gateway/quickstart-openai" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
<Metadata text="integrations/gateway/quickstart-openai" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
<Metadata text="integrations/gateway/quickstart-openai-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
</Step>
<Step title="Send a request">
Run the snippet above. Once the request completes, Inference platform captures it automatically.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to inspect the request and metrics.
</Step>
</Steps>
<Note>
Need a different provider? See the [Gateway overview](/integrations/gateway/overview#supported-providers) for per-provider guides, or use any OpenAI-compatible endpoint through the `x-inference-provider-url` header.
</Note>
</Tab>
</Tabs>
That's it. Every request now flows through Gateway and gets captured automatically.
## Headers [#headers]
These headers control routing, authentication, and how the request gets tagged in the dashboard. The only one required for every request is `Authorization`. Add the others as needed.
| Header | Required | Description |
| ------------------------------ | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>`. Authenticates the request to the Inference platform and selects the project scope. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `x-inference-provider-api-key` | When proxying a provider | Your downstream provider's API key (OpenAI, Bedrock, Groq, Cerebras, etc.). The gateway forwards it as bearer auth so your code never has to. For Anthropic's native `/v1/messages` route, use `x-api-key` instead. |
| `x-inference-provider` | Optional | Forces routing to a specific provider (`openai`, `anthropic`, `cerebras`, `vertex-ai`, `gemini`). When omitted, the gateway infers the provider from the SDK path or base URL. Set this only when you want to override that inference. |
| `x-inference-provider-url` | Optional | Routes to any OpenAI-compatible provider by base URL, even one without a dedicated integration. For third-party OpenAI-compatible URLs, the gateway infers OpenAI automatically. Pair with `x-inference-provider` only when you want to force a specific provider name. |
| `x-inference-environment` | Optional | Tags the request with an environment name like `production`, `staging`, or `development`. Filterable in the dashboard. |
| `x-inference-task-id` | Optional | Groups requests under a logical task such as `summarize-docs` or `chat-support`. Useful for filtering, analytics, and building datasets. |
| `x-inference-metadata-*` | Optional | Attach arbitrary metadata to a request. The `x-inference-metadata-` prefix is stripped to form the key (e.g., `x-inference-metadata-chat-id: abc123` stores `chat-id: abc123`). Filter inferences and create datasets in the dashboard using these keys. |
### Provider base URLs [#provider-base-urls]
The base URL you point your SDK at determines which provider the gateway forwards to. Most providers don't need an explicit `x-inference-provider` header, the gateway figures it out from the URL.
| Provider | Base URL | Note |
| -------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| OpenAI | `https://api.openai.com/v1` | Default routing, no provider header needed. |
| OpenRouter | `https://openrouter.ai/api/v1` | No provider header needed unless you want to force `openai`. |
| Amazon Bedrock | `https://bedrock-mantle.{region}.api.aws/v1` | Use a Bedrock bearer/API key in `x-inference-provider-api-key`. |
| Anthropic | `https://api.anthropic.com/v1` | No provider header needed for `/v1/messages` or `api.anthropic.com`. |
| Google Gemini | `https://generativelanguage.googleapis.com` | Use `/v1beta/models/*` native paths, or `/v1beta/openai` for OpenAI-compatible calls. |
| Vertex AI | `https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/endpoints/openapi` | Set `x-inference-provider: vertex-ai`. |
| Azure OpenAI | `https://{resource}.openai.azure.com/openai/deployments/{deployment}` | |
| Groq | `https://api.groq.com/openai/v1` | |
| Together AI | `https://api.together.xyz/v1` | |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` | |
| Perplexity | `https://api.perplexity.ai` | |
| Mistral | `https://api.mistral.ai/v1` | |
| DeepSeek | `https://api.deepseek.com/v1` | |
| Cerebras | `https://api.cerebras.ai/v1` | |
| Inference.net | `https://api.inference.net/v1` | |
## What gets captured [#what-gets-captured]
Once traffic is flowing, Inference platform records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency, including time to first token (TTFT) and tokens per second
* Token counts (input and output)
* Error rates and status codes
* Model and provider
## Where to find your data [#where-to-find-your-data]
* **[Metrics Explorer](/platform/gateway/metrics-explorer)** for cost, latency, errors, and usage across all your LLM calls
* **[Inference Viewer](/platform/gateway/inference-viewer)** to browse and filter individual requests and responses
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Gateway overview" icon="satellite-dish" href="/integrations/gateway/overview">
Routing headers, supported providers, and the full set of OpenAI-compatible base URLs.
</Card>
<Card title="Connect more providers" icon="plug" href="/integrations/gateway/overview#supported-providers">
Set up Anthropic, Vertex AI, Amazon Bedrock, Gemini, OpenRouter, Cerebras, Groq, and more.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
</CardGroup>
# Install with AI (https://docs.inference.net/integrations/install-with-ai)
The [Inference CLI](/cli/overview) is the fastest way to connect your app to the Inference platform. It scans your codebase, finds your LLM clients, and either routes them through the gateway, adds trace collection, or both — guided by an AI coding agent (Claude Code, OpenCode, or Codex).
Want your AI coding assistant to query Inference platform resources directly? Configure the [MCP server](/integrations/mcp-server).
<Info>
Install with AI works with OpenAI, Anthropic, Amazon Bedrock, Gemini, Vertex AI, Groq, Cerebras, OpenRouter, LangChain and more.
</Info>
## What it sets up [#what-it-sets-up]
When you run `inf instrument`, you'll be asked which Inference platform product to set up:
| Mode | What it does |
| --------- | ---------------------------------------------------------------------------------------------------------------- |
| `gateway` | Routes every LLM call through the Inference Gateway so cost, latency, and usage telemetry land in Observability. |
| `tracing` | Adds the Inference Tracing SDK and instruments your agents, tools, and provider calls with span trees. |
| `both` | Sets up gateway routing first, then trace collection — in a single agent session. |
Pass `--mode gateway`, `--mode tracing`, or `--mode both` to skip the prompt (useful in CI or scripted runs).
<Steps>
<Step title="Install the CLI">
Install the Inference CLI globally.
<Metadata text="integrations/install-with-ai/install-cli" />
```bash
npm install -g @inference/cli
```
</Step>
<Step title="Sign in">
Sign in with your Inference account. Your browser will open to authenticate.
<Metadata text="integrations/install-with-ai/sign-in" />
```bash
inf auth login
```
<Tip>
Running in CI or another headless environment? Use `inf auth set-key` instead of browser login.
</Tip>
</Step>
<Step title="Run instrumentation in your project">
Navigate to your project root and run instrumentation.
<Metadata text="get-started/record-first-call-ai-instrument" />
```bash
cd /path/to/your/project && inf instrument
```
The command guides you through the following workflow:
* Pick the project to instrument into. If you don't have one, choose **Create a new project** and name it — the CLI creates it on the spot.
* Pick what to instrument: gateway, tracing, or both.
* Choose whether to create a new API key for the project.
* **Yes**: the CLI mints a project-scoped key and writes it to `.env` as `INFERENCE_API_KEY`.
* **No**: the CLI writes `INFERENCE_API_KEY=PASTE_YOUR_API_KEY_HERE` to `.env` and prints the dashboard link where you can create a key yourself. Replace the placeholder with the real key before running your app. If `.env` already has a real key, the CLI leaves it unchanged.
* Select a coding agent to use: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain, etc.
* For **gateway**: redirect base URLs to the proxy, add routing and task ID headers.
* For **tracing**: install the Inference Tracing SDK, initialize it at the app entrypoint, and add spans around agents, tools, and provider calls.
* Review the generated changes before applying them.
<Tip>
Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would to produce inference requests. Requests from your application are now routed through the gateway and will appear in the dashboard.
</Step>
<Step title="Verify it worked">
Open the [dashboard](https://inference.net/dashboard) to see request details, traces, and analytics. You can also verify from the CLI:
<Metadata text="integrations/install-with-ai/verify-instrumentation" />
```bash
inf inference list
```
</Step>
</Steps>
<Tip>
The CLI writes `INFERENCE_API_KEY` to `.env` for you during `inf instrument`. If you skipped key creation, create a key in the dashboard under **Settings → API Keys** and replace `PASTE_YOUR_API_KEY_HERE` in `.env`. Keep `.env` out of git — add it to `.gitignore`.
</Tip>
## Supported AI coding agents [#supported-ai-coding-agents]
| Agent | Binary |
| ----------- | ---------- |
| Claude Code | `claude` |
| OpenCode | `opencode` |
| Codex | `codex` |
## Supported providers [#supported-providers]
**Built-in:** OpenAI, Anthropic
**OpenAI-compatible via `x-inference-provider-url`:** Amazon Bedrock, Google Gemini, Vertex AI, Together AI, Groq, Fireworks AI, Mistral AI, Cerebras, Perplexity, DeepSeek, OpenRouter, Azure OpenAI, and any OpenAI-compatible endpoint.
**Native provider APIs:** Vertex AI native Gemini and Anthropic-on-Vertex are supported through the manual gateway headers documented in the [Vertex AI guide](/integrations/model-providers/vertex-ai).
# MCP Server (https://docs.inference.net/integrations/mcp-server)
The Inference MCP server lets compatible AI clients query and operate Inference platform resources from your project. Use it to inspect projects, models, datasets, rubrics, evals, training jobs, deployments, inferences, traces, spans, and HALO agent-trace reports without leaving the client.
The most common workflow is applying HALO fixes: ask your assistant to apply the suggested fixes for an agent, and it pulls the HALO report and edits your code directly. See [Example prompts](#example-prompts).
## Add the server [#add-the-server]
Use this URL everywhere:
```text
https://mcp.inference.net/mcp
```
Or open [mcp.inference.net](https://mcp.inference.net/) and click a client button.
The first request opens a browser. Sign in with your Inference account, pick **one project** or **every project in the team**, and leave access on **read-only** unless you want the agent to create datasets, run evals, launch training, or change deployments.
No API key goes in the config.
<Info>
**Read-only is the default.** A read grant can browse everything but cannot change resources. Turn on write only when you want MCP tools to mutate the project.
</Info>
## Desktop [#desktop]
### Claude Desktop [#claude-desktop]
1. Open **Customize → Connectors**.
2. Add a custom connector with `https://mcp.inference.net/mcp`.
3. Click **Connect** and complete sign-in.
### Codex Desktop [#codex-desktop]
1. Open **Settings → MCP servers**.
2. Add a Streamable HTTP server with `https://mcp.inference.net/mcp`.
3. Click **Authenticate**.
### ChatGPT [#chatgpt]
1. Enable Developer Mode.
2. Create a custom connector with `https://mcp.inference.net/mcp`.
3. Complete the OAuth prompt.
## Editors [#editors]
### Claude Code [#claude-code]
<Metadata text="integrations/mcp-server/claude-code" />
```text
claude mcp add --transport http inference https://mcp.inference.net/mcp
```
Complete the browser sign-in when prompted.
### Cursor [#cursor]
Click **Add to Cursor** on [mcp.inference.net](https://mcp.inference.net/), or add:
<Metadata text="integrations/mcp-server/cursor" />
```json
{
"mcpServers": {
"inference": {
"url": "https://mcp.inference.net/mcp"
}
}
}
```
### VS Code [#vs-code]
<Metadata text="integrations/mcp-server/vscode" />
```json
{
"servers": {
"inference": {
"type": "http",
"url": "https://mcp.inference.net/mcp"
}
}
}
```
### Codex CLI [#codex-cli]
<Metadata text="integrations/mcp-server/codex" />
```text
codex mcp add inference --url https://mcp.inference.net/mcp
```
Then authenticate when the client asks.
## How access is scoped [#how-access-is-scoped]
OAuth consent binds the connection to **one project** or **every current project in the team**, plus either **read** or **read+write**. A single-project grant makes `project_id` optional. A team-wide grant can see every current and future project in that team; tools that need a project then require `project_id` — call `list_projects` first.
* **One project per connection.** Pick a single project at consent when you want implicit `project_id` resolution.
* **All projects in the team.** Pick this when the agent should work across the team. New projects created later are included automatically.
* **Switching scope.** Add the server again and pick a different project or team-wide access at consent. There is no in-session project switch.
* Call `whoami` to see the current team, scoped projects, and permissions.
Header API keys still work for CI and for clients that cannot do OAuth. See [Advanced / CI](#advanced--ci).
## Example prompts [#example-prompts]
You drive the MCP server in natural language. Your assistant picks the right tools and resolves the project from your grant.
### Apply HALO fixes for an agent [#apply-halo-fixes-for-an-agent]
```text
Apply the HALO suggested fixes for the customer-support-agent.
```
If you don't name an agent:
```text
Apply the latest HALO fixes.
```
### Run HALO on demand or on a schedule [#run-halo-on-demand-or-on-a-schedule]
```text
Run HALO on the customer-support-agent over the last 24 hours, then apply the fixes.
```
```text
Create a daily HALO schedule for the customer-support-agent at 9am UTC.
Pause the HALO schedule for the customer-support-agent.
```
<Warning>
Starting a HALO run and creating a schedule consume credits and compute, and require write access. A schedule fires on its cadence until you pause or archive it. Your assistant should confirm the agent, time window, and prompt before starting.
</Warning>
### More examples [#more-examples]
```text
What project is this connection scoped to?
List the agents in this project and their execution counts.
Show me the latest HALO runs.
Summarize the errors in the last 50 inferences.
Find the slowest traces from today.
List my training jobs and their status.
Create an eval dataset from successful traffic over the last week.
```
<Note>
Read actions work with a read-only grant. Write actions (creating datasets, running evals, launching training jobs, changing deployments) need write access.
</Note>
## Advanced / CI [#advanced--ci]
Clients that cannot run OAuth can still send a project API key.
```text
URL: https://mcp.inference.net/mcp
Authorization: Bearer <your-project-api-key>
```
<Metadata text="integrations/mcp-server/claude-code-api-key" />
```text
export INFERENCE_API_KEY="<your-project-api-key>"
claude mcp add --transport http inference https://mcp.inference.net/mcp \
--header "Authorization: Bearer $INFERENCE_API_KEY"
```
<Metadata text="integrations/mcp-server/codex-api-key" />
```text
codex mcp add inference \
--url https://mcp.inference.net/mcp \
--bearer-token-env-var INFERENCE_API_KEY
```
Create keys on the [API Keys](https://inference.net/dashboard/api-keys) page. Keep them out of source control.
## Troubleshooting [#troubleshooting]
| Error | What it means | How to fix |
| ------------------------ | ----------------------------------------------------- | ----------------------------------------------------------------------------------- |
| `401` | No credential, or the token/key is invalid. | Complete browser sign-in, or check `Authorization: Bearer <sk-inference-…>` for CI. |
| `403` | The grant or key does not have the permission needed. | Re-consent with write, or use a key with write scope. |
| `Project ID is required` | Session auth, or a multi-project key with no default. | Name the project, or connect with a single-project OAuth grant. |
| `Project not found` | The requested project is outside this connection. | Re-add the server and pick that project. |
# Amazon Bedrock (https://docs.inference.net/integrations/model-providers/amazon-bedrock)
Route Amazon Bedrock requests through the Inference gateway to get cost tracking, latency monitoring, and analytics. Amazon Bedrock exposes OpenAI-compatible Chat Completions endpoints and Anthropic Messages endpoints on Bedrock Mantle, so you can keep using the OpenAI or Anthropic SDKs with the `x-inference-provider-url` header.
<Info>
Prefer automatic setup? Run
`inf instrument`
to instrument your codebase in seconds.
[Learn more](/integrations/install-with-ai)
</Info>
<Note>
This guide covers Bedrock Mantle with a Bedrock bearer/API key. Bedrock Runtime
`Converse`, `InvokeModel`, and SigV4-only calls use different request formats
and are not covered by this Gateway setup.
</Note>
## OpenAI-compatible Chat Completions [#openai-compatible-chat-completions]
Use this path for Bedrock models that support the OpenAI-compatible Chat Completions API.
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Amazon Bedrock API key** — from your AWS account. Bedrock also recognizes this as `AWS_BEARER_TOKEN_BEDROCK`.
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/amazon-bedrock/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-api-key>
export AWS_REGION=us-east-1
export BEDROCK_BASE_URL=https://bedrock-mantle.${AWS_REGION}.api.aws/v1
export BEDROCK_MODEL=openai.gpt-oss-120b
export BEDROCK_ANTHROPIC_BASE_URL=https://bedrock-mantle.${AWS_REGION}.api.aws/anthropic
export BEDROCK_ANTHROPIC_MODEL=anthropic.claude-sonnet-4-6-v1
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and the `x-inference-provider-url` header tells the gateway to forward requests to Amazon Bedrock.
<CodeGroup>
<Metadata text="integrations/amazon-bedrock/setup" />
```typescript TypeScript
import OpenAI from "openai";
const bedrockApiKey = process.env.AWS_BEARER_TOKEN_BEDROCK;
const bedrockBaseUrl =
process.env.BEDROCK_BASE_URL ?? "https://bedrock-mantle.us-east-1.api.aws/v1";
if (!bedrockApiKey) {
console.log("Set AWS_BEARER_TOKEN_BEDROCK to run this example.");
process.exit(0);
}
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": bedrockApiKey,
"x-inference-provider-url": bedrockBaseUrl,
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: process.env.BEDROCK_MODEL ?? "openai.gpt-oss-120b",
messages: [{ role: "user", content: "Reply with exactly OK." }],
max_tokens: 32,
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/amazon-bedrock/setup" />
```python Python
import os
bedrock_api_key = os.getenv("AWS_BEARER_TOKEN_BEDROCK")
bedrock_base_url = os.getenv(
"BEDROCK_BASE_URL", "https://bedrock-mantle.us-east-1.api.aws/v1"
)
if not bedrock_api_key:
print("Set AWS_BEARER_TOKEN_BEDROCK to run this example.")
raise SystemExit(0)
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": bedrock_api_key,
"x-inference-provider-url": bedrock_base_url,
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model=os.getenv("BEDROCK_MODEL", "openai.gpt-oss-120b"),
messages=[{"role": "user", "content": "Reply with exactly OK."}],
max_tokens=32,
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/amazon-bedrock/setup-curl" />
```bash cURL
if [ -z "$AWS_BEARER_TOKEN_BEDROCK" ]; then
echo "Set AWS_BEARER_TOKEN_BEDROCK to run this example."
exit 0
fi
BEDROCK_BASE_URL=${BEDROCK_BASE_URL:-https://bedrock-mantle.us-east-1.api.aws/v1}
BEDROCK_MODEL=${BEDROCK_MODEL:-openai.gpt-oss-120b}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $AWS_BEARER_TOKEN_BEDROCK" \
-H "x-inference-provider-url: $BEDROCK_BASE_URL" \
-H "Content-Type: application/json" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "'"${BEDROCK_MODEL}"'",
"max_tokens": 32,
"messages": [{"role": "user", "content": "Reply with exactly OK."}]
}'
```
</CodeGroup>
</Step>
</Steps>
For the Bedrock Runtime OpenAI-compatible endpoint, set `BEDROCK_BASE_URL` to `https://bedrock-runtime.${AWS_REGION}.amazonaws.com/v1`.
## Anthropic Messages on Bedrock [#anthropic-messages-on-bedrock]
Use this path for Claude models on Bedrock that support the Anthropic Messages API. The key detail is `x-inference-provider: anthropic`: it tells Inference platform to use Anthropic Messages extraction and forward your Bedrock API key as downstream `x-api-key`.
<Steps>
<Step title="Set environment variables">
Reuse the same Inference platform and Bedrock keys from above, then set the Bedrock Anthropic endpoint and model:
<Metadata text="integrations/amazon-bedrock/anthropic-env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-api-key>
export AWS_REGION=us-east-1
export BEDROCK_ANTHROPIC_BASE_URL=https://bedrock-mantle.${AWS_REGION}.api.aws/anthropic
export BEDROCK_ANTHROPIC_MODEL=anthropic.claude-sonnet-4-6-v1
```
</Step>
<Step title="Update your code">
Point the Anthropic SDK at Inference Gateway. Keep the Bedrock key in the SDK `apiKey` for SDK compatibility, and also pass it as `x-inference-provider-api-key` so Inference platform can forward it to Bedrock as downstream `x-api-key`.
<CodeGroup>
<Metadata text="integrations/amazon-bedrock/anthropic-setup" />
```typescript TypeScript
const bedrockApiKey = process.env.AWS_BEARER_TOKEN_BEDROCK;
const bedrockAnthropicBaseUrl =
process.env.BEDROCK_ANTHROPIC_BASE_URL ??
"https://bedrock-mantle.us-east-1.api.aws/anthropic";
if (!bedrockApiKey) {
console.log("Set AWS_BEARER_TOKEN_BEDROCK to run this example.");
process.exit(0);
}
const { default: Anthropic } = await import("@anthropic-ai/sdk");
const client = new Anthropic({
baseURL: "https://api.inference.net",
apiKey: bedrockApiKey,
defaultHeaders: {
"Authorization": `Bearer ${process.env.INFERENCE_API_KEY}`,
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": bedrockApiKey,
"x-inference-provider-url": bedrockAnthropicBaseUrl,
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.messages.create({
model: process.env.BEDROCK_ANTHROPIC_MODEL ?? "anthropic.claude-sonnet-4-6-v1",
max_tokens: 1024,
messages: [{ role: "user", content: "Reply with exactly OK." }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/amazon-bedrock/anthropic-setup" />
```python Python
import os
bedrock_api_key = os.getenv("AWS_BEARER_TOKEN_BEDROCK")
bedrock_anthropic_base_url = os.getenv(
"BEDROCK_ANTHROPIC_BASE_URL",
"https://bedrock-mantle.us-east-1.api.aws/anthropic",
)
if not bedrock_api_key:
print("Set AWS_BEARER_TOKEN_BEDROCK to run this example.")
raise SystemExit(0)
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.inference.net",
api_key=bedrock_api_key,
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": bedrock_api_key,
"x-inference-provider-url": bedrock_anthropic_base_url,
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.messages.create(
model=os.getenv("BEDROCK_ANTHROPIC_MODEL", "anthropic.claude-sonnet-4-6-v1"),
max_tokens=1024,
messages=[{"role": "user", "content": "Reply with exactly OK."}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/amazon-bedrock/anthropic-setup-curl" />
```bash cURL
if [ -z "$AWS_BEARER_TOKEN_BEDROCK" ]; then
echo "Set AWS_BEARER_TOKEN_BEDROCK to run this example."
exit 0
fi
BEDROCK_ANTHROPIC_BASE_URL=${BEDROCK_ANTHROPIC_BASE_URL:-https://bedrock-mantle.us-east-1.api.aws/anthropic}
BEDROCK_ANTHROPIC_MODEL=${BEDROCK_ANTHROPIC_MODEL:-anthropic.claude-sonnet-4-6-v1}
curl https://api.inference.net/v1/messages \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider: anthropic" \
-H "x-inference-provider-url: $BEDROCK_ANTHROPIC_BASE_URL" \
-H "x-inference-provider-api-key: $AWS_BEARER_TOKEN_BEDROCK" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "'"${BEDROCK_ANTHROPIC_MODEL}"'",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Reply with exactly OK."}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Anthropic (https://docs.inference.net/integrations/model-providers/anthropic)
Route your Anthropic requests through the Inference gateway to get cost tracking, latency monitoring, and analytics. Anthropic uses its native SDK and the `/v1/messages` endpoint, not the OpenAI-compatible path.
<Info>
This page is for routing to Anthropic's own Claude models with your Anthropic API key. To call Inference-hosted models like
`glm-5.2`
with the Anthropic SDK, see
[Anthropic SDK](/api/anthropic-sdk)
. That only needs your Inference API key.
</Info>
<Info>
Prefer automatic setup? Run
`inf instrument`
to instrument your codebase in seconds.
[Learn more](/integrations/install-with-ai)
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Anthropic API key** — from your [Anthropic console](https://console.anthropic.com/settings/keys)
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/anthropic/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export ANTHROPIC_API_KEY=<your-anthropic-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Because the Anthropic SDK sends `apiKey` as the `x-api-key` header, you pass your Anthropic key there and add the Inference project key as an `Authorization` header.
<CodeGroup>
<Metadata text="integrations/anthropic/setup" />
```typescript TypeScript
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "https://api.inference.net",
apiKey: process.env.ANTHROPIC_API_KEY,
defaultHeaders: {
"Authorization": `Bearer ${process.env.INFERENCE_API_KEY}`,
"x-inference-provider": "anthropic",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.messages.create({
model: "claude-opus-4-6",
max_tokens: 1024,
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/anthropic/setup" />
```python Python
import os
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.inference.net",
api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/anthropic/setup-curl" />
```bash cURL
curl https://api.inference.net/v1/messages \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-H "x-inference-provider: anthropic" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "claude-opus-4-6",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Cerebras (https://docs.inference.net/integrations/model-providers/cerebras)
Route your Cerebras requests through the Inference gateway to get cost tracking, latency monitoring, and analytics. Cerebras has a dedicated provider routing ID, so you use the OpenAI SDK with `x-inference-provider: cerebras`.
<Info>
Prefer automatic setup? Run
`inf instrument`
to instrument your codebase in seconds.
[Learn more](/integrations/install-with-ai)
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Cerebras API key** — from your [Cerebras dashboard](https://cloud.cerebras.ai/)
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/cerebras/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export CEREBRAS_API_KEY=<your-cerebras-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and your Cerebras key goes in `x-inference-provider-api-key`.
<CodeGroup>
<Metadata text="integrations/cerebras/setup" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.CEREBRAS_API_KEY,
"x-inference-provider": "cerebras",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "llama3.1-8b",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/cerebras/setup" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["CEREBRAS_API_KEY"],
"x-inference-provider": "cerebras",
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="llama3.1-8b",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/cerebras/setup-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $CEREBRAS_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: cerebras" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "llama3.1-8b",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Google Gemini (https://docs.inference.net/integrations/model-providers/google-gemini)
Route direct Gemini API calls through Inference Gateway to get request observability, latency tracking, and persisted request and response payloads. This guide covers Google's Gemini API at `generativelanguage.googleapis.com`, not Vertex AI.
Use the native Gemini paths when you want Google's `generateContent` and `streamGenerateContent` request format:
| Gemini API operation | Inference platform path |
| ------------------------ | ---------------------------------------------- |
| Non-streaming generation | `/v1beta/models/{model}:generateContent` |
| Streaming generation | `/v1beta/models/{model}:streamGenerateContent` |
The gateway defaults these paths to the `gemini` provider. You can still set `x-inference-provider: gemini` explicitly to make routing obvious.
<Info>
Looking for Gemini through Google Cloud Vertex AI instead? Use the [Vertex AI guide](/integrations/model-providers/vertex-ai).
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Gemini API key** — from [Google AI Studio](https://aistudio.google.com/apikey)
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/google-gemini/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export GEMINI_API_KEY=<your-gemini-api-key>
export GEMINI_MODEL=gemini-3-flash-preview
```
</Step>
<Step title="Use the Google Gen AI SDK">
The Google Gen AI SDK can point at Inference Gateway with `httpOptions.baseUrl`. The SDK sends your Gemini key as `x-goog-api-key`; Inference platform forwards that header downstream and uses `Authorization` for your Inference platform project key.
<CodeGroup>
<Metadata text="integrations/google-gemini/sdk-typescript" />
```typescript TypeScript
import { GoogleGenAI } from "@google/genai";
const ai = new GoogleGenAI({
apiKey: process.env.GEMINI_API_KEY,
httpOptions: {
baseUrl: "https://api.inference.net",
apiVersion: "v1beta",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"x-inference-provider": "gemini",
"x-inference-environment": "production",
"x-inference-task-id": "gemini-direct",
},
},
});
const response = await ai.models.generateContent({
model: process.env.GEMINI_MODEL ?? "gemini-3-flash-preview",
contents: "Reply with exactly OK.",
config: { maxOutputTokens: 128, temperature: 0 },
});
console.log(response.text);
const stream = await ai.models.generateContentStream({
model: process.env.GEMINI_MODEL ?? "gemini-3-flash-preview",
contents: "Reply with exactly OK.",
config: { maxOutputTokens: 128, temperature: 0 },
});
for await (const chunk of stream) {
process.stdout.write(chunk.text ?? "");
}
```
<Metadata text="integrations/google-gemini/sdk-python" />
```python Python
import os
from google import genai
from google.genai import types
client = genai.Client(
api_key=os.environ["GEMINI_API_KEY"],
http_options=types.HttpOptions(
base_url="https://api.inference.net",
api_version="v1beta",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "gemini",
"x-inference-environment": "production",
"x-inference-task-id": "gemini-direct",
},
),
)
response = client.models.generate_content(
model=os.getenv("GEMINI_MODEL", "gemini-3-flash-preview"),
contents="Reply with exactly OK.",
config=types.GenerateContentConfig(max_output_tokens=128, temperature=0),
)
print(response.text)
for chunk in client.models.generate_content_stream(
model=os.getenv("GEMINI_MODEL", "gemini-3-flash-preview"),
contents="Reply with exactly OK.",
config=types.GenerateContentConfig(max_output_tokens=128, temperature=0),
):
print(chunk.text or "", end="")
```
</CodeGroup>
</Step>
<Step title="Use cURL for raw Gemini paths">
Raw HTTP callers can pass the Gemini key as `x-inference-provider-api-key`. Inference platform converts that to `x-goog-api-key` when forwarding to Gemini.
<CodeGroup>
<Metadata text="integrations/google-gemini/curl-non-streaming" />
```bash cURL
curl "https://api.inference.net/v1beta/models/${GEMINI_MODEL}:generateContent" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: gemini" \
-H "x-inference-provider-api-key: ${GEMINI_API_KEY}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: gemini-direct" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 128, "temperature": 0 }
}'
```
<Metadata text="integrations/google-gemini/curl-streaming" />
```bash cURL
curl "https://api.inference.net/v1beta/models/${GEMINI_MODEL}:streamGenerateContent?alt=sse" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: gemini" \
-H "x-inference-provider-api-key: ${GEMINI_API_KEY}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: gemini-direct-stream" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 128, "temperature": 0 }
}'
```
</CodeGroup>
</Step>
</Steps>
## Headers [#headers]
| Header | Required | Description |
| ------------------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `Authorization` | Yes | `Bearer <your-project-api-key>` authenticates the request to the Inference platform and links telemetry to your project. |
| `x-inference-provider` | No | Set to `gemini` to make routing explicit. Native Gemini paths default to Gemini when omitted. |
| `x-inference-provider-api-key` | Yes for cURL | Your Gemini API key. Inference platform forwards it to Gemini as `x-goog-api-key`. |
| `x-inference-environment` | No | Tags requests with an environment, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics. |
## Supported paths [#supported-paths]
Inference platform currently supports the direct Gemini generation paths:
* `/v1beta/models/{model}:generateContent`
* `/v1beta/models/{model}:streamGenerateContent`
* `/v1/models/{model}:generateContent`
* `/v1/models/{model}:streamGenerateContent`
Other Gemini API paths should be called directly until they are explicitly supported by the gateway.
## OpenAI-compatible endpoint [#openai-compatible-endpoint]
If you would rather use the OpenAI request format (for example, to reuse an existing OpenAI SDK setup), Gemini exposes an OpenAI-compatible surface at `https://generativelanguage.googleapis.com/v1beta/openai`. Inference platform can route to it by combining the OpenAI-format path with a provider URL override:
| Header | Value |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `x-inference-provider` | `gemini` |
| `x-inference-provider-url` | `https://generativelanguage.googleapis.com/v1beta/openai` |
| `x-inference-provider-api-key` | Your Gemini API key. Inference platform forwards it as `Authorization: Bearer <key>` because the OpenAI-compat endpoint requires bearer auth. |
<Metadata text="integrations/google-gemini/curl-openai-compat" />
```bash cURL
curl "https://api.inference.net/v1/chat/completions" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: gemini" \
-H "x-inference-provider-url: https://generativelanguage.googleapis.com/v1beta/openai" \
-H "x-inference-provider-api-key: ${GEMINI_API_KEY}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: gemini-openai-compat" \
-d '{
"model": "'"${GEMINI_MODEL}"'",
"messages": [{ "role": "user", "content": "Reply with exactly OK." }],
"max_completion_tokens": 32,
"temperature": 0
}'
```
The native `:generateContent` paths above remain the recommended surface — they expose Gemini features (system instructions, thinking traces, response schemas, image inputs) that the OpenAI-compat shim does not pass through.
# Groq (https://docs.inference.net/integrations/model-providers/groq)
Route your Groq requests through the Inference gateway to get cost tracking, latency monitoring, and analytics. Groq is OpenAI-compatible, so you use the OpenAI SDK with the `x-inference-provider-url` header to specify Groq's base URL.
<Info>
Prefer automatic setup? Run
`inf instrument`
to instrument your codebase in seconds.
[Learn more](/integrations/install-with-ai)
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Groq API key** — from your [Groq console](https://console.groq.com/keys)
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/groq/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export GROQ_API_KEY=<your-groq-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and the `x-inference-provider-url` header tells the gateway to forward requests to Groq.
<CodeGroup>
<Metadata text="integrations/groq/setup" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.GROQ_API_KEY,
"x-inference-provider-url": "https://api.groq.com/openai/v1",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "llama-3.3-70b-versatile",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/groq/setup" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["GROQ_API_KEY"],
"x-inference-provider-url": "https://api.groq.com/openai/v1",
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="llama-3.3-70b-versatile",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/groq/setup-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $GROQ_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider-url: https://api.groq.com/openai/v1" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "llama-3.3-70b-versatile",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# OpenAI (https://docs.inference.net/integrations/model-providers/openai)
Route your OpenAI requests through the Inference gateway to get cost tracking, latency monitoring, and analytics. You keep your existing OpenAI API key — just point the SDK at the gateway and add a few headers.
<Info>
Prefer automatic setup? Run
`inf instrument`
to instrument your codebase in seconds.
[Learn more](/integrations/install-with-ai)
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenAI API key** — from your [OpenAI account](https://platform.openai.com/api-keys)
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/openai/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey` to authenticate with the gateway, and your OpenAI key goes in `x-inference-provider-api-key` so the gateway can forward it to OpenAI.
<CodeGroup>
<Metadata text="integrations/openai/setup" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/openai/setup" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/openai/setup-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# OpenRouter (https://docs.inference.net/integrations/model-providers/openrouter)
Route your OpenRouter requests through the Inference gateway to access hundreds of models while getting full observability. OpenRouter is OpenAI-compatible, so you use the OpenAI SDK with the `x-inference-provider-url` header.
<Info>
Prefer automatic setup? Run
`inf instrument`
to instrument your codebase in seconds.
[Learn more](/integrations/install-with-ai)
</Info>
<Info>
Inference also exposes an OpenRouter-compatible model listing at [`https://api.inference.net/openrouter/v1/models`](https://api.inference.net/openrouter/v1/models). It is public (no auth) and mirrors the shared model catalog.
</Info>
## Setup [#setup]
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenRouter API key** — from your [OpenRouter account](https://openrouter.ai/keys)
</Step>
<Step title="Set environment variables">
<Metadata text="integrations/openrouter/env-vars" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export OPENROUTER_API_KEY=<your-openrouter-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and the `x-inference-provider-url` header tells the gateway to forward requests to OpenRouter.
<CodeGroup>
<Metadata text="integrations/openrouter/setup" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENROUTER_API_KEY,
"x-inference-provider-url": "https://openrouter.ai/api/v1",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "z-ai/glm-5-turbo",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata text="integrations/openrouter/setup" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENROUTER_API_KEY"],
"x-inference-provider-url": "https://openrouter.ai/api/v1",
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="z-ai/glm-5-turbo",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata text="integrations/openrouter/setup-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider-url: https://openrouter.ai/api/v1" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "z-ai/glm-5-turbo",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Vertex AI (https://docs.inference.net/integrations/model-providers/vertex-ai)
Inference platform can proxy Vertex AI in two modes:
* **OpenAI-compatible Vertex endpoint**: use the OpenAI SDK and set `x-inference-provider-url` to your Vertex `/endpoints/openapi` URL.
* **Native Vertex APIs**: call the Vertex model operation path through Inference Gateway and set `x-inference-provider-url` to the global or regional `aiplatform.googleapis.com` base URL.
Use `x-inference-provider: vertex-ai` for both modes so Inference platform applies Vertex-specific URL and authentication handling.
<Info>
This guide is for Gemini and Anthropic through Google Cloud Vertex AI. For
direct Gemini API calls using `generativelanguage.googleapis.com`, see the
[Google Gemini guide](/integrations/model-providers/google-gemini).
</Info>
## Supported Vertex endpoints [#supported-vertex-endpoints]
| API shape | Inference platform path example | Streaming |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------- |
| OpenAI-compatible Gemini | `/v1/chat/completions` with `x-inference-provider-url: .../endpoints/openapi` | Yes |
| Native Gemini | `/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent` | No |
| Native Gemini stream | `/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:streamGenerateContent` | Yes |
| Anthropic on Vertex | `/v1/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:rawPredict` | No |
| Anthropic on Vertex stream | `/v1/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:streamRawPredict` | Yes |
| Native Gemini with API key | `/v1/publishers/google/models/{model}:generateContent` with `x-inference-provider-api-key: $GEMINI_VERTEX_API_KEY` | No |
## Environment [#environment]
```bash
export INFERENCE_API_KEY="<your-inference-project-api-key>"
export GOOGLE_CLOUD_PROJECT="<your-gcp-project-id>"
export VERTEX_LOCATION="global"
# For OpenAI-compatible Vertex and Anthropic-on-Vertex.
export VERTEX_AI_ACCESS_TOKEN="$(gcloud auth print-access-token)"
# Optional for native Gemini on Vertex. Google API keys are forwarded as ?key=...
export GEMINI_VERTEX_API_KEY="<your-google-api-key>"
```
For regional Vertex endpoints, use the regional host:
```bash
export VERTEX_BASE_URL="https://${VERTEX_LOCATION}-aiplatform.googleapis.com"
```
For the `global` location, use:
```bash
export VERTEX_BASE_URL="https://aiplatform.googleapis.com"
```
## OpenAI-compatible Vertex [#openai-compatible-vertex]
Use this path when you want to keep the OpenAI SDK shape for Vertex Gemini models.
```typescript
import OpenAI from "openai";
const projectId = process.env.GOOGLE_CLOUD_PROJECT!;
const location = process.env.VERTEX_LOCATION ?? "global";
const vertexHost =
location === "global"
? "https://aiplatform.googleapis.com"
: `https://${location}-aiplatform.googleapis.com`;
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider": "vertex-ai",
"x-inference-provider-api-key": process.env.VERTEX_AI_ACCESS_TOKEN!,
"x-inference-provider-url": `${vertexHost}/v1/projects/${projectId}/locations/${location}/endpoints/openapi`,
"x-inference-environment": "production",
},
});
const response = await client.chat.completions.create(
{
model: "google/gemini-2.0-flash-001",
messages: [{ role: "user", content: "Reply with exactly OK." }],
max_tokens: 32,
},
{
headers: { "x-inference-task-id": "vertex-openai-compatible" },
},
);
```
```python
import os
from openai import OpenAI
project_id = os.environ["GOOGLE_CLOUD_PROJECT"]
location = os.getenv("VERTEX_LOCATION", "global")
vertex_host = (
"https://aiplatform.googleapis.com"
if location == "global"
else f"https://{location}-aiplatform.googleapis.com"
)
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider": "vertex-ai",
"x-inference-provider-api-key": os.environ["VERTEX_AI_ACCESS_TOKEN"],
"x-inference-provider-url": f"{vertex_host}/v1/projects/{project_id}/locations/{location}/endpoints/openapi",
"x-inference-environment": "production",
},
)
response = client.chat.completions.create(
model="google/gemini-2.0-flash-001",
messages=[{"role": "user", "content": "Reply with exactly OK."}],
max_tokens=32,
extra_headers={"x-inference-task-id": "vertex-openai-compatible"},
)
```
## Native Gemini on Vertex [#native-gemini-on-vertex]
Use native Gemini paths when you need Vertex's `generateContent` or `streamGenerateContent` request format.
```bash
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/google/models/gemini-2.0-flash-001:generateContent" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-native-gemini" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 64, "temperature": 0 }
}'
```
If you are using a Google API key instead of an OAuth access token for Gemini, pass `GEMINI_VERTEX_API_KEY` as `x-inference-provider-api-key`. Inference platform forwards Google API keys to Vertex as the `key` query parameter.
```bash
curl "https://api.inference.net/v1/publishers/google/models/gemini-2.0-flash-001:generateContent" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${GEMINI_VERTEX_API_KEY}" \
-H "x-inference-provider-url: https://aiplatform.googleapis.com" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-native-gemini-api-key" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 64, "temperature": 0 }
}'
```
For Gemini streaming, use `:streamGenerateContent`. Add `?alt=sse` if you want Vertex to return server-sent events.
```bash
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/google/models/gemini-2.0-flash-001:streamGenerateContent?alt=sse" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-native-gemini-stream" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 64, "temperature": 0 }
}'
```
## Anthropic on Vertex [#anthropic-on-vertex]
Anthropic models on Vertex use Vertex operation paths and Anthropic's Vertex payload shape. Use a Google Cloud OAuth access token or service-account-minted access token as `x-inference-provider-api-key`.
```bash
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/anthropic/models/claude-sonnet-4-5:rawPredict" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-anthropic" \
-d '{
"anthropic_version": "vertex-2023-10-16",
"max_tokens": 512,
"messages": [
{
"role": "user",
"content": "Reply with exactly OK."
}
]
}'
```
For streaming Anthropic responses on Vertex, use `:streamRawPredict` and include `"stream": true` in the body.
```bash
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/anthropic/models/claude-sonnet-4-5:streamRawPredict" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-anthropic-stream" \
-d '{
"anthropic_version": "vertex-2023-10-16",
"max_tokens": 512,
"stream": true,
"messages": [
{
"role": "user",
"content": "Reply with exactly OK."
}
]
}'
```
## Header summary [#header-summary]
| Header | Required | Notes |
| ------------------------------ | -------- | -------------------------------------------------------------------- |
| `Authorization` | Yes | Your Inference project API key. |
| `x-inference-provider` | Yes | Set to `vertex-ai`. |
| `x-inference-provider-api-key` | Yes | Google API key for native Gemini, or OAuth2 access token for Vertex. |
| `x-inference-provider-url` | Yes | Vertex host or `/endpoints/openapi` URL, depending on the API shape. |
| `x-inference-environment` | No | Dashboard environment tag. |
| `x-inference-task-id` | No | Dashboard task grouping. |
# Integrations (https://docs.inference.net/integrations/overview)
Inference platform has three ways to connect with your stack. **Tracing integrations** use a lightweight SDK to collect the full shape of an LLM operation directly from your code, including agent runs, tool calls, framework steps, and spans you add yourself. **Gateway integrations** route your existing provider calls through Inference Gateway with a one-line base URL change, no SDK swap required. **MCP** connects compatible AI coding assistants to Inference platform resources with your project API key.
Browse the documented integrations below. If you do not see a gateway provider yet, Inference platform can still route many OpenAI-compatible endpoints through `x-inference-provider-url`.
**Jump to:** [Tracing integrations](#traces-integrations) · [Gateway integrations](#gateway-integrations) · [Routing headers](#routing-headers) · [OpenAI-compatible providers](#supported-openai-compatible-provider-urls)
## Traces Integrations [#traces-integrations]
Traces integrations use the `@inference/tracing` (TypeScript) or `inference-tracing` (Python) SDK to collect OpenInference-shaped spans directly from LLM SDKs, agent frameworks, and your own orchestration code. A single `setup()` call instruments the providers or frameworks you enable. Spans are exported over OTLP and grouped in the Inference platform by service, trace, and task.
Use Traces when you need:
* Full agent run trees, not just individual requests
* Tool calls, tool results, and multi-step framework spans
* Visibility into work that never touches Inference Gateway (local models, custom routing, non-HTTP orchestration)
<Card title="Traces overview" icon="route" href="/integrations/traces/overview">
Learn what gets captured and how to get started with Inference Tracing.
</Card>
### Browse Tracing Integrations [#browse-tracing-integrations]
<TracesCatalog />
## Gateway Integrations [#gateway-integrations]
Gateway integrations route requests through Inference Gateway with a one-line base URL change. You keep your existing provider API keys. Your Inference project API key authenticates requests to the gateway, and a small set of headers control routing, environments, and task grouping.
Because requests flow through the gateway, Inference platform can measure performance metrics that are invisible to application code: time to first token (TTFT), tokens per second, and end-to-end latency across providers. These are captured automatically without any changes to your request logic.
<Card title="Gateway overview" icon="satellite-dish" href="/integrations/gateway/overview">
Routing headers, supported providers, and the full Gateway setup reference.
</Card>
### Browse Gateway Integrations [#browse-gateway-integrations]
<GatewayCatalog />
## Routing Headers [#routing-headers]
| Header | Required | Description |
| ------------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>` authenticates the request to the gateway and links it to your project. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `x-inference-provider-api-key` | Yes | Your provider API key or token, such as OpenAI, Bedrock, Groq, Gemini, or a Google Cloud Vertex credential. The gateway forwards it downstream. For Anthropic's native SDK, use `x-api-key` instead. |
| `x-inference-provider` | No | Forces routing to a specific provider, such as `openai`, `anthropic`, `gemini`, `vertex-ai`, or `cerebras`. Usually inferred from the SDK, path, or `x-inference-provider-url`; set it only to override that inference. |
| `x-inference-environment` | No | Tags requests with an environment, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics. |
| `x-inference-provider-url` | No | Routes to any OpenAI-compatible provider by specifying its base URL. For Vertex native APIs, set this to the global or regional `aiplatform.googleapis.com` base URL. |
## Supported OpenAI-compatible Provider URLs [#supported-openai-compatible-provider-urls]
Any OpenAI-compatible provider can be used via the `x-inference-provider-url` header, even when it does not have a dedicated guide in the catalog yet.
| Provider | Base URL |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OpenAI | `https://api.openai.com/v1` |
| OpenRouter | `https://openrouter.ai/api/v1` |
| Amazon Bedrock | `https://bedrock-mantle.{region}.api.aws/v1` |
| Anthropic | `https://api.anthropic.com/v1` |
| Google Gemini | `https://generativelanguage.googleapis.com` for native Gemini paths; `https://generativelanguage.googleapis.com/v1beta/openai` for OpenAI-compatible calls |
| Vertex AI | `https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/endpoints/openapi` |
| Azure OpenAI | `https://{resource}.openai.azure.com/openai/deployments/{deployment}` |
| Groq | `https://api.groq.com/openai/v1` |
| Together AI | `https://api.together.xyz/v1` |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` |
| Perplexity | `https://api.perplexity.ai` |
| Mistral | `https://api.mistral.ai/v1` |
| DeepSeek | `https://api.deepseek.com/v1` |
| Cerebras | `https://api.cerebras.ai/v1` |
| Inference.net | `https://api.inference.net/v1` |
For regional Vertex AI endpoints, use `https://{location}-aiplatform.googleapis.com/v1/projects/{project}/locations/{location}/endpoints/openapi` instead of the global host.
# Agent Identity (https://docs.inference.net/integrations/traces/agent-identity)
Agent dashboards group executions by agent identity. For production agents,
emit a stable `agent.id` and a readable `agent.name`. Keep `agent.id` stable
across deploys, environments, and display-name changes; use `agent.name` as the
human-readable label.
If you only provide `agent.name`, Inference platform can still show observed agents, but
renames and duplicate names are harder to group. Prefer both fields whenever you
control the agent wrapper.
## Agent Span Wrapper [#agent-span-wrapper]
Use `agentSpan()` or `agent_span()` around the product-level agent run. Child
model and tool spans created inside the wrapper inherit the active agent
identity in supported Inference Tracing integrations.
<CodeGroup>
<Metadata text="integrations/traces/agent-identity-manual" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-refund-1842",
userId: "user_8675309",
role: "refunds",
system: "openai",
},
async (span) => {
const input = "Review refund request #1842.";
span.setInput(input);
const response = await client.responses.create({
model: "gpt-4o-mini",
input,
});
span.setOutput(response.output_text);
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/agent-identity-manual" />
```python Python
import os
from inference_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-refund-1842",
user_id="user_8675309",
agent_role="refunds",
system="openai",
) as span:
user_input = "Review refund request #1842."
span.set_input(user_input)
response = client.responses.create(model="gpt-4o-mini", input=user_input)
span.set_output(response.output_text)
tracing.shutdown()
```
</CodeGroup>
## Choosing IDs [#choosing-ids]
Use IDs that are stable, readable, and unique within a project:
| Good | Avoid |
| ---------------------- | ---------------------------------------- |
| `support-triage-agent` | `SupportAgent v2` |
| `refund-review-agent` | random UUID per process |
| `billing-agent-prod` | user-specific or request-specific values |
`agent.role` is optional. Use it when one workflow has multiple agents with
clear responsibilities, such as `triage`, `refunds`, or `billing`.
## Conversation Grouping [#conversation-grouping]
`sessionId` / `session_id` is an optional field that emits `session.id` on the
agent span. Use it to group agent runs that belong to the same conversation,
chat thread, or multi-turn request. Pass an ID that is stable for one
conversation and is already available in the calling code — do not treat it as
process-wide setup.
<CodeGroup>
<Metadata text="integrations/traces/agent-identity-session" />
```typescript TypeScript
await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: conversationId,
role: "support",
system: "openai",
},
async (span) => {
// ...
},
);
```
<Metadata text="integrations/traces/agent-identity-session" />
```python Python
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id=conversation_id,
agent_role="support",
system="openai",
) as span:
...
```
</CodeGroup>
## User Grouping [#user-grouping]
`userId` / `user_id` is an optional field that emits `user.id` on the agent
span. The trace dashboard lets you filter to all traces for that
user.
## What the Inference platform Uses [#what-the-inference-platform-uses]
Inference platform reads `agent.id` first and falls back to `agent.name` when no stable ID
is present. `session.id` is read separately for conversation grouping, and
`user.id` for filtering traces by user. The raw attributes remain on the span,
and ingestion promotes them into dashboard fields used by the Agents page.
# Vercel AI SDK Traces (https://docs.inference.net/integrations/traces/ai-sdk)
The Vercel AI SDK emits native OpenTelemetry spans when
`experimental_telemetry` is enabled. Inference platform provides the tracer provider and a
small helper that wires those AI SDK spans into your Inference platform trace export.
Use this guide when your app calls the `ai` package directly through
`generateText`, `streamText`, structured outputs, tools, or `ToolLoopAgent`.
The same setup works with AI SDK providers such as `@ai-sdk/openai`,
`@ai-sdk/anthropic`, and `@ai-sdk/openai-compatible`.
<Note>
For AI SDK v6, including `ai@6.0.138`, prefer the per-call
`createAISdkTelemetrySettings(...)` helper shown below. Inference platform also
registers a module-level telemetry integration for calls that pass
`experimental_telemetry: { isEnabled: true, functionId }`, but AI SDK v6
module lifecycle events do not expose the operation name. The per-call helper
preserves exact `ai.generateText`, `ai.streamText`, and model-step span names.
</Note>
## What Is Captured [#what-is-captured]
* `ai.generateText` operation spans and `ai.generateText.doGenerate` model-step
spans
* `ai.streamText` operation spans and `ai.streamText.doStream` model-step spans
* `ai.toolCall` spans for client-side tool execution
* `ToolLoopAgent.generate()` and `ToolLoopAgent.stream()` activity through the
same native AI SDK spans
* Prompt text or prompt messages, response text, structured output metadata, and
streamed text
* Tool call names, IDs, arguments, and tool results
* Token usage including input, output, total, cached input, and reasoning tokens
when the provider returns them
* `operation.name` values that include your `functionId`
* Custom metadata passed through `experimental_telemetry`
## Install [#install]
<Metadata text="integrations/traces/ai-sdk-install" />
```bash TypeScript
bun add @inference/tracing ai
```
Install the provider package you use with the AI SDK:
```bash TypeScript
bun add @ai-sdk/openai-compatible
```
Other common provider packages include `@ai-sdk/openai`, `@ai-sdk/anthropic`,
and `@ai-sdk/google`.
## Configure Export [#configure-export]
Set your Inference Tracing OTLP endpoint and token in the runtime environment. Short-lived
scripts should also set a stable service name so traces are easy to find.
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
export INFERENCE_API_KEY="..."
export INFERENCE_SERVICE_NAME="ai-sdk-worker"
```
If you route model calls through an OpenAI-compatible gateway, configure the AI
SDK provider separately:
```bash
export INFERENCE_BASE_URL="https://api.inference.net/v1"
export INFERENCE_API_KEY="..."
export INFERENCE_MODEL="meta-llama/Llama-3.1-8B-Instruct"
```
## Initialize Tracing [#initialize-tracing]
Initialize Inference Tracing before the first AI SDK call. Import the AI SDK
namespace and pass it to `setup()` so auto-detection and integration status can
see the installed module.
<Metadata text="integrations/traces/ai-sdk-init" />
```typescript TypeScript
import * as ai from "ai";
import { setup } from "@inference/tracing";
import { createAISdkTelemetrySettings } from "@inference/tracing/ai-sdk";
const tracing = await setup({
serviceName: process.env.INFERENCE_SERVICE_NAME ?? "ai-sdk-worker",
modules: { aiSdk: ai },
});
const telemetry = (functionId: string) =>
createAISdkTelemetrySettings(tracing.tracer, {
functionId,
metadata: {
route: "support-summary",
environment: process.env.NODE_ENV ?? "development",
},
});
```
Pass `experimental_telemetry: telemetry("...")` on every AI SDK call you want
to trace. The AI SDK does not apply telemetry settings globally.
If `setup({ modules: { aiSdk: ai } })` reports the AI SDK integration as
installed but you do not see spans, verify that your actual `generateText`,
`streamText`, or `ToolLoopAgent` call receives the `experimental_telemetry`
object created by `createAISdkTelemetrySettings(...)`.
## Provider Setup [#provider-setup]
This example uses an OpenAI-compatible provider, which works with the Inference platform
Gateway and other OpenAI-compatible endpoints.
<Metadata text="integrations/traces/ai-sdk-provider" />
```typescript TypeScript
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const provider = createOpenAICompatible({
name: "inference",
baseURL: process.env.INFERENCE_BASE_URL ?? "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
supportsStructuredOutputs: true,
headers: {
"x-inference-environment": "development",
"x-inference-metadata-app": "ai-sdk-worker",
},
});
const model = provider(
process.env.INFERENCE_MODEL ?? "meta-llama/Llama-3.1-8B-Instruct",
);
```
`includeUsage: true` is useful because usage metadata is what populates token
columns in the Inference platform. Some providers only return token counts for non-streaming
calls or only after a stream finishes.
## Basic Generation [#basic-generation]
<Metadata text="integrations/traces/ai-sdk-generate-text" />
```typescript TypeScript
import { generateText } from "ai";
const result = await generateText({
model,
system: "You answer in one concise sentence.",
prompt: "Summarize why trace trees are useful.",
experimental_telemetry: telemetry("support-summary-generate"),
});
console.log(result.text);
```
Expected spans:
* `ai.generateText`
* `ai.generateText.doGenerate`
Expected promoted fields include `llm_model_name`, `input_tokens`,
`output_tokens`, `total_tokens`, `input`, and `output` when the provider returns
the corresponding AI SDK attributes.
## Streaming [#streaming]
`streamText()` produces a streaming operation span and a model-step span. Consume
the stream before process shutdown so the AI SDK can finish the span and record
the reconstructed response text.
<Metadata text="integrations/traces/ai-sdk-stream-text" />
```typescript TypeScript
import { streamText } from "ai";
const result = streamText({
model,
prompt: "Stream a six-word sentence about observability.",
experimental_telemetry: telemetry("support-summary-stream"),
});
let text = "";
for await (const chunk of result.textStream) {
text += chunk;
}
console.log(text);
```
Expected spans:
* `ai.streamText`
* `ai.streamText.doStream`
For short-lived scripts, call `await tracing.shutdown()` after the stream is
fully consumed.
## Tool Calling [#tool-calling]
Tool calls create both model spans and client-side `ai.toolCall` spans. The
model-step span records the tool call requested by the model. The tool span
records your local `execute()` call and its result.
<Metadata text="integrations/traces/ai-sdk-tools" />
```typescript TypeScript
import { generateText, stepCountIs, tool } from "ai";
import { z } from "zod";
const result = await generateText({
model,
prompt: "Use the weather tool for Paris, then summarize the result.",
stopWhen: stepCountIs(2),
tools: {
weather: tool({
description: "Get the current weather for a city.",
inputSchema: z.object({ city: z.string() }),
execute: async ({ city }) => ({
city,
temperatureC: 21,
condition: "clear",
}),
}),
},
toolChoice: { type: "tool", toolName: "weather" },
experimental_telemetry: telemetry("weather-tool-generate"),
});
console.log(result.text);
```
Expected spans:
* `ai.generateText`
* one or more `ai.generateText.doGenerate` model-step spans
* `ai.toolCall` spans for executed tools
Useful raw attributes:
| Attribute | Meaning |
| ----------------------- | -------------------------------------------------------- |
| `ai.response.toolCalls` | Tool calls requested by the model on a model-step span |
| `ai.toolCall.name` | Tool name on the client-side tool span |
| `ai.toolCall.id` | Tool call ID that links model request and tool execution |
| `ai.toolCall.args` | JSON arguments passed to `execute()` |
| `ai.toolCall.result` | JSON result returned by `execute()` |
## Structured Output [#structured-output]
Structured outputs are traced through the same `generateText` operation shape.
The response text or parsed output is preserved in AI SDK attributes when the
provider returns it.
<Metadata text="integrations/traces/ai-sdk-structured-output" />
```typescript TypeScript
import { generateText, Output } from "ai";
import { z } from "zod";
const result = await generateText({
model,
prompt: "Extract city, temperatureC, and condition from: Paris is clear and 21C.",
output: Output.object({
name: "weather_report",
schema: z.object({
city: z.string(),
temperatureC: z.number(),
condition: z.string(),
}),
}),
experimental_telemetry: telemetry("weather-structured-output"),
});
console.log(result.output);
```
Use `supportsStructuredOutputs: true` on OpenAI-compatible providers when the
downstream model endpoint supports native structured outputs.
## Agents [#agents]
The AI SDK's `ToolLoopAgent` accepts `experimental_telemetry` in the agent
constructor. Agent calls then emit the same native AI SDK spans as core
functions:
* `agent.generate()` emits `ai.generateText` and `ai.generateText.doGenerate`
* `agent.stream()` emits `ai.streamText` and `ai.streamText.doStream`
* agent tool execution emits `ai.toolCall`
There is no separate `ai.agent` span today. Infer the agent loop from the
parent/child relationships, repeated model-step spans, tool-call spans, and the
`functionId` you choose.
For the Agents dashboard, wrap `ToolLoopAgent` calls with `agentSpan()` and use
the same stable agent ID for generate and stream paths.
<Metadata text="integrations/traces/ai-sdk-agent-tool" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
import { ToolLoopAgent, stepCountIs, tool } from "ai";
import { z } from "zod";
const weatherAgent = new ToolLoopAgent({
model,
instructions:
"Use available tools to answer weather questions, then give a concise final answer.",
stopWhen: stepCountIs(2),
tools: {
weather: tool({
description: "Get the current weather for a city.",
inputSchema: z.object({ city: z.string() }),
execute: async ({ city }) => ({
city,
temperatureC: 21,
condition: "clear",
}),
}),
},
toolChoice: { type: "tool", toolName: "weather" },
experimental_telemetry: telemetry("weather-agent-generate"),
});
const answer = await agentSpan(
{
agentId: "weather-agent",
agentName: "Weather Agent",
spanName: "weather-agent.run",
sessionId: "conversation-weather-paris",
role: "weather",
system: "ai-sdk",
},
async (span) => {
const prompt = "Use the weather tool for Paris, then answer in one sentence.";
span.setInput(prompt);
const output = await weatherAgent.generate({ prompt });
span.setOutput(output.text);
return output;
},
);
console.log(answer.text);
console.log(answer.steps.length);
```
## Streaming Agent [#streaming-agent]
<Metadata text="integrations/traces/ai-sdk-agent-stream" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
import { ToolLoopAgent } from "ai";
const streamingAgent = new ToolLoopAgent({
model,
instructions: "Stream concise answers.",
experimental_telemetry: telemetry("weather-agent-stream"),
});
await agentSpan(
{
agentId: "weather-agent",
agentName: "Weather Agent",
spanName: "weather-agent.run",
sessionId: "conversation-weather-paris",
role: "weather",
system: "ai-sdk",
},
async (span) => {
const prompt = "Stream a short weather summary.";
span.setInput(prompt);
const stream = await streamingAgent.stream({ prompt });
let output = "";
for await (const chunk of stream.textStream) {
output += chunk;
process.stdout.write(chunk);
}
span.setOutput(output);
},
);
```
## Next.js Route Handler [#nextjs-route-handler]
For request/response applications, initialize tracing in a module that is loaded
before route handlers call the AI SDK. How you flush depends on where the
handler runs:
* **Long-running Node server** (a container, or `next start` on your own host):
the batch processor flushes on its interval while the process stays up. Do not
flush per request; call `shutdown()` only on `SIGTERM`.
* **Serverless or edge functions** (Vercel, Lambda): the function can freeze
between invocations before the batch flushes, so flush per invocation with
`tracing.provider.forceFlush()`, never `shutdown()`. AI SDK spans only finish
once the stream is fully produced, so flush from the stream's `onFinish` and
`onError` callbacks, not before returning the response. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle).
On Vercel, keep AI SDK trace-emitting route handlers on the Node.js runtime when
using Inference Tracing's Node OpenTelemetry setup. Vercel also provides `@vercel/otel`
for framework-level OpenTelemetry initialization and custom exporters, but a
manual Inference platform sanity span appearing in the dashboard means your exporter path
is already able to reach Inference platform.
<Metadata text="integrations/traces/ai-sdk-next-route" />
```typescript TypeScript
// app/api/chat/route.ts
import * as ai from "ai";
import { streamText } from "ai";
import { setup } from "@inference/tracing";
import { createAISdkTelemetrySettings } from "@inference/tracing/ai-sdk";
const tracing = await setup({
serviceName: "next-ai-sdk-app",
modules: { aiSdk: ai },
});
const telemetry = (functionId: string) =>
createAISdkTelemetrySettings(tracing.tracer, {
functionId,
metadata: { route: "/api/chat" },
});
export const runtime = "nodejs";
export async function POST(request: Request) {
const { prompt } = await request.json();
const result = streamText({
model,
prompt,
experimental_telemetry: telemetry("chat-route-stream"),
});
return result.toUIMessageStreamResponse();
}
```
On Vercel or another serverless runtime, flush once the stream's spans finish so
they are not lost when the function freezes:
<Metadata text="integrations/traces/ai-sdk-next-route-serverless" />
```typescript TypeScript
export async function POST(request: Request) {
const { prompt } = await request.json();
const result = streamText({
model,
prompt,
experimental_telemetry: telemetry("chat-route-stream"),
// The function can freeze after the response returns, so flush once the
// stream's spans are done. Reuse the provider; never shutdown() here.
onFinish: () => tracing.provider.forceFlush(),
onError: () => tracing.provider.forceFlush(),
});
return result.toUIMessageStreamResponse();
}
```
## Verify Traces [#verify-traces]
Filter by the service name you configured:
```bash
inf trace list --range 1h --service ai-sdk-worker --limit 10
```
Look for `ai.generateText`, `ai.streamText`, and `ai.toolCall` spans. If you set
distinct `functionId` values, you can also search for the corresponding
`operation.name` attributes in the trace detail view.
For `ToolLoopAgent` workflows that use `agentSpan()`, also look for the outer
AGENT span with `agent.id` and `agent.name`; the AI SDK operation spans should
appear underneath it.
## Attribute Reference [#attribute-reference]
Inference Tracing promotes stable AI SDK attributes into canonical columns and preserves
all raw attributes for inspection.
| Inference field | AI SDK attribute |
| ------------------- | ------------------------------------------------------ |
| `llm_model_name` | `ai.model.id` |
| `input_tokens` | `ai.usage.inputTokens` or `ai.usage.promptTokens` |
| `output_tokens` | `ai.usage.outputTokens` or `ai.usage.completionTokens` |
| `total_tokens` | `ai.usage.totalTokens` or `ai.usage.tokens` |
| `cache_read_tokens` | `ai.usage.cachedInputTokens` |
| `reasoning_tokens` | `ai.usage.reasoningTokens` |
| `input_messages` | `ai.prompt.messages` |
| `input` | `ai.prompt` |
| `output` | `ai.response.text` |
Observation kinds are inferred from `ai.operationId`:
| `ai.operationId` shape | Observation kind |
| ----------------------------------------------- | ---------------- |
| `ai.generateText`, `ai.generateText.doGenerate` | `LLM` |
| `ai.streamText`, `ai.streamText.doStream` | `LLM` |
| `ai.generateObject`, `ai.streamObject` | `LLM` |
| `ai.toolCall` | `TOOL` |
| `ai.embed`, `ai.embedMany` | `EMBEDDING` |
## Common Gotchas [#common-gotchas]
* Pass `experimental_telemetry` on every AI SDK call or agent you want traced.
* Use a stable `functionId`; it appears in `operation.name` and makes filtering
easier.
* Set `includeUsage: true` on OpenAI-compatible providers when available.
* Fully consume streams before process exit.
* Call `await tracing.shutdown()` in scripts, CLIs, tests, and job workers that
exit after a run.
* Do not call `shutdown()` after each request in a long-running server.
* On serverless or edge (Vercel functions, Lambda), flush per invocation with
`tracing.provider.forceFlush()` instead of `shutdown()`, since the process can
freeze before the batch exports.
* If tool calls appear on model spans but no `ai.toolCall` span appears, confirm
the tool has an `execute()` function and is executed client-side.
# Anthropic Traces (https://docs.inference.net/integrations/traces/anthropic)
Inference platform instruments Anthropic Messages API calls in TypeScript and Python. The
span includes content blocks, tool-use blocks, model name, invocation
parameters, finish reason, usage, and prompt-cache token details when Anthropic
returns them.
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/anthropic-install-typescript" />
```bash TypeScript
bun add @inference/tracing @anthropic-ai/sdk
```
<Metadata text="integrations/traces/anthropic-install-python" />
```bash Python
pip install 'inference-tracing[anthropic]'
```
</CodeGroup>
## Basic Messages Call [#basic-messages-call]
<CodeGroup>
<Metadata text="integrations/traces/anthropic-basic" />
```typescript TypeScript
import Anthropic from "@anthropic-ai/sdk";
import { setup } from "@inference/tracing";
const tracing = await setup({ modules: { anthropic: Anthropic } });
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const message = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 128,
messages: [{ role: "user", content: "Respond with just the word hello." }],
});
console.log(message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/anthropic-basic" />
```python Python
import os
from anthropic import Anthropic
from inference_tracing import setup
tracing = setup()
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=128,
messages=[{"role": "user", "content": "Respond with just the word hello."}],
)
print(message.content)
tracing.shutdown()
```
</CodeGroup>
## Anthropic Inside An Agent [#anthropic-inside-an-agent]
<CodeGroup>
<Metadata text="integrations/traces/anthropic-agent-identity" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "research-agent",
agentName: "Research Agent",
spanName: "research-agent.run",
sessionId: "conversation-research-notes",
role: "research",
system: "anthropic",
},
async (span) => {
const input = "Summarize the latest customer note.";
span.setInput(input);
const message = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 128,
messages: [{ role: "user", content: input }],
});
span.setOutput(message.content);
},
);
```
<Metadata text="integrations/traces/anthropic-agent-identity" />
```python Python
from inference_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="research-agent",
agent_name="Research Agent",
span_name="research-agent.run",
session_id="conversation-research-notes",
agent_role="research",
system="anthropic",
) as span:
user_input = "Summarize the latest customer note."
span.set_input(user_input)
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=128,
messages=[{"role": "user", "content": user_input}],
)
span.set_output([block.model_dump() for block in message.content])
```
</CodeGroup>
## Tool Use Round Trip [#tool-use-round-trip]
Anthropic tool use is a two-turn pattern: the assistant returns a `tool_use`
block, then your app returns a matching `tool_result` block. Inference platform records
both sides of that relationship on the auto-emitted `LLM` span — what the
model asked for and the result you passed back.
<Tip>
The capture below is the **model-side** view: it shows the request/response
the LLM saw. To also capture the **caller-side** view (the actual function
that ran, its input, output, and duration), wrap the tool function in a
`TOOL` span using
[Manual spans](/integrations/traces/manual-spans#tool-chain-and-retriever-spans).
For a full agent loop, see the
[Production Agent Example](/integrations/traces/production-agent-example).
</Tip>
<Metadata text="integrations/traces/anthropic-tool-use" />
```typescript TypeScript
const tools: Anthropic.Tool[] = [
{
name: "lookup_order",
description: "Look up an order by ID.",
input_schema: {
type: "object",
properties: { orderId: { type: "string" } },
required: ["orderId"],
},
},
];
const messages: Anthropic.MessageParam[] = [
{ role: "user", content: "Check order ABC-123." },
];
const first = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 256,
tools,
messages,
});
const toolUse = first.content.find(
(block): block is Anthropic.ToolUseBlock => block.type === "tool_use",
);
if (toolUse != null) {
messages.push({ role: "assistant", content: first.content });
const args = toolUse.input as { orderId: string };
messages.push({
role: "user",
content: [
{
type: "tool_result",
tool_use_id: toolUse.id,
content: JSON.stringify({ orderId: args.orderId, status: "shipped" }),
},
],
});
const final = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 256,
tools,
messages,
});
console.log(final.content);
}
```
## Prompt Caching [#prompt-caching]
When Anthropic returns cache creation and cache read token counts, Inference platform maps
them into OpenInference token detail attributes.
<Metadata text="integrations/traces/anthropic-prompt-caching" />
```typescript TypeScript
const longSystem =
"You are a careful, terse assistant. Answer in one sentence.\n\n" +
"Reference document:\n" +
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. ".repeat(300);
const params: Anthropic.MessageCreateParamsNonStreaming = {
model: "claude-haiku-4-5",
max_tokens: 64,
system: [
{
type: "text",
text: longSystem,
cache_control: { type: "ephemeral" },
},
],
messages: [{ role: "user", content: "Is the document about lorem ipsum?" }],
};
const first = await client.messages.create(params);
const second = await client.messages.create(params);
console.log(first.usage.cache_creation_input_tokens ?? 0);
console.log(second.usage.cache_read_input_tokens ?? 0);
```
# Attributes And Span Kinds (https://docs.inference.net/integrations/traces/attributes)
Both Inference Tracing SDKs export the OpenInference attribute keys and span
kinds they emit. Use them when you author manual spans, when you need to
filter on a specific attribute from the CLI, or when you want to understand
what each span in the trace tree should carry.
`Attr` is not a whitelist. A span is an ordinary OpenTelemetry span, so you can
set any key you like with `setAttribute` (see
[custom attributes](/integrations/traces/manual-spans#adding-custom-attributes)).
The keys below are the ones the dashboard *understands*: set `Attr.MODEL_NAME`
(`llm.model_name`) and it shows the model and computes cost; set
`Attr.SESSION_ID` (`session.id`) and it groups the conversation. `Attr` just
spares you from typing, and mistyping, the wire keys by hand.
The wire-format values are byte-identical to the upstream
[OpenInference](https://github.com/Arize-ai/openinference) semantic conventions,
so OpenInference-aware viewers render Inference platform spans without configuration.
<CodeGroup>
<Metadata text="integrations/traces/attributes-import" />
```typescript TypeScript
import { Attr, SpanKindValues } from "@inference/tracing";
```
<Metadata text="integrations/traces/attributes-import" />
```python Python
from inference_tracing import Attr, SpanKindValues
```
</CodeGroup>
## Span Kinds [#span-kinds]
`SpanKindValues` is the canonical set of values for the
`openinference.span.kind` attribute. Pick the kind that best describes the
work the span wraps.
| Kind | Constant | Use For | Required Attributes | Common Optional Attributes |
| ----------- | -------------------------- | ---------------------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `AGENT` | `SpanKindValues.AGENT` | The outer span around an agent run | `SPAN_KIND`, `AGENT_ID` | `AGENT_NAME`, `AGENT_ROLE`, `SESSION_ID`, `USER_ID`, `SYSTEM`, `INPUT_VALUE`, `OUTPUT_VALUE`, `MODEL_NAME`, token counts |
| `LLM` | `SpanKindValues.LLM` | One LLM call (chat, completion, embedding-as-LLM) | `SPAN_KIND`, `MODEL_NAME` | `SYSTEM`, `LLM_SYSTEM`, `LLM_PROVIDER`, `INVOCATION_PARAMETERS`, `STREAMING`, `FINISH_REASON`, message attributes, token counts |
| `TOOL` | `SpanKindValues.TOOL` | A tool invocation by an agent | `SPAN_KIND`, `TOOL_NAME` | `TOOL_CALL_ID`, `INPUT_VALUE`, `OUTPUT_VALUE`, `INPUT_MIME_TYPE`, `OUTPUT_MIME_TYPE` |
| `CHAIN` | `SpanKindValues.CHAIN` | A chain step that is not itself an LLM call (router, postprocessor, planner) | `SPAN_KIND` | `INPUT_VALUE`, `OUTPUT_VALUE`, free-form attributes |
| `RETRIEVER` | `SpanKindValues.RETRIEVER` | A vector-search or document-lookup call | `SPAN_KIND` | `INPUT_VALUE` (the query), `OUTPUT_VALUE` (the results) |
| `EMBEDDING` | `SpanKindValues.EMBEDDING` | An embedding-model call your code makes directly | `SPAN_KIND`, `MODEL_NAME` | `INPUT_VALUE`, token counts |
The patched provider SDKs emit `LLM`-kind spans automatically with all required
and most optional attributes filled in. You only need to author `LLM` spans
manually if you are wrapping an SDK Inference platform does not patch.
## Attribute Constants [#attribute-constants]
All attributes are exported on the `Attr` object. The constant name is the
left-hand column; the wire-format key in the right-hand column is what
actually goes on the span.
### Span Identity [#span-identity]
| Constant | Wire key | On |
| ----------------- | ------------------------- | ------------------------------------------- |
| `Attr.SPAN_KIND` | `openinference.span.kind` | Every span |
| `Attr.SESSION_ID` | `session.id` | Any span, used for conversation grouping |
| `Attr.USER_ID` | `user.id` | Any span, used for filtering traces by user |
### Agent Identity [#agent-identity]
Set on `AGENT` spans, copied onto child `LLM`/`TOOL` spans by the per-SDK
patchers and by `agentSpan()` / `agent_span()` when the child is created in
the active context.
| Constant | Wire key | Purpose |
| ----------------- | ------------ | --------------------------------------------------------- |
| `Attr.AGENT_ID` | `agent.id` | Stable identifier for grouping in the Agents dashboard |
| `Attr.AGENT_NAME` | `agent.name` | Human-readable label |
| `Attr.AGENT_ROLE` | `agent.role` | Role in a multi-agent workflow (e.g. `triage`, `refunds`) |
### Inputs And Outputs [#inputs-and-outputs]
| Constant | Wire key | Notes |
| ----------------------- | ------------------ | ------------------------------------------------- |
| `Attr.INPUT_VALUE` | `input.value` | Stringified input. JSON-encode structured values. |
| `Attr.OUTPUT_VALUE` | `output.value` | Stringified output. |
| `Attr.INPUT_MIME_TYPE` | `input.mime_type` | Typically `"application/json"` or `"text/plain"`. |
| `Attr.OUTPUT_MIME_TYPE` | `output.mime_type` | Same. |
### Model And Provider [#model-and-provider]
Set on `LLM` and `EMBEDDING` spans. Also useful on `AGENT` spans to declare the
provider the agent runs on.
| Constant | Wire key | Notes |
| ---------------------------- | --------------------------- | ---------------------------------------------------------------- |
| `Attr.MODEL_NAME` | `llm.model_name` | Prefer the model the API echoed back. |
| `Attr.SYSTEM` | `gen_ai.system` | Provider identifier on AGENT spans (`"openai"`, `"anthropic"`). |
| `Attr.LLM_SYSTEM` | `llm.system` | Provider identifier on LLM spans. |
| `Attr.LLM_PROVIDER` | `llm.provider` | Alternate provider identifier on LLM spans. |
| `Attr.INVOCATION_PARAMETERS` | `llm.invocation_parameters` | JSON of the request parameters (temperature, max\_tokens, etc.). |
| `Attr.STREAMING` | `llm.streaming` | Boolean. Did the call stream. |
| `Attr.FINISH_REASON` | `llm.finish_reason` | Provider finish reason. |
### Token Usage [#token-usage]
| Constant | Wire key |
| --------------------------------------- | ---------------------------------------------- |
| `Attr.TOKEN_COUNT_PROMPT` | `llm.token_count.prompt` |
| `Attr.TOKEN_COUNT_COMPLETION` | `llm.token_count.completion` |
| `Attr.TOKEN_COUNT_TOTAL` | `llm.token_count.total` |
| `Attr.TOKEN_COUNT_PROMPT_CACHE_WRITE` | `llm.token_count.prompt_details.cache_write` |
| `Attr.TOKEN_COUNT_PROMPT_CACHE_READ` | `llm.token_count.prompt_details.cache_read` |
| `Attr.TOKEN_COUNT_COMPLETION_REASONING` | `llm.token_count.completion_details.reasoning` |
The per-SDK patchers fill these in automatically when the provider returns
usage. For manual spans, use the helpers on the
[span handle](/integrations/traces/handle-api):
* TypeScript: `span.recordTokens({ prompt, completion, total })` to pass the
counts directly.
* Python: `span.record_tokens(prompt=..., completion=..., total=...)` for
manual counts, or `span.record_usage(response.usage)` to let the SDK
normalize an OpenAI- or Anthropic-shaped usage object (including cache
fields).
### Tools [#tools]
Set on `TOOL` spans. Also set on `LLM` spans by the provider patchers when the
LLM emits a `tool_use` block, so the dashboard can show the tool the model
asked for.
| Constant | Wire key |
| ---------------------------- | ----------------------- |
| `Attr.TOOL_NAME` | `tool.name` |
| `Attr.TOOL_CALL_ID` | `tool_call.id` |
| `Attr.AGENT_TOOL_CALL_COUNT` | `agent.tool_call_count` |
| `Attr.AGENT_LLM_CALL_COUNT` | `agent.llm_call_count` |
### Messages [#messages]
`LLM` spans emit per-message attributes for each input and output message. The
keys are indexed and machine-generated. Use the `OpenInferenceAttribute`
helper to compose them rather than hand-rolling the strings.
<Metadata text="integrations/traces/attributes-messages-ts" />
```typescript TypeScript
import { OpenInferenceAttribute } from "@inference/tracing";
const prefix = OpenInferenceAttribute.inputMessagePrefix(0);
span.setAttribute(OpenInferenceAttribute.role(prefix), "user");
span.setAttribute(OpenInferenceAttribute.content(prefix), "Hello");
```
The wire keys take the form `llm.input_messages.0.message.role`,
`llm.input_messages.0.message.content`, `llm.output_messages.0.message.role`,
and so on. For tool-call messages, the helper also generates
`...tool_calls.0.tool_call.function.name`,
`...tool_calls.0.tool_call.function.arguments`, and
`...tool_calls.0.tool_call.id`.
You rarely need to author these by hand; the per-SDK patchers emit them.
This block is here so the keys are searchable when you are debugging captured
output.
## Using Attributes From The CLI [#using-attributes-from-the-cli]
The same attribute keys are filterable from `inf trace` and `inf span`:
<Metadata text="integrations/traces/attributes-cli" />
```bash
# Find all TOOL spans for one tool name
inf span list --kind TOOL --metadata "tool.name=lookup_order"
# Find traces from one conversation
inf trace list --metadata "session.id=conversation-ticket-123"
# Find all traces for one user
inf trace list --metadata "user.id=user_8675309"
# Find expensive LLM spans, sorted by cost
inf span list --kind LLM --filter "cost_total>0.05" --sort cost_total --order desc
```
See [`inf span`](/cli/spans) and [`inf trace`](/cli/traces) for the full
filter syntax.
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
The typed methods on `agentSpan` / `manual_span` handles that write these attributes for you.
</Card>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Author TOOL, CHAIN, and RETRIEVER spans inside your agent loop.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Pick stable `agent.id` and `session.id` values for the Agents dashboard.
</Card>
<Card title="CLI span reference" icon="terminal" href="/cli/spans">
Filter, search, and inspect spans by attribute from the terminal.
</Card>
</CardGroup>
# Claude Agent SDK Traces (https://docs.inference.net/integrations/traces/claude-agent-sdk)
Use this integration for applications that call the Claude Agent SDK, formerly
the Claude Code SDK. Python can patch `query()` during `setup()` before import.
TypeScript uses an explicit wrapper because ESM namespace bindings cannot be
safely patched in place.
The Claude Agent SDK integration emits an AGENT span for each query loop. To
group those executions under a stable custom ID in the Agents dashboard, wrap
the product-level query with `agentSpan()` / `agent_span()` and use the SDK
wrapper inside it.
If your app shells out to the `claude` binary instead of importing
`@anthropic-ai/claude-agent-sdk` or `claude_agent_sdk`, use
[Claude Code SDK traces](/integrations/traces/claude-code-sdk).
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/claude-agent-sdk-install-typescript" />
```bash TypeScript
bun add @inference/tracing @anthropic-ai/claude-agent-sdk
```
<Metadata text="integrations/traces/claude-agent-sdk-install-python" />
```bash Python
pip install 'inference-tracing[claude-agent-sdk]'
```
</CodeGroup>
## TypeScript Wrapped Query [#typescript-wrapped-query]
<Metadata text="integrations/traces/claude-agent-sdk-ts" />
```typescript TypeScript
import { query } from "@anthropic-ai/claude-agent-sdk";
import { setup, wrapClaudeAgentSdkQuery } from "@inference/tracing";
const tracing = await setup({ serviceName: "claude-agent" });
const tracedQuery = wrapClaudeAgentSdkQuery(query);
const stream = tracedQuery({
prompt:
"Count the number of files matching *.md under the current directory tree. " +
"Use the Bash tool. Reply with just the integer.",
options: {
maxTurns: 4,
allowedTools: ["Bash"],
permissionMode: "bypassPermissions",
},
});
for await (const message of stream) {
console.log(message);
}
await tracing.shutdown();
```
## Stable Agent Identity [#stable-agent-identity]
Use a stable `agent.id` for the logical agent or workflow you operate. The
Claude Agent SDK span remains visible inside the trace, and the outer AGENT span
provides the dashboard grouping key.
<CodeGroup>
<Metadata text="integrations/traces/claude-agent-sdk-identity-ts" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
const prompt = "Review the current diff and list risky changes.";
await agentSpan(
{
agentId: "claude-review-agent",
agentName: "Claude Review Agent",
spanName: "claude-review.run",
sessionId: "conversation-pr-review-101",
role: "code-review",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const stream = tracedQuery({ prompt, options: { maxTurns: 4 } });
let output = "";
for await (const message of stream) {
output += JSON.stringify(message) + "\n";
}
span.setOutput(output.trim());
},
);
```
<Metadata text="integrations/traces/claude-agent-sdk-identity-python" />
```python Python
from inference_tracing import agent_span
prompt = "Review the current diff and list risky changes."
async def traced_review() -> None:
options = ClaudeAgentOptions(max_turns=4)
with agent_span(
tracing.tracer,
agent_id="claude-review-agent",
agent_name="Claude Review Agent",
span_name="claude-review.run",
session_id="conversation-pr-review-101",
agent_role="code-review",
system="anthropic",
) as span:
span.set_input(prompt)
output = []
async for message in query(prompt=prompt, options=options):
output.append(str(message))
span.set_output("\n".join(output))
```
</CodeGroup>
## Python Query Loop [#python-query-loop]
<Metadata text="integrations/traces/claude-agent-sdk-python" />
```python Python
import asyncio
from inference_tracing import setup
tracing = setup(service_name="claude-agent")
from claude_agent_sdk import ClaudeAgentOptions, query
async def main() -> None:
options = ClaudeAgentOptions(
max_turns=4,
allowed_tools=["Bash"],
permission_mode="bypassPermissions",
)
async for message in query(
prompt=(
"Count files matching *.md under the current directory tree. "
"Use the Bash tool. Reply with just the integer."
),
options=options,
):
print(message)
asyncio.run(main())
tracing.shutdown()
```
## What To Look For [#what-to-look-for]
* An AGENT span for the query run
* An outer AGENT span with `agent.id` when you add the stable identity wrapper
* Nested LLM turns from the Claude Agent SDK
* Tool-use and tool-result data when built-in tools are used
* Final assistant output captured on the span
# Claude Code SDK Traces (https://docs.inference.net/integrations/traces/claude-code-sdk)
Use this guide when your application invokes Claude Code as a CLI process, such
as `claude -p`, or when you have an existing Claude Code wrapper that shells out
to the `claude` binary. The trace shape is an explicit AGENT span with the
prompt as input and Claude Code output as output.
<Info>
Anthropic renamed the Claude Code SDK to the Claude Agent SDK. If your app
imports `@anthropic-ai/claude-agent-sdk` or `claude_agent_sdk`, use
[Claude Agent SDK traces](/integrations/traces/claude-agent-sdk). If your app
invokes the `claude` binary directly, use the subprocess pattern below.
</Info>
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/claude-code-sdk-install-typescript" />
```bash TypeScript
bun add @inference/tracing
npm install -g @anthropic-ai/claude-code
```
<Metadata text="integrations/traces/claude-code-sdk-install-python" />
```bash Python
pip install inference-tracing python-dotenv
npm install -g @anthropic-ai/claude-code
```
</CodeGroup>
## TypeScript CLI Invocation [#typescript-cli-invocation]
Resolve the `claude` binary once, then wrap each non-interactive invocation with
`agentSpan()`. Set `CLAUDE_BIN` when Claude Code is installed outside `PATH` or
when you want to pin a specific executable.
<Metadata text="integrations/traces/claude-code-sdk-typescript-cli" />
```typescript TypeScript
import { execFile, execSync } from "node:child_process";
import { promisify } from "node:util";
import { agentSpan, setup } from "@inference/tracing";
const execFileP = promisify(execFile);
function resolveClaudeBin(): string {
if (process.env.CLAUDE_BIN != null && process.env.CLAUDE_BIN !== "") {
return process.env.CLAUDE_BIN;
}
const pathSansNodeModules = (process.env.PATH ?? "")
.split(":")
.filter((pathEntry) => !pathEntry.includes("/node_modules/"))
.join(":");
return execSync("which claude", {
env: { ...process.env, PATH: pathSansNodeModules },
stdio: ["pipe", "pipe", "pipe"],
})
.toString()
.trim();
}
const tracing = await setup({ serviceName: "claude-code-worker" });
const prompt = "Inspect this repository and list the top three TODO comments.";
await agentSpan(
{
agentId: "claude-code-repo-worker",
agentName: "Claude Code",
spanName: "claude-code.invocation",
sessionId: "conversation-cli-hello",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const { stdout } = await execFileP(resolveClaudeBin(), ["-p", prompt], {
encoding: "utf-8",
timeout: 120_000,
});
span.setOutput(stdout.trim());
},
);
await tracing.shutdown();
```
## Python CLI Invocation [#python-cli-invocation]
Use the same pattern in Python when a worker, job, or automation script shells
out to Claude Code.
<Metadata text="integrations/traces/claude-code-sdk-python-cli" />
```python Python
import os
import shutil
import subprocess
import sys
from inference_tracing import agent_span, setup
from dotenv import load_dotenv
def resolve_claude_bin() -> str:
override = os.environ.get("CLAUDE_BIN")
if override:
return override
resolved = shutil.which("claude")
if resolved is None:
print("`claude` binary not found on PATH.", file=sys.stderr)
sys.exit(1)
return resolved
load_dotenv()
tracing = setup(service_name="claude-code-worker")
prompt = "Inspect this repository and list the top three TODO comments."
with agent_span(
tracing.tracer,
agent_id="claude-code-repo-worker",
agent_name="Claude Code",
span_name="claude-code.invocation",
session_id="conversation-cli-hello",
system="anthropic",
) as span:
span.set_input(prompt)
completed = subprocess.run(
[resolve_claude_bin(), "-p", prompt],
capture_output=True,
text=True,
timeout=120,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
## Agent SDK Query Loop [#agent-sdk-query-loop]
If you are using Claude Code programmatically through the Agent SDK, trace the
SDK's `query()` loop instead of tracing a raw subprocess.
<CodeGroup>
<Metadata text="integrations/traces/claude-code-sdk-agent-typescript" />
```typescript TypeScript
import { query } from "@anthropic-ai/claude-agent-sdk";
import { setup, wrapClaudeAgentSdkQuery } from "@inference/tracing";
const tracing = await setup({ serviceName: "claude-code-agent" });
const tracedQuery = wrapClaudeAgentSdkQuery(query);
const stream = tracedQuery({
prompt: "Count Markdown files in this repository. Use the Bash tool.",
options: {
maxTurns: 4,
allowedTools: ["Bash"],
permissionMode: "bypassPermissions",
},
});
for await (const message of stream) {
console.log(message);
}
await tracing.shutdown();
```
<Metadata text="integrations/traces/claude-code-sdk-agent-python" />
```python Python
import asyncio
from inference_tracing import setup
tracing = setup(service_name="claude-code-agent")
from claude_agent_sdk import ClaudeAgentOptions, query
async def main() -> None:
options = ClaudeAgentOptions(
max_turns=4,
allowed_tools=["Bash"],
permission_mode="bypassPermissions",
)
async for message in query(
prompt="Count Markdown files in this repository. Use the Bash tool.",
options=options,
):
print(message)
asyncio.run(main())
tracing.shutdown()
```
</CodeGroup>
## What To Look For [#what-to-look-for]
* An AGENT span named `claude-code.invocation` for direct CLI subprocess work
* `agent.id=claude-code-repo-worker` for stable Agents dashboard grouping
* `agent.name=ClaudeCode` and `gen_ai.system=anthropic` on the span
* Prompt input and stdout output captured on the span
* Error status and exception details if the subprocess exits non-zero
* Agent SDK query spans when using the `claude_agent_sdk` package
# Cursor SDK Traces (https://docs.inference.net/integrations/traces/cursor-sdk)
Inference platform instruments the Cursor SDK in TypeScript. Initialize tracing before
calling `Agent.create()` so the `Agent` static methods and the returned
`SDKAgent` / `Run` objects are patched before any run is observed.
Use this guide for Node applications that run programmatic Cursor agents with
`@cursor/sdk`. Cursor publishes a TypeScript SDK only, so there is no Python
equivalent for this integration.
<Info>
Cursor's SDK streams runs over HTTP/2 via `@connectrpc/connect-node`. Run
Inference Tracing-enabled Cursor apps under Node (>= 22). Bun's HTTP/2 client currently
emits `NGHTTP2_FRAME_SIZE_ERROR` mid-stream against `api.cursor.com`.
</Info>
## What Is Captured [#what-is-captured]
* One AGENT span named `Cursor Agent Run` per observed run, started lazily when
`run.stream()`, `run.wait()`, `run.conversation()`, or `run.cancel()` is first
called
* `cursor.agent_id`, `cursor.run_id`, `cursor.run_status`, `cursor.duration_ms`,
request IDs, and model metadata (`llm.model_name`,
`llm.invocation_parameters`)
* `input.value` from `agent.send()` and `output.value` from streamed assistant
text, `run.wait()` results, or `run.conversation()` turns
* TOOL child spans for streamed `tool_call` events, including `tool.name`,
`tool_call.id`, JSON arguments, and tool results
* Aggregate counts: `agent.tool_call_count`, `agent.llm_call_count`
* Error status and exception details when run streaming, waiting, or cancellation
fails
## Install [#install]
<CodeGroup>
```bash TypeScript
bun add @inference/tracing @cursor/sdk
```
</CodeGroup>
## TypeScript Cursor Agent Run [#typescript-cursor-agent-run]
Initialize tracing before creating Cursor agents. Auto-instrumentation detects
`@cursor/sdk` when it is installed in the project, so the smoothest path is just
`setup()`.
<Metadata text="integrations/traces/cursor-sdk-ts" />
```typescript TypeScript
import { Agent } from "@cursor/sdk";
import { setup } from "@inference/tracing";
const tracing = await setup({
serviceName: "cursor-agent-runner",
});
const agent = await Agent.create({
apiKey: process.env.CURSOR_API_KEY!,
model: { id: "composer-2" },
local: { cwd: process.cwd() },
});
try {
const run = await agent.send("Summarize what this repository does");
for await (const event of run.stream()) {
console.log(event.type);
}
const result = await run.wait();
console.log(`run ${run.id} ${result.status}`);
} finally {
await agent[Symbol.asyncDispose]();
await tracing.shutdown();
}
```
For manual initialization, pass the SDK namespace into the granular entry point:
```typescript TypeScript
import * as CursorSdk from "@cursor/sdk";
import { setup } from "@inference/tracing";
import { instrumentCursorSdk } from "@inference/tracing/cursor-sdk";
const tracing = await setup({ autoInstrument: false });
instrumentCursorSdk(CursorSdk, tracing);
```
## Lazy Span Lifecycle [#lazy-span-lifecycle]
The AGENT span starts when the application first observes a run through
`run.stream()`, `run.wait()`, `run.conversation()`, or `run.cancel()`, not when
`agent.send()` returns. This avoids leaking open spans for fire-and-handoff
workflows where one process kicks off a Cursor run and another process observes
it later. Each unique observed run produces one AGENT span; observing the same
run twice in the same process does not double-emit.
## Stable Agent Identity [#stable-agent-identity]
Cursor's SDK exposes its own run and agent identifiers, which Inference platform preserves
as `cursor.run_id` and `cursor.agent_id`. For the canonical Agents dashboard
grouping key, wrap the Cursor run you operate with `agentSpan()` and pass your
stable product ID as `agentId`.
<Metadata text="integrations/traces/cursor-sdk-agent-identity" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "cursor-repo-maintainer",
agentName: "Cursor Repo Maintainer",
spanName: "cursor-repo-maintainer.run",
sessionId: "conversation-repo-tour",
role: "code-maintenance",
system: "cursor",
},
async (span) => {
const input = "Summarize what this repository does";
span.setInput(input);
const run = await agent.send(input);
const result = await run.wait();
span.setOutput(result);
},
);
```
## Verify in the Inference platform [#verify-in-the-inference-platform]
Filter traces by your `service.name` (for example `cursor-agent-runner`). A
successful run should show one `Cursor Agent Run` AGENT span with `cursor.run_id`
and the model name, plus nested TOOL spans for each tool call the agent
produced.
For short-lived scripts, always call `tracing.shutdown()` before process exit so
batched spans are flushed to the Inference platform.
# ElevenLabs Agents Traces (https://docs.inference.net/integrations/traces/elevenlabs)
Inference platform instruments the ElevenLabs Agents SDK in TypeScript and Python.
Initialize tracing before starting a conversation so the SDK conversation
lifecycle and client-tool callbacks are patched before the session begins.
Use this guide for applications that use `@elevenlabs/client` or the Python
`elevenlabs.conversational_ai.conversation` APIs.
## What Is Captured [#what-is-captured]
* One AGENT span named `ElevenLabs Conversation` for each conversation session
* User and agent transcript messages as OpenInference message attributes
* ElevenLabs metadata such as agent ID, conversation ID, user ID, auth mode, and
text-only mode
* TOOL child spans for registered client tools, including tool name, arguments,
result, and errors
* Error status and exception details when session startup, tool execution, or
shutdown fails
## Install [#install]
<CodeGroup>
```bash TypeScript
bun add @inference/tracing @elevenlabs/client
```
```bash Python
pip install 'inference-tracing[elevenlabs]'
```
</CodeGroup>
## TypeScript Conversation Session [#typescript-conversation-session]
Pass the ElevenLabs SDK namespace into `setup()` before calling
`Conversation.startSession()`.
```typescript TypeScript
import * as ElevenLabs from "@elevenlabs/client";
import { Conversation } from "@elevenlabs/client";
import { setup } from "@inference/tracing";
const tracing = await setup({
serviceName: "voice-support",
modules: { elevenlabs: ElevenLabs },
});
const conversation = await Conversation.startSession({
agentId: process.env.ELEVENLABS_AGENT_ID!,
textOnly: true,
userId: "user_123",
clientTools: {
lookupAppointment: async ({ user_id }) => {
return JSON.stringify({
user_id,
starts_at: "2026-04-29T10:30:00Z",
});
},
},
onMessage: (message) => {
console.log(message.role, message.message);
},
});
conversation.sendUserMessage("When is my next appointment?");
await conversation.endSession();
await tracing.shutdown();
```
For private agents, use the same tracing setup and pass the `signedUrl` or
`conversationToken` options that your ElevenLabs app already uses. Inference platform
keeps the same outer `ElevenLabs Conversation` span shape.
## Python Conversation Session [#python-conversation-session]
Call `setup()` before constructing and starting the `Conversation`. This patches
the installed ElevenLabs SDK and records the session lifecycle automatically.
```python Python
import os
from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import ClientTools, Conversation
from inference_tracing import setup
tracing = setup(service_name="voice-support")
client = ElevenLabs(api_key=os.environ.get("ELEVENLABS_API_KEY"))
client_tools = ClientTools()
client_tools.register(
"lookupAppointment",
lambda params: {
"user_id": params["user_id"],
"starts_at": "2026-04-29T10:30:00Z",
},
)
conversation = Conversation(
client,
os.environ["ELEVENLABS_AGENT_ID"],
user_id="user_123",
requires_auth=bool(os.environ.get("ELEVENLABS_API_KEY")),
audio_interface=None,
client_tools=client_tools,
callback_agent_response=lambda text: print(f"Agent: {text}"),
callback_user_transcript=lambda text: print(f"User: {text}"),
)
conversation.start_session()
conversation.send_user_message("When is my next appointment?")
conversation.end_session()
conversation_id = conversation.wait_for_session_end()
print(f"Conversation ID: {conversation_id}")
tracing.shutdown()
```
`audio_interface=None` keeps the example text-only. For voice conversations, use
your normal ElevenLabs audio interface; the instrumentation records the same
conversation span and transcript callbacks.
## Stable Agent Identity [#stable-agent-identity]
<CodeGroup>
<Metadata text="integrations/traces/elevenlabs-agent-identity-ts" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "voice-support-agent",
agentName: "Voice Support Agent",
spanName: "voice-support.run",
sessionId: "conversation-voice-1",
role: "support",
system: "elevenlabs",
},
async (span) => {
span.setInput("When is my next appointment?");
const conversation = await Conversation.startSession({
agentId: process.env.ELEVENLABS_AGENT_ID!,
textOnly: true,
userId: "user_123",
});
conversation.sendUserMessage("When is my next appointment?");
await conversation.endSession();
span.setOutput("conversation ended");
},
);
```
<Metadata text="integrations/traces/elevenlabs-agent-identity-python" />
```python Python
from inference_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="voice-support-agent",
agent_name="Voice Support Agent",
span_name="voice-support.run",
session_id="conversation-voice-1",
agent_role="support",
system="elevenlabs",
) as span:
span.set_input("When is my next appointment?")
conversation.start_session()
conversation.send_user_message("When is my next appointment?")
conversation.end_session()
conversation_id = conversation.wait_for_session_end()
span.set_output({"conversation_id": conversation_id})
```
</CodeGroup>
## Verify in the Inference platform [#verify-in-the-inference-platform]
Filter traces by `service.name=voice-support`. A successful session should show
an `ElevenLabs Conversation` AGENT span with nested TOOL spans when your agent
calls `lookupAppointment`. When you add the wrapper above, the trace also has an
outer AGENT span with your stable `agent.id`.
For short-lived scripts, always call `tracing.shutdown()` before process exit so
batched spans are flushed to the Inference platform.
# Vercel Eve Traces (https://docs.inference.net/integrations/traces/eve)
Inference platform traces Eve through Eve's native `agent/instrumentation.ts` hook. Eve
already emits OpenTelemetry spans for agent turns, model calls, sub-agent
invocations, and tools; Inference platform installs its OpenTelemetry provider from that
hook and enriches the exported spans with OpenInference attributes and `$eve.*`
workflow tags.
Use this guide for TypeScript Eve agents. If your app calls the Vercel AI SDK
directly outside Eve, use [Vercel AI SDK traces](/integrations/traces/ai-sdk)
for those direct `generateText` or `streamText` calls.
<Info>
Eve uses the presence of `agent/instrumentation.ts` as the telemetry
enablement signal. For Eve apps, export `defineCatalystEveInstrumentation()`
from that file instead of calling `setup()` in your agent code.
</Info>
## What Is Captured [#what-is-captured]
* Eve turn spans such as `ai.eve.turn` as OpenInference CHAIN spans
* Eve `invoke_agent` spans as AGENT spans
* AI SDK v7 model spans inside Eve as LLM spans, including model, provider,
input/output messages, finish reason, and token usage when the provider
returns them
* Eve `execute_tool` spans as TOOL spans, including tool name, call ID,
arguments, result, and errors when available
* Eve session, turn, parent, root, and trigger metadata
* `$eve.*` aggregate fields such as model, input tokens, output tokens, cache
tokens, and tool count
* Custom runtime metadata added from `defineCatalystEveInstrumentation()`
Inference Tracing sets `recordInputs` and `recordOutputs` to `true` by default so the
dashboard can show model and tool IO. Set either option to `false` when a
deployment should avoid exporting full prompts, responses, tool arguments, or
tool results.
## Install [#install]
Install Inference Tracing in the same package where your Eve agent runs.
<Metadata text="integrations/traces/eve-install[series=eve_setup]" />
```bash TypeScript
bun add @inference/tracing eve
```
Install the AI SDK provider package your Eve agent uses. For Inference Gateway or
another OpenAI-compatible endpoint:
<Metadata text="integrations/traces/eve-install-provider[series=eve_setup]" />
```bash TypeScript
bun add @ai-sdk/openai-compatible
```
## Configure Export [#configure-export]
Set the Inference platform traces endpoint and token before the Eve process starts.
<Metadata text="integrations/traces/eve-env[series=eve_setup]" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="eve-weather-agent"
export INFERENCE_SERVICE_VERSION="2026.06.17"
```
If your Eve model calls go through Inference Gateway, configure that provider
separately:
<Metadata text="integrations/traces/eve-gateway-env[series=eve_setup]" />
```bash
export INFERENCE_API_KEY="<your-api-key>"
export INFERENCE_BASE_URL="https://api.inference.net/v1"
export INFERENCE_MODEL="claude-haiku-4-5"
```
## Add Eve Instrumentation [#add-eve-instrumentation]
Create `agent/instrumentation.ts` at the root of your Eve agent. Eve loads this
file during agent startup.
<Metadata text="integrations/traces/eve-instrumentation[series=eve_setup]" />
```typescript TypeScript
import { defineCatalystEveInstrumentation } from "@inference/tracing/eve";
export default defineCatalystEveInstrumentation({
functionId: "weather-agent",
serviceName: "eve-weather-agent",
metadata: {
"deployment.environment": process.env.NODE_ENV ?? "development",
},
});
```
`functionId` becomes Eve's AI SDK telemetry function ID. Use a stable value for
the logical agent or workflow. `serviceName` becomes the OpenTelemetry
`service.name` resource attribute; when omitted, Inference platform uses Eve's agent name.
## Agent Provider Example [#agent-provider-example]
Your `agent/agent.ts` keeps using Eve normally. This example uses an
OpenAI-compatible AI SDK provider pointed at Inference Gateway.
<Metadata text="integrations/traces/eve-agent-provider[series=eve_setup]" />
```typescript TypeScript
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { defineAgent } from "eve";
const inference = createOpenAICompatible({
name: "inference.net",
baseURL: process.env.INFERENCE_BASE_URL ?? "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
});
export default defineAgent({
model: inference(process.env.INFERENCE_MODEL ?? "claude-haiku-4-5"),
modelContextWindowTokens: 200_000,
});
```
`includeUsage: true` lets the provider return token counts in the Inference platform.
`modelContextWindowTokens` is useful when Eve cannot infer context-window
metadata from a custom AI SDK provider model.
## Tool Spans [#tool-spans]
Eve tools are captured automatically when the runtime emits `execute_tool`
spans. You do not need to wrap the tool manually.
<Metadata text="integrations/traces/eve-tool[series=eve_setup]" />
```typescript TypeScript
import { defineTool } from "eve/tools";
import { never } from "eve/tools/approval";
import { z } from "zod";
export default defineTool({
needsApproval: never(),
description: "Get the current weather for a city.",
inputSchema: z.object({
city: z.string(),
}),
async execute(input) {
return {
city: input.city,
temperatureF: 72,
condition: "Sunny",
summary: `Sunny in ${input.city} with a light breeze.`,
};
},
});
```
When this tool runs, Inference platform records a TOOL span with `tool.name`,
`tool_call.id`, `input.value`, and `output.value`.
## Existing Eve Hooks [#existing-eve-hooks]
If you already use Eve instrumentation events, pass them into
`defineCatalystEveInstrumentation()`. Inference platform composes your `step.started`
handler with its own handler and merges the returned runtime context.
<Metadata text="integrations/traces/eve-existing-hooks[series=eve_setup]" />
```typescript TypeScript
import { defineCatalystEveInstrumentation } from "@inference/tracing/eve";
export default defineCatalystEveInstrumentation({
functionId: "support-agent",
serviceName: "eve-support-agent",
recordInputs: false,
recordOutputs: false,
metadata: {
"deployment.environment": "production",
},
events: {
"step.started": (input) => ({
runtimeContext: {
"customer.channel": input.channel.kind ?? "unknown",
},
}),
},
});
```
Only primitive metadata values are exported as attributes. Use strings, numbers,
or booleans for custom runtime context.
## Options [#options]
| Option | Purpose |
| --------------- | --------------------------------------------------------------------------------------------- |
| `functionId` | Stable Eve/AI SDK telemetry function ID. Defaults to Eve's agent name when omitted. |
| `serviceName` | OpenTelemetry `service.name`. Defaults to Eve's agent name when omitted. |
| `recordInputs` | Whether Eve/AI SDK records full inputs. Defaults to `true`. |
| `recordOutputs` | Whether Eve/AI SDK records full outputs. Defaults to `true`. |
| `metadata` | Primitive values merged into Eve's AI SDK runtime context and exported as attributes. |
| `events` | Eve instrumentation event hooks. Inference platform composes `step.started` with your hook. |
| `setup` | Optional callback invoked from Eve's startup hook after Inference Tracing setup is requested. |
Other Inference platform `setup()` options, such as `batching`, can also be
passed through. `autoInstrument` and `modules` are intentionally managed by
the Eve integration.
## Verify in the Inference platform [#verify-in-the-inference-platform]
Run your Eve agent, trigger a turn, then open the Inference platform and filter by your
`service.name`, for example `eve-weather-agent`.
A successful Eve trace should include:
* An `ai.eve.turn` CHAIN span with `$eve.parent`, `$eve.root`, and `$eve.trigger`
* An `invoke_agent` AGENT span with `agent.name` and `gen_ai.system=eve`
* One or more LLM spans for the nested AI SDK model calls
* TOOL spans for any executed Eve tools
* Token usage and model fields when the provider returns usage metadata
For local one-off smoke tests, `batching: "simple"` can make spans export as
soon as they end. For long-lived Eve processes, the default batch exporter is
usually the better fit.
## Troubleshooting [#troubleshooting]
If Eve traces do not appear:
* Confirm the file is named `agent/instrumentation.ts` and is inside the Eve
agent root.
* Confirm `INFERENCE_OTLP_ENDPOINT` and `INFERENCE_API_KEY` are available to
the Eve process.
* Use a stable `serviceName` and filter by that value in the Inference platform.
* Run Eve in a Node-compatible runtime. Inference platform configures the Node
OpenTelemetry tracer provider for this integration.
* If model spans appear without token counts, set `includeUsage: true` on the
AI SDK provider when the provider supports it.
* If Eve cannot resolve model context metadata for a custom provider, set
`modelContextWindowTokens` in `defineAgent()`.
# Snippets (https://docs.inference.net/integrations/traces/examples)
These examples are copy-paste ready. Each one shows what gets captured and
links to the integration page that covers the surface in depth.
For setup and configuration, start with the
[Traces Quickstart](/integrations/traces/quickstart). For an end-to-end view
of a real production agent, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## OpenAI Chat Completion [#openai-chat-completion]
Initialize tracing before constructing the OpenAI client. The SDK patches Chat
Completions and emits an `LLM` span with input messages, output messages,
model name, invocation parameters, finish reason, and token counts.
<CodeGroup>
<Metadata text="integrations/traces/examples-openai-chat-ts" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
serviceName: "checkout-agent",
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You answer in one short sentence." },
{ role: "user", content: "Summarize order ABC-123." },
],
max_tokens: 80,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-openai-chat-python" />
```python Python
import os
from inference_tracing import setup
from openai import OpenAI
tracing = setup(service_name="checkout-agent")
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You answer in one short sentence."},
{"role": "user", "content": "Summarize order ABC-123."},
],
max_tokens=80,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
See [OpenAI traces](/integrations/traces/openai) for tool calls, structured
outputs, and the Responses API.
## OpenAI Tool Round Trip [#openai-tool-round-trip]
Tool calls are captured on the model span. The first turn records the assistant
tool call and arguments; the second turn records the tool result in the input
message list.
<CodeGroup>
<Metadata text="integrations/traces/examples-openai-tool-ts" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
const tools = [
{
type: "function" as const,
function: {
name: "get_weather",
description: "Look up the current weather in a city.",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
];
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: "user", content: "What's the weather in San Francisco?" },
];
const first = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
const toolCalls = first.choices[0]?.message.tool_calls ?? [];
messages.push({ role: "assistant", content: null, tool_calls: toolCalls });
for (const toolCall of toolCalls) {
const args = JSON.parse(toolCall.function.arguments) as { city: string };
messages.push({
role: "tool",
tool_call_id: toolCall.id,
content: JSON.stringify({
city: args.city,
tempF: 62,
condition: "sunny",
}),
});
}
const final = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
console.log(final.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-openai-tool-python" />
```python Python
import json
from inference_tracing import setup
from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
tracing = setup()
client = OpenAI()
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Look up the current weather in a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
},
]
messages: list[ChatCompletionMessageParam] = [
{"role": "user", "content": "What's the weather in San Francisco?"},
]
first = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
tool_calls = first.choices[0].message.tool_calls or []
messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [tc.model_dump() for tc in tool_calls],
},
)
for tool_call in tool_calls:
args = json.loads(tool_call.function.arguments)
messages.append(
{
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps(
{"city": args["city"], "temp_f": 62, "condition": "sunny"},
),
},
)
final = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
print(final.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
This captures the *model-side* view of tool calling: what the LLM asked for
and the result you passed back. For a *caller-side* view that wraps the actual
function execution in its own `TOOL` span, see
[Manual spans](/integrations/traces/manual-spans#tool-chain-and-retriever-spans).
## OpenAI Structured Output [#openai-structured-output]
Structured-output requests keep the schema in `llm.invocation_parameters` and
the model response in `output.value`.
<CodeGroup>
<Metadata text="integrations/traces/examples-openai-structured-ts" />
```typescript TypeScript
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "user",
content: "Extract the city, temperature, and unit from: 72F in Berlin.",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "weather_report",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
city: { type: "string" },
temperature: { type: "number" },
unit: { type: "string", enum: ["F", "C"] },
},
required: ["city", "temperature", "unit"],
},
},
},
});
console.log(response.choices[0]?.message.content);
```
<Metadata text="integrations/traces/examples-openai-structured-python" />
```python Python
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "user",
"content": "Extract the city, temperature, and unit from: 72F in Berlin.",
},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "weather_report",
"strict": True,
"schema": {
"type": "object",
"additionalProperties": False,
"properties": {
"city": {"type": "string"},
"temperature": {"type": "number"},
"unit": {"type": "string", "enum": ["F", "C"]},
},
"required": ["city", "temperature", "unit"],
},
},
},
)
print(response.choices[0].message.content)
```
</CodeGroup>
## OpenAI Responses API [#openai-responses-api]
The Responses API is traced separately from Chat Completions. Function-call
items are normalized into the same OpenInference tool-call attributes used by
Chat Completions, so the dashboard renders them the same way.
<CodeGroup>
<Metadata text="integrations/traces/examples-openai-responses-ts" />
```typescript TypeScript
const response = await client.responses.create({
model: "gpt-4o-mini",
input: "In one sentence, what is OpenTelemetry?",
});
console.log(response.output_text);
```
<Metadata text="integrations/traces/examples-openai-responses-python" />
```python Python
response = client.responses.create(
model="gpt-4o-mini",
input="In one sentence, what is OpenTelemetry?",
)
print(response.output_text)
```
</CodeGroup>
## Anthropic Messages [#anthropic-messages]
Anthropic Messages calls emit `LLM` spans with user and assistant content
blocks, model name, invocation parameters, finish reason, and usage.
<CodeGroup>
<Metadata text="integrations/traces/examples-anthropic-basic-ts" />
```typescript TypeScript
import Anthropic from "@anthropic-ai/sdk";
import { setup } from "@inference/tracing";
const tracing = await setup({ modules: { anthropic: Anthropic } });
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const message = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 128,
messages: [{ role: "user", content: "Respond with just the word hello." }],
});
console.log(message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-anthropic-basic-python" />
```python Python
import os
from anthropic import Anthropic
from inference_tracing import setup
tracing = setup()
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=128,
messages=[{"role": "user", "content": "Respond with just the word hello."}],
)
print(message.content)
tracing.shutdown()
```
</CodeGroup>
## Anthropic Prompt Caching [#anthropic-prompt-caching]
When Anthropic returns prompt-cache usage fields, Inference platform maps them to
OpenInference token detail attributes so they show up alongside the regular
token counts on the LLM span.
<CodeGroup>
<Metadata text="integrations/traces/examples-anthropic-cache-ts" />
```typescript TypeScript
const longSystem =
"You are a careful, terse assistant. Answer in one sentence.\n\n" +
"Reference document:\n" +
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. ".repeat(300);
const params: Anthropic.MessageCreateParamsNonStreaming = {
model: "claude-haiku-4-5",
max_tokens: 64,
system: [
{
type: "text",
text: longSystem,
cache_control: { type: "ephemeral" },
},
],
messages: [{ role: "user", content: "Is the document about lorem ipsum?" }],
};
const first = await client.messages.create(params);
const second = await client.messages.create(params);
console.log(first.usage.cache_creation_input_tokens ?? 0);
console.log(second.usage.cache_read_input_tokens ?? 0);
```
<Metadata text="integrations/traces/examples-anthropic-cache-python" />
```python Python
long_system = (
"You are a careful, terse assistant. Answer in one sentence.\n\n"
"Reference document:\n"
+ "Lorem ipsum dolor sit amet, consectetur adipiscing elit. " * 300
)
params = dict(
model="claude-haiku-4-5",
max_tokens=64,
system=[
{
"type": "text",
"text": long_system,
"cache_control": {"type": "ephemeral"},
},
],
messages=[{"role": "user", "content": "Is the document about lorem ipsum?"}],
)
first = client.messages.create(**params)
second = client.messages.create(**params)
print(first.usage.cache_creation_input_tokens)
print(second.usage.cache_read_input_tokens)
```
</CodeGroup>
The cache attributes show up on the LLM span as
`llm.token_count.prompt_details.cache_write` and
`llm.token_count.prompt_details.cache_read`. See the
[Attributes reference](/integrations/traces/attributes#token-usage) for the
full set of token-detail keys.
## Manual Parent Around Automatic Children [#manual-parent-around-automatic-children]
Use an outer agent span around orchestration code when you want nested LLM,
tool, or framework spans grouped under one product-level operation. Spans
created inside the callback auto-parent under the agent span via OTel context
propagation.
<CodeGroup>
<Metadata text="integrations/traces/examples-manual-parent-ts" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-ticket-123",
system: "openai",
},
async (span) => {
const ticket = { id: "ticket_123", orderId: "ABC-123" };
span.setInput(ticket);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "user", content: `Review refund for ${ticket.orderId}` },
],
});
span.setOutput({ decision: response.choices[0]?.message.content });
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-manual-parent-python" />
```python Python
from inference_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI()
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-ticket-123",
system="openai",
) as span:
ticket = {"id": "ticket_123", "order_id": "ABC-123"}
span.set_input(ticket)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": f"Review refund for {ticket['order_id']}"},
],
)
decision = response.choices[0].message.content
span.set_output({"decision": decision})
tracing.shutdown()
```
</CodeGroup>
## OpenAI Agents With Outer Span [#openai-agents-with-outer-span]
Pair OpenAI Agents with OpenAI instrumentation. Use `agentSpan()` /
`agent_span()` for an explicit outer span; nested OpenAI calls are captured
automatically and parent under it.
<CodeGroup>
<Metadata text="integrations/traces/examples-openai-agents-ts" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run, tool } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const lookupOrder = tool({
name: "lookup_order",
description: "Look up an order by ID.",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) =>
JSON.stringify({ orderId, status: "shipped" }),
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Use tools to help customers with orders.",
tools: [lookupOrder],
model: "gpt-4o-mini",
});
const userMessage = "Where is order ABC-123?";
await agentSpan(
{
agentId: "support-agent-prod",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
system: "openai",
},
async (span) => {
span.setInput(userMessage);
const result = await run(supportAgent, userMessage, { maxTurns: 4 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-openai-agents-python" />
```python Python
import asyncio
import json
from agents import Agent, Runner, function_tool
from inference_tracing import agent_span, setup
tracing = setup()
@function_tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, "status": "shipped"})
async def run_support_agent() -> str:
agent = Agent(
name="SupportAgent",
instructions="Use tools to help customers with orders.",
tools=[lookup_order],
model="gpt-4o-mini",
)
user_message = "Where is order ABC-123?"
with agent_span(
tracing.tracer,
agent_id="support-agent-prod",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
system="openai",
) as span:
span.set_input(user_message)
result = await Runner.run(agent, input=user_message, max_turns=4)
output = str(result.final_output or "")
span.set_output(output)
return output
print(asyncio.run(run_support_agent()))
tracing.shutdown()
```
</CodeGroup>
## OpenAI Agents Handoff [#openai-agents-handoff]
Handoffs create a useful trace tree when wrapped in an outer agent span: the
triage agent, specialist agent, model calls, and tools are all grouped under
one customer request.
<Metadata text="integrations/traces/examples-openai-agents-handoff-ts" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, handoff, run, tool } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const issueRefund = tool({
name: "issue_refund",
description: "Issue a refund for an order.",
parameters: z.object({ orderId: z.string(), amount: z.number() }),
execute: async ({ orderId, amount }) =>
JSON.stringify({ ok: true, orderId, refundId: "RFD-2201", amount }),
});
const refundsAgent = new Agent({
name: "RefundsAgent",
instructions: "Handle refund requests and use issue_refund.",
tools: [issueRefund],
model: "gpt-4o-mini",
});
const billingAgent = new Agent({
name: "BillingAgent",
instructions: "Answer billing questions. Do not issue refunds.",
model: "gpt-4o-mini",
});
const triageAgent = new Agent({
name: "TriageAgent",
instructions: "Route refund requests to RefundsAgent.",
handoffs: [handoff(refundsAgent), handoff(billingAgent)],
model: "gpt-4o-mini",
});
await agentSpan(
{
agentId: "triage-agent-prod",
agentName: "Triage Agent",
spanName: "triage-agent.run",
sessionId: "conversation-refund-abc-123",
system: "openai",
},
async (span) => {
const input = "I need a refund for order ABC-123, total $42.50.";
span.setInput(input);
const result = await run(triageAgent, input, { maxTurns: 8 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
## Pi Agent With Outer Span [#pi-agent-with-outer-span]
Current Pi Agent applications use an explicit `Models` collection. Pass that
collection to `setup()` before you register providers. The model spans then
nest under the outer agent span.
<Metadata text="integrations/traces/examples-pi-agent-ts[series=pi_agent_example]" />
```typescript TypeScript
import { Agent } from "@earendil-works/pi-agent-core";
import { createModels } from "@earendil-works/pi-ai";
import { anthropicProvider } from "@earendil-works/pi-ai/providers/anthropic";
import { agentSpan, setup } from "@inference/tracing";
const models = createModels();
const tracing = await setup({
serviceName: "pi-agent",
autoInstrument: false,
modules: { piAgent: models },
});
models.setProvider(anthropicProvider());
const model = models.getModel("anthropic", "claude-sonnet-4-6");
if (!model) throw new Error("Pi model not found");
const agent = new Agent({
initialState: { systemPrompt: "Answer concisely.", model },
streamFn: models.streamSimple.bind(models),
});
await agentSpan(
{
agentId: "pi-support-agent",
agentName: "Pi Support Agent",
spanName: "pi-support-agent.run",
sessionId: "conversation-abc-123",
system: "pi-agent",
},
async (span) => {
const input = "Summarize order ABC-123.";
span.setInput(input);
await agent.prompt(input);
span.setOutput(agent.state.messages.at(-1));
},
);
await tracing.shutdown();
```
See [Pi Agent traces](/integrations/traces/pi-agent) for tool execution and
manual setup.
## LangChain Agent With Tools [#langchain-agent-with-tools]
LangChain instrumentation hooks the callback manager. You do not need to wrap
each tool or model call manually; chain, LLM, and tool spans are emitted from
the framework callbacks.
If the same agent is already wrapped with LangSmith `@traceable`, keep that
decorator in place and install the `langsmith` extra. Inference platform uses the
LangSmith OTel span as the active parent, so the LangChain and provider spans
stay grouped under the decorated run.
<CodeGroup>
<Metadata text="integrations/traces/examples-langchain-ts" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import { ChatAnthropic } from "@langchain/anthropic";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { createAgent, tool } from "langchain";
import { z } from "zod";
const tracing = await setup({
modules: { langchainCallbacksManager: CallbackManagerModule },
});
const lookupOrder = tool(
({ orderId }) => JSON.stringify({ orderId, status: "shipped", total: 42.5 }),
{
name: "lookup_order",
description: "Look up an order by ID.",
schema: z.object({ orderId: z.string() }),
},
);
const cancelOrder = tool(
({ orderId, reason }) => JSON.stringify({ ok: true, orderId, reason }),
{
name: "cancel_order",
description: "Cancel a not-yet-delivered order.",
schema: z.object({ orderId: z.string(), reason: z.string() }),
},
);
const agent = createAgent({
model: new ChatAnthropic({ model: "claude-haiku-4-5", maxTokens: 512 }),
tools: [lookupOrder, cancelOrder],
systemPrompt: "Use tools to resolve order issues.",
});
const result = await agent.invoke({
messages: [{ role: "user", content: "Cancel order ABC-123." }],
});
console.log(result.messages.at(-1)?.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-langchain-python" />
```python Python
import json
from inference_tracing import setup
from langchain.agents import create_agent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
tracing = setup()
ORDERS = {"ABC-123": {"status": "shipped", "total": 42.5}}
@tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, **ORDERS[order_id]})
@tool
def cancel_order(order_id: str, reason: str) -> str:
"""Cancel a not-yet-delivered order."""
return json.dumps({"ok": True, "order_id": order_id, "reason": reason})
llm = ChatAnthropic(model_name="claude-haiku-4-5", max_tokens=512)
agent = create_agent(
llm,
tools=[lookup_order, cancel_order],
system_prompt="Use tools to resolve order issues.",
)
result = agent.invoke(
{"messages": [{"role": "user", "content": "Cancel order ABC-123."}]},
)
print(result["messages"][-1].content)
tracing.shutdown()
```
</CodeGroup>
## Pydantic AI Structured Agent (Python) [#pydantic-ai-structured-agent-python]
Pydantic AI ships native OpenTelemetry instrumentation. Inference platform registers its
provider and enables Pydantic AI instrumentation during `setup()`.
<Metadata text="integrations/traces/examples-pydantic-ai-python" />
```python Python
from inference_tracing import setup
from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext
class CityWeather(BaseModel):
city: str
temp_c: float = Field(description="Temperature in Celsius.")
condition: str
class WeatherReport(BaseModel):
cities: list[CityWeather]
summary: str
tracing = setup()
agent = Agent(
"openai:gpt-4o-mini",
output_type=WeatherReport,
system_prompt="Use get_weather for every requested city.",
)
@agent.tool
def get_weather(_ctx: RunContext[None], city: str) -> str:
"""Look up current weather for a city."""
return f'{{"city": "{city}", "temp_c": 12, "condition": "overcast"}}'
result = agent.run_sync("What's the weather in Paris and Tokyo?")
print(result.output.summary)
tracing.shutdown()
```
## Claude Agent SDK [#claude-agent-sdk]
Python can patch the SDK during `setup()` before `query` is imported.
TypeScript uses an explicit wrapper because ESM namespace bindings cannot be
safely patched.
<CodeGroup>
<Metadata text="integrations/traces/examples-claude-agent-sdk-ts" />
```typescript TypeScript
import { query } from "@anthropic-ai/claude-agent-sdk";
import { setup, wrapClaudeAgentSdkQuery } from "@inference/tracing";
const tracing = await setup();
const tracedQuery = wrapClaudeAgentSdkQuery(query);
const stream = tracedQuery({
prompt: "Count files matching *.md under the current directory.",
options: {
maxTurns: 4,
allowedTools: ["Bash"],
permissionMode: "bypassPermissions",
},
});
for await (const message of stream) {
console.log(message);
}
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-claude-agent-sdk-python" />
```python Python
from dotenv import load_dotenv
from inference_tracing import setup
load_dotenv()
tracing = setup()
from claude_agent_sdk import ClaudeAgentOptions, query # noqa: E402
options = ClaudeAgentOptions(
max_turns=4,
allowed_tools=["Bash"],
permission_mode="bypassPermissions",
)
async for message in query(
prompt="Count files matching *.md under the current directory.",
options=options,
):
print(message)
tracing.shutdown()
```
</CodeGroup>
## CLI Or Subprocess Work [#cli-or-subprocess-work]
When a tool has no instrumentable SDK, wrap the subprocess call in an agent
span and set the input, output, and token usage when available.
<CodeGroup>
<Metadata text="integrations/traces/examples-cli-ts" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import { Codex } from "@openai/codex-sdk";
const tracing = await setup();
const codex = new Codex({ apiKey: process.env.OPENAI_API_KEY });
await agentSpan(
{
agentId: "codex-prod",
agentName: "Codex",
system: "openai",
spanName: "codex.invocation",
sessionId: "conversation-cli-hello",
},
async (span) => {
const prompt = "Reply with just the word hello.";
span.setInput(prompt);
const thread = codex.startThread({
skipGitRepoCheck: true,
sandboxMode: "read-only",
});
const turn = await thread.run(prompt);
span.setOutput(turn.finalResponse ?? "");
if (turn.usage != null) {
span.recordTokens({
prompt: turn.usage.input_tokens ?? 0,
completion: turn.usage.output_tokens ?? 0,
});
}
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/examples-cli-python" />
```python Python
import subprocess
from inference_tracing import agent_span, setup
tracing = setup()
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="codex-prod",
agent_name="Codex",
system="openai",
span_name="codex.invocation",
session_id="conversation-cli-hello",
) as span:
span.set_input(prompt)
completed = subprocess.run(
["codex", "exec", "--skip-git-repo-check", prompt],
capture_output=True,
text=True,
timeout=120,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
</CodeGroup>
## Recommended Reading Order [#recommended-reading-order]
1. [Traces Quickstart](/integrations/traces/quickstart) — install, configure
export, capture your first span.
2. [OpenAI traces](/integrations/traces/openai) or
[Anthropic traces](/integrations/traces/anthropic) — the provider you use
first.
3. [Manual spans](/integrations/traces/manual-spans) — tool, chain, and
retriever spans inside your agent loop.
4. [Production Agent Example](/integrations/traces/production-agent-example) — a
production-shaped agent end to end.
5. [Agent identity](/integrations/traces/agent-identity) — stable IDs for
dashboard grouping.
6. [Troubleshooting](/integrations/traces/troubleshooting) — missing spans,
missing attributes, shutdown.
# Span Handle API (https://docs.inference.net/integrations/traces/handle-api)
`agentSpan()` / `agent_span()` and `manualSpan()` / `manual_span()` yield a
handle to the active span. The handle exposes a typed surface for the
OpenInference attributes you typically want to set, plus an escape hatch to
the raw OTel span for anything outside that vocabulary. The same handle is
used by both helpers across both languages.
This page is the full reference for that handle. For when to call each method,
see the [Manual spans guide](/integrations/traces/manual-spans). For the wire
keys each method writes, see the
[Attributes reference](/integrations/traces/attributes).
## Shape [#shape]
<CodeGroup>
<Metadata text="integrations/traces/handle-api-shape-ts" />
```typescript TypeScript
interface AgentSpanHandle {
setInput(value: unknown): void;
setOutput(value: unknown): void;
setModel(model: string): void;
setTool(options: { name: string; callId?: string }): void;
recordTokens(tokens: {
prompt?: number;
completion?: number;
total?: number;
}): void;
recordUsage(usage: unknown): TokenUsage;
setAttribute(key: string, value: unknown): void;
setAttributes(attributes: Record<string, unknown>): void;
raw: Span; // OTel @opentelemetry/api Span
}
```
<Metadata text="integrations/traces/handle-api-shape-python" />
```python Python
class AgentSpanHandle:
span: Span # OTel opentelemetry.trace.Span
def set_input(self, value: Any) -> None: ...
def set_output(self, value: Any) -> None: ...
def set_model(self, model: str) -> None: ...
def set_tool(self, *, name: str, call_id: str | None = None) -> None: ...
def record_tokens(
self,
*,
prompt: int | None = None,
completion: int | None = None,
total: int | None = None,
) -> None: ...
def record_usage(self, usage: Any) -> None: ...
def set_attribute(self, key: str, value: Any) -> None: ...
def set_attributes(self, attributes: Mapping[str, Any]) -> None: ...
```
</CodeGroup>
## `setInput` / `set_input` [#setinput--set_input]
Records the input that started this unit of work.
* Strings are stored as-is on `input.value`.
* Non-strings are JSON-stringified.
When the input is structured and you want viewers to render it as JSON, set
`input.mime_type` yourself:
<CodeGroup>
<Metadata text="integrations/traces/handle-api-set-input-ts" />
```typescript TypeScript
span.setInput("Summarize order ABC-123");
// or
span.setInput({ ticketId: "ticket_123", orderId: "ABC-123" });
span.setAttribute(Attr.INPUT_MIME_TYPE, "application/json");
```
<Metadata text="integrations/traces/handle-api-set-input-python" />
```python Python
span.set_input("Summarize order ABC-123")
# or
span.set_input({"ticket_id": "ticket_123", "order_id": "ABC-123"})
span.set_attribute(Attr.INPUT_MIME_TYPE, "application/json")
```
</CodeGroup>
## `setOutput` / `set_output` [#setoutput--set_output]
Records the final output of this unit of work. Same coercion rules as
`setInput`. Call it once near the end of the callback, after the work has
produced its result.
## `setModel` / `set_model` [#setmodel--set_model]
Records the model name on `llm.model_name`. Useful on `AGENT` spans when the
agent picks a model dynamically and you want the model surfaced at the agent
level. The per-SDK patchers already set this on `LLM` child spans, so you
typically do not need it there.
```typescript
span.setModel("claude-haiku-4-5");
```
## `recordTokens` / `record_tokens` [#recordtokens--record_tokens]
Records token usage from explicit counts. Use this when you have aggregate
counts across an agent run, or when you are wrapping an SDK Inference platform does not
patch and you want to attach the provider's reported counts.
Any field may be omitted; only the provided fields are written. If you want
`total` on the span, pass it explicitly — `recordTokens` / `record_tokens`
does not infer it from `prompt` and `completion`. (The `recordUsage` /
`record_usage` helper documented below does infer `total` when the provider's
usage payload omits it.)
<CodeGroup>
<Metadata text="integrations/traces/handle-api-record-tokens-ts" />
```typescript TypeScript
span.recordTokens({ prompt: 820, completion: 160, total: 980 });
```
<Metadata text="integrations/traces/handle-api-record-tokens-python" />
```python Python
span.record_tokens(prompt=820, completion=160)
```
</CodeGroup>
## `recordUsage` / `record_usage` [#recordusage--record_usage]
Records token usage from a provider-shaped `usage` object. The SDK normalizes
the common shapes so you do not have to: OpenAI Chat Completions
(`prompt_tokens` / `completion_tokens`), the newer Responses spelling
(`input_tokens` / `output_tokens`), Anthropic cache accounting fields
(`cache_creation_input_tokens` / `cache_read_input_tokens`), and OpenAI's
`prompt_tokens_details.cached_tokens` /
`completion_tokens_details.reasoning_tokens`. When `total` is missing from
the payload but `prompt` and `completion` are present, the helper infers it.
The TypeScript version returns the normalized `TokenUsage` so the caller can
read the numbers back without re-parsing.
<CodeGroup>
<Metadata text="integrations/traces/handle-api-record-usage-ts" />
```typescript TypeScript
const response = await client.chat.completions.create({ /* ... */ });
if (response.usage) {
span.recordUsage(response.usage);
}
// Works with Anthropic responses too:
const message = await anthropicClient.messages.create({ /* ... */ });
span.recordUsage(message.usage);
```
<Metadata text="integrations/traces/handle-api-record-usage-python" />
```python Python
response = client.chat.completions.create(...)
if response.usage is not None:
span.record_usage(response.usage)
# Works with Anthropic responses too:
message = anthropic_client.messages.create(...)
span.record_usage(message.usage)
```
</CodeGroup>
<Note>
For Anthropic responses, `recordUsage` / `record_usage` folds
`cache_creation_input_tokens` and `cache_read_input_tokens` into the recorded
prompt token count, then also records them as separate cache-write and
cache-read attributes. The prompt count on the span therefore reflects the
total tokens that hit the model, not just the uncached prompt.
</Note>
## `setTool` / `set_tool` [#settool--set_tool]
Records tool identity on a `TOOL` span: writes `tool.name` and, when supplied,
`tool_call.id`. The `manualSpan` / `manual_span` helper accepts the same
fields as options, so `setTool` is rarely needed when authoring a TOOL span
from scratch — use it when the tool identity is only available partway
through the callback body.
<CodeGroup>
<Metadata text="integrations/traces/handle-api-set-tool-ts" />
```typescript TypeScript
await manualSpan(
{ spanName: "lookup_order.tool", spanKind: SpanKindValues.TOOL },
async (span) => {
span.setTool({ name: "lookup_order", callId: toolCallId });
span.setInput(args);
const result = await TOOLS.lookup_order(args);
span.setOutput(result);
},
);
```
<Metadata text="integrations/traces/handle-api-set-tool-python" />
```python Python
with manual_span(
tracing.tracer,
name="lookup_order.tool",
span_kind=SpanKindValues.TOOL,
) as span:
span.set_tool(name="lookup_order", call_id=tool_call_id)
span.set_input(args)
result = TOOLS["lookup_order"](**args)
span.set_output(result)
```
</CodeGroup>
## `setAttribute` / `set_attribute` / `set_attributes` [#setattribute--set_attribute--set_attributes]
Adds one or many domain-specific attributes with OTel-safe value normalization.
Strings, booleans, numbers, and homogeneous primitive arrays pass through
unchanged; mappings and heterogeneous arrays are JSON-stringified.
<CodeGroup>
<Metadata text="integrations/traces/handle-api-set-attributes-ts" />
```typescript TypeScript
span.setAttribute("app.tenant_id", tenantId);
span.setAttributes({
"app.request_channel": "slack",
"app.deploy_env": process.env.DEPLOY_ENV ?? "dev",
});
```
<Metadata text="integrations/traces/handle-api-set-attributes-python" />
```python Python
span.set_attribute("app.tenant_id", tenant_id)
span.set_attributes({
"app.request_channel": "slack",
"app.deploy_env": os.environ.get("DEPLOY_ENV", "dev"),
})
```
</CodeGroup>
## `raw` / `.span` [#raw--span]
The underlying OTel span. Use this when you need behavior outside the typed
surface:
* Adding domain attributes.
* Recording span events (`span.raw.addEvent("rate_limit_hit", { retry_after: 30 })`).
* Recording an exception without ending the span yourself
(`span.raw.recordException(err)`).
* Reading the span context to propagate to a background job
(`span.raw.spanContext()`).
The escape hatch is `span.raw` in TypeScript and `span.span` in Python.
## Value Coercion Rules [#value-coercion-rules]
| Input | Coerced to |
| ------------------------------- | ------------------------------------------------------------------------------------------------- |
| `string` | Stored verbatim. |
| `number`, `boolean` | Stored as-is. OTel handles these as primitive attribute values. |
| Plain object | `JSON.stringify(value)` (TS) / `json.dumps(value, default=str)` (Python). |
| Array of homogeneous primitives | Stored as an OTel attribute array. |
| Array containing mixed types | JSON-stringified. |
| `Error` / `Exception` | Not coerced. Use `span.raw.recordException(err)` instead, which produces an OTel exception event. |
| `bigint` (TS) | Cast yourself before passing — OTel does not accept `bigint`. |
The `setInput` and `setOutput` methods apply these rules to their single
value. The other helpers write to one attribute key each and apply the rules
to that value.
## Status Behavior [#status-behavior]
The handle does not expose `setStatus` directly. The wrapping context manager
sets status for you:
* Body returns normally → status `OK`.
* Body raises → exception is recorded as a span event, status is set to
`ERROR` with the exception's message, then the exception re-raises.
If you need fine-grained status control mid-callback (for example, to mark a
partial-success state), set it on the raw span:
<Metadata text="integrations/traces/handle-api-status" />
```typescript TypeScript
import { SpanStatusCode } from "@opentelemetry/api";
if (result.partial) {
span.raw.setStatus({
code: SpanStatusCode.ERROR,
message: `partial result: ${result.missingFields.join(", ")}`,
});
}
```
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Manual spans guide" icon="pen-nib" href="/integrations/traces/manual-spans">
Use the handle to author AGENT, TOOL, CHAIN, and RETRIEVER spans.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
All `Attr.*` constants and `SpanKindValues` values.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool execution, end to end.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
Debug missing spans, missing attributes, and unexpected status codes.
</Card>
</CardGroup>
# LangChain Traces (https://docs.inference.net/integrations/traces/langchain)
Inference platform hooks LangChain callback managers so runnable calls emit spans without
manual span creation. In agent workflows, the trace tree includes chain, model,
and tool spans with parent-child relationships preserved by LangChain callbacks.
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/langchain-install-typescript" />
```bash TypeScript
bun add @inference/tracing @langchain/core langchain @langchain/anthropic zod
```
<Metadata text="integrations/traces/langchain-install-python" />
```bash Python
pip install 'inference-tracing[langchain]'
```
</CodeGroup>
## TypeScript Agent With Tools [#typescript-agent-with-tools]
Pass the callback manager module to `setup()`. Inference platform patches the static
configuration path LangChain uses when constructing callback managers.
<Metadata text="integrations/traces/langchain-agent-ts" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import { ChatAnthropic } from "@langchain/anthropic";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { createAgent, tool } from "langchain";
import { z } from "zod";
const tracing = await setup({
serviceName: "support-agent",
modules: { langchainCallbacksManager: CallbackManagerModule },
});
const lookupOrder = tool(
({ orderId }) => JSON.stringify({ orderId, status: "shipped", total: 42.5 }),
{
name: "lookup_order",
description: "Look up an order by ID.",
schema: z.object({ orderId: z.string() }),
},
);
const cancelOrder = tool(
({ orderId, reason }) => JSON.stringify({ ok: true, orderId, reason }),
{
name: "cancel_order",
description: "Cancel a not-yet-delivered order.",
schema: z.object({ orderId: z.string(), reason: z.string() }),
},
);
const agent = createAgent({
model: new ChatAnthropic({ model: "claude-haiku-4-5", maxTokens: 512 }),
tools: [lookupOrder, cancelOrder],
systemPrompt: "Use tools to resolve order issues.",
});
const result = await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
role: "support",
system: "langchain",
},
async (span) => {
const input = "Cancel order ABC-123.";
span.setInput(input);
const output = await agent.invoke({
messages: [{ role: "user", content: input }],
});
span.setOutput(output.messages.at(-1)?.content);
return output;
},
);
console.log(result.messages.at(-1)?.content);
await tracing.shutdown();
```
## Python Agent With Tools [#python-agent-with-tools]
<Metadata text="integrations/traces/langchain-agent-python" />
```python Python
import json
from inference_tracing import agent_span, setup
from langchain.agents import create_agent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
tracing = setup(service_name="support-agent")
@tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, "status": "shipped", "total": 42.5})
@tool
def cancel_order(order_id: str, reason: str) -> str:
"""Cancel a not-yet-delivered order."""
return json.dumps({"ok": True, "order_id": order_id, "reason": reason})
llm = ChatAnthropic(model_name="claude-haiku-4-5", max_tokens=512)
agent = create_agent(
llm,
tools=[lookup_order, cancel_order],
system_prompt="Use tools to resolve order issues.",
)
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
agent_role="support",
system="langchain",
) as span:
user_input = "Cancel order ABC-123."
span.set_input(user_input)
result = agent.invoke(
{"messages": [{"role": "user", "content": user_input}]},
)
span.set_output(result["messages"][-1].content)
print(result["messages"][-1].content)
tracing.shutdown()
```
## What To Look For [#what-to-look-for]
* A top-level LangChain chain or agent span
* An outer AGENT span with `agent.id=support-agent` when you use the wrapper
* Nested LLM spans for model calls
* Tool spans named after LangChain tools
* Tool input and output attributes on tool spans
* Token counts on model spans when the provider returns usage
# Langfuse Traces (https://docs.inference.net/integrations/traces/langfuse)
If your application already uses Langfuse, this is just an env-var switch. Keep all of your existing Langfuse tracing code and point the Langfuse SDK at the Inference platform by changing three environment variables. Your spans stream to the Inference platform **Trajectories** dashboard with no SDK swap and no code changes. Inference platform accepts the Langfuse ingestion and OTEL endpoints and converts traces, generations, spans, usage, costs, inputs, outputs, metadata, users, sessions, and tags into platform spans.
This is the fastest way to try Inference platform: flip the env vars, see your traces, and switch over fully later for more control. Spans show up in the Trajectories dashboard out of the box but to get the most out of it, set an agent identity on your top-level span so runs group in the **Agents** dashboard and become available to Halo. See [Group by agent](#group-by-agent).
## Configure Langfuse [#configure-langfuse]
Use your Inference API key as the Langfuse secret key. The public key is only present for Langfuse SDK compatibility and its value is never used. Set it to `pk-inference`, but any non-empty value works.
<Metadata text="integrations/traces/langfuse-env" />
```bash
export LANGFUSE_HOST="https://telemetry.inference.net"
export LANGFUSE_PUBLIC_KEY="pk-inference"
export LANGFUSE_SECRET_KEY="<your-inference-api-key>"
```
The Inference API key must have write access to the project that should receive the traces.
## Python [#python]
<Metadata text="integrations/traces/langfuse-python" />
```python Python
from langfuse import Langfuse, observe, get_client
langfuse = Langfuse(
public_key="pk-inference",
secret_key="<your-inference-api-key>",
base_url="https://telemetry.inference.net",
)
@observe(as_type="generation", name="answer-question")
def answer_question(question: str) -> str:
client = get_client()
# Set identity the Langfuse way. These map to user.id, session.id, and tags in the Inference platform.
client.update_current_trace(
user_id="user-123",
session_id="session-abc",
tags=["prod"],
)
client.update_current_generation(
model="gpt-4o-mini",
usage_details={"input": 12, "output": 8, "total": 20},
)
return f"Answer: {question}"
print(answer_question("What is Inference platform?"))
langfuse.flush()
langfuse.shutdown()
```
## TypeScript [#typescript]
<Metadata text="integrations/traces/langfuse-typescript" />
```typescript TypeScript
import { observe, getClient } from "@langfuse/tracing";
process.env.LANGFUSE_HOST = "https://telemetry.inference.net";
process.env.LANGFUSE_PUBLIC_KEY = "pk-inference";
process.env.LANGFUSE_SECRET_KEY = "<your-inference-api-key>";
const answerQuestion = observe(
async (question: string) => {
const client = getClient();
// Set identity the Langfuse way. These map to user.id, session.id, and tags in the Inference platform.
client.updateCurrentTrace({
userId: "user-123",
sessionId: "session-abc",
tags: ["prod"],
});
client.updateCurrentGeneration({
model: "gpt-4o-mini",
usageDetails: { input: 12, output: 8, total: 20 },
});
return `Answer: ${question}`;
},
{ name: "answer-question", asType: "generation" },
);
console.log(await answerQuestion("What is Inference platform?"));
await getClient().flushAsync();
```
Langfuse's OpenTelemetry-based SDKs are also supported with the same host, public key, and secret key settings above. If you are not using Langfuse and are configuring a generic OpenTelemetry exporter, use Inference platform's standard OTLP endpoint instead.
## Send to Both Langfuse and Inference platform [#send-to-both-langfuse-and-inference-platform]
The setup above points the Langfuse SDK entirely at the Inference platform, so all of that
SDK's traffic comes to us. If you want to keep your existing Langfuse pipeline
and mirror the same traces to the Inference platform, you can fan out to both at once. This
is the lowest-risk way to evaluate Inference platform: Langfuse keeps working untouched
while Inference platform receives a copy of everything.
### TypeScript: add a second span processor [#typescript-add-a-second-span-processor]
The Langfuse JS/TS SDK is OpenTelemetry-based, and OpenTelemetry broadcasts
every span to every registered span processor by default. Keep your existing
`LangfuseSpanProcessor` pointed at Langfuse and add a second one pointed at
Inference platform. Your `observe` calls and the rest of your tracing code do not change.
<Metadata text="integrations/traces/langfuse-dual-typescript" />
```typescript instrumentation.ts
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";
const sdk = new NodeSDK({
spanProcessors: [
// Your existing Langfuse destination.
new LangfuseSpanProcessor({
baseUrl: "https://cloud.langfuse.com",
publicKey: process.env.LANGFUSE_PUBLIC_KEY,
secretKey: process.env.LANGFUSE_SECRET_KEY,
}),
// Inference platform destination. The secret key is your Inference API key. The public
// key is required by the SDK but ignored by the Inference platform, so any non-empty
// value works.
new LangfuseSpanProcessor({
baseUrl: "https://telemetry.inference.net",
publicKey: "pk-inference",
secretKey: "<your-inference-api-key>",
}),
],
});
sdk.start();
```
Every span now streams to both backends. Each processor batches and flushes
independently, so a delay or outage on one does not block the other.
### Python: fan out with an OpenTelemetry Collector [#python-fan-out-with-an-opentelemetry-collector]
The Python SDK does not broadcast across destinations. Its multi-client mode
routes each span to a single project based on a public-key attribute, so it is
not a true dual-send: you have to thread `langfuse_public_key` through every
`@observe`, OpenAI wrapper, and Langchain handler call to pick the destination,
and spans without that attribute land everywhere by accident. It is fragile and
not the documented path for sending to both.
The clean, SDK-agnostic way to dual-send from Python (or any runtime) is an
OpenTelemetry Collector with two OTLP exporters. Both Langfuse and Inference platform
accept OTLP, so point your app at a local collector and let the collector ship
to both. Your application code does not change.
<Metadata text="integrations/traces/langfuse-dual-collector" />
```yaml otel-collector.yaml
receivers:
otlp:
protocols:
http:
grpc:
exporters:
# base64-encode "<langfuse-public-key>:<langfuse-secret-key>" for the header.
otlphttp/langfuse:
endpoint: https://cloud.langfuse.com/api/public/otel
encoding: json
headers:
Authorization: "Basic <base64 of langfuse-public-key:langfuse-secret-key>"
# base64-encode "pk-inference:<your-inference-api-key>" for the header.
# "encoding: json" is required — the ingest endpoints reject OTLP/protobuf.
otlphttp/inference:
endpoint: https://telemetry.inference.net/api/public/otel
encoding: json
headers:
Authorization: "Basic <base64 of pk-inference:your-inference-api-key>"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp/langfuse, otlphttp/inference]
```
Point your Langfuse SDK (or any OTLP exporter) at the collector's OTLP endpoint
and every trace is mirrored to both backends. This is the same fan-out pattern
Inference platform uses internally to ship telemetry to multiple sinks.
<Note>
The collector approach works for any OTLP trace source, not just Langfuse. If
you already export OpenTelemetry traces, add a second OTLP exporter for
Inference platform and you are done.
</Note>
## What Maps Into the Inference platform [#what-maps-into-the-inference-platform]
| Langfuse field | Inference field |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Trace ID | Stable Inference trace ID derived from the Langfuse trace ID |
| Observation ID | Stable Inference span ID derived from the Langfuse observation ID |
| Trace name | `langfuse.trace.name` (shown in the Trajectories view; does **not** create an agent on its own — see [Group by agent](#group-by-agent)) |
| User and session | `user.id`, `session.id` |
| Agent name / id | `agent.name`, `agent.id` — only when you set them explicitly (see [Group by agent](#group-by-agent)) |
| Generation model | `llm.model_name`, `llm.provider` when inferable |
| Usage and cost | Prompt/completion/total token columns and cost columns |
| Input and output | Span input/output and LLM message columns |
| Metadata and tags | Preserved under `langfuse.*` attributes; tags also under `source.tags` |
| Errors and warnings | Span status and status message |
<Note>
This is v1 of the Langfuse integration — scores, evaluations, and richer
mapping are coming later. Langfuse scores and evaluations are not supported at
this time. Pointing
`LANGFUSE_HOST` at the Inference platform routes all of that SDK's traffic to us. To keep
sending to Langfuse as well, see [Send to both Langfuse and
Inference platform](#send-to-both-langfuse-and-inference-platform). Scores emitted through the
SDK are not ingested at this point in time.
</Note>
## Group by agent [#group-by-agent]
Pointing the base URL at the Inference platform sends your spans — but Inference platform does **not**
treat every trace as an agent. A trace name, or even a Langfuse `agent`-type
observation, is not enough on its own; otherwise every traced workload would
flood the Agents dashboard. Agent grouping is **opt-in**: set `agent.name` (and
optionally a stable `agent.id`) on the **top-level span of your agent**, and
Inference platform groups that whole trace — and every child span — under it in the
**Agents** dashboard. This is also what makes the trace available to
[Halo](/get-started/analyze-traces), which operates on agents.
The Langfuse SDKs are OpenTelemetry-based and forward raw OTel attributes
untouched, so set the two attributes directly on the active span. Do this
**only on the top-level agent span** — children inherit grouping through the
trace and should not carry their own agent identity.
* `agent.name` — the human-readable label the Agents dashboard groups on.
* `agent.id` — a stable identifier that survives display-name changes. When
set, it wins over `agent.name`, so renaming the agent doesn't split its
history. Omit it if you don't need stable grouping.
<Metadata text="integrations/traces/langfuse-agent-typescript" />
```typescript TypeScript
import { observe } from "@langfuse/tracing";
import { trace } from "@opentelemetry/api";
const runAgent = observe(
async (question: string) => {
// Mark this top-level span as an agent. Set ONLY here, not on children.
trace.getActiveSpan()?.setAttributes({
"agent.name": "deep-research",
"agent.id": "deep-research-v1", // stable id; optional
});
// ... your agent loop, tool calls, and nested generations ...
return "answer";
},
{ name: "deep-research", asType: "agent" },
);
```
<Metadata text="integrations/traces/langfuse-agent-python" />
```python Python
from langfuse import observe
from opentelemetry import trace
@observe(as_type="agent", name="deep-research")
def run_agent(question: str) -> str:
# Mark this top-level span as an agent. Set ONLY here, not on children.
span = trace.get_current_span()
span.set_attribute("agent.name", "deep-research")
span.set_attribute("agent.id", "deep-research-v1") # stable id; optional
# ... your agent loop, tool calls, and nested generations ...
return "answer"
```
## View and Analyze [#view-and-analyze]
Once traces arrive, open the Inference.net dashboard and use the Trajectories tab to
inspect individual trace trees. Traces whose top-level span carries an
`agent.name` / `agent.id` (see [Group by agent](#group-by-agent)) are grouped
in the Agents dashboard, where Halo can run analysis over them.
For the best Halo reports, keep `agent.id` (or `agent.name`) stable per agent or
workflow and set `userId` / `sessionId` when available.
## Want More Control? [#want-more-control]
The Langfuse drop-in is the easy on-ramp: no code changes, just env vars. If you
want finer control over how traces are grouped, such as stable agent IDs,
explicit agent and manual spans, and a fully shaped trace tree, switch to the
first-party Inference Tracing SDK. Start with the
[Quickstart](/integrations/traces/quickstart), then see
[Agent identity](/integrations/traces/agent-identity) for grouping executions in
the Agents dashboard.
# LangGraph Traces (https://docs.inference.net/integrations/traces/langgraph)
Inference platform uses the LangChain callback path for LangGraph workflows. A compiled
graph emits a graph-level CHAIN span and node-level CHAIN spans parented under
the graph run.
If the graph is the implementation of a product agent, wrap `graph.invoke()`
with `agentSpan()` / `agent_span()` and set a stable `agent.id` so executions
group in the Agents dashboard.
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/langgraph-install-typescript" />
```bash TypeScript
bun add @inference/tracing @langchain/core @langchain/langgraph
```
<Metadata text="integrations/traces/langgraph-install-python" />
```bash Python
pip install 'inference-tracing[langgraph]'
```
</CodeGroup>
## TypeScript State Graph [#typescript-state-graph]
<Metadata text="integrations/traces/langgraph-state-ts" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { Annotation, END, START, StateGraph } from "@langchain/langgraph";
const tracing = await setup({
serviceName: "counter-graph",
modules: { langchainCallbacksManager: CallbackManagerModule },
});
const State = Annotation.Root({
count: Annotation<number>({
reducer: (_left, right) => right,
default: () => 0,
}),
});
const graph = new StateGraph(State)
.addNode("increment", (state) => ({ count: state.count + 1 }))
.addNode("double", (state) => ({ count: state.count * 2 }))
.addEdge(START, "increment")
.addEdge("increment", "double")
.addEdge("double", END)
.compile();
const result = await agentSpan(
{
agentId: "counter-graph-agent",
agentName: "Counter Graph Agent",
spanName: "counter-graph.run",
sessionId: "conversation-counter-1",
role: "workflow",
system: "langgraph",
},
async (span) => {
const input = { count: 1 };
span.setInput(input);
const output = await graph.invoke(input);
span.setOutput(output);
return output;
},
);
console.log(result);
await tracing.shutdown();
```
## Python State Graph [#python-state-graph]
<Metadata text="integrations/traces/langgraph-state-python" />
```python Python
from inference_tracing import agent_span, setup
from langgraph.graph import END, START, StateGraph
from typing_extensions import TypedDict
class GraphState(TypedDict):
count: int
tracing = setup(service_name="counter-graph")
def increment(state: GraphState) -> GraphState:
return {"count": state["count"] + 1}
def double(state: GraphState) -> GraphState:
return {"count": state["count"] * 2}
builder = StateGraph(GraphState)
builder.add_node("increment", increment)
builder.add_node("double", double)
builder.add_edge(START, "increment")
builder.add_edge("increment", "double")
builder.add_edge("double", END)
graph = builder.compile()
with agent_span(
tracing.tracer,
agent_id="counter-graph-agent",
agent_name="Counter Graph Agent",
span_name="counter-graph.run",
session_id="conversation-counter-1",
agent_role="workflow",
system="langgraph",
) as span:
input_state = {"count": 1}
span.set_input(input_state)
result = graph.invoke(input_state)
span.set_output(result)
print(result)
tracing.shutdown()
```
## What To Look For [#what-to-look-for]
* A graph span named `LangGraph`
* Node spans named after graph nodes
* `openinference.span.kind=CHAIN` on graph and node spans
* Node spans parented under the graph span
* An outer AGENT span with your stable `agent.id` when you wrap agent-style
graph invocations
For agent-style graphs, model and tool spans created inside node functions
appear below the active graph or node context.
# LangSmith Traces (https://docs.inference.net/integrations/traces/langsmith)
If your application already uses LangSmith tracing, Inference platform can bridge
LangSmith OpenTelemetry spans to the Inference platform tracer provider. This keeps
LangSmith `traceable` functions and framework integrations visible with the rest
of your application traces.
If a LangSmith-traced function is the boundary of an agent run, add an outer
`agentSpan()` / `agent_span()` with a stable `agent.id`. LangSmith spans remain
visible under that AGENT span.
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/langsmith-install-typescript" />
```bash TypeScript
bun add @inference/tracing langsmith
```
<Metadata text="integrations/traces/langsmith-install-python" />
```bash Python
pip install 'inference-tracing[langsmith]'
```
</CodeGroup>
## TypeScript Traceable Function [#typescript-traceable-function]
<Metadata text="integrations/traces/langsmith-traceable-ts" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import { Client } from "langsmith";
import { traceable } from "langsmith/traceable";
process.env.LANGSMITH_TRACING = "true";
process.env.LANGSMITH_TRACING_MODE = "otel";
const tracing = await setup({ serviceName: "langsmith-worker" });
const client = new Client({ tracingMode: "otel" });
const answerQuestion = traceable(
async (input: { question: string }) => ({
answer: input.question.toUpperCase(),
}),
{
name: "answerQuestion",
run_type: "tool",
client,
tracer: tracing.provider.getTracer("langsmith-app"),
},
);
console.log(await answerQuestion({ question: "hi" }));
await client.flush();
await tracing.shutdown();
```
## Python Traceable Function [#python-traceable-function]
<Metadata text="integrations/traces/langsmith-traceable-python" />
```python Python
import os
from inference_tracing import setup
from langsmith import Client, traceable
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_TRACING_MODE"] = "otel"
tracing = setup(service_name="langsmith-worker")
client = Client()
@traceable(name="answer_question", run_type="tool", client=client, enabled=True)
def answer_question(question: str) -> str:
return question.upper()
print(answer_question("hi"))
client.flush()
tracing.shutdown()
```
## Hybrid Mode [#hybrid-mode]
If `LANGSMITH_TRACING=true` is set and `LANGSMITH_TRACING_MODE` is not set,
Inference Tracing defaults LangSmith to `hybrid` mode. That keeps existing LangSmith
tracing active while also routing OpenTelemetry spans through the Inference platform.
<Metadata text="integrations/traces/langsmith-hybrid-mode" />
```bash
export LANGSMITH_TRACING=true
# Inference Tracing sets LANGSMITH_TRACING_MODE=hybrid when unset.
```
## Stable Agent Identity [#stable-agent-identity]
Use this pattern around LangSmith-traced work when you want the Agents dashboard
to group executions by your product agent ID.
<CodeGroup>
<Metadata text="integrations/traces/langsmith-agent-identity-ts" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "question-answer-agent",
agentName: "Question Answer Agent",
spanName: "question-answer.run",
sessionId: "conversation-qa-1",
role: "qa",
system: "langsmith",
},
async (span) => {
const input = { question: "hi" };
span.setInput(input);
const output = await answerQuestion(input);
span.setOutput(output);
},
);
```
<Metadata text="integrations/traces/langsmith-agent-identity-python" />
```python Python
from inference_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="question-answer-agent",
agent_name="Question Answer Agent",
span_name="question-answer.run",
session_id="conversation-qa-1",
agent_role="qa",
system="langsmith",
) as span:
question = "hi"
span.set_input(question)
answer = answer_question(question)
span.set_output(answer)
```
</CodeGroup>
# LiveKit Agents Traces (https://docs.inference.net/integrations/traces/livekit-agents)
Inference platform traces LiveKit Agents by connecting LiveKit's native OpenTelemetry
tracer to the Inference Tracing provider. Your LiveKit worker keeps using `AgentSession`,
rooms, tools, and model plugins normally; Inference platform enriches the spans LiveKit
already emits with OpenInference attributes before export.
Use this guide for applications built with `@livekit/agents` in TypeScript or
`livekit-agents` in Python.
## What Is Captured [#what-is-captured]
* `agent_session` spans as OpenInference AGENT spans
* `llm_node`, `llm_request`, and `llm_request_run` spans as LLM spans
* LiveKit function tools as TOOL spans, including tool name, call ID,
arguments, output, and errors when available
* LiveKit `lk.*` and `gen_ai.*` attributes, mapped into model, provider, token,
input, and output attributes when LiveKit provides them
* Other LiveKit telemetry spans as CHAIN spans so the full session timeline
remains visible
Inference Tracing sets `agent.name` on LiveKit `agent_session` spans from
`lk.agent_name` or `lk.agent_label` when LiveKit provides it. Use stable
LiveKit agent class names or labels, and add a manual `agent_span` around your
own worker/session boundary when your product needs a canonical `agent.id` for
dashboard grouping.
<Info>
Inference platform does not monkey patch `AgentSession` methods. The integration uses
LiveKit's public telemetry provider hooks, so LiveKit helper objects and
streaming behavior are preserved.
</Info>
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/livekit-agents-install-typescript" />
```bash TypeScript
bun add @inference/tracing @livekit/agents
```
<Metadata text="integrations/traces/livekit-agents-install-python" />
```bash Python
pip install 'inference-tracing[livekit-agents]'
```
</CodeGroup>
## Configure Export [#configure-export]
Set a stable service name so LiveKit sessions are easy to find in the Inference platform.
<Metadata text="integrations/traces/livekit-agents-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="livekit-agent"
export INFERENCE_SERVICE_VERSION="2026.05.06"
```
## Initialize Tracing [#initialize-tracing]
Initialize tracing before starting LiveKit agent sessions or workers.
<CodeGroup>
<Metadata text="integrations/traces/livekit-agents-setup-typescript" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import * as LiveKitAgents from "@livekit/agents";
const tracing = await setup({
serviceName: "livekit-agent",
modules: { livekitAgents: LiveKitAgents },
});
// Start your LiveKit worker or AgentSession normally.
// LiveKit telemetry now exports through the Inference platform.
await tracing.shutdown();
```
<Metadata text="integrations/traces/livekit-agents-setup-python" />
```python Python
from inference_tracing import setup
tracing = setup(service_name="livekit-agent")
# Start your LiveKit worker or AgentSession normally.
# LiveKit telemetry now exports through the Inference platform.
tracing.shutdown()
```
</CodeGroup>
## Text-Only Agent Example [#text-only-agent-example]
This Python example starts a LiveKit text-only `AgentSession`. It works for smoke
tests and CI because it does not require audio input or output, but it still runs
the real LiveKit session path.
<Metadata text="integrations/traces/livekit-agents-python-text-session" />
```python Python
import os
from dotenv import load_dotenv
from inference_tracing import setup
from livekit import agents
from livekit.agents import Agent, AgentSession, room_io
from livekit.plugins import openai
load_dotenv()
async def entrypoint(ctx: agents.JobContext):
await ctx.connect()
tracing = setup(service_name="livekit-support-agent")
session = AgentSession(llm=openai.LLM(model=os.getenv("OPENAI_MODEL", "gpt-4o-mini")))
await session.start(
agent=Agent(
instructions="You are a concise support agent. Use tools when needed."
),
room=ctx.room,
room_options=room_io.RoomOptions(
text_input=True,
text_output=True,
audio_input=False,
audio_output=False,
),
record=False,
)
await session.generate_reply(
user_input="Reply with one sentence confirming you are online.",
input_modality="text",
).wait_for_playout()
await session.aclose()
tracing.shutdown()
if __name__ == "__main__":
agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))
```
## Function Tools [#function-tools]
LiveKit tool spans are captured automatically when the LiveKit runtime emits
`function_tool` telemetry. The span includes the tool name, tool call ID, JSON
arguments, and output when available.
<Metadata text="integrations/traces/livekit-agents-python-tool" />
```python Python
from livekit.agents import Agent, function_tool
class FrontDeskAgent(Agent):
def __init__(self) -> None:
super().__init__(
instructions="Help users find appointment information.",
)
@function_tool
async def lookup_appointment(self, user_id: str) -> str:
return f"{user_id} has an appointment at 10:30 AM."
```
When `lookup_appointment` runs inside an `AgentSession`, Inference platform shows a TOOL
span under the LiveKit session trace.
## Stable Agent Identity [#stable-agent-identity]
If your application owns a stable agent ID outside LiveKit's runtime telemetry,
wrap the session work with `agent_span()` and start the LiveKit session inside
that context.
<Metadata text="integrations/traces/livekit-agents-agent-identity-python" />
```python Python
from inference_tracing import agent_span
async def start_traced_session(ctx, session) -> None:
with agent_span(
tracing.tracer,
agent_id="front-desk-agent",
agent_name="Front Desk Agent",
span_name="front-desk-agent.run",
session_id="conversation-front-desk-1",
agent_role="front-desk",
system="livekit",
) as span:
span.set_input({"room": ctx.room.name})
await session.start(agent=FrontDeskAgent(), room=ctx.room)
span.set_output({"status": "session-started"})
```
## Verify in the Inference platform [#verify-in-the-inference-platform]
Open the Inference platform and filter traces by your `INFERENCE_SERVICE_NAME`, for example
`livekit-agent` or `livekit-support-agent`.
A successful LiveKit trace should include:
* An `agent_session` AGENT span for the session
* One or more LLM spans for LiveKit model nodes or requests
* TOOL spans for any function tools the agent called
* Your stable `agent.id` on the outer AGENT span when you add the identity
wrapper above
For short-lived scripts, always call `tracing.shutdown()` before process exit so
batched spans are flushed.
## Troubleshooting [#troubleshooting]
If you do not see LiveKit spans:
* Initialize tracing before starting the worker or `AgentSession`.
* Set a stable `INFERENCE_SERVICE_NAME` and filter by that value.
* Confirm `INFERENCE_OTLP_ENDPOINT` and `INFERENCE_API_KEY` are present in the
process environment.
* Call `tracing.shutdown()` before a script exits.
* For Python OpenAI-backed LiveKit agents, use current Inference Tracing packages
so OpenAI streaming context managers are preserved.
# Manual Spans (https://docs.inference.net/integrations/traces/manual-spans)
Inference platform's per-SDK instrumentation captures provider calls automatically: every
SDK you pass to `setup({ modules })` emits an LLM span on its own. But a real
agent does work the SDK never sees, like running tools, retrieving documents,
routing requests, validating output, or shelling out to a CLI. Manual spans are
how you put that work into the same trace tree, next to the auto-captured LLM
spans.
There are two shapes you author by hand, and everything on this page is one of
them:
* **The agent span** wraps a whole run. It is the parent row everything else
nests under, and it carries the agent identity and session. You author it with
`agentSpan()` / `agent_span()`. This is the same wrapper the integration
guides put around a run.
* **Child spans** sit inside that run, one per non-LLM step: a tool call, a
retrieval, a routing step. You author them with `manualSpan()` /
`manual_span()`, choosing the span kind that matches the step.
If you have not yet wired up `setup()`, start with the
[Traces Quickstart](/integrations/traces/quickstart).
## Span Kinds You Author Manually [#span-kinds-you-author-manually]
| Kind | Use it for |
| ----------- | ---------------------------------------------------------------------------------------------------------- |
| `AGENT` | The outer span around an agent run. Carries `agent.id`, `agent.name`, optional `session.id` and `user.id`. |
| `TOOL` | One invocation of a tool function by an agent. Carries `tool.name`, optional `tool_call.id`. |
| `CHAIN` | A chain step that is not itself an LLM call (routing logic, prompt assembly, post-processing). |
| `RETRIEVER` | A vector-search or document-lookup call. |
| `EMBEDDING` | An embedding-model call your code makes directly. |
| `LLM` | A model call made outside an instrumented SDK (rare, prefer the SDK patches). |
These six are the whole set. Under the hood a span is an ordinary
OpenTelemetry span, so you can attach any attribute you want to it (see
[Adding Custom Attributes](#adding-custom-attributes)). The *kind* is different:
it is a fixed vocabulary from
[OpenInference](https://github.com/Arize-ai/openinference) that tells the
dashboard how to render the span. A `TOOL` span gets a tool name and an
arguments/result panel, a `RETRIEVER` span gets a query and a document list, an
`LLM` span gets messages, token counts, and cost. Pick the kind that matches how
you want the step to appear; when none of the specific kinds fit, use `CHAIN`,
the generic "this is a step" kind. The kind strings come from `SpanKindValues`,
and the [Attributes reference](/integrations/traces/attributes) lists which
attributes each kind expects.
## Agent Spans [#agent-spans]
Use `agentSpan()` (TypeScript) or `agent_span()` (Python) as the outermost
wrapper around an agent run. Pass a stable `agentId` so the Inference platform Agents
dashboard groups executions correctly across renames and deploys. Pass
`sessionId` for one conversation, not as process-wide setup. Pass `userId` to
record the end user the run acts on behalf of so you can filter traces by user
on the dashboard. The wrapper also accepts free-form `metadata` and `tags`.
<CodeGroup>
<Metadata text="integrations/traces/manual-agent-span" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-ticket-123",
userId: "user_8675309",
role: "refunds",
system: "openai",
},
async (span) => {
const ticket = { id: "ticket_123", orderId: "ABC-123" };
span.setInput(ticket);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "user", content: `Review refund for ${ticket.orderId}` },
],
});
span.setOutput({ decision: response.choices[0]?.message.content });
if (response.usage != null) {
span.recordTokens({
prompt: response.usage.prompt_tokens ?? 0,
completion: response.usage.completion_tokens ?? 0,
});
}
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/manual-agent-span" />
```python Python
import os
from inference_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-ticket-123",
user_id="user_8675309",
agent_role="refunds",
system="openai",
) as span:
ticket = {"id": "ticket_123", "order_id": "ABC-123"}
span.set_input(ticket)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": f"Review refund for {ticket['order_id']}"},
],
)
span.set_output({"decision": response.choices[0].message.content})
if response.usage is not None:
span.record_usage(response.usage)
tracing.shutdown()
```
</CodeGroup>
The OpenAI call inside the callback runs in the agent span's active OTel context,
so the auto-emitted LLM span automatically parents under the AGENT span. That
is how a full trace tree forms with no extra plumbing.
See the [Handle API reference](/integrations/traces/handle-api) for the full
set of methods on the `span` argument.
## Tool, Chain, And Retriever Spans [#tool-chain-and-retriever-spans]
When the work inside an agent loop is a tool call, a retrieval step, or a chain
step, wrap it in a child span with the right `SpanKind`. The child span
automatically parents under the active AGENT span and inherits its `agent.id`.
`manualSpan()` / `manual_span()` is the general-purpose helper. Pass
`spanKind` / `span_kind`, optional typed fields (`toolName` / `tool_name`,
`toolCallId` / `tool_call_id`, `model`, `usage`, `input`, `output`), and any
extra `attributes`. It defaults to `CHAIN`; pass `TOOL`, `RETRIEVER`, or
`EMBEDDING` when authoring the matching shape.
### Tool spans [#tool-spans]
<CodeGroup>
<Metadata text="integrations/traces/manual-tool-span-typescript" />
```typescript TypeScript
import { manualSpan, SpanKindValues } from "@inference/tracing";
async function executeTool(
name: string,
args: Record<string, unknown>,
callId: string,
) {
return await manualSpan(
{
spanName: `${name}.tool`,
spanKind: SpanKindValues.TOOL,
toolName: name,
toolCallId: callId,
input: args,
},
async (span) => {
const result = await TOOLS[name](args);
span.setOutput(result);
return result;
},
);
}
```
<Metadata text="integrations/traces/manual-tool-span-python" />
```python Python
from inference_tracing import SpanKindValues, manual_span
def execute_tool(name: str, args: dict, call_id: str) -> dict:
with manual_span(
tracing.tracer,
name=f"{name}.tool",
span_kind=SpanKindValues.TOOL,
tool_name=name,
tool_call_id=call_id,
input=args,
) as span:
result = TOOLS[name](**args)
span.set_output(result)
return result
```
</CodeGroup>
<Note>
`input` / `output` do not set the MIME type. If you want viewers to render
them as JSON, set `Attr.INPUT_MIME_TYPE` / `Attr.OUTPUT_MIME_TYPE`
explicitly.
</Note>
### Chain spans [#chain-spans]
Use `CHAIN` for routing logic, prompt assembly, post-processing, or any other
non-LLM step that benefits from its own span. `metadata` and `tags` are
free-form attributes that show up under the span's attributes view in the
dashboard.
<CodeGroup>
<Metadata text="integrations/traces/manual-chain-span-typescript" />
```typescript TypeScript
import { manualSpan, SpanKindValues } from "@inference/tracing";
await manualSpan(
{
spanName: "provider_router/select_model",
spanKind: SpanKindValues.CHAIN,
input: { route: "support", priority: "high" },
metadata: { customer_tier: "enterprise" },
tags: ["router", "support"],
},
async (span) => {
const selected = chooseModelForRequest();
span.setOutput({ model: selected.model, reason: selected.reason });
},
);
```
<Metadata text="integrations/traces/manual-chain-span-python" />
```python Python
from inference_tracing import SpanKindValues, manual_span
with manual_span(
tracing.tracer,
name="provider_router/select_model",
span_kind=SpanKindValues.CHAIN,
input={"route": "support", "priority": "high"},
metadata={"customer_tier": "enterprise"},
tags=["router", "support"],
) as span:
selected = choose_model_for_request()
span.set_output({"model": selected.model, "reason": selected.reason})
```
</CodeGroup>
### Retriever spans [#retriever-spans]
Use `RETRIEVER` for vector-store or document-lookup calls. Record the query as
input and the result list as output.
<CodeGroup>
<Metadata text="integrations/traces/manual-retriever-span-typescript" />
```typescript TypeScript
import { manualSpan, SpanKindValues } from "@inference/tracing";
await manualSpan(
{
spanName: "vector_store.search",
spanKind: SpanKindValues.RETRIEVER,
input: { query, k: 8 },
},
async (span) => {
const docs = await vectorStore.search(query, { k: 8 });
span.setOutput(docs.map((d) => ({ id: d.id, score: d.score })));
},
);
```
<Metadata text="integrations/traces/manual-retriever-span-python" />
```python Python
from inference_tracing import SpanKindValues, manual_span
with manual_span(
tracing.tracer,
name="vector_store.search",
span_kind=SpanKindValues.RETRIEVER,
input={"query": query, "k": 8},
) as span:
docs = vector_store.search(query, k=8)
span.set_output([{"id": d.id, "score": d.score} for d in docs])
```
</CodeGroup>
### Escape Hatch [#escape-hatch]
`manualSpan` is the canonical path for every non-AGENT manual span. The only
reason to reach below it is `tracing.tracer.startActiveSpan()` directly, when
you need behavior the helper doesn't expose, like mid-callback span events paired
with custom status transitions, or a span lifetime that doesn't match a single
callback. With raw `startActiveSpan` you own status, exception recording, and
`span.end()` yourself.
For a full example that ties an AGENT span, several TOOL spans, and the
patched provider LLM spans together, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## Adding Custom Attributes [#adding-custom-attributes]
The typed handle covers the OpenInference vocabulary. For domain-specific
attributes (tenant ID, request channel, deploy environment), use
`setAttribute` / `set_attribute` on the handle. The helper normalizes values
into OTel-safe types (primitives and primitive arrays pass through; mixed
arrays and plain objects are JSON-stringified).
<CodeGroup>
<Metadata text="integrations/traces/manual-custom-attributes-ts" />
```typescript TypeScript
await agentSpan(
{ agentId: "support-agent", spanName: "support-agent.run" },
async (span) => {
span.setAttribute("app.tenant_id", tenantId);
span.setAttributes({
"app.request_channel": "slack",
"app.deploy_env": process.env.DEPLOY_ENV ?? "dev",
});
// ...
},
);
```
<Metadata text="integrations/traces/manual-custom-attributes-python" />
```python Python
with agent_span(
tracing.tracer,
agent_id="support-agent",
span_name="support-agent.run",
) as span:
span.set_attribute("app.tenant_id", tenant_id)
span.set_attributes({
"app.request_channel": "slack",
"app.deploy_env": os.environ.get("DEPLOY_ENV", "dev"),
})
# ...
```
</CodeGroup>
For anything outside what the handle provides (span events, mid-callback
exception recording, span-context propagation to background jobs), the raw
OTel span is available as `span.raw` (TypeScript) or `span.span` (Python).
Custom attributes appear under the span's attributes view in the dashboard and
in `inf span get <trace-id> <span-id> --view attributes`. They are also
filterable from the CLI with `--metadata "app.tenant_id=acme"`.
## How Identity And Context Propagate [#how-identity-and-context-propagate]
The agent span runs its callback inside an active OTel context that carries:
* The OTel `Span` itself, so child spans created by `tracer.startActiveSpan` or
by patched provider SDKs auto-parent under it.
* A Inference-specific agent identity record (`agent.id`, `agent.name`,
`agent.role`), which the per-SDK patchers copy onto LLM child spans so the
Agents dashboard groups model calls under the right agent.
You do not need to pass `agentId` to every child span. If you create a tool
span inside an agent span, the agent identity flows through automatically.
```
AGENT support-agent.run agent.id=support-agent, session.id=conv-42
├── LLM OpenAI chat (auto-emitted, inherits agent.id)
├── TOOL lookup_order.tool tool.name=lookup_order
│ └── LLM OpenAI chat (nested call from inside the tool)
├── CHAIN prompt_builder.run
└── TOOL issue_refund.tool tool.name=issue_refund
```
If you need to inspect or copy the active identity (for example, to tag a
background job span with the same `agent.id`), import the helpers:
<Metadata text="integrations/traces/manual-active-identity" />
```typescript TypeScript
import { getActiveAgentIdentity } from "@inference/tracing";
const identity = getActiveAgentIdentity();
if (identity?.id) {
jobQueue.enqueue({ agentId: identity.id, payload });
}
```
## CLI And Subprocess Wrappers [#cli-and-subprocess-wrappers]
When the work is a shell-out to a CLI (Claude Code, Codex, a custom binary),
wrap the process call in an agent span and set the input and output yourself.
<CodeGroup>
<Metadata text="integrations/traces/manual-subprocess" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
const execFileP = promisify(execFile);
const tracing = await setup();
const prompt = "Reply with just the word hello.";
await agentSpan(
{
agentId: "claude-code-prod",
agentName: "Claude Code",
spanName: "claude-code.invocation",
sessionId: "conversation-cli-hello",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const { stdout } = await execFileP("claude", ["-p", prompt], {
encoding: "utf-8",
timeout: 60_000,
});
span.setOutput(stdout.trim());
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/manual-subprocess" />
```python Python
import subprocess
from inference_tracing import agent_span, setup
tracing = setup()
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="claude-code-prod",
agent_name="Claude Code",
span_name="claude-code.invocation",
session_id="conversation-cli-hello",
system="anthropic",
) as span:
span.set_input(prompt)
completed = subprocess.run(
["claude", "-p", prompt],
capture_output=True,
text=True,
timeout=60,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
</CodeGroup>
## Status Semantics [#status-semantics]
All Inference platform manual-span helpers follow the same status rules:
* Callback returns normally → span ends with `OK`.
* Callback throws → the exception is recorded as a span event, the span ends
with `ERROR` and the error message, and the original exception re-throws so
caller error handling is unaffected.
If you reach for `tracer.startActiveSpan()` directly (the OTel escape hatch
mentioned above), you own status and `span.end()` yourself, which is the
price of the lower-level API.
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
Every method on the AGENT / TOOL / CHAIN span handle, with value-coercion rules.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
All `Attr.*` constants and `SpanKindValues` with the attributes each kind expects.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool execution, end to end.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Pick stable `agent.id` and `session.id` values for the Agents dashboard.
</Card>
</CardGroup>
# OpenAI Agents Traces (https://docs.inference.net/integrations/traces/openai-agents)
Pair OpenAI Agents instrumentation with OpenAI instrumentation so nested model
calls are captured. Once `setup()` runs (pass both modules in TypeScript;
Python auto-detects installed SDKs), the agent run, its tool calls, handoffs,
and nested model calls are all captured automatically. Wrap the run in
`agentSpan()` / `agent_span()` when you want stable identity and session
grouping.
<Info>
Three layers, in increasing order of effort:
* **`setup()`** traces the whole run automatically (pass the SDK modules in TypeScript; Python auto-detects). Enough on its own.
* **`agentSpan()` / `agent_span()`** adds a stable `agentId`, `agentName`, `role`, and `sessionId` so the Agents dashboard groups runs by conversation, plus your high-level input and output.
* **`manualSpan()` / `manual_span()`** captures steps the SDK never sees: your own retrieval, routing, or sub-steps inside a tool.
For the full breakdown of when to reach for each, see [How Tracing Fits Together](/integrations/traces/overview#how-tracing-fits-together).
</Info>
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/openai-agents-install-typescript" />
```bash TypeScript
bun add @inference/tracing openai @openai/agents zod
```
<Metadata text="integrations/traces/openai-agents-install-python" />
```bash Python
pip install 'inference-tracing[openai-agents]'
```
</CodeGroup>
## Configure Export [#configure-export]
Set the Inference Tracing endpoint and token before your app starts. `setup()` reads
these in every example below. Generate a token at
[API Keys](https://inference.net/dashboard/api-keys).
<Metadata text="integrations/traces/openai-agents-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="support-agent"
```
## Minimal Setup [#minimal-setup]
Pass the SDK modules to `setup()` in TypeScript, or let it auto-detect in
Python, then run your agent. The run, any tool calls, and the nested model calls
are all captured with no further code. This is everything you need for a single
one-shot agent.
<CodeGroup>
<Metadata text="integrations/traces/openai-agents-minimal-typescript" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run } from "@openai/agents";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Help customers with order questions.",
model: "gpt-4o-mini",
});
// No agentSpan needed: the run and its nested model calls are captured
// automatically.
const result = await run(supportAgent, "Where is order ABC-123?");
console.log(result.finalOutput);
await tracing.shutdown();
```
<Metadata text="integrations/traces/openai-agents-minimal-python" />
```python Python
from inference_tracing import setup
from agents import Agent, Runner
tracing = setup()
support_agent = Agent(
name="SupportAgent",
instructions="Help customers with order questions.",
model="gpt-4o-mini",
)
# No agent_span needed: the run and its nested model calls are captured
# automatically.
result = Runner.run_sync(support_agent, "Where is order ABC-123?")
print(result.final_output)
tracing.shutdown()
```
</CodeGroup>
## Group Under An Agent [#group-under-an-agent]
Wrap the run in `agentSpan()` to give it stable identity and session grouping.
This is the recommended shape for anything beyond a one-shot script. Pass
`userId` to record `user.id` so you can filter traces by user on the dashboard,
just like `sessionId`. For arbitrary keys, attach any custom attributes you want
to filter traces by inside the callback (`organization.id`, `chat.id`,
`order.id`). See
[Adding Custom Attributes](/integrations/traces/manual-spans#adding-custom-attributes).
<CodeGroup>
<Metadata text="integrations/traces/openai-agents-single-typescript" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run, tool } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const lookupOrder = tool({
name: "lookup_order",
description: "Look up an order by ID.",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) =>
JSON.stringify({ orderId, status: "shipped", total: 42.5 }),
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Use tools to help customers with order questions.",
tools: [lookupOrder],
model: "gpt-4o-mini",
});
const userMessage = "Where is order ABC-123?";
await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
userId: "user_8675309",
role: "support",
system: "openai",
},
async (span) => {
span.setInput(userMessage);
// For arbitrary keys, attach whatever your app keys on. Any attribute is
// filterable from the dashboard and CLI, e.g.
// `inf span list --metadata "organization.id=org_42"`.
span.setAttribute("organization.id", "org_42");
const result = await run(supportAgent, userMessage, { maxTurns: 4 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/openai-agents-single-python" />
```python Python
import json
from inference_tracing import agent_span, setup
from agents import Agent, Runner, function_tool
tracing = setup()
@function_tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, "status": "shipped", "total": 42.5})
support_agent = Agent(
name="SupportAgent",
instructions="Use tools to help customers with order questions.",
tools=[lookup_order],
model="gpt-4o-mini",
)
user_message = "Where is order ABC-123?"
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
user_id="user_8675309",
agent_role="support",
system="openai",
) as span:
span.set_input(user_message)
# For arbitrary keys, attach whatever your app keys on. Any attribute is
# filterable from the dashboard and CLI, e.g.
# `inf span list --metadata "organization.id=org_42"`.
span.set_attribute("organization.id", "org_42")
result = Runner.run_sync(support_agent, user_message, max_turns=4)
span.set_output(str(result.final_output))
tracing.shutdown()
```
</CodeGroup>
## Multi-Agent Handoff [#multi-agent-handoff]
Wrap the triage request once. The trace tree groups the triage agent,
specialist agent, nested model calls, and tools under the customer request.
<CodeGroup>
<Metadata text="integrations/traces/openai-agents-handoff-typescript" />
```typescript TypeScript
import { Agent, handoff, run, tool } from "@openai/agents";
const issueRefund = tool({
name: "issue_refund",
description: "Issue a refund for an order.",
parameters: z.object({ orderId: z.string(), amount: z.number() }),
execute: async ({ orderId, amount }) =>
JSON.stringify({ ok: true, orderId, refundId: "RFD-2201", amount }),
});
const refundsAgent = new Agent({
name: "RefundsAgent",
instructions: "Handle refund requests and use issue_refund.",
tools: [issueRefund],
model: "gpt-4o-mini",
});
const billingAgent = new Agent({
name: "BillingAgent",
instructions: "Answer billing questions. Do not issue refunds.",
model: "gpt-4o-mini",
});
const triageAgent = new Agent({
name: "TriageAgent",
instructions: "Route refund requests to RefundsAgent.",
handoffs: [handoff(refundsAgent), handoff(billingAgent)],
model: "gpt-4o-mini",
});
await agentSpan(
{
agentId: "triage-agent",
agentName: "Triage Agent",
spanName: "triage-agent.run",
sessionId: "conversation-refund-abc-123",
role: "triage",
system: "openai",
},
async (span) => {
const input = "I need a refund for order ABC-123, total $42.50.";
span.setInput(input);
const result = await run(triageAgent, input, { maxTurns: 8 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
```
<Metadata text="integrations/traces/openai-agents-handoff-python" />
```python Python
import json
from inference_tracing import agent_span
from agents import Agent, Runner, function_tool, handoff
@function_tool
def issue_refund(order_id: str, amount: float) -> str:
"""Issue a refund for an order."""
return json.dumps(
{"ok": True, "order_id": order_id, "refund_id": "RFD-2201", "amount": amount}
)
refunds_agent = Agent(
name="RefundsAgent",
instructions="Handle refund requests and use issue_refund.",
tools=[issue_refund],
model="gpt-4o-mini",
)
billing_agent = Agent(
name="BillingAgent",
instructions="Answer billing questions. Do not issue refunds.",
model="gpt-4o-mini",
)
triage_agent = Agent(
name="TriageAgent",
instructions="Route refund requests to RefundsAgent.",
handoffs=[handoff(refunds_agent), handoff(billing_agent)],
model="gpt-4o-mini",
)
with agent_span(
tracing.tracer,
agent_id="triage-agent",
agent_name="Triage Agent",
span_name="triage-agent.run",
session_id="conversation-refund-abc-123",
agent_role="triage",
system="openai",
) as span:
user_input = "I need a refund for order ABC-123, total $42.50."
span.set_input(user_input)
result = Runner.run_sync(triage_agent, user_input, max_turns=8)
span.set_output(str(result.final_output))
```
</CodeGroup>
Use one stable ID for the outer customer-facing run. If you also emit separate
AGENT spans for handoff specialists, give each specialist its own stable
`agent.id`, such as `refunds-agent` or `billing-agent`.
## When To Add Manual Spans [#when-to-add-manual-spans]
The Agents SDK only captures the work it runs for you: the model calls, and the
tools you register with `tool()`. Anything you do yourself inside the
`agentSpan` callback is invisible to it. You do not wrap the SDK's own tool
calls in `manualSpan()`, those are already captured, and doing so would just
produce a duplicate span. You reach for `manualSpan()` for the steps the SDK
never sees.
Two common cases:
1. **A step around the run.** You fetch context from a vector store before the
run, or validate and reshape the output after it. Neither is an SDK call, so
author a `RETRIEVER` or `CHAIN` span yourself.
2. **Sub-steps inside a tool.** The SDK emits one TOOL span per tool
invocation. If that tool's `execute` does something you want broken out (a
vector search, a downstream API call, a transform), wrap it in a child span.
It nests under the SDK's TOOL span automatically.
<CodeGroup>
<Metadata text="integrations/traces/openai-agents-manual-typescript" />
```typescript TypeScript
import { manualSpan, SpanKindValues } from "@inference/tracing";
// Case 2: a sub-step inside a registered tool. The SDK already opened a TOOL
// span for this call; the RETRIEVER span below nests inside it.
const lookupOrder = tool({
name: "lookup_order",
description: "Look up an order by ID.",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) => {
const docs = await manualSpan(
{
spanName: "order_docs.search",
spanKind: SpanKindValues.RETRIEVER,
input: { orderId, k: 4 },
},
async (span) => {
const results = await vectorStore.search(orderId, { k: 4 });
span.setOutput(results.map((d) => ({ id: d.id, score: d.score })));
return results;
},
);
return JSON.stringify({ orderId, status: "shipped", context: docs.length });
},
});
await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
role: "support",
system: "openai",
},
async (span) => {
const userMessage = "Where is order ABC-123?";
span.setInput(userMessage);
// Case 1: a pre-run step the SDK never sees. You classify the request
// yourself before handing it to the agent.
const route = await manualSpan(
{
spanName: "router.classify",
spanKind: SpanKindValues.CHAIN,
input: { userMessage },
},
async (chain) => {
const decision = classifyRequest(userMessage);
chain.setOutput(decision);
return decision;
},
);
const result = await run(supportAgent, userMessage, { maxTurns: 4 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
```
<Metadata text="integrations/traces/openai-agents-manual-python" />
```python Python
from inference_tracing import agent_span, manual_span, SpanKindValues
from agents import Runner, function_tool
# Case 2: a sub-step inside a registered tool. The SDK already opened a TOOL
# span for this call; the RETRIEVER span below nests inside it.
@function_tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
with manual_span(
tracing.tracer,
name="order_docs.search",
span_kind=SpanKindValues.RETRIEVER,
input={"order_id": order_id, "k": 4},
) as span:
results = vector_store.search(order_id, k=4)
span.set_output([{"id": d.id, "score": d.score} for d in results])
return json.dumps(
{"order_id": order_id, "status": "shipped", "context": len(results)}
)
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
agent_role="support",
system="openai",
) as span:
user_message = "Where is order ABC-123?"
span.set_input(user_message)
# Case 1: a pre-run step the SDK never sees. You classify the request
# yourself before handing it to the agent.
with manual_span(
tracing.tracer,
name="router.classify",
span_kind=SpanKindValues.CHAIN,
input={"user_message": user_message},
) as chain:
decision = classify_request(user_message)
chain.set_output(decision)
result = Runner.run_sync(support_agent, user_message, max_turns=4)
span.set_output(str(result.final_output))
```
</CodeGroup>
The rule of thumb: if the Agents SDK runs the code (a model call, a registered
tool), it is captured for you. If you run the code (retrieval, routing,
validation, or a sub-step inside a tool), wrap it in `manualSpan`. See
[Manual Spans](/integrations/traces/manual-spans) for the full set of span
kinds and the handle API.
## Flushing [#flushing]
Every example above ends with `await tracing.shutdown()`, which flushes batched
spans before the process exits. Long-lived servers and serverless or edge
runtimes need a different flush strategy, since the process does not exit after
each run. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle)
for all three shapes.
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Author TOOL, CHAIN, and RETRIEVER spans for work the SDK does not capture.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Pick stable `agent.id` and `session.id` values for the Agents dashboard.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool spans, end to end.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
Every `Attr.*` constant and `SpanKindValues` value the SDK emits.
</Card>
</CardGroup>
# OpenAI Traces (https://docs.inference.net/integrations/traces/openai)
Inference platform instruments the OpenAI SDK in both TypeScript and Python. Initialize
tracing before constructing clients so the OpenAI prototypes are patched before
application calls start.
## What Is Captured [#what-is-captured]
* Chat Completions request and response messages
* Responses API input and output text
* Tool call names, IDs, and JSON arguments
* Tool result messages passed back into later turns
* JSON schema response format in invocation parameters
* Model name, finish reason, usage, and token counts
## Install [#install]
<CodeGroup>
<Metadata text="integrations/traces/openai-install-typescript" />
```bash TypeScript
bun add @inference/tracing openai
```
<Metadata text="integrations/traces/openai-install-python" />
```bash Python
pip install 'inference-tracing[openai]'
```
</CodeGroup>
## Basic Chat [#basic-chat]
<CodeGroup>
<Metadata text="integrations/traces/openai-basic-chat" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
serviceName: "checkout-agent",
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You answer in one short sentence." },
{ role: "user", content: "Summarize order ABC-123." },
],
max_tokens: 80,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/openai-basic-chat" />
```python Python
import os
from inference_tracing import setup
from openai import OpenAI
tracing = setup(service_name="checkout-agent")
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You answer in one short sentence."},
{"role": "user", "content": "Summarize order ABC-123."},
],
max_tokens=80,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
## OpenAI Inside An Agent [#openai-inside-an-agent]
<CodeGroup>
<Metadata text="integrations/traces/openai-agent-identity" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "checkout-agent",
agentName: "Checkout Agent",
spanName: "checkout-agent.run",
sessionId: "conversation-checkout-1",
role: "checkout",
system: "openai",
},
async (span) => {
const input = "Summarize order ABC-123.";
span.setInput(input);
const response = await client.responses.create({
model: "gpt-4o-mini",
input,
});
span.setOutput(response.output_text);
},
);
```
<Metadata text="integrations/traces/openai-agent-identity" />
```python Python
from inference_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="checkout-agent",
agent_name="Checkout Agent",
span_name="checkout-agent.run",
session_id="conversation-checkout-1",
agent_role="checkout",
system="openai",
) as span:
user_input = "Summarize order ABC-123."
span.set_input(user_input)
response = client.responses.create(model="gpt-4o-mini", input=user_input)
span.set_output(response.output_text)
```
</CodeGroup>
## Tool Calling [#tool-calling]
The first model call records the assistant tool call. The second model call
records the tool result message and the final answer.
<Tip>
The capture below is the **model-side** view of tool calling — the
request/response the LLM saw. To also capture the **caller-side** view (the
actual function that ran, its input, output, and duration), wrap the tool
function in a `TOOL` span using
[Manual spans](/integrations/traces/manual-spans#tool-chain-and-retriever-spans).
For a full agent loop, see the
[Production Agent Example](/integrations/traces/production-agent-example).
</Tip>
<CodeGroup>
<Metadata text="integrations/traces/openai-tool-calling" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const tools = [
{
type: "function" as const,
function: {
name: "get_weather",
description: "Look up the current weather in a city.",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
];
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: "user", content: "What's the weather in San Francisco?" },
];
const first = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
const toolCalls = first.choices[0]?.message.tool_calls ?? [];
messages.push({ role: "assistant", content: null, tool_calls: toolCalls });
for (const toolCall of toolCalls) {
const args = JSON.parse(toolCall.function.arguments) as { city: string };
messages.push({
role: "tool",
tool_call_id: toolCall.id,
content: JSON.stringify({ city: args.city, tempF: 62 }),
});
}
const final = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
console.log(final.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/openai-tool-calling" />
```python Python
import json
import os
from inference_tracing import setup
from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Look up the current weather in a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
},
]
messages: list[ChatCompletionMessageParam] = [
{"role": "user", "content": "What's the weather in San Francisco?"},
]
first = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
tool_calls = first.choices[0].message.tool_calls or []
messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [tool_call.model_dump() for tool_call in tool_calls],
},
)
for tool_call in tool_calls:
args = json.loads(tool_call.function.arguments)
messages.append(
{
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({"city": args["city"], "temp_f": 62}),
},
)
final = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
print(final.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
## Structured Outputs [#structured-outputs]
`response_format` is included in invocation parameters, so you can inspect which
schema constrained the model call.
<Metadata text="integrations/traces/openai-structured-output" />
```typescript TypeScript
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "user",
content: "Extract the city, temperature, and unit from: 72F in Berlin.",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "weather_report",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
city: { type: "string" },
temperature: { type: "number" },
unit: { type: "string", enum: ["F", "C"] },
},
required: ["city", "temperature", "unit"],
},
},
},
});
console.log(response.choices[0]?.message.content);
```
## Responses API [#responses-api]
Responses API calls are traced with the same OpenInference attribute families as
Chat Completions.
<Metadata text="integrations/traces/openai-responses-api" />
```typescript TypeScript
const response = await client.responses.create({
model: "gpt-4o-mini",
input: "In one sentence, what is OpenTelemetry?",
});
console.log(response.output_text);
```
# OpenCode Traces (https://docs.inference.net/integrations/traces/opencode)
[OpenCode](https://opencode.ai) is an open-source coding agent you run in your
terminal. The Inference platform OpenCode plugin records those coding sessions so you can
review what the agent did: model calls, tool calls, token usage, cost, finish
reasons, errors, prompts, responses, and tool payloads.
This page is for observing a coding tool **you use**. To trace an agent or
application **you build**, start with the [Tracing Quickstart](/integrations/traces/quickstart)
and the SDK integrations.
<Info>
OpenCode installs npm plugins automatically from your `opencode.json`; you do
not need to add this package to your app's dependencies.
</Info>
## Install [#install]
Add the plugin to your OpenCode config, either in your project `opencode.json`
or globally at `~/.config/opencode/opencode.json`.
```json opencode.json
{
"$schema": "https://opencode.ai/config.json",
"plugin": ["@inference/trace-opencode"]
}
```
## Configure [#configure]
Opt in with your inference.net API key. `INFERENCE_API_KEY` alone never turns
recording on (many environments export it for gateway calls), so pair it with
`TRACE_TO_INFERENCE=true`.
```bash
export TRACE_TO_INFERENCE=true
export INFERENCE_API_KEY="<your-inference-api-key>"
export INFERENCE_SERVICE_NAME="my-opencode"
```
Then use OpenCode as usual.
```bash
opencode run "add a health check endpoint and a test for it"
```
## What Gets Recorded [#what-gets-recorded]
Every turn, meaning one prompt and the agent's response to it, becomes a trace
in the Inference platform. Spans carry the OpenCode `session.id`, so all turns from the same
coding session can be grouped together.
* The `AGENT` root span records session id, service identity, call counts, stop
reason, and error status.
* `LLM` spans record model/provider details, finish reason,
prompt/cache/completion/reasoning tokens, and cost.
* `TOOL` spans record shell, file, and MCP tool calls with timing and error
status.
By default, spans include prompts, model output, tool arguments, and tool
output. Set `INFERENCE_REDACT_CONTENT=true` to keep spans metadata-only.
```text
Turn (AGENT, session.id=ses_...)
anthropic/claude-... (LLM: tokens, cache, finish reason, cost)
bash: ... (TOOL: name, timing, status)
read: file.ts (TOOL)
anthropic/claude-... (LLM: final reply)
```
Use these traces to audit what a session did, debug failed turns, compare how
models behave on similar tasks, or track coding-agent cost. Turn on content
redaction when you only need metadata and timing.
## Configuration Reference [#configuration-reference]
The plugin uses the same env-var convention as the TypeScript and Python
tracing SDKs.
| Environment variable | Default | Description |
| ------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INFERENCE_API_KEY` | - | inference.net API key used as the ingest bearer token. Requires `TRACE_TO_INFERENCE=true` to enable recording. |
| `INFERENCE_OTLP_ENDPOINT` | `https://telemetry.inference.net` | OTLP/HTTP trace ingest endpoint. |
| `INFERENCE_SERVICE_NAME` | `opencode` | Stable OTel `service.name`, also used as the OpenCode agent id. |
| `TRACE_TO_INFERENCE` | off | Set `true` to enable with `INFERENCE_API_KEY`, or for tokenless custom collectors with an explicit non-hosted endpoint. Set `false` to force-disable. |
| `INFERENCE_REDACT_CONTENT` | `false` | Set `true` to keep prompts, model outputs, tool arguments, and tool outputs off traces. |
| `INFERENCE_ADDITIONAL_METADATA` | - | JSON object attached to each turn's metadata. |
| `INFERENCE_DEBUG` | `false` | Verbose plugin logging through OpenCode's log API. |
### Local Collector [#local-collector]
To export to a local collector without an auth token, set both an explicit
endpoint and `TRACE_TO_INFERENCE=true`.
```bash
export INFERENCE_OTLP_ENDPOINT="http://localhost:4318"
export TRACE_TO_INFERENCE=true
```
## Privacy [#privacy]
By default the plugin records prompts, model output, file content read or
written by tools, and shell command output. Set
`INFERENCE_REDACT_CONTENT=true` when you want metadata-only traces.
## Troubleshooting [#troubleshooting]
If no traces appear, check:
* `TRACE_TO_INFERENCE=true` and `INFERENCE_API_KEY` are set in the
environment that launches OpenCode.
* `INFERENCE_SERVICE_NAME` is the value you are filtering for in the dashboard.
* `INFERENCE_OTLP_ENDPOINT` is `https://telemetry.inference.net` unless you are
intentionally using a custom collector.
* OpenCode has restarted after you added the plugin to `opencode.json`.
# Traces (https://docs.inference.net/integrations/traces/overview)
Inference Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Spans are emitted as OpenInference-shaped OpenTelemetry, exported over OTLP/HTTP, and shown in the Inference platform as trace trees, agent sessions, and per-span IO.
Use tracing for agent improvement. Once spans are flowing, point [Halo](https://github.com/context-labs/halo), our open-source agent-loop optimizer, at your traces and get back concrete suggestions for fixing prompts, tools, and harness logic. See [Analyze your traces](/get-started/analyze-traces) for the dashboard and Halo flow.
<Info>
Tracing and the Inference Gateway are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For gateway-only setups, see [model provider integrations](/integrations/overview).
</Info>
## How Tracing Fits Together [#how-tracing-fits-together]
Tracing has three layers. You add them in order, and most apps stop at the
second.
**1. Automatic SDK spans.** Pass an SDK to `setup({ modules })` and every call
through it becomes its own span, with input messages, output, model name,
finish reason, and token counts. You change nothing at the call site. For a
one-shot script, or a batch of unrelated LLM calls, this is all you need.
**2. An agent span around the run.** When one request drives several model or
tool calls (an agent run, a conversation turn, a workflow), wrap it in
`agentSpan`. The agent span is the parent row the SDK spans nest under, so you
see "Support Agent" with "openai chat" and "lookup\_order" beneath it instead of
three orphan rows. It carries `agentId`, `agentName`, `role`, and `sessionId`
for grouping in the Agents dashboard, and holds the run's high-level input and
output. Without it the SDK spans are still captured, just as a flat list with no
agent or conversation to group them by.
**3. Manual spans for your own steps.** When a step inside the run is not an SDK
call (a tool you execute yourself, a retrieval, a router, a validation pass),
wrap it in `manualSpan` with the matching span kind. It nests under the agent
span next to the auto-captured LLM spans, so the trace tree reflects everything
the run did, not just the model calls.
| You're doing... | What you need |
| ------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| One-shot call, or independent calls with no orchestration | `setup({ modules })`. Done. |
| Several calls per request, one agent | Add `agentSpan({ agentId, agentName, sessionId })` around the run |
| Multi-agent system | A separate `agentSpan` per agent, each with its own `agentId` and `role` |
| A non-SDK step inside the run (tool, retrieval, router, validator) | `manualSpan({ spanKind: TOOL / CHAIN / RETRIEVER })` inside the agent span |
| Subprocess or CLI call | `agentSpan` around it (no SDK to auto-patch) |
| Framework with native capture (LangGraph, OpenAI Agents, Cursor SDK, Vercel Eve, Vercel AI SDK, LiveKit, ElevenLabs, Claude Agent SDK, Pi Agent, PI AI) | Use the framework guide's setup path; add `agentSpan` when you need stable identity and session grouping |
<Tip>
Set `sessionId`, `agentId`, and `agentName` once on the outer `agentSpan`.
Every span underneath inherits them, so you should not be setting these in
multiple places.
</Tip>
## Packages [#packages]
Two packages, one shape. Install whichever matches your runtime.
| Runtime | Install | Import | Source |
| ----------------------- | ------------------------------------------------------------------------ | -------------------------------------------- | ------ |
| TypeScript / Node / Bun | [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) | `import { setup } from "@inference/tracing"` | npm |
| Python (3.10+) | [`inference-tracing`](https://pypi.org/project/inference-tracing/) | `from inference_tracing import setup` | PyPI |
Both packages share the same `setup()` entry point, the same env-var contract, and the same OpenInference attribute conventions. Pick whichever fits your stack and write the same code shape in either language.
<Tip>
Provider SDKs (`openai`, `@anthropic-ai/sdk`, `langchain`, etc.) are optional peer dependencies. Install only the ones you use. For Python, install per-integration extras: `pip install 'inference-tracing[openai,langchain]'`.
</Tip>
## Getting Started [#getting-started]
The fastest path is the [Inference CLI](/cli/overview):
<Metadata text="integrations/traces/overview-inf-instrument" />
```bash
inf instrument --mode tracing
```
`tracing` mode installs the SDK and wires up `setup()`. `both` mode does the same plus gateway routing in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See [Install with AI](/integrations/install-with-ai) for the full flow.
Prefer to wire it up by hand? Start with the path that matches what you want to instrument:
1. [Quickstart](/integrations/traces/quickstart) installs the SDK, configures export, and captures your first provider span. Both `Install with AI` and `Install manually` paths included.
2. [Provider and framework guides](#supported-trace-integrations) cover OpenAI, Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, Claude Agent SDK, Claude Code, Pydantic AI, ElevenLabs Agents, Vercel Eve, Pi Agent, PI AI, Cursor SDK, the Vercel AI SDK, and other supported surfaces.
3. [Manual spans](/integrations/traces/manual-spans) wrap custom agents, subprocesses, tools, retrieval, routing, and unsupported SDKs.
4. [Agent identity](/integrations/traces/agent-identity) sets stable `agent.id` values so the Agents dashboard groups executions correctly.
## Supported Trace Integrations [#supported-trace-integrations]
* [OpenAI](/integrations/traces/openai): Trace Chat Completions, tool calls, structured outputs, and Responses API calls.
* [Anthropic](/integrations/traces/anthropic): Trace Messages API calls, tool use round trips, and prompt-cache usage.
* [LangChain](/integrations/traces/langchain): Capture chains, agents, model calls, and tools through callback instrumentation.
* [LangGraph](/integrations/traces/langgraph): Capture graph and node spans while preserving parent-child relationships.
* [LangSmith](/integrations/traces/langsmith): Bridge LangSmith OpenTelemetry spans to the Inference platform.
* [OpenAI Agents](/integrations/traces/openai-agents): Trace agent runs, tools, handoffs, and nested OpenAI model calls.
* [LiveKit Agents](/integrations/traces/livekit-agents): Trace LiveKit sessions, model nodes, and function tools through native OTel spans.
* [ElevenLabs Agents](/integrations/traces/elevenlabs): Trace conversation sessions, transcripts, and client tool calls.
* [Vercel Eve](/integrations/traces/eve): Trace Eve agent turns, model calls, sub-agent invocations, tools, and `$eve.*` workflow tags.
* [Pi Agent](/integrations/traces/pi-agent): Trace current `@earendil-works` Pi agents, model turns, streams, tool calls, usage, and costs.
* [PI AI](/integrations/traces/pi-ai): Trace legacy `@mariozechner/pi-ai` model turns, streams, tool calls, usage, costs, and agent identity.
* [Cursor SDK](/integrations/traces/cursor-sdk): Trace Cursor agent runs, streamed messages, and tool calls.
* [Vercel AI SDK](/integrations/traces/ai-sdk): Trace generateText, streamText, ToolLoopAgent, tool calls, structured output, and usage.
* [Claude Agent SDK](/integrations/traces/claude-agent-sdk): Trace Claude Agent SDK query loops and yielded messages.
* [Claude Code SDK](/integrations/traces/claude-code-sdk): Trace Claude Code CLI subprocess calls and SDK-style agent invocations.
* [OpenCode](/integrations/traces/opencode): Record your own OpenCode coding sessions with LLM calls, tool calls, token usage, cost, and errors.
* [Pydantic AI](/integrations/traces/pydantic-ai): Trace Pydantic AI agent runs and tool calls (Python only).
* [Manual spans](/integrations/traces/manual-spans): Wrap custom agents, CLI calls, provider routing, and unsupported SDKs.
* [Agent identity](/integrations/traces/agent-identity): Set stable agent IDs, display names, and roles for dashboard grouping.
## What Gets Captured [#what-gets-captured]
| Span data | Examples |
| ------------------ | ------------------------------------------------------------------------ |
| Inputs and outputs | `input.value`, `output.value` |
| Messages | user, system, assistant, tool, and tool-result messages |
| Tool calls | tool names, IDs, JSON arguments, and tool results |
| Model metadata | `llm.model_name`, `llm.invocation_parameters`, `gen_ai.system` |
| Usage | prompt, completion, total, and prompt-cache token counts when available |
| Agent structure | agent spans, framework spans, tool spans, graph/node spans |
| Agent identity | `agent.id`, `agent.name`, and `agent.role` for Agents dashboard grouping |
| Errors | exception status and error details on failed spans |
All keys follow [OpenInference](https://github.com/Arize-ai/openinference) conventions, so the same spans can be consumed by any OpenInference-aware viewer.
## Configuration [#configuration]
Both SDKs read the same env vars and accept the same options on `setup()`.
### Env Vars And Identity [#env-vars-and-identity]
| Env var | Purpose |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `INFERENCE_OTLP_ENDPOINT` | OTLP/HTTP trace ingest endpoint. Set to `https://telemetry.inference.net` for the hosted Inference platform backend. |
| `INFERENCE_API_KEY` | Bearer token for ingest auth. Generate one at [API Keys](https://inference.net/dashboard/api-keys). |
| `INFERENCE_SERVICE_NAME` | OTel `service.name` resource attribute. Use a stable value per deployed service. |
| `INFERENCE_SERVICE_VERSION` | OTel `service.version` resource attribute. |
| `INFERENCE_DEBUG` | Enable SDK debug logging. |
Precedence: explicit options on `setup()` win, then `INFERENCE_*` env vars, then SDK defaults.
Ingest is OTLP/JSON only. The exporters built into the SDKs send JSON by
default; if you point a custom exporter at `INFERENCE_OTLP_ENDPOINT`, force
JSON encoding (`encoding: json`) or the ingest endpoint returns 415.
### Instrumentation Control [#instrumentation-control]
Two options control which SDKs Inference platform patches at setup time.
| Option | Default | Purpose |
| ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `modules` (TypeScript) | `{}` | Map of SDK modules to patch. Pass `{ openai: OpenAI, anthropic: Anthropic }`. Entries listed here are always patched. Python auto-detects installed packages and does not need this option. |
| `autoInstrument` | `true` | When `true`, `setup()` also scans `node_modules` and patches any other supported SDK it finds. When `false`, only the SDKs you list in `modules:` get patched, nothing else. |
Three patterns, in order of how often they show up in real apps:
**Auto-detect everything.** Call `setup()` with no `modules:`. The auto-scan finds every supported SDK and patches it. Easiest path for scripts, demos, and small apps.
```ts
const tracing = await setup();
```
**Explicit modules plus auto-detect.** Pass the SDKs you use to `modules:`. Those entries are patched directly (no resolver gymnastics), and `setup()` also auto-detects anything else.
```ts
const tracing = await setup({ modules: { openai: OpenAI } });
```
**Explicit only.** Pass `modules:` and set `autoInstrument: false`. Only the SDKs you list get patched, even if a transitive dep happens to pull in another supported one. Cleanest pattern for production services that know exactly which providers they call.
```ts
const tracing = await setup({
modules: { anthropic: Anthropic },
autoInstrument: false,
});
```
<Tip>
Pass only the SDK modules you actually use to `modules`. Auto-instrumentation
patches prototypes, so patching an SDK you do not call adds no runtime cost
but a small import cost.
</Tip>
### Imports (TypeScript) [#imports-typescript]
For almost all setups, `@inference/tracing` is the only import you need.
`setup()` auto-patches every supported global SDK it can find (OpenAI,
Anthropic, LangChain, LangGraph, ElevenLabs, LiveKit, legacy PI AI, and the
rest). Current Pi Agent apps use per-instance `Models` collections, so pass the
collection as `modules.piAgent`.
| Export | What it does |
| ------------------------- | ------------------------------------------------------------------------------------------------------- |
| `setup` | Initializes tracing. Pass the SDK modules you use; it patches them and returns a handle for shutdown. |
| `agentSpan` | Opens a parent span that groups everything an agent run does. Use one per top-level invocation. |
| `manualSpan` | Opens a custom span for work that is not an SDK call: tool execution, retrieval, validation, etc. |
| `wrapClaudeAgentSdkQuery` | Wraps a Claude Agent SDK `query()` call so each tool turn becomes a span. |
| `Attr`, `SpanKindValues` | OpenInference attribute keys and span-kind enums for setting custom attributes. |
| Error types | `CatalystTracingError`, `InvalidSdkInputError`, `InvalidTracerInputError`, for catching setup failures. |
#### When you need a subpath [#when-you-need-a-subpath]
Four cases call for importing from a per-integration subpath:
1. **You're using the Vercel AI SDK.** The AI SDK has its own
`experimental_telemetry` option, so there is nothing for `setup()` to
patch. You import `createAISdkTelemetrySettings` and pass its result into
each `generateText` / `streamText` call.
2. **You're using Vercel Eve.** Eve enables telemetry by loading
`agent/instrumentation.ts`, so you export `defineCatalystEveInstrumentation()`
from `@inference/tracing/eve` in that file.
3. **You're adding tracing after creating a Pi `Models` collection.** Call
`instrumentPiAgent(models, tracing)` before the first model call.
4. **You're installing instrumentation on a custom tracer provider** instead of the one `setup()` registers. Rare in practice; most apps want the standard flow. The subpath helpers (`installOpenAIInstrumentation`, etc.) accept an explicit provider argument.
If neither applies, you can skip the table below.
| Import | Exports | Use For |
| ------------------------------ | -------------------------------------------------------------------------------- | ------------- |
| `@inference/tracing/ai-sdk` | `createAISdkTelemetrySettings`, `instrumentAISdk`, `installAISdkInstrumentation` | Vercel AI SDK |
| `@inference/tracing/openai` | `instrumentOpenAI`, `installOpenAIInstrumentation` | OpenAI |
| `@inference/tracing/anthropic` | `instrumentAnthropic`, `installAnthropicInstrumentation` | Anthropic |
| `@inference/tracing/langchain` | `instrumentLangChain`, `installLangChainInstrumentation` | LangChain |
| `@inference/tracing/langgraph` | `instrumentLangGraph`, `installLangChainInstrumentation` | LangGraph |
LangGraph re-exports the LangChain instrumentation: there is no separate
`installLangGraphInstrumentation`. LangGraph graphs are captured through the
LangChain callback path.
\| `@inference/tracing/langsmith` | `installLangSmith` | LangSmith bridge |
\| `@inference/tracing/openai-agents` | `instrumentOpenAIAgents` | `@openai/agents` |
\| `@inference/tracing/claude-agent-sdk` | `wrapClaudeAgentSdkQuery` | Claude Agent SDK |
\| `@inference/tracing/elevenlabs` | `instrumentElevenLabs`, `installElevenLabsInstrumentation` | ElevenLabs Agents |
\| `@inference/tracing/eve` | `defineCatalystEveInstrumentation`, `installEveSpanEnrichment` | Vercel Eve |
\| `@inference/tracing/pi-agent` | `instrumentPiAgent`, `installPiAgentInstrumentation` | Current Pi Agent (`@earendil-works`) |
\| `@inference/tracing/pi-ai` | `instrumentPiAi`, `installPiAiInstrumentation` | Legacy PI AI (`@mariozechner`) |
\| `@inference/tracing/cursor-sdk` | `instrumentCursorSdk`, `installCursorSdkInstrumentation` | Cursor SDK |
\| `@inference/tracing/livekit-agents` | `instrumentLiveKitAgents`, `installLiveKitAgentsInstrumentation` | LiveKit Agents |
\| `@inference/tracing/semconv` | `OpenInferenceAttribute`, `MessageRole` (plus `Attr` and `SpanKindValues`, also on the root) | OpenInference constants |
Python uses one import root (`inference_tracing`) and gates each
integration behind a pip extra (`pip install 'inference-tracing[openai,anthropic]'`).
### The `setup()` Return Value [#the-setup-return-value]
`setup()` returns a `CatalystTracing` handle:
| Field | Purpose |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tracer` | The OpenTelemetry `Tracer` for authoring manual spans. In Python, pass it to `agent_span` / `manual_span`. In TypeScript, `agentSpan` / `manualSpan` resolve it automatically after `setup()`, so this is an escape hatch — use it for `tracer.startActiveSpan(...)` directly. |
| `provider` | The OTel tracer provider, in case you want to register additional span processors or exporters. |
| `installed` | Per-integration install result: which SDKs were detected, which were patched, and any errors. Useful in CI and on startup logs. |
| `shutdown()` | Flush and close tracing. Call this on `SIGTERM` for long-lived services, before `process.exit(0)` for scripts. |
For long-lived and serverless processes, memoize the `setup()` promise and
flush correctly for the process shape. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle).
## How It Works [#how-it-works]
1. `setup()` configures an OpenTelemetry tracer provider with a Inference Tracing OTLP/HTTP exporter.
2. The SDK auto-patches any supported provider or framework SDK it finds. In TypeScript you pass them via `modules`. In Python it auto-detects installed packages.
3. Your app runs normally. Spans are batched and exported in the background.
4. Spans land in the Inference platform grouped by service, trace, span, and task metadata. View them in the [Trajectories tab or the Agents tab](/get-started/analyze-traces).
5. Run analysis on them with [Halo](https://github.com/context-labs/halo) to understand your traces and gain insights you can pass into your favorite coding agent.
## Inspect Traces from the CLI [#inspect-traces-from-the-cli]
After spans are flowing to the Inference platform, you can inspect the same trace data from the terminal:
<Metadata text="integrations/traces/overview-cli-inspect" />
```bash
# Browse recent traces
inf trace list --range 1h
# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline
# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io
```
See [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) for the full command reference.
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Quickstart" icon="rocket" href="/integrations/traces/quickstart">
Install an SDK, configure export, and capture your first model span.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool spans, end to end.
</Card>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Author AGENT, TOOL, CHAIN, and RETRIEVER spans.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
Every `Attr.*` constant and `SpanKindValues` value.
</Card>
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
Every method on the span handle yielded by `agentSpan` / `manualSpan` and the Python equivalents.
</Card>
<Card title="Instrumentation examples" icon="code" href="/integrations/traces/examples">
Copyable examples for providers, agents, tool loops, and prompt caching.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable IDs and readable names to agent spans.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
Debug missing spans, missing attributes, and shutdown behavior.
</Card>
</CardGroup>
# Pi Agent Traces (https://docs.inference.net/integrations/traces/pi-agent)
<Info>
Using legacy `@mariozechner/pi-ai`? Use the
[PI AI integration](/integrations/traces/pi-ai).
</Info>
Inference platform instruments current Pi Agent applications built with
`@earendil-works/pi-agent-core` and `@earendil-works/pi-ai`. Each Pi `Models`
collection owns its providers. Pass that collection to the Inference platform so each model
turn emits an OpenInference LLM span.
This guide is tested with `@earendil-works/pi-agent-core@0.84.1` and
`@earendil-works/pi-ai@0.84.1`.
Pi Agent is available for TypeScript. There is no Python equivalent for this
integration.
## What Is Captured [#what-is-captured]
* One LLM span per model turn, named like `pi-agent.<provider>.turn`
* Calls through `stream`, `streamSimple`, `complete`, and `completeSimple`
* System prompts, input messages, assistant output, model name, and provider
* Tool call IDs, names, and JSON arguments from assistant messages
* Token usage, prompt cache read/write counts, finish reason, and total cost
* Errors, aborts, and exception details
* Active `agentSpan()` identity, including `agent.id`, `agent.name`,
`agent.role`, and `session.id`
## Install [#install]
<Metadata text="integrations/traces/pi-agent-install" />
```bash TypeScript
bun add @inference/tracing@0.1.9 @earendil-works/pi-agent-core@0.84.1 @earendil-works/pi-ai@0.84.1
```
## Configure Export [#configure-export]
Set the Inference Tracing endpoint and token before your app starts. Generate a token at
[API Keys](https://inference.net/dashboard/api-keys).
<Metadata text="integrations/traces/pi-agent-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="pi-agent"
export ANTHROPIC_API_KEY="<your-anthropic-api-key>"
```
## Initialize Tracing [#initialize-tracing]
Create the Pi `Models` collection before tracing setup. Pass the collection as
`piAgent`. You can add providers before or after setup.
<Metadata text="integrations/traces/pi-agent-init[series=pi_agent_setup]" />
```typescript TypeScript
import { createModels } from "@earendil-works/pi-ai";
import { anthropicProvider } from "@earendil-works/pi-ai/providers/anthropic";
import { setup } from "@inference/tracing";
const models = createModels();
const tracing = await setup({
serviceName: process.env.INFERENCE_SERVICE_NAME ?? "pi-agent",
autoInstrument: false,
modules: { piAgent: models },
});
models.setProvider(anthropicProvider());
```
<Warning>
Pi uses a separate `Models` collection for each application. Inference platform cannot
find that collection through package auto-detection. Pass it as
`modules.piAgent` or call `instrumentPiAgent(models, tracing)`.
</Warning>
For manual initialization, use the Pi Agent subpath helper before the agent
makes its first model call.
<Metadata text="integrations/traces/pi-agent-manual-init[series=pi_agent_manual]" />
```typescript TypeScript
import { createModels } from "@earendil-works/pi-ai";
import { setup } from "@inference/tracing";
import { instrumentPiAgent } from "@inference/tracing/pi-agent";
const models = createModels();
const tracing = await setup({ autoInstrument: false });
instrumentPiAgent(models, tracing);
```
## Run An Agent [#run-an-agent]
Pass `models.streamSimple.bind(models)` to the current Pi `Agent`. Wrap the run
in `agentSpan()` to group all model turns under one stable agent identity.
The following example uses an Anthropic model. It assumes you ran the setup
block above.
<Metadata text="integrations/traces/pi-agent-run[series=pi_agent_setup]" />
```typescript TypeScript
import { Agent } from "@earendil-works/pi-agent-core";
import { agentSpan } from "@inference/tracing";
const model = models.getModel("anthropic", "claude-sonnet-4-6");
if (!model) throw new Error("Pi model not found");
const agent = new Agent({
initialState: {
systemPrompt: "You answer order questions in one short sentence.",
model,
},
sessionId: "order-abc-123",
streamFn: models.streamSimple.bind(models),
});
await agentSpan(
{
agentId: "pi-support-agent",
agentName: "Pi Support Agent",
spanName: "pi-support-agent.run",
sessionId: "order-abc-123",
role: "support",
system: "pi-agent",
},
async (span) => {
const input = "Summarize order ABC-123.";
span.setInput(input);
await agent.prompt(input);
span.setOutput(agent.state.messages.at(-1));
},
);
await tracing.shutdown();
```
Expected spans:
* `pi-support-agent.run` AGENT span
* One or more `pi-agent.anthropic.turn` LLM child spans
## Trace Tool Execution [#trace-tool-execution]
Pi returns model tool calls in assistant messages and executes `AgentTool`
functions locally. Inference platform records the requested tool name, ID, and arguments
on the LLM span. Wrap local execution with `manualSpan()` when you also want a
TOOL span for the work.
<Metadata text="integrations/traces/pi-agent-tool[series=pi_agent_tool]" />
```typescript TypeScript
import { type AgentTool } from "@earendil-works/pi-agent-core";
import { Type } from "@earendil-works/pi-ai";
import { manualSpan, SpanKindValues } from "@inference/tracing";
const parameters = Type.Object({ orderId: Type.String() });
const lookupOrder: AgentTool<typeof parameters> = {
name: "lookup_order",
label: "Look up order",
description: "Look up an order by ID.",
parameters,
execute: async (toolCallId, { orderId }) =>
manualSpan(
{
spanName: "lookup_order",
spanKind: SpanKindValues.TOOL,
toolName: "lookup_order",
toolCallId,
input: { orderId },
},
async (span) => {
const order = { orderId, status: "shipped", eta: "Friday" };
span.setOutput(order);
return {
content: [{ type: "text" as const, text: JSON.stringify(order) }],
details: order,
};
},
),
};
```
Add `lookupOrder` to `initialState.tools`. A tool round trip then produces:
* A `pi-agent.<provider>.turn` LLM span with the requested tool call
* A `lookup_order` TOOL span for local execution
* Another `pi-agent.<provider>.turn` LLM span for the final answer
## Verify in the Inference platform [#verify-in-the-inference-platform]
Filter traces by your `service.name`, for example `pi-agent`. A successful run
shows the AGENT span with nested Pi Agent LLM spans. Each LLM span includes
input/output, model metadata, usage, finish reason, and tool call attributes.
If no Pi Agent spans appear:
* Pass the `Models` collection as `modules: { piAgent: models }`.
* Instrument the collection before the agent makes its first model call.
* Add providers through the instrumented collection's `setProvider()` method.
* Consume streaming results or await `agent.prompt()` before shutdown.
* Call `await tracing.shutdown()` before a short-lived process exits.
# PI AI Traces (https://docs.inference.net/integrations/traces/pi-ai)
<Info>
Using current `@earendil-works/pi-ai` or `@earendil-works/pi-agent-core`?
Use the [Pi Agent integration](/integrations/traces/pi-agent).
</Info>
Inference platform instruments `@mariozechner/pi-ai` in TypeScript. PI AI is a unified
LLM API and provider registry. Inference platform patches the registered PI AI providers
so each model turn through `stream`, `streamSimple`, `complete`, or
`completeSimple` emits an OpenInference LLM span.
Use this guide for Node applications that use PI AI for model calls, tool
calling, cross-provider handoffs, or agent workflows. PI AI is a TypeScript
package, so there is no Python equivalent for this integration.
This page covers the legacy `@mariozechner` package family. Inference platform continues
to support it for existing applications.
## What Is Captured [#what-is-captured]
* One LLM span per PI AI model turn, named like `pi-ai.<provider>.turn`
* `stream` and `streamSimple` calls, plus `complete` and `completeSimple`
calls because PI AI completes by resolving the underlying stream
* System prompts, input messages, assistant output, model name, provider, and
invocation parameters
* Tool call IDs, names, and JSON arguments from assistant messages
* Token usage, prompt cache read/write counts, and total cost when PI AI
returns them
* Finish reason, errors, aborts, and exception details
* Active `agentSpan()` identity, including `agent.id`, `agent.name`,
`agent.role`, and `session.id`
## Install [#install]
<Metadata text="integrations/traces/pi-ai-install" />
```bash TypeScript
bun add @inference/tracing @mariozechner/pi-ai
```
## Configure Export [#configure-export]
Set the Inference Tracing endpoint and token before your app starts. Generate a token at
[API Keys](https://inference.net/dashboard/api-keys).
<Metadata text="integrations/traces/pi-ai-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="pi-ai-agent"
export OPENAI_API_KEY="<your-openai-api-key>"
```
## Initialize Tracing [#initialize-tracing]
Initialize Inference platform before making PI AI calls. Import the PI AI namespace and
pass it to `setup()` so Inference platform patches the same provider registry your app
uses.
<Metadata text="integrations/traces/pi-ai-init" />
```typescript TypeScript
import * as piAi from "@mariozechner/pi-ai";
import { setup } from "@inference/tracing";
const tracing = await setup({
serviceName: process.env.INFERENCE_SERVICE_NAME ?? "pi-ai-agent",
modules: { piAi },
});
```
For explicit manual initialization, disable auto-instrumentation and use the PI
AI subpath helper.
<Metadata text="integrations/traces/pi-ai-manual-init" />
```typescript TypeScript
import * as piAi from "@mariozechner/pi-ai";
import { setup } from "@inference/tracing";
import { instrumentPiAi } from "@inference/tracing/pi-ai";
const tracing = await setup({ autoInstrument: false });
instrumentPiAi(piAi, tracing);
```
## Complete Call [#complete-call]
`completeSimple()` is the smallest path. PI AI resolves it through a provider
stream, so Inference platform emits the LLM span when the call finishes.
The examples below assume you initialized `tracing` with the setup block above.
<Metadata text="integrations/traces/pi-ai-complete" />
```typescript TypeScript
import { completeSimple, getModel, type Context } from "@mariozechner/pi-ai";
const model = getModel("openai", "gpt-4o-mini");
const context: Context = {
systemPrompt: "You answer in one concise sentence.",
messages: [{ role: "user", content: "Why are trace trees useful?" }],
};
const response = await completeSimple(model, context);
console.log(response.content);
```
Expected span:
* `pi-ai.openai.turn`
Expected promoted fields include `llm.model_name`, `input.value`,
`output.value`, `llm.token_count.prompt`, `llm.token_count.completion`,
`llm.token_count.total`, and `llm.invocation_parameters` when PI AI returns the
corresponding data.
## Streaming [#streaming]
For streaming calls, consume the stream and call `await stream.result()` before
process shutdown. Inference platform finishes the span when PI AI emits a done/error event
or when `.result()` resolves.
<Metadata text="integrations/traces/pi-ai-stream" />
```typescript TypeScript
import { getModel, streamSimple, type Context } from "@mariozechner/pi-ai";
const model = getModel("openai", "gpt-4o-mini");
const context: Context = {
messages: [{ role: "user", content: "Stream a sentence about observability." }],
};
const stream = streamSimple(model, context);
for await (const event of stream) {
if (event.type === "text_delta") {
process.stdout.write(event.delta);
}
}
const finalMessage = await stream.result();
console.log("\nFinished:", finalMessage.stopReason);
```
For short-lived scripts, call `await tracing.shutdown()` after the stream is
fully consumed.
## Tool Loop [#tool-loop]
PI AI returns tool calls in assistant message content. Inference platform records the tool
call name, ID, and arguments on the PI AI LLM span. Your local tool execution is
application code, so wrap it with `manualSpan()` if you also want a TOOL child
span for the work your app performs.
<Metadata text="integrations/traces/pi-ai-tool-loop" />
```typescript TypeScript
import {
Type,
completeSimple,
getModel,
type Context,
type Tool,
} from "@mariozechner/pi-ai";
import { manualSpan, SpanKindValues } from "@inference/tracing";
const model = getModel("openai", "gpt-4o-mini");
const tools: Tool[] = [
{
name: "lookup_order",
description: "Look up an order by ID.",
parameters: Type.Object({
orderId: Type.String({ description: "Customer order ID." }),
}),
},
];
const context: Context = {
systemPrompt: "Use tools when you need order data.",
messages: [{ role: "user", content: "Where is order ABC-123?" }],
tools,
};
const firstTurn = await completeSimple(model, context);
context.messages.push(firstTurn);
for (const block of firstTurn.content) {
if (block.type !== "toolCall" || block.name !== "lookup_order") continue;
const args = block.arguments as { orderId: string };
const order = await manualSpan(
{
spanName: "lookup_order",
spanKind: SpanKindValues.TOOL,
toolName: block.name,
toolCallId: block.id,
input: args,
},
async (span) => {
const result = {
orderId: args.orderId,
status: "shipped",
eta: "Friday",
};
span.setOutput(result);
return result;
},
);
context.messages.push({
role: "toolResult",
toolCallId: block.id,
toolName: block.name,
content: [{ type: "text", text: JSON.stringify(order) }],
isError: false,
timestamp: Date.now(),
});
}
const finalTurn = await completeSimple(model, context);
console.log(finalTurn.content);
```
Expected spans:
* `pi-ai.openai.turn` for the first model turn with the tool call
* `lookup_order` TOOL span when you wrap local execution with `manualSpan()`
* another `pi-ai.openai.turn` for the continuation after the tool result
## Stable Agent Identity [#stable-agent-identity]
Wrap the full PI AI run in `agentSpan()` when one user request can produce
multiple model turns or tool executions. PI AI LLM spans inherit the active
agent identity and nest under the AGENT span.
<Metadata text="integrations/traces/pi-ai-agent-identity" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
import { completeSimple, getModel, type Context } from "@mariozechner/pi-ai";
const model = getModel("openai", "gpt-4o-mini");
await agentSpan(
{
agentId: "pi-ai-support-agent",
agentName: "PI AI Support Agent",
spanName: "pi-ai-support-agent.run",
sessionId: "conversation-order-abc-123",
role: "support",
system: "pi-ai",
},
async (span) => {
const input = "Summarize order ABC-123 for the customer.";
const context: Context = {
messages: [{ role: "user", content: input }],
};
span.setInput(input);
const response = await completeSimple(model, context);
span.setOutput(response.content);
},
);
```
## Verify in the Inference platform [#verify-in-the-inference-platform]
Filter traces by your `service.name`, for example `pi-ai-agent`. A successful
run should show PI AI LLM spans named by provider, such as
`pi-ai.openai.turn`, with captured input/output, model metadata, usage, finish
reason, and tool call attributes. If you wrap the run in `agentSpan()`, the same
LLM spans should show the inherited `agent.id`, `agent.name`, and `agent.role`.
If no PI AI spans appear:
* Initialize Inference platform before making PI AI calls.
* Pass the PI AI namespace directly with `modules: { piAi }` or
`instrumentPiAi(piAi, tracing)`.
* Consume streaming results or call `.result()` so PI AI can finish the turn.
* Call `await tracing.shutdown()` before process exit in short-lived scripts.
# Production Agent Example (https://docs.inference.net/integrations/traces/production-agent-example)
This example walks through a realistic agent loop end to end: a long-lived
server that handles incoming messages, runs an LLM-driven agent that calls
several custom tools, and exports a clean OpenInference trace tree to
Inference platform. Every piece in this guide maps to a real pattern used by agents in
production today.
By the end you will have:
* A boot-time `setup()` that runs once per process.
* A request handler that wraps the whole agent run in an `AGENT` span with
stable `agent.id` and per-conversation `session.id`.
* Custom tool execution wrapped in `TOOL` spans with `tool.name` and
`tool_call.id`.
* Auto-emitted `LLM` child spans from the patched Anthropic SDK, nested under
the agent span by OTel context propagation.
* Domain-specific attributes (tenant, channel, viewer role) on the agent span
for filtering in the dashboard.
* A graceful shutdown that flushes batched spans on `SIGTERM`.
<Info>
This example **hand-rolls the agent loop** with the raw Anthropic SDK, so you
author the `TOOL` spans yourself (Step 3). The patched SDK captures the `LLM`
calls automatically, but your tools run in your own code, so nothing emits a
tool span unless you do.
If you use a framework that runs the tools for you (OpenAI Agents, LangGraph),
the framework emits the `TOOL` spans and you skip Step 3. There you would only
add manual spans for steps the framework never sees, like a retrieval inside a
tool. See [OpenAI Agents](/integrations/traces/openai-agents) for that path.
</Info>
## Step 1 — Bootstrap Tracing Once [#step-1--bootstrap-tracing-once]
Tracing should initialize **once per process**, not per request. For a
long-lived server, that means a memoized `setup()` call that any code path
can `await`.
<Metadata text="integrations/traces/cookbook-bootstrap" />
```typescript TypeScript
// tracing.ts
import Anthropic from "@anthropic-ai/sdk";
import { setup, type CatalystTracing } from "@inference/tracing";
let tracingPromise: Promise<CatalystTracing> | null = null;
export function initTracing(): Promise<CatalystTracing> {
if (!tracingPromise) {
tracingPromise = setup({
serviceName: process.env.SERVICE_NAME ?? "customer-support",
serviceVersion: process.env.SERVICE_VERSION,
endpoint: process.env.INFERENCE_OTLP_ENDPOINT,
token: process.env.INFERENCE_API_KEY,
modules: { anthropic: Anthropic },
});
}
return tracingPromise;
}
export async function shutdownTracing(): Promise<void> {
if (!tracingPromise) return;
const tracing = await tracingPromise;
await tracing.shutdown();
}
```
```typescript TypeScript (server entrypoint)
// server.ts
import { initTracing, shutdownTracing } from "./tracing.ts";
await initTracing(); // patches Anthropic before any client is constructed
const server = startServer();
for (const signal of ["SIGTERM", "SIGINT"] as const) {
process.on(signal, async () => {
await shutdownTracing();
server.close(() => process.exit(0));
});
}
```
Two things to notice:
1. **`initTracing()` runs before the first Anthropic client is constructed.**
The per-SDK patchers work by mutating the SDK's prototype, so `setup()`
has to win the race.
2. **`shutdown()` runs on `SIGTERM`, not per request.** Spans are batched and
exported in the background; calling `shutdown()` per request would force
synchronous flushes and add latency.
## Step 2 — Define The Request Boundary [#step-2--define-the-request-boundary]
Each incoming message becomes one trace, rooted at one `AGENT` span. The
agent span carries the identifiers Inference platform uses for grouping in the Agents
dashboard.
<Metadata text="integrations/traces/cookbook-request-boundary" />
```typescript TypeScript
import { agentSpan } from "@inference/tracing";
import { initTracing } from "./tracing.ts";
import { runAgent } from "./agent.ts";
export interface IncomingMessage {
conversationId: string;
text: string;
channel: "slack" | "email" | "web";
tenantId: string;
viewer: { id: string; role: "admin" | "member" };
}
export async function handleMessage(msg: IncomingMessage): Promise<string> {
const tracing = await initTracing();
return await agentSpan(
{
agentId: "customer-support-prod",
agentName: "Customer Support Agent",
role: "support",
system: "anthropic",
sessionId: msg.conversationId,
spanName: "customer-support.run",
},
async (span) => {
// Domain attributes for dashboard filtering.
span.raw.setAttribute("app.tenant_id", msg.tenantId);
span.raw.setAttribute("app.channel", msg.channel);
span.raw.setAttribute("app.viewer.id", msg.viewer.id);
span.raw.setAttribute("app.viewer.role", msg.viewer.role);
span.setInput(msg.text);
const response = await runAgent(msg);
span.setOutput(response);
return response;
},
);
}
```
The four `app.*` attributes are *outside* the OpenInference vocabulary. They
go on the raw OTel span and become filter facets in the dashboard. Use the
same naming convention (a stable prefix for your app, dot-separated keys) so
you can find them easily under `inf trace list --metadata "app.channel=slack"`.
## Step 3 — Author Tool Spans Around Each Tool Call [#step-3--author-tool-spans-around-each-tool-call]
When the LLM emits a `tool_use` block, your code runs the actual tool
function. Wrap that execution in a `TOOL` span so the trace tree shows what
the tool received, what it returned, and how long it took.
<Metadata text="integrations/traces/cookbook-tool-span" />
```typescript TypeScript
// tools.ts
import { manualSpan, SpanKindValues } from "@inference/tracing";
import { initTracing } from "./tracing.ts";
export type ToolName = "lookup_order" | "issue_refund" | "send_email";
export type ToolArgs = Record<string, unknown>;
export type ToolResult = Record<string, unknown>;
const TOOL_IMPLS: Record<ToolName, (args: ToolArgs) => Promise<ToolResult>> = {
lookup_order: async ({ orderId }) => ({ orderId, status: "shipped" }),
issue_refund: async ({ orderId, amount }) => ({
ok: true,
orderId,
amount,
refundId: "RFD-" + Math.floor(Math.random() * 9999),
}),
send_email: async ({ to, subject }) => ({ ok: true, to, subject }),
};
export async function executeTool(
name: ToolName,
args: ToolArgs,
toolCallId: string,
): Promise<ToolResult> {
const tracing = await initTracing();
return await manualSpan(
{
spanName: `${name}.tool`,
spanKind: SpanKindValues.TOOL,
toolName: name,
toolCallId,
input: args,
},
async (span) => {
const result = await TOOL_IMPLS[name](args);
span.setOutput(result);
return result;
},
);
}
```
`manualSpan` writes `openinference.span.kind=TOOL`, `tool.name`,
`tool_call.id`, and `input.value` from the options. Set `input.mime_type`
with `Attr.INPUT_MIME_TYPE` when you want the input rendered as JSON. The
callback only needs to set the output. Span end, status, and exception
recording are all handled — if the tool throws, the exception is recorded
on the span, the span ends with `ERROR`, and the original exception
re-throws so the agent loop can see it.
Because `executeTool` runs inside the active context established by
`agentSpan` upstream, the `TOOL` span automatically parents under the agent
span. No span IDs need to be threaded through.
<Tip>
If your tool needs behavior `manualSpan` does not provide — for instance,
recording a span event mid-callback while keeping the span alive past the
callback return — drop down to `tracing.tracer.startActiveSpan` and manage
status / `span.end()` yourself. See
[Manual spans → Escape hatch](/integrations/traces/manual-spans#escape-hatch).
</Tip>
## Step 4 — Wire The Agent Loop [#step-4--wire-the-agent-loop]
The agent loop alternates between calling the LLM and executing tool calls
the LLM requests. Both sides are now instrumented.
<Metadata text="integrations/traces/cookbook-agent-loop" />
```typescript TypeScript
// agent.ts
import Anthropic from "@anthropic-ai/sdk";
import { executeTool, type ToolName } from "./tools.ts";
import type { IncomingMessage } from "./handler.ts";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const TOOLS: Anthropic.Tool[] = [
{
name: "lookup_order",
description: "Look up an order by ID.",
input_schema: {
type: "object",
properties: { orderId: { type: "string" } },
required: ["orderId"],
},
},
{
name: "issue_refund",
description: "Issue a refund.",
input_schema: {
type: "object",
properties: {
orderId: { type: "string" },
amount: { type: "number" },
},
required: ["orderId", "amount"],
},
},
];
export async function runAgent(msg: IncomingMessage): Promise<string> {
const messages: Anthropic.MessageParam[] = [
{ role: "user", content: msg.text },
];
for (let turn = 0; turn < 8; turn++) {
// The patched Anthropic SDK emits an LLM span automatically, parented
// under the active agent span via OTel context propagation.
const response = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 1024,
tools: TOOLS,
messages,
});
if (response.stop_reason === "end_turn") {
return textOf(response.content);
}
if (response.stop_reason !== "tool_use") {
return textOf(response.content);
}
messages.push({ role: "assistant", content: response.content });
const toolResults: Anthropic.ToolResultBlockParam[] = [];
for (const block of response.content) {
if (block.type !== "tool_use") continue;
const result = await executeTool(
block.name as ToolName,
block.input as Record<string, unknown>,
block.id,
);
toolResults.push({
type: "tool_result",
tool_use_id: block.id,
content: JSON.stringify(result),
});
}
messages.push({ role: "user", content: toolResults });
}
return "Max turns reached.";
}
function textOf(content: Anthropic.ContentBlock[]): string {
return content
.filter((b): b is Anthropic.TextBlock => b.type === "text")
.map((b) => b.text)
.join("");
}
```
Three observations:
1. **No tracing imports in the inner loop.** The agent code looks the same as
it would without tracing. The instrumentation is at the boundaries
(`setup()`, `agentSpan()`, `executeTool()`).
2. **The patched Anthropic SDK does the LLM-span work.** We pass
`modules: { anthropic: Anthropic }` to `setup()`, and from then on every
`client.messages.create()` call emits an `LLM` span with input messages,
output content blocks, model, finish reason, and token usage.
3. **Tool spans are caller-side.** They wrap the real function execution, not
the message round-trip. The model-side view of the tool call is captured
on the parent `LLM` span automatically; the caller-side view is the
`TOOL` span we author.
## Step 5 — Verify In The Dashboard And CLI [#step-5--verify-in-the-dashboard-and-cli]
Send a request through the server, then check the resulting trace:
<Metadata text="integrations/traces/cookbook-verify" />
```bash
# Find the most recent trace from this service
inf trace list --service customer-support --limit 1
# Open its span tree
inf trace get <trace-id> --view tree
# Inspect a TOOL span's input and output
inf span list --trace-id <trace-id> --kind TOOL
inf span get <trace-id> <span-id> --view io
# Filter on a domain attribute
inf trace list --metadata "app.channel=slack" --range 1h
```
The trace tree should show a single AGENT root with the LLM and TOOL spans
nested beneath it.
## Common Variations [#common-variations]
### Multi-Tenant Service With Per-Request Identity [#multi-tenant-service-with-per-request-identity]
If `agent.id` itself depends on the request (for example, a multi-tenant
service that runs different agent personas per customer), compute it in the
handler:
<Metadata text="integrations/traces/cookbook-multitenant" />
```typescript TypeScript
const agentId = `support-${msg.tenantId}-prod`;
await agentSpan(
{
agentId,
agentName: `${tenantConfig.displayName} Support`,
role: "support",
sessionId: msg.conversationId,
spanName: "customer-support.run",
},
async (span) => { /* ... */ },
);
```
Stable IDs matter more than human-friendly ones. Prefer `support-acme-prod`
over `support-acme-2024-v2` — the Agents dashboard uses the ID to group runs
across deploys.
### Background Jobs Triggered From The Agent [#background-jobs-triggered-from-the-agent]
If your tool launches a background job that itself does LLM work, capture
the active identity and pass it into the job so the background span can be
filtered together with its originating conversation:
<Metadata text="integrations/traces/cookbook-background" />
```typescript TypeScript
import { getActiveAgentIdentity } from "@inference/tracing";
async function executeTool_enqueueReport(args: ToolArgs): Promise<ToolResult> {
const identity = getActiveAgentIdentity();
await jobQueue.enqueue("generate-report", {
...args,
contextAgentId: identity?.id,
contextSessionId: identity?.id ? identity.id : undefined,
});
return { ok: true };
}
```
The background worker can then set `agent.id` and `session.id` on its own
agent span so the two pieces of work share dashboard grouping.
### Streaming Responses [#streaming-responses]
When the agent streams output back to the user, set the span output once at
the end, after the stream completes. The patched SDK already handles
streaming LLM calls correctly; the outer agent span just needs the final
text:
<Metadata text="integrations/traces/cookbook-streaming" />
```typescript TypeScript
const result = await agentSpan(options, async (span) => {
span.setInput(msg.text);
let final = "";
for await (const chunk of streamAgent(msg)) {
final += chunk;
streamToClient(chunk); // forward each chunk to the client as it arrives
}
span.setOutput(final);
return final;
});
```
### Custom Span Events [#custom-span-events]
For mid-callback events that are not span attributes — a rate-limit retry,
a fallback to a smaller model, a cache miss — use `span.raw.addEvent`:
<Metadata text="integrations/traces/cookbook-events" />
```typescript TypeScript
span.raw.addEvent("rate_limit_retry", {
attempt: 2,
retry_after_ms: 1500,
});
```
Events appear under the `--view events` flag of `inf span get` and on the
span detail page.
## What To Test [#what-to-test]
| Behavior | How to verify |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `setup()` runs before the first SDK call | Search server logs for the Inference Tracing init message; confirm it precedes any Anthropic request log. |
| LLM spans parent under the agent span | `inf trace get <id> --view tree` shows a single AGENT root with LLM and TOOL children. |
| Tool span has `tool.name` and `tool_call.id` | `inf span get <id> --view attributes` |
| Errors mark the span `ERROR` | Force a tool to throw; confirm the span status is `ERROR` and the trace status is `ERROR`. |
| Spans flush on `SIGTERM` | Send `SIGTERM` to the server right after a request; the trace should still appear in the Inference platform. |
| Domain attributes are filterable | `inf trace list --metadata "app.tenant_id=acme"` returns the expected traces. |
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
The full surface for AGENT, TOOL, CHAIN, and RETRIEVER spans.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
All `Attr.*` constants and `SpanKindValues` with the attributes each kind expects.
</Card>
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
Every method on the span handle and how it coerces values.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
Debug missing spans, missing attributes, and shutdown behavior.
</Card>
</CardGroup>
# Pydantic AI Traces (https://docs.inference.net/integrations/traces/pydantic-ai)
Pydantic AI ships native OpenTelemetry instrumentation. Inference platform registers its
TracerProvider and enables Pydantic AI instrumentation during `setup()`.
## Install [#install]
<Metadata text="integrations/traces/pydantic-ai-install" />
```bash Python
pip install 'inference-tracing[pydantic-ai]'
```
## Structured Agent With Tools [#structured-agent-with-tools]
<Metadata text="integrations/traces/pydantic-ai-agent" />
```python Python
from inference_tracing import agent_span, setup
from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext
class CityWeather(BaseModel):
city: str
temp_c: float = Field(description="Temperature in Celsius.")
condition: str
class WeatherReport(BaseModel):
cities: list[CityWeather]
summary: str = Field(description="One sentence comparing the conditions.")
tracing = setup(service_name="weather-agent")
agent = Agent(
"openai:gpt-4o-mini",
output_type=WeatherReport,
system_prompt="Use get_weather for every requested city.",
)
@agent.tool
def get_weather(_ctx: RunContext[None], city: str) -> str:
"""Look up current weather for a city."""
weather = {
"Paris": {"temp_c": 12, "condition": "overcast"},
"Tokyo": {"temp_c": 18, "condition": "sunny"},
}
record = weather.get(city, {"temp_c": 0, "condition": "unknown"})
return (
f'{{"city": "{city}", "temp_c": {record["temp_c"]}, '
f'"condition": "{record["condition"]}"}}'
)
with agent_span(
tracing.tracer,
agent_id="weather-agent",
agent_name="Weather Agent",
span_name="weather-agent.run",
session_id="conversation-weather-paris",
agent_role="weather",
system="pydantic-ai",
) as span:
user_input = "What's the weather in Paris and Tokyo?"
span.set_input(user_input)
result = agent.run_sync(user_input)
span.set_output(result.output.model_dump())
print(result.output.summary)
tracing.shutdown()
```
## What To Look For [#what-to-look-for]
* Agent run spans from Pydantic AI
* An outer AGENT span with `agent.id=weather-agent` when you use the wrapper
* Tool spans for `get_weather`
* Model spans from the provider used by Pydantic AI
* Structured `WeatherReport` output in the captured span data
# Traces Quickstart (https://docs.inference.net/integrations/traces/quickstart)
This page is the SDK-focused quickstart: install the [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) (TypeScript) or [`inference-tracing`](https://pypi.org/project/inference-tracing/) (Python) package, point it at the Inference platform, and capture your first span. If you'd rather see the higher-level flow, start with [Capture your first trace](/get-started/capture-first-trace).
The example below uses OpenAI because it's the smallest end-to-end trace. The same export configuration applies to Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, ElevenLabs Agents, Vercel Eve, Pi Agent, legacy PI AI, Cursor SDK, Claude Agent SDK, Pydantic AI, the Vercel AI SDK, and manual spans. Each framework guide shows the exact setup hook for that SDK.
## Choose a setup path [#choose-a-setup-path]
Installing with AI is the quickest. Use the manual flow if you want to wire it up yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to launch a coding agent like [Claude Code](https://code.claude.com/docs/en/overview), OpenCode, or Codex to install the tracing SDK, configure export, and wire up your LLM clients.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata text="integrations/traces/quickstart-ai-auth" />
```bash
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run tracing instrumentation in your project">
From your project root, run instrumentation in tracing mode.
<Metadata text="integrations/traces/quickstart-ai-instrument" />
```bash
cd /path/to/your/project && inf instrument --mode tracing
```
The command guides you through the following workflow:
* Select a coding agent: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients and agent frameworks.
* Install `@inference/tracing` or `inference-tracing` plus the right per-integration extras.
* Wire `setup()` into your app entrypoint so spans start before clients are constructed.
* Add stable service and agent identity so traces group cleanly in the dashboard.
* Review the generated changes before applying them.
<Tip>
Pick `both` instead of `tracing` to also route requests through the Inference Gateway in the same pass.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would. Traces stream to the Inference platform as your code executes.
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and filter by your service name to see the captured trace tree.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to wire it up yourself. The example below uses OpenAI. For other providers and frameworks, see the [per-integration guides](/integrations/traces/overview#supported-trace-integrations).
<Steps>
<Step title="Install the SDK">
Provider and framework SDKs are optional peers. Install the ones you use alongside the tracing package. For Python, add per-integration extras to the install string.
<CodeGroup>
<Metadata text="integrations/traces/quickstart-install-typescript" />
```bash TypeScript
# Pick your package manager
bun add @inference/tracing openai
npm install @inference/tracing openai
pnpm add @inference/tracing openai
```
<Metadata text="integrations/traces/quickstart-install-python" />
```bash Python
pip install 'inference-tracing[openai]'
# Multiple integrations at once
pip install 'inference-tracing[openai,anthropic,langchain]'
# Everything
pip install 'inference-tracing[all]'
```
</CodeGroup>
Available Python extras: `openai`, `anthropic`, `langchain`, `langgraph`, `langsmith`, `openai-agents`, `claude-agent-sdk`, `pydantic-ai`, `elevenlabs`, `livekit-agents`, `all`.
</Step>
<Step title="Configure export">
Set the Inference platform traces endpoint and token before your app starts.
<Metadata text="integrations/traces/quickstart-env" />
```bash
export INFERENCE_OTLP_ENDPOINT="https://telemetry.inference.net"
# Get your API key from https://inference.net/dashboard/api-keys/
export INFERENCE_API_KEY="<your-token>"
export INFERENCE_SERVICE_NAME="checkout-agent"
```
<Tip>
Use a stable `INFERENCE_SERVICE_NAME` per deployed service. It makes traces easier to filter and compare across environments.
</Tip>
You can also pass these as options to `setup()` instead of env vars. See the [configuration reference](/integrations/traces/overview#configuration).
</Step>
<Step title="Initialize tracing early">
Call `setup()` before constructing clients from instrumented SDKs. In TypeScript, pass the SDK modules you want patched. In Python, `setup()` auto-detects installed packages.
<CodeGroup>
<Metadata text="integrations/traces/quickstart-openai" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Reply with just the word hello." }],
max_tokens: 16,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata text="integrations/traces/quickstart-openai" />
```python Python
import os
from inference_tracing import setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply with just the word hello."}],
max_tokens=16,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
If the process is short-lived, always call `shutdown()` before exit so batched spans are flushed.
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and navigate to the Agents or Trajectories tab. You'll see an LLM span with input messages, output messages, model name, invocation parameters, finish reason, and token counts.
</Step>
</Steps>
<Note>
Need a different provider or framework? See the [supported integrations](/integrations/traces/overview#supported-trace-integrations) list.
</Note>
</Tab>
</Tabs>
That's it. Spans are streaming to the Inference platform and your first trace is ready to inspect.
What you have so far is one LLM span per call, captured automatically. That's enough for a one-shot script, but real apps usually run several calls per user request, and you'll want those grouped under a named agent and session in the dashboard. That's the next step. If you used Install with AI, the agent likely already wired this up for you; read on to see what it set up and why.
## Group calls under an agent [#group-calls-under-an-agent]
The example above is a one-shot LLM call. Once your app runs multiple LLM calls as part of a logical unit (an agent run, a conversation turn, a workflow), wrap that unit in `agentSpan` so the LLM spans nest under an `AGENT` row carrying `agent.id`, `agent.name`, and `session.id`. The Agents dashboard groups on those attributes.
<CodeGroup>
<Metadata text="integrations/traces/quickstart-agent-typescript" />
```typescript TypeScript
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
await agentSpan(
{
agentId: "hello-agent",
agentName: "Hello Agent",
sessionId: "session-001",
},
async (span) => {
span.setInput("Reply with just the word hello.");
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Reply with just the word hello." }],
max_tokens: 16,
});
span.setOutput(response.choices[0]?.message.content ?? "");
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/quickstart-agent-python" />
```python Python
import os
from inference_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="hello-agent",
agent_name="Hello Agent",
session_id="session-001",
) as span:
span.set_input("Reply with just the word hello.")
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply with just the word hello."}],
max_tokens=16,
)
span.set_output(response.choices[0].message.content or "")
tracing.shutdown()
```
</CodeGroup>
The OpenAI LLM span still appears, but now nested under your `agentSpan` row instead of as an orphan. Real agents run many LLM calls per session; the outer span is what makes them findable as one thing.
## Wrap your own code [#wrap-your-own-code]
For non-LLM steps inside an agent loop (a tool call, a retrieval, a custom router, an evaluator, a CLI subprocess), wrap them with `manualSpan`. Combined with the `agentSpan` above, you get a full trace tree: an outer AGENT row, inner SDK rows, and inner manual rows, all parented correctly.
<CodeGroup>
<Metadata text="integrations/traces/quickstart-manual-typescript" />
```typescript TypeScript
import {
SpanKindValues,
agentSpan,
manualSpan,
setup,
} from "@inference/tracing";
const tracing = await setup();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
},
async (span) => {
span.setInput("Review refund request #1842");
const decision = await runRefundReview();
span.setOutput(decision.summary);
},
);
// manualSpan authors TOOL / CHAIN / RETRIEVER / EMBEDDING spans.
await manualSpan(
{
spanName: "rag.retrieve",
spanKind: SpanKindValues.RETRIEVER,
input: { query, k: 8 },
},
async (span) => {
const docs = await retrieve(query);
span.setOutput(docs);
},
);
await tracing.shutdown();
```
<Metadata text="integrations/traces/quickstart-manual-python" />
```python Python
from inference_tracing import (
SpanKindValues,
agent_span,
manual_span,
setup,
)
tracing = setup()
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
span_name="refund-review.run",
) as span:
span.set_input("Review refund request #1842")
decision = run_refund_review()
span.set_output(decision.summary)
# manual_span authors TOOL / CHAIN / RETRIEVER / EMBEDDING spans.
with manual_span(
tracing.tracer,
name="rag.retrieve",
span_kind=SpanKindValues.RETRIEVER,
input={"query": query, "k": 8},
) as span:
docs = retrieve(query)
span.set_output(docs)
tracing.shutdown()
```
</CodeGroup>
For the full manual-span surface (tools, retrievers, embeddings, agent identity), see [Manual spans](/integrations/traces/manual-spans) and [Agent identity](/integrations/traces/agent-identity).
## Flushing and process lifecycle [#flushing-and-process-lifecycle]
Spans are batched and exported in the background, so a process that exits or
freezes before the batch flushes drops them. How you flush depends on the
process shape:
* **Short-lived script:** call `await tracing.shutdown()` before exit. It
force-flushes, then tears the provider down. The examples above do this.
* **Long-lived service** (HTTP server, Slack bot, queue worker): call `setup()`
**once per process** before the first SDK client is constructed, memoize the
result so any handler can `await` it, and call `shutdown()` only on `SIGTERM`.
Never per request, since that forces a synchronous flush and adds latency.
* **Serverless or edge** (Lambda, Cloudflare Workers): memoize `setup()` the
same way, but flush per invocation with `tracing.provider.forceFlush()`
instead of `shutdown()`, since the provider must survive for the next warm
invocation.
### Long-lived service [#long-lived-service]
<CodeGroup>
<Metadata text="integrations/traces/quickstart-server-typescript" />
```typescript TypeScript
// tracing.ts — memoized setup
import OpenAI from "openai";
import { setup, type CatalystTracing } from "@inference/tracing";
let tracingPromise: Promise<CatalystTracing> | null = null;
export function initTracing(): Promise<CatalystTracing> {
if (!tracingPromise) {
tracingPromise = setup({ modules: { openai: OpenAI } });
}
return tracingPromise;
}
export async function shutdownTracing(): Promise<void> {
if (!tracingPromise) return;
const tracing = await tracingPromise;
await tracing.shutdown();
}
```
```typescript TypeScript (server entrypoint)
// server.ts
import { initTracing, shutdownTracing } from "./tracing.ts";
await initTracing(); // patches OpenAI before the first client is constructed
const server = startServer();
for (const signal of ["SIGTERM", "SIGINT"] as const) {
process.on(signal, async () => {
await shutdownTracing();
server.close(() => process.exit(0));
});
}
```
<Metadata text="integrations/traces/quickstart-server-python" />
```python Python
# tracing.py — memoized setup
from threading import Lock
from inference_tracing import CatalystTracing, setup
_tracing: CatalystTracing | None = None
_lock = Lock()
def get_tracing() -> CatalystTracing:
global _tracing
if _tracing is None:
with _lock:
if _tracing is None:
_tracing = setup()
return _tracing
def shutdown_tracing() -> None:
if _tracing is not None:
_tracing.shutdown()
```
```python Python (server entrypoint)
# server.py
import signal
from tracing import get_tracing, shutdown_tracing
get_tracing() # registers instrumentation before app code runs
def handle_signal(_signum, _frame):
shutdown_tracing()
raise SystemExit(0)
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
run_server()
```
</CodeGroup>
### Serverless and edge runtimes [#serverless-and-edge-runtimes]
On Lambda, Cloudflare Workers, or any runtime that freezes the process between
invocations, the background batch processor may never run, so spans are dropped.
Memoize `setup()` the same way as a long-lived service (the provider is reused
across warm invocations), but flush at the end of **each invocation** with
`tracing.provider.forceFlush()` rather than calling `shutdown()`. Reserve
`shutdown()` for real process teardown, since it tears down the provider the
next warm invocation needs.
<CodeGroup>
<Metadata text="integrations/traces/quickstart-serverless-typescript" />
```typescript TypeScript
import { initTracing } from "./tracing.ts";
export async function handler(event: { message: string }) {
const tracing = await initTracing(); // memoized setup(), patches once
const reply = await answerQuestion(event.message);
// Flush before the runtime freezes the process. Do not shutdown(): the next
// warm invocation reuses this provider.
await tracing.provider.forceFlush();
return reply;
}
```
<Metadata text="integrations/traces/quickstart-serverless-python" />
```python Python
from tracing import get_tracing
def handler(event):
tracing = get_tracing() # memoized setup(), patches once
reply = answer_question(event["message"])
# Flush before the runtime freezes the process. Do not shutdown(): the next
# warm invocation reuses this provider.
tracing.provider.force_flush()
return reply
```
</CodeGroup>
### Selective instrumentation [#selective-instrumentation]
`setup()` auto-instruments every supported SDK it detects. If you want
explicit control — for example, to instrument OpenAI but skip LangChain — set
`autoInstrument: false` and call the targeted helper yourself:
<Metadata text="integrations/traces/quickstart-selective-typescript" />
```typescript TypeScript
import { setup } from "@inference/tracing";
import { instrumentOpenAI } from "@inference/tracing/openai";
import OpenAI from "openai";
const tracing = await setup({ autoInstrument: false });
instrumentOpenAI(OpenAI, tracing);
```
The per-integration entry points are listed in the
[overview's configuration section](/integrations/traces/overview#configuration).
For a full production-shaped server with custom tool spans and domain
attributes, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## Verify [#verify]
Open the Inference.net dashboard and navigate to the Agents or Trajectories tab. The trace should include an OpenAI LLM span with input messages, output messages, model name, invocation parameters, finish reason, and token counts. Any custom spans you added show up as parent or sibling nodes in the trace tree.
If you don't see anything, see [Troubleshooting](/integrations/traces/troubleshooting).
## Next Steps [#next-steps]
<CardGroup cols="2">
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Walk trace trees in the dashboard and run Halo to find what to improve.
</Card>
<Card title="OpenAI tracing" icon="sparkles" href="/integrations/traces/openai">
Add tool calls, structured outputs, and Responses API examples.
</Card>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Wrap custom agents, CLI calls, and unsupported SDKs.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so the Agents dashboard groups runs correctly.
</Card>
</CardGroup>
# Traces Troubleshooting (https://docs.inference.net/integrations/traces/troubleshooting)
Use this checklist when traces do not appear in the Inference platform or when spans are
missing expected attributes.
## No Spans Appear [#no-spans-appear]
### Confirm Export Configuration [#confirm-export-configuration]
<Metadata text="integrations/traces/troubleshooting-env" />
```bash
echo "$INFERENCE_OTLP_ENDPOINT"
echo "$INFERENCE_API_KEY"
echo "$INFERENCE_SERVICE_NAME"
```
### Enable Debug Logging [#enable-debug-logging]
<Metadata text="integrations/traces/troubleshooting-debug" />
```bash
export INFERENCE_DEBUG=true
```
### Initialize Before Clients [#initialize-before-clients]
Call `setup()` before constructing provider or framework clients. Many
integrations patch shared prototypes or callback configuration paths.
<Metadata text="integrations/traces/troubleshooting-setup-order" />
```typescript TypeScript
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
```
## Spans Disappear In Short-Lived Processes [#spans-disappear-in-short-lived-processes]
Batch exporters flush on an interval. CLIs, scripts, workers, and tests should
call `shutdown()` before the process exits.
<CodeGroup>
<Metadata text="integrations/traces/troubleshooting-shutdown" />
```typescript TypeScript
await tracing.shutdown();
```
<Metadata text="integrations/traces/troubleshooting-shutdown" />
```python Python
tracing.shutdown()
```
</CodeGroup>
### 415 Unsupported Media Type [#415-unsupported-media-type]
The ingest endpoints accept OTLP/**JSON only** (`Content-Type:
application/json`). The SDKs' built-in exporters send JSON by default, but a
custom exporter (for example an OpenTelemetry Collector) defaults to
OTLP/protobuf, which the ingest endpoint rejects with `415 Unsupported Media
Type`. Force JSON encoding on your exporter, such as `encoding: json` on an
`otlphttp` collector exporter. Do the same for the
[Langfuse OTEL routes](/integrations/traces/langfuse), which also accept JSON
only.
## Framework Spans Are Missing [#framework-spans-are-missing]
| Integration | Check |
| ------------------------ | --------------------------------------------------------------------------- |
| LangChain / LangGraph JS | Pass `@langchain/core/callbacks/manager` as a namespace to `setup()` |
| LangSmith | Set `LANGSMITH_TRACING=true` and use `otel` or `hybrid` mode |
| Claude Agent SDK TS | Use `wrapClaudeAgentSdkQuery(query)` |
| OpenAI Agents | Pair `openaiAgents` with `openai` in `setup()` |
| Vercel Eve | Export `defineCatalystEveInstrumentation()` from `agent/instrumentation.ts` |
## Attributes Are Missing [#attributes-are-missing]
Some attributes only appear when the upstream provider returns the data. For
example, token counts depend on provider usage fields, and Anthropic prompt-cache
details appear only when prompt caching actually engages for the request.
## Next Steps [#next-steps]
If setup order, export configuration, and shutdown are correct, reduce the issue
to the smallest provider call and compare it with [Traces Quickstart](/integrations/traces/quickstart).
# Inference platform by Inference.net (https://docs.inference.net/introduction)
Inference platform is a platform for understanding, evaluating, and improving the AI systems you ship.
[Create an account](https://inference.net/register) to get started with the Inference platform.
## The Inference platform [#the-inference-platform]
Inference platform is built around two products. Use them on their own or together.
<CardGroup cols="2">
<Card title="Tracing" icon="route" href="/get-started/capture-first-trace">
Capture full traces of every LLM call, tool call, and agent step. Then let Halo, our open-source agent-loop optimizer, read your traces and tell you what to fix in prompts, tools, and the harness itself.
</Card>
<Card title="Gateway" icon="satellite-dish" href="/get-started/record-first-call">
Route LLM traffic through Inference Gateway to capture every request, watch usage and cost, build datasets, run evals, and fine-tune and deploy task-specific models on dedicated GPUs.
</Card>
</CardGroup>
Tracing is for agent improvement. Gateway is for inference observability and training task-specific models from your captured traffic. Most teams start with one and add the other later. They also compose: when your agent calls LLMs through Gateway, trace spans line up with the same requests Gateway is recording.
## Quick Start [#quick-start]
### Tracing [#tracing]
<CardGroup cols="2">
<Card title="Capture your first trace" icon="route" href="/get-started/capture-first-trace">
Install the tracing SDK and capture LLM calls, tool calls, and agent steps in minutes.
</Card>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Inspect trace trees in the dashboard and run Halo to find what to improve.
</Card>
<Card title="Label traffic with Signals" icon="signal-stream" href="/get-started/signals">
Define plain-language classifiers that turn your agent's traffic into quality metrics.
</Card>
<Card title="Compare Models with AutoEvals" icon="scale-balanced" href="/get-started/compare-models-auto-evals">
Sample production traffic, replay it against candidate models, and find a better fit.
</Card>
</CardGroup>
### Gateway [#gateway]
<CardGroup cols="2">
<Card title="Record your first LLM call" icon="bolt" href="/get-started/record-first-call">
Route traffic through Inference Gateway to capture LLM calls and view metrics.
</Card>
<Card title="Compare models with AutoEvals" icon="scale-balanced" href="/get-started/compare-models-auto-evals">
Replay your traffic against candidate models and get a recommended winner.
</Card>
<Card title="Train and deploy a model" icon="brain" href="/get-started/train-and-deploy">
The full loop: data, training, and a production endpoint.
</Card>
<Card title="Use the Inference API" icon="code" href="/api/api-quickstart">
Call open-source, popular closed-source, and custom models with one API key.
</Card>
</CardGroup>
## Inference platform [#inference-platform]
<CardGroup cols="2">
<Card title="Tracing" icon="route" href="/integrations/traces/overview">
Collect OpenInference-shaped traces from LLM SDKs, agent frameworks, and custom application code.
</Card>
<Card title="Gateway" icon="satellite-dish" href="/platform/gateway/overview">
Capture your LLM traffic and get metrics and visibility into your production usage.
</Card>
<Card title="Datasets" icon="file-code" href="/platform/datasets/overview">
Create and manage datasets for evaluation and training.
</Card>
<Card title="Evaluate" icon="scale-balanced" href="/platform/eval/overview">
Compare models on your live traffic with AutoEvals to measure quality across candidates.
</Card>
<Card title="Train" icon="brain" href="/platform/train/overview">
Train a custom model on your production data to improve performance, lower latency and cost.
</Card>
<Card title="Deploy" icon="rocket" href="/platform/deploy/overview">
Deploy a model to a dedicated GPU to use in production.
</Card>
</CardGroup>
# Build a Dataset from Traffic (https://docs.inference.net/platform/datasets/build-from-traffic)
The most useful datasets come from real production traffic. Inference platform lets you filter your captured inferences and save the results as a dataset, ready for evals or training.
You can create a dataset from traffic in two places: the **Create Dataset** button in the Datasets tab, or **Save as Dataset** in the [Inference Viewer](/platform/gateway/inference-viewer). Both follow the same flow.
## The flow [#the-flow]
<Steps>
<Step title="Filter your traffic">
Filter by model, task, provider, status code, or any tracked dimension until you have a representative slice of traffic.
</Step>
<Step title="Choose a dataset type">
Decide whether this will be an **eval dataset** or a **training dataset**. Remember the [zero-overlap rule](/platform/datasets/overview#the-zero-overlap-rule), training and eval data must never share examples.
</Step>
<Step title="Save as dataset">
Name the dataset and save. It's immediately available for evals or training.
</Step>
</Steps>
## Getting clean samples [#getting-clean-samples]
The quality of your dataset depends on how well you filter. A few tips:
* **Filter by [task](/platform/gateway/tasks)** to get samples for a specific objective rather than a mix of everything
* **Errors are opt-in** — only successful inferences (2xx) are included by default. Add error status codes to the filter only when you specifically want failure cases (e.g. for training a model to handle edge cases)
* **Check the date range** - a dataset pulled from a single day might not capture the full variety of inputs your app sees
## Eval vs training: different goals, different data [#eval-vs-training-different-goals-different-data]
**Eval datasets** should be small, stable, and challenging. Pick examples that represent the hard cases — the ones where you're not sure the model will get it right. These become your benchmark, so don't change them often.
**Training datasets** should be large, diverse, and representative. The more variety, the better the model generalizes. Iterate on these as you learn what the model struggles with.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Upload your own data" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have curated data? Upload it directly.
</Card>
<Card title="Dataset formats" icon="file-code" href="/platform/datasets/formats">
Supported schemas and validation rules.
</Card>
</CardGroup>
# Dataset Formats and Schemas (https://docs.inference.net/platform/datasets/formats)
Datasets can be created from [captured traffic](/platform/datasets/build-from-traffic) or [uploaded as JSONL files](/platform/datasets/upload-a-dataset). This page covers the supported formats.
## Supported formats [#supported-formats]
The system auto-detects the format from the first valid line. All rows in a file must use the same format.
<Warning>
You cannot mix source-backed and Hugging Face rows in the same file. Mixed-format files fail validation.
</Warning>
### Source-backed format [#source-backed-format]
Each line has a top-level `request` and optional `response` object containing raw provider bodies.
| Field | Required | Description |
| ---------- | -------- | --------------------------------------------------------------- |
| `request` | Yes | Raw provider request body that includes a usable `model` value |
| `response` | No | Raw provider response body, or `null` if you only have requests |
Validation notes:
* The request must include a usable model value.
* `response` may be omitted or set to `null` if you only have request-side data.
<Metadata text="datasets/format-source-backed" />
```json
{"request":{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}],"temperature":0.7,"max_tokens":100},"response":{"id":"chatcmpl-123","object":"chat.completion","created":1700000000,"model":"gpt-4","choices":[{"index":0,"message":{"role":"assistant","content":"Hi there!"},"finish_reason":"stop"}]}}
```
#### Tag rows with a task [#tag-rows-with-a-task]
Source-backed rows can be assigned to a [task](/platform/gateway/create-a-task) by setting `x-inference-task-id` inside `request.metadata`. Rows that share a task ID are grouped under that task in the dashboard, exactly like live gateway traffic tagged with the `x-inference-task-id` header.
<Metadata text="datasets/format-source-backed-task-id" />
```json
{"request":{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}],"metadata":{"x-inference-task-id":"support-summaries"}},"response":{"choices":[{"index":0,"message":{"role":"assistant","content":"Hi there!"},"finish_reason":"stop"}]}}
```
<Note>
There is no upload-level task flag — the task ID must be present on **every line** you want grouped. Lines without the metadata key land outside any task. (The `--task` flag on `inf dataset create` filters captured traffic by task; it does not tag uploads.)
</Note>
#### Multimodal rows [#multimodal-rows]
Requests may include image content using standard OpenAI-style `image_url` content parts — either `data:` base64 URLs or public HTTPS URLs. Image parts are preserved through the import unchanged.
<Metadata text="datasets/format-source-backed-images" />
```json
{"request":{"model":"google/gemma-3-27b-instruct/bf-16","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image"},{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,/9j/4AAQ..."}}]},"metadata":{"x-inference-task-id":"image-descriptions"}},"response":{"choices":[{"index":0,"message":{"role":"assistant","content":"A photo of ..."},"finish_reason":"stop"}]}}
```
### Hugging Face format [#hugging-face-format]
Each line has a top-level `messages` array with `role`/`content` objects.
| Field | Required | Description |
| ---------- | -------- | ------------------------------------------------------------ |
| `messages` | Yes | Array of `{ role, content }` objects (at least one required) |
| `id` | No | Optional row identifier (stored in metadata) |
| `tools` | No | Optional top-level tool definitions |
Valid roles: `system`, `user`, `assistant`, `tool`.
Additional supported fields:
* `content` may be a string or an array of content parts for multimodal rows.
* Assistant messages may include `tool_calls`.
* Tool messages must include `tool_call_id`.
* Top-level `tools` are preserved on import.
<Metadata text="datasets/format-huggingface" />
```json
{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 2+2?"},{"role":"assistant","content":"4"}]}
```
When importing Hugging Face-format rows, the system:
* Treats the last assistant turn as the imported response and earlier turns as request context
* Synthesizes request/response payloads so evals and detail views work
* Sets `request_model` to `unknown-imported-model`
* Sets token usage and costs to zero
* Stores the original row `id` in metadata as `importOriginalRowId`
## Validation behavior [#validation-behavior]
* Files must be valid JSONL.
* Invalid rows are reported with line numbers in the upload status details.
* Uploads can complete with partial failures if at least one row imports successfully.
* If every row fails validation, the upload status is `failed`.
## Upload limits [#upload-limits]
| Limit | Value |
| ------------------ | --------- |
| Maximum file size | 100 GB |
| Maximum line count | 5,000,000 |
## Download formats [#download-formats]
Datasets can be downloaded in two formats:
| Format | Description | Best for |
| ----------------- | ------------------------------------ | -------------------------------- |
| **Hugging Face** | `{ id, messages }` per row (default) | Training, fine-tuning, sharing |
| **Source-backed** | `{ request, response }` per row | Re-uploading, round-trip testing |
In the UI, click **Download** and choose the format. In the CLI:
<Metadata text="datasets/download-cli" />
```bash
# Default (Hugging Face)
inf dataset download my-dataset
# Source-backed format
inf dataset download my-dataset --format source-backed
```
<Note>
Hugging Face exports skip rows with empty message arrays. Source-backed exports include all rows with a valid request payload. Row counts may differ between formats.
</Note>
# Datasets (https://docs.inference.net/platform/datasets/overview)
Datasets are collections of LLM inputs and outputs used for evaluation and fine-tuning. They can come from two places: your live production traffic captured through [Gateway](/platform/gateway/overview), or files you [upload directly](/platform/datasets/upload-a-dataset).
Everything downstream depends on good data. Evals need representative examples to measure model quality. Training needs diverse, high-quality samples to teach a model your task. Datasets are where both start.
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/WIohm_V4aHo?list=PLJzp7SN2tfJsRAU9VGSfSo60CyDJzqhLP&rel=0" title="Turn Production Traffic Into LLM Training Data | Inference platform" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Types of datasets [#types-of-datasets]
| Type | Purpose | How it evolves |
| -------------------- | --------------------------------------------- | ---------------------------------------------------------------------------- |
| **Eval dataset** | Measures model quality against a rubric | Stays stable, a fixed set of challenging examples that act as your benchmark |
| **Training dataset** | Data the model learns from during fine-tuning | Changes often as you iterate on data quality and coverage |
### The zero-overlap rule [#the-zero-overlap-rule]
Inference platform automatically enforces zero-overlap between training and eval datasets. If a training dataset overlaps with an eval dataset, the overlapping data will be excluded from the training dataset when a new training run is created.
## Key concepts [#key-concepts]
| Concept | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Build from traffic** | Filter your captured production inferences and save them as a dataset. The best datasets come from real usage. |
| **Upload** | Bring your own JSONL files when you have curated data or are migrating from another platform. |
| **Dataset format** | The schema your data needs to follow. See [Dataset Formats](/platform/datasets/formats) for supported fields and validation rules. |
| **Task tags** | Use [task tags](/platform/gateway/tasks) when building from traffic to filter by objective. This gives you clean, focused samples instead of mixed traffic. |
## Tips for good datasets [#tips-for-good-datasets]
* **Diverse training data** leads to models that generalize well. If your training data isn't heterogeneous, the trained model won't handle edge cases.
* **Stable eval data** gives you a consistent benchmark. Don't change your eval dataset frequently, it's the measuring stick.
* **Start with production traffic** when possible. Real user inputs reflect the actual distribution of requests your model will see, and they're harder to fake than synthetic data.
* **Use task tags** to filter by objective before saving a dataset. A dataset scoped to a single task is almost always more useful than one built from mixed traffic.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Build from traffic" icon="satellite-dish" href="/platform/datasets/build-from-traffic">
Turn filtered production traffic into a dataset.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Bring your own JSONL files.
</Card>
<Card title="Compare models with AutoEvals" icon="scale-balanced" href="/get-started/compare-models-auto-evals">
Replay your traffic against candidate models and get a recommended winner.
</Card>
<Card title="Train a custom model" icon="brain" href="/get-started/train-and-deploy">
Use your dataset to fine-tune a task-specific model.
</Card>
</CardGroup>
# Upload a Dataset (https://docs.inference.net/platform/datasets/upload-a-dataset)
If you already have curated data from annotation pipelines, synthetic generation, or another platform, you can upload it as JSONL instead of building a dataset from captured traffic.
Uploads and datasets are separate objects in the Inference platform:
* An **upload** is the imported JSONL file plus its validation and processing status.
* A **dataset** is the stable collection you use for evals, training, and download.
You upload the file first, then create an **eval** or **training** dataset from that uploaded data once processing finishes.
## How to upload [#how-to-upload]
<Tabs>
<Tab title="Dashboard">
1. Go to **Datasets** in the dashboard
2. Click **Upload Data**
3. Select your `.jsonl` file
4. Give the upload a name and start the import
The upload appears in **Datasets > Uploads**, where you can track processing and review any validation errors.
</Tab>
<Tab title="CLI">
<Metadata text="platform/datasets/upload-cli" />
```bash
inf dataset upload path/to/data.jsonl --name support-summaries
```
If you omit `--name`, the CLI uses the filename without the extension. By default the CLI waits for processing and prints the detected format plus the processed line count. Use `--no-wait` to return after the transfer completes.
</Tab>
</Tabs>
<Note>
The upload command does not ask whether the data is for evals or training. You choose **eval** vs **training** when you create a dataset from the completed upload.
</Note>
## After upload [#after-upload]
1. Wait for the upload to finish processing in **Datasets > Uploads**.
2. Open the dataset creation flow and select the upload as your source.
3. Choose whether the resulting dataset is **eval** or **training**.
Successful uploads become a reusable source in the same dataset creation flow you use for traffic-backed datasets.
## Supported formats [#supported-formats]
Two JSONL formats are supported. See [Dataset Formats](/platform/datasets/formats) for full schemas, required fields, and validation rules.
| Format | Structure | Best for |
| ----------------- | -------------------------------- | --------------------------------------------- |
| **Source-backed** | `{ request, response }` per line | Round-tripping data captured from providers |
| **Hugging Face** | `{ messages }` per line | Standard training/eval format, easy to create |
The system auto-detects the format from the first valid line. Every row in the file must use the same format.
## Tag uploaded rows with a task [#tag-uploaded-rows-with-a-task]
Source-backed rows can populate a [task](/platform/gateway/create-a-task) on the dashboard. Set `x-inference-task-id` in each line's `request.metadata` — rows sharing a task ID are grouped under that task in your project's **Tasks** view, the same way live gateway traffic is grouped by the `x-inference-task-id` header.
<Metadata text="platform/datasets/upload-task-id" />
```json
{"request":{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}],"metadata":{"x-inference-task-id":"support-summaries"}},"response":{"choices":[{"index":0,"message":{"role":"assistant","content":"Hi!"},"finish_reason":"stop"}]}}
```
Uploaded requests may also include images as standard OpenAI-style `image_url` content parts (base64 `data:` URLs or public HTTPS URLs) — see [Dataset Formats](/platform/datasets/formats#multimodal-rows) for a full multimodal example.
<Note>
* There is no `--task` flag on `inf dataset upload`; the task ID lives in the data itself and must be on every line you want grouped.
* Uploaded rows are timestamped at processing time, not with historical timestamps from your data — the task's time-series charts show the whole batch at upload time.
</Note>
## Validation behavior [#validation-behavior]
* Invalid rows are reported with line numbers in the upload status details.
* Uploads can complete with some failed rows if at least one row imports successfully.
* Mixed-format files are treated as a fatal error and fail the upload.
* Source-backed rows must include a usable model value in the request.
## Upload limits [#upload-limits]
| Limit | Value |
| ------------------ | --------- |
| Maximum file size | 100 GB |
| Maximum line count | 5,000,000 |
## Next steps [#next-steps]
<CardGroup cols="3">
<Card title="Build from traffic instead" icon="satellite-dish" href="/platform/datasets/build-from-traffic">
Pull datasets directly from your captured production traffic.
</Card>
<Card title="CLI Command Reference" icon="terminal" href="/cli/datasets">
Upload from the terminal with `inf dataset upload`.
</Card>
<Card title="Dataset formats reference" icon="file-code" href="/platform/datasets/formats">
Full schema details and validation rules.
</Card>
</CardGroup>
# Call Your Deployment (https://docs.inference.net/platform/deploy/call-your-deployment)
Deployments expose an OpenAI-compatible chat completions endpoint. Point any OpenAI SDK client at the Inference base URL and set the `model` to your deployment's model path.
<CodeGroup>
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "acme-corp/my-model",
messages: [{ role: "user", content: "Hello, world!" }],
});
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="acme-corp/my-model",
messages=[{"role": "user", "content": "Hello, world!"}],
)
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "acme-corp/my-model",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
Structured outputs, function calling, and other chat completions features all work the same way. You can also tag requests with `x-inference-task-id` to group calls by objective — see [Tasks](/platform/gateway/tasks) for details.
## Where to find your model path [#where-to-find-your-model-path]
The model path is shown on your deployment's detail page in the dashboard. It's your team slug followed by the name you chose when creating the deployment (e.g. `acme-corp/my-model`).
# Deploy a Hugging Face Model (https://docs.inference.net/platform/deploy/deploy-a-hf-model)
<Note>
**Beta.** Custom model registration is in beta. Reach out to the team to have it enabled for your account before you start.
</Note>
Custom models let you deploy any public or private Hugging Face model on dedicated GPUs. You register the model in the dashboard, choose the GPUs it can run on, and create a deployment. The deployment gets an OpenAI-compatible endpoint you call like any other.
<Steps>
<Step title="Open the My Models tab">
Open the [dashboard](https://inference.net/dashboard), go to **Models**, and switch to the **My Models** tab. Custom models appear next to your fine-tuned models.
<Frame caption="The My Models tab lists your registered custom models.">
<img src="/images/deploy/custom-models/models-my-models.png" alt="Models page with the My Models tab selected" />
</Frame>
</Step>
<Step title="Create a new model">
Click **Create new Model** on the My Models tab. This opens the new model form, where you register a Hugging Face model your team can deploy.
<Frame caption="The new model form.">
<img src="/images/deploy/custom-models/new-model-form.png" alt="New model form with empty fields" />
</Frame>
</Step>
<Step title="Fill in the model details">
Fill in the model details and pick the GPUs you want the model to run on.
<Note>
You can pick 1-2 H100s or H200s only. If you need other cards or larger instance sizes, [reach out to the team](https://inference.net/meet-with-us/).
</Note>
<Frame caption="A filled-in form for nvidia/Kimi-K3-NVFP4.">
<img src="/images/deploy/custom-models/new-model-filled.png" alt="New model form filled in for nvidia/Kimi-K3-NVFP4" />
</Frame>
</Step>
<Step title="Tweak the engine config">
Once the model is created, you can tweak its engine config. The config controls how the model is served: the engine (vLLM or SGLang), the compatible GPUs, and the engine flags. Saving creates a new version of the config. Running deployments keep their current version until they roll forward.
<Frame caption="Edit engine config, showing engine choice, compatible GPUs, and engine flags.">
<img src="/images/deploy/custom-models/engine-config.png" alt="Edit engine config panel" />
</Frame>
</Step>
<Step title="Create the deployment">
Click the deploy button to create a deployment. Pick a deployment name, an instance size, and create the deployment. Your endpoint is ready after warm-up. See [Deploy a Trained Model](/platform/deploy/deploy-a-model) for what happens next.
<Frame caption="The create deployment page for a custom model.">
<img src="/images/deploy/custom-models/create-deployment.png" alt="Create deployment page showing identity, model, and rate estimate" />
</Frame>
</Step>
</Steps>
## After the deployment is live [#after-the-deployment-is-live]
Once the endpoint is up, [call it](/platform/deploy/call-your-deployment) with the same OpenAI-compatible API you use elsewhere. Set `model` to your deployment's model path, for example `harry/kimi-k3-nvfp4-testing`.
# Deploy a Trained Model (https://docs.inference.net/platform/deploy/deploy-a-model)
When training completes, the model is automatically registered and ready to deploy. No manual promotion step.
Deployments are created within a project, and access is scoped to that project — sibling projects in the same team cannot call them. When you create a deployment, you give it a name — the model path becomes your team/org slug followed by that name (e.g. `acme-corp/my-model`).
## Starting a deployment [#starting-a-deployment]
There are a few ways to get started:
* From the **Dedicated** page — click **Create Deployment**
* From a **completed training run** — click deploy directly from the training detail page
* From the **Models** page — open the model's detail page and click deploy
All three paths lead to the same flow.
## The flow [#the-flow]
<Steps>
<Step title="Name the deployment">
Give it a descriptive name. This becomes the second part of your model path (e.g. `acme-corp/my-model`).
</Step>
<Step title="Deploy">
Select your instance type and count, then click deploy.
</Step>
<Step title="Wait for warm-up">
The deployment takes a few minutes to 20–30 minutes to come online. This time is spent allocating compute and spinning up the GPU.
</Step>
</Steps>
## After deployment [#after-deployment]
Once the endpoint is live, you can [call it](/platform/deploy/call-your-deployment) using the same OpenAI-compatible API you're already using. Just swap in your model path.
{/* TODO: screenshot of the deployment page */}
# Manage and Monitor (https://docs.inference.net/platform/deploy/manage-and-monitor)
<Frame>
<img src="/images/deploy/deploy-overview.png" alt="Deployments list with per-deployment metrics" />
</Frame>
## Lifecycle operations [#lifecycle-operations]
* **Start** — bring a stopped deployment back online
* **Stop** — take the deployment offline. Stopping archives the deployment; it stays listed and can be started again.
There's no delete action — the lifecycle is Start and Stop. An archived deployment keeps its identifiers but is not serving.
## Scaling [#scaling]
Your model deploys on a dedicated GPU. To scale, edit the instance configuration from the deployment's settings — the GPU family, GPUs per instance, and instance count you chose at creation. You can increase or decrease compute this way. There is no auto-scaling.
## Monitoring [#monitoring]
Once your deployment is live, click into it from the **Dedicated** page to see metrics and individual inference calls. You get the same [Gateway](/platform/gateway/overview) experience — latency, error rates, token usage, and full request/response payloads — scoped to that deployment.
<Frame caption="Deployment detail page showing metrics and inference calls.">
<img src="/images/deploy/deploy-metrics.png" alt="Deployment metrics and inference calls" />
</Frame>
## The loop continues [#the-loop-continues]
Your custom model is live. Use [Gateway](/platform/gateway/overview) to watch its production performance, run [AutoEvals](/platform/eval/auto-evals/run-an-auto-eval) to catch regressions, and [train](/platform/train/overview) the next version when you're ready.
<CardGroup cols="3">
<Card title="Gateway" icon="satellite-dish" href="/platform/gateway/overview">
Monitor production traffic.
</Card>
<Card title="Eval" icon="flask" href="/platform/eval/auto-evals/run-an-auto-eval">
Catch quality regressions.
</Card>
<Card title="Train" icon="brain" href="/platform/train/overview">
Build the next version.
</Card>
</CardGroup>
# Open Source Models (https://docs.inference.net/platform/deploy/open-source-models)
Open source models on Inference.net are served serverless. Call them through the OpenAI-compatible API with your Inference API key and pay per token. There is nothing to deploy.
## Call an open source model [#call-an-open-source-model]
Use the model id as the `model` parameter, the same as any other serverless call:
<CodeGroup>
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "glm-5.2",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="glm-5.2",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
```rust Rust
use serde_json::{json, Value};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let api_key = std::env::var("INFERENCE_API_KEY")?;
let response: Value = reqwest::Client::new()
.post("https://api.inference.net/v1/chat/completions")
.bearer_auth(api_key)
.json(&json!({
"model": "glm-5.2",
"messages": [{"role": "user", "content": "Hello, world!"}]
}))
.send()
.await?
.json()
.await?;
println!(
"{}",
response["choices"][0]["message"]["content"]
.as_str()
.unwrap_or_default()
);
Ok(())
}
```
```go Go
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "glm-5.2",
"messages": []map[string]string{
{"role": "user", "content": "Hello, world!"},
},
})
req, _ := http.NewRequest("POST", "https://api.inference.net/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFERENCE_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
var result struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
panic(err)
}
fmt.Println(result.Choices[0].Message.Content)
}
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "glm-5.2",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
Browse the available models at [inference.net/models](https://inference.net/models). See the [API quickstart](/api/api-quickstart) for full examples.
## Dedicated deployments [#dedicated-deployments]
You can also run an open source model on its own dedicated GPU — your own instance, no shared capacity. Open source catalog models with weights available are deployable through the same create-deployment flow you use for trained models: open the **Create Deployment** picker and select the model from the open source catalog, not just your fine-tuned models.
Bringing your own weights is supported too, via [registering a Hugging Face model](/platform/deploy/deploy-a-hf-model) as a custom model first. Once registered, that model appears in the deployment picker under **My Models**. See [Deploy a Trained Model](/platform/deploy/deploy-a-model) for the create-deployment flow.
# Deploy (https://docs.inference.net/platform/deploy/overview)
Deploy gives you a dedicated GPU serving your fine-tuned model. The API is OpenAI-compatible, so switching from an off-the-shelf model to your custom model is a one-line code change. This is the last step in the loop and the beginning of the next one.
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/7AAf7Y7Qe24?list=PLJzp7SN2tfJsRAU9VGSfSo60CyDJzqhLP&rel=0" title="Deploy a Production-Ready Fine-Tuned LLM | Inference platform" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Key concepts [#key-concepts]
| Concept | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Dedicated GPU** | Your model runs on its own GPU. No shared infrastructure, no noisy neighbors. Instance type and count are chosen at deploy time from the model's allowed instance types; the default is 1×H100. |
| **OpenAI-compatible API** | Same base URL, same API key, just swap the model parameter. Structured outputs, function calling, and all standard API features work the same way. |
| **Project-scoped** | Deployments are created within a project, and access is scoped to that project. Sibling projects in the same team cannot call them. The model path is your team/org slug followed by the deployment name you choose (e.g. `acme-corp/my-model`). |
| **The improvement loop** | Deploy → observe production performance → run evals to catch regressions → train the next version. The loop continues. |
## What you can deploy [#what-you-can-deploy]
* Models trained on the Inference platform
* Open source catalog models with weights available
* Hugging Face models you've registered as custom models
* Served via an OpenAI-compatible API (chat completions endpoint)
* Same base URL and API key as the rest of the Inference API
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Deploy a trained model" icon="server" href="/platform/deploy/deploy-a-model">
Name it, click deploy, start serving.
</Card>
<Card title="Deploy a hugging face model" icon="rocket" href="/platform/deploy/deploy-a-hf-model">
Deploy existing models.
</Card>
<Card title="Call your deployment" icon="code" href="/platform/deploy/call-your-deployment">
One line of code to switch over.
</Card>
<Card title="Manage and monitor" icon="sliders" href="/platform/deploy/manage-and-monitor">
Lifecycle operations, scaling, and the improvement loop.
</Card>
</CardGroup>
# Serverless Deployments (https://docs.inference.net/platform/deploy/serverless-deployments)
Most models on Inference.net are already serverless: the open-source and
closed-source models in the catalog are callable by any account and billed per
token. See the [API quickstart](/api/api-quickstart) for how to call them.
This page covers **serverless deployments**, which extend that model to
deployments. Most deployments are private: only the owning team can call them,
and they bill by GPU capacity. A deployment can also be serverless-enabled by
the Inference team, which makes it callable by every account on the platform
and billed per token, or offered for free.
## Calling a serverless deployment [#calling-a-serverless-deployment]
Serverless deployments work exactly like any other model on the
OpenAI-compatible API. Use the deployment's model path as the `model`:
<CodeGroup>
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "inference-net/example-model",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="inference-net/example-model",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
```rust Rust
use serde_json::{json, Value};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let api_key = std::env::var("INFERENCE_API_KEY")?;
let response: Value = reqwest::Client::new()
.post("https://api.inference.net/v1/chat/completions")
.bearer_auth(api_key)
.json(&json!({
"model": "inference-net/example-model",
"messages": [{"role": "user", "content": "Hello, world!"}]
}))
.send()
.await?
.json()
.await?;
println!(
"{}",
response["choices"][0]["message"]["content"]
.as_str()
.unwrap_or_default()
);
Ok(())
}
```
```go Go
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "inference-net/example-model",
"messages": []map[string]string{
{"role": "user", "content": "Hello, world!"},
},
})
req, _ := http.NewRequest("POST", "https://api.inference.net/v1/chat/completions", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFERENCE_API_KEY"))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
var result struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
panic(err)
}
fmt.Println(result.Choices[0].Message.Content)
}
```
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "inference-net/example-model",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
Streaming, structured outputs, and the other chat completions features work
the same as on catalog models.
## Billing [#billing]
Serverless deployments are priced in **USD per 1M tokens**, with separate
input and output rates set per deployment. Usage is billed to the calling
team's credit balance like any other serverless inference:
* Requests are authorized against your credit balance up front; if the
balance can't cover the estimated cost, the API responds with `402`.
* The actual charge is settled when the inference completes, from the real
token counts reported by the serving engine.
* Failed inferences are never billed.
* Charges appear in your usage dashboard under the deployment's model path.
### Super-admin pricing rules [#super-admin-pricing-rules]
Super-admins can add fixed effective pricing rules under a deployment's **Serverless Options** settings. A rule can target one team, or all teams when Team ID is blank, and can be scheduled with a start and end time. Matching team rules take precedence over all-team rules; otherwise callers receive the deployment's standard prices.
The optional percentage field is a calculator: it converts the displayed base prices into exact fixed rates when the rule is saved. Later changes to the standard deployment price do not silently change an existing rule.
For partner catalogs such as OpenRouter, a native user discount is advertised only when the partner team's input, cached-input, and output rates are all reduced by the same proportion. Otherwise the partner-specific effective prices are advertised directly.
## Free deployments [#free-deployments]
A serverless deployment with **no prices set is public and free**: anyone on
the platform can call it and no credits are charged or required. Free
deployments still count against your standard serverless rate limits.
## Limits [#limits]
Serverless deployment requests share your team's serverless inference rate
limits. Context-window limits are enforced by the deployment's engine rather
than the platform catalog, so an oversized prompt is rejected by the model
itself.
# Interpret the Results (https://docs.inference.net/platform/eval/auto-evals/interpret-results)
A completed AutoEval run produces a report on the **Compare Models** tab, and every judged sample behind it is open for inspection in the Sample Viewer. This page walks through both, top to bottom.
If you have not started a run yet, see [Run an AutoEval](/platform/eval/auto-evals/run-an-auto-eval).
## The verdict [#the-verdict]
The report leads with the recommendation: either **Recommended model** with a specific alternative, or **Keep current model**.
<Frame>
<img src="/images/autoevals/run_completed.png" alt="The RECOMMENDED MODEL hero showing a model name, the Eval score vs current percentage delta and Monthly savings stats, the Switch in Code button, the judge's rationale, and the Key findings list below" />
</Frame>
Next to the verdict:
* **Eval score vs current**: how much better or worse the recommended model scored than your current one.
* **Monthly savings**: what switching would save, projected from the real traffic volume in the sampled window.
* **Rationale**: the analysis agent's reasoning for the pick, shown inline under the verdict.
* **Switch in Code**: a ready-made prompt you can paste into a coding agent to migrate your task or agent off the current model.
Some runs finish without a switch recommendation. The eval results below the verdict still show how each model scored.
## Key findings and run summary [#key-findings-and-run-summary]
**Key findings** lists the most important observations from the judged samples, marked as positive, caution, or negative. Below it, the **Run summary** is a written analysis of the run with citations that link directly into the underlying requests and traces, so every claim can be checked against real traffic.
Findings often go beyond model choice: the analysis agent also records configuration problems and failure modes it noticed in your traffic.
## Quality versus cost [#quality-versus-cost]
The scatter chart plots every candidate by score against cost. Up and to the left is better: higher quality at lower cost. The current and recommended models are labeled. A second tab, **Similarity vs. cost**, shows the same chart with the similarity score on the y-axis.
<Frame>
<img src="/images/autoevals/quality_vs_cost_graph.png" alt="The Quality vs. cost scatter chart with one point per candidate, the recommended model emphasized and labeled 'recommended', the current model labeled 'current', and the Quality vs. cost and Similarity vs. cost tabs" />
</Frame>
## The leaderboard [#the-leaderboard]
The **Comparison Scores** table ranks every candidate against your current model.
<Frame>
<img src="/images/autoevals/eval_scores_table.png" alt="The Comparison Scores leaderboard with rank, Model, Samples, Quality, Similarity, P90 Latency, and Price columns, with the current model and recommended model badged" />
</Frame>
| Column | Meaning |
| ------------------ | ----------------------------------------------------------------------------------------------------- |
| **Quality** | Average blind quality score, 1 to 10. Your current model's production responses are the baseline row. |
| **Similarity** | Average similarity to your current model's behavior, 1 to 10. |
| **Samples** | How many samples were judged for this model, with any failures. |
| **Cost Delta** | Projected monthly savings vs your current model (cheaper is green). |
| **P90 Latency** | Response latency for the replayed samples. |
| **Price in · out** | The model's per-token pricing. |
Sort the table by any column's header, such as quality, similarity, cost delta, or latency. Agentic runs add per-rubric tabs above the table to rank by one rubric or across all of them. Click a row to expand the judge's verdict on that model: a narrative summary of how it behaved, and a **Compare Responses** button that opens the raw results.
Models that could not be scored appear as muted rows with the skip reason.
### Reading quality against similarity [#reading-quality-against-similarity]
The two scores answer different questions, and the combination matters:
* **High quality, high similarity**: behaves like your current model and scores well. The safest switch.
* **High quality, low similarity**: scores well but behaves differently (different tool choices, different response style). Test it before routing traffic to it.
* **Low quality**: not a candidate, regardless of similarity.
## The Sample Viewer [#the-sample-viewer]
Every score in the report is backed by judged samples you can read. The **Compare Responses** button on the recommended model and the expanded rows opens the eval viewer for the run.
The eval viewer shows:
* **Score distribution** per model, so you can see spread and outliers rather than just averages. Click a model to filter.
* **Comparison grid**: one row per sample, with the original input and output pinned on the left and one column per model. Click any cell to open the full result, including the judge's scored assessment.
* **Sort controls**: sort samples by any model's score, highest or lowest first. Sorting by your current model, lowest first, surfaces the traffic your production model handles worst.
Click **View in Sample Viewer** to review samples one at a time. The Sample Viewer shows the judged turn highlighted in the original conversation, then every model's version of the response with the judge's reasoning under each one. Use the arrow keys to move between samples.
## When a run ends without results [#when-a-run-ends-without-results]
Not every run produces a verdict. The report states what happened and why:
| State | Meaning |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **This run was cancelled** | Someone cancelled the run before it finished. |
| **This run failed** | The run hit an error mid-scoring. No verdict was produced and partial scores were discarded. |
| **Insights could not be generated** | Scoring finished but the analysis stage failed. The scores still render; only the verdict and summary are missing. |
| **No models were scored** | The run finished without evals. The report shows the reason: your team's credit balance could not cover the eval cost, the window had no traffic that qualified for sampling, or no eligible candidate models were enabled. |
Each state offers **Run new comparison** to try again, and the credit case links to your plan and usage.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Run an AutoEval" icon="play" href="/platform/eval/auto-evals/run-an-auto-eval">
Configuration, sampling, scoring, and limits.
</Card>
<Card title="Train your own model" icon="brain" href="/platform/train/autotrainer">
Push past off-the-shelf quality with a model distilled from your traffic.
</Card>
</CardGroup>
# Run an AutoEval (https://docs.inference.net/platform/eval/auto-evals/run-an-auto-eval)
AutoEvals sample your recent production traffic, replay each sampled turn against a set of candidate models, score every result with LLM judges, and publish a model recommendation with supporting evidence. You do not build a dataset or write a rubric.
In the dashboard, AutoEvals appear as the **Compare Models** tab on each task and agent. This page covers everything up to a completed run. For reading the report, see [Interpret the Results](/platform/eval/auto-evals/interpret-results).
## Prerequisites [#prerequisites]
A run is always scoped to one entity: a Gateway [task](/platform/gateway/tasks) or a traced agent. That scope is where the run samples its traffic, so you need one of these integrations first:
| Scope | Integration | What gets sampled |
| --------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| **Task** | LLM calls routed through the [Gateway](/platform/gateway/overview) | Successful recorded requests for that task. Requests without a task ID group under the **Default** task. |
| **Agent** | [Traces](/get-started/capture-first-trace) reported from your app | Successful LLM spans captured for that agent. |
There is no minimum traffic requirement to start a run manually, but the sample can only be as representative as the traffic in the window you pick.
## Where to find it [#where-to-find-it]
* **Tasks**: **Gateway → Tasks**, open a task, select the **Compare Models** tab.
* **Agents**: **Agents**, open an agent, select the **Compare Models** tab.
The tab lists every comparison for that scope with its creation time, status, and source (**Manual** runs show who started them, **Automated** runs are scheduled by the platform). Click **View run** to open any report.
<Frame>
<img src="/images/autoevals/compare_models_tab.png" alt="The Compare Models tab on a task, showing the 'Compare Models' empty state and the Run Comparison button in the header" />
</Frame>
## Start a run [#start-a-run]
Click **Run Comparison** to open the comparison dialog.
<Frame>
<img src="/images/autoevals/run_comparison_clicked.png" alt="The comparison dialog with the current model's pricing, the Traffic window and Samples per model selectors, the spend summary, and the model catalog with recommended models pre-selected" />
</Frame>
The dialog has three parts:
### Current model and spend [#current-model-and-spend]
The left panel shows the model you run today: its per-token pricing and your recent spend on this task across 1, 7, and 30 day windows. This is the baseline every candidate is compared against.
### Traffic window [#traffic-window]
* **Traffic window**: the window to sample from. Presets range from 1 hour to 30 days, and 30 days is both the default and the maximum. The panel shows how many turns are available in the window.
* **Samples per model**: how many turns to sample, up to 100 (the default). Fewer samples make the run cheaper and faster, at the cost of statistical confidence.
### Model catalog [#model-catalog]
The catalog lists every callable candidate model with its quality score, per-token pricing, blended price, and projected savings against your current model. Models are grouped by how their blended price compares to yours, and the models recommended for your task (at or below your blended cost, best quality per dollar) come pre-selected.
Check or uncheck models to set the candidate list. Your selection is saved per task or agent and restored on the next run; **Reset to recommended** returns to the default set. The left rail shows the blended savings range of the current selection, and the footer shows how many models are selected.
If the run's estimated eval cost exceeds your team's credit balance, the dialog warns **Not enough credits to start this run**. Add credits or reduce the sample count and enabled models.
Click **Run Comparison**. The dialog closes and takes you straight to the new run's report.
## While the run executes [#while-the-run-executes]
The report shows an **Analysis in progress** card with a three-step pipeline. Results fill in as models finish scoring.
| Stage | What happens |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| **Sampling traffic** | Inference platform pins a deterministic sample of the scope's requests. |
| **Scoring candidate models** | Every candidate re-runs the samples and is judged. |
| **Generating insights** | An analysis agent summarizes the eval results into a verdict, summary, and insights. |
Per-model results appear below the pipeline as each model finishes: how many samples it completed, and any failures.
<Frame>
<img src="/images/autoevals/run_in_progress.png" alt="The Analysis in progress card with the Sampling traffic, Scoring candidate models, and Generating insights steps" />
</Frame>
You can **Cancel** a run while it executes. Cancelling stops sampling, scoring, and insight generation, and marks the run as cancelled. Scoring already in flight may take a moment to settle.
When the run finishes, Inference platform sends an email or Slack notification with the headline result and a link to the report.
## How a run works [#how-a-run-works]
### Sampling [#sampling]
Inference platform draws up to your **Samples per model** count of individual assistant turns from the window. Sampling is recency weighted, so newer conversations are more likely to be picked, and deterministic, so the same configuration reproduces the same sample. Each conversation contributes at most 5 turns, spread across the early, middle, and late parts of the conversation. The pinned sample is saved as an eval dataset, so you can inspect exactly what was judged.
### Scoring [#scoring]
Each sampled turn is replayed against every candidate model: the conversation up to that turn, with the same tools available, is sent to the candidate, which produces its own version of the response. LLM judges then score every response against two fixed rubrics:
| Rubric | What the judge sees | Score |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| **Action quality** | The conversation and one response, graded blind. The judge never knows which model wrote it, and never sees the alternative. Your original production responses are graded the same way, as the baseline. | 1 to 10 |
| **Behavioral similarity** | The candidate's response next to your original response, plus similar excerpts from the rest of the sample. Measures how closely the candidate matches your current behavior. | 1 to 10 |
Blind grading matters: judges shown two responses side by side tend to favor the incumbent. Grading each response in isolation removes that bias. Each rubric uses a judge model pinned by the platform; the [eval viewer](/platform/eval/auto-evals/interpret-results#the-sample-viewer) shows which judge scored each rubric.
### Recommendation [#recommendation]
Once the scores settle, an analysis agent reviews the scored results together with the sampled conversations. It writes the run summary, picks the model recommendation (weighing quality, cost, and latency, and allowed to conclude "keep the current model"), and records insights about problems it found in the traffic.
## Candidate models [#candidate-models]
Runs compare your traffic against candidate models from the catalog (GPT, Claude, Gemini, Grok, DeepSeek, GLM, Kimi, Nemotron, Qwen, and others). The recommended set — the models at or below your blended cost with the best quality per dollar — comes pre-selected. Use the checkboxes in the run dialog's model catalog to set the candidate list per task or agent.
## Automatic runs [#automatic-runs]
Inference platform can start your team's first comparison automatically. Automatic runs are available on Growth and higher plans, and start once a task or agent accumulates enough recent traffic (about 1,000 requests in the trailing 7 days). An automatic run shows **Automated** as its source and a **Free run** badge: its eval cost does not use your team credits.
After the first automatic run, you start runs yourself with **Run Comparison**.
## Cost and limits [#cost-and-limits]
The judge and replay calls in the scoring stage bill as normal eval usage from your team credits, gated by your credit balance at launch and the platform's concurrent-run limit. With the default 100 samples and all candidates enabled, a full run requests roughly 3,500 eval samples: every candidate plus your baseline is graded for quality, and every candidate is graded for similarity.
To reduce a run's cost, lower **Samples per model** or deselect candidate models you would not switch to.
If your credit balance would not cover the run's estimated eval cost, the run is refused with **Not enough credits to start this run** — add credits or shrink the selection.
The sampling and analysis stages are included; you are not billed for the analysis agent.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Interpret the results" icon="chart-column" href="/platform/eval/auto-evals/interpret-results">
Read the verdict, the leaderboard, and individual judged samples.
</Card>
<Card title="Train your own model" icon="brain" href="/platform/train/autotrainer">
When the best off-the-shelf model is not enough, AutoTrainer distills one from your traffic.
</Card>
</CardGroup>
# Eval (https://docs.inference.net/platform/eval/overview)
Model providers ship updates constantly and prompts drift. Eval gives you a repeatable way to measure model quality and to compare model options for a given task, so you can decide with data instead of intuition.
## AutoEvals: compare models on your traffic [#autoevals-compare-models-on-your-traffic]
AutoEvals answer one question: is there a better model for this workload than the one you run today? A run samples your recent production traffic from a Gateway task or a traced agent, replays each sample against a set of candidate models, scores every output with LLM judges, and publishes a model recommendation with the evidence behind it.
You do not build a dataset or write a rubric. Start a run from the **Compare Models** tab on any task or agent, and read the results down to the individual judged sample.
<CardGroup cols="2">
<Card title="Run an AutoEval" icon="play" href="/platform/eval/auto-evals/run-an-auto-eval">
From integration to a completed run: configuration, sampling, scoring, and limits.
</Card>
<Card title="Interpret the results" icon="chart-column" href="/platform/eval/auto-evals/interpret-results">
The recommendation, the leaderboard, and the Sample Viewer.
</Card>
</CardGroup>
## How scoring works [#how-scoring-works]
Each sampled input is replayed against every candidate model, and an LLM judge scores each output against two fixed rubrics, one for quality and one for similarity to your current model. Scores aggregate into a leaderboard, and the run recommends the model that wins on quality at the lowest cost. You can open any judged sample to see the exact reasoning behind its score.
## When to go further [#when-to-go-further]
AutoEvals tell you whether a better off-the-shelf model exists for a task you already run. When the best available model still is not good enough, [AutoTrainer](/platform/train/autotrainer) distills the task onto a smaller, cheaper custom model from the same traffic.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Run an AutoEval" icon="play" href="/platform/eval/auto-evals/run-an-auto-eval">
Configuration, sampling, scoring, and limits.
</Card>
<Card title="Interpret the results" icon="chart-column" href="/platform/eval/auto-evals/interpret-results">
Read the verdict, the leaderboard, and individual judged samples.
</Card>
<Card title="Compare Models get-started" icon="scale-balanced" href="/get-started/compare-models-auto-evals">
A guided first run from integration to a completed comparison.
</Card>
<Card title="Train a custom model" icon="brain" href="/platform/train/autotrainer">
Push past off-the-shelf quality with a model distilled from your traffic.
</Card>
</CardGroup>
# How to Create a Task (https://docs.inference.net/platform/gateway/create-a-task)
Task are automatically created when you send requests with a `x-inference-task-id` header. The task appears in the dashboard as soon as the first tagged request comes through.
## Set the task header [#set-the-task-header]
Add `x-inference-task-id` to your request. The value is whatever name makes sense for the objective — `document-summary`, `ticket-classifier`, `extract-product-data`, etc.
You can set the task per request, or put it in `defaultHeaders` on the client if all calls from that client belong to the same task.
**Per request** (use when one client serves multiple tasks):
<CodeGroup>
<Metadata text="observe/create-task-header[series=create-task]" />
```typescript TypeScript
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Summarize this document..." }],
}, {
headers: { "x-inference-task-id": "document-summary" },
});
```
<Metadata text="observe/create-task-header[series=create-task]" />
```python Python
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Summarize this document..."}],
extra_headers={"x-inference-task-id": "document-summary"},
)
```
<Metadata text="observe/create-task-header" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-H "x-inference-task-id: document-summary" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Summarize this document..."}]
}'
```
</CodeGroup>
**In default headers** (use when a client is dedicated to one task):
<CodeGroup>
<Metadata text="observe/create-task-default-header[series=create-task-default]" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
"x-inference-task-id": "document-summary",
"x-inference-environment": "production",
},
});
```
<Metadata text="observe/create-task-default-header[series=create-task-default]" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
"x-inference-task-id": "document-summary",
"x-inference-environment": "production",
},
)
```
</CodeGroup>
## The default task [#the-default-task]
If you don't set `x-inference-task-id`, the request is grouped under the **default** task. Your traffic is still captured, but it's all in one bucket. Once you start tagging, you can break down metrics, run evals, and build datasets per task.
## Assign task IDs automatically [#assign-task-ids-automatically]
Use **[Install with AI](/integrations/install-with-ai)** if you want the CLI to add task IDs for you. It runs `inf instrument`, scans your codebase, and names each call site based on what the prompt is doing — for example, `document-summary` or `ticket-classification`. These are starting points; you can rename them by editing the `x-inference-task-id` header value in the generated code.
# Inference Viewer (https://docs.inference.net/platform/gateway/inference-viewer)
The Inference Viewer is a filterable table of every LLM call that flows through the Inference platform. Use it to inspect individual requests, debug issues, and find samples to save as datasets for evals and training.
<Frame>
<img src="/images/observe/observe-inferences.png" alt="Inference Viewer" />
</Frame>
## Table columns [#table-columns]
Each row in the table represents a single LLM call. The visible columns are configurable, and include:
| Column | Description |
| --------------------------------- | ------------------------------------------------ |
| **Time** | When the request was sent |
| **Status** | HTTP status code |
| **Model** | The model used |
| **API URL** | The downstream provider endpoint |
| **Environment** | Environment tag (production, staging, etc.) |
| **Task** | The task ID, if one was set |
| **Input / Output / Total Tokens** | Token counts for the request and response |
| **Cached Tokens** | Tokens served from cache |
| **Reasoning Tokens** | Tokens used for reasoning (where applicable) |
| **Cost** | Total cost, with input and output cost breakdown |
| **Duration** | End-to-end request latency |
| **TTFT** | Time to first token (for streaming requests) |
| **Request / Response Size** | Payload sizes in bytes |
You can sort by time, status, cost, duration, tokens, or payload size.
## Filtering [#filtering]
The filter builder lets you combine multiple conditions to narrow down your traffic. Filters are available for both categorical and numeric fields.
**Categorical filters:**
* **Model** - filter to specific models
* **Provider** - filter by upstream provider
* **Task** - filter by task ID
* **Environment** - filter by environment tag
* **Status** - filter by HTTP status code or range (success, error, 2xx, 4xx, 5xx, or specific codes like 429)
* **Streaming** - filter streaming vs non-streaming requests
**Numeric filters:**
* **Duration** - filter by latency (e.g. requests slower than 5s)
* **Cost** - filter by cost (e.g. requests costing more than $0.05)
* **Input / Output Tokens** - filter by token count (e.g. input > 5k tokens)
* **Request / Response Size** - filter by payload size in bytes
**Quick filters** are available for common queries: input tokens > 5k, cost > $0.05, duration > 5s, and status = error.
## Detail view [#detail-view]
Click on any row to open the detail panel. This shows the full picture of a single inference:
* Full request and response payloads (viewable as raw JSON)
* Cost breakdown (input, output, reasoning, cached)
* Token breakdown (input, output, reasoning, cached) with visual bars
* Duration and time to first token
* Model, provider, task, and environment
* Streaming status
* Request metadata (key-value pairs)
* Geolocation (country, city)
## Save as dataset [#save-as-dataset]
You can build datasets from live traffic directly in the Inference Viewer or from the [Datasets tab](/platform/datasets/build-from-traffic). Apply filters to get a representative slice of your data, then click **Save as Dataset** to create an eval or training dataset from the filtered results.
The dataset creation flow:
1. Apply your filters to narrow down the traffic
2. Review the matching inferences
3. Optionally set a limit on how many inferences to include
4. Choose whether this is an eval dataset or a training dataset
5. Name the dataset and save
The saved dataset is immediately available for [training a model](/get-started/train-and-deploy).
<Tip>
Use [task tags](/platform/gateway/tasks) to filter by objective before saving a dataset. This gives you clean, focused samples instead of a mix of unrelated traffic.
</Tip>
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Build a dataset from traffic" icon="database" href="/platform/datasets/build-from-traffic">
Step-by-step guide for turning filtered traffic into datasets.
</Card>
<Card title="Compare models with AutoEvals" icon="scale-balanced" href="/get-started/compare-models-auto-evals">
Replay your traffic against candidate models and get a recommended winner.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have data? Upload a JSONL file directly.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group calls by objective for better filtering and per-feature metrics.
</Card>
</CardGroup>
# Integrate (https://docs.inference.net/platform/gateway/integrate)
Production LLM traffic serves as the backbone for model optimization in the Inference platform. Inference Gateway records LLM traffic between your application and your current LLM provider, and stores it for later evaluation and training.
There are two ways to capture production LLM traffic through Gateway: use the [Inference CLI](/cli/overview) to automatically instrument your codebase, or wire it up manually.
To get started with the Inference platform, create a free account at [inference.net](https://inference.net/register).
## Choose a setup path [#choose-a-setup-path]
Installing with AI is quickest. Use the manual flow if you want to review each change yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to automatically initialize a coding agent like [Claude Code](https://code.claude.com/docs/en/overview) to scan your codebase, update your LLM clients, and add required request metadata.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata text="get-started/record-first-call-ai-auth" />
```bash
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run instrumentation in your project">
Navigate to your project root and run instrumentation.
<Metadata text="get-started/record-first-call-ai-instrument" />
```bash
cd /path/to/your/project && inf instrument
```
The command guides you through the following workflow:
* Select a coding agent to use: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain,etc
* Redirect base URLs to the gateway
* Add routing headers so requests are authenticated, forwarded, and traced
* Add task IDs so each call site is grouped automatically in the dashboard
* Review the generated changes before applying them
<Tip>
Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would to produce inference requests. Requests from your application are now routed through the gateway and will appear in the dashboard.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to see request details, traces, and analytics.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to review each change yourself. The example below uses OpenAI. For Anthropic, Cerebras, Groq, and other providers, see the [Integrations guide](/integrations/overview).
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenAI API key** — from your [OpenAI account](https://platform.openai.com/api-keys)
Set them as environment variables:
<Metadata text="get-started/record-first-call-manual-env" />
```bash
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point your SDK at `https://api.inference.net/v1` and use your Inference project API key as the SDK's `apiKey`. Your provider's API key goes in the `x-inference-provider-api-key` header so the gateway can forward it. The gateway adds roughly 10ms of latency and forwards your requests to the provider as-is.
<CodeGroup>
<Metadata text="get-started/record-first-call-manual-openai" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
<Metadata text="get-started/record-first-call-manual-openai" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
<Metadata text="get-started/record-first-call-manual-openai-curl" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
</Step>
<Step title="Send a request">
Run the snippet above from your application or terminal. Once the request completes, Inference platform will capture it automatically.
</Step>
<Step title="View your results">
Open the [Inference dashboard](https://inference.net/dashboard) to inspect the request, traces, and metrics.
</Step>
</Steps>
<Note>
Need provider-specific manual instructions? See [Integrations Overview](/integrations/overview).
</Note>
</Tab>
</Tabs>
That's it. Every request now flows through the Inference platform and gets captured automatically.
## What gets captured [#what-gets-captured]
Once traffic is flowing, Inference platform records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency (end-to-end and time to first token)
* Token counts (input and output)
* Error rates and status codes
* Model and provider
## Where to find your data [#where-to-find-your-data]
* **[Metrics Explorer](/platform/gateway/metrics-explorer)** - dashboards for cost, latency, errors, and usage across all your LLM calls
* **[Inference Viewer](/platform/gateway/inference-viewer)** - browse and filter individual requests and responses
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Connect more providers" icon="plug" href="/integrations/overview">
Set up Anthropic, Cerebras, Groq, and other providers.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have data? Upload a JSONL file to start evaluating or training.
</Card>
</CardGroup>
# Metrics Explorer (https://docs.inference.net/platform/gateway/metrics-explorer)
The Metrics Explorer gives you an at-a-glance view of your LLM usage. See trends, spot anomalies, and break down spending by model, task, or provider.
<Frame>
<img src="/images/observe/observe-metrics.png" alt="Metrics Explorer dashboard" />
</Frame>
## Summary statistics [#summary-statistics]
At the top of the dashboard, you'll see headline numbers for the selected time range:
* **Total inferences** and error count/rate
* **Total cost**, broken down by input, cached input, output, and reasoning tokens
* **Total tokens**, broken down by input (including cached) and output (including reasoning)
* **Average tokens per request** for both input and output
* **Streaming request count**
## Time series charts [#time-series-charts]
Below the summary, time series charts show how your usage changes over time:
| Chart | What it shows |
| ----------------------- | ----------------------------------------------------------------- |
| **Requests** | Total request volume over time |
| **Error rate** | Success rate percentage over time |
| **Cost** | Cost breakdown over time (input, cached input, output, reasoning) |
| **Tokens** | Token usage breakdown over time |
| **Duration** | Latency percentiles (p50, p75, p90, p99) |
| **Time to first token** | TTFT percentiles (p50, p75, p90, p99), useful for streaming |
| **Payload size** | Average request and response sizes |
| **Peak throughput** | Peak requests per time bucket |
| **Model distribution** | Request count broken down by model |
## Breakdown charts [#breakdown-charts]
The dashboard also includes distribution views:
* **Model cost comparison** - which models are costing you the most
* **HTTP code breakdown** - distribution of status codes
* **Failures breakdown** - error types and frequency
* **Request types** - streaming vs non-streaming split
## Filtering [#filtering]
You can scope the dashboard to a specific slice of traffic using filters:
* **Task** - see metrics for a single task (requires [task tagging](/platform/gateway/tasks))
* **Model** - filter to a specific model
* **Provider** - filter by upstream provider
* **Environment** - filter by environment tag (production, staging, etc.)
* **Status** - filter by HTTP status code or range (2xx, 4xx, 5xx, specific codes)
## Time controls [#time-controls]
* **Time range picker** - select the date range for the dashboard (default: last 24 hours)
* **Granularity selector** - control how data points are bucketed (hourly, daily, etc.)
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Inference Viewer" icon="list" href="/platform/gateway/inference-viewer">
Drill into individual LLM calls to inspect requests, responses, and errors.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn your traffic into datasets for evals and training.
</Card>
</CardGroup>
# Gateway (https://docs.inference.net/platform/gateway/overview)
Inference Gateway captures LLM requests flowing through your products. It stores the raw request, response, and metadata associated with each invocation of an LLM.
Recorded data is used to provide in-depth metrics and visibility into your LLM token usage, cost, latency, and error rates, across all your providers in a single, unified view. Additionally, this data is used to power downstream model evaluation and training.
Inference Gateway supports all major LLM providers and frameworks. For agents,
tools, framework runs, and custom orchestration, Inference Tracing captures full
trace trees and individual spans in addition to gateway inferences. View the
[integrations guide](/integrations/overview) for in-depth instructions.
{/* # If you want to use historical data for evaluation and training, skip to [uploading a dataset](/platform/datasets/upload-a-dataset). */}
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/S5ddaKQ-REU?list=PLJzp7SN2tfJsRAU9VGSfSo60CyDJzqhLP&rel=0" title="Observe Every LLM Call in One Place | Inference platform" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Key concepts [#key-concepts]
| Concept | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Gateway** | Edge layer between your app and LLM provider. Records traffic with \< 10ms overhead. |
| **Inference** | A single LLM call stored by Gateway. Includes request, response, cost, latency, & token counts. |
| **Trace** | A multi-step execution captured through OpenTelemetry. Useful for agents, tools, framework runs, and custom orchestration. |
| **Span** | One step inside a trace, such as a model call, tool call, retriever, graph node, or custom application operation. |
| **Task** | A user-defined objective (like "summarize docs" or "classify tickets") that groups related inferences so you can track each AI feature independently. |
| **Metrics** | Aggregated cost, latency, error rates, and token usage across your inferences. Filterable by model, task, or provider. |
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Set up tasks" icon="bullseye" href="/platform/gateway/tasks">
Group your LLM calls by objective.
</Card>
<Card title="Integrate with your LLM provider" icon="plug" href="/platform/gateway/integrate">
Connect your app and start capturing traffic.
</Card>
<Card title="Metrics Explorer" icon="chart-line" href="/platform/gateway/metrics-explorer">
See your LLM usage dashboards.
</Card>
<Card title="Inference Viewer" icon="list" href="/platform/gateway/inference-viewer">
Browse individual LLM calls.
</Card>
<Card title="Trace CLI" icon="terminal" href="/cli/traces">
Inspect trace trees, span timelines, facets, and exports from the terminal.
</Card>
</CardGroup>
# Prompt Versions (https://docs.inference.net/platform/gateway/prompt-versions)
Every time Inference platform sees a new prompt for a task, it records a hash of the prompt content and treats it as a new version. You don't need to do anything to enable this. It happens automatically as requests flow through the gateway.
## How it works [#how-it-works]
Inference platform hashes the system message of each request. When the hash changes (because you updated your system prompt), a new version is created under that task. The previous versions are preserved, so you have a full history of every system prompt your task has used.
This means you can:
* See which prompt version a specific inference used
* View the full history of prompt changes for a task
* Filter inferences by prompt version to compare behavior across changes
## What triggers a new version [#what-triggers-a-new-version]
A change to the system message creates a new version. Changes to user messages, parameters like `temperature`, or the model do not currently trigger a new version.
## Coming Soon [#coming-soon]
* **Broader hashing** - future versions will include user message templates, model, and parameter changes in the version hash
* **Dataset filtering by version** - build training and eval datasets tied to a specific version of a prompt, so you can train on data from the version that performed best
# Tasks Overview (https://docs.inference.net/platform/gateway/tasks)
A task is a user-defined objective like "summarize this document," "classify this ticket," or "extract these fields." Tagging your LLM calls with a `task-id` groups them by function, not just which model or prompt was used. If you don't set a task, calls are automatically grouped under a `default` task so nothing gets lost.
We recommend using [Install with AI](/integrations/install-with-ai) to automatically add tags to your LLM calls.
<Tip>
To get the most out of Inference platform, we **highly recommend** adding tags to your LLM calls.
</Tip>
## Why tasks matter [#why-tasks-matter]
Once you have more than one AI feature, tasks let you:
* **Track metrics per feature** — cost, latency, and error rates for each objective independently
* **Run evals per task** — measure whether a specific capability is getting better or worse
* **Build focused datasets** — filter by task to get clean, relevant samples for training
* **Experiment safely** — change the model or prompt for one task without affecting others
The task is the stable anchor. You might swap models, rewrite prompts, or redesign your agent, but the task stays the same.
## How to tag a call [#how-to-tag-a-call]
Set the `x-inference-task-id` header on your request. The task appears in the dashboard automatically once the first tagged request comes through. See [How to Create a Task](/platform/gateway/create-a-task) for the full setup.
<CodeGroup>
<Metadata text="observe/tasks-tag[series=tasks]" />
```typescript TypeScript
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Summarize this document..." }],
}, {
headers: { "x-inference-task-id": "document-summary" },
});
```
<Metadata text="observe/tasks-tag[series=tasks]" />
```python Python
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Summarize this document..."}],
extra_headers={"x-inference-task-id": "document-summary"},
)
```
<Metadata text="observe/tasks-tag" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-H "x-inference-task-id: document-summary" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Summarize this document..."}]
}'
```
</CodeGroup>
## Using tasks across the platform [#using-tasks-across-the-platform]
Once calls are tagged, tasks appear as a filter and grouping dimension everywhere:
* **Metrics Explorer** — break down cost, latency, and errors by task
* **Inference Viewer** — filter to see all calls for a specific task
* **Datasets** — filter by task to build focused eval and training datasets
* **AutoTrainer** — point [AutoTrainer](/platform/train/autotrainer) at a task to distill a custom model from that task's traffic
## Train a model on a task [#train-a-model-on-a-task]
A task is also the unit of training. [AutoTrainer](/platform/train/autotrainer) attaches to one task: it collects that task's successful responses as training samples and trains a distilled model for it automatically. If you prefer to stay hands-on, skip AutoTrainer and [train manually](/platform/train/launch-a-run) later, using a dataset built from the task's traffic. Nothing about tagging changes either way, so you can start with tags today and decide on training when you are ready.
{/* ## Coming soon
Agent traces inside the Inference platform are currently in private beta. [Contact us](mailto:support@inference.net) for early access. */}
# After Training Completes (https://docs.inference.net/platform/train/after-training)
When training completes, you get a registered model artifact that's ready to deploy. No manual promotion step — it goes straight from "done" to "deployable."
## What you see [#what-you-see]
* **Final eval results** — the trained model's scores on your eval dataset, compared against the baseline from before training started
* **Model artifact** — registered in the platform and visible on the **Models** page
* **Create Deployment button** — appears on the training details page and on your model's row in the Models page
<Frame>
<img src="/images/training/training-complete.png" alt="Completed training run with final eval results and Create Deployment button" />
</Frame>
## Is it actually better? [#is-it-actually-better]
Compare the trained model's final eval scores against the off-the-shelf models you benchmarked earlier. If the trained model scores higher, it's ready to deploy. If not, you may need to iterate on your training data or recipe. To compare it against other models on your live traffic, run an [AutoEval](/platform/eval/auto-evals/run-an-auto-eval).
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Deploy your model" icon="server" href="/platform/deploy/deploy-a-model">
Ship it to a dedicated GPU.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/platform/train/troubleshooting">
If training failed or results aren't what you expected.
</Card>
</CardGroup>
# AutoTrainer (https://docs.inference.net/platform/train/autotrainer)
<Note>
**Open beta.** AutoTrainer is in open beta. It is available to all users. If you have problems, [contact us](https://inference.net/meet-with-us/).
</Note>
<Note>
**Automatic deployment is temporarily disabled.** We are opening it up soon. Until then, training still runs end to end, and the trained model is yours to keep: download the weights from the training page and deploy them anywhere, on any provider or your own hardware. You can also [deploy it on our platform](/platform/deploy/deploy-a-model) or [contact us](https://inference.net/meet-with-us/) for a dedicated deployment sized to your needs. See [After training completes](#after-training-completes).
</Note>
AutoTrainer is an automatic version of the [training loop](/platform/train/overview). You point an existing chat completions call at AutoTrainer and select a **teacher model**. The teacher model is a large model that already does the task well. The teacher model serves your requests, and each successful response becomes a training sample. When there are sufficient samples, the platform trains a distilled model, deploys it, and moves your traffic to it. Your endpoint and your code do not change. Your cost and latency decrease.
Use AutoTrainer for high-volume, single-purpose tasks that have a stable prompt, for example extraction, classification, summarization, and tagging. If you want control of datasets and recipes, use a [manual training run](/platform/train/launch-a-run).
Not ready to train your own model? [AutoEvals](/platform/eval/auto-evals/run-an-auto-eval) compare your task's live traffic against a set of catalog models and recommend the best off-the-shelf option. Start there to find the right model for your task, then come back to AutoTrainer when you want better cost and latency than any off-the-shelf model gives you.
## How it works [#how-it-works]
AutoTrainer is tied to one [task](/platform/gateway/tasks). Each AutoTrainer request must identify its task. Samples collect for each task separately. Each task trains its own distilled model and serves a maximum of one distilled model at a time. If you have two different workloads, give each workload its own task.
Each task moves through three stages:
<Steps>
<Step title="Collecting">
The teacher model serves your requests. The platform records each successful teacher response as a training sample. Each sample counts toward your `minSamples` target.
</Step>
<Step title="Training">
When the task has `minSamples` successful samples, and `autoTrain` is on, the platform builds training and eval datasets from the samples. It then trains a distilled model with a pre-configured recipe. The teacher model continues to serve your requests during training.
</Step>
<Step title="Live">
When `autoDeploy` is on, the platform deploys the trained model to a dedicated GPU and does a smoke test. Then it moves your traffic to the trained model automatically. You do not change your code. `autoDeploy` is temporarily disabled; until it opens up, deploy the trained model yourself as described in [After training completes](#after-training-completes).
</Step>
</Steps>
{/* TODO:MEDIA - Lifecycle diagram: request flow routing to teacher during collecting/training, then cutting over to the distilled model once live. Commented out until the media is ready.
<div style={{
border: "2px dashed #999",
borderRadius: "12px",
backgroundColor: "var(--inference-white-darker, #e8e0d9)",
padding: "2rem",
textAlign: "center",
color: "#666",
fontSize: "0.9rem",
margin: "1.5rem 0"
}}>
<p style={{margin: 0, fontWeight: 500}}>📍 TODO:MEDIA</p>
<p style={{margin: "0.5rem 0 0 0"}}>Lifecycle diagram: collecting → training → live, showing traffic on the teacher model until the distilled model takes over.</p>
</div>
*/}
<Note>
AutoTrainer does not stop your service. If training, deployment, or routing fails, your requests go to the teacher model and continue to operate.
</Note>
## Quickstart [#quickstart]
The fastest way to use AutoTrainer is `createAutoTrainClient` from [`@inference/sdk`](https://www.npmjs.com/package/@inference/sdk). This function binds a task, a teacher model, a system prompt, and an optional output schema into one typed client.
<Steps>
<Step title="Install the SDK">
<Metadata text="platform/train/autotrainer-install" />
```bash
npm install @inference/sdk openai zod
```
</Step>
<Step title="Create an AutoTrainer client">
<Metadata text="platform/train/autotrainer-quickstart[series=autotrainer-quickstart]" />
```typescript
import OpenAI from "openai";
import { z } from "zod";
import { createInferenceClient } from "@inference/sdk";
const client = createInferenceClient({
openai: OpenAI,
apiKey: process.env.INFERENCE_API_KEY,
});
const personExtractor = client.createAutoTrainClient({
task: "person-extraction",
teacherModel: "glm-5.2",
systemPrompt: "Extract information about people from text.",
schema: z.object({
name: z.string(),
age: z.number(),
}),
config: {
minSamples: 100,
autoTrain: true,
autoDeploy: true,
},
});
```
</Step>
<Step title="Call it like any other model">
<Metadata text="platform/train/autotrainer-quickstart[series=autotrainer-quickstart]" />
```typescript
// Typed as { name: string; age: number }
const person = await personExtractor.run({
input: "Hello, my name is John and I am 30 years old.",
});
```
The teacher model serves the first `minSamples` successful calls. Then the platform trains and deploys the distilled model. After that, the same call returns responses from the distilled model.
</Step>
</Steps>
For free-form text tasks, omit `schema`. Then `run()` resolves to the raw response string.
`run()` also accepts these optional fields together with `input`:
| Field | Description |
| ------------- | --------------------------------------------------------------------------------------------------------------- |
| `metadata` | Key-value tags for the request. The SDK sends them as `x-inference-metadata-*` headers for filters and metrics. |
| `environment` | Environment label for the request, for example `production` or `staging`. |
| `projectId` | Replaces the client's project for this call. |
## Use the API directly [#use-the-api-directly]
The SDK is a convenience, not a requirement. An AutoTrainer request is a standard OpenAI-compatible chat completions call plus four `x-inference-*` headers. You can send it from all languages that have an HTTP client, for example Rust, Go, Java, or Ruby.
To make a direct call:
* Set the request `model` to `"auto-train"`. This value is only a marker. The server selects the teacher model or the distilled model.
* Send all four `x-inference-*` configuration headers with each request. The SDK adds these headers for you. Direct calls must include them.
<CodeGroup>
<Metadata text="platform/train/autotrainer-direct[series=autotrainer-direct]" />
```typescript TypeScript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create(
{
model: "auto-train",
messages: [
{ role: "system", content: "Extract information about people from text." },
{ role: "user", content: "John is 30 years old." },
],
},
{
headers: {
"x-inference-task-id": "person-extraction",
"x-inference-auto-train-teacher-model": "glm-5.2",
"x-inference-config-min-samples": "100",
"x-inference-config-auto-train": "true",
"x-inference-config-auto-deploy": "true",
},
},
);
console.log(response.choices[0].message.content);
```
<Metadata text="platform/train/autotrainer-direct[series=autotrainer-direct]" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="auto-train",
messages=[
{"role": "system", "content": "Extract information about people from text."},
{"role": "user", "content": "John is 30 years old."},
],
extra_headers={
"x-inference-task-id": "person-extraction",
"x-inference-auto-train-teacher-model": "glm-5.2",
"x-inference-config-min-samples": "100",
"x-inference-config-auto-train": "true",
"x-inference-config-auto-deploy": "true",
},
)
print(response.choices[0].message.content)
```
<Metadata text="platform/train/autotrainer-direct" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-task-id: person-extraction" \
-H "x-inference-auto-train-teacher-model: glm-5.2" \
-H "x-inference-config-min-samples: 100" \
-H "x-inference-config-auto-train: true" \
-H "x-inference-config-auto-deploy: true" \
-d '{
"model": "auto-train",
"messages": [
{"role": "system", "content": "Extract information about people from text."},
{"role": "user", "content": "John is 30 years old."}
]
}'
```
<Metadata text="platform/train/autotrainer-direct" />
```rust Rust
use reqwest::Client;
use serde_json::{json, Value};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let response: Value = Client::new()
.post("https://api.inference.net/v1/chat/completions")
.bearer_auth(std::env::var("INFERENCE_API_KEY")?)
.header("x-inference-task-id", "person-extraction")
.header("x-inference-auto-train-teacher-model", "glm-5.2")
.header("x-inference-config-min-samples", "100")
.header("x-inference-config-auto-train", "true")
.header("x-inference-config-auto-deploy", "true")
.json(&json!({
"model": "auto-train",
"messages": [
{"role": "system", "content": "Extract information about people from text."},
{"role": "user", "content": "John is 30 years old."}
]
}))
.send()
.await?
.json()
.await?;
println!("{}", response["choices"][0]["message"]["content"]);
Ok(())
}
```
<Metadata text="platform/train/autotrainer-direct" />
```go Go
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
)
func main() {
body, _ := json.Marshal(map[string]any{
"model": "auto-train",
"messages": []map[string]string{
{"role": "system", "content": "Extract information about people from text."},
{"role": "user", "content": "John is 30 years old."},
},
})
req, _ := http.NewRequest(
"POST",
"https://api.inference.net/v1/chat/completions",
bytes.NewReader(body),
)
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFERENCE_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("x-inference-task-id", "person-extraction")
req.Header.Set("x-inference-auto-train-teacher-model", "glm-5.2")
req.Header.Set("x-inference-config-min-samples", "100")
req.Header.Set("x-inference-config-auto-train", "true")
req.Header.Set("x-inference-config-auto-deploy", "true")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
var result struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
panic(err)
}
fmt.Println(result.Choices[0].Message.Content)
}
```
<Metadata text="platform/train/autotrainer-direct" />
```ruby Ruby
require "json"
require "net/http"
require "uri"
uri = URI("https://api.inference.net/v1/chat/completions")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{ENV.fetch("INFERENCE_API_KEY")}"
request["Content-Type"] = "application/json"
request["x-inference-task-id"] = "person-extraction"
request["x-inference-auto-train-teacher-model"] = "glm-5.2"
request["x-inference-config-min-samples"] = "100"
request["x-inference-config-auto-train"] = "true"
request["x-inference-config-auto-deploy"] = "true"
request.body = JSON.generate({
model: "auto-train",
messages: [
{ role: "system", content: "Extract information about people from text." },
{ role: "user", content: "John is 30 years old." }
]
})
response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
http.request(request)
end
result = JSON.parse(response.body)
puts result.dig("choices", 0, "message", "content")
```
</CodeGroup>
## Configuration reference [#configuration-reference]
| SDK option | Header | Default | Description |
| ------------------- | -------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task` | `x-inference-task-id` | Required | Stable identifier for the task that AutoTrainer distills. Samples collect for each task separately. Each task serves a maximum of one distilled model. |
| `teacherModel` | `x-inference-auto-train-teacher-model` | Required | The model that serves requests during sample collection. The distilled model learns from its responses. All models that the gateway can route are applicable. |
| `config.minSamples` | `x-inference-config-min-samples` | Required | The number of successful teacher responses to collect before training starts. The minimum value is 2. |
| `config.autoTrain` | `x-inference-config-auto-train` | `true` | Controls if training starts automatically when the task has `minSamples` samples. Set it to `false` to collect samples without training. |
| `config.autoDeploy` | `x-inference-config-auto-deploy` | `true` | Controls if the platform deploys the trained model and moves the traffic automatically. Set it to `false` to train without a traffic change. The teacher model then continues to serve requests. Temporarily disabled; see [After training completes](#after-training-completes). |
Notes on the values:
* **`teacherModel`**: Select a model that already does the task well. The quality of the distilled model is limited by the quality of the teacher samples. For models on Inference.net, use the model identifier, for example `glm-5.2`, with your Inference API key. Provider models, for example OpenAI or Anthropic, also operate here. Use the same provider headers as other [gateway requests](/api/api-quickstart).
* **`minSamples`**: Set a value that is sufficient for the full range of your real inputs. More diverse samples make a better distilled model. For most tasks, a few hundred samples is a good initial value. Only successful teacher responses count toward the target. Errors and refused requests do not count.
* **`autoTrain` / `autoDeploy`**: These are independent gates. With `autoTrain: true` and `autoDeploy: false`, the platform trains a model, but the teacher model continues to serve. Use this configuration to examine the trained model before you move the traffic.
You can change `minSamples`, `autoTrain`, and `autoDeploy` at all times. The most recent values on incoming requests apply.
## Keep the task identity stable [#keep-the-task-identity-stable]
The combination of the **task, the system prompt, the output schema, and the teacher model** identifies what AutoTrainer distills. Keep all four stable so that the samples collect toward the same model.
If you change the prompt, the schema, or the teacher model, a new identity starts. A task serves a maximum of one distilled model at a time:
* The new identity collects samples from zero. The teacher model serves its requests. The previous model continues to serve requests that use the old prompt or schema.
* When the replacement model completes training, the platform moves the task's deployment to the new model. During the change, the teacher model serves the task. The change usually takes some minutes. Then the traffic moves to the new model.
* The platform records each retrain as a new version of the same deployment. The version history is the retrain record for the task.
* If you use an old prompt or schema again, its old model does not come back. A retired identity routes to the teacher model permanently.
## Structured output [#structured-output]
Give a Zod schema to `createAutoTrainClient` as shown in the [quickstart](#quickstart). Then `run()` returns parsed, typed output. The SDK sends an OpenAI structured-outputs `response_format`, and the schema becomes part of the task identity.
When you call the API directly:
* For structured output, use `response_format` with the type `json_schema`. AutoTrainer requests do not support the `json_object` type.
* Each AutoTrainer request must include a `system` or `developer` message.
* Requests with a schema and requests without a schema are different identities. Do not mix them in one task.
## Monitor progress [#monitor-progress]
AutoTrainer progress shows on the page of the task that you defined. To see it:
1. Open your project in the dashboard.
2. Go to **Observability**, then **Tasks**.
3. Select the task that you set in your AutoTrainer requests.
4. Select the **Auto-training** tab.
The tab shows the samples collected toward the target, the `autoTrain` and `autoDeploy` flags, and links to the training job, the model, and the deployment when the platform creates them.
{/* TODO:MEDIA - Screenshot of the task Auto-training tab showing sample progress, config flags, and links to the training job and deployment. Commented out until the media is ready.
<div style={{
border: "2px dashed #999",
borderRadius: "12px",
backgroundColor: "var(--inference-white-darker, #e8e0d9)",
padding: "2rem",
textAlign: "center",
color: "#666",
fontSize: "0.9rem",
margin: "1.5rem 0"
}}>
<p style={{margin: 0, fontWeight: 500}}>📍 TODO:MEDIA</p>
<p style={{margin: "0.5rem 0 0 0"}}>Screenshot of the task's Auto-training tab in the dashboard.</p>
</div>
*/}
## After training completes [#after-training-completes]
The trained model is yours. You own the weights, and you can deploy them anywhere: on our platform, on another provider, or on your own hardware.
* **Check the evals.** Training runs evals automatically, and the results show on the training job page. To compare it against other models on your live traffic, run an [AutoEval](/platform/eval/auto-evals/run-an-auto-eval) from the **Compare Models** tab.
* **Download the weights.** Open the training job page and download the model weights. What you do with them is up to you.
* **Deploy it.** [Deploy the model on our platform](/platform/deploy/deploy-a-model), or [contact us](https://inference.net/meet-with-us/) and we will help set up a dedicated deployment sized to your traffic and latency needs.
## Billing [#billing]
There is no separate AutoTrainer fee. You pay for the parts that it controls:
* **Teacher requests** bill the same as other inference requests through the gateway.
* **The training run** bills the same as a [manual training run](/platform/train/launch-a-run).
* **The deployment** bills the same as a dedicated deployment. Retrains use the task's current deployment again. They do not add a new deployment.
When the distilled model is live, the cost for each request decreases from the teacher-model price to your dedicated deployment price.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Tasks" icon="tag" href="/platform/gateway/tasks">
How task tags group requests for metrics, evals, and training.
</Card>
<Card title="Manage deployments" icon="server" href="/platform/deploy/manage-and-monitor">
Monitor, version, and scale the deployment that serves your distilled model.
</Card>
</CardGroup>
# Choose a Recipe (https://docs.inference.net/platform/train/choose-a-recipe)
A recipe is a pre-configured training setup. It includes a vetted base model, optimized training parameters, and compute configuration. You don't need ML expertise to pick one.
## What a recipe includes [#what-a-recipe-includes]
* **Base model** — selected by the Inference team for quality on the task type
* **Optimized training parameters** — learning rate, epochs, and other hyperparameters
* **Compute configuration** — minimum 8 GPUs per training run
## How to choose [#how-to-choose]
Pick based on **task difficulty and capability needs**, not specific model names. Start small and scale up only if your eval results show the smaller recipe isn't cutting it.
| Tier | Best for |
| ---------- | ------------------------------------------------------------------------------------------------ |
| **Tiny** | High-throughput tasks where speed matters most — simple classification, extraction, tagging |
| **Small** | Fast inference with more capability — structured output, entity extraction, routing |
| **Medium** | Good balance of speed and intelligence — summarization, Q\&A, agentic tasks that need to be fast |
| **Large** | Complex reasoning, multi-step tasks, difficult agentic use cases |
Some recipes offer specific capabilities (like multimodal support) that are only available with certain base models. Choose those when your task requires them.
## Available recipes [#available-recipes]
These are the pre-built recipes on the platform, each configured and tested by the Inference team. Your project's own recipes appear alongside these in the recipe picker.
| Recipe | Base Model | Parameters | Compute | Description |
| ---------- | ------------- | ---------- | ------- | ----------------------------------------------------------------------- |
| **Tiny** | Qwen 3.5 0.8B | 0.8B | 8x H100 | Tiny and incredibly fast. Best for high-volume, low-complexity tasks. |
| **Small** | Qwen 3.5 4B | 4B | 8x H100 | Small and incredibly fast. A good default when latency is the priority. |
| **Medium** | Qwen 3.5 9B | 9B | 8x H100 | Good balance of speed and intelligence. Good as a fast agentic model. |
| **Large** | Qwen 3.5 27B | 27B | 8x H200 | Large and intelligent. Great for difficult agentic use cases. |
All recipes run on 8 GPUs and include optimized training parameters: the **Tiny**, **Small**, and **Medium** tiers use 8x H100, and **Large** uses 8x H200. You don't need to configure any of this — just pick the tier that fits your task.
# Launch a Training Run (https://docs.inference.net/platform/train/launch-a-run)
Once you have your [prerequisites](/platform/train/overview#prerequisites) ready, launching a training run takes a few minutes.
<Note>
This guide shows the manual path. If you do not want to manage datasets and deployment, [AutoTrainer](/platform/train/autotrainer) does these steps automatically from your live traffic.
</Note>
## Start a new job [#start-a-new-job]
Go to **Fine-tuning** in the dashboard sidebar and click **New Training Job**.
<Frame caption="New Training Job form in the dashboard.">
<img src="/images/training/training-create.png" alt="New Training Job form in the dashboard" />
</Frame>
<Steps>
<Step title="Select a training dataset">
Choose the dataset the model will learn from. You can [build a dataset from live traffic](/platform/datasets/build-from-traffic), [upload a JSONL file](/platform/datasets/upload-a-dataset), or use [task tags](/platform/gateway/tasks) to filter captured traffic for clean, focused samples.
</Step>
<Step title="Select an eval dataset">
Choose the dataset used for [evaluations throughout training](/platform/train/mid-training-evals). Must have zero overlap with training data. The platform judges the model against a default rubric automatically.
</Step>
<Step title="Select a recipe">
Choose a [recipe](/platform/train/choose-a-recipe) based on task difficulty and capability needs.
</Step>
<Step title="Start training">
Review your selections and click **Start Training** to queue the job.
</Step>
</Steps>
## Training job lifecycle [#training-job-lifecycle]
Once started, your job moves through these statuses:
| Status | What's happening |
| ---------------------- | --------------------------------------------------------------------- |
| **Exporting datasets** | Your training and eval data is being prepared |
| **Queued** | Job is waiting for available compute |
| **Starting** | GPUs are being allocated and the training environment is initializing |
| **Running** | The model is actively training |
| **Cycling** | The run is rolling over to a new provider instance |
| **Completed** | Training finished successfully |
If something goes wrong, the job will show one of these:
| Status | What it means |
| ------------- | ------------------------------------------------------------------------------------------------------------- |
| **Failed** | Something went wrong — check the [logs](/platform/train/mid-training-evals#logs) on the training details page |
| **Cancelled** | The run was cancelled |
| **Timed out** | The run exceeded the maximum allowed duration |
Training can take anywhere from around 10 minutes to 10+ hours depending on dataset size and recipe. Once running, [monitor progress](/platform/train/mid-training-evals) from the training details page.
## How training is scored [#how-training-is-scored]
At each checkpoint, the platform runs your eval dataset through the model-in-progress and an LLM judge scores the outputs against built-in quality and similarity rubrics. Mid-training evals use those scores to decide when the model has learned enough, and when to stop early to prevent overfitting.
The eval dataset drives this: pick a small, stable set of hard examples that stress-test the model, and keep it disjoint from your training data. If a model trains on eval examples, the eval becomes meaningless.
# Monitor a Training Run (https://docs.inference.net/platform/train/mid-training-evals)
Once a training job starts, the training details page gives you real-time visibility into what's happening. A progress bar shows percentage complete and the current status of the run.
## Training graphs [#training-graphs]
Four graphs update as training progresses:
| Graph | What it measures | What to look for |
| ----------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Loss** | How far the model's predictions are from expected output | Decreasing = learning. Flattening = model has learned what it can from the data. |
| **Learning rate** | How much weights update at each training step | Warm-up then decay schedule — configured by the recipe automatically. |
| **Gradient norm** | Gradient magnitude during backpropagation | Steady or decreasing = stable. Persistent spikes may indicate a data quality issue. |
| **Eval score** | Average score on the eval dataset at each checkpoint | Trending up = model is improving at your task. This is the most direct signal that training is working. |
<Frame caption="Training graphs showing loss, learning rate, gradient norm, and eval score during an active run.">
<img src="/images/training/training-graphs.png" alt="Training details page showing the four training graphs" />
</Frame>
## Evaluations [#evaluations]
The platform runs evaluations at three points during a training job:
1. **Before training** — establishes a baseline score for the model before any weight updates
2. **During training** — at each checkpoint, the model runs your eval dataset and an LLM judge scores the outputs against built-in quality and similarity rubrics
3. **After training** — a final evaluation on the completed model
## Checkpoints [#checkpoints]
Training saves checkpoints at regular intervals. If a run fails after a checkpoint, it can be resumed from the last saved state rather than starting over.
## Logs [#logs]
The **Logs** tab shows output from all GPUs during training. Use it to debug issues or see what's happening under the hood. You can filter logs by type — `warn`, `error`, and others — to focus on what matters.
# Train (https://docs.inference.net/platform/train/overview)
<Warning>
**Enterprise required.** Training is available on Enterprise plans. [Contact us](https://inference.net/meet-with-us/) to enable it for your team.
</Warning>
<Note>
You do not have to manage the training loop yourself. [AutoTrainer](/platform/train/autotrainer) collects samples from your live traffic, trains a distilled model, and moves your traffic to it automatically. It is in open beta and available to all users.
</Note>
Fine-tuning takes a small open-source model and trains it on your data so it gets really good at one specific task. The result is a model that's smaller, faster, and cheaper to run than the general-purpose model it replaces, while being more accurate for your workload. You don't need to be an ML engineer to use it.
Training is one optimization lever, not the only one. Many teams get massive value from [observability](/platform/gateway/overview) + [AutoEvals](/platform/eval/auto-evals/run-an-auto-eval) + prompt tuning alone. Fine-tuning is for when you've confirmed via evals that off-the-shelf models aren't good enough for a specific task, and you have the data to prove it.
<Frame>
<iframe style="{ width: "100%", aspectRatio: "16 / 9", border: 0, display: "block" }" src="https://www.youtube.com/embed/m3Ayw92jpL0?list=PLJzp7SN2tfJsRAU9VGSfSo60CyDJzqhLP&rel=0" title="Fine-Tune a Custom LLM on Your Own Data | Inference platform" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" />
</Frame>
## Prerequisites [#prerequisites]
Training requires two things before you start:
1. **A training dataset** - the data the model learns from. Build it from [captured traffic](/platform/datasets/build-from-traffic) or [upload your own](/platform/datasets/upload-a-dataset).
2. **An eval dataset** - measures learning progress. Must have **zero overlap** with training data.
## Key concepts [#key-concepts]
| Concept | Description |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Recipe** | A pre-configured training setup with a vetted base model, optimized parameters, and compute config. You pick by task difficulty, the platform handles the ML complexity. |
| **Training dataset** | The data the model learns from. Diversity and quality matter most. Build from captured traffic or upload your own. |
| **Eval dataset** | A separate dataset that measures learning progress. Must have zero overlap with training data to prevent overfitting. |
| **Rubric** | The quality criteria that guide training. Mid-training evals score the model against it. If the rubric is wrong, the model optimizes for the wrong thing. |
| **Mid-training evals** | Periodic quality checks during training run on a fixed schedule. They give you visibility into whether training is working — if scores degrade, you spot it and stop the run yourself. |
## Why fine-tune [#why-fine-tune]
* **Reduce latency** - a smaller, task-specific model responds faster than a general-purpose one
* **Reduce cost** - smaller models cost less to serve at scale
* **Improve accuracy** - a model trained on your data is optimized for exactly what you need
* **Maintain ownership** - you own the model artifact and control where it runs
## What makes good training data [#what-makes-good-training-data]
The quality of your trained model depends directly on the quality of your data. A few principles:
* **Diversity matters most.** Training data should cover the range of inputs the model will see in production — different phrasings, edge cases, varying complexity. A narrow dataset produces a narrow model.
* **Real traffic beats synthetic data.** Production inputs reflect what users actually send. [Build datasets from live traffic](/platform/datasets/build-from-traffic) when possible.
* **Scope to a single task.** A dataset built from mixed traffic teaches the model many things poorly. Use [task tags](/platform/gateway/tasks) to filter for one objective at a time.
* **More is generally better**, but quality trumps volume. A thousand clean, diverse examples outperform ten thousand repetitive ones.
For eval data, the goal is different: pick a small, stable set of hard examples that stress-test the model. Don't change your eval dataset often — it's your benchmark. See [Datasets](/platform/datasets/overview) for more on building both types.
## Next steps [#next-steps]
<CardGroup cols="2">
<Card title="Choose a recipe" icon="book-open" href="/platform/train/choose-a-recipe">
Pick a pre-configured training setup for your task.
</Card>
<Card title="Launch a training run" icon="rocket" href="/platform/train/launch-a-run">
End-to-end flow from datasets to queued job.
</Card>
<Card title="AutoTrainer" icon="wand-magic-sparkles" href="/platform/train/autotrainer">
Distill a task onto a smaller model automatically from your live traffic.
</Card>
<Card title="Monitor a training run" icon="chart-line" href="/platform/train/mid-training-evals">
Track progress, graphs, and logs during training.
</Card>
<Card title="Deploy a trained model" icon="server" href="/platform/deploy/deploy-a-model">
Ship your model to a dedicated GPU.
</Card>
</CardGroup>
# Troubleshooting Training Failures (https://docs.inference.net/platform/train/troubleshooting)
Training runs should generally succeed without intervention. If a run fails, it usually indicates an underlying issue worth investigating.
## Common failures [#common-failures]
| Failure | What happened | What to do |
| ------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Model collapse / divergence** | Model trained but performs poorly. May have overfit despite good eval scores. | Check data quality and diversity. Try a different recipe. |
| **Insufficient data** | Model fails to generalize well. | Add more representative samples to the training dataset. |
| **Poor data quality** | Training data isn't diverse or heterogeneous enough. | Curate a more varied dataset. Ensure it covers edge cases. |
| **Limit reached** | Your provider cluster's concurrent training slots are full or the run needs billing authorization. | Wait for an active run to finish, or authorize billing, then retry. |
## What you see on failure [#what-you-see-on-failure]
* An error message on the training run
* A description of the failure condition
* A retry button
## When to reach out [#when-to-reach-out]
If training fails unexpectedly, contact support. Failures usually indicate something worth looking into — the team can help diagnose whether it's a data issue, a recipe mismatch, or a platform problem.
<Card title="Talk to an engineer" icon="calendar" href="https://inference.net/meet-with-us/">
Meet with our team to discuss training failures or get help with your approach.
</Card>
## Training succeeded? [#training-succeeded]
Model trained and eval scores look good? Deploy it to a dedicated GPU and start serving traffic.
<Card title="Deploy your model" icon="server" href="/platform/deploy/overview">
Ship your trained model to production.
</Card>
# API Keys and Authentication (https://docs.inference.net/reference/api-keys)
## Managing API keys [#managing-api-keys]
A default API key is created for your personal organization's first project when you sign up. Every other team and project starts without keys — create them from the [API Keys](https://inference.net/dashboard/api-keys) page in the sidebar, where you can also delete existing ones.
The API Keys page shows all keys across your organization and which project each one belongs to. When you click **Create API Key**, the new key is scoped to whichever project you currently have selected.
## Project scoping [#project-scoping]
API keys are scoped to a project. The key you use determines which project's resources you're interacting with — custom models, datasets, training runs, evals, and observability metrics are all grouped by the project the key belongs to.
## Permissions [#permissions]
When creating an API key, you can grant **read** access, **write** access, or both. For example, a read-only key for pulling metrics, or a key with write access for uploading datasets and triggering training runs.
## Authentication [#authentication]
Pass the key as a Bearer token in the `Authorization` header, or set it as your API key in whichever SDK you're using. The same key works across:
* **Inference API** — calling models (both off-the-shelf and custom deployments)
* **Inference platform** — SDK and CLI operations
* **Observability** — routing traffic through the gateway
* **MCP server** — connecting compatible AI coding assistants to Inference platform resources
## Best practices [#best-practices]
* Keep API keys out of source control
* Rotate keys periodically
* Use environment variables to store keys in your application
# Billing and Credits (https://docs.inference.net/reference/billing)
## What costs what [#what-costs-what]
| Activity | How it's billed |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Inference calls** (API) | Per token, varies by model |
| **Eval judge calls** | Per token (these are full LLM inferences) |
| **Training compute** | Per GPU-hour — H100 at $4/GPU/hr, H200 at $5/GPU/hr. Built-in recipes run 8 GPUs on a single node ($32/hr or $40/hr per node) |
<Note>
Dedicated deployments currently run under a preview gate: every plan is capped at one active deployment, and per-hour deployment billing is not yet enabled. Storage (training datasets, model artifacts, trace dataset exports) is not metered.
</Note>
## Credits [#credits]
Your account has a credit balance. All platform activity draws from this balance. Make sure you have sufficient credits before launching training runs or deploying models — both will fail if credits run out.
## Checking your usage [#checking-your-usage]
View your current balance and usage breakdown in the [Inference.net dashboard](https://inference.net/dashboard).
# Glossary (https://docs.inference.net/reference/glossary)
| Term | Definition |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Task** | A user-defined objective that groups LLM calls (e.g., "summarize document," "classify ticket"). Tasks persist even as the implementation changes. |
| **Rubric** | A plain English description of how to judge model output on a quality dimension. Scored numerically by an LLM judge. |
| **Recipe** | A pre-configured training setup including base model, training parameters, and compute config. |
| **Dataset** | A curated set of inference samples used for evaluation or training. |
| **Eval dataset** | A dataset used to measure model quality. Should remain stable over time. Must not overlap with training data. |
| **Training dataset** | A dataset the model learns from during fine-tuning. Evolves as you iterate on data quality. |
| **Inference** | A single LLM request-response pair captured by the platform. |
| **Deployment** | A trained model running on a dedicated GPU, accessible via an OpenAI-compatible API. |
| **Mid-training eval** | A periodic evaluation run during training that scores model checkpoints against the rubric. |
| **LLM-as-a-judge** | The evaluation mechanism where an LLM scores model outputs against rubric criteria. |
| **TTFT** | Time to first token. Measures streaming responsiveness. |
| **Overfitting** | When a model memorizes training data instead of learning generalizable patterns. Detected by degrading eval scores. |
| **Distillation** | Training a smaller model to replicate the quality of a larger model, reducing cost and latency. |
| **Agent** | An entity that groups runs sharing a stable `agentId`/`agentName` into one workspace in the Agents dashboard, where its metrics, sessions, trajectories, signals, and HALO analysis live. |
| **Trajectory** | The user-facing name for a trace in the dashboard. The URL slug and CLI still say `traces`. |
| **Signal** | A plain-language classifier you define once; an LLM judge labels each span, trace, or session into metrics you can chart, filter, and alert on. |
| **HALO** | "Hierarchical Agent Loop Optimization." The open-source agent-loop engine that reads traces, finds systemic failure modes across runs, and writes concrete fixes cited to specific traces. |
| **AutoEval** | An automated comparison that samples recent traffic, replays it against candidate models, scores outputs with LLM judges, and recommends a better fit with supporting evidence. |
# Rate Limits (https://docs.inference.net/reference/rate-limits)
For detailed rate limit information including current limits by tier, see the [Inference API rate limits](/api/rate-limits) page.
## What happens when you hit a limit [#what-happens-when-you-hit-a-limit]
You'll receive a `429 Too Many Requests` response. Back off and retry with exponential backoff.
## Deployment-specific limits [#deployment-specific-limits]
Self-serve deployments on a single GPU have inherent throughput limits. If traffic exceeds capacity, requests slow down and eventually return 429s. For higher throughput, see [Scale to Production](/platform/deploy/scale-to-production).
## Requesting higher limits [#requesting-higher-limits]
If you need higher rate limits, [talk to the team](https://inference.net/meet-with-us/).
# Inference platform workflow (https://docs.inference.net/workflow)
Inference platform is built around two products.
* **Tracing** is for agent improvement. The tracing SDKs capture the full execution of your agents and AI apps: LLM calls, tool calls, framework steps, and any custom spans you wrap. Halo, our open-source agent-loop optimizer, reads those traces and writes up what to fix in prompts, tools, and the harness itself.
* **Gateway** is for inference observability and training task-specific models. Gateway sits between your app and your LLM provider, recording every request. From that recorded traffic, you can build datasets, run evals, fine-tune smaller models, and deploy them on dedicated GPUs.
Tracing and Gateway stand alone. Many teams start with one and add the other later. They also compose: when your agent calls LLMs through Gateway, the trace spans and the gateway records line up against the same requests.
The platform also serves models through an OpenAI-compatible API: open-source models, popular closed-source models, and Inference.net-trained models (like [Schematron](/workhorse-models/schematron) for structured data extraction).
***
## Tracing [#tracing]
Capture full traces of your agents and AI apps. The Inference Tracing SDKs collect LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Then Halo (our open-source [agent-loop optimizer](https://github.com/context-labs/halo)) reads your traces and surfaces concrete things to improve in your prompts, tools, and agent harness.
**What you'll do:**
* [Capture your first trace](/get-started/capture-first-trace) by installing the tracing SDK and pointing it at the Inference platform
* [Analyze your traces](/get-started/analyze-traces) in the Traces and Agents dashboards, then run Halo to find what to fix
* Instrument [providers and frameworks](/integrations/traces/overview) like OpenAI, Anthropic, LangChain, LangGraph, Vercel AI SDK, OpenAI Agents, LiveKit, Pi Agent, legacy PI AI, Pydantic AI, and more
* Add stable [agent identity](/integrations/traces/agent-identity) so the Agents dashboard groups runs correctly
**Outcome:** Deep visibility into how your agents behave end to end, plus an automated reviewer that points you to the highest-impact fixes.
<Card title="Get started with Tracing" icon="arrow-right" href="/get-started/capture-first-trace">
Install the SDK and capture your first trace.
</Card>
***
## Gateway [#gateway]
Record and analyze your production LLM traffic. Inference Gateway sits between your app and your LLM provider, capturing every request, response, cost, and latency metric with less than 10ms of overhead. Keep using any provider or model — Gateway is transparent.
**What you'll do:**
* [Integrate with your LLM provider](/platform/gateway/integrate) to start capturing traffic
* [Define tasks](/platform/gateway/tasks) to group LLM calls by objective (e.g. "summarize docs", "classify tickets")
* [Explore metrics](/platform/gateway/metrics-explorer) for cost, latency, errors, and token usage across all your calls
* [Browse individual inferences](/platform/gateway/inference-viewer) to inspect raw requests and responses
**Outcome:** Full visibility into how your AI features perform in production — broken down by model, task, and provider.
<Card title="Get started with Gateway" icon="arrow-right" href="/platform/gateway/overview">
Set up Gateway and start capturing LLM traffic.
</Card>
***
## Datasets [#datasets]
Curate collections of LLM inputs and outputs for evaluation and training. Datasets can come from your live production traffic captured through Gateway, or from files you upload directly.
**What you'll do:**
* [Build datasets from traffic](/platform/datasets/build-from-traffic) by filtering captured inferences and saving them
* [Upload your own data](/platform/datasets/upload-a-dataset) as JSONL files when you have curated examples or are migrating from another platform
* Understand [dataset formats](/platform/datasets/formats) and the schema your data needs to follow
**Outcome:** Clean, representative datasets scoped to specific tasks — ready to power evals and training.
<Card title="Get started with Datasets" icon="arrow-right" href="/platform/datasets/overview">
Build or upload your first dataset.
</Card>
***
## Eval [#eval]
Measure model quality on your live traffic with AutoEvals. The platform samples recent production requests, replays each against a set of candidate models, scores every output with LLM judges, and recommends the best model with the evidence behind it.
**What you'll do:**
* [Run an AutoEval](/platform/eval/auto-evals/run-an-auto-eval) from the **Compare Models** tab on a task or agent
* [Interpret the results](/platform/eval/auto-evals/interpret-results) — the recommendation, the leaderboard, and individual judged samples
* Follow the guided [Compare Models walkthrough](/get-started/compare-models-auto-evals) from integration to a completed comparison
**Outcome:** A data-driven answer to which model is best for your workload, and a validated quality signal that can guide training.
<Card title="Get started with Eval" icon="arrow-right" href="/platform/eval/overview">
Compare models on your traffic with AutoEvals.
</Card>
***
## Train [#train]
Fine-tune a task-specific model on your production data. The result is a model that's smaller, faster, and cheaper to run than the general-purpose model it replaces — while being more accurate for your workload. You don't need to be an ML engineer to use it.
**What you'll do:**
* [Choose a recipe](/platform/train/choose-a-recipe) — a pre-configured training setup with a vetted base model and optimized parameters
* [Launch a training run](/platform/train/launch-a-run) with your training dataset, eval dataset, and rubric
* [Monitor mid-training evals](/platform/train/mid-training-evals) to track quality scores as the model learns
**Outcome:** A trained, task-specific model that's been validated against your eval dataset — ready to deploy.
<Card title="Get started with Train" icon="arrow-right" href="/platform/train/overview">
Fine-tune a model on your data.
</Card>
***
## Deploy [#deploy]
Ship your trained model to a dedicated GPU with an OpenAI-compatible API. The API uses the same base URL and API key as the rest of the Inference platform — switching from an off-the-shelf model to your custom model is a one-line code change.
**What you'll do:**
* [Deploy a trained model](/platform/deploy/deploy-a-model) to a dedicated GPU in a few clicks
* [Call your deployment](/platform/deploy/call-your-deployment) using the same OpenAI-compatible SDK you already use
* [Manage and monitor](/platform/deploy/manage-and-monitor) your deployment lifecycle, scaling, and performance
**Outcome:** A production endpoint serving your custom model — and the beginning of the next improvement loop. Deploy, observe, eval, retrain.
<Card title="Get started with Deploy" icon="arrow-right" href="/platform/deploy/overview">
Ship your model to a dedicated GPU.
</Card>
***
## Pick your starting point [#pick-your-starting-point]
<CardGroup cols="2">
<Card title="Capture your first trace" icon="route" href="/get-started/capture-first-trace">
Install the tracing SDK and capture LLM calls, tool calls, and agent steps.
</Card>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Inspect trace trees and run Halo to find what to improve.
</Card>
<Card title="Record your first LLM call" icon="bolt" href="/get-started/record-first-call">
Route traffic through Inference Gateway to capture LLM calls and view metrics.
</Card>
<Card title="Compare models with AutoEvals" icon="scale-balanced" href="/get-started/compare-models-auto-evals">
Replay your traffic against candidate models and get a recommended winner.
</Card>
<Card title="Train and deploy a model" icon="brain" href="/get-started/train-and-deploy">
The full loop: data, training, and a production endpoint.
</Card>
<Card title="Use the Inference API" icon="code" href="/api/api-quickstart">
Call open-source, popular closed-source, and Inference.net models directly.
</Card>
</CardGroup>
# ClipTagger (https://docs.inference.net/workhorse-models/cliptagger)
<img src="/images/cliptagger/grass-x-inference.png" style="{ width: '100%', borderRadius: '0.5rem', margin: '1rem 0' }" />
<Warning>
ClipTagger-12b is currently in **private preview** and is not publicly available. To request access, email [support@inference.net](mailto:support@inference.net) or [schedule a call](https://inference.net/meet-with-us/). The examples below assume you have been granted access.
</Warning>
## Introduction [#introduction]
Programmatic video understanding enables developers to build applications that extract structured information from video content at scale, transforming raw frames into searchable, actionable data. By processing video frames through specialized vision language models, developers can build systems that understand visual content as deeply as they understand text.
[ClipTagger-12b](https://huggingface.co/inference-net/ClipTagger-12b) is a custom 12-billion parameter [vision language model](https://huggingface.co/blog/vlms) (VLM) designed for video understanding at massive scale. The model was trained by the [Inference.net](https://inference.net) research team in collaboration with [Grass](https://grass.io) to meet their trillion-scale video frame captioning needs.
Read the [announcement blog post](https://inference.net/blog/cliptagger-12b).
On our benchmarks, ClipTagger-12b achieves quality on par with GPT-4.1 and outperforms Claude 4 Sonnet, while delivering 15–17x lower costs. We use Gemini-2.5-Pro as an independent judge to evaluate caption quality. The model enables teams operating at scale to perform temporal video understanding, content indexing, and automated video analysis workflows at state-of-the-art model quality while maintaining control over their data and costs.
## Key Features [#key-features]
* **Frontier-quality** – Comparable to GPT-4.1 and better than Claude 4 Sonnet.
* **Cost-Efficient** – 15x lower cost than GPT-4.1 and 17x lower than Claude 4 Sonnet.
* **Production-Ready** – Battle-tested on trillion-scale video frame captioning workloads.
* **Temporal Consistency** – Maintains semantic consistency across video frames.
* **Open Source** – The model is open source and available on [Hugging Face](https://huggingface.co/inference-net/cliptagger-12b).
## Processing at Scale [#processing-at-scale]
For large-scale workloads, we recommend using our asynchronous APIs:
<CardGroup cols="2">
<Card title="Batch API" icon="list-check" href="/api/async-inference/batch-api">
Handle huge frame queues asynchronously. Submit up to 50,000 requests in one batch, track progress, and get results within 24 hours with optional webhooks.
</Card>
<Card title="Group API" icon="layer-group" href="/api/async-inference/group">
Best for smaller collections of up to 50 frames. Send a simple JSON payload, monitor the whole group as one job, and receive a single callback when it finishes.
</Card>
</CardGroup>
## Getting Started [#getting-started]
You'll need an [Inference.net](https://inference.net) account and API key. See our [Quick Start Guide](/api/api-quickstart) for setup instructions.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) and set the base URL to `https://api.inference.net/v1`.
<CodeGroup>
<Metadata text="video-understanding/import_openai[series=video_understanding]" />
```typescript TypeScript
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
<Metadata text="video-understanding/import_openai[series=video_understanding]" />
```python Python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ.get("INFERENCE_API_KEY"),
)
```
</CodeGroup>
## Basic Usage [#basic-usage]
The model processes a single video frame per request. In order to process a video, you'll need to send a request for each frame you want to analyze.
**Processing every frame of a video is not always necessary.** Depending on your use case, you may want to process a subset of frames, such as keyframes or frames with high motion.
You can use our [Group API](/api/async-inference/group) to submit multiple frames at once, and received a response once all frames have been processed. This is often ideal for processing 10-20 frames per video.
The model requires a specific system prompt and user prompt to generate structured output. Here's a complete example:
<CodeGroup>
<Metadata text="video-understanding/basic_usage[series=video_understanding]" />
```typescript TypeScript
import fs from "fs";
import { z } from "zod";
// Define the response schema using Zod
const ClipTaggerResponseSchema = z.object({
description: z.string(),
objects: z.array(z.string()).max(10),
actions: z.array(z.string()).max(5),
environment: z.string(),
content_type: z.string(),
specific_style: z.string(),
production_quality: z.string(),
summary: z.string(),
logos: z.array(z.string())
});
// Type inference from the schema
type ClipTaggerResponse = z.infer<typeof ClipTaggerResponseSchema>;
// System and user prompts (use exactly as shown for best results)
const SYSTEM_PROMPT = "You are an image annotation API trained to analyze YouTube video keyframes. You will be given instructions on the output format, what to caption, and how to perform your job. Follow those instructions. For descriptions and summaries, provide them directly and do not lead them with 'This image shows' or 'This keyframe displays...', just get right into the details.";
const USER_PROMPT = `You are an image annotation API trained to analyze YouTube video keyframes. You must respond with a valid JSON object matching the exact structure below.
Your job is to extract detailed **factual elements directly visible** in the image. Do not speculate or interpret artistic intent, camera focus, or composition. Do not include phrases like "this appears to be", "this looks like", or anything about the image itself. Describe what **is physically present in the frame**, and nothing more.
Return JSON in this structure:
{
"description": "A detailed, factual account of what is visibly happening (4 sentences max). Only mention concrete elements or actions that are clearly shown. Do not include anything about how the image is styled, shot, or composed. Do not lead the description with something like 'This image shows' or 'this keyframe is...', just get right into the details.",
"objects": ["object1 with relevant visual details", "object2 with relevant visual details", ...],
"actions": ["action1 with participants and context", "action2 with participants and context", ...],
"environment": "Detailed factual description of the setting and atmosphere based on visible cues (e.g., interior of a classroom with fluorescent lighting, or outdoor forest path with snow-covered trees).",
"content_type": "The type of content it is, e.g. 'real-world footage', 'video game', 'animation', 'cartoon', 'CGI', 'VTuber', etc.",
"specific_style": "Specific genre, aesthetic, or platform style (e.e., anime, 3D animation, mobile gameplay, vlog, tutorial, news broadcast, etc.)",
"production_quality": "Visible production level: e.g., 'professional studio', 'amateur handheld', 'webcam recording', 'TV broadcast', etc.",
"summary": "One clear, comprehensive sentence summarizing the visual content of the frame. Like the description, get right to the point.",
"logos": ["logo1 with visual description", "logo2 with visual description", ...]
}
Rules:
- Be specific and literal. Focus on what is explicitly visible.
- Do NOT include interpretations of emotion, mood, or narrative unless it's visually explicit.
- No artistic or cinematic analysis.
- Always include the language of any text in the image if present as an object, e.g. "English text", "Japanese text", "Russian text", etc.
- Maximum 10 objects and 5 actions.
- Return an empty array for 'logos' if none are present.
- Always output strictly valid JSON with proper escaping.
- Output **only the JSON**, no extra text or explanation.`;
// Function to encode image from URL
async function encodeImageUrl(imageUrl) {
const response = await fetch(imageUrl);
const buffer = Buffer.from(await response.arrayBuffer());
return buffer.toString('base64');
}
// Example usage
const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg";
const base64Image = await encodeImageUrl(imageUrl);
const messages = [
{ role: "system", content: SYSTEM_PROMPT },
{
role: "user",
content: [
{ type: "text", text: USER_PROMPT },
{
type: "image_url",
image_url: {
url: `data:image/jpeg;base64,${base64Image}`,
detail: "high"
},
},
],
},
];
const response = await openai.chat.completions.create({
model: "inference-net/cliptagger-12b",
messages: messages,
temperature: 0.1,
max_tokens: 2000,
response_format: { type: "json_object" },
});
// Parse and validate the JSON response
const rawResult = JSON.parse(response.choices[0].message.content);
const result = ClipTaggerResponseSchema.parse(rawResult); // This will throw if the response doesn't match the schema
// Now 'result' is fully typed as ClipTaggerResponse
console.log(JSON.stringify(result, null, 2));
// You can access typed properties
console.log(`Description: ${result.description}`);
console.log(`Objects found: ${result.objects.length}`);
```
<Metadata text="video-understanding/basic_usage[series=video_understanding]" />
```python Python
import base64
import requests
import json
from typing import List
from pydantic import BaseModel, conlist
# Define the response model using Pydantic
class ClipTaggerResponse(BaseModel):
description: str
objects: conlist(str, max_length=10)
actions: conlist(str, max_length=5)
environment: str
content_type: str
specific_style: str
production_quality: str
summary: str
logos: List[str]
# System and user prompts (use exactly as shown for best results)
SYSTEM_PROMPT = "You are an image annotation API trained to analyze YouTube video keyframes. You will be given instructions on the output format, what to caption, and how to perform your job. Follow those instructions. For descriptions and summaries, provide them directly and do not lead them with 'This image shows' or 'This keyframe displays...', just get right into the details."
USER_PROMPT = """
You are an image annotation API trained to analyze YouTube video keyframes. You must respond with a valid JSON object matching the exact structure below.
Your job is to extract detailed **factual elements directly visible** in the image. Do not speculate or interpret artistic intent, camera focus, or composition. Do not include phrases like "this appears to be", "this looks like", or anything about the image itself. Describe what **is physically present in the frame**, and nothing more.
Return JSON in this structure:
{
"description": "A detailed, factual account of what is visibly happening (4 sentences max). Only mention concrete elements or actions that are clearly shown. Do not include anything about how the image is styled, shot, or composed. Do not lead the description with something like 'This image shows' or 'this keyframe is...', just get right into the details.",
"objects": ["object1 with relevant visual details", "object2 with relevant visual details", ...],
"actions": ["action1 with participants and context", "action2 with participants and context", ...],
"environment": "Detailed factual description of the setting and atmosphere based on visible cues (e.g., interior of a classroom with fluorescent lighting, or outdoor forest path with snow-covered trees).",
"content_type": "The type of content it is, e.g. 'real-world footage', 'video game', 'animation', 'cartoon', 'CGI', 'VTuber', etc.",
"specific_style": "Specific genre, aesthetic, or platform style (e.e., anime, 3D animation, mobile gameplay, vlog, tutorial, news broadcast, etc.)",
"production_quality": "Visible production level: e.g., 'professional studio', 'amateur handheld', 'webcam recording', 'TV broadcast', etc.",
"summary": "One clear, comprehensive sentence summarizing the visual content of the frame. Like the description, get right to the point.",
"logos": ["logo1 with visual description", "logo2 with visual description", ...]
}
Rules:
- Be specific and literal. Focus on what is explicitly visible.
- Do NOT include interpretations of emotion, mood, or narrative unless it's visually explicit.
- No artistic or cinematic analysis.
- Always include the language of any text in the image if present as an object, e.g. "English text", "Japanese text", "Russian text", etc.
- Maximum 10 objects and 5 actions.
- Return an empty array for 'logos' if none are present.
- Always output strictly valid JSON with proper escaping.
- Output **only the JSON**, no extra text or explanation.
"""
# Function to encode image from URL
def encode_image_url(image_url):
response = requests.get(image_url)
return base64.b64encode(response.content).decode('utf-8')
# Example usage
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
base64_image = encode_image_url(image_url)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": [
{"type": "text", "text": USER_PROMPT},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}",
"detail": "high"
},
},
],
},
]
response = client.chat.completions.create(
model="inference-net/cliptagger-12b",
messages=messages,
temperature=0.1,
max_tokens=2000,
response_format={"type": "json_object"},
)
# Parse and validate the JSON response
raw_result = json.loads(response.choices[0].message.content)
result = ClipTaggerResponse(**raw_result) # This will raise ValidationError if the response doesn't match the schema
# Now 'result' is a typed Pydantic model instance
print(result.model_dump_json(indent=2))
# You can access typed properties
print(f"Description: {result.description}")
print(f"Objects found: {len(result.objects)}")
```
</CodeGroup>
<Note>
ClipTagger-12b requires these exact system and user prompts for optimal performance. The model was specifically trained with this prompt structure to ensure consistent, high-quality output.
</Note>
## Output [#output]
We're going to use the image below as an example.
<img src="/images/cliptagger/cliptagger-example.png" style="{ width: '100%', borderRadius: '0.5rem', margin: '1rem 0' }" />
The model reliably outputs this exact JSON structure for every frame, with all fields always present. Even when certain elements aren't detected (like `actions` or `logos` in the example above), the model returns empty arrays rather than omitting fields.
```json
{
"description": "A wooden boardwalk path extends from the foreground into the distance, cutting through a field of tall, vibrant green grass. The path is flanked on both sides by the dense grass. In the background, a line of trees is visible on the horizon under a blue sky with scattered white clouds.",
"objects": [
"Wooden boardwalk",
"Tall green grass",
"Blue sky",
"White clouds",
"Trees"
],
"actions": [],
"environment": "An outdoor, natural landscape, likely a marsh or wetland, on a clear day. The scene is characterized by a wooden boardwalk, lush green vegetation, and a bright blue sky with wispy clouds.",
"content_type": "real-world footage",
"specific_style": "landscape photography",
"production_quality": "professional photography",
"summary": "A wooden boardwalk path winds through a lush green field under a bright blue sky with scattered clouds.",
"logos": []
}
```
<Note>
The JSON response schema shown above is the model's standard output format. All fields will always be present in the response, with empty arrays returned when no relevant elements are detected (e.g., no logos or actions in the frame).
</Note>
## Keyframe selection [#keyframe-selection]
There are several ways to select keyframes for a video.
### 1. Frame-by-frame analysis [#1-frame-by-frame-analysis]
The simplest way to select keyframes is to process every frame of the video. This is the most accurate way to select keyframes, but it is also the most expensive.
## Example Use Cases [#example-use-cases]
**Video Search & Discovery** - Build searchable video databases by indexing structured metadata and tracking object persistence across frames.
**Content Moderation** - Automated content analysis with consistent categorization for content type detection and quality verification.
**Accessibility** - Generate consistent alt-text and scene summaries for video content with frame-by-frame descriptions.
**Ad Verification** - Ensure sponsored content compliance by tracking product visibility and logo appearances.
## Best Practices [#best-practices]
1. **Use the exact prompts** – The provided system and user prompts are optimized for best results
2. **Set low temperature** – Use `temperature: 0.1` for consistent output
3. **Enable JSON mode** – Always set `response_format: {"type": "json_object"}`
4. **Process frames systematically** – Maintain temporal order for better analysis
5. **Batch similar content** – Group frames from the same video or scene
## Limitations [#limitations]
* Single video frame per request only. Use the [Group API](/api/async-inference/group) to process multiple frames at once.
* Maximum image size: 1MB
* Supported formats: JPEG, PNG, WebP, GIF
* English-only descriptions (though it can identify text in other languages)
## Support [#support]
For technical support or custom deployment options:
* Email: [support@inference.net](mailto:support@inference.net)
* [Schedule a call](https://inference.net/sales) for a consultation
<Note>
cliptagger-12B was developed in partnership with Grass to process over a billion videos while maintaining perfect schema adherence.
</Note>
# Schematron (https://docs.inference.net/workhorse-models/schematron)
## Introduction [#introduction]
Schematron is a family of long-context models we trained to extract clean, typed JSON from messy HTML. Purpose-built for web scraping, product data ingestion, and converting arbitrary pages into structured data, these models take a user-defined schema and reliably output conforming JSON.
Schematron V2 Small delivers the highest extraction quality, approaching frontier-model accuracy on complex schemas. Schematron V2 Turbo is optimized for throughput and cost, processing pages significantly faster while still exceeding the quality of the previous-generation Schematron 3B. Both models excel at processing long HTML inputs with strict schema adherence.
To learn more about how we trained Schematron and what it can do, see the [announcement blog post](https://inference.net/blog/schematron?utm_source=docs).
<Tip>
For best results, pre-clean HTML with `lxml` to remove scripts, styles, and boilerplate before sending it to the model. See [Preprocess HTML (recommended)](#preprocess-html-recommended).
Alternatives like [Readability](https://github.com/mozilla/readability) or regex can work; `lxml` is simply recommended for this model because it matches how the training data was preprocessed. When preprocessing, err on the side of removing less content, as the model will still be able to extract the information you need.
</Tip>
## Quickstart [#quickstart]
<CardGroup cols="3">
<Card title="Schematron V2 Small" icon="rocket" href="https://inference.net/models/schematron-v2-small">
Highest extraction quality for complex schemas and long pages.
</Card>
<Card title="Schematron V2 Turbo" icon="bolt" href="https://inference.net/models/schematron-v2-turbo">
Optimized for maximum throughput and lowest cost at scale.
</Card>
<Card title="Notebook Example" icon="code" href="https://github.com/context-labs/inference-samples/blob/main/examples/schematron-scrape-companies/schematron-scrape-companies.ipynb">
End-to-end company scraping walkthrough using Schematron.
</Card>
</CardGroup>
<Note>
You'll need an Inference.net account, API key, and the OpenAI SDK configured with `base_url="https://api.inference.net/v1"`. See the [Quick Start Guide](/api/api-quickstart) for setup details.
</Note>
<Note>
**TypeScript Users**: The examples below use OpenAI SDK v6 and Zod v3. Install the compatible version with:
```bash
npm install zod@^3
```
</Note>
## Basic Usage: Extract from HTML into a Pydantic model [#basic-usage-extract-from-html-into-a-pydantic-model]
Use a typed model (e.g., Pydantic) to define your schema. Pass the model's JSON Schema and the raw HTML to the model, enable JSON mode, and validate the response.
<Info>
This model does not use user/system prompts. It only uses the schema to extract the data.
</Info>
<CodeGroup>
<Metadata text="json-extraction/basic_usage[series=basic_usage]" />
```typescript TypeScript
import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
const openai = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
// 1) Define your schema (nested data and lists are supported)
const Product = z.object({
name: z.string(),
price: z
.number()
.describe(
"Primary price of the product."
),
specs: z
.record(z.string())
.describe(
"Specs of the product."
),
tags: z
.array(z.string())
.describe("Tags assigned of the product."),
});
// 2) Messy HTML (could be the full page; trim to the relevant region when possible)
const html = `
<div id="item">
<h2 class="title">MacBook Pro M3</h2>
<p>Price: <b>$2,499.99</b> USD</p>
<ul info>
<li>RAM: 16GB</li>
<li>Storage: 512GB SSD</li>
</ul>
<span class="tag">laptop</span>
<span class="tag">professional</span>
<span class="tag">macbook</span>
<span class="tag">apple</span>
</div>
`;
// 3) Chat Completions extraction (typed validation)
const resp = await openai.chat.completions.parse({
model: "inference-net/schematron-v2-small",
messages: [{ role: "user", content: html }],
response_format: zodResponseFormat(Product, "product"),
});
console.log(resp.choices[0].message.content);
```
<Metadata text="json-extraction/basic_usage[series=basic_usage]" />
```python Python
import os
from pydantic import BaseModel, Field
from openai import OpenAI
# 1) Define your schema (nested data and lists are supported)
class Product(BaseModel):
name: str
price: float = Field(
...,
description="Primary price of the product.",
)
specs: dict[str, str] = Field(
default_factory=dict,
description="Specs of the product.",
)
tags: list[str] = Field(
default_factory=list,
description="Tags assigned of the product.",
)
# 2) Messy HTML (could be the full page; trim to the relevant region when possible)
html = """
<div id="item">
<h2 class="title">MacBook Pro M3</h2>
<p>Price: <b>$2,499.99</b> USD</p>
<ul info>
<li>RAM: 16GB</li>
<li>Storage: 512GB SSD</li>
</ul>
<span class="tag">laptop</span>
<span class="tag">professional</span>
<span class="tag">macbook</span>
<span class="tag">apple</span>
</div>
"""
# 3) Client setup
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ.get("INFERENCE_API_KEY"),
)
resp = client.beta.chat.completions.parse(
model="inference-net/schematron-v2-small",
messages=[
{"role": "user", "content": html},
],
response_format=Product,
)
print(resp.choices[0].message.parsed.model_dump_json(indent=2))
```
<Metadata text="json-extraction/basic_usage" />
```bash cURL
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "inference-net/schematron-v2-small",
"messages": [
{ "role": "user", "content": "<div id=\"item\">\n <h2 class=\"title\">MacBook Pro M3</h2>\n <p>Price: <b>$2,499.99</b> USD</p>\n <ul info>\n <li>RAM: 16GB</li>\n <li>Storage: 512GB SSD</li>\n </ul>\n <span class=\"tag\">laptop</span>\n <span class=\"tag\">professional</span>\n <span class=\"tag\">macbook</span>\n <span class=\"tag\">apple</span>\n</div>" }
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "product",
"strict": true,
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number", "description": "Primary price of the product." },
"specs": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Specs of the product." },
"tags": { "type": "array", "items": { "type": "string" }, "description": "Tags assigned of the product." }
},
"required": ["name", "price", "specs", "tags"],
"additionalProperties": false
}
}
}
}'
```
</CodeGroup>
## Output [#output]
For the HTML above, Schematron will produce strictly valid JSON that conforms to your schema. A representative output is:
<Metadata text="json-extraction/output" />
```json
{
"name": "MacBook Pro M3",
"price": 2499.99,
"specs": {
"RAM": "16GB",
"Storage": "512GB SSD"
},
"tags": [
"laptop",
"professional",
"macbook",
"apple"
]
}
```
## Best Practices [#best-practices]
1. **Keep temperature at zero** – This model is trained to perform best with 0 temperature.
2. **Provide a clear schema** – Pass a JSON Schema or typed model (e.g., `Product.model_json_schema()`). Include required fields and types, optionally describing them directly within the schema (supported by Pydantic and Zod). For example:
<CodeGroup>
```typescript TypeScript
const Product = z.object({
name: z.string().describe("Exact product name as shown in the title or primary heading."),
price: z
.number()
.describe(
"Primary price of the product."
),
specs: z.object({
ram: z.string().describe("RAM capacity."),
storage: z.string().describe("Storage capacity and type."),
}).describe("Key attributes describing the product."),
});
```
```python Python
from pydantic import BaseModel, Field
class Specs(BaseModel):
ram: str = Field(..., description="RAM capacity.")
storage: str = Field(..., description="Storage capacity and type.")
class Product(BaseModel):
name: str = Field(
...,
description=(
"Exact product name as shown in the title or primary heading."
),
)
price: float = Field(
...,
description="Primary price of the product.",
)
specs: Specs = Field(
...,
description="Key attributes describing the product.",
)
```
</CodeGroup>
4. **Pre-clean HTML (recommended)** – Remove scripts, styles, and boilerplate to improve accuracy and reduce tokens. See [Preprocess HTML (recommended)](#preprocess-html-recommended). Alternatives (Readability, Trafilatura, BeautifulSoup, regex) are fine; `lxml` aligns with training.
5. **Validate on ingest** – Parse with Pydantic (or your validator) and handle validation errors explicitly. Although Schematron should always return valid JSON matching your schema, it's still a good idea to validate the response.
## Preprocess HTML (recommended) [#preprocess-html-recommended]
Schematron models were trained on HTML that had been pre-cleaned with `lxml.html.clean.Cleaner` to strip scripts, styles, and inline JavaScript. Aligning your preprocessing with training typically improves extraction quality and consistency.
<Metadata text="json-extraction/schema_examples[series=best_practices]" />
```python Python
import lxml.html as LH
from lxml.html.clean import Cleaner
HTML_CLEANER = Cleaner(
scripts=True,
javascript=True,
style=True,
inline_style=True,
safe_attrs_only=False,
)
def strip_noise(html: str) -> str:
"""Remove scripts, styles, and JavaScript from HTML using lxml."""
if not html or not html.strip():
return ""
try:
doc = LH.fromstring(html)
cleaned = HTML_CLEANER.clean_html(doc)
return LH.tostring(cleaned, encoding="unicode")
except Exception:
return ""
```
Notes:
* You do not have to use `lxml`. Alternatives like Readability (often more aggressive), Trafilatura, BeautifulSoup, or even targeted regex can be acceptable.
* Use whichever tool best fits your content; `lxml` is recommended for this model because it matches how the training data was preprocessed.
## Capabilities & Access [#capabilities--access]
### Key Features [#key-features]
* **Schema-first extraction** – Drive output with your own JSON Schema or a typed model (e.g. Pydantic) and get strictly formatted JSON back.
* **Long-context HTML** – Trained for long documents with up to 128K-token context and robust to noisy markup.
* **Strict JSON mode** – 100% schema adherence.
* **Cost-efficient quality** – Matches the quality of frontier models at a significantly lower cost.
### Models [#models]
* **inference-net/schematron-v2-small** – Best quality for complex schemas or very long pages.
* **inference-net/schematron-v2-turbo** – Optimized for maximum throughput and lowest cost at scale.
### Access Schematron [#access-schematron]
**Serverless API**
* [Schematron V2 Small](https://inference.net/models/schematron-v2-small)
* [Schematron V2 Turbo](https://inference.net/models/schematron-v2-turbo)
## Processing at Scale [#processing-at-scale]
For large-scale extraction, use our asynchronous APIs:
<CardGroup cols="2">
<Card title="Batch API" icon="list-check" href="/api/async-inference/batch-api">
Submit up to 50,000 extraction jobs at once, watch their status asynchronously, and receive optional webhook updates within a 24-hour window.
</Card>
<Card title="Group API" icon="layer-group" href="/api/async-inference/group">
Send smaller groups (≤ 50 requests) in one JSON payload, follow the entire group as a single job, and get one callback when everything is done.
</Card>
</CardGroup>
## Limitations [#limitations]
* **Context window** – Up to 128K tokens per request. Truncate or chunk very large pages.
* **Ambiguous fields** – If a field requires summarization or synthesis, ensure your schema/description is explicit about expectations. Schematron does not support a prompt or directions, instead all information needs to be passed through the schema.
## Legacy Models [#legacy-models]
<Warning>
The first-generation **Schematron 3B** (`inference-net/schematron-3b`) and **Schematron 8B** (`inference-net/schematron-8b`) models have been retired. Requests to these identifiers are now transparently routed to their V2 successors to avoid breakage:
* Calls to `inference-net/schematron-3b` are served by `inference-net/schematron-v2-turbo`.
* Calls to `inference-net/schematron-8b` are served by `inference-net/schematron-v2-small`.
V2 pricing applies on these rewritten requests — see the pricing pages for [Schematron V2 Turbo](https://inference.net/models/schematron-v2-turbo) and [Schematron V2 Small](https://inference.net/models/schematron-v2-small) for current per-token rates. We strongly recommend updating your model identifiers explicitly so your code, dashboards, and receipts all reflect the model you intend to call.
The original open-source weights remain available on Hugging Face for self-hosting:
* [Schematron-8B on Hugging Face](https://huggingface.co/inference-net/Schematron-8B)
* [Schematron-3B on Hugging Face](https://huggingface.co/inference-net/Schematron-3B)
</Warning>
## Support [#support]
For technical support or custom deployment options:
* Email: [support@inference.net](mailto:support@inference.net)
* [Schedule a call](https://inference.net/sales)