# API Quickstart
Source: https://docs.inference.net/api/api-quickstart
Get started with the Inference.net API
The Inference.net API is OpenAI-compatible, so you can use the OpenAI SDK or plain HTTP to make requests. There are three ways to use it:
1. **Call an open-source model** — run models hosted on Inference.net directly.
2. **Proxy through Catalyst** — route requests to any provider (OpenAI, Anthropic, etc.) through the Catalyst gateway for observability, evals, and cost tracking.
3. **Call your custom model** — hit a model you've fine-tuned and deployed on the platform.
## Get an API Key
Visit [inference.net](https://inference.net) and create an account.
On the dashboard, go to the **API Keys** tab in the left sidebar. Create a new key or use the default key.
```bash theme={"system"}
export INFERENCE_API_KEY=
```
***
## 1. Call an Open-Source Model
Run open-source models hosted on Inference.net. No provider API key needed — just your Inference API key. Browse available models at [inference.net/models](https://inference.net/models).
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [{ role: "user", content: "What is the meaning of life?" }],
stream: true,
});
for await (const chunk of response) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[{"role": "user", "content": "What is the meaning of life?"}],
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="", flush=True)
```
```bash cURL theme={"system"}
curl -N https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "user", "content": "What is the meaning of life?"}
],
"stream": true
}'
```
This works with any model on the platform, including our purpose-built [Schematron](/workhorse-models/schematron) models for structured data extraction.
***
## 2. Proxy Through Catalyst
Route requests to any LLM provider (OpenAI, Anthropic, Groq, etc.) through the Catalyst gateway. You keep your existing provider API key — the gateway adds observability, cost tracking, and eval-readiness with roughly 10ms of added latency.
Your Inference project API key authenticates with the gateway. Your provider API key is forwarded to the provider via the `x-inference-provider-api-key` header.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "What is the meaning of life?" }],
});
console.log(response.choices[0].message.content);
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "What is the meaning of life?"}],
)
print(response.choices[0].message.content)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "x-inference-provider: openai" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4.1",
"messages": [
{"role": "user", "content": "What is the meaning of life?"}
]
}'
```
For detailed setup guides per provider (Anthropic, Groq, Cerebras, OpenRouter, and more), see the [Integrations](/integrations/overview) docs.
***
## 3. Call Your Custom Model
Hit a model you've fine-tuned and deployed on Inference.net. The model path is your team slug followed by the deployment name, shown on your deployment's detail page in the dashboard.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "your-team/your-model",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="your-team/your-model",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "your-team/your-model",
"messages": [
{"role": "user", "content": "Hello, world!"}
]
}'
```
Learn more about deploying models in the [Deploy](/platform/deploy/overview) docs.
***
## Headers Reference
| Header | Required | Description |
| ------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer ` — authenticates the request. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `Content-Type` | Yes | Must be `application/json`. |
| `x-inference-provider` | Proxy only | Routes the request to the correct provider: `openai`, `anthropic`, `groq`, `cerebras`, etc. |
| `x-inference-provider-api-key` | Proxy only | Your provider's API key. The gateway forwards it downstream. For Anthropic's native SDK, use `x-api-key` instead. |
| `x-inference-provider-url` | No | Routes to any OpenAI-compatible provider by base URL, even if it doesn't have a dedicated integration. |
| `x-inference-environment` | No | Tags requests with an environment label, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics in the dashboard. |
| `x-inference-metadata-*` | No | Attach arbitrary metadata to a request. The prefix is stripped to form the key — e.g., `x-inference-metadata-chat-id: abc123` stores `chat-id: abc123`. You can filter inferences and create datasets based on these keys in the dashboard. |
## Supported Request Parameters
The API supports the standard OpenAI chat completions parameters:
| Parameter | Type | Description |
| ------------------- | --------- | ---------------------------------------------------------------------------------------------------- |
| `model` | `string` | The model to use. |
| `messages` | `array` | The conversation messages. |
| `stream` | `boolean` | Whether to stream the response. |
| `max_tokens` | `integer` | Maximum number of tokens to generate. |
| `temperature` | `number` | Sampling temperature (0–2). |
| `top_p` | `number` | Nucleus sampling threshold. |
| `frequency_penalty` | `number` | Penalizes repeated tokens based on frequency. |
| `presence_penalty` | `number` | Penalizes tokens based on whether they've appeared. |
| `response_format` | `object` | Set to `{"type": "json_object"}` or a JSON schema for [structured outputs](/api/structured-outputs). |
| `tools` | `array` | Tool/function definitions for [function calling](/api/function-calling). |
Need a parameter that isn't listed here? [Contact us](mailto:support@inference.net) and we'll add it.
## Next Steps
Set up Catalyst with OpenAI, Anthropic, Groq, and other providers.
Get typed JSON responses from your API calls.
Process up to 50,000 requests in a single batch job.
Explore all models available on Inference.net.
# Batch API
Source: https://docs.inference.net/api/async-inference/batch-api
Process jobs asynchronously with Batch API.
Learn how to use our OpenAI-compatible Batch API to send asynchronous groups of inference requests to Inference.net, with nearly unlimited rate limits and fast completion times. The service is ideal for processing a large number of jobs that don't require immediate responses.
Batch API is currently compatible with all the [models](https://inference.net/models) we offer.
Use [https://batch.inference.net/v1](https://batch.inference.net/v1) for all Batch API requests (including Files and Batches). Do not use [https://api.inference.net/v1](https://api.inference.net/v1) for batch jobs.
## Overview
While some uses require you to send synchronous requests, there are many cases where requests do not need an immediate response or rate limits prevent you from executing a large number of queries quickly. Batch processing jobs are often helpful in use cases like:
1. Extracting structured data from a large number of documents.
2. Generating synthetic data for training.
3. Translating a large number of documents into other languages.
4. Summarizing a large number of customer interactions.
Inference.net's Batch API offers a straightforward set of endpoints that allow you to upload a batch of requests, kick off a batch processing job, query for the status of the batch, and eventually retrieve the collected results when the batch is complete.
Compared to using standard endpoints directly, Batch API has:
1. **Higher rate limits:** Substantially more headroom compared to the [synchronous APIs](/api/rate-limits).
2. **Fast completion times:** Each batch completes within 24 hours (and often much more quickly).
## Getting Started
You'll need an Inference.net account and API key to use the Batch API. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net's Batch API using the OpenAI SDK, set the base URL to `https://batch.inference.net/v1`.
In this example, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://batch.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://batch.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL theme={"system"}
export INFERENCE_API_KEY=
# All Batch API requests use https://batch.inference.net/v1
```
## Running A Batch Processing Job
### 1. Preparing Your Batch File
Prepare a `.jsonl` file where each line is a separate JSON object that represents an individual request.
Each JSON object must be on a single line and cannot contain any line breaks.
Each JSON object must include the following fields:
* `custom_id`: A unique identifier for the request. This is used to reference the request's results after completion. It must be unique for each request in the file.
* `method`: The HTTP method to use for the request. Currently, only `POST` is supported.
* `url`: The URL to send the request to. Currently, only `/v1/chat/completions` and `/v1/completions` are supported.
* `body`: The request body, which contains the input for the inference request. The parameters in each line's `body` field are the same as the parameters for the underlying endpoint specified by the `url` field. See this [example](/quickstart#test-request) for more details.
Here's an example of an input file with 2 requests using the `/v1/chat/completions` endpoint.
```jsonl theme={"system"}
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}], "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "messages": [{"role": "system", "content": "You are an unhelpful assistant."}, {"role": "user", "content": "What is the capital of Belgium?"}], "max_tokens": 1000}}
```
And here is an example of an input file using the `/v1/completions` endpoint:
```jsonl theme={"system"}
{"custom_id": "request-1", "method": "POST", "url": "/v1/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "prompt": "What is the capital of France?", "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/completions", "body": {"model": "google/gemma-3-27b-instruct/bf-16", "prompt": "What is the capital of Belgium?", "max_tokens": 1000}}
```
### 2. Uploading Your Batch Input File
In order to create a Batch Processing job, you must first upload your input file.
```typescript TypeScript theme={"system"}
import fs from "fs";
const batchInputFile = await client.files.create({
file: fs.createReadStream("batchinput.jsonl"),
purpose: "batch",
});
console.log(batchInputFile);
```
```python Python theme={"system"}
batch_input_file = client.files.create(
file=open("batchinput.jsonl", "rb"),
purpose="batch",
)
print(batch_input_file)
```
```bash cURL theme={"system"}
curl https://batch.inference.net/v1/files \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-F purpose="batch" \
-F file="@batchinput.jsonl"
```
The response will look similar to this, depending on the language you are using:
```json JSON theme={"system"}
{
"id": "file-abc123"
}
```
### 3. Starting the Batch Processing Job
Once you've successfully uploaded your input file, you can use the ID of the file to create a batch.
In this case, let's assume the file ID is `file-abc123`.
For now, the completion window can only be set to `24h`.
To associate custom metadata with the batch, you can provide an optional `metadata` parameter.
This metadata is not used by Inference.net to complete requests, but it is included when retrieving the status of a batch so that you can associate custom metadata with the batch.
> **Note:** The Batch Processing job will begin processing immediately after creation.
Create the Batch
```typescript TypeScript theme={"system"}
import type { BatchCreateParams } from "openai/resources/batches";
const batch = await client.batches.create({
input_file_id: batchInputFile.id,
endpoint: "/v1/chat/completions",
completion_window: "24h",
metadata: {
description: "nightly eval job",
},
// Optional. Must be HTTPS.
webhook_url: "https://example.com/my_webhook",
} as BatchCreateParams);
console.log(batch);
```
```python Python theme={"system"}
batch = client.batches.create(
input_file_id=batch_input_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={
"description": "nightly eval job",
},
# Optional. Must be HTTPS.
extra_body={
"webhook_url": "https://example.com/my_webhook",
},
)
print(batch)
```
```bash cURL theme={"system"}
curl https://batch.inference.net/v1/batches \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file-abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"metadata": {
"description": "nightly eval job"
},
"webhook_url": "https://example.com/webhook"
}'
```
This request will return a batch object with metadata about your batch:
```json JSON theme={"system"}
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"errors": null,
"input_file_id": "file-abc123",
"completion_window": "24h",
"status": "validating",
"output_file_id": null,
"error_file_id": null,
"created_at": 1714508499,
"in_progress_at": 1714508500,
"expires_at": 1714536634,
"completed_at": null,
"failed_at": null,
"expired_at": null,
"request_counts": {
"total": 2,
"completed": 0,
"failed": 0
},
"metadata": {
"description": "nightly eval job"
}
}
```
Inference.net supports a `webhook_url` that you can set to receive a webhook notification when the batch is complete.
The `webhook_url` must be an HTTPS URL that can receive POST requests.
If no metadata is provided when the batch is created, the `metadata` field will be null.
Your webhook will receive a POST with a request JSON body that looks like this:
```json JSON theme={"system"}
{
"batch_id": "batch_abc123",
"status": "completed",
"metadata": {
"description": "nightly eval job"
}
}
```
The `webhook_url` parameter is not part of the official OpenAI SDK types. In TypeScript, cast the params as `BatchCreateParams` to avoid type errors. In Python, pass it via `extra_body`.
### 4. Checking the Status of a Batch
You can check the status of a batch at any time, which will also return a Batch object.
Check the status of a batch by retrieving it using the Batch ID assigned to it by Inference.net (represented here by `batch_abc123`).
```typescript TypeScript theme={"system"}
const retrievedBatch = await client.batches.retrieve(batch.id);
console.log(retrievedBatch);
```
```python Python theme={"system"}
retrieved_batch = client.batches.retrieve(batch.id)
print(retrieved_batch)
```
```bash cURL theme={"system"}
curl https://batch.inference.net/v1/batches/batch_abc123 \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json"
```
The status of a given Batch object can be any of the following:
| Status | Description |
| ------------ | ------------------------------------------------------------------------------ |
| validating | the input file is being validated before the batch can begin |
| failed | the input file has failed the validation process |
| in\_progress | the input file was successfully validated and the batch is currently being run |
| finalizing | the batch has completed and the results are being prepared |
| completed | the batch has been completed and the results are ready |
| expired | the batch was not able to be completed within the 24-hour time window |
| cancelling | the batch is being cancelled (may take up to 10 minutes) |
| cancelled | the batch was cancelled |
### 5. Retrieving the Results
You will receive an email notification when the batch is complete.
Once the batch is complete, you can download the output by making a request against the Files API using the `output_file_id` field from the Batch object.
Similarly, you can retrieve the error file (containing all failed requests) by making a request against the Files API using the `error_file_id` field from the Batch object.
```typescript TypeScript theme={"system"}
const fileResponse = await client.files.content(batch.output_file_id);
const fileContents = await fileResponse.text();
console.log(fileContents);
```
```python Python theme={"system"}
file_response = client.files.content(batch.output_file_id)
print(file_response.text)
```
```bash cURL theme={"system"}
curl https://batch.inference.net/v1/files/output-file-id/content \
-H "Authorization: Bearer $INFERENCE_API_KEY" > batch_output.jsonl
```
The output `.jsonl` file will have one response line for every successful request line in the input file. Any failed requests in the batch will have their error information written to an error file that can be found via the batch's `error_file_id`.
> Note that the output line order **may not match** the input line order.
Instead of relying on order to process your results, use the custom\_id field which will be present in each line of your output file and allow you to map requests in your input to results in your output.
```jsonl theme={"system"}
{"id": "batch_req_123", "custom_id": "request-2", "response": {"status_code": 200, "request_id": "req_123", "body": {"id": "chatcmpl-123", "object": "chat.completion", "created": 1711652795, "model": "google/gemma-3-27b-instruct/bf-16", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello."}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 2, "total_tokens": 24}, "system_fingerprint": "fp_123"}}, "error": null}
{"id": "batch_req_456", "custom_id": "request-1", "response": {"status_code": 200, "request_id": "req_789", "body": {"id": "chatcmpl-abc", "object": "chat.completion", "created": 1711652789, "model": "google/gemma-3-27b-instruct/bf-16", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I assist you today?"}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 20, "completion_tokens": 9, "total_tokens": 29}, "system_fingerprint": "fp_3ba"}}, "error": null}
```
## Listing All Batches
At any time, you can see all your batches. For users with many batches, you can use the `limit` and `after` parameters to paginate your results.
If an `after` parameter is provided, the list will return batches after the specified batch ID.
```typescript TypeScript theme={"system"}
const list = await client.batches.list({
limit: 10,
after: batch.id,
});
for await (const b of list) {
console.log(b);
}
```
```python Python theme={"system"}
batches = client.batches.list(limit=10, after=batch.id)
for b in batches:
print(b)
```
```bash cURL theme={"system"}
curl 'https://batch.inference.net/v1/batches?limit=10&after=batch_abc123' \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json"
```
## Batch Expiration
Batches that do not complete in time eventually move to an `expired` state; unfinished requests within that batch are cancelled, and any responses to completed requests are made available via the batch's output file. You will only be charged for tokens consumed from any completed requests.
Expired requests will be written to your error file with the message as shown below. You can use the `custom_id` to retrieve the request data for expired requests.
```jsonl theme={"system"}
{"id": "batch_req_123", "custom_id": "request-3", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
{"id": "batch_req_123", "custom_id": "request-7", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
```
## Rate Limits
Batch API rate limits are separate from existing per-model rate limits. A single batch may include up to 50,000 requests, and a batch input file can be up to 200 MB in size. If you need higher rate limits, please contact us at [support@inference.net](mailto:support@inference.net).
## Compatibility Notes
### 1. Batch Cancellation
Although the OpenAI SDK supports the ability to cancel an in-progress batch, Inference.net does not currently support batch cancellation. This is under development and will be available soon.
### 2. Model Availability
Inference.net's Batch Processing is compatible with all of Inference.net's supported models. See our list of [supported models](https://inference.net/models) for a complete list.
# Group API
Source: https://docs.inference.net/api/async-inference/group
Submit multiple asynchronous inference requests as a single group for easier tracking and webhook notifications.
Learn how to use our Group API to submit multiple inference requests together, perfect for processing related tasks that need to be tracked as a unit. The Group API supports both chat completions and text completions with up to 50 requests per group.
Group API is available for both `/v1/slow/group/chat/completions` and `/v1/slow/group/completions` endpoints.
You should not mix completion and chat-completion requests in the same group.
## Overview
The Group API provides a streamlined way to submit multiple asynchronous inference requests as a single unit. Unlike the Batch API which requires JSONL file uploads, the Group API accepts requests directly in the request body, making it ideal for:
* **Small to medium batches:** Process up to 50 requests at once
* **Related tasks:** Group related inference requests together
* **Webhook notifications:** Get notified when all requests in a group complete
* **Simpler integration:** No file uploads or JSONL formatting required
* **Faster implementation:** Direct JSON API calls without file management
## Group API vs Batch API
| Feature | Group API | Batch API |
| ---------------- | ----------------------------------- | ---------------------- |
| Maximum requests | 50 | 1,000,000 |
| Input format | JSON array in request body | JSONL file upload |
| File management | Not required | Required |
| Use case | Small batches, quick implementation | Large-scale processing |
| Webhook support | Yes | Yes |
| Completion time | 1-72 hours | 1-72 hours |
## Getting Started
### 1. Submit a Group Request
Submit multiple requests together by sending them as an array in the request body:
```typescript TypeScript theme={"system"}
const response = await fetch(
"https://api.inference.net/v1/slow/group/chat/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
{
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of France?" },
],
max_tokens: 100,
},
{
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of Germany?" },
],
max_tokens: 100,
},
],
webhook_id: "my-webhook-123", // Optional: attach a webhook for notifications
}),
},
);
const result = await response.json();
console.log(result); // { groupId: "group_abc123", groupSize: 2 }
```
```python Python theme={"system"}
import os
import requests
response = requests.post(
"https://api.inference.net/v1/slow/group/chat/completions",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
"max_tokens": 100,
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"},
],
"max_tokens": 100,
},
],
"webhook_id": "my-webhook-123", # Optional: attach a webhook for notifications
},
)
result = response.json()
print(result) # {"groupId": "group_abc123", "groupSize": 2}
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/group/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"}
],
"max_tokens": 100
}
],
"webhook_id": "my-webhook-123"
}'
```
The response will include a group ID and the number of requests:
```json JSON theme={"system"}
{
"groupId": "group_xY3kL9mN2pQ",
"groupSize": 2
}
```
### 2. Retrieve Group Results
Once your group is processed, retrieve all generation results using the group ID:
```typescript TypeScript theme={"system"}
const response = await fetch(
`https://api.inference.net/v1/slow/group/${groupId}/generations`,
{
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
},
},
);
const result = await response.json();
console.log(result.generations); // Array of all completed generations
```
```python Python theme={"system"}
import os
import requests
response = requests.get(
f"https://api.inference.net/v1/slow/group/{group_id}/generations",
headers={"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}"},
)
result = response.json()
print(result["generations"]) # Array of all completed generations
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/group/group_xY3kL9mN2pQ/generations \
-H "Authorization: Bearer $INFERENCE_API_KEY"
```
The response includes all generations in the group:
```json JSON theme={"system"}
{
"generations": [
{
"_id": "gen_abc123",
"state": "Success",
"request": {
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100
},
"response": {
"id": "gen_abc123",
"object": "chat.completion",
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}
},
{
"_id": "gen_def456",
"state": "Success",
"request": {
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"}
],
"max_tokens": 100
},
"response": {
"id": "gen_def456",
"object": "chat.completion",
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of Germany is Berlin."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}
}
]
}
```
## Using Webhooks
Attach a webhook to receive notifications when your group completes processing. Include the `webhook_id` when submitting the group:
```typescript TypeScript theme={"system"}
const response = await fetch(
"https://api.inference.net/v1/slow/group/chat/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
/* your requests */
],
webhook_id: "my-webhook-123",
}),
},
);
```
```python Python theme={"system"}
import os
import requests
response = requests.post(
"https://api.inference.net/v1/slow/group/chat/completions",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"requests": [
# your requests
],
"webhook_id": "my-webhook-123",
},
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/group/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type": "application/json" \
-d '{
"requests": [ ... ],
"webhook_id": "my-webhook-123"
}'
```
When all requests in the group complete, your webhook will receive a notification with:
* Group ID
* Completion status
* Summary of successful and failed requests
* Custom IDs for each request (if provided)
See our [Webhook Documentation](/api/async-inference/webhooks/getting-started-with-webhooks) for setup instructions.
## Text Completions Support
The Group API also supports text completions:
```typescript TypeScript theme={"system"}
const response = await fetch(
"https://api.inference.net/v1/slow/group/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
{
model: "google/gemma-3-27b-instruct/bf-16",
prompt: "The capital of France is",
max_tokens: 10,
},
{
model: "google/gemma-3-27b-instruct/bf-16",
prompt: "The capital of Germany is",
max_tokens: 10,
},
],
}),
},
);
```
```python Python theme={"system"}
import os
import requests
response = requests.post(
"https://api.inference.net/v1/slow/group/completions",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of France is",
"max_tokens": 10,
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of Germany is",
"max_tokens": 10,
},
],
},
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/group/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"requests": [
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of France is",
"max_tokens": 10
},
{
"model": "google/gemma-3-27b-instruct/bf-16",
"prompt": "The capital of Germany is",
"max_tokens": 10
}
]
}'
```
## Limits and Constraints
* **Maximum requests per group:** 50
* **Request format:** Direct JSON (no JSONL files required)
* **Supported endpoints:**
* `/v1/slow/group/chat/completions`
* `/v1/slow/group/completions`
* **Completion time:** 24-72 hours
* **Request expiration:** Groups expire after 72 hours if not completed
## Best Practices
1. **Group related requests:** Use groups for requests that logically belong together (e.g., analyzing multiple documents from the same source).
2. **Use webhooks for notifications:** Instead of polling, configure webhooks to be notified when your group completes.
3. **Handle individual failures:** Some requests in a group may fail while others succeed. Check each generation's status.
4. **Stay under limits:** Keep groups to 50 requests or less. For larger batches, use the [Batch API](/api/async-inference/batch-api).
5. **Include metadata:** Add custom IDs or metadata to your requests for easier tracking:
```json JSON theme={"system"}
{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [{"role": "user", "content": "..."}],
"metadata": {
"custom_id": "doc_123",
"type": "summary"
}
}
```
## Error Handling
The API validates your request structure immediately. Common errors include:
```json JSON theme={"system"}
{
"error": {
"message": "Invalid request body.",
"type": "BadRequestError",
"fields": {
"_errors": ["Unrecognized key(s) in object: 'webhook_url'"]
}
}
}
```
Ensure you use the correct field names:
* `webhook_id` (correct)
* `webhook_url` (incorrect — this is for the Batch API)
## When to Use Group API
Choose the Group API when you need:
* Quick implementation without file management
* To process 50 or fewer related requests
* Webhook notifications for a set of requests
* Simple JSON-based integration
For larger workloads (50+ requests), consider using the [Batch API](/api/async-inference/batch-api) instead.
# Overview
Source: https://docs.inference.net/api/async-inference/overview
Make cost-effective inference requests with flexible completion times.
Learn how to use our OpenAI-compatible Asynchronous Inference API to send individual inference requests that complete within 24-72 hours at reduced costs. Simply use `/v1/slow` instead of `/v1/` in your API calls to access this feature.
Background inference is cheaper, and easier to build with when your application isn't serving real-time inference.
Asynchronous Inference API is compatible with all the [models](https://inference.net/models) we offer.
Webhook support is only available for `/chat/completions` calls. Support for `/completions` will come later.
## Overview
The Asynchronous Inference API provides a simple way to make cost-effective inference requests when immediate responses aren't required. By using the `/v1/slow` prefix instead of `/v1/`, you can:
1. **Get immediate request IDs:** Your API call returns instantly with a unique ID.
2. **Save on costs:** Enjoy significantly cheaper pricing compared to synchronous requests.
3. **Flexible completion:** Requests complete within 24-72 hours.
4. **Same familiar API:** Uses the exact same request format as our standard endpoints.
This API is perfect for use cases like:
* Large-scale content generation
* Batch document processing
* Non-urgent data analysis
* Cost-sensitive workloads
* Background processing tasks
## Getting Started
Using the Asynchronous Inference API is as simple as changing your base URL from `/v1/` to `/v1/slow/`. The API maintains full compatibility with the OpenAI SDK.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1/slow", // Note the /v1/slow prefix
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1/slow", # Note the /v1/slow prefix
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL theme={"system"}
# Use /v1/slow/ instead of /v1/ in the URL
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ ... }'
```
## Making Asynchronous Requests
### 1. Submit a Request
Make requests exactly as you would with the standard API, but responses will include a request ID instead of the completion result:
```typescript TypeScript theme={"system"}
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of France?" },
],
max_tokens: 1000,
});
console.log(response.id); // Returns immediately with request ID
```
```python Python theme={"system"}
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
max_tokens=1000,
)
print(response.id) # Returns immediately with request ID
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 1000
}'
```
The initial response will include a unique request ID:
```json JSON theme={"system"}
{
"id": "N2mZQjrvh-k_m8nMMN7Jn",
"choices": [],
"created": 1749061362809,
"model": "google/gemma-3-27b-instruct/bf-16",
"object": "chat.completion"
}
```
### 2. Retrieve Results
Once your request is processed, retrieve the results using the generation endpoint:
```typescript TypeScript theme={"system"}
const response = await fetch(
`https://api.inference.net/v1/generation/${generationId}`,
{
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
},
},
);
const result = await response.json();
```
```python Python theme={"system"}
import os
import requests
response = requests.get(
f"https://api.inference.net/v1/generation/{generation_id}",
headers={"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}"},
)
result = response.json()
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/generation/N2mZQjrvh-k_m8nMMN7Jn \
-H "Authorization: Bearer $INFERENCE_API_KEY"
```
The completed response includes both the original request and the generation result:
```json JSON theme={"system"}
{
"request": {
"messages": [
{"content": "You are a helpful assistant.", "role": "system"},
{"content": "What is the meaning of life?", "role": "user"}
],
"model": "google/gemma-3-27b-instruct/bf-16",
"stream": false,
"max_tokens": 8,
"metadata": {"webhook_id": "mPufxRcrw"}
},
"response": {
"id": "N2mZQjrvh-k_m8nMMN7Jn",
"object": "chat.completion",
"created": 1749061362,
"model": "google/gemma-3-27b-instruct/bf-16",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The meaning of life is a complex and",
"reasoning_content": null,
"tool_calls": null
},
"logprobs": null,
"finish_reason": "length",
"matched_stop": null
}
],
"usage": {
"prompt_tokens": 48,
"total_tokens": 56,
"completion_tokens": 8,
"prompt_tokens_details": null
},
"system_fingerprint": ""
},
"state": "Success",
"stateMessage": "Generation successful",
"finishedAt": "2025-06-04T18:22:42.912Z"
}
```
## Request States
Asynchronous requests can have the following states:
| Status | Description |
| ----------- | ------------------------------------------------- |
| Queued | Request received and queued for processing |
| In Progress | Request is currently being processed |
| Success | Request completed successfully, results available |
| Failed | Request failed due to an error |
## Best Practices
1. **Store Request IDs:** Always save the returned request ID for later retrieval.
2. **Use Webhooks:** Instead of polling, set up webhooks for real-time notifications when requests complete. See our [Getting Started with Webhooks](/api/async-inference/webhooks/getting-started-with-webhooks) guide.
3. **Handle Failures:** Have a fallback plan for requests that fail during processing.
4. **Batch When Possible:** For multiple requests, consider using our [Batch API](/api/async-inference/batch-api) for better organization.
## Supported Endpoints
The Asynchronous Inference API supports the following endpoints:
* `/v1/slow/chat/completions`
* `/v1/slow/completions`
Simply replace `/v1/` with `/v1/slow/` in your existing code to use asynchronous processing.
## Pricing and Limits
* **Pricing:** Significantly reduced compared to synchronous requests (contact sales for specific rates)
* **Completion Time:** 24-72 hours
* **Rate Limits:** More generous than synchronous endpoints
* **Request Expiration:** Requests expire after 72 hours if not completed
For specific pricing information and higher rate limits, please contact [support@inference.net](mailto:support@inference.net).
# Getting Started With Webhooks
Source: https://docs.inference.net/api/async-inference/webhooks/getting-started-with-webhooks
Everything you need to know to get started with webhooks.
Webhook support is currently available for `/chat/completions` and `/embeddings` calls. Support for `/completions` will come later.
## Overview
Webhooks provide an efficient push-based notification system for tracking generation completions in real-time. Rather than repeatedly polling the API to check generation status, webhooks automatically notify your application when generations complete, enabling streamlined workflows and better resource utilization.
## Key Benefits
* **Resource Efficiency**: Eliminate unnecessary API calls for status checks
* **Real-time Updates**: Receive notifications within milliseconds of generation completion
* **Scalability**: Handle thousands of concurrent generations efficiently
* **Improved User Experience**: Update your UI instantly when results are ready
## Getting Started
### Step 1: Create a Webhook Endpoint
Your application needs an HTTPS endpoint capable of receiving POST requests. The endpoint should:
1. Accept JSON payloads
2. Respond with HTTP 200 status immediately
3. Process the webhook data asynchronously
```typescript TypeScript theme={"system"}
import express from "express";
const app = express();
app.post("/webhooks/inference", express.json(), (req, res) => {
const { event, webhook_id, generation_id, data } = req.body;
// Verify webhook source via headers
const webhookId = req.headers["x-inference-webhook-id"];
if (event === "generation.completed") {
console.log(`Generation ${generation_id} completed with status: ${data.state}`);
// Process asynchronously
setImmediate(() => {
processGenerationResult(data);
});
} else if (event === "async-embedding.completed") {
console.log(`Embedding ${generation_id} completed with status: ${data.state}`);
setImmediate(() => {
processEmbeddingResult(data);
});
}
// Always respond immediately
res.status(200).json({ received: true });
});
app.listen(3000, () => {
console.log("Webhook receiver listening on port 3000");
});
```
```python Python theme={"system"}
from fastapi import FastAPI, Request, BackgroundTasks
from pydantic import BaseModel
from typing import Optional, Dict, Any
app = FastAPI()
class WebhookPayload(BaseModel):
event: str
timestamp: str
webhook_id: str
generation_id: Optional[str] = None
data: Dict[str, Any]
def process_generation(payload: WebhookPayload):
"""Process generation result asynchronously"""
if payload.event == "generation.completed":
print(f"Processing generation {payload.generation_id}")
# Your processing logic here
elif payload.event == "async-embedding.completed":
print(f"Processing embedding {payload.generation_id}")
# Your embedding processing logic here
@app.post("/webhooks/inference")
async def handle_webhook(
payload: WebhookPayload,
request: Request,
background_tasks: BackgroundTasks,
):
# Verify webhook source
webhook_id = request.headers.get("x-inference-webhook-id")
# Queue for background processing
background_tasks.add_task(process_generation, payload)
return {"received": True}
```
Go Example
```go theme={"system"}
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
)
type WebhookPayload struct {
Event string `json:"event"`
Timestamp string `json:"timestamp"`
WebhookID string `json:"webhook_id"`
GenerationID string `json:"generation_id,omitempty"`
Data map[string]interface{} `json:"data"`
}
func handleWebhook(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "Failed to read body", http.StatusBadRequest)
return
}
var payload WebhookPayload
if err := json.Unmarshal(body, &payload); err != nil {
http.Error(w, "Invalid JSON", http.StatusBadRequest)
return
}
// Verify webhook source
webhookID := r.Header.Get("X-Inference-Webhook-ID")
// Process asynchronously
go func() {
if payload.Event == "generation.completed" {
fmt.Printf("Processing generation %s\n", payload.GenerationID)
// Your processing logic here
} else if payload.Event == "async-embedding.completed" {
fmt.Printf("Processing embedding %s\n", payload.GenerationID)
// Your embedding processing logic here
}
}()
// Respond immediately
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(map[string]bool{"received": true})
}
func main() {
http.HandleFunc("/webhooks/inference", handleWebhook)
fmt.Println("Webhook server listening on :3000")
http.ListenAndServe(":3000", nil)
}
```
### Step 2: Deploy Your Endpoint
Your webhook endpoint must be publicly accessible via HTTPS. For development environments, consider using:
* **ngrok**: `ngrok http 3000`
* **Cloudflare Tunnel**: Provides a stable URL
* **localtunnel**: `lt --port 3000`
### Step 3: Register Your Webhook
1. Navigate to the inference.net dashboard
2. Go to **API Keys** → **Webhooks** in the sidebar
3. Click **Create Webhook**
4. Enter a descriptive name and your HTTPS endpoint URL
5. Save your webhook
You'll receive a webhook identifier (e.g., `AhALzdz8S`) that you'll use when creating generations.
### Step 4: Link Webhook to Generations
Include the webhook identifier in the metadata when creating a generation:
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1/slow",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [{ role: "user", content: "Explain quantum computing" }],
// @ts-expect-error metadata is not in the OpenAI SDK types
metadata: {
webhook_id: "AhALzdz8S",
},
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1/slow",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[{"role": "user", "content": "Explain quantum computing"}],
extra_body={
"metadata": {"webhook_id": "AhALzdz8S"},
},
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "user", "content": "Explain quantum computing"}
],
"metadata": {
"webhook_id": "AhALzdz8S"
}
}'
```
When the generation completes, your webhook endpoint will receive a notification.
## Webhook Events
### generation.completed
Sent when a generation finishes processing (successfully or with failure):
```json JSON theme={"system"}
{
"event": "generation.completed",
"timestamp": "2025-01-03T06:46:22.838Z",
"webhook_id": "AhALzdz8S",
"generation_id": "XBKcs7F1s2oJ_AHiLMbF4",
"data": {
"state": "Success",
"stateMessage": "Generation successful",
"request": {
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing"}
],
"model": "google/gemma-3-27b-instruct/bf-16",
"stream": false,
"max_tokens": 100,
"metadata": {
"webhook_id": "AhALzdz8S"
}
},
"response": {
"id": "XBKcs7F1s2oJ_AHiLMbF4",
"object": "chat.completion",
"created": 1748933182,
"model": "google/gemma-3-27b-instruct/bf-16",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "Quantum computing is a revolutionary approach..."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 28,
"completion_tokens": 42,
"total_tokens": 70
}
},
"finishedAt": "2025-01-03T06:46:22.307Z"
}
}
```
The `response` object is compatible with the types exported from the official OpenAI SDKs.
```typescript TypeScript theme={"system"}
import type { OpenAI } from "openai";
const response = responseJsonObject as OpenAI.Chat.Completions.ChatCompletion;
```
```python Python theme={"system"}
from openai.types.chat.chat_completion import ChatCompletion
response: ChatCompletion = webhook_payload.data["response"]
```
### async-embedding.completed
Sent when an async embedding request finishes processing:
```json JSON theme={"system"}
{
"event": "async-embedding.completed",
"timestamp": "2025-01-15T10:30:00Z",
"webhook_id": "AhALzdz8S",
"generation_id": "EMB_abc123",
"data": {
"state": "Success",
"stateMessage": "Embeddings generated successfully",
"request": {
"model": "qwen/qwen3-embedding-4b",
"input": ["text1", "text2"],
"metadata": { "webhook_id": "AhALzdz8S" }
},
"response": {
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.0023064255, -0.009327292]
}
],
"model": "qwen/qwen3-embedding-4b",
"usage": {
"prompt_tokens": 100,
"total_tokens": 100
}
},
"finishedAt": "2025-01-15T10:30:00Z"
}
}
```
The `response` object follows the standard OpenAI embeddings format.
## Headers
All webhook requests include the following headers:
| Header | Description | Example |
| --------------------------- | ------------------------------------- | --------------------------- |
| `X-Inference-Event` | Event type | `generation.completed` |
| `X-Inference-Webhook-ID` | Webhook identifier | `AhALzdz8S` |
| `X-Inference-Generation-ID` | Generation ID (for completion events) | `XBKcs7F1s2oJ_AHiLMbF4` |
| `User-Agent` | inference.net webhook agent | `inference.net-Webhook/1.0` |
| `Content-Type` | Always `application/json` | `application/json` |
### (Security) Verifying the request source
The `X-Inference-Webhook-ID` is a good way to verify that the payload you're receiving is officially coming from our API.
This ID is unique to your webhook, and is completely private to you and your team.
If the ID does not match what you see in the dashboard, your endpoint has most likely been discovered by a malicious actor.
## Testing Webhooks
You can test your webhook endpoint from the dashboard:
1. Navigate to **Webhooks** in the dashboard
2. Find your webhook in the list
3. Click the menu and select **Test**
4. Check your endpoint logs for the test payload
A successful test will show a green success indicator in the dashboard.
## Best Practices
### 1. Respond Immediately
Your endpoint must respond within 30 seconds. Always return a 200 status immediately and process the webhook asynchronously:
```typescript TypeScript theme={"system"}
// Correct approach
app.post("/webhook", (req, res) => {
res.status(200).send("OK");
processWebhookAsync(req.body);
});
// Incorrect approach — may timeout
app.post("/webhook", async (req, res) => {
await heavyProcessing(req.body); // Risk of timeout
res.status(200).send("OK");
});
```
```python Python theme={"system"}
# Correct approach — process in background
@app.post("/webhook")
async def handle_webhook(
payload: WebhookPayload,
background_tasks: BackgroundTasks,
):
background_tasks.add_task(process_webhook, payload)
return {"received": True}
# Incorrect approach — may timeout
@app.post("/webhook")
async def handle_webhook(payload: WebhookPayload):
await heavy_processing(payload) # Risk of timeout
return {"received": True}
```
### 2. Implement Idempotency
Failed webhooks may be retried. Use the `generation_id` to ensure you don't process the same event twice:
```typescript TypeScript theme={"system"}
const processedGenerations = new Set();
async function processWebhook(payload: any) {
if (processedGenerations.has(payload.generation_id)) {
return; // Already processed
}
processedGenerations.add(payload.generation_id);
// Process the generation
}
```
```python Python theme={"system"}
processed_generations: set[str] = set()
def process_webhook(payload: WebhookPayload):
if payload.generation_id in processed_generations:
return # Already processed
processed_generations.add(payload.generation_id)
# Process the generation
```
### 3. Validate Webhook Source
Always verify that webhooks originate from inference.net by checking the presence of expected headers:
```typescript TypeScript theme={"system"}
function validateWebhookSource(headers: Record): boolean {
const requiredHeaders = ["x-inference-webhook-id", "x-inference-event"];
return requiredHeaders.every((header) => headers[header]);
}
```
```python Python theme={"system"}
def validate_webhook_source(headers: dict) -> bool:
required_headers = ["x-inference-webhook-id", "x-inference-event"]
return all(headers.get(h) for h in required_headers)
```
### 4. Handle Errors Gracefully
Implement proper error handling to prevent individual failures from affecting your entire system:
```typescript TypeScript theme={"system"}
async function handleWebhook(payload: any) {
try {
await processWebhook(payload);
} catch (error) {
console.error("Webhook processing failed:", error);
// Log to monitoring service
// Return 200 to prevent unnecessary retries
}
}
```
```python Python theme={"system"}
async def handle_webhook(payload: WebhookPayload):
try:
await process_webhook(payload)
except Exception as error:
print(f"Webhook processing failed: {error}")
# Log to monitoring service
# Return 200 to prevent unnecessary retries
```
### 5. Monitor Webhook Processing
Track key metrics to ensure reliable webhook processing:
* Webhook receipt rate
* Processing success/failure rates
* Average processing time
* Queue depth (if using queues)
## Troubleshooting
### Not Receiving Webhooks
1. **Check webhook status**: Ensure your webhook is not disabled in the dashboard
2. **Test connectivity**: Use the test feature in the dashboard
3. **Verify URL**: Confirm your endpoint is publicly accessible via HTTPS
4. **Check logs**: Review both your server logs and any reverse proxy logs
5. **Validate metadata**: Ensure you're including the correct `webhook_id` in generation requests
### Webhooks Arriving Late
* Verify your endpoint responds quickly (\< 1 second ideally)
* Check that you're not performing heavy processing before responding
* Monitor your server load and resource usage
### Duplicate Webhook Deliveries
* Implement idempotency using the `generation_id`
* Ensure your endpoint always returns 200 OK for successful receipt
* Check for any errors in your webhook processing that might trigger retries
## Frequently Asked Questions
**Q: What happens if my endpoint is down?**
A: Failed webhook deliveries are retried up to 3 times with exponential backoff. After all retries are exhausted, the delivery is marked as failed.
**Q: What's the webhook timeout?**
A: Webhook endpoints must respond within 30 seconds. Timeouts are treated as failures and will trigger retries.
**Q: Can I filter which events I receive?**
A: Currently, webhooks receive all event types. Event filtering is planned for a future update.
**Q: How secure are webhooks?**
A: All webhooks are sent over HTTPS. You should validate the webhook source using the provided headers. HMAC signature verification is planned for additional security.
**Q: What's the maximum payload size?**
A: Webhook payloads can be up to 10MB, though typical payloads range from 5-50KB.
**Q: Can I replay missed webhooks?**
A: Webhook replay functionality is not currently available. As a fallback, you can poll the generation status endpoint.
## Support
For assistance with webhooks:
* Email: [support@inference.net](mailto:support@inference.net)
* Discord: Join our developer community
* Documentation: [https://docs.inference.net](https://docs.inference.net)
* Issues: Report bugs via our support portal
# Webhooks: Quick Reference
Source: https://docs.inference.net/api/async-inference/webhooks/quick-reference
Quick reference of webhook support for asynchronous inference
## Dashboard Management
Webhooks are managed through the inference.net dashboard:
1. Navigate to **Webhooks** on the sidebar
2. Create, test, archive, or restore webhooks through the UI
3. Copy your webhook identifier for use in generation requests
## Payload Structures
### generation.completed
```json JSON theme={"system"}
{
"event": "generation.completed",
"timestamp": "ISO 8601 timestamp",
"webhook_id": "webhook identifier",
"generation_id": "generation ID",
"data": {
"state": "Success|Failed",
"stateMessage": "Human readable status",
"request": { /* Original request with metadata */ },
"response": { /* OpenAI format response */ },
"finishedAt": "ISO 8601 timestamp"
}
}
```
### async-embedding.completed
```json JSON theme={"system"}
{
"event": "async-embedding.completed",
"timestamp": "ISO 8601 timestamp",
"webhook_id": "webhook identifier",
"generation_id": "generation ID",
"data": {
"state": "Success|Failed",
"stateMessage": "Human readable status",
"request": { /* Original embeddings request with metadata */ },
"response": { /* OpenAI format embeddings response */ },
"finishedAt": "ISO 8601 timestamp"
}
}
```
### slow-group.completed
```json JSON theme={"system"}
{
"event": "slow-group.completed",
"timestamp": "ISO 8601 timestamp",
"group_id": "group ID",
"data": {
"group_size": 2,
"status": "completed",
"generations": [
{
"generationId": "generation ID",
"state": "Success|Failed",
"stateMessage": "Human readable status",
"request": { /* Original request */ },
"response": { /* OpenAI format response */ },
"finishedAt": "ISO 8601 timestamp or null"
}
]
}
}
```
## Headers
| Header | Description | Example |
| --------------------------- | ----------------------------- | ------------------------------------------------------------------------------ |
| `X-Inference-Event` | Event type | `generation.completed`, `async-embedding.completed`, or `slow-group.completed` |
| `X-Inference-Webhook-ID` | Webhook identifier | `AhALzdz8S` |
| `X-Inference-Generation-ID` | Generation ID (if applicable) | `XBKcs7F1s2oJ_AHiLMbF4` |
| `X-Inference-Group-ID` | Group ID (for group events) | `GRP_XYZ123` |
| `User-Agent` | inference.net webhook agent | `inference.net-Webhook/1.0` |
| `Content-Type` | Always `application/json` | `application/json` |
## Using Webhooks in Generations
Include the webhook identifier in your generation request metadata:
### Chat Completions
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1/slow",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [{ role: "user", content: "Hello!" }],
// @ts-expect-error metadata is not in the OpenAI SDK types
metadata: { webhook_id: "YOUR_WEBHOOK_IDENTIFIER" },
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1/slow",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[{"role": "user", "content": "Hello!"}],
extra_body={"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"}},
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/slow/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [{"role": "user", "content": "Hello!"}],
"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"}
}'
```
### Embeddings
```typescript TypeScript theme={"system"}
const embeddingResponse = await fetch(
"https://api.inference.net/v1/async/embeddings",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "qwen/qwen3-embedding-4b",
input: ["Text to embed", "Another text to embed"],
metadata: { webhook_id: "YOUR_WEBHOOK_IDENTIFIER" },
}),
},
);
```
```python Python theme={"system"}
import os
import requests
response = requests.post(
"https://api.inference.net/v1/async/embeddings",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
},
json={
"model": "qwen/qwen3-embedding-4b",
"input": ["Text to embed", "Another text to embed"],
"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"},
},
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/async/embeddings \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3-embedding-4b",
"input": ["Text to embed", "Another text to embed"],
"metadata": {"webhook_id": "YOUR_WEBHOOK_IDENTIFIER"}
}'
```
## Minimal Webhook Handler Examples
```typescript TypeScript theme={"system"}
app.post("/webhook", express.json(), (req, res) => {
res.status(200).json({ received: true });
if (req.body.event === "generation.completed") {
setImmediate(() => {
console.log("Generation completed:", req.body.generation_id);
// Your processing logic here
});
} else if (req.body.event === "async-embedding.completed") {
setImmediate(() => {
console.log("Embedding completed:", req.body.generation_id);
console.log("Number of embeddings:", req.body.data.response.data.length);
});
} else if (req.body.event === "slow-group.completed") {
setImmediate(() => {
console.log("Group completed:", req.body.group_id);
console.log("Group size:", req.body.data.group_size);
req.body.data.generations.forEach((gen: any) => {
console.log(`Generation ${gen.generationId}: ${gen.state}`);
});
});
}
});
```
```python Python theme={"system"}
@app.post("/webhook")
async def handle_webhook(payload: dict, background_tasks: BackgroundTasks):
background_tasks.add_task(process_webhook, payload)
return {"received": True}
def process_webhook(payload):
if payload["event"] == "generation.completed":
print(f"Processing generation {payload['generation_id']}")
# Your processing logic here
elif payload["event"] == "async-embedding.completed":
print(f"Processing embedding {payload['generation_id']}")
print(f"Number of embeddings: {len(payload['data']['response']['data'])}")
elif payload["event"] == "slow-group.completed":
print(f"Processing group {payload['group_id']}")
print(f"Group size: {payload['data']['group_size']}")
for gen in payload["data"]["generations"]:
print(f"Generation {gen['generationId']}: {gen['state']}")
```
## Timing & Limits
| Metric | Value | Notes |
| ---------------- | ---------------- | ----------------------------- |
| Response timeout | 30 seconds | Must respond within this time |
| Retry attempts | 3 | With exponential backoff |
| Max payload size | 10MB | Typical: 5-50KB |
| Delivery time | Under 60 seconds | From completion to webhook |
## Response Codes
| Code | Meaning | Retry? |
| ------- | ------------------ | ------ |
| 200-299 | Success | No |
| 400-499 | Client error | No |
| 500-599 | Server error | Yes |
| Timeout | No response in 30s | Yes |
## Best Practices Checklist
* [ ] Respond with 200 OK immediately
* [ ] Process webhook data asynchronously
* [ ] Implement idempotency with generation\_id or group\_id
* [ ] Validate webhook source via headers
* [ ] Handle errors gracefully
* [ ] Monitor webhook processing
* [ ] Use HTTPS endpoint
* [ ] Set up proper error logging
* [ ] Test webhook with dashboard test feature
* [ ] Implement timeout handling
* [ ] Handle both individual and group completions
## Common Issues & Solutions
| Issue | Solution |
| ---------------------- | ---------------------------------------------------------------------------- |
| Not receiving webhooks | Check webhook not disabled in dashboard, test connectivity, verify HTTPS URL |
| Duplicate webhooks | Implement idempotency, ensure 200 OK response |
| Webhooks timing out | Respond immediately, process asynchronously |
| Invalid payload | Validate against documented schema |
| Test webhook fails | Check endpoint is publicly accessible, returns 200 OK |
## Support Resources
* [Full Documentation](https://docs.inference.net)
* [API Reference](https://docs.inference.net/api)
* [Support](mailto:support@inference.net)
* Discord Community
# Data Retention
Source: https://docs.inference.net/api/data-retention
Understand how Inference.net handles request data, observability records, and retention controls.
Inference.net is designed to support production workloads without treating captured request data casually.
## Core principles
* Request data is not used for model training by default
* Secrets and similar sensitive values are stripped where possible
* Platform data is encrypted in transit and at rest
* Retention should match the operational need of the workflow
## Direct API vs Gateway
The direct API and Gateway are different product paths, but the same general rule applies: only keep what is operationally useful, and use project-level controls and data curation intentionally.
For the workflow-first entry point into traffic capture, start with [Integrate with Your LLM Provider](/platform/gateway/integrate).
## Recommended operational pattern
* Use environments and task IDs to segment traffic
* Create long-lived datasets only for the examples you want to preserve
* Review retention expectations before broad production rollout
## Need a specific retention policy?
If you need a specific policy, no-retention handling, or help mapping the platform into your internal compliance requirements, [meet with our team](https://inference.net/meet-with-us/).
# Function Calling
Source: https://docs.inference.net/api/function-calling
Enable models to fetch data and take actions.
Support for Function Calling is in beta and will be available soon.
## Introduction
**Function calling** provides a powerful and flexible way for models to interface with your code or external services, and has two primary use cases:
| | |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fetching Data | Retrieve up-to-date information to incorporate into the model's response (RAG). Useful for searching knowledge bases and retrieving specific data from APIs (e.g. current weather data). |
| Taking Action | Perform actions like submitting a form, calling APIs, modifying application state (UI/frontend or backend), or taking agentic workflow actions (like handing off the conversation). |
If you only want the model to produce JSON, see our docs on [structured outputs](/api/structured-outputs).
## Getting Started
You'll need an Inference.net account and API key to use Function Calling. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net using the OpenAI SDK, you will need to set the base URL to `https://api.inference.net/v1`.
In this example, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL theme={"system"}
export INFERENCE_API_KEY=
# All requests use:
# -H "Authorization: Bearer $INFERENCE_API_KEY"
# -H "Content-Type: application/json"
```
## Overview
You can extend the capabilities of models by giving them access to functions that you define called `tools`. This is also called "function calling".
With function calling, you'll tell the model what tools are available to it, and it will decide which one to use.
You'll then execute the function code, send back the results, and the model will incorporate them into its final response.
![Function Calling Diagram Steps]()
### Sample function
Let's look at the steps to allow a model to use a real `get_weather` function defined below:
```typescript TypeScript theme={"system"}
async function getWeather(latitude: number, longitude: number) {
const response = await fetch(
`https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}¤t=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m`
);
const data = await response.json();
return data.current.temperature_2m;
}
```
```python Python theme={"system"}
import requests
def get_weather(latitude, longitude):
response = requests.get(
f"https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}"
f"¤t=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m"
)
data = response.json()
return data["current"]["temperature_2m"]
```
All functions must return strings. You can format the string as JSON or another format if you like, but the return type itself must be a string.
Unlike the diagram earlier, this function expects precise `latitude` and `longitude` instead of a general `location` parameter.
### Step By Step Example
#### Step 1: Call model with get\_weather tool defined
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const tools: OpenAI.ChatCompletionTool[] = [{
type: "function",
function: {
name: "get_weather",
description: "Get current temperature for provided coordinates in celsius.",
parameters: {
type: "object",
properties: {
lat: { type: "number" },
lon: { type: "number" },
},
required: ["lat", "lon"],
additionalProperties: false,
},
strict: true,
},
}];
const messages: OpenAI.ChatCompletionMessageParam[] = [
{
role: "system",
content: "You are a helpful assistant that can answer questions and uses tools to get information.",
},
{
role: "user",
content: "What's the weather like in Paris today?",
},
];
const completion = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages,
tools,
});
```
```python Python theme={"system"}
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for provided coordinates in celsius.",
"parameters": {
"type": "object",
"properties": {
"lat": {"type": "number"},
"lon": {"type": "number"},
},
"required": ["lat", "lon"],
"additionalProperties": False,
},
"strict": True,
},
}]
messages = [
{"role": "system", "content": "You are a helpful assistant that can answer questions and uses tools to get information."},
{"role": "user", "content": "What's the weather like in Paris today?"},
]
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=messages,
tools=tools,
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant that can answer questions and uses tools to get information."},
{"role": "user", "content": "What'\''s the weather like in Paris today?"}
],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for provided coordinates in celsius.",
"parameters": {
"type": "object",
"properties": {
"lat": { "type": "number" },
"lon": { "type": "number" }
},
"required": ["lat", "lon"],
"additionalProperties": false
},
"strict": true
}
}]
}'
```
Less powerful models may not reliably respond with tool calls, and may not provide all requested parameters.
Experiment with system prompts and other models to find the best results.
#### Step 2: Pull the selected function call from the model's response
```json JSON theme={"system"}
[
{
"id": "call_12345xyz",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{\"lat\":48.8566,\"lon\":2.3522}"
}
}
]
```
#### Step 3: Execute the `get_weather` function
```typescript TypeScript theme={"system"}
const toolCall = completion.choices[0].message.tool_calls[0];
const args = JSON.parse(toolCall.function.arguments);
const result = await getWeather(args.lat, args.lon);
```
```python Python theme={"system"}
tool_call = completion.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)
result = get_weather(args["lat"], args["lon"])
```
#### Step 4: Supply result and call model again
```typescript TypeScript theme={"system"}
messages.push(completion.choices[0].message); // append model's function call message
messages.push({ // append result message
role: "tool",
tool_call_id: toolCall.id,
content: result.toString(),
});
const completion2 = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages,
tools,
});
console.log(completion2.choices[0].message.content);
```
```python Python theme={"system"}
messages.append(completion.choices[0].message) # append model's function call message
messages.append({ # append result message
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result),
})
completion_2 = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=messages,
tools=tools,
)
print(completion_2.choices[0].message.content)
```
#### Output
```json JSON theme={"system"}
"The current temperature in Paris is 14°C (57.2°F)."
```
## Defining functions
Functions can be set in the `tools` parameter of each API request.
A function is defined by its schema, which informs the model what it does and what input arguments it expects. It comprises the following fields:
| Field | Description |
| ----------- | --------------------------------------------------- |
| name | The function's name (e.g. get\_weather) |
| description | Details on when and how to use the function |
| parameters | JSON schema defining the function's input arguments |
Take a look at this example:
```json JSON theme={"system"}
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Retrieves current weather for the given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia"
},
"units": {
"type": "string",
"enum": [
"celsius",
"fahrenheit"
],
"description": "Units the temperature will be returned in."
}
},
"required": [
"location",
"units"
],
"additionalProperties": false
},
"strict": true
}
}
```
Because the `parameters` are defined by a [JSON schema](https://json-schema.org/), you can leverage many of its rich features like property types, enums, and descriptions.
### SDK Helpers
While you can define function schemas directly, [OpenAI's SDKs](https://platform.openai.com/docs/libraries) have helpers to convert `pydantic` and `zod` objects into schemas.
Not all `pydantic` and `zod` features are currently supported by Function Calling, but simple, flat schemas are supported.
Here is an example of how to use the SDK to define a schema.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
import { z } from "zod";
import { zodFunction } from "openai/helpers/zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const GetWeatherParameters = z.object({
location: z.string().describe("City and country e.g. Bogotá, Colombia"),
});
const tools = [
zodFunction({ name: "getWeather", parameters: GetWeatherParameters }),
];
const messages: OpenAI.ChatCompletionMessageParam[] = [
{ role: "user", content: "What's the weather like in Paris today?" },
];
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages,
tools,
});
console.log(response.choices[0].message.tool_calls);
```
```python Python theme={"system"}
import os
from openai import OpenAI, pydantic_function_tool
from pydantic import BaseModel, Field
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
class GetWeather(BaseModel):
location: str = Field(
...,
description="City and country e.g. Bogotá, Colombia",
)
tools = [pydantic_function_tool(GetWeather)]
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful assistant that can answer questions and help with tasks."},
{"role": "user", "content": "What's the weather like in Paris today?"},
],
tools=tools,
)
print(completion.choices[0].message.tool_calls)
```
### Best practices for defining functions
1. **Write clear and detailed function names, parameter descriptions, and instructions.**
* **Explicitly describe the purpose of the function and each parameter** (and its format), and what the output represents.
* **Use the system prompt to describe when (and when not) to use each function.** Generally, tell the model *exactly* what to do.
* **Include examples and edge cases**, especially to rectify any recurring failures.
2. **Apply software engineering best practices.**
* **Make the functions obvious and intuitive**. ([principle of least surprise](https://en.wikipedia.org/wiki/Principle_of_least_astonishment))
* **Use enums** and object structure to make invalid states unrepresentable. (e.g. `toggle_light(on: bool, off: bool)` allows for invalid calls)
* **Pass the intern test.** Can an intern/human correctly use the function given nothing but what you gave the model? (If not, what questions do they ask you? Add the answers to the prompt.)
3. **Offload the burden from the model and use code where possible.**
* **Don't make the model fill arguments you already know.** For example, if you already have an `order_id` based on a previous menu, don't have an `order_id` param – instead, have no params `submit_refund()` and pass the `order_id` with code.
* **Combine functions that are always called in sequence.** For example, if you always call `mark_location()` after `query_location()`, just move the marking logic into the query function call.
4. **Keep the number of functions small for higher accuracy.**
* **Evaluate your performance** with different numbers of functions.
* **Aim for fewer than 20 functions** at any one time, though this is just a soft suggestion.
## Streaming
Streaming can be used to surface progress by showing which function is called as the model fills its arguments, and even displaying the arguments in real time.
Streaming function calls is very similar to streaming regular responses: you set `stream` to `true` and get chunks with `delta` objects.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const tools: OpenAI.ChatCompletionTool[] = [{
type: "function",
function: {
name: "get_weather",
description: "Get current temperature for a given location.",
parameters: {
type: "object",
properties: {
location: {
type: "string",
description: "City and country e.g. Bogotá, Colombia",
},
},
required: ["location"],
additionalProperties: false,
},
strict: true,
},
}];
const stream = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [{ role: "user", content: "What's the weather like in Paris today?" }],
tools,
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0].delta;
console.log(delta.tool_calls);
}
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia",
},
},
"required": ["location"],
"additionalProperties": False,
},
"strict": True,
},
}]
stream = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[{"role": "user", "content": "What's the weather like in Paris today?"}],
tools=tools,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
print(delta.tool_calls)
```
```bash cURL theme={"system"}
curl -N https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"stream": true,
"messages": [
{"role": "user", "content": "What'\''s the weather like in Paris today?"}
],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia"
}
},
"required": ["location"],
"additionalProperties": false
},
"strict": true
}
}]
}'
```
Output of `delta.tool_calls`:
```txt TXT theme={"system"}
[{"index": 0, "id": "call_DdmO9pD3xa9XTPNJ32zg2hcA", "function": {"arguments": "", "name": "get_weather"}, "type": "function"}]
[{"index": 0, "id": null, "function": {"arguments": "{\"", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "location", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "\":\"", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "Paris", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": ",", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": " France", "name": null}, "type": null}]
[{"index": 0, "id": null, "function": {"arguments": "\"}", "name": null}, "type": null}]
null
```
Instead of aggregating chunks into a single `content` string, however, you're aggregating chunks into an encoded `arguments` JSON object.
When the model calls one or more functions the `tool_calls` field of each `delta` will be populated. Each `tool_call` contains the following fields:
| Field | Description |
| -------- | ------------------------------------------------------- |
| index | Identifies which function call the delta is for |
| id | Tool call id. |
| function | Function call delta (name and arguments) |
| type | Type of tool\_call (always function for function calls) |
Many of these fields are only set for the first `delta` of each tool call, like `id`, `function.name`, and `type`.
Below is a code snippet demonstrating how to aggregate the `delta` objects into a final `tool_calls` object.
```typescript TypeScript theme={"system"}
const finalToolCalls: Record = {};
for await (const chunk of stream) {
const toolCalls = chunk.choices[0].delta.tool_calls || [];
for (const toolCall of toolCalls) {
const { index } = toolCall;
if (!finalToolCalls[index]) {
finalToolCalls[index] = toolCall;
}
finalToolCalls[index].function.arguments += toolCall.function.arguments;
}
}
```
```python Python theme={"system"}
final_tool_calls = {}
for chunk in stream:
for tool_call in chunk.choices[0].delta.tool_calls or []:
index = tool_call.index
if index not in final_tool_calls:
final_tool_calls[index] = tool_call
final_tool_calls[index].function.arguments += tool_call.function.arguments
```
Accumulated final\_tool\_calls\[0]
```json JSON theme={"system"}
{
"index": 0,
"id": "call_RzfkBpJgzeR0S242qfvjadNe",
"function": {
"name": "get_weather",
"arguments": "{\"location\":\"Paris, France\"}"
}
}
```
# Rate Limits
Source: https://docs.inference.net/api/rate-limits
Rate limits for the Inference.net API
## Generation Rate Limits
Rate limits for model inference requests are based on your account tier:
| Tier | Requests per minute (RPM) |
| ------------ | ------------------------------------ |
| Free | 30 |
| Paid | 250 |
| Custom teams | 500 - 1,000,000 (per-team overrides) |
## Deployed Model Rate Limits
Models you deploy on the platform have a rate limit of **300 RPM per instance**. If you need higher throughput, scale up the number of instances — total RPM equals the number of instances multiplied by 300.
## Batch API Rate Limits
* **Batch file upload:** 1 per minute
* Batch processing rate limits are separate from generation rate limits. See the [Batch API](/api/async-inference/batch-api) docs for details.
## General API Rate Limits
The Catalyst gateway enforces a general rate limit of **100 requests per second** per API key or IP address. This applies across all endpoints.
## Increasing Your Limits
If you need higher rate limits, [contact us](mailto:support@inference.net) or use the support chat to request a custom tier.
# Structured Outputs
Source: https://docs.inference.net/api/structured-outputs
Ensure responses adhere to a JSON schema.
When using Structured Outputs, always include instructions in the system prompt to respond in JSON format.
For example: "You are a helpful assistant. Respond in JSON format."
## Introduction
JSON is one of the most widely used formats in the world for applications to exchange data.
Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied [JSON Schema](https://json-schema.org/overview/what-is-jsonschema), so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value.
Some benefits of Structured Outputs include:
1. **Reliable type-safety:** No need to validate or retry incorrectly formatted responses
2. **Simpler prompting:** No need for strongly worded prompts to achieve consistent formatting
## Getting Started
You'll need an Inference.net account and API key to use Structured Outputs. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net using the OpenAI SDK, you will need to set the base URL to `https://api.inference.net/v1`.
In this example, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
```
```bash cURL theme={"system"}
export INFERENCE_API_KEY=
# All requests use:
# -H "Authorization: Bearer $INFERENCE_API_KEY"
# -H "Content-Type: application/json"
```
## When to use Structured Outputs
Structured Outputs are suitable when you want to indicate a structured schema for use when the model responds to the user.
For example, if you are building a math tutoring application, you might want the assistant to respond to your user using a specific JSON Schema so that you can generate a UI that displays different parts of the model's output in distinct ways.
Put simply:
* If you are connecting the model to tools, functions, data, etc. in your system, then you should use function calling
* If you want to structure the model's output when it responds to the user, then you should use a structured `response_format`
### Structured Outputs vs JSON mode
Structured Outputs is the evolution of [JSON mode](#json-mode). While both ensure valid JSON is produced, only Structured Outputs ensure schema adherance. Both Structured Outputs and JSON mode are supported in the Chat Completions API and Batch API.
We recommend always using Structured Outputs instead of JSON mode when possible.
| | Structured Outputs | JSON Mode |
| ------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------ |
| Outputs valid JSON | Yes | Yes |
| Adheres to schema | Yes (see supported schemas) | No |
| Enabling | `response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ... } }` | `response_format: { type: "json_object" }` |
## Example
### Chain of thought
You can ask the model to output an answer in a structured, step-by-step way, to guide the user through the solution.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const Step = z.object({
explanation: z.string(),
output: z.string(),
});
const MathReasoning = z.object({
steps: z.array(Step),
final_answer: z.string(),
});
const completion = await client.chat.completions.parse({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format." },
{ role: "user", content: "how can I solve 8x + 7 = -23" },
],
response_format: zodResponseFormat(MathReasoning, "math_reasoning"),
});
const math_reasoning = completion.choices[0].message.parsed;
console.log(math_reasoning);
```
```python Python theme={"system"}
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."},
{"role": "user", "content": "how can I solve 8x + 7 = -23"},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "math_reasoning",
"strict": True,
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": {"type": "string"},
"output": {"type": "string"},
},
"required": ["explanation", "output"],
"additionalProperties": False,
},
},
"final_answer": {"type": "string"},
},
"required": ["steps", "final_answer"],
"additionalProperties": False,
},
},
},
)
math_reasoning = json.loads(completion.choices[0].message.content)
print(json.dumps(math_reasoning, indent=2))
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{
"role": "system",
"content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."
},
{
"role": "user",
"content": "how can I solve 8x + 7 = -23"
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "math_reasoning",
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": { "type": "string" },
"output": { "type": "string" }
},
"required": ["explanation", "output"],
"additionalProperties": false
}
},
"final_answer": { "type": "string" }
},
"required": ["steps", "final_answer"],
"additionalProperties": false
},
"strict": true
}
}
}'
```
#### Example response
```json JSON theme={"system"}
{
"steps": [
{
"explanation": "Start with the equation 8x + 7 = -23.",
"output": "8x + 7 = -23"
},
{
"explanation": "Subtract 7 from both sides to isolate the term with the variable.",
"output": "8x = -23 - 7"
},
{
"explanation": "Simplify the right side of the equation.",
"output": "8x = -30"
},
{
"explanation": "Divide both sides by 8 to solve for x.",
"output": "x = -30 / 8"
},
{
"explanation": "Simplify the fraction.",
"output": "x = -15 / 4"
}
],
"final_answer": "x = -15 / 4"
}
```
### Defining Schemas with the SDK
The OpenAI SDK makes it easy to define object schemas using [Zod](https://zod.dev/) (TypeScript) or [Pydantic](https://docs.pydantic.dev/) (Python). Below, you can see how to extract information from unstructured text that conforms to a schema defined in code.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const CalendarEvent = z.object({
name: z.string(),
date: z.string(),
participants: z.array(z.string()),
});
const completion = await client.chat.completions.parse({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "Extract the event information. Respond in JSON format." },
{ role: "user", content: "Alice and Bob are going to a science fair on Friday." },
],
response_format: zodResponseFormat(CalendarEvent, "event"),
});
const event = completion.choices[0].message.parsed;
console.log(event);
```
```python Python theme={"system"}
import os
from pydantic import BaseModel
from openai import OpenAI
class CalendarEvent(BaseModel):
name: str
date: str
participants: list[str]
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.beta.chat.completions.parse(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "Extract the event information. Respond in JSON format."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
],
response_format=CalendarEvent,
)
event = completion.choices[0].message.parsed
print(event.model_dump_json(indent=2))
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "Extract the event information. Respond in JSON format."},
{"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "event",
"strict": true,
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"date": { "type": "string" },
"participants": { "type": "array", "items": { "type": "string" } }
},
"required": ["name", "date", "participants"],
"additionalProperties": false
}
}
}
}'
```
Some tips:
* Name keys clearly and intuitively
* Create clear titles and descriptions for important keys in your structure
* Create and use evals to determine the structure that works best for your use case
## Step By Step Example — Parsing The Model's Output
You can use the OpenAI SDK to parse the model's output into a typed object automatically.
### Step 1: Define your object
First you must define an object or data structure to represent the JSON Schema that the model should be constrained to follow.
```typescript TypeScript theme={"system"}
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
const Step = z.object({
explanation: z.string(),
output: z.string(),
});
const MathResponse = z.object({
steps: z.array(Step),
final_answer: z.string(),
});
```
```python Python theme={"system"}
from pydantic import BaseModel
class Step(BaseModel):
explanation: str
output: str
class MathResponse(BaseModel):
steps: list[Step]
final_answer: str
```
### Step 2: Supply your object in the API call
You can use the `parse` method to automatically parse the JSON response into the object you defined.
Under the hood, the SDK takes care of supplying the JSON schema corresponding to your data structure, and then parsing the response as an object.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const completion = await client.chat.completions.parse({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format." },
{ role: "user", content: "how can I solve 8x + 7 = -23" },
],
response_format: zodResponseFormat(MathResponse, "math_response"),
});
console.log(completion.choices[0].message.parsed);
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.beta.chat.completions.parse(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."},
{"role": "user", "content": "how can I solve 8x + 7 = -23"},
],
response_format=MathResponse,
)
print(completion.choices[0].message.parsed.model_dump_json(indent=2))
```
### Handling edge cases
In some cases, the model might not generate a valid response that matches the provided JSON schema.
This can happen if for example you reach a max tokens limit and the response is incomplete.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
import { z } from "zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const Step = z.object({
explanation: z.string(),
output: z.string(),
});
const MathResponse = z.object({
steps: z.array(Step),
final_answer: z.string(),
});
try {
const completion = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{
role: "system",
content: "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format.",
},
{
role: "user",
content: "how can I solve 8x + 7 = -23",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "math_response",
schema: {
type: "object",
properties: {
steps: {
type: "array",
items: {
type: "object",
properties: {
explanation: { type: "string" },
output: { type: "string" },
},
required: ["explanation", "output"],
additionalProperties: false,
},
},
final_answer: { type: "string" },
},
required: ["steps", "final_answer"],
additionalProperties: false,
},
strict: true,
},
},
max_tokens: 20, // intentionally low to demonstrate truncation handling
});
const message = completion.choices[0].message;
if (completion.choices[0].finish_reason === "length") {
console.log("Model did not return a complete response — parsing may fail.");
throw new Error("Incomplete response");
}
try {
const parsed = MathResponse.parse(JSON.parse(message.content));
console.log("Parsed math response:", parsed);
} catch (zodError) {
console.error("Response does not match the expected schema:", message.content);
}
} catch (e) {
console.error(e);
}
```
```python Python theme={"system"}
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
try:
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{
"role": "system",
"content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format.",
},
{"role": "user", "content": "how can I solve 8x + 7 = -23"},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "math_response",
"strict": True,
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": {"type": "string"},
"output": {"type": "string"},
},
"required": ["explanation", "output"],
"additionalProperties": False,
},
},
"final_answer": {"type": "string"},
},
"required": ["steps", "final_answer"],
"additionalProperties": False,
},
},
},
max_tokens=20, # intentionally low to demonstrate truncation handling
)
if completion.choices[0].finish_reason == "length":
raise Exception("Incomplete response")
math_response = json.loads(completion.choices[0].message.content)
print(json.dumps(math_response, indent=2))
except Exception as e:
print(str(e))
```
```bash cURL theme={"system"}
# Use max_tokens to demonstrate truncation handling.
# Check finish_reason in the response — "length" means the output was truncated.
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{
"role": "system",
"content": "You are a helpful math tutor. Guide the user through the solution step by step. Respond in JSON format."
},
{
"role": "user",
"content": "how can I solve 8x + 7 = -23"
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "math_response",
"schema": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"type": "object",
"properties": {
"explanation": { "type": "string" },
"output": { "type": "string" }
},
"required": ["explanation", "output"],
"additionalProperties": false
}
},
"final_answer": { "type": "string" }
},
"required": ["steps", "final_answer"],
"additionalProperties": false
},
"strict": true
}
},
"max_tokens": 20
}'
```
## Streaming
You can use streaming to process model responses as they are being generated, and parse them as structured data.
That way, you don't have to wait for the entire response to complete before handling it. This is particularly useful if you would like to display JSON fields one by one, or handle function call arguments as soon as they are available.
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const EntitiesSchema = z.object({
attributes: z.array(z.string()),
colors: z.array(z.string()),
animals: z.array(z.string()),
});
const stream = client.chat.completions
.stream({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "Extract entities from the input text. Respond in JSON format." },
{
role: "user",
content: "The quick brown fox jumps over the lazy dog with piercing blue eyes",
},
],
response_format: zodResponseFormat(EntitiesSchema, "entities"),
})
.on("content.delta", ({ snapshot, parsed }) => {
console.log("content:", snapshot);
console.log("parsed:", parsed);
console.log();
})
.on("content.done", (props) => {
console.log(props);
});
await stream.done();
const finalCompletion = await stream.finalChatCompletion();
console.log(finalCompletion);
```
```python Python theme={"system"}
import os
from typing import List
from pydantic import BaseModel
from openai import OpenAI
class EntitiesModel(BaseModel):
attributes: List[str]
colors: List[str]
animals: List[str]
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
with client.chat.completions.stream(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{
"role": "system",
"content": "Extract entities from the input text. Respond in JSON format.",
},
{
"role": "user",
"content": "The quick brown fox jumps over the lazy dog with piercing blue eyes",
},
],
response_format=EntitiesModel,
) as stream:
for event in stream:
if event.type == "content.delta":
if event.parsed is not None:
print("content.delta parsed:", event.parsed)
elif event.type == "content.done":
print("content.done")
elif event.type == "error":
print("Error in stream:", event.error)
final_completion = stream.get_final_completion()
print("Final completion:", final_completion)
```
```bash cURL theme={"system"}
# Streaming structured outputs works the same as regular streaming.
# Add "stream": true and parse SSE chunks as they arrive.
curl -N https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"stream": true,
"messages": [
{"role": "system", "content": "Extract entities from the input text. Respond in JSON format."},
{"role": "user", "content": "The quick brown fox jumps over the lazy dog with piercing blue eyes"}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "entities",
"strict": true,
"schema": {
"type": "object",
"properties": {
"attributes": { "type": "array", "items": { "type": "string" } },
"colors": { "type": "array", "items": { "type": "string" } },
"animals": { "type": "array", "items": { "type": "string" } }
},
"required": ["attributes", "colors", "animals"],
"additionalProperties": false
}
}
}
}'
```
## Supported schemas
Structured Outputs supports a subset of the [JSON Schema](https://json-schema.org/docs) language.
#### Supported types
The following types are supported for Structured Outputs:
* String
* Number
* Boolean
* Integer
* Object
* Array
* Enum
* anyOf
#### Required Fields And Additional Properties
To use Structured Outputs, all properties on all objects must be specified as `required`.
Also, `additionalProperties` must be set to `false`.
In the following example, note how both `location` and `unit` are listed as required properties.
```json JSON theme={"system"}
{
"name": "get_weather",
"description": "Fetches the weather in the given location",
"strict": true,
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The location to get the weather for"
},
"unit": {
"type": "string",
"description": "The unit to return the temperature in",
"enum": ["F", "C"]
}
},
"additionalProperties": false,
"required": ["location", "unit"]
}
}
```
#### Schema Limitations Depend on the Model
Limitations on the number of properties, enum values, and total string size may vary depending on the model you are using.
#### Key ordering
When using Structured Outputs, outputs will be produced in the same order as the ordering of keys in the schema.
## JSON mode
When using JSON mode, always instruct the model to produce JSON in the system prompt.
For example: "You are a helpful assistant. Respond in JSON format."
JSON mode is a more basic version of the Structured Outputs feature. While JSON mode ensures that model output is valid JSON, Structured Outputs reliably matches the model's output to the schema you specify. We recommend you use Structured Outputs if it is supported for your use case.
When JSON mode is turned on, the model's output is ensured to be valid JSON, except for in some edge cases that you should detect and handle appropriately.
To turn on JSON mode with the Chat Completions API you can set the `response_format` to `{ "type": "json_object" }`.
Important notes:
* When using JSON mode, you must always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate non-JSON or an unending stream of whitespace.
* JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. You should use Structured Outputs to ensure it matches your schema, or if that is not possible, you should use a validation library and potentially retries to ensure that the output matches your desired schema.
* Your application must detect and handle the edge cases that can result in the model output not being a complete JSON object (see below)
* Some models will include a triple backtick / JSON code format block around the JSON response. This should be detected and handled appropriately.
### Handling JSON Mode edge cases
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
try {
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{
role: "system",
content: "You are a helpful assistant designed to output JSON.",
},
{ role: "user", content: "Who won the world series in 2020? Please respond in the format {winner: ...}" },
],
response_format: { type: "json_object" },
});
// Check if the response was truncated due to context length
if (response.choices[0].finish_reason === "length") {
// Handle incomplete JSON
}
// Check if the model refused the request
if (response.choices[0].message.refusal) {
console.log(response.choices[0].message.refusal);
}
// Check if content was filtered
if (response.choices[0].finish_reason === "content_filter") {
// Handle filtered content
}
if (response.choices[0].finish_reason === "stop") {
console.log(JSON.parse(response.choices[0].message.content));
}
} catch (e) {
console.error(e);
}
```
```python Python theme={"system"}
import os
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
try:
response = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{"role": "system", "content": "You are a helpful assistant designed to output JSON."},
{"role": "user", "content": "Who won the world series in 2020? Please respond in the format {winner: ...}"},
],
response_format={"type": "json_object"},
)
message = response.choices[0]
# Check if the response was truncated due to context length
if message.finish_reason == "length":
raise Exception("Incomplete response — output was truncated")
# Check if the model refused the request
if hasattr(message.message, "refusal") and message.message.refusal:
print(message.message.refusal)
# Check if content was filtered
if message.finish_reason == "content_filter":
raise Exception("Response filtered")
if message.finish_reason == "stop":
print(json.loads(message.message.content))
except Exception as e:
print(str(e))
```
```bash cURL theme={"system"}
# JSON mode is enabled with response_format type "json_object".
# Check finish_reason in the response to handle edge cases:
# "stop" — complete JSON returned
# "length" — output truncated, JSON may be incomplete
# "content_filter" — content was filtered
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant designed to output JSON."},
{"role": "user", "content": "Who won the world series in 2020? Please respond in the format {winner: ...}"}
],
"response_format": { "type": "json_object" }
}'
```
# Vision
Source: https://docs.inference.net/api/vision
Use models to extract information from images.
## Introduction
**Vision Models** are *multi-modal models* that accept both text and images as input.
You can use vision models to extract information from images (for example, by asking the model to describe the image).
This guide explains how to use Vision Models with the Inference API.
## Getting Started
You'll need an Inference.net account and API key. See our [Quick Start Guide](/api/api-quickstart) for instructions on how to create an account and get an API key.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) for your language of choice.
To connect to Inference.net using the OpenAI SDK, you will need to set the base URL to `https://api.inference.net/v1`.
In the following examples, we are reading the API key from the environment variable `INFERENCE_API_KEY`.
## Step By Step Example
To use image inputs with the Inference API:
1. Encode your image as a base64 string
2. Include the base64 string in a [Data URI](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data) with an image mimetype (e.g. `image/png`)
3. Include the Data URI in the `content` array of a user message
4. Send the request to the Inference API and inspect the response
### Step 1: Encode your image as a Data URI
```typescript TypeScript theme={"system"}
const url = "https://upload.wikimedia.org/wikipedia/commons/3/3f/Crystal_Project_bug.png";
const response = await fetch(url);
const buffer = Buffer.from(await response.arrayBuffer());
const base64 = buffer.toString("base64");
const dataUri = `data:image/png;base64,${base64}`;
```
```python Python theme={"system"}
import base64
import requests
url = "https://upload.wikimedia.org/wikipedia/commons/3/3f/Crystal_Project_bug.png"
response = requests.get(url)
image_data = response.content
encoded_string = base64.b64encode(image_data).decode("utf-8")
data_uri = f"data:image/png;base64,{encoded_string}"
```
```bash cURL theme={"system"}
# Encode an image file to a base64 data URI:
DATA_URI="data:image/png;base64,$(base64 -i image.png)"
# Or fetch and encode from a URL:
DATA_URI="data:image/png;base64,$(curl -s https://upload.wikimedia.org/wikipedia/commons/3/3f/Crystal_Project_bug.png | base64)"
```
### Step 2: Structure and send your request
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const completion = await client.chat.completions.create({
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{
role: "system",
content: "You are a helpful assistant that can answer questions about the image.",
},
{
role: "user",
content: [
{
type: "image_url",
image_url: { url: dataUri },
},
{
type: "text",
text: "What is in this image?",
},
],
},
],
});
console.log(completion.choices[0].message.content);
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
completion = client.chat.completions.create(
model="google/gemma-3-27b-instruct/bf-16",
messages=[
{
"role": "system",
"content": "You are a helpful assistant that can answer questions about the image.",
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": data_uri},
},
{
"type": "text",
"text": "What is in this image?",
},
],
},
],
)
print(completion.choices[0].message.content)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant that can answer questions about the image."
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": { "url": "'"$DATA_URI"'" }
},
{
"type": "text",
"text": "What is in this image?"
}
]
}
]
}'
```
## Limitations
* We do not support sending images from a URL directly into the request body.
* Supported image formats include webp, png, gif and jpg/jpeg.
* The total size of the request body must be less than 1MB.
* Each request can contain a maximum of 2 images.
## Token Usage
Using images in a request counts towards the total token usage for a request.
The exact token count will vary, but a handy approximation of the number of tokens used by an image is the following formula:
```
h = max(2, min(1, HEIGHT / 560))
w = max(2, min(1, WIDTH / 560))
tokens = h * w * 1601
```
In plain English:
1. The image height and width in pixels are both divided by 560
2. The resulting height and width are clamped between 1 and 2
3. Finally, the height and width are multiplied together and then multiplied by 1,601
Here is a table of image dimensions and their corresponding estimated token counts:
| Height | Width | Tokens | Note |
| ------ | ------ | ------ | -------------------------------------------------------------- |
| 32px | 32px | 1,601 | Images smaller than 560x560 are still considered 560x560 |
| 560px | 560px | 1,601 | |
| 1120px | 1120px | 6,404 | 6,404 is the approximate maximum token usage of a single image |
The above formula and table is an approximation. We suggest that you:
* Explicitly check your image dimensions before submitting them to the API to avoid high token usage.
* Monitor your token usage and adjust your requests if necessary.
See the [Models](https://inference.net/models) page for current pricing per token for Vision Models.
# Authentication
Source: https://docs.inference.net/cli/authentication
Sign in interactively, use an API key for CI, and manage CLI credentials.
The CLI supports two auth modes: browser-based **session login** for interactive use and a **project API key** for CI and headless environments.
## `inf auth login`
Sign in through your browser using the OAuth 2.0 device authorization flow. Opens a verification URL, displays a user code, and polls until you approve in the browser.
```bash theme={"system"}
inf auth login
```
After sign-in, the CLI stores a session token in `~/.inf/config.json` and activates an organization (team). If your account belongs to multiple teams, `inf auth login` prompts you to choose one in an interactive terminal.
### Options
| Option | Required | Description |
| --------------------- | -------- | -------------------------------------------------------- |
| `--team ` | No | Activate a specific team by ID, slug, or exact team name |
### Team selection
Use `--team` when you know which team you want to activate, or when running in a non-interactive shell.
```bash theme={"system"}
inf auth login --team acme
```
You can pass a team ID, slug, or exact team name:
```bash theme={"system"}
inf auth login --team team_abc123
inf auth login --team acme
inf auth login --team "Acme Research"
```
If the selected team is different from the previously active team, the CLI clears the stored active project and then tries to auto-select a project from the newly active team. You can always run `inf project list` and `inf project switch ` to pick a different project.
In non-interactive environments, `inf auth login` cannot prompt for a team. If you belong to multiple teams and omit `--team`, the CLI falls back to the first team returned by the auth API. Pass `--team ` to make the selected team deterministic.
Session login requires a browser, so `inf auth login` is not suitable for CI or other headless environments. Use `inf auth set-key` or the `INF_API_KEY` env var there instead.
## `inf auth set-key`
Store a project API key on disk for headless or CI authentication.
```bash theme={"system"}
inf auth set-key
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ----------------------------------------------- |
| `key` | Yes | A project API key (starts with `sk-inference-`) |
After saving, the CLI validates the key by fetching the project list. A successful fetch auto-selects the first project as active.
### Example
```bash theme={"system"}
inf auth set-key sk-inference-...
```
## `inf auth status`
Show who you're signed in as, which auth method the CLI is using, the active team and project, and the API URL.
```bash theme={"system"}
inf auth status
```
## `inf auth logout`
Sign out, clear the session token / API key from `~/.inf/config.json`, and forget the active project and team.
```bash theme={"system"}
inf auth logout
```
## Credential resolution order
When multiple credentials are present, the CLI picks the first match:
1. `INF_API_KEY` environment variable
2. API key stored via `inf auth set-key`
3. Session token stored via `inf auth login`
`inf instrument` is the one exception — it **rejects** `INF_API_KEY` and requires a session login, because it needs to fetch your project's default API key on your behalf. Unset `INF_API_KEY` and run `inf auth login` before running `inf instrument`.
## Configuration
The CLI stores configuration at `~/.inf/config.json`, created automatically on first login. Tokens are stored with `0600` permissions.
### Environment variables
| Variable | Description | Default |
| ---------------- | -------------------------------------------------------------------------------------- | ----------------------------------------- |
| `INF_API_KEY` | API key for authentication. Takes precedence over stored credentials | — |
| `INF_API_URL` | Override the API base URL | `https://observability-api.inference.net` |
| `INF_PROJECT_ID` | Override the active project for any invocation (equivalent to `--project ` global) | — |
# Dashboard
Source: https://docs.inference.net/cli/dashboard
Launch the interactive terminal dashboard for a live overview of your project.
`inf dashboard` launches a full-screen interactive terminal UI that gives you a live overview of training runs, evaluations, datasets, and inferences in your active project.
**Alias:** `inf dash`
## `inf dashboard`
Launch the TUI. The dashboard reads your active project from `~/.inf/config.json`, or use the global `--project ` flag to target a different project for this session.
```bash theme={"system"}
inf dashboard
```
### Keyboard Shortcuts
| Key | Action |
| ------------------- | ----------------------------- |
| `1` – `4` | Switch between tabs directly |
| `Tab` / `Shift+Tab` | Cycle through tabs |
| `j` / `k` | Navigate up and down in lists |
| `Enter` | Drill into the selected item |
| `r` | Refresh the current view |
| `/` | Open the command palette |
| `q` or `Ctrl+C` | Quit the dashboard |
### Tabs
The dashboard provides four tabs for navigating your project data:
| Tab | Key | Description |
| ---------- | --- | ---------------------------------------------------- |
| Training | `1` | View training runs with status, progress, and loss |
| Evals | `2` | Browse eval run groups and individual run results |
| Datasets | `3` | List filtered datasets with export status |
| Inferences | `4` | View recent inferences with token counts and latency |
Select any item in a list and press `Enter` to open a detail panel with more information.
### Examples
```bash theme={"system"}
# Launch for the active project
inf dashboard
# Launch for a specific project without changing your stored config
inf dashboard --project proj_789ghi012jkl
# Use the alias
inf dash
```
The dashboard requires a terminal that supports modern rendering. Most standard terminal emulators (Terminal.app, iTerm2, Alacritty, Windows Terminal) work well.
# Datasets
Source: https://docs.inference.net/cli/datasets
Upload JSONL inference data, materialize eval/training datasets, and download them.
Use `inf dataset` to upload JSONL inference data and manage datasets created from captured traffic, existing uploads, or JSONL files on disk. Materialized datasets feed into [`inf eval run`](/cli/evals) for evals and into training jobs.
**Alias:** `inf datasets`
## `inf dataset upload`
Import a JSONL file into the active project as an upload entry. An upload is the raw material you can then materialize into an eval or training dataset. The CLI validates the file locally, uploads it in parts, waits for processing to finish, and prints the detected format plus the processed line count.
```bash theme={"system"}
inf dataset upload
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | -------------------------------- |
| `file` | Yes | Path to the JSONL file to upload |
### Options
| Flag | Required | Description | Default |
| ------------------- | -------- | ---------------------------------------------------------------- | -------------------------- |
| `-n, --name ` | No | Upload name shown in Catalyst | Filename without extension |
| `--no-wait` | No | Return after the transfer finishes instead of polling processing | Off |
Uploaded data appears in **Datasets → Uploads** in the dashboard. Once processing completes, create an eval or training dataset from that upload — either with [`inf dataset create --upload-id`](#inf-dataset-create) below or in the dashboard.
### Examples
```bash theme={"system"}
# Use the filename as the upload name and wait for processing
inf dataset upload ./data/support-summaries.jsonl
# Set a custom upload name
inf dataset upload ./data/support-summaries.jsonl --name support-summaries-v2
# Return after the transfer completes, without waiting for processing
inf dataset upload ./data/support-summaries.jsonl --no-wait
```
## `inf dataset create`
Materialize an eval or training dataset from captured traffic, an existing upload, or a JSONL file on disk. The file-backed path uploads, waits for processing, and materializes in one command.
```bash theme={"system"}
inf dataset create -n -t [source-flags…]
```
### Options
| Flag | Required | Description | Default |
| ---------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `-n, --name ` | Yes | Dataset name | — |
| `-t, --type ` | Yes | `eval` or `training` | — |
| `-f, --file ` | No | JSONL file on disk — uploads, waits for processing, then materializes from that upload | — |
| `--upload-id ` | No | Materialize from an existing upload | — |
| `--task ` | No | Filter captured traffic by task ID | — |
| `--model ` | No | Filter captured traffic by model ID | — |
| `--since ` | No | Start of the time window for traffic filters (ISO 8601 or `YYYY-MM-DD HH:MM:SS`) | 30 days ago |
| `--until ` | No | End of the time window for traffic filters | 1 minute from now |
| `--limit ` | No | Cap on the number of inferences included | — |
| `--status ` | No | Status filter: `success` (default), `2xx`, or a specific code like `200` — datasets reject non-success traffic unless you override | `success` |
| `--description ` | No | Free-text dataset description | — |
`--file` and `--upload-id` are mutually exclusive — `--file` creates a new upload automatically. Date values accept ISO 8601 (`2026-04-01T00:00:00Z`) or ClickHouse format (`2026-04-01 00:00:00`).
Dataset materialization runs asynchronously. The command prints the dataset ID and points at `inf dataset get ` to check progress.
### Examples
```bash theme={"system"}
# One-command: upload a JSONL file and materialize an eval dataset
inf dataset create -n demo-eval -t eval --file ./samples.jsonl
# Materialize from an existing upload
inf dataset create -n training-v1 -t training --upload-id up_abc123
# Filter captured traffic by task + time window
inf dataset create -n support-eval -t eval \
--task support-tickets \
--since 2026-04-01 \
--until 2026-04-14
# Cap to 1,000 rows (only successful traffic is included by default)
inf dataset create -n small-eval -t eval --task support-tickets --limit 1000
```
## `inf dataset list`
Display datasets in the active project.
```bash theme={"system"}
inf dataset list
```
**Alias:** `inf dataset ls`
### Options
| Flag | Required | Description | Default |
| ----------------- | -------- | ------------------------- | ------- |
| `-l, --limit ` | No | Maximum number of results | `20` |
The table shows the dataset ID (8-char prefix), name, type, inference count, export status, and creation date. Use `--json` to get full UUIDs for scripting.
### Examples
```bash theme={"system"}
# Default table view
inf dataset list
# More results
inf dataset list --limit 100
# Pipe full UUIDs into another command
inf dataset list --json | jq -r '.[].id'
```
## `inf dataset get`
View detailed information about a specific dataset — ID, name, type, inference count, export status, source project, and creation date.
```bash theme={"system"}
inf dataset get
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ------------------------------------------------- |
| `id` | Yes | Dataset ID, UUID prefix (4+ chars), or exact name |
## `inf dataset download`
Download a dataset as a JSONL file. If the server-side export isn't ready yet, the CLI requests it and polls until it's ready before downloading.
```bash theme={"system"}
inf dataset download [id]
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ------------------------------------------------------------------------------------------------------------------------ |
| `id` | No | Dataset ID, UUID prefix (4+ chars), or exact name. If omitted in an interactive terminal, the CLI prompts you to choose. |
### Options
| Flag | Required | Description | Default |
| ----------------------- | -------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `-o, --output ` | No | Output file path | `.jsonl` for Hugging Face, `.source-backed.jsonl` for source-backed |
| `-f, --format ` | No | Download format: `huggingface` or `source-backed` | Prompted in a TTY; otherwise `huggingface` |
The CLI resolves dataset IDs by exact ID, UUID prefix, or exact name.
### Examples
```bash theme={"system"}
# Download to the default filename
inf dataset download ds_abc123
# Download to a specific file
inf dataset download ds_abc123 --output ./data/my-dataset.jsonl
# Download source-backed JSONL (request/response objects)
inf dataset download customer-support-eval --format source-backed
# Pick interactively when no id is provided
inf dataset download
```
# Evals
Source: https://docs.inference.net/cli/evals
Create rubrics, launch eval runs, and inspect results from the terminal.
Run and inspect model evaluations from the command line. Manage rubrics (the judge prompts evals run against), list and inspect run groups, launch new runs, and browse eval-ready datasets.
**Alias:** `inf evals`
The full eval loop is paste-able from the terminal:
```bash theme={"system"}
# 1. Create a rubric from a markdown file
inf eval rubric create -n support-tickets-v1 -f ./rubric.md
# → Rubric rub_abc12 / version rv_xyz45 created.
# 2. Materialize an eval dataset (traffic-backed, upload-backed, or from a file)
inf dataset create -n demo-eval -t eval --file ./samples.jsonl
# → Dataset ds_def78 created.
# 3. Launch an eval run group
inf eval run \
--rubric-id rub_abc12 \
--dataset-id ds_def78 \
--models openai:gpt-5.2,anthropic:claude-sonnet-4-6 \
--judge-model anthropic:claude-sonnet-4-6
# → Run group rg_20260415_152340 created.
# 4. Track progress
inf eval get rg_20260415_152340
```
Route IDs look like `:` (e.g. `openai:gpt-5.2`). Use [`inf models list`](/cli/models#inf-models-list) to discover every route ID available to your team — see [Route IDs](/cli/models#route-ids) for the full format.
## `inf eval rubric create`
Create a rubric — the judge prompt an eval run scores responses against. Rubrics live in the active project, carry versioned prompt content, and are passed to `inf eval run` by ID. The template must contain the placeholder `{{ eval_model_response }}` where the model's response will be injected for scoring.
```bash theme={"system"}
inf eval rubric create -n -f
```
### Options
| Flag | Required | Description | Default |
| ------------------- | -------- | ------------------------------------------------------------ | -------------- |
| `-n, --name ` | Yes | Rubric name | — |
| `-f, --file ` | Yes | Path to a markdown file containing the judge prompt template | — |
| `--max-score ` | No | Maximum score for the rubric (2–100) | `10` |
| `--project-id ` | No | Project to create the rubric in | Active project |
Prints the rubric ID and the first version ID. Use them directly with `inf eval run`.
### Examples
```bash theme={"system"}
# Create a rubric with the default 0–10 scoring scale
inf eval rubric create -n support-tickets-v1 -f ./rubric.md
# Create a rubric with a 0–100 scale
inf eval rubric create -n quality-v2 -f ./quality.md --max-score 100
```
## `inf eval rubric get`
Get details of a rubric — ID, name, latest version number, version count, score range, and a preview of the template.
```bash theme={"system"}
inf eval rubric get
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ---------------------------------------------------- |
| `id` | Yes | Full UUID, 4+ character prefix, or exact rubric name |
Ambiguous prefixes print the candidate list and abort.
## `inf eval rubric delete`
Archive (soft-delete) a rubric. Rubrics cannot be hard-deleted — archiving hides them from `inf eval rubrics` but preserves their eval history. Restore from the dashboard if needed.
```bash theme={"system"}
inf eval rubric delete
```
**Alias:** `inf eval rubric archive ` — both names do the same thing; use whichever reads clearer in your script.
### Arguments
| Argument | Required | Description |
| -------- | -------- | ---------------------------------------------------- |
| `id` | Yes | Full UUID, 4+ character prefix, or exact rubric name |
### Options
| Flag | Required | Description | Default |
| ----------- | --------------------------- | ---------------------------- | ------- |
| `-y, --yes` | Yes in non-TTY environments | Skip the confirmation prompt | Off |
In an interactive terminal, the CLI asks for confirmation unless `-y` is passed. In non-TTY environments (CI, scripts) the command refuses to run without `-y`.
### Examples
```bash theme={"system"}
# Archive interactively (prompts for confirmation)
inf eval rubric delete support-tickets-v1
# Archive non-interactively
inf eval rubric archive rub_abc12 --yes
```
## `inf eval rubrics`
List rubrics in the active project.
```bash theme={"system"}
inf eval rubrics
```
**Alias:** `inf eval defs`
### Options
| Flag | Required | Description | Default |
| -------------------- | -------- | ------------------------ | ------- |
| `--include-archived` | No | Include archived rubrics | Off |
Shows the rubric ID (8-char prefix), name, latest version, total version count, and creation date. Use `--json` for full UUIDs.
## `inf eval run`
Launch a new eval run group against one or more models, scored by a judge model.
```bash theme={"system"}
inf eval run \
--rubric-id \
--dataset-id \
--models \
--judge-model
```
### Options
| Flag | Required | Description | Default |
| -------------------------- | -------- | -------------------------------------------------------------------------- | -------------- |
| `--rubric-id ` | Yes | Rubric ID | — |
| `--dataset-id ` | Yes | Eval-type dataset ID (create one with `inf dataset create -t eval`) | — |
| `--models ` | Yes | Comma-separated model route IDs — run `inf models list` to discover them | — |
| `--judge-model ` | Yes | Route ID of the judge model — run `inf models list --judge-only` to filter | — |
| `--rubric-version-id ` | No | Pin to a specific rubric version | Latest version |
| `--sample-size ` | No | Samples drawn from the dataset per model (1–100) | `100` |
| `-n, --name ` | No | Display name for the run group | Auto-generated |
Prints the run group ID and an `inf eval get ` follow-up command to track progress.
### Examples
```bash theme={"system"}
# Launch a run against two models with a third as judge
inf eval run \
--rubric-id rub_abc12 \
--dataset-id ds_def78 \
--models openai:gpt-5.2,anthropic:claude-sonnet-4-6 \
--judge-model anthropic:claude-sonnet-4-6
# Pin to a specific rubric version
inf eval run \
--rubric-id rub_abc12 \
--rubric-version-id rv_xyz45 \
--dataset-id ds_def78 \
--models openai:gpt-5.2 \
--judge-model anthropic:claude-sonnet-4-6
```
## `inf eval list`
List eval run groups for a given rubric.
```bash theme={"system"}
inf eval list --rubric-id
```
**Alias:** `inf eval ls`
### Options
| Flag | Required | Description | Default |
| -------------------------- | -------- | ----------------------------------- | ------- |
| `--rubric-id ` | Yes | Rubric ID to list runs for | — |
| `--rubric-version-id ` | No | Filter by a specific rubric version | — |
Shows the run group ID (8-char prefix), rubric version, model count, derived status (`pending`, `running`, `failed`, or `completed`), and creation date.
## `inf eval get`
View detailed information about a specific eval run group.
```bash theme={"system"}
inf eval get
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | --------------------- |
| `id` | Yes | The eval run group ID |
### Output
The detail view covers the run group itself, followed by a sub-table of individual runs:
| Field | Description |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | Run group ID |
| `rubricId` / `rubricVersionId` | Rubric and pinned version |
| `evalDatasetId` | Dataset the run group scored |
| `judgeProvider` / `judgeModelId` | Judge model scoring the responses |
| `models` | How many models were evaluated in this run group |
| `created` | Run group creation timestamp |
| Runs sub-table | One row per model: run ID, provider, model, status, average score, failed sample count, `completed/total` samples. When avg score is `—`, the adjacent `N failed` hint shows how many samples the judge couldn't score. |
## `inf eval datasets`
List datasets available for evaluations (type = `eval`).
```bash theme={"system"}
inf eval datasets
```
### Options
| Flag | Required | Description | Default |
| -------------------- | -------- | ------------------------- | ------- |
| `-l, --limit ` | No | Maximum number of results | `50` |
| `--include-archived` | No | Include archived datasets | Off |
Eval datasets are materialized via [`inf dataset create -t eval …`](/cli/datasets#inf-dataset-create) or the dashboard. The output shows the dataset ID (8-char prefix), name, inference count, and creation date.
# HALO
Source: https://docs.inference.net/cli/halo
Run HALO agent-trace analyses, schedule recurring reports, and pull report markdown from the CLI.
HALO analyzes your agent's traces and produces a markdown report that flags anomalies, errors, inefficiencies, and opportunities to improve reliability, latency, cost, and tool usage. The report is written to read like a brief you can paste straight into a coding agent to apply the fixes.
Use `inf halo` to kick off a one-off analysis, schedule recurring ones, watch a run reach completion, and read the resulting report.
## How HALO is organized
* A **run** is one analysis of one agent over one time window. Each completed run produces exactly one report.
* A **conversation** is the thread a run lives in. The first assistant message is the original report; any follow-up questions you ask add more runs and more assistant messages to the same conversation.
* A **schedule** fires recurring runs for one agent on a cadence (hourly, daily, weekly, or monthly).
The report markdown is the assistant message content inside a conversation. There is no separate "report" artifact to download, so reading a report means reading its conversation.
## Quickstart: run an analysis and read the report
```bash theme={"system"}
# 1. Find the agent you want to analyze
inf halo agent list
# 2. Start a run for that agent over the last 24h
inf halo run create --agent-uuid
# 3. Wait for it to finish (prints the run id from step 2)
inf halo run poll
# 4. Read the report (the assistant message in the conversation)
inf halo conversation get
```
`inf halo run create` prints both the `run-id` and the `conversation-id`. The report prints in full from `inf halo conversation get` — no `--json` required. Add `--json` only when you want the raw structured payload for scripting.
## `inf halo agent`
List the agents HALO sees in your project's recent traces. You need an agent's UUID to start a run or create a schedule.
```bash theme={"system"}
inf halo agent list
```
**Alias:** `inf halo agent ls`
### Options
| Flag | Required | Description | Default |
| ---------------------- | -------- | ----------------------------------------------------------------------------------------- | ------- |
| `--time-range ` | No | Trace window to scan for agents: `1h`, `6h`, `12h`, `1d`, `3d`, `7d`, `14d`, `30d`, `90d` | `30d` |
## `inf halo run`
Create and inspect HALO runs.
### `inf halo run create`
Start a manual analysis for one agent over a time window.
```bash theme={"system"}
inf halo run create --agent-uuid
```
In an interactive terminal you can omit `--agent-uuid` to pick from recent agents. In non-interactive or `--json` mode, `--agent-uuid` is required.
#### Options
| Flag | Required | Description | Default |
| ---------------------------- | ------------------------------- | ------------------------------------------------------------ | ------------------------------- |
| `--agent-uuid ` | No (required non-interactively) | Agent UUID from `inf halo agent list` | Interactive picker |
| `--prompt ` | No | Prompt steering the analysis | Default HALO prompt |
| `--window-start-at ` | No | Window start, ISO datetime | Derived from `--lookback-hours` |
| `--window-end-at ` | No | Window end, ISO datetime | Now |
| `--lookback-hours ` | No | Window length when `--window-start-at` is omitted | `24` |
| `--model-id ` | No | Catalog HALO model id | Catalog default |
| `--span-limit ` | No | Per-run span cap | `10000` |
| `--max-subagent-depth ` | No | Engine recursion ceiling (1–5) | `3` |
| `--max-turns ` | No | Per-agent turn ceiling (1–100) | `50` |
| `--agent-time-range ` | No | Trace window preset used when picking an agent interactively | `30d` |
### `inf halo run poll`
Poll a run until it reaches a terminal state (`completed`, `failed`, `cancelled`, `timed_out`, or `no_traces`). Exits `0` only on `completed`.
```bash theme={"system"}
inf halo run poll
```
#### Options
| Flag | Required | Description | Default |
| ---------------------- | -------- | ------------- | --------------- |
| `--interval ` | No | Poll interval | `5` |
| `--timeout ` | No | Total timeout | `1800` (30 min) |
### `inf halo run get`
Fetch a run's status and its referenced trace dataset. This returns run metadata, not the report text — read the conversation for the report.
```bash theme={"system"}
inf halo run get
```
### `inf halo run events`
List the structured event timeline for a run (started, heartbeat, agent steps, completed, failed).
```bash theme={"system"}
inf halo run events
```
#### Options
| Flag | Required | Description | Default |
| ------------- | -------- | -------------------- | ------- |
| `--limit ` | No | Max events to return | `100` |
### `inf halo run cancel`
Request cancellation of an in-flight run.
```bash theme={"system"}
inf halo run cancel --reason "no longer needed"
```
#### Options
| Flag | Required | Description | Default |
| ----------------- | -------- | ------------------------------------ | ---------------- |
| `--reason ` | No | Reason recorded for the cancellation | `user-cancelled` |
## `inf halo conversation`
Inspect HALO conversations and read their reports.
**Alias:** `inf halo conv`
### `inf halo conversation list`
List conversations in the active project, newest first.
```bash theme={"system"}
inf halo conversation list
```
**Alias:** `inf halo conversation ls`
#### Options
| Flag | Required | Description | Default |
| ------------- | -------- | --------------------------- | ------- |
| `--limit ` | No | Max conversations to return | `50` |
### `inf halo conversation get`
Print a conversation with its messages, runs, and trace datasets. The assistant messages are the report markdown — the first assistant message is the original report; later ones answer follow-up questions in the same thread. Message content prints in full.
```bash theme={"system"}
inf halo conversation get
```
Add `--json` to get the raw payload (every message, run, and trace dataset) for scripting.
## `inf halo schedule`
Manage recurring HALO analyses. Each schedule targets one agent and fires runs on a cadence.
### `inf halo schedule list`
```bash theme={"system"}
inf halo schedule list
```
**Alias:** `inf halo schedule ls`
| Flag | Required | Description | Default |
| -------------------- | -------- | -------------------------- | ------- |
| `--include-archived` | No | Include archived schedules | `false` |
### `inf halo schedule get`
```bash theme={"system"}
inf halo schedule get
```
### `inf halo schedule runs`
List recent runs fired by a schedule.
```bash theme={"system"}
inf halo schedule runs
```
| Flag | Required | Description | Default |
| ------------- | -------- | ------------------ | ------- |
| `--limit ` | No | Max runs to return | `50` |
### `inf halo schedule create`
```bash theme={"system"}
inf halo schedule create \
--title "Daily checkout-agent review" \
--agent-uuid \
--frequency daily \
--hours 9 \
--minutes 0 \
--timezone America/Los_Angeles
```
| Flag | Required | Description | Default |
| -------------------------- | ------------------------ | -------------------------------------------------------- | ------------------- |
| `--title ` | Yes | Human-readable schedule title | - |
| `--agent-uuid <id>` | Yes | Agent the schedule analyzes (from `inf halo agent list`) | - |
| `--frequency <freq>` | Yes | `hourly`, `daily`, `weekly`, or `monthly` | - |
| `--prompt <text>` | No | Prompt steering each run | Default HALO prompt |
| `--hours <list>` | For daily/weekly/monthly | Comma-separated hours (0–23) | - |
| `--minutes <list>` | Yes | Comma-separated minutes (0–59) | - |
| `--days-of-week <list>` | For weekly | Comma-separated days (0=Sun…6=Sat) | - |
| `--days-of-month <list>` | For monthly | Comma-separated days (1–31) | - |
| `--timezone <tz>` | No | IANA timezone | `UTC` |
| `--model-id <id>` | No | Catalog HALO model id | Catalog default |
| `--span-limit <n>` | No | Per-run span cap | `10000` |
| `--lookback-hours <n>` | No | How far back each run looks, independent of cadence | `24` |
| `--max-subagent-depth <n>` | No | Engine recursion ceiling (1–5) | `3` |
| `--max-turns <n>` | No | Per-agent turn ceiling (1–100) | `50` |
| `--enabled <bool>` | No | Whether the schedule fires | `true` |
### `inf halo schedule update`
Update mutable fields on a schedule. Accepts the same flags as `create` (all optional), keyed by `<schedule-id>`.
<Metadata />
```bash theme={"system"}
# Pause a schedule
inf halo schedule update <schedule-id> --enabled false
# Repoint it at a different agent and widen the window
inf halo schedule update <schedule-id> --agent-uuid <other-agent> --lookback-hours 168
```
### `inf halo schedule archive` / `unarchive`
Archive a schedule so it stops firing, or restore it later.
<Metadata />
```bash theme={"system"}
inf halo schedule archive <schedule-id>
inf halo schedule unarchive <schedule-id>
```
`archive` prompts for confirmation. Pass `-y` / `--yes` to skip the prompt.
## Common workflows
<Metadata />
```bash theme={"system"}
# Run an analysis end-to-end and capture the report markdown to a file
RUN=$(inf halo run create --agent-uuid <agent-uuid> --json)
RUN_ID=$(echo "$RUN" | jq -r '.runId')
CONV_ID=$(echo "$RUN" | jq -r '.conversationId')
inf halo run poll "$RUN_ID"
# Extract just the original report (first assistant message) as markdown
inf halo conversation get "$CONV_ID" --json \
| jq -r '.messages | map(select(.role == "assistant"))[0].content' \
> halo-report.md
```
## Related commands
* [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) inspect the underlying traces HALO analyzes.
* The [MCP server](/integrations/mcp-server) exposes the same reports to AI coding assistants via `list_halo_conversations` and `get_halo_conversation`.
# Inferences
Source: https://docs.inference.net/cli/inferences
List, filter, sort, and export inference requests and responses captured by Gateway.
Inspect inference requests and responses captured by Gateway. List and filter inferences in the active project, sort by tokens / cost / latency, discover the available filter values, and fetch the complete stored request and response bodies.
**Alias:** `inf inferences`
## `inf inference list`
List and filter inferences in the active project.
<Metadata />
```bash theme={"system"}
inf inference list
```
**Alias:** `inf inference ls`
### Options
| Flag | Required | Description | Default |
| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `-l, --limit <n>` | No | Maximum number of results (1–100) | `20` |
| `--sort <key>` | No | Sort key: `sent_at`, `http_code`, `cost`, `duration`, `tokens`, `input_tokens`, `output_tokens`, `input_size`, `output_size` | `sent_at` |
| `--order <asc\|desc>` | No | Sort order | `desc` |
| `--filter <expr>` | No | Numeric filter `<field><op><number>` (repeatable). Fields: `inputTokens`, `outputTokens`, `inputSizeBytes`, `outputSizeBytes`, `durationMs`, `totalCost`. Ops: `> >= < <= = !=` | — |
| `--metadata <expr>` | No | Metadata filter `<key><op><value>` (repeatable). Ops: `= != ~ !~`, plus `<key> is_empty` / `<key> is_not_empty` | — |
| `-m, --model <name>` | No | Filter by model (repeatable or comma-separated) | All models |
| `--provider <name>` | No | Filter by downstream provider (repeatable or comma-separated) | All |
| `--environment <name>` | No | Filter by environment (repeatable or comma-separated) | All |
| `--status <code>` | No | Filter by HTTP status code, e.g. `200,500` (repeatable or comma-separated) | All |
| `--task <id>` | No | Filter by task ID — matches the `x-inference-task-id` header set by [`inf instrument --task-id`](/cli/instrument) (repeatable or comma-separated) | — |
| `--upload <id>` | No | Filter by upload ID (repeatable or comma-separated) | — |
| `--stream <bool>` | No | Filter by streamed responses (`true` / `false`) | — |
| `--from <iso>` / `--to <iso>` | No | Absolute time range (ISO-8601). Use both together. | — |
| `--range <preset>` | No | Relative time range: `1h, 6h, 12h, 1d, 3d, 7d, 14d, 30d, 90d, all` | — |
| `--cursor <cursor>` | No | Pagination cursor (from a previous `--json` response) | — |
<Note>
Don't hardcode filter values — run [`inf inference facets`](#inf-inference-facets) to discover the valid models, providers, environments, tasks, and metadata keys for the active project, along with the full filter reference.
</Note>
The human-readable table shows the inference ID, model, status code (color-coded), input/output token counts, latency, cost, and timestamp. Add the global `--json` flag to emit the **full result object** — `{ items, nextCursor, hasMore, totalCount }` — so scripts can read `.items` and paginate by passing `.nextCursor` back via `--cursor`.
### Examples
<Metadata />
```bash theme={"system"}
# List the 10 most recent inferences
inf inference list --limit 10
# Largest-input requests first — e.g. find prompts over 39k tokens
inf inference list --sort input_tokens --order desc --limit 50
inf inference list --filter inputTokens>39000
# Combine numeric, array, and metadata filters over a time window
inf inference list --model meta-llama/llama-3.1-8b-instruct/fp-8 --status 200 --range 7d
inf inference list --filter durationMs>5000 --metadata user_id=abc123
# Paginate from a script — JSON output includes nextCursor
cursor=$(inf --json inference list --sort input_tokens --order desc | jq -r '.nextCursor')
inf --json inference list --sort input_tokens --order desc --cursor "$cursor"
```
## `inf inference facets`
Probe the active project for the values you can filter on — models, providers, environments, task IDs, and metadata keys — plus the full numeric/metadata/sort reference. Values are discovered from your data, not hardcoded, which makes this the starting point for building a filtered `inf inference list` query (and a one-call way for an AI agent to learn the filter surface).
**Alias:** `inf inference filters`
<Metadata />
```bash theme={"system"}
inf inference facets
```
Add the global `--json` flag to emit a machine-readable object combining the probed values with a static `reference` (numeric fields, operators, and sort keys).
## `inf inference get`
Fetch the **complete stored request and response** for a specific inference — method, path, headers, and the full request/response bodies (returned only when payload storage was enabled for the request; otherwise the sides come back `null`). Useful for debugging specific calls, inspecting model behavior, or archiving payloads.
<Metadata />
```bash theme={"system"}
inf inference get <id>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | -------------------------------------------- |
| `id` | Yes | Full inference UUID or a 4+ character prefix |
### Options
| Flag | Required | Description | Default |
| --------------------- | -------- | ----------------------------------------------------------------------------------------------- | ------- |
| `--request` | No | Output only the request (omit the response) | Both |
| `--response` | No | Output only the response (omit the request) | Both |
| `--body` | No | Output only the raw body content (no method/path/headers). Requires `--request` or `--response` | — |
| `-o, --output <path>` | No | Write the output to a file instead of stdout | stdout |
Add the global `--json` flag to print the full `{ request, response }` payload (method, path, headers, and complete bodies) as clean JSON for piping into `jq` or saving.
### Examples
<Metadata />
```bash theme={"system"}
# Using a 4+ character UUID prefix (human-readable view)
inf inference get inf_abc1
# Pull a full UUID from list output, then fetch it
inf inference list --json | jq -r '.items[0].id' | xargs inf inference get
# Full request + response as clean JSON, then extract the response body
inf --json inference get <id> | jq '.response.body'
# Dump just the raw response body to a file
inf inference get <id> --response --body -o response.json
```
# Instrument
Source: https://docs.inference.net/cli/instrument
Automatically instrument your codebase to route LLM calls through Inference.net Catalyst using an AI coding agent.
`inf instrument` hands your codebase to an AI coding agent (Claude Code, OpenCode, or Codex) to wire up Catalyst observability for you. It scans your project, finds your LLM clients — OpenAI, Anthropic, LangChain, Amazon Bedrock, Gemini, Groq, Cerebras, OpenRouter, and others — and rewrites them to route through the Inference.net gateway so every call is captured in Gateway.
<Info>
This is the fastest way to connect an existing app to Catalyst. You don't need to know the SDK changes in advance — the agent figures it out from your code.
</Info>
## Prerequisites
* A signed-in CLI session. `inf instrument` requires a session login — API-key auth is rejected. Run `inf auth login` first.
* A supported AI coding agent on your `PATH`. Install one of:
| Agent | Binary | Install |
| ----------- | ---------- | ------------------------------------------------ |
| Claude Code | `claude` | [docs.anthropic.com/en/docs/claude-code][claude] |
| OpenCode | `opencode` | [opencode.ai][opencode] |
| Codex | `codex` | [github.com/openai/codex][codex] |
[claude]: https://docs.anthropic.com/en/docs/claude-code
[opencode]: https://opencode.ai
[codex]: https://github.com/openai/codex
## `inf instrument`
Instrument the current working directory. The command walks you through:
1. Confirming the target project (auto-selected if you only have one).
2. Fetching your project's default API key.
3. Detecting installed coding agents and picking one (prompted if you have more than one).
4. Downloading the instrumentation skill and building the agent prompt.
5. Launching the agent interactively so you can review and approve the changes.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project
inf instrument
```
### Options
| Flag | Required | Description | Default |
| ---------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `--dry-run` | No | Ask the agent to preview changes without modifying files | Off |
| `--agent <name>` | No | Preselect the coding agent by binary name: `claude`, `opencode`, or `codex`. Skips the interactive picker — useful in CI, scripts, or when driving `inf instrument` from another agent | Prompt |
| `-y, --yes` | No | Skip the "Instrument this project?" confirmation | Off |
| `--print-prompt` | No | Fetch the instrumentation skill, build the full agent prompt, print it to stdout, and exit without spawning an agent. Lets another orchestrator (your own agent harness, a CI job) drive the agent itself | Off |
| `--task-id <id>` | No | Default task id the agent tags every call with via the `x-inference-task-id` header. `inf inference list --task …` and `inf dataset create --task …` filter on this. 1–64 alphanumeric chars plus `._:-` | `default` |
### What the agent does
Once the agent launches, it:
* Scans your codebase for LLM clients (OpenAI, Anthropic, LangChain, Amazon Bedrock, Gemini, Groq, Cerebras, OpenRouter, and others).
* Redirects base URLs to the Inference.net gateway.
* Adds routing headers so requests are authenticated, forwarded, and traced.
* Tags every call with the default `x-inference-task-id` so call sites group automatically in Gateway.
* Shows you a diff before applying changes.
Your app keeps using the same provider SDKs it already has — `inf instrument` rewrites the client construction, not your call sites.
### Examples
<Metadata />
```bash theme={"system"}
# Preview what would change without touching any files
inf instrument --dry-run
# Use Claude Code non-interactively (skip the agent picker and the confirmation)
inf instrument --agent claude --yes
# Tag every instrumented call with a custom task id for filtering later
inf instrument --task-id support-chatbot
# Print the prompt for your own agent orchestrator instead of spawning one
inf instrument --print-prompt > instrument-prompt.md
```
### Verify instrumentation
After the agent finishes, run your app to generate a few LLM calls, then confirm they're captured:
<Metadata />
```bash theme={"system"}
inf inference list
```
Or open the observability dashboard link the command prints on completion.
## Supported providers
**Built-in:** OpenAI, Anthropic.
**OpenAI-compatible via `x-inference-provider-url`:** Amazon Bedrock, Google Gemini, Together AI, Groq, Fireworks AI, Mistral AI, Cerebras, Perplexity, DeepSeek, OpenRouter, Azure OpenAI, and any OpenAI-compatible endpoint.
## Troubleshooting
**`inf instrument now requires a session login. Unset INF_API_KEY and run inf auth login.`**
The command refuses to run with `INF_API_KEY` set in your environment. Unset it and sign in with `inf auth login` — instrumentation needs your session to fetch the project's default API key on your behalf.
**`No supported AI coding agent found.`**
Install one of Claude Code, OpenCode, or Codex from the links in [Prerequisites](#prerequisites). `inf instrument` detects the binary on your `PATH`.
**`--agent <name> is not installed (or not supported).`**
The binary name must match exactly: `claude`, `opencode`, or `codex`. Check `which claude` (or the name you passed) resolves to an executable.
**`No default API key found for project …`**
Open the dashboard → the relevant project → **API Keys** and create one. `inf instrument` needs a project-scoped key to embed in the agent prompt.
**The agent exited with an error.**
`inf instrument` prints the dashboard URL for manual integration on failure. You can also rerun with `-v` to see debug output, including the skill URL and prompt length.
# Models
Source: https://docs.inference.net/cli/models
Browse callable models, providers, capabilities, and pricing from the terminal.
`inf models` lets you browse every model available to your active team — both platform-provided models and any BYOK (bring-your-own-key) routes your team has configured.
**Alias:** `inf model`
`inf eval run` takes model route IDs via `--models` and `--judge-model`. `inf models list` is where you discover those route IDs.
## Route IDs
Route IDs look like `<provider>:<model-alias>` — for example `openai:gpt-5.2`, `anthropic:claude-sonnet-4-6`, `cerebras:llama-3.3-70b`. They are the canonical identifier the CLI and API use to address a specific model route, and they're what [`inf eval run`](/cli/evals#inf-eval-run) expects for `--models` and `--judge-model`. Use `inf models list --json` to dump every route ID available to your team.
## `inf models list`
Display every callable model visible to the active team, with provider, scope, capability flags, context window, and per-million-token pricing.
<Metadata />
```bash theme={"system"}
inf models list
```
**Alias:** `inf models ls`
### Options
| Flag | Required | Description | Default |
| ------------------- | -------- | --------------------------------------------------------------------------------------------------------- | ------------- |
| `--provider <name>` | No | Filter by provider name (case-insensitive exact match) — e.g. `openai`, `anthropic`, `google`, `cerebras` | All providers |
| `--scope <scope>` | No | Filter by scope: `platform` (inf-public catalog) or `byok` (your team's own provider keys) | Both |
| `--judge-only` | No | Show only models that can act as a judge in evals | Off |
### Output
In table mode (default), each row shows:
| Column | Description |
| ---------- | ------------------------------------------------------------------------- |
| `Model` | Canonical alias (e.g. `gpt-5.2`, `claude-sonnet-4-6`, `gemini-2.5-flash`) |
| `Provider` | Provider brand (OpenAI, Anthropic, Google, Cerebras, …) |
| `Scope` | `platform` (inf-public) or `byok` (team-owned route) |
| `Context` | Max context window, rounded to the nearest 1k tokens |
| `Struct.` | Whether the model supports structured outputs (`yes` / `no`) |
| `Tools` | Whether the model supports tool / function calling |
| `Reason.` | Whether the model has a reasoning mode |
| `Judge` | Whether the model is allow-listed as an eval judge |
| `$/1M In` | Input price per million tokens |
| `$/1M Out` | Output price per million tokens |
### Examples
<Metadata />
```bash theme={"system"}
# All callable models, table view
inf models list
# Only OpenAI routes
inf models list --provider openai
# Only models allow-listed as judges (for inf eval run --judge-model)
inf models list --judge-only
# Only BYOK routes
inf models list --scope byok
# Machine-readable — full routeId per row, good for piping into inf eval run
inf models list --json | jq -r '.[] | select(.judgeCapable == true) | .routeId'
```
### JSON mode
`inf models list --json` emits the full enriched record per model, including the `routeId` string that `inf eval run --models` and `--judge-model` expect:
```json theme={"system"}
[
{
"routeId": "openai:gpt-5.2-2025-12-11",
"canonicalAlias": "gpt-5.2",
"displayName": "GPT 5.2",
"providerName": "OpenAI",
"providerSlug": "openai",
"scope": "platform",
"maxContextSize": 128000,
"structuredOutputs": true,
"tools": true,
"reasoning": false,
"judgeCapable": true,
"costInputPerMToken": 2.5,
"costOutputPerMToken": 10
}
]
```
<Tip>
`inf models list --json | jq '.[] | select(.scope == "platform")'` is a quick way to prune your eval model set to just platform routes.
</Tip>
# Install CLI
Source: https://docs.inference.net/cli/overview
The Inference CLI (`inf`) drives Catalyst from the terminal. It serves two main paths:
1. **Instrument your codebase** so every LLM call your app makes is captured in Gateway — `inf instrument` hands the job to an AI coding agent (Claude Code, OpenCode, or Codex) and walks you through the diff.
2. **Operate the platform programmatically** — browse models, manage rubrics and eval runs, upload and materialize datasets, queue and monitor training runs, and inspect captured inferences, traces, and spans without opening the dashboard.
Sign up for an account at [Inference.net](https://inference.net/register) to get started.
<Warning>
The CLI is currently in beta. Please report any issues you find.
</Warning>
## Quick Start
<Steps>
<Step title="Install">
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli
```
</Step>
<Step title="Sign in">
<Metadata />
```bash theme={"system"}
inf auth login
```
</Step>
<Step title="Check status">
<Metadata />
```bash theme={"system"}
inf auth status
```
</Step>
</Steps>
Run `inf --help` at any time to see every command. Having trouble? [Send us a message](mailto:support@inference.net) or tag us at [x.com/@inference\_net](https://x.com/inference_net).
## Global Options
These flags work on every command.
| Flag | Description |
| -------------------- | --------------------------------------------------- |
| `--json` | Output as JSON (preserves full UUIDs for scripting) |
| `-v, --verbose` | Verbose debug output |
| `-p, --project <id>` | Override the active project for this invocation |
| `-V, --version` | Show CLI version |
| `-h, --help` | Show help |
<Tip>
Tables show UUIDs as readable 8-character prefixes. For scripting, always use `--json` — it preserves full UUIDs so you can round-trip values between commands (e.g. `inf dataset list --json | jq -r '.[0].id' | xargs inf dataset get`).
</Tip>
## Commands
| Command | Description |
| ----------------------------------- | --------------------------------------------------------------------- |
| [`inf instrument`](/cli/instrument) | Instrument your codebase for Catalyst observability using an AI agent |
| [`inf auth`](/cli/authentication) | Sign in, sign out, and check authentication status |
| [`inf project`](/cli/projects) | List, switch between, and inspect projects |
| [`inf models`](/cli/models) | Browse callable models with capabilities and pricing |
| [`inf eval`](/cli/evals) | Manage rubrics, launch eval runs, inspect results |
| [`inf dataset`](/cli/datasets) | Upload JSONL data, create eval/training datasets, download |
| [`inf training`](/cli/training) | Queue training runs, monitor progress, view logs, and poll status |
| [`inf inference`](/cli/inferences) | View inference requests and responses captured by Gateway |
| [`inf trace`](/cli/traces) | Browse trace trees, timelines, facets, and exports |
| [`inf span`](/cli/spans) | Search spans and inspect span IO, attributes, events, and links |
| [`inf dashboard`](/cli/dashboard) | Launch the interactive terminal dashboard |
## Explore the CLI
<CardGroup>
<Card title="Instrument your codebase" icon="wand-magic-sparkles" href="/cli/instrument">
Hand your project to an AI coding agent that wires up Catalyst for you.
</Card>
<Card title="Authentication" icon="key" href="/cli/authentication">
Browser and headless authentication, env vars, and config.
</Card>
<Card title="Projects" icon="book" href="/cli/projects">
Switch between projects and inspect the active project.
</Card>
<Card title="Models" icon="sparkles" href="/cli/models">
Browse callable models, capabilities, and pricing.
</Card>
<Card title="Evals" icon="flask" href="/cli/evals">
Manage rubrics, launch eval runs, and inspect results.
</Card>
<Card title="Datasets" icon="file-code" href="/cli/datasets">
Upload JSONL files and materialize eval or training datasets.
</Card>
<Card title="Training" icon="graduation-cap" href="/cli/training">
Queue training runs, monitor progress, view logs, and poll for completion.
</Card>
<Card title="Inferences" icon="code" href="/cli/inferences">
Inspect request and response payloads captured by Gateway.
</Card>
<Card title="Traces" icon="waypoints" href="/cli/traces">
Browse trace trees, timelines, facets, and exports from the terminal.
</Card>
<Card title="Spans" icon="git-branch" href="/cli/spans">
Search spans and inspect captured IO, attributes, events, and links.
</Card>
<Card title="TUI Dashboard" icon="computer" href="/cli/dashboard">
Interactive terminal UI for training runs, evals, datasets, and inferences.
</Card>
</CardGroup>
## Usage telemetry
The CLI reports anonymized usage events so we can prioritize the commands our
customers actually rely on.
* **What we capture:** the command name, CLI version, OS and CPU architecture,
the JavaScript runtime, and the flag names (never flag *values*) you used.
When you run the CLI inside a git repository we also capture the repo
`owner/name` and current branch from your `origin` remote — this is most
useful for `inf instrument`, where we want to understand which codebases
Catalyst gets wired into.
* **What we never capture:** argument values, environment variables, file
contents, API keys, or any data you pass to a command.
* **When we don't capture anything:** events are only sent once you are
authenticated. Commands run before `inf auth login` / `inf auth set-key`
emit no events.
* **How it works:** events are sent fire-and-forget over tRPC with a short
timeout, so telemetry never slows down or blocks your command. Failures are
silent.
# Projects
Source: https://docs.inference.net/cli/projects
List, switch, and inspect your Inference.net projects.
Projects organize your training runs, evaluations, datasets, and inferences. Most CLI commands operate on the active project, which is stored in `~/.inf/config.json` and settable per-invocation with the global `--project <id>` flag.
## `inf project list`
Display every project you have access to in the active team. The active project is flagged with a green dot (`●`).
<Metadata />
```bash theme={"system"}
inf project list
```
**Alias:** `inf project ls`
### Example
<Metadata />
```bash theme={"system"}
inf project list
```
## `inf project switch`
Set a different project as the active project. The CLI validates that the project exists and that you have access before saving.
<Metadata />
```bash theme={"system"}
inf project switch <project-id>
```
**Alias:** `inf project use`
### Arguments
| Argument | Required | Description |
| ------------ | -------- | --------------------------------- |
| `project-id` | Yes | The ID of the project to activate |
### Examples
<Metadata />
```bash theme={"system"}
# Switch to the staging project
inf project switch proj_789ghi012jkl
# Same, using the `use` alias
inf project use proj_789ghi012jkl
```
<Tip>
Temporarily override the active project for a single command without changing your stored config using the global `--project` flag:
```bash theme={"system"}
inf training list --project proj_789ghi012jkl
```
</Tip>
## `inf project current`
Show details about the currently active project — project ID, name, team ID, and creation date. If no project is active, the command errors and points you at `inf project switch <id>`.
<Metadata />
```bash theme={"system"}
inf project current
```
# Spans
Source: https://docs.inference.net/cli/spans
Search spans, inspect span payloads, and debug trace steps from the CLI.
Use `inf span` when you already know you need span-level detail instead of a whole trace tree. It lets you search spans across traces, focus on root or entrypoint spans, inspect captured input/output, and print OpenTelemetry attributes for a single step.
**Alias:** `inf spans`
## `inf span list`
Display spans in the active project.
<Metadata />
```bash theme={"system"}
inf span list
```
**Alias:** `inf span ls`
### Options
| Flag | Required | Description | Default |
| --------------------------------- | -------- | ------------------------------------------------------------------------------------ | ---------------- |
| `-l, --limit <n>` | No | Maximum number of spans | `25` |
| `--cursor <cursor>` | No | Pagination cursor from a previous response | - |
| `--sort <key>` | No | Sort by `start_time`, `duration_ns`, `cost_total`, or `total_tokens` | Server default |
| `--order <asc\|desc>` | No | Sort order | Server default |
| `--trace-id <id>` | No | Limit results to one trace | - |
| `--text <query>` | No | Search captured span input/output text | - |
| `--scope <all\|root\|entrypoint>` | No | Span scope filter | `all` |
| `--range <preset>` | No | Relative time range: `1h`, `6h`, `12h`, `1d`, `3d`, `7d`, `14d`, `30d`, `90d`, `all` | `1d` |
| `--from <iso>` | No | Start time. Must be used with `--to` | - |
| `--to <iso>` | No | End time. Must be used with `--from` | - |
| `--kind <kind>` | No | Observation kind. Repeat or comma-separate values like `LLM,TOOL,AGENT` | All kinds |
| `--provider <name>` | No | LLM provider filter | All providers |
| `--model <name>` | No | LLM model filter | All models |
| `--service <name>` | No | OpenTelemetry `service.name` filter | All services |
| `--environment <name>` | No | Deployment environment filter | All environments |
| `--user <id>` | No | User ID filter | All users |
| `--session <id>` | No | Session ID filter | All sessions |
| `--agent <name>` | No | Agent name filter | All agents |
| `--status <ok\|error>` | No | Span status filter | All statuses |
| `--filter <expr>` | No | Advanced typed field filter. Repeatable | - |
| `--metadata <expr>` | No | Span attribute filter. Repeatable | - |
| `--resource <expr>` | No | Resource attribute filter. Repeatable | - |
Use `--json` to preserve full trace IDs and span IDs for scripting.
### Examples
<Metadata />
```bash theme={"system"}
# Recent spans
inf span list
# Root spans for failed traces in the last day
inf span list --scope root --status error --range 1d
# Spans inside a specific trace
inf span list --trace-id 2f0d2b...
# Search captured IO text for a string
inf span list --text "rate limit"
# Find expensive model spans
inf span list --kind LLM --filter "cost_total>0.05" --sort cost_total --order desc
```
## `inf span get`
Open a single span by trace ID and span ID.
<Metadata />
```bash theme={"system"}
inf span get <trace-id> <span-id>
```
**Alias:** `inf span show`
### Arguments
| Argument | Required | Description |
| ---------- | -------- | ---------------------------- |
| `trace-id` | Yes | Trace ID containing the span |
| `span-id` | Yes | Span ID |
### Options
| Flag | Required | Description | Default |
| --------------------- | -------- | ---------------------------------------------------------- | --------- |
| `--view <view>` | No | `summary`, `io`, `attributes`, `events`, `links`, or `raw` | `summary` |
| `-o, --output <path>` | No | Write raw JSON to a file | - |
### Views
| View | Use it for |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `summary` | Span identity, timing, status, model/provider, token counts, cost, task/session metadata, and domain-specific summary fields |
| `io` | Captured input, output, input messages, output messages, and retrieval documents |
| `attributes` | Span and resource attributes across string, integer, and floating-point maps |
| `events` | Raw OpenTelemetry events JSON |
| `links` | Raw OpenTelemetry links JSON |
| `raw` | Full span JSON exactly as returned by the API |
### Examples
<Metadata />
```bash theme={"system"}
# Human-readable summary
inf span get 2f0d2b... 91f8c4...
# Inspect captured prompt/response payloads
inf span get 2f0d2b... 91f8c4... --view io
# Inspect OpenTelemetry attributes
inf span get 2f0d2b... 91f8c4... --view attributes
# Save raw span JSON
inf span get 2f0d2b... 91f8c4... --view raw -o span.json
```
## `inf span facets`
Inspect available span filter values and counts.
<Metadata />
```bash theme={"system"}
inf span facets
```
`inf span facets` accepts the same filters as [`inf span list`](#inf-span-list), plus:
| Flag | Required | Description | Default |
| ----------------------------- | -------- | --------------------------------------------------------- | -------------------- |
| `--facet <id>` | No | Facet ID. Repeat or comma-separate values | All supported facets |
| `--search <facet=query>` | No | Search within a facet's values. Repeatable | - |
| `--attribute-key <scope:key>` | No | Fetch values for one `span:` or `resource:` attribute key | - |
| `-l, --limit <n>` | No | Maximum options per facet | `50` |
### Examples
<Metadata />
```bash theme={"system"}
# Discover span facets for recent traffic
inf span facets --range 1d
# Search model facet values
inf span facets --facet llm_model_name --search llm_model_name=claude
# List values for a span attribute
inf span facets --attribute-key span:gen_ai.operation.name
```
## Advanced filter syntax
The span filter language matches [`inf trace`](/cli/traces) except trace aggregate fields such as `span_count` and `llm_span_count` are not valid for span queries.
| Expression | Meaning |
| --------------------------------------- | ----------------------------------------------------------------------------------------- |
| `duration_ms>500` | Numeric comparison. `duration` and `duration_ms` are converted to nanoseconds for the API |
| `total_tokens between:100,500` | Numeric range |
| `span_name~retrieval` | String contains |
| `llm_provider=anthropic` | String equality |
| `service_name in:api,worker` | String set membership |
| `agent_name not-empty` | Non-empty string field |
| `--metadata gen_ai.operation.name=chat` | Span attribute equality |
| `--resource service.version~2026` | Resource attribute contains |
## Common workflows
<Metadata />
```bash theme={"system"}
# Pick a trace, then inspect its LLM spans
TRACE_ID=$(inf trace list --json | jq -r '.traces[0].traceId')
inf span list --trace-id "$TRACE_ID" --kind LLM --json
# Open the first span from that trace
SPAN_ID=$(inf span list --trace-id "$TRACE_ID" --json | jq -r '.spans[0].spanId')
inf span get "$TRACE_ID" "$SPAN_ID" --view io
```
## Related commands
* [`inf trace`](/cli/traces) opens whole traces, trees, timelines, and trace exports.
* [`inf inference`](/cli/inferences) inspects gateway request/response records.
* [`inf dashboard`](/cli/dashboard) launches the interactive terminal dashboard.
# Traces
Source: https://docs.inference.net/cli/traces
Browse trace trees, inspect trace facets, and manage trace exports from the CLI.
Use `inf trace` to inspect multi-step LLM workflows captured by Catalyst Tracing. It mirrors the dashboard trace viewer for headless debugging: list trace summaries, open a trace tree, render a timeline, inspect captured conversation messages, discover filter facets, and queue/download trace exports.
**Alias:** `inf traces`
## `inf trace upload`
Upload a JSONL trace file into the active project. The CLI validates the format locally (OTLP or Langfuse export), uploads it in parts, waits for processing to finish, and prints the upload ID plus line counts. Same flow as the dashboard **Upload** button and [`inf dataset upload`](/cli/datasets#inf-dataset-upload).
<Metadata />
```bash theme={"system"}
inf trace upload <file>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ----------------------------- |
| `file` | Yes | Path to a `.jsonl` trace file |
### Options
| Flag | Required | Description | Default |
| ------------------- | -------- | -------------------------------------------------------------- | -------------------------- |
| `-n, --name <name>` | No | Upload name shown in Catalyst | Filename without extension |
| `--no-wait` | No | Return after the transfer completes without polling processing | Off |
Uploaded traces appear under the **Traces** tab once processing completes. Filter them with all-time range and the upload ID (`trace_import_id` in filters):
<Metadata />
```bash theme={"system"}
# Upload and wait for processing (default)
inf trace upload ./exports/langfuse-traces.jsonl
# Custom upload name
inf trace upload ./traces.jsonl --name prod-replay-2026-04-01
# Queue processing but do not wait
inf trace upload ./traces.jsonl --no-wait
# List traces from a completed upload
inf trace list --range all --filter "trace_import_id=<upload-id>"
```
## `inf trace list`
Display trace summaries in the active project.
<Metadata />
```bash theme={"system"}
inf trace list
```
**Alias:** `inf trace ls`
### Options
| Flag | Required | Description | Default |
| ---------------------- | -------- | ------------------------------------------------------------------------------------------------- | ---------------- |
| `-l, --limit <n>` | No | Maximum number of traces | `25` |
| `--cursor <cursor>` | No | Pagination cursor from a previous response | - |
| `--sort <key>` | No | Sort by `start_time`, `duration`, `total_cost`, `total_tokens`, `span_count`, or `llm_span_count` | Server default |
| `--order <asc\|desc>` | No | Sort order | Server default |
| `--range <preset>` | No | Relative time range: `1h`, `6h`, `12h`, `1d`, `3d`, `7d`, `14d`, `30d`, `90d`, `all` | `1d` |
| `--from <iso>` | No | Start time. Must be used with `--to` | - |
| `--to <iso>` | No | End time. Must be used with `--from` | - |
| `--kind <kind>` | No | Observation kind. Repeat or comma-separate values like `LLM,TOOL,AGENT` | All kinds |
| `--provider <name>` | No | LLM provider filter | All providers |
| `--model <name>` | No | LLM model filter | All models |
| `--service <name>` | No | OpenTelemetry `service.name` filter | All services |
| `--environment <name>` | No | Deployment environment filter | All environments |
| `--user <id>` | No | User ID filter | All users |
| `--session <id>` | No | Session ID filter | All sessions |
| `--agent <name>` | No | Agent name filter | All agents |
| `--status <ok\|error>` | No | Trace status filter | All statuses |
| `--filter <expr>` | No | Advanced typed field filter. Repeatable | - |
| `--metadata <expr>` | No | Span attribute filter. Repeatable | - |
| `--resource <expr>` | No | Resource attribute filter. Repeatable | - |
Use `--json` for the raw API payload with full trace IDs, pagination cursors, and aggregate counts.
### Examples
<Metadata />
```bash theme={"system"}
# Recent traces
inf trace list
# Failed LLM traces from the last hour
inf trace list --range 1h --kind LLM --status error
# Slow traces by duration, sorted longest first
inf trace list --filter "duration_ms>5000" --sort duration --order desc
# Filter by service and model
inf trace list --service api --model claude-sonnet-4-5
```
## `inf trace get`
Open a single trace and its spans.
<Metadata />
```bash theme={"system"}
inf trace get <trace-id>
```
**Alias:** `inf trace show`
### Arguments
| Argument | Required | Description |
| ---------- | -------- | ----------- |
| `trace-id` | Yes | Trace ID |
### Options
| Flag | Required | Description | Default |
| --------------------- | -------- | ---------------------------------------------------------- | --------- |
| `--view <view>` | No | `summary`, `tree`, `timeline`, `thread`, `spans`, or `raw` | `summary` |
| `--span <span-id>` | No | Highlight a selected span in the tree view | - |
| `--limit <n>` | No | Maximum spans to fetch for this trace | `1000` |
| `--cursor <cursor>` | No | Span pagination cursor | - |
| `-o, --output <path>` | No | Write raw JSON to a file | - |
The default summary view prints trace metadata and an ASCII span tree. Use `--view raw` or global `--json` when piping trace data into scripts.
### Examples
<Metadata />
```bash theme={"system"}
# Summary plus span tree
inf trace get 2f0d2b...
# Text waterfall ordered by span start time
inf trace get 2f0d2b... --view timeline
# Conversation-style view from captured input/output messages
inf trace get 2f0d2b... --view thread
# Persist raw trace + spans JSON for local analysis
inf trace get 2f0d2b... --view raw -o trace.json
```
## `inf trace facets`
Inspect available trace filter values and counts. This is useful when building a repeatable query and you do not know the exact model, service, environment, or attribute values in a project.
<Metadata />
```bash theme={"system"}
inf trace facets
```
### Options
`inf trace facets` accepts the same filter flags as [`inf trace list`](#inf-trace-list), plus:
| Flag | Required | Description | Default |
| ----------------------------- | -------- | --------------------------------------------------------- | -------------------- |
| `--facet <id>` | No | Facet ID. Repeat or comma-separate values | All supported facets |
| `--search <facet=query>` | No | Search within a facet's values. Repeatable | - |
| `--attribute-key <scope:key>` | No | Fetch values for one `span:` or `resource:` attribute key | - |
| `-l, --limit <n>` | No | Maximum options per facet | `50` |
### Examples
<Metadata />
```bash theme={"system"}
# Show all default trace facets
inf trace facets --range 7d
# Find model facet values containing "claude"
inf trace facets --facet llm_model_name --search llm_model_name=claude
# Explore values for an OpenTelemetry span attribute
inf trace facets --attribute-key span:gen_ai.operation.name
```
## Advanced filter syntax
`--filter`, `--metadata`, and `--resource` are repeatable. Quote expressions so your shell does not interpret operators.
| Expression | Meaning |
| --------------------------------------- | ----------------------------------------------------------------------------------------- |
| `duration_ms>500` | Numeric comparison. `duration` and `duration_ms` are converted to nanoseconds for the API |
| `total_tokens between:100,500` | Numeric range |
| `span_count>=3` | Trace aggregate filter |
| `llm_model_name~claude` | String contains |
| `service_name in:api,worker` | String set membership |
| `agent_name not-empty` | Non-empty string field |
| `--metadata gen_ai.operation.name=chat` | Span attribute equality |
| `--resource service.version~2026` | Resource attribute contains |
## `inf trace export`
Trace exports create downloadable JSONL files for offline analysis, support escalations, or long-running review workflows.
### `inf trace export create`
Queue an export job.
<Metadata />
```bash theme={"system"}
inf trace export create --range 7d --kind LLM
```
| Flag | Required | Description | Default |
| ---------------------------------- | -------- | ----------------------------------------- | --------------------- |
| `--from <iso>` / `--to <iso>` | No | Explicit export window | - |
| `--range <preset>` | No | Relative export window | `1d` |
| `--kind <kind>` | No | Observation kind filter | All kinds |
| `--trace-id <id>` | No | Trace ID. Repeat or comma-separate values | - |
| `--session <id>` | No | Session ID filter | - |
| `--user <id>` | No | User ID filter | - |
| `--agent <name>` | No | Agent name filter | - |
| `--model <name>` | No | LLM model filter | - |
| `--status-code <OK\|ERROR\|UNSET>` | No | Span status code filter | - |
| `--attribute <key=value>` | No | Attribute equality match. Repeatable | - |
| `--wait` | No | Poll until the export finishes | Off |
| `--download` | No | Download after the export is ready | Off |
| `-o, --output <path>` | No | Download path | Generated from job ID |
<Metadata />
```bash theme={"system"}
# Queue and print a job ID
inf trace export create --range 30d --kind LLM
# Queue, wait, and download in one command
inf trace export create --range 7d --status-code ERROR --wait --download -o failed-traces.jsonl
```
### `inf trace export list`
<Metadata />
```bash theme={"system"}
inf trace export list
```
### `inf trace export status`
<Metadata />
```bash theme={"system"}
inf trace export status <job-id>
```
### `inf trace export download`
<Metadata />
```bash theme={"system"}
inf trace export download <job-id> -o traces.jsonl
```
## Related commands
* [`inf span`](/cli/spans) lists and opens individual spans.
* [`inf inference`](/cli/inferences) inspects gateway request/response records.
* [`inf dashboard`](/cli/dashboard) launches the interactive terminal dashboard.
# Training
Source: https://docs.inference.net/cli/training
Queue training runs, discover recipes and base models, monitor progress, and surface failures.
Kick off and manage model training runs from the command line. Discover recipes and trainable base models, queue new runs (with override flags for the trickier `trainingConfig` knobs), cancel in-flight jobs, and zoom in on errors without scrolling through raw logs.
**Alias:** `inf train`
The full training loop is paste-able from the terminal:
<Metadata />
```bash theme={"system"}
# 1. Materialize training and eval datasets
inf dataset create -n my-train-split -t training --file ./train.jsonl
inf dataset create -n my-eval-split -t eval --file ./eval.jsonl
# → Datasets ds_trn_abc12 / ds_evl_def34 created.
# 2. Create (or reuse) a rubric the evals will score against
inf eval rubric create -n my-rubric -f ./rubric.md
# → Rubric rub_xyz56 / version rv_ver78 created.
# 3. Pick a recipe and base model
inf training recipes
inf training models
# 4. Queue the training run
inf training create \
--name distill-hardreasoning-qwen3.5-4b-v2 \
--recipe inf-public-training-recipe:qwen-3.5-4b-fft \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56 \
--sample-packing false \
--num-epochs 5 \
--task-id distill-hardreasoning-v2
# → Training job job_90ab12 queued.
# 5. Track progress until it finishes
inf training poll job_90ab12
```
## `inf training models`
Discover the base models you can fine-tune and (with `--judge`) the judge models that can score checkpoints.
<Metadata />
```bash theme={"system"}
inf training models
```
### Options
| Flag | Required | Description | Default |
| --------- | -------- | ------------------------------------------------------------------- | ------- |
| `--judge` | No | List judge models instead of base models (use with `--judge-model`) | Off |
The table shows each model's canonical alias, full name, and ID prefix. Pair with `--json` when scripting to preserve full IDs — those full IDs are what you pass to `--base-model` / `--judge-model` on `inf training create`.
### Examples
<Metadata />
```bash theme={"system"}
# List base models you can fine-tune
inf training models
# List judge models instead
inf training models --judge
# Dump full IDs for scripting
inf training models --json | jq -r '.[].id'
```
## `inf training recipes`
Recipes bundle a base model + judge model + GPU plan + full `trainingConfig`. `inf training recipes` lists everything visible to the active project (public recipes + the project's own recipes).
<Metadata />
```bash theme={"system"}
inf training recipes
```
### Options
| Flag | Required | Description | Default |
| -------------------- | -------- | -------------------------------------- | ------- |
| `--include-archived` | No | Include archived recipes | Off |
| `--public-only` | No | Show only public recipes | Off |
| `--project-only` | No | Show only the active project's recipes | Off |
<Warning>
Only super-admins can fork a public recipe into a project recipe. If you need to customize a recipe's `trainingConfig`, use the override flags on [`inf training create`](#inf-training-create) rather than trying to clone the recipe.
</Warning>
## `inf training recipes get`
Inspect a specific recipe, including its full `trainingConfig`. Useful for spotting knobs you may want to override at queue time.
<Metadata />
```bash theme={"system"}
inf training recipes get <id>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ----------- |
| `id` | Yes | Recipe ID |
### Examples
<Metadata />
```bash theme={"system"}
# List recipes visible to the active project
inf training recipes
# Inspect a specific recipe (including its trainingConfig)
inf training recipes get inf-public-training-recipe:qwen-3.5-4b-fft
# Only show project-owned recipes
inf training recipes --project-only
```
## `inf training create`
Queue a new training run. You can either specify a recipe (recommended — it pre-fills base model, judge, GPU plan, and `trainingConfig`) or pass individual flags. Override flags let you tweak specific `trainingConfig` fields without forking the recipe.
<Metadata />
```bash theme={"system"}
inf training create \
--name <name> \
--training-dataset <id> \
--eval-dataset <id> \
--rubric <id>
```
**Alias:** `inf training queue`
### Options
| Flag | Required | Description | Default |
| -------------------------- | -------- | ----------------------------------------------------------------------------------------------------- | -------------- |
| `-n, --name <name>` | Yes | Display name for the run | — |
| `--training-dataset <id>` | Yes | Training-type dataset ID (also accepts `--training-dataset-id`) | — |
| `--eval-dataset <id>` | Yes | Eval-type dataset ID (also accepts `--eval-dataset-id`) | — |
| `--rubric <id>` | Yes | Rubric ID (also accepts `--rubric-id`) | — |
| `--rubric-version-id <id>` | No | Pin to a specific rubric version | Latest version |
| `--recipe <id>` | No | Training recipe ID — pre-fills base model, judge, GPU plan, and `trainingConfig` (also `--recipe-id`) | — |
| `--base-model <id>` | No | Base model to fine-tune (overrides the recipe, also `--base-model-id`) | Recipe default |
| `--judge-model <id>` | No | Judge model for evals (overrides the recipe, also `--judge-model-id`) | Recipe default |
| `--num-nodes <n>` | No | Number of training nodes | Recipe default |
| `--gpus-per-node <n>` | No | GPUs per node (1–8) | Recipe default |
| `--task-id <id>` | No | Associate the run with an existing task | — |
| `--sample-packing <bool>` | No | Override `trainingConfig.sample_packing` (`true`/`false`) | Recipe default |
| `--num-epochs <n>` | No | Override `trainingConfig.num_epochs` | Recipe default |
| `--max-steps <n>` | No | Override `trainingConfig.max_steps` (use `-1` for "no cap") | Recipe default |
| `--learning-rate <n>` | No | Override `trainingConfig.learning_rate` | Recipe default |
| `--dry-run` | No | Print the exact tRPC payload that would be sent and exit — nothing is created | Off |
GPU hardware selection is managed server-side and is not configurable from the CLI. When `--rubric-version-id` is omitted, the CLI fetches the latest version of `--rubric` before queueing — pin a version if you need reproducibility across re-runs. Prints the new training job ID along with an `inf training poll <id>` follow-up command. The run status starts as `queued` while datasets are prepared, then moves to `running`.
### Escape hatches for recipe-pinned configs
The public recipes are opinionated: `sample_packing` is often `true`, and `num_epochs` / `max_steps` are tuned for the typical dataset size. When your dataset is small or shaped differently, those defaults can cause `torchrun` to crash (for example, with `args.max_steps must be set to a positive value if dataloader does not have a length, was -1` — which is what happens when sample packing collapses a tiny dataset into fewer batches than the FSDP shard count).
The `--sample-packing`, `--num-epochs`, `--max-steps`, and `--learning-rate` flags give you override knobs **without** needing to fork the recipe (only super-admins can). Pair them with `--dry-run` to sanity-check the resulting payload before burning a job slot.
### Examples
<Metadata />
```bash theme={"system"}
# Minimal queue with a recipe
inf training create \
--name distill-v1 \
--recipe inf-public-training-recipe:qwen-3.5-4b-fft \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56
# Override recipe defaults for a small dataset and verify the payload first
inf training create \
--name distill-hardreasoning-qwen3.5-4b-v2 \
--recipe inf-public-training-recipe:qwen-3.5-4b-fft \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56 \
--sample-packing false \
--num-epochs 5 \
--dry-run
# Use the `queue` alias
inf training queue \
--name distill-v1 \
--base-model model_abc \
--judge-model model_xyz \
--training-dataset ds_trn_abc12 \
--eval-dataset ds_evl_def34 \
--rubric rub_xyz56
```
## `inf training list`
Display a table of training runs in the active project.
<Metadata />
```bash theme={"system"}
inf training list
```
**Alias:** `inf training ls`
### Options
| Flag | Required | Description | Default |
| ----------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------- | ------------ |
| `-s, --status <status>` | No | Filter by status: `exporting_datasets`, `queued`, `running`, `cycling`, `completed`, `failed`, `cancelled`, or `timed_out` | All statuses |
| `-l, --limit <n>` | No | Maximum number of results | `20` |
| `--offset <n>` | No | Offset for pagination | `0` |
The table shows the run ID (8-char prefix), name, status (color-coded), base model, progress (`currentStep/totalSteps`), current loss, and creation date.
### Examples
<Metadata />
```bash theme={"system"}
# Show only running training jobs
inf training list --status running
# Get the first 50 training runs
inf training list --limit 50
# Page through results
inf training list --limit 20 --offset 40
```
## `inf training get`
View detailed information about a specific training run. Without `--error`, prints the full detail view. With `--error`, dumps only the fields that matter when a run crashed — ideal for triaging `failed` jobs from CI or a script.
<Metadata />
```bash theme={"system"}
inf training get <id>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options
| Flag | Required | Description | Default |
| --------- | -------- | -------------------------------------------------------------------------------------------------- | ------- |
| `--error` | No | Print only the status, status detail, and error message in a highlighted block (for `failed` runs) | Off |
### Output
The default detail view includes every field on the training job:
| Field | Description |
| ----------------------------------------- | ------------------------------------------------------------------------------- |
| `id`, `name` | Run identifier and display name |
| `status` | One of the status values listed under [`inf training list`](#inf-training-list) |
| `baseModelId` | Base model the run fine-tunes from |
| `adapter` | Adapter type (e.g. LoRA) |
| `currentStep` / `totalSteps` | Progress counters |
| `currentLoss` | Most recent training loss |
| `numNodes` | Number of nodes participating in the run |
| `provider` / `providerRunId` | Underlying training provider and their internal run ID |
| `evalDatasetName` / `rubricName` | Eval configuration (if the run is configured for mid-training evals) |
| `filteredDatasetName` | Training dataset name |
| `startedAt` / `completedAt` / `createdAt` | Lifecycle timestamps |
| `statusDetail` / `errorMessage` | Populated when the run fails or ends in a non-success state |
With `--error`, the output collapses to just `status`, `statusDetail`, and `errorMessage`.
### Examples
<Metadata />
```bash theme={"system"}
# Full detail view
inf training get job_abc123
# Just the error payload for a failed run
inf training get job_abc123 --error
```
## `inf training cancel`
Cancel a `queued` or `running` training job. Completed and already-cancelled jobs will reject the call.
<Metadata />
```bash theme={"system"}
inf training cancel <id>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options
| Flag | Required | Description | Default |
| ----------- | --------------------------- | ---------------------------- | ------- |
| `-y, --yes` | Yes in non-TTY environments | Skip the confirmation prompt | Off |
In an interactive terminal, the CLI asks for confirmation unless `-y` is passed. In non-TTY environments (CI, scripts) the command refuses to run without `-y`.
### Examples
<Metadata />
```bash theme={"system"}
# Cancel interactively
inf training cancel job_abc123
# Cancel non-interactively (required in CI)
inf training cancel job_abc123 --yes
```
## `inf training logs`
Stream or view log entries for a training run. Logs are color-coded by level, and each line is prefixed with the training phase it came from (`torchrun_init`, `training`, `inference_export`, …) so you can tell setup crashes apart from training crashes at a glance.
<Metadata />
```bash theme={"system"}
inf training logs <id>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options
| Flag | Required | Description | Default |
| ----------------- | -------- | ----------------------------------------------------------------------------- | ---------- |
| `-l, --limit <n>` | No | Maximum number of log entries | `50` |
| `--level <level>` | No | Filter by log level (e.g. `error`, `warn`, `info`) | All levels |
| `--phase <phase>` | No | Filter by training phase (substring match — e.g. `torchrun_init`, `training`) | No filter |
| `-f, --follow` | No | Continuously poll for new logs (like `tail -f`) | Off |
Log entries are timestamped and color-coded: errors in red, warnings in yellow, info in blue. In follow mode, the CLI polls for new logs every 3 seconds until you press `Ctrl+C`.
### Examples
<Metadata />
```bash theme={"system"}
# View the last 50 log entries
inf training logs job_abc123
# Stream logs in real time
inf training logs job_abc123 --follow
# Only show error logs
inf training logs job_abc123 --level error
# Only show setup-phase logs (everything before training starts)
inf training logs job_abc123 --phase torchrun_init
```
## `inf training poll`
Wait for a training run to complete, printing status updates as the status changes.
<Metadata />
```bash theme={"system"}
inf training poll <id>
```
### Arguments
| Argument | Required | Description |
| -------- | -------- | ------------------- |
| `id` | Yes | The training job ID |
### Options
| Flag | Required | Description | Default |
| -------------------------- | -------- | ------------------------ | ------- |
| `-i, --interval <seconds>` | No | Poll interval in seconds | `10` |
The CLI prints a status line each time the status changes, showing the status, progress, and current loss. It exits automatically when the run reaches `completed`, `failed`, `cancelled`, or `timed_out`. On a `failed` exit, the CLI points you at `inf training get <id> --error` for the full error payload.
### Examples
<Metadata />
```bash theme={"system"}
# Poll every 10 seconds (default)
inf training poll job_abc123
# Poll every 30 seconds
inf training poll job_abc123 --interval 30
```
# Analyze Your Traces
Source: https://docs.inference.net/get-started/analyze-traces
Once traces are flowing into Catalyst, the next step is figuring out what they're telling you. The dashboard gives you two places to look: the **Traces** tab for browsing every span across your account, and the **Agents** tab for a per-agent view with overview, sessions, traces, and analysis. Halo is our open-source agent-loop optimizer, hosted right inside the Agents dashboard, that reads your traces and writes up what to improve.
This guide assumes you've already [captured your first trace](/get-started/capture-first-trace). If not, start there.
## Two places to look
<CardGroup>
<Card title="Traces tab" icon="route">
Everything you've captured, across every service and agent. Filter by service, agent, time range, status, model, token usage, latency, errors, and custom span attributes. Open any trace to walk the tree.
</Card>
<Card title="Agents tab" icon="robot">
A per-agent workspace. Pick an agent and you get four sub-tabs: Overview, Sessions, Traces, and Analysis.
</Card>
</CardGroup>
## Inside the Agents tab
Click into any agent and you'll see four sub-tabs scoped to that agent.
| Sub-tab | What it shows |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Overview** | High-level metrics for the agent. Run counts, error rate, latency, token usage, and cost over time. |
| **Sessions** | One row per agent session. Click a session to see the full conversation, every tool call, and every span in order. This is where you go to understand "what happened in this one run." |
| **Traces** | The same trace data as the global Traces tab, pre-filtered to this agent. Filter further by status, time range, model, or any span attribute. |
| **Analysis** | The Halo workspace. Run Halo on the agent's traces, read past reports, and configure scheduled runs. |
<Tip>
The Agents dashboard groups runs by `agent.id` and `agent.name` from your spans. If your agent runs aren't grouping the way you expect, see [Agent identity](/integrations/traces/agent-identity).
</Tip>
## Run Halo on your traces
[Halo](https://github.com/context-labs/halo) (Hierarchical Agent Loop Optimization) is an open-source RLM-based engine for analyzing agent traces and finding things to improve. It reads OpenTelemetry-compatible spans, decomposes them to find systemic failure modes across many runs, and writes up concrete fixes with citations back to specific traces.
You can run Halo two ways:
* **Self-hosted** from the [open-source repo](https://github.com/context-labs/halo). `pip install halo-engine`, point it at a JSONL trace file, and go.
* **Hosted** inside Catalyst. The Agents tab's Analysis sub-tab runs the same engine against the traces you've already collected, with no extra setup, no trace export, and no separate pipeline.
The hosted version is where most teams start. Open an agent, open Analysis, and either run Halo on demand or put it on a schedule.
### Run Halo on demand
<Steps>
<Step title="Open the Analysis sub-tab">
From the Agents tab, pick the agent you want to investigate and click **Analysis**.
</Step>
<Step title="Pick a trace window">
Choose the time range Halo should review. Tighter windows give Halo more focused signal. A single problem agent over the last 24 hours beats a firehose of everything from the last month.
</Step>
<Step title="Write a prompt (or use the default)">
The default prompt is the same one we use internally most of the time:
<Metadata />
```text theme={"system"}
Analyze the traces in this window. Understand the agent's activity
and identify anomalies, errors, inefficiencies, and opportunities
to improve reliability, latency, cost, and tool usage. Highlight
the top recurring failures, notable tool-call patterns, wasted or
redundant work, slow or high-cost paths, and concrete fixes you'd
recommend. Cite specific trace IDs for every key finding.
```
You can also ask Halo anything specific: "Why is the refund agent timing out on Tuesdays?", "Which tool calls are returning empty results most often?", "Find redundant LLM calls in the planning loop."
</Step>
<Step title="Read the report">
Halo returns a ranked list of findings with evidence pulled directly from your traces. Each finding cites the trace IDs it came from, so you can click straight from a finding into the trace tree that produced it.
</Step>
<Step title="Apply the changes and re-run">
Update prompts, tools, or harness logic based on the findings. Capture a new window of traces, and run Halo again to confirm the issue is gone. This is the HALO loop: trace, analyze, fix, repeat.
</Step>
</Steps>
### Schedule recurring runs
For agents you ship to production, the higher-leverage move is putting Halo on a schedule so it reviews recent traces automatically.
<Steps>
<Step title="Open the schedule settings">
From the Analysis sub-tab, open the schedules sheet and create a new schedule.
</Step>
<Step title="Pick a cadence and window">
Hourly, daily, weekly, or monthly. The lookback window pre-fills to match the cadence (a daily schedule defaults to a 24-hour window) but you can override it. The runtime caps any single window at 30 days.
</Step>
<Step title="Set the prompt">
The schedule prompt seeds from the same default shown above. Customize it per schedule when you want a recurring run focused on a specific failure mode.
</Step>
<Step title="Review reports as they land">
Each scheduled run produces a new report in the Analysis history rail. Read the latest one, jump back to past reports to track regressions, and ignore runs where Halo finds nothing actionable.
</Step>
</Steps>
## Inspect traces from the CLI
If you'd rather stay in the terminal, the [Inference CLI](/cli/overview) reads the same trace data the dashboard does.
<Metadata />
```bash theme={"system"}
# Browse recent traces
inf trace list --range 1h
# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline
# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io
```
See [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) for the full reference.
## Next steps
<CardGroup>
<Card title="Halo on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO engine, methodology, and benchmarks. MIT licensed.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so Halo and the Agents dashboard group runs correctly.
</Card>
<Card title="Capture more of your stack" icon="puzzle-piece" href="/integrations/traces/overview">
Add tracing to your other providers, frameworks, and agent runtimes.
</Card>
<Card title="Wrap custom work" icon="pen-nib" href="/integrations/traces/manual-spans">
Add spans around retrieval, routing, subprocesses, and unsupported SDKs.
</Card>
</CardGroup>
# Capture Your First Trace
Source: https://docs.inference.net/get-started/capture-first-trace
Catalyst Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, and any custom spans you add. Drop the SDK into your app, point it at Catalyst, and traces start flowing.
This guide gets you from zero to a captured trace. The example uses OpenAI because it is the smallest end-to-end setup. The same flow works for Anthropic, LangChain, LangGraph, the Vercel AI SDK, Vercel Eve, OpenAI Agents, LiveKit Agents, PI AI, Pydantic AI, and the other [supported integrations](/integrations/traces/overview).
Either path below installs and wires up the Catalyst tracing SDK: [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) on npm for TypeScript, or [`inference-catalyst-tracing`](https://pypi.org/project/inference-catalyst-tracing/) on PyPI for Python.
To get started with Catalyst, create a free account at [inference.net](https://inference.net/register).
## Choose a setup path
Installing with AI is the quickest. Use the manual flow if you want to review each change yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to launch a coding agent like [Claude Code](https://code.claude.com/docs/en/overview) to install the tracing SDK, configure export, and wire up your existing LLM clients.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run tracing instrumentation in your project">
From your project root, run instrumentation in tracing mode.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project && inf instrument --mode tracing
```
The command guides you through the following workflow:
* Select a coding agent: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients and agent frameworks.
* Install the tracing SDK and configure export to Catalyst.
* Wire `setup()` into your app entrypoint so spans start before clients are constructed.
* Add stable service and agent identity so traces group cleanly in the dashboard.
* Review the generated changes before applying them.
<Tip>
Pick `both` instead of `tracing` if you also want to route requests through the Catalyst Gateway in the same pass.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would. Traces stream to Catalyst as your code executes.
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and filter by your service name to see the captured trace tree.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to wire it up yourself. The example below uses OpenAI. For other providers and frameworks, see the [Tracing integrations guide](/integrations/traces/overview).
<Steps>
<Step title="Install the SDK">
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing openai
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[openai]'
```
</CodeGroup>
</Step>
<Step title="Configure export">
Set the Catalyst traces endpoint and token before your app starts.
<Metadata />
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
# Get your API key from https://inference.net/dashboard/api-keys/
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="checkout-agent"
```
<Tip>
Use a stable `CATALYST_SERVICE_NAME` per deployed service. It makes traces easier to filter and compare across environments.
</Tip>
</Step>
<Step title="Initialize tracing early">
Call `setup()` before constructing clients from instrumented SDKs.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Reply with just the word hello." }],
max_tokens: 16,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply with just the word hello."}],
max_tokens=16,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
If the process is short-lived, always call `shutdown()` before exit so batched spans are flushed.
<Tip>
Longer-lived processes flush differently. A long-lived server (HTTP,
Slack bot, queue worker) memoizes `setup()` and calls `shutdown()` on
`SIGTERM`, not per request. A serverless or edge function instead
flushes per invocation with `tracing.provider.forceFlush()`. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle)
in the quickstart.
</Tip>
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and navigate to the Agents or Traces tab. You'll see an LLM span with input messages, output messages, model name, invocation parameters, finish reason, and token counts.
</Step>
</Steps>
### Group calls under an agent
A single LLM call is captured automatically. To get an `AGENT` row with stable `agent.id`, `agent.name`, and `session.id` for dashboard grouping, wrap your code in `agentSpan`. Use `manualSpan` inside it for non-LLM steps like tools, retrieval, and validation. The example below reuses the `tracing` and `client` from the step above.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, manualSpan, SpanKindValues } from "@inference/tracing";
await agentSpan(
tracing.tracer,
{
agentId: "hello-agent",
agentName: "Hello Agent",
sessionId: "session-001",
},
async (span) => {
const prompt = "Reply with just the word hello.";
span.setInput(prompt);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: prompt }],
max_tokens: 16,
});
const reply = response.choices[0]?.message.content ?? "";
await manualSpan(
tracing.tracer,
{ spanName: "validate_reply", spanKind: SpanKindValues.CHAIN, input: reply },
async (step) => step.setOutput({ ok: reply.toLowerCase().includes("hello") }),
);
span.setOutput(reply);
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span, manual_span, SpanKindValues
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="hello-agent",
agent_name="Hello Agent",
session_id="session-001",
) as span:
span.set_input(prompt)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
max_tokens=16,
)
reply = response.choices[0].message.content or ""
with manual_span(
tracing.tracer,
name="validate_reply",
span_kind=SpanKindValues.CHAIN,
input=reply,
) as step:
step.set_output({"ok": "hello" in reply.lower()})
span.set_output(reply)
tracing.shutdown()
```
</CodeGroup>
The trace now shows an `AGENT` row for "Hello Agent" with the LLM call and the `validate_reply` `CHAIN` row nested under it. For the full surface (framework integrations, multi-agent setups, identity propagation), see the [tracing integrations guide](/integrations/traces/overview).
<Note>
Need a different provider or framework? See [Tracing integrations](/integrations/traces/overview) for OpenAI, Anthropic, LangChain, LangGraph, the Vercel AI SDK, Vercel Eve, OpenAI Agents, LiveKit, Claude Agent SDK, PI AI, Pydantic AI, and more.
</Note>
</Tab>
</Tabs>
That's it. Spans are streaming to Catalyst and your trace is ready to inspect.
## What gets captured
| Span data | Examples |
| ------------------ | ---------------------------------------------------------- |
| Inputs and outputs | `input.value`, `output.value` |
| Messages | user, system, assistant, tool, and tool-result messages |
| Tool calls | tool names, IDs, JSON arguments, and tool results |
| Model metadata | model name, provider/system, invocation parameters |
| Usage | prompt, completion, total, and prompt-cache token counts |
| Agent structure | agent spans, framework spans, tool spans, graph/node spans |
| Errors | exception status and error details on failed spans |
## Next steps
<CardGroup>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Inspect trace trees in the dashboard and run Halo to find what to improve.
</Card>
<Card title="Add more integrations" icon="puzzle-piece" href="/integrations/traces/overview">
Instrument Anthropic, LangChain, LangGraph, Vercel AI SDK, Vercel Eve, PI AI, agent frameworks, and more.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so the Agents dashboard groups runs correctly.
</Card>
<Card title="Wrap custom work" icon="pen-nib" href="/integrations/traces/manual-spans">
Add spans around your own orchestration, retrieval, and routing code.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool spans, end to end.
</Card>
</CardGroup>
# Record Your First LLM Call
Source: https://docs.inference.net/get-started/record-first-call
In Catalyst, production LLM traffic serves as the backbone for model optimization. Catalyst Gateway records LLM traffic between your application and your current LLM provider, and stores it for later evaluation and training.
This guide shows two ways to capture your first LLM call through Gateway: use the [Inference CLI](/cli/overview) to automatically instrument your codebase, or wire it up manually.
To get started with Catalyst, create a free account at [inference.net](https://inference.net/register).
## Choose a setup path
Installing with AI is quickest. Use the manual flow if you want to review each change yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to automatically initialize a coding agent like [Claude Code](https://code.claude.com/docs/en/overview) to scan your codebase, update your LLM clients, and add required request metadata.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run instrumentation in your project">
Navigate to your project root and run instrumentation.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project && inf instrument
```
The command guides you through the following workflow:
* Select a coding agent to use: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain,etc
* Redirect base URLs to the gateway
* Add routing headers so requests are authenticated, forwarded, and traced
* Add task IDs so each call site is grouped automatically in the dashboard
* Review the generated changes before applying them
<Tip>
Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would to produce inference requests. Requests from your application are now routed through the gateway and will appear in the dashboard.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to see request details, traces, and analytics.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to review each change yourself. The example below uses OpenAI. For Anthropic, Cerebras, Groq, and other providers, see the [Integrations guide](/integrations/overview).
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenAI API key** — from your [OpenAI account](https://platform.openai.com/api-keys)
Set them as environment variables:
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point your SDK at `https://api.inference.net/v1` and use your Catalyst project API key as the SDK's `apiKey`. Your provider's API key goes in the `x-inference-provider-api-key` header so the gateway can forward it. The gateway adds roughly 10ms of latency and forwards your requests to the provider as-is.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
</Step>
<Step title="Send a request">
Run the snippet above from your application or terminal. Once the request completes, Catalyst will capture it automatically.
</Step>
<Step title="View your results">
Open the [Inference Catalyst dashboard](https://inference.net/dashboard) to inspect the request, traces, and metrics.
</Step>
</Steps>
<Note>
Need provider-specific manual instructions? See [Integrations Overview](/integrations/overview).
</Note>
</Tab>
</Tabs>
That's it. Every request now flows through Catalyst and gets captured automatically.
## What gets captured
Once traffic is flowing, Catalyst records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency (end-to-end and time to first token)
* Token counts (input and output)
* Error rates and status codes
* Model and provider
## Where to find your data
* **[Metrics Explorer](/platform/gateway/metrics-explorer)** - dashboards for cost, latency, errors, and usage across all your LLM calls
* **[Inference Viewer](/platform/gateway/inference-viewer)** - browse and filter individual requests and responses
## Next steps
<CardGroup>
<Card title="Connect more providers" icon="plug" href="/integrations/overview">
Set up Anthropic, Cerebras, Groq, and other providers.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have data? Upload a JSONL file to start evaluating or training.
</Card>
</CardGroup>
# Run Your First Eval
Source: https://docs.inference.net/get-started/run-first-eval
An eval measures which model is better for your task, and by how much. You define a rubric that describes what "good" looks like, run your data through candidate models, and let an LLM judge score the outputs. This is how you know whether a smaller, cheaper model can replace the one you're using today.
This guide uses the **Customer Support Chatbot** demo project, which comes pre-loaded with a dataset and rubric so you can run an eval immediately — no data required. Once you've seen how it works, you can apply the same process to your own data.
## Start the demo project
If you haven't already, create the demo project:
1. From the dashboard, navigate to the **Learn** page (or the **Create a Project** page).
2. Find **Customer Support Chatbot** and click **Start with demo project**.
<Frame>
<img alt="Customer Support Chatbot demo project card with Start with demo project button" />
</Frame>
This creates a new project in your account pre-loaded with everything you need:
| Artifact | Name | Purpose |
| ---------------- | ------------------------ | ---------------------------------------------------------------------------------------------- |
| Eval dataset | `customer-support-eval` | Sample customer support conversations to evaluate against |
| Training dataset | `customer-support-train` | Used later for [training a model](/get-started/train-and-deploy) |
| Rubric | Customer support rubric | Defines what a good customer support response looks like — tone, format, and accuracy criteria |
## Run an eval
<Steps>
<Step title="Navigate to Evals">
Open your **Customer Support Chatbot** project and go to the **Evals** tab. Click **New Eval**.
</Step>
<Step title="Select the rubric and dataset">
The demo project's rubric and the `customer-support-eval` dataset are already available in your project. Select them.
<Frame>
<img alt="Eval setup form with rubric and dataset selected" />
</Frame>
</Step>
<Step title="Pick models to compare">
Choose two or more models to evaluate. You can pick any combination from the model catalog — OpenAI, Anthropic, open-source, or any other available model. For a quick comparison, try picking a large model and a smaller one to see how they stack up.
</Step>
<Step title="Run the eval">
Click **Run**. Each sample from the dataset is sent to each model, and an LLM judge scores every response against the rubric.
</Step>
<Step title="Compare the results">
When the eval completes, the comparison view shows side-by-side scores across all models and samples. Look at overall scores to see which model wins, and drill into individual samples to understand where models differ.
<Frame>
<img alt="Eval results comparison view showing scores across models" />
</Frame>
</Step>
</Steps>
## What you just learned
* **Rubrics** define your quality bar in plain English — the LLM judge uses them to score outputs
* **Evals** run your data through multiple models and score the results, giving you a data-driven comparison
* You can re-run evals anytime — after changing the rubric, adding models, or later after [training a custom model](/get-started/train-and-deploy) to see how it compares
## Next steps
<CardGroup>
<Card title="Train a custom model" icon="brain" href="/get-started/train-and-deploy">
Use the same demo project to train and deploy a model.
</Card>
<Card title="Write a rubric" icon="pen" href="/platform/eval/write-a-rubric">
Learn how to write your own rubrics for your specific use case.
</Card>
<Card title="Read the results" icon="chart-column" href="/platform/eval/read-the-results">
Deep dive on interpreting the comparison view.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/overview">
Create datasets from your own data — captured traffic or uploaded files.
</Card>
</CardGroup>
# Signals
Source: https://docs.inference.net/get-started/signals
Define plain-language signals that automatically classify your agent's traffic (sentiment, jailbreak attempts, NSFW content, task outcomes, anything you can describe) and alert on them.
Once traces are flowing, signals turn them into structured labels. A signal is a plain-language classifier you define once and Catalyst runs continuously against one of your agents. Describe what you want to detect ("is this NSFW?", "did the user get frustrated?", "was the task completed?") and every matching trace gets labeled automatically by an LLM judge, with the results queryable right alongside your other agent metrics.
Signals are how you evaluate the things default metrics miss. For a user-facing agent they tell you how people are actually interacting with it (frustration, sentiment, jailbreak attempts). For a non-user-facing agent they tell you how the agent itself is behaving (did it refuse, did it complete the task, did it stay on policy).
<Tip>
Want the full walkthrough with a worked example instead of the reference? See the guide [Measure your agent's quality with Signals](/guides/measure-agent-quality).
</Tip>
## Before you start
Signals run on captured traffic, so you need traces flowing first. This guide assumes you've already [captured your first trace](/get-started/capture-first-trace). You also need:
* **An agent with traces installed.** Signals are always scoped to a single agent, so there is nothing to label until that agent is emitting traces.
* **A stable, consistent `agentId`.** Signals are per agent, so your traces need a consistent agent identity to group by. Set the `agentId` once and keep it the same across runs. If you instrumented with the CLI this is usually already set. See [Set agent identity](/integrations/traces/agent-identity).
* **A `sessionId`, if you want session-scoped signals.** Session signals classify a whole conversation, which requires that your traces carry a `sessionId` (your conversation or chat ID) so Catalyst can assemble the conversation. See [Choose a scope](#choose-a-scope).
## Where to find signals
You can get to signals two ways in the dashboard:
* The **Signals** tab, which lists every signal and is the main table view across your agents.
* The **Agents** tab, where you pick an agent and open its **Signals** view to see and create signals scoped to just that agent.
## How signals work
A signal evaluates traffic for one of your agents. When you activate it, Catalyst samples incoming traffic for that agent, sends each sampled target's input and output to an LLM judge with the prompt you wrote, and writes the label back. You can then filter, chart, and break down your traces by that label.
<Steps>
<Step title="You define the signal">
Pick a scope and classifier type, write a prompt describing what to look for, and set a sample rate.
</Step>
<Step title="Catalyst samples matching traffic">
For an active signal, a deterministic share of the agent's traffic (the sample rate) is selected for classification. Sampling is deterministic, so the same target always resolves the same way.
</Step>
<Step title="A judge labels each target">
The judge reads the input and output of each sampled span, trace, or session and returns a label that conforms to your classifier type.
</Step>
<Step title="Labels land on your traces">
Results are stored against each labeled target and surfaced in the dashboard, where you can filter by label value and watch the label distribution over time.
</Step>
</Steps>
## Choose a scope
A signal's **scope** is the unit it labels. You pick it when you create the signal, and it's fixed for the life of the signal (everything else is editable). Scope determines how much context the judge sees on each call.
<CardGroup>
<Card title="Span" icon="dot">
A single model call. The narrowest scope. Useful for narrow, call-level checks, but often too granular if you care about the interaction as a whole.
</Card>
<Card title="Trace" icon="route">
One turn, or request. The judge sees the whole turn rather than a single call. A good fit for request-level outcomes.
</Card>
<Card title="Session" icon="comments">
The full conversation. Usually the most useful scope for understanding a user, since the judge sees the entire back-and-forth. Requires a `sessionId` on your traces so the conversation can be assembled.
</Card>
</CardGroup>
<Note>
Session scope only works if your traces carry a `sessionId`. If you haven't set one, add it before creating a session-scoped signal. See [Set agent identity](/integrations/traces/agent-identity).
</Note>
## Two classifier types
When you create a signal you choose how it labels each target.
<CardGroup>
<Card title="Binary (yes / no)" icon="toggle-on">
A true/false classifier. Use it for "is this X or not" questions: NSFW content, jailbreak attempts, refusals, or any flag you want to filter on. No labels to configure, just a prompt.
</Card>
<Card title="String (enumerated labels)" icon="tags">
A classifier that returns one of a fixed set of labels you define. Use it when there's more than two outcomes: sentiment (positive / neutral / negative), task outcome (completed / partial / failed / abandoned), and so on. Define between 2 and 10 labels.
</Card>
</CardGroup>
## Create a signal
Create a signal from the **Signals** tab, or from a specific agent's **Signals** view under the **Agents** tab. Either way the signal is scoped to one agent.
<Steps>
<Step title="Open the signal editor for your agent">
From the Signals tab or an agent's Signals view, pick the agent whose traffic you want to label and create a new signal.
</Step>
<Step title="Name the signal">
Give it a short, descriptive name like `NSFW` or `User frustration`. The name is how the label shows up everywhere else in the dashboard.
</Step>
<Step title="Choose a scope">
Pick **Span**, **Trace**, or **Session** (see [Choose a scope](#choose-a-scope)). This is the one setting you can't change later, so pick the unit you actually want to reason about. Session is usually the most useful for understanding a user.
</Step>
<Step title="Choose a classifier type">
Pick **Binary (yes / no)** or **String (enumerated labels)**. For a string classifier, add the 2 to 10 labels the judge is allowed to return.
</Step>
<Step title="Write the prompt">
Describe what the signal should classify. This is the instruction the judge follows on every target, so be specific about what counts as each outcome. For a binary signal, describe what makes a target a "yes." For a string signal, describe when to pick each label.
</Step>
<Step title="Set the sample rate">
The sample rate is the share of matching traffic that gets classified, and lower rates cost less. 100% runs on every target, which is often unnecessary at high volume, so to spend fewer credits pick a lower rate like 25% or 50% and the signal only runs on that share of incoming traffic. Start lower on high-volume agents and raise it once you trust the labels. Common presets are 10%, 25%, 50%, and 100%.
</Step>
<Step title="Save as a draft or activate">
**Save draft** keeps the signal unpublished so you can keep tuning it. **Activate** publishes it and starts live classification on new traffic.
</Step>
</Steps>
<Tip>
Don't want to start from scratch? Use **Start from a template** to prefill the classifier type, prompt, and labels for a common signal, then edit from there. See [Templates](#templates) below.
</Tip>
## Test before you activate
Before you commit a signal to live traffic, run it against recent traffic to preview how it labels it. A test run classifies a small sample synchronously and shows you the label distribution and per-target results, and **nothing is saved**. It's a preview only and doesn't affect your signal or store any labels.
<Steps>
<Step title="Open the tester">
From the signal, choose **Test it**.
</Step>
<Step title="Pick a sample size and time range">
Choose how many recent targets to classify (1 to 100) and the window to pull them from.
</Step>
<Step title="Read the preview">
You'll see the label distribution across the sample plus a per-target breakdown, including which targets got flagged. If the labels don't match your intent, adjust the prompt or labels and test again.
</Step>
</Steps>
<Note>
Test results are not persisted. They exist only to help you tune the prompt before activating.
</Note>
## Activate and run live
Activating a signal starts live classification: as new traffic arrives for the agent, the configured share of it gets sampled and labeled automatically. You don't have to do anything else, and labels accumulate as traffic flows.
A signal is always in one of three states:
| State | What it means |
| ------------ | ------------------------------------------------------------------ |
| **Draft** | Saved but not running. Nothing is being classified. |
| **Active** | Live. New matching traffic is sampled and labeled. |
| **Disabled** | Paused. Live classification has stopped, but past labels are kept. |
You can disable an active signal at any time to stop classification without losing the labels you've already collected, and re-enable it later.
## Backfill historical data
Live classification only labels traffic that arrives after you activate. To label traffic you already captured, run a **manual run** (backfill) at any time. Unlike a test, a manual run **saves** its labels: they're stored against your traces and tied to the run, exactly like live labels.
<Steps>
<Step title="Open the manual run dialog">
From the signal, choose **Manual run / Backfill**.
</Step>
<Step title="Pick a time range and sample rate">
Choose the historical window to apply the signal to, and the share of matching traffic in that window to classify.
</Step>
<Step title="Start the run">
The run executes in the background, classifying past traffic across the window. Results land in the same place as live labels as the run progresses.
</Step>
</Steps>
<Tip>
Backfill a representative window first to sanity-check the labels at scale before running it over a long history. A manual run classifies real traffic and counts toward usage.
</Tip>
## Read the results
There are a few places to read what a signal found:
* **The Signals tab** is the main table across your agents, with each signal's current state and headline numbers at a glance.
* **The agent's Overview** under the Agents tab has per-signal graphs: trends and volume over time, plus a range of metrics for each signal so you can see how a label is moving.
* **The signal detail view** has a table of every classified target (the spans, traces, or sessions the signal labeled). Click into any row to open the underlying trace, span, or session and read the actual conversation that produced the label.
Labeled targets render their label as a colored chip. For a binary signal, "yes" and "no" get distinct colors; for a string signal, each label gets its own color. From there you can:
* **Filter by label value** to pull up just the targets a signal flagged (for example, every session labeled "yes" by a jailbreak signal).
* **Watch the distribution over time** to see how a label trends across hours or days.
* **Jump straight to the underlying trace, span, or session** to see the full context and conversation behind any labeled target.
## Alert on a signal
Once a signal is running, set up alerts so you hear about changes without watching the dashboard. Alerts are configured per signal, and you can get notified through the Slack integration or by email.
An alert fires when a metric crosses a condition over a window. You choose:
* **The metric.** What to watch, depending on the classifier type:
* **Label volume** (any signal): how many labels came in.
* **True rate** (binary signals): the share of labels that are "yes."
* **Value count** (string signals): how many labels landed on one specific value.
* **Value share** (string signals): that value's share of all labels.
* **The comparison.** Either a **percentage change** versus the prior equal-length window (for example, "this value's count is up 10% in the last 24 hours") or an **absolute threshold** (for example, "true rate is below 80%").
* **The window** the comparison runs over, from 5 minutes up to 48 hours.
* **A minimum label count** so quiet periods don't trip the alert on statistically meaningless deltas.
* **A cooldown** so a flapping condition doesn't bombard you. After a firing, the next one is suppressed until the cooldown elapses.
You can **backtest** an alert against recent history to see when it would have fired before you turn it on, and pause or re-enable any alert at any time.
<Tip>
Start with a wide window, a sensible minimum label count, and a cooldown. Tighten the threshold once you've seen how the metric actually moves in the backtest.
</Tip>
## Versioning and editing
Signals are versioned. Editing a signal (changing its prompt, labels, classifier type, or sample rate) creates a new version rather than mutating the old one, and the dashboard shows each version after every edit. The current version is the one powering live classification, and labels record which version produced them, so you can change a signal's definition without losing the history of what earlier versions decided.
**Scope is the exception: it's fixed once a signal is created.** Everything else is editable, but to label a different unit you create a new signal.
When you no longer need a signal, archive it. Archiving stops it and removes it from your active list while preserving the labels it produced.
## Templates
To get started quickly, create a signal **from a template** and edit from there. Built-in templates include:
| Template | Type | What it flags |
| ---------------------- | ------ | ----------------------------------------------------------------------------------------------------------- |
| **NSFW** | Binary | Spans whose content is sexually explicit, graphic, or otherwise not safe for work. |
| **Jailbreak attempt** | Binary | Spans where the user tries to bypass the model's safety guardrails or system instructions. |
| **Laziness / refusal** | Binary | Spans where the assistant refuses, stalls, or gives a low-effort non-answer instead of completing the task. |
| **User frustration** | Binary | Spans where the user expresses frustration, annoyance, or dissatisfaction with the assistant. |
| **Sentiment** | String | The overall sentiment the user expresses: positive, neutral, or negative. |
| **Task outcome** | String | Whether the task the user asked for was completed: completed, partial, failed, or abandoned. |
Templates are just a starting point. You can write any prompt you want and build a signal from scratch, with either classifier type.
## Manage signals from the CLI and MCP
The dashboard isn't the only way in. Everything above is also available through the [Inference CLI](/cli/overview) and the [MCP server](/integrations/mcp-server), so you can create, run, and read signals from your terminal or straight from an AI coding assistant.
* **CLI.** The `inf signals` command group lists, creates, edits, activates, disables, and archives signals, tests them, kicks off manual runs, and reads labels and distributions.
* **MCP.** The Inference MCP server exposes the same operations as tools (creating signals, activating and disabling them, running backfills, configuring alerts, and querying labels and distributions), so an assistant can set up and inspect signals on your behalf.
## Feed signals into Halo
Signals pair naturally with [Halo](/get-started/analyze-traces), our agent-loop optimizer. The targets a signal flags are exactly the traffic worth digging into:
* **Improve a behavior.** Point Halo at the spans, traces, or sessions a signal flagged and ask how to fix what they have in common (the refusals, the frustrated sessions, the failed tasks).
* **Decide what to measure.** Talk to Halo about your traces to surface which signals would be most valuable to add in the first place.
## Next steps
<CardGroup>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Run Halo on your traces to find systemic failure modes and concrete fixes.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so signals attach to the right agent and group cleanly.
</Card>
<Card title="Capture more of your stack" icon="puzzle-piece" href="/integrations/traces/overview">
Add tracing to more providers, frameworks, and agent runtimes so signals have more to label.
</Card>
<Card title="Wrap custom work" icon="pen-nib" href="/integrations/traces/manual-spans">
Add spans around retrieval, routing, and subprocesses so signals can classify them too.
</Card>
</CardGroup>
# Train and Deploy a Custom Model
Source: https://docs.inference.net/get-started/train-and-deploy
Train a task-specific model using demo data, deploy it, and see how it performs.
When off-the-shelf models aren't good enough for a specific task, fine-tune one that is. A task-specific model is typically smaller, faster, and cheaper to run than the general-purpose model it replaces, while being more accurate for your workload.
This guide uses the **Customer Support Chatbot** demo project to walk through the full loop: launching a training job, monitoring progress, deploying the result, and evaluating the trained model. No data of your own is needed — the demo project comes with everything pre-loaded.
## Start the demo project
If you already created the demo project during [Run Your First Eval](/get-started/run-first-eval), you're all set — open it from the dashboard and skip to the next section.
Otherwise:
1. From the dashboard, navigate to the **Learn** page (or the **Create a Project** page).
2. Find **Customer Support Chatbot** and click **Start with demo project**.
The project comes pre-loaded with three artifacts that map directly to what a training job requires:
| Artifact | Name | Role in training |
| ---------------- | ------------------------ | -------------------------------------------------------------------------------------------- |
| Training dataset | `customer-support-train` | The data the model learns from |
| Eval dataset | `customer-support-eval` | A held-out set used to measure learning progress — must have zero overlap with training data |
| Rubric | Customer support rubric | Defines the quality criteria the LLM judge scores against during and after training |
## Train a model
<Steps>
<Step title="Create a new training job">
Open the **Training** tab in your project and click **New Training Job**.
Select the three inputs from your demo project:
1. **Training dataset** — `customer-support-train`
2. **Eval dataset** — `customer-support-eval`
3. **Rubric** — the customer support rubric
<Frame>
<img alt="Training job setup form with demo datasets and rubric selected" />
</Frame>
</Step>
<Step title="Choose a recipe">
Next, pick a [recipe](/platform/train/choose-a-recipe) — a pre-configured training setup with a base model, optimized parameters, and compute config. For this demo, the **smallest recipe** works well and will finish the fastest.
</Step>
<Step title="Launch training">
Review your selections and click **Start Training**. The job will begin shortly.
</Step>
<Step title="Monitor training progress">
During training, the platform periodically runs the `customer-support-eval` dataset through your model-in-progress and scores the outputs using the rubric. You can watch these mid-training eval scores update in real time.
* **Scores improving** — training is on track and continues
* **Scores degrading** — training stops early to prevent overfitting
<Frame>
<img alt="Mid-training eval chart showing scores improving over time" />
</Frame>
See [Monitor a Training Run](/platform/train/mid-training-evals) for more on reading these charts.
</Step>
</Steps>
## Deploy the trained model
<Steps>
<Step title="Deploy">
When training completes, your model is automatically registered and ready to deploy. Navigate to **Deployments**, name your deployment, and click **Deploy**. The GPU spins up in a few minutes depending on model size.
</Step>
<Step title="Call your model">
Once deployed, you call it the same way you'd call any model through the Inference API — same base URL, same headers — just swap the `model` parameter to your trained model's identifier.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "your-org/your-trained-model",
messages: [
{
role: "user",
content: "I was charged twice for my subscription, can you help?",
},
],
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="your-org/your-trained-model",
messages=[
{
"role": "user",
"content": "I was charged twice for my subscription, can you help?",
}
],
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "your-org/your-trained-model",
"messages": [
{
"role": "user",
"content": "I was charged twice for my subscription, can you help?"
}
]
}'
```
</CodeGroup>
Replace `your-org/your-trained-model` with the model identifier shown on your deployment page. See [Call Your Deployment](/platform/deploy/call-your-deployment) for the full setup guide.
</Step>
</Steps>
## Evaluate the trained model
Once a model has finished training, you can run evals against it alongside any other model — no deployment required. This lets you iterate on your data and retrain for better results before you deploy and take it to production.
<Steps>
<Step title="Run an eval with your trained model">
Go to the **Evals** tab and create a new eval. Select the `customer-support-eval` dataset and the project rubric — the same ones used during training. This time, add your **trained model** alongside one or more off-the-shelf models.
</Step>
<Step title="Compare the results">
The comparison view shows how your trained model scores against the others on the same rubric. Since the model was trained specifically on this task, you should see it perform competitively — often matching or beating larger general-purpose models on quality, while being smaller and cheaper to run. If the results aren't where you want them, refine your training data and retrain before deploying.
<Frame>
<img alt="Eval comparison view showing trained model scores vs off-the-shelf models" />
</Frame>
</Step>
</Steps>
## What you just learned
* **Training** teaches a model your specific task using your data, with the rubric guiding quality during the process
* **Mid-training evals** give you visibility into whether training is working before it finishes
* **Deployment** puts the trained model behind the same API you already use — no code changes beyond swapping the model name
* **Post-training evals** let you validate that the trained model actually outperforms alternatives on your criteria
## Next steps
<CardGroup>
<Card title="Choose a recipe" icon="book-open" href="/platform/train/choose-a-recipe">
Understand recipe tiers and how to pick the right one for your task.
</Card>
<Card title="Launch a training run" icon="rocket" href="/platform/train/launch-a-run">
The full training flow with your own data — cost and duration estimates included.
</Card>
<Card title="Call your deployment" icon="code" href="/platform/deploy/call-your-deployment">
Full production setup for calling your deployed model.
</Card>
<Card title="Monitor with Gateway" icon="chart-line" href="/platform/gateway/overview">
Track your deployed model's cost, latency, and quality over time.
</Card>
</CardGroup>
# HALO Desktop: Optimize an Agent End to End on Your Machine
Source: https://docs.inference.net/guides/halo-desktop-app
Run the full trace, analyze, fix loop entirely on your own machine with the free HALO desktop app. Load demo traces from Hugging Face, explore them in a local timeline, run HALO with your own model provider, and hand the report to Claude Code, Cursor, or Codex. No account, no cloud.
The [Try HALO with a Demo Repo](/guides/try-halo-demo-repo) guide runs this same loop against the hosted [Catalyst](/integrations/traces/quickstart) dashboard. This one does the whole thing **on your machine** with the free, open-source [HALO desktop app](https://github.com/context-labs/halo). Traces live in a local SQLite database, HALO runs against a model provider you configure, and nothing leaves your laptop. No Inference account required.
You will install the app, load a bundled demo dataset of about 1,000 real agent runs with one click, explore the traces in the local viewer, run HALO over them, and open the resulting report directly in a coding agent to apply the fixes. The traces come from a search agent with deliberate, documented flaws, so HALO has something real to find.
<Note>
**Desktop or hosted?** The desktop app is the best way to try HALO end to end for free and keep your traces fully local. For continuous production observability across a team (always-on ingest, multiple users, scheduled HALO runs, Signals, and alerts), the [hosted version](/guides/try-halo-demo-repo) is the better fit. You can start local and move to hosted later without changing your instrumentation; it is the same Catalyst tracing under the hood. See [Self-hosting and production](#self-hosting-and-production).
</Note>
## Before you start
You need:
* The **HALO desktop app** (installed in [Step 1](#step-1-install-the-desktop-app)). HALO v1 ships for **Apple Silicon Macs** and **Ubuntu/Debian Linux (x64)**. Intel Macs, other Linux distributions, and Windows are not supported yet.
* A **model provider key** for running HALO. The desktop app supports OpenAI-compatible and Anthropic-compatible endpoints (OpenAI, Anthropic, OpenRouter, [Inference](https://inference.net), or your own endpoint), and you bring your own key. The app stores it locally and it never leaves your machine.
* A **coding agent** for the last step: Claude Code, Cursor, or Codex. The report opens in any of them with one click, and there is a copy-paste path for everything else.
<Note>
**What "no account" means.** The desktop app, the local trace timeline, and the HALO engine all run with no Inference account and no sign-in. The one thing that always needs a key is a model provider, because HALO calls an LLM to do the analysis. That key can be any provider you already have.
</Note>
## Step 1: Install the desktop app
Install with the one-line installer:
```bash theme={"system"}
curl -fsSL https://inference.net/halo/install.sh | sh
```
The installer detects your platform and downloads the matching release: a signed, notarized DMG on Apple Silicon macOS, or a `-Setup.tar.gz` on Ubuntu/Debian x64. You can also grab a build directly from the [releases page](https://github.com/context-labs/halo/releases).
<Note>
HALO v1 supports **Apple Silicon Macs** and **Ubuntu/Debian Linux (x64)** only. The installer stops with a clear message on Intel Macs, other Linux distributions, and Windows. On a clean Ubuntu/Debian box you may also need a few system libraries first: `sudo apt-get install -y libgtk-3-0 libwebkit2gtk-4.1-0 libayatana-appindicator3-1`.
</Note>
## Step 2: Load the demo traces
Launch the app. After a short welcome, onboarding lands on the **Import Demo Traces** screen. This is the front door for getting data into HALO: import history from **Langfuse** or **Phoenix**, upload a **JSONL** trace export, connect a **local agent**, or load a demo dataset.
<Frame>
<img alt="The Import Agent Traces onboarding screen with cards for Import from Langfuse, Import from Phoenix, Import JSONL File, Connect Local Agent, Load Demo Traces, and Read Documentation" />
</Frame>
For this guide, click **Load Demo Traces**. The dialog explains exactly what it does: HALO downloads an allowlisted JSONL export from the public [`inference-net/SearchAgentDemoTraces`](https://huggingface.co/datasets/inference-net/SearchAgentDemoTraces) dataset on Hugging Face, caches it beside your local app data, and queues the normal JSONL importer. No arbitrary remote URLs are accepted.
<Frame>
<img alt="The Load Demo Traces dialog showing the SearchAgentDemoTraces Hugging Face dataset, with Source, Cache, and Import cards and a Load Traces button" />
</Frame>
Click **Load Traces**. The dataset is about 1,000 real runs of a pre-instrumented web-search agent (more on it in [Step 5](#step-5-open-a-trace)), and the import usually finishes in under a minute. You can watch it complete on the **Imports** page, which tracks everything you have ever brought into HALO:
<Frame>
<img alt="The Imports page showing a completed file import of 1,005 traces and 16,174 observations" />
</Frame>
<Tip>
You can come back to this screen anytime. The **Import Data** button in the top right of the Data and Imports pages reopens the same import options, and **Settings → Replay onboarding** restarts the whole flow.
</Tip>
## Step 3: Add a model provider
HALO analysis runs against a model you choose, with your own key. Open **Settings** and click **Add Provider** under **Model providers**.
<Frame>
<img alt="The Settings page showing the Model providers section with an Add Provider button, Data Management with database details, and the HALO Configuration section with the engine installed" />
</Frame>
Pick a provider type, which prefills the base URL, then paste your API key:
* **OpenAI** sets `https://api.openai.com/v1`.
* **Anthropic** sets `https://api.anthropic.com/v1`.
* **Custom OpenAI-compatible** lets you point at OpenRouter, Inference (`https://api.inference.net/v1`), or any other compatible endpoint.
<Frame>
<img alt="The Add provider dialog with a provider type selector, name, base URL, and API key fields" />
</Frame>
The key is stored locally in SQLite, masked in the UI, and never leaves the machine.
While you are in Settings, check the **HALO Configuration** section at the bottom. HALO runs as a local Python engine; this section shows its install status and runtime checks (`git`, `uv`, Python 3.12).
## Step 4: Find your way around
The app has four main pages, all in the left sidebar:
* **Data** is the trace timeline: every trace and session in your local workspace, with search, filters, and summary stats.
* **Analysis** is where HALO runs live: start new runs, watch them stream, and revisit every completed report.
* **Imports** tracks trace history brought in from other tools, with per-import status and counts.
* **Settings** holds model providers, the local database details, and the HALO engine install.
Open **Data**. The stats bar across the top summarizes the workspace (traces, spans, LLM spans, errors, tokens, cost), and each row is one run with its input, output, and duration. Failed runs are tinted red, so problems jump out while scrolling. The left rail filters by time window, status, source, service, agent, and model, and the **Traces / Sessions** toggle switches between individual runs and grouped sessions.
<Frame>
<img alt="The Data page showing about 1,000 imported traces with filters on the left, summary stats across the top, and failed runs highlighted in red" />
</Frame>
<Warning>
Imported traces keep their **original timestamps**, not the time you imported them. Set the time window to **All time** so the demo dataset shows up, and do the same when you configure the HALO run in [Step 6](#step-6-run-halo).
</Warning>
## Step 5: Open a trace
The demo traces all come from one agent, **Traceable Search Agent**: a compact web-search agent that plans in a scratchpad, searches the web through Tavily, extracts pages, scores sources, and writes a final answer. It ships with deliberate, documented flaws (a heuristic source-quality score, an unstructured scratchpad, no enforced answer schema), which is exactly what makes this dataset a good HALO sandbox. The code lives in [`context-labs/SearchAgentDemo`](https://github.com/context-labs/SearchAgentDemo) (Python) and [`context-labs/SearchAgentDemoTS`](https://github.com/context-labs/SearchAgentDemoTS) (TypeScript); you will clone one in [Step 7](#step-7-hand-the-report-to-a-coding-agent).
Click any row on the Data page to open the trace viewer. The **Timeline** view is a waterfall of every span in the run: the agent loop at the top, then each LLM call, tool call, and retriever call laid out against the clock. It makes the shape of a run obvious at a glance, including what ran serially that could have been parallel, and where the time actually went. Use the **Cost heatmap** toggle to shade spans by spend, and zoom with the controls or by double-clicking a span.
<Frame>
<img alt="The Timeline view of a single trace, showing a waterfall of OpenAI Chat Completions calls, web search tool calls, and nested tavily.search retriever spans across an 8 second run" />
</Frame>
Switch to the **Conversation** view to read the same run as a thread: the user's question, the agent's answer, and an expandable list of every span in between. Click any span to inspect it in the side panel, with the full input and output, model, duration, and cost. Open the first LLM span and you can read the agent's entire system prompt, including its operating loop and its known rough edges.
<Frame>
<img alt="The Conversation view of a trace with the span list expanded and a side panel showing an LLM span's system prompt, model, and timing details" />
</Frame>
Skim a handful of runs, especially the red ones. You will spot failed searches and retry loops by eye. That is the point of the next step: HALO reads all 1,000 runs and finds the patterns for you.
## Step 6: Run HALO
Click **Run Analysis** in the top right of the Data page (or start from the **Analysis** page). The dialog is where you scope the run:
* **Session group or Trace group** picks the unit of analysis.
* The filters (search, window, status, source, service, agent, observed model, scope) narrow which telemetry HALO reads. For the demo dataset, set the window to **All time** and leave the rest open. The live counts at the bottom confirm what the run will cover, about 1,000 traces here.
* **Provider and model** select the provider you saved in [Step 3](#step-3-add-a-model-provider) and which of its models does the analysis.
* **Title and prompt** name the run and tell HALO what to look for.
<Frame>
<img alt="The Run Analysis dialog with Session group and Trace group tabs, telemetry filters, live counts showing 1,000 traces, and provider and model selectors" />
</Frame>
A specific prompt gives sharper results. This run used:
```text theme={"system"}
Analyze these traces. Identify the most important failures, latency bottlenecks,
confusing tool behavior, and concrete improvements for the developer.
```
Click **Start run**. The run moves through **Queued → Export → Analysis → Done**, and you can watch every agent step stream in as the engine digs through the traces. The right rail tracks the run details: target, provider, model, and the trace, session, and span counts as they are exported.
<Frame>
<img alt="A HALO run just after starting, showing the analysis prompt, the Queued, Export, Analysis, Done status pipeline, and run details in the right rail" />
</Frame>
Expect several minutes for a dataset this size; this run covered 1,000 traces and 16,000+ spans in about nine minutes. When it finishes, the report renders in place: ranked findings with citations back to the exact traces each one came from. Click a citation chip to drop into that trace and confirm the finding before you act on it.
<Frame>
<img alt="A completed HALO report showing ranked findings with trace citations, section chips for failures and latency bottlenecks, and an Act on this report panel in the right rail" />
</Frame>
On this dataset, HALO's top findings included:
* **A deterministic tool-argument failure.** The model passes `topic="health"` or `topic="sports"` to the Tavily search tool, which only accepts `general`, `news`, or `finance`, so those searches fail every single time. The fix is to encode the constraint as an enum in the tool schema.
* **Error-prone search and extraction that triggers rework loops.** Failed searches and extractions send the agent back through extra LLM deliberation and retries, burning latency and tokens.
* **Latency dominated by sequential LLM calls.** The slowest runs stack around six back-to-back chat completions plus serial searches that could run concurrently.
* **Confusing telemetry.** The tool wrapper span and the underlying provider span both report ERROR for the same root cause, and the wrapper span is missing its real inputs, which makes debugging and error counts misleading.
The report closes with a prioritized action plan, ordered by return on effort:
<Frame>
<img alt="The developer-focused action plan from the HALO report, listing five fixes ranked by ROI, from Tavily topic correctness through cleaning up tracing semantics" />
</Frame>
<Tip>
The chat box under the report asks follow-up questions by re-running HALO over the same traces with full context. Use it to drill into a finding ("show me more traces with the topic failure") without configuring a new run.
</Tip>
## Step 7: Hand the report to a coding agent
The report is a ranked, cited to-do list for the agent's codebase, and the desktop app hands it off directly. The **Act on this report** panel next to every completed run has one-click buttons for **Open in Cursor**, **Open in Claude Code**, and **Open in Codex**, plus **Copy prompt** for any other tool. No MCP server, no wiring.
<Frame>
<img alt="The Act on this report panel with Open in Cursor, Open in Claude Code, and Open in Codex buttons and a Copy prompt option" />
</Frame>
The demo traces came from the search agent repo, so that is where the fixes belong. Clone it first, in whichever language you prefer (the two flavors have the same tools, the same deliberate flaws, and produce the same findings):
<CodeGroup>
```bash Python theme={"system"}
git clone https://github.com/context-labs/SearchAgentDemo.git
cd SearchAgentDemo
```
```bash TypeScript theme={"system"}
git clone https://github.com/context-labs/SearchAgentDemoTS.git
cd SearchAgentDemoTS
```
</CodeGroup>
Then click the button for your coding agent and run it from the cloned repo. Every run saves its full report to disk (the paths are listed under **Artifacts** in the run details), and the handoff prompt points the coding agent at that file with instructions along the lines of:
```text theme={"system"}
Read the HALO trace-analysis report at
~/Library/Application Support/net.inference.halo/halo-runs/<run-id>/report.md
and act on its findings in this codebase: fix the identified failures and
implement the recommended improvements.
```
For this dataset, the fixes land in the tool schemas (`tools.py` / `tools.ts`), the agent loop (`agent.py` / `agent.ts`), and the tracing setup. The report's citations mean the coding agent is not guessing: every recommendation traces back to specific runs you can open in the viewer.
## Step 8: Close the loop
A fix is not done until new traces from the changed code confirm it. This is where the desktop app's live mode comes in: the same agent repo can stream traces straight into your local timeline while it runs, with nothing crossing the network beyond the agent's own model calls.
1. In the app, open **Import Data → Connect Local Agent**. It shows the local ingest endpoint. Copy the repo's env file and point the agent at it:
```bash theme={"system"}
cp .env.example .env
```
```bash theme={"system"}
# .env
INFERENCE_API_KEY=sk-... # powers the agent's model calls
CATALYST_OTLP_ENDPOINT=http://127.0.0.1:8799/v1/traces # send traces to the desktop app, not the cloud
```
`INFERENCE_API_KEY` is what the agent uses to call its model; point it at any provider by also setting `INFERENCE_BASE_URL` (it defaults to Inference's OpenAI-compatible endpoint). The `CATALYST_OTLP_ENDPOINT` override is the whole trick: instead of shipping spans to the cloud, the agent posts them to the desktop app listening on `127.0.0.1:8799`.
2. Run a batch from the repo's bundled query dataset against your patched code. The query set is fixed, so the only variable between before and after is your fix:
<CodeGroup>
```bash Python theme={"system"}
uv run search-agent-batch --limit 5
```
```bash TypeScript theme={"system"}
bun run search-agent-batch --limit 5
```
</CodeGroup>
Watch the traces land in the Data page as each run finishes.
3. Run HALO again, scoped to the new traces (a recent time window or the new session IDs), and compare the two reports. The findings you fixed should be gone or diminished; pick up the next one.
That is the whole HALO loop, start to finish, running entirely on your machine. Once it clicks here, point the same loop at your own agent.
## Bring your own traces
You do not have to use the demo dataset. The same **Import Agent Traces** options from [Step 2](#step-2-load-the-demo-traces) work on your real data, which makes the desktop app a low-commitment way to try HALO on the agent you actually run:
* **Import from Langfuse.** Paste a Langfuse project's API URL and key pair (public and secret). HALO pulls your trace history over the Langfuse public API and stores it locally. If you would rather keep your existing pipeline and try HALO without importing, you can instead point the [Langfuse SDK at Catalyst](/integrations/traces/langfuse) with an env-var switch.
* **Import from Phoenix.** Bring historical traces from an Arize Phoenix project into the local timeline the same way.
* **Import JSONL.** Any JSONL trace export with one span per line works, including OpenTelemetry and Catalyst exports.
* **Connect Local Agent.** Stream traces live from any Catalyst- or OpenTelemetry-instrumented agent by pointing its OTLP endpoint at `http://127.0.0.1:8799/v1/traces`, as in [Step 8](#step-8-close-the-loop).
<Tip>
HALO groups and analyzes traces by agent identity. For the sharpest reports from imported data, make sure your top-level spans carry a stable `agent.name` (and ideally `agent.id`). For Langfuse specifically, see [Group by agent](/integrations/traces/langfuse#group-by-agent).
</Tip>
## Self-hosting and production
A common question: can you run this in production? Two answers, depending on what you mean.
* **The desktop app** is a single-user, local-first tool. It is ideal for exploring, debugging, and iterating on an agent on your own machine. It is not designed to be a shared, always-on production service.
* **The HALO engine** is open source and MIT licensed ([`context-labs/halo`](https://github.com/context-labs/halo), [`halo-engine` on PyPI](https://pypi.org/project/halo-engine/)). You can self-host it: install the CLI, point it at exported traces, and run it anywhere, including in CI against a fresh trace export on every deploy. That is the self-hosted production path for the analysis step itself.
For continuous, team-scale observability (always-on trace ingest, multiple users, scheduled HALO runs, [Signals](/guides/measure-agent-quality), and alerting), the [hosted Catalyst platform](/guides/try-halo-demo-repo) is the path we recommend. The instrumentation is identical, so anything you build locally moves over by changing where traces are sent.
## Where to go next
<CardGroup>
<Card title="Try the hosted version" icon="cloud" href="/guides/try-halo-demo-repo">
The same loop on the hosted Catalyst dashboard, with scheduled runs, Signals, and the MCP server.
</Card>
<Card title="Do this on your own agent" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Run the trace, analyze, fix loop against your real app from instrumentation onward.
</Card>
<Card title="Bring your Langfuse traces" icon="link" href="/integrations/traces/langfuse">
Point the Langfuse SDK at Catalyst with an env-var switch, no code changes.
</Card>
<Card title="HALO on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO desktop app, engine, and methodology. MIT licensed.
</Card>
</CardGroup>
# Measure Your Agent's Quality with Signals
Source: https://docs.inference.net/guides/measure-agent-quality
Define plain-language signals and let an LLM judge label every span, trace, or session automatically and turn your agent's traffic into quality metrics you can chart, filter, and get alerted on.
Your dashboard tracks the metrics every agent has: run count, error rate, latency, tokens, cost. However, these metrics don't accurately tell you whether the agent actually did its job, or whether the people using it are happy. Did it finish the task or quietly give up halfway? Is anyone getting frustrated? Those answers live in the content of your traces, and reading traces by hand obviously doesn't scale.
Signals fix that. A signal is a plain-language classifier you define once and Catalyst runs continuously against your traffic as it comes in. You describe the signal and an LLM judge figures out if that applies to the span, trace, or session automatically. The labels become metrics you can chart, filter by, and alert on.
<Frame>
<img alt="Traces flow in, a signal you define classifies each span, trace, or session with an LLM judge, and the labels land back on your traffic as metrics you can filter, chart, and alert on" />
</Frame>
Signals are fully configurable to your needs but their are two overarching things you are typically measuring:
* **How users interact with your agent** — sentiment, frustration, jailbreak attempts, NSFW content.
* **How your agent behaves** — whether it completed the task, refused, or stalled.
You can interact with signals via the **dashboard**, through the [MCP server](/integrations/mcp-server), and from the [CLI](/cli/overview) (`inf signals`). This guide leads with the dashboard.
## Before you start
Signals run on your traces, so you need traces flowing first. You need:
* A free [Inference account](https://inference.net/register).
* **Tracing installed and traces arriving.** If you haven't done this, start with [Optimize an Agent End to End](/guides/optimize-an-agent-end-to-end) or the [Tracing Quickstart](/integrations/traces/quickstart) and come back once traces show up.
* **A stable agent identity.** Signals are scoped to one agent, so your runs need a consistent `agentId` to group on. If you instrumented with the CLI this is likely already set.
```typescript theme={"system"}
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "gator-flue-agent", // Stable ID every signal attaches to. Keep it constant across deploys.
agentName: "Gator Flue Agent",
sessionId: chatId, // Needed for session-scoped signals. Think Slack thread, chat, or job ID - Basically a single conversation.
},
async (span) => {
span.setInput(input);
const result = await run(agent, input);
span.setOutput(String(result.finalOutput ?? ""));
},
);
```
<Note>
Signals group on `agentId`. If you want **session-scoped** signals (Step 2), you also need a stable `sessionId` on your traces. See [Agent identity](/integrations/traces/agent-identity).
</Note>
## Step 1: Find Signals in the dashboard
There are two ways in:
* The top-level **Signals** tab lists every signal across all your agents in one table, with each one's agent, scope, type, a live result breakdown, trend, and volume.
* Inside an agent (**Agents** → your agent → **Signals**) you see just that agent's signals, with their overview graphs next to its other metrics.
Either way, **signals are per agent** — each one classifies the traffic of a single `agentId`. Here's the project-wide table for our team, with the spread we run on Gator:
<Frame>
<img alt="The top-level Signals tab showing a table of every signal across the project — session sentiment, sentiment, task outcome, laziness / refusal, and user frustration — each with its agent, type, live result breakdown, 7-day trend, and volume" />
</Frame>
## Step 2: Create a signal
Open the agent's **Signals** view and create a new signal. You can write your own from scratch or start from a template that prefills the type, prompt, and labels for a common case:
| Template | Type | What it flags |
| ---------------------- | ------ | -------------------------------------------------------------------------- |
| **Task outcome** | String | Whether the task was completed / partial / failed / abandoned. |
| **Sentiment** | String | Overall user sentiment: positive / neutral / negative. |
| **User frustration** | Binary | The user expressing annoyance, repeated correction, or dissatisfaction. |
| **Laziness / refusal** | Binary | The assistant refusing, stalling, or giving a low-effort non-answer. |
| **Jailbreak attempt** | Binary | The user trying to bypass safety guardrails or override the system prompt. |
| **NSFW** | Binary | Content that's sexually explicit, graphic, or not safe for work. |
You're not limited to these — write any prompt and classify anything you can describe. Either way, every signal comes down to a few decisions.
### Classifier type
<CardGroup>
<Card title="Binary (yes / no)" icon="toggle-on">
A true/false classifier for "is this X or not" questions: frustration, refusals, jailbreak attempts, NSFW. No labels to configure, just a prompt.
</Card>
<Card title="String (enumerated labels)" icon="tags">
Returns one of a fixed set of labels you define. Use it when there are more than two outcomes: task outcome (completed / partial / failed / abandoned), sentiment (positive / neutral / negative). Define 2 to 10 labels.
</Card>
</CardGroup>
### Scope
A signal classifies one **unit** of traffic at a time:
| Scope | What the judge reads | Best for |
| ----------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| **Span** | A single model call. | Fine-grained checks on one step. Can be a lot of labels on a busy agent. |
| **Trace** | One full request or turn. | Request-level questions about a single run. |
| **Session** | The whole conversation or job, end to end. | Understanding a complete interaction: overall sentiment, or whether a multi-turn task ultimately got done. |
<Warning>
**Session scope needs a `sessionId` on your traces.** Without it there are no sessions to classify. Also note: **scope is fixed once a signal is created** — you can edit the prompt, labels, and sample rate later, but not the scope. Pick deliberately.
</Warning>
The same question can be worth running at more than one scope. We track sentiment two ways on Gator: once per session for the overall vibe of a conversation, and again at a finer scope to spot where things turn. That's why you'll see both in the screenshots.
### Sentiment
Every signal also carries a **sentiment** that tells Catalyst whether an outcome is good, bad, or neutral. This is what lets alerting reason about direction instead of just movement. A **negative** result trending up, or a **positive** result trending down, is worth flagging; a **neutral** result is just neutral and won't trigger anything on its own.
You set sentiment based on the classifier type:
* **Binary** signals get one sentiment for the whole signal. A Jailbreak attempt or NSFW signal is negative, so any climb is bad news.
* **String** signals get one sentiment per label. On a Sentiment classifier you'd mark `negative` as negative, `positive` as positive, and `neutral` as neutral, so a rise in negative labels or a drop in positive ones gets caught automatically.
This mostly pays off in [alerting](#step-8-get-notified): because each label already knows whether it's good or bad, an alert can watch for "the bad thing going up" without you spelling out the direction every time.
### Sample rate
A sliding selector sets how much of your matching traffic actually gets classified, anywhere from a small fraction up to 100%. Every classification is a judge call you pay for, so this is the main cost lever on a signal.
**100% labels every span, trace, or session.** That's fine on a low-volume agent, but if you're running a lot of traffic it's usually far more than you need and can get expensive fast. A 25% or 50% sample gives you a representative trend at a fraction of the spend; raise it later once you trust the labels or want finer resolution. You can change it anytime.
### Write the prompt and save
Whatever you're classifying, the **prompt** is the instruction the judge follows on every target, so be specific about what counts as each outcome (and, for a string classifier, what each label means). When it's written, give the signal a short, readable name and **save it as a draft** — that keeps it unpublished so you can test before going live.
<Frame>
<img alt="The create-signal form (shown here for a Jailbreak attempt signal at session scope), with the template picker, scope and binary type selectors, the classifier prompt, sample-rate presets, and an inline 'try it before you commit' preview" />
</Frame>
<Tip>
Prefer to script it? `inf signals create` does the same from the terminal, and the [MCP server](/integrations/mcp-server) exposes `create_signal` so your coding agent can set one up for you.
</Tip>
## Step 3: Test before you activate
Before committing a signal to live traffic, run it against recent data to preview how it labels. A test classifies a small sample (1–100 recent targets) and shows you the distribution plus the label on each one. **Nothing is saved** — it's a preview, so it doesn't touch your signal or store labels.
If the judge disagrees with you on a few, tighten the prompt and test again. This is the cheapest place to get a signal right, before any labels exist.
<Frame>
<img alt="The Test Signal dialog noting results are not persisted, with a sample size and time range, the positive / neutral / negative distribution across the sample, and a per-row table of each span and its label" />
</Frame>
<Note>The CLI equivalent is `inf signals verify`.</Note>
## Step 4: Activate, pause, and resume
When the test looks right, activate the signal. Live classification starts immediately: as new traffic arrives, the configured share is sampled and labeled automatically.
A signal is always in one of three states, and you can move between them anytime:
| State | What it means |
| ------------ | ----------------------------------------------------------------------- |
| **Draft** | Saved but not running. |
| **Active** | Live. New matching targets are sampled and labeled. |
| **Disabled** | Paused. Classification stops, but every label collected so far is kept. |
Pause an active signal whenever you want and resume it later without losing history. Sampling is deterministic, so the share you set is the share you pay for.
## Step 5: Backfill historical data
Live classification only labels traffic that arrives after you activate. To label data you already captured — so your charts have history on day one — run a **manual run** (backfill): pick a historical window and a sample rate, and it classifies past targets in the background. Unlike a test, a backfill **saves** its labels, exactly like live ones. You can run one at any point, not just at creation.
<Frame>
<img alt="The Manual Run / Backfill dialog applying a signal to a historical time range, with From and To dates and a sample-rate slider for the percentage of past targets to classify" />
</Frame>
<Tip>
Backfill a short window first to sanity-check labels at scale before running over months of history — a manual run classifies real targets and counts toward usage. CLI: `inf signals run create`.
</Tip>
## Step 6: Read the results
This is where signals become metrics. Each agent's **Signals** view rolls up every signal it runs: a summary up top (how many are active, total classifications, the busiest signal) and a card per signal with its label breakdown and trend, sitting right next to run count, latency, and cost.
<Frame>
<img alt="The agent's Signals overview showing summary tiles for active signals, total classifications, and busiest signal, with a card and trend graph for each signal: session sentiment, sentiment, task outcome, and laziness / refusal" />
</Frame>
Open any signal and its detail page charts its classifications over time and the full label distribution.
<Frame>
<img alt="A signal detail page for Task outcome, with stat tiles for classifications, top label, sample rate, and live version, a stacked bar chart of classifications over time, and a donut of the completed / partial / failed / abandoned distribution" />
</Frame>
Below the charts is a **table of every classified target**, with its label. Click any row to drop into that span, trace, or session and read the **actual conversation** behind the label: the full run, the tool calls, the output the judge saw. You can also filter the main **Traces** view by a label to pull up just the flagged runs.
<Frame>
<img alt="The labeled-traces table on a signal detail page, one row per classified target with its label, version, session, and time, each clickable through to the underlying trace" />
</Frame>
## Step 7: Edit and version
Editing a signal's prompt, labels, or sample rate **creates a new version** rather than overwriting the old one. The dashboard shows each version, and every label records which one produced it, so you can sharpen a definition over time without losing the history of what earlier versions decided. (Scope is the one thing fixed at creation.)
When a signal has served its purpose, **archive** it: it stops and leaves your active list, but its labels are preserved.
## Step 8: Get notified
A trend you have to remember to check is one you'll miss. There are two ways to stay on top of your signals without logging in: **targeted alerts** for a specific condition, and a **daily overview** that summarizes everything.
### Targeted alerts
An alert watches one signal and fires when its numbers cross a line. They're configured per signal (a signal can have several), and a global **Alerts** page lists every alert across all your signals.
You build an alert by picking a **metric** to watch, the **direction and threshold** that count as a problem (a level it crosses, or a percentage it moves up or down), and the **time window** to measure over. Which metrics are available depends on the signal's type:
| Metric | Works on | Measures |
| ---------------- | -------- | --------------------------------------- |
| **True rate** | Binary | Share of targets labeled "yes." |
| **Value count** | String | Count of a specific label. |
| **Value share** | String | Share of targets with a specific label. |
| **Label volume** | Any | Total labels produced. |
The threshold reads one of two ways: **percentage change** (the metric moves up or down by X% versus the prior window) or an **absolute level** (it crosses a fixed number). You also set the rolling **window** (5 minutes to 48 hours), an optional **minimum label count** so low volume doesn't trip it, and a **cooldown** so repeat firings don't bury you. As you tune it, the form previews where the alert *would* have fired over the last 7 days.
The **sentiment** you set in Step 2 is what makes the direction obvious here: you watch negative labels for an increase and positive labels for a decrease, since both mean things are getting worse. Neutral labels are rarely worth alerting on at all.
<Frame>
<img alt="The alert create form: a signal and name, the condition built from a metric, a decreases-by / increases-by percentage or absolute level, and a window, plus a minimum label count, a Slack connect prompt, a cooldown selector, and a 7-day 'would have fired' preview chart" />
</Frame>
By default an alert emails your **whole team**, and any member can mute an individual alert for themselves. Connect **Slack** once and you can route an alert's firings to a specific channel instead. A firing tells you the signal, the metric, the threshold, what was observed, and when:
<Frame>
<img alt="A Slack signal-alert message for Jailbreak attempt reading 'Label volume above 1 over 5m', with fields for the signal, metric, comparison, threshold, observed value, window, and trigger time, and a View alert button" />
</Frame>
<Tip>The CLI mirrors all of this: `inf signals alerts backtest` to preview, then `inf signals alerts create`.</Tip>
### Daily overview
Targeted alerts are for when something specific goes wrong. The **daily overview** is for confirming everything's humming along without opening the dashboard. Opt in and you get a once-a-day email (or Slack) summarizing all your signals: how many were classified, and the biggest movers since yesterday. It's the "is anything off?" glance you can do from your inbox.
<Frame>
<img alt="A daily signal overview email summarizing the day across signals — total classified, the biggest mover called out at the top, and a card per signal with its volume and percentage change — plus a View signals button" />
</Frame>
## Step 9: Hand it to HALO
Signals tell you **what** is happening and **how often** — failed tasks spiked overnight, frustration is up this week. [HALO](/guides/optimize-an-agent-end-to-end) tells you **why**, and what to change.
The handoff runs through the [MCP server](/integrations/mcp-server). With Inference connected to your coding agent, you go from a signal to a fix in plain language:
```text theme={"system"}
The Task outcome signal on gator-flue-agent is showing more failures this week.
Pull the failed sessions, figure out what's going wrong, and propose a fix.
```
It works the other way too — ask HALO what to watch:
```text theme={"system"}
Look at gator-flue-agent's recent traces and tell me which signals would catch its most common failure modes.
```
HALO reads the flagged traces, finds the systemic cause, and writes back concrete fixes with citations to the exact runs. See the [MCP server guide](/integrations/mcp-server) for setup.
## Where to go next
You now have quality metrics that didn't exist an hour ago — defined in a sentence, computed on every run, charted, and wired to notifications — with no extra instrumentation.
<CardGroup>
<Card title="Signals reference" icon="signal-stream" href="/get-started/signals">
Full reference for classifier types, scopes, sample rates, states, testing, backfills, and templates.
</Card>
<Card title="Optimize an agent end to end" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Run HALO on the traces your signals flag to find the root cause and apply the fixes.
</Card>
<Card title="Try HALO on a demo repo" icon="flask" href="/guides/try-halo-demo-repo">
Run the full trace → HALO → fix loop on a ready-made instrumented agent in about fifteen minutes.
</Card>
<Card title="Connect the MCP server" icon="plug" href="/integrations/mcp-server">
Create signals, read labels, and hand flagged traces to HALO from your coding agent.
</Card>
<Card title="Set agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent and session IDs so signals attach to the right agent and scope cleanly.
</Card>
</CardGroup>
# Optimize an Agent End to End with Catalyst Tracing and HALO
Source: https://docs.inference.net/guides/optimize-an-agent-end-to-end
Most agents have problems you can't always see from the outside: silent failures that still report `OK`, tool calls firing twice, a consistently slow call quietly burning latency and spend. They're hard to catch, and with most tracing platforms you're left parsing spans yourself.
Plugging your traces into an LLM doesn't always fix it either. Traces are usually a huge amount of context such that a general model can't reliably find the patterns across runs that are causing your agent to underperform. That's why we built HALO (Hierarchical Agent Loop Optimizer): a novel way to inspect traces with an RLM, tuned for context-ingestion efficiency so you get maximum signal out of every run.
This guide walks the full loop: **trace, measure, analyze, fix, repeat.** You'll install Catalyst tracing, record real traces, explore them in the dashboard, run HALO to find what to improve, and apply the fixes it hands back.
We follow a real example the whole way through: our own internal GTM agent with significant day-to-day traffic.
<Frame>
<img alt="Install tracing once, then a repeating loop: capture traces, run HALO, apply fixes, and back to capturing traces" />
</Frame>
Prefer to watch? The video below walks through the same loop end to end. It covers the same ground as this guide in video form, so use whichever you like (or both).
<Frame>
<iframe title="Optimize an Agent End to End with Catalyst Tracing and HALO" />
</Frame>
## Before you start
You need:
* A free [Inference account](https://inference.net/register).
* An app or agent that makes LLM calls. Any provider or framework works (OpenAI, Anthropic, Gemini, LangChain, LangGraph, the Vercel AI SDK, OpenAI Agents, and [more](/integrations/traces/overview)).
* An API key from the [dashboard](https://inference.net/dashboard/api-keys).
## Step 1: Install tracing
Tracing captures the full execution of your agent: every LLM call, tool call, framework step, and any custom spans you wrap. HALO reads those traces, so the richer your traces, the better the analysis.
The fastest path is to let the [Inference CLI](/cli/overview) drive a coding agent that wires the SDK in for you.
<Steps>
<Step title="Install the CLI and sign in">
```bash theme={"system"}
npm install -g @inference/cli && inf auth login
```
Your browser opens to authenticate.
</Step>
<Step title="Run instrumentation in your project">
From your project root:
```bash theme={"system"}
cd /path/to/your/project && inf instrument --mode tracing
```
The CLI scans your codebase for LLM clients and agent frameworks, installs the Catalyst tracing SDK, wires `setup()` into your entrypoint, adds stable service and agent identity, and shows you every change before applying it.
</Step>
</Steps>
Prefer to wire it in yourself? Install the Catalyst tracing SDK directly — [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) for TypeScript or [`inference-catalyst-tracing`](https://pypi.org/project/inference-catalyst-tracing/) for Python:
<CodeGroup>
```bash bun theme={"system"}
bun add @inference/tracing
```
```bash npm theme={"system"}
npm install @inference/tracing
```
```bash pnpm theme={"system"}
pnpm add @inference/tracing
```
```bash yarn theme={"system"}
yarn add @inference/tracing
```
```bash pip theme={"system"}
pip install inference-catalyst-tracing
```
</CodeGroup>
The [Tracing Quickstart](/integrations/traces/quickstart) guide has the full manual setup for TypeScript and Python, plus how to wrap an agent span so your runs group cleanly in the dashboard.
## Step 2: Record traces
Run your app the way you normally would. Traces stream to Catalyst as your code executes.
The goal at this stage is **volume and variety**: the more real runs you capture, the more signal HALO has to work with. Exercise the paths you actually care about, including the ones that go wrong. Errors, retries, and slow paths are exactly what HALO is looking for.
<Tip>You can still experiment with HALO using development data but the full value comes from being able to analyze traces of production level traffic.</Tip>
Confirm traces are arriving before moving on. You can check from the CLI:
```bash theme={"system"}
inf trace list --range 1h
```
Or just open the **Traces** tab in the dashboard and you should see a list of traces in the table if everything was set up correctly.
<Frame>
<img alt="The Traces tab in the Inference dashboard showing a live list of captured traces with columns for time, root span, kind, service, environment, and agent" />
</Frame>
If you see rows here, tracing is live and you're ready to start digging in.
## Step 3: Explore a trace
Click any trace and a detail sheet slides open. This is where the real picture of a run lives, and it has a few tabs worth knowing.
### The trace tree
The first tab shows the top-level trace and every span nested underneath it. Click into any span to expand it. For each one you get:
* **Inputs and outputs**, the exact payload that went in and came back out.
* **Span attributes**, including cost and input/output token counts per span, so you can see precisely which step spent what.
* **Raw JSON**, the unmodified span data, which you can also download for offline use or to feed into your own tooling.
This is the view you reach for when you want to know exactly what happened on a single run, step by step.
<Frame>
<img alt="The trace detail sheet open on the trace/spans tab, with the span tree on one side and an expanded span showing its inputs, outputs, span attributes including cost and token counts, and the raw JSON and download controls" />
</Frame>
### The timeline
Switch to the timeline tab and the same run is plotted along a timeline. You see the total length of the run, then the duration of each individual span (each agent step, tool call, MCP call, and so on) stacked underneath. This is the fastest way to spot what's actually slow: a single tool call eating ten seconds jumps right out, where it would be easy to miss in the tree.
<Frame>
<img alt="The trace detail sheet on the timeline tab, showing the total run duration up top and the per-span duration bars for tool calls, MCP calls, and model calls below" />
</Frame>
### The thread
The thread tab is the human-readable view of the whole run. It renders the entire conversation top to bottom, from the first message to the last, the way you'd read a chat. For a chat-style agent that's the back-and-forth with the user. For a job that kicks off subagents, it's the data those subagents return and what the main agent did with it. When you just want to read what happened without parsing spans, this is the tab.
<Frame>
<img alt="The trace detail sheet on the thread tab, showing the full run rendered as a readable conversation from first message to final output" />
</Frame>
## Step 4: Filter and search
A list of traces is only useful if you can find the one you want. The Traces tab gives you a deep filter panel down the left side: filter by status, model, service, provider, user, session, numeric ranges like token count and cost, and any custom span attribute you've attached.
Search is where it gets powerful. This is not metadata search on names or IDs. It's a deep search across the full content inside every span: the messages, the tool inputs, the tool outputs, the errors, and any custom attributes you've attached. Type a single word that shows up deep inside a chat conversation, a tool's JSON payload, or an error string, and it surfaces every trace where that word appears, combined with whatever filters you have selected. Finding the one run where a user said "refund" and the model errored is a single query, not an afternoon of scrolling.
<Frame>
<img alt="The Traces tab with a deep search for a keyword and status and model filters applied, showing matching traces with the search term highlighted inside the chat content" />
</Frame>
## Step 5: Group your runs into agents
Everything so far works for plain LLM traces, and that alone is useful. But where this really pays off is with full **agent loops**.
Any run you wrap with a stable agent identity (an `agentId`, an `agentName`, and a few optional fields) gets promoted out of the raw trace list and rolled up under the **Agents** tab. Instead of scattered individual calls, you get one workspace per agent: its metrics, its sessions, its traces, and its HALO analysis, all grouped together.
If you instrumented with the CLI or an AI coding agent, it likely set an `agentId` for you. It's worth a quick look to make sure that ID is the stable, readable value you actually want, because that's the key everything groups on. Depending on your framework you may need to tweak where the wrapper goes.
Here's what setting it looks like with the OpenAI Agents SDK. The same shape applies to any framework:
<CodeGroup>
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run } from "@openai/agents";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Help customers with order questions.",
model: "gpt-4o-mini",
});
await agentSpan(
{
agentId: "support-agent", // Stable ID everything groups on. Keep it constant across deploys and renames.
agentName: "Support Agent", // Human-readable label shown in the Agents dashboard.
spanName: "support-agent.run", // Name of the top-level span for this run.
sessionId: conversationId, // Ties multi-turn runs into one conversation. Think chatId or like a Slack threadId.
userId: "user_8675309", // Optional. Lets you filter every trace down to a single user.
role: "support", // Optional. Useful when one workflow has several agents (triage, refunds, billing).
system: "openai", // The framework or provider powering the run.
},
async (span) => {
const input = "Where is order ABC-123?";
span.setInput(input);
const result = await run(supportAgent, input);
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
```python Python theme={"system"}
from agents import Agent, Runner
from inference_catalyst_tracing import agent_span, setup
tracing = setup()
support_agent = Agent(
name="SupportAgent",
instructions="Help customers with order questions.",
model="gpt-4o-mini",
)
with agent_span(
tracing.tracer,
agent_id="support-agent", # Stable ID everything groups on. Keep it constant across deploys and renames.
agent_name="Support Agent", # Human-readable label shown in the Agents dashboard.
span_name="support-agent.run", # Name of the top-level span for this run.
session_id=conversation_id, # Ties multi-turn runs into one conversation. Think chatId or like a Slack threadId.
user_id="user_8675309", # Optional. Lets you filter every trace down to a single user.
role="support", # Optional. Useful when one workflow has several agents (triage, refunds, billing).
system="openai", # The framework or provider powering the run.
) as span:
user_message = "Where is order ABC-123?"
span.set_input(user_message)
result = await Runner.run(support_agent, input=user_message)
span.set_output(str(result.final_output or ""))
tracing.shutdown()
```
</CodeGroup>
<Note>
Agents group on `agentId`. Set it once on the top-level span and every run for that agent rolls up together. If your runs aren't grouping the way you expect, the ID is almost always the thing to check. See [Agent identity](/integrations/traces/agent-identity).
</Note>
## Step 6: Explore your agent
Open the **Agents** tab and you get a list of every agent you've traced, grouped by `agentId`. Each one is a card with its name, some high-level info, and a small graph of its usage, so you can see all your agents and how busy they've been at a glance.
<Frame>
<img alt="The Agents tab showing the list of traced agents grouped by agentId, where each card has the agent name, high-level info, and a usage graph" />
</Frame>
Click into an agent and you land in its workspace. This is where most teams spend their time. It has a handful of sub-tabs.
### Overview
The overview is your agent's home page. It rolls up run counts, error rate, latency, token usage, and cost, all scoped to this one agent, and charts them over time so you can spot a spike in errors or watch cost creep up after a change. It's the first place to look when you want to know how the agent has been behaving lately.
<Frame>
<img alt="The Overview sub-tab for a single agent, showing the metric tiles up top and the time-series charts for run count, error rate, latency, token usage, and cost" />
</Frame>
<Tip>
Want to track more than the built-in metrics? **Signals** turn your traces into structured labels you define in plain language — "is this NSFW?", "did the user get frustrated?", "was the task completed?" An LLM judge labels matching spans automatically, so the result charts right here alongside your other metrics and becomes something you can filter and break down by. See [Measure Your Agent's Quality with Signals](/guides/measure-agent-quality) for the full walkthrough.
</Tip>
### Sessions
A **session** is the whole back-and-forth rolled up into a single view. Think of it as a conversation. You set the `sessionId` to something like a chat ID or a Slack thread ID, and every trace tagged with that ID becomes part of the same session. The `sessionId` should be unique and persistent for a single instance of a conversation that has a clear start and end.
The Sessions sub-tab lists one row per session with high-level metrics, and you can drill into any session to see each trace and span inside it, with the exact same trace detail (tree, timeline, thread) you saw back in Step 3.
<Frame>
<img alt="The Sessions sub-tab showing one row per session with high-level metrics, with a session expanded into the traces it contains" />
</Frame>
### Traces
The Traces sub-tab is the same trace view from Steps 3 and 4, just pre-filtered to this agent. All the same deep search and filtering applies, scoped to the runs that belong here.
### Analysis
The Analysis sub-tab is the HALO workspace, and it's where the next step happens.
## Step 7: Run HALO
Most observability tools stop at charts and leave the digging to you. [HALO](https://github.com/context-labs/halo) does the digging, with an architecture built specifically for this problem. HALO is an open-source agent-loop optimizer hosted right inside the Agents dashboard, and the key detail is that it's an RLM unlike a regular general purpose model.
An [RLM (Recursive Language Model)](https://github.com/alexzhang13/rlm) swaps the usual `llm.completion(prompt)` for an `rlm.completion(prompt)`. Instead of cramming every trace into one context window, it holds your traces as a variable in a code environment and lets the model programmatically examine, decompose, and recursively call sub-models over them. That distinction matters here. Traces are enormous, and a general-purpose model run over them either blows past its context window or overfits to the error in a single trace instead of generalizing to the systemic, harness-level problem behind it. Because an RLM's context is effectively unbounded, HALO can surface trends and recurring patterns across thousands of runs over time while still catching the granular one-off failure buried deep in a single span. As far as we know, no other trace-analysis tool is built this way.
From there the flow is straightforward: HALO decomposes your traces, identifies the systemic failure modes, and writes up concrete fixes with citations back to the exact traces each finding came from. This guide uses the hosted version, which runs the same engine against the traces you've already collected with no extra setup. You can also [self-host it](https://github.com/context-labs/halo) and point it at an exported trace file.
Open the **Analysis** sub-tab. Your previous HALO runs and chats live in a list down the left. On the right is a prompt window you can type anything into. We give you a sensible default prompt that works reasonably well out of the box, but you'll get sharper results the more specific you are. You can also set the time range HALO should analyze, plus advanced options like span limit, max depth, and max turns.
<Frame>
<img alt="The Analysis sub-tab, with the list of previous HALO runs and chats on the left and the prompt window on the right, showing the time-range picker and the advanced options for span limit, max depth, and max turns" />
</Frame>
A few tips for the prompt:
* The default is a good general starting point. Tighten it to your actual question for a sharper answer: "Why is the enrichment agent timing out?", "Which tool calls return empty results most often?", "Find redundant LLM calls in the planning loop."
* Tighter time windows give HALO more focused signal. A single problem agent over the last 24 hours beats a firehose of everything from the last month.
### Put HALO on a schedule
For agents already in production, you don't want to remember to run HALO by hand. Open the schedule create sheet and set it on a repeating schedule (hourly, daily, weekly) so it reviews recent traces automatically and you read reports as they land.
You can set up as many schedules as you want, each with its own prompt, so different schedules analyze your traces from different perspectives. One could watch for cost regressions daily while another hunts for reliability issues weekly.
<Frame>
<img alt="The HALO schedule create sheet, showing the schedule cadence, the prompt field, and the time range, with the ability to create multiple schedules each with its own prompt" />
</Frame>
## Step 8: Read the report and keep chatting
HALO works through the traces in the window and produces a report. It's ranked by impact and it goes deep, so the report is long enough that it'll scroll well past the fold.
On our own GTM agent, this exact run surfaced things we'd have spent days finding by hand, and in some cases never found at all, because the spans looked healthy:
* Tools being called with **invalid inputs**.
* Queries that were **failing without bubbling up**, so the span still showed an `ok` status while the work underneath had actually broken.
* **Duplicate tool calls** that repeated the same work for no benefit.
* A ranked **list of recommended fixes** for each finding.
Every finding cites the specific trace IDs it came from, so you can click straight from a finding into the trace tree that produced it and confirm it matches what you're seeing before you act.
We shipped fixes for the top findings the same afternoon: the silent query failures now surface as real errors instead of hiding behind an `ok` status, and the redundant tool calls are gone. That's the whole point of the loop. It turns invisible problems into a short, ranked to-do list.
<Frame>
<img alt="A completed HALO report showing ranked findings like invalid tool inputs, silent query failures on ok-status spans, and duplicate tool calls, along with a list of recommended fixes" />
</Frame>
Two things make the report genuinely actionable:
1. **Hand it to a coding agent.** Copy a finding and its recommended fix straight into your coding agent and let it make the edit. The fixes are usually specific enough to apply directly: tightening a prompt, adding a guardrail, removing a redundant call, adjusting a tool description.
2. **Keep chatting with HALO.** The report isn't the end of the conversation. Once it lands you can ask follow-ups in the same thread: "give me more concise ways to fix these," "what are the highest priorities in your opinion?", or "which tool calls are the most expensive, and what alternatives would get the same result?" If you're not getting the results you want, adjust your prompt and run it again.
## Step 9: Apply the fixes
A finding is only useful once it's in your code, and this is the step where the whole loop pays off. There are three ways to get a fix from a HALO report into your codebase, from most automated to most manual. Pick whichever fits how you work.
### Let your coding agent apply them (the MCP)
Connect the [Inference MCP server](/integrations/mcp-server) to your coding agent (Claude Code, Cursor, or any MCP client) and just ask it to pull the report and apply the fixes:
```text theme={"system"}
Apply the the latest fixes from the HALO report for the Gator Flue Agent
```
<Frame>
<img alt="A coding agent connected to the Inference MCP server pulling the latest HALO report for the gator flow agent and editing the prompts, tool definitions, and harness code directly in the repo" />
</Frame>
Just ask in natural language and your assistant resolves the agent name to its identity, finds the HALO reports, and edits your prompts, tool definitions, and harness logic directly, grounded in the same findings and trace citations you read in the dashboard.
That's the moment the loop closes itself. The same traces you've been sending all along get analyzed, turned into a ranked to-do list, and applied to your harness from a single line of natural language to make changes. See the [MCP server guide](/integrations/mcp-server) for setup and more example prompts.
### From the CLI
Prefer the terminal? The [`inf halo`](/cli/halo) commands pull the same report from the command line. Read it with `inf halo conversation get <conversation-id>` and pipe it into whatever coding agent you use, or read it inline.
### Copy it yourself
The manual path always works, and we mentioned it back in Step 8: copy the recommendation (and any suggested change) straight from the report into your coding agent, or make the edit by hand.
1. Copy the recommendation from the report, or paste it into your coding agent.
2. Make the edit in your prompt, tool definition, or harness logic.
3. Save and move on to the next finding.
## Step 10: Close the loop
A fix isn't done until you've confirmed it worked.
1. Ship the change and let your app run.
2. Capture a fresh window of traces.
3. Run HALO again over the new window (or wait for the next scheduled run).
4. Confirm the finding is gone, and pick up the next one.
That's the HALO loop. Each pass tightens your agent: fewer errors, lower latency, less wasted spend. The teams that get the most out of HALO run this continuously rather than once.
## What's coming
The loop above works today. And because all of it runs off the traces you're already sending, the features we're building next light up on that same data with no new instrumentation. Install tracing today and these turn on automatically as they ship, against the history you've already collected:
* **Connect your GitHub repo.** Link the repo behind your agent so HALO can read your actual code as additional context, then ground its suggestions in your real prompts, tool definitions, and harness logic instead of inferring them from traces alone.
<Info>
All of this builds on the traces you're already collecting. The earlier you instrument, the more history HALO has to work with the day these land.
</Info>
## Where to go next
That's the whole loop. If you haven't instrumented yet, [create a free account](https://inference.net/register) and run `inf instrument --mode tracing` in your project. Your first useful HALO report is about 20 minutes away.
<Tip>
Your traces are yours. We don't train on them, and we never use your data to improve our own models. What we do give you is the option to put that data to work for *you*: when you're ready, you can [train your own custom model](/get-started/train-and-deploy) on the runs you've already captured, entirely within our platform and under your control. A small model fine-tuned on your agent's real traffic is often faster and cheaper than a general-purpose one at the same task, and instrumenting today means that dataset is sitting ready whenever you want it. That's the second half of the loop: optimize the harness with HALO, and optimize the model with your own traces.
</Tip>
<CardGroup>
<Card title="Try it on a demo repo first" icon="flask" href="/guides/try-halo-demo-repo">
Run this whole loop on a ready-made, pre-instrumented agent before pointing it at your own.
</Card>
<Card title="Run inference through the gateway" icon="plug" href="/integrations/gateway/quickstart">
One API for every model, with usage, latency, and cost traced automatically.
</Card>
<Card title="Train a custom model" icon="graduation-cap" href="/get-started/train-and-deploy">
Turn the traces you've captured into a fine-tuned model that's faster and cheaper at your task.
</Card>
<Card title="HALO on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO engine, methodology, and benchmarks. MIT licensed.
</Card>
<Card title="Tracing integrations" icon="diagram-project" href="/integrations/traces/overview">
Every framework and provider Catalyst tracing supports, with setup for each.
</Card>
</CardGroup>
# Guides
Source: https://docs.inference.net/guides/overview
End-to-end walkthroughs that string the Catalyst platform together into a single workflow.
Guides are end-to-end walkthroughs. Where the rest of the docs explain one feature at a time, a guide takes you from an empty project to a finished outcome, stitching tracing, metrics, Halo, and your coding agent into one continuous loop.
## Start here
<CardGroup>
<Card title="Optimize an agent end to end" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Install tracing, record real traces, read the metrics, run Halo to find what to fix, then apply the fixes by hand or by pulling them straight into your coding agent. The full Halo loop, start to finish.
</Card>
<Card title="Try HALO with a demo repo" icon="flask" href="/guides/try-halo-demo-repo">
Skip instrumentation. Clone a ready-to-run search agent, seed real traces, run Halo to find what's wrong, and let your coding agent apply the fix through the MCP server. The full loop in about fifteen minutes.
</Card>
<Card title="Run HALO on your machine with the desktop app" icon="desktop" href="/guides/halo-desktop-app">
The same loop, fully local. Stream traces from a demo agent into the free HALO desktop app, run Halo with your own model provider, and apply the fixes in your editor. No account, no cloud.
</Card>
<Card title="Measure your agent's quality with Signals" icon="signal-stream" href="/guides/measure-agent-quality">
Turn raw traces into metrics. Define plain-language classifiers — frustration, sentiment, task outcome — and let an LLM judge label every span automatically, then filter and chart your traces by them.
</Card>
</CardGroup>
## What you need
Every guide assumes you have:
* A free [Inference account](https://inference.net/register).
* An app or agent that makes LLM calls (any provider or framework).
* A terminal with Node.js or Python, depending on your stack.
More guides are on the way. If there's a workflow you want walked through end to end, [let us know](mailto:support@inference.net).
# Try HALO End to End with a Demo Repo
Source: https://docs.inference.net/guides/try-halo-demo-repo
Clone a ready-to-run instrumented search agent, seed real traces, run HALO to find what's wrong, and let your coding agent apply the fix through the MCP server. The full loop in about fifteen minutes with zero instrumentation work.
<Note>
This guide uses the **hosted** version of HALO on the [Catalyst](/integrations/traces/quickstart) dashboard. Want to run the same loop entirely on your own machine, with traces that never leave your laptop and no account required? See [HALO Desktop: Optimize an Agent End to End on Your Machine](/guides/halo-desktop-app).
</Note>
The [Optimize an Agent End to End](/guides/optimize-an-agent-end-to-end) guide walks the full **trace, measure, analyze, fix** loop against *your* agent. This one hands you a working agent so you can feel the whole loop before you touch your own code.
[`context-labs/SearchAgentDemo`](https://github.com/context-labs/SearchAgentDemo) is a small search agent that comes already instrumented with Catalyst tracing. You get traces into your project (upload a pre-run dataset or generate your own), run [HALO](https://github.com/context-labs/halo) (our open-source agent-loop optimizer) over them, and then connect the [MCP server](/integrations/mcp-server) so your coding agent pulls the HALO report and edits the repo for you. The repo even ships with a handful of deliberate, documented flaws, so HALO has something real to find and you get to watch it close the loop.
The only hard requirement is a free Inference account. We even publish a [pre-run trace dataset](#step-3-get-traces-into-your-project), so you can be looking at a HALO report in a couple of minutes without running the agent at all. And when you do want to generate your own, the agent has a mock-search mode, so you don't even need a Tavily key.
<Note>
**Pick your language.** The demo ships in two behaviorally identical flavors: [`context-labs/SearchAgentDemo`](https://github.com/context-labs/SearchAgentDemo) (Python) and [`context-labs/SearchAgentDemoTS`](https://github.com/context-labs/SearchAgentDemoTS) (TypeScript). Same tools, same deliberate flaws, same dataset, same traces, same HALO findings. Use the language tabs in each step; everything between the steps (the dashboard, HALO, and MCP) is identical.
</Note>
<Frame>
<img alt="The HALO loop as a cycle: get traces (upload or generate), run HALO to find patterns across runs, apply fixes via MCP in your editor, re-run and compare to confirm the fix, and repeat" />
</Frame>
## Before you start
You need:
* A free [Inference account](https://inference.net/register).
* An API key from the [dashboard](https://inference.net/dashboard/api-keys).
* For the **Python** repo: [`uv`](https://docs.astral.sh/uv/) installed (the repo uses it for dependency management), plus `git` and Python 3.11+. For the **TypeScript** repo: [`bun`](https://bun.sh) installed, plus `git`.
* A coding agent with MCP support for the last few steps (Claude Code, Cursor, or any MCP client).
Optional:
* A [Tavily](https://tavily.com) API key for real web search. Without one, run the agent in `--mock-search` mode. Traces still flow, they're just more uniform.
## Step 1: Clone the demo repo
<CodeGroup>
```bash Python theme={"system"}
git clone https://github.com/context-labs/SearchAgentDemo.git && cd SearchAgentDemo
```
```bash TypeScript theme={"system"}
git clone https://github.com/context-labs/SearchAgentDemoTS.git && cd SearchAgentDemoTS
```
</CodeGroup>
It's a single [OpenAI Agents SDK](/integrations/traces/openai-agents) agent with multi-turn tool calling, instrumented end to end with the [Catalyst tracing SDK](/integrations/traces/quickstart). The layout is small on purpose:
<CodeGroup>
```text Python theme={"system"}
src/search_agent_example/
agent.py Agent definition and instructions
tools.py Scratchpad, search, extract, source scoring, and claim comparison tools
search_clients.py Tavily and mock search clients
cli.py Single-query traced runner
batch.py Dataset traced runner
data/
queries.jsonl 50 starter queries
search-agent-demo-traces.jsonl.gz ~1,000 pre-run traces (gzipped)
docs/ HALO notes and known limitations
```
```text TypeScript theme={"system"}
src/
agent.ts Agent definition and instructions
tools.ts Scratchpad, search, extract, source scoring, and claim comparison tools
searchClients.ts Tavily and mock search clients
cli.ts Single-query traced runner
batch.ts Dataset traced runner
data/
queries.jsonl 50 starter queries
search-agent-demo-traces.jsonl.gz ~1,000 pre-run traces (gzipped)
docs/ HALO notes and known limitations
```
</CodeGroup>
The agent runs a real loop: it plans in a scratchpad, searches the web, extracts pages, scores sources, compares claims, and writes a final answer. Every run is wrapped in an agent span with a stable identity (`agentId: traceable-search-agent`), and the Tavily calls inside the tools add manual `RETRIEVER` spans, so the traces have genuine structure for HALO to dig into.
<Tip>
This repo ships with a set of **deliberate, documented flaws**: a heuristic source-quality score, a loose unstructured scratchpad, a shallow lexical claim comparison, minimal URL deduplication, truncated page extraction. They're listed in `docs/known_limitations.md`. They're exactly the kind of systemic, harness-level issues HALO is built to surface, which is what makes this a good sandbox.
</Tip>
## Step 2: Install and configure
Install dependencies:
<CodeGroup>
```bash Python theme={"system"}
uv sync --extra dev
```
```bash TypeScript theme={"system"}
bun install
```
</CodeGroup>
Create your `.env` from the example:
```bash theme={"system"}
cp .env.example .env
```
Now open `.env` and paste in **one thing**, your [Inference API key](https://inference.net/dashboard/api-keys):
```bash theme={"system"}
INFERENCE_API_KEY=sk-... # the only required value
MODEL_ID=gpt-4.1-mini # already set for you; any tool-capable Inference model works
TAVILY_API_KEY= # leave blank for now; only needed for real web search (Step 3)
```
That single key does double duty: it powers the agent's model calls **and** sends the traces to Catalyst. The model defaults to `gpt-4.1-mini` (cheap, reliable tool-calling), so unless you want to change it, your only edit is pasting the key.
<Note>
One key, two jobs. `INFERENCE_API_KEY` authenticates the agent's model calls against Inference's OpenAI-compatible endpoint (`https://api.inference.net/v1`) and is copied into `CATALYST_OTLP_TOKEN` so traces flow to `https://telemetry.inference.net`. Both are wired up for you, no other configuration needed. Prefer a different OpenAI-compatible provider? Set `INFERENCE_BASE_URL` and `INFERENCE_API_KEY` to theirs.
</Note>
## Step 3: Get traces into your project
HALO needs a body of traces to analyze. There are two ways to get them, and they're not exclusive:
* **Option A, upload the pre-run dataset (fastest).** We already ran the agent across the full query set many times and published the resulting traces. Download once, upload into your project, and you're at HALO in two minutes with no model or search spend.
* **Option B, generate your own.** Run the repo locally to produce fresh traces. Slower and costs a little, but they're *your* runs, and you can keep generating after you change the code.
Most people should start with Option A to feel the loop, then switch to Option B once they want to iterate on the harness.
### Option A: Upload the pre-run dataset
The repo **ships the pre-run dataset with it**, no download needed. We ran the agent across the full query set many times and committed the resulting traces as a gzipped OTLP JSONL file at `data/search-agent-demo-traces.jsonl.gz` (\~20 MB). It's roughly **1,000 traces** with real search variance across the whole query set, far more, and more varied, than you'd want to generate by hand. Because they came from this repo, they already carry the `traceable-search-agent` identity and the same span shape, so they behave exactly like runs you'd produce yourself.
Decompress it first (expands to one \~200 MB JSONL file). The `-k` flag keeps the original `.gz` around:
```bash theme={"system"}
gunzip -k data/search-agent-demo-traces.jsonl.gz
```
That leaves `data/search-agent-demo-traces.jsonl` ready to upload. Two ways:
**From the dashboard.** Open the **Traces** tab, click **imports**, and select the file. The dashboard validates it, processes it, and the traces show up in the table when it's done.
<Frame>
<img alt="The Traces view with the Upload / import dialog for bringing the pre-run trace dataset into the project" />
</Frame>
**From the CLI.** [`inf trace upload`](/cli/traces#inf-trace-upload) does the same thing and waits for processing to finish:
```bash theme={"system"}
inf trace upload ./data/search-agent-demo-traces.jsonl --name search-agent-demo
```
It prints an **upload ID** when it's done. Use it to pull up just this set (note the all-time range, since these traces are timestamped when they were originally generated):
```bash theme={"system"}
inf trace list --range all --filter "trace_import_id=<upload-id>"
```
<Note>
Uploading traces is a **dashboard or CLI** action. The [MCP server](/integrations/mcp-server) reads, exports, and analyzes traces (it doesn't upload them), so do the import here, then drive HALO over the uploaded traces from your coding agent in Steps 5–7.
</Note>
<Warning>
Because the dataset was generated earlier, its traces land at their **original timestamps**, not "now." When you run HALO in Step 5, set the time range wide (or "all time") so it actually covers them. The same goes for finding them in the dashboard: widen the range if the default last-hour view looks empty.
</Warning>
Once the upload finishes, skip ahead to [Step 4](#step-4-see-your-traces-in-the-dashboard). You have everything HALO needs.
### Option B: Generate your own traces
The repo is a **command-line tool**. There's no web UI and no server to start. You run it from the terminal, and every run does two things: it answers the query, and it ships a full [trace](/integrations/traces/overview) to Catalyst. Those traces *are* the data HALO analyzes, so "generating data" just means running the agent a handful of times. You view and analyze it afterward in the [dashboard](https://inference.net/dashboard) (Step 4 on).
Two ways to run it:
* **`search-agent "<query>"`** runs one query, one trace. Good for a smoke test.
* **`search-agent-batch`** runs a slice of the bundled 50-query dataset, one trace per row. This is how you build up enough varied runs for a meaningful HALO report.
Start with a single query to confirm everything's connected. Use mock search for a free smoke test:
<CodeGroup>
```bash Python theme={"system"}
uv run search-agent "What changed in the latest Python release?" --mock-search
```
```bash TypeScript theme={"system"}
bun run search-agent "What changed in the latest Python release?" --mock-search
```
</CodeGroup>
Or run a real search if you set a Tavily key:
<CodeGroup>
```bash Python theme={"system"}
uv run search-agent "What are the latest CISA recommendations for defending against ransomware?"
```
```bash TypeScript theme={"system"}
bun run search-agent "What are the latest CISA recommendations for defending against ransomware?"
```
</CodeGroup>
The command prints the final answer and the trace `session_id`. That same `session_id` shows up in the dashboard so you can find the run.
Now seed a **batch**. HALO works best with repeated behavior across varied tasks, so the repo ships a 50-query starter dataset. Run a slice of it:
<CodeGroup>
```bash Python theme={"system"}
uv run search-agent-batch --limit 5
```
```bash TypeScript theme={"system"}
bun run search-agent-batch --limit 5
```
</CodeGroup>
Each row gets a stable session ID like `dataset-q001` and trace attributes for `demo.query_id`, `demo.category`, and `demo.dataset`, so the runs group and filter cleanly. Run a few different slices to build up variety:
<CodeGroup>
```bash Python theme={"system"}
uv run search-agent-batch --start 20 --limit 5
```
```bash TypeScript theme={"system"}
bun run search-agent-batch --start 20 --limit 5
```
</CodeGroup>
Each command prints its output to the terminal as it goes. There's nothing else to run; once a command finishes, the traces are already in Catalyst.
<Accordion title="Command-line options">
Both commands take the same core flags (drop the `--` value examples in as needed):
| Flag | Applies to | Default | What it does |
| ------------------- | ---------- | ------------------------------- | ------------------------------------------------------------------------------------ |
| `--mock-search` | both | off | Use deterministic local results instead of Tavily (free, no Tavily key, no network). |
| `--limit <n>` | batch | `3` | How many dataset rows to run. |
| `--start <n>` | batch | `0` | Zero-based offset into the dataset, so you can run different slices. |
| `--session-id <id>` | single | random | Stable session ID for grouping a run in the dashboard. |
| `--user-id <id>` | both | `demo-user` / `demo-batch-user` | User ID recorded on the trace (batch defaults to `demo-batch-user`). |
| `--max-turns <n>` | both | `10` | Cap on the agent's tool-calling loop (2–20). |
So a free, no-key run of five dataset rows is `search-agent-batch --limit 5 --mock-search`, and a real-search slice starting at row 20 is `search-agent-batch --start 20 --limit 5`.
</Accordion>
<Tip>
Aim for **at least 20 real-search traces** before you read too much into a HALO report. Mock-search traces are great for verifying instrumentation, but they're too uniform for serious harness analysis. Start with `--limit 3` to control model and search spend, then widen.
</Tip>
## Step 4: See your traces in the dashboard
Open the **Agents** tab in the [dashboard](https://inference.net/dashboard). Whether you uploaded the pre-run dataset or generated your own, the traces roll up under a single agent, **Traceable Search Agent** (`traceable-search-agent`), because every span carries that stable `agentId`.
<Frame>
<img alt="The Traceable Search Agent's traces in the dashboard, one row per run, grouped under the agent" />
</Frame>
Click into the agent and then into any run to open the trace detail. You get the same three views the e2e guide covers in depth: the **trace tree** (every span, with inputs, outputs, cost, and tokens), the **timeline** (what's actually slow), and the **thread** (the whole run as a readable conversation). For this agent you'll see the agent loop, the model calls, the tool calls, and the manual `tavily.search` and `tavily.extract` retriever spans nested underneath.
<Frame>
<img alt="A single run open on the trace detail view, showing the agent loop, tool calls, and nested retriever spans with their inputs and outputs" />
</Frame>
<Tip>
New to the trace views? The [Optimize an Agent End to End](/guides/optimize-an-agent-end-to-end#step-3-explore-a-trace) guide breaks down the tree, timeline, thread, and deep search in detail. Everything there applies here, scoped to this one agent.
</Tip>
## Step 5: Run HALO
Open the agent's **Analysis** sub-tab. This is the [HALO](https://github.com/context-labs/halo) workspace. There's a prompt window on the right with a sensible default, plus a time-range picker and advanced options (span limit, max depth, max turns). Set the range to cover your traces and run it. If you uploaded the pre-run dataset, widen it to **all time**, since those traces keep their original timestamps.
A tighter, more specific prompt gives sharper results. Because this repo has known weak spots, try aiming HALO at them:
```text theme={"system"}
Review the search agent's traces. Where is it wasting tool calls, over-trusting
weak sources, or losing evidence between search and final answer? Rank the issues
by impact and cite the traces.
```
<Frame>
<img alt="The Analysis sub-tab with a prompt entered and HALO just started, the previous runs listed on the left" />
</Frame>
HALO works through the traces and writes a ranked report with citations back to the exact runs each finding came from. Because the repo ships real flaws, you should expect findings in the neighborhood of:
* **Over-trusting the source-quality heuristic.** `assess_source` scores domains with simple rules and can overrate weak institutional pages.
* **A loose scratchpad.** Notes are plain text with no schema, so stale or vague notes leak into the final answer.
* **Shallow claim comparison.** `compare_claims` only checks lexical overlap, so it misses real contradictions.
* **Minimal deduplication.** Repeated near-duplicate sources waste tool calls and bias synthesis.
* **Truncated extraction.** `extract_page` caps content, so key evidence can fall outside the slice.
<Frame>
<img alt="A completed HALO report with ranked findings, recommended fixes, and citations back to the exact traces each finding came from" />
</Frame>
Click any finding's citation to drop straight into the trace that produced it and confirm it matches before you act. And keep chatting: once the report lands you can ask follow-ups in the same thread: "which of these is highest impact?", "show me the cheapest fix for the wasted tool calls."
### Optional: Connect your GitHub repo
You can connect your GitHub repo so HALO sees your actual code alongside your traces. With the source in hand, HALO grounds its findings in the real functions and prompts behind each trace and points to concrete fixes with real code examples, instead of reasoning from the traces alone. Once connected, HALO has access to the repo on every run and every follow-up question in the thread.
For this demo it isn't really necessary; the deliberate flaws show up clearly from the traces. But for a real repo full of agent code it makes a real difference, and we recommend it for sharper results.
<Frame>
<img alt="Connecting a GitHub repo to HALO so it can read the agent's source code alongside the traces" />
</Frame>
## Step 6: Connect the MCP server
This is the part worth doing slowly, because it's where the loop closes itself. Connect the [Inference MCP server](/integrations/mcp-server) to your coding agent so it can read your HALO reports and edit the cloned repo directly.
The fastest setup, for Claude Code:
```bash theme={"system"}
claude mcp add --transport http inference https://mcp.inference.net/mcp \
--header "Authorization: Bearer $INFERENCE_API_KEY"
```
For Cursor or any other MCP client, see the [MCP server guide](/integrations/mcp-server) for the exact config. Use the same API key you put in your `.env`.
Open your coding agent **inside the repo you cloned** so it can both read the HALO report (through the MCP) and edit the agent's code (on disk).
## Step 7: Apply the fix from your editor
Now just ask, in plain language:
```text theme={"system"}
Pull the latest HALO report for the Traceable Search Agent and apply the top fixes
to this repo.
```
Your coding agent resolves the agent name to its identity, fetches the HALO report and its trace citations through the MCP, and edits the relevant files (the prompts in `agent.py` / `agent.ts`, the tool definitions in `tools.py` / `tools.ts`, the harness logic), grounded in the same findings you just read in the dashboard.
That's the moment the loop closes itself: the traces you seeded got analyzed, turned into a ranked to-do list, and applied to the harness from a single line of natural language.
<Tip>
Prefer the terminal? The [`inf halo`](/cli/halo) commands pull the same report from the command line. Read it with `inf halo conversation get <conversation-id>` and pipe it into whatever coding agent you use. And the manual path always works too: copy a recommendation straight out of the report into your editor.
</Tip>
## Step 8: Close the loop
A fix isn't done until you've confirmed it worked, and confirming means new traces from the changed code, so this is the point where everyone generates their own runs, including if you took the upload shortcut in Step 3. The demo repo makes the comparison clean because the query set is fixed: run the same slice before and after and the only variable is your fix. (If you haven't set up the repo yet, do [Steps 1–2](#step-1-clone-the-demo-repo) first.)
1. Run a slice of the bundled queries against your patched code. `--limit 5` runs the first 5 of the 50 queries in `data/queries.jsonl`; keep it the same on both sides of the fix so the comparison is apples to apples:
<CodeGroup>
```bash Python theme={"system"}
uv run search-agent-batch --limit 5
```
```bash TypeScript theme={"system"}
bun run search-agent-batch --limit 5
```
</CodeGroup>
2. Run HALO again over the new window (or set it on a [schedule](/guides/optimize-an-agent-end-to-end#put-halo-on-a-schedule)).
3. Compare the two reports. The findings you fixed should be gone or diminished; pick up the next one.
That's the whole HALO loop, start to finish, on a repo you didn't have to write. Once it clicks here, point the same loop at your own agent.
We hope running it on a sandbox gave you a feel for what HALO can do on your own projects: turn a pile of traces into a ranked, cited list of real problems, and then into fixes, without you reading spans by hand. HALO is fully open source ([`context-labs/halo`](https://github.com/context-labs/halo), MIT licensed). Read the engine and methodology, or self-host it and point it at your own exported traces.
## Where to go next
<CardGroup>
<Card title="Do this on your own agent" icon="arrows-rotate" href="/guides/optimize-an-agent-end-to-end">
Run the same trace → HALO → fix loop against your real app. Start with `inf instrument --mode tracing`.
</Card>
<Card title="Measure agent quality with Signals" icon="signal-stream" href="/guides/measure-agent-quality">
Turn the traces you seeded into quality metrics you can chart, filter, and get alerted on.
</Card>
<Card title="Connect the MCP server" icon="plug" href="/integrations/mcp-server">
Full setup and more example prompts for driving HALO from your coding agent.
</Card>
<Card title="HALO on GitHub" icon="github" href="https://github.com/context-labs/halo">
The open-source HALO engine, methodology, and benchmarks. MIT licensed.
</Card>
</CardGroup>
# ElevenLabs
Source: https://docs.inference.net/integrations/agent-platforms/elevenlabs
Route an ElevenLabs Agent's LLM calls through Inference Catalyst to any supported model — including Anthropic Claude — for full observability.
ElevenLabs Agents lets you point an agent at a Custom LLM that speaks the OpenAI Chat Completions API. The Inference Catalyst gateway accepts that exact format and can forward to any supported provider, so you can run an ElevenLabs agent on Anthropic Claude (or any other model in the catalog) while keeping cost tracking, latency monitoring, and trace capture in one place.
<Frame>
<iframe title="ElevenLabs + Inference Catalyst walkthrough" />
</Frame>
This guide configures an ElevenLabs Agent to call **Claude Sonnet 4.6** through the gateway.
<Info>ElevenLabs always sends OpenAI Chat Completions to the Custom LLM URL. To reach Claude, we route through Anthropic's [OpenAI-compatible endpoint](https://platform.claude.com/docs/en/api/openai-sdk) using the `x-inference-provider-url` header — no schema translation needed on either side.</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Anthropic API key** — from your [Anthropic console](https://console.anthropic.com/settings/keys)
</Step>
<Step title="Open the agent's LLM settings">
In the ElevenLabs dashboard, open your agent and click into the **LLM** panel. Set the LLM to **Custom LLM**.
</Step>
<Step title="Configure the Server URL">
Under **Server URL**, leave the endpoint dropdown on **Chat Completions** and set the base URL to:
```
https://api.inference.net/v1
```
ElevenLabs appends `/chat/completions` automatically, so the final request goes to `https://api.inference.net/v1/chat/completions`.
</Step>
<Step title="Set the Model ID">
In **Model ID**, enter the upstream model you want the gateway to route to:
```
claude-sonnet-4-6
```
You can swap this for any other supported model (e.g. `gpt-4.1`, `claude-opus-4-6`) without changing anything else.
</Step>
<Step title="Add your gateway API key">
Under **API Key**, choose **Secret** → **Add new secret** and paste your **Inference Catalyst project API key**. ElevenLabs sends this as `Authorization: Bearer <key>`, which is how the gateway authenticates the request.
<Warning>Do not paste your Anthropic key here. The Anthropic key is forwarded separately as a request header in the next step.</Warning>
</Step>
<Step title="Add the routing headers">
Under **Request headers**, click **Add header** and add the following entries. `x-inference-provider-url` tells the gateway to forward the chat-completions request to Anthropic's OpenAI-compatible endpoint, and `x-inference-provider-api-key` carries the Anthropic key it should use upstream.
| Header | Value |
| ------------------------------ | --------------------------- |
| `x-inference-provider-url` | `https://api.anthropic.com` |
| `x-inference-provider-api-key` | *your Anthropic API key* |
| `x-inference-environment` | `production` |
For the Anthropic key, click the `{{ }}` icon and reference an environment variable (e.g. `{{ ANTHROPIC_API_KEY }}`) rather than pasting the secret inline.
<Note>The provider URL must be the **bare host** (`https://api.anthropic.com`) — not `https://api.anthropic.com/v1`. The gateway appends `/v1/chat/completions` itself; including `/v1` yields a 404.</Note>
<Warning>Do **not** set `x-inference-provider: anthropic` here. That value targets Anthropic's native `/v1/messages` shape, but ElevenLabs sends chat completions — the override URL above is what makes this work.</Warning>
</Step>
<Step title="Tag requests with a Task ID">
Add one more header so this agent's calls are grouped under a dedicated task in your Catalyst dashboard. Tasks let you split out cost, latency, and traces per use case (e.g. one agent per task, or one task per tool flow).
| Header | Value |
| --------------------- | ------------- |
| `x-inference-task-id` | `voice-agent` |
Pick any stable string — it just needs to stay consistent across requests you want grouped together. If the task ID doesn't already exist in your project, the gateway creates it on first use. To split traffic further (e.g. staging vs. production agents), use distinct IDs like `voice-agent-prod` and `voice-agent-staging` and filter by them in the dashboard.
</Step>
<Step title="Capture ElevenLabs system variables as metadata">
ElevenLabs exposes per-conversation system variables (agent ID, conversation ID, caller ID, etc.) that can be injected into request headers. The Catalyst gateway treats any header prefixed with `x-inference-metadata-` as searchable metadata: the prefix is stripped and the rest becomes a key you can filter on in the dashboard. Combine the two and every LLM call is tagged with the ElevenLabs context it came from.
For each variable you want to capture, click **Add header**, switch the **Type** tab to **Variable**, and configure:
| Header name | Dynamic variable |
| ----------------------------------------- | ---------------------------- |
| `x-inference-metadata-agent-id` | `system__agent_id` |
| `x-inference-metadata-conversation-id` | `system__conversation_id` |
| `x-inference-metadata-caller-id` | `system__caller_id` |
| `x-inference-metadata-called-number` | `system__called_number` |
| `x-inference-metadata-call-sid` | `system__call_sid` |
| `x-inference-metadata-call-duration-secs` | `system__call_duration_secs` |
ElevenLabs substitutes the variable value into the header on every request, and the gateway stores the stripped key (`agent-id`, `conversation-id`, …) against the inference. In the Catalyst dashboard you can then filter by any of these — e.g. pull every LLM call from a single conversation by `conversation-id`, or cost-per-caller by `caller-id` — and use the same keys to slice datasets you build off the traffic.
<Tip>The metadata key is just `<header name minus the prefix>`. Keep it short and lowercase-with-dashes — that's how it'll show up in the dashboard's filter UI. Add only the variables you actually plan to filter or group by; extra metadata is free to send but adds noise.</Tip>
</Step>
<Step title="Test the connection">
Click **Test Connection** at the bottom of the LLM panel. ElevenLabs will issue a probe request through the gateway to Anthropic. A green status confirms the agent can reach Claude Sonnet 4.6 via Catalyst. Save the agent and start a session — the call will appear in your [dashboard](https://inference.net/dashboard) within a few seconds.
</Step>
</Steps>
## Equivalent cURL
If you want to verify the gateway path outside ElevenLabs, this is the same request the agent issues:
```bash theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-url: https://api.anthropic.com" \
-H "x-inference-provider-api-key: $ANTHROPIC_API_KEY" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: voice-agent" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-4-6",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
# LangChain
Source: https://docs.inference.net/integrations/frameworks/langchain
Route supported LangChain OpenAI and Anthropic wrappers through Inference.net while preserving provider routing, environments, and task metadata.
Use this guide when your application talks to models through **LangChain wrappers** instead of the direct provider SDKs.
Inference.net uses the same gateway pattern here as everywhere else: requests go through the Inference.net gateway, the gateway authenticates with `INFERENCE_API_KEY`, and the gateway forwards requests to the downstream provider using the provider key you supply in `x-inference-provider-api-key`.
The important detail is that LangChain's wrapper surfaces are **not identical** to the direct OpenAI and Anthropic SDK constructors. This page documents the wrapper-specific patterns that we validated for Python and TypeScript.
## Supported wrappers in this guide
| Language | Supported now | Notes |
| ---------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Python | `ChatOpenAI`, `ChatAnthropic`, `init_chat_model(..., provider="openai" / "anthropic")` | Validated against the local gateway and observability stack |
| TypeScript | `ChatOpenAI`, `ChatAnthropic` | Validated against the local gateway and observability stack |
## Deferred and unsupported wrappers
These cases are **not** covered in this pass:
* `ChatGoogleGenerativeAI`
* native Google SDKs such as `genai.Client()`
* `ChatNVIDIA`
* config-only model strings in TOML or YAML when there is no concrete wrapper instantiation site to rewrite
## The LangChain-specific differences
Compared with the direct SDK patterns, LangChain changes a few important things:
1. **Anthropic wrapper auth is pragmatic, not pure.**
The LangChain Anthropic wrappers still expect an Anthropic API key field, so the examples below keep `anthropic_api_key` / `apiKey` set to the real Anthropic key for wrapper compatibility. Gateway auth still goes through `Authorization: Bearer <INFERENCE_API_KEY>`, and downstream provider auth still goes through `x-inference-provider-api-key`.
2. **OpenAI-compatible routing still uses the OpenAI wrappers.**
For custom OpenAI-compatible providers, keep the LangChain OpenAI wrapper pointed at the Inference.net gateway and move the original provider URL into `x-inference-provider-url`.
3. **Task IDs are still supported.**
LangChain can pass request-level task IDs through the validated wrappers, but the exact call shape differs by wrapper. The precise task-ID call patterns are documented below.
## Python
### Python `ChatOpenAI`
Use `ChatOpenAI` for direct OpenAI routing by pointing the wrapper at the gateway, authenticating the gateway with `INFERENCE_API_KEY`, and forwarding the downstream OpenAI key in `x-inference-provider-api-key`.
<Metadata />
```python theme={"system"}
import os
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
model="gpt-4o-mini",
temperature=0,
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider": "openai",
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-environment": "production",
},
)
```
### Python `ChatAnthropic`
Use the pragmatic LangChain pattern for Anthropic wrappers:
* keep `anthropic_api_key` set to the real Anthropic key
* point `anthropic_api_url` at the Inference.net gateway
* send gateway auth through `Authorization`
* send downstream provider auth through `x-inference-provider-api-key`
<Metadata />
```python theme={"system"}
import os
from langchain_anthropic import ChatAnthropic
model = ChatAnthropic(
model="claude-sonnet-4-5-20250929",
temperature=0,
anthropic_api_url="https://api.inference.net",
anthropic_api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": os.environ["ANTHROPIC_API_KEY"],
"x-inference-environment": "production",
},
)
```
### Python `init_chat_model`
For the validated OpenAI and Anthropic paths, `init_chat_model` passes the provider-specific kwargs through to the underlying LangChain wrapper.
#### OpenAI via `init_chat_model`
<Metadata />
```python theme={"system"}
import os
from langchain.chat_models import init_chat_model
model = init_chat_model(
model="openai:gpt-4o-mini",
temperature=0,
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider": "openai",
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-environment": "production",
},
)
```
#### Anthropic via `init_chat_model`
<Metadata />
```python theme={"system"}
import os
from langchain.chat_models import init_chat_model
model = init_chat_model(
model="anthropic:claude-sonnet-4-5-20250929",
temperature=0,
anthropic_api_url="https://api.inference.net",
anthropic_api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": os.environ["ANTHROPIC_API_KEY"],
"x-inference-environment": "production",
},
)
```
## TypeScript
### TypeScript `ChatOpenAI`
For LangChain JS/TS, `ChatOpenAI` uses `configuration.baseURL` and `configuration.defaultHeaders` for gateway routing.
<Metadata />
```typescript theme={"system"}
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gpt-4o-mini",
temperature: 0,
apiKey: process.env.INFERENCE_API_KEY!,
configuration: {
baseURL: "https://api.inference.net/v1",
defaultHeaders: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-environment": "production",
},
},
});
```
### TypeScript `ChatAnthropic`
For LangChain JS/TS, `ChatAnthropic` uses `anthropicApiUrl` and `clientOptions.defaultHeaders`.
Use the pragmatic wrapper-compatible pattern:
* keep `apiKey` set to the real Anthropic key
* point `anthropicApiUrl` at the Inference.net gateway
* send gateway auth through `Authorization`
* send downstream provider auth through `x-inference-provider-api-key`
<Metadata />
```typescript theme={"system"}
import { ChatAnthropic } from "@langchain/anthropic";
const model = new ChatAnthropic({
model: "claude-sonnet-4-5-20250929",
temperature: 0,
apiKey: process.env.ANTHROPIC_API_KEY!,
anthropicApiUrl: "https://api.inference.net",
clientOptions: {
defaultHeaders: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY!}`,
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": process.env.ANTHROPIC_API_KEY!,
"x-inference-environment": "production",
},
},
});
```
## Custom OpenAI-compatible providers
When you use a LangChain OpenAI wrapper against Gemini, Together AI, Groq, Fireworks, Mistral, OpenRouter, or another OpenAI-compatible provider, keep the wrapper pointed at the Inference.net gateway and move the original provider URL into `x-inference-provider-url`.
### Python custom provider via `ChatOpenAI`
<Metadata />
```python theme={"system"}
import os
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
model="gemini-2.5-flash",
temperature=0,
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-url": "https://generativelanguage.googleapis.com/v1beta/openai",
"x-inference-provider-api-key": os.environ["GEMINI_API_KEY"],
"x-inference-environment": "production",
},
)
```
### TypeScript custom provider via `ChatOpenAI`
<Metadata />
```typescript theme={"system"}
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gemini-2.5-flash",
temperature: 0,
apiKey: process.env.INFERENCE_API_KEY!,
configuration: {
baseURL: "https://api.inference.net/v1",
defaultHeaders: {
"x-inference-provider-url": "https://generativelanguage.googleapis.com/v1beta/openai",
"x-inference-provider-api-key": process.env.GEMINI_API_KEY!,
"x-inference-environment": "production",
},
},
});
```
When `x-inference-provider-url` is present, you usually do **not** need `x-inference-provider`. The gateway can infer the provider protocol from the overridden URL.
## Task IDs
The wrappers in this guide support request-level task IDs on a **shared client**. You do **not** need to create separate clients per task for the validated wrappers below.
The important detail is that the call shape differs by wrapper.
### Python task IDs
For the validated Python wrappers in this guide, pass task IDs through `extra_headers`.
#### Python `ChatOpenAI` and `ChatAnthropic`
<Metadata />
```python theme={"system"}
def invoke_with_task(model, question: str, task_id: str):
return model.invoke(
question,
extra_headers={"x-inference-task-id": task_id},
)
```
This same `extra_headers` pattern also works for the validated `init_chat_model(..., provider="openai")` and `init_chat_model(..., provider="anthropic")` paths.
### TypeScript task IDs
For TypeScript, the exact call shape depends on the wrapper.
#### TypeScript `ChatAnthropic`
Use `headers` directly in the second argument to `invoke()`.
<Metadata />
```typescript theme={"system"}
import { ChatAnthropic } from "@langchain/anthropic";
async function invokeAnthropicWithTask(
model: ChatAnthropic,
question: string,
taskId: string,
) {
return model.invoke(question, {
headers: { "x-inference-task-id": taskId },
});
}
```
#### TypeScript `ChatOpenAI`
Use `options.headers` in the second argument to `invoke()`.
<Metadata />
```typescript theme={"system"}
import { ChatOpenAI } from "@langchain/openai";
async function invokeOpenAIWithTask(
model: ChatOpenAI,
question: string,
taskId: string,
) {
return model.invoke(question, {
options: {
headers: { "x-inference-task-id": taskId },
},
});
}
```
### Why this matters
LangChain does not normalize transport-level request options across wrappers. That is why the TypeScript OpenAI and Anthropic wrappers use different task-ID call shapes, even though both ultimately set the same `x-inference-task-id` header on the proxied request.
### Framework-owned invocation loops
Some frameworks accept a LangChain model object and then own the actual `.invoke()` / `.ainvoke()` calls internally. In those cases you may not have a wrapper call site where you can attach per-request task IDs yourself.
Use a client-level fallback instead:
#### Python fallback
<Metadata />
```python theme={"system"}
import os
from langchain_anthropic import ChatAnthropic
model = ChatAnthropic(
model="claude-sonnet-4-5-20250929",
temperature=0,
anthropic_api_url="https://api.inference.net",
anthropic_api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": os.environ["ANTHROPIC_API_KEY"],
"x-inference-task-id": "research-agent",
},
)
```
#### TypeScript fallback
<Metadata />
```typescript theme={"system"}
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gpt-4o-mini",
temperature: 0,
apiKey: process.env.INFERENCE_API_KEY!,
configuration: {
baseURL: "https://api.inference.net/v1",
defaultHeaders: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-task-id": "research-agent",
},
},
});
```
This is less granular than a per-request task ID, but it still gives you task grouping when the framework owns the invocation loop.
# Vercel AI SDK
Source: https://docs.inference.net/integrations/frameworks/vercel-ai-sdk
Route Vercel AI SDK traffic through Inference.net using the OpenAI-compatible provider, preserving provider routing, environments, and task metadata.
Use this guide when your application talks to models through the **Vercel AI SDK** (the `ai` package) with `generateText`, `streamText`, `generateObject`, `streamObject`, or `ToolLoopAgent`, instead of the direct provider SDKs.
Inference.net uses the same gateway pattern here as everywhere else: requests go through the Inference.net gateway, the gateway authenticates with `INFERENCE_API_KEY`, and the gateway forwards requests to the downstream provider using the provider key you supply in `x-inference-provider-api-key`.
<Note>
Use the OpenAI-compatible provider (`@ai-sdk/openai-compatible`), **not** the AI SDK Gateway provider (`@ai-sdk/gateway` / `createGateway()`). `createGateway()` speaks Vercel's proprietary AI Gateway protocol and cannot be pointed at an OpenAI-compatible base URL, so it cannot route through Inference.net. `createOpenAICompatible({ baseURL })` sends standard OpenAI-compatible requests that the gateway proxies as-is.
</Note>
## Install
<Metadata />
```bash theme={"system"}
npm install ai @ai-sdk/openai-compatible
```
## Set environment variables
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Provider API key** (in this example, OpenAI) — from your [OpenAI account](https://platform.openai.com/api-keys)
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
## Configure the provider
Point `createOpenAICompatible` at the gateway. Your project API key goes in `apiKey` to authenticate the gateway, and your provider key goes in `x-inference-provider-api-key` so the gateway can forward it downstream.
<Metadata />
```typescript theme={"system"}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const inference = createOpenAICompatible({
name: "inference",
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
headers: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-environment": "production",
},
});
const model = inference("gpt-4.1");
```
`includeUsage: true` is useful because usage metadata is what populates token columns in the dashboard. Some providers only return token counts for non-streaming calls or after a stream finishes.
## Generate text
<Metadata />
```typescript theme={"system"}
import { generateText } from "ai";
const { text } = await generateText({
model,
prompt: "Hello, world!",
});
console.log(text);
```
## Stream text
<Metadata />
```typescript theme={"system"}
import { streamText } from "ai";
const result = streamText({
model,
prompt: "Stream a short sentence about observability.",
});
for await (const chunk of result.textStream) {
process.stdout.write(chunk);
}
```
## Custom OpenAI-compatible providers
To route to Gemini, Together AI, Groq, Fireworks, Mistral, OpenRouter, or another OpenAI-compatible provider, keep the provider pointed at the Inference.net gateway and move the original provider URL into `x-inference-provider-url`.
<Metadata />
```typescript theme={"system"}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const gemini = createOpenAICompatible({
name: "inference-gemini",
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
headers: {
"x-inference-provider-url": "https://generativelanguage.googleapis.com/v1beta/openai",
"x-inference-provider-api-key": process.env.GEMINI_API_KEY!,
"x-inference-environment": "production",
},
});
const model = gemini("gemini-2.5-flash");
```
When `x-inference-provider-url` is present, you usually do **not** need `x-inference-provider`. The gateway can infer the provider protocol from the overridden URL.
## Task IDs and per-request metadata
You can attach routing metadata at two levels:
1. **Provider-level headers** apply to every call made through that provider client. This is the simplest option when a framework owns the invocation loop and you have no per-call site.
2. **Per-request headers** use the `headers` option on `generateText`, `streamText`, `generateObject`, and `streamObject`. These are forwarded to the gateway on that request, so you can add or override routing headers per call while sharing one provider client.
### Provider-level task ID
Add `x-inference-task-id` to the provider `headers` so every request is grouped under the same task.
<Metadata />
```typescript theme={"system"}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const inference = createOpenAICompatible({
name: "inference",
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
headers: {
"x-inference-provider": "openai",
"x-inference-provider-api-key": process.env.OPENAI_API_KEY!,
"x-inference-task-id": "research-agent",
},
});
```
### Per-request task ID
<Metadata />
```typescript theme={"system"}
import { generateText } from "ai";
const { text } = await generateText({
model,
prompt: "Summarize the incident report.",
headers: { "x-inference-task-id": "summarize-incident" },
});
```
The same `headers` option accepts any routing header, including `x-inference-environment` and `x-inference-metadata-*`, so you can tag environment and arbitrary metadata per call.
## Add tracing
<Note>
Gateway captures one record per LLM request through the proxy. To also capture the full agent hierarchy (`generateText` / `streamText` / tool spans) from inside your code, add [Vercel AI SDK tracing](/integrations/traces/ai-sdk). Gateway and tracing are independent and can be used together.
</Note>
## Next steps
<CardGroup>
<Card title="Gateway overview" icon="satellite-dish" href="/integrations/gateway/overview">
Routing headers, supported providers, and the full set of OpenAI-compatible base URLs.
</Card>
<Card title="Vercel AI SDK tracing" icon="chart-network" href="/integrations/traces/ai-sdk">
Capture native AI SDK spans for the full agent and tool-call hierarchy.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
</CardGroup>
# Gateway
Source: https://docs.inference.net/integrations/gateway/overview
Route LLM traffic through the Catalyst Gateway with a one-line base URL change. Capture requests, watch usage, build datasets, and run evals on the same traffic.
Catalyst Gateway is a transparent proxy that sits between your app and your LLM provider. Point your SDK at `https://api.inference.net/v1`, add a couple of headers, and every request is captured with cost, latency, and full request/response payloads. You keep your existing provider API keys. Gateway adds roughly 10ms of overhead and forwards requests as-is.
Use Gateway for inference observability and training. Once traffic is flowing, you can:
* Watch cost, latency, error rates, and token usage in [Metrics Explorer](/platform/gateway/metrics-explorer)
* Browse individual requests in the [Inference Viewer](/platform/gateway/inference-viewer)
* Build [datasets from traffic](/platform/datasets/build-from-traffic) and run [evals](/platform/eval/overview)
* [Fine-tune a task-specific model](/platform/train/overview) on your captured data and [deploy](/platform/deploy/overview) it on a dedicated GPU
<Info>
Gateway and Catalyst Tracing are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For traces, see the [Tracing overview](/integrations/traces/overview).
</Info>
## Getting Started
The fastest path is the [Inference CLI](/cli/overview):
<Metadata />
```bash theme={"system"}
inf instrument --mode gateway
```
`gateway` mode points your existing LLM clients at the Catalyst Gateway. `both` mode does that plus installs the tracing SDK in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See [Install with AI](/integrations/install-with-ai) for the full flow.
Prefer to wire it up by hand? Start with the [Gateway quickstart](/integrations/gateway/quickstart) or pick your provider below.
## Officially Supported Providers
These providers have dedicated guides with copy-paste setup. Any other OpenAI-compatible provider works too via the `x-inference-provider-url` header. See [supported OpenAI-compatible providers](#supported-openai-compatible-provider-urls) below.
* [OpenAI](/integrations/model-providers/openai): Chat Completions and Responses API, including tool calls and structured outputs.
* [Anthropic](/integrations/model-providers/anthropic): Messages API with tool use, prompt caching, and streaming.
* [Vertex AI](/integrations/model-providers/vertex-ai): Google Cloud Vertex AI for Gemini and Anthropic models on GCP.
* [Amazon Bedrock](/integrations/model-providers/amazon-bedrock): Bedrock OpenAI-compatible Chat Completions through AWS-hosted models.
* [Google Gemini](/integrations/model-providers/google-gemini): Native Gemini API and the OpenAI-compatible Gemini endpoint.
* [OpenRouter](/integrations/model-providers/openrouter): Route across many models through OpenRouter's OpenAI-compatible API.
* [Cerebras](/integrations/model-providers/cerebras): Cerebras inference with full request capture.
* [Groq](/integrations/model-providers/groq): Groq's low-latency inference endpoint.
* [LangChain](/integrations/frameworks/langchain): Use Gateway from LangChain by setting the base URL on the chat model.
* [Vercel AI SDK](/integrations/frameworks/vercel-ai-sdk): Route the `ai` package through Gateway with the OpenAI-compatible provider.
* [ElevenLabs](/integrations/agent-platforms/elevenlabs): Route ElevenLabs Agents' LLM calls through Catalyst.
## Routing Headers
| Header | Required | Description |
| ------------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>` authenticates the request to the gateway and links it to your project. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `x-inference-provider-api-key` | Yes | Your provider API key or token, such as OpenAI, Bedrock, Groq, Gemini, or a Google Cloud Vertex credential. The gateway forwards it downstream. For Anthropic's native SDK, use `x-api-key` instead. |
| `x-inference-provider` | No | Forces routing to a specific provider, such as `openai`, `anthropic`, `gemini`, `vertex-ai`, or `cerebras`. Usually inferred from the SDK, path, or `x-inference-provider-url`; set it only to override that inference. |
| `x-inference-environment` | No | Tags requests with an environment, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics. |
| `x-inference-provider-url` | No | Routes to any OpenAI-compatible provider by specifying its base URL. For Vertex native APIs, set this to the global or regional `aiplatform.googleapis.com` base URL. |
## Supported OpenAI-compatible Provider URLs
Any OpenAI-compatible provider can be used via the `x-inference-provider-url` header, even when it does not have a dedicated guide in the catalog yet.
| Provider | Base URL |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OpenAI | `https://api.openai.com/v1` |
| OpenRouter | `https://openrouter.ai/api` |
| Amazon Bedrock | `https://bedrock-mantle.{region}.api.aws/v1` |
| Anthropic | `https://api.anthropic.com/v1` |
| Google Gemini | `https://generativelanguage.googleapis.com` for native Gemini paths; `https://generativelanguage.googleapis.com/v1beta/openai` for OpenAI-compatible calls |
| Vertex AI | `https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/endpoints/openapi` |
| Azure OpenAI | `https://{resource}.openai.azure.com/openai/deployments/{deployment}` |
| Groq | `https://api.groq.com/openai/v1` |
| Together AI | `https://api.together.xyz/v1` |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` |
| Perplexity | `https://api.perplexity.ai` |
| Mistral | `https://api.mistral.ai/v1` |
| DeepSeek | `https://api.deepseek.com/v1` |
| Cerebras | `https://api.cerebras.ai/v1` |
| Inference.net | `https://api.inference.net/v1` |
For Bedrock Runtime's OpenAI-compatible endpoint, use `https://bedrock-runtime.{region}.amazonaws.com/v1` instead of the Bedrock Mantle host. For regional Vertex AI endpoints, use `https://{location}-aiplatform.googleapis.com/v1/projects/{project}/locations/{location}/endpoints/openapi` instead of the global host.
## What Gets Captured
Once traffic is flowing through Gateway, Catalyst records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency, including time to first token (TTFT) and tokens per second
* Token counts (input and output)
* Error rates and status codes
* Model and provider
These metrics are captured automatically without any changes to your request logic. TTFT and tokens-per-second are specifically the metrics that are invisible to application code; only a proxy can measure them.
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/integrations/gateway/quickstart">
Install with AI or wire it up by hand and capture your first LLM call.
</Card>
<Card title="Record your first LLM call" icon="bolt" href="/get-started/record-first-call">
The higher-level Get Started flow with the same setup paths.
</Card>
<Card title="Metrics Explorer" icon="chart-line" href="/platform/gateway/metrics-explorer">
Watch cost, latency, errors, and token usage across all your calls.
</Card>
<Card title="Tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective with `x-inference-task-id`.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
<Card title="Run an eval" icon="flask" href="/platform/eval/overview">
Compare models side by side on your captured traffic.
</Card>
</CardGroup>
# Gateway Quickstart
Source: https://docs.inference.net/integrations/gateway/quickstart
Route your first LLM request through the Catalyst Gateway and see it land in the dashboard.
This page is the gateway-focused quickstart. Point your SDK at `https://api.inference.net/v1`, add a couple of headers, and Catalyst captures every request with cost, latency, and full request/response payloads. If you'd rather see the higher-level Get Started flow, start with [Record your first LLM call](/get-started/record-first-call).
The example below uses OpenAI. For other providers (Anthropic, Vertex AI, Amazon Bedrock, Gemini, OpenRouter, Cerebras, Groq, LangChain, ElevenLabs), see the [Gateway overview](/integrations/gateway/overview#supported-providers).
## Choose a setup path
Installing with AI is the quickest. Use the manual flow if you want to wire it up yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to launch a coding agent like [Claude Code](https://code.claude.com/docs/en/overview), OpenCode, or Codex to scan your codebase, update your LLM clients, and add the routing headers.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run gateway instrumentation in your project">
From your project root, run instrumentation in gateway mode.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project && inf instrument --mode gateway
```
The command guides you through the following workflow:
* Select a coding agent: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain, etc.
* Redirect base URLs to the Catalyst Gateway.
* Add routing headers so requests are authenticated, forwarded, and tagged.
* Add task IDs so each call site is grouped automatically in the dashboard.
* Review the generated changes before applying them.
<Tip>
Pick `both` instead of `gateway` to also install the tracing SDK in the same pass. Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would. Requests now flow through Gateway and appear in the dashboard.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to see request details and analytics.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to wire it up yourself. The example below uses OpenAI. For Anthropic, Vertex AI, Amazon Bedrock, Gemini, OpenRouter, Cerebras, Groq, LangChain, and ElevenLabs, see the per-provider guides linked from the [Gateway overview](/integrations/gateway/overview#supported-providers).
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Provider API key** (in this example, OpenAI) from your [OpenAI account](https://platform.openai.com/api-keys)
Set them as environment variables:
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point your SDK at `https://api.inference.net/v1` and use your Catalyst project API key as the SDK's `apiKey`. Your provider's API key goes in the `x-inference-provider-api-key` header so the gateway can forward it. The gateway adds roughly 10ms of latency and forwards your requests to the provider as-is.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
</Step>
<Step title="Send a request">
Run the snippet above. Once the request completes, Catalyst captures it automatically.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to inspect the request and metrics.
</Step>
</Steps>
<Note>
Need a different provider? See the [Gateway overview](/integrations/gateway/overview#supported-providers) for per-provider guides, or use any OpenAI-compatible endpoint through the `x-inference-provider-url` header.
</Note>
</Tab>
</Tabs>
That's it. Every request now flows through Gateway and gets captured automatically.
## Headers
These headers control routing, authentication, and how the request gets tagged in the dashboard. The only one required for every request is `Authorization`. Add the others as needed.
| Header | Required | Description |
| ------------------------------ | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>`. Authenticates the request to Catalyst and selects the project scope. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `x-inference-provider-api-key` | When proxying a provider | Your downstream provider's API key (OpenAI, Bedrock, Groq, Cerebras, etc.). The gateway forwards it as bearer auth so your code never has to. For Anthropic's native `/v1/messages` route, use `x-api-key` instead. |
| `x-inference-provider` | Optional | Forces routing to a specific provider (`openai`, `anthropic`, `groq`, `cerebras`, `vertex-ai`, `gemini`). When omitted, the gateway infers the provider from the SDK path or base URL. Set this only when you want to override that inference. |
| `x-inference-provider-url` | Optional | Routes to any OpenAI-compatible provider by base URL, even one without a dedicated integration. For third-party OpenAI-compatible URLs, the gateway infers OpenAI automatically. Pair with `x-inference-provider` only when you want to force a specific provider name. |
| `x-inference-environment` | Optional | Tags the request with an environment name like `production`, `staging`, or `development`. Filterable in the dashboard. |
| `x-inference-task-id` | Optional | Groups requests under a logical task such as `summarize-docs` or `chat-support`. Useful for filtering, analytics, and building datasets. |
| `x-inference-metadata-*` | Optional | Attach arbitrary metadata to a request. The `x-inference-metadata-` prefix is stripped to form the key (e.g., `x-inference-metadata-chat-id: abc123` stores `chat-id: abc123`). Filter inferences and create datasets in the dashboard using these keys. |
### Provider base URLs
The base URL you point your SDK at determines which provider the gateway forwards to. Most providers don't need an explicit `x-inference-provider` header, the gateway figures it out from the URL.
| Provider | Base URL | Note |
| -------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| OpenAI | `https://api.openai.com/v1` | Default routing, no provider header needed. |
| OpenRouter | `https://openrouter.ai/api` | No provider header needed unless you want to force `openai`. |
| Amazon Bedrock | `https://bedrock-mantle.{region}.api.aws/v1` | Use a Bedrock bearer/API key in `x-inference-provider-api-key`. |
| Anthropic | `https://api.anthropic.com/v1` | No provider header needed for `/v1/messages` or `api.anthropic.com`. |
| Google Gemini | `https://generativelanguage.googleapis.com` | Use `/v1beta/models/*` native paths, or `/v1beta/openai` for OpenAI-compatible calls. |
| Vertex AI | `https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/endpoints/openapi` | Set `x-inference-provider: vertex-ai`. |
| Azure OpenAI | `https://{resource}.openai.azure.com/openai/deployments/{deployment}` | |
| Groq | `https://api.groq.com/openai/v1` | |
| Together AI | `https://api.together.xyz/v1` | |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` | |
| Perplexity | `https://api.perplexity.ai` | |
| Mistral | `https://api.mistral.ai/v1` | |
| DeepSeek | `https://api.deepseek.com/v1` | |
| Cerebras | `https://api.cerebras.ai/v1` | |
| Inference.net | `https://api.inference.net/v1` | |
## What gets captured
Once traffic is flowing, Catalyst records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency, including time to first token (TTFT) and tokens per second
* Token counts (input and output)
* Error rates and status codes
* Model and provider
## Where to find your data
* **[Metrics Explorer](/platform/gateway/metrics-explorer)** for cost, latency, errors, and usage across all your LLM calls
* **[Inference Viewer](/platform/gateway/inference-viewer)** to browse and filter individual requests and responses
## Next steps
<CardGroup>
<Card title="Gateway overview" icon="satellite-dish" href="/integrations/gateway/overview">
Routing headers, supported providers, and the full set of OpenAI-compatible base URLs.
</Card>
<Card title="Connect more providers" icon="plug" href="/integrations/gateway/overview#supported-providers">
Set up Anthropic, Vertex AI, Amazon Bedrock, Gemini, OpenRouter, Cerebras, Groq, and more.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
</CardGroup>
# Install with AI
Source: https://docs.inference.net/integrations/install-with-ai
Use the Inference CLI and an AI coding agent to instrument your codebase automatically.
The [Inference CLI](/cli/overview) is the fastest way to connect your app to Catalyst. It scans your codebase, finds your LLM clients, and either routes them through the gateway, adds trace collection, or both — guided by an AI coding agent (Claude Code, OpenCode, or Codex).
Want your AI coding assistant to query Catalyst resources directly? Configure the [MCP server](/integrations/mcp-server).
<Info>
Install with AI works with OpenAI, Anthropic, Amazon Bedrock, Gemini, Vertex AI, Groq, Cerebras, OpenRouter, LangChain and more.
</Info>
## What it sets up
When you run `inf instrument`, you'll be asked which Catalyst product to set up:
| Mode | What it does |
| --------- | ------------------------------------------------------------------------------------------------------------- |
| `gateway` | Routes every LLM call through the Catalyst proxy so cost, latency, and usage telemetry land in Observability. |
| `tracing` | Adds the Catalyst tracing SDK and instruments your agents, tools, and provider calls with span trees. |
| `both` | Sets up gateway routing first, then trace collection — in a single agent session. |
Pass `--mode gateway`, `--mode tracing`, or `--mode both` to skip the prompt (useful in CI or scripted runs).
<Steps>
<Step title="Install the CLI">
Install the Inference CLI globally.
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli
```
</Step>
<Step title="Sign in">
Sign in with your Inference account. Your browser will open to authenticate.
<Metadata />
```bash theme={"system"}
inf auth login
```
<Tip>
Running in CI or another headless environment? Use `inf auth set-key` instead of browser login.
</Tip>
</Step>
<Step title="Run instrumentation in your project">
Navigate to your project root and run instrumentation.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project && inf instrument
```
The command guides you through the following workflow:
* Pick what to instrument: gateway, tracing, or both.
* Select a coding agent to use: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain, etc.
* For **gateway**: redirect base URLs to the proxy, add routing and task ID headers.
* For **tracing**: install the Catalyst tracing SDK, initialize it at the app entrypoint, and add spans around agents, tools, and provider calls.
* Review the generated changes before applying them.
<Tip>
Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would to produce inference requests. Requests from your application are now routed through the gateway and will appear in the dashboard.
</Step>
<Step title="Verify it worked">
Open the [dashboard](https://inference.net/dashboard) to see request details, traces, and analytics. You can also verify from the CLI:
<Metadata />
```bash theme={"system"}
inf inference list
```
</Step>
</Steps>
<Tip>
Add `INFERENCE_API_KEY` to your `.env` file so the instrumentation works across environments. Find your key in the [dashboard](https://inference.net/dashboard) under **API Keys**.
</Tip>
## Supported AI coding agents
| Agent | Binary |
| ----------- | ---------- |
| Claude Code | `claude` |
| OpenCode | `opencode` |
| Codex | `codex` |
## Supported providers
**Built-in:** OpenAI, Anthropic
**OpenAI-compatible via `x-inference-provider-url`:** Amazon Bedrock, Google Gemini, Vertex AI, Together AI, Groq, Fireworks AI, Mistral AI, Cerebras, Perplexity, DeepSeek, OpenRouter, Azure OpenAI, and any OpenAI-compatible endpoint.
**Native provider APIs:** Vertex AI native Gemini and Anthropic-on-Vertex are supported through the manual gateway headers documented in the [Vertex AI guide](/integrations/model-providers/vertex-ai).
# MCP Server
Source: https://docs.inference.net/integrations/mcp-server
Connect AI coding assistants to Catalyst resources with your Inference API key.
The Inference MCP server lets compatible AI coding assistants query and operate Catalyst resources from your project. Use it to inspect projects, models, datasets, rubrics, evals, training jobs, deployments, inferences, traces, spans, and HALO agent-trace reports without leaving your MCP client.
The most common workflow is applying HALO fixes: ask your assistant to apply the suggested fixes for an agent, and it pulls the HALO report and edits your code directly. See [Example prompts](#example-prompts).
## How your API key scopes access
MCP authentication uses a project API key. The key is scoped to one or more projects within a team, and the MCP server forwards it to Catalyst APIs, where permissions are enforced. Authentication is by API key only.
Because the key already identifies its project, you rarely need to name one. Every tool that takes a project treats it as optional and resolves it from the key automatically, so prompts like "list recent inferences" or "show the latest HALO runs" work without specifying a project.
* **Single-project key (most common):** all tools target that project. You never pass a project.
* **Multi-project key:** tools use the key's default project unless you name a specific one. Mention the project in your prompt to switch.
* **Switching projects:** to work in a project the key isn't scoped to, use a different key — there is no in-session project switch. If you regularly work across projects, add the MCP server multiple times under different names, each with its own key (e.g. `inference-prod`, `inference-staging`), so all projects stay available at once.
Call the `whoami` tool at any time to see the current team, the projects the key can access, and the project it resolves to by default.
<Info>
**Read vs. write keys.** A read-only key is the safer default: it can browse and read everything but cannot change anything, so the MCP server can never modify your resources. Use a key with **write** access only when you want MCP tools to make changes, such as creating datasets, running evals, launching training jobs, or changing deployments. Read and write scopes are set per project when you create the key.
</Info>
## Configure your client
<Steps>
<Step title="Create or choose a project API key">
Open [API Keys](https://inference.net/dashboard/api-keys) in the dashboard and choose a key scoped to the project you want your MCP client to access.
</Step>
<Step title="Add the MCP server to your tool">
Use `https://mcp.inference.net/mcp` as the server URL and pass your key in the `Authorization` header. Examples for common clients are below.
</Step>
<Step title="Reload and test">
Restart or reload your MCP client, then ask it to run `whoami` to confirm the key is forwarded correctly and to see which project it resolves to.
</Step>
</Steps>
## Claude Code
Set `INFERENCE_API_KEY` in your shell, then add the hosted MCP server:
<Metadata />
```text theme={"system"}
export INFERENCE_API_KEY="<your-project-api-key>"
claude mcp add --transport http inference https://mcp.inference.net/mcp \
--header "Authorization: Bearer $INFERENCE_API_KEY"
```
## Cursor
Add the server to `.cursor/mcp.json` for one project, or `~/.cursor/mcp.json` for all projects.
<Metadata />
```json theme={"system"}
{
"mcpServers": {
"inference": {
"url": "https://mcp.inference.net/mcp",
"headers": {
"Authorization": "Bearer ${env:INFERENCE_API_KEY}"
}
}
}
}
```
## VS Code
Create or edit `.vscode/mcp.json`. VS Code will prompt for the key the first time it starts the server and store it securely.
<Metadata />
```json theme={"system"}
{
"inputs": [
{
"type": "promptString",
"id": "inference-api-key",
"description": "Inference API key",
"password": true
}
],
"servers": {
"inference": {
"type": "http",
"url": "https://mcp.inference.net/mcp",
"headers": {
"Authorization": "Bearer ${input:inference-api-key}"
}
}
}
}
```
## Windsurf
Edit `~/.codeium/windsurf/mcp_config.json` and add the server:
<Metadata />
```json theme={"system"}
{
"mcpServers": {
"inference": {
"serverUrl": "https://mcp.inference.net/mcp",
"headers": {
"Authorization": "Bearer ${env:INFERENCE_API_KEY}"
}
}
}
}
```
## Codex
Set `INFERENCE_API_KEY`, then add the server with the Codex CLI:
<Metadata />
```text theme={"system"}
export INFERENCE_API_KEY="<your-project-api-key>"
codex mcp add inference \
--url https://mcp.inference.net/mcp \
--bearer-token-env-var INFERENCE_API_KEY
```
## OpenCode
Add the server under `mcp` in your OpenCode config:
<Metadata />
```json theme={"system"}
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"inference": {
"type": "remote",
"url": "https://mcp.inference.net/mcp",
"enabled": true,
"headers": {
"Authorization": "Bearer <your-project-api-key>"
}
}
}
}
```
## Gemini CLI
Add the server to `~/.gemini/settings.json` or `.gemini/settings.json`:
<Metadata />
```json theme={"system"}
{
"mcpServers": {
"inference": {
"httpUrl": "https://mcp.inference.net/mcp",
"headers": {
"Authorization": "Bearer <your-project-api-key>"
}
}
}
}
```
## Other clients
Use Streamable HTTP with this URL and header. The client must support custom headers; OAuth-only or SSE-only clients are not supported.
<Metadata />
```text theme={"system"}
URL: https://mcp.inference.net/mcp
Authorization: Bearer <your-project-api-key>
```
## Example prompts
You drive the MCP server in natural language. Your assistant picks the right tools and resolves the project from your key, so you can describe what you want rather than name tools or IDs.
### Apply HALO fixes for an agent
HALO analyzes an agent's traces and writes a report with suggested fixes. To apply them, name the agent in plain language:
```text theme={"system"}
Apply the HALO suggested fixes for the customer-support-agent.
```
Your assistant resolves the agent name to its identity, finds the most recent HALO conversation for it, pulls the report, and applies the changes in your codebase. If you don't name an agent:
```text theme={"system"}
Apply the latest HALO fixes.
```
it lists recent HALO reports for the project and asks which one you mean before editing.
### Run HALO on demand or on a schedule
If there's no recent report, ask your assistant to run HALO now, then apply the result once it finishes:
```text theme={"system"}
Run HALO on the customer-support-agent over the last 24 hours, then apply the fixes.
```
You can also manage recurring HALO schedules:
```text theme={"system"}
Create a daily HALO schedule for the customer-support-agent at 9am UTC.
Pause the HALO schedule for the customer-support-agent.
```
<Warning>
Starting a HALO run and creating a schedule consume credits and compute, and require a write-scoped key. A schedule fires on its cadence until you pause or archive it. Your assistant should confirm the agent, time window, and prompt before starting.
</Warning>
### More examples
```text theme={"system"}
What project is this key scoped to?
List the agents in this project and their execution counts.
Show me the latest HALO runs.
Summarize the errors in the last 50 inferences.
Find the slowest traces from today.
List my training jobs and their status.
Create an eval dataset from successful traffic over the last week.
```
<Note>
Read actions (listing, summarizing, reading reports) work with a read-only key. Write actions (creating datasets, running evals, launching training jobs, changing deployments) need a key with write access.
</Note>
## Troubleshooting
| Error | What it means | How to fix |
| ------------------------ | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `401` | The `Authorization` header is missing or the API key is invalid. | Check that the header is `Authorization: Bearer <your-project-api-key>` and that the key starts with `sk-inference-`. |
| `403` | The key is valid but does not have the permission needed for the tool. | Use a key with the required read or write scope for that action. |
| `Project ID is required` | The key is scoped to multiple projects and has no default, so a tool could not pick one. | Name the project in your prompt, or run `whoami` to see the key's projects. |
| `Project not found` | The requested project is outside the key's scope. | Use a key scoped to that project; a single key cannot reach projects it isn't scoped to. |
<Tip>
Keep project API keys out of source control. Prefer your MCP client's secret storage or environment-variable support when available.
</Tip>
# Amazon Bedrock
Source: https://docs.inference.net/integrations/model-providers/amazon-bedrock
Route Amazon Bedrock model calls through Inference Catalyst for full observability.
Route Amazon Bedrock requests through the Inference Catalyst gateway to get cost tracking, latency monitoring, and analytics. Amazon Bedrock exposes OpenAI-compatible Chat Completions endpoints and Anthropic Messages endpoints on Bedrock Mantle, so you can keep using the OpenAI or Anthropic SDKs with the `x-inference-provider-url` header.
<Info>Prefer automatic setup? Run `inf instrument` to instrument your codebase in seconds. [Learn more](/integrations/install-with-ai)</Info>
<Note>
This guide covers Bedrock Mantle with a Bedrock bearer/API key. Bedrock Runtime
`Converse`, `InvokeModel`, and SigV4-only calls use different request formats
and are not covered by this Gateway setup.
</Note>
## OpenAI-compatible Chat Completions
Use this path for Bedrock models that support the OpenAI-compatible Chat Completions API.
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Amazon Bedrock API key** — from your AWS account. Bedrock also recognizes this as `AWS_BEARER_TOKEN_BEDROCK`.
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-api-key>
export AWS_REGION=us-east-1
export BEDROCK_BASE_URL=https://bedrock-mantle.${AWS_REGION}.api.aws/v1
export BEDROCK_MODEL=openai.gpt-oss-120b
export BEDROCK_ANTHROPIC_BASE_URL=https://bedrock-mantle.${AWS_REGION}.api.aws/anthropic
export BEDROCK_ANTHROPIC_MODEL=anthropic.claude-sonnet-4-6-v1
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and the `x-inference-provider-url` header tells the gateway to forward requests to Amazon Bedrock.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const bedrockApiKey = process.env.AWS_BEARER_TOKEN_BEDROCK;
const bedrockBaseUrl =
process.env.BEDROCK_BASE_URL ?? "https://bedrock-mantle.us-east-1.api.aws/v1";
if (!bedrockApiKey) {
console.log("Set AWS_BEARER_TOKEN_BEDROCK to run this example.");
process.exit(0);
}
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": bedrockApiKey,
"x-inference-provider-url": bedrockBaseUrl,
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: process.env.BEDROCK_MODEL ?? "openai.gpt-oss-120b",
messages: [{ role: "user", content: "Reply with exactly OK." }],
max_tokens: 32,
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
bedrock_api_key = os.getenv("AWS_BEARER_TOKEN_BEDROCK")
bedrock_base_url = os.getenv(
"BEDROCK_BASE_URL", "https://bedrock-mantle.us-east-1.api.aws/v1"
)
if not bedrock_api_key:
print("Set AWS_BEARER_TOKEN_BEDROCK to run this example.")
raise SystemExit(0)
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": bedrock_api_key,
"x-inference-provider-url": bedrock_base_url,
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model=os.getenv("BEDROCK_MODEL", "openai.gpt-oss-120b"),
messages=[{"role": "user", "content": "Reply with exactly OK."}],
max_tokens=32,
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
if [ -z "$AWS_BEARER_TOKEN_BEDROCK" ]; then
echo "Set AWS_BEARER_TOKEN_BEDROCK to run this example."
exit 0
fi
BEDROCK_BASE_URL=${BEDROCK_BASE_URL:-https://bedrock-mantle.us-east-1.api.aws/v1}
BEDROCK_MODEL=${BEDROCK_MODEL:-openai.gpt-oss-120b}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $AWS_BEARER_TOKEN_BEDROCK" \
-H "x-inference-provider-url: $BEDROCK_BASE_URL" \
-H "Content-Type: application/json" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "'"${BEDROCK_MODEL}"'",
"max_tokens": 32,
"messages": [{"role": "user", "content": "Reply with exactly OK."}]
}'
```
</CodeGroup>
</Step>
</Steps>
For the Bedrock Runtime OpenAI-compatible endpoint, set `BEDROCK_BASE_URL` to `https://bedrock-runtime.${AWS_REGION}.amazonaws.com/v1`.
## Anthropic Messages on Bedrock
Use this path for Claude models on Bedrock that support the Anthropic Messages API. The key detail is `x-inference-provider: anthropic`: it tells Catalyst to use Anthropic Messages extraction and forward your Bedrock API key as downstream `x-api-key`.
<Steps>
<Step title="Set environment variables">
Reuse the same Catalyst and Bedrock keys from above, then set the Bedrock Anthropic endpoint and model:
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-api-key>
export AWS_REGION=us-east-1
export BEDROCK_ANTHROPIC_BASE_URL=https://bedrock-mantle.${AWS_REGION}.api.aws/anthropic
export BEDROCK_ANTHROPIC_MODEL=anthropic.claude-sonnet-4-6-v1
```
</Step>
<Step title="Update your code">
Point the Anthropic SDK at the Catalyst gateway. Keep the Bedrock key in the SDK `apiKey` for SDK compatibility, and also pass it as `x-inference-provider-api-key` so Catalyst can forward it to Bedrock as downstream `x-api-key`.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const bedrockApiKey = process.env.AWS_BEARER_TOKEN_BEDROCK;
const bedrockAnthropicBaseUrl =
process.env.BEDROCK_ANTHROPIC_BASE_URL ??
"https://bedrock-mantle.us-east-1.api.aws/anthropic";
if (!bedrockApiKey) {
console.log("Set AWS_BEARER_TOKEN_BEDROCK to run this example.");
process.exit(0);
}
const { default: Anthropic } = await import("@anthropic-ai/sdk");
const client = new Anthropic({
baseURL: "https://api.inference.net",
apiKey: bedrockApiKey,
defaultHeaders: {
"Authorization": `Bearer ${process.env.INFERENCE_API_KEY}`,
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": bedrockApiKey,
"x-inference-provider-url": bedrockAnthropicBaseUrl,
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.messages.create({
model: process.env.BEDROCK_ANTHROPIC_MODEL ?? "anthropic.claude-sonnet-4-6-v1",
max_tokens: 1024,
messages: [{ role: "user", content: "Reply with exactly OK." }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
bedrock_api_key = os.getenv("AWS_BEARER_TOKEN_BEDROCK")
bedrock_anthropic_base_url = os.getenv(
"BEDROCK_ANTHROPIC_BASE_URL",
"https://bedrock-mantle.us-east-1.api.aws/anthropic",
)
if not bedrock_api_key:
print("Set AWS_BEARER_TOKEN_BEDROCK to run this example.")
raise SystemExit(0)
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.inference.net",
api_key=bedrock_api_key,
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-provider-api-key": bedrock_api_key,
"x-inference-provider-url": bedrock_anthropic_base_url,
"x-inference-environment": os.getenv("APP_ENV", "development"),
},
)
response = client.messages.create(
model=os.getenv("BEDROCK_ANTHROPIC_MODEL", "anthropic.claude-sonnet-4-6-v1"),
max_tokens=1024,
messages=[{"role": "user", "content": "Reply with exactly OK."}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
if [ -z "$AWS_BEARER_TOKEN_BEDROCK" ]; then
echo "Set AWS_BEARER_TOKEN_BEDROCK to run this example."
exit 0
fi
BEDROCK_ANTHROPIC_BASE_URL=${BEDROCK_ANTHROPIC_BASE_URL:-https://bedrock-mantle.us-east-1.api.aws/anthropic}
BEDROCK_ANTHROPIC_MODEL=${BEDROCK_ANTHROPIC_MODEL:-anthropic.claude-sonnet-4-6-v1}
curl https://api.inference.net/v1/messages \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider: anthropic" \
-H "x-inference-provider-url: $BEDROCK_ANTHROPIC_BASE_URL" \
-H "x-inference-provider-api-key: $AWS_BEARER_TOKEN_BEDROCK" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "'"${BEDROCK_ANTHROPIC_MODEL}"'",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Reply with exactly OK."}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Anthropic
Source: https://docs.inference.net/integrations/model-providers/anthropic
Route Anthropic requests through Inference Catalyst for full observability.
Route your Anthropic requests through the Inference Catalyst gateway to get cost tracking, latency monitoring, and analytics. Anthropic uses its native SDK and the `/v1/messages` endpoint, not the OpenAI-compatible path.
<Info>Prefer automatic setup? Run `inf instrument` to instrument your codebase in seconds. [Learn more](/integrations/install-with-ai)</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Anthropic API key** — from your [Anthropic console](https://console.anthropic.com/settings/keys)
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export ANTHROPIC_API_KEY=<your-anthropic-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Because the Anthropic SDK sends `apiKey` as the `x-api-key` header, you pass your Anthropic key there and add the Inference project key as an `Authorization` header.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "https://api.inference.net",
apiKey: process.env.ANTHROPIC_API_KEY,
defaultHeaders: {
"Authorization": `Bearer ${process.env.INFERENCE_API_KEY}`,
"x-inference-provider": "anthropic",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.messages.create({
model: "claude-opus-4-6",
max_tokens: 1024,
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.inference.net",
api_key=os.environ["ANTHROPIC_API_KEY"],
default_headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "anthropic",
"x-inference-environment": env = os.getenv("APP_ENV", "development"),
},
)
response = client.messages.create(
model="claude-opus-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/messages \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-H "x-inference-provider: anthropic" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "claude-opus-4-6",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Cerebras
Source: https://docs.inference.net/integrations/model-providers/cerebras
Route Cerebras requests through Inference Catalyst for full observability.
Route your Cerebras requests through the Inference Catalyst gateway to get cost tracking, latency monitoring, and analytics. Cerebras has a dedicated provider routing ID, so you use the OpenAI SDK with `x-inference-provider: cerebras`.
<Info>Prefer automatic setup? Run `inf instrument` to instrument your codebase in seconds. [Learn more](/integrations/install-with-ai)</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Cerebras API key** — from your [Cerebras dashboard](https://cloud.cerebras.ai/)
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export CEREBRAS_API_KEY=<your-cerebras-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and your Cerebras key goes in `x-inference-provider-api-key`.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.CEREBRAS_API_KEY,
"x-inference-provider": "cerebras",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "llama3.1-8b",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["CEREBRAS_API_KEY"],
"x-inference-provider": "cerebras",
"x-inference-environment": env = os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="llama3.1-8b",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $CEREBRAS_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: cerebras" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "llama3.1-8b",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Google Gemini
Source: https://docs.inference.net/integrations/model-providers/google-gemini
Route direct Gemini API calls through Catalyst using native Gemini endpoints.
Route direct Gemini API calls through the Catalyst gateway to get request observability, latency tracking, and persisted request and response payloads. This guide covers Google's Gemini API at `generativelanguage.googleapis.com`, not Vertex AI.
Use the native Gemini paths when you want Google's `generateContent` and `streamGenerateContent` request format:
| Gemini API operation | Catalyst path |
| ------------------------ | ---------------------------------------------- |
| Non-streaming generation | `/v1beta/models/{model}:generateContent` |
| Streaming generation | `/v1beta/models/{model}:streamGenerateContent` |
The gateway defaults these paths to the `gemini` provider. You can still set `x-inference-provider: gemini` explicitly to make routing obvious.
<Info>
Looking for Gemini through Google Cloud Vertex AI instead? Use the [Vertex AI guide](/integrations/model-providers/vertex-ai).
</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Gemini API key** — from [Google AI Studio](https://aistudio.google.com/apikey)
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export GEMINI_API_KEY=<your-gemini-api-key>
export GEMINI_MODEL=gemini-3-flash-preview
```
</Step>
<Step title="Use the Google Gen AI SDK">
The Google Gen AI SDK can point at the Catalyst gateway with `httpOptions.baseUrl`. The SDK sends your Gemini key as `x-goog-api-key`; Catalyst forwards that header downstream and uses `Authorization` for your Catalyst project key.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { GoogleGenAI } from "@google/genai";
const ai = new GoogleGenAI({
apiKey: process.env.GEMINI_API_KEY,
httpOptions: {
baseUrl: "https://api.inference.net",
apiVersion: "v1beta",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"x-inference-provider": "gemini",
"x-inference-environment": "production",
"x-inference-task-id": "gemini-direct",
},
},
});
const response = await ai.models.generateContent({
model: process.env.GEMINI_MODEL ?? "gemini-3-flash-preview",
contents: "Reply with exactly OK.",
config: { maxOutputTokens: 128, temperature: 0 },
});
console.log(response.text);
const stream = await ai.models.generateContentStream({
model: process.env.GEMINI_MODEL ?? "gemini-3-flash-preview",
contents: "Reply with exactly OK.",
config: { maxOutputTokens: 128, temperature: 0 },
});
for await (const chunk of stream) {
process.stdout.write(chunk.text ?? "");
}
```
<Metadata />
```python Python theme={"system"}
import os
from google import genai
from google.genai import types
client = genai.Client(
api_key=os.environ["GEMINI_API_KEY"],
http_options=types.HttpOptions(
base_url="https://api.inference.net",
api_version="v1beta",
headers={
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"x-inference-provider": "gemini",
"x-inference-environment": "production",
"x-inference-task-id": "gemini-direct",
},
),
)
response = client.models.generate_content(
model=os.getenv("GEMINI_MODEL", "gemini-3-flash-preview"),
contents="Reply with exactly OK.",
config=types.GenerateContentConfig(max_output_tokens=128, temperature=0),
)
print(response.text)
for chunk in client.models.generate_content_stream(
model=os.getenv("GEMINI_MODEL", "gemini-3-flash-preview"),
contents="Reply with exactly OK.",
config=types.GenerateContentConfig(max_output_tokens=128, temperature=0),
):
print(chunk.text or "", end="")
```
</CodeGroup>
</Step>
<Step title="Use cURL for raw Gemini paths">
Raw HTTP callers can pass the Gemini key as `x-inference-provider-api-key`. Catalyst converts that to `x-goog-api-key` when forwarding to Gemini.
<CodeGroup>
<Metadata />
```bash cURL theme={"system"}
curl "https://api.inference.net/v1beta/models/${GEMINI_MODEL}:generateContent" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: gemini" \
-H "x-inference-provider-api-key: ${GEMINI_API_KEY}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: gemini-direct" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 128, "temperature": 0 }
}'
```
<Metadata />
```bash cURL theme={"system"}
curl "https://api.inference.net/v1beta/models/${GEMINI_MODEL}:streamGenerateContent?alt=sse" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: gemini" \
-H "x-inference-provider-api-key: ${GEMINI_API_KEY}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: gemini-direct-stream" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 128, "temperature": 0 }
}'
```
</CodeGroup>
</Step>
</Steps>
## Headers
| Header | Required | Description |
| ------------------------------ | ------------ | ---------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>` authenticates the request to Catalyst and links telemetry to your project. |
| `x-inference-provider` | No | Set to `gemini` to make routing explicit. Native Gemini paths default to Gemini when omitted. |
| `x-inference-provider-api-key` | Yes for cURL | Your Gemini API key. Catalyst forwards it to Gemini as `x-goog-api-key`. |
| `x-inference-environment` | No | Tags requests with an environment, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics. |
## Supported paths
Catalyst currently supports the direct Gemini generation paths:
* `/v1beta/models/{model}:generateContent`
* `/v1beta/models/{model}:streamGenerateContent`
* `/v1/models/{model}:generateContent`
* `/v1/models/{model}:streamGenerateContent`
Other Gemini API paths should be called directly until they are explicitly supported by the gateway.
## OpenAI-compatible endpoint
If you would rather use the OpenAI request format (for example, to reuse an existing OpenAI SDK setup), Gemini exposes an OpenAI-compatible surface at `https://generativelanguage.googleapis.com/v1beta/openai`. Catalyst can route to it by combining the OpenAI-format path with a provider URL override:
| Header | Value |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| `x-inference-provider` | `gemini` |
| `x-inference-provider-url` | `https://generativelanguage.googleapis.com/v1beta/openai` |
| `x-inference-provider-api-key` | Your Gemini API key. Catalyst forwards it as `Authorization: Bearer <key>` because the OpenAI-compat endpoint requires bearer auth. |
<Metadata />
```bash cURL theme={"system"}
curl "https://api.inference.net/v1/chat/completions" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: gemini" \
-H "x-inference-provider-url: https://generativelanguage.googleapis.com/v1beta/openai" \
-H "x-inference-provider-api-key: ${GEMINI_API_KEY}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: gemini-openai-compat" \
-d '{
"model": "'"${GEMINI_MODEL}"'",
"messages": [{ "role": "user", "content": "Reply with exactly OK." }],
"max_completion_tokens": 32,
"temperature": 0
}'
```
The native `:generateContent` paths above remain the recommended surface — they expose Gemini features (system instructions, thinking traces, response schemas, image inputs) that the OpenAI-compat shim does not pass through.
# Groq
Source: https://docs.inference.net/integrations/model-providers/groq
Route Groq requests through Inference Catalyst for full observability.
Route your Groq requests through the Inference Catalyst gateway to get cost tracking, latency monitoring, and analytics. Groq is OpenAI-compatible, so you use the OpenAI SDK with the `x-inference-provider-url` header to specify Groq's base URL.
<Info>Prefer automatic setup? Run `inf instrument` to instrument your codebase in seconds. [Learn more](/integrations/install-with-ai)</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **Groq API key** — from your [Groq console](https://console.groq.com/keys)
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export GROQ_API_KEY=<your-groq-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and the `x-inference-provider-url` header tells the gateway to forward requests to Groq.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.GROQ_API_KEY,
"x-inference-provider-url": "https://api.groq.com/openai/v1",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "llama-3.3-70b-versatile",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["GROQ_API_KEY"],
"x-inference-provider-url": "https://api.groq.com/openai/v1",
"x-inference-environment": env = os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="llama-3.3-70b-versatile",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $GROQ_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider-url: https://api.groq.com/openai/v1" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "llama-3.3-70b-versatile",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# OpenAI
Source: https://docs.inference.net/integrations/model-providers/openai
Route OpenAI requests through Inference Catalyst for full observability.
Route your OpenAI requests through the Inference Catalyst gateway to get cost tracking, latency monitoring, and analytics. You keep your existing OpenAI API key — just point the SDK at the gateway and add a few headers.
<Info>Prefer automatic setup? Run `inf instrument` to instrument your codebase in seconds. [Learn more](/integrations/install-with-ai)</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenAI API key** — from your [OpenAI account](https://platform.openai.com/api-keys)
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey` to authenticate with the gateway, and your OpenAI key goes in `x-inference-provider-api-key` so the gateway can forward it to OpenAI.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
"x-inference-environment": env = os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# OpenRouter
Source: https://docs.inference.net/integrations/model-providers/openrouter
Route OpenRouter requests through Inference Catalyst for full observability.
Route your OpenRouter requests through the Inference Catalyst gateway to access hundreds of models while getting full observability. OpenRouter is OpenAI-compatible, so you use the OpenAI SDK with the `x-inference-provider-url` header.
<Info>Prefer automatic setup? Run `inf instrument` to instrument your codebase in seconds. [Learn more](/integrations/install-with-ai)</Info>
## Setup
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenRouter API key** — from your [OpenRouter account](https://openrouter.ai/keys)
</Step>
<Step title="Set environment variables">
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export OPENROUTER_API_KEY=<your-openrouter-api-key>
```
</Step>
<Step title="Update your code">
Point the SDK at the gateway. Your project API key goes in `apiKey`, and the `x-inference-provider-url` header tells the gateway to forward requests to OpenRouter.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENROUTER_API_KEY,
"x-inference-provider-url": "https://openrouter.ai/api",
"x-inference-environment": process.env.NODE_ENV,
},
});
const response = await client.chat.completions.create({
model: "z-ai/glm-5-turbo",
messages: [{ role: "user", content: "Hello" }],
}, {
headers: { "x-inference-task-id": "default" },
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENROUTER_API_KEY"],
"x-inference-provider-url": "https://openrouter.ai/api",
"x-inference-environment": env = os.getenv("APP_ENV", "development"),
},
)
response = client.chat.completions.create(
model="z-ai/glm-5-turbo",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"x-inference-task-id": "default"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider-url: https://openrouter.ai/api" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: default" \
-d '{
"model": "z-ai/glm-5-turbo",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
</CodeGroup>
</Step>
</Steps>
# Vertex AI
Source: https://docs.inference.net/integrations/model-providers/vertex-ai
Route Vertex AI Gemini and Anthropic model calls through Catalyst.
Catalyst can proxy Vertex AI in two modes:
* **OpenAI-compatible Vertex endpoint**: use the OpenAI SDK and set `x-inference-provider-url` to your Vertex `/endpoints/openapi` URL.
* **Native Vertex APIs**: call the Vertex model operation path through the Catalyst gateway and set `x-inference-provider-url` to the global or regional `aiplatform.googleapis.com` base URL.
Use `x-inference-provider: vertex-ai` for both modes so Catalyst applies Vertex-specific URL and authentication handling.
<Info>
This guide is for Gemini and Anthropic through Google Cloud Vertex AI. For
direct Gemini API calls using `generativelanguage.googleapis.com`, see the
[Google Gemini guide](/integrations/model-providers/google-gemini).
</Info>
## Supported Vertex endpoints
| API shape | Catalyst path example | Streaming |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------- |
| OpenAI-compatible Gemini | `/v1/chat/completions` with `x-inference-provider-url: .../endpoints/openapi` | Yes |
| Native Gemini | `/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent` | No |
| Native Gemini stream | `/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:streamGenerateContent` | Yes |
| Anthropic on Vertex | `/v1/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:rawPredict` | No |
| Anthropic on Vertex stream | `/v1/projects/{project}/locations/{location}/publishers/anthropic/models/{model}:streamRawPredict` | Yes |
| Native Gemini with API key | `/v1/publishers/google/models/{model}:generateContent` with `x-inference-provider-api-key: $GEMINI_VERTEX_API_KEY` | No |
## Environment
```bash theme={"system"}
export INFERENCE_API_KEY="<your-catalyst-project-api-key>"
export GOOGLE_CLOUD_PROJECT="<your-gcp-project-id>"
export VERTEX_LOCATION="global"
# For OpenAI-compatible Vertex and Anthropic-on-Vertex.
export VERTEX_AI_ACCESS_TOKEN="$(gcloud auth print-access-token)"
# Optional for native Gemini on Vertex. Google API keys are forwarded as ?key=...
export GEMINI_VERTEX_API_KEY="<your-google-api-key>"
```
For regional Vertex endpoints, use the regional host:
```bash theme={"system"}
export VERTEX_BASE_URL="https://${VERTEX_LOCATION}-aiplatform.googleapis.com"
```
For the `global` location, use:
```bash theme={"system"}
export VERTEX_BASE_URL="https://aiplatform.googleapis.com"
```
## OpenAI-compatible Vertex
Use this path when you want to keep the OpenAI SDK shape for Vertex Gemini models.
```typescript theme={"system"}
import OpenAI from "openai";
const projectId = process.env.GOOGLE_CLOUD_PROJECT!;
const location = process.env.VERTEX_LOCATION ?? "global";
const vertexHost =
location === "global"
? "https://aiplatform.googleapis.com"
: `https://${location}-aiplatform.googleapis.com`;
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider": "vertex-ai",
"x-inference-provider-api-key": process.env.VERTEX_AI_ACCESS_TOKEN!,
"x-inference-provider-url": `${vertexHost}/v1/projects/${projectId}/locations/${location}/endpoints/openapi`,
"x-inference-environment": "production",
},
});
const response = await client.chat.completions.create(
{
model: "google/gemini-2.0-flash-001",
messages: [{ role: "user", content: "Reply with exactly OK." }],
max_tokens: 32,
},
{
headers: { "x-inference-task-id": "vertex-openai-compatible" },
},
);
```
```python theme={"system"}
import os
from openai import OpenAI
project_id = os.environ["GOOGLE_CLOUD_PROJECT"]
location = os.getenv("VERTEX_LOCATION", "global")
vertex_host = (
"https://aiplatform.googleapis.com"
if location == "global"
else f"https://{location}-aiplatform.googleapis.com"
)
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider": "vertex-ai",
"x-inference-provider-api-key": os.environ["VERTEX_AI_ACCESS_TOKEN"],
"x-inference-provider-url": f"{vertex_host}/v1/projects/{project_id}/locations/{location}/endpoints/openapi",
"x-inference-environment": "production",
},
)
response = client.chat.completions.create(
model="google/gemini-2.0-flash-001",
messages=[{"role": "user", "content": "Reply with exactly OK."}],
max_tokens=32,
extra_headers={"x-inference-task-id": "vertex-openai-compatible"},
)
```
## Native Gemini on Vertex
Use native Gemini paths when you need Vertex's `generateContent` or `streamGenerateContent` request format.
```bash theme={"system"}
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/google/models/gemini-2.0-flash-001:generateContent" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-native-gemini" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 64, "temperature": 0 }
}'
```
If you are using a Google API key instead of an OAuth access token for Gemini, pass `GEMINI_VERTEX_API_KEY` as `x-inference-provider-api-key`. Catalyst forwards Google API keys to Vertex as the `key` query parameter.
```bash theme={"system"}
curl "https://api.inference.net/v1/publishers/google/models/gemini-2.0-flash-001:generateContent" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${GEMINI_VERTEX_API_KEY}" \
-H "x-inference-provider-url: https://aiplatform.googleapis.com" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-native-gemini-api-key" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 64, "temperature": 0 }
}'
```
For Gemini streaming, use `:streamGenerateContent`. Add `?alt=sse` if you want Vertex to return server-sent events.
```bash theme={"system"}
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/google/models/gemini-2.0-flash-001:streamGenerateContent?alt=sse" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-native-gemini-stream" \
-d '{
"contents": [
{
"role": "user",
"parts": [{ "text": "Reply with exactly OK." }]
}
],
"generationConfig": { "maxOutputTokens": 64, "temperature": 0 }
}'
```
## Anthropic on Vertex
Anthropic models on Vertex use Vertex operation paths and Anthropic's Vertex payload shape. Use a Google Cloud OAuth access token or service-account-minted access token as `x-inference-provider-api-key`.
```bash theme={"system"}
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/anthropic/models/claude-sonnet-4-5:rawPredict" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-anthropic" \
-d '{
"anthropic_version": "vertex-2023-10-16",
"max_tokens": 512,
"messages": [
{
"role": "user",
"content": "Reply with exactly OK."
}
]
}'
```
For streaming Anthropic responses on Vertex, use `:streamRawPredict` and include `"stream": true` in the body.
```bash theme={"system"}
curl "https://api.inference.net/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${VERTEX_LOCATION}/publishers/anthropic/models/claude-sonnet-4-5:streamRawPredict" \
-H "Authorization: Bearer ${INFERENCE_API_KEY}" \
-H "Content-Type: application/json" \
-H "x-inference-provider: vertex-ai" \
-H "x-inference-provider-api-key: ${VERTEX_AI_ACCESS_TOKEN}" \
-H "x-inference-provider-url: ${VERTEX_BASE_URL}" \
-H "x-inference-environment: production" \
-H "x-inference-task-id: vertex-anthropic-stream" \
-d '{
"anthropic_version": "vertex-2023-10-16",
"max_tokens": 512,
"stream": true,
"messages": [
{
"role": "user",
"content": "Reply with exactly OK."
}
]
}'
```
## Header summary
| Header | Required | Notes |
| ------------------------------ | -------- | -------------------------------------------------------------------- |
| `Authorization` | Yes | Your Catalyst project API key. |
| `x-inference-provider` | Yes | Set to `vertex-ai`. |
| `x-inference-provider-api-key` | Yes | Google API key for native Gemini, or OAuth2 access token for Vertex. |
| `x-inference-provider-url` | Yes | Vertex host or `/endpoints/openapi` URL, depending on the API shape. |
| `x-inference-environment` | No | Dashboard environment tag. |
| `x-inference-task-id` | No | Dashboard task grouping. |
# Integrations
Source: https://docs.inference.net/integrations/overview
Connect Catalyst to your existing tooling, model providers, and frameworks.
Catalyst has three ways to connect with your stack. **Tracing integrations** use a lightweight SDK to collect the full shape of an LLM operation directly from your code, including agent runs, tool calls, framework steps, and spans you add yourself. **Gateway integrations** proxy your existing provider calls through Catalyst with a one-line base URL change, no SDK swap required. **MCP** connects compatible AI coding assistants to Catalyst resources with your project API key.
Browse the documented integrations below. If you do not see a gateway provider yet, Catalyst can still route many OpenAI-compatible endpoints through `x-inference-provider-url`.
**Jump to:** [Tracing integrations](#traces-integrations) · [Gateway integrations](#gateway-integrations) · [Routing headers](#routing-headers) · [OpenAI-compatible providers](#supported-openai-compatible-provider-urls)
## Traces Integrations
Traces integrations use the `@inference/tracing` (TypeScript) or `inference-catalyst-tracing` (Python) SDK to collect OpenInference-shaped spans directly from LLM SDKs, agent frameworks, and your own orchestration code. A single `setup()` call instruments the providers or frameworks you enable. Spans are exported over OTLP and grouped in Catalyst by service, trace, and task.
Use Traces when you need:
* Full agent run trees, not just individual requests
* Tool calls, tool results, and multi-step framework spans
* Visibility into work that never touches the Catalyst gateway (local models, custom routing, non-HTTP orchestration)
<Card title="Traces overview" icon="route" href="/integrations/traces/overview">
Learn what gets captured and how to get started with Catalyst Tracing.
</Card>
### Browse Tracing Integrations
<TracesCatalog />
## Gateway Integrations
Gateway integrations route requests through the Catalyst gateway with a one-line base URL change. You keep your existing provider API keys. Your Catalyst project API key authenticates requests to the gateway, and a small set of headers control routing, environments, and task grouping.
Because requests flow through the gateway, Catalyst can measure performance metrics that are invisible to application code: time to first token (TTFT), tokens per second, and end-to-end latency across providers. These are captured automatically without any changes to your request logic.
<Card title="Gateway overview" icon="satellite-dish" href="/integrations/gateway/overview">
Routing headers, supported providers, and the full Gateway setup reference.
</Card>
### Browse Gateway Integrations
<GatewayCatalog />
## Routing Headers
| Header | Required | Description |
| ------------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <your-project-api-key>` authenticates the request to the gateway and links it to your project. For OpenAI-compatible SDKs, set this as the SDK's `apiKey`. |
| `x-inference-provider-api-key` | Yes | Your provider API key or token, such as OpenAI, Bedrock, Groq, Gemini, or a Google Cloud Vertex credential. The gateway forwards it downstream. For Anthropic's native SDK, use `x-api-key` instead. |
| `x-inference-provider` | No | Forces routing to a specific provider, such as `openai`, `anthropic`, `gemini`, `vertex-ai`, or `cerebras`. Usually inferred from the SDK, path, or `x-inference-provider-url`; set it only to override that inference. |
| `x-inference-environment` | No | Tags requests with an environment, such as `production` or `staging`. |
| `x-inference-task-id` | No | Groups requests under a logical task for filtering and analytics. |
| `x-inference-provider-url` | No | Routes to any OpenAI-compatible provider by specifying its base URL. For Vertex native APIs, set this to the global or regional `aiplatform.googleapis.com` base URL. |
## Supported OpenAI-compatible Provider URLs
Any OpenAI-compatible provider can be used via the `x-inference-provider-url` header, even when it does not have a dedicated guide in the catalog yet.
| Provider | Base URL |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OpenAI | `https://api.openai.com/v1` |
| OpenRouter | `https://openrouter.ai/api` |
| Amazon Bedrock | `https://bedrock-mantle.{region}.api.aws/v1` |
| Anthropic | `https://api.anthropic.com/v1` |
| Google Gemini | `https://generativelanguage.googleapis.com` for native Gemini paths; `https://generativelanguage.googleapis.com/v1beta/openai` for OpenAI-compatible calls |
| Vertex AI | `https://aiplatform.googleapis.com/v1/projects/{project}/locations/global/endpoints/openapi` |
| Azure OpenAI | `https://{resource}.openai.azure.com/openai/deployments/{deployment}` |
| Groq | `https://api.groq.com/openai/v1` |
| Together AI | `https://api.together.xyz/v1` |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` |
| Perplexity | `https://api.perplexity.ai` |
| Mistral | `https://api.mistral.ai/v1` |
| DeepSeek | `https://api.deepseek.com/v1` |
| Cerebras | `https://api.cerebras.ai/v1` |
| Inference.net | `https://api.inference.net/v1` |
For regional Vertex AI endpoints, use `https://{location}-aiplatform.googleapis.com/v1/projects/{project}/locations/{location}/endpoints/openapi` instead of the global host.
# Agent Identity
Source: https://docs.inference.net/integrations/traces/agent-identity
Add stable agent IDs and display names so Catalyst groups agent executions correctly.
Agent dashboards group executions by agent identity. For production agents,
emit a stable `agent.id` and a readable `agent.name`. Keep `agent.id` stable
across deploys, environments, and display-name changes; use `agent.name` as the
human-readable label.
If you only provide `agent.name`, Catalyst can still show observed agents, but
renames and duplicate names are harder to group. Prefer both fields whenever you
control the agent wrapper.
## Agent Span Wrapper
Use `agentSpan()` or `agent_span()` around the product-level agent run. Child
model and tool spans created inside the wrapper inherit the active agent
identity in supported Catalyst tracing integrations.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-refund-1842",
userId: "user_8675309",
role: "refunds",
system: "openai",
},
async (span) => {
const input = "Review refund request #1842.";
span.setInput(input);
const response = await client.responses.create({
model: "gpt-4o-mini",
input,
});
span.setOutput(response.output_text);
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-refund-1842",
user_id="user_8675309",
agent_role="refunds",
system="openai",
) as span:
user_input = "Review refund request #1842."
span.set_input(user_input)
response = client.responses.create(model="gpt-4o-mini", input=user_input)
span.set_output(response.output_text)
tracing.shutdown()
```
</CodeGroup>
## Choosing IDs
Use IDs that are stable, readable, and unique within a project:
| Good | Avoid |
| ---------------------- | ---------------------------------------- |
| `support-triage-agent` | `SupportAgent v2` |
| `refund-review-agent` | random UUID per process |
| `billing-agent-prod` | user-specific or request-specific values |
`agent.role` is optional. Use it when one workflow has multiple agents with
clear responsibilities, such as `triage`, `refunds`, or `billing`.
## Conversation Grouping
`sessionId` / `session_id` is an optional field that emits `session.id` on the
agent span. Use it to group agent runs that belong to the same conversation,
chat thread, or multi-turn request. Pass an ID that is stable for one
conversation and is already available in the calling code — do not treat it as
process-wide setup.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: conversationId,
role: "support",
system: "openai",
},
async (span) => {
// ...
},
);
```
<Metadata />
```python Python theme={"system"}
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id=conversation_id,
agent_role="support",
system="openai",
) as span:
...
```
</CodeGroup>
## User Grouping
`userId` / `user_id` is an optional field that emits `user.id` on the agent
span. The trace dashboard lets you filter to all traces for that
user.
## What Catalyst Uses
Catalyst reads `agent.id` first and falls back to `agent.name` when no stable ID
is present. `session.id` is read separately for conversation grouping, and
`user.id` for filtering traces by user. The raw attributes remain on the span,
and ingestion promotes them into dashboard fields used by the Agents page.
# Vercel AI SDK Traces
Source: https://docs.inference.net/integrations/traces/ai-sdk
Trace Vercel AI SDK generateText, streamText, ToolLoopAgent, tool calls, structured output, and usage through Catalyst.
The Vercel AI SDK emits native OpenTelemetry spans when
`experimental_telemetry` is enabled. Catalyst provides the tracer provider and a
small helper that wires those AI SDK spans into your Catalyst trace export.
Use this guide when your app calls the `ai` package directly through
`generateText`, `streamText`, structured outputs, tools, or `ToolLoopAgent`.
The same setup works with AI SDK providers such as `@ai-sdk/openai`,
`@ai-sdk/anthropic`, and `@ai-sdk/openai-compatible`.
<Note>
For AI SDK v6, including `ai@6.0.138`, prefer the per-call
`createAISdkTelemetrySettings(...)` helper shown below. Catalyst also
registers a module-level telemetry integration for calls that pass
`experimental_telemetry: { isEnabled: true, functionId }`, but AI SDK v6
module lifecycle events do not expose the operation name. The per-call helper
preserves exact `ai.generateText`, `ai.streamText`, and model-step span names.
</Note>
## What Is Captured
* `ai.generateText` operation spans and `ai.generateText.doGenerate` model-step
spans
* `ai.streamText` operation spans and `ai.streamText.doStream` model-step spans
* `ai.toolCall` spans for client-side tool execution
* `ToolLoopAgent.generate()` and `ToolLoopAgent.stream()` activity through the
same native AI SDK spans
* Prompt text or prompt messages, response text, structured output metadata, and
streamed text
* Tool call names, IDs, arguments, and tool results
* Token usage including input, output, total, cached input, and reasoning tokens
when the provider returns them
* `operation.name` values that include your `functionId`
* Custom metadata passed through `experimental_telemetry`
## Install
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing ai
```
Install the provider package you use with the AI SDK:
```bash TypeScript theme={"system"}
bun add @ai-sdk/openai-compatible
```
Other common provider packages include `@ai-sdk/openai`, `@ai-sdk/anthropic`,
and `@ai-sdk/google`.
## Configure Export
Set your Catalyst OTLP endpoint and token in the runtime environment. Short-lived
scripts should also set a stable service name so traces are easy to find.
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
export CATALYST_OTLP_TOKEN="..."
export CATALYST_SERVICE_NAME="ai-sdk-worker"
```
If you route model calls through an OpenAI-compatible gateway, configure the AI
SDK provider separately:
```bash theme={"system"}
export INFERENCE_BASE_URL="https://api.inference.net/v1"
export INFERENCE_API_KEY="..."
export INFERENCE_MODEL="meta-llama/Llama-3.1-8B-Instruct"
```
## Initialize Tracing
Initialize Catalyst tracing before the first AI SDK call. Import the AI SDK
namespace and pass it to `setup()` so auto-detection and integration status can
see the installed module.
<Metadata />
```typescript TypeScript theme={"system"}
import * as ai from "ai";
import { setup } from "@inference/tracing";
import { createAISdkTelemetrySettings } from "@inference/tracing/ai-sdk";
const tracing = await setup({
serviceName: process.env.CATALYST_SERVICE_NAME ?? "ai-sdk-worker",
modules: { aiSdk: ai },
});
const telemetry = (functionId: string) =>
createAISdkTelemetrySettings(tracing.tracer, {
functionId,
metadata: {
route: "support-summary",
environment: process.env.NODE_ENV ?? "development",
},
});
```
Pass `experimental_telemetry: telemetry("...")` on every AI SDK call you want
to trace. The AI SDK does not apply telemetry settings globally.
If `setup({ modules: { aiSdk: ai } })` reports the AI SDK integration as
installed but you do not see spans, verify that your actual `generateText`,
`streamText`, or `ToolLoopAgent` call receives the `experimental_telemetry`
object created by `createAISdkTelemetrySettings(...)`.
## Provider Setup
This example uses an OpenAI-compatible provider, which works with Catalyst
Gateway and other OpenAI-compatible endpoints.
<Metadata />
```typescript TypeScript theme={"system"}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
const provider = createOpenAICompatible({
name: "inference",
baseURL: process.env.INFERENCE_BASE_URL ?? "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
supportsStructuredOutputs: true,
headers: {
"x-inference-environment": "development",
"x-inference-metadata-app": "ai-sdk-worker",
},
});
const model = provider(
process.env.INFERENCE_MODEL ?? "meta-llama/Llama-3.1-8B-Instruct",
);
```
`includeUsage: true` is useful because usage metadata is what populates token
columns in Catalyst. Some providers only return token counts for non-streaming
calls or only after a stream finishes.
## Basic Generation
<Metadata />
```typescript TypeScript theme={"system"}
import { generateText } from "ai";
const result = await generateText({
model,
system: "You answer in one concise sentence.",
prompt: "Summarize why trace trees are useful.",
experimental_telemetry: telemetry("support-summary-generate"),
});
console.log(result.text);
```
Expected spans:
* `ai.generateText`
* `ai.generateText.doGenerate`
Expected promoted fields include `llm_model_name`, `input_tokens`,
`output_tokens`, `total_tokens`, `input`, and `output` when the provider returns
the corresponding AI SDK attributes.
## Streaming
`streamText()` produces a streaming operation span and a model-step span. Consume
the stream before process shutdown so the AI SDK can finish the span and record
the reconstructed response text.
<Metadata />
```typescript TypeScript theme={"system"}
import { streamText } from "ai";
const result = streamText({
model,
prompt: "Stream a six-word sentence about observability.",
experimental_telemetry: telemetry("support-summary-stream"),
});
let text = "";
for await (const chunk of result.textStream) {
text += chunk;
}
console.log(text);
```
Expected spans:
* `ai.streamText`
* `ai.streamText.doStream`
For short-lived scripts, call `await tracing.shutdown()` after the stream is
fully consumed.
## Tool Calling
Tool calls create both model spans and client-side `ai.toolCall` spans. The
model-step span records the tool call requested by the model. The tool span
records your local `execute()` call and its result.
<Metadata />
```typescript TypeScript theme={"system"}
import { generateText, stepCountIs, tool } from "ai";
import { z } from "zod";
const result = await generateText({
model,
prompt: "Use the weather tool for Paris, then summarize the result.",
stopWhen: stepCountIs(2),
tools: {
weather: tool({
description: "Get the current weather for a city.",
inputSchema: z.object({ city: z.string() }),
execute: async ({ city }) => ({
city,
temperatureC: 21,
condition: "clear",
}),
}),
},
toolChoice: { type: "tool", toolName: "weather" },
experimental_telemetry: telemetry("weather-tool-generate"),
});
console.log(result.text);
```
Expected spans:
* `ai.generateText`
* one or more `ai.generateText.doGenerate` model-step spans
* `ai.toolCall` spans for executed tools
Useful raw attributes:
| Attribute | Meaning |
| ----------------------- | -------------------------------------------------------- |
| `ai.response.toolCalls` | Tool calls requested by the model on a model-step span |
| `ai.toolCall.name` | Tool name on the client-side tool span |
| `ai.toolCall.id` | Tool call ID that links model request and tool execution |
| `ai.toolCall.args` | JSON arguments passed to `execute()` |
| `ai.toolCall.result` | JSON result returned by `execute()` |
## Structured Output
Structured outputs are traced through the same `generateText` operation shape.
The response text or parsed output is preserved in AI SDK attributes when the
provider returns it.
<Metadata />
```typescript TypeScript theme={"system"}
import { generateText, Output } from "ai";
import { z } from "zod";
const result = await generateText({
model,
prompt: "Extract city, temperatureC, and condition from: Paris is clear and 21C.",
output: Output.object({
name: "weather_report",
schema: z.object({
city: z.string(),
temperatureC: z.number(),
condition: z.string(),
}),
}),
experimental_telemetry: telemetry("weather-structured-output"),
});
console.log(result.output);
```
Use `supportsStructuredOutputs: true` on OpenAI-compatible providers when the
downstream model endpoint supports native structured outputs.
## Agents
The AI SDK's `ToolLoopAgent` accepts `experimental_telemetry` in the agent
constructor. Agent calls then emit the same native AI SDK spans as core
functions:
* `agent.generate()` emits `ai.generateText` and `ai.generateText.doGenerate`
* `agent.stream()` emits `ai.streamText` and `ai.streamText.doStream`
* agent tool execution emits `ai.toolCall`
There is no separate `ai.agent` span today. Infer the agent loop from the
parent/child relationships, repeated model-step spans, tool-call spans, and the
`functionId` you choose.
For the Agents dashboard, wrap `ToolLoopAgent` calls with `agentSpan()` and use
the same stable agent ID for generate and stream paths.
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
import { ToolLoopAgent, stepCountIs, tool } from "ai";
import { z } from "zod";
const weatherAgent = new ToolLoopAgent({
model,
instructions:
"Use available tools to answer weather questions, then give a concise final answer.",
stopWhen: stepCountIs(2),
tools: {
weather: tool({
description: "Get the current weather for a city.",
inputSchema: z.object({ city: z.string() }),
execute: async ({ city }) => ({
city,
temperatureC: 21,
condition: "clear",
}),
}),
},
toolChoice: { type: "tool", toolName: "weather" },
experimental_telemetry: telemetry("weather-agent-generate"),
});
const answer = await agentSpan(
{
agentId: "weather-agent",
agentName: "Weather Agent",
spanName: "weather-agent.run",
sessionId: "conversation-weather-paris",
role: "weather",
system: "ai-sdk",
},
async (span) => {
const prompt = "Use the weather tool for Paris, then answer in one sentence.";
span.setInput(prompt);
const output = await weatherAgent.generate({ prompt });
span.setOutput(output.text);
return output;
},
);
console.log(answer.text);
console.log(answer.steps.length);
```
## Streaming Agent
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
import { ToolLoopAgent } from "ai";
const streamingAgent = new ToolLoopAgent({
model,
instructions: "Stream concise answers.",
experimental_telemetry: telemetry("weather-agent-stream"),
});
await agentSpan(
{
agentId: "weather-agent",
agentName: "Weather Agent",
spanName: "weather-agent.run",
sessionId: "conversation-weather-paris",
role: "weather",
system: "ai-sdk",
},
async (span) => {
const prompt = "Stream a short weather summary.";
span.setInput(prompt);
const stream = await streamingAgent.stream({ prompt });
let output = "";
for await (const chunk of stream.textStream) {
output += chunk;
process.stdout.write(chunk);
}
span.setOutput(output);
},
);
```
## Next.js Route Handler
For request/response applications, initialize tracing in a module that is loaded
before route handlers call the AI SDK. How you flush depends on where the
handler runs:
* **Long-running Node server** (a container, or `next start` on your own host):
the batch processor flushes on its interval while the process stays up. Do not
flush per request; call `shutdown()` only on `SIGTERM`.
* **Serverless or edge functions** (Vercel, Lambda): the function can freeze
between invocations before the batch flushes, so flush per invocation with
`tracing.provider.forceFlush()`, never `shutdown()`. AI SDK spans only finish
once the stream is fully produced, so flush from the stream's `onFinish` and
`onError` callbacks, not before returning the response. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle).
On Vercel, keep AI SDK trace-emitting route handlers on the Node.js runtime when
using Catalyst's Node OpenTelemetry setup. Vercel also provides `@vercel/otel`
for framework-level OpenTelemetry initialization and custom exporters, but a
manual Catalyst sanity span appearing in the dashboard means your exporter path
is already able to reach Catalyst.
<Metadata />
```typescript TypeScript theme={"system"}
// app/api/chat/route.ts
import * as ai from "ai";
import { streamText } from "ai";
import { setup } from "@inference/tracing";
import { createAISdkTelemetrySettings } from "@inference/tracing/ai-sdk";
const tracing = await setup({
serviceName: "next-ai-sdk-app",
modules: { aiSdk: ai },
});
const telemetry = (functionId: string) =>
createAISdkTelemetrySettings(tracing.tracer, {
functionId,
metadata: { route: "/api/chat" },
});
export const runtime = "nodejs";
export async function POST(request: Request) {
const { prompt } = await request.json();
const result = streamText({
model,
prompt,
experimental_telemetry: telemetry("chat-route-stream"),
});
return result.toUIMessageStreamResponse();
}
```
On Vercel or another serverless runtime, flush once the stream's spans finish so
they are not lost when the function freezes:
<Metadata />
```typescript TypeScript theme={"system"}
export async function POST(request: Request) {
const { prompt } = await request.json();
const result = streamText({
model,
prompt,
experimental_telemetry: telemetry("chat-route-stream"),
// The function can freeze after the response returns, so flush once the
// stream's spans are done. Reuse the provider; never shutdown() here.
onFinish: () => tracing.provider.forceFlush(),
onError: () => tracing.provider.forceFlush(),
});
return result.toUIMessageStreamResponse();
}
```
## Verify Traces
Filter by the service name you configured:
```bash theme={"system"}
inf trace list --range 1h --service ai-sdk-worker --limit 10
```
Look for `ai.generateText`, `ai.streamText`, and `ai.toolCall` spans. If you set
distinct `functionId` values, you can also search for the corresponding
`operation.name` attributes in the trace detail view.
For `ToolLoopAgent` workflows that use `agentSpan()`, also look for the outer
AGENT span with `agent.id` and `agent.name`; the AI SDK operation spans should
appear underneath it.
## Attribute Reference
Catalyst promotes stable AI SDK attributes into canonical columns and preserves
all raw attributes for inspection.
| Catalyst field | AI SDK attribute |
| ------------------- | ------------------------------------------------------ |
| `llm_model_name` | `ai.model.id` |
| `input_tokens` | `ai.usage.inputTokens` or `ai.usage.promptTokens` |
| `output_tokens` | `ai.usage.outputTokens` or `ai.usage.completionTokens` |
| `total_tokens` | `ai.usage.totalTokens` or `ai.usage.tokens` |
| `cache_read_tokens` | `ai.usage.cachedInputTokens` |
| `reasoning_tokens` | `ai.usage.reasoningTokens` |
| `input_messages` | `ai.prompt.messages` |
| `input` | `ai.prompt` |
| `output` | `ai.response.text` |
Observation kinds are inferred from `ai.operationId`:
| `ai.operationId` shape | Observation kind |
| ----------------------------------------------- | ---------------- |
| `ai.generateText`, `ai.generateText.doGenerate` | `LLM` |
| `ai.streamText`, `ai.streamText.doStream` | `LLM` |
| `ai.generateObject`, `ai.streamObject` | `LLM` |
| `ai.toolCall` | `TOOL` |
| `ai.embed`, `ai.embedMany` | `EMBEDDING` |
## Common Gotchas
* Pass `experimental_telemetry` on every AI SDK call or agent you want traced.
* Use a stable `functionId`; it appears in `operation.name` and makes filtering
easier.
* Set `includeUsage: true` on OpenAI-compatible providers when available.
* Fully consume streams before process exit.
* Call `await tracing.shutdown()` in scripts, CLIs, tests, and job workers that
exit after a run.
* Do not call `shutdown()` after each request in a long-running server.
* On serverless or edge (Vercel functions, Lambda), flush per invocation with
`tracing.provider.forceFlush()` instead of `shutdown()`, since the process can
freeze before the batch exports.
* If tool calls appear on model spans but no `ai.toolCall` span appears, confirm
the tool has an `execute()` function and is executed client-side.
# Anthropic Traces
Source: https://docs.inference.net/integrations/traces/anthropic
Trace Anthropic Messages API calls, tool use, and prompt caching.
Catalyst instruments Anthropic Messages API calls in TypeScript and Python. The
span includes content blocks, tool-use blocks, model name, invocation
parameters, finish reason, usage, and prompt-cache token details when Anthropic
returns them.
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing @anthropic-ai/sdk
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[anthropic]'
```
</CodeGroup>
## Basic Messages Call
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import Anthropic from "@anthropic-ai/sdk";
import { setup } from "@inference/tracing";
const tracing = await setup({ modules: { anthropic: Anthropic } });
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const message = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 128,
messages: [{ role: "user", content: "Respond with just the word hello." }],
});
console.log(message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from anthropic import Anthropic
from inference_catalyst_tracing import setup
tracing = setup()
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=128,
messages=[{"role": "user", "content": "Respond with just the word hello."}],
)
print(message.content)
tracing.shutdown()
```
</CodeGroup>
## Anthropic Inside An Agent
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "research-agent",
agentName: "Research Agent",
spanName: "research-agent.run",
sessionId: "conversation-research-notes",
role: "research",
system: "anthropic",
},
async (span) => {
const input = "Summarize the latest customer note.";
span.setInput(input);
const message = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 128,
messages: [{ role: "user", content: input }],
});
span.setOutput(message.content);
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="research-agent",
agent_name="Research Agent",
span_name="research-agent.run",
session_id="conversation-research-notes",
agent_role="research",
system="anthropic",
) as span:
user_input = "Summarize the latest customer note."
span.set_input(user_input)
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=128,
messages=[{"role": "user", "content": user_input}],
)
span.set_output([block.model_dump() for block in message.content])
```
</CodeGroup>
## Tool Use Round Trip
Anthropic tool use is a two-turn pattern: the assistant returns a `tool_use`
block, then your app returns a matching `tool_result` block. Catalyst records
both sides of that relationship on the auto-emitted `LLM` span — what the
model asked for and the result you passed back.
<Tip>
The capture below is the **model-side** view: it shows the request/response
the LLM saw. To also capture the **caller-side** view (the actual function
that ran, its input, output, and duration), wrap the tool function in a
`TOOL` span using
[Manual spans](/integrations/traces/manual-spans#tool-chain-and-retriever-spans).
For a full agent loop, see the
[Production Agent Example](/integrations/traces/production-agent-example).
</Tip>
<Metadata />
```typescript TypeScript theme={"system"}
const tools: Anthropic.Tool[] = [
{
name: "lookup_order",
description: "Look up an order by ID.",
input_schema: {
type: "object",
properties: { orderId: { type: "string" } },
required: ["orderId"],
},
},
];
const messages: Anthropic.MessageParam[] = [
{ role: "user", content: "Check order ABC-123." },
];
const first = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 256,
tools,
messages,
});
const toolUse = first.content.find(
(block): block is Anthropic.ToolUseBlock => block.type === "tool_use",
);
if (toolUse != null) {
messages.push({ role: "assistant", content: first.content });
const args = toolUse.input as { orderId: string };
messages.push({
role: "user",
content: [
{
type: "tool_result",
tool_use_id: toolUse.id,
content: JSON.stringify({ orderId: args.orderId, status: "shipped" }),
},
],
});
const final = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 256,
tools,
messages,
});
console.log(final.content);
}
```
## Prompt Caching
When Anthropic returns cache creation and cache read token counts, Catalyst maps
them into OpenInference token detail attributes.
<Metadata />
```typescript TypeScript theme={"system"}
const longSystem =
"You are a careful, terse assistant. Answer in one sentence.\n\n" +
"Reference document:\n" +
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. ".repeat(300);
const params: Anthropic.MessageCreateParamsNonStreaming = {
model: "claude-haiku-4-5",
max_tokens: 64,
system: [
{
type: "text",
text: longSystem,
cache_control: { type: "ephemeral" },
},
],
messages: [{ role: "user", content: "Is the document about lorem ipsum?" }],
};
const first = await client.messages.create(params);
const second = await client.messages.create(params);
console.log(first.usage.cache_creation_input_tokens ?? 0);
console.log(second.usage.cache_read_input_tokens ?? 0);
```
# Attributes And Span Kinds
Source: https://docs.inference.net/integrations/traces/attributes
Reference for the OpenInference attribute constants and span-kind enum exported by the Catalyst tracing SDKs.
Both Catalyst tracing SDKs export the OpenInference attribute keys and span
kinds they emit. Use them when you author manual spans, when you need to
filter on a specific attribute from the CLI, or when you want to understand
what each span in the trace tree should carry.
`Attr` is not a whitelist. A span is an ordinary OpenTelemetry span, so you can
set any key you like with `setAttribute` (see
[custom attributes](/integrations/traces/manual-spans#adding-custom-attributes)).
The keys below are the ones the dashboard *understands*: set `Attr.MODEL_NAME`
(`llm.model_name`) and it shows the model and computes cost; set
`Attr.SESSION_ID` (`session.id`) and it groups the conversation. `Attr` just
spares you from typing, and mistyping, the wire keys by hand.
The wire-format values are byte-identical to the upstream
[OpenInference](https://github.com/Arize-ai/openinference) semantic conventions,
so OpenInference-aware viewers render Catalyst spans without configuration.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { Attr, SpanKindValues } from "@inference/tracing";
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import Attr, SpanKindValues
```
</CodeGroup>
## Span Kinds
`SpanKindValues` is the canonical set of values for the
`openinference.span.kind` attribute. Pick the kind that best describes the
work the span wraps.
| Kind | Constant | Use For | Required Attributes | Common Optional Attributes |
| ----------- | -------------------------- | ---------------------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `AGENT` | `SpanKindValues.AGENT` | The outer span around an agent run | `SPAN_KIND`, `AGENT_ID` | `AGENT_NAME`, `AGENT_ROLE`, `SESSION_ID`, `USER_ID`, `SYSTEM`, `INPUT_VALUE`, `OUTPUT_VALUE`, `MODEL_NAME`, token counts |
| `LLM` | `SpanKindValues.LLM` | One LLM call (chat, completion, embedding-as-LLM) | `SPAN_KIND`, `MODEL_NAME` | `SYSTEM`, `LLM_SYSTEM`, `LLM_PROVIDER`, `INVOCATION_PARAMETERS`, `STREAMING`, `FINISH_REASON`, message attributes, token counts |
| `TOOL` | `SpanKindValues.TOOL` | A tool invocation by an agent | `SPAN_KIND`, `TOOL_NAME` | `TOOL_CALL_ID`, `INPUT_VALUE`, `OUTPUT_VALUE`, `INPUT_MIME_TYPE`, `OUTPUT_MIME_TYPE` |
| `CHAIN` | `SpanKindValues.CHAIN` | A chain step that is not itself an LLM call (router, postprocessor, planner) | `SPAN_KIND` | `INPUT_VALUE`, `OUTPUT_VALUE`, free-form attributes |
| `RETRIEVER` | `SpanKindValues.RETRIEVER` | A vector-search or document-lookup call | `SPAN_KIND` | `INPUT_VALUE` (the query), `OUTPUT_VALUE` (the results) |
| `EMBEDDING` | `SpanKindValues.EMBEDDING` | An embedding-model call your code makes directly | `SPAN_KIND`, `MODEL_NAME` | `INPUT_VALUE`, token counts |
The patched provider SDKs emit `LLM`-kind spans automatically with all required
and most optional attributes filled in. You only need to author `LLM` spans
manually if you are wrapping an SDK Catalyst does not patch.
## Attribute Constants
All attributes are exported on the `Attr` object. The constant name is the
left-hand column; the wire-format key in the right-hand column is what
actually goes on the span.
### Span Identity
| Constant | Wire key | On |
| ----------------- | ------------------------- | ------------------------------------------- |
| `Attr.SPAN_KIND` | `openinference.span.kind` | Every span |
| `Attr.SESSION_ID` | `session.id` | Any span, used for conversation grouping |
| `Attr.USER_ID` | `user.id` | Any span, used for filtering traces by user |
### Agent Identity
Set on `AGENT` spans, copied onto child `LLM`/`TOOL` spans by the per-SDK
patchers and by `agentSpan()` / `agent_span()` when the child is created in
the active context.
| Constant | Wire key | Purpose |
| ----------------- | ------------ | --------------------------------------------------------- |
| `Attr.AGENT_ID` | `agent.id` | Stable identifier for grouping in the Agents dashboard |
| `Attr.AGENT_NAME` | `agent.name` | Human-readable label |
| `Attr.AGENT_ROLE` | `agent.role` | Role in a multi-agent workflow (e.g. `triage`, `refunds`) |
### Inputs And Outputs
| Constant | Wire key | Notes |
| ----------------------- | ------------------ | ------------------------------------------------- |
| `Attr.INPUT_VALUE` | `input.value` | Stringified input. JSON-encode structured values. |
| `Attr.OUTPUT_VALUE` | `output.value` | Stringified output. |
| `Attr.INPUT_MIME_TYPE` | `input.mime_type` | Typically `"application/json"` or `"text/plain"`. |
| `Attr.OUTPUT_MIME_TYPE` | `output.mime_type` | Same. |
### Model And Provider
Set on `LLM` and `EMBEDDING` spans. Also useful on `AGENT` spans to declare the
provider the agent runs on.
| Constant | Wire key | Notes |
| ---------------------------- | --------------------------- | ---------------------------------------------------------------- |
| `Attr.MODEL_NAME` | `llm.model_name` | Prefer the model the API echoed back. |
| `Attr.SYSTEM` | `gen_ai.system` | Provider identifier on AGENT spans (`"openai"`, `"anthropic"`). |
| `Attr.LLM_SYSTEM` | `llm.system` | Provider identifier on LLM spans. |
| `Attr.LLM_PROVIDER` | `llm.provider` | Alternate provider identifier on LLM spans. |
| `Attr.INVOCATION_PARAMETERS` | `llm.invocation_parameters` | JSON of the request parameters (temperature, max\_tokens, etc.). |
| `Attr.STREAMING` | `llm.streaming` | Boolean. Did the call stream. |
| `Attr.FINISH_REASON` | `llm.finish_reason` | Provider finish reason. |
### Token Usage
| Constant | Wire key |
| --------------------------------------- | ---------------------------------------------- |
| `Attr.TOKEN_COUNT_PROMPT` | `llm.token_count.prompt` |
| `Attr.TOKEN_COUNT_COMPLETION` | `llm.token_count.completion` |
| `Attr.TOKEN_COUNT_TOTAL` | `llm.token_count.total` |
| `Attr.TOKEN_COUNT_PROMPT_CACHE_WRITE` | `llm.token_count.prompt_details.cache_write` |
| `Attr.TOKEN_COUNT_PROMPT_CACHE_READ` | `llm.token_count.prompt_details.cache_read` |
| `Attr.TOKEN_COUNT_COMPLETION_REASONING` | `llm.token_count.completion_details.reasoning` |
The per-SDK patchers fill these in automatically when the provider returns
usage. For manual spans, use the helpers on the
[span handle](/integrations/traces/handle-api):
* TypeScript: `span.recordTokens({ prompt, completion, total })` to pass the
counts directly.
* Python: `span.record_tokens(prompt=..., completion=..., total=...)` for
manual counts, or `span.record_usage(response.usage)` to let the SDK
normalize an OpenAI- or Anthropic-shaped usage object (including cache
fields).
### Tools
Set on `TOOL` spans. Also set on `LLM` spans by the provider patchers when the
LLM emits a `tool_use` block, so the dashboard can show the tool the model
asked for.
| Constant | Wire key |
| ---------------------------- | ----------------------- |
| `Attr.TOOL_NAME` | `tool.name` |
| `Attr.TOOL_CALL_ID` | `tool_call.id` |
| `Attr.AGENT_TOOL_CALL_COUNT` | `agent.tool_call_count` |
| `Attr.AGENT_LLM_CALL_COUNT` | `agent.llm_call_count` |
### Messages
`LLM` spans emit per-message attributes for each input and output message. The
keys are indexed and machine-generated. Use the `OpenInferenceAttribute`
helper to compose them rather than hand-rolling the strings.
<Metadata />
```typescript TypeScript theme={"system"}
import { OpenInferenceAttribute } from "@inference/tracing";
const prefix = OpenInferenceAttribute.inputMessagePrefix(0);
span.setAttribute(OpenInferenceAttribute.role(prefix), "user");
span.setAttribute(OpenInferenceAttribute.content(prefix), "Hello");
```
The wire keys take the form `llm.input_messages.0.message.role`,
`llm.input_messages.0.message.content`, `llm.output_messages.0.message.role`,
and so on. For tool-call messages, the helper also generates
`...tool_calls.0.tool_call.function.name`,
`...tool_calls.0.tool_call.function.arguments`, and
`...tool_calls.0.tool_call.id`.
You rarely need to author these by hand; the per-SDK patchers emit them.
This block is here so the keys are searchable when you are debugging captured
output.
## Using Attributes From The CLI
The same attribute keys are filterable from `inf trace` and `inf span`:
<Metadata />
```bash theme={"system"}
# Find all TOOL spans for one tool name
inf span list --kind TOOL --metadata "tool.name=lookup_order"
# Find traces from one conversation
inf trace list --metadata "session.id=conversation-ticket-123"
# Find all traces for one user
inf trace list --metadata "user.id=user_8675309"
# Find expensive LLM spans, sorted by cost
inf span list --kind LLM --filter "cost_total>0.05" --sort cost_total --order desc
```
See [`inf span`](/cli/spans) and [`inf trace`](/cli/traces) for the full
filter syntax.
## Next Steps
<CardGroup>
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
The typed methods on `agentSpan` / `manual_span` handles that write these attributes for you.
</Card>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Author TOOL, CHAIN, and RETRIEVER spans inside your agent loop.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Pick stable `agent.id` and `session.id` values for the Agents dashboard.
</Card>
<Card title="CLI span reference" icon="terminal" href="/cli/spans">
Filter, search, and inspect spans by attribute from the terminal.
</Card>
</CardGroup>
# Claude Agent SDK Traces
Source: https://docs.inference.net/integrations/traces/claude-agent-sdk
Trace Claude Agent SDK query loops and yielded agent messages.
Use this integration for applications that call the Claude Agent SDK, formerly
the Claude Code SDK. Python can patch `query()` during `setup()` before import.
TypeScript uses an explicit wrapper because ESM namespace bindings cannot be
safely patched in place.
The Claude Agent SDK integration emits an AGENT span for each query loop. To
group those executions under a stable custom ID in the Agents dashboard, wrap
the product-level query with `agentSpan()` / `agent_span()` and use the SDK
wrapper inside it.
If your app shells out to the `claude` binary instead of importing
`@anthropic-ai/claude-agent-sdk` or `claude_agent_sdk`, use
[Claude Code SDK traces](/integrations/traces/claude-code-sdk).
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing @anthropic-ai/claude-agent-sdk
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[claude-agent-sdk]'
```
</CodeGroup>
## TypeScript Wrapped Query
<Metadata />
```typescript TypeScript theme={"system"}
import { query } from "@anthropic-ai/claude-agent-sdk";
import { setup, wrapClaudeAgentSdkQuery } from "@inference/tracing";
const tracing = await setup({ serviceName: "claude-agent" });
const tracedQuery = wrapClaudeAgentSdkQuery(query);
const stream = tracedQuery({
prompt:
"Count the number of files matching *.md under the current directory tree. " +
"Use the Bash tool. Reply with just the integer.",
options: {
maxTurns: 4,
allowedTools: ["Bash"],
permissionMode: "bypassPermissions",
},
});
for await (const message of stream) {
console.log(message);
}
await tracing.shutdown();
```
## Stable Agent Identity
Use a stable `agent.id` for the logical agent or workflow you operate. The
Claude Agent SDK span remains visible inside the trace, and the outer AGENT span
provides the dashboard grouping key.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
const prompt = "Review the current diff and list risky changes.";
await agentSpan(
{
agentId: "claude-review-agent",
agentName: "Claude Review Agent",
spanName: "claude-review.run",
sessionId: "conversation-pr-review-101",
role: "code-review",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const stream = tracedQuery({ prompt, options: { maxTurns: 4 } });
let output = "";
for await (const message of stream) {
output += JSON.stringify(message) + "\n";
}
span.setOutput(output.trim());
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span
prompt = "Review the current diff and list risky changes."
async def traced_review() -> None:
options = ClaudeAgentOptions(max_turns=4)
with agent_span(
tracing.tracer,
agent_id="claude-review-agent",
agent_name="Claude Review Agent",
span_name="claude-review.run",
session_id="conversation-pr-review-101",
agent_role="code-review",
system="anthropic",
) as span:
span.set_input(prompt)
output = []
async for message in query(prompt=prompt, options=options):
output.append(str(message))
span.set_output("\n".join(output))
```
</CodeGroup>
## Python Query Loop
<Metadata />
```python Python theme={"system"}
import asyncio
from inference_catalyst_tracing import setup
tracing = setup(service_name="claude-agent")
from claude_agent_sdk import ClaudeAgentOptions, query
async def main() -> None:
options = ClaudeAgentOptions(
max_turns=4,
allowed_tools=["Bash"],
permission_mode="bypassPermissions",
)
async for message in query(
prompt=(
"Count files matching *.md under the current directory tree. "
"Use the Bash tool. Reply with just the integer."
),
options=options,
):
print(message)
asyncio.run(main())
tracing.shutdown()
```
## What To Look For
* An AGENT span for the query run
* An outer AGENT span with `agent.id` when you add the stable identity wrapper
* Nested LLM turns from the Claude Agent SDK
* Tool-use and tool-result data when built-in tools are used
* Final assistant output captured on the span
# Claude Code SDK Traces
Source: https://docs.inference.net/integrations/traces/claude-code-sdk
Trace Claude Code CLI and SDK-style invocations with OpenInference AGENT spans.
Use this guide when your application invokes Claude Code as a CLI process, such
as `claude -p`, or when you have an existing Claude Code wrapper that shells out
to the `claude` binary. The trace shape is an explicit AGENT span with the
prompt as input and Claude Code output as output.
<Info>
Anthropic renamed the Claude Code SDK to the Claude Agent SDK. If your app
imports `@anthropic-ai/claude-agent-sdk` or `claude_agent_sdk`, use
[Claude Agent SDK traces](/integrations/traces/claude-agent-sdk). If your app
invokes the `claude` binary directly, use the subprocess pattern below.
</Info>
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing
npm install -g @anthropic-ai/claude-code
```
<Metadata />
```bash Python theme={"system"}
pip install inference-catalyst-tracing python-dotenv
npm install -g @anthropic-ai/claude-code
```
</CodeGroup>
## TypeScript CLI Invocation
Resolve the `claude` binary once, then wrap each non-interactive invocation with
`agentSpan()`. Set `CLAUDE_BIN` when Claude Code is installed outside `PATH` or
when you want to pin a specific executable.
<Metadata />
```typescript TypeScript theme={"system"}
import { execFile, execSync } from "node:child_process";
import { promisify } from "node:util";
import { agentSpan, setup } from "@inference/tracing";
const execFileP = promisify(execFile);
function resolveClaudeBin(): string {
if (process.env.CLAUDE_BIN != null && process.env.CLAUDE_BIN !== "") {
return process.env.CLAUDE_BIN;
}
const pathSansNodeModules = (process.env.PATH ?? "")
.split(":")
.filter((pathEntry) => !pathEntry.includes("/node_modules/"))
.join(":");
return execSync("which claude", {
env: { ...process.env, PATH: pathSansNodeModules },
stdio: ["pipe", "pipe", "pipe"],
})
.toString()
.trim();
}
const tracing = await setup({ serviceName: "claude-code-worker" });
const prompt = "Inspect this repository and list the top three TODO comments.";
await agentSpan(
{
agentId: "claude-code-repo-worker",
agentName: "Claude Code",
spanName: "claude-code.invocation",
sessionId: "conversation-cli-hello",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const { stdout } = await execFileP(resolveClaudeBin(), ["-p", prompt], {
encoding: "utf-8",
timeout: 120_000,
});
span.setOutput(stdout.trim());
},
);
await tracing.shutdown();
```
## Python CLI Invocation
Use the same pattern in Python when a worker, job, or automation script shells
out to Claude Code.
<Metadata />
```python Python theme={"system"}
import os
import shutil
import subprocess
import sys
from inference_catalyst_tracing import agent_span, setup
from dotenv import load_dotenv
def resolve_claude_bin() -> str:
override = os.environ.get("CLAUDE_BIN")
if override:
return override
resolved = shutil.which("claude")
if resolved is None:
print("`claude` binary not found on PATH.", file=sys.stderr)
sys.exit(1)
return resolved
load_dotenv()
tracing = setup(service_name="claude-code-worker")
prompt = "Inspect this repository and list the top three TODO comments."
with agent_span(
tracing.tracer,
agent_id="claude-code-repo-worker",
agent_name="Claude Code",
span_name="claude-code.invocation",
session_id="conversation-cli-hello",
system="anthropic",
) as span:
span.set_input(prompt)
completed = subprocess.run(
[resolve_claude_bin(), "-p", prompt],
capture_output=True,
text=True,
timeout=120,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
## Agent SDK Query Loop
If you are using Claude Code programmatically through the Agent SDK, trace the
SDK's `query()` loop instead of tracing a raw subprocess.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { query } from "@anthropic-ai/claude-agent-sdk";
import { setup, wrapClaudeAgentSdkQuery } from "@inference/tracing";
const tracing = await setup({ serviceName: "claude-code-agent" });
const tracedQuery = wrapClaudeAgentSdkQuery(query);
const stream = tracedQuery({
prompt: "Count Markdown files in this repository. Use the Bash tool.",
options: {
maxTurns: 4,
allowedTools: ["Bash"],
permissionMode: "bypassPermissions",
},
});
for await (const message of stream) {
console.log(message);
}
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import asyncio
from inference_catalyst_tracing import setup
tracing = setup(service_name="claude-code-agent")
from claude_agent_sdk import ClaudeAgentOptions, query
async def main() -> None:
options = ClaudeAgentOptions(
max_turns=4,
allowed_tools=["Bash"],
permission_mode="bypassPermissions",
)
async for message in query(
prompt="Count Markdown files in this repository. Use the Bash tool.",
options=options,
):
print(message)
asyncio.run(main())
tracing.shutdown()
```
</CodeGroup>
## What To Look For
* An AGENT span named `claude-code.invocation` for direct CLI subprocess work
* `agent.id=claude-code-repo-worker` for stable Agents dashboard grouping
* `agent.name=ClaudeCode` and `gen_ai.system=anthropic` on the span
* Prompt input and stdout output captured on the span
* Error status and exception details if the subprocess exits non-zero
* Agent SDK query spans when using the `claude_agent_sdk` package
# Cursor SDK Traces
Source: https://docs.inference.net/integrations/traces/cursor-sdk
Trace Cursor agent runs, streamed messages, wait results, and tool calls.
Catalyst instruments the Cursor SDK in TypeScript. Initialize tracing before
calling `Agent.create()` so the `Agent` static methods and the returned
`SDKAgent` / `Run` objects are patched before any run is observed.
Use this guide for Node applications that run programmatic Cursor agents with
`@cursor/sdk`. Cursor publishes a TypeScript SDK only, so there is no Python
equivalent for this integration.
<Info>
Cursor's SDK streams runs over HTTP/2 via `@connectrpc/connect-node`. Run
Catalyst-traced Cursor apps under Node (>= 22). Bun's HTTP/2 client currently
emits `NGHTTP2_FRAME_SIZE_ERROR` mid-stream against `api.cursor.com`.
</Info>
## What Is Captured
* One AGENT span named `Cursor Agent Run` per observed run, started lazily when
`run.stream()`, `run.wait()`, `run.conversation()`, or `run.cancel()` is first
called
* `cursor.agent_id`, `cursor.run_id`, `cursor.run_status`, `cursor.duration_ms`,
request IDs, and model metadata (`llm.model_name`,
`llm.invocation_parameters`)
* `input.value` from `agent.send()` and `output.value` from streamed assistant
text, `run.wait()` results, or `run.conversation()` turns
* TOOL child spans for streamed `tool_call` events, including `tool.name`,
`tool_call.id`, JSON arguments, and tool results
* Aggregate counts: `agent.tool_call_count`, `agent.llm_call_count`
* Error status and exception details when run streaming, waiting, or cancellation
fails
## Install
<CodeGroup>
```bash TypeScript theme={"system"}
bun add @inference/tracing @cursor/sdk
```
</CodeGroup>
## TypeScript Cursor Agent Run
Initialize tracing before creating Cursor agents. Auto-instrumentation detects
`@cursor/sdk` when it is installed in the project, so the smoothest path is just
`setup()`.
<Metadata />
```typescript TypeScript theme={"system"}
import { Agent } from "@cursor/sdk";
import { setup } from "@inference/tracing";
const tracing = await setup({
serviceName: "cursor-agent-runner",
});
const agent = await Agent.create({
apiKey: process.env.CURSOR_API_KEY!,
model: { id: "composer-2" },
local: { cwd: process.cwd() },
});
try {
const run = await agent.send("Summarize what this repository does");
for await (const event of run.stream()) {
console.log(event.type);
}
const result = await run.wait();
console.log(`run ${run.id} ${result.status}`);
} finally {
await agent[Symbol.asyncDispose]();
await tracing.shutdown();
}
```
For manual initialization, pass the SDK namespace into the granular entry point:
```typescript TypeScript theme={"system"}
import * as CursorSdk from "@cursor/sdk";
import { setup } from "@inference/tracing";
import { instrumentCursorSdk } from "@inference/tracing/cursor-sdk";
const tracing = await setup({ autoInstrument: false });
instrumentCursorSdk(CursorSdk, tracing);
```
## Lazy Span Lifecycle
The AGENT span starts when the application first observes a run through
`run.stream()`, `run.wait()`, `run.conversation()`, or `run.cancel()`, not when
`agent.send()` returns. This avoids leaking open spans for fire-and-handoff
workflows where one process kicks off a Cursor run and another process observes
it later. Each unique observed run produces one AGENT span; observing the same
run twice in the same process does not double-emit.
## Stable Agent Identity
Cursor's SDK exposes its own run and agent identifiers, which Catalyst preserves
as `cursor.run_id` and `cursor.agent_id`. For the canonical Agents dashboard
grouping key, wrap the Cursor run you operate with `agentSpan()` and pass your
stable product ID as `agentId`.
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "cursor-repo-maintainer",
agentName: "Cursor Repo Maintainer",
spanName: "cursor-repo-maintainer.run",
sessionId: "conversation-repo-tour",
role: "code-maintenance",
system: "cursor",
},
async (span) => {
const input = "Summarize what this repository does";
span.setInput(input);
const run = await agent.send(input);
const result = await run.wait();
span.setOutput(result);
},
);
```
## Verify In Catalyst
Filter traces by your `service.name` (for example `cursor-agent-runner`). A
successful run should show one `Cursor Agent Run` AGENT span with `cursor.run_id`
and the model name, plus nested TOOL spans for each tool call the agent
produced.
For short-lived scripts, always call `tracing.shutdown()` before process exit so
batched spans are flushed to Catalyst.
# ElevenLabs Agents Traces
Source: https://docs.inference.net/integrations/traces/elevenlabs
Trace ElevenLabs Agents conversation sessions, transcripts, and client tool calls.
Catalyst instruments the ElevenLabs Agents SDK in TypeScript and Python.
Initialize tracing before starting a conversation so the SDK conversation
lifecycle and client-tool callbacks are patched before the session begins.
Use this guide for applications that use `@elevenlabs/client` or the Python
`elevenlabs.conversational_ai.conversation` APIs.
## What Is Captured
* One AGENT span named `ElevenLabs Conversation` for each conversation session
* User and agent transcript messages as OpenInference message attributes
* ElevenLabs metadata such as agent ID, conversation ID, user ID, auth mode, and
text-only mode
* TOOL child spans for registered client tools, including tool name, arguments,
result, and errors
* Error status and exception details when session startup, tool execution, or
shutdown fails
## Install
<CodeGroup>
```bash TypeScript theme={"system"}
bun add @inference/tracing @elevenlabs/client
```
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[elevenlabs]'
```
</CodeGroup>
## TypeScript Conversation Session
Pass the ElevenLabs SDK namespace into `setup()` before calling
`Conversation.startSession()`.
```typescript TypeScript theme={"system"}
import * as ElevenLabs from "@elevenlabs/client";
import { Conversation } from "@elevenlabs/client";
import { setup } from "@inference/tracing";
const tracing = await setup({
serviceName: "voice-support",
modules: { elevenlabs: ElevenLabs },
});
const conversation = await Conversation.startSession({
agentId: process.env.ELEVENLABS_AGENT_ID!,
textOnly: true,
userId: "user_123",
clientTools: {
lookupAppointment: async ({ user_id }) => {
return JSON.stringify({
user_id,
starts_at: "2026-04-29T10:30:00Z",
});
},
},
onMessage: (message) => {
console.log(message.role, message.message);
},
});
conversation.sendUserMessage("When is my next appointment?");
await conversation.endSession();
await tracing.shutdown();
```
For private agents, use the same tracing setup and pass the `signedUrl` or
`conversationToken` options that your ElevenLabs app already uses. Catalyst
keeps the same outer `ElevenLabs Conversation` span shape.
## Python Conversation Session
Call `setup()` before constructing and starting the `Conversation`. This patches
the installed ElevenLabs SDK and records the session lifecycle automatically.
```python Python theme={"system"}
import os
from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import ClientTools, Conversation
from inference_catalyst_tracing import setup
tracing = setup(service_name="voice-support")
client = ElevenLabs(api_key=os.environ.get("ELEVENLABS_API_KEY"))
client_tools = ClientTools()
client_tools.register(
"lookupAppointment",
lambda params: {
"user_id": params["user_id"],
"starts_at": "2026-04-29T10:30:00Z",
},
)
conversation = Conversation(
client,
os.environ["ELEVENLABS_AGENT_ID"],
user_id="user_123",
requires_auth=bool(os.environ.get("ELEVENLABS_API_KEY")),
audio_interface=None,
client_tools=client_tools,
callback_agent_response=lambda text: print(f"Agent: {text}"),
callback_user_transcript=lambda text: print(f"User: {text}"),
)
conversation.start_session()
conversation.send_user_message("When is my next appointment?")
conversation.end_session()
conversation_id = conversation.wait_for_session_end()
print(f"Conversation ID: {conversation_id}")
tracing.shutdown()
```
`audio_interface=None` keeps the example text-only. For voice conversations, use
your normal ElevenLabs audio interface; the instrumentation records the same
conversation span and transcript callbacks.
## Stable Agent Identity
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "voice-support-agent",
agentName: "Voice Support Agent",
spanName: "voice-support.run",
sessionId: "conversation-voice-1",
role: "support",
system: "elevenlabs",
},
async (span) => {
span.setInput("When is my next appointment?");
const conversation = await Conversation.startSession({
agentId: process.env.ELEVENLABS_AGENT_ID!,
textOnly: true,
userId: "user_123",
});
conversation.sendUserMessage("When is my next appointment?");
await conversation.endSession();
span.setOutput("conversation ended");
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="voice-support-agent",
agent_name="Voice Support Agent",
span_name="voice-support.run",
session_id="conversation-voice-1",
agent_role="support",
system="elevenlabs",
) as span:
span.set_input("When is my next appointment?")
conversation.start_session()
conversation.send_user_message("When is my next appointment?")
conversation.end_session()
conversation_id = conversation.wait_for_session_end()
span.set_output({"conversation_id": conversation_id})
```
</CodeGroup>
## Verify In Catalyst
Filter traces by `service.name=voice-support`. A successful session should show
an `ElevenLabs Conversation` AGENT span with nested TOOL spans when your agent
calls `lookupAppointment`. When you add the wrapper above, the trace also has an
outer AGENT span with your stable `agent.id`.
For short-lived scripts, always call `tracing.shutdown()` before process exit so
batched spans are flushed to Catalyst.
# Vercel Eve Traces
Source: https://docs.inference.net/integrations/traces/eve
Trace Eve agent turns, model calls, tool execution, session lineage, and AI SDK v7 telemetry through Catalyst.
Catalyst traces Eve through Eve's native `agent/instrumentation.ts` hook. Eve
already emits OpenTelemetry spans for agent turns, model calls, sub-agent
invocations, and tools; Catalyst installs its OpenTelemetry provider from that
hook and enriches the exported spans with OpenInference attributes and `$eve.*`
workflow tags.
Use this guide for TypeScript Eve agents. If your app calls the Vercel AI SDK
directly outside Eve, use [Vercel AI SDK traces](/integrations/traces/ai-sdk)
for those direct `generateText` or `streamText` calls.
<Info>
Eve uses the presence of `agent/instrumentation.ts` as the telemetry
enablement signal. For Eve apps, export `defineCatalystEveInstrumentation()`
from that file instead of calling `setup()` in your agent code.
</Info>
## What Is Captured
* Eve turn spans such as `ai.eve.turn` as OpenInference CHAIN spans
* Eve `invoke_agent` spans as AGENT spans
* AI SDK v7 model spans inside Eve as LLM spans, including model, provider,
input/output messages, finish reason, and token usage when the provider
returns them
* Eve `execute_tool` spans as TOOL spans, including tool name, call ID,
arguments, result, and errors when available
* Eve session, turn, parent, root, and trigger metadata
* `$eve.*` aggregate fields such as model, input tokens, output tokens, cache
tokens, and tool count
* Custom runtime metadata added from `defineCatalystEveInstrumentation()`
Catalyst sets `recordInputs` and `recordOutputs` to `true` by default so the
dashboard can show model and tool IO. Set either option to `false` when a
deployment should avoid exporting full prompts, responses, tool arguments, or
tool results.
## Install
Install Catalyst tracing in the same package where your Eve agent runs.
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing eve
```
Install the AI SDK provider package your Eve agent uses. For Catalyst Gateway or
another OpenAI-compatible endpoint:
<Metadata />
```bash TypeScript theme={"system"}
bun add @ai-sdk/openai-compatible
```
## Configure Export
Set the Catalyst traces endpoint and token before the Eve process starts.
<Metadata />
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="eve-weather-agent"
export CATALYST_SERVICE_VERSION="2026.06.17"
```
If your Eve model calls go through Catalyst Gateway, configure that provider
separately:
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY="<your-api-key>"
export INFERENCE_BASE_URL="https://api.inference.net/v1"
export INFERENCE_MODEL="claude-haiku-4-5"
```
## Add Eve Instrumentation
Create `agent/instrumentation.ts` at the root of your Eve agent. Eve loads this
file during agent startup.
<Metadata />
```typescript TypeScript theme={"system"}
import { defineCatalystEveInstrumentation } from "@inference/tracing/eve";
export default defineCatalystEveInstrumentation({
functionId: "weather-agent",
serviceName: "eve-weather-agent",
metadata: {
"deployment.environment": process.env.NODE_ENV ?? "development",
},
});
```
`functionId` becomes Eve's AI SDK telemetry function ID. Use a stable value for
the logical agent or workflow. `serviceName` becomes the OpenTelemetry
`service.name` resource attribute; when omitted, Catalyst uses Eve's agent name.
## Agent Provider Example
Your `agent/agent.ts` keeps using Eve normally. This example uses an
OpenAI-compatible AI SDK provider pointed at Catalyst Gateway.
<Metadata />
```typescript TypeScript theme={"system"}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { defineAgent } from "eve";
const inference = createOpenAICompatible({
name: "inference.net",
baseURL: process.env.INFERENCE_BASE_URL ?? "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY!,
includeUsage: true,
});
export default defineAgent({
model: inference(process.env.INFERENCE_MODEL ?? "claude-haiku-4-5"),
modelContextWindowTokens: 200_000,
});
```
`includeUsage: true` lets the provider return token counts for Catalyst columns.
`modelContextWindowTokens` is useful when Eve cannot infer context-window
metadata from a custom AI SDK provider model.
## Tool Spans
Eve tools are captured automatically when the runtime emits `execute_tool`
spans. You do not need to wrap the tool manually.
<Metadata />
```typescript TypeScript theme={"system"}
import { defineTool } from "eve/tools";
import { never } from "eve/tools/approval";
import { z } from "zod";
export default defineTool({
needsApproval: never(),
description: "Get the current weather for a city.",
inputSchema: z.object({
city: z.string(),
}),
async execute(input) {
return {
city: input.city,
temperatureF: 72,
condition: "Sunny",
summary: `Sunny in ${input.city} with a light breeze.`,
};
},
});
```
When this tool runs, Catalyst records a TOOL span with `tool.name`,
`tool_call.id`, `input.value`, and `output.value`.
## Existing Eve Hooks
If you already use Eve instrumentation events, pass them into
`defineCatalystEveInstrumentation()`. Catalyst composes your `step.started`
handler with its own handler and merges the returned runtime context.
<Metadata />
```typescript TypeScript theme={"system"}
import { defineCatalystEveInstrumentation } from "@inference/tracing/eve";
export default defineCatalystEveInstrumentation({
functionId: "support-agent",
serviceName: "eve-support-agent",
recordInputs: false,
recordOutputs: false,
metadata: {
"deployment.environment": "production",
},
events: {
"step.started": (input) => ({
runtimeContext: {
"customer.channel": input.channel.kind ?? "unknown",
},
}),
},
});
```
Only primitive metadata values are exported as attributes. Use strings, numbers,
or booleans for custom runtime context.
## Options
| Option | Purpose |
| --------------- | -------------------------------------------------------------------------------------------- |
| `functionId` | Stable Eve/AI SDK telemetry function ID. Defaults to Eve's agent name when omitted. |
| `serviceName` | OpenTelemetry `service.name`. Defaults to Eve's agent name when omitted. |
| `recordInputs` | Whether Eve/AI SDK records full inputs. Defaults to `true`. |
| `recordOutputs` | Whether Eve/AI SDK records full outputs. Defaults to `true`. |
| `metadata` | Primitive values merged into Eve's AI SDK runtime context and exported as attributes. |
| `events` | Eve instrumentation event hooks. Catalyst composes `step.started` with your hook. |
| `setup` | Optional callback invoked from Eve's startup hook after Catalyst tracing setup is requested. |
Other Catalyst `setup()` options, such as `batching` and `resourceAttributes`,
can also be passed through. `autoInstrument` and `modules` are intentionally
managed by the Eve integration.
## Verify In Catalyst
Run your Eve agent, trigger a turn, then open Catalyst and filter by your
`service.name`, for example `eve-weather-agent`.
A successful Eve trace should include:
* An `ai.eve.turn` CHAIN span with `$eve.parent`, `$eve.root`, and `$eve.trigger`
* An `invoke_agent` AGENT span with `agent.name` and `gen_ai.system=eve`
* One or more LLM spans for the nested AI SDK model calls
* TOOL spans for any executed Eve tools
* Token usage and model fields when the provider returns usage metadata
For local one-off smoke tests, `batching: "simple"` can make spans export as
soon as they end. For long-lived Eve processes, the default batch exporter is
usually the better fit.
## Troubleshooting
If Eve traces do not appear:
* Confirm the file is named `agent/instrumentation.ts` and is inside the Eve
agent root.
* Confirm `CATALYST_OTLP_ENDPOINT` and `CATALYST_OTLP_TOKEN` are available to
the Eve process.
* Use a stable `serviceName` and filter by that value in Catalyst.
* Run Eve in a Node-compatible runtime. Catalyst configures the Node
OpenTelemetry tracer provider for this integration.
* If model spans appear without token counts, set `includeUsage: true` on the
AI SDK provider when the provider supports it.
* If Eve cannot resolve model context metadata for a custom provider, set
`modelContextWindowTokens` in `defineAgent()`.
# Snippets
Source: https://docs.inference.net/integrations/traces/examples
Copyable tracing patterns for providers, frameworks, agents, tool loops, structured outputs, prompt caching, handoffs, and custom subprocess work.
These examples are copy-paste ready. Each one shows what gets captured and
links to the integration page that covers the surface in depth.
For setup and configuration, start with the
[Traces Quickstart](/integrations/traces/quickstart). For an end-to-end view
of a real production agent, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## OpenAI Chat Completion
Initialize tracing before constructing the OpenAI client. The SDK patches Chat
Completions and emits an `LLM` span with input messages, output messages,
model name, invocation parameters, finish reason, and token counts.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
serviceName: "checkout-agent",
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You answer in one short sentence." },
{ role: "user", content: "Summarize order ABC-123." },
],
max_tokens: 80,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import setup
from openai import OpenAI
tracing = setup(service_name="checkout-agent")
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You answer in one short sentence."},
{"role": "user", "content": "Summarize order ABC-123."},
],
max_tokens=80,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
See [OpenAI traces](/integrations/traces/openai) for tool calls, structured
outputs, and the Responses API.
## OpenAI Tool Round Trip
Tool calls are captured on the model span. The first turn records the assistant
tool call and arguments; the second turn records the tool result in the input
message list.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
const tools = [
{
type: "function" as const,
function: {
name: "get_weather",
description: "Look up the current weather in a city.",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
];
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: "user", content: "What's the weather in San Francisco?" },
];
const first = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
const toolCalls = first.choices[0]?.message.tool_calls ?? [];
messages.push({ role: "assistant", content: null, tool_calls: toolCalls });
for (const toolCall of toolCalls) {
const args = JSON.parse(toolCall.function.arguments) as { city: string };
messages.push({
role: "tool",
tool_call_id: toolCall.id,
content: JSON.stringify({
city: args.city,
tempF: 62,
condition: "sunny",
}),
});
}
const final = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
console.log(final.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import json
from inference_catalyst_tracing import setup
from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
tracing = setup()
client = OpenAI()
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Look up the current weather in a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
},
]
messages: list[ChatCompletionMessageParam] = [
{"role": "user", "content": "What's the weather in San Francisco?"},
]
first = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
tool_calls = first.choices[0].message.tool_calls or []
messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [tc.model_dump() for tc in tool_calls],
},
)
for tool_call in tool_calls:
args = json.loads(tool_call.function.arguments)
messages.append(
{
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps(
{"city": args["city"], "temp_f": 62, "condition": "sunny"},
),
},
)
final = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
print(final.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
This captures the *model-side* view of tool calling: what the LLM asked for
and the result you passed back. For a *caller-side* view that wraps the actual
function execution in its own `TOOL` span, see
[Manual spans](/integrations/traces/manual-spans#tool-chain-and-retriever-spans).
## OpenAI Structured Output
Structured-output requests keep the schema in `llm.invocation_parameters` and
the model response in `output.value`.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "user",
content: "Extract the city, temperature, and unit from: 72F in Berlin.",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "weather_report",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
city: { type: "string" },
temperature: { type: "number" },
unit: { type: "string", enum: ["F", "C"] },
},
required: ["city", "temperature", "unit"],
},
},
},
});
console.log(response.choices[0]?.message.content);
```
<Metadata />
```python Python theme={"system"}
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "user",
"content": "Extract the city, temperature, and unit from: 72F in Berlin.",
},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "weather_report",
"strict": True,
"schema": {
"type": "object",
"additionalProperties": False,
"properties": {
"city": {"type": "string"},
"temperature": {"type": "number"},
"unit": {"type": "string", "enum": ["F", "C"]},
},
"required": ["city", "temperature", "unit"],
},
},
},
)
print(response.choices[0].message.content)
```
</CodeGroup>
## OpenAI Responses API
The Responses API is traced separately from Chat Completions. Function-call
items are normalized into the same OpenInference tool-call attributes used by
Chat Completions, so the dashboard renders them the same way.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.responses.create({
model: "gpt-4o-mini",
input: "In one sentence, what is OpenTelemetry?",
});
console.log(response.output_text);
```
<Metadata />
```python Python theme={"system"}
response = client.responses.create(
model="gpt-4o-mini",
input="In one sentence, what is OpenTelemetry?",
)
print(response.output_text)
```
</CodeGroup>
## Anthropic Messages
Anthropic Messages calls emit `LLM` spans with user and assistant content
blocks, model name, invocation parameters, finish reason, and usage.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import Anthropic from "@anthropic-ai/sdk";
import { setup } from "@inference/tracing";
const tracing = await setup({ modules: { anthropic: Anthropic } });
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const message = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 128,
messages: [{ role: "user", content: "Respond with just the word hello." }],
});
console.log(message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from anthropic import Anthropic
from inference_catalyst_tracing import setup
tracing = setup()
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
message = client.messages.create(
model="claude-haiku-4-5",
max_tokens=128,
messages=[{"role": "user", "content": "Respond with just the word hello."}],
)
print(message.content)
tracing.shutdown()
```
</CodeGroup>
## Anthropic Prompt Caching
When Anthropic returns prompt-cache usage fields, Catalyst maps them to
OpenInference token detail attributes so they show up alongside the regular
token counts on the LLM span.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const longSystem =
"You are a careful, terse assistant. Answer in one sentence.\n\n" +
"Reference document:\n" +
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. ".repeat(300);
const params: Anthropic.MessageCreateParamsNonStreaming = {
model: "claude-haiku-4-5",
max_tokens: 64,
system: [
{
type: "text",
text: longSystem,
cache_control: { type: "ephemeral" },
},
],
messages: [{ role: "user", content: "Is the document about lorem ipsum?" }],
};
const first = await client.messages.create(params);
const second = await client.messages.create(params);
console.log(first.usage.cache_creation_input_tokens ?? 0);
console.log(second.usage.cache_read_input_tokens ?? 0);
```
<Metadata />
```python Python theme={"system"}
long_system = (
"You are a careful, terse assistant. Answer in one sentence.\n\n"
"Reference document:\n"
+ "Lorem ipsum dolor sit amet, consectetur adipiscing elit. " * 300
)
params = dict(
model="claude-haiku-4-5",
max_tokens=64,
system=[
{
"type": "text",
"text": long_system,
"cache_control": {"type": "ephemeral"},
},
],
messages=[{"role": "user", "content": "Is the document about lorem ipsum?"}],
)
first = client.messages.create(**params)
second = client.messages.create(**params)
print(first.usage.cache_creation_input_tokens)
print(second.usage.cache_read_input_tokens)
```
</CodeGroup>
The cache attributes show up on the LLM span as
`llm.token_count.prompt_details.cache_write` and
`llm.token_count.prompt_details.cache_read`. See the
[Attributes reference](/integrations/traces/attributes#token-usage) for the
full set of token-detail keys.
## Manual Parent Around Automatic Children
Use an outer agent span around orchestration code when you want nested LLM,
tool, or framework spans grouped under one product-level operation. Spans
created inside the callback auto-parent under the agent span via OTel context
propagation.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-ticket-123",
system: "openai",
},
async (span) => {
const ticket = { id: "ticket_123", orderId: "ABC-123" };
span.setInput(ticket);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "user", content: `Review refund for ${ticket.orderId}` },
],
});
span.setOutput({ decision: response.choices[0]?.message.content });
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI()
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-ticket-123",
system="openai",
) as span:
ticket = {"id": "ticket_123", "order_id": "ABC-123"}
span.set_input(ticket)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": f"Review refund for {ticket['order_id']}"},
],
)
decision = response.choices[0].message.content
span.set_output({"decision": decision})
tracing.shutdown()
```
</CodeGroup>
## OpenAI Agents With Outer Span
Pair OpenAI Agents with OpenAI instrumentation. Use `agentSpan()` /
`agent_span()` for an explicit outer span; nested OpenAI calls are captured
automatically and parent under it.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run, tool } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const lookupOrder = tool({
name: "lookup_order",
description: "Look up an order by ID.",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) =>
JSON.stringify({ orderId, status: "shipped" }),
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Use tools to help customers with orders.",
tools: [lookupOrder],
model: "gpt-4o-mini",
});
const userMessage = "Where is order ABC-123?";
await agentSpan(
{
agentId: "support-agent-prod",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
system: "openai",
},
async (span) => {
span.setInput(userMessage);
const result = await run(supportAgent, userMessage, { maxTurns: 4 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import asyncio
import json
from agents import Agent, Runner, function_tool
from inference_catalyst_tracing import agent_span, setup
tracing = setup()
@function_tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, "status": "shipped"})
async def run_support_agent() -> str:
agent = Agent(
name="SupportAgent",
instructions="Use tools to help customers with orders.",
tools=[lookup_order],
model="gpt-4o-mini",
)
user_message = "Where is order ABC-123?"
with agent_span(
tracing.tracer,
agent_id="support-agent-prod",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
system="openai",
) as span:
span.set_input(user_message)
result = await Runner.run(agent, input=user_message, max_turns=4)
output = str(result.final_output or "")
span.set_output(output)
return output
print(asyncio.run(run_support_agent()))
tracing.shutdown()
```
</CodeGroup>
## OpenAI Agents Handoff
Handoffs create a useful trace tree when wrapped in an outer agent span: the
triage agent, specialist agent, model calls, and tools are all grouped under
one customer request.
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, handoff, run, tool } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const issueRefund = tool({
name: "issue_refund",
description: "Issue a refund for an order.",
parameters: z.object({ orderId: z.string(), amount: z.number() }),
execute: async ({ orderId, amount }) =>
JSON.stringify({ ok: true, orderId, refundId: "RFD-2201", amount }),
});
const refundsAgent = new Agent({
name: "RefundsAgent",
instructions: "Handle refund requests and use issue_refund.",
tools: [issueRefund],
model: "gpt-4o-mini",
});
const billingAgent = new Agent({
name: "BillingAgent",
instructions: "Answer billing questions. Do not issue refunds.",
model: "gpt-4o-mini",
});
const triageAgent = new Agent({
name: "TriageAgent",
instructions: "Route refund requests to RefundsAgent.",
handoffs: [handoff(refundsAgent), handoff(billingAgent)],
model: "gpt-4o-mini",
});
await agentSpan(
{
agentId: "triage-agent-prod",
agentName: "Triage Agent",
spanName: "triage-agent.run",
sessionId: "conversation-refund-abc-123",
system: "openai",
},
async (span) => {
const input = "I need a refund for order ABC-123, total $42.50.";
span.setInput(input);
const result = await run(triageAgent, input, { maxTurns: 8 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
## LangChain Agent With Tools
LangChain instrumentation hooks the callback manager. You do not need to wrap
each tool or model call manually; chain, LLM, and tool spans are emitted from
the framework callbacks.
If the same agent is already wrapped with LangSmith `@traceable`, keep that
decorator in place and install the `langsmith` extra. Catalyst uses the
LangSmith OTel span as the active parent, so the LangChain and provider spans
stay grouped under the decorated run.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import { ChatAnthropic } from "@langchain/anthropic";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { createAgent, tool } from "langchain";
import { z } from "zod";
const tracing = await setup({
modules: { langchainCallbacksManager: CallbackManagerModule },
});
const lookupOrder = tool(
({ orderId }) => JSON.stringify({ orderId, status: "shipped", total: 42.5 }),
{
name: "lookup_order",
description: "Look up an order by ID.",
schema: z.object({ orderId: z.string() }),
},
);
const cancelOrder = tool(
({ orderId, reason }) => JSON.stringify({ ok: true, orderId, reason }),
{
name: "cancel_order",
description: "Cancel a not-yet-delivered order.",
schema: z.object({ orderId: z.string(), reason: z.string() }),
},
);
const agent = createAgent({
model: new ChatAnthropic({ model: "claude-haiku-4-5", maxTokens: 512 }),
tools: [lookupOrder, cancelOrder],
systemPrompt: "Use tools to resolve order issues.",
});
const result = await agent.invoke({
messages: [{ role: "user", content: "Cancel order ABC-123." }],
});
console.log(result.messages.at(-1)?.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import json
from inference_catalyst_tracing import setup
from langchain.agents import create_agent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
tracing = setup()
ORDERS = {"ABC-123": {"status": "shipped", "total": 42.5}}
@tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, **ORDERS[order_id]})
@tool
def cancel_order(order_id: str, reason: str) -> str:
"""Cancel a not-yet-delivered order."""
return json.dumps({"ok": True, "order_id": order_id, "reason": reason})
llm = ChatAnthropic(model_name="claude-haiku-4-5", max_tokens_to_sample=512)
agent = create_agent(
llm,
tools=[lookup_order, cancel_order],
system_prompt="Use tools to resolve order issues.",
)
result = agent.invoke(
{"messages": [{"role": "user", "content": "Cancel order ABC-123."}]},
)
print(result["messages"][-1].content)
tracing.shutdown()
```
</CodeGroup>
## Pydantic AI Structured Agent (Python)
Pydantic AI ships native OpenTelemetry instrumentation. Catalyst registers its
provider and enables Pydantic AI instrumentation during `setup()`.
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import setup
from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext
class CityWeather(BaseModel):
city: str
temp_c: float = Field(description="Temperature in Celsius.")
condition: str
class WeatherReport(BaseModel):
cities: list[CityWeather]
summary: str
tracing = setup()
agent = Agent(
"openai:gpt-4o-mini",
output_type=WeatherReport,
system_prompt="Use get_weather for every requested city.",
)
@agent.tool
def get_weather(_ctx: RunContext[None], city: str) -> str:
"""Look up current weather for a city."""
return f'{{"city": "{city}", "temp_c": 12, "condition": "overcast"}}'
result = agent.run_sync("What's the weather in Paris and Tokyo?")
print(result.output.summary)
tracing.shutdown()
```
## Claude Agent SDK
Python can patch the SDK during `setup()` before `query` is imported.
TypeScript uses an explicit wrapper because ESM namespace bindings cannot be
safely patched.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { query } from "@anthropic-ai/claude-agent-sdk";
import { setup, wrapClaudeAgentSdkQuery } from "@inference/tracing";
const tracing = await setup();
const tracedQuery = wrapClaudeAgentSdkQuery(query);
const stream = tracedQuery({
prompt: "Count files matching *.md under the current directory.",
options: {
maxTurns: 4,
allowedTools: ["Bash"],
permissionMode: "bypassPermissions",
},
});
for await (const message of stream) {
console.log(message);
}
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
from dotenv import load_dotenv
from inference_catalyst_tracing import setup
load_dotenv()
tracing = setup()
from claude_agent_sdk import ClaudeAgentOptions, query # noqa: E402
options = ClaudeAgentOptions(
max_turns=4,
allowed_tools=["Bash"],
permission_mode="bypassPermissions",
)
async for message in query(
prompt="Count files matching *.md under the current directory.",
options=options,
):
print(message)
tracing.shutdown()
```
</CodeGroup>
## CLI Or Subprocess Work
When a tool has no instrumentable SDK, wrap the subprocess call in an agent
span and set the input, output, and token usage when available.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import { Codex } from "@openai/codex-sdk";
const tracing = await setup();
const codex = new Codex({ apiKey: process.env.OPENAI_API_KEY });
await agentSpan(
{
agentId: "codex-prod",
agentName: "Codex",
system: "openai",
spanName: "codex.invocation",
sessionId: "conversation-cli-hello",
},
async (span) => {
const prompt = "Reply with just the word hello.";
span.setInput(prompt);
const thread = codex.startThread({
skipGitRepoCheck: true,
sandboxMode: "read-only",
});
const turn = await thread.run(prompt);
span.setOutput(turn.finalResponse ?? "");
if (turn.usage != null) {
span.recordTokens({
prompt: turn.usage.input_tokens ?? 0,
completion: turn.usage.output_tokens ?? 0,
});
}
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import subprocess
from inference_catalyst_tracing import agent_span, setup
tracing = setup()
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="codex-prod",
agent_name="Codex",
system="openai",
span_name="codex.invocation",
session_id="conversation-cli-hello",
) as span:
span.set_input(prompt)
completed = subprocess.run(
["codex", "exec", "--skip-git-repo-check", prompt],
capture_output=True,
text=True,
timeout=120,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
</CodeGroup>
## Recommended Reading Order
1. [Traces Quickstart](/integrations/traces/quickstart) — install, configure
export, capture your first span.
2. [OpenAI traces](/integrations/traces/openai) or
[Anthropic traces](/integrations/traces/anthropic) — the provider you use
first.
3. [Manual spans](/integrations/traces/manual-spans) — tool, chain, and
retriever spans inside your agent loop.
4. [Production Agent Example](/integrations/traces/production-agent-example) — a
production-shaped agent end to end.
5. [Agent identity](/integrations/traces/agent-identity) — stable IDs for
dashboard grouping.
6. [Troubleshooting](/integrations/traces/troubleshooting) — missing spans,
missing attributes, shutdown.
# Span Handle API
Source: https://docs.inference.net/integrations/traces/handle-api
Reference for the typed span handle yielded by agentSpan, agent_span, manualSpan, and manual_span.
`agentSpan()` / `agent_span()` and `manualSpan()` / `manual_span()` yield a
handle to the active span. The handle exposes a typed surface for the
OpenInference attributes you typically want to set, plus an escape hatch to
the raw OTel span for anything outside that vocabulary. The same handle is
used by both helpers across both languages.
This page is the full reference for that handle. For when to call each method,
see the [Manual spans guide](/integrations/traces/manual-spans). For the wire
keys each method writes, see the
[Attributes reference](/integrations/traces/attributes).
## Shape
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
interface AgentSpanHandle {
setInput(value: unknown): void;
setOutput(value: unknown): void;
setModel(model: string): void;
setTool(options: { name: string; callId?: string }): void;
recordTokens(tokens: {
prompt?: number;
completion?: number;
total?: number;
}): void;
recordUsage(usage: unknown): TokenUsage;
setAttribute(key: string, value: unknown): void;
setAttributes(attributes: Record<string, unknown>): void;
raw: Span; // OTel @opentelemetry/api Span
}
```
<Metadata />
```python Python theme={"system"}
class AgentSpanHandle:
span: Span # OTel opentelemetry.trace.Span
def set_input(self, value: Any) -> None: ...
def set_output(self, value: Any) -> None: ...
def set_model(self, model: str) -> None: ...
def set_tool(self, *, name: str, call_id: str | None = None) -> None: ...
def record_tokens(
self,
*,
prompt: int | None = None,
completion: int | None = None,
total: int | None = None,
) -> None: ...
def record_usage(self, usage: Any) -> None: ...
def set_attribute(self, key: str, value: Any) -> None: ...
def set_attributes(self, attributes: Mapping[str, Any]) -> None: ...
```
</CodeGroup>
## `setInput` / `set_input`
Records the input that started this unit of work.
* Strings are stored as-is on `input.value`.
* Non-strings are JSON-stringified.
If the input is structured, also set `input.mime_type = "application/json"`
so downstream viewers know to render it as JSON.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
span.setInput("Summarize order ABC-123");
// or
span.setInput({ ticketId: "ticket_123", orderId: "ABC-123" });
span.setAttribute(Attr.INPUT_MIME_TYPE, "application/json");
```
<Metadata />
```python Python theme={"system"}
span.set_input("Summarize order ABC-123")
# or
span.set_input({"ticket_id": "ticket_123", "order_id": "ABC-123"})
span.set_attribute(Attr.INPUT_MIME_TYPE, "application/json")
```
</CodeGroup>
## `setOutput` / `set_output`
Records the final output of this unit of work. Same coercion rules as
`setInput`. Call it once near the end of the callback, after the work has
produced its result.
## `setModel` / `set_model`
Records the model name on `llm.model_name`. Useful on `AGENT` spans when the
agent picks a model dynamically and you want the model surfaced at the agent
level. The per-SDK patchers already set this on `LLM` child spans, so you
typically do not need it there.
```typescript theme={"system"}
span.setModel("claude-haiku-4-5");
```
## `recordTokens` / `record_tokens`
Records token usage from explicit counts. Use this when you have aggregate
counts across an agent run, or when you are wrapping an SDK Catalyst does not
patch and you want to attach the provider's reported counts.
Any field may be omitted; only the provided fields are written. If you want
`total` on the span, pass it explicitly — `recordTokens` / `record_tokens`
does not infer it from `prompt` and `completion`. (The `recordUsage` /
`record_usage` helper documented below does infer `total` when the provider's
usage payload omits it.)
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
span.recordTokens({ prompt: 820, completion: 160, total: 980 });
```
<Metadata />
```python Python theme={"system"}
span.record_tokens(prompt=820, completion=160)
```
</CodeGroup>
## `recordUsage` / `record_usage`
Records token usage from a provider-shaped `usage` object. The SDK normalizes
the common shapes so you do not have to: OpenAI Chat Completions
(`prompt_tokens` / `completion_tokens`), the newer Responses spelling
(`input_tokens` / `output_tokens`), Anthropic cache accounting fields
(`cache_creation_input_tokens` / `cache_read_input_tokens`), and OpenAI's
`prompt_tokens_details.cached_tokens` /
`completion_tokens_details.reasoning_tokens`. When `total` is missing from
the payload but `prompt` and `completion` are present, the helper infers it.
The TypeScript version returns the normalized `TokenUsage` so the caller can
read the numbers back without re-parsing.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.chat.completions.create({ /* ... */ });
if (response.usage) {
span.recordUsage(response.usage);
}
// Works with Anthropic responses too:
const message = await anthropicClient.messages.create({ /* ... */ });
span.recordUsage(message.usage);
```
<Metadata />
```python Python theme={"system"}
response = client.chat.completions.create(...)
if response.usage is not None:
span.record_usage(response.usage)
# Works with Anthropic responses too:
message = anthropic_client.messages.create(...)
span.record_usage(message.usage)
```
</CodeGroup>
<Note>
For Anthropic responses, `recordUsage` / `record_usage` folds
`cache_creation_input_tokens` and `cache_read_input_tokens` into the recorded
prompt token count, then also records them as separate cache-write and
cache-read attributes. The prompt count on the span therefore reflects the
total tokens that hit the model, not just the uncached prompt.
</Note>
## `setTool` / `set_tool`
Records tool identity on a `TOOL` span: writes `tool.name` and, when supplied,
`tool_call.id`. The `manualSpan` / `manual_span` helper accepts the same
fields as options, so `setTool` is rarely needed when authoring a TOOL span
from scratch — use it when the tool identity is only available partway
through the callback body.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
await manualSpan(
{ spanName: "lookup_order.tool", spanKind: SpanKindValues.TOOL },
async (span) => {
span.setTool({ name: "lookup_order", callId: toolCallId });
span.setInput(args);
const result = await TOOLS.lookup_order(args);
span.setOutput(result);
},
);
```
<Metadata />
```python Python theme={"system"}
with manual_span(
tracing.tracer,
name="lookup_order.tool",
span_kind=SpanKindValues.TOOL,
) as span:
span.set_tool(name="lookup_order", call_id=tool_call_id)
span.set_input(args)
result = TOOLS["lookup_order"](**args)
span.set_output(result)
```
</CodeGroup>
## `setAttribute` / `set_attribute` / `set_attributes`
Adds one or many domain-specific attributes with OTel-safe value normalization.
Strings, booleans, numbers, and homogeneous primitive arrays pass through
unchanged; mappings and heterogeneous arrays are JSON-stringified.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
span.setAttribute("app.tenant_id", tenantId);
span.setAttributes({
"app.request_channel": "slack",
"app.deploy_env": process.env.DEPLOY_ENV ?? "dev",
});
```
<Metadata />
```python Python theme={"system"}
span.set_attribute("app.tenant_id", tenant_id)
span.set_attributes({
"app.request_channel": "slack",
"app.deploy_env": os.environ.get("DEPLOY_ENV", "dev"),
})
```
</CodeGroup>
## `raw` / `.span`
The underlying OTel span. Use this when you need behavior outside the typed
surface:
* Adding domain attributes.
* Recording span events (`span.raw.addEvent("rate_limit_hit", { retry_after: 30 })`).
* Recording an exception without ending the span yourself
(`span.raw.recordException(err)`).
* Reading the span context to propagate to a background job
(`span.raw.spanContext()`).
The escape hatch is `span.raw` in TypeScript and `span.span` in Python.
## Value Coercion Rules
| Input | Coerced to |
| ------------------------------- | ------------------------------------------------------------------------------------------------- |
| `string` | Stored verbatim. |
| `number`, `boolean` | Stored as-is. OTel handles these as primitive attribute values. |
| Plain object | `JSON.stringify(value)` (TS) / `json.dumps(value, default=str)` (Python). |
| Array of homogeneous primitives | Stored as an OTel attribute array. |
| Array containing mixed types | JSON-stringified. |
| `Error` / `Exception` | Not coerced. Use `span.raw.recordException(err)` instead, which produces an OTel exception event. |
| `bigint` (TS) | Cast yourself before passing — OTel does not accept `bigint`. |
The `setInput` and `setOutput` methods apply these rules and also set the
matching MIME type attribute when the value is structured. The other helpers
write to one attribute key each and apply the rules to that value.
## Status Behavior
The handle does not expose `setStatus` directly. The wrapping context manager
sets status for you:
* Body returns normally → status `OK`.
* Body raises → exception is recorded as a span event, status is set to
`ERROR` with the exception's message, then the exception re-raises.
If you need fine-grained status control mid-callback (for example, to mark a
partial-success state), set it on the raw span:
<Metadata />
```typescript TypeScript theme={"system"}
import { SpanStatusCode } from "@opentelemetry/api";
if (result.partial) {
span.raw.setStatus({
code: SpanStatusCode.ERROR,
message: `partial result: ${result.missingFields.join(", ")}`,
});
}
```
## Next Steps
<CardGroup>
<Card title="Manual spans guide" icon="pen-nib" href="/integrations/traces/manual-spans">
Use the handle to author AGENT, TOOL, CHAIN, and RETRIEVER spans.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
All `Attr.*` constants and `SpanKindValues` values.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool execution, end to end.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
Debug missing spans, missing attributes, and unexpected status codes.
</Card>
</CardGroup>
# LangChain Traces
Source: https://docs.inference.net/integrations/traces/langchain
Capture LangChain chains, agents, LLM calls, and tools through Catalyst callback instrumentation.
Catalyst hooks LangChain callback managers so runnable calls emit spans without
manual span creation. In agent workflows, the trace tree includes chain, model,
and tool spans with parent-child relationships preserved by LangChain callbacks.
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing @langchain/core langchain @langchain/anthropic zod
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[langchain]'
```
</CodeGroup>
## TypeScript Agent With Tools
Pass the callback manager module to `setup()`. Catalyst patches the static
configuration path LangChain uses when constructing callback managers.
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import { ChatAnthropic } from "@langchain/anthropic";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { createAgent, tool } from "langchain";
import { z } from "zod";
const tracing = await setup({
serviceName: "support-agent",
modules: { langchainCallbacksManager: CallbackManagerModule },
});
const lookupOrder = tool(
({ orderId }) => JSON.stringify({ orderId, status: "shipped", total: 42.5 }),
{
name: "lookup_order",
description: "Look up an order by ID.",
schema: z.object({ orderId: z.string() }),
},
);
const cancelOrder = tool(
({ orderId, reason }) => JSON.stringify({ ok: true, orderId, reason }),
{
name: "cancel_order",
description: "Cancel a not-yet-delivered order.",
schema: z.object({ orderId: z.string(), reason: z.string() }),
},
);
const agent = createAgent({
model: new ChatAnthropic({ model: "claude-haiku-4-5", maxTokens: 512 }),
tools: [lookupOrder, cancelOrder],
systemPrompt: "Use tools to resolve order issues.",
});
const result = await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
role: "support",
system: "langchain",
},
async (span) => {
const input = "Cancel order ABC-123.";
span.setInput(input);
const output = await agent.invoke({
messages: [{ role: "user", content: input }],
});
span.setOutput(output.messages.at(-1)?.content);
return output;
},
);
console.log(result.messages.at(-1)?.content);
await tracing.shutdown();
```
## Python Agent With Tools
<Metadata />
```python Python theme={"system"}
import json
from inference_catalyst_tracing import agent_span, setup
from langchain.agents import create_agent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
tracing = setup(service_name="support-agent")
@tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, "status": "shipped", "total": 42.5})
@tool
def cancel_order(order_id: str, reason: str) -> str:
"""Cancel a not-yet-delivered order."""
return json.dumps({"ok": True, "order_id": order_id, "reason": reason})
llm = ChatAnthropic(model_name="claude-haiku-4-5", max_tokens_to_sample=512)
agent = create_agent(
llm,
tools=[lookup_order, cancel_order],
system_prompt="Use tools to resolve order issues.",
)
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
agent_role="support",
system="langchain",
) as span:
user_input = "Cancel order ABC-123."
span.set_input(user_input)
result = agent.invoke(
{"messages": [{"role": "user", "content": user_input}]},
)
span.set_output(result["messages"][-1].content)
print(result["messages"][-1].content)
tracing.shutdown()
```
## What To Look For
* A top-level LangChain chain or agent span
* An outer AGENT span with `agent.id=support-agent` when you use the wrapper
* Nested LLM spans for model calls
* Tool spans named after LangChain tools
* Tool input and output attributes on tool spans
* Token counts on model spans when the provider returns usage
# Langfuse Traces
Source: https://docs.inference.net/integrations/traces/langfuse
Send Langfuse SDK traces to Catalyst without changing your tracing code.
If your application already uses Langfuse, this is just an env-var switch. Keep all of your existing Langfuse tracing code and point the Langfuse SDK at Catalyst by changing three environment variables. Your spans stream into the Catalyst **Traces** dashboard with no SDK swap and no code changes. Catalyst accepts the Langfuse ingestion and OTEL endpoints and converts traces, generations, spans, usage, costs, inputs, outputs, metadata, users, sessions, and tags into Catalyst spans.
This is the fastest way to try Catalyst: flip the env vars, see your traces, and switch over fully later for more control. Spans show up in the Traces dashboard out of the box but to get the most out of it, set an agent identity on your top-level span so runs group in the **Agents** dashboard and become available to Halo. See [Group by agent](#group-by-agent).
## Configure Langfuse
Use your Catalyst API key as the Langfuse secret key. The public key is only present for Langfuse SDK compatibility and its value is never used. Set it to `pk-catalyst`, but any non-empty value works.
<Metadata />
```bash theme={"system"}
export LANGFUSE_HOST="https://telemetry.inference.net"
export LANGFUSE_PUBLIC_KEY="pk-catalyst"
export LANGFUSE_SECRET_KEY="<your-catalyst-api-key>"
```
The Catalyst API key must have write access to the project that should receive the traces.
## Python
<Metadata />
```python Python theme={"system"}
from langfuse import Langfuse, observe, get_client
langfuse = Langfuse(
public_key="pk-catalyst",
secret_key="<your-catalyst-api-key>",
base_url="https://telemetry.inference.net",
)
@observe(as_type="generation", name="answer-question")
def answer_question(question: str) -> str:
client = get_client()
# Set identity the Langfuse way. These map to user.id, session.id, and tags in Catalyst.
client.update_current_trace(
user_id="user-123",
session_id="session-abc",
tags=["prod"],
)
client.update_current_generation(
model="gpt-4o-mini",
usage_details={"input": 12, "output": 8, "total": 20},
)
return f"Answer: {question}"
print(answer_question("What is Catalyst?"))
langfuse.flush()
langfuse.shutdown()
```
## TypeScript
<Metadata />
```typescript TypeScript theme={"system"}
import { observe, getClient } from "@langfuse/tracing";
process.env.LANGFUSE_HOST = "https://telemetry.inference.net";
process.env.LANGFUSE_PUBLIC_KEY = "pk-catalyst";
process.env.LANGFUSE_SECRET_KEY = "<your-catalyst-api-key>";
const answerQuestion = observe(
async (question: string) => {
const client = getClient();
// Set identity the Langfuse way. These map to user.id, session.id, and tags in Catalyst.
client.updateCurrentTrace({
userId: "user-123",
sessionId: "session-abc",
tags: ["prod"],
});
client.updateCurrentGeneration({
model: "gpt-4o-mini",
usageDetails: { input: 12, output: 8, total: 20 },
});
return `Answer: ${question}`;
},
{ name: "answer-question", asType: "generation" },
);
console.log(await answerQuestion("What is Catalyst?"));
await getClient().flushAsync();
```
Langfuse's OpenTelemetry-based SDKs are also supported with the same host, public key, and secret key settings above. If you are not using Langfuse and are configuring a generic OpenTelemetry exporter, use Catalyst's standard OTLP endpoint instead.
## Send to Both Langfuse and Catalyst
The setup above points the Langfuse SDK entirely at Catalyst, so all of that
SDK's traffic comes to us. If you want to keep your existing Langfuse pipeline
and mirror the same traces into Catalyst, you can fan out to both at once. This
is the lowest-risk way to evaluate Catalyst: Langfuse keeps working untouched
while Catalyst receives a copy of everything.
### TypeScript: add a second span processor
The Langfuse JS/TS SDK is OpenTelemetry-based, and OpenTelemetry broadcasts
every span to every registered span processor by default. Keep your existing
`LangfuseSpanProcessor` pointed at Langfuse and add a second one pointed at
Catalyst. Your `observe` calls and the rest of your tracing code do not change.
<Metadata />
```typescript instrumentation.ts theme={"system"}
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";
const sdk = new NodeSDK({
spanProcessors: [
// Your existing Langfuse destination.
new LangfuseSpanProcessor({
baseUrl: "https://cloud.langfuse.com",
publicKey: process.env.LANGFUSE_PUBLIC_KEY,
secretKey: process.env.LANGFUSE_SECRET_KEY,
}),
// Catalyst destination. The secret key is your Catalyst API key. The public
// key is required by the SDK but ignored by Catalyst, so any non-empty
// value works.
new LangfuseSpanProcessor({
baseUrl: "https://telemetry.inference.net",
publicKey: "pk-catalyst",
secretKey: "<your-catalyst-api-key>",
}),
],
});
sdk.start();
```
Every span now streams to both backends. Each processor batches and flushes
independently, so a delay or outage on one does not block the other.
### Python: fan out with an OpenTelemetry Collector
The Python SDK does not broadcast across destinations. Its multi-client mode
routes each span to a single project based on a public-key attribute, so it is
not a true dual-send: you have to thread `langfuse_public_key` through every
`@observe`, OpenAI wrapper, and Langchain handler call to pick the destination,
and spans without that attribute land everywhere by accident. It is fragile and
not the documented path for sending to both.
The clean, SDK-agnostic way to dual-send from Python (or any runtime) is an
OpenTelemetry Collector with two OTLP exporters. Both Langfuse and Catalyst
accept OTLP, so point your app at a local collector and let the collector ship
to both. Your application code does not change.
<Metadata />
```yaml otel-collector.yaml theme={"system"}
receivers:
otlp:
protocols:
http:
grpc:
exporters:
# base64-encode "<langfuse-public-key>:<langfuse-secret-key>" for the header.
otlphttp/langfuse:
endpoint: https://cloud.langfuse.com/api/public/otel
headers:
Authorization: "Basic <base64 of langfuse-public-key:langfuse-secret-key>"
# base64-encode "pk-catalyst:<your-catalyst-api-key>" for the header.
otlphttp/catalyst:
endpoint: https://telemetry.inference.net/api/public/otel
headers:
Authorization: "Basic <base64 of pk-catalyst:your-catalyst-api-key>"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp/langfuse, otlphttp/catalyst]
```
Point your Langfuse SDK (or any OTLP exporter) at the collector's OTLP endpoint
and every trace is mirrored to both backends. This is the same fan-out pattern
Catalyst uses internally to ship telemetry to multiple sinks.
<Note>
The collector approach works for any OTLP trace source, not just Langfuse. If
you already export OpenTelemetry traces, add a second OTLP exporter for
Catalyst and you are done.
</Note>
## What Maps Into Catalyst
| Langfuse field | Catalyst field |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| Trace ID | Stable Catalyst trace ID derived from the Langfuse trace ID |
| Observation ID | Stable Catalyst span ID derived from the Langfuse observation ID |
| Trace name | `langfuse.trace.name` (shown in the Traces view; does **not** create an agent on its own — see [Group by agent](#group-by-agent)) |
| User and session | `user.id`, `session.id` |
| Agent name / id | `agent.name`, `agent.id` — only when you set them explicitly (see [Group by agent](#group-by-agent)) |
| Generation model | `llm.model_name`, `llm.provider` when inferable |
| Usage and cost | Prompt/completion/total token columns and cost columns |
| Input and output | Span input/output and LLM message columns |
| Metadata and tags | Preserved under `langfuse.*` and `halo.source.*` attributes |
| Errors and warnings | Span status and status message |
<Note>
This is v1 of the Langfuse integration — scores, evaluations, and richer
mapping are coming later. Langfuse scores and evaluations are not supported at
this time. Pointing
`LANGFUSE_HOST` at Catalyst routes all of that SDK's traffic to us. To keep
sending to Langfuse as well, see [Send to both Langfuse and
Catalyst](#send-to-both-langfuse-and-catalyst). Scores emitted through the
SDK are not ingested at this point in time.
</Note>
## Group by agent
Pointing the base URL at Catalyst sends your spans — but Catalyst does **not**
treat every trace as an agent. A trace name, or even a Langfuse `agent`-type
observation, is not enough on its own; otherwise every traced workload would
flood the Agents dashboard. Agent grouping is **opt-in**: set `agent.name` (and
optionally a stable `agent.id`) on the **top-level span of your agent**, and
Catalyst groups that whole trace — and every child span — under it in the
**Agents** dashboard. This is also what makes the trace available to
[Halo](/platform/gateway/overview), which operates on agents.
The Langfuse SDKs are OpenTelemetry-based and forward raw OTel attributes
untouched, so set the two attributes directly on the active span. Do this
**only on the top-level agent span** — children inherit grouping through the
trace and should not carry their own agent identity.
* `agent.name` — the human-readable label the Agents dashboard groups on.
* `agent.id` — a stable identifier that survives display-name changes. When
set, it wins over `agent.name`, so renaming the agent doesn't split its
history. Omit it if you don't need stable grouping.
<Metadata />
```typescript TypeScript theme={"system"}
import { observe } from "@langfuse/tracing";
import { trace } from "@opentelemetry/api";
const runAgent = observe(
async (question: string) => {
// Mark this top-level span as an agent. Set ONLY here, not on children.
trace.getActiveSpan()?.setAttributes({
"agent.name": "deep-research",
"agent.id": "deep-research-v1", // stable id; optional
});
// ... your agent loop, tool calls, and nested generations ...
return "answer";
},
{ name: "deep-research", asType: "agent" },
);
```
<Metadata />
```python Python theme={"system"}
from langfuse import observe
from opentelemetry import trace
@observe(as_type="agent", name="deep-research")
def run_agent(question: str) -> str:
# Mark this top-level span as an agent. Set ONLY here, not on children.
span = trace.get_current_span()
span.set_attribute("agent.name", "deep-research")
span.set_attribute("agent.id", "deep-research-v1") # stable id; optional
# ... your agent loop, tool calls, and nested generations ...
return "answer"
```
## View and Analyze
Once traces arrive, open the Catalyst dashboard and use the Traces tab to
inspect individual trace trees. Traces whose top-level span carries an
`agent.name` / `agent.id` (see [Group by agent](#group-by-agent)) are grouped
in the Agents dashboard, where Halo can run analysis over them.
For the best Halo reports, keep `agent.id` (or `agent.name`) stable per agent or
workflow and set `userId` / `sessionId` when available.
## Want More Control?
The Langfuse drop-in is the easy on-ramp: no code changes, just env vars. If you
want finer control over how traces are grouped, such as stable agent IDs,
explicit agent and manual spans, and a fully shaped trace tree, switch to the
first-party Catalyst tracing SDK. Start with the
[Quickstart](/integrations/traces/quickstart), then see
[Agent identity](/integrations/traces/agent-identity) for grouping executions in
the Agents dashboard.
# LangGraph Traces
Source: https://docs.inference.net/integrations/traces/langgraph
Trace LangGraph workflows while preserving graph and node parent-child spans.
Catalyst uses the LangChain callback path for LangGraph workflows. A compiled
graph emits a graph-level CHAIN span and node-level CHAIN spans parented under
the graph run.
If the graph is the implementation of a product agent, wrap `graph.invoke()`
with `agentSpan()` / `agent_span()` and set a stable `agent.id` so executions
group in the Agents dashboard.
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing @langchain/core @langchain/langgraph
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[langgraph]'
```
</CodeGroup>
## TypeScript State Graph
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import * as CallbackManagerModule from "@langchain/core/callbacks/manager";
import { Annotation, END, START, StateGraph } from "@langchain/langgraph";
const tracing = await setup({
serviceName: "counter-graph",
modules: { langchainCallbacksManager: CallbackManagerModule },
});
const State = Annotation.Root({
count: Annotation<number>({
reducer: (_left, right) => right,
default: () => 0,
}),
});
const graph = new StateGraph(State)
.addNode("increment", (state) => ({ count: state.count + 1 }))
.addNode("double", (state) => ({ count: state.count * 2 }))
.addEdge(START, "increment")
.addEdge("increment", "double")
.addEdge("double", END)
.compile();
const result = await agentSpan(
{
agentId: "counter-graph-agent",
agentName: "Counter Graph Agent",
spanName: "counter-graph.run",
sessionId: "conversation-counter-1",
role: "workflow",
system: "langgraph",
},
async (span) => {
const input = { count: 1 };
span.setInput(input);
const output = await graph.invoke(input);
span.setOutput(output);
return output;
},
);
console.log(result);
await tracing.shutdown();
```
## Python State Graph
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span, setup
from langgraph.graph import END, START, StateGraph
from typing_extensions import TypedDict
class GraphState(TypedDict):
count: int
tracing = setup(service_name="counter-graph")
def increment(state: GraphState) -> GraphState:
return {"count": state["count"] + 1}
def double(state: GraphState) -> GraphState:
return {"count": state["count"] * 2}
builder = StateGraph(GraphState)
builder.add_node("increment", increment)
builder.add_node("double", double)
builder.add_edge(START, "increment")
builder.add_edge("increment", "double")
builder.add_edge("double", END)
graph = builder.compile()
with agent_span(
tracing.tracer,
agent_id="counter-graph-agent",
agent_name="Counter Graph Agent",
span_name="counter-graph.run",
session_id="conversation-counter-1",
agent_role="workflow",
system="langgraph",
) as span:
input_state = {"count": 1}
span.set_input(input_state)
result = graph.invoke(input_state)
span.set_output(result)
print(result)
tracing.shutdown()
```
## What To Look For
* A graph span named `LangGraph`
* Node spans named after graph nodes
* `openinference.span.kind=CHAIN` on graph and node spans
* Node spans parented under the graph span
* An outer AGENT span with your stable `agent.id` when you wrap agent-style
graph invocations
For agent-style graphs, model and tool spans created inside node functions
appear below the active graph or node context.
# LangSmith Traces
Source: https://docs.inference.net/integrations/traces/langsmith
Bridge LangSmith OpenTelemetry spans into the Catalyst tracer provider.
If your application already uses LangSmith tracing, Catalyst can bridge
LangSmith OpenTelemetry spans into the Catalyst tracer provider. This keeps
LangSmith `traceable` functions and framework integrations visible with the rest
of your application traces.
If a LangSmith-traced function is the boundary of an agent run, add an outer
`agentSpan()` / `agent_span()` with a stable `agent.id`. LangSmith spans remain
visible under that AGENT span.
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing langsmith
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[langsmith]'
```
</CodeGroup>
## TypeScript Traceable Function
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import { Client } from "langsmith";
import { traceable } from "langsmith/traceable";
process.env.LANGSMITH_TRACING = "true";
process.env.LANGSMITH_TRACING_MODE = "otel";
const tracing = await setup({ serviceName: "langsmith-worker" });
const client = new Client({ tracingMode: "otel" });
const answerQuestion = traceable(
async (input: { question: string }) => ({
answer: input.question.toUpperCase(),
}),
{
name: "answerQuestion",
run_type: "tool",
client,
tracer: tracing.provider.getTracer("langsmith-app"),
},
);
console.log(await answerQuestion({ question: "hi" }));
await client.flush();
await tracing.shutdown();
```
## Python Traceable Function
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import setup
from langsmith import Client, traceable
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_TRACING_MODE"] = "otel"
tracing = setup(service_name="langsmith-worker")
client = Client()
@traceable(name="answer_question", run_type="tool", client=client, enabled=True)
def answer_question(question: str) -> str:
return question.upper()
print(answer_question("hi"))
client.flush()
tracing.shutdown()
```
## Hybrid Mode
If `LANGSMITH_TRACING=true` is set and `LANGSMITH_TRACING_MODE` is not set,
Catalyst defaults LangSmith to `hybrid` mode. That keeps existing LangSmith
tracing active while also routing OpenTelemetry spans through Catalyst.
<Metadata />
```bash theme={"system"}
export LANGSMITH_TRACING=true
# Catalyst sets LANGSMITH_TRACING_MODE=hybrid when unset.
```
## Stable Agent Identity
Use this pattern around LangSmith-traced work when you want the Agents dashboard
to group executions by your product agent ID.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "question-answer-agent",
agentName: "Question Answer Agent",
spanName: "question-answer.run",
sessionId: "conversation-qa-1",
role: "qa",
system: "langsmith",
},
async (span) => {
const input = { question: "hi" };
span.setInput(input);
const output = await answerQuestion(input);
span.setOutput(output);
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="question-answer-agent",
agent_name="Question Answer Agent",
span_name="question-answer.run",
session_id="conversation-qa-1",
agent_role="qa",
system="langsmith",
) as span:
question = "hi"
span.set_input(question)
answer = answer_question(question)
span.set_output(answer)
```
</CodeGroup>
# LiveKit Agents Traces
Source: https://docs.inference.net/integrations/traces/livekit-agents
Trace LiveKit Agents sessions, model calls, and function tools through LiveKit's native OpenTelemetry spans.
Catalyst traces LiveKit Agents by connecting LiveKit's native OpenTelemetry
tracer to the Catalyst provider. Your LiveKit worker keeps using `AgentSession`,
rooms, tools, and model plugins normally; Catalyst enriches the spans LiveKit
already emits with OpenInference attributes before export.
Use this guide for applications built with `@livekit/agents` in TypeScript or
`livekit-agents` in Python.
## What Is Captured
* `agent_session` spans as OpenInference AGENT spans
* `llm_node`, `llm_request`, and `llm_request_run` spans as LLM spans
* LiveKit function tools as TOOL spans, including tool name, call ID,
arguments, output, and errors when available
* LiveKit `lk.*` and `gen_ai.*` attributes, mapped into model, provider, token,
input, and output attributes when LiveKit provides them
* Other LiveKit telemetry spans as CHAIN spans so the full session timeline
remains visible
Catalyst sets `agent.name` on LiveKit `agent_session` spans from
`lk.agent_name` or `lk.agent_label` when LiveKit provides it. Use stable
LiveKit agent class names or labels, and add a manual `agent_span` around your
own worker/session boundary when your product needs a canonical `agent.id` for
dashboard grouping.
<Info>
Catalyst does not monkey patch `AgentSession` methods. The integration uses
LiveKit's public telemetry provider hooks, so LiveKit helper objects and
streaming behavior are preserved.
</Info>
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing @livekit/agents
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[livekit-agents]'
```
</CodeGroup>
## Configure Export
Set a stable service name so LiveKit sessions are easy to find in Catalyst.
<Metadata />
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="livekit-agent"
export CATALYST_SERVICE_VERSION="2026.05.06"
```
## Initialize Tracing
Initialize tracing before starting LiveKit agent sessions or workers.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import * as LiveKitAgents from "@livekit/agents";
const tracing = await setup({
serviceName: "livekit-agent",
modules: { livekitAgents: LiveKitAgents },
});
// Start your LiveKit worker or AgentSession normally.
// LiveKit telemetry now exports through Catalyst.
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import setup
tracing = setup(service_name="livekit-agent")
# Start your LiveKit worker or AgentSession normally.
# LiveKit telemetry now exports through Catalyst.
tracing.shutdown()
```
</CodeGroup>
## Text-Only Agent Example
This Python example starts a LiveKit text-only `AgentSession`. It works for smoke
tests and CI because it does not require audio input or output, but it still runs
the real LiveKit session path.
<Metadata />
```python Python theme={"system"}
import os
from dotenv import load_dotenv
from inference_catalyst_tracing import setup
from livekit import agents
from livekit.agents import Agent, AgentSession, room_io
from livekit.plugins import openai
load_dotenv()
async def entrypoint(ctx: agents.JobContext):
await ctx.connect()
tracing = setup(service_name="livekit-support-agent")
session = AgentSession(llm=openai.LLM(model=os.getenv("OPENAI_MODEL", "gpt-4o-mini")))
await session.start(
agent=Agent(
instructions="You are a concise support agent. Use tools when needed."
),
room=ctx.room,
room_options=room_io.RoomOptions(
text_input=True,
text_output=True,
audio_input=False,
audio_output=False,
),
record=False,
)
await session.generate_reply(
user_input="Reply with one sentence confirming you are online.",
input_modality="text",
).wait_for_playout()
await session.aclose()
tracing.shutdown()
if __name__ == "__main__":
agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))
```
## Function Tools
LiveKit tool spans are captured automatically when the LiveKit runtime emits
`function_tool` telemetry. The span includes the tool name, tool call ID, JSON
arguments, and output when available.
<Metadata />
```python Python theme={"system"}
from livekit.agents import Agent, function_tool
class FrontDeskAgent(Agent):
def __init__(self) -> None:
super().__init__(
instructions="Help users find appointment information.",
)
@function_tool
async def lookup_appointment(self, user_id: str) -> str:
return f"{user_id} has an appointment at 10:30 AM."
```
When `lookup_appointment` runs inside an `AgentSession`, Catalyst shows a TOOL
span under the LiveKit session trace.
## Stable Agent Identity
If your application owns a stable agent ID outside LiveKit's runtime telemetry,
wrap the session work with `agent_span()` and start the LiveKit session inside
that context.
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span
async def start_traced_session(ctx, session) -> None:
with agent_span(
tracing.tracer,
agent_id="front-desk-agent",
agent_name="Front Desk Agent",
span_name="front-desk-agent.run",
session_id="conversation-front-desk-1",
agent_role="front-desk",
system="livekit",
) as span:
span.set_input({"room": ctx.room.name})
await session.start(agent=FrontDeskAgent(), room=ctx.room)
span.set_output({"status": "session-started"})
```
## Verify In Catalyst
Open Catalyst and filter traces by your `CATALYST_SERVICE_NAME`, for example
`livekit-agent` or `livekit-support-agent`.
A successful LiveKit trace should include:
* An `agent_session` AGENT span for the session
* One or more LLM spans for LiveKit model nodes or requests
* TOOL spans for any function tools the agent called
* Your stable `agent.id` on the outer AGENT span when you add the identity
wrapper above
For short-lived scripts, always call `tracing.shutdown()` before process exit so
batched spans are flushed.
## Troubleshooting
If you do not see LiveKit spans:
* Initialize tracing before starting the worker or `AgentSession`.
* Set a stable `CATALYST_SERVICE_NAME` and filter by that value.
* Confirm `CATALYST_OTLP_ENDPOINT` and `CATALYST_OTLP_TOKEN` are present in the
process environment.
* Call `tracing.shutdown()` before a script exits.
* For Python OpenAI-backed LiveKit agents, use current Catalyst tracing packages
so OpenAI streaming context managers are preserved.
# Manual Spans
Source: https://docs.inference.net/integrations/traces/manual-spans
Author OpenInference-shaped spans for tools, retrievers, custom routers, CLI subprocesses, and anything else the SDK does not patch automatically.
Catalyst's per-SDK instrumentation captures provider calls automatically: every
SDK you pass to `setup({ modules })` emits an LLM span on its own. But a real
agent does work the SDK never sees, like running tools, retrieving documents,
routing requests, validating output, or shelling out to a CLI. Manual spans are
how you put that work into the same trace tree, next to the auto-captured LLM
spans.
There are two shapes you author by hand, and everything on this page is one of
them:
* **The agent span** wraps a whole run. It is the parent row everything else
nests under, and it carries the agent identity and session. You author it with
`agentSpan()` / `agent_span()`. This is the same wrapper the integration
guides put around a run.
* **Child spans** sit inside that run, one per non-LLM step: a tool call, a
retrieval, a routing step. You author them with `manualSpan()` /
`manual_span()`, choosing the span kind that matches the step.
If you have not yet wired up `setup()`, start with the
[Traces Quickstart](/integrations/traces/quickstart).
## Span Kinds You Author Manually
| Kind | Use it for |
| ----------- | ---------------------------------------------------------------------------------------------------------- |
| `AGENT` | The outer span around an agent run. Carries `agent.id`, `agent.name`, optional `session.id` and `user.id`. |
| `TOOL` | One invocation of a tool function by an agent. Carries `tool.name`, optional `tool_call.id`. |
| `CHAIN` | A chain step that is not itself an LLM call (routing logic, prompt assembly, post-processing). |
| `RETRIEVER` | A vector-search or document-lookup call. |
| `EMBEDDING` | An embedding-model call your code makes directly. |
| `LLM` | A model call made outside an instrumented SDK (rare, prefer the SDK patches). |
These six are the whole set. Under the hood a span is an ordinary
OpenTelemetry span, so you can attach any attribute you want to it (see
[Adding Custom Attributes](#adding-custom-attributes)). The *kind* is different:
it is a fixed vocabulary from
[OpenInference](https://github.com/Arize-ai/openinference) that tells the
dashboard how to render the span. A `TOOL` span gets a tool name and an
arguments/result panel, a `RETRIEVER` span gets a query and a document list, an
`LLM` span gets messages, token counts, and cost. Pick the kind that matches how
you want the step to appear; when none of the specific kinds fit, use `CHAIN`,
the generic "this is a step" kind. The kind strings come from `SpanKindValues`,
and the [Attributes reference](/integrations/traces/attributes) lists which
attributes each kind expects.
## Agent Spans
Use `agentSpan()` (TypeScript) or `agent_span()` (Python) as the outermost
wrapper around an agent run. Pass a stable `agentId` so the Catalyst Agents
dashboard groups executions correctly across renames and deploys. Pass
`sessionId` for one conversation, not as process-wide setup. Pass `userId` to
record the end user the run acts on behalf of so you can filter traces by user
on the dashboard. The wrapper also accepts free-form `metadata` and `tags`.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-ticket-123",
userId: "user_8675309",
role: "refunds",
system: "openai",
},
async (span) => {
const ticket = { id: "ticket_123", orderId: "ABC-123" };
span.setInput(ticket);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "user", content: `Review refund for ${ticket.orderId}` },
],
});
span.setOutput({ decision: response.choices[0]?.message.content });
if (response.usage != null) {
span.recordTokens({
prompt: response.usage.prompt_tokens ?? 0,
completion: response.usage.completion_tokens ?? 0,
});
}
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-ticket-123",
user_id="user_8675309",
agent_role="refunds",
system="openai",
) as span:
ticket = {"id": "ticket_123", "order_id": "ABC-123"}
span.set_input(ticket)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": f"Review refund for {ticket['order_id']}"},
],
)
span.set_output({"decision": response.choices[0].message.content})
if response.usage is not None:
span.record_usage(response.usage)
tracing.shutdown()
```
</CodeGroup>
The OpenAI call inside the callback runs in the agent span's active OTel context,
so the auto-emitted LLM span automatically parents under the AGENT span. That
is how a full trace tree forms with no extra plumbing.
See the [Handle API reference](/integrations/traces/handle-api) for the full
set of methods on the `span` argument.
## Tool, Chain, And Retriever Spans
When the work inside an agent loop is a tool call, a retrieval step, or a chain
step, wrap it in a child span with the right `SpanKind`. The child span
automatically parents under the active AGENT span and inherits its `agent.id`.
`manualSpan()` / `manual_span()` is the general-purpose helper. Pass
`spanKind` / `span_kind`, optional typed fields (`toolName` / `tool_name`,
`toolCallId` / `tool_call_id`, `model`, `usage`, `input`, `output`), and any
extra `attributes`. It defaults to `CHAIN`; pass `TOOL`, `RETRIEVER`, or
`EMBEDDING` when authoring the matching shape.
### Tool spans
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
async function executeTool(
name: string,
args: Record<string, unknown>,
callId: string,
) {
return await manualSpan(
{
spanName: `${name}.tool`,
spanKind: SpanKindValues.TOOL,
toolName: name,
toolCallId: callId,
input: args,
},
async (span) => {
const result = await TOOLS[name](args);
span.setOutput(result);
return result;
},
);
}
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import SpanKindValues, manual_span
def execute_tool(name: str, args: dict, call_id: str) -> dict:
with manual_span(
tracing.tracer,
name=f"{name}.tool",
span_kind=SpanKindValues.TOOL,
tool_name=name,
tool_call_id=call_id,
input=args,
) as span:
result = TOOLS[name](**args)
span.set_output(result)
return result
```
</CodeGroup>
### Chain spans
Use `CHAIN` for routing logic, prompt assembly, post-processing, or any other
non-LLM step that benefits from its own span. `metadata` and `tags` are
free-form attributes that show up under the span's attributes view in the
dashboard.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
await manualSpan(
{
spanName: "provider_router/select_model",
spanKind: SpanKindValues.CHAIN,
input: { route: "support", priority: "high" },
metadata: { customer_tier: "enterprise" },
tags: ["router", "support"],
},
async (span) => {
const selected = chooseModelForRequest();
span.setOutput({ model: selected.model, reason: selected.reason });
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import SpanKindValues, manual_span
with manual_span(
tracing.tracer,
name="provider_router/select_model",
span_kind=SpanKindValues.CHAIN,
input={"route": "support", "priority": "high"},
metadata={"customer_tier": "enterprise"},
tags=["router", "support"],
) as span:
selected = choose_model_for_request()
span.set_output({"model": selected.model, "reason": selected.reason})
```
</CodeGroup>
### Retriever spans
Use `RETRIEVER` for vector-store or document-lookup calls. Record the query as
input and the result list as output.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
await manualSpan(
{
spanName: "vector_store.search",
spanKind: SpanKindValues.RETRIEVER,
input: { query, k: 8 },
},
async (span) => {
const docs = await vectorStore.search(query, { k: 8 });
span.setOutput(docs.map((d) => ({ id: d.id, score: d.score })));
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import SpanKindValues, manual_span
with manual_span(
tracing.tracer,
name="vector_store.search",
span_kind=SpanKindValues.RETRIEVER,
input={"query": query, "k": 8},
) as span:
docs = vector_store.search(query, k=8)
span.set_output([{"id": d.id, "score": d.score} for d in docs])
```
</CodeGroup>
### Escape Hatch
`manualSpan` is the canonical path for every non-AGENT manual span. The only
reason to reach below it is `tracing.tracer.startActiveSpan()` directly, when
you need behavior the helper doesn't expose, like mid-callback span events paired
with custom status transitions, or a span lifetime that doesn't match a single
callback. With raw `startActiveSpan` you own status, exception recording, and
`span.end()` yourself.
For a full example that ties an AGENT span, several TOOL spans, and the
patched provider LLM spans together, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## Adding Custom Attributes
The typed handle covers the OpenInference vocabulary. For domain-specific
attributes (tenant ID, request channel, deploy environment), use
`setAttribute` / `set_attribute` on the handle. The helper normalizes values
into OTel-safe types (primitives and primitive arrays pass through; mixed
arrays and plain objects are JSON-stringified).
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
await agentSpan(
{ agentId: "support-agent", spanName: "support-agent.run" },
async (span) => {
span.setAttribute("app.tenant_id", tenantId);
span.setAttributes({
"app.request_channel": "slack",
"app.deploy_env": process.env.DEPLOY_ENV ?? "dev",
});
// ...
},
);
```
<Metadata />
```python Python theme={"system"}
with agent_span(
tracing.tracer,
agent_id="support-agent",
span_name="support-agent.run",
) as span:
span.set_attribute("app.tenant_id", tenant_id)
span.set_attributes({
"app.request_channel": "slack",
"app.deploy_env": os.environ.get("DEPLOY_ENV", "dev"),
})
# ...
```
</CodeGroup>
For anything outside what the handle provides (span events, mid-callback
exception recording, span-context propagation to background jobs), the raw
OTel span is available as `span.raw` (TypeScript) or `span.span` (Python).
Custom attributes appear under the span's attributes view in the dashboard and
in `inf span get <trace-id> <span-id> --view attributes`. They are also
filterable from the CLI with `--metadata "app.tenant_id=acme"`.
## How Identity And Context Propagate
The agent span runs its callback inside an active OTel context that carries:
* The OTel `Span` itself, so child spans created by `tracer.startActiveSpan` or
by patched provider SDKs auto-parent under it.
* A Catalyst-specific agent identity record (`agent.id`, `agent.name`,
`agent.role`), which the per-SDK patchers copy onto LLM child spans so the
Agents dashboard groups model calls under the right agent.
You do not need to pass `agentId` to every child span. If you create a tool
span inside an agent span, the agent identity flows through automatically.
```
AGENT support-agent.run agent.id=support-agent, session.id=conv-42
├── LLM OpenAI chat (auto-emitted, inherits agent.id)
├── TOOL lookup_order.tool tool.name=lookup_order
│ └── LLM OpenAI chat (nested call from inside the tool)
├── CHAIN prompt_builder.run
└── TOOL issue_refund.tool tool.name=issue_refund
```
If you need to inspect or copy the active identity (for example, to tag a
background job span with the same `agent.id`), import the helpers:
<Metadata />
```typescript TypeScript theme={"system"}
import { getActiveAgentIdentity } from "@inference/tracing";
const identity = getActiveAgentIdentity();
if (identity?.id) {
jobQueue.enqueue({ agentId: identity.id, payload });
}
```
## CLI And Subprocess Wrappers
When the work is a shell-out to a CLI (Claude Code, Codex, a custom binary),
wrap the process call in an agent span and set the input and output yourself.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
const execFileP = promisify(execFile);
const tracing = await setup();
const prompt = "Reply with just the word hello.";
await agentSpan(
{
agentId: "claude-code-prod",
agentName: "Claude Code",
spanName: "claude-code.invocation",
sessionId: "conversation-cli-hello",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const { stdout } = await execFileP("claude", ["-p", prompt], {
encoding: "utf-8",
timeout: 60_000,
});
span.setOutput(stdout.trim());
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import subprocess
from inference_catalyst_tracing import agent_span, setup
tracing = setup()
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="claude-code-prod",
agent_name="Claude Code",
span_name="claude-code.invocation",
session_id="conversation-cli-hello",
system="anthropic",
) as span:
span.set_input(prompt)
completed = subprocess.run(
["claude", "-p", prompt],
capture_output=True,
text=True,
timeout=60,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
</CodeGroup>
## Status Semantics
All Catalyst manual-span helpers follow the same status rules:
* Callback returns normally → span ends with `OK`.
* Callback throws → the exception is recorded as a span event, the span ends
with `ERROR` and the error message, and the original exception re-throws so
caller error handling is unaffected.
If you reach for `tracer.startActiveSpan()` directly (the OTel escape hatch
mentioned above), you own status and `span.end()` yourself, which is the
price of the lower-level API.
## Next Steps
<CardGroup>
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
Every method on the AGENT / TOOL / CHAIN span handle, with value-coercion rules.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
All `Attr.*` constants and `SpanKindValues` with the attributes each kind expects.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool execution, end to end.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Pick stable `agent.id` and `session.id` values for the Agents dashboard.
</Card>
</CardGroup>
# OpenAI Traces
Source: https://docs.inference.net/integrations/traces/openai
Trace OpenAI Chat Completions, tool calls, structured outputs, and Responses API calls.
Catalyst instruments the OpenAI SDK in both TypeScript and Python. Initialize
tracing before constructing clients so the OpenAI prototypes are patched before
application calls start.
## What Is Captured
* Chat Completions request and response messages
* Responses API input and output text
* Tool call names, IDs, and JSON arguments
* Tool result messages passed back into later turns
* JSON schema response format in invocation parameters
* Model name, finish reason, usage, and token counts
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing openai
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[openai]'
```
</CodeGroup>
## Basic Chat
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
serviceName: "checkout-agent",
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You answer in one short sentence." },
{ role: "user", content: "Summarize order ABC-123." },
],
max_tokens: 80,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import setup
from openai import OpenAI
tracing = setup(service_name="checkout-agent")
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You answer in one short sentence."},
{"role": "user", "content": "Summarize order ABC-123."},
],
max_tokens=80,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
## OpenAI Inside An Agent
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
await agentSpan(
{
agentId: "checkout-agent",
agentName: "Checkout Agent",
spanName: "checkout-agent.run",
sessionId: "conversation-checkout-1",
role: "checkout",
system: "openai",
},
async (span) => {
const input = "Summarize order ABC-123.";
span.setInput(input);
const response = await client.responses.create({
model: "gpt-4o-mini",
input,
});
span.setOutput(response.output_text);
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span
with agent_span(
tracing.tracer,
agent_id="checkout-agent",
agent_name="Checkout Agent",
span_name="checkout-agent.run",
session_id="conversation-checkout-1",
agent_role="checkout",
system="openai",
) as span:
user_input = "Summarize order ABC-123."
span.set_input(user_input)
response = client.responses.create(model="gpt-4o-mini", input=user_input)
span.set_output(response.output_text)
```
</CodeGroup>
## Tool Calling
The first model call records the assistant tool call. The second model call
records the tool result message and the final answer.
<Tip>
The capture below is the **model-side** view of tool calling — the
request/response the LLM saw. To also capture the **caller-side** view (the
actual function that ran, its input, output, and duration), wrap the tool
function in a `TOOL` span using
[Manual spans](/integrations/traces/manual-spans#tool-chain-and-retriever-spans).
For a full agent loop, see the
[Production Agent Example](/integrations/traces/production-agent-example).
</Tip>
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const tools = [
{
type: "function" as const,
function: {
name: "get_weather",
description: "Look up the current weather in a city.",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
];
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{ role: "user", content: "What's the weather in San Francisco?" },
];
const first = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
const toolCalls = first.choices[0]?.message.tool_calls ?? [];
messages.push({ role: "assistant", content: null, tool_calls: toolCalls });
for (const toolCall of toolCalls) {
const args = JSON.parse(toolCall.function.arguments) as { city: string };
messages.push({
role: "tool",
tool_call_id: toolCall.id,
content: JSON.stringify({ city: args.city, tempF: 62 }),
});
}
const final = await client.chat.completions.create({
model: "gpt-4o-mini",
messages,
tools,
});
console.log(final.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import json
import os
from inference_catalyst_tracing import setup
from openai import OpenAI
from openai.types.chat import ChatCompletionMessageParam
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Look up the current weather in a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
},
]
messages: list[ChatCompletionMessageParam] = [
{"role": "user", "content": "What's the weather in San Francisco?"},
]
first = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
tool_calls = first.choices[0].message.tool_calls or []
messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [tool_call.model_dump() for tool_call in tool_calls],
},
)
for tool_call in tool_calls:
args = json.loads(tool_call.function.arguments)
messages.append(
{
"role": "tool",
"tool_call_id": tool_call.id,
"content": json.dumps({"city": args["city"], "temp_f": 62}),
},
)
final = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
)
print(final.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
## Structured Outputs
`response_format` is included in invocation parameters, so you can inspect which
schema constrained the model call.
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "user",
content: "Extract the city, temperature, and unit from: 72F in Berlin.",
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "weather_report",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
city: { type: "string" },
temperature: { type: "number" },
unit: { type: "string", enum: ["F", "C"] },
},
required: ["city", "temperature", "unit"],
},
},
},
});
console.log(response.choices[0]?.message.content);
```
## Responses API
Responses API calls are traced with the same OpenInference attribute families as
Chat Completions.
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.responses.create({
model: "gpt-4o-mini",
input: "In one sentence, what is OpenTelemetry?",
});
console.log(response.output_text);
```
# OpenAI Agents Traces
Source: https://docs.inference.net/integrations/traces/openai-agents
Trace OpenAI Agents runs, tool calls, handoffs, and nested OpenAI model calls.
Pair OpenAI Agents instrumentation with OpenAI instrumentation so nested model
calls are captured. Once `setup()` runs (pass both modules in TypeScript;
Python auto-detects installed SDKs), the agent run, its tool calls, handoffs,
and nested model calls are all captured automatically. Wrap the run in
`agentSpan()` / `agent_span()` when you want stable identity and session
grouping.
<Info>
Three layers, in increasing order of effort:
* **`setup()`** traces the whole run automatically (pass the SDK modules in TypeScript; Python auto-detects). Enough on its own.
* **`agentSpan()` / `agent_span()`** adds a stable `agentId`, `agentName`, `role`, and `sessionId` so the Agents dashboard groups runs by conversation, plus your high-level input and output.
* **`manualSpan()` / `manual_span()`** captures steps the SDK never sees: your own retrieval, routing, or sub-steps inside a tool.
For the full breakdown of when to reach for each, see [How Tracing Fits Together](/integrations/traces/overview#how-tracing-fits-together).
</Info>
## Install
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing openai @openai/agents zod
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[openai-agents]'
```
</CodeGroup>
## Configure Export
Set the Catalyst endpoint and token before your app starts. `setup()` reads
these in every example below. Generate a token at
[API Keys](https://inference.net/dashboard/api-keys).
<Metadata />
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="support-agent"
```
## Minimal Setup
Pass the SDK modules to `setup()` in TypeScript, or let it auto-detect in
Python, then run your agent. The run, any tool calls, and the nested model calls
are all captured with no further code. This is everything you need for a single
one-shot agent.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run } from "@openai/agents";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Help customers with order questions.",
model: "gpt-4o-mini",
});
// No agentSpan needed: the run and its nested model calls are captured
// automatically.
const result = await run(supportAgent, "Where is order ABC-123?");
console.log(result.finalOutput);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import setup
from agents import Agent, Runner
tracing = setup()
support_agent = Agent(
name="SupportAgent",
instructions="Help customers with order questions.",
model="gpt-4o-mini",
)
# No agent_span needed: the run and its nested model calls are captured
# automatically.
result = Runner.run_sync(support_agent, "Where is order ABC-123?")
print(result.final_output)
tracing.shutdown()
```
</CodeGroup>
## Group Under An Agent
Wrap the run in `agentSpan()` to give it stable identity and session grouping.
This is the recommended shape for anything beyond a one-shot script. Pass
`userId` to record `user.id` so you can filter traces by user on the dashboard,
just like `sessionId`. For arbitrary keys, attach any custom attributes you want
to filter traces by inside the callback (`organization.id`, `chat.id`,
`order.id`). See
[Adding Custom Attributes](/integrations/traces/manual-spans#adding-custom-attributes).
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import * as agents from "@openai/agents";
import { Agent, run, tool } from "@openai/agents";
import OpenAI from "openai";
import { z } from "zod";
const tracing = await setup({
modules: { openai: OpenAI, openaiAgents: agents },
});
const lookupOrder = tool({
name: "lookup_order",
description: "Look up an order by ID.",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) =>
JSON.stringify({ orderId, status: "shipped", total: 42.5 }),
});
const supportAgent = new Agent({
name: "SupportAgent",
instructions: "Use tools to help customers with order questions.",
tools: [lookupOrder],
model: "gpt-4o-mini",
});
const userMessage = "Where is order ABC-123?";
await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
userId: "user_8675309",
role: "support",
system: "openai",
},
async (span) => {
span.setInput(userMessage);
// For arbitrary keys, attach whatever your app keys on. Any attribute is
// filterable from the dashboard and CLI, e.g.
// `inf span list --metadata "organization.id=org_42"`.
span.setAttribute("organization.id", "org_42");
const result = await run(supportAgent, userMessage, { maxTurns: 4 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import json
from inference_catalyst_tracing import agent_span, setup
from agents import Agent, Runner, function_tool
tracing = setup()
@function_tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
return json.dumps({"order_id": order_id, "status": "shipped", "total": 42.5})
support_agent = Agent(
name="SupportAgent",
instructions="Use tools to help customers with order questions.",
tools=[lookup_order],
model="gpt-4o-mini",
)
user_message = "Where is order ABC-123?"
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
user_id="user_8675309",
agent_role="support",
system="openai",
) as span:
span.set_input(user_message)
# For arbitrary keys, attach whatever your app keys on. Any attribute is
# filterable from the dashboard and CLI, e.g.
# `inf span list --metadata "organization.id=org_42"`.
span.set_attribute("organization.id", "org_42")
result = Runner.run_sync(support_agent, user_message, max_turns=4)
span.set_output(str(result.final_output))
tracing.shutdown()
```
</CodeGroup>
## Multi-Agent Handoff
Wrap the triage request once. The trace tree groups the triage agent,
specialist agent, nested model calls, and tools under the customer request.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { Agent, handoff, run, tool } from "@openai/agents";
const issueRefund = tool({
name: "issue_refund",
description: "Issue a refund for an order.",
parameters: z.object({ orderId: z.string(), amount: z.number() }),
execute: async ({ orderId, amount }) =>
JSON.stringify({ ok: true, orderId, refundId: "RFD-2201", amount }),
});
const refundsAgent = new Agent({
name: "RefundsAgent",
instructions: "Handle refund requests and use issue_refund.",
tools: [issueRefund],
model: "gpt-4o-mini",
});
const billingAgent = new Agent({
name: "BillingAgent",
instructions: "Answer billing questions. Do not issue refunds.",
model: "gpt-4o-mini",
});
const triageAgent = new Agent({
name: "TriageAgent",
instructions: "Route refund requests to RefundsAgent.",
handoffs: [handoff(refundsAgent), handoff(billingAgent)],
model: "gpt-4o-mini",
});
await agentSpan(
{
agentId: "triage-agent",
agentName: "Triage Agent",
spanName: "triage-agent.run",
sessionId: "conversation-refund-abc-123",
role: "triage",
system: "openai",
},
async (span) => {
const input = "I need a refund for order ABC-123, total $42.50.";
span.setInput(input);
const result = await run(triageAgent, input, { maxTurns: 8 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
```
<Metadata />
```python Python theme={"system"}
import json
from inference_catalyst_tracing import agent_span
from agents import Agent, Runner, function_tool, handoff
@function_tool
def issue_refund(order_id: str, amount: float) -> str:
"""Issue a refund for an order."""
return json.dumps(
{"ok": True, "order_id": order_id, "refund_id": "RFD-2201", "amount": amount}
)
refunds_agent = Agent(
name="RefundsAgent",
instructions="Handle refund requests and use issue_refund.",
tools=[issue_refund],
model="gpt-4o-mini",
)
billing_agent = Agent(
name="BillingAgent",
instructions="Answer billing questions. Do not issue refunds.",
model="gpt-4o-mini",
)
triage_agent = Agent(
name="TriageAgent",
instructions="Route refund requests to RefundsAgent.",
handoffs=[handoff(refunds_agent), handoff(billing_agent)],
model="gpt-4o-mini",
)
with agent_span(
tracing.tracer,
agent_id="triage-agent",
agent_name="Triage Agent",
span_name="triage-agent.run",
session_id="conversation-refund-abc-123",
agent_role="triage",
system="openai",
) as span:
user_input = "I need a refund for order ABC-123, total $42.50."
span.set_input(user_input)
result = Runner.run_sync(triage_agent, user_input, max_turns=8)
span.set_output(str(result.final_output))
```
</CodeGroup>
Use one stable ID for the outer customer-facing run. If you also emit separate
AGENT spans for handoff specialists, give each specialist its own stable
`agent.id`, such as `refunds-agent` or `billing-agent`.
## When To Add Manual Spans
The Agents SDK only captures the work it runs for you: the model calls, and the
tools you register with `tool()`. Anything you do yourself inside the
`agentSpan` callback is invisible to it. You do not wrap the SDK's own tool
calls in `manualSpan()`, those are already captured, and doing so would just
produce a duplicate span. You reach for `manualSpan()` for the steps the SDK
never sees.
Two common cases:
1. **A step around the run.** You fetch context from a vector store before the
run, or validate and reshape the output after it. Neither is an SDK call, so
author a `RETRIEVER` or `CHAIN` span yourself.
2. **Sub-steps inside a tool.** The SDK emits one TOOL span per tool
invocation. If that tool's `execute` does something you want broken out (a
vector search, a downstream API call, a transform), wrap it in a child span.
It nests under the SDK's TOOL span automatically.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
// Case 2: a sub-step inside a registered tool. The SDK already opened a TOOL
// span for this call; the RETRIEVER span below nests inside it.
const lookupOrder = tool({
name: "lookup_order",
description: "Look up an order by ID.",
parameters: z.object({ orderId: z.string() }),
execute: async ({ orderId }) => {
const docs = await manualSpan(
{
spanName: "order_docs.search",
spanKind: SpanKindValues.RETRIEVER,
input: { orderId, k: 4 },
},
async (span) => {
const results = await vectorStore.search(orderId, { k: 4 });
span.setOutput(results.map((d) => ({ id: d.id, score: d.score })));
return results;
},
);
return JSON.stringify({ orderId, status: "shipped", context: docs.length });
},
});
await agentSpan(
{
agentId: "support-agent",
agentName: "Support Agent",
spanName: "support-agent.run",
sessionId: "conversation-order-abc-123",
role: "support",
system: "openai",
},
async (span) => {
const userMessage = "Where is order ABC-123?";
span.setInput(userMessage);
// Case 1: a pre-run step the SDK never sees. You classify the request
// yourself before handing it to the agent.
const route = await manualSpan(
{
spanName: "router.classify",
spanKind: SpanKindValues.CHAIN,
input: { userMessage },
},
async (chain) => {
const decision = classifyRequest(userMessage);
chain.setOutput(decision);
return decision;
},
);
const result = await run(supportAgent, userMessage, { maxTurns: 4 });
span.setOutput(String(result.finalOutput ?? ""));
},
);
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span, manual_span, SpanKindValues
from agents import Runner, function_tool
# Case 2: a sub-step inside a registered tool. The SDK already opened a TOOL
# span for this call; the RETRIEVER span below nests inside it.
@function_tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
with manual_span(
tracing.tracer,
name="order_docs.search",
span_kind=SpanKindValues.RETRIEVER,
input={"order_id": order_id, "k": 4},
) as span:
results = vector_store.search(order_id, k=4)
span.set_output([{"id": d.id, "score": d.score} for d in results])
return json.dumps(
{"order_id": order_id, "status": "shipped", "context": len(results)}
)
with agent_span(
tracing.tracer,
agent_id="support-agent",
agent_name="Support Agent",
span_name="support-agent.run",
session_id="conversation-order-abc-123",
agent_role="support",
system="openai",
) as span:
user_message = "Where is order ABC-123?"
span.set_input(user_message)
# Case 1: a pre-run step the SDK never sees. You classify the request
# yourself before handing it to the agent.
with manual_span(
tracing.tracer,
name="router.classify",
span_kind=SpanKindValues.CHAIN,
input={"user_message": user_message},
) as chain:
decision = classify_request(user_message)
chain.set_output(decision)
result = Runner.run_sync(support_agent, user_message, max_turns=4)
span.set_output(str(result.final_output))
```
</CodeGroup>
The rule of thumb: if the Agents SDK runs the code (a model call, a registered
tool), it is captured for you. If you run the code (retrieval, routing,
validation, or a sub-step inside a tool), wrap it in `manualSpan`. See
[Manual Spans](/integrations/traces/manual-spans) for the full set of span
kinds and the handle API.
## Flushing
Every example above ends with `await tracing.shutdown()`, which flushes batched
spans before the process exits. Long-lived servers and serverless or edge
runtimes need a different flush strategy, since the process does not exit after
each run. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle)
for all three shapes.
## Next Steps
<CardGroup>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Author TOOL, CHAIN, and RETRIEVER spans for work the SDK does not capture.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Pick stable `agent.id` and `session.id` values for the Agents dashboard.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool spans, end to end.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
Every `Attr.*` constant and `SpanKindValues` value the SDK emits.
</Card>
</CardGroup>
# OpenCode Traces
Source: https://docs.inference.net/integrations/traces/opencode
Record traces of your own OpenCode coding sessions to Catalyst.
[OpenCode](https://opencode.ai) is an open-source coding agent you run in your
terminal. The Catalyst OpenCode plugin records those coding sessions so you can
review what the agent did: model calls, tool calls, token usage, cost, finish
reasons, errors, prompts, responses, and tool payloads.
This page is for observing a coding tool **you use**. To trace an agent or
application **you build**, start with the [Tracing Quickstart](/integrations/traces/quickstart)
and the SDK integrations.
<Info>
OpenCode installs npm plugins automatically from your `opencode.json`; you do
not need to add this package to your app's dependencies.
</Info>
## Install
Add the plugin to your OpenCode config, either in your project `opencode.json`
or globally at `~/.config/opencode/opencode.json`.
```json opencode.json theme={"system"}
{
"$schema": "https://opencode.ai/config.json",
"plugin": ["@inference/trace-opencode"]
}
```
## Configure
Set your Catalyst OTLP token. Recording turns on automatically when the token is
present.
```bash theme={"system"}
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="my-opencode"
```
Then use OpenCode as usual.
```bash theme={"system"}
opencode run "add a health check endpoint and a test for it"
```
## What Gets Recorded
Every turn, meaning one prompt and the agent's response to it, becomes a trace
in Catalyst. Spans carry the OpenCode `session.id`, so all turns from the same
coding session can be grouped together.
* The `AGENT` root span records session id, service identity, call counts, stop
reason, and error status.
* `LLM` spans record model/provider details, finish reason,
prompt/cache/completion/reasoning tokens, and cost.
* `TOOL` spans record shell, file, and MCP tool calls with timing and error
status.
By default, spans include prompts, model output, tool arguments, and tool
output. Set `CATALYST_REDACT_CONTENT=true` to keep spans metadata-only.
```text theme={"system"}
Turn (AGENT, session.id=ses_...)
anthropic/claude-... (LLM: tokens, cache, finish reason, cost)
bash: ... (TOOL: name, timing, status)
read: file.ts (TOOL)
anthropic/claude-... (LLM: final reply)
```
Use these traces to audit what a session did, debug failed turns, compare how
models behave on similar tasks, or track coding-agent cost. Turn on content
redaction when you only need metadata and timing.
## Configuration Reference
The plugin uses the same Catalyst tracing env-var convention as the TypeScript
and Python tracing SDKs.
| Environment variable | Default | Description |
| ------------------------------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `CATALYST_OTLP_TOKEN` | - | Catalyst ingest token. Recording starts when this is set. |
| `CATALYST_OTLP_ENDPOINT` | `https://telemetry.inference.net` | OTLP/HTTP trace ingest endpoint. |
| `CATALYST_SERVICE_NAME` | `opencode` | Stable OTel `service.name`, also used as the OpenCode agent id. |
| `TRACE_TO_CATALYST` | on when a token is set | Set `false` to disable. Set `true` only for tokenless custom collectors with an explicit non-hosted endpoint. |
| `CATALYST_REDACT_CONTENT` | `false` | Set `true` to keep prompts, model outputs, tool arguments, and tool outputs off traces. |
| `CATALYST_ADDITIONAL_METADATA` | - | JSON object attached to each turn's metadata. |
| `CATALYST_DEBUG` | `false` | Verbose plugin logging through OpenCode's log API. |
### Local Collector
To export to a local collector without an auth token, set both an explicit
endpoint and `TRACE_TO_CATALYST=true`.
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="http://localhost:4318"
export TRACE_TO_CATALYST=true
```
## Privacy
By default the plugin records prompts, model output, file content read or
written by tools, and shell command output. Set
`CATALYST_REDACT_CONTENT=true` when you want metadata-only traces.
## Troubleshooting
If no traces appear, check:
* `CATALYST_OTLP_TOKEN` is set in the environment that launches OpenCode.
* `CATALYST_SERVICE_NAME` is the value you are filtering for in Catalyst.
* `CATALYST_OTLP_ENDPOINT` is `https://telemetry.inference.net` unless you are
intentionally using a custom collector.
* OpenCode has restarted after you added the plugin to `opencode.json`.
# Traces
Source: https://docs.inference.net/integrations/traces/overview
Capture the full execution of your AI apps and agents with first-party OpenInference-shaped tracing SDKs for TypeScript and Python.
Catalyst Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Spans are emitted as OpenInference-shaped OpenTelemetry, exported over OTLP/HTTP, and shown in Catalyst as trace trees, agent sessions, and per-span IO.
Use tracing for agent improvement. Once spans are flowing, point [Halo](https://github.com/context-labs/halo), our open-source agent-loop optimizer, at your traces and get back concrete suggestions for fixing prompts, tools, and harness logic. See [Analyze your traces](/get-started/analyze-traces) for the dashboard and Halo flow.
<Info>
Tracing and the Catalyst Gateway are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For gateway-only setups, see [model provider integrations](/integrations/overview).
</Info>
## How Tracing Fits Together
Tracing has three layers. You add them in order, and most apps stop at the
second.
**1. Automatic SDK spans.** Pass an SDK to `setup({ modules })` and every call
through it becomes its own span, with input messages, output, model name,
finish reason, and token counts. You change nothing at the call site. For a
one-shot script, or a batch of unrelated LLM calls, this is all you need.
**2. An agent span around the run.** When one request drives several model or
tool calls (an agent run, a conversation turn, a workflow), wrap it in
`agentSpan`. The agent span is the parent row the SDK spans nest under, so you
see "Support Agent" with "openai chat" and "lookup\_order" beneath it instead of
three orphan rows. It carries `agentId`, `agentName`, `role`, and `sessionId`
for grouping in the Agents dashboard, and holds the run's high-level input and
output. Without it the SDK spans are still captured, just as a flat list with no
agent or conversation to group them by.
**3. Manual spans for your own steps.** When a step inside the run is not an SDK
call (a tool you execute yourself, a retrieval, a router, a validation pass),
wrap it in `manualSpan` with the matching span kind. It nests under the agent
span next to the auto-captured LLM spans, so the trace tree reflects everything
the run did, not just the model calls.
| You're doing... | What you need |
| --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| One-shot call, or independent calls with no orchestration | `setup({ modules })`. Done. |
| Several calls per request, one agent | Add `agentSpan({ agentId, agentName, sessionId })` around the run |
| Multi-agent system | A separate `agentSpan` per agent, each with its own `agentId` and `role` |
| A non-SDK step inside the run (tool, retrieval, router, validator) | `manualSpan({ spanKind: TOOL / CHAIN / RETRIEVER })` inside the agent span |
| Subprocess or CLI call | `agentSpan` around it (no SDK to auto-patch) |
| Framework with native capture (LangGraph, OpenAI Agents, Cursor SDK, Vercel Eve, Vercel AI SDK, LiveKit, ElevenLabs, Claude Agent SDK, PI AI) | Use the framework guide's setup path; add `agentSpan` when you need stable identity and session grouping |
<Tip>
Set `sessionId`, `agentId`, and `agentName` once on the outer `agentSpan`.
Every span underneath inherits them, so you should not be setting these in
multiple places.
</Tip>
## Packages
Two packages, one shape. Install whichever matches your runtime.
| Runtime | Install | Import | Source |
| ----------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------- | ------ |
| TypeScript / Node / Bun | [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) | `import { setup } from "@inference/tracing"` | npm |
| Python (3.10+) | [`inference-catalyst-tracing`](https://pypi.org/project/inference-catalyst-tracing/) | `from inference_catalyst_tracing import setup` | PyPI |
Both packages share the same `setup()` entry point, the same env-var contract, and the same OpenInference attribute conventions. Pick whichever fits your stack and write the same code shape in either language.
<Tip>
Provider SDKs (`openai`, `@anthropic-ai/sdk`, `langchain`, etc.) are optional peer dependencies. Install only the ones you use. For Python, install per-integration extras: `pip install 'inference-catalyst-tracing[openai,langchain]'`.
</Tip>
## Getting Started
The fastest path is the [Inference CLI](/cli/overview):
<Metadata />
```bash theme={"system"}
inf instrument --mode tracing
```
`tracing` mode installs the SDK and wires up `setup()`. `both` mode does the same plus gateway routing in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See [Install with AI](/integrations/install-with-ai) for the full flow.
Prefer to wire it up by hand? Start with the path that matches what you want to instrument:
1. [Quickstart](/integrations/traces/quickstart) installs the SDK, configures export, and captures your first provider span. Both `Install with AI` and `Install manually` paths included.
2. [Provider and framework guides](#supported-trace-integrations) cover OpenAI, Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, Claude Agent SDK, Claude Code, Pydantic AI, ElevenLabs Agents, Vercel Eve, PI AI, Cursor SDK, the Vercel AI SDK, and other supported surfaces.
3. [Manual spans](/integrations/traces/manual-spans) wrap custom agents, subprocesses, tools, retrieval, routing, and unsupported SDKs.
4. [Agent identity](/integrations/traces/agent-identity) sets stable `agent.id` values so the Agents dashboard groups executions correctly.
## Supported Trace Integrations
* [OpenAI](/integrations/traces/openai): Trace Chat Completions, tool calls, structured outputs, and Responses API calls.
* [Anthropic](/integrations/traces/anthropic): Trace Messages API calls, tool use round trips, and prompt-cache usage.
* [LangChain](/integrations/traces/langchain): Capture chains, agents, model calls, and tools through callback instrumentation.
* [LangGraph](/integrations/traces/langgraph): Capture graph and node spans while preserving parent-child relationships.
* [LangSmith](/integrations/traces/langsmith): Bridge LangSmith OpenTelemetry spans into Catalyst.
* [OpenAI Agents](/integrations/traces/openai-agents): Trace agent runs, tools, handoffs, and nested OpenAI model calls.
* [LiveKit Agents](/integrations/traces/livekit-agents): Trace LiveKit sessions, model nodes, and function tools through native OTel spans.
* [ElevenLabs Agents](/integrations/traces/elevenlabs): Trace conversation sessions, transcripts, and client tool calls.
* [Vercel Eve](/integrations/traces/eve): Trace Eve agent turns, model calls, sub-agent invocations, tools, and `$eve.*` workflow tags.
* [PI AI](/integrations/traces/pi-ai): Trace PI AI model turns, streams, tool calls, usage, costs, and agent identity.
* [Cursor SDK](/integrations/traces/cursor-sdk): Trace Cursor agent runs, streamed messages, and tool calls.
* [Vercel AI SDK](/integrations/traces/ai-sdk): Trace generateText, streamText, ToolLoopAgent, tool calls, structured output, and usage.
* [Claude Agent SDK](/integrations/traces/claude-agent-sdk): Trace Claude Agent SDK query loops and yielded messages.
* [Claude Code SDK](/integrations/traces/claude-code-sdk): Trace Claude Code CLI subprocess calls and SDK-style agent invocations.
* [OpenCode](/integrations/traces/opencode): Record your own OpenCode coding sessions with LLM calls, tool calls, token usage, cost, and errors.
* [Pydantic AI](/integrations/traces/pydantic-ai): Trace Pydantic AI agent runs and tool calls (Python only).
* [Manual spans](/integrations/traces/manual-spans): Wrap custom agents, CLI calls, provider routing, and unsupported SDKs.
* [Agent identity](/integrations/traces/agent-identity): Set stable agent IDs, display names, and roles for dashboard grouping.
## What Gets Captured
| Span data | Examples |
| ------------------ | ------------------------------------------------------------------------ |
| Inputs and outputs | `input.value`, `output.value` |
| Messages | user, system, assistant, tool, and tool-result messages |
| Tool calls | tool names, IDs, JSON arguments, and tool results |
| Model metadata | `llm.model_name`, `llm.invocation_parameters`, `gen_ai.system` |
| Usage | prompt, completion, total, and prompt-cache token counts when available |
| Agent structure | agent spans, framework spans, tool spans, graph/node spans |
| Agent identity | `agent.id`, `agent.name`, and `agent.role` for Agents dashboard grouping |
| Errors | exception status and error details on failed spans |
All keys follow [OpenInference](https://github.com/Arize-ai/openinference) conventions, so the same spans can be consumed by any OpenInference-aware viewer.
## Configuration
Both SDKs read the same env vars and accept the same options on `setup()`.
### Env Vars And Identity
| Env var | Purpose |
| -------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `CATALYST_OTLP_ENDPOINT` | OTLP/HTTP trace ingest endpoint. Set to `https://telemetry.inference.net` for the hosted Catalyst backend. |
| `CATALYST_OTLP_TOKEN` | Bearer token for ingest auth. Generate one at [API Keys](https://inference.net/dashboard/api-keys). |
| `CATALYST_SERVICE_NAME` | OTel `service.name` resource attribute. Use a stable value per deployed service. |
| `CATALYST_SERVICE_VERSION` | OTel `service.version` resource attribute. |
| `CATALYST_DEBUG` | Enable SDK debug logging. |
Precedence: explicit options on `setup()` win, then `CATALYST_*` env vars, then SDK defaults. Legacy `OTLP_ENDPOINT`, `OTLP_INGEST_TOKEN`, and `SERVICE_NAME` are still accepted but prefer `CATALYST_*` in new setups.
### Instrumentation Control
Two options control which SDKs Catalyst patches at setup time.
| Option | Default | Purpose |
| ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `modules` (TypeScript) | `{}` | Map of SDK modules to patch. Pass `{ openai: OpenAI, anthropic: Anthropic }`. Entries listed here are always patched. Python auto-detects installed packages and does not need this option. |
| `autoInstrument` | `true` | When `true`, `setup()` also scans `node_modules` and patches any other supported SDK it finds. When `false`, only the SDKs you list in `modules:` get patched, nothing else. |
Three patterns, in order of how often they show up in real apps:
**Auto-detect everything.** Call `setup()` with no `modules:`. The auto-scan finds every supported SDK and patches it. Easiest path for scripts, demos, and small apps.
```ts theme={"system"}
const tracing = await setup();
```
**Explicit modules plus auto-detect.** Pass the SDKs you use to `modules:`. Those entries are patched directly (no resolver gymnastics), and `setup()` also auto-detects anything else.
```ts theme={"system"}
const tracing = await setup({ modules: { openai: OpenAI } });
```
**Explicit only.** Pass `modules:` and set `autoInstrument: false`. Only the SDKs you list get patched, even if a transitive dep happens to pull in another supported one. Cleanest pattern for production services that know exactly which providers they call.
```ts theme={"system"}
const tracing = await setup({
modules: { anthropic: Anthropic },
autoInstrument: false,
});
```
<Tip>
Pass only the SDK modules you actually use to `modules`. Auto-instrumentation
patches prototypes, so patching an SDK you do not call adds no runtime cost
but a small import cost.
</Tip>
### Imports (TypeScript)
For almost all setups, `@inference/tracing` is the only import you need.
`setup()` auto-patches every supported SDK it can find (OpenAI, Anthropic,
LangChain, LangGraph, ElevenLabs, LiveKit, PI AI, and the rest), so you do not need
to reach for the per-integration helpers yourself.
| Export | What it does |
| ------------------------- | ----------------------------------------------------------------------------------------------------- |
| `setup` | Initializes tracing. Pass the SDK modules you use; it patches them and returns a handle for shutdown. |
| `agentSpan` | Opens a parent span that groups everything an agent run does. Use one per top-level invocation. |
| `manualSpan` | Opens a custom span for work that is not an SDK call: tool execution, retrieval, validation, etc. |
| `wrapClaudeAgentSdkQuery` | Wraps a Claude Agent SDK `query()` call so each tool turn becomes a span. |
| `Attr`, `SpanKindValues` | OpenInference attribute keys and span-kind enums for setting custom attributes. |
| Error types | `CatalystInstrumentationError` and friends, for catching setup failures. |
#### When you need a subpath
Three cases call for importing from a per-integration subpath:
1. **You're using the Vercel AI SDK.** The AI SDK has its own
`experimental_telemetry` option, so there is nothing for `setup()` to
patch. You import `createAISdkTelemetrySettings` and pass its result into
each `generateText` / `streamText` call.
2. **You're using Vercel Eve.** Eve enables telemetry by loading
`agent/instrumentation.ts`, so you export `defineCatalystEveInstrumentation()`
from `@inference/tracing/eve` in that file.
3. **You're installing instrumentation on a custom tracer provider** instead of the one `setup()` registers. Rare in practice; most apps want the standard flow. The subpath helpers (`installOpenAIInstrumentation`, etc.) accept an explicit provider argument.
If neither applies, you can skip the table below.
| Import | Exports | Use For |
| ------------------------------------- | -------------------------------------------------------------------------------------------- | ----------------------- |
| `@inference/tracing/ai-sdk` | `createAISdkTelemetrySettings`, `instrumentAISdk`, `installAISdkInstrumentation` | Vercel AI SDK |
| `@inference/tracing/openai` | `instrumentOpenAI`, `installOpenAIInstrumentation` | OpenAI |
| `@inference/tracing/anthropic` | `instrumentAnthropic`, `installAnthropicInstrumentation` | Anthropic |
| `@inference/tracing/langchain` | `instrumentLangChain`, `installLangChainInstrumentation` | LangChain |
| `@inference/tracing/langgraph` | `instrumentLangGraph`, `installLangChainInstrumentation` | LangGraph |
| `@inference/tracing/langsmith` | `installLangSmith` | LangSmith bridge |
| `@inference/tracing/openai-agents` | `instrumentOpenAIAgents` | `@openai/agents` |
| `@inference/tracing/claude-agent-sdk` | `wrapClaudeAgentSdkQuery` | Claude Agent SDK |
| `@inference/tracing/elevenlabs` | `instrumentElevenLabs`, `installElevenLabsInstrumentation` | ElevenLabs Agents |
| `@inference/tracing/eve` | `defineCatalystEveInstrumentation`, `installEveSpanEnrichment` | Vercel Eve |
| `@inference/tracing/pi-ai` | `instrumentPiAi`, `installPiAiInstrumentation` | PI AI |
| `@inference/tracing/cursor-sdk` | `instrumentCursorSdk`, `installCursorSdkInstrumentation` | Cursor SDK |
| `@inference/tracing/livekit-agents` | `instrumentLiveKitAgents`, `installLiveKitAgentsInstrumentation` | LiveKit Agents |
| `@inference/tracing/semconv` | `OpenInferenceAttribute`, `MessageRole` (plus `Attr` and `SpanKindValues`, also on the root) | OpenInference constants |
Python uses one import root (`inference_catalyst_tracing`) and gates each
integration behind a pip extra (`pip install 'inference-catalyst-tracing[openai,anthropic]'`).
### The `setup()` Return Value
`setup()` returns a `CatalystTracing` handle:
| Field | Purpose |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tracer` | The OpenTelemetry `Tracer` for authoring manual spans. In Python, pass it to `agent_span` / `manual_span`. In TypeScript, `agentSpan` / `manualSpan` resolve it automatically after `setup()`, so this is an escape hatch — use it for `tracer.startActiveSpan(...)` directly. |
| `provider` | The OTel tracer provider, in case you want to register additional span processors or exporters. |
| `installResults` | Per-integration install result: which SDKs were detected, which were patched, and any errors. Useful in CI and on startup logs. |
| `shutdown()` | Flush and close tracing. Call this on `SIGTERM` for long-lived services, before `process.exit(0)` for scripts. |
For long-lived and serverless processes, memoize the `setup()` promise and
flush correctly for the process shape. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle).
## How It Works
1. `setup()` configures an OpenTelemetry tracer provider with a Catalyst OTLP/HTTP exporter.
2. The SDK auto-patches any supported provider or framework SDK it finds. In TypeScript you pass them via `modules`. In Python it auto-detects installed packages.
3. Your app runs normally. Spans are batched and exported in the background.
4. Spans land in Catalyst grouped by service, trace, span, and task metadata. View them in the [Traces tab or the Agents tab](/get-started/analyze-traces).
5. Run analysis on them with [Halo](https://github.com/context-labs/halo) to understand your traces and gain insights you can pass into your favorite coding agent.
## Inspect Traces from the CLI
After spans are flowing into Catalyst, you can inspect the same trace data from the terminal:
<Metadata />
```bash theme={"system"}
# Browse recent traces
inf trace list --range 1h
# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline
# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io
```
See [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) for the full command reference.
## Next Steps
<CardGroup>
<Card title="Quickstart" icon="rocket" href="/integrations/traces/quickstart">
Install an SDK, configure export, and capture your first model span.
</Card>
<Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
A production-shaped agent with custom tool spans, end to end.
</Card>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Author AGENT, TOOL, CHAIN, and RETRIEVER spans.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
Every `Attr.*` constant and `SpanKindValues` value.
</Card>
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
Every method on the span handle yielded by `agentSpan` / `manualSpan` and the Python equivalents.
</Card>
<Card title="Instrumentation examples" icon="code" href="/integrations/traces/examples">
Copyable examples for providers, agents, tool loops, and prompt caching.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable IDs and readable names to agent spans.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
Debug missing spans, missing attributes, and shutdown behavior.
</Card>
</CardGroup>
# PI AI Traces
Source: https://docs.inference.net/integrations/traces/pi-ai
Trace PI AI model turns, streamed output, tool calls, token usage, costs, and agent identity through Catalyst.
Catalyst instruments `@mariozechner/pi-ai` in TypeScript. PI AI is a unified
LLM API and provider registry. Catalyst patches the registered PI AI providers
so each model turn through `stream`, `streamSimple`, `complete`, or
`completeSimple` emits an OpenInference LLM span.
Use this guide for Node applications that use PI AI for model calls, tool
calling, cross-provider handoffs, or agent workflows. PI AI is a TypeScript
package, so there is no Python equivalent for this integration.
## What Is Captured
* One LLM span per PI AI model turn, named like `pi-ai.<provider>.turn`
* `stream` and `streamSimple` calls, plus `complete` and `completeSimple`
calls because PI AI completes by resolving the underlying stream
* System prompts, input messages, assistant output, model name, provider, and
invocation parameters
* Tool call IDs, names, and JSON arguments from assistant messages
* Token usage, prompt cache read/write counts, and total cost when PI AI
returns them
* Finish reason, errors, aborts, and exception details
* Active `agentSpan()` identity, including `agent.id`, `agent.name`,
`agent.role`, and `session.id`
## Install
<Metadata />
```bash TypeScript theme={"system"}
bun add @inference/tracing @mariozechner/pi-ai
```
## Configure Export
Set the Catalyst endpoint and token before your app starts. Generate a token at
[API Keys](https://inference.net/dashboard/api-keys).
<Metadata />
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="pi-ai-agent"
export OPENAI_API_KEY="<your-openai-api-key>"
```
## Initialize Tracing
Initialize Catalyst before making PI AI calls. Import the PI AI namespace and
pass it to `setup()` so Catalyst patches the same provider registry your app
uses.
<Metadata />
```typescript TypeScript theme={"system"}
import * as piAi from "@mariozechner/pi-ai";
import { setup } from "@inference/tracing";
const tracing = await setup({
serviceName: process.env.CATALYST_SERVICE_NAME ?? "pi-ai-agent",
modules: { piAi },
});
```
For explicit manual initialization, disable auto-instrumentation and use the PI
AI subpath helper.
<Metadata />
```typescript TypeScript theme={"system"}
import * as piAi from "@mariozechner/pi-ai";
import { setup } from "@inference/tracing";
import { instrumentPiAi } from "@inference/tracing/pi-ai";
const tracing = await setup({ autoInstrument: false });
instrumentPiAi(piAi, tracing);
```
## Complete Call
`completeSimple()` is the smallest path. PI AI resolves it through a provider
stream, so Catalyst emits the LLM span when the call finishes.
The examples below assume you initialized `tracing` with the setup block above.
<Metadata />
```typescript TypeScript theme={"system"}
import { completeSimple, getModel, type Context } from "@mariozechner/pi-ai";
const model = getModel("openai", "gpt-4o-mini");
const context: Context = {
systemPrompt: "You answer in one concise sentence.",
messages: [{ role: "user", content: "Why are trace trees useful?" }],
};
const response = await completeSimple(model, context);
console.log(response.content);
```
Expected span:
* `pi-ai.openai.turn`
Expected promoted fields include `llm.model_name`, `input.value`,
`output.value`, `llm.token_count.prompt`, `llm.token_count.completion`,
`llm.token_count.total`, and `llm.invocation_parameters` when PI AI returns the
corresponding data.
## Streaming
For streaming calls, consume the stream and call `await stream.result()` before
process shutdown. Catalyst finishes the span when PI AI emits a done/error event
or when `.result()` resolves.
<Metadata />
```typescript TypeScript theme={"system"}
import { getModel, streamSimple, type Context } from "@mariozechner/pi-ai";
const model = getModel("openai", "gpt-4o-mini");
const context: Context = {
messages: [{ role: "user", content: "Stream a sentence about observability." }],
};
const stream = streamSimple(model, context);
for await (const event of stream) {
if (event.type === "text_delta") {
process.stdout.write(event.delta);
}
}
const finalMessage = await stream.result();
console.log("\nFinished:", finalMessage.stopReason);
```
For short-lived scripts, call `await tracing.shutdown()` after the stream is
fully consumed.
## Tool Loop
PI AI returns tool calls in assistant message content. Catalyst records the tool
call name, ID, and arguments on the PI AI LLM span. Your local tool execution is
application code, so wrap it with `manualSpan()` if you also want a TOOL child
span for the work your app performs.
<Metadata />
```typescript TypeScript theme={"system"}
import {
Type,
completeSimple,
getModel,
type Context,
type Tool,
} from "@mariozechner/pi-ai";
import { manualSpan, SpanKindValues } from "@inference/tracing";
const model = getModel("openai", "gpt-4o-mini");
const tools: Tool[] = [
{
name: "lookup_order",
description: "Look up an order by ID.",
parameters: Type.Object({
orderId: Type.String({ description: "Customer order ID." }),
}),
},
];
const context: Context = {
systemPrompt: "Use tools when you need order data.",
messages: [{ role: "user", content: "Where is order ABC-123?" }],
tools,
};
const firstTurn = await completeSimple(model, context);
context.messages.push(firstTurn);
for (const block of firstTurn.content) {
if (block.type !== "toolCall" || block.name !== "lookup_order") continue;
const args = block.arguments as { orderId: string };
const order = await manualSpan(
{
spanName: "lookup_order",
spanKind: SpanKindValues.TOOL,
toolName: block.name,
toolCallId: block.id,
input: args,
},
async (span) => {
const result = {
orderId: args.orderId,
status: "shipped",
eta: "Friday",
};
span.setOutput(result);
return result;
},
);
context.messages.push({
role: "toolResult",
toolCallId: block.id,
toolName: block.name,
content: [{ type: "text", text: JSON.stringify(order) }],
isError: false,
timestamp: Date.now(),
});
}
const finalTurn = await completeSimple(model, context);
console.log(finalTurn.content);
```
Expected spans:
* `pi-ai.openai.turn` for the first model turn with the tool call
* `lookup_order` TOOL span when you wrap local execution with `manualSpan()`
* another `pi-ai.openai.turn` for the continuation after the tool result
## Stable Agent Identity
Wrap the full PI AI run in `agentSpan()` when one user request can produce
multiple model turns or tool executions. PI AI LLM spans inherit the active
agent identity and nest under the AGENT span.
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
import { completeSimple, getModel, type Context } from "@mariozechner/pi-ai";
const model = getModel("openai", "gpt-4o-mini");
await agentSpan(
{
agentId: "pi-ai-support-agent",
agentName: "PI AI Support Agent",
spanName: "pi-ai-support-agent.run",
sessionId: "conversation-order-abc-123",
role: "support",
system: "pi-ai",
},
async (span) => {
const input = "Summarize order ABC-123 for the customer.";
const context: Context = {
messages: [{ role: "user", content: input }],
};
span.setInput(input);
const response = await completeSimple(model, context);
span.setOutput(response.content);
},
);
```
## Verify In Catalyst
Filter traces by your `service.name`, for example `pi-ai-agent`. A successful
run should show PI AI LLM spans named by provider, such as
`pi-ai.openai.turn`, with captured input/output, model metadata, usage, finish
reason, and tool call attributes. If you wrap the run in `agentSpan()`, the same
LLM spans should show the inherited `agent.id`, `agent.name`, and `agent.role`.
If no PI AI spans appear:
* Initialize Catalyst before making PI AI calls.
* Pass the PI AI namespace directly with `modules: { piAi }` or
`instrumentPiAi(piAi, tracing)`.
* Consume streaming results or call `.result()` so PI AI can finish the turn.
* Call `await tracing.shutdown()` before process exit in short-lived scripts.
# Production Agent Example
Source: https://docs.inference.net/integrations/traces/production-agent-example
A production-shaped agent with custom tool execution, end to end. Memoized setup, parent agent span, per-tool TOOL spans, domain attributes, and graceful shutdown.
This example walks through a realistic agent loop end to end: a long-lived
server that handles incoming messages, runs an LLM-driven agent that calls
several custom tools, and exports a clean OpenInference trace tree to
Catalyst. Every piece in this guide maps to a real pattern used by agents in
production today.
By the end you will have:
* A boot-time `setup()` that runs once per process.
* A request handler that wraps the whole agent run in an `AGENT` span with
stable `agent.id` and per-conversation `session.id`.
* Custom tool execution wrapped in `TOOL` spans with `tool.name` and
`tool_call.id`.
* Auto-emitted `LLM` child spans from the patched Anthropic SDK, nested under
the agent span by OTel context propagation.
* Domain-specific attributes (tenant, channel, viewer role) on the agent span
for filtering in the dashboard.
* A graceful shutdown that flushes batched spans on `SIGTERM`.
<Info>
This example **hand-rolls the agent loop** with the raw Anthropic SDK, so you
author the `TOOL` spans yourself (Step 3). The patched SDK captures the `LLM`
calls automatically, but your tools run in your own code, so nothing emits a
tool span unless you do.
If you use a framework that runs the tools for you (OpenAI Agents, LangGraph),
the framework emits the `TOOL` spans and you skip Step 3. There you would only
add manual spans for steps the framework never sees, like a retrieval inside a
tool. See [OpenAI Agents](/integrations/traces/openai-agents) for that path.
</Info>
## Step 1 — Bootstrap Tracing Once
Tracing should initialize **once per process**, not per request. For a
long-lived server, that means a memoized `setup()` call that any code path
can `await`.
<Metadata />
```typescript TypeScript theme={"system"}
// tracing.ts
import Anthropic from "@anthropic-ai/sdk";
import { setup, type CatalystTracing } from "@inference/tracing";
let tracingPromise: Promise<CatalystTracing> | null = null;
export function initTracing(): Promise<CatalystTracing> {
if (!tracingPromise) {
tracingPromise = setup({
serviceName: process.env.SERVICE_NAME ?? "customer-support",
serviceVersion: process.env.SERVICE_VERSION,
endpoint: process.env.CATALYST_OTLP_ENDPOINT,
token: process.env.CATALYST_OTLP_TOKEN,
modules: { anthropic: Anthropic },
});
}
return tracingPromise;
}
export async function shutdownTracing(): Promise<void> {
if (!tracingPromise) return;
const tracing = await tracingPromise;
await tracing.shutdown();
}
```
```typescript TypeScript (server entrypoint) theme={"system"}
// server.ts
import { initTracing, shutdownTracing } from "./tracing.ts";
await initTracing(); // patches Anthropic before any client is constructed
const server = startServer();
for (const signal of ["SIGTERM", "SIGINT"] as const) {
process.on(signal, async () => {
await shutdownTracing();
server.close(() => process.exit(0));
});
}
```
Two things to notice:
1. **`initTracing()` runs before the first Anthropic client is constructed.**
The per-SDK patchers work by mutating the SDK's prototype, so `setup()`
has to win the race.
2. **`shutdown()` runs on `SIGTERM`, not per request.** Spans are batched and
exported in the background; calling `shutdown()` per request would force
synchronous flushes and add latency.
## Step 2 — Define The Request Boundary
Each incoming message becomes one trace, rooted at one `AGENT` span. The
agent span carries the identifiers Catalyst uses for grouping in the Agents
dashboard.
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan } from "@inference/tracing";
import { initTracing } from "./tracing.ts";
import { runAgent } from "./agent.ts";
export interface IncomingMessage {
conversationId: string;
text: string;
channel: "slack" | "email" | "web";
tenantId: string;
viewer: { id: string; role: "admin" | "member" };
}
export async function handleMessage(msg: IncomingMessage): Promise<string> {
const tracing = await initTracing();
return await agentSpan(
{
agentId: "customer-support-prod",
agentName: "Customer Support Agent",
role: "support",
system: "anthropic",
sessionId: msg.conversationId,
spanName: "customer-support.run",
},
async (span) => {
// Domain attributes for dashboard filtering.
span.raw.setAttribute("app.tenant_id", msg.tenantId);
span.raw.setAttribute("app.channel", msg.channel);
span.raw.setAttribute("app.viewer.id", msg.viewer.id);
span.raw.setAttribute("app.viewer.role", msg.viewer.role);
span.setInput(msg.text);
const response = await runAgent(msg);
span.setOutput(response);
return response;
},
);
}
```
The four `app.*` attributes are *outside* the OpenInference vocabulary. They
go on the raw OTel span and become filter facets in the dashboard. Use the
same naming convention (a stable prefix for your app, dot-separated keys) so
you can find them easily under `inf trace list --metadata "app.channel=slack"`.
## Step 3 — Author Tool Spans Around Each Tool Call
When the LLM emits a `tool_use` block, your code runs the actual tool
function. Wrap that execution in a `TOOL` span so the trace tree shows what
the tool received, what it returned, and how long it took.
<Metadata />
```typescript TypeScript theme={"system"}
// tools.ts
import { manualSpan, SpanKindValues } from "@inference/tracing";
import { initTracing } from "./tracing.ts";
export type ToolName = "lookup_order" | "issue_refund" | "send_email";
export type ToolArgs = Record<string, unknown>;
export type ToolResult = Record<string, unknown>;
const TOOL_IMPLS: Record<ToolName, (args: ToolArgs) => Promise<ToolResult>> = {
lookup_order: async ({ orderId }) => ({ orderId, status: "shipped" }),
issue_refund: async ({ orderId, amount }) => ({
ok: true,
orderId,
amount,
refundId: "RFD-" + Math.floor(Math.random() * 9999),
}),
send_email: async ({ to, subject }) => ({ ok: true, to, subject }),
};
export async function executeTool(
name: ToolName,
args: ToolArgs,
toolCallId: string,
): Promise<ToolResult> {
const tracing = await initTracing();
return await manualSpan(
{
spanName: `${name}.tool`,
spanKind: SpanKindValues.TOOL,
toolName: name,
toolCallId,
input: args,
},
async (span) => {
const result = await TOOL_IMPLS[name](args);
span.setOutput(result);
return result;
},
);
}
```
`manualSpan` writes `openinference.span.kind=TOOL`, `tool.name`,
`tool_call.id`, `input.value`, and `input.mime_type` from the options. The
callback only needs to set the output. Span end, status, and exception
recording are all handled — if the tool throws, the exception is recorded
on the span, the span ends with `ERROR`, and the original exception
re-throws so the agent loop can see it.
Because `executeTool` runs inside the active context established by
`agentSpan` upstream, the `TOOL` span automatically parents under the agent
span. No span IDs need to be threaded through.
<Tip>
If your tool needs behavior `manualSpan` does not provide — for instance,
recording a span event mid-callback while keeping the span alive past the
callback return — drop down to `tracing.tracer.startActiveSpan` and manage
status / `span.end()` yourself. See
[Manual spans → Escape hatch](/integrations/traces/manual-spans#escape-hatch).
</Tip>
## Step 4 — Wire The Agent Loop
The agent loop alternates between calling the LLM and executing tool calls
the LLM requests. Both sides are now instrumented.
<Metadata />
```typescript TypeScript theme={"system"}
// agent.ts
import Anthropic from "@anthropic-ai/sdk";
import { executeTool, type ToolName } from "./tools.ts";
import type { IncomingMessage } from "./handler.ts";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const TOOLS: Anthropic.Tool[] = [
{
name: "lookup_order",
description: "Look up an order by ID.",
input_schema: {
type: "object",
properties: { orderId: { type: "string" } },
required: ["orderId"],
},
},
{
name: "issue_refund",
description: "Issue a refund.",
input_schema: {
type: "object",
properties: {
orderId: { type: "string" },
amount: { type: "number" },
},
required: ["orderId", "amount"],
},
},
];
export async function runAgent(msg: IncomingMessage): Promise<string> {
const messages: Anthropic.MessageParam[] = [
{ role: "user", content: msg.text },
];
for (let turn = 0; turn < 8; turn++) {
// The patched Anthropic SDK emits an LLM span automatically, parented
// under the active agent span via OTel context propagation.
const response = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 1024,
tools: TOOLS,
messages,
});
if (response.stop_reason === "end_turn") {
return textOf(response.content);
}
if (response.stop_reason !== "tool_use") {
return textOf(response.content);
}
messages.push({ role: "assistant", content: response.content });
const toolResults: Anthropic.ToolResultBlockParam[] = [];
for (const block of response.content) {
if (block.type !== "tool_use") continue;
const result = await executeTool(
block.name as ToolName,
block.input as Record<string, unknown>,
block.id,
);
toolResults.push({
type: "tool_result",
tool_use_id: block.id,
content: JSON.stringify(result),
});
}
messages.push({ role: "user", content: toolResults });
}
return "Max turns reached.";
}
function textOf(content: Anthropic.ContentBlock[]): string {
return content
.filter((b): b is Anthropic.TextBlock => b.type === "text")
.map((b) => b.text)
.join("");
}
```
Three observations:
1. **No tracing imports in the inner loop.** The agent code looks the same as
it would without tracing. The instrumentation is at the boundaries
(`setup()`, `agentSpan()`, `executeTool()`).
2. **The patched Anthropic SDK does the LLM-span work.** We pass
`modules: { anthropic: Anthropic }` to `setup()`, and from then on every
`client.messages.create()` call emits an `LLM` span with input messages,
output content blocks, model, finish reason, and token usage.
3. **Tool spans are caller-side.** They wrap the real function execution, not
the message round-trip. The model-side view of the tool call is captured
on the parent `LLM` span automatically; the caller-side view is the
`TOOL` span we author.
## Step 5 — Verify In The Dashboard And CLI
Send a request through the server, then check the resulting trace:
<Metadata />
```bash theme={"system"}
# Find the most recent trace from this service
inf trace list --service customer-support --limit 1
# Open its span tree
inf trace get <trace-id> --view tree
# Inspect a TOOL span's input and output
inf span list --trace-id <trace-id> --kind TOOL
inf span get <trace-id> <span-id> --view io
# Filter on a domain attribute
inf trace list --metadata "app.channel=slack" --range 1h
```
The trace tree should show a single AGENT root with the LLM and TOOL spans
nested beneath it.
## Common Variations
### Multi-Tenant Service With Per-Request Identity
If `agent.id` itself depends on the request (for example, a multi-tenant
service that runs different agent personas per customer), compute it in the
handler:
<Metadata />
```typescript TypeScript theme={"system"}
const agentId = `support-${msg.tenantId}-prod`;
await agentSpan(
{
agentId,
agentName: `${tenantConfig.displayName} Support`,
role: "support",
sessionId: msg.conversationId,
spanName: "customer-support.run",
},
async (span) => { /* ... */ },
);
```
Stable IDs matter more than human-friendly ones. Prefer `support-acme-prod`
over `support-acme-2024-v2` — the Agents dashboard uses the ID to group runs
across deploys.
### Background Jobs Triggered From The Agent
If your tool launches a background job that itself does LLM work, capture
the active identity and pass it into the job so the background span can be
filtered together with its originating conversation:
<Metadata />
```typescript TypeScript theme={"system"}
import { getActiveAgentIdentity } from "@inference/tracing";
async function executeTool_enqueueReport(args: ToolArgs): Promise<ToolResult> {
const identity = getActiveAgentIdentity();
await jobQueue.enqueue("generate-report", {
...args,
contextAgentId: identity?.id,
contextSessionId: identity?.id ? identity.id : undefined,
});
return { ok: true };
}
```
The background worker can then set `agent.id` and `session.id` on its own
agent span so the two pieces of work share dashboard grouping.
### Streaming Responses
When the agent streams output back to the user, set the span output once at
the end, after the stream completes. The patched SDK already handles
streaming LLM calls correctly; the outer agent span just needs the final
text:
<Metadata />
```typescript TypeScript theme={"system"}
await agentSpan(options, async (span) => {
span.setInput(msg.text);
let final = "";
for await (const chunk of streamAgent(msg)) {
final += chunk;
yield chunk; // back to the caller
}
span.setOutput(final);
});
```
### Custom Span Events
For mid-callback events that are not span attributes — a rate-limit retry,
a fallback to a smaller model, a cache miss — use `span.raw.addEvent`:
<Metadata />
```typescript TypeScript theme={"system"}
span.raw.addEvent("rate_limit_retry", {
attempt: 2,
retry_after_ms: 1500,
});
```
Events appear under the `--view events` flag of `inf span get` and on the
span detail page.
## What To Test
| Behavior | How to verify |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `setup()` runs before the first SDK call | Search server logs for the Catalyst tracing init message; confirm it precedes any Anthropic request log. |
| LLM spans parent under the agent span | `inf trace get <id> --view tree` shows a single AGENT root with LLM and TOOL children. |
| Tool span has `tool.name` and `tool_call.id` | `inf span get <id> --view attributes` |
| Errors mark the span `ERROR` | Force a tool to throw; confirm the span status is `ERROR` and the trace status is `ERROR`. |
| Spans flush on `SIGTERM` | Send `SIGTERM` to the server right after a request; the trace should still appear in Catalyst. |
| Domain attributes are filterable | `inf trace list --metadata "app.tenant_id=acme"` returns the expected traces. |
## Next Steps
<CardGroup>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
The full surface for AGENT, TOOL, CHAIN, and RETRIEVER spans.
</Card>
<Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
All `Attr.*` constants and `SpanKindValues` with the attributes each kind expects.
</Card>
<Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
Every method on the span handle and how it coerces values.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
Debug missing spans, missing attributes, and shutdown behavior.
</Card>
</CardGroup>
# Pydantic AI Traces
Source: https://docs.inference.net/integrations/traces/pydantic-ai
Trace Pydantic AI agents, tool calls, and structured outputs through native OpenTelemetry instrumentation.
Pydantic AI ships native OpenTelemetry instrumentation. Catalyst registers its
TracerProvider and enables Pydantic AI instrumentation during `setup()`.
## Install
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[pydantic-ai]'
```
## Structured Agent With Tools
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import agent_span, setup
from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext
class CityWeather(BaseModel):
city: str
temp_c: float = Field(description="Temperature in Celsius.")
condition: str
class WeatherReport(BaseModel):
cities: list[CityWeather]
summary: str = Field(description="One sentence comparing the conditions.")
tracing = setup(service_name="weather-agent")
agent = Agent(
"openai:gpt-4o-mini",
output_type=WeatherReport,
system_prompt="Use get_weather for every requested city.",
)
@agent.tool
def get_weather(_ctx: RunContext[None], city: str) -> str:
"""Look up current weather for a city."""
weather = {
"Paris": {"temp_c": 12, "condition": "overcast"},
"Tokyo": {"temp_c": 18, "condition": "sunny"},
}
record = weather.get(city, {"temp_c": 0, "condition": "unknown"})
return (
f'{{"city": "{city}", "temp_c": {record["temp_c"]}, '
f'"condition": "{record["condition"]}"}}'
)
with agent_span(
tracing.tracer,
agent_id="weather-agent",
agent_name="Weather Agent",
span_name="weather-agent.run",
session_id="conversation-weather-paris",
agent_role="weather",
system="pydantic-ai",
) as span:
user_input = "What's the weather in Paris and Tokyo?"
span.set_input(user_input)
result = agent.run_sync(user_input)
span.set_output(result.output.model_dump())
print(result.output.summary)
tracing.shutdown()
```
## What To Look For
* Agent run spans from Pydantic AI
* An outer AGENT span with `agent.id=weather-agent` when you use the wrapper
* Tool spans for `get_weather`
* Model spans from the provider used by Pydantic AI
* Structured `WeatherReport` output in the captured span data
# Traces Quickstart
Source: https://docs.inference.net/integrations/traces/quickstart
Install a Catalyst tracing SDK, configure export, and capture your first trace.
This page is the SDK-focused quickstart: install the [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing) (TypeScript) or [`inference-catalyst-tracing`](https://pypi.org/project/inference-catalyst-tracing/) (Python) package, point it at Catalyst, and capture your first span. If you'd rather see the higher-level flow, start with [Capture your first trace](/get-started/capture-first-trace).
The example below uses OpenAI because it's the smallest end-to-end trace. The same export configuration applies to Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, ElevenLabs Agents, Vercel Eve, PI AI, Cursor SDK, Claude Agent SDK, Pydantic AI, the Vercel AI SDK, and manual spans. Each framework guide shows the exact setup hook for that SDK.
## Choose a setup path
Installing with AI is the quickest. Use the manual flow if you want to wire it up yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to launch a coding agent like [Claude Code](https://code.claude.com/docs/en/overview), OpenCode, or Codex to install the tracing SDK, configure export, and wire up your LLM clients.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run tracing instrumentation in your project">
From your project root, run instrumentation in tracing mode.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project && inf instrument --mode tracing
```
The command guides you through the following workflow:
* Select a coding agent: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients and agent frameworks.
* Install `@inference/tracing` or `inference-catalyst-tracing` plus the right per-integration extras.
* Wire `setup()` into your app entrypoint so spans start before clients are constructed.
* Add stable service and agent identity so traces group cleanly in the dashboard.
* Review the generated changes before applying them.
<Tip>
Pick `both` instead of `tracing` to also route requests through the Catalyst Gateway in the same pass.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would. Traces stream to Catalyst as your code executes.
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and filter by your service name to see the captured trace tree.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to wire it up yourself. The example below uses OpenAI. For other providers and frameworks, see the [per-integration guides](/integrations/traces/overview#supported-trace-integrations).
<Steps>
<Step title="Install the SDK">
Provider and framework SDKs are optional peers. Install the ones you use alongside the tracing package. For Python, add per-integration extras to the install string.
<CodeGroup>
<Metadata />
```bash TypeScript theme={"system"}
# Pick your package manager
bun add @inference/tracing openai
npm install @inference/tracing openai
pnpm add @inference/tracing openai
```
<Metadata />
```bash Python theme={"system"}
pip install 'inference-catalyst-tracing[openai]'
# Multiple integrations at once
pip install 'inference-catalyst-tracing[openai,anthropic,langchain]'
# Everything
pip install 'inference-catalyst-tracing[all]'
```
</CodeGroup>
Available Python extras: `openai`, `anthropic`, `langchain`, `langgraph`, `langsmith`, `openai-agents`, `claude-agent-sdk`, `pydantic-ai`, `elevenlabs`, `livekit-agents`, `all`.
</Step>
<Step title="Configure export">
Set the Catalyst traces endpoint and token before your app starts.
<Metadata />
```bash theme={"system"}
export CATALYST_OTLP_ENDPOINT="https://telemetry.inference.net"
# Get your API key from https://inference.net/dashboard/api-keys/
export CATALYST_OTLP_TOKEN="<your-token>"
export CATALYST_SERVICE_NAME="checkout-agent"
```
<Tip>
Use a stable `CATALYST_SERVICE_NAME` per deployed service. It makes traces easier to filter and compare across environments.
</Tip>
You can also pass these as options to `setup()` instead of env vars. See the [configuration reference](/integrations/traces/overview#configuration).
</Step>
<Step title="Initialize tracing early">
Call `setup()` before constructing clients from instrumented SDKs. In TypeScript, pass the SDK modules you want patched. In Python, `setup()` auto-detects installed packages.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({
modules: { openai: OpenAI },
});
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Reply with just the word hello." }],
max_tokens: 16,
});
console.log(response.choices[0]?.message.content);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply with just the word hello."}],
max_tokens=16,
)
print(response.choices[0].message.content)
tracing.shutdown()
```
</CodeGroup>
If the process is short-lived, always call `shutdown()` before exit so batched spans are flushed.
</Step>
<Step title="View your trace">
Open the [dashboard](https://inference.net/dashboard) and navigate to the Agents or Traces tab. You'll see an LLM span with input messages, output messages, model name, invocation parameters, finish reason, and token counts.
</Step>
</Steps>
<Note>
Need a different provider or framework? See the [supported integrations](/integrations/traces/overview#supported-trace-integrations) list.
</Note>
</Tab>
</Tabs>
That's it. Spans are streaming to Catalyst and your first trace is ready to inspect.
What you have so far is one LLM span per call, captured automatically. That's enough for a one-shot script, but real apps usually run several calls per user request, and you'll want those grouped under a named agent and session in the dashboard. That's the next step. If you used Install with AI, the agent likely already wired this up for you; read on to see what it set up and why.
## Group calls under an agent
The example above is a one-shot LLM call. Once your app runs multiple LLM calls as part of a logical unit (an agent run, a conversation turn, a workflow), wrap that unit in `agentSpan` so the LLM spans nest under an `AGENT` row carrying `agent.id`, `agent.name`, and `session.id`. The Agents dashboard groups on those attributes.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
await agentSpan(
{
agentId: "hello-agent",
agentName: "Hello Agent",
sessionId: "session-001",
},
async (span) => {
span.setInput("Reply with just the word hello.");
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Reply with just the word hello." }],
max_tokens: 16,
});
span.setOutput(response.choices[0]?.message.content ?? "");
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
import os
from inference_catalyst_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="hello-agent",
agent_name="Hello Agent",
session_id="session-001",
) as span:
span.set_input("Reply with just the word hello.")
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply with just the word hello."}],
max_tokens=16,
)
span.set_output(response.choices[0].message.content or "")
tracing.shutdown()
```
</CodeGroup>
The OpenAI LLM span still appears, but now nested under your `agentSpan` row instead of as an orphan. Real agents run many LLM calls per session; the outer span is what makes them findable as one thing.
## Wrap your own code
For non-LLM steps inside an agent loop (a tool call, a retrieval, a custom router, an evaluator, a CLI subprocess), wrap them with `manualSpan`. Combined with the `agentSpan` above, you get a full trace tree: an outer AGENT row, inner SDK rows, and inner manual rows, all parented correctly.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import {
SpanKindValues,
agentSpan,
manualSpan,
setup,
} from "@inference/tracing";
const tracing = await setup();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
},
async (span) => {
span.setInput("Review refund request #1842");
const decision = await runRefundReview();
span.setOutput(decision.summary);
},
);
// manualSpan authors TOOL / CHAIN / RETRIEVER / EMBEDDING spans.
await manualSpan(
{
spanName: "rag.retrieve",
spanKind: SpanKindValues.RETRIEVER,
input: { query, k: 8 },
},
async (span) => {
const docs = await retrieve(query);
span.setOutput(docs);
},
);
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
from inference_catalyst_tracing import (
SpanKindValues,
agent_span,
manual_span,
setup,
)
tracing = setup()
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
span_name="refund-review.run",
) as span:
span.set_input("Review refund request #1842")
decision = run_refund_review()
span.set_output(decision.summary)
# manual_span authors TOOL / CHAIN / RETRIEVER / EMBEDDING spans.
with manual_span(
tracing.tracer,
name="rag.retrieve",
span_kind=SpanKindValues.RETRIEVER,
input={"query": query, "k": 8},
) as span:
docs = retrieve(query)
span.set_output(docs)
tracing.shutdown()
```
</CodeGroup>
For the full manual-span surface (tools, retrievers, embeddings, agent identity), see [Manual spans](/integrations/traces/manual-spans) and [Agent identity](/integrations/traces/agent-identity).
## Flushing and process lifecycle
Spans are batched and exported in the background, so a process that exits or
freezes before the batch flushes drops them. How you flush depends on the
process shape:
* **Short-lived script:** call `await tracing.shutdown()` before exit. It
force-flushes, then tears the provider down. The examples above do this.
* **Long-lived service** (HTTP server, Slack bot, queue worker): call `setup()`
**once per process** before the first SDK client is constructed, memoize the
result so any handler can `await` it, and call `shutdown()` only on `SIGTERM`.
Never per request, since that forces a synchronous flush and adds latency.
* **Serverless or edge** (Lambda, Cloudflare Workers): memoize `setup()` the
same way, but flush per invocation with `tracing.provider.forceFlush()`
instead of `shutdown()`, since the provider must survive for the next warm
invocation.
### Long-lived service
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
// tracing.ts — memoized setup
import OpenAI from "openai";
import { setup, type CatalystTracing } from "@inference/tracing";
let tracingPromise: Promise<CatalystTracing> | null = null;
export function initTracing(): Promise<CatalystTracing> {
if (!tracingPromise) {
tracingPromise = setup({ modules: { openai: OpenAI } });
}
return tracingPromise;
}
export async function shutdownTracing(): Promise<void> {
if (!tracingPromise) return;
const tracing = await tracingPromise;
await tracing.shutdown();
}
```
```typescript TypeScript (server entrypoint) theme={"system"}
// server.ts
import { initTracing, shutdownTracing } from "./tracing.ts";
await initTracing(); // patches OpenAI before the first client is constructed
const server = startServer();
for (const signal of ["SIGTERM", "SIGINT"] as const) {
process.on(signal, async () => {
await shutdownTracing();
server.close(() => process.exit(0));
});
}
```
<Metadata />
```python Python theme={"system"}
# tracing.py — memoized setup
from threading import Lock
from inference_catalyst_tracing import CatalystTracing, setup
_tracing: CatalystTracing | None = None
_lock = Lock()
def get_tracing() -> CatalystTracing:
global _tracing
if _tracing is None:
with _lock:
if _tracing is None:
_tracing = setup()
return _tracing
def shutdown_tracing() -> None:
if _tracing is not None:
_tracing.shutdown()
```
```python Python (server entrypoint) theme={"system"}
# server.py
import signal
from tracing import get_tracing, shutdown_tracing
get_tracing() # registers instrumentation before app code runs
def handle_signal(_signum, _frame):
shutdown_tracing()
raise SystemExit(0)
signal.signal(signal.SIGTERM, handle_signal)
signal.signal(signal.SIGINT, handle_signal)
run_server()
```
</CodeGroup>
### Serverless and edge runtimes
On Lambda, Cloudflare Workers, or any runtime that freezes the process between
invocations, the background batch processor may never run, so spans are dropped.
Memoize `setup()` the same way as a long-lived service (the provider is reused
across warm invocations), but flush at the end of **each invocation** with
`tracing.provider.forceFlush()` rather than calling `shutdown()`. Reserve
`shutdown()` for real process teardown, since it tears down the provider the
next warm invocation needs.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import { initTracing } from "./tracing.ts";
export async function handler(event: { message: string }) {
const tracing = await initTracing(); // memoized setup(), patches once
const reply = await answerQuestion(event.message);
// Flush before the runtime freezes the process. Do not shutdown(): the next
// warm invocation reuses this provider.
await tracing.provider.forceFlush();
return reply;
}
```
<Metadata />
```python Python theme={"system"}
from tracing import get_tracing
def handler(event):
tracing = get_tracing() # memoized setup(), patches once
reply = answer_question(event["message"])
# Flush before the runtime freezes the process. Do not shutdown(): the next
# warm invocation reuses this provider.
tracing.provider.force_flush()
return reply
```
</CodeGroup>
### Selective instrumentation
`setup()` auto-instruments every supported SDK it detects. If you want
explicit control — for example, to instrument OpenAI but skip LangChain — set
`autoInstrument: false` and call the targeted helper yourself:
<Metadata />
```typescript TypeScript theme={"system"}
import { setup } from "@inference/tracing";
import { instrumentOpenAI } from "@inference/tracing/openai";
import OpenAI from "openai";
const tracing = await setup({ autoInstrument: false });
instrumentOpenAI(OpenAI, tracing);
```
The per-integration entry points are listed in the
[overview's configuration section](/integrations/traces/overview#configuration).
For a full production-shaped server with custom tool spans and domain
attributes, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## Verify
Open the Catalyst dashboard and navigate to the Agents or Traces tab. The trace should include an OpenAI LLM span with input messages, output messages, model name, invocation parameters, finish reason, and token counts. Any custom spans you added show up as parent or sibling nodes in the trace tree.
If you don't see anything, see [Troubleshooting](/integrations/traces/troubleshooting).
## Next Steps
<CardGroup>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Walk trace trees in the dashboard and run Halo to find what to improve.
</Card>
<Card title="OpenAI tracing" icon="sparkles" href="/integrations/traces/openai">
Add tool calls, structured outputs, and Responses API examples.
</Card>
<Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
Wrap custom agents, CLI calls, and unsupported SDKs.
</Card>
<Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
Add stable agent IDs so the Agents dashboard groups runs correctly.
</Card>
</CardGroup>
# Traces Troubleshooting
Source: https://docs.inference.net/integrations/traces/troubleshooting
Debug missing spans, missing attributes, and shutdown behavior.
Use this checklist when traces do not appear in Catalyst or when spans are
missing expected attributes.
## No Spans Appear
### Confirm Export Configuration
<Metadata />
```bash theme={"system"}
echo "$CATALYST_OTLP_ENDPOINT"
echo "$CATALYST_OTLP_TOKEN"
echo "$CATALYST_SERVICE_NAME"
```
### Enable Debug Logging
<Metadata />
```bash theme={"system"}
export CATALYST_DEBUG=true
```
### Initialize Before Clients
Call `setup()` before constructing provider or framework clients. Many
integrations patch shared prototypes or callback configuration paths.
<Metadata />
```typescript TypeScript theme={"system"}
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
```
## Spans Disappear In Short-Lived Processes
Batch exporters flush on an interval. CLIs, scripts, workers, and tests should
call `shutdown()` before the process exits.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
await tracing.shutdown();
```
<Metadata />
```python Python theme={"system"}
tracing.shutdown()
```
</CodeGroup>
## Framework Spans Are Missing
| Integration | Check |
| ------------------------ | --------------------------------------------------------------------------- |
| LangChain / LangGraph JS | Pass `@langchain/core/callbacks/manager` as a namespace to `setup()` |
| LangSmith | Set `LANGSMITH_TRACING=true` and use `otel` or `hybrid` mode |
| Claude Agent SDK TS | Use `wrapClaudeAgentSdkQuery(query)` |
| OpenAI Agents | Pair `openaiAgents` with `openai` in `setup()` |
| Vercel Eve | Export `defineCatalystEveInstrumentation()` from `agent/instrumentation.ts` |
## Attributes Are Missing
Some attributes only appear when the upstream provider returns the data. For
example, token counts depend on provider usage fields, and Anthropic prompt-cache
details appear only when prompt caching actually engages for the request.
## Next Steps
If setup order, export configuration, and shutdown are correct, reduce the issue
to the smallest provider call and compare it with [Traces Quickstart](/integrations/traces/quickstart).
# Catalyst by Inference.net
Source: https://docs.inference.net/introduction
Catalyst is a platform for understanding, evaluating, and improving the AI systems you ship.
[Create an account](https://inference.net/register) to get started with Catalyst.
## The Catalyst platform
Catalyst is built around two products. Use them on their own or together.
<CardGroup>
<Card title="Tracing" icon="route" href="/get-started/capture-first-trace">
Capture full traces of every LLM call, tool call, and agent step. Then let Halo, our open-source agent-loop optimizer, read your traces and tell you what to fix in prompts, tools, and the harness itself.
</Card>
<Card title="Gateway" icon="satellite-dish" href="/get-started/record-first-call">
Route LLM traffic through Catalyst Gateway to capture every request, watch usage and cost, build datasets, run evals, and fine-tune and deploy task-specific models on dedicated GPUs.
</Card>
</CardGroup>
Tracing is for agent improvement. Gateway is for inference observability and training task-specific models from your captured traffic. Most teams start with one and add the other later. They also compose: when your agent calls LLMs through Gateway, trace spans line up with the same requests Gateway is recording.
## Quick Start
### Tracing
<CardGroup>
<Card title="Capture your first trace" icon="route" href="/get-started/capture-first-trace">
Install the tracing SDK and capture LLM calls, tool calls, and agent steps in minutes.
</Card>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Inspect trace trees in the dashboard and run Halo to find what to improve.
</Card>
</CardGroup>
### Gateway
<CardGroup>
<Card title="Record your first LLM call" icon="bolt" href="/get-started/record-first-call">
Route traffic through the Catalyst gateway to capture LLM calls and view metrics.
</Card>
<Card title="Run your first eval" icon="flask" href="/get-started/run-first-eval">
Define quality, measure it, and compare models side by side.
</Card>
<Card title="Train and deploy a model" icon="brain" href="/get-started/train-and-deploy">
The full loop: data, training, and a production endpoint.
</Card>
<Card title="Use the Inference API" icon="code" href="/api/api-quickstart">
Call open-source and custom models running on Inference.net.
</Card>
</CardGroup>
## Catalyst Platform
<CardGroup>
<Card title="Tracing" icon="route" href="/integrations/traces/overview">
Collect OpenInference-shaped traces from LLM SDKs, agent frameworks, and custom application code.
</Card>
<Card title="Gateway" icon="satellite-dish" href="/platform/gateway/overview">
Capture your LLM traffic and get metrics and visibility into your production usage.
</Card>
<Card title="Datasets" icon="file-code" href="/platform/datasets/overview">
Create and manage datasets for evaluation and training.
</Card>
<Card title="Evaluate" icon="scale-balanced" href="/platform/eval/overview">
Evaluate models to measure quality across model candidates.
</Card>
<Card title="Train" icon="brain" href="/platform/train/overview">
Train a custom model on your production data to improve performance, lower latency and cost.
</Card>
<Card title="Deploy" icon="rocket" href="/platform/deploy/overview">
Deploy a model to a dedicated GPU to use in production.
</Card>
</CardGroup>
# Build a Dataset from Traffic
Source: https://docs.inference.net/platform/datasets/build-from-traffic
Turn production traffic into datasets for evaluation and training.
The most useful datasets come from real production traffic. Catalyst lets you filter your captured inferences and save the results as a dataset, ready for evals or training.
You can create a dataset from traffic in two places: the **Create Dataset** button in the Datasets tab, or **Save as Dataset** in the [Inference Viewer](/platform/gateway/inference-viewer). Both follow the same flow.
## The flow
<Steps>
<Step title="Filter your traffic">
Filter by model, task, provider, status code, or any tracked dimension until you have a representative slice of traffic.
</Step>
<Step title="Choose a dataset type">
Decide whether this will be an **eval dataset** or a **training dataset**. Remember the [zero-overlap rule](/platform/datasets/overview#the-zero-overlap-rule), training and eval data must never share examples.
</Step>
<Step title="Save as dataset">
Name the dataset and save. It's immediately available for evals or training.
</Step>
</Steps>
## Getting clean samples
The quality of your dataset depends on how well you filter. A few tips:
* **Filter by [task](/platform/gateway/tasks)** to get samples for a specific objective rather than a mix of everything
* **Exclude errors** unless you specifically want failure cases (e.g. for training a model to handle edge cases)
* **Check the date range** - a dataset pulled from a single day might not capture the full variety of inputs your app sees
## Eval vs training: different goals, different data
**Eval datasets** should be small, stable, and challenging. Pick examples that represent the hard cases — the ones where you're not sure the model will get it right. These become your benchmark, so don't change them often.
**Training datasets** should be large, diverse, and representative. The more variety, the better the model generalizes. Iterate on these as you learn what the model struggles with.
## Next steps
<CardGroup>
<Card title="Upload your own data" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have curated data? Upload it directly.
</Card>
<Card title="Dataset formats" icon="file-code" href="/platform/datasets/formats">
Supported schemas and validation rules.
</Card>
</CardGroup>
# Dataset Formats and Schemas
Source: https://docs.inference.net/platform/datasets/formats
JSONL upload formats, required fields, validation rules, and upload limits.
Datasets can be created from [captured traffic](/platform/datasets/build-from-traffic) or [uploaded as JSONL files](/platform/datasets/upload-a-dataset). This page covers the supported formats.
## Supported formats
The system auto-detects the format from the first valid line. All rows in a file must use the same format.
<Warning>
You cannot mix source-backed and Hugging Face rows in the same file. Mixed-format files fail validation.
</Warning>
### Source-backed format
Each line has a top-level `request` and optional `response` object containing raw provider bodies.
| Field | Required | Description |
| ---------- | -------- | --------------------------------------------------------------- |
| `request` | Yes | Raw provider request body that includes a usable `model` value |
| `response` | No | Raw provider response body, or `null` if you only have requests |
Validation notes:
* The request must include a usable model value.
* `response` may be omitted or set to `null` if you only have request-side data.
<Metadata />
```json theme={"system"}
{"request":{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}],"temperature":0.7,"max_tokens":100},"response":{"id":"chatcmpl-123","object":"chat.completion","created":1700000000,"model":"gpt-4","choices":[{"index":0,"message":{"role":"assistant","content":"Hi there!"},"finish_reason":"stop"}]}}
```
### Hugging Face format
Each line has a top-level `messages` array with `role`/`content` objects.
| Field | Required | Description |
| ---------- | -------- | ------------------------------------------------------------ |
| `messages` | Yes | Array of `{ role, content }` objects (at least one required) |
| `id` | No | Optional row identifier (stored in metadata) |
| `tools` | No | Optional top-level tool definitions |
Valid roles: `system`, `user`, `assistant`, `tool`.
Additional supported fields:
* `content` may be a string or an array of content parts for multimodal rows.
* Assistant messages may include `tool_calls`.
* Tool messages must include `tool_call_id`.
* Top-level `tools` are preserved on import.
<Metadata />
```json theme={"system"}
{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 2+2?"},{"role":"assistant","content":"4"}]}
```
When importing Hugging Face-format rows, the system:
* Treats the last assistant turn as the imported response and earlier turns as request context
* Synthesizes request/response payloads so evals and detail views work
* Sets `request_model` to `unknown-imported-model`
* Sets token usage and costs to zero
* Stores the original row `id` in metadata as `importOriginalRowId`
## Validation behavior
* Files must be valid JSONL.
* Invalid rows are reported with line numbers in the upload status details.
* Uploads can complete with partial failures if at least one row imports successfully.
* If every row fails validation, the upload status is `failed`.
## Upload limits
| Limit | Value |
| ------------------ | --------- |
| Maximum file size | 10 GB |
| Maximum line count | 1,000,000 |
## Download formats
Datasets can be downloaded in two formats:
| Format | Description | Best for |
| ----------------- | ------------------------------------ | -------------------------------- |
| **Hugging Face** | `{ id, messages }` per row (default) | Training, fine-tuning, sharing |
| **Source-backed** | `{ request, response }` per row | Re-uploading, round-trip testing |
In the UI, click **Download** and choose the format. In the CLI:
<Metadata />
```bash theme={"system"}
# Default (Hugging Face)
inf dataset download my-dataset
# Source-backed format
inf dataset download my-dataset --format source-backed
```
<Note>
Hugging Face exports skip rows with empty message arrays. Source-backed exports include all rows with a valid request payload. Row counts may differ between formats.
</Note>
# Datasets
Source: https://docs.inference.net/platform/datasets/overview
Curate datasets from production traffic or your own files for evals and training.
Datasets are collections of LLM inputs and outputs used for evaluation and fine-tuning. They can come from two places: your live production traffic captured through [Gateway](/platform/gateway/overview), or files you [upload directly](/platform/datasets/upload-a-dataset).
Everything downstream depends on good data. Evals need representative examples to measure model quality. Training needs diverse, high-quality samples to teach a model your task. Datasets are where both start.
<Frame>
<iframe title="Turn Production Traffic Into LLM Training Data | Catalyst" />
</Frame>
## Types of datasets
| Type | Purpose | How it evolves |
| -------------------- | --------------------------------------------- | ---------------------------------------------------------------------------- |
| **Eval dataset** | Measures model quality against a rubric | Stays stable, a fixed set of challenging examples that act as your benchmark |
| **Training dataset** | Data the model learns from during fine-tuning | Changes often as you iterate on data quality and coverage |
### The zero-overlap rule
Catalyst automatically enforces zero-overlap between training and eval datasets. If a training dataset overlaps with an eval dataset, the overlapping data will be excluded from the training dataset when a new training run is created.
## Key concepts
| Concept | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Build from traffic** | Filter your captured production inferences and save them as a dataset. The best datasets come from real usage. |
| **Upload** | Bring your own JSONL files when you have curated data or are migrating from another platform. |
| **Dataset format** | The schema your data needs to follow. See [Dataset Formats](/platform/datasets/formats) for supported fields and validation rules. |
| **Task tags** | Use [task tags](/platform/gateway/tasks) when building from traffic to filter by objective. This gives you clean, focused samples instead of mixed traffic. |
## Tips for good datasets
* **Diverse training data** leads to models that generalize well. If your training data isn't heterogeneous, the trained model won't handle edge cases.
* **Stable eval data** gives you a consistent benchmark. Don't change your eval dataset frequently, it's the measuring stick.
* **Start with production traffic** when possible. Real user inputs reflect the actual distribution of requests your model will see, and they're harder to fake than synthetic data.
* **Use task tags** to filter by objective before saving a dataset. A dataset scoped to a single task is almost always more useful than one built from mixed traffic.
## Next steps
<CardGroup>
<Card title="Build from traffic" icon="satellite-dish" href="/platform/datasets/build-from-traffic">
Turn filtered production traffic into a dataset.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Bring your own JSONL files.
</Card>
<Card title="Set up your first eval" icon="flask" href="/get-started/run-first-eval">
Use your dataset to compare models.
</Card>
<Card title="Train a custom model" icon="brain" href="/get-started/train-and-deploy">
Use your dataset to fine-tune a task-specific model.
</Card>
</CardGroup>
# Upload a Dataset
Source: https://docs.inference.net/platform/datasets/upload-a-dataset
Import JSONL inference data, then turn it into eval or training datasets.
If you already have curated data from annotation pipelines, synthetic generation, or another platform, you can upload it as JSONL instead of building a dataset from captured traffic.
Uploads and datasets are separate objects in Catalyst:
* An **upload** is the imported JSONL file plus its validation and processing status.
* A **dataset** is the stable collection you use for evals, training, and download.
You upload the file first, then create an **eval** or **training** dataset from that uploaded data once processing finishes.
## How to upload
<Tabs>
<Tab title="Dashboard">
1. Go to **Datasets** in the dashboard
2. Click **Upload Data**
3. Select your `.jsonl` file
4. Give the upload a name and start the import
The upload appears in **Datasets > Uploads**, where you can track processing and review any validation errors.
</Tab>
<Tab title="CLI">
<Metadata />
```bash theme={"system"}
inf dataset upload path/to/data.jsonl --name support-summaries
```
If you omit `--name`, the CLI uses the filename without the extension. By default the CLI waits for processing and prints the detected format plus the processed line count. Use `--no-wait` to return after the transfer completes.
</Tab>
</Tabs>
<Note>
The upload command does not ask whether the data is for evals or training. You choose **eval** vs **training** when you create a dataset from the completed upload.
</Note>
## After upload
1. Wait for the upload to finish processing in **Datasets > Uploads**.
2. Open the dataset creation flow and select the upload as your source.
3. Choose whether the resulting dataset is **eval** or **training**.
Successful uploads become a reusable source in the same dataset creation flow you use for traffic-backed datasets.
## Supported formats
Two JSONL formats are supported. See [Dataset Formats](/platform/datasets/formats) for full schemas, required fields, and validation rules.
| Format | Structure | Best for |
| ----------------- | -------------------------------- | --------------------------------------------- |
| **Source-backed** | `{ request, response }` per line | Round-tripping data captured from providers |
| **Hugging Face** | `{ messages }` per line | Standard training/eval format, easy to create |
The system auto-detects the format from the first valid line. Every row in the file must use the same format.
## Validation behavior
* Invalid rows are reported with line numbers in the upload status details.
* Uploads can complete with some failed rows if at least one row imports successfully.
* Mixed-format files are treated as a fatal error and fail the upload.
* Source-backed rows must include a usable model value in the request.
## Upload limits
| Limit | Value |
| ------------------ | --------- |
| Maximum file size | 10 GB |
| Maximum line count | 1,000,000 |
## Next steps
<CardGroup>
<Card title="Build from traffic instead" icon="satellite-dish" href="/platform/datasets/build-from-traffic">
Pull datasets directly from your captured production traffic.
</Card>
<Card title="CLI Command Reference" icon="terminal" href="/cli/datasets">
Upload from the terminal with `inf dataset upload`.
</Card>
<Card title="Dataset formats reference" icon="file-code" href="/platform/datasets/formats">
Full schema details and validation rules.
</Card>
</CardGroup>
```
```
# Call Your Deployment
Source: https://docs.inference.net/platform/deploy/call-your-deployment
Connect to your deployed model using the OpenAI-compatible API.
Deployments expose an OpenAI-compatible chat completions endpoint. Point any OpenAI SDK client at the Inference base URL and set the `model` to your deployment's model path.
<CodeGroup>
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
const response = await client.chat.completions.create({
model: "acme-corp/my-model",
messages: [{ role: "user", content: "Hello, world!" }],
});
```
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
)
response = client.chat.completions.create(
model="acme-corp/my-model",
messages=[{"role": "user", "content": "Hello, world!"}],
)
```
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "acme-corp/my-model",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
Structured outputs, function calling, and other chat completions features all work the same way. You can also tag requests with `x-inference-task-id` to group calls by objective — see [Tasks](/platform/gateway/tasks) for details.
## Where to find your model path
The model path is shown on your deployment's detail page in the dashboard. It's your team slug followed by the name you chose when creating the deployment (e.g. `acme-corp/my-model`).
# Deploy a Trained Model
Source: https://docs.inference.net/platform/deploy/deploy-a-model
Go from a completed training run to a live endpoint in a few clicks.
When training completes, the model is automatically registered and ready to deploy. No manual promotion step.
Deployments are scoped to your team, not a specific project. When you create a deployment, you give it a name — the model path becomes your team slug followed by that name (e.g. `acme-corp/my-model`).
## Starting a deployment
There are a few ways to get started:
* From the **Deployments** page — click **Create**
* From a **completed training run** — click deploy directly from the training detail page
* From the **Models** page — click deploy in the row for your model
All three paths lead to the same flow.
## The flow
<Steps>
<Step title="Name the deployment">
Give it a descriptive name. This becomes the second part of your model path (e.g. `acme-corp/my-model`).
</Step>
<Step title="Deploy">
Select your instance configuration and click deploy. If you need more compute than a single GPU, you can reach out to the team directly from this page.
</Step>
<Step title="Wait for warm-up">
The deployment takes a few minutes to 20–30 minutes to come online. This time is spent allocating compute and spinning up the GPU.
</Step>
</Steps>
## After deployment
Once the endpoint is live, you can [call it](/platform/deploy/call-your-deployment) using the same OpenAI-compatible API you're already using. Just swap in your model path.
# Manage and Monitor
Source: https://docs.inference.net/platform/deploy/manage-and-monitor
Start, stop, and delete deployments. Monitor production performance and scale when you need to.
<Frame>
<img alt="Deployments list with per-deployment metrics" />
</Frame>
## Lifecycle operations
* **Start** — bring a stopped deployment back online
* **Stop** — take the deployment offline
* **Delete** — remove the deployment entirely
## Scaling
By default, your model deploys on a single dedicated GPU. If you need additional GPUs or auto-scaling, [reach out to our team](https://inference.net/meet-with-us/).
## Monitoring
Once your deployment is live, click into it from the **Deployments** page to see metrics and individual inference calls. You get the same [Gateway](/platform/gateway/overview) experience — latency, error rates, token usage, and full request/response payloads — scoped to that deployment.
<Frame>
<img alt="Deployment metrics and inference calls" />
</Frame>
## The loop continues
Your custom model is live. Use [Gateway](/platform/gateway/overview) to watch its production performance, run [evals](/platform/eval/overview) to catch regressions, and [train](/platform/train/overview) the next version when you're ready.
<CardGroup>
<Card title="Gateway" icon="satellite-dish" href="/platform/gateway/overview">
Monitor production traffic.
</Card>
<Card title="Eval" icon="flask" href="/platform/eval/overview">
Catch quality regressions.
</Card>
<Card title="Train" icon="brain" href="/platform/train/overview">
Build the next version.
</Card>
</CardGroup>
# Open Source Models
Source: https://docs.inference.net/platform/deploy/open-source-models
Deploy off-the-shelf open source models or bring your own trained models.
<Info>
This feature is on the roadmap. Today, only models trained on the platform can be deployed.
</Info>
## What's planned
* **Deploy off-the-shelf OSS models** — run popular open source models on dedicated GPUs without going through training. Pick a model, select your instance, and deploy.
* **Bring your own trained models** — deploy models you've already fine-tuned outside the platform. Upload your weights and serve them on the same infrastructure.
## Why this matters
Not every deployment starts with training on the Inference platform. Some teams want dedicated GPU serving for an existing open source model, or they've already fine-tuned a model elsewhere and want to host it. This feature will bring all of that into the same deployment workflow — same API, same monitoring, same scaling options.
<Card title="Want early access?" icon="envelope" href="https://inference.net/meet-with-us/">
Talk to our team if you'd like to be notified when open source and custom model deployments are available.
</Card>
# Deploy
Source: https://docs.inference.net/platform/deploy/overview
Dedicated GPU infrastructure for serving trained models via an OpenAI-compatible API.
Deploy gives you a dedicated GPU serving your fine-tuned model. The API is OpenAI-compatible, so switching from an off-the-shelf model to your custom model is a one-line code change. This is the last step in the loop and the beginning of the next one.
<Frame>
<iframe title="Deploy a Production-Ready Fine-Tuned LLM | Catalyst" />
</Frame>
## Key concepts
| Concept | Description |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Dedicated GPU** | Your model runs on its own GPU. No shared infrastructure, no noisy neighbors. Compute is determined by the recipe used during training. |
| **OpenAI-compatible API** | Same base URL, same API key, just swap the model parameter. Structured outputs, function calling, and all standard API features work the same way. |
| **Team-scoped** | Deployments belong to a team, not a project. The model path is your team slug followed by the deployment name you choose (e.g. `acme-corp/my-model`). |
| **The improvement loop** | Deploy → observe production performance → run evals to catch regressions → train the next version. The loop continues. |
## What you can deploy
* Models trained on the Inference platform
* Served via an OpenAI-compatible API (chat completions endpoint)
* Same base URL and API key as the rest of the Inference API
## Next steps
<CardGroup>
<Card title="Deploy a trained model" icon="server" href="/platform/deploy/deploy-a-model">
Name it, click deploy, start serving.
</Card>
<Card title="Call your deployment" icon="code" href="/platform/deploy/call-your-deployment">
One line of code to switch over.
</Card>
<Card title="Manage and monitor" icon="sliders" href="/platform/deploy/manage-and-monitor">
Lifecycle operations, scaling, and the improvement loop.
</Card>
</CardGroup>
# Serverless Deployments
Source: https://docs.inference.net/platform/deploy/serverless-deployments
Platform-wide deployments billed per token, or offered for free.
Most deployments are private: only the owning team can call them, and they bill
by GPU capacity. A deployment can also be **serverless-enabled** by the
Inference team, which makes it callable by every account on the platform and
billed per token — or offered for free.
## Calling a serverless deployment
Serverless deployments work exactly like any other model on the
OpenAI-compatible API. Use the deployment's model path as the `model`:
```bash theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "inference-net/example-model",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
Streaming, structured outputs, and the other chat completions features work
the same as on catalog models.
## Billing
Serverless deployments are priced in **USD per 1M tokens**, with separate
input and output rates set per deployment. Usage is billed to the calling
team's credit balance like any other serverless inference:
* Requests are authorized against your credit balance up front; if the
balance can't cover the estimated cost, the API responds with `402`.
* The actual charge is settled when the inference completes, from the real
token counts reported by the serving engine.
* Failed inferences are never billed.
* Charges appear in your usage dashboard under the deployment's model path.
## Free deployments
A serverless deployment with **no prices set is public and free**: anyone on
the platform can call it and no credits are charged or required. Free
deployments still count against your standard serverless rate limits.
## Limits
Serverless deployment requests share your team's serverless inference rate
limits. Context-window limits are enforced by the deployment's engine rather
than the platform catalog, so an oversized prompt is rejected by the model
itself.
# How LLM-as-a-Judge Works
Source: https://docs.inference.net/platform/eval/llm-as-a-judge
The evaluation mechanism that scores model outputs against your rubric criteria.
LLM-as-a-judge is the evaluation mechanism in Catalyst. An LLM reads your rubric, looks at the output being evaluated, and returns a numerical score. These scores are then aggregated into a final evaluation result.
## How it works
1. The judge model receives the **rubric**, the **conversation context**, and the **output to judge**
2. It evaluates the output against the rubric criteria
3. It returns a **judgment with a numerical score** within the rubric's defined range
## Choosing a judge model
Use the smartest available model as your judge. You want the judge to be more capable than the models being evaluated, since a weaker judge may not reliably distinguish quality differences.
You select the judge model when running an eval.
## Building the judge context
The judge is only given the context that you [define in your rubric](/platform/eval/write-a-rubric). Make sure to build the proper rubric and verify the sample contents during your first evaluation run. There is a defined way to create the variables that will be filled in during an execution of an eval.
## Cost
Judge calls are full LLM inferences, so they have real cost. For offline evals with a bounded dataset, this is usually manageable. For online evals at scale (coming soon), sample rate controls will help manage cost.
## What the judge doesn't do
The judge scores against your rubric. It doesn't independently decide what "good" means. If the rubric is vague or measures the wrong thing, the judge will faithfully score against bad criteria. This is why [validating your rubric](/platform/eval/write-a-rubric) before using it in training is important.
# Offline vs Online Evaluation
Source: https://docs.inference.net/platform/eval/offline-vs-online
Running evals against collected samples vs scoring live production traffic.
## Offline evaluation
Offline eval is the standard flow: run a dataset through models, score the outputs, compare. You control what gets evaluated and when.
This is what's available today. The inputs typically come from captured production traffic, but the outputs are re-generated. You select which models to run against rather than judging the original production outputs.
## Online evaluation
<Info>
Online evaluation is coming soon.
</Info>
Online eval will score live production traffic as it flows through Catalyst. Instead of re-running samples through models, it judges the actual outputs your users are seeing.
How it differs from offline:
* **Sample rate controls** - evaluate a percentage of traffic to manage cost
* **Real outputs** - judge what your model actually produced, not a re-run
* **Continuous** - always running, not triggered manually
## Current limitation
Today, you always select a model to generate new outputs for evaluation. You can't run a rubric directly against captured production outputs. This means offline evals answer "how would model X perform on this data?" rather than "how did my production model actually perform?"
Near-term, the platform will support judging captured production outputs directly against rubrics, bridging toward full online evaluation.
# Eval
Source: https://docs.inference.net/platform/eval/overview
Measure model quality with rubrics scored by LLM judges. Know which model is better and by how much.
Model providers ship updates constantly and prompts drift. Eval gives you a repeatable way to measure model quality before and after every change. They can also help compare model options for a given task.
If you're planning to [fine-tune a custom model](/platform/train/overview), run evals first. A validated rubric and eval dataset are prerequisites for training. They're the measuring stick that determines when the model has learned enough, or when to stop to prevent overfitting.
<Frame>
<iframe title="Evaluate LLMs with LLM-as-a-Judge | Catalyst" />
</Frame>
## How it works
1. **Define a [rubric](/platform/eval/write-a-rubric)** - describe what "good" looks like in plain English
2. **Pick a dataset** - samples from [captured traffic](/platform/datasets/build-from-traffic) or [uploaded JSONL](/platform/datasets/upload-a-dataset)
3. **Select models** - the candidates you want to compare
4. **Run the eval** - each sample goes through each model, and an [LLM judge](/platform/eval/llm-as-a-judge) scores every output
5. **[Compare results](/platform/eval/read-the-results)** - side-by-side scores show which model wins
## Key concepts
| Concept | Description |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **LLM-as-a-judge** | A capable LLM reads your rubric, examines the model output, and returns a scored judgment. |
| **Rubric** | A plain English description of a quality dimension, scored numerically. Defines what "good" means for your use case. |
| **Direct rubric** | The LLM judge grades the model output directly against the rubric, without comparing it to a reference answer. |
| **Adherence rubric** | The LLM judge grades the model output based on how closely it matches a reference response. |
| **Eval dataset** | A stable, curated set of challenging examples that acts as your benchmark. Pick the hard cases. |
| **Offline vs online** | Offline evals run against collected samples. Online evals score live traffic as it flows through. Offline is available today; online is coming soon. |
| **Train-eval splits** | Training and eval data must never overlap. If a model trains on eval examples, the eval becomes meaningless. See the [zero-overlap rule](/platform/datasets/overview#the-zero-overlap-rule). |
## Next steps
<CardGroup>
<Card title="Writing rubrics" icon="pen" href="/platform/eval/write-a-rubric">
Create rubrics from templates, AI generation, or plain English.
</Card>
<Card title="Run a model comparison" icon="scale-balanced" href="/platform/eval/run-a-comparison">
Compare models head to head on your data.
</Card>
<Card title="How LLM-as-a-Judge works" icon="gavel" href="/platform/eval/llm-as-a-judge">
Understand the evaluation mechanism.
</Card>
<Card title="Read the results" icon="chart-column" href="/platform/eval/read-the-results">
Interpret scores and make decisions.
</Card>
</CardGroup>
# Read the Results
Source: https://docs.inference.net/platform/eval/read-the-results
Interpret the side-by-side comparison view and decide which model wins.
After an eval completes, the comparison view shows how each model performed across every sample and rubric dimension.
## What the comparison shows
* **Side-by-side plots** highlighting where models differ in quality
* **Full scores table** across all models and samples
* **Per-sample breakdown** so you can see where specific models excel or struggle
<img alt="Eval comparison view showing side-by-side model scores and the scores table." />
## How to read the results
Look for:
* **Overall winner** - which model has the highest average score across your rubric
* **Edge cases** - samples where one model significantly outperforms another
* **Rubric dimensions** - if you have multiple rubrics, check whether models trade off on different quality dimensions (e.g. one model is more accurate but another has better tone)
While the aggregate scores can inform you on how different models perform in a nutshell, it is recommended to analyze individual samples in the sample viewer. This will help to understand specific model quirks.
## Making decisions
* **Which model to use in production** - the one that best matches your quality criteria
* **Whether to train a custom model** - if no off-the-shelf model scores well enough, [fine-tuning](/platform/train/overview) is the next step
* **Whether the rubric needs work** - if scores don't align with your intuition, iterate on the rubric before changing models
# Run a Model Comparison
Source: https://docs.inference.net/platform/eval/run-a-comparison
Run your eval dataset through multiple models and score the outputs against your rubric.
Pick a rubric, pick a dataset, pick models, and run. Catalyst handles execution and scoring.
## Step by step
<Steps>
<Step title="Select a rubric">
Choose the [rubric](/platform/eval/write-a-rubric) that defines your quality criteria.
</Step>
<Step title="Select an eval dataset">
Choose the dataset containing your evaluation samples. This can come from [captured traffic](/platform/datasets/build-from-traffic) or a [JSONL upload](/platform/datasets/upload-a-dataset).
</Step>
<Step title="Select models">
Pick one or more models to evaluate. You can choose from a wide range of models including OpenAI, Anthropic, open-source, or your own custom trained models.
</Step>
<Step title="Run the eval">
Each sample from the dataset runs through each selected model. Each output gets scored by the [LLM judge](/platform/eval/llm-as-a-judge) using your rubric.
</Step>
</Steps>
<img alt="Eval setup flow showing rubric, dataset, and model selection in the dashboard." />
## How the math works
The eval is a cross-product of samples and models:
* 10 samples across 3 models = 30 inference outputs
* Each output gets scored = 30 judge calls
* Results: per-sample scores for every model
## Next steps
Once the eval completes, go to [Read the Results](/platform/eval/read-the-results) to interpret the comparison view.
# Writing Rubrics
Source: https://docs.inference.net/platform/eval/write-a-rubric
Create evaluation rubrics from templates, AI generation, or plain English.
A rubric defines what "good" means for your use case. It's a plain English description of a quality dimension, scored numerically, that the [LLM judge](/platform/eval/llm-as-a-judge) uses to score model outputs.
## Three ways to create a rubric
<CardGroup>
<Card title="AI Generate from data" icon="sparkles">
Point the generator at an existing dataset. It analyzes your inputs and outputs and suggests rubric dimensions relevant to your data.
</Card>
<Card title="Start from a template" icon="copy">
Pick from pre-built rubrics for common quality dimensions like accuracy, helpfulness, tone, or format compliance. Customize from there.
</Card>
<Card title="Write your own" icon="pen">
Describe the quality dimension in plain English, define what each score level means, and set the scoring range.
</Card>
</CardGroup>
All three paths start from the Evals page in the dashboard. Generated rubrics and templates are starting points. Review and refine before running evals.
<img alt="Rubric creation UI showing the three creation paths: AI generate from data, start from a template, and write your own." />
## Template variables
Rubrics use three template variables that inject context from your data into the prompt sent to the judge:
| Variable | What it contains | Required |
| ----------------------------- | ------------------------------------------------ | ------------ |
| `{{ conversation_context }}` | The input messages and conversation history | Recommended |
| `{{ conversation_response }}` | The reference/original response from the dataset | Recommended |
| `{{ eval_model_response }}` | The output being scored | **Required** |
Every rubric must include `{{ eval_model_response }}`. Using all three gives the judge the full picture: the input, what was originally produced, and the output it needs to score.
## Scoring range
You set the max score when creating a rubric. The default range is **1-10**, which gives the judge enough room to distinguish meaningful quality differences. You can adjust this to fit your use case. A smaller range like 1-3 works for simpler pass/fail dimensions, while the default 1-10 is a good fit for most evaluations.
## Writing effective rubrics
Task-specific rubrics produce sharper, more useful results than broad ones.
| Vague | Specific |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| "Is this response accurate?" | "Does the extracted JSON contain all required fields with correct types?" |
| "Is the tone appropriate?" | "Does the response match the brand voice: professional, concise, no hedging?" |
| "Is this helpful?" | "Does the summary capture the three most important points from the source document?" |
Describe what separates a high score from a low score. A rubric without clear score descriptions will produce inconsistent results.
It is also recommended to thoroughly describe all levels of a judge score. If you are setting your max score at 10, this means listing out the qualities of a response at each score level. Alternatively, you can create combined criterias that add up to your max score (e.g. 3 points for clarity, 3 points for informativeness, 3 points for similarity to original, 1 point for tone).
## Versioning
You can create different versions of a rubric to test against. This lets you iterate on scoring criteria and compare how different rubric versions evaluate the same data, which is useful for dialing in what you actually care about before committing to a rubric for training.
## Validate before training
If you plan to use a rubric for [training](/platform/train/overview), run it against your eval dataset first. Mid-training evals use the rubric to decide when to stop. If the rubric measures the wrong thing, the model optimizes for the wrong objective.
## Direct vs Adherence Rubrics
A direct rubric is something that judges the evaluated model response without comparing it to the original stored response. These rubrics only have `{{ conversation_context }}` and `{{ eval_model_response }}` and omit `{{ conversation_response }}` since there is no comparison.
An adherence rubric is something that judges an evaluated model response against a reference response. This is in cases where you want to see how similar other model responses are against your current model. These rubrics contain all three of `{{ conversation_context }}`, `{{ eval_model_response }}`, and `{{ conversation_response }}`.
# How to Create a Task
Source: https://docs.inference.net/platform/gateway/create-a-task
Task are automatically created when you send requests with a `x-inference-task-id` header. The task appears in the dashboard as soon as the first tagged request comes through.
## Set the task header
Add `x-inference-task-id` to your request. The value is whatever name makes sense for the objective — `document-summary`, `ticket-classifier`, `extract-product-data`, etc.
You can set the task per request, or put it in `defaultHeaders` on the client if all calls from that client belong to the same task.
**Per request** (use when one client serves multiple tasks):
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Summarize this document..." }],
}, {
headers: { "x-inference-task-id": "document-summary" },
});
```
<Metadata />
```python Python theme={"system"}
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Summarize this document..."}],
extra_headers={"x-inference-task-id": "document-summary"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-H "x-inference-task-id: document-summary" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Summarize this document..."}]
}'
```
</CodeGroup>
**In default headers** (use when a client is dedicated to one task):
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
"x-inference-task-id": "document-summary",
"x-inference-environment": "production",
},
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
"x-inference-task-id": "document-summary",
"x-inference-environment": "production",
},
)
```
</CodeGroup>
## The default task
If you don't set `x-inference-task-id`, the request is grouped under the **default** task. Your traffic is still captured, but it's all in one bucket. Once you start tagging, you can break down metrics, run evals, and build datasets per task.
## Assign task IDs automatically
Use **[Install with AI](/integrations/install-with-ai)** if you want the CLI to add task IDs for you. It runs `inf instrument`, scans your codebase, and names each call site based on what the prompt is doing — for example, `document-summary` or `ticket-classification`. These are starting points; you can rename them by editing the `x-inference-task-id` header value in the generated code.
# Inference Viewer
Source: https://docs.inference.net/platform/gateway/inference-viewer
Browse, filter, and inspect individual LLM requests and responses.
The Inference Viewer is a filterable table of every LLM call that flows through Catalyst. Use it to inspect individual requests, debug issues, and find samples to save as datasets for evals and training.
<Frame>
<img alt="Inference Viewer" />
</Frame>
## Table columns
Each row in the table represents a single LLM call. The visible columns are configurable, and include:
| Column | Description |
| --------------------------------- | ------------------------------------------------ |
| **Time** | When the request was sent |
| **Status** | HTTP status code |
| **Model** | The model used |
| **API URL** | The downstream provider endpoint |
| **Environment** | Environment tag (production, staging, etc.) |
| **Task** | The task ID, if one was set |
| **Input / Output / Total Tokens** | Token counts for the request and response |
| **Cached Tokens** | Tokens served from cache |
| **Reasoning Tokens** | Tokens used for reasoning (where applicable) |
| **Cost** | Total cost, with input and output cost breakdown |
| **Duration** | End-to-end request latency |
| **TTFT** | Time to first token (for streaming requests) |
| **Request / Response Size** | Payload sizes in bytes |
You can sort by time, status, cost, duration, tokens, or payload size.
## Filtering
The filter builder lets you combine multiple conditions to narrow down your traffic. Filters are available for both categorical and numeric fields.
**Categorical filters:**
* **Model** - filter to specific models
* **Provider** - filter by upstream provider
* **Task** - filter by task ID
* **Environment** - filter by environment tag
* **Status** - filter by HTTP status code or range (success, error, 2xx, 4xx, 5xx, or specific codes like 429)
* **Streaming** - filter streaming vs non-streaming requests
**Numeric filters:**
* **Duration** - filter by latency (e.g. requests slower than 5s)
* **Cost** - filter by cost (e.g. requests costing more than \$0.05)
* **Input / Output Tokens** - filter by token count (e.g. input > 5k tokens)
* **Request / Response Size** - filter by payload size in bytes
**Quick filters** are available for common queries: input tokens > 5k, cost > \$0.05, duration > 5s, and status = error.
## Detail view
Click on any row to open the detail panel. This shows the full picture of a single inference:
* Full request and response payloads (viewable as raw JSON)
* Cost breakdown (input, output, reasoning, cached)
* Token breakdown (input, output, reasoning, cached) with visual bars
* Duration and time to first token
* Model, provider, task, and environment
* Streaming status
* Request metadata (key-value pairs)
* Geolocation (country, city)
## Save as dataset
You can build datasets from live traffic directly in the Inference Viewer or from the [Datasets tab](/platform/datasets/build-from-traffic). Apply filters to get a representative slice of your data, then click **Save as Dataset** to create an eval or training dataset from the filtered results.
The dataset creation flow:
1. Apply your filters to narrow down the traffic
2. Review the matching inferences
3. Optionally set a limit on how many inferences to include
4. Choose whether this is an eval dataset or a training dataset
5. Name the dataset and save
The saved dataset is immediately available for [running evals](/get-started/run-first-eval) or [training a model](/get-started/train-and-deploy).
<Tip>
Use [task tags](/platform/gateway/tasks) to filter by objective before saving a dataset. This gives you clean, focused samples instead of a mix of unrelated traffic.
</Tip>
## Next steps
<CardGroup>
<Card title="Build a dataset from traffic" icon="database" href="/platform/datasets/build-from-traffic">
Step-by-step guide for turning filtered traffic into datasets.
</Card>
<Card title="Set up your first eval" icon="flask" href="/get-started/run-first-eval">
Use your dataset to compare models with rubric-based scoring.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have data? Upload a JSONL file directly.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group calls by objective for better filtering and per-feature metrics.
</Card>
</CardGroup>
# Integrate
Source: https://docs.inference.net/platform/gateway/integrate
Collect data from your AI application for evaluation and training
Production LLM traffic serves as the backbone for model optimization in Catalyst. Catalyst Gateway records LLM traffic between your application and your current LLM provider, and stores it for later evaluation and training.
There are two ways to capture production LLM traffic through Gateway: use the [Inference CLI](/cli/overview) to automatically instrument your codebase, or wire it up manually.
To get started with Catalyst, create a free account at [inference.net](https://inference.net/register).
## Choose a setup path
Installing with AI is quickest. Use the manual flow if you want to review each change yourself.
<Tabs>
<Tab title="Install with AI">
Use the [Inference CLI](/cli/overview) to automatically initialize a coding agent like [Claude Code](https://code.claude.com/docs/en/overview) to scan your codebase, update your LLM clients, and add required request metadata.
<Steps>
<Step title="Install the CLI and authenticate">
Install the Inference CLI globally and log in. Your browser will open to authenticate.
<Metadata />
```bash theme={"system"}
npm install -g @inference/cli && inf auth login
```
</Step>
<Step title="Run instrumentation in your project">
Navigate to your project root and run instrumentation.
<Metadata />
```bash theme={"system"}
cd /path/to/your/project && inf instrument
```
The command guides you through the following workflow:
* Select a coding agent to use: Claude Code, OpenCode, or Codex.
* Scan your codebase for LLM clients such as OpenAI, Anthropic, LangChain,etc
* Redirect base URLs to the gateway
* Add routing headers so requests are authenticated, forwarded, and traced
* Add task IDs so each call site is grouped automatically in the dashboard
* Review the generated changes before applying them
<Tip>
Run `inf instrument --dry-run` to preview changes without modifying any files.
</Tip>
</Step>
<Step title="Run your app">
Run your application how you normally would to produce inference requests. Requests from your application are now routed through the gateway and will appear in the dashboard.
</Step>
<Step title="View your results">
Open the [dashboard](https://inference.net/dashboard) to see request details, traces, and analytics.
</Step>
</Steps>
<Note>
Want the full canonical guide for this workflow? See [Install with AI](/integrations/install-with-ai).
</Note>
</Tab>
<Tab title="Install manually">
Use this path if you want to review each change yourself. The example below uses OpenAI. For Anthropic, Cerebras, Groq, and other providers, see the [Integrations guide](/integrations/overview).
<Steps>
<Step title="Get your API keys">
You need two keys:
* **Inference Catalyst project API key** — from your [dashboard](https://inference.net/dashboard) under **API Keys**
* **OpenAI API key** — from your [OpenAI account](https://platform.openai.com/api-keys)
Set them as environment variables:
<Metadata />
```bash theme={"system"}
export INFERENCE_API_KEY=<your-project-api-key>
export OPENAI_API_KEY=<your-openai-api-key>
```
</Step>
<Step title="Update your code">
Point your SDK at `https://api.inference.net/v1` and use your Catalyst project API key as the SDK's `apiKey`. Your provider's API key goes in the `x-inference-provider-api-key` header so the gateway can forward it. The gateway adds roughly 10ms of latency and forwards your requests to the provider as-is.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
defaultHeaders: {
"x-inference-provider-api-key": process.env.OPENAI_API_KEY,
"x-inference-provider": "openai",
},
});
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Hello, world!" }],
});
console.log(response.choices[0].message.content);
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
default_headers={
"x-inference-provider-api-key": os.environ["OPENAI_API_KEY"],
"x-inference-provider": "openai",
},
)
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Hello, world!"}],
)
print(response.choices[0].message.content)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'
```
</CodeGroup>
</Step>
<Step title="Send a request">
Run the snippet above from your application or terminal. Once the request completes, Catalyst will capture it automatically.
</Step>
<Step title="View your results">
Open the [Inference Catalyst dashboard](https://inference.net/dashboard) to inspect the request, traces, and metrics.
</Step>
</Steps>
<Note>
Need provider-specific manual instructions? See [Integrations Overview](/integrations/overview).
</Note>
</Tab>
</Tabs>
That's it. Every request now flows through Catalyst and gets captured automatically.
## What gets captured
Once traffic is flowing, Catalyst records:
* The full request and response payloads
* Cost per call and aggregate spend
* Latency (end-to-end and time to first token)
* Token counts (input and output)
* Error rates and status codes
* Model and provider
## Where to find your data
* **[Metrics Explorer](/platform/gateway/metrics-explorer)** - dashboards for cost, latency, errors, and usage across all your LLM calls
* **[Inference Viewer](/platform/gateway/inference-viewer)** - browse and filter individual requests and responses
## Next steps
<CardGroup>
<Card title="Connect more providers" icon="plug" href="/integrations/overview">
Set up Anthropic, Cerebras, Groq, and other providers.
</Card>
<Card title="Organize with tasks" icon="bullseye" href="/platform/gateway/tasks">
Group LLM calls by feature or objective to track metrics separately.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn captured traffic into datasets for evals and training.
</Card>
<Card title="Upload a dataset" icon="upload" href="/platform/datasets/upload-a-dataset">
Already have data? Upload a JSONL file to start evaluating or training.
</Card>
</CardGroup>
# Metrics Explorer
Source: https://docs.inference.net/platform/gateway/metrics-explorer
Dashboards for cost, latency, errors, and token usage across all your LLM calls.
The Metrics Explorer gives you an at-a-glance view of your LLM usage. See trends, spot anomalies, and break down spending by model, task, or provider.
<Frame>
<img alt="Metrics Explorer dashboard" />
</Frame>
## Summary statistics
At the top of the dashboard, you'll see headline numbers for the selected time range:
* **Total inferences** and error count/rate
* **Total cost**, broken down by input, cached input, output, and reasoning tokens
* **Total tokens**, broken down by input (including cached) and output (including reasoning)
* **Average tokens per request** for both input and output
* **Streaming request count**
## Time series charts
Below the summary, time series charts show how your usage changes over time:
| Chart | What it shows |
| ----------------------- | ----------------------------------------------------------------- |
| **Requests** | Total request volume over time |
| **Error rate** | Success rate percentage over time |
| **Cost** | Cost breakdown over time (input, cached input, output, reasoning) |
| **Tokens** | Token usage breakdown over time |
| **Duration** | Latency percentiles (p50, p75, p90, p99) |
| **Time to first token** | TTFT percentiles (p50, p75, p90, p99), useful for streaming |
| **Payload size** | Average request and response sizes |
| **Peak throughput** | Peak requests per time bucket |
| **Model distribution** | Request count broken down by model |
## Breakdown charts
The dashboard also includes distribution views:
* **Model cost comparison** - which models are costing you the most
* **HTTP code breakdown** - distribution of status codes
* **Failures breakdown** - error types and frequency
* **Request types** - streaming vs non-streaming split
## Filtering
You can scope the dashboard to a specific slice of traffic using filters:
* **Task** - see metrics for a single task (requires [task tagging](/platform/gateway/tasks))
* **Model** - filter to a specific model
* **Provider** - filter by upstream provider
* **Environment** - filter by environment tag (production, staging, etc.)
* **Status** - filter by HTTP status code or range (2xx, 4xx, 5xx, specific codes)
## Time controls
* **Time range picker** - select the date range for the dashboard (default: last 24 hours)
* **Granularity selector** - control how data points are bucketed (hourly, daily, etc.)
## Next steps
<CardGroup>
<Card title="Inference Viewer" icon="list" href="/platform/gateway/inference-viewer">
Drill into individual LLM calls to inspect requests, responses, and errors.
</Card>
<Card title="Build a dataset" icon="database" href="/platform/datasets/build-from-traffic">
Turn your traffic into datasets for evals and training.
</Card>
</CardGroup>
# Gateway
Source: https://docs.inference.net/platform/gateway/overview
Record and analyze your production LLM traffic
Catalyst Gateway captures LLM requests flowing through your products. It stores the raw request, response, and metadata associated with each invocation of an LLM.
Recorded data is used to provide in-depth metrics and visibility into your LLM token usage, cost, latency, and error rates, across all your providers in a single, unified view. Additionally, this data is used to power downstream model evaluation and training.
Catalyst Gateway supports all major LLM providers and frameworks. For agents,
tools, framework runs, and custom orchestration, Catalyst Tracing captures full
trace trees and individual spans in addition to gateway inferences. View the
[integrations guide](/integrations/overview) for in-depth instructions.
<Frame>
<iframe title="Observe Every LLM Call in One Place | Catalyst" />
</Frame>
## Key concepts
| Concept | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Gateway** | Edge layer between your app and LLM provider. Records traffic with \< 10ms overhead. |
| **Inference** | A single LLM call stored by Gateway. Includes request, response, cost, latency, & token counts. |
| **Trace** | A multi-step execution captured through OpenTelemetry. Useful for agents, tools, framework runs, and custom orchestration. |
| **Span** | One step inside a trace, such as a model call, tool call, retriever, graph node, or custom application operation. |
| **Task** | A user-defined objective (like "summarize docs" or "classify tickets") that groups related inferences so you can track each AI feature independently. |
| **Metrics** | Aggregated cost, latency, error rates, and token usage across your inferences. Filterable by model, task, or provider. |
## Next steps
<CardGroup>
<Card title="Set up tasks" icon="bullseye" href="/platform/gateway/tasks">
Group your LLM calls by objective.
</Card>
<Card title="Integrate with your LLM provider" icon="plug" href="/platform/gateway/integrate">
Connect your app and start capturing traffic.
</Card>
<Card title="Metrics Explorer" icon="chart-line" href="/platform/gateway/metrics-explorer">
See your LLM usage dashboards.
</Card>
<Card title="Inference Viewer" icon="list" href="/platform/gateway/inference-viewer">
Browse individual LLM calls.
</Card>
<Card title="Trace CLI" icon="terminal" href="/cli/traces">
Inspect trace trees, span timelines, facets, and exports from the terminal.
</Card>
</CardGroup>
# Prompt Versions
Source: https://docs.inference.net/platform/gateway/prompt-versions
Catalyst records a hash of each prompt so you can track changes and see which inferences used which version.
Every time Catalyst sees a new prompt for a task, it records a hash of the prompt content and treats it as a new version. You don't need to do anything to enable this. It happens automatically as requests flow through the gateway.
## How it works
Catalyst hashes the system message of each request. When the hash changes (because you updated your system prompt), a new version is created under that task. The previous versions are preserved, so you have a full history of every system prompt your task has used.
This means you can:
* See which prompt version a specific inference used
* View the full history of prompt changes for a task
* Filter inferences by prompt version to compare behavior across changes
## What triggers a new version
A change to the system message creates a new version. Changes to user messages, parameters like `temperature`, or the model do not currently trigger a new version.
## Coming Soon
* **Broader hashing** - future versions will include user message templates, model, and parameter changes in the version hash
* **Dataset filtering by version** - build training and eval datasets tied to a specific version of a prompt, so you can train on data from the version that performed best
# Tasks Overview
Source: https://docs.inference.net/platform/gateway/tasks
Group LLM calls by objective to track metrics, run evaluations, and train models
A task is a user-defined objective like "summarize this document," "classify this ticket," or "extract these fields." Tagging your LLM calls with a `task-id` groups them by function, not just which model or prompt was used. If you don't set a task, calls are automatically grouped under a `default` task so nothing gets lost.
We recommend using [Install with AI](/integrations/install-with-ai) to automatically add tags to your LLM calls.
<Tip>
To get the most out of Catalyst, we **highly recommend** adding tags to your LLM calls.
</Tip>
## Why tasks matter
Once you have more than one AI feature, tasks let you:
* **Track metrics per feature** — cost, latency, and error rates for each objective independently
* **Run evals per task** — measure whether a specific capability is getting better or worse
* **Build focused datasets** — filter by task to get clean, relevant samples for training
* **Experiment safely** — change the model or prompt for one task without affecting others
The task is the stable anchor. You might swap models, rewrite prompts, or redesign your agent, but the task stays the same.
## How to tag a call
Set the `x-inference-task-id` header on your request. The task appears in the dashboard automatically once the first tagged request comes through. See [How to Create a Task](/platform/gateway/create-a-task) for the full setup.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
const response = await client.chat.completions.create({
model: "gpt-4.1",
messages: [{ role: "user", content: "Summarize this document..." }],
}, {
headers: { "x-inference-task-id": "document-summary" },
});
```
<Metadata />
```python Python theme={"system"}
response = client.chat.completions.create(
model="gpt-4.1",
messages=[{"role": "user", "content": "Summarize this document..."}],
extra_headers={"x-inference-task-id": "document-summary"},
)
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "x-inference-provider-api-key: $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-H "x-inference-provider: openai" \
-H "x-inference-task-id: document-summary" \
-d '{
"model": "gpt-4.1",
"messages": [{"role": "user", "content": "Summarize this document..."}]
}'
```
</CodeGroup>
## Using tasks across the platform
Once calls are tagged, tasks appear as a filter and grouping dimension everywhere:
* **Metrics Explorer** — break down cost, latency, and errors by task
* **Inference Viewer** — filter to see all calls for a specific task
* **Datasets** — filter by task to build focused eval and training datasets
# After Training Completes
Source: https://docs.inference.net/platform/train/after-training
What you get when training finishes and how to evaluate the result before deploying.
When training completes, you get a registered model artifact that's ready to deploy. No manual promotion step — it goes straight from "done" to "deployable."
## What you see
* **Final eval results** — the trained model's scores on your eval dataset, compared against the baseline from before training started
* **Model artifact** — registered in the platform and visible on the **Models** page
* **Deploy button** — appears on the training details page and on your model's row in the Models page
<Frame>
<img alt="Completed training run with final eval results and Deploy button" />
</Frame>
## Is it actually better?
Compare the trained model's final eval scores against the off-the-shelf models you benchmarked earlier in [Eval](/platform/eval/overview). If the trained model scores higher on your rubric, it's ready to deploy. If not, you may need to iterate on your training data or recipe.
## Next steps
<CardGroup>
<Card title="Deploy your model" icon="server" href="/platform/deploy/deploy-a-model">
Ship it to a dedicated GPU.
</Card>
<Card title="Troubleshooting" icon="wrench" href="/platform/train/troubleshooting">
If training failed or results aren't what you expected.
</Card>
</CardGroup>
# Choose a Recipe
Source: https://docs.inference.net/platform/train/choose-a-recipe
Pre-configured training setups that abstract away base model selection and training parameters.
A recipe is a pre-configured training setup. It includes a vetted base model, optimized training parameters, and compute configuration. You don't need ML expertise to pick one.
## What a recipe includes
* **Base model** — selected by the Inference team for quality on the task type
* **Optimized training parameters** — learning rate, epochs, and other hyperparameters
* **Compute configuration** — minimum 8 GPUs per training run
## How to choose
Pick based on **task difficulty and capability needs**, not specific model names. Start small and scale up only if your eval results show the smaller recipe isn't cutting it.
| Tier | Best for |
| ---------- | ------------------------------------------------------------------------------------------------ |
| **Tiny** | High-throughput tasks where speed matters most — simple classification, extraction, tagging |
| **Small** | Fast inference with more capability — structured output, entity extraction, routing |
| **Medium** | Good balance of speed and intelligence — summarization, Q\&A, agentic tasks that need to be fast |
| **Large** | Complex reasoning, multi-step tasks, difficult agentic use cases |
Some recipes offer specific capabilities (like multimodal support) that are only available with certain base models. Choose those when your task requires them.
## Available recipes
These are the pre-built recipes currently available on the platform. Each one has been configured and tested by the Inference team.
| Recipe | Base Model | Parameters | Description |
| ---------- | ------------- | ---------- | ----------------------------------------------------------------------- |
| **Tiny** | Qwen 3.5 0.8B | 0.8B | Tiny and incredibly fast. Best for high-volume, low-complexity tasks. |
| **Small** | Qwen 3.5 4B | 4B | Small and incredibly fast. A good default when latency is the priority. |
| **Medium** | Qwen 3.5 9B | 9B | Good balance of speed and intelligence. Good as a fast agentic model. |
| **Large** | Qwen 3.5 27B | 27B | Large and intelligent. Great for difficult agentic use cases. |
All recipes use 8x H100 GPUs and include optimized training parameters. You don't need to configure any of this — just pick the tier that fits your task.
# Launch a Training Run
Source: https://docs.inference.net/platform/train/launch-a-run
Start a training job from the dashboard — select your datasets, rubric, and recipe, then start training.
Once you have your [prerequisites](/platform/train/overview#prerequisites) ready, launching a training run takes a few minutes.
## Start a new job
Go to the **Training** tab in the dashboard and click **New Training Job**.
<Frame>
<img alt="New Training Job form in the dashboard" />
</Frame>
<Steps>
<Step title="Select a training dataset">
Choose the dataset the model will learn from. You can [build a dataset from live traffic](/platform/datasets/build-from-traffic), [upload a JSONL file](/platform/datasets/upload-a-dataset), or use [task tags](/platform/gateway/tasks) to filter captured traffic for clean, focused samples.
</Step>
<Step title="Select an eval dataset">
Choose the dataset used for [evaluations throughout training](/platform/train/mid-training-evals). Must have zero overlap with training data.
</Step>
<Step title="Select a rubric">
Choose the [rubric](/platform/eval/write-a-rubric) that defines your quality criteria. Make sure you've validated it against your eval dataset first.
</Step>
<Step title="Select a recipe">
Choose a [recipe](/platform/train/choose-a-recipe) based on task difficulty and capability needs.
</Step>
<Step title="Start training">
Review your selections and click **Start Training** to queue the job.
</Step>
</Steps>
## Training job lifecycle
Once started, your job moves through these statuses:
| Status | What's happening |
| ---------------------- | --------------------------------------------------------------------- |
| **Exporting datasets** | Your training and eval data is being prepared |
| **Queued** | Job is waiting for available compute |
| **Starting** | GPUs are being allocated and the training environment is initializing |
| **Running** | The model is actively training |
| **Completed** | Training finished successfully |
If something goes wrong, the job will show one of these:
| Status | What it means |
| ------------- | ------------------------------------------------------------------------------------------------------------- |
| **Failed** | Something went wrong — check the [logs](/platform/train/mid-training-evals#logs) on the training details page |
| **Cancelled** | The run was cancelled |
| **Timed out** | The run exceeded the maximum allowed duration |
Training can take anywhere from around 10 minutes to 10+ hours depending on dataset size and recipe. Once running, [monitor progress](/platform/train/mid-training-evals) from the training details page.
# Monitor a Training Run
Source: https://docs.inference.net/platform/train/mid-training-evals
Track training progress with real-time graphs, eval scores, and GPU logs.
Once a training job starts, the training details page gives you real-time visibility into what's happening. A progress bar shows percentage complete and the current status of the run.
## Training graphs
Four graphs update as training progresses:
| Graph | What it measures | What to look for |
| ----------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Loss** | How far the model's predictions are from expected output | Decreasing = learning. Flattening = model has learned what it can from the data. |
| **Learning rate** | How much weights update at each training step | Warm-up then decay schedule — configured by the recipe automatically. |
| **Gradient norm** | Gradient magnitude during backpropagation | Steady or decreasing = stable. Persistent spikes may indicate a data quality issue. |
| **Eval score** | Average score on the eval dataset at each checkpoint | Trending up = model is improving at your task. This is the most direct signal that training is working. |
<Frame>
<img alt="Training details page showing the four training graphs" />
</Frame>
## Evaluations
The platform runs evaluations at three points during a training job:
1. **Before training** — establishes a baseline score for the model before any weight updates
2. **During training** — at each checkpoint, the model runs your eval dataset and an LLM judge scores the outputs using your rubric
3. **After training** — a final evaluation on the completed model
## Checkpoints
Training saves checkpoints at regular intervals. If a run fails after a checkpoint, it can be resumed from the last saved state rather than starting over.
## Logs
The **Logs** tab shows output from all GPUs during training. Use it to debug issues or see what's happening under the hood. You can filter logs by type — `warn`, `error`, and others — to focus on what matters.
# Train
Source: https://docs.inference.net/platform/train/overview
Fine-tune task-specific models using your data and eval criteria. The platform handles base model selection, parameters, and compute.
<Warning>
**Enterprise required.** Training is available on Enterprise plans. [Contact us](https://inference.net/meet-with-us/) to enable it for your team.
</Warning>
Fine-tuning takes a small open-source model and trains it on your data so it gets really good at one specific task. The result is a model that's smaller, faster, and cheaper to run than the general-purpose model it replaces, while being more accurate for your workload. You don't need to be an ML engineer to use it.
Training is one optimization lever, not the only one. Many teams get massive value from [observability](/platform/gateway/overview) + [evals](/platform/eval/overview) + prompt tuning alone. Fine-tuning is for when you've confirmed via evals that off-the-shelf models aren't good enough for a specific task, and you have the data to prove it.
<Frame>
<iframe title="Fine-Tune a Custom LLM on Your Own Data | Catalyst" />
</Frame>
## Prerequisites
Training requires three things before you start:
1. **A training dataset** - the data the model learns from. Build it from [captured traffic](/platform/datasets/build-from-traffic) or [upload your own](/platform/datasets/upload-a-dataset).
2. **An eval dataset** - measures learning progress. Must have **zero overlap** with training data.
3. **A validated rubric** - run it against your eval dataset first to confirm it measures what you care about. See [Set Up Your First Eval](/get-started/run-first-eval).
## Key concepts
| Concept | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Recipe** | A pre-configured training setup with a vetted base model, optimized parameters, and compute config. You pick by task difficulty, the platform handles the ML complexity. |
| **Training dataset** | The data the model learns from. Diversity and quality matter most. Build from captured traffic or upload your own. |
| **Eval dataset** | A separate dataset that measures learning progress. Must have zero overlap with training data to prevent overfitting. |
| **Rubric** | The quality criteria that guide training. Mid-training evals use it to decide when to stop. If the rubric is wrong, the model optimizes for the wrong thing. |
| **Mid-training evals** | Periodic quality checks during training. If scores improve, training continues. If they degrade, training stops early to prevent overfitting. |
## Why fine-tune
* **Reduce latency** - a smaller, task-specific model responds faster than a general-purpose one
* **Reduce cost** - smaller models cost less to serve at scale
* **Improve accuracy** - a model trained on your data and scored against your rubric is optimized for exactly what you need
* **Maintain ownership** - you own the model artifact and control where it runs
## What makes good training data
The quality of your trained model depends directly on the quality of your data. A few principles:
* **Diversity matters most.** Training data should cover the range of inputs the model will see in production — different phrasings, edge cases, varying complexity. A narrow dataset produces a narrow model.
* **Real traffic beats synthetic data.** Production inputs reflect what users actually send. [Build datasets from live traffic](/platform/datasets/build-from-traffic) when possible.
* **Scope to a single task.** A dataset built from mixed traffic teaches the model many things poorly. Use [task tags](/platform/gateway/tasks) to filter for one objective at a time.
* **More is generally better**, but quality trumps volume. A thousand clean, diverse examples outperform ten thousand repetitive ones.
For eval data, the goal is different: pick a small, stable set of hard examples that stress-test the model. Don't change your eval dataset often — it's your benchmark. See [Datasets](/platform/datasets/overview) for more on building both types.
## Next steps
<CardGroup>
<Card title="Choose a recipe" icon="book-open" href="/platform/train/choose-a-recipe">
Pick a pre-configured training setup for your task.
</Card>
<Card title="Launch a training run" icon="rocket" href="/platform/train/launch-a-run">
End-to-end flow from datasets to queued job.
</Card>
<Card title="Monitor a training run" icon="chart-line" href="/platform/train/mid-training-evals">
Track progress, graphs, and logs during training.
</Card>
<Card title="Deploy a trained model" icon="server" href="/platform/deploy/deploy-a-model">
Ship your model to a dedicated GPU.
</Card>
</CardGroup>
# Troubleshooting Training Failures
Source: https://docs.inference.net/platform/train/troubleshooting
Common training failures, what they look like, and how to recover.
Training runs should generally succeed without intervention. If a run fails, it usually indicates an underlying issue worth investigating.
## Common failures
| Failure | What happened | What to do |
| ------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| **Model collapse / divergence** | Model trained but performs poorly. May have overfit despite good eval scores. | Check data quality and diversity. Try a different recipe. |
| **Insufficient data** | Model fails to generalize well. | Add more representative samples to the training dataset. |
| **Poor data quality** | Training data isn't diverse or heterogeneous enough. | Curate a more varied dataset. Ensure it covers edge cases. |
| **Rate limited** | Too many concurrent training runs. The platform rate-limits usage to prevent abuse. | Wait for your current run to finish, then retry. |
## What you see on failure
* An error message on the training run
* A description of the failure condition
* A retry button
## When to reach out
If training fails unexpectedly, contact support. Failures usually indicate something worth looking into — the team can help diagnose whether it's a data issue, a recipe mismatch, or a platform problem.
<Card title="Talk to an engineer" icon="calendar" href="https://inference.net/meet-with-us/">
Meet with our team to discuss training failures or get help with your approach.
</Card>
## Training succeeded?
Model trained and eval scores look good? Deploy it to a dedicated GPU and start serving traffic.
<Card title="Deploy your model" icon="server" href="/platform/deploy/overview">
Ship your trained model to production.
</Card>
# API Keys and Authentication
Source: https://docs.inference.net/reference/api-keys
Where to find your API key, how authentication works, and security best practices.
## Managing API keys
A default API key is created for you when you create a new project. You can create additional keys or delete existing ones from the [API Keys](https://inference.net/dashboard/api-keys) page in the sidebar.
The API Keys page shows all keys across your organization and which project each one belongs to. When you click **Create API Key**, the new key is scoped to whichever project you currently have selected.
## Project scoping
API keys are scoped to a project. The key you use determines which project's resources you're interacting with — custom models, datasets, training runs, evals, and observability metrics are all grouped by the project the key belongs to.
## Permissions
When creating an API key, you can grant **read** access, **write** access, or both. For example, a read-only key for pulling metrics, or a key with write access for uploading datasets and triggering training runs.
## Authentication
Pass the key as a Bearer token in the `Authorization` header, or set it as your API key in whichever SDK you're using. The same key works across:
* **Inference API** — calling models (both off-the-shelf and custom deployments)
* **Catalyst platform** — SDK and CLI operations
* **Observability** — routing traffic through the gateway
* **MCP server** — connecting compatible AI coding assistants to Catalyst resources
## Best practices
* Keep API keys out of source control
* Rotate keys periodically
* Use environment variables to store keys in your application
# Glossary
Source: https://docs.inference.net/reference/glossary
Quick-reference definitions for Catalyst concepts and terminology.
| Term | Definition |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Task** | A user-defined objective that groups LLM calls (e.g., "summarize document," "classify ticket"). Tasks persist even as the implementation changes. |
| **Rubric** | A plain English description of how to judge model output on a quality dimension. Scored numerically by an LLM judge. |
| **Recipe** | A pre-configured training setup including base model, training parameters, and compute config. |
| **Dataset** | A curated set of inference samples used for evaluation or training. |
| **Eval dataset** | A dataset used to measure model quality. Should remain stable over time. Must not overlap with training data. |
| **Training dataset** | A dataset the model learns from during fine-tuning. Evolves as you iterate on data quality. |
| **Inference** | A single LLM request-response pair captured by the platform. |
| **Deployment** | A trained model running on a dedicated GPU, accessible via an OpenAI-compatible API. |
| **Mid-training eval** | A periodic evaluation run during training that scores model checkpoints against the rubric. |
| **LLM-as-a-judge** | The evaluation mechanism where an LLM scores model outputs against rubric criteria. |
| **TTFT** | Time to first token. Measures streaming responsiveness. |
| **Overfitting** | When a model memorizes training data instead of learning generalizable patterns. Detected by degrading eval scores. |
| **Distillation** | Training a smaller model to replicate the quality of a larger model, reducing cost and latency. |
# Rate Limits
Source: https://docs.inference.net/reference/rate-limits
Request limits, what happens when you hit them, and how to request higher limits.
For detailed rate limit information including current limits by tier, see the [Inference API rate limits](/api/rate-limits) page.
## What happens when you hit a limit
You'll receive a `429 Too Many Requests` response. Back off and retry with exponential backoff.
## Deployment-specific limits
Self-serve deployments on a single GPU have inherent throughput limits. If traffic exceeds capacity, requests slow down and eventually return 429s. For higher throughput, see [Scale to Production](/platform/deploy/scale-to-production).
## Requesting higher limits
If you need higher rate limits, [talk to the team](https://inference.net/meet-with-us/).
# Catalyst Workflow
Source: https://docs.inference.net/workflow
How Tracing and Gateway fit together across the Catalyst platform.
Catalyst is built around two products.
* **Tracing** is for agent improvement. The tracing SDKs capture the full execution of your agents and AI apps: LLM calls, tool calls, framework steps, and any custom spans you wrap. Halo, our open-source agent-loop optimizer, reads those traces and writes up what to fix in prompts, tools, and the harness itself.
* **Gateway** is for inference observability and training task-specific models. Gateway sits between your app and your LLM provider, recording every request. From that recorded traffic, you can build datasets, run evals, fine-tune smaller models, and deploy them on dedicated GPUs.
Tracing and Gateway stand alone. Many teams start with one and add the other later. They also compose: when your agent calls LLMs through Gateway, the trace spans and the gateway records line up against the same requests.
The platform also provides access to open-source and Inference.net-trained models (like [Schematron](/workhorse-models/schematron) for structured data extraction) through an OpenAI-compatible API.
***
## Tracing
Capture full traces of your agents and AI apps. The Catalyst tracing SDKs collect LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Then Halo (our open-source [agent-loop optimizer](https://github.com/context-labs/halo)) reads your traces and surfaces concrete things to improve in your prompts, tools, and agent harness.
**What you'll do:**
* [Capture your first trace](/get-started/capture-first-trace) by installing the tracing SDK and pointing it at Catalyst
* [Analyze your traces](/get-started/analyze-traces) in the Traces and Agents dashboards, then run Halo to find what to fix
* Instrument [providers and frameworks](/integrations/traces/overview) like OpenAI, Anthropic, LangChain, LangGraph, Vercel AI SDK, OpenAI Agents, LiveKit, PI AI, Pydantic AI, and more
* Add stable [agent identity](/integrations/traces/agent-identity) so the Agents dashboard groups runs correctly
**Outcome:** Deep visibility into how your agents behave end to end, plus an automated reviewer that points you to the highest-impact fixes.
<Card title="Get started with Tracing" icon="arrow-right" href="/get-started/capture-first-trace">
Install the SDK and capture your first trace.
</Card>
***
## Gateway
Record and analyze your production LLM traffic. Catalyst Gateway sits between your app and your LLM provider, capturing every request, response, cost, and latency metric with less than 10ms of overhead. Keep using any provider or model — Gateway is transparent.
**What you'll do:**
* [Integrate with your LLM provider](/platform/gateway/integrate) to start capturing traffic
* [Define tasks](/platform/gateway/tasks) to group LLM calls by objective (e.g. "summarize docs", "classify tickets")
* [Explore metrics](/platform/gateway/metrics-explorer) for cost, latency, errors, and token usage across all your calls
* [Browse individual inferences](/platform/gateway/inference-viewer) to inspect raw requests and responses
**Outcome:** Full visibility into how your AI features perform in production — broken down by model, task, and provider.
<Card title="Get started with Gateway" icon="arrow-right" href="/platform/gateway/overview">
Set up Gateway and start capturing LLM traffic.
</Card>
***
## Datasets
Curate collections of LLM inputs and outputs for evaluation and training. Datasets can come from your live production traffic captured through Gateway, or from files you upload directly.
**What you'll do:**
* [Build datasets from traffic](/platform/datasets/build-from-traffic) by filtering captured inferences and saving them
* [Upload your own data](/platform/datasets/upload-a-dataset) as JSONL files when you have curated examples or are migrating from another platform
* Understand [dataset formats](/platform/datasets/formats) and the schema your data needs to follow
**Outcome:** Clean, representative datasets scoped to specific tasks — ready to power evals and training.
<Card title="Get started with Datasets" icon="arrow-right" href="/platform/datasets/overview">
Build or upload your first dataset.
</Card>
***
## Eval
Measure model quality with rubrics scored by LLM judges. Define what "good" looks like for your use case, then score model outputs systematically across candidates. Evals tell you which model is better and by how much — so you can make decisions with data instead of intuition.
**What you'll do:**
* [Write a rubric](/platform/eval/write-a-rubric) that describes your quality criteria in plain English — from a template, AI generation, or scratch
* [Run a model comparison](/platform/eval/run-a-comparison) to score multiple models side by side on your dataset
* Understand how [LLM-as-a-judge](/platform/eval/llm-as-a-judge) scoring works under the hood
* [Read the results](/platform/eval/read-the-results) to interpret scores and decide which model wins
**Outcome:** A repeatable, data-driven way to measure model quality before and after every change — and a validated rubric that can guide training.
<Card title="Get started with Eval" icon="arrow-right" href="/platform/eval/overview">
Define quality, measure it, and compare models.
</Card>
***
## Train
Fine-tune a task-specific model on your production data. The result is a model that's smaller, faster, and cheaper to run than the general-purpose model it replaces — while being more accurate for your workload. You don't need to be an ML engineer to use it.
**What you'll do:**
* [Choose a recipe](/platform/train/choose-a-recipe) — a pre-configured training setup with a vetted base model and optimized parameters
* [Launch a training run](/platform/train/launch-a-run) with your training dataset, eval dataset, and rubric
* [Monitor mid-training evals](/platform/train/mid-training-evals) to track quality scores as the model learns
**Outcome:** A trained, task-specific model that's been validated against your rubric — ready to deploy.
<Card title="Get started with Train" icon="arrow-right" href="/platform/train/overview">
Fine-tune a model on your data.
</Card>
***
## Deploy
Ship your trained model to a dedicated GPU with an OpenAI-compatible API. The API uses the same base URL and API key as the rest of the Inference platform — switching from an off-the-shelf model to your custom model is a one-line code change.
**What you'll do:**
* [Deploy a trained model](/platform/deploy/deploy-a-model) to a dedicated GPU in a few clicks
* [Call your deployment](/platform/deploy/call-your-deployment) using the same OpenAI-compatible SDK you already use
* [Manage and monitor](/platform/deploy/manage-and-monitor) your deployment lifecycle, scaling, and performance
**Outcome:** A production endpoint serving your custom model — and the beginning of the next improvement loop. Deploy, observe, eval, retrain.
<Card title="Get started with Deploy" icon="arrow-right" href="/platform/deploy/overview">
Ship your model to a dedicated GPU.
</Card>
***
## Pick your starting point
<CardGroup>
<Card title="Capture your first trace" icon="route" href="/get-started/capture-first-trace">
Install the tracing SDK and capture LLM calls, tool calls, and agent steps.
</Card>
<Card title="Analyze your traces" icon="microscope" href="/get-started/analyze-traces">
Inspect trace trees and run Halo to find what to improve.
</Card>
<Card title="Record your first LLM call" icon="bolt" href="/get-started/record-first-call">
Route traffic through the Catalyst gateway to capture LLM calls and view metrics.
</Card>
<Card title="Run your first eval" icon="flask" href="/get-started/run-first-eval">
Define quality, measure it, and compare models side by side.
</Card>
<Card title="Train and deploy a model" icon="brain" href="/get-started/train-and-deploy">
The full loop: data, training, and a production endpoint.
</Card>
<Card title="Use the Inference API" icon="code" href="/api/api-quickstart">
Access open-source and Inference.net models directly.
</Card>
</CardGroup>
# ClipTagger
Source: https://docs.inference.net/workhorse-models/cliptagger
Programmatic video understanding built for massive scale
<img />
## Introduction
Programmatic video understanding enables developers to build applications that extract structured information from video content at scale, transforming raw frames into searchable, actionable data. By processing video frames through specialized vision language models, developers can build systems that understand visual content as deeply as they understand text.
[ClipTagger-12b](https://huggingface.co/inference-net/ClipTagger-12b) is a custom 12-billion parameter [vision language model](https://huggingface.co/blog/vlms) (VLM) designed for video understanding at massive scale. The model was trained by the [Inference.net](https://inference.net) research team in collaboration with [Grass](https://grass.io) to meet their trillion-scale video frame captioning needs.
Read the [announcement blog post](https://inference.net/blog/cliptagger-12b).
On our benchmarks, ClipTagger-12b achieves quality on par with GPT-4.1 and outperforms Claude 4 Sonnet, while delivering 15–17x lower costs. We use Gemini-2.5-Pro as an independent judge to evaluate caption quality. The model enables teams operating at scale to perform temporal video understanding, content indexing, and automated video analysis workflows at state-of-the-art model quality while maintaining control over their data and costs.
## Key Features
* **Frontier-quality** – Comparable to GPT-4.1 and better than Claude 4 Sonnet.
* **Cost-Efficient** – 15x lower cost than GPT-4.1 and 17x lower than Claude 4 Sonnet.
* **Production-Ready** – Battle-tested on trillion-scale video frame captioning workloads.
* **Temporal Consistency** – Maintains semantic consistency across video frames.
* **Open Source** – The model is open source and available on [Hugging Face](https://huggingface.co/inference-net/cliptagger-12b).
## Processing at Scale
For large-scale workloads, we recommend using our asynchronous APIs:
<CardGroup>
<Card title="Batch API" icon="list-check" href="/api/async-inference/batch-api">
Handle huge frame queues asynchronously. Submit up to 50,000 requests in one batch, track progress, and get results within 24 hours with optional webhooks.
</Card>
<Card title="Group API" icon="layer-group" href="/api/async-inference/group">
Best for smaller collections of up to 50 frames. Send a simple JSON payload, monitor the whole group as one job, and receive a single callback when it finishes.
</Card>
</CardGroup>
## Getting Started
You'll need an [Inference.net](https://inference.net) account and API key. See our [Quick Start Guide](/api/api-quickstart) for setup instructions.
Install the [OpenAI SDK](https://platform.openai.com/docs/libraries) and set the base URL to `https://api.inference.net/v1`.
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
```
<Metadata />
```python Python theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ.get("INFERENCE_API_KEY"),
)
```
</CodeGroup>
## Basic Usage
The model processes a single video frame per request. In order to process a video, you'll need to send a request for each frame you want to analyze.
**Processing every frame of a video is not always necessary.** Depending on the you case, you may want to process a subset of frames, such as keyframes or frames with high motion.
You can use our [Group API](/api/async-inference/group) to submit multiple frames at once, and received a response once all frames have been processed. This is often ideal for processing 10-20 frames per video.
The model requires a specific system prompt and user prompt to generate structured output. Here's a complete example:
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import fs from "fs";
import { z } from "zod";
// Define the response schema using Zod
const ClipTaggerResponseSchema = z.object({
description: z.string(),
objects: z.array(z.string()).max(10),
actions: z.array(z.string()).max(5),
environment: z.string(),
content_type: z.string(),
specific_style: z.string(),
production_quality: z.string(),
summary: z.string(),
logos: z.array(z.string())
});
// Type inference from the schema
type ClipTaggerResponse = z.infer<typeof ClipTaggerResponseSchema>;
// System and user prompts (use exactly as shown for best results)
const SYSTEM_PROMPT = "You are an image annotation API trained to analyze YouTube video keyframes. You will be given instructions on the output format, what to caption, and how to perform your job. Follow those instructions. For descriptions and summaries, provide them directly and do not lead them with 'This image shows' or 'This keyframe displays...', just get right into the details.";
const USER_PROMPT = `You are an image annotation API trained to analyze YouTube video keyframes. You must respond with a valid JSON object matching the exact structure below.
Your job is to extract detailed **factual elements directly visible** in the image. Do not speculate or interpret artistic intent, camera focus, or composition. Do not include phrases like "this appears to be", "this looks like", or anything about the image itself. Describe what **is physically present in the frame**, and nothing more.
Return JSON in this structure:
{
"description": "A detailed, factual account of what is visibly happening (4 sentences max). Only mention concrete elements or actions that are clearly shown. Do not include anything about how the image is styled, shot, or composed. Do not lead the description with something like 'This image shows' or 'this keyframe is...', just get right into the details.",
"objects": ["object1 with relevant visual details", "object2 with relevant visual details", ...],
"actions": ["action1 with participants and context", "action2 with participants and context", ...],
"environment": "Detailed factual description of the setting and atmosphere based on visible cues (e.g., interior of a classroom with fluorescent lighting, or outdoor forest path with snow-covered trees).",
"content_type": "The type of content it is, e.g. 'real-world footage', 'video game', 'animation', 'cartoon', 'CGI', 'VTuber', etc.",
"specific_style": "Specific genre, aesthetic, or platform style (e.e., anime, 3D animation, mobile gameplay, vlog, tutorial, news broadcast, etc.)",
"production_quality": "Visible production level: e.g., 'professional studio', 'amateur handheld', 'webcam recording', 'TV broadcast', etc.",
"summary": "One clear, comprehensive sentence summarizing the visual content of the frame. Like the description, get right to the point.",
"logos": ["logo1 with visual description", "logo2 with visual description", ...]
}
Rules:
- Be specific and literal. Focus on what is explicitly visible.
- Do NOT include interpretations of emotion, mood, or narrative unless it's visually explicit.
- No artistic or cinematic analysis.
- Always include the language of any text in the image if present as an object, e.g. "English text", "Japanese text", "Russian text", etc.
- Maximum 10 objects and 5 actions.
- Return an empty array for 'logos' if none are present.
- Always output strictly valid JSON with proper escaping.
- Output **only the JSON**, no extra text or explanation.`;
// Function to encode image from URL
async function encodeImageUrl(imageUrl) {
const response = await fetch(imageUrl);
const buffer = Buffer.from(await response.arrayBuffer());
return buffer.toString('base64');
}
// Example usage
const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg";
const base64Image = await encodeImageUrl(imageUrl);
const messages = [
{ role: "system", content: SYSTEM_PROMPT },
{
role: "user",
content: [
{ type: "text", text: USER_PROMPT },
{
type: "image_url",
image_url: {
url: `data:image/jpeg;base64,${base64Image}`,
detail: "high"
},
},
],
},
];
const response = await openai.chat.completions.create({
model: "inference-net/cliptagger-12b",
messages: messages,
temperature: 0.1,
max_tokens: 2000,
response_format: { type: "json_object" },
});
// Parse and validate the JSON response
const rawResult = JSON.parse(response.choices[0].message.content);
const result = ClipTaggerResponseSchema.parse(rawResult); // This will throw if the response doesn't match the schema
// Now 'result' is fully typed as ClipTaggerResponse
console.log(JSON.stringify(result, null, 2));
// You can access typed properties
console.log(`Description: ${result.description}`);
console.log(`Objects found: ${result.objects.length}`);
```
<Metadata />
```python Python theme={"system"}
import base64
import requests
import json
from typing import List
from pydantic import BaseModel, conlist
# Define the response model using Pydantic
class ClipTaggerResponse(BaseModel):
description: str
objects: conlist(str, max_length=10)
actions: conlist(str, max_length=5)
environment: str
content_type: str
specific_style: str
production_quality: str
summary: str
logos: List[str]
# System and user prompts (use exactly as shown for best results)
SYSTEM_PROMPT = "You are an image annotation API trained to analyze YouTube video keyframes. You will be given instructions on the output format, what to caption, and how to perform your job. Follow those instructions. For descriptions and summaries, provide them directly and do not lead them with 'This image shows' or 'This keyframe displays...', just get right into the details."
USER_PROMPT = """
You are an image annotation API trained to analyze YouTube video keyframes. You must respond with a valid JSON object matching the exact structure below.
Your job is to extract detailed **factual elements directly visible** in the image. Do not speculate or interpret artistic intent, camera focus, or composition. Do not include phrases like "this appears to be", "this looks like", or anything about the image itself. Describe what **is physically present in the frame**, and nothing more.
Return JSON in this structure:
{
"description": "A detailed, factual account of what is visibly happening (4 sentences max). Only mention concrete elements or actions that are clearly shown. Do not include anything about how the image is styled, shot, or composed. Do not lead the description with something like 'This image shows' or 'this keyframe is...', just get right into the details.",
"objects": ["object1 with relevant visual details", "object2 with relevant visual details", ...],
"actions": ["action1 with participants and context", "action2 with participants and context", ...],
"environment": "Detailed factual description of the setting and atmosphere based on visible cues (e.g., interior of a classroom with fluorescent lighting, or outdoor forest path with snow-covered trees).",
"content_type": "The type of content it is, e.g. 'real-world footage', 'video game', 'animation', 'cartoon', 'CGI', 'VTuber', etc.",
"specific_style": "Specific genre, aesthetic, or platform style (e.e., anime, 3D animation, mobile gameplay, vlog, tutorial, news broadcast, etc.)",
"production_quality": "Visible production level: e.g., 'professional studio', 'amateur handheld', 'webcam recording', 'TV broadcast', etc.",
"summary": "One clear, comprehensive sentence summarizing the visual content of the frame. Like the description, get right to the point.",
"logos": ["logo1 with visual description", "logo2 with visual description", ...]
}
Rules:
- Be specific and literal. Focus on what is explicitly visible.
- Do NOT include interpretations of emotion, mood, or narrative unless it's visually explicit.
- No artistic or cinematic analysis.
- Always include the language of any text in the image if present as an object, e.g. "English text", "Japanese text", "Russian text", etc.
- Maximum 10 objects and 5 actions.
- Return an empty array for 'logos' if none are present.
- Always output strictly valid JSON with proper escaping.
- Output **only the JSON**, no extra text or explanation.
"""
# Function to encode image from URL
def encode_image_url(image_url):
response = requests.get(image_url)
return base64.b64encode(response.content).decode('utf-8')
# Example usage
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
base64_image = encode_image_url(image_url)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": [
{"type": "text", "text": USER_PROMPT},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}",
"detail": "high"
},
},
],
},
]
response = client.chat.completions.create(
model="inference-net/cliptagger-12b",
messages=messages,
temperature=0.1,
max_tokens=2000,
response_format={"type": "json_object"},
)
# Parse and validate the JSON response
raw_result = json.loads(response.choices[0].message.content)
result = ClipTaggerResponse(**raw_result) # This will raise ValidationError if the response doesn't match the schema
# Now 'result' is a typed Pydantic model instance
print(result.model_dump_json(indent=2))
# You can access typed properties
print(f"Description: {result.description}")
print(f"Objects found: {len(result.objects)}")
```
</CodeGroup>
<Note>
ClipTagger-12b requires these exact system and user prompts for optimal performance. The model was specifically trained with this prompt structure to ensure consistent, high-quality output.
</Note>
## Output
We're going to use the image below as an example.
<img />
The model reliably outputs this exact JSON structure for every frame, with all fields always present. Even when certain elements aren't detected (like `actions` or `logos` in the example above), the model returns empty arrays rather than omitting fields.
```json theme={"system"}
{
"description": "A wooden boardwalk path extends from the foreground into the distance, cutting through a field of tall, vibrant green grass. The path is flanked on both sides by the dense grass. In the background, a line of trees is visible on the horizon under a blue sky with scattered white clouds.",
"objects": [
"Wooden boardwalk",
"Tall green grass",
"Blue sky",
"White clouds",
"Trees"
],
"actions": [],
"environment": "An outdoor, natural landscape, likely a marsh or wetland, on a clear day. The scene is characterized by a wooden boardwalk, lush green vegetation, and a bright blue sky with wispy clouds.",
"content_type": "real-world footage",
"specific_style": "landscape photography",
"production_quality": "professional photography",
"summary": "A wooden boardwalk path winds through a lush green field under a bright blue sky with scattered clouds.",
"logos": []
}
```
<Note>
The JSON response schema shown above is the model's standard output format. All fields will always be present in the response, with empty arrays returned when no relevant elements are detected (e.g., no logos or actions in the frame).
</Note>
## Keyframe selection
There are several ways to select keyframes for a video.
### 1. Frame-by-frame analysis
The simplest way to select keyframes is to process every frame of the video. This is the most accurate way to select keyframes, but it is also the most expensive.
## Example Use Cases
**Video Search & Discovery** - Build searchable video databases by indexing structured metadata and tracking object persistence across frames.
**Content Moderation** - Automated content analysis with consistent categorization for content type detection and quality verification.
**Accessibility** - Generate consistent alt-text and scene summaries for video content with frame-by-frame descriptions.
**Ad Verification** - Ensure sponsored content compliance by tracking product visibility and logo appearances.
## Best Practices
1. **Use the exact prompts** – The provided system and user prompts are optimized for best results
2. **Set low temperature** – Use `temperature: 0.1` for consistent output
3. **Enable JSON mode** – Always set `response_format: {"type": "json_object"}`
4. **Process frames systematically** – Maintain temporal order for better analysis
5. **Batch similar content** – Group frames from the same video or scene
## Limitations
* Single video frame per request only. Use the [Group API](/api/async-inference/group) to process multiple frames at once.
* Maximum image size: 1MB
* Supported formats: JPEG, PNG, WebP, GIF
* English-only descriptions (though it can identify text in other languages)
## Support
For technical support or custom deployment options:
* Email: [support@inference.net](mailto:support@inference.net)
* [Schedule a call](https://inference.net/sales) for a consultation
<Note>
cliptagger-12B was developed in partnership with Grass to process over a billion videos while maintaining perfect schema adherence.
</Note>
# Schematron
Source: https://docs.inference.net/workhorse-models/schematron
Schema-guided extraction from messy HTML
## Introduction
Schematron is a family of long-context models we trained to extract clean, typed JSON from messy HTML. Purpose-built for web scraping, product data ingestion, and converting arbitrary pages into structured data, these models take a user-defined schema and reliably output conforming JSON.
Schematron V2 Small delivers the highest extraction quality, approaching frontier-model accuracy on complex schemas. Schematron V2 Turbo is optimized for throughput and cost, processing pages significantly faster while still exceeding the quality of the previous-generation Schematron 3B. Both models excel at processing long HTML inputs with strict schema adherence.
To learn more about how we trained Schematron and what it can do, see the [announcement blog post](https://inference.net/blog/schematron?utm_source=docs).
<Tip>
For best results, pre-clean HTML with `lxml` to remove scripts, styles, and boilerplate before sending it to the model. See [Preprocess HTML (recommended)](#preprocess-html-recommended).
Alternatives like [Readability](https://github.com/mozilla/readability) or regex can work; `lxml` is simply recommended for this model because it matches how the training data was preprocessed. When preprocessing, err on the side of removing less content, as the model will still be able to extract the information you need.
</Tip>
## Quickstart
<CardGroup>
<Card title="Schematron V2 Small" icon="rocket" href="https://inference.net/models/schematron-v2-small">
Highest extraction quality for complex schemas and long pages.
</Card>
<Card title="Schematron V2 Turbo" icon="bolt" href="https://inference.net/models/schematron-v2-turbo">
Optimized for maximum throughput and lowest cost at scale.
</Card>
<Card title="Notebook Example" icon="code" href="https://github.com/context-labs/inference-samples/blob/main/examples/schematron-scrape-companies/schematron-scrape-companies.ipynb">
End-to-end company scraping walkthrough using Schematron.
</Card>
</CardGroup>
<Note>
You'll need an Inference.net account, API key, and the OpenAI SDK configured with `base_url="https://api.inference.net/v1"`. See the [Quick Start Guide](/api/api-quickstart) for setup details.
</Note>
<Note>
**TypeScript Users**: The examples below use OpenAI SDK v6 and Zod v3. Install the compatible version with:
```bash theme={"system"}
npm install zod@^3
```
</Note>
## Basic Usage: Extract from HTML into a Pydantic model
Use a typed model (e.g., Pydantic) to define your schema. Pass the model's JSON Schema and the raw HTML to the model, enable JSON mode, and validate the response.
<Info>
This model does not use user/system prompts. It only uses the schema to extract the data.
</Info>
<CodeGroup>
<Metadata />
```typescript TypeScript theme={"system"}
import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
const openai = new OpenAI({
baseURL: "https://api.inference.net/v1",
apiKey: process.env.INFERENCE_API_KEY,
});
// 1) Define your schema (nested data and lists are supported)
const Product = z.object({
name: z.string(),
price: z
.number()
.describe(
"Primary price of the product."
),
specs: z
.record(z.string())
.describe(
"Specs of the product."
),
tags: z
.array(z.string())
.describe("Tags assigned of the product."),
});
// 2) Messy HTML (could be the full page; trim to the relevant region when possible)
const html = `
<div id="item">
<h2 class="title">MacBook Pro M3</h2>
<p>Price: <b>$2,499.99</b> USD</p>
<ul info>
<li>RAM: 16GB</li>
<li>Storage: 512GB SSD</li>
</ul>
<span class="tag">laptop</span>
<span class="tag">professional</span>
<span class="tag">macbook</span>
<span class="tag">apple</span>
</div>
`;
// 3) Chat Completions extraction (typed validation)
const resp = await openai.chat.completions.parse({
model: "inference-net/schematron-v2-small",
messages: [{ role: "user", content: html }],
response_format: zodResponseFormat(Product, "product"),
});
console.log(resp.choices[0].message.content);
```
<Metadata />
```python Python theme={"system"}
import os
from pydantic import BaseModel, Field
from openai import OpenAI
# 1) Define your schema (nested data and lists are supported)
class Product(BaseModel):
name: str
price: float = Field(
...,
description="Primary price of the product.",
)
specs: dict[str, str] = Field(
default_factory=dict,
description="Specs of the product.",
)
tags: list[str] = Field(
default_factory=list,
description="Tags assigned of the product.",
)
# 2) Messy HTML (could be the full page; trim to the relevant region when possible)
html = """
<div id="item">
<h2 class="title">MacBook Pro M3</h2>
<p>Price: <b>$2,499.99</b> USD</p>
<ul info>
<li>RAM: 16GB</li>
<li>Storage: 512GB SSD</li>
</ul>
<span class="tag">laptop</span>
<span class="tag">professional</span>
<span class="tag">macbook</span>
<span class="tag">apple</span>
</div>
"""
# 3) Client setup
client = OpenAI(
base_url="https://api.inference.net/v1",
api_key=os.environ.get("INFERENCE_API_KEY"),
)
resp = client.beta.chat.completions.parse(
model="inference-net/schematron-v2-small",
messages=[
{"role": "user", "content": html},
],
response_format=Product,
)
print(resp.choices[0].message.parsed.model_dump_json(indent=2))
```
<Metadata />
```bash cURL theme={"system"}
curl https://api.inference.net/v1/chat/completions \
-H "Authorization: Bearer $INFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "inference-net/schematron-v2-small",
"messages": [
{ "role": "user", "content": "<div id=\"item\">\n <h2 class=\"title\">MacBook Pro M3</h2>\n <p>Price: <b>$2,499.99</b> USD</p>\n <ul info>\n <li>RAM: 16GB</li>\n <li>Storage: 512GB SSD</li>\n </ul>\n <span class=\"tag\">laptop</span>\n <span class=\"tag\">professional</span>\n <span class=\"tag\">macbook</span>\n <span class=\"tag\">apple</span>\n</div>" }
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "product",
"strict": true,
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number", "description": "Primary price of the product." },
"specs": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Specs of the product." },
"tags": { "type": "array", "items": { "type": "string" }, "description": "Tags assigned of the product." }
},
"required": ["name", "price", "specs", "tags"],
"additionalProperties": false
}
}
}
}'
```
</CodeGroup>
## Output
For the HTML above, Schematron will produce strictly valid JSON that conforms to your schema. A representative output is:
<Metadata />
```json theme={"system"}
{
"name": "MacBook Pro M3",
"price": 2499.99,
"specs": {
"RAM": "16GB",
"Storage": "512GB SSD"
},
"tags": [
"laptop",
"professional",
"macbook",
"apple"
]
}
```
## Best Practices
1. **Keep temperature at zero** – This model is trained to perform best with 0 temperature.
2. **Provide a clear schema** – Pass a JSON Schema or typed model (e.g., `Product.model_json_schema()`). Include required fields and types, optionally describing them directly within the schema (supported by Pydantic and Zod). For example:
<CodeGroup>
```typescript TypeScript theme={"system"}
const Product = z.object({
name: z.string().describe("Exact product name as shown in the title or primary heading."),
price: z
.number()
.describe(
"Primary price of the product."
),
specs: z.object({
ram: z.string().describe("RAM capacity."),
storage: z.string().describe("Storage capacity and type."),
}).describe("Key attributes describing the product."),
});
```
```python Python theme={"system"}
from pydantic import BaseModel, Field
class Specs(BaseModel):
ram: str = Field(..., description="RAM capacity.")
storage: str = Field(..., description="Storage capacity and type.")
class Product(BaseModel):
name: str = Field(
...,
description=(
"Exact product name as shown in the title or primary heading."
),
)
price: float = Field(
...,
description="Primary price of the product.",
)
specs: Specs = Field(
...,
description="Key attributes describing the product.",
)
```
</CodeGroup>
4. **Pre-clean HTML (recommended)** – Remove scripts, styles, and boilerplate to improve accuracy and reduce tokens. See [Preprocess HTML (recommended)](#preprocess-html-recommended). Alternatives (Readability, Trafilatura, BeautifulSoup, regex) are fine; `lxml` aligns with training.
5. **Validate on ingest** – Parse with Pydantic (or your validator) and handle validation errors explicitly. Although Schematron should always return valid JSON matching your schema, it's still a good idea to validate the response.
## Preprocess HTML (recommended)
Schematron models were trained on HTML that had been pre-cleaned with `lxml.html.clean.Cleaner` to strip scripts, styles, and inline JavaScript. Aligning your preprocessing with training typically improves extraction quality and consistency.
<Metadata />
```python Python theme={"system"}
import lxml.html as LH
from lxml.html.clean import Cleaner
HTML_CLEANER = Cleaner(
scripts=True,
javascript=True,
style=True,
inline_style=True,
safe_attrs_only=False,
)
def strip_noise(html: str) -> str:
"""Remove scripts, styles, and JavaScript from HTML using lxml."""
if not html or not html.strip():
return ""
try:
doc = LH.fromstring(html)
cleaned = HTML_CLEANER.clean_html(doc)
return LH.tostring(cleaned, encoding="unicode")
except Exception:
return ""
```
Notes:
* You do not have to use `lxml`. Alternatives like Readability (often more aggressive), Trafilatura, BeautifulSoup, or even targeted regex can be acceptable.
* Use whichever tool best fits your content; `lxml` is recommended for this model because it matches how the training data was preprocessed.
## Capabilities & Access
### Key Features
* **Schema-first extraction** – Drive output with your own JSON Schema or a typed model (e.g. Pydantic) and get strictly formatted JSON back.
* **Long-context HTML** – Trained for long documents with up to 128K-token context and robust to noisy markup.
* **Strict JSON mode** – 100% schema adherence.
* **Cost-efficient quality** – Matches the quality of frontier models at a significantly lower cost.
### Models
* **inference-net/schematron-v2-small** – Best quality for complex schemas or very long pages.
* **inference-net/schematron-v2-turbo** – Optimized for maximum throughput and lowest cost at scale.
### Access Schematron
**Serverless API**
* [Schematron V2 Small](https://inference.net/models/schematron-v2-small)
* [Schematron V2 Turbo](https://inference.net/models/schematron-v2-turbo)
## Processing at Scale
For large-scale extraction, use our asynchronous APIs:
<CardGroup>
<Card title="Batch API" icon="list-check" href="/api/async-inference/batch-api">
Submit up to 50,000 extraction jobs at once, watch their status asynchronously, and receive optional webhook updates within a 24-hour window.
</Card>
<Card title="Group API" icon="layer-group" href="/api/async-inference/group">
Send smaller groups (≤ 50 requests) in one JSON payload, follow the entire group as a single job, and get one callback when everything is done.
</Card>
</CardGroup>
## Limitations
* **Context window** – Up to 128K tokens per request. Truncate or chunk very large pages.
* **Ambiguous fields** – If a field requires summarization or synthesis, ensure your schema/description is explicit about expectations. Schematron does not support a prompt or directions, instead all information needs to be passed through the schema.
## Legacy Models
<Warning>
The first-generation **Schematron 3B** (`inference-net/schematron-3b`) and **Schematron 8B** (`inference-net/schematron-8b`) models have been retired. Requests to these identifiers are now transparently routed to their V2 successors to avoid breakage:
* Calls to `inference-net/schematron-3b` are served by `inference-net/schematron-v2-turbo`.
* Calls to `inference-net/schematron-8b` are served by `inference-net/schematron-v2-small`.
V2 pricing applies on these rewritten requests — see the pricing pages for [Schematron V2 Turbo](https://inference.net/models/schematron-v2-turbo) and [Schematron V2 Small](https://inference.net/models/schematron-v2-small) for current per-token rates. We strongly recommend updating your model identifiers explicitly so your code, dashboards, and receipts all reflect the model you intend to call.
The original open-source weights remain available on Hugging Face for self-hosting:
* [Schematron-8B on Hugging Face](https://huggingface.co/inference-net/Schematron-8B)
* [Schematron-3B on Hugging Face](https://huggingface.co/inference-net/Schematron-3B)
</Warning>
## Support
For technical support or custom deployment options:
* Email: [support@inference.net](mailto:support@inference.net)
* [Schedule a call](https://inference.net/sales)