Traces - Inference.net Documentation

Catalyst Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Spans are emitted as OpenInference-shaped OpenTelemetry, exported over OTLP/HTTP, and shown in Catalyst as trace trees, agent sessions, and per-span IO. Use tracing for agent improvement. Once spans are flowing, point Halo, our open-source agent-loop optimizer, at your traces and get back concrete suggestions for fixing prompts, tools, and harness logic. See Analyze your traces for the dashboard and Halo flow.

Tracing and the Catalyst Gateway are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For gateway-only setups, see model provider integrations.

How Tracing Fits Together

Tracing has three layers. You add them in order, and most apps stop at the second. 1. Automatic SDK spans. Pass an SDK to setup({ modules }) and every call through it becomes its own span, with input messages, output, model name, finish reason, and token counts. You change nothing at the call site. For a one-shot script, or a batch of unrelated LLM calls, this is all you need. 2. An agent span around the run. When one request drives several model or tool calls (an agent run, a conversation turn, a workflow), wrap it in agentSpan. The agent span is the parent row the SDK spans nest under, so you see “Support Agent” with “openai chat” and “lookup_order” beneath it instead of three orphan rows. It carries agentId, agentName, role, and sessionId for grouping in the Agents dashboard, and holds the run’s high-level input and output. Without it the SDK spans are still captured, just as a flat list with no agent or conversation to group them by. 3. Manual spans for your own steps. When a step inside the run is not an SDK call (a tool you execute yourself, a retrieval, a router, a validation pass), wrap it in manualSpan with the matching span kind. It nests under the agent span next to the auto-captured LLM spans, so the trace tree reflects everything the run did, not just the model calls.

You’re doing…	What you need
One-shot call, or independent calls with no orchestration	`setup({ modules })`. Done.
Several calls per request, one agent	Add `agentSpan({ agentId, agentName, sessionId })` around the run
Multi-agent system	A separate `agentSpan` per agent, each with its own `agentId` and `role`
A non-SDK step inside the run (tool, retrieval, router, validator)	`manualSpan({ spanKind: TOOL / CHAIN / RETRIEVER })` inside the agent span
Subprocess or CLI call	`agentSpan` around it (no SDK to auto-patch)
Framework with native capture (LangGraph, OpenAI Agents, Cursor SDK, Vercel Eve, Vercel AI SDK, LiveKit, ElevenLabs, Claude Agent SDK, PI AI)	Use the framework guide’s setup path; add `agentSpan` when you need stable identity and session grouping

Set sessionId, agentId, and agentName once on the outer agentSpan. Every span underneath inherits them, so you should not be setting these in multiple places.

Packages

Two packages, one shape. Install whichever matches your runtime.

Runtime	Install	Import	Source
TypeScript / Node / Bun	`@inference/tracing`	`import { setup } from "@inference/tracing"`	npm
Python (3.10+)	`inference-catalyst-tracing`	`from inference_catalyst_tracing import setup`	PyPI

Both packages share the same setup() entry point, the same env-var contract, and the same OpenInference attribute conventions. Pick whichever fits your stack and write the same code shape in either language.

Provider SDKs (openai, @anthropic-ai/sdk, langchain, etc.) are optional peer dependencies. Install only the ones you use. For Python, install per-integration extras: pip install 'inference-catalyst-tracing[openai,langchain]'.

Getting Started

The fastest path is the Inference CLI:

inf instrument --mode tracing

tracing mode installs the SDK and wires up setup(). both mode does the same plus gateway routing in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See Install with AI for the full flow. Prefer to wire it up by hand? Start with the path that matches what you want to instrument:

Quickstart installs the SDK, configures export, and captures your first provider span. Both Install with AI and Install manually paths included.
Provider and framework guides cover OpenAI, Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, Claude Agent SDK, Claude Code, Pydantic AI, ElevenLabs Agents, Vercel Eve, PI AI, Cursor SDK, the Vercel AI SDK, and other supported surfaces.
Manual spans wrap custom agents, subprocesses, tools, retrieval, routing, and unsupported SDKs.
Agent identity sets stable agent.id values so the Agents dashboard groups executions correctly.

Supported Trace Integrations

OpenAI: Trace Chat Completions, tool calls, structured outputs, and Responses API calls.
Anthropic: Trace Messages API calls, tool use round trips, and prompt-cache usage.
LangChain: Capture chains, agents, model calls, and tools through callback instrumentation.
LangGraph: Capture graph and node spans while preserving parent-child relationships.
LangSmith: Bridge LangSmith OpenTelemetry spans into Catalyst.
OpenAI Agents: Trace agent runs, tools, handoffs, and nested OpenAI model calls.
LiveKit Agents: Trace LiveKit sessions, model nodes, and function tools through native OTel spans.
ElevenLabs Agents: Trace conversation sessions, transcripts, and client tool calls.
Vercel Eve: Trace Eve agent turns, model calls, sub-agent invocations, tools, and $eve.* workflow tags.
PI AI: Trace PI AI model turns, streams, tool calls, usage, costs, and agent identity.
Cursor SDK: Trace Cursor agent runs, streamed messages, and tool calls.
Vercel AI SDK: Trace generateText, streamText, ToolLoopAgent, tool calls, structured output, and usage.
Claude Agent SDK: Trace Claude Agent SDK query loops and yielded messages.
Claude Code SDK: Trace Claude Code CLI subprocess calls and SDK-style agent invocations.
OpenCode: Record your own OpenCode coding sessions with LLM calls, tool calls, token usage, cost, and errors.
Pydantic AI: Trace Pydantic AI agent runs and tool calls (Python only).
Manual spans: Wrap custom agents, CLI calls, provider routing, and unsupported SDKs.
Agent identity: Set stable agent IDs, display names, and roles for dashboard grouping.

What Gets Captured

Span data	Examples
Inputs and outputs	`input.value`, `output.value`
Messages	user, system, assistant, tool, and tool-result messages
Tool calls	tool names, IDs, JSON arguments, and tool results
Model metadata	`llm.model_name`, `llm.invocation_parameters`, `gen_ai.system`
Usage	prompt, completion, total, and prompt-cache token counts when available
Agent structure	agent spans, framework spans, tool spans, graph/node spans
Agent identity	`agent.id`, `agent.name`, and `agent.role` for Agents dashboard grouping
Errors	exception status and error details on failed spans

All keys follow OpenInference conventions, so the same spans can be consumed by any OpenInference-aware viewer.

Configuration

Both SDKs read the same env vars and accept the same options on setup().

Env Vars And Identity

Env var	Purpose
`CATALYST_OTLP_ENDPOINT`	OTLP/HTTP trace ingest endpoint. Set to `https://telemetry.inference.net` for the hosted Catalyst backend.
`CATALYST_OTLP_TOKEN`	Bearer token for ingest auth. Generate one at API Keys.
`CATALYST_SERVICE_NAME`	OTel `service.name` resource attribute. Use a stable value per deployed service.
`CATALYST_SERVICE_VERSION`	OTel `service.version` resource attribute.
`CATALYST_DEBUG`	Enable SDK debug logging.

Precedence: explicit options on setup() win, then CATALYST_* env vars, then SDK defaults. Legacy OTLP_ENDPOINT, OTLP_INGEST_TOKEN, and SERVICE_NAME are still accepted but prefer CATALYST_* in new setups.

Instrumentation Control

Two options control which SDKs Catalyst patches at setup time.

Option	Default	Purpose
`modules` (TypeScript)	`{}`	Map of SDK modules to patch. Pass `{ openai: OpenAI, anthropic: Anthropic }`. Entries listed here are always patched. Python auto-detects installed packages and does not need this option.
`autoInstrument`	`true`	When `true`, `setup()` also scans `node_modules` and patches any other supported SDK it finds. When `false`, only the SDKs you list in `modules:` get patched, nothing else.

Three patterns, in order of how often they show up in real apps: Auto-detect everything. Call setup() with no modules:. The auto-scan finds every supported SDK and patches it. Easiest path for scripts, demos, and small apps.

const tracing = await setup();

Explicit modules plus auto-detect. Pass the SDKs you use to modules:. Those entries are patched directly (no resolver gymnastics), and setup() also auto-detects anything else.

const tracing = await setup({ modules: { openai: OpenAI } });

Explicit only. Pass modules: and set autoInstrument: false. Only the SDKs you list get patched, even if a transitive dep happens to pull in another supported one. Cleanest pattern for production services that know exactly which providers they call.

const tracing = await setup({
  modules: { anthropic: Anthropic },
  autoInstrument: false,
});

Pass only the SDK modules you actually use to modules. Auto-instrumentation patches prototypes, so patching an SDK you do not call adds no runtime cost but a small import cost.

Imports (TypeScript)

For almost all setups, @inference/tracing is the only import you need. setup() auto-patches every supported SDK it can find (OpenAI, Anthropic, LangChain, LangGraph, ElevenLabs, LiveKit, PI AI, and the rest), so you do not need to reach for the per-integration helpers yourself.

Export	What it does
`setup`	Initializes tracing. Pass the SDK modules you use; it patches them and returns a handle for shutdown.
`agentSpan`	Opens a parent span that groups everything an agent run does. Use one per top-level invocation.
`manualSpan`	Opens a custom span for work that is not an SDK call: tool execution, retrieval, validation, etc.
`wrapClaudeAgentSdkQuery`	Wraps a Claude Agent SDK `query()` call so each tool turn becomes a span.
`Attr`, `SpanKindValues`	OpenInference attribute keys and span-kind enums for setting custom attributes.
Error types	`CatalystInstrumentationError` and friends, for catching setup failures.

When you need a subpath

Three cases call for importing from a per-integration subpath:

You’re using the Vercel AI SDK. The AI SDK has its own experimental_telemetry option, so there is nothing for setup() to patch. You import createAISdkTelemetrySettings and pass its result into each generateText / streamText call.
You’re using Vercel Eve. Eve enables telemetry by loading agent/instrumentation.ts, so you export defineCatalystEveInstrumentation() from @inference/tracing/eve in that file.
You’re installing instrumentation on a custom tracer provider instead of the one setup() registers. Rare in practice; most apps want the standard flow. The subpath helpers (installOpenAIInstrumentation, etc.) accept an explicit provider argument.

If neither applies, you can skip the table below.

Import	Exports	Use For
`@inference/tracing/ai-sdk`	`createAISdkTelemetrySettings`, `instrumentAISdk`, `installAISdkInstrumentation`	Vercel AI SDK
`@inference/tracing/openai`	`instrumentOpenAI`, `installOpenAIInstrumentation`	OpenAI
`@inference/tracing/anthropic`	`instrumentAnthropic`, `installAnthropicInstrumentation`	Anthropic
`@inference/tracing/langchain`	`instrumentLangChain`, `installLangChainInstrumentation`	LangChain
`@inference/tracing/langgraph`	`instrumentLangGraph`, `installLangChainInstrumentation`	LangGraph
`@inference/tracing/langsmith`	`installLangSmith`	LangSmith bridge
`@inference/tracing/openai-agents`	`instrumentOpenAIAgents`	`@openai/agents`
`@inference/tracing/claude-agent-sdk`	`wrapClaudeAgentSdkQuery`	Claude Agent SDK
`@inference/tracing/elevenlabs`	`instrumentElevenLabs`, `installElevenLabsInstrumentation`	ElevenLabs Agents
`@inference/tracing/eve`	`defineCatalystEveInstrumentation`, `installEveSpanEnrichment`	Vercel Eve
`@inference/tracing/pi-ai`	`instrumentPiAi`, `installPiAiInstrumentation`	PI AI
`@inference/tracing/cursor-sdk`	`instrumentCursorSdk`, `installCursorSdkInstrumentation`	Cursor SDK
`@inference/tracing/livekit-agents`	`instrumentLiveKitAgents`, `installLiveKitAgentsInstrumentation`	LiveKit Agents
`@inference/tracing/semconv`	`OpenInferenceAttribute`, `MessageRole` (plus `Attr` and `SpanKindValues`, also on the root)	OpenInference constants

Python uses one import root (inference_catalyst_tracing) and gates each integration behind a pip extra (pip install 'inference-catalyst-tracing[openai,anthropic]').

The `setup()` Return Value

setup() returns a CatalystTracing handle:

Field	Purpose
`tracer`	The OpenTelemetry `Tracer` for authoring manual spans. In Python, pass it to `agent_span` / `manual_span`. In TypeScript, `agentSpan` / `manualSpan` resolve it automatically after `setup()`, so this is an escape hatch — use it for `tracer.startActiveSpan(...)` directly.
`provider`	The OTel tracer provider, in case you want to register additional span processors or exporters.
`installResults`	Per-integration install result: which SDKs were detected, which were patched, and any errors. Useful in CI and on startup logs.
`shutdown()`	Flush and close tracing. Call this on `SIGTERM` for long-lived services, before `process.exit(0)` for scripts.

For long-lived and serverless processes, memoize the setup() promise and flush correctly for the process shape. See Flushing and process lifecycle.

How It Works

setup() configures an OpenTelemetry tracer provider with a Catalyst OTLP/HTTP exporter.
The SDK auto-patches any supported provider or framework SDK it finds. In TypeScript you pass them via modules. In Python it auto-detects installed packages.
Your app runs normally. Spans are batched and exported in the background.
Spans land in Catalyst grouped by service, trace, span, and task metadata. View them in the Traces tab or the Agents tab.
Run analysis on them with Halo to understand your traces and gain insights you can pass into your favorite coding agent.

Inspect Traces from the CLI

After spans are flowing into Catalyst, you can inspect the same trace data from the terminal:

# Browse recent traces
inf trace list --range 1h

# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline

# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io

See inf trace and inf span for the full command reference.

Next Steps

Quickstart

Install an SDK, configure export, and capture your first model span.

Production agent example

A production-shaped agent with custom tool spans, end to end.

Manual spans

Author AGENT, TOOL, CHAIN, and RETRIEVER spans.

Attributes reference

Every Attr.* constant and SpanKindValues value.

Handle API reference

Every method on the span handle yielded by agentSpan / manualSpan and the Python equivalents.

Instrumentation examples

Copyable examples for providers, agents, tool loops, and prompt caching.

Agent identity

Add stable IDs and readable names to agent spans.

Troubleshooting

Debug missing spans, missing attributes, and shutdown behavior.

​How Tracing Fits Together

​Packages

​Getting Started

​Supported Trace Integrations

​What Gets Captured

​Configuration

​Env Vars And Identity

​Instrumentation Control

​Imports (TypeScript)

​When you need a subpath

​The setup() Return Value

​How It Works

​Inspect Traces from the CLI

​Next Steps

Quickstart

Production agent example

Manual spans

Attributes reference

Handle API reference

Instrumentation examples

Agent identity

Troubleshooting

How Tracing Fits Together

Packages

Getting Started

Supported Trace Integrations

What Gets Captured

Configuration

Env Vars And Identity

Instrumentation Control

Imports (TypeScript)

When you need a subpath

The `setup()` Return Value

How It Works

Inspect Traces from the CLI

Next Steps