Skip to main content
Catalyst Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Spans are emitted as OpenInference-shaped OpenTelemetry, exported over OTLP/HTTP, and shown in Catalyst as trace trees, agent sessions, and per-span IO. Use tracing for agent improvement. Once spans are flowing, point Halo, our open-source agent-loop optimizer, at your traces and get back concrete suggestions for fixing prompts, tools, and harness logic. See Analyze your traces for the dashboard and Halo flow.
Tracing and the Catalyst Gateway are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For gateway-only setups, see model provider integrations.

​
How Tracing Fits Together

Tracing has three layers. You add them in order, and most apps stop at the second. 1. Automatic SDK spans. Pass an SDK to setup({ modules }) and every call through it becomes its own span, with input messages, output, model name, finish reason, and token counts. You change nothing at the call site. For a one-shot script, or a batch of unrelated LLM calls, this is all you need. 2. An agent span around the run. When one request drives several model or tool calls (an agent run, a conversation turn, a workflow), wrap it in agentSpan. The agent span is the parent row the SDK spans nest under, so you see “Support Agent” with “openai chat” and “lookup_order” beneath it instead of three orphan rows. It carries agentId, agentName, role, and sessionId for grouping in the Agents dashboard, and holds the run’s high-level input and output. Without it the SDK spans are still captured, just as a flat list with no agent or conversation to group them by. 3. Manual spans for your own steps. When a step inside the run is not an SDK call (a tool you execute yourself, a retrieval, a router, a validation pass), wrap it in manualSpan with the matching span kind. It nests under the agent span next to the auto-captured LLM spans, so the trace tree reflects everything the run did, not just the model calls.
You’re doing…What you need
One-shot call, or independent calls with no orchestrationsetup({ modules }). Done.
Several calls per request, one agentAdd agentSpan({ agentId, agentName, sessionId }) around the run
Multi-agent systemA separate agentSpan per agent, each with its own agentId and role
A non-SDK step inside the run (tool, retrieval, router, validator)manualSpan({ spanKind: TOOL / CHAIN / RETRIEVER }) inside the agent span
Subprocess or CLI callagentSpan around it (no SDK to auto-patch)
Framework with native capture (LangGraph, OpenAI Agents, Cursor SDK, Vercel AI SDK, LiveKit, ElevenLabs, Claude Agent SDK, PI AI)setup() captures the run; add agentSpan for stable identity and session grouping
Set sessionId, agentId, and agentName once on the outer agentSpan. Every span underneath inherits them, so you should not be setting these in multiple places.

​
Packages

Two packages, one shape. Install whichever matches your runtime.
RuntimeInstallImportSource
TypeScript / Node / Bun@inference/tracingimport { setup } from "@inference/tracing"npm
Python (3.10+)inference-catalyst-tracingfrom inference_catalyst_tracing import setupPyPI
Both packages share the same setup() entry point, the same env-var contract, and the same OpenInference attribute conventions. Pick whichever fits your stack and write the same code shape in either language.
Provider SDKs (openai, @anthropic-ai/sdk, langchain, etc.) are optional peer dependencies. Install only the ones you use. For Python, install per-integration extras: pip install 'inference-catalyst-tracing[openai,langchain]'.

​
Getting Started

The fastest path is the Inference CLI: 
inf instrument --mode tracing
tracing mode installs the SDK and wires up setup(). both mode does the same plus gateway routing in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See Install with AI for the full flow. Prefer to wire it up by hand? Start with the path that matches what you want to instrument:
  1. Quickstart installs the SDK, configures export, and captures your first provider span. Both Install with AI and Install manually paths included.
  2. Provider and framework guides cover OpenAI, Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, Claude Agent SDK, Claude Code, Pydantic AI, ElevenLabs Agents, PI AI, Cursor SDK, the Vercel AI SDK, and other supported surfaces.
  3. Manual spans wrap custom agents, subprocesses, tools, retrieval, routing, and unsupported SDKs.
  4. Agent identity sets stable agent.id values so the Agents dashboard groups executions correctly.

​
Supported Trace Integrations

  • OpenAI: Trace Chat Completions, tool calls, structured outputs, and Responses API calls.
  • Anthropic: Trace Messages API calls, tool use round trips, and prompt-cache usage.
  • LangChain: Capture chains, agents, model calls, and tools through callback instrumentation.
  • LangGraph: Capture graph and node spans while preserving parent-child relationships.
  • LangSmith: Bridge LangSmith OpenTelemetry spans into Catalyst.
  • OpenAI Agents: Trace agent runs, tools, handoffs, and nested OpenAI model calls.
  • LiveKit Agents: Trace LiveKit sessions, model nodes, and function tools through native OTel spans.
  • ElevenLabs Agents: Trace conversation sessions, transcripts, and client tool calls.
  • PI AI: Trace PI AI model turns, streams, tool calls, usage, costs, and agent identity.
  • Cursor SDK: Trace Cursor agent runs, streamed messages, and tool calls.
  • Vercel AI SDK: Trace generateText, streamText, ToolLoopAgent, tool calls, structured output, and usage.
  • Claude Agent SDK: Trace Claude Agent SDK query loops and yielded messages.
  • Claude Code SDK: Trace Claude Code CLI subprocess calls and SDK-style agent invocations.
  • Pydantic AI: Trace Pydantic AI agent runs and tool calls (Python only).
  • Manual spans: Wrap custom agents, CLI calls, provider routing, and unsupported SDKs.
  • Agent identity: Set stable agent IDs, display names, and roles for dashboard grouping.

​
What Gets Captured

Span dataExamples
Inputs and outputsinput.value, output.value
Messagesuser, system, assistant, tool, and tool-result messages
Tool callstool names, IDs, JSON arguments, and tool results
Model metadatallm.model_name, llm.invocation_parameters, gen_ai.system
Usageprompt, completion, total, and prompt-cache token counts when available
Agent structureagent spans, framework spans, tool spans, graph/node spans
Agent identityagent.id, agent.name, and agent.role for Agents dashboard grouping
Errorsexception status and error details on failed spans
All keys follow OpenInference conventions, so the same spans can be consumed by any OpenInference-aware viewer.

​
Configuration

Both SDKs read the same env vars and accept the same options on setup().

​
Env Vars And Identity

Env varPurpose
CATALYST_OTLP_ENDPOINTOTLP/HTTP trace ingest endpoint. Set to https://telemetry.inference.net for the hosted Catalyst backend.
CATALYST_OTLP_TOKENBearer token for ingest auth. Generate one at API Keys.
CATALYST_SERVICE_NAMEOTel service.name resource attribute. Use a stable value per deployed service.
CATALYST_SERVICE_VERSIONOTel service.version resource attribute.
CATALYST_DEBUGEnable SDK debug logging.
Precedence: explicit options on setup() win, then CATALYST_* env vars, then SDK defaults. Legacy OTLP_ENDPOINT, OTLP_INGEST_TOKEN, and SERVICE_NAME are still accepted but prefer CATALYST_* in new setups.

​
Instrumentation Control

Two options control which SDKs Catalyst patches at setup time.
OptionDefaultPurpose
modules (TypeScript){}Map of SDK modules to patch. Pass { openai: OpenAI, anthropic: Anthropic }. Entries listed here are always patched. Python auto-detects installed packages and does not need this option.
autoInstrumenttrueWhen true, setup() also scans node_modules and patches any other supported SDK it finds. When false, only the SDKs you list in modules: get patched, nothing else.
Three patterns, in order of how often they show up in real apps: Auto-detect everything. Call setup() with no modules:. The auto-scan finds every supported SDK and patches it. Easiest path for scripts, demos, and small apps. 
const tracing = await setup();
Explicit modules plus auto-detect. Pass the SDKs you use to modules:. Those entries are patched directly (no resolver gymnastics), and setup() also auto-detects anything else. 
const tracing = await setup({ modules: { openai: OpenAI } });
Explicit only. Pass modules: and set autoInstrument: false. Only the SDKs you list get patched, even if a transitive dep happens to pull in another supported one. Cleanest pattern for production services that know exactly which providers they call. 
const tracing = await setup({
  modules: { anthropic: Anthropic },
  autoInstrument: false,
});
Pass only the SDK modules you actually use to modules. Auto-instrumentation patches prototypes, so patching an SDK you do not call adds no runtime cost but a small import cost.

​
Imports (TypeScript)

For almost all setups, @inference/tracing is the only import you need. setup() auto-patches every supported SDK it can find (OpenAI, Anthropic, LangChain, LangGraph, ElevenLabs, LiveKit, PI AI, and the rest), so you do not need to reach for the per-integration helpers yourself.
ExportWhat it does
setupInitializes tracing. Pass the SDK modules you use; it patches them and returns a handle for shutdown.
agentSpanOpens a parent span that groups everything an agent run does. Use one per top-level invocation.
manualSpanOpens a custom span for work that is not an SDK call: tool execution, retrieval, validation, etc.
wrapClaudeAgentSdkQueryWraps a Claude Agent SDK query() call so each tool turn becomes a span.
Attr, SpanKindValuesOpenInference attribute keys and span-kind enums for setting custom attributes.
Error typesCatalystInstrumentationError and friends, for catching setup failures.

​
When you need a subpath

Two cases call for importing from a per-integration subpath:
  1. You’re using the Vercel AI SDK. The AI SDK has its own experimental_telemetry option, so there is nothing for setup() to patch. You import createAISdkTelemetrySettings and pass its result into each generateText / streamText call.
  2. You’re installing instrumentation on a custom tracer provider instead of the one setup() registers. Rare in practice; most apps want the standard flow. The subpath helpers (installOpenAIInstrumentation, etc.) accept an explicit provider argument.
If neither applies, you can skip the table below.
ImportExportsUse For
@inference/tracing/ai-sdkcreateAISdkTelemetrySettings, instrumentAISdk, installAISdkInstrumentationVercel AI SDK
@inference/tracing/openaiinstrumentOpenAI, installOpenAIInstrumentationOpenAI
@inference/tracing/anthropicinstrumentAnthropic, installAnthropicInstrumentationAnthropic
@inference/tracing/langchaininstrumentLangChain, installLangChainInstrumentationLangChain
@inference/tracing/langgraphinstrumentLangGraph, installLangChainInstrumentationLangGraph
@inference/tracing/langsmithinstallLangSmithLangSmith bridge
@inference/tracing/openai-agentsinstrumentOpenAIAgents@openai/agents
@inference/tracing/claude-agent-sdkwrapClaudeAgentSdkQueryClaude Agent SDK
@inference/tracing/elevenlabsinstrumentElevenLabs, installElevenLabsInstrumentationElevenLabs Agents
@inference/tracing/pi-aiinstrumentPiAi, installPiAiInstrumentationPI AI
@inference/tracing/cursor-sdkinstrumentCursorSdk, installCursorSdkInstrumentationCursor SDK
@inference/tracing/livekit-agentsinstrumentLiveKitAgents, installLiveKitAgentsInstrumentationLiveKit Agents
@inference/tracing/semconvOpenInferenceAttribute, MessageRole (plus Attr and SpanKindValues, also on the root)OpenInference constants
Python uses one import root (inference_catalyst_tracing) and gates each integration behind a pip extra (pip install 'inference-catalyst-tracing[openai,anthropic]').

​
The setup() Return Value

setup() returns a CatalystTracing handle:
FieldPurpose
tracerThe OpenTelemetry Tracer for authoring manual spans. In Python, pass it to agent_span / manual_span. In TypeScript, agentSpan / manualSpan resolve it automatically after setup(), so this is an escape hatch — use it for tracer.startActiveSpan(...) directly.
providerThe OTel tracer provider, in case you want to register additional span processors or exporters.
installResultsPer-integration install result: which SDKs were detected, which were patched, and any errors. Useful in CI and on startup logs.
shutdown()Flush and close tracing. Call this on SIGTERM for long-lived services, before process.exit(0) for scripts.
For long-lived and serverless processes, memoize the setup() promise and flush correctly for the process shape. See Flushing and process lifecycle.

​
How It Works

  1. setup() configures an OpenTelemetry tracer provider with a Catalyst OTLP/HTTP exporter.
  2. The SDK auto-patches any supported provider or framework SDK it finds. In TypeScript you pass them via modules. In Python it auto-detects installed packages.
  3. Your app runs normally. Spans are batched and exported in the background.
  4. Spans land in Catalyst grouped by service, trace, span, and task metadata. View them in the Traces tab or the Agents tab.
  5. Run analysis on them with Halo to understand your traces and gain insights you can pass into your favorite coding agent.

​
Inspect Traces from the CLI

After spans are flowing into Catalyst, you can inspect the same trace data from the terminal: 
# Browse recent traces
inf trace list --range 1h

# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline

# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io
See inf trace and inf span for the full command reference.

​
Next Steps

Quickstart

Install an SDK, configure export, and capture your first model span.

Production agent example

A production-shaped agent with custom tool spans, end to end.

Manual spans

Author AGENT, TOOL, CHAIN, and RETRIEVER spans.

Attributes reference

Every Attr.* constant and SpanKindValues value.

Handle API reference

Every method on the span handle yielded by agentSpan / manualSpan and the Python equivalents.

Instrumentation examples

Copyable examples for providers, agents, tool loops, and prompt caching.

Agent identity

Add stable IDs and readable names to agent spans.

Troubleshooting

Debug missing spans, missing attributes, and shutdown behavior.