> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inference.net/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces

> Capture the full execution of your AI apps and agents with first-party OpenInference-shaped tracing SDKs for TypeScript and Python.

Catalyst Tracing captures the full execution of your AI apps and agents: LLM calls, tool calls, framework steps, agent runs, and any custom spans you wrap. Spans are emitted as OpenInference-shaped OpenTelemetry, exported over OTLP/HTTP, and shown in Catalyst as trace trees, agent sessions, and per-span IO.

Use tracing for agent improvement. Once spans are flowing, point [Halo](https://github.com/context-labs/halo), our open-source agent-loop optimizer, at your traces and get back concrete suggestions for fixing prompts, tools, and harness logic. See [Analyze your traces](/get-started/analyze-traces) for the dashboard and Halo flow.

<Info>
  Tracing and the Catalyst Gateway are independent. Gateway captures one record per LLM request through a proxy. Tracing captures the full agent hierarchy from inside your code. Use them on their own or together. For gateway-only setups, see [model provider integrations](/integrations/overview).
</Info>

## How Tracing Fits Together

Tracing has three layers. You add them in order, and most apps stop at the
second.

**1. Automatic SDK spans.** Pass an SDK to `setup({ modules })` and every call
through it becomes its own span, with input messages, output, model name,
finish reason, and token counts. You change nothing at the call site. For a
one-shot script, or a batch of unrelated LLM calls, this is all you need.

**2. An agent span around the run.** When one request drives several model or
tool calls (an agent run, a conversation turn, a workflow), wrap it in
`agentSpan`. The agent span is the parent row the SDK spans nest under, so you
see "Support Agent" with "openai chat" and "lookup\_order" beneath it instead of
three orphan rows. It carries `agentId`, `agentName`, `role`, and `sessionId`
for grouping in the Agents dashboard, and holds the run's high-level input and
output. Without it the SDK spans are still captured, just as a flat list with no
agent or conversation to group them by.

**3. Manual spans for your own steps.** When a step inside the run is not an SDK
call (a tool you execute yourself, a retrieval, a router, a validation pass),
wrap it in `manualSpan` with the matching span kind. It nests under the agent
span next to the auto-captured LLM spans, so the trace tree reflects everything
the run did, not just the model calls.

| You're doing...                                                                                                                               | What you need                                                                                            |
| --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| One-shot call, or independent calls with no orchestration                                                                                     | `setup({ modules })`. Done.                                                                              |
| Several calls per request, one agent                                                                                                          | Add `agentSpan({ agentId, agentName, sessionId })` around the run                                        |
| Multi-agent system                                                                                                                            | A separate `agentSpan` per agent, each with its own `agentId` and `role`                                 |
| A non-SDK step inside the run (tool, retrieval, router, validator)                                                                            | `manualSpan({ spanKind: TOOL / CHAIN / RETRIEVER })` inside the agent span                               |
| Subprocess or CLI call                                                                                                                        | `agentSpan` around it (no SDK to auto-patch)                                                             |
| Framework with native capture (LangGraph, OpenAI Agents, Cursor SDK, Vercel Eve, Vercel AI SDK, LiveKit, ElevenLabs, Claude Agent SDK, PI AI) | Use the framework guide's setup path; add `agentSpan` when you need stable identity and session grouping |

<Tip>
  Set `sessionId`, `agentId`, and `agentName` once on the outer `agentSpan`.
  Every span underneath inherits them, so you should not be setting these in
  multiple places.
</Tip>

## Packages

Two packages, one shape. Install whichever matches your runtime.

| Runtime                 | Install                                                                              | Import                                         | Source |
| ----------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------- | ------ |
| TypeScript / Node / Bun | [`@inference/tracing`](https://www.npmjs.com/package/@inference/tracing)             | `import { setup } from "@inference/tracing"`   | npm    |
| Python (3.10+)          | [`inference-catalyst-tracing`](https://pypi.org/project/inference-catalyst-tracing/) | `from inference_catalyst_tracing import setup` | PyPI   |

Both packages share the same `setup()` entry point, the same env-var contract, and the same OpenInference attribute conventions. Pick whichever fits your stack and write the same code shape in either language.

<Tip>
  Provider SDKs (`openai`, `@anthropic-ai/sdk`, `langchain`, etc.) are optional peer dependencies. Install only the ones you use. For Python, install per-integration extras: `pip install 'inference-catalyst-tracing[openai,langchain]'`.
</Tip>

## Getting Started

The fastest path is the [Inference CLI](/cli/overview):

<Metadata text="integrations/traces/overview-inf-instrument" />

```bash theme={"system"}
inf instrument --mode tracing
```

`tracing` mode installs the SDK and wires up `setup()`. `both` mode does the same plus gateway routing in one pass. The command launches your choice of coding agent (Claude Code, OpenCode, or Codex) to make the edits. See [Install with AI](/integrations/install-with-ai) for the full flow.

Prefer to wire it up by hand? Start with the path that matches what you want to instrument:

1. [Quickstart](/integrations/traces/quickstart) installs the SDK, configures export, and captures your first provider span. Both `Install with AI` and `Install manually` paths included.
2. [Provider and framework guides](#supported-trace-integrations) cover OpenAI, Anthropic, LangChain, LangGraph, LangSmith, OpenAI Agents, LiveKit Agents, Claude Agent SDK, Claude Code, Pydantic AI, ElevenLabs Agents, Vercel Eve, PI AI, Cursor SDK, the Vercel AI SDK, and other supported surfaces.
3. [Manual spans](/integrations/traces/manual-spans) wrap custom agents, subprocesses, tools, retrieval, routing, and unsupported SDKs.
4. [Agent identity](/integrations/traces/agent-identity) sets stable `agent.id` values so the Agents dashboard groups executions correctly.

## Supported Trace Integrations

* [OpenAI](/integrations/traces/openai): Trace Chat Completions, tool calls, structured outputs, and Responses API calls.
* [Anthropic](/integrations/traces/anthropic): Trace Messages API calls, tool use round trips, and prompt-cache usage.
* [LangChain](/integrations/traces/langchain): Capture chains, agents, model calls, and tools through callback instrumentation.
* [LangGraph](/integrations/traces/langgraph): Capture graph and node spans while preserving parent-child relationships.
* [LangSmith](/integrations/traces/langsmith): Bridge LangSmith OpenTelemetry spans into Catalyst.
* [OpenAI Agents](/integrations/traces/openai-agents): Trace agent runs, tools, handoffs, and nested OpenAI model calls.
* [LiveKit Agents](/integrations/traces/livekit-agents): Trace LiveKit sessions, model nodes, and function tools through native OTel spans.
* [ElevenLabs Agents](/integrations/traces/elevenlabs): Trace conversation sessions, transcripts, and client tool calls.
* [Vercel Eve](/integrations/traces/eve): Trace Eve agent turns, model calls, sub-agent invocations, tools, and `$eve.*` workflow tags.
* [PI AI](/integrations/traces/pi-ai): Trace PI AI model turns, streams, tool calls, usage, costs, and agent identity.
* [Cursor SDK](/integrations/traces/cursor-sdk): Trace Cursor agent runs, streamed messages, and tool calls.
* [Vercel AI SDK](/integrations/traces/ai-sdk): Trace generateText, streamText, ToolLoopAgent, tool calls, structured output, and usage.
* [Claude Agent SDK](/integrations/traces/claude-agent-sdk): Trace Claude Agent SDK query loops and yielded messages.
* [Claude Code SDK](/integrations/traces/claude-code-sdk): Trace Claude Code CLI subprocess calls and SDK-style agent invocations.
* [OpenCode](/integrations/traces/opencode): Record your own OpenCode coding sessions with LLM calls, tool calls, token usage, cost, and errors.
* [Pydantic AI](/integrations/traces/pydantic-ai): Trace Pydantic AI agent runs and tool calls (Python only).
* [Manual spans](/integrations/traces/manual-spans): Wrap custom agents, CLI calls, provider routing, and unsupported SDKs.
* [Agent identity](/integrations/traces/agent-identity): Set stable agent IDs, display names, and roles for dashboard grouping.

## What Gets Captured

| Span data          | Examples                                                                 |
| ------------------ | ------------------------------------------------------------------------ |
| Inputs and outputs | `input.value`, `output.value`                                            |
| Messages           | user, system, assistant, tool, and tool-result messages                  |
| Tool calls         | tool names, IDs, JSON arguments, and tool results                        |
| Model metadata     | `llm.model_name`, `llm.invocation_parameters`, `gen_ai.system`           |
| Usage              | prompt, completion, total, and prompt-cache token counts when available  |
| Agent structure    | agent spans, framework spans, tool spans, graph/node spans               |
| Agent identity     | `agent.id`, `agent.name`, and `agent.role` for Agents dashboard grouping |
| Errors             | exception status and error details on failed spans                       |

All keys follow [OpenInference](https://github.com/Arize-ai/openinference) conventions, so the same spans can be consumed by any OpenInference-aware viewer.

## Configuration

Both SDKs read the same env vars and accept the same options on `setup()`.

### Env Vars And Identity

| Env var                    | Purpose                                                                                                    |
| -------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `CATALYST_OTLP_ENDPOINT`   | OTLP/HTTP trace ingest endpoint. Set to `https://telemetry.inference.net` for the hosted Catalyst backend. |
| `CATALYST_OTLP_TOKEN`      | Bearer token for ingest auth. Generate one at [API Keys](https://inference.net/dashboard/api-keys).        |
| `CATALYST_SERVICE_NAME`    | OTel `service.name` resource attribute. Use a stable value per deployed service.                           |
| `CATALYST_SERVICE_VERSION` | OTel `service.version` resource attribute.                                                                 |
| `CATALYST_DEBUG`           | Enable SDK debug logging.                                                                                  |

Precedence: explicit options on `setup()` win, then `CATALYST_*` env vars, then SDK defaults. Legacy `OTLP_ENDPOINT`, `OTLP_INGEST_TOKEN`, and `SERVICE_NAME` are still accepted but prefer `CATALYST_*` in new setups.

### Instrumentation Control

Two options control which SDKs Catalyst patches at setup time.

| Option                 | Default | Purpose                                                                                                                                                                                     |
| ---------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `modules` (TypeScript) | `{}`    | Map of SDK modules to patch. Pass `{ openai: OpenAI, anthropic: Anthropic }`. Entries listed here are always patched. Python auto-detects installed packages and does not need this option. |
| `autoInstrument`       | `true`  | When `true`, `setup()` also scans `node_modules` and patches any other supported SDK it finds. When `false`, only the SDKs you list in `modules:` get patched, nothing else.                |

Three patterns, in order of how often they show up in real apps:

**Auto-detect everything.** Call `setup()` with no `modules:`. The auto-scan finds every supported SDK and patches it. Easiest path for scripts, demos, and small apps.

```ts theme={"system"}
const tracing = await setup();
```

**Explicit modules plus auto-detect.** Pass the SDKs you use to `modules:`. Those entries are patched directly (no resolver gymnastics), and `setup()` also auto-detects anything else.

```ts theme={"system"}
const tracing = await setup({ modules: { openai: OpenAI } });
```

**Explicit only.** Pass `modules:` and set `autoInstrument: false`. Only the SDKs you list get patched, even if a transitive dep happens to pull in another supported one. Cleanest pattern for production services that know exactly which providers they call.

```ts theme={"system"}
const tracing = await setup({
  modules: { anthropic: Anthropic },
  autoInstrument: false,
});
```

<Tip>
  Pass only the SDK modules you actually use to `modules`. Auto-instrumentation
  patches prototypes, so patching an SDK you do not call adds no runtime cost
  but a small import cost.
</Tip>

### Imports (TypeScript)

For almost all setups, `@inference/tracing` is the only import you need.
`setup()` auto-patches every supported SDK it can find (OpenAI, Anthropic,
LangChain, LangGraph, ElevenLabs, LiveKit, PI AI, and the rest), so you do not need
to reach for the per-integration helpers yourself.

| Export                    | What it does                                                                                          |
| ------------------------- | ----------------------------------------------------------------------------------------------------- |
| `setup`                   | Initializes tracing. Pass the SDK modules you use; it patches them and returns a handle for shutdown. |
| `agentSpan`               | Opens a parent span that groups everything an agent run does. Use one per top-level invocation.       |
| `manualSpan`              | Opens a custom span for work that is not an SDK call: tool execution, retrieval, validation, etc.     |
| `wrapClaudeAgentSdkQuery` | Wraps a Claude Agent SDK `query()` call so each tool turn becomes a span.                             |
| `Attr`, `SpanKindValues`  | OpenInference attribute keys and span-kind enums for setting custom attributes.                       |
| Error types               | `CatalystInstrumentationError` and friends, for catching setup failures.                              |

#### When you need a subpath

Three cases call for importing from a per-integration subpath:

1. **You're using the Vercel AI SDK.** The AI SDK has its own
   `experimental_telemetry` option, so there is nothing for `setup()` to
   patch. You import `createAISdkTelemetrySettings` and pass its result into
   each `generateText` / `streamText` call.
2. **You're using Vercel Eve.** Eve enables telemetry by loading
   `agent/instrumentation.ts`, so you export `defineCatalystEveInstrumentation()`
   from `@inference/tracing/eve` in that file.
3. **You're installing instrumentation on a custom tracer provider** instead of the one `setup()` registers. Rare in practice; most apps want the standard flow. The subpath helpers (`installOpenAIInstrumentation`, etc.) accept an explicit provider argument.

If neither applies, you can skip the table below.

| Import                                | Exports                                                                                      | Use For                 |
| ------------------------------------- | -------------------------------------------------------------------------------------------- | ----------------------- |
| `@inference/tracing/ai-sdk`           | `createAISdkTelemetrySettings`, `instrumentAISdk`, `installAISdkInstrumentation`             | Vercel AI SDK           |
| `@inference/tracing/openai`           | `instrumentOpenAI`, `installOpenAIInstrumentation`                                           | OpenAI                  |
| `@inference/tracing/anthropic`        | `instrumentAnthropic`, `installAnthropicInstrumentation`                                     | Anthropic               |
| `@inference/tracing/langchain`        | `instrumentLangChain`, `installLangChainInstrumentation`                                     | LangChain               |
| `@inference/tracing/langgraph`        | `instrumentLangGraph`, `installLangChainInstrumentation`                                     | LangGraph               |
| `@inference/tracing/langsmith`        | `installLangSmith`                                                                           | LangSmith bridge        |
| `@inference/tracing/openai-agents`    | `instrumentOpenAIAgents`                                                                     | `@openai/agents`        |
| `@inference/tracing/claude-agent-sdk` | `wrapClaudeAgentSdkQuery`                                                                    | Claude Agent SDK        |
| `@inference/tracing/elevenlabs`       | `instrumentElevenLabs`, `installElevenLabsInstrumentation`                                   | ElevenLabs Agents       |
| `@inference/tracing/eve`              | `defineCatalystEveInstrumentation`, `installEveSpanEnrichment`                               | Vercel Eve              |
| `@inference/tracing/pi-ai`            | `instrumentPiAi`, `installPiAiInstrumentation`                                               | PI AI                   |
| `@inference/tracing/cursor-sdk`       | `instrumentCursorSdk`, `installCursorSdkInstrumentation`                                     | Cursor SDK              |
| `@inference/tracing/livekit-agents`   | `instrumentLiveKitAgents`, `installLiveKitAgentsInstrumentation`                             | LiveKit Agents          |
| `@inference/tracing/semconv`          | `OpenInferenceAttribute`, `MessageRole` (plus `Attr` and `SpanKindValues`, also on the root) | OpenInference constants |

Python uses one import root (`inference_catalyst_tracing`) and gates each
integration behind a pip extra (`pip install 'inference-catalyst-tracing[openai,anthropic]'`).

### The `setup()` Return Value

`setup()` returns a `CatalystTracing` handle:

| Field            | Purpose                                                                                                                                                                                                                                                                        |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `tracer`         | The OpenTelemetry `Tracer` for authoring manual spans. In Python, pass it to `agent_span` / `manual_span`. In TypeScript, `agentSpan` / `manualSpan` resolve it automatically after `setup()`, so this is an escape hatch — use it for `tracer.startActiveSpan(...)` directly. |
| `provider`       | The OTel tracer provider, in case you want to register additional span processors or exporters.                                                                                                                                                                                |
| `installResults` | Per-integration install result: which SDKs were detected, which were patched, and any errors. Useful in CI and on startup logs.                                                                                                                                                |
| `shutdown()`     | Flush and close tracing. Call this on `SIGTERM` for long-lived services, before `process.exit(0)` for scripts.                                                                                                                                                                 |

For long-lived and serverless processes, memoize the `setup()` promise and
flush correctly for the process shape. See
[Flushing and process lifecycle](/integrations/traces/quickstart#flushing-and-process-lifecycle).

## How It Works

1. `setup()` configures an OpenTelemetry tracer provider with a Catalyst OTLP/HTTP exporter.
2. The SDK auto-patches any supported provider or framework SDK it finds. In TypeScript you pass them via `modules`. In Python it auto-detects installed packages.
3. Your app runs normally. Spans are batched and exported in the background.
4. Spans land in Catalyst grouped by service, trace, span, and task metadata. View them in the [Traces tab or the Agents tab](/get-started/analyze-traces).
5. Run analysis on them with [Halo](https://github.com/context-labs/halo) to understand your traces and gain insights you can pass into your favorite coding agent.

## Inspect Traces from the CLI

After spans are flowing into Catalyst, you can inspect the same trace data from the terminal:

<Metadata text="integrations/traces/overview-cli-inspect" />

```bash theme={"system"}
# Browse recent traces
inf trace list --range 1h

# Open a trace tree or timeline
inf trace get <trace-id> --view tree
inf trace get <trace-id> --view timeline

# Search spans and inspect captured IO
inf span list --trace-id <trace-id> --kind LLM
inf span get <trace-id> <span-id> --view io
```

See [`inf trace`](/cli/traces) and [`inf span`](/cli/spans) for the full command reference.

## Next Steps

<CardGroup cols={2}>
  <Card title="Quickstart" icon="rocket" href="/integrations/traces/quickstart">
    Install an SDK, configure export, and capture your first model span.
  </Card>

  <Card title="Production agent example" icon="kitchen-set" href="/integrations/traces/production-agent-example">
    A production-shaped agent with custom tool spans, end to end.
  </Card>

  <Card title="Manual spans" icon="pen-nib" href="/integrations/traces/manual-spans">
    Author AGENT, TOOL, CHAIN, and RETRIEVER spans.
  </Card>

  <Card title="Attributes reference" icon="tags" href="/integrations/traces/attributes">
    Every `Attr.*` constant and `SpanKindValues` value.
  </Card>

  <Card title="Handle API reference" icon="book" href="/integrations/traces/handle-api">
    Every method on the span handle yielded by `agentSpan` / `manualSpan` and the Python equivalents.
  </Card>

  <Card title="Instrumentation examples" icon="code" href="/integrations/traces/examples">
    Copyable examples for providers, agents, tool loops, and prompt caching.
  </Card>

  <Card title="Agent identity" icon="fingerprint" href="/integrations/traces/agent-identity">
    Add stable IDs and readable names to agent spans.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/integrations/traces/troubleshooting">
    Debug missing spans, missing attributes, and shutdown behavior.
  </Card>
</CardGroup>