> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inference.net/llms.txt
> Use this file to discover all available pages before exploring further.
# Manual Spans
> Author OpenInference-shaped spans for tools, retrievers, custom routers, CLI subprocesses, and anything else the SDK does not patch automatically.
Catalyst's per-SDK instrumentation captures provider calls automatically: every
SDK you pass to `setup({ modules })` emits an LLM span on its own. But a real
agent does work the SDK never sees, like running tools, retrieving documents,
routing requests, validating output, or shelling out to a CLI. Manual spans are
how you put that work into the same trace tree, next to the auto-captured LLM
spans.
There are two shapes you author by hand, and everything on this page is one of
them:
* **The agent span** wraps a whole run. It is the parent row everything else
nests under, and it carries the agent identity and session. You author it with
`agentSpan()` / `agent_span()`. This is the same wrapper the integration
guides put around a run.
* **Child spans** sit inside that run, one per non-LLM step: a tool call, a
retrieval, a routing step. You author them with `manualSpan()` /
`manual_span()`, choosing the span kind that matches the step.
If you have not yet wired up `setup()`, start with the
[Traces Quickstart](/integrations/traces/quickstart).
## Span Kinds You Author Manually
| Kind | Use it for |
| ----------- | ---------------------------------------------------------------------------------------------------------- |
| `AGENT` | The outer span around an agent run. Carries `agent.id`, `agent.name`, optional `session.id` and `user.id`. |
| `TOOL` | One invocation of a tool function by an agent. Carries `tool.name`, optional `tool_call.id`. |
| `CHAIN` | A chain step that is not itself an LLM call (routing logic, prompt assembly, post-processing). |
| `RETRIEVER` | A vector-search or document-lookup call. |
| `EMBEDDING` | An embedding-model call your code makes directly. |
| `LLM` | A model call made outside an instrumented SDK (rare, prefer the SDK patches). |
These six are the whole set. Under the hood a span is an ordinary
OpenTelemetry span, so you can attach any attribute you want to it (see
[Adding Custom Attributes](#adding-custom-attributes)). The *kind* is different:
it is a fixed vocabulary from
[OpenInference](https://github.com/Arize-ai/openinference) that tells the
dashboard how to render the span. A `TOOL` span gets a tool name and an
arguments/result panel, a `RETRIEVER` span gets a query and a document list, an
`LLM` span gets messages, token counts, and cost. Pick the kind that matches how
you want the step to appear; when none of the specific kinds fit, use `CHAIN`,
the generic "this is a step" kind. The kind strings come from `SpanKindValues`,
and the [Attributes reference](/integrations/traces/attributes) lists which
attributes each kind expects.
## Agent Spans
Use `agentSpan()` (TypeScript) or `agent_span()` (Python) as the outermost
wrapper around an agent run. Pass a stable `agentId` so the Catalyst Agents
dashboard groups executions correctly across renames and deploys. Pass
`sessionId` for one conversation, not as process-wide setup. Pass `userId` to
record the end user the run acts on behalf of so you can filter traces by user
on the dashboard. The wrapper also accepts free-form `metadata` and `tags`.
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";
const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();
await agentSpan(
{
agentId: "refund-review-agent",
agentName: "Refund Review Agent",
spanName: "refund-review.run",
sessionId: "conversation-ticket-123",
userId: "user_8675309",
role: "refunds",
system: "openai",
},
async (span) => {
const ticket = { id: "ticket_123", orderId: "ABC-123" };
span.setInput(ticket);
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "user", content: `Review refund for ${ticket.orderId}` },
],
});
span.setOutput({ decision: response.choices[0]?.message.content });
if (response.usage != null) {
span.recordTokens({
prompt: response.usage.prompt_tokens ?? 0,
completion: response.usage.completion_tokens ?? 0,
});
}
},
);
await tracing.shutdown();
```
```python Python theme={"system"}
import os
from inference_catalyst_tracing import agent_span, setup
from openai import OpenAI
tracing = setup()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
with agent_span(
tracing.tracer,
agent_id="refund-review-agent",
agent_name="Refund Review Agent",
span_name="refund-review.run",
session_id="conversation-ticket-123",
user_id="user_8675309",
agent_role="refunds",
system="openai",
) as span:
ticket = {"id": "ticket_123", "order_id": "ABC-123"}
span.set_input(ticket)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": f"Review refund for {ticket['order_id']}"},
],
)
span.set_output({"decision": response.choices[0].message.content})
if response.usage is not None:
span.record_usage(response.usage)
tracing.shutdown()
```
The OpenAI call inside the callback runs in the agent span's active OTel context,
so the auto-emitted LLM span automatically parents under the AGENT span. That
is how a full trace tree forms with no extra plumbing.
See the [Handle API reference](/integrations/traces/handle-api) for the full
set of methods on the `span` argument.
## Tool, Chain, And Retriever Spans
When the work inside an agent loop is a tool call, a retrieval step, or a chain
step, wrap it in a child span with the right `SpanKind`. The child span
automatically parents under the active AGENT span and inherits its `agent.id`.
`manualSpan()` / `manual_span()` is the general-purpose helper. Pass
`spanKind` / `span_kind`, optional typed fields (`toolName` / `tool_name`,
`toolCallId` / `tool_call_id`, `model`, `usage`, `input`, `output`), and any
extra `attributes`. It defaults to `CHAIN`; pass `TOOL`, `RETRIEVER`, or
`EMBEDDING` when authoring the matching shape.
### Tool spans
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
async function executeTool(
name: string,
args: Record,
callId: string,
) {
return await manualSpan(
{
spanName: `${name}.tool`,
spanKind: SpanKindValues.TOOL,
toolName: name,
toolCallId: callId,
input: args,
},
async (span) => {
const result = await TOOLS[name](args);
span.setOutput(result);
return result;
},
);
}
```
```python Python theme={"system"}
from inference_catalyst_tracing import SpanKindValues, manual_span
def execute_tool(name: str, args: dict, call_id: str) -> dict:
with manual_span(
tracing.tracer,
name=f"{name}.tool",
span_kind=SpanKindValues.TOOL,
tool_name=name,
tool_call_id=call_id,
input=args,
) as span:
result = TOOLS[name](**args)
span.set_output(result)
return result
```
### Chain spans
Use `CHAIN` for routing logic, prompt assembly, post-processing, or any other
non-LLM step that benefits from its own span. `metadata` and `tags` are
free-form attributes that show up under the span's attributes view in the
dashboard.
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
await manualSpan(
{
spanName: "provider_router/select_model",
spanKind: SpanKindValues.CHAIN,
input: { route: "support", priority: "high" },
metadata: { customer_tier: "enterprise" },
tags: ["router", "support"],
},
async (span) => {
const selected = chooseModelForRequest();
span.setOutput({ model: selected.model, reason: selected.reason });
},
);
```
```python Python theme={"system"}
from inference_catalyst_tracing import SpanKindValues, manual_span
with manual_span(
tracing.tracer,
name="provider_router/select_model",
span_kind=SpanKindValues.CHAIN,
input={"route": "support", "priority": "high"},
metadata={"customer_tier": "enterprise"},
tags=["router", "support"],
) as span:
selected = choose_model_for_request()
span.set_output({"model": selected.model, "reason": selected.reason})
```
### Retriever spans
Use `RETRIEVER` for vector-store or document-lookup calls. Record the query as
input and the result list as output.
```typescript TypeScript theme={"system"}
import { manualSpan, SpanKindValues } from "@inference/tracing";
await manualSpan(
{
spanName: "vector_store.search",
spanKind: SpanKindValues.RETRIEVER,
input: { query, k: 8 },
},
async (span) => {
const docs = await vectorStore.search(query, { k: 8 });
span.setOutput(docs.map((d) => ({ id: d.id, score: d.score })));
},
);
```
```python Python theme={"system"}
from inference_catalyst_tracing import SpanKindValues, manual_span
with manual_span(
tracing.tracer,
name="vector_store.search",
span_kind=SpanKindValues.RETRIEVER,
input={"query": query, "k": 8},
) as span:
docs = vector_store.search(query, k=8)
span.set_output([{"id": d.id, "score": d.score} for d in docs])
```
### Escape Hatch
`manualSpan` is the canonical path for every non-AGENT manual span. The only
reason to reach below it is `tracing.tracer.startActiveSpan()` directly, when
you need behavior the helper doesn't expose, like mid-callback span events paired
with custom status transitions, or a span lifetime that doesn't match a single
callback. With raw `startActiveSpan` you own status, exception recording, and
`span.end()` yourself.
For a full example that ties an AGENT span, several TOOL spans, and the
patched provider LLM spans together, see the
[Production Agent Example](/integrations/traces/production-agent-example).
## Adding Custom Attributes
The typed handle covers the OpenInference vocabulary. For domain-specific
attributes (tenant ID, request channel, deploy environment), use
`setAttribute` / `set_attribute` on the handle. The helper normalizes values
into OTel-safe types (primitives and primitive arrays pass through; mixed
arrays and plain objects are JSON-stringified).
```typescript TypeScript theme={"system"}
await agentSpan(
{ agentId: "support-agent", spanName: "support-agent.run" },
async (span) => {
span.setAttribute("app.tenant_id", tenantId);
span.setAttributes({
"app.request_channel": "slack",
"app.deploy_env": process.env.DEPLOY_ENV ?? "dev",
});
// ...
},
);
```
```python Python theme={"system"}
with agent_span(
tracing.tracer,
agent_id="support-agent",
span_name="support-agent.run",
) as span:
span.set_attribute("app.tenant_id", tenant_id)
span.set_attributes({
"app.request_channel": "slack",
"app.deploy_env": os.environ.get("DEPLOY_ENV", "dev"),
})
# ...
```
For anything outside what the handle provides (span events, mid-callback
exception recording, span-context propagation to background jobs), the raw
OTel span is available as `span.raw` (TypeScript) or `span.span` (Python).
Custom attributes appear under the span's attributes view in the dashboard and
in `inf span get --view attributes`. They are also
filterable from the CLI with `--metadata "app.tenant_id=acme"`.
## How Identity And Context Propagate
The agent span runs its callback inside an active OTel context that carries:
* The OTel `Span` itself, so child spans created by `tracer.startActiveSpan` or
by patched provider SDKs auto-parent under it.
* A Catalyst-specific agent identity record (`agent.id`, `agent.name`,
`agent.role`), which the per-SDK patchers copy onto LLM child spans so the
Agents dashboard groups model calls under the right agent.
You do not need to pass `agentId` to every child span. If you create a tool
span inside an agent span, the agent identity flows through automatically.
```
AGENT support-agent.run agent.id=support-agent, session.id=conv-42
├── LLM OpenAI chat (auto-emitted, inherits agent.id)
├── TOOL lookup_order.tool tool.name=lookup_order
│ └── LLM OpenAI chat (nested call from inside the tool)
├── CHAIN prompt_builder.run
└── TOOL issue_refund.tool tool.name=issue_refund
```
If you need to inspect or copy the active identity (for example, to tag a
background job span with the same `agent.id`), import the helpers:
```typescript TypeScript theme={"system"}
import { getActiveAgentIdentity } from "@inference/tracing";
const identity = getActiveAgentIdentity();
if (identity?.id) {
jobQueue.enqueue({ agentId: identity.id, payload });
}
```
## CLI And Subprocess Wrappers
When the work is a shell-out to a CLI (Claude Code, Codex, a custom binary),
wrap the process call in an agent span and set the input and output yourself.
```typescript TypeScript theme={"system"}
import { agentSpan, setup } from "@inference/tracing";
import { execFile } from "node:child_process";
import { promisify } from "node:util";
const execFileP = promisify(execFile);
const tracing = await setup();
const prompt = "Reply with just the word hello.";
await agentSpan(
{
agentId: "claude-code-prod",
agentName: "Claude Code",
spanName: "claude-code.invocation",
sessionId: "conversation-cli-hello",
system: "anthropic",
},
async (span) => {
span.setInput(prompt);
const { stdout } = await execFileP("claude", ["-p", prompt], {
encoding: "utf-8",
timeout: 60_000,
});
span.setOutput(stdout.trim());
},
);
await tracing.shutdown();
```
```python Python theme={"system"}
import subprocess
from inference_catalyst_tracing import agent_span, setup
tracing = setup()
prompt = "Reply with just the word hello."
with agent_span(
tracing.tracer,
agent_id="claude-code-prod",
agent_name="Claude Code",
span_name="claude-code.invocation",
session_id="conversation-cli-hello",
system="anthropic",
) as span:
span.set_input(prompt)
completed = subprocess.run(
["claude", "-p", prompt],
capture_output=True,
text=True,
timeout=60,
check=True,
)
span.set_output(completed.stdout.strip())
tracing.shutdown()
```
## Status Semantics
All Catalyst manual-span helpers follow the same status rules:
* Callback returns normally → span ends with `OK`.
* Callback throws → the exception is recorded as a span event, the span ends
with `ERROR` and the error message, and the original exception re-throws so
caller error handling is unaffected.
If you reach for `tracer.startActiveSpan()` directly (the OTel escape hatch
mentioned above), you own status and `span.end()` yourself, which is the
price of the lower-level API.
## Next Steps
Every method on the AGENT / TOOL / CHAIN span handle, with value-coercion rules.
All `Attr.*` constants and `SpanKindValues` with the attributes each kind expects.
A production-shaped agent with custom tool execution, end to end.
Pick stable `agent.id` and `session.id` values for the Agents dashboard.