Manual Spans - Inference.net Documentation

Catalyst’s per-SDK instrumentation captures provider calls automatically: every SDK you pass to setup({ modules }) emits an LLM span on its own. But a real agent does work the SDK never sees, like running tools, retrieving documents, routing requests, validating output, or shelling out to a CLI. Manual spans are how you put that work into the same trace tree, next to the auto-captured LLM spans. There are two shapes you author by hand, and everything on this page is one of them:

The agent span wraps a whole run. It is the parent row everything else nests under, and it carries the agent identity and session. You author it with agentSpan() / agent_span(). This is the same wrapper the integration guides put around a run.
Child spans sit inside that run, one per non-LLM step: a tool call, a retrieval, a routing step. You author them with manualSpan() / manual_span(), choosing the span kind that matches the step.

If you have not yet wired up setup(), start with the Traces Quickstart.

Span Kinds You Author Manually

Kind	Use it for
`AGENT`	The outer span around an agent run. Carries `agent.id`, `agent.name`, optional `session.id` and `user.id`.
`TOOL`	One invocation of a tool function by an agent. Carries `tool.name`, optional `tool_call.id`.
`CHAIN`	A chain step that is not itself an LLM call (routing logic, prompt assembly, post-processing).
`RETRIEVER`	A vector-search or document-lookup call.
`EMBEDDING`	An embedding-model call your code makes directly.
`LLM`	A model call made outside an instrumented SDK (rare, prefer the SDK patches).

These six are the whole set. Under the hood a span is an ordinary OpenTelemetry span, so you can attach any attribute you want to it (see Adding Custom Attributes). The kind is different: it is a fixed vocabulary from OpenInference that tells the dashboard how to render the span. A TOOL span gets a tool name and an arguments/result panel, a RETRIEVER span gets a query and a document list, an LLM span gets messages, token counts, and cost. Pick the kind that matches how you want the step to appear; when none of the specific kinds fit, use CHAIN, the generic “this is a step” kind. The kind strings come from SpanKindValues, and the Attributes reference lists which attributes each kind expects.

Agent Spans

Use agentSpan() (TypeScript) or agent_span() (Python) as the outermost wrapper around an agent run. Pass a stable agentId so the Catalyst Agents dashboard groups executions correctly across renames and deploys. Pass sessionId for one conversation, not as process-wide setup. Pass userId to record the end user the run acts on behalf of so you can filter traces by user on the dashboard. The wrapper also accepts free-form metadata and tags.

import { agentSpan, setup } from "@inference/tracing";
import OpenAI from "openai";

const tracing = await setup({ modules: { openai: OpenAI } });
const client = new OpenAI();

await agentSpan(
  {
    agentId: "refund-review-agent",
    agentName: "Refund Review Agent",
    spanName: "refund-review.run",
    sessionId: "conversation-ticket-123",
    userId: "user_8675309",
    role: "refunds",
    system: "openai",
  },
  async (span) => {
    const ticket = { id: "ticket_123", orderId: "ABC-123" };
    span.setInput(ticket);

    const response = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "user", content: `Review refund for ${ticket.orderId}` },
      ],
    });

    span.setOutput({ decision: response.choices[0]?.message.content });
    if (response.usage != null) {
      span.recordTokens({
        prompt: response.usage.prompt_tokens ?? 0,
        completion: response.usage.completion_tokens ?? 0,
      });
    }
  },
);

await tracing.shutdown();

The OpenAI call inside the callback runs in the agent span’s active OTel context, so the auto-emitted LLM span automatically parents under the AGENT span. That is how a full trace tree forms with no extra plumbing. See the Handle API reference for the full set of methods on the span argument.

Tool, Chain, And Retriever Spans

When the work inside an agent loop is a tool call, a retrieval step, or a chain step, wrap it in a child span with the right SpanKind. The child span automatically parents under the active AGENT span and inherits its agent.id. manualSpan() / manual_span() is the general-purpose helper. Pass spanKind / span_kind, optional typed fields (toolName / tool_name, toolCallId / tool_call_id, model, usage, input, output), and any extra attributes. It defaults to CHAIN; pass TOOL, RETRIEVER, or EMBEDDING when authoring the matching shape.

Tool spans

import { manualSpan, SpanKindValues } from "@inference/tracing";

async function executeTool(
  name: string,
  args: Record<string, unknown>,
  callId: string,
) {
  return await manualSpan(
    {
      spanName: `${name}.tool`,
      spanKind: SpanKindValues.TOOL,
      toolName: name,
      toolCallId: callId,
      input: args,
    },
    async (span) => {
      const result = await TOOLS[name](args);
      span.setOutput(result);
      return result;
    },
  );
}

Chain spans

Use CHAIN for routing logic, prompt assembly, post-processing, or any other non-LLM step that benefits from its own span. metadata and tags are free-form attributes that show up under the span’s attributes view in the dashboard.

import { manualSpan, SpanKindValues } from "@inference/tracing";

await manualSpan(
  {
    spanName: "provider_router/select_model",
    spanKind: SpanKindValues.CHAIN,
    input: { route: "support", priority: "high" },
    metadata: { customer_tier: "enterprise" },
    tags: ["router", "support"],
  },
  async (span) => {
    const selected = chooseModelForRequest();
    span.setOutput({ model: selected.model, reason: selected.reason });
  },
);

Retriever spans

Use RETRIEVER for vector-store or document-lookup calls. Record the query as input and the result list as output.

import { manualSpan, SpanKindValues } from "@inference/tracing";

await manualSpan(
  {
    spanName: "vector_store.search",
    spanKind: SpanKindValues.RETRIEVER,
    input: { query, k: 8 },
  },
  async (span) => {
    const docs = await vectorStore.search(query, { k: 8 });
    span.setOutput(docs.map((d) => ({ id: d.id, score: d.score })));
  },
);

Escape Hatch

manualSpan is the canonical path for every non-AGENT manual span. The only reason to reach below it is tracing.tracer.startActiveSpan() directly, when you need behavior the helper doesn’t expose, like mid-callback span events paired with custom status transitions, or a span lifetime that doesn’t match a single callback. With raw startActiveSpan you own status, exception recording, and span.end() yourself. For a full example that ties an AGENT span, several TOOL spans, and the patched provider LLM spans together, see the Production Agent Example.

Adding Custom Attributes

The typed handle covers the OpenInference vocabulary. For domain-specific attributes (tenant ID, request channel, deploy environment), use setAttribute / set_attribute on the handle. The helper normalizes values into OTel-safe types (primitives and primitive arrays pass through; mixed arrays and plain objects are JSON-stringified).

await agentSpan(
  { agentId: "support-agent", spanName: "support-agent.run" },
  async (span) => {
    span.setAttribute("app.tenant_id", tenantId);
    span.setAttributes({
      "app.request_channel": "slack",
      "app.deploy_env": process.env.DEPLOY_ENV ?? "dev",
    });
    // ...
  },
);

For anything outside what the handle provides (span events, mid-callback exception recording, span-context propagation to background jobs), the raw OTel span is available as span.raw (TypeScript) or span.span (Python). Custom attributes appear under the span’s attributes view in the dashboard and in inf span get <trace-id> <span-id> --view attributes. They are also filterable from the CLI with --metadata "app.tenant_id=acme".

How Identity And Context Propagate

The agent span runs its callback inside an active OTel context that carries:

The OTel Span itself, so child spans created by tracer.startActiveSpan or by patched provider SDKs auto-parent under it.
A Catalyst-specific agent identity record (agent.id, agent.name, agent.role), which the per-SDK patchers copy onto LLM child spans so the Agents dashboard groups model calls under the right agent.

You do not need to pass agentId to every child span. If you create a tool span inside an agent span, the agent identity flows through automatically.

AGENT  support-agent.run         agent.id=support-agent, session.id=conv-42
├── LLM    OpenAI chat            (auto-emitted, inherits agent.id)
├── TOOL   lookup_order.tool      tool.name=lookup_order
│   └── LLM    OpenAI chat        (nested call from inside the tool)
├── CHAIN  prompt_builder.run
└── TOOL   issue_refund.tool      tool.name=issue_refund

If you need to inspect or copy the active identity (for example, to tag a background job span with the same agent.id), import the helpers:

TypeScript

import { getActiveAgentIdentity } from "@inference/tracing";

const identity = getActiveAgentIdentity();
if (identity?.id) {
  jobQueue.enqueue({ agentId: identity.id, payload });
}

CLI And Subprocess Wrappers

When the work is a shell-out to a CLI (Claude Code, Codex, a custom binary), wrap the process call in an agent span and set the input and output yourself.

import { agentSpan, setup } from "@inference/tracing";
import { execFile } from "node:child_process";
import { promisify } from "node:util";

const execFileP = promisify(execFile);
const tracing = await setup();
const prompt = "Reply with just the word hello.";

await agentSpan(
  {
    agentId: "claude-code-prod",
    agentName: "Claude Code",
    spanName: "claude-code.invocation",
    sessionId: "conversation-cli-hello",
    system: "anthropic",
  },
  async (span) => {
    span.setInput(prompt);
    const { stdout } = await execFileP("claude", ["-p", prompt], {
      encoding: "utf-8",
      timeout: 60_000,
    });
    span.setOutput(stdout.trim());
  },
);

await tracing.shutdown();

Status Semantics

All Catalyst manual-span helpers follow the same status rules:

Callback returns normally → span ends with OK.
Callback throws → the exception is recorded as a span event, the span ends with ERROR and the error message, and the original exception re-throws so caller error handling is unaffected.

If you reach for tracer.startActiveSpan() directly (the OTel escape hatch mentioned above), you own status and span.end() yourself, which is the price of the lower-level API.

Next Steps

Handle API reference

Every method on the AGENT / TOOL / CHAIN span handle, with value-coercion rules.

Attributes reference

All Attr.* constants and SpanKindValues with the attributes each kind expects.

Production agent example

A production-shaped agent with custom tool execution, end to end.

Agent identity

Pick stable agent.id and session.id values for the Agents dashboard.

​Span Kinds You Author Manually

​Agent Spans

​Tool, Chain, And Retriever Spans

​Tool spans

​Chain spans

​Retriever spans

​Escape Hatch

​Adding Custom Attributes

​How Identity And Context Propagate

​CLI And Subprocess Wrappers

​Status Semantics

​Next Steps