Skip to main content
Your dashboard tracks the metrics every agent has: run count, error rate, latency, tokens, cost. However, these metrics don’t accurately tell you whether the agent actually did its job, or whether the people using it are happy. Did it finish the task or quietly give up halfway? Is anyone getting frustrated? Those answers live in the content of your traces, and reading traces by hand obviously doesn’t scale. Signals fix that. A signal is a plain-language classifier you define once and Catalyst runs continuously against your traffic as it comes in. You describe the signal and an LLM judge figures out if that applies to the span, trace, or session automatically. The labels become metrics you can chart, filter by, and alert on.
Traces flow in, a signal you define classifies each span, trace, or session with an LLM judge, and the labels land back on your traffic as metrics you can filter, chart, and alert on
Signals are fully configurable to your needs but their are two overarching things you are typically measuring:
  • How users interact with your agent — sentiment, frustration, jailbreak attempts, NSFW content.
  • How your agent behaves — whether it completed the task, refused, or stalled.
You can interact with signals via the dashboard, through the MCP server, and from the CLI (inf signals). This guide leads with the dashboard.

​
Before you start

Signals run on your traces, so you need traces flowing first. You need:
  • A free Inference account.
  • Tracing installed and traces arriving. If you haven’t done this, start with Optimize an Agent End to End or the Tracing Quickstart and come back once traces show up.
  • A stable agent identity. Signals are scoped to one agent, so your runs need a consistent agentId to group on. If you instrumented with the CLI this is likely already set.
import { agentSpan } from "@inference/tracing";

await agentSpan(
  {
    agentId: "gator-flue-agent", // Stable ID every signal attaches to. Keep it constant across deploys.
    agentName: "Gator Flue Agent",
    sessionId: chatId,         // Needed for session-scoped signals. Think Slack thread, chat, or job ID - Basically a single conversation.
  },
  async (span) => {
    span.setInput(input);
    const result = await run(agent, input);
    span.setOutput(String(result.finalOutput ?? ""));
  },
);
Signals group on agentId. If you want session-scoped signals (Step 2), you also need a stable sessionId on your traces. See Agent identity.

​
Step 1: Find Signals in the dashboard

There are two ways in:
  • The top-level Signals tab lists every signal across all your agents in one table, with each one’s agent, scope, type, a live result breakdown, trend, and volume.
  • Inside an agent (Agents → your agent → Signals) you see just that agent’s signals, with their overview graphs next to its other metrics.
Either way, signals are per agent — each one classifies the traffic of a single agentId. Here’s the project-wide table for our team, with the spread we run on Gator:
The top-level Signals tab showing a table of every signal across the project — session sentiment, sentiment, task outcome, laziness / refusal, and user frustration — each with its agent, type, live result breakdown, 7-day trend, and volume

​
Step 2: Create a signal

Open the agent’s Signals view and create a new signal. You can write your own from scratch or start from a template that prefills the type, prompt, and labels for a common case:
TemplateTypeWhat it flags
Task outcomeStringWhether the task was completed / partial / failed / abandoned.
SentimentStringOverall user sentiment: positive / neutral / negative.
User frustrationBinaryThe user expressing annoyance, repeated correction, or dissatisfaction.
Laziness / refusalBinaryThe assistant refusing, stalling, or giving a low-effort non-answer.
Jailbreak attemptBinaryThe user trying to bypass safety guardrails or override the system prompt.
NSFWBinaryContent that’s sexually explicit, graphic, or not safe for work.
You’re not limited to these — write any prompt and classify anything you can describe. Either way, every signal comes down to a few decisions.

​
Classifier type

Binary (yes / no)

A true/false classifier for “is this X or not” questions: frustration, refusals, jailbreak attempts, NSFW. No labels to configure, just a prompt.

String (enumerated labels)

Returns one of a fixed set of labels you define. Use it when there are more than two outcomes: task outcome (completed / partial / failed / abandoned), sentiment (positive / neutral / negative). Define 2 to 10 labels.

​
Scope

A signal classifies one unit of traffic at a time:
ScopeWhat the judge readsBest for
SpanA single model call.Fine-grained checks on one step. Can be a lot of labels on a busy agent.
TraceOne full request or turn.Request-level questions about a single run.
SessionThe whole conversation or job, end to end.Understanding a complete interaction: overall sentiment, or whether a multi-turn task ultimately got done.
Session scope needs a sessionId on your traces. Without it there are no sessions to classify. Also note: scope is fixed once a signal is created — you can edit the prompt, labels, and sample rate later, but not the scope. Pick deliberately.
The same question can be worth running at more than one scope. We track sentiment two ways on Gator: once per session for the overall vibe of a conversation, and again at a finer scope to spot where things turn. That’s why you’ll see both in the screenshots.

​
Sentiment

Every signal also carries a sentiment that tells Catalyst whether an outcome is good, bad, or neutral. This is what lets alerting reason about direction instead of just movement. A negative result trending up, or a positive result trending down, is worth flagging; a neutral result is just neutral and won’t trigger anything on its own. You set sentiment based on the classifier type:
  • Binary signals get one sentiment for the whole signal. A Jailbreak attempt or NSFW signal is negative, so any climb is bad news.
  • String signals get one sentiment per label. On a Sentiment classifier you’d mark negative as negative, positive as positive, and neutral as neutral, so a rise in negative labels or a drop in positive ones gets caught automatically.
This mostly pays off in alerting: because each label already knows whether it’s good or bad, an alert can watch for “the bad thing going up” without you spelling out the direction every time.

​
Sample rate

A sliding selector sets how much of your matching traffic actually gets classified, anywhere from a small fraction up to 100%. Every classification is a judge call you pay for, so this is the main cost lever on a signal. 100% labels every span, trace, or session. That’s fine on a low-volume agent, but if you’re running a lot of traffic it’s usually far more than you need and can get expensive fast. A 25% or 50% sample gives you a representative trend at a fraction of the spend; raise it later once you trust the labels or want finer resolution. You can change it anytime.

​
Write the prompt and save

Whatever you’re classifying, the prompt is the instruction the judge follows on every target, so be specific about what counts as each outcome (and, for a string classifier, what each label means). When it’s written, give the signal a short, readable name and save it as a draft — that keeps it unpublished so you can test before going live.
The create-signal form (shown here for a Jailbreak attempt signal at session scope), with the template picker, scope and binary type selectors, the classifier prompt, sample-rate presets, and an inline 'try it before you commit' preview
Prefer to script it? inf signals create does the same from the terminal, and the MCP server exposes create_signal so your coding agent can set one up for you.

​
Step 3: Test before you activate

Before committing a signal to live traffic, run it against recent data to preview how it labels. A test classifies a small sample (1–100 recent targets) and shows you the distribution plus the label on each one. Nothing is saved — it’s a preview, so it doesn’t touch your signal or store labels. If the judge disagrees with you on a few, tighten the prompt and test again. This is the cheapest place to get a signal right, before any labels exist.
The Test Signal dialog noting results are not persisted, with a sample size and time range, the positive / neutral / negative distribution across the sample, and a per-row table of each span and its label
The CLI equivalent is inf signals verify.

​
Step 4: Activate, pause, and resume

When the test looks right, activate the signal. Live classification starts immediately: as new traffic arrives, the configured share is sampled and labeled automatically. A signal is always in one of three states, and you can move between them anytime:
StateWhat it means
DraftSaved but not running.
ActiveLive. New matching targets are sampled and labeled.
DisabledPaused. Classification stops, but every label collected so far is kept.
Pause an active signal whenever you want and resume it later without losing history. Sampling is deterministic, so the share you set is the share you pay for.

​
Step 5: Backfill historical data

Live classification only labels traffic that arrives after you activate. To label data you already captured — so your charts have history on day one — run a manual run (backfill): pick a historical window and a sample rate, and it classifies past targets in the background. Unlike a test, a backfill saves its labels, exactly like live ones. You can run one at any point, not just at creation.
The Manual Run / Backfill dialog applying a signal to a historical time range, with From and To dates and a sample-rate slider for the percentage of past targets to classify
Backfill a short window first to sanity-check labels at scale before running over months of history — a manual run classifies real targets and counts toward usage. CLI: inf signals run create.

​
Step 6: Read the results

This is where signals become metrics. Each agent’s Signals view rolls up every signal it runs: a summary up top (how many are active, total classifications, the busiest signal) and a card per signal with its label breakdown and trend, sitting right next to run count, latency, and cost.
The agent's Signals overview showing summary tiles for active signals, total classifications, and busiest signal, with a card and trend graph for each signal: session sentiment, sentiment, task outcome, and laziness / refusal
Open any signal and its detail page charts its classifications over time and the full label distribution.
A signal detail page for Task outcome, with stat tiles for classifications, top label, sample rate, and live version, a stacked bar chart of classifications over time, and a donut of the completed / partial / failed / abandoned distribution
Below the charts is a table of every classified target, with its label. Click any row to drop into that span, trace, or session and read the actual conversation behind the label: the full run, the tool calls, the output the judge saw. You can also filter the main Traces view by a label to pull up just the flagged runs.
The labeled-traces table on a signal detail page, one row per classified target with its label, version, session, and time, each clickable through to the underlying trace

​
Step 7: Edit and version

Editing a signal’s prompt, labels, or sample rate creates a new version rather than overwriting the old one. The dashboard shows each version, and every label records which one produced it, so you can sharpen a definition over time without losing the history of what earlier versions decided. (Scope is the one thing fixed at creation.) When a signal has served its purpose, archive it: it stops and leaves your active list, but its labels are preserved.

​
Step 8: Get notified

A trend you have to remember to check is one you’ll miss. There are two ways to stay on top of your signals without logging in: targeted alerts for a specific condition, and a daily overview that summarizes everything.

​
Targeted alerts

An alert watches one signal and fires when its numbers cross a line. They’re configured per signal (a signal can have several), and a global Alerts page lists every alert across all your signals. You build an alert by picking a metric to watch, the direction and threshold that count as a problem (a level it crosses, or a percentage it moves up or down), and the time window to measure over. Which metrics are available depends on the signal’s type:
MetricWorks onMeasures
True rateBinaryShare of targets labeled “yes.”
Value countStringCount of a specific label.
Value shareStringShare of targets with a specific label.
Label volumeAnyTotal labels produced.
The threshold reads one of two ways: percentage change (the metric moves up or down by X% versus the prior window) or an absolute level (it crosses a fixed number). You also set the rolling window (5 minutes to 48 hours), an optional minimum label count so low volume doesn’t trip it, and a cooldown so repeat firings don’t bury you. As you tune it, the form previews where the alert would have fired over the last 7 days. The sentiment you set in Step 2 is what makes the direction obvious here: you watch negative labels for an increase and positive labels for a decrease, since both mean things are getting worse. Neutral labels are rarely worth alerting on at all.
The alert create form: a signal and name, the condition built from a metric, a decreases-by / increases-by percentage or absolute level, and a window, plus a minimum label count, a Slack connect prompt, a cooldown selector, and a 7-day 'would have fired' preview chart
By default an alert emails your whole team, and any member can mute an individual alert for themselves. Connect Slack once and you can route an alert’s firings to a specific channel instead. A firing tells you the signal, the metric, the threshold, what was observed, and when:
A Slack signal-alert message for Jailbreak attempt reading 'Label volume above 1 over 5m', with fields for the signal, metric, comparison, threshold, observed value, window, and trigger time, and a View alert button
The CLI mirrors all of this: inf signals alerts backtest to preview, then inf signals alerts create.

​
Daily overview

Targeted alerts are for when something specific goes wrong. The daily overview is for confirming everything’s humming along without opening the dashboard. Opt in and you get a once-a-day email (or Slack) summarizing all your signals: how many were classified, and the biggest movers since yesterday. It’s the “is anything off?” glance you can do from your inbox.
A daily signal overview email summarizing the day across signals — total classified, the biggest mover called out at the top, and a card per signal with its volume and percentage change — plus a View signals button

​
Step 9: Hand it to HALO

Signals tell you what is happening and how often — failed tasks spiked overnight, frustration is up this week. HALO tells you why, and what to change. The handoff runs through the MCP server. With Inference connected to your coding agent, you go from a signal to a fix in plain language: 
The Task outcome signal on gator-flue-agent is showing more failures this week.
Pull the failed sessions, figure out what's going wrong, and propose a fix.
It works the other way too — ask HALO what to watch: 
Look at gator-flue-agent's recent traces and tell me which signals would catch its most common failure modes.
HALO reads the flagged traces, finds the systemic cause, and writes back concrete fixes with citations to the exact runs. See the MCP server guide for setup.

​
Where to go next

You now have quality metrics that didn’t exist an hour ago — defined in a sentence, computed on every run, charted, and wired to notifications — with no extra instrumentation.

Signals reference

Full reference for classifier types, scopes, sample rates, states, testing, backfills, and templates.

Optimize an agent end to end

Run HALO on the traces your signals flag to find the root cause and apply the fixes.

Connect the MCP server

Create signals, read labels, and hand flagged traces to HALO from your coding agent.

Set agent identity

Add stable agent and session IDs so signals attach to the right agent and scope cleanly.