Signals

Once traces are flowing, signals turn them into structured labels. A signal is a plain-language classifier you define once and Catalyst runs continuously against one of your agents. Describe what you want to detect (“is this NSFW?”, “did the user get frustrated?”, “was the task completed?”) and every matching trace gets labeled automatically by an LLM judge, with the results queryable right alongside your other agent metrics. Signals are how you evaluate the things default metrics miss. For a user-facing agent they tell you how people are actually interacting with it (frustration, sentiment, jailbreak attempts). For a non-user-facing agent they tell you how the agent itself is behaving (did it refuse, did it complete the task, did it stay on policy).

​

Before you start

Signals run on captured traffic, so you need traces flowing first. This guide assumes you’ve already captured your first trace. You also need:

• An agent with traces installed. Signals are always scoped to a single agent, so there is nothing to label until that agent is emitting traces.
• A stable, consistent agentId. Signals are per agent, so your traces need a consistent agent identity to group by. Set the agentId once and keep it the same across runs. If you instrumented with the CLI this is usually already set. See Set agent identity.
• A sessionId, if you want session-scoped signals. Session signals classify a whole conversation, which requires that your traces carry a sessionId (your conversation or chat ID) so Catalyst can assemble the conversation. See Choose a scope.

​

Where to find signals

You can get to signals two ways in the dashboard:

• The Signals tab, which lists every signal and is the main table view across your agents.
• The Agents tab, where you pick an agent and open its Signals view to see and create signals scoped to just that agent.

​

How signals work

A signal evaluates traffic for one of your agents. When you activate it, Catalyst samples incoming traffic for that agent, sends each sampled target’s input and output to an LLM judge with the prompt you wrote, and writes the label back. You can then filter, chart, and break down your traces by that label.

​

Choose a scope

A signal’s scope is the unit it labels. You pick it when you create the signal, and it’s fixed for the life of the signal (everything else is editable). Scope determines how much context the judge sees on each call.

​

Two classifier types

When you create a signal you choose how it labels each target.

​

Create a signal

Create a signal from the Signals tab, or from a specific agent’s Signals view under the Agents tab. Either way the signal is scoped to one agent.

​

Test before you activate

Before you commit a signal to live traffic, run it against recent traffic to preview how it labels it. A test run classifies a small sample synchronously and shows you the label distribution and per-target results, and nothing is saved. It’s a preview only and doesn’t affect your signal or store any labels.

​

Activate and run live

Activating a signal starts live classification: as new traffic arrives for the agent, the configured share of it gets sampled and labeled automatically. You don’t have to do anything else, and labels accumulate as traffic flows. A signal is always in one of three states: You can disable an active signal at any time to stop classification without losing the labels you’ve already collected, and re-enable it later.

​

Backfill historical data

Live classification only labels traffic that arrives after you activate. To label traffic you already captured, run a manual run (backfill) at any time. Unlike a test, a manual run saves its labels: they’re stored against your traces and tied to the run, exactly like live labels.

​

Read the results

There are a few places to read what a signal found:

• The Signals tab is the main table across your agents, with each signal’s current state and headline numbers at a glance.
• The agent’s Overview under the Agents tab has per-signal graphs: trends and volume over time, plus a range of metrics for each signal so you can see how a label is moving.
• The signal detail view has a table of every classified target (the spans, traces, or sessions the signal labeled). Click into any row to open the underlying trace, span, or session and read the actual conversation that produced the label.

Labeled targets render their label as a colored chip. For a binary signal, “yes” and “no” get distinct colors; for a string signal, each label gets its own color. From there you can:

• Filter by label value to pull up just the targets a signal flagged (for example, every session labeled “yes” by a jailbreak signal).
• Watch the distribution over time to see how a label trends across hours or days.
• Jump straight to the underlying trace, span, or session to see the full context and conversation behind any labeled target.

​

Alert on a signal

Once a signal is running, set up alerts so you hear about changes without watching the dashboard. Alerts are configured per signal, and you can get notified through the Slack integration or by email. An alert fires when a metric crosses a condition over a window. You choose:

• The metric. What to watch, depending on the classifier type:
  • Label volume (any signal): how many labels came in.
  • True rate (binary signals): the share of labels that are “yes.”
  • Value count (string signals): how many labels landed on one specific value.
  • Value share (string signals): that value’s share of all labels.
• The comparison. Either a percentage change versus the prior equal-length window (for example, “this value’s count is up 10% in the last 24 hours”) or an absolute threshold (for example, “true rate is below 80%”).
• The window the comparison runs over, from 5 minutes up to 48 hours.
• A minimum label count so quiet periods don’t trip the alert on statistically meaningless deltas.
• A cooldown so a flapping condition doesn’t bombard you. After a firing, the next one is suppressed until the cooldown elapses.

You can backtest an alert against recent history to see when it would have fired before you turn it on, and pause or re-enable any alert at any time.

​

Versioning and editing

Signals are versioned. Editing a signal (changing its prompt, labels, classifier type, or sample rate) creates a new version rather than mutating the old one, and the dashboard shows each version after every edit. The current version is the one powering live classification, and labels record which version produced them, so you can change a signal’s definition without losing the history of what earlier versions decided. Scope is the exception: it’s fixed once a signal is created. Everything else is editable, but to label a different unit you create a new signal. When you no longer need a signal, archive it. Archiving stops it and removes it from your active list while preserving the labels it produced.

​

Templates

To get started quickly, create a signal from a template and edit from there. Built-in templates include: Templates are just a starting point. You can write any prompt you want and build a signal from scratch, with either classifier type.

​

Manage signals from the CLI and MCP

The dashboard isn’t the only way in. Everything above is also available through the Inference CLI and the MCP server, so you can create, run, and read signals from your terminal or straight from an AI coding assistant.

• CLI. The inf signals command group lists, creates, edits, activates, disables, and archives signals, tests them, kicks off manual runs, and reads labels and distributions.
• MCP. The Inference MCP server exposes the same operations as tools (creating signals, activating and disabling them, running backfills, configuring alerts, and querying labels and distributions), so an assistant can set up and inspect signals on your behalf.

​

Feed signals into Halo

Signals pair naturally with Halo, our agent-loop optimizer. The targets a signal flags are exactly the traffic worth digging into:

• Improve a behavior. Point Halo at the spans, traces, or sessions a signal flagged and ask how to fix what they have in common (the refusals, the frustrated sessions, the failed tasks).
• Decide what to measure. Talk to Halo about your traces to surface which signals would be most valuable to add in the first place.

​

Next steps