Alerts

Every alert rule is a windowed metric threshold: pick a metric, a threshold, and a window, and the rule fires when the metric measured over the trailing window exceeds the threshold. Rules are managed from the Alerts page in the dashboard or via the API.

Metrics

Rate thresholds are fractions, not percentages — use 0.3 to mean 30%. Latency thresholds are milliseconds. Latency rules window on the session's ended_at, so an in-progress call only enters evaluation once it ends.

How evaluation works

A sweeper evaluates every enabled rule every 30 seconds over a rolling window: each tick re-computes the metric over the trailing window_minutes (minimum 15), so data continuously ages in and out.

A rule fires when both hold:

• the window contains at least min_samples observations, and
• the measured value is strictly greater than threshold_value.

min_samples is the noise gate. With one session in the window, a single failed outcome is a 100% fail rate and a p95 over one turn is just that turn — raise min_samples so a rule waits for meaningful volume before paging you.

Suppression

Without a cooldown, a rolling window that crosses its threshold would re-fire on every 30-second tick until the offending data ages out — ~30 duplicate webhooks for one incident. Instead, each firing stamps the rule, and the rule is skipped from evaluation for one full window length.

Because the cooldown equals the window, the data behind a firing has aged out of the window by the time the rule is eligible again — a re-fire always reflects new bad data. A sustained incident therefore alerts once per window (e.g. every 15 minutes), not once total.

Two consequences worth knowing:

• Editing a rule does not reset suppression. A threshold change saved mid-cooldown takes effect at the next evaluation, which is whenever the cooldown expires.
• The stamp is claimed atomically inside the firing transaction, so even two sweepers racing produce one firing, never two.

Rule fields

Webhook delivery

A firing produces one JSON request to webhook_url:

{
  "type": "alert.triggered",
  "firing_id": "3fdc253e-…",
  "rule": {
    "id": "e4e4d9d1-…",
    "name": "Poor call voice quality",
    "metric": "latency_llm_ttft_p95",
    "judge_name": null,
    "threshold_value": 2000,
    "window_minutes": 15
  },
  "window": { "start": "2026-06-10T14:04:01Z", "end": "2026-06-10T14:19:01Z" },
  "matched_count": 5,
  "total_count": null,
  "observed_value": 2412.5,
  "agent_id": null,
  "account_id": null,
  "sample_session_ids": ["ws-4b0f79854c3c5d3a"],
  "fired_at": "2026-06-10T14:19:01.861Z"
}

For rate metrics, matched_count / total_count are the fraction's numerator and denominator. For latency metrics, total_count is null and matched_count is the number of turns in the window; sample_session_ids lists up to 20 sessions containing over-threshold turns.

Every request carries identifying headers, plus a signature when the rule has a secret:

Retries

Delivery is at-least-once. A response outside 2xx (or a 10-second timeout) schedules a retry; the backoff after the initial attempt is 30s, 2m, 10m, 30m, 2h — 6 attempts total before the firing is marked failed. All delivery state lives in Postgres, so retries survive restarts. Every attempt — success or failure, including test sends — is recorded in the alert_webhook_attempts audit table, exposed via the API and on the dashboard's Alerts page.

API routes

Running the sweeper

The sweeper runs inside the API process by default (ALERT_SWEEPER=inline) — single-container deploys need zero extra configuration. For deployments with a dedicated background worker (bun run start:worker), set ALERT_SWEEPER=off on the API so exactly one sweeper is active; the worker always sweeps. Both processes drain gracefully on SIGTERM/SIGINT.

A misconfigured second sweeper degrades to wasted work, not duplicate alerts — both the suppression stamp and the delivery claim are atomic.