Tuning a surveillance alert queue
A surveillance rule fires 2,823 alerts to catch 193 real ones. What raising the threshold saves, what it costs, and how to price every step.
Latest month, firm-wide: 4,310 alerts across eight rule types, 77.3 percent of them false positives. 2,872 went to a reviewer and 1,438 auto-closed on enrichment. 255 became cases, 80 were substantiated, 37 escalated. The queue reads healthy at a median 5.6 days to disposition until the next number: 18.0 percent of open alerts are past the 15-day review SLA.
The deliverable here is a tuning memo for one rule, excessive trading on an annualized turnover ratio. It answers three questions before anyone touches a setting: what the rule costs to review today, what a higher threshold saves, and how many real findings that gives up. The working version is the supervision and trade surveillance demo, where the threshold is a slider.
Synthetic data throughout. Seeded generator, no real firm, branch or representative.
One rule, thirteen settings
- At its current 3.0 threshold the rule fires 2,823 alerts. 193 are real; 2,630 are noise.
- The panel draws one population of 6,000 accounts once, each with a turnover ratio and a hidden flag for genuinely problematic activity. Every threshold is a recount of that same population.
- The invariant that proves it: true positives plus missed findings equals 196 at all thirteen settings. The rule can only move findings between caught and missed.
- At 3.0 the true positives are 7 percent of the bar; at 8.0 they are 64 percent. Tightening always helps precision. The question is what precision costs.
What a higher threshold saves
What it costs
Moving from 3.0 to the recommended 4.5:
- False positives fall 67 percent, from 2,630 to 878.
- Reviewer hours fall 62 percent, from 988.0 to 372.8: 615.2 hours returned.
- Missed findings rise from 3 to 9.
- 615.2 hours over 6 findings is about 103 reviewer hours per finding given up.
That last number goes in front of the supervisory principal, because it is the only form of the tradeoff a person can argue with. "Cuts false positives by 67 percent" sells itself. "One finding per 103 hours saved" starts a conversation about risk appetite.
Price every step
| Step | Hours saved | Findings given up | Hours per finding |
|---|---|---|---|
| 2.0 to 2.5 | 346.5 | 1 | 346.5 |
| 2.5 to 3.0 | 332.2 | 2 | 166.1 |
| 3.0 to 3.5 | 271.9 | 0 | free |
| 3.5 to 4.0 | 203.0 | 1 | 203.0 |
| 4.0 to 4.5 | 140.3 | 5 | 28.1 |
| 4.5 to 5.0 | 97.7 | 4 | 24.4 |
| 5.0 to 5.5 | 67.6 | 4 | 16.9 |
| 5.5 to 6.0 | 43.0 | 6 | 7.2 |
| 6.0 to 6.5 | 37.5 | 10 | 3.8 |
| 6.5 to 7.0 | 22.7 | 8 | 2.8 |
| 7.0 to 7.5 | 19.6 | 13 | 1.5 |
| 7.5 to 8.0 | 11.9 | 9 | 1.3 |
- The step from 3.0 to 3.5 is free on this data: 271.9 hours back for zero more missed findings.
- Past 4.0 the price of a finding collapses, from 203.0 hours at one step to 28.1 at the next and 1.3 at the far end. That is where tuning becomes a coverage cut.
- The flat stretch from 3.0 to 4.0, shaded on the hours chart, is the honest gain. Everything past it is a policy decision.
Be careful with the free step. With 196 problem accounts and half-point increments, a zero is partly an artifact of bin width. Test a free step before banking it.
What the numbers say
- Excessive trading is the loudest rule, 1,218 of the month's 4,310 alerts, and 93 percent of what it fires at today's setting is noise.
- The 77.3 percent false-positive rate and the 18.0 percent past-SLA backlog are one problem measured twice: reviewers are reviewing the wrong alerts.
- The same team owes an inspection on each of its 120 branches under the FINRA Rule 3110 cycle. It has completed 104 (86.7 percent), with 16 overdue. Freed hours have an obvious home.
How I would run this on the job
From 2015 to 2017 I was a branch office examiner at Advisor Group, now Osaic, conducting roughly 100 to 120 branch office examinations a year under FINRA Rule 3110: the firm's own internal inspections of its own branches. I later worked in regional supervision as a manager and then a director, and after that on the supervisory controls team. This queue is the artifact I spent those years inside.
- Start from the disposition funnel, not the alert count. A rule whose alerts rarely reach a case is a tuning candidate; one whose alerts often do stays, however loud.
- Fix the population before touching the threshold, so the curve moves only because the rule did.
- Interrogate the label. On the job it comes from closed cases and is only as honest as the reviewers' notes, so sample and re-read dispositions first.
- Take the flat stretch, then stop. Steps past it change what the firm is willing to miss, and that call belongs to the supervisory principal.
- Deliver a one-page memo: current and proposed settings, alerts before and after, findings given up, hours returned, the curve attached, and the six findings written down as the price.
- Re-run it quarterly on high-volume rules and annually on the rest, with the prior memo attached.
The systems I have worked supervision in are FIS Protegent Surveillance for the alert queue, NetX360, Envestnet WMP and Wealthscape for account and trading detail, and Smarsh for communications review. The tuning curve belongs upstream in SQL or Python against the dispositioned case history, with the threshold table versioned so a change leaves a dated record with a name on it.