Context
The operating situation
Compliance teams could only review yesterday's trading. By the time a pattern was flagged, the activity was a day old and the window to act on it had usually closed.
Challenge
The operating problem
Surveillance ran as an overnight batch, so suspicious activity surfaced a day after it happened. The platform was also near its throughput ceiling, which capped how much market activity could be covered at all.
Approach
Decisions that shaped it
- Repartitioned the Spark workflows so processing scaled horizontally instead of hitting a single ceiling.
- Parallelized point-in-time calculations that had previously forced sequential passes.
- Added market-data capture and moved result delivery onto serverless services.
- Rebuilt telemetry and release practice so regressions surfaced in minutes rather than in the next batch.
Solution
What was delivered
A re-architected surveillance pipeline: partitioned Spark processing, parallel point-in-time calculation, captured market data, and serverless result delivery - detecting inside 15-minute windows instead of overnight.
Outcome
Documented outcome
Detection moved from T+1 batch to 15-minute near-real-time windows, throughput rose from 8M to 750M events per day, and processing held 99.9% accuracy.
8M to 750M events per day describes this surveillance platform's throughput, not firm-wide volume, and the two figures are its own before and after. The separate data-retrieval platform's volumes are a different system and are not combined with these. Detection latency is a platform capability, not a regulatory outcome or a compliance guarantee. The former employer stays anonymized.
Controls
What kept it safe to operate
- Processing accuracy measured and held at 99.9% as throughput scaled, so speed did not quietly cost correctness.
- Telemetry and release practice rebuilt alongside the pipeline; mean time to recovery fell 65%.
- Detection windows are a monitoring improvement, not an assertion about what the firm's analysts concluded.