Automating Pre-Trade Market Data Validation

The Challenge

Market data was reviewed manually each morning by an operations analyst, who scanned for obviously wrong values. The process caught large, obvious errors but missed smaller anomalies - a price that was plausible but still wrong - that only became apparent once they affected valuation.

Existing Environment

A gas trading and risk platform receiving daily price curves from multiple sources, with manual visual review as the only validation step.

Root Cause

There was no systematic, rules-based validation layer - detection depended entirely on an analyst noticing an anomaly by eye, which does not scale and is inherently inconsistent.

Solution

We designed a rules-based and statistical validation layer that ran automatically on every incoming price curve: bounds checks against historical ranges, day-over-day change thresholds, and cross-source comparison where multiple providers quoted the same instrument. Flagged values were held for review before being released to the ETRM rather than failing silently or passing through unchecked.

Validation pipeline

Implementation

• Defined validation rules with input from the trading and risk teams
• Built the validation layer to run automatically on every data load
• Created a review queue for flagged values with a clear approve/reject workflow
• Tuned thresholds over several weeks to balance false positives against missed errors

Result

• No further undetected bad-data incidents in the twelve months following rollout
• The operations analyst's role shifted from manual scanning to reviewing a small number of flagged exceptions
• Review time per day dropped substantially despite the same volume of incoming data

Technologies