$540k of phantom data, and the real lesson underneath it

The problem

Traders on FXVPS run automated strategies across MetaTrader 4 and MetaTrader 5. They expect a single, trustworthy view of performance, but their trade history arrives from two platforms that each behave as though they are the source of truth.

Ingest both streams naively and the cracks begin to appear. Duplicate trades are counted twice, trading volume is inflated, and balance and credit events quietly masquerade as P&L. None of the numbers look obviously wrong. They look plausible, which is exactly what makes reconciliation bugs so dangerous.

What I found

A cross-platform reconciliation pass uncovered $540k of phantom data across seven accounts: duplicated volume and positions that did not correspond to anything real.

Once that noise was removed, a further $4k of genuine accounting drift became visible underneath. The real discrepancy had been there all along; it was simply buried beneath much larger inconsistencies.

What I built

The fix was not a more complicated reconciliation algorithm. It started with making the source of truth explicit.

• Platform-aware duplicate detection.
• Balance and credit filtering.
• Broker-anchored P&L reconciliation.

Rather than trusting either trading platform’s local view, every account’s P&L is re-anchored to the broker’s records. The migration itself was ordered index-first, allowing the reconciliation job to complete within the host’s boot window instead of competing with the rest of the startup process.

The real lesson

Most data problems are not really data problems. They are undeclared source-of-truth problems.

The interesting engineering challenge was never the deduplication maths. It was deciding, before writing a line of reconciliation code, which system was allowed to be right when two systems disagreed. Once that decision existed on paper, the reconciliation logic became straightforward.

Name the source of truth first.

The rest of the pipeline tends to follow.