Real-time payments data for a UK challenger bank.

The bank’s core processed payments in real time, but everything downstream of it lived on overnight batch: fraud analytics ran on yesterday’s transactions, customer-facing spend insights lagged a day, and finance reconciliation started each morning with a multi-hour ETL window that had begun to overrun.

The core itself was off-limits — rightly. It was stable, regulated, and supplied by a vendor whose change process ran in quarters. Every previous proposal had foundered on the assumption that fresher data required touching it. The brief: real-time data for the rest of the bank, with the core left exactly as it was.

Change data capture was the unlock. Debezium read the core database’s replication log — a read-only, vendor-supported interface — and published every transaction to Kafka within seconds of commit. No core code changed; the vendor relationship stayed simple.

The raw CDC feed is an implementation detail, so we put a translation layer in front of it: stream processors turned table-level changes into well-named domain events — payment.settled, account.balance-updated — governed by a schema registry with enforced compatibility rules. Consumers integrate against business language, not the vendor’s table structure, which means a future core migration will not break them.

Consumers moved one at a time, with batch running in parallel and outputs reconciled nightly until each team had weeks of evidence the stream matched. Fraud analytics went first — freshest-data value — then spend insights, then finance. Batch was switched off consumer by consumer, with reconciliation reports as the retirement criterion.

Source       ·  Core banking DB → Debezium CDC (log-based, read-only)
Backbone     ·  Kafka (MSK), schema registry, enforced compatibility
Translation  ·  Stream processors: CDC rows → domain events
Consumers    ·  Fraud, customer insights, finance reconciliation
Migration    ·  Parallel-run per consumer, nightly reconciliation

Data freshness across the bank went from 24 hours to under 60 seconds, with zero downtime through the entire migration. Fraud models began scoring on live transactions, catching patterns the overnight run had been seeing a day late. The morning ETL window — and its overrun alerts — disappeared. At handover, 14 consumers ran on the backbone, operated by the bank’s own platform team with our runbooks.