Real-Time ODS for Capital Markets: The Micro-Batch Era Is Ending

For a decade, capital markets firms accepted an operational data store that ran a few minutes behind the market. That tradeoff is closing. Trading is moving toward a 23-hour day, settlement has compressed to T+1, and the lakehouse can finally process data as fast as the tape moves. The minutes-behind operational data store is turning into a liability.

Why the ODS question changed in 2026

For most of the last decade, intraday risk had a comfortable floor. Markets opened, markets closed, and a desk that reconciled positions every few minutes stayed inside the risk window for the trading day. Batch-oriented operational data stores fit that world, and the cost savings were real, so firms accepted the lag.

Two structural shifts broke that assumption this year. First, the major US exchanges are moving toward extended sessions, with 23-hour, five-day trading on the 2026 roadmap. A trading day with almost no close removes the natural reconciliation window that batch architectures quietly depended on. Second, T+1 settlement has raised the velocity of collateral and funding. Margin calls, collateral moves, and funding decisions now run on a clock that a minutes-behind position view cannot keep up with.

Volume is also getting burstier. Intraday spikes are wider, and during volatile sessions the gap between firms running resilient real-time infrastructure and firms on platforms built for a calmer market becomes visible fast. The operational data store sits right in the middle of that gap. It is the layer that decides whether the risk desk is looking at the market or looking at history.

What an operational data store actually does here

An operational data store sits between transactional systems (order management, execution platforms, core banking) and analytical consumers (risk engines, profit and loss calculators, regulatory reports). Its job is to take live operational data, normalize it, and serve it at speeds no batch process can match.

In a capital markets context, the operational data store has to answer questions like these in seconds, not minutes:

• What is our firm-wide position in a given name across every desk, right now?
• What is the fixed income desk's intraday profit and loss as of ten seconds ago?
• Which trades from the last five minutes breach our value-at-risk limits?
• What is our total exposure to a single counterparty across all products?

If any of those take longer than a few seconds during a volatile session, the architecture has a problem. In a 23-hour trading world, that problem runs all day.

The three-layer reference architecture

I've deployed this pattern at multiple financial institutions on Databricks and Microsoft Fabric. It's not the only way to build an ODS, but it handles the latency, cost, and governance requirements better than the alternatives I've tested.

Layer 1: Streaming ingestion

Trade events, market data ticks, and position updates flow in through Kafka or Azure Event Hubs. Databricks Structured Streaming consumes these and writes to Delta Lake tables with autocompaction and optimized write enabled.

The design decision at this layer used to be a compromise. You picked trigger.availableNow for micro-batch (ten to thirty second latency) or trigger.processingTime("0 seconds") for near-continuous micro-batch, and for true sub-second work you bolted on a separate Apache Flink engine. Databricks Real-Time Mode, generally available since March 2026, removes that compromise. It brings millisecond-class latency to the same Structured Streaming code, so the dual-engine architecture is no longer the price of going fast. Early adopters report latency cut by more than 80 percent without rewriting their pipelines.

My guidance to clients is not "switch everything to Real-Time Mode." It is to keep micro-batch where ten-second latency is genuinely fine, because the cost difference is real, and to move position aggregation and limit monitoring to Real-Time Mode, because in a 23-hour trading day those layers no longer have a slower option.

Layer 2: Materialized views and aggregation

Raw trade events feed downstream views that pre-compute the aggregations your consumers actually need: position by instrument, P&L by desk, exposure by counterparty. Delta Lake's Change Data Feed (CDF) makes this incremental, so you're never rescanning full tables.

Layer 3: Serving and consumption

Different consumers need different serving patterns. Here's how it breaks down:

Where Databricks and Fabric each fit

I get this question in every presales conversation: "Should we use Databricks or Fabric?" The short answer for capital markets: both, for different things.

• Databricks handles streaming ingestion, complex transformations, ML feature engineering, and Unity Catalog governance. Think of it as your processing engine.
• Fabric handles business analytics, Power BI integration, and data sharing with non-engineering teams via OneLake. Think of it as your consumption layer.

The Databricks-to-Fabric connector through OneLake shortcuts makes this practical. Delta tables written by Databricks show up directly in Fabric without any data copying. Your engineers get Spark; your analysts get Power BI; the data stays in one place.

Four mistakes that still kill ODS latency

Real-Time Mode raises the ceiling, but it does not fix a badly shaped data store. I have debugged enough slow deployments to see the same four patterns repeat, and none of them are solved by a faster engine:

1. Over-indexing on Z-ordering. Z-ordering helps with ad-hoc queries, but for fixed aggregation patterns, pre-materialized views are faster every time. Don't optimize the scan. Eliminate it.
2. Ignoring partition pruning. Partition by trade date and instrument class. If your queries can't skip 90%+ of the data, rethink your partitioning.
3. One cluster for everything. Your ingestion workload (write-heavy, small cluster) and your aggregation workload (compute-heavy, autoscaling) have completely different resource profiles. Separate them.
4. Batch-window governance. If data quality checks run nightly but data arrives continuously, you'll discover issues 23 hours too late. Move quality gates into the streaming pipeline itself.

Governance without the performance tax

Unity Catalog makes this possible. Row-level security, column masking, and attribute-based access control let you serve the same Delta tables to the trading desk and the compliance team. Each group sees only what they're authorized to see, and you maintain a single copy of the data.

Data lineage tracking (automatic in Unity Catalog) gives you a full audit trail from the source trade event through to the downstream risk report. When an examiner asks how a number was calculated, you have the answer ready.

What I'm telling clients to do now

The technology arrived the same year the market made it mandatory. That alignment is rare, and it changes the conversation I have with capital markets clients.

• Treat the real-time operational data store as a 2026 deliverable, not a roadmap item. Extended hours and T+1 are not waiting for a modernization cycle. The risk window has already shrunk.
• Migrate the latency-sensitive layers first. Position aggregation, limit monitoring, and collateral views are where minutes-behind hurts most. Move those to Real-Time Mode and leave genuinely batch-tolerant layers where they are.
• Collapse the transactional layer into the lakehouse where it fits. Lakebase puts a managed Postgres engine next to the Delta tables. An operational data store that used to need a separate operational database can consolidate, which cuts cost and shrinks the reconciliation surface.
• Design for portability. Delta Lake and open table formats keep you from getting locked to one vendor's real-time story while the streaming market keeps consolidating around you.

This is the work IBM Consulting is doing with capital markets clients now. Not picking a product, but sequencing the migration so the risk desk sees the benefit first, the governance story holds up to an examiner, and the architecture survives the next vendor shift. The platforms are ready. The discipline is in the rollout.

The best real-time operational data store is not the one with the lowest benchmark latency. It is the one where the risk desk, the regulator, and the finance team all trust the same number at the same moment.