DEV Community: Derek Francis

Six Years In, Cloud Waste Is Still the Number One Job

Derek Francis — Sun, 06 Sep 2026 18:31:39 +0000

"First they asked us to fix cloud. Then fix the software mess. Now it’s fix the contract and license mess, now fix the data center."

That is a practitioner quoted in the FinOps Foundation’s State of FinOps 2026 report, and it is a fair summary of what has happened to the discipline. The survey behind it collected responses from 1,192 practitioners who together account for more than $83 billion in annual cloud spend, spread across EMEA, North America, Asia Pacific and South and Central America.

After six annual editions of tooling, certification, dashboards and vendor consolidation, the single top priority reported by those practitioners is still workload optimisation and waste reduction. The same thing it was in 2020.

That deserves a moment of honesty. If the top problem has not shifted in six years, the approach to it probably needs to.

Why waste keeps coming back

Why does the same problem survive six years of sustained attention? Because waste is not a one-time cleanup. It is a byproduct of how software gets built. Every sprint creates new resources and only some sprints remove old ones. An environment spun up for a load test in March is still running in September because nobody remembers whose it was, and the person who created it moved teams in May.

Dashboards make this visible. They do not make it stop. The organisations that genuinely bend the curve move the decision left, into the moment infrastructure gets defined rather than the moment the bill arrives. In practice that means budget limits enforced in the pipeline, mandatory ownership tags that fail a deployment when they are missing, and automatic expiry on anything created outside production.

Firms that bring in Cloud Engineering Services for this reason are usually not buying another cost report. They are buying the guardrails that make the report boring.

AI arrived faster than anyone planned for

Here is the most striking movement in the 2026 data. 98 percent of respondents now manage AI spend. Two years ago that figure was 31 percent.

Have you seen a cost category move that fast before? A line item that barely existed is now near-universal, FinOps for AI is the top forward-looking priority for the coming twelve months, and AI cost management is the skill practitioners most want to build.

AI spend also behaves differently from ordinary compute. It is bursty, it is often billed per token or per request rather than per hour, and the people generating the cost are frequently not the people who understand the pricing model. So the familiar playbook of reserved instances and rightsizing does not map cleanly onto it.

The remit keeps widening

The survey shows scope expanding well past public cloud. 90 percent of respondents now manage SaaS or plan to, up 25 points. 64 percent manage licensing. 57 percent manage private cloud and 48 percent manage data centre spend. 28 percent have been handed labour costs as well.

Is that scope creep or seniority? A bit of both. 78 percent of these teams now report to a CTO or CIO, up 18 points against 2023, and teams with VP-level engagement carry two to four times more influence over technology selection.

Where to start if you are behind

Pick your three largest cost centres and find out who owns each one by name. Then make one change in the pipeline instead of one more change in a dashboard. Ownership tags enforced at deploy time will do more for next quarter’s bill than any number of weekly cost reviews.

And if your AI spend is still landing on a general compute line, split it out now. You will want the history when someone asks why it tripled.

About Author

Shefali Vasave

Shefali manages content marketing at Opus Technologies, a domain-native engineering partner for banks, payment providers, and fintechs, and writes on the various aspects of financial institutions navigating change in a real-time, digital-first world.

Your Pipelines Got Faster. Your Checks Did Not.

Derek Francis — Sun, 06 Sep 2026 18:31:14 +0000

Two numbers from the same survey sit a page apart and tell opposite stories. 72 percent of data teams say they are prioritising AI-assisted coding in their development workflow. Only 24 percent say they are prioritising AI-assisted pipeline management, which covers testing, observability and everything else that tells you when a pipeline broke.

Both figures come from the State of Analytics Engineering report that dbt Labs published in April 2026, based on 363 data professionals surveyed between late 2025 and early 2026.

So the writing got faster and the checking did not. You can probably guess what happens next.

The same report found that 71 percent of data professionals named incorrect or hallucinated outputs reaching stakeholders as a top concern. That worry is rational. When a team triples the number of models it ships in a quarter without touching its test coverage, the failure rate per model stays flat while the absolute number of failures climbs.

Trust went up, which is the strange part

Here is the finding that complicates the story. Trust in data jumped from 66 percent to 83 percent year on year, the largest single-year rise the report has recorded, and speed as a stated priority climbed from 50 percent to 71 percent over the same period.

What do you get when you read those together? An organisation that is moving faster, feels better about its data, and has fewer people watching the pipes. That combination has a shelf life.

Ownership is the quiet failure

41 percent of organisations in the survey reported ambiguous data ownership. This is the one that rarely appears in a postmortem but sits underneath most of them.

Ask a plain question inside your own company. When the revenue dashboard is wrong on a Monday morning, who is expected to notice, and who is expected to fix it? If the honest answer is "whoever gets pinged first", you do not have an ownership model. You have a rota that happens to hold while nothing serious breaks.

Ambiguity here is expensive because it delays detection, not because it delays repair. Repair is usually quick. Detection is what takes three days.

The budget squeeze is real

One more pair of numbers from the report. 57 percent of respondents reported increased warehouse and compute spending, while only 36 percent reported increased team budgets. That means more data, more models and more compute handled by roughly the same number of engineers.

This is the pressure that has pushed a lot of organisations toward external Data Engineering Services for the unglamorous layer, meaning contracts and tests on critical tables, freshness monitoring, lineage a non-engineer can actually read, and ownership records that survive someone changing jobs.

What to do this quarter

Start with your five most-viewed dashboards and trace every table that feeds them. Put a test on each one. Nothing elaborate, just a not-null check, a row-count range and a freshness threshold.

Then name an owner for each of those tables. Name a person, not a team.

Lastly, measure how long it takes you to notice a break rather than how long it takes to fix one. Most teams have never measured the first number and are quietly shocked when they finally do.

None of this is exciting work. It does not demo well and it will never earn a slide in the quarterly review. But it is the difference between a data team that gets faster and a data team that only gets louder.

May your freshness checks fire before your CFO does.

About Author

Shefali Vasave

The 18 Percent Problem in Banking AI

Derek Francis — Sun, 06 Sep 2026 18:30:16 +0000

On 24 June 2026, Personetics published a survey of 902 banking professionals drawn from more than 30 countries. Close to 80 percent of them described fully operationalised generative AI as significant or transformational for their institution. Only 18 percent said the technology was actually running inside day-to-day operations.

That distance between belief and practice is the real state of AI in banking today. Almost everyone is convinced. Very few have shipped.

So what is holding the other 82 percent back? The same survey points at something far less interesting than model architecture. 56 percent named data silos between business lines as a major barrier, and 31 percent said their hardest challenge was keeping AI outputs accurate and compliant. Neither of those gets solved by a better model.

The bottleneck sits below the model

Consider what a single personalised offer has to touch inside a retail bank. Core banking, the card processor, the mortgage book, the CRM, the consent register and the marketing suppression list. Each of those systems was bought in a different decade by a different team working from a different definition of "customer", and none of them were designed on the assumption that a language model would one day need to read all six at once and then explain its reasoning to a regulator.

Personetics found that the average institution takes 12 weeks to move a personalised offer from concept to launch. Twelve weeks. That number has very little to do with AI capability and almost everything to do with how long it takes six systems to agree on who the customer is.

This is why the institutions doing well here look unglamorous from the outside. Teams that build Enterprise AI solutions for financial services that survive an audit tend to spend their first months on entitlements, lineage, retention rules and a single customer definition, long before anyone argues about which model to use.

The regulatory clock moved, but not by much

Europe handed out a small reprieve this year. On 6 May 2026 the EU institutions reached a provisional political agreement on the Digital Omnibus package, confirmed by member states on 13 May, which pushes compliance for stand-alone high-risk systems under Annex III of the AI Act from 2 August 2026 out to 2 December 2027. AI embedded in regulated products moves to 2 August 2028.

Does that buy banks breathing room? A little. But those changes only take legal effect once the Omnibus is formally adopted and published in the Official Journal, and 2 August 2026 remains live for the Article 50 transparency obligations, which is where most customer-facing banking use cases sit anyway.

Truth be told, sixteen extra months is not much when the obstacle is a data estate assembled over three decades.

What the leaders actually did

So what separates the institutions in production from the ones still demoing? Three habits keep showing up.

They picked one workflow and finished it. A narrow, well-instrumented use case in collections or complaint handling beats nine simultaneous pilots that all stall at legal review.

They wrote down what "wrong" looks like before launch. Accuracy is not a feeling. If you cannot describe the failure you are watching for, your monitoring will not catch it.

And they gave the work an owner with a budget. Innovation teams produce demos. Line-of-business owners produce systems that people actually use on a Monday morning.

Yes, none of this is new advice. It is roughly the same advice that applied to data warehousing in 2010 and to mobile banking in 2015, which is probably why it keeps getting skipped.

The 18 percent are not smarter than everyone else. They just started at the boring end.

About Author

Shefali Vasave

Modernizing Legacy Core Platform Without Disruption

Derek Francis — Fri, 07 Aug 2026 09:10:31 +0000

Core banking and payment platforms rarely die. That is the whole difficulty. A switch installed in the 1990s will keep authorising transactions long after the people who configured it have collected their pensions. It does not fall over. It just makes everything around it slower.

The tax that appears on no invoice

Legacy platform cost is mostly indirect, which is exactly why it survives budget season.

The direct costs are visible enough. Licences, specialised hardware, contractors who still know the language. Those get scrutinised every year and survive because replacing them looks expensive by comparison.

The indirect costs never appear as a line item. A new product takes eleven months instead of three because the core cannot represent the data model. A partner integration needs a custom file transfer because the platform has no usable API. Compliance reporting requires people reconciling extracts by hand because the system was never built to answer that question.

Each of those lands in someone else's budget. Added together they usually exceed the platform's running cost. Nobody adds them together.

Standards move whether you do or not

Then there is external pressure. Payment messaging standards have shifted toward richer, structured formats. Real-time settlement is now an expectation rather than a premium feature. Regulators increasingly ask for data that older schemas simply do not carry.

A platform that cannot represent structured remittance information or extended party detail does not fail a mandate loudly. It fails quietly, by requiring workarounds: mapping layers, truncation rules, manual exception queues. The workarounds become permanent.

Eventually they become the thing that has to be modernised too.

Why big-bang replacement keeps disappointing

The clean answer is to replace the platform outright. Buy or build something modern, migrate, decommission the old one.

Programs like this have a poor record, and not because the technology is hard. They struggle because the old system's behaviour was never fully documented. Twenty years of edge cases live in code nobody has read end to end. Every one of them was a real business requirement once. You find out which ones still matter during a cutover weekend, which is the worst possible time to find out.

They also outlast the organisation's attention span. Sponsors change. Priorities move. A four-year program with nothing visible until year three is a program waiting to be cancelled by someone who was not in the room when it was approved.

Phased, boring, effective

The approach that tends to work is considerably less dramatic. Put an API layer in front of the core so new products stop depending on its internals. Move the capabilities that change most often out first, running old and new in parallel until the new path has earned trust. Instrument everything, because you cannot safely change what you cannot observe.

Most platform modernization services follow some version of that sequence: assess and prioritise, design the target architecture, then migrate capability by capability instead of all at once. Slower on paper. Finishes more often.

The other benefit is optionality. Each phase delivers something usable on its own. If funding shifts or leadership changes, you are left with real improvements rather than a half-migrated estate and an awkward status report.

Start with the knowledge of the current system

If there is one thing worth doing before any of the engineering, it is capturing what the current system actually does. Not what the design documents claim it does. What it does, including the parts that look like bugs and turn out to be deliberate.

That work is tedious and gets no applause, and it is often the difference between a modernization that lands and one that gets written off. The people holding that knowledge are, in most institutions, closer to retirement than to their next promotion.
Ask them now, while asking is still an option.

About Author

Derek Francis

Derek manages content marketing at Opus Technologies, a domain-native engineering partner for banks, payment providers, and fintechs, and writes on the various aspects of financial institutions navigating change in a real-time, digital-first world.

Why Cloud Migration Bills Go Up Instead of Down

Derek Francis — Fri, 07 Aug 2026 08:33:21 +0000

Most cloud programs in banking begin with a savings number. Somebody models the current infrastructure spend, applies a percentage, and the business case gets signed. Two years later the workloads have moved and the monthly invoice is higher than the data centre it replaced.

Nobody lied in the business case. The math was just applied to the wrong variable.

Simply Moving Servers is Not Modernizing

The quickest way into the cloud is to pick up what you have and set it down somewhere else. Same application, same database, same architecture, different landlord. Projects that take this route usually finish close to schedule, which keeps everyone happy right up until the first full quarter of billing.

Here is the mechanic behind it. On premises, a server running at 12 percent utilisation at three in the morning costs nothing extra. You already bought it. The waste is invisible because it was capitalised years ago. In the cloud, that same idle capacity bills by the hour, every hour, indefinitely.

Autoscaling is supposed to solve this. It only works if the application can genuinely run on two instances overnight and forty at peak. A monolithic card management system written in 2004 cannot. It was designed for a world where scaling meant ordering a bigger box and waiting six weeks for delivery.

So a workload that was quietly inefficient in the data centre becomes an inefficiency you now pay for, line by line, on a statement someone in finance reads every month.

The Rising Bills Due to Unattributed Spend

Then there is the operating side. On-premises, a small number of people controlled provisioning, and getting new capacity meant asking them. In the cloud, a development team can spin up an environment on a Tuesday afternoon and forget it exists. Multiply that across forty teams and the rounding error stops being a rounding error.

Untagged resources are the usual culprit. If you cannot attribute spend to a team or a product, you cannot credibly ask anyone to reduce it. Finance sees one large number, engineering sees no signal, and the conversation goes nowhere for another quarter.
Storage tiering is the other one. Transaction logs that must be retained for seven years sit in hot storage because nobody wrote a lifecycle policy. That is a configuration choice, not a cloud problem, and fixing it is often worth more than any renegotiation with the provider.

Doing It The Right Way

The institutions that get this right tend to do three fairly unglamorous things.
They assess before they move. Which applications benefit from elasticity, which are fine where they are, and which should be switched off rather than migrated. A surprising share of most legacy estates consists of systems nobody has logged into in years.

Migrating those is pure cost transfer with extra project risk.

They pick refactoring targets deliberately. Not everything needs to become microservices. Usually a handful of components drive most of the compute cost and most of the release friction, and those are the ones worth breaking apart first. The rest can stay as they are until there is a reason to touch them. This sequencing is broadly why structured cloud modernization services open with a readiness assessment and a total cost of ownership model rather than a migration calendar.

And they give cost an owner. Someone reviews the number monthly and has the authority to turn things off. Without that, optimisation happens once, gets celebrated, and then decays over the following year.

The Uncomfortable Truth

Cloud economics only work when the software above them is built to take advantage of them. Elastic infrastructure underneath rigid applications gives you the cost profile of renting with none of the flexibility of owning.

That is a harder conversation than a migration timeline, because it means conceding the application portfolio needs work and not just the hosting arrangement. It is also the conversation that decides whether year two of the program reads better than year one.

Migration is a project. Cost efficiency is a habit. Programs that confuse the two end up explaining an invoice to a CFO who was promised the opposite.

About Author

Derek Francis

Why Reliable Software Can Still Slow Your Team Down

Derek Francis — Fri, 07 Aug 2026 08:25:15 +0000

Ask a payments team which of their applications causes the most trouble and you will sometimes get an odd answer. Not the one that breaks. The one that never breaks.
There is usually a system somewhere in the estate that has run without incident for years. It processes exactly what it is supposed to process. It has never caused an outage. And when a product manager asks for a modest change to it, the estimate comes back in months.

That gap between “it works” and “we can change it” is where a lot of engineering budget quietly disappears.

Reliability and changeability are different properties

Software can be extremely dependable and extremely rigid at the same time. It usually gets that way honestly. Every urgent fix, every regulator-driven patch, every acquisition whose data had to be mapped in over a weekend, all of it left a layer behind. None of those decisions were wrong at the time. The accumulation is the problem.

The symptoms are recognisable. A change to one module triggers regression testing across six others because nobody is confident about the coupling. Business logic lives in three places and the team disagrees about which one is authoritative. The test suite covers the paths that were easy to test rather than the ones that carry money.

None of this appears in an uptime report. It appears in the roadmap, as items that keep getting moved to next quarter.

The knowledge problem underneath

Then there is the human side. The people who built the system have moved on. What remains is documentation written for a version that no longer exists and two or three engineers who understand the thing by memory rather than by reading it.

This creates a bottleneck that compounds. Every change routes through the same small group. Onboarding a new engineer takes months because the knowledge is not written anywhere useful. And those engineers know they are indispensable, which is not a comfortable position for them either.

Teams often treat this as a staffing issue. It is an architecture issue wearing a staffing costume. Systems that are hard to reason about require specialists. Systems with clear boundaries and honest tests can be worked on by anyone competent.

So what’s the solution?

The instinct is to rewrite. Occasionally that is the right call. More often it becomes a two-year program that runs out of political capital in month fourteen, leaving you maintaining two systems instead of one.

The less satisfying alternative works more reliably. Find the parts of the codebase that change most frequently and improve those first. Wrap the stable parts in interfaces so callers stop depending on internals. Write tests around behaviour you are about to change rather than around everything. Extract one capability at a time and prove the pattern before scaling it across the estate.

That is the working logic behind most application engineering services engagements: concentrate effort on the code paths that carry delivery risk, and leave the rest alone until there is a reason to disturb it.

One more thing, frequently skipped: make deployment cheap. Teams instinctively avoid changing systems that are painful to release. Fix the release pipeline and some of the appetite for change comes back on its own, without anyone being told to be braver.

The metric that matters

Uptime tells you the system is running. It tells you nothing about whether the business can get what it needs out of it.
A better question for a quarterly review: how long does it take to get a small, well-understood change into production? If the answer is weeks, that number is your actual constraint, no matter how green the dashboards look.

Stable software that resists change is not an asset holding steady. It is an asset turning into a liability on a timeline nobody is tracking.

About Author

Derek Francis

How Fintechs Are Using Machine Learning to Cut Payment Fraud Without Killing Conversion

Derek Francis — Tue, 23 Jun 2026 07:00:21 +0000

Machine learning has quietly become the default tool for payment fraud detection, and the reason is a trade-off every fraud team knows too well. Tighten the rules and you block more fraud, but you also block real customers who just wanted to buy something. Loosen the rules and conversion improves, right up until losses start eating the margin. Push too far in either direction and someone in a leadership meeting is going to ask hard questions.

For a long time, the lever for managing this was static rules. Block transactions over a certain amount from a certain region. Flag anything that looks unusual. Rules are easy to understand, which is their charm, and rigid, which is their curse. A rule cannot tell the difference between a genuine customer behaving slightly differently and an actual fraudster. It just sees a threshold and reacts. The result is a pile of false positives, which is a polite way of saying you annoyed your best customers.

And that is not a soft cost. Industry analyses have repeatedly found that the revenue lost to false declines runs at well over ten times the value of the fraud actually stopped, and roughly a third of wrongly declined shoppers never come back to that merchant. A blunt rule does not just leak money on the fraud side. It quietly bleeds it on the conversion side too.

This is where machine learning changed the conversation. Instead of asking whether a transaction crosses a fixed line, an ML model asks how this transaction compares to everything it has learned about normal and abnormal behavior. It weighs dozens of signals at once.
Spending patterns, device fingerprints, timing, location, the rhythm of how someone types or navigates. No human-written rule can juggle that many variables in real time. A model can.

The payoff is sharper decisions. A good fraud model can spot the subtle signs of account takeover while waving through a loyal customer who happens to be shopping from a new city on holiday. That is the dream. Fewer real frauds slipping past, fewer legitimate buyers getting rejected at checkout. Both numbers move in the right direction at the same time, which static rules could never quite manage.

But machine learning is not a magic switch, and pretending otherwise is how teams get burned. A few realities deserve respect.
Models drift. Fraud patterns shift constantly because the people committing fraud adapt. A model trained on last year's behavior slowly goes stale.

Without retraining and monitoring, your sharp detector quietly turns dull, and you might not notice until losses climb. The scale of this is well documented. A 2023 McKinsey survey found that around 40% of companies deploying AI models saw noticeable performance degradation within the first year, driven by exactly this kind of drift. Gartner has put the figure higher still, with a majority of enterprises reporting measurable model degradation inside twelve months and most failing to catch it early. In fraud, where adversaries adapt on purpose, that clock runs faster.

False positives still need a human story. Even a great model gets things wrong sometimes. What matters is the experience around the rejection. A smooth step-up verification, a quick way for a wronged customer to recover, a feedback loop that teaches the model from its mistakes. The model is one piece of a larger system, not the whole thing.

Explainability is not optional in finance. When a regulator or a customer asks why a transaction was blocked, "the model said so" is not an answer anyone accepts. This is not a preference, it is supervisory expectation. The Federal Reserve’s SR 11-7 guidance on model risk management requires regulated institutions to validate and explain the models behind their decisions, and the EU’s Digital Operational Resilience Act adds its own weight to model governance and accountability for institutions operating in Europe.

“The model said so” does not satisfy either. Fintechs need models whose decisions can be interpreted and defended. This is one of the reasons purpose-built AI & ML services for financial institutions tend to outperform generic, off-the-shelf fraud tools. The domain context, the regulatory weight, and the cost of a wrong call are simply different in payments than they are anywhere else.

The fintechs winning at this are not the ones with the fanciest algorithms. They are the ones who treat fraud detection as a living system. They tune for the business outcome, not just the accuracy score. They measure the cost of a blocked good customer alongside the cost of a missed fraud, and they optimize the balance rather than chasing one number.

Because in the end, fraud prevention is not really about catching every bad actor. It is about protecting the business while keeping the door open for the people you actually want to serve. Machine learning, used with judgment, is the closest thing the industry has found to threading that needle. The technology is impressive. The discipline is the moat. The fintechs that built that discipline early are the ones now posting authorization rates their competitors are still trying to explain, and that gap only widens from here.

Derek Francis

Why Payment Data Pipelines Break Under Real-Time Load (And How Banks Fix the Latency Problem)

Derek Francis — Tue, 23 Jun 2026 06:45:34 +0000

Payment data pipelines fail in ways that ruin a payments engineer’s week, and the failures rhyme. The dashboards froze. Fraud scores arrived after the transaction had already cleared. Settlement reports came in stale. Nobody slept. The frustrating part is that the same data architecture had run fine for years. So, what changed?

The honest answer is that batch thinking does not survive contact with real-time payments. A lot of banks built their data foundations in an era when nightly jobs were good enough. Load the warehouse overnight, run the reports in the morning, move on. That rhythm worked when money moved slowly. It does not work when a customer expects an instant confirmation and a fraud engine has milliseconds to make a call.

Here is where things crack. Real-time payment rails push a constant stream of events instead of a tidy nightly dump. Your pipeline now has to ingest, transform, and serve data while transactions are still happening. Add ISO 20022 into the mix and the pressure climbs. ISO 20022 messages are rich. They carry far more structured detail than the old formats, which is wonderful for analytics and miserable for a pipeline that was never designed to parse that much context at speed. This is not a fringe concern either. Swift reported that by the time its MT/ISO 20022 coexistence period closed in November 2025, around 80% of daily traffic was already running on the ISO 20022 format, with more than 3.1 million of these messages exchanged every day. The rich-data era is the default now, not the roadmap.

Then there is the fraud-scoring window. Fraud models need fresh features. Account behaviour over the last few minutes, velocity checks, device signals. If your pipeline takes thirty seconds to surface that data, the fraud decision is already too late. You are essentially detecting fraud after the loss.

That gap between when data is created and when it becomes usable is the silent killer in most payment systems. And the cost of getting it wrong runs in both directions. Javelin Strategy & Research estimated that wrongly declined transactions cost merchants roughly $118 billion in a single year, against about $9 billion in genuine card fraud over the same period. When your features arrive late, you do not just miss fraud, you also misjudge good customers, and the second mistake is the more expensive one. So how do teams actually fix it? A few moves matter more than the rest.

First, separate the hot path from the cold path. Not every piece of data needs to move in real time. Fraud scoring and authorization need low latency. Quarterly trend analysis does not. Mixing both in one pipeline means the slow stuff drags down the fast stuff. Splitting them lets each run at its own pace.

Second, treat data domains as first-class citizens. Payments, cards, risk, and customer data each have their own shape and their own owners. When you map these domains deliberately instead of dumping everything into one lake, you get clarity, and clarity is what keeps latency predictable. This domain-driven approach is exactly the kind of foundation that strong data engineering services are built to deliver, especially in a payments context where the cost of a stale record is measured in real money.

Third, build observability in from the start. You cannot fix what you cannot see. Most pipeline failures are not sudden. They creep. A queue backs up a little, then a little more, until one Monday morning everything is an hour behind. Good observability catches the creep early, before it becomes a 2 a.m. incident.

Fourth, push governance into the pipeline rather than bolting it on afterward. In financial services, a fast pipeline that cannot prove its data lineage is a liability. Embedding governance and quality checks at the point of ingestion means you stay both quick and auditable. This is no longer just good hygiene. Under the EU’s Digital Operational Resilience Act, in force since 17 January 2025, financial entities can be fined up to 2% of global annual turnover for failures tied to operational resilience and data lineage. A pipeline that cannot show where a record came from and how it was transformed is now a regulatory exposure, not merely a technical one.

None of this is glamorous. There is no single product you buy that makes the latency problem disappear. It is architecture, discipline, and a willingness to stop pretending that batch habits scale into a real-time world.

The banks that get this right share one trait. They stopped treating data as something you store and report on later. They started treating it as something that flows, continuously, and engineered for that reality. The latency problem is rarely a hardware problem. It is a design problem. And design problems, unlike server crashes, can actually be solved for good.

Derek Francis