DEV Community: Audacia

Fabric IQ and Ontology: Giving Data a Shared Business Meaning

Audacia — Thu, 13 Aug 2026 07:30:00 +0000

Every organisation has some version of the same challenge:

What counts as an "active customer"?
Does revenue get measured net or gross?
Does a "sale" include returns?

The answer exists somewhere: in a finance director's head, in an outdated document, or as a DAX measure in a Power BI model that only one person knows how to maintain. A new analyst joins the team, asks the question, and gets a different answer depending on who they ask.

This ambiguity has always carried a cost in wasted time and inconsistent reporting. However it becomes sharper with AI agents in the mix, because an agent cannot ask a colleague to clarify what "active" means. It works only with the definitions available to it, and inconsistent definitions produce inconsistent answers.

This is a live problem across the industry, not one specific to any single vendor. Databricks recently took Unity Catalog Business Semantics to general availability and open-sourced its core implementation in Apache Spark, explicitly framing the effort around giving analysts, engineers and AI agents a single trusted source of metric definitions, and Snowflake released Semantic Views for the same purpose.

Microsoft's answer sits within Fabric IQ, a workload in Microsoft Fabric built around a feature called Ontology, currently in public preview. This article looks at what Fabric IQ and Ontology are, how an ontology differs from the semantic models many organisations already use, where the approach earns real business value, and where the platform trade-offs sit. It follows on from our earlier piece, A Look at Microsoft Fabric, which covered Fabric's approach to unifying data across engineering, analytics and data science on a single platform.

What Fabric IQ Is

Fabric's original pitch was about unifying where data lives. OneLake gave engineers, analysts and data scientists a shared foundation to work from, removing the friction of exporting data between disconnected tools.

Fabric IQ extends that unification a layer further, toward unifying what the data means. It brings business context into the platform through two core items, Ontology and semantic models, both built on OneLake. Semantic models continue to deliver trusted metrics for reporting. Ontology defines the shared business language behind those metrics. A graph component supports relationship and impact analysis across entities, and data and operations agents draw on all of it to interact with live and historical data through consistent definitions rather than raw schemas.

What an Ontology Actually Is

In Fabric IQ, an ontology is a structured description of the concepts that make up a business and the relationships between them. A customer is a concept. An order is a concept. An order belongs to a customer. Each concept carries defined properties, a clear description, and relationships to other concepts.

The useful comparison for anyone who has built a semantic model in Power BI is to notice what's similar and what's different. A semantic model is a technical artefact with tables, columns, relationships and DAX measures, tied to how the underlying data is physically structured. An ontology operates a level up, describing what a concept means independent of how the data behind it is stored. A single ontology concept, such as "Customer", might map to one table, several tables across different systems, or a calculation spanning multiple sources.

Ontologies and semantic models are designed to work together rather than replace one another. Ontologies can be generated directly from existing Power BI semantic models, so organisations with mature reporting already in place have a starting point rather than a blank page. An ontology is built from a small number of components: entity types, the reusable logical model of a concept such as Customer or Shipment; entity instances, the concrete occurrences of that concept populated from real data; and properties and relationships that attach detail, such as a customer's name or the fact that a customer places an order. Underlying all of this is a queryable graph, built from data bindings and relationship definitions, that allows relationships to be traced rather than buried inside join logic.

Why This Matters for AI

Reporting and analytics have functioned without a formal ontology layer for years, because a trained analyst can navigate ambiguity and apply judgement when a definition is unclear. However, AI agents need a concrete, structured answer about what a question means before they can act on it, and cannot rely on judgement to fill the gap.

Agents that draw on an ontology see business entities, their relationships and the rules that govern them, rather than raw tables and columns. This gives them a structured map of the business to reason over, producing more consistent, explainable responses.

Ontology also supports a natural language query layer, NL2Ontology, which converts business questions phrased in plain language into structured queries against the ontology's definitions, respecting the filters, units and validity rules already published there rather than guessing at joins. Business rules embedded directly in the ontology can trigger alerts or automated actions when a defined condition occurs, such as inventory falling below a threshold, without custom code.

The Business Case: Cost, Risk and Speed

An ontology layer earns its place through three mechanisms.

Cost reduction through reuse:

Once a concept such as Customer is defined centrally, every subsequent project draws on that definition rather than rebuilding it. A churn model, a customer service agent and a regulatory report can all reference the same entity with the same properties and constraints, removing the duplicated engineering effort of maintaining separate versions of "what a customer is" across teams. This is the same logic Databricks cites from early Unity Catalog Business Semantics adopters, where standardising metric definitions cut the ongoing workload of reconciling conflicting numbers across dashboards.

Risk reduction through traceability:

Every ontology-bound answer traces back to its source data and definition. In audit-sensitive and regulated environments, an agent that can show which entity, property and rule produced a given answer sits in a materially stronger position during a compliance review than one that cannot account for its own reasoning. This matters increasingly as agents move from producing dashboards to taking operational actions.

Speed to value in agent deployment:

Building an AI agent typically involves substantial work translating business intent into queries against unfamiliar table structures, work usually repeated for every new agent. With shared business definitions in place, that translation is done once and inherited by everything built afterwards. Microsoft's own framing of this is that agents gain the operational context needed to understand how a business runs, making their behaviour explainable rather than dependent on how one developer happened to interpret the schema.

Taken together, these mechanisms point toward a platform that becomes more valuable with each project built on it, rather than one where every new initiative starts its definitional work from zero.

How to Prepare Your Data for Fabric IQ

Understanding what an ontology is and why it matters naturally raises a more concrete question: what does an organisation actually need in place before it can build one?

Data that already lives in OneLake:

Ontology only binds to data inside Microsoft Fabric - static data in OneLake lakehouse tables, or time series data in OneLake or an Eventhouse. Data sitting outside Fabric needs to be brought into OneLake first, whether through direct ingestion, mirroring or shortcuts, before it can be modelled as part of an ontology.

Two starting points, depending on what already exists:

Organisations with an established Power BI semantic model can generate an ontology directly from it, inheriting entity types and relationships from work already done. Organisations without one build the ontology manually, binding entity types directly to OneLake tables and defining relationships from scratch. The first path is considerably faster.

Clean, well-structured source data:

Data should already be organised and have gone through the ETL a business requires before binding begins, with all the information needed to model it present in the source. Ontology doesn't clean or transform data on the way in. It expects data that's already fit for purpose, meaning governance and quality foundations matter as much here as anywhere.

Technical requirements before design starts:

Ontology has specific requirements for the lakehouse tables it binds to. They must be managed tables without OneLake security enabled, and without column mapping enabled, which can be triggered automatically by certain naming conventions or import-mode semantic model tables. These are the kind of details that catch teams out mid-build rather than at the planning stage, so it's worth an early technical review against an organisation's actual table structures rather than assuming compatibility.

A defined entity key for every concept:

Each entity type needs at least one property, or combination of properties, that uniquely identifies each record, established during static data binding before any time series or streaming data can be layered on top. This is standard data modelling discipline, but it means the exercise of properly defining a business concept, rather than simply pointing at a table, has to happen before any binding work begins.

Considerations:

It's worth considering that adopting Fabric IQ's ontology means investing engineering effort in Microsoft's specific modelling approach, tied to OneLake and the Fabric tenant. That is a commitment for anyone weighing platform risk, and it sits within a wider industry response to the same problem, not a decision made in isolation.

Databricks has taken a different position on portability, open-sourcing the core implementation of Unity Catalog Business Semantics in Apache Spark and stating explicitly that its goal is business meaning that is open and portable across an existing ecosystem, without lock-in.

Snowflake's Semantic Views take a similarly warehouse-native approach, while tools such as dbt's MetricFlow are built to be vendor-agnostic by design, generating SQL that runs against whichever warehouse an organisation uses rather than tying metric definitions to one platform.

None of this means Fabric IQ is the wrong choice for organisations already committed to the Microsoft stack. It does mean the decision to build into it should be weighed against the same questions any platform-specific investment deserves: how difficult would it be to migrate this definitional work elsewhere, and how much of the organisation's data strategy is already tied to this vendor regardless of this specific feature.

What This Means for Tech Leaders

Fabric IQ and Ontology are in public preview, not general availability, and the tooling reflects that stage. Defining concepts, properties and relationships currently involves more manual configuration than the finished product is likely to require, and for an enterprise modelling hundreds of concepts, that overhead is considerable.

Readiness for this depends on groundwork that has nothing to do with Fabric IQ specifically. A mature Power BI semantic model gives an ontology something to bootstrap from rather than starting from scratch. Established data governance, with clear ownership and lineage, gives the ontology layer trustworthy data to bind to. Organisations without that foundation are likely to find that an ontology surfaces the same definitional disputes it's meant to resolve, in a new location rather than a resolved one.

There's also a skills dimension worth planning for. Ontology modelling is a distinct discipline from both data engineering and traditional BI development, closer to the kind of conceptual modelling more familiar in data architecture and knowledge management. Teams comfortable building semantic models in Power BI will find some concepts transfer, but should expect a genuine learning curve rather than treating this as an extension of existing DAX skills.

Enterprise controls are maturing alongside the feature, with Fabric IQ now supporting granular sharing and permissions management for ontology items, and Azure Private Link integration for network isolation, which is a reasonable signal that Microsoft is treating this as a governed enterprise capability rather than an experimental add-on.

Conclusion

Fabric's original contribution was to unify where enterprise data lives. Fabric IQ, through Ontology, is Microsoft's attempt to unify what that data means, and it arrives alongside comparable moves from Databricks and Snowflake that point to the same conclusion from different directions: AI agents need governed business meaning, not just access to tables.

For organisations already running mature Power BI semantic models and established data governance on Fabric, a scoped pilot ontology covering one well-understood domain is a reasonable step to take, ahead of general availability, precisely because the bootstrap path from existing semantic models lowers the initial cost of finding out whether the approach fits.

For organisations earlier in their data governance maturity, or invested primarily in a different platform, the more useful action is tracking how Fabric IQ, Unity Catalog Business Semantics and Snowflake's Semantic Views mature over the next year, rather than committing engineering effort to any one of them today. The underlying idea, giving data a consistent, governed meaning that both people and machines can share, is worth planning for regardless of which vendor's version an organisation eventually adopts.

A Testing Strategy for Legacy Modernisation

Audacia — Thu, 06 Aug 2026 07:30:00 +0000

Every legacy modernisation programme runs into the same uncomfortable question sooner or later: how do you prove a new system works correctly, when nobody can fully define what "correctly" means for the one it's replacing? The specification lives in code written years ago, in the muscle memory of staff who've long since moved on, and in workarounds nobody documented because they were never meant to be permanent.

The scale of the risk this creates is well established. The Standish Group's CHAOS Report, drawing on a database of more than 25,000 projects, found that only 53% of legacy modernisation projects met their definition of success: delivered on time, on budget, with a satisfactory result. The remaining 47% were challenged or failed outright. Industry analysis of ERP modernisation specifically shows a similarly stark picture. Gartner predicts that by 2027, more than 70% of recently implemented ERP initiatives will fail to fully meet their original business goals, with as many as 25% failing outright.

Every gap in testing strategy surfaces later as a cost somewhere. It might be rework, when a defect discovered post-launch has to be fixed under pressure; compliance exposure, when a regulator or auditor asks what changed and nobody can answer with evidence; or a delayed go-live, when a critical discrepancy is found too late to fix without pushing the release date. Each of these lands on a desk as a budget overrun, a reputational risk, or both. A deliberate testing strategy converts that difficulty into decisions made early and visibly, giving teams a documented basis for what changed and why.

This article looks at why legacy systems resist conventional testing approaches, how characterisation testing addresses that problem directly, what a testing strategy for modernisation needs to look like, and how to measure whether it's working. It's a natural companion to our earlier piece, Managing Hidden Waterfalls in Legacy Modernisation Projects, which looks at the delivery and methodology side of this same challenge; this piece goes deeper into the testing discipline itself.

Why Legacy Systems Resist Conventional Testing Approaches

Conventional testing assumes a specification exists, whether a requirements document, a user story, or at a minimum a shared understanding of what correct behaviour looks like. Legacy modernisation removes that assumption. The system itself becomes the closest thing to a specification, and it's an unreliable one: business logic embedded in code that was never formally documented, exceptions handled through branches and workarounds nobody captured at the time, and user behaviour that has adapted around system quirks until those quirks became de facto requirements nobody thinks to mention.

This creates a genuine testing problem alongside the documentation gap. A test asserts that a given input should produce a given output, but writing that assertion requires knowing what the correct output is. When the only available answer is "whatever the old system currently does," testing becomes an exercise in discovering behaviour that was never written down in the first place.

The people who could fill that gap through institutional memory are often unavailable by the time a modernisation programme starts. Systems that have been running for a decade or more have usually outlived several rounds of staff turnover, and the workaround someone built in 2014 to handle an edge case in a specific regional process may now be triggering silently on a small fraction of transactions, invisible until someone tries to replace the system and it stops happening.

Characterisation Testing: Capturing Behaviour Before You Change It

This is precisely the problem characterisation testing was designed to solve. The term comes from Michael Feathers' influential work on legacy code, which defines a characterisation test as one that documents the actual behaviour of a piece of code, rather than asserting what that behaviour should be. Instead of writing a test that says "this function should return X," a characterisation test observes what the function currently returns for a given input and records that as the baseline, regardless of whether the result looks correct.

This represents a genuine shift in what testing is for. It stops being a check against a known standard of correctness and becomes a check against continuity: does the system still do what it did yesterday? That distinction matters enormously in legacy modernisation, because the immediate goal is to make existing behaviour visible, so a deliberate decision can be made about which parts of it to preserve and which to change.

In practice, building characterisation tests means running the legacy system with a wide, representative range of real inputs, recording its outputs, and turning those observed input-output pairs into an automated regression suite. That suite then becomes the safety net against which any replacement or refactored system is measured. If the new system produces a different output for the same input, that difference gets surfaced explicitly, as either an intentional improvement or an accidental regression, months before it reaches a customer or an auditor.

Building a Testing Strategy for Modernisation

Characterisation testing gives a starting point, but a full testing strategy for legacy modernisation needs several layers working together.

Characterisation tests on the legacy system

As described above, this establishes the behavioural baseline before any redesign work begins. This should happen early, ideally during discovery, not retrofitted once development is already underway.

Parallel run and shadow testing

Running the old and new systems side by side against the same live inputs, then comparing outputs, catches discrepancies that static test suites miss, particularly those arising from data conditions or usage patterns nobody anticipated when writing test cases. This is especially valuable for financial, regulatory or high-volume transactional systems, where the cost of an undetected behavioural difference is high.

Risk-prioritised regression coverage

Not every piece of legacy behaviour deserves equal testing investment. Coverage should be weighted toward the areas carrying the highest business, regulatory or financial consequence if something breaks.

A deliberate decision process for behavioural change

Every discrepancy between old and new behaviour needs to be explicitly classified: is this a bug in the legacy system that we're correcting, or a regression we've accidentally introduced? Without this discipline, teams tend to default to reproducing every legacy quirk faithfully, which is exactly the trap "Managing Hidden Waterfalls" describes: testing effort shifting entirely toward proving parity with the old system, at the expense of validating that the new one actually solves the problem it was commissioned to solve.

The Role of AI in Legacy Testing

AI-assisted tooling has a real, measurable role in this work, and it's worth being specific about where that role sits. Generating characterisation tests at scale, by running legacy code against large volumes of synthetic or historical inputs and automatically capturing outputs, is exactly the kind of repetitive, high-volume task these tools are well suited to. The same applies to analysing legacy codebases to extract embedded business rules, map dependency chains, and flag areas of dead or redundant code, work that would otherwise consume significant manual analyst time on systems where documentation is thin or absent.

The 2026 State of Testing research found that large enterprises are already leaning on this capability heavily, with 65.5% using AI specifically for regression suite and script maintenance work, a meaningfully higher figure than smaller organisations report. This lines up with where the technique is genuinely strongest: sustaining and extending test coverage on systems that already have some automated baseline in place.

Deciding which legacy behaviours represent genuine business rules worth preserving requires domain knowledge and business context that sits with the team. The tooling can surface candidates at scale; the judgement about which ones matter stays a human responsibility.

Measuring Whether the Strategy Is Working

A testing strategy needs a way to show whether it's succeeding, beyond the general sense that testing volume has increased. Two measures matter most in legacy modernisation specifically.

Parity rate against the characterisation baseline:

This tracks the proportion of characterisation tests where the new system's output matches the legacy system's recorded behaviour, and the proportion where it deliberately diverges. A high, stable parity rate combined with a small, explicitly reviewed set of intentional divergences is a strong signal that the programme understands what it's changing and why. A parity rate that's low or declining late in delivery is an early warning that the team is discovering legacy behaviour later than it should.

Defect escape rate post-migration:

This measures how many behavioural discrepancies are caught before go-live versus reported by users or discovered in production afterwards. It gives a clearer picture of testing effectiveness than raw test count or coverage percentage, both of which can look healthy while missing exactly the undocumented edge cases legacy systems are prone to hiding.

Why Testing Needs to Start at Discovery

The organisational pattern that undermines legacy modernisation testing most consistently is treating it as something that happens at the end, once development is largely complete. By that point, the programme has usually already committed to an architecture and a delivery timeline that leaves little room to properly characterise the systems it depends on, and testing becomes a compressed, high-pressure activity squeezed into the final weeks before go-live.

Characterisation work belongs alongside early technical discovery, feeding directly into design decisions about what the new system needs to replicate and what it's free to change. This also gives programme leadership something more useful than a green sprint burndown chart: concrete evidence, in the form of a growing characterisation suite, of how well the team actually understands the system it's replacing.

Conclusion

Legacy modernisation testing starts from a genuinely different premise than testing greenfield applications. There's no clean specification to validate against, only years of accumulated behaviour that nobody wrote down. Characterisation testing gives teams a disciplined way to make that behaviour visible before they change it, turning an implicit, undocumented specification into an explicit, testable one.

Treating testing as a discipline that shapes a programme from its earliest stages is good practice for any software delivery, not something unique to legacy modernisation. What legacy modernisation changes is the cost of skipping it. Without a written specification to fall back on, there's no shortcut back to "what the system is supposed to do" once development is already underway. Organisations that invest in characterising legacy behaviour early, prioritise coverage by business risk, maintain a deliberate process for classifying behavioural change, and measure parity and defect escape rate are the ones that avoid discovering, late and expensively, that the system they built doesn't actually handle the behaviour the business depends on.

From Proof of Concept to Production: What AI Projects Actually Require

Audacia — Thu, 30 Jul 2026 07:30:00 +0000

The gap between a proof of concept and a production system is where a significant proportion of AI initiatives fail to progress. Understanding what that transition involves, and planning for it from the outset, is one of the more important things a technology leader can do before committing investment.

This blog draws on Episode 1 of Technically Speaking and covers the lessons our guests have taken from productionising AI.

Overview

Clear definition of success

In conventional software projects, requirements can be defined in a reasonably deterministic way. There is a clear vision for what the system needs to do, and delivery can be measured against it. However, in AI projects, outputs are non-deterministic: the same input can produce different results depending on context, phrasing and model behaviour.

This changes how teams need to approach scoping, measurement and validation. The question of project success shifts from whether an output meets a specification to how accurately it performs against a defined benchmark. This means that a 'gold standard' set of reference outputs, against which model performance can be measured, needs to be established before development begins. A model averaging a score of 8 out of 10 dropping to 7.5 following a model upgrade is concrete, actionable information. It makes it considerably easier to establish whether a project is ready to proceed to the next stage.

The rationale extends beyond the initial build. Models get upgraded, providers deprecate versions and data changes over time as business processes evolve. Without an evaluation framework in place from the start, there is no reliable basis for understanding whether a change to the model or the underlying data has improved performance, degraded it or introduced different failure modes.

Data readiness

Data readiness is a more front-loaded concern in AI projects than in most software delivery projects. Access to data of sufficient quality and volume needs to be established before a project begins, because without it the project cannot proceed. Finding out early is considerably less costly than finding it out late.

This means data assessment belongs in pre-discovery. Organisations that are mapping out where AI could add value across their operations often identify pockets of data that could support several potential use cases, but which first need to be structured, cleansed or made accessible. Use case road mapping should factor in the data engineering work that may be needed before any AI development can begin.

For more traditional machine learning applications - predictive and classification models, rather than large language model applications - the data dependency is even more pronounced. These models rely on historical data to classify or predict future outcomes, which means the first question is often not whether AI can solve the problem, but whether the organisation is even recording the right data. In some cases, the answer is no, and the prerequisite work is building the capability to capture it. The implication is that the short-term plan may be a data engineering project, not an AI project, which can introduce a lag of months before an AI project can begin.

Planned productionisation

Productionisation is often treated as the final step in an AI project when it is more accurately described as a distinct phase with its own requirements, timelines and risks. In regulated environments in particular, it can be more demanding than the development work that precedes it.

Going live as a service typically requires satisfying formal transition criteria: signoffs from security, testing and operations teams, alongside documentation that may not have been prioritised during a more exploratory proof of concept phase. For AI systems specifically, cyber security testing introduces challenges that have no real equivalent in traditional software projects. AI applications, such as large language models, present an expanded attack surface, including vectors such as prompt injection, data poisoning and exploits that target model behaviour rather than conventional code vulnerabilities. The specialist skills required to assess these risks are still relatively scarce, which means that security review can become a material bottleneck if it is not planned for early in the project timeline.

Adoption and change management

On the human side of productionisation, a technically sound AI system can still fail to deliver value if adoption is handled poorly.

Two practices make a consistent difference to adoption. Firstly, involving sceptics during the design process, rather than waiting until rollout, surfaces the objections that are most likely to obstruct take-up and gives the team an opportunity to address them before they become entrenched. If those sceptics can be converted into advocates through that process, they tend to be considerably more credible and effective champions than people who were enthusiastic from the start.

The second is to think carefully about how users interact with the system. Presenting people with an open-ended prompt interface is not always the most effective approach, particularly for users who are not accustomed to interacting with technology through natural language. Structured inputs, guided forms and clearly defined interaction patterns often produce better outcomes and reduce the amount of training required.

This reflects the broader reality of how AI projects are constituted. A substantial proportion of what gets built in an AI initiative is conventional software engineering and data engineering. The AI model is often a relatively contained component within a larger system, and the quality of that surrounding system matters considerably to whether the overall solution delivers its intended value.

Governance

A common assumption is that governance and guardrails slow down AI adoption. Experience from organisations that have deployed successfully tends to suggest the opposite - clear governance is often what makes adoption possible at all.

Defining clearly what data can be used with AI tools and making explicit that individuals remain accountable for any output they act on, gives people a basis for using the technology with confidence. Without an approved tooling framework, uncertainty about what is permitted can lead many people to avoid the tools entirely, or use publicly available tools, potentially exposing sensitive data to systems outside the organisation's control and contributing to external model training in ways that may not be intended or sanctioned.

For people to remain accountable, the right processes need to be in place. AI systems in most production contexts are not making autonomous decisions. They are reducing the effort involved in reaching a decision or producing a draft output that a human reviews and acts on. Keeping humans in the loop and ensuring that outputs are always editable maintains accountability for AI decisions - and tends to support adoption, because users are more willing to trust a system they can oversee and correct.

Ongoing ownership and operational costs

An AI system needs ongoing ownership, monitoring and support after go-live. Models change, sometimes during active development, and organisations need a process for evaluating and responding to those changes without disrupting production systems. Additionally, data drifts over time and performance can degrade in ways that are not immediately visible.

The cost implications also need to be considered carefully. Infrastructure costs that appear manageable during a proof of concept, running against limited data at low volumes, can scale significantly when the system is handling production-level usage. Token costs for large language model applications are a specific area to watch. Provider pricing has been subsidised during the early adoption phase, and there are reasonable grounds to expect that will change as the market matures. Stress-testing the return on investment against scenarios involving meaningful price increases is a sensible part of any business case.

Prioritised use cases

Early AI deployments, when they go well, can create increased internal demand. That is a positive outcome, but it requires a disciplined approach to prioritisation of use-cases.

It is worth being clear about what kind of value is being pursued. For example, in most cases, the goal is not to reduce headcount but to free up specialist capacity for higher-value activity. In environments where people take years to develop the expertise the organisation depends on, the productivity argument for AI is about removing low-value, time-consuming work from those specialists, not replacing them.

Pre-discovery benchmarking is a useful tool here: measuring how long a task currently takes and using that as the basis for estimating the value of any efficiency gain. Some teams find it difficult to express their work in those terms, but even approximate figures provide a consistent basis for comparing use cases and making a reasoned case for where investment will have the most impact.

It also needs to be considered whether AI is the right tool for the value the organisation is trying to add. It could transpire that a traditional software engineering project is needed, or that a tool already exists, in which case investment is better directed elsewhere.

Ability to stop

The decision to continue an AI project needs to be subjected to sufficient scrutiny at the right points. Insufficient upfront business analysis means teams can find themselves well into a build before discovering that the underlying use case is not well-suited to an AI approach, or that a more straightforward software solution would deliver equivalent value with considerably less complexity.

What protects against this is building genuine decision points into the project structure: stages at which the question of whether to continue is asked honestly, with objective criteria for the answer. Phase-by-phase contracts, rather than large fixed-scope agreements, support this kind of structured reassessment. So does a regular delivery cadence that gives stakeholders something concrete to evaluate at each stage, rather than deferring judgement to a single release event.

It becomes progressively harder to stop a project as investment accumulates, regardless of whether the evidence supports continuing. Teams that handle this well are typically operating with a leadership culture that treats an early, well-evidenced decision to stop as a legitimate and valuable outcome, rather than a failure to be managed.

What's next for AI productionisation?

Agentic AI
Agentic AI, in which systems take sequences of actions autonomously rather than producing outputs for human review, is likely to be a significant area of development and investment over the next few years. The potential value is substantial, but so are the governance and risk management questions it raises. The sensible starting point is human-in-the-loop systems that allow organisations to gather real evidence about how reliably the AI performs at each decision point, and to use that evidence as the basis for decisions about where further automation is appropriate.

LLM Token FinOps
The economics of large language model deployment are also likely to evolve. As provider subsidies are withdrawn, organisations that have not built robust evaluation frameworks will find it difficult to assess whether their existing tools remain cost-effective, or whether alternative approaches such as smaller, locally-hosted models would better serve their needs. Those that can measure performance objectively will be better placed to make those judgements.

Standard practice
Looking further ahead, the expectation is that AI will become a standard part of how organisations design and improve their services, rather than a distinct category of initiative requiring its own governance and planning apparatus. Getting to that point requires working through the current phase carefully, building the patterns and the institutional knowledge that make AI deployment reliable and repeatable, rather than treating each new project as an experiment conducted from scratch.

This was an episode of Technically Speaking, Audacia's podcast series for technology leaders navigating real decisions in complex organisations.

Future episodes will explore the challenges shaping technology leadership today, whether that is navigating the governance questions that come with AI, rethinking what technology leadership looks like or building the infrastructure and culture that makes it all possible.

Listen on your preferred platform here.

If there are topics you'd like to see covered, we'd welcome your input at podcast@audacia.co.uk

Cybersecurity: The Human Problem AI Can't Fix

Audacia — Thu, 23 Jul 2026 11:32:34 +0000

This blog captures the key themes from the launch event of Technically Speaking, Audacia's new podcast - held as part of Leeds Digital Mini Festival.

Technology and security leaders from financial services, government and software came together to explore the 'Human Problem' in cybersecurity. Chaired by Philip White, Managing Director at Audacia, with panelists Rob Devany (Leeds Building Society), Nayza Fioritta Neves Ferro (Lhasa), Jennifer Anderson (National Wealth Fund) and Richard Brown (Audacia). The views here reflect the collective discussion rather than any single individual or organisation.

Cybersecurity has topped the CIO priority list for four consecutive years. Budgets have grown, frameworks have matured and security tooling has never been more advanced. However, the data tells a consistent story: between 68% and 95% of data breaches involve a human element, and 66% of CISOs say their greatest vulnerability is still people – whether that's through a compromised credential, a phishing click or a misconfigured system.

Start with Risk Appetite

The starting point for any security programme should be a conversation at board level about what level of risk the organisation is willing to accept.

Different organisations will have different risk appetites; a government body managing critical infrastructure funding has a very different threat profile to a fintech operating under competitive delivery pressure. The important thing is that everyone understands and agrees with the defined appetite.

A common failure of communication is what practitioners call the "watermelon" problem: green on the outside, red on the inside. An organisation declares a robust risk stance at board level, but day-to-day behaviour doesn’t align. The danger is that this misalignment falls hardest on whoever is accountable for security - when something goes wrong, the declared appetite and the actual practice are suddenly at odds.

Coordinating risk posture and everyday actions means going beyond a headline position and working through what the agreed appetite looks like in practice: how data is shared, how suppliers are granted access and what employees can and can't do on company devices. These operational details are what gives the policy meaning.

Internal Threats, External Threats and the Supply Chain Problem

Practitioners tend to think about their threat landscape in two buckets: internal threats and external threats. Both require active management, and the discipline of RAG-rating each area against a clear framework helps boards understand the current position without drowning in technical detail.

Supply chain sits uncomfortably across both; a third-party supplier with access to internal systems is an internal risk managed through an external relationship. Organisations need effective security controls in place with suppliers, even when delivery timelines create pressure to cut corners. One approach is to issue suppliers with managed virtual environments rather than allowing them to connect from their own hardware.

For software development teams, the harder problem is open source. It is practically impossible to formally vet every dependency in a modern codebase, and yet open source is foundational to how software gets built. One counter-intuitive shift is in how teams handle dependency updates. For years, applying updates immediately was considered good practice but that assumption no longer holds. Threat actors have learned to inject malicious code into open-source packages and push them as updates, meaning a developer who patches on day one may be the first to pull compromised code into their codebase. Tools like Dependabot now support cool-down windows, giving security vendors time to identify and flag suspicious packages before they reach production.

There is also a simpler lever that gets less attention: reducing the number of dependencies in the first place. Every library added to a codebase is a maintenance commitment and a potential attack surface. In an era when AI can generate functional code quickly, the case for pulling in an external package to handle something straightforward is weaker than it used to be. If the function is stable and well-understood, writing and owning it directly can be the more defensible choice. The JavaScript ecosystem has long illustrated what the opposite looks like - deep dependency trees where a single unmaintained package can cause widespread disruption.

The broader principle is visibility. Knowing exactly what your software estate contains - through practices like maintaining a software bill of materials (SBOM) - means that when a vulnerability is disclosed, organisations can pinpoint their exposure immediately rather than working backwards through layers of dependencies.

The Human Attack Surface

The most technically sophisticated attack campaigns get significant attention. But the more common scenario is simpler: someone has a distracted afternoon, a link looks plausible and they click it. The breach that follows may not be detected for weeks.

Eliminating human error is impossible, therefore the appropriate response to this is to build enough redundancy into the system that a single mistake doesn't become a significant incident. Two practical approaches are worth prioritising:

Security Operations Centres (SOCs)

24/7 monitoring of an organisation's estate means that suspicious behaviour is caught by the security team before the affected individual realises something has gone wrong. Outsourcing the SOC function is a practical option for organisations that can't sustain it internally.

Self-service incident response

Giving employees a quick, easy way to flag their own mistakes - for example, a one-click option to revoke all active sessions if they suspect they've been compromised - reduces the window of exposure. A culture should be established to reassure employees that if they do make a mistake, reporting it quickly is the right move, and the organisation has the tools to contain the damage.

Why Security Training Usually Fails

Annual compliance-driven security training has become a standard fixture in most organisations. Its impact on actual incident rates is harder to demonstrate. A useful exercise is to consider whether the number of security incidents would change if the training programme were removed entirely. For many organisations, the impact would be negligible.

What seems to work better involves several shifts:

Frequency over volume: Bite-size monthly content has more lasting effect than an annual session.
Personas over policies: Training built around realistic, role-specific scenarios has a greater impact than generic guidance.
Peer storytelling: When someone in the team shares a near-miss in their own words the message can carry more weight than top-down instruction. This can look like making real, anonymised examples a regular part of team meetings.
Culture over compliance: Physical security is a useful illustration: when teams stop thinking of locking up the office as a "security task" and start thinking of it as helping their colleagues, behaviour changes in a way that no policy document achieves. The framing shift should be that security isn't something the IT team does to protect the organisation from its own employees. It's embedded culturally as something everyone contributes to.

AI Is Changing the Threat Landscape

AI is shifting the cybersecurity problem in two directions at once.

On the threat side, social engineering attacks are becoming harder to detect. Phishing emails that once had obvious tells such as poor grammar, generic greetings and implausible pretexts, now pass convincingly as legitimate communications. At the same time, the volume is increasing and the targeting is more precise. Overall, the human attack surface is being exploited more efficiently.

On the development side, the volume of AI-generated code entering production is creating a new category of risk. Frontier technology companies are now shipping thousands of code changes per week written entirely by AI. Most organisations won't move at that pace, but it’s clear that AI is becoming a standard part of the development workflow, and the security implications of that haven't been fully worked through.

The practical answer is to make sure the foundations that make safe, fast change possible are already in place. In a human sense this means equipping teams with the right skills; developers need to understand how to use AI tooling effectively, including what to verify and what still requires human judgement. In terms of technology: automated testing, CI/CD pipelines, static analysis and code review processes, are the guardrails that determine whether AI-generated code is an accelerant or a liability. Teams without these foundations will simply move faster in the wrong direction.

However, the same AI capabilities making attacks harder to detect are also being applied defensively. Security teams are using AI to identify anomalies, accelerate threat detection and work through vulnerability data at scale.

The Pace-of-Change Paradox

One of the more counter-intuitive tensions in modern security practice is whether the risk of changing too slowly is now greater than the risk of changing too fast.

For example, delaying the adoption of a new tool - an AI assistant, a cloud service, a development platform - because of security due diligence processes can push teams towards workarounds. Those workarounds sit outside the visibility and control of the security function. The organisation ends up less secure than if it had moved faster and managed the risk through proper channels.

Rather than rushing due diligence, the constructive response is better-leveraged diligence. Many organisations will start research on a new tool from scratch. However, most tools under consideration have already been adopted by comparable organisations. Finding out what they encountered, and what they did about it through peer networks and industry benchmarks is a faster route to a defensible decision than starting anew. The same principle extends beyond tool adoption. The earlier advice to delay dependency updates rather than apply them on day one is an example not of moving too slowly but of better-leveraged diligence in practice.

Three Priorities for Security Leaders

For anyone newly put in charge of cybersecurity in their organisation, three recommendations stand out above the rest:

Focus on people first

Tools and budgets matter, but the most common point of failure is human. Getting people genuinely engaged, not just compliant, tends to deliver more lasting results than technical controls alone.

Know your framework

For those newer to the security function, the National Cyber Security Centre (NCSC) publishes practical, accessible guidance calibrated to organisation size and sector. Starting there, rather than trying to synthesise a position from scratch, is a more reliable foundation.

Build a sounding board

The risk landscape, once you start looking at it seriously, is genuinely overwhelming. Having trusted peers to pressure-test thinking, and to pull focus back to what's within control, is as important as technical knowledge.

Introducing Technically Speaking

Technically Speaking is a new podcast from Audacia, hosted by Technical Director Richard Brown. Each episode is a conversation with a senior technology leader working through the questions shaping modern technology in practice.

In Episode #1, host Richard Brown is joined by Amy De-Balsi, an independent programme manager working in a highly regulated public sector environment, and Adam Brookes, Head of Consulting at Audacia, to explore why AI initiatives so often stall between proof of concept and production, and what it actually takes to close that gap.

Listen now on your preferred platform

If there are topics you'd like to see covered, we'd welcome your input at podcast@audacia.co.uk

Engineer Experience: Why & How to Measure It | Audacia Insights

Audacia — Tue, 16 Jun 2026 09:00:00 +0000

Engineer Experience examines how people, processes and tools affect productivity amongst software engineering teams. Based on a Tech Talk delivered by Technical Director Richard Brown, this blog explores the ‘why’ and ‘how’ of measuring Engineer Experience.

What Is Engineer Experience?

According to GitHub, engineer experience, sometimes called developer experience or DevEx, refers to the systems, technologies, processes and cultures that influence the effectiveness of software development.

Engineers are typically the most expensive resource on any software project. When engineers constantly face bottlenecks, blockers, inefficient tooling, slow laptops or build queues that stretch for hours, the entire team slows down and struggles to deliver value to end users. Whereas the more efficiently a developer is able to work, the better value for money an organisation can offer.

Bringing engineer experience front and centre means recognising that investing some time, effort or money can unblock engineers to deliver much faster. The return on investment is often relatively high compared to the input.

Why Measure Engineer Experience?

Often, surveys can reveal what metrics can’t. Metrics like DORA measurements, build times and deployment frequency provide useful data. However, without asking engineers directly, organisations can miss potential blockers and inefficiencies. The best way to measure engineer experience is through surveys that ask engineers themselves about their day-to-day work.

Measuring engineer experience delivers four key benefits:

Identify inefficiencies

Surveying engineers helps identify inefficiencies by allowing organisations to drill down into process details to find what causes them. For example, metrics might show that deployments reach production only once per month, but survey answers can reveal why.

Evaluate tools and technologies

Engineers working directly with software, frameworks and tech stacks are best positioned to say whether their tools are fit for purpose or their tech stack is up to date. Engineering leaders might struggle to keep pace with every new development, especially in AI tools where capabilities proliferate rapidly, but engineering teams usually maintain deep knowledge in their specific areas. Asking them about the tools and technologies they use surfaces insights about how effective those choices are for solving daily problems.

Recognise pain points

Job satisfaction matters enormously in any role, and in software development the best engineers are in very high demand, making it crucial to avoid having people dissatisfied in their roles. Whether it stems from being unable to use particular technologies, waiting three hours every time they queue a build, or working in silos with no visibility of other teams, a few stacked frustrations can prompt engineers to look elsewhere. Identifying pain points means they can be addressed before they become retention issues.

Build engagement

People genuinely appreciate being asked their opinion and having it valued. Engineering leaders who mandate decisions — frameworks, IDEs, product licences — without input quickly leave people feeling disempowered and lacking buy-in. Asking for opinions and taking them on board creates genuine engagement and commitment to organisational goals.

Best Practices for Engineer Experience Surveys

Several practices help organisations run effective engineer experience surveys and extract maximum value from the results.

Ask about things within your control

Only ask questions about factors that can actually be changed. For example, if an organisation only uses JavaScript and there are constraints that mean this cannot be changed, asking about satisfaction with the programming language only draws attention to dissatisfaction without offering a path to improvement. Instead, focus on areas where action is possible.

Focus on what’s important to your organisation

Template surveys and example questions exist online and make excellent starting points, but every organisation differs. Use templates as foundations, then add, remove and modify questions to ensure the survey speaks to specific organisational needs and priorities.

Be consistent

Running a single survey provides a pulse check on engineer sentiment, but the real value emerges from running questionnaires at a regular cadence. Instead of changing questions between surveys, maintain a high proportion of consistent questions as a baseline. This allows organisations to track trends and determine whether changes implemented between surveys have influenced engineer sentiment. For example, average scores moving from 3.5 to 4 out of 5 between surveys suggests that any change implemented in-between was positive and worth continuing.

Include role-specific questions

Most organisations employ different types of engineers — software engineers, test engineers, cloud engineers, data engineers, data scientists etc. Include universal questions aimed at everyone as well as subsets of questions targeting individual roles. For example, ask testers about splits between automation and manual testing, developers about unit testing and cloud engineers about cloud provider choices or infrastructure-as-code toolchains.

Consider anonymity carefully

Anonymous surveys encourage honest, robust feedback, but they can be harder to action. Without follow-up questions or context, it becomes difficult to fully understand the motivation behind comments. For example, in an organisation of 100 engineers, if 97 rate documentation as fantastic but three suggest room for improvement, then the big picture suggests that documentation is a low priority pain point. However, if those three engineers all make up one team, then the feedback becomes very different and targeted improvement for that specific team becomes necessary.

One solution can be to maintain anonymity whilst also collecting role, team or project information. This preserves anonymity (teams are typically larger than three people) whilst enabling identification of systematic trends within specific roles or teams.

Balance quantitative and qualitative feedback

Quantitative questions, scoring from one to five, enable easy trend measurement and quick identification of pain points (typically the lowest-scoring questions). Qualitative free-text responses provide the crucial context that makes feedback actionable. Someone scoring something one out of five is useful, but someone scoring one out of five and explaining why is far more valuable.

Follow through with change

Actioning feedback is the most critical practice in this list. At Audacia we break results down by project and meet with lead developers to identify next steps. These decisions feed into sprint retrospectives or targeted improvement initiatives.

We never make surveys mandatory, but by demonstrating that feedback leads to real change engagement stays high. Without this follow-through engagement rates drop as people gradually disengage from the process. If feedback can’t be actioned because of constraints such as finite time and budget, then clearly communicate those. Opening this conversation makes people aware of leadership constraints, whilst maintaining trust.

Key Survey Areas

At Audacia, we structure our survey questions around these key areas chosen to reflect what matters most to our engineers and the organisation. These areas can change year-on-year depending on organisational activity. Every business is different, and the areas you prioritise should reflect that.

Documentation and knowledge sharing

These practices address how quickly engineers can onboard to new projects or code areas. For consultancies, this becomes especially critical because project teams work in entirely different business domains and build very different software, making knowledge silos easy to form.

Can engineers search documentation to answer questions about how code works or why architectural decisions were made?
Or must they ask specific individuals, creating bottlenecks?

Technical standards

Technical standards should help organisations ship better code faster. Surveys provide a pulse check on whether standards are helping or hindering. Too many standards, poorly discoverable standards or frequently changing standards slow teams down rather than speed them up. On the other hand, clear, concise, easily discoverable standards speed teams up and improve code quality.

Agile processes and project delivery

Determine whether agile processes help teams work effectively, or whether they fail in practice.

Are stand-ups efficient or do they stretch to two hours because discussion is too detailed?
Do retrospectives happen?
Do teams action retrospective items?
Are teams continuously improving?

Additionally asking about project delivery can surface inefficiencies. For example, builds taking two hours because of massive queues, or deployments only being possible at specific times because they rely on one person.

Observability

This matters particularly when supporting software in production.

When bugs appear or production goes down, how do teams find out?
Do customers call to report problems, or do alerts arrive proactively, enabling teams to inform customers about issues and mitigation plans?
When bugs occur, can teams diagnose them quickly?

Strong observability, seeing exceptions logged and tracking paths through systems, enables rapid bug diagnosis, faster patches and reduced impact on end users.

Quality

This topic covers broad territory.

Do code reviews add value or slow teams down?
Are automated tests written?
Are those tests run regularly?
Do they deliver valuable feedback?
Do tests catch regression bugs?

Practices like code reviews and automated testing policies are important, but they can easily become bottlenecks. Asking engineers about the value they receive identifies where policies fail to add value and simply slow people down.

Tools, technologies, and practices

These often have the biggest impact on engineer job satisfaction. Engineers spend most days writing code, living in their IDEs, working with programming languages and frameworks. An engineer could face frustration if they work with outdated frameworks approaching end-of-support whilst colleagues on other teams use the latest frameworks. Identifying these pockets of dissatisfaction — particularly when responses are sliced by team or project — enables targeted improvements.

AI tools and adoption

Engineer sentiment around AI is increasingly worth measuring on its own terms. Attitudes vary widely from enthusiastic adopters to sceptics, and understanding where your teams sit helps inform decisions around tooling and training.

Useful questions to ask include:

Do engineers feel positively about the role of AI in engineering generally?
Do they think the organisation is adopting AI at the right pace?
Are engineers satisfied with the AI tools available to them?
Do they have sufficient quota within those tools to use them in their day-to-day work?

Conclusion

Engineer experience surveys provide a valuable mechanism for gathering feedback that metrics alone cannot capture. By customising surveys to organisational priorities and running them consistently, organisations can measure progress, identify systematic issues and make targeted improvements. This ongoing practice of measuring and improving engineer experience leads to better delivery outcomes, higher engineer satisfaction and stronger retention.

Watch the full talk

Author

Richard Brown is the Technical Director at Audacia, where he is responsible for steering the technical direction of the company and maintaining standards across development and testing.

AI-Assisted Engineering: Building the Foundations for Adoption and Scale

Audacia — Tue, 28 Apr 2026 07:30:00 +0000

Based on a Tech Talk delivered by Technical Director Richard Brown, this blog explores how to successfully implement AI-assisted engineering.

Rolling out AI to engineering teams successfully depends on three interconnected phases: establishing solid foundations, executing a structured rollout and committing to continuous improvement. Each phase builds on the previous one to create sustainable AI adoption that delivers real value. These observations come from our internal AI deployment at Audacia and supporting other organisations through similar transformations.

The stages of AI assisted engineering - foundations, rollout and improvement represented in a diagram.

What are the foundations of AI-assisted engineering?

Before rolling out AI-assisted engineering, three foundations need to be in place: a clear AI governance policy that defines what people can and can't do, solid delivery practices that AI can amplify rather than expose, and documented coding standards that can be fed directly into AI tools as context.

AI Governance

Research into AI adoption across organisations reveals that providing guardrails and governance strongly correlates with successful AI adoption. On the other hand, adoption rates drop when organisations impose no constraints. Without defined boundaries, people become nervous and uncertain about what they're allowed to do.

A key first step to laying the foundations for AI is publishing an AI policy, which removes this ambiguity. The policy should clarify acceptable use – which tools are approved, licensing requirements, privacy controls and IP indemnity policies – as well as defining overarching principles that must be followed when using AI tools. The principles we use at Audacia are:

Accountability: People remain responsible for the quality and accuracy of their work, including any AI generated output used.

Fairness: AI models can reflect biases in their training data, so output should be reviewed carefully wherever it influences decisions that affect people.

Maintainability: AI-generated code should meet the same standards for readability and structure as anything written by a human.

Privacy: AI tools must handle data appropriately, with clear retention policies and care taken over what is shared with external models.

Security: AI tools warrant particular scrutiny – tools must meet appropriate security standards, and applications built with AI capabilities need proper security testing.

Transparency: AI involvement should be clear to all stakeholders/consumers, whether in code, content or decisions.

Solid Delivery Practices

The annual DORA (DevOps Research and Assessment) report recently focused on AI-assisted engineering and its impact across the software industry. The key takeaway that stands out is that AI serves as an amplifier; robust, mature practices accelerate the positive impact of AI, and poor processes provide a blocker to successful adoption.

In practice this means that if work flows smoothly through the pipeline, code review and pair programming processes are robust, testing is comprehensive and deployments are quick, AI will generally make those things even better. However, if single points of failure exist and bottlenecks plague your processes, then AI will amplify those problems too. Making sure solid foundations exist before introducing AI ensures that the right practices get amplified.

Many of these solid delivery foundations represent established good practice. For example, small batch sizes matter – including committing often, submitting small pull requests, breaking down user stories to smaller chunks, and reducing lengthy feature branches and troublesome merge conflicts. Additionally, unit testing becomes critical when generating more code faster – a comprehensive automated test suite ensures increased velocity doesn't introduce regression bugs.

Lastly, recognising that software engineering represents just one part of the overall delivery lifecycle is crucial. Speeding up software engineers achieves nothing if bottlenecks exist elsewhere in the lifecycle – if test environments aren't fit for purpose or requirements can't be defined quickly enough, faster developers simply wait for requirements or work on the wrong things. Ultimately, the entire development lifecycle needs examination, not just isolated parts.

Documented Standards

AI-assisted engineering demands documented standards – without them, there is nothing to govern the code AI generates. If your organisation doesn't have established coding standards, this is a good opportunity to document them.

Coding standards need to be communicated effectively to AI tools. Having Copilot or Claude generate extensive code that engineers must immediately rewrite or refactor to fit coding standards creates inefficiency. To address this, your organisational standards should be part of the context used by large language models generate content. Technologies like MCP servers allow coding standards to be pulled into the LLM context window, helping the model understand what 'good' looks like.

Ideally, these foundational pieces should be in place before rollouts begin – that's not always feasible, so laying the foundations and rollout can happen in parallel. However, before rollout, at a minimum, the overall software development lifecycle needs examination, and people need information about what they are and aren't allowed to do with AI.

How do you roll out AI tools across an engineering team?

Rolling out AI tools requires choosing the right tool for each team's environment, with data security and longevity in mind, and establishing benchmarks to evaluate new tools consistently. Equally important is role-specific training that sets realistic expectations – using AI effectively is a skill that doesn't automatically come with technical ability.

Choosing the Right Tools

The obvious first step to rolling out AI is giving people access to the right AI tools – such as GitHub Copilot, Claude Code, etc. Choosing the right tools for the right team requires meeting engineers where they spend most of their time. For example, if developers work primarily in VS Code, Copilot often makes sense because of deep integration. Data security should also inform this choice – tools must have appropriate terms of service, clear data retention policies and transparent privacy policies.

Beyond the initial selection, tool choice needs to account for longevity. The AI landscape moves quickly, and the right tool today might not be the right tool in six months. This makes ongoing assessment a responsibility in itself – staying aware of new models and services as they emerge rather than treating the initial rollout as a settled decision.

To help with ongoing tool assessment organisations can establish benchmarks to test against. Measuring tools against dimensions like correctness (does it meet the requirement given), autonomy (how much rework is required), quality (does it generate good code), even token usage (how efficiently does it get to a solution), gives teams a reliable basis for comparison. A representative codebase or application – something that genuinely reflects the organisation's work – makes this practical. For example, when a new model or plugin ships it can be tested against something real rather than abstract criteria.

Training and Upskilling

Alongside providing access to tools, ensure guidance is provided on how best to use them. Whether called prompt engineering, vibe coding, or something else, using AI as a software engineer is a skill in itself – and the best software engineers aren't necessarily the best at using AI, and vice versa.

Any training should present a balanced attitude to AI. Opinions tend to be polarised – hype at one end, dismissal at the other – with the truth somewhere in the middle. AI has limitations, and sometimes stepping outside the AI loop entirely is needed. Without these realistic expectations, the first roadblock an engineer faces can lead to disillusionment and disengagement.

Making training role-specific matters too. General organisational training is useful, but software engineers, test engineers and other teams each need specific guidance. Like tooling, meeting people where they are is essential. Finally, training must evolve. As new employees join and tools update, the ability to deliver ongoing, up-to-date guidance is what turns a one-off rollout into lasting capability.

How do you sustain and improve AI adoption over time?

Sustaining AI adoption long-term comes down to culture, clarity and measurement. Building communities that encourage open knowledge sharing, defining where AI works well and where human judgement is still needed, and tracking delivery velocity and quality all ensure AI continues to add value rather than becoming shelfware or a crutch.

Building Communities

Embedding AI adoption into an organisation’s culture, through building passionate communities, is what will drive AI forward and continuously improve how it gets used. As part of this, AI adoption shouldn't be seen as a top-down mandate – instead everyone should have a voice. Newer engineers in particular grow into the industry as AI natives. This technology moves so fast and is so new that everyone has valid opinions worth hearing.

These communities should also promote knowledge sharing. Whether through existing communities of practice, online forums, Slack channels, Teams channels, etc., building communities around AI creates space for sharing success stories, discussing challenges and asking questions. To avoid knowledge silos forming, employ a "no stupid questions" culture, creating a space where everyone learns from each other. Otherwise, instead of moving forward as an organisation, only localised progress occurs.

Developing Use Cases

Developing specific use cases helps teams understand the boundaries of AI. There are some cases AI struggles with, and some it handles incredibly well. Tasks which are generally repetitive, somewhat tedious work can be effectively offloaded to AI agents. For example, AI has proven reliable for upgrading codebases from one framework version to another, assuming the upgrade path is well-documented.

However, understanding boundaries remains important. Where does manual intervention still matter? When developing use cases involving AI agents, how much freedom and control should those agents have? Where does a human need to remain in the loop? Defining these boundaries ties back to acceptable use policies.

Tracking Progress

Tracking key metrics is important in ensuring AI is being used effectively, but also that individuals aren’t over-depending on AI without critical thinking. Maintaining the ability to solve novel problems without becoming overly dependent is crucial. Key metrics to track include delivery velocity and quality – as AI adoption increases, hopefully the amount of value delivered increases too, and quality should, at minimum, remain constant.

Summary

Successful AI adoption in engineering requires more than tool access. It starts with clear policies, strong delivery practices and documented standards, moves into carefully chosen tools and realistic training, and succeeds long-term through knowledge-sharing communities, well-defined use cases and consistent progress tracking.

Above all, AI-assisted engineering is not a one-off endeavour. Treating adoption as a continuous process of revisiting foundations, refreshing training and reassessing tooling as the landscape shifts, is what separates organisations that sustain value from those where initial momentum fades.

Author

Richard Brown is the Technical Director at Audacia, where he is responsible for steering the technical direction of the company and maintaining standards across development and testing.

Watch the Tech Talk

A look at Microsoft Fabric

Audacia — Mon, 30 Mar 2026 09:00:00 +0000

Running data functions at large organisations might look like one set of tools for ingestion, another for storage, something else for transformation, a separate analytics layer, and a BI platform bolted on top. Each being the right choice at the time, however, collectively, they've become a problem.

This can end up with a setup where:

Data engineers spend their time preparing and packaging data to hand off to analytics teams
Analysts build reports in tools that sit outside the engineering environment, often working from copies or extracts rather than a single source of truth
Data scientists operate in yet another silo, pulling data into notebooks and models that live separately from everything else

Handoffs can lead to delays, and each disconnected tool can add governance complexity as well as creating a significant cumulative cost - in time, money, and organisational friction.

The industry has been moving towards platform consolidation for years, and the major cloud providers have all made progress in this direction. Microsoft's entry with Fabric represents an attempt to bring the entire data lifecycle, from raw ingestion to executive dashboard, into a single, unified environment.

For those evaluating where Fabric fits, the challenge is understanding what Fabric actually changes, who benefits most, and whether the strategic shift is worth pursuing.

Understanding the spectrum

Before assessing any platform, it helps to step back and consider where your organisation sits with regards to data structuring. Different businesses need different levels of sophistication, and understanding the different requirements helps clarify where Fabric adds value and where simpler solutions might still serve you well.

At the most straightforward level, simple databases serve a clear and important purpose. Relational databases like SQL Server or PostgreSQL handle structured data storage and retrieval for individual applications effectively. If your needs are transactional, such as powering a web application, managing customer records, or supporting a single product, a well-designed database does the job without unnecessary complexity. Many teams start here, and for contained use cases, there's no reason to move beyond it.

As organisations grow and the demand for cross-functional reporting increases, data warehouses become the natural next step. Platforms like Azure Synapse Analytics, Snowflake, or Google BigQuery are designed to aggregate data from multiple sources into a structured, optimised environment built for analytical queries. This is the traditional backbone of enterprise business intelligence where data is extracted from operational systems, transformed into consistent schemas, and made available for reporting and analysis. For organisations that need reliable, governed analytics across departments, a data warehouse remains a solid foundation.

The challenge arises when the warehouse alone is no longer enough. Modern data demands often include unstructured data, real-time streaming, machine learning workloads, and self-service analytics - none of which a traditional warehouse handles natively. This is where organisations start layering in additional tools such as a lakehouse for unstructured data, a Spark environment for data science, a separate streaming platform for real-time use cases, and a BI tool on top. Each addition solves a problem, but each also introduces another integration point, another security model to manage, and another team boundary to navigate.

Unified platforms like Microsoft Fabric sit at the far end of this spectrum. Rather than asking organisations to assemble their own stack from best-of-breed components, Fabric brings storage, engineering, warehousing, data science, real-time analytics, and business intelligence together in a single environment. For those operating at scale, with multiple data teams and increasingly complex requirements, the cost of maintaining a fragmented stack can become harder to justify.

Understanding where your organisation sits on this spectrum matters because the value of Fabric depends heavily on context. An organisation running a handful of straightforward reporting use cases may find a warehouse and Power BI perfectly sufficient. An organisation juggling data engineering, science, streaming, and BI workloads across five different platforms will feel the consolidation benefits immediately.

Microsoft Fabric: The umbrella explained

Fabric can be easy to misunderstand if you approach it as simply another Microsoft product release. In reality, it's an umbrella platform - a unified SaaS offering that brings together multiple previously separate data services under a common foundation.

At the base of everything sits OneLake, Fabric's unified data layer. OneLake acts as a single storage foundation for your entire organisation's data, regardless of whether that data is structured, semi-structured, or unstructured. Every service within Fabric reads from and writes to OneLake, which means there's one copy of the data, one set of access controls, and one lineage trail. This is the architectural decision that makes the rest of the consolidation possible. A shared data layer means the services built on top of it genuinely share a foundation, rather than simply being co-located.

Built on top of that foundation, Fabric consolidates several core services that organisations have traditionally sourced and managed independently.

Data Factory handles data integration and orchestration. If you're currently running ETL or ELT pipelines to move data between systems, Data Factory provides that capability natively within Fabric. It connects to a wide range of source systems and allows you to build, schedule, and monitor data movement and transformation workflows without reaching for a separate integration tool.

Data Engineering provides a Spark-based environment for large-scale data processing. Data engineers can work with notebooks and Spark jobs directly within the Fabric environment, processing large volumes of data without needing a standalone Spark cluster or a separate Databricks workspace. The data they process lives in OneLake, immediately accessible to every other service.

Data Warehousing delivers a T-SQL-based analytical data warehouse. For organisations with teams skilled in SQL, this provides a familiar interface for building and querying structured analytical models without the need to provision and manage separate warehouse infrastructure.

Data Science supports machine learning and advanced analytics workloads. Data scientists can build, train, and deploy models within the same environment where the data engineering and warehousing work happens. This reduces the friction that typically exists when models need to move between teams or when data needs to be extracted into separate science environments.

Real-Time Analytics addresses streaming and event-driven data. For organisations working with IoT data, application telemetry, or any use case that requires near-instant insight from data as it arrives, this service provides real-time ingestion and querying capabilities natively within the platform.

Power BI, already the dominant enterprise BI tool in many organisations, is integrated directly into Fabric rather than sitting alongside it as a separate product. Reports and dashboards connect directly to data in OneLake, with no need to extract, export, or duplicate data into a separate BI layer.

Data Activator adds an automation layer, allowing organisations to set up alerts and trigger actions based on data conditions. Rather than building custom monitoring solutions, teams can define rules that respond automatically when data meets certain thresholds or patterns.

Comparable tools for each of these capabilities exist elsewhere in the market. Where Fabric distinguishes itself is in the shared foundation. These services share OneLake, share a security model, share a governance framework, and share a licensing structure. They were built on a common platform rather than bundling together tools.

Data consolidation

For data teams, the most compelling argument for Fabric is often operational rather than technical. The way most large organisations currently work with data involves a series of handoffs between teams that can create unnecessary friction.

Consider a workflow of:

A data engineering team builds and maintains pipelines that ingest raw data from source systems, transform it, and load it into a warehouse or lakehouse. Once the data is structured and validated, it's made available, often through a separate access layer or export process, to an analytics team.
The analytics team then builds reports and dashboards in a BI tool like Power BI, Tableau, or Looker.
If a data science team is involved, they'll often pull data into yet another environment to build models, the outputs of which may then need to be fed back into the warehouse for the analytics team to report on.

This also creates an opportunity for data to drift out of sync, for definitions to diverge, and for governance to become fragmented. With each of these handoffs also being a potential point of failure, introducing latency, and requiring coordination between teams who may be using different tools, different interfaces, and different mental models of the same data.

Fabric's consolidation directly addresses this. When Power BI sits within the same platform as the data engineering and warehousing layers, the gap between "data is ready" and "report is built" shrinks dramatically. An analyst building a Power BI report in Fabric is working directly with data in OneLake, the same data the engineering team just processed, governed by the same access controls, with the same lineage. There's no export, no separate connection to configure and no waiting for data to appear in a different system.

Similarly, when data scientists work within the same environment, they can access the data they need directly rather than extracting it into a standalone notebook server or requesting access through a separate process. They work on the same platform, with the same data, subject to the same governance. The output of their models can be written back to OneLake and immediately consumed by BI reports or downstream applications.

Organisational roles and specialisms remain important in this model. Data engineering, analytics, and data science are distinct disciplines with distinct skills, and Fabric doesn't change that, however it does reduce the friction between them. Teams still specialise, but they collaborate on a shared platform rather than throwing work in silos between disconnected tools.

The governance implications are equally significant. In a fragmented stack, security and access controls need to be configured and maintained separately across each tool; data lineage is difficult to track end-to-end when data passes through multiple systems; and compliance reporting requires pulling information from multiple audit logs. However, in Fabric, a single security model covers the entire lifecycle. Access controls set at the OneLake level apply consistently whether the data is being accessed by an engineer in a Spark notebook, an analyst in Power BI, or a scientist in a machine learning experiment.

For organisations operating in regulated industries such as financial services, healthcare or the public sector, this unified governance model creates a significant reduction in compliance risk and audit complexity.

Considerations

Understanding what Fabric offers is a useful starting point. The harder work is deciding whether and how to adopt it. There are several strategic dimensions worth considering:

Market positioning

Fabric exists within a competitive landscape. Databricks offers a strong lakehouse platform with deep data science capabilities, while Snowflake provides a mature, cloud-agnostic data warehousing experience, and AWS has its own suite of data services. Each has genuine strengths, and the right choice depends on the specific context. Fabric's distinctive advantage lies in its breadth and native integration with the Microsoft ecosystem. If your organisation already runs on Azure, uses Microsoft 365, and has Power BI embedded across business teams, Fabric offers a consolidation path that leverages existing investments and skills. Organisations whose stacks are primarily built on AWS or GCP will need to weigh that integration benefit against the switching costs involved.

Migration reality

No large organisation is going to rip and replace its entire data infrastructure overnight, and Fabric doesn't require that. A more realistic approach is phased adoption - identifying workloads where consolidation delivers the most immediate value and starting there. Power BI teams that currently connect to external data sources can be a logical first choice, with data engineering teams managing complex pipeline orchestration across multiple tools being another. Starting with high-friction, high-visibility workloads can help to build internal confidence and demonstrate value before committing to broader migration.

Skills and team readiness

Fabric lowers certain barriers, such as analysts can do more without engineering support, and the shared environment reduces the need for manual handoffs. At the same time, adopting any new platform requires an investment in learning. Teams will need to understand OneLake's storage model, the nuances of each service within Fabric, and how governance works across the unified environment. Planning for this upskilling alongside the technical migration is essential.

Governance and compliance

For organisations in regulated sectors, Fabric's unified security and lineage model is a significant draw. Having a single place to manage access controls, audit data movement, and trace lineage from source to report simplifies compliance in a way that fragmented stacks struggle to match.

Platform maturity

Fabric is still evolving. Some components are more mature than others and Microsoft continues to ship updates and new capabilities at pace. Early adopters should be prepared for a platform that is moving quickly, with all the opportunity and occasional rough edges that brings. Evaluating Fabric today means accepting that some features may still be maturing while recognising that Microsoft's investment and trajectory suggest significant development ahead.

Summary

The fragmented data stack served its purpose for a long time. It allowed organisations to adopt best-of-breed tools for each stage of the data lifecycle and build capabilities incrementally. But the operational and strategic costs of maintaining that fragmentation are growing, and the expectations placed on data teams - to deliver faster, govern better, and do more with less - are only increasing.

Microsoft Fabric represents a credible path towards consolidation. By bringing the full data lifecycle under one roof, sharing a common data layer, and unifying governance across every workload, it addresses many of the friction points that data teams deal with daily.

Whether Fabric is the right move for an organisation depends on the current stack, their team's capabilities, and overall strategic direction. For data leaders already embedded in the Microsoft ecosystem and feeling the strain of a fragmented infrastructure, it can be a good option to evaluate.

Author

Chris is a Lead Data Scientist, with a background in astrophysics, and has over 4 years’ experience in providing data strategies insights using computational models and machine learning methodology. Chris has worked with a number of organisations across industries to successfully deliver AI projects, from PoC development and use case validation, through to model training and maintenance.

Testing AI: How to Effectively Evaluate LLMs

Audacia — Mon, 23 Mar 2026 10:00:00 +0000

Traditional software testing rests on a basic assumption that given the same input, the system produces the same output. A test case defines expected behaviour, and a test passes or fails based on whether the output matches. This assumption – deterministic behaviour with verifiable correctness – is the foundation on which decades of quality assurance practices have been built.

However, this can break down with large language models. An LLM may produce a different response to the same prompt on successive runs. Its outputs are sensitive to context, prompt phrasing, temperature settings and the interaction between retrieved documents and parametric knowledge. It can produce responses that are fluent, confident and completely wrong - a failure mode that traditional testing has no framework for detecting. And unlike a conventional software bug, which typically manifests consistently and can be reproduced, AI system failures are often probabilistic, context-dependent and difficult to predict.

For engineering leaders, this creates a new problem. Organisations are deploying LLM-powered features at pace, such as customer-facing chatbots, internal knowledge assistants, AI-augmented search, automated document processing, coding assistants and increasingly autonomous agentic workflows. However, the testing and evaluation practices for these systems are struggling to keep up.

The World Quality Report 2025, surveying over 2,000 senior executives across 22 countries, found that hallucination and reliability concerns are now among the top barriers to generative AI adoption in quality engineering, cited by 60% of respondents - a challenge that barely registered two years ago.

This article looks at what testing looks like for AI systems, why it is fundamentally different from traditional software testing, and how organisations can build the evaluation capability required to deploy LLMs responsibly.

Why Traditional Testing Fails for AI Systems

The differences between testing traditional software and testing AI systems are not differences of degree but of kind.

In conventional software, correctness is binary. A function either returns the right value or it does not. Test cases can enumerate expected input-output pairs, and 100% pass rates are achievable and expected. The system under test is deterministic - run the same test twice, get the same result. And when a test fails, the failure is reproducible, allowing engineers to diagnose and fix the root cause.

Little of these properties hold for LLM-powered systems. There is no single "correct" response to most natural language queries. A question about company policy might have multiple valid phrasings, levels of detail and degrees of nuance. The system is non-deterministic by design (temperature and sampling parameters introduce controlled randomness). And failures, such as hallucinations, reasoning errors, safety violations and biased outputs, may occur intermittently, triggered by specific combinations of context, phrasing and retrieved information that are difficult to anticipate or reproduce.

This means testing AI systems is an evaluation discipline rather than a verification discipline. Instead of asking "does this pass or fail?", organisations must ask "how well does this system perform across a range of scenarios, and is the distribution of performance acceptable for our use case?" This requires statistical thinking, domain-specific quality criteria and continuous evaluation rather than one-off test suites.

The Hallucination Problem: Scale and Consequences

Hallucination - where an LLM generates content that is fluent and confident but factually incorrect or unsupported by source material - is the most visible failure mode and the one that most concerns enterprise adopters.

Vectara's Hallucination Leaderboard, which benchmarks LLMs for factual consistency in summarisation tasks, found that even frontier reasoning models, including GPT-5, Claude Sonnet 4.5, Grok-4, and DeepSeek-R1, all exhibited hallucination rates exceeding 10% on their updated, more challenging benchmark. The recently released Gemini-3-pro demonstrated a 13.6% hallucination rate and did not make the top-25 list.

These are the best available systems, evaluated on a straightforward summarisation task, not adversarial conditions or edge cases.

The academic community is also grappling with how to define and categorise hallucinations consistently. The HalluLens benchmark, presented at ACL 2025, identified a fundamental challenge in existing benchmarks often conflating hallucination with factuality, despite these being distinct problems requiring different evaluation approaches. HalluLens proposes a taxonomy distinguishing between extrinsic hallucinations (where generated content deviates from or contradicts source material the model had access to) and intrinsic hallucinations (where the model contradicts its own earlier outputs). This distinction matters for enterprise applications because the mitigation strategies differ, with extrinsic hallucination being a retrieval and grounding problem, while intrinsic hallucination is a consistency and reasoning problem.

The real-world consequences of inadequate hallucination testing are already visible and increasingly costly.

Air Canada lost a legal case after its chatbot fabricated a bereavement discount policy that did not exist – the airline was held liable for the AI's invention.
New York City's public-facing chatbot provided illegal advice to business owners about regulatory requirements.
And a GPTZero analysis of over 4,000 papers accepted at NeurIPS 2025 found that dozens contained fabricated AI-generated citations – invented authors, titles and journals that passed peer review undetected.

These incidents share a common root cause in systems being deployed without adequate evaluation of their failure modes under realistic conditions.

What LLM Evaluation Looks Like

Practitioners are converging on a multi-dimensional evaluation approach that moves well beyond traditional pass/fail testing. The emerging consensus spans at least seven dimensions: accuracy, safety, bias, hallucination, robustness, latency and security. Each requires different evaluation methods, and the relative importance of each dimension varies by use case – a customer service chatbot has different critical dimensions than a code generation tool or a medical information system.

Benchmark suites

Benchmark suites are the most familiar evaluation approach, adapted from academic AI research. Standardised benchmarks test model capabilities across reasoning, knowledge, coding and other dimensions. However, generic benchmarks have significant limitations for enterprise use. Many models now saturate standard benchmarks like MMLU (exceeding 90% accuracy), which has driven the development of harder alternatives. More fundamentally, a model's score on a general benchmark tells you little about how it will perform on your specific domain, data and use cases. Organisations deploying LLMs need domain-specific evaluation datasets that reflect the actual questions their users ask, the documents their RAG systems retrieve, and the edge cases their particular deployment will encounter.

LLM-as-judge approaches

LLM-as-judge approaches use one language model to evaluate the outputs of another. This approach is both practical and scalable, allowing automated evaluation of thousands of responses without human reviewers, with tools like DeepEval and RAGAS making this accessible. But the approach does have an inherent risk. If both the generating model and the evaluating model are prone to hallucination, they may reinforce each other's errors, creating what researchers describe as a "hallucination echo chamber." Effective LLM-as-judge implementations mitigate this through multi-model consensus (using several different models as judges), structured evaluation rubrics that constrain the judge's assessment to specific, verifiable dimensions and periodic calibration against human judgement.

Red-teaming and adversarial testing

Red-teaming and adversarial testing deliberately probe the system for failure modes. This includes testing for prompt injection (where adversarial inputs manipulate the model's behaviour), safety violations (where the model produces harmful or inappropriate content), and edge cases where the model's confidence exceeds its accuracy. Red-teaming is particularly important for customer-facing AI systems, where an adversarial user may deliberately attempt to exploit the system. The EU AI Act explicitly requires adversarial testing for general-purpose AI models, making this a compliance requirement rather than a best practice.

Human evaluation

Human evaluation remains essential for high-stakes use cases. Automated metrics cannot fully capture whether a response is genuinely helpful, appropriately nuanced, or safe in context. Human evaluation is expensive and slow, which makes it impractical for comprehensive testing, but it serves a critical role in calibrating automated evaluation systems and validating performance on the most important and sensitive scenarios.

Continuous evaluation in production

Continuous evaluation in production closes the loop. Unlike traditional software where testing occurs before deployment, AI systems require ongoing monitoring because their performance depends on inputs that cannot be fully anticipated. This includes tracking hallucination rates on real user queries, monitoring for distribution shift (where the types of questions users ask diverge from what the system was evaluated on), and collecting user feedback to identify failure patterns that pre-deployment testing missed.

Testing RAG Systems: Where Retrieval Meets Generation

Retrieval-augmented generation (RAG), where an LLM's responses are grounded in documents retrieved from an organisational knowledge base, is the most common enterprise LLM deployment pattern. It is also where testing becomes particularly nuanced, because failures can originate in the retrieval step, the generation step or the interaction between the two.

A RAG system can fail in several distinct ways. The retrieval component may return irrelevant documents, missing the information needed to answer the query. It may return relevant documents but rank them poorly, burying the critical information below less relevant content. The generation component may ignore the retrieved context and rely on its parametric knowledge instead, producing a plausible but ungrounded answer. Or it may hallucinate details that are not present in any of the retrieved documents, fabricating specifics while appearing to cite its sources.

Testing RAG systems therefore requires evaluating each component independently and the system as a whole. Retrieval quality can be measured through precision (what proportion of retrieved documents are relevant?) and recall (what proportion of relevant documents are retrieved?). Generation quality requires checking faithfulness (does the response accurately reflect the retrieved content?), relevance (does the response actually answer the question?) and completeness (does it include all pertinent information from the retrieved documents?).

The challenge is that these evaluations require ground-truth datasets specific to the organisation's knowledge base and user queries. Off-the-shelf benchmarks do not test whether your RAG system correctly answers questions about your company's policies, products or processes. Building these evaluation datasets, such as curating representative questions, establishing correct answers and maintaining them as the knowledge base evolves, is one of the most labour-intensive but essential aspects of AI testing. Enterprise research has found that content quality and organisation within the knowledge base itself often has a larger impact on RAG performance than the choice of model or retrieval architecture, which means testing must extend to the data layer, not just the AI components.

Testing Agentic AI: The Next Frontier

The testing challenge compounds further as organisations move from simple question-answering systems to agentic AI – systems that can plan multi-step tasks, use tools and take actions in the real world. An agentic workflow might involve an AI system that receives a customer request, retrieves relevant information from multiple sources, reasons about the best course of action and executes a series of steps (updating a database, sending a communication, triggering a workflow) with minimal human intervention.

Testing agentic systems requires evaluating not just the quality of individual outputs but the correctness of entire decision chains. Does the agent correctly decompose a complex task into appropriate sub-tasks? Does it select the right tools for each step? Does it handle errors and unexpected conditions gracefully? Does it know when to escalate to a human rather than proceeding autonomously?

These questions go beyond hallucination testing into territory that more closely resembles integration testing and end-to-end workflow validation. However, with the added complexity that the system's behaviour is non-deterministic and its decision-making is opaque.

The real-world consequences of inadequate agentic AI testing have already surfaced: in one widely reported incident, an autonomous AI coding agent deleted a company's primary database during a self-directed "cleanup" operation, violating a direct instruction prohibiting modifications. The root cause was not a hallucination but a reasoning failure, where the agent decided that a database cleanup was appropriate despite an explicit code freeze instruction, and no separation existed between test and production environments.

For engineering leaders, agentic AI testing demands a combination of traditional integration testing principles (test the workflow end-to-end, validate boundary conditions, verify error handling) with AI-specific evaluation (assess the quality of the agent's reasoning, its compliance with guardrails and its behaviour under adversarial or unexpected conditions). Sandbox environments with realistic but non-production data become essential, as does the ability to replay and analyse the agent's decision chain after the fact.

The Regulatory Dimension

The regulatory environment is adding both urgency and specificity to AI testing requirements.

The EU AI Act, now entering enforcement, establishes graduated testing obligations based on risk classification. High-risk AI systems, which include those used in employment, credit decisions, education and critical infrastructure, require comprehensive testing for accuracy, robustness, cybersecurity and non-discrimination before deployment, with ongoing monitoring obligations thereafter.

General-purpose AI models face model evaluation requirements including adversarial testing. Organisations deploying LLM-powered features must be able to demonstrate that they have tested their systems against these criteria – a compliance requirement that many have not yet begun to address.

The UK's approach differs in structure but converges in its implications. Rather than prescriptive legislation, UK regulators are applying existing regulatory frameworks, through the FCA, ICO, CMA and sector-specific regulators, to AI systems within their remit. The ICO's guidance on AI and data protection, for instance, requires organisations to demonstrate that AI systems processing personal data are accurate, fair and transparent. The practical effect is similar in that organisations must be able to evidence that they have evaluated their AI systems' behaviour against relevant quality and safety criteria.

The EU Cyber Resilience Act adds another layer for AI-powered software products, requiring that products be developed according to secure-by-design principles, free from known exploitable vulnerabilities and supported by ongoing security updates. For AI systems that interact with external inputs (user queries, retrieved documents, API calls), this implies testing for adversarial inputs, prompt injection and data leakage – categories that traditional security testing does not cover.

Building AI Testing Capability

Perhaps the most practical challenge facing engineering leaders is where AI testing capability should sit organisationally and what skills it requires.

AI evaluation requires a blend of competencies. It requires understanding of ML evaluation methodology, such as benchmark design, statistical analysis of non-deterministic outputs and evaluation metric selection. It also requires domain expertise to define what "correct" means for specific use cases – a question that is ultimately a business judgement rather than a technical one. As well as this, it requires prompt engineering capability to design effective evaluation prompts and adversarial test cases. And lastly, it requires the infrastructure skills to build and run evaluation pipelines at scale, integrate monitoring into production systems, and maintain evaluation datasets as the system and its usage evolve.

Some organisations are embedding this capability within existing QA teams, extending their remit to encompass AI evaluation alongside traditional testing. Others are building dedicated AI quality or AI evaluation functions, sometimes within ML engineering teams, sometimes as standalone roles. Neither approach has emerged as clearly superior. The right answer depends on the organisation's AI maturity, the scale and criticality of its AI deployments, and whether the dominant challenge is evaluation methodology (which favours ML expertise) or integration with existing quality processes (which favours QA expertise).

What is clear is that there is a skills gap. The World Quality Report found that 50% of organisations lack AI/ML expertise, unchanged from the prior year, and that generative AI has emerged as the single most in-demand skill for quality engineers (63%), ahead of core quality engineering fundamentals (60%).

PractiTest's State of Testing 2026 data reinforces this from the practitioner perspective. Testing professionals who actively use AI tools are significantly less anxious about their future and earn a measurable salary premium, suggesting that the market is already pricing in AI evaluation capability.

From Optional to Essential

The window during which AI testing could be treated as an emerging discipline is closing. Organisations are deploying LLM-powered systems into production, customers and employees are interacting with them daily, and the failure modes are documented and increasingly expensive.

The hallucination rates are quantified, with even frontier models exceeding 10% on rigorous benchmarks. The regulatory requirements are specific, with the EU AI Act mandating testing that most organisations cannot yet perform. And the deployment patterns are growing more complex, with RAG systems compounding retrieval and generation failures, while agentic workflows are introducing autonomous decision-making with real-world consequences.

The Veracode research on AI-generated code security showed the same pattern – newer, larger models do not produce more secure code, highlighting that these are not problems that will be solved with the next model release. Instead, teams require exploration and investment into testing capability, evaluation infrastructure and the organisational capacity to assess and manage the risks inherent in deploying probabilistic systems.

Author

Richard Brown is the Technical Director at Audacia, where he is responsible for steering the technical direction of the company and maintaining standards across development and testing.

Why AI Governance is Key to Scaling AI

Audacia — Mon, 16 Mar 2026 10:00:00 +0000

Governance is the aspect of AI that most reliably triggers resistance from delivery teams. The perception, which can often be well-founded in experience, is that governance means delays, committees, paperwork and risk management leading to blockers.

This perception is understandable but can lead to significant risk. Understandable, because many organisations have governance frameworks that aren’t necessarily suited to the iterative, experimental nature of AI development. However the absence of governance does not eliminate risk, but rather it means that risks are often uncovered in production, where the consequences are most severe and the cost of remediation is highest.

The organisations that are scaling AI successfully have resolved this tension - not by choosing between speed and governance, but by fundamentally rethinking what governance means in the context of AI. They have made it proportionate, embedded and automated, with the evidence showing that this approach can help to accelerate delivery, not slow it down.

The Cost of Ungoverned AI

McKinsey's 2025 State of AI survey found that 51% of organisations report at least one negative AI-related incident in the past 12 months. The most commonly cited incidents involved inaccuracy, followed by compliance failures, reputational damage, privacy breaches and unauthorised actions by AI systems.

These are risks affecting the majority of organisations deploying AI at any meaningful scale, and they are growing. The average organisation is now actively managing around four types of AI risk, up from approximately two in 2022, with inaccuracy, cybersecurity, privacy and regulatory risk most frequently addressed. Explainability – the ability to understand and explain why an AI system produced a particular output – stands out as a risk that many organisations experience but fewer have robust controls for.

Deloitte's 2026 State of AI in the Enterprise report adds a governance dimension specific to the emerging agentic AI frontier, with only one in five companies having a mature governance model for autonomous AI agents. As AI systems move from answering questions to taking independent action, the governance gap becomes a genuine operational risk.

The business case for governance is fundamental to building the organisational trust required to scale AI beyond pilots. Without governance, boards can hesitate to approve production deployment, business stakeholders can question the reliability of AI outputs, regulators can ask questions that cannot be answered, and individual AI initiatives that might otherwise create value remain confined to sandboxes because teams don’t have the confidence to release them.

The EU AI Act: A New Regulatory Baseline

The most significant regulatory development for enterprise AI is the EU AI Act – the first comprehensive AI legislation globally. Its phased implementation timeline is now well underway and directly affects any organisation operating in or serving EU markets.

The Act entered into force on 1 August 2024. Within this, prohibited AI practices – including social scoring and certain forms of biometric categorisation – have been banned since February 2025. Obligations for general-purpose AI (GPAI) models, including transparency and documentation requirements, became applicable in August 2025. The penalty regime is now active, with fines of up to €35 million or 7% of global turnover for prohibited practices, and up to €15 million or 3% for other infringements.

The most consequential deadline for enterprises is August 2026, when the comprehensive compliance framework for high-risk AI systems takes effect. This covers AI used in areas including biometrics, critical infrastructure, education, employment, essential services, law enforcement and border management. Organisations deploying AI in these domains will need to demonstrate risk management systems, data governance measures, technical documentation, human oversight mechanisms and conformity assessments.

For UK organisations, the Act has extraterritorial reach - if the output of an AI system is used within the EU, the obligations apply regardless of where the provider is based. Any UK enterprise with EU customers, operations or supply chain connections should therefore understand and plan for compliance.

The UK Approach: Principles-Based but Tightening

The UK has deliberately chosen a different path from the EU's prescriptive legislation. As of early 2026, the UK has not adopted a single cross-economy AI law. Instead, it relies on existing sector regulators to apply current frameworks to AI within their domains – a principles-based, outcomes-focused approach.

In financial services – the UK sector furthest advanced in AI adoption – this approach is well-articulated. The FCA confirmed in December 2025 that it will not introduce AI-specific rules, citing the technology's rapid evolution. Instead, it relies on existing frameworks including the Consumer Duty, Senior Managers and Certification Regime (SM&CR), and operational resilience requirements. FCA's position is that these technology-agnostic frameworks already cover the key risks associated with AI deployment – accountability, transparency, consumer protection and resilience.

The Bank of England and FCA's third survey of AI in UK financial services, published in November 2024, found that 75% of firms are already using AI, with a further 10% planning to adopt within three years. Foundation models account for 17% of use cases, though most deployments remain low materiality. Lloyds' 2025 Financial Institutions Sentiment Survey reported that 59% of institutions now see measurable productivity gains from AI, up from 32% a year earlier.

But "principles-based" does not mean "relaxed." The FCA's Chief Data Officer has noted that advances in AI may require modified approaches to firm risk management and governance, and that regulation will need to adapt. The Treasury Committee published a report on AI in financial services in January 2026, examining both opportunities and risks. And the UK government appointed two AI Champions for financial services – signalling that regulatory attention is intensifying.

For organisations outside financial services, the landscape is less codified but no less important. The ICO's existing guidance on automated decision-making under UK GDPR applies to any AI system that processes personal data. Sector-specific regulators in healthcare (MHRA, CQC), energy (Ofgem), and other domains are developing their own positions. And the UK government's AI Opportunities Action Plan, published in early 2025, signals a direction of travel toward greater expectations around safety, transparency and accountability – even without prescriptive legislation.

The practical implication for UK enterprises is that the absence of an AI-specific law does not mean the absence of regulatory obligation. Existing frameworks already create accountability for AI outcomes, and the direction of travel – both domestically and through the extraterritorial reach of the EU AI Act – is clearly toward greater scrutiny.

Three Principles for AI Governance

The organisations succeeding with AI governance share three design principles that distinguish their approach from the heavyweight, process-oriented governance models that have historically frustrated delivery teams.

Proportionate governance

Proportionate governance calibrates the level of oversight to the level of risk. Not every AI application carries the same risk profile. A model that recommends internal knowledge articles requires a fundamentally different governance posture than a model that makes credit decisions or informs clinical diagnoses.

A practical risk-tiering framework – typically three or four tiers – allows low-risk use cases to move quickly with lightweight review, while high-risk applications receive the scrutiny they demand. The key dimensions for tiering include: the impact on individuals if the model produces an incorrect output, the regulatory sensitivity of the domain, the degree of human oversight in the workflow and the nature of the data being processed (particularly personal or sensitive data). This approach avoids the bottleneck of treating every AI initiative as though it were mission-critical.

Embedded governance

Embedded governance builds compliance checks into the development process rather than imposing them as a gate at the end. This includes bias testing as part of model evaluation, data privacy assessments as part of pipeline design, explainability requirements as part of model selection and risk assessment as part of use case approval.

When governance is embedded, it does not create a bottleneck at deployment. Instead, it prevents the far more costly rework that comes from discovering compliance issues after a model has been built, tested and handed to the operations team. The shift is from governance as a stage gate to governance as a continuous practice – present throughout the development lifecycle, not concentrated at a single approval point.

Automated governance

Automated governance leverages tooling to enforce standards without human bottlenecks. Automated checks for data quality thresholds, model performance metrics, bias indicators and audit logging can be built into CI/CD pipelines, ensuring that governance is consistently applied without requiring manual review for every model update or retraining cycle.

Cisco's AI Readiness Index found that 97% of the most AI-ready organisations ("Pacesetters") deploy AI at the scale and speed necessary to realise value, compared to just 41% overall – and that 84% of these Pacesetters have comprehensive change management plans, versus 35% of all companies. This highlights that governance and speed are not in tension for the most advanced organisations, they can in fact be mutually reinforcing.

Building the Governance Framework: Components

For organisations looking to establish or strengthen their AI governance, several components form the foundation.

An AI risk register and use case inventory

Before governance can be applied proportionately, the organisation needs visibility into what AI is being used, where and at what risk level. This sounds quite simple, but many organisations – particularly those where AI adoption has been bottom-up and decentralised – lack a comprehensive view of their AI estate. The inventory should capture each use case, its risk tier, its data sources, its intended users and its current lifecycle stage.

Clear roles and accountability

Governance requires named individuals accountable for AI risk. In the UK financial services context, the SM&CR already provides this structure – the Senior Manager responsible for AI outcomes is personally accountable. Outside regulated sectors, the principle still applies: someone senior must own AI governance, with authority to approve, escalate or halt deployments based on risk assessment.

Model documentation standards

Each AI model in production should be accompanied by documentation covering its purpose, training data, performance metrics, known limitations, bias assessments and monitoring arrangements. This documentation serves multiple purposes – it enables effective oversight, supports regulatory compliance, facilitates knowledge transfer when team members change and provides the audit trail that boards and regulators increasingly expect.

Monitoring and incident management

Governance does not end at deployment. Production AI systems require ongoing monitoring for model drift (degradation in performance as real-world data diverges from training data), data quality issues, emerging biases and unexpected behaviours. A clear incident management process – defining how AI-related issues are detected, escalated, investigated and remediated – is essential, particularly given how many organisations have already experienced at least one negative AI incident.

Regular review and adaptation

The governance framework itself should evolve. The regulatory landscape is changing rapidly – the EU AI Act's high-risk obligations take effect in August 2026, UK regulatory expectations continue to sharpen, and the technology itself is advancing at pace. A governance framework designed for today's AI capabilities will need updating as agentic systems, multimodal models and new deployment patterns continue to evolve.

Governance as Competitive Advantage

It is tempting to view governance as a cost centre – an overhead imposed by regulators and risk committees that can add little to the value AI delivers.

However, governance is what gives the board confidence to approve production deployment, as well as allow the use of AI in customer-facing and decision-critical contexts rather than confining it to internal experimentation. And it prevents the compliance complexity when regulatory expectations tighten.

BCG's research found that AI leaders follow a 10-20-70 resource allocation: 10% to algorithms, 20% to technology and data and 70% to people and processes – the category that includes governance, change management and organisational readiness. With the organisations investing most heavily in governance the same ones generating the most value from AI.

The lack of governance can be one of the main reasons that AI projects stall. It can lead to eroding trust, increased compliance risk, rework, ultimately keeping promising AI initiatives confined to sandboxes. However, if governance is built it in from the start, proportionate to risk, embedded in the development lifecycle and automated where possible, it can be the element that leads to production success.

Author
Chris is a Lead Data Scientist, with a background in astrophysics, and has over 4 years’ experience in providing data strategies insights using computational models and machine learning methodology. Chris has worked with a number of organisations across industries to successfully deliver AI projects, from PoC development and use case validation, through to model training and maintenance.

Managing Hidden Waterfalls in Legacy Modernisation Projects

Audacia — Mon, 09 Mar 2026 08:30:00 +0000

Why agile delivery fails in legacy heavy environments without structural preparation.

Agile remains the dominant model for modern software delivery for good reasons. Iterative development, fast feedback loops and the ability to adapt to new information are essential in complex, evolving systems. However, when agile is introduced into legacy-heavy organisations without accounting for institutional constraints, its effectiveness can diminish over time. What begins as an agile programme can often shift imperceptibly into a sequential delivery model beneath the surface.

In 2019, 29% of organisations reported using waterfall delivery models. By 2022, that figure had risen to 43%. Not because teams chose to abandon agile, but because the environments they were delivering into quietly forced the shift.

Teams start with discovery and prototyping, iterate rapidly and validate assumptions early. But as delivery progresses, unaddressed constraints begin to emerge, such as undocumented legacy behaviours, regulatory edge cases or operational workarounds that were never captured as formal requirements. At this point, the legacy system reasserts itself as a source of truth.

Agile ceremonies may continue, but the programme becomes more reactive. The goal can subtly shift from solving user problems to reproducing historical behaviour. What remains is a hybrid model becoming agile in appearance, yet waterfall in substance - the hidden waterfall.

Legacy Replacement as High-Risk

The data on legacy modernisation is consistent. These programmes fail more often, and more visibly, than greenfield initiatives.

A review of ERP project outcomes by KPC Team places failure or severe underperformance rates between 55% and 75%, depending on scope and definition. With ERP Today highlighting testing, data quality and scope volatility as common points of failure.

Data migration projects carry even higher risk. According to Oracle, over 80% of data migration initiatives either overrun, underdeliver or fail entirely. Most often due to undocumented dependencies and inadequate validation. Factors such as schema drift, semantic inconsistencies and legacy entanglement are reported as persistent blockers to successful transformation.

Most digital transformation efforts in large organisations are not pure greenfield builds. They are legacy replacement or coexistence programmes - subject to all the structural, technical and operational complexity that entails.

Why Legacy Becomes the Specification

A common misstep in legacy modernisation is the assumption that existing systems simply encode outdated implementations of known requirements. In practice, legacy systems carry decades of organisational memory, much of it undocumented.

Research in requirements engineering reveals several persistent patterns in legacy systems:

Business logic is embedded in code rather than documentation
Exceptions are handled through hidden branches or procedural workarounds
User behaviours evolve around system constraints, becoming de facto requirements

When delivery teams attempt to define new system requirements without examining these embedded behaviours, they quickly encounter gaps. At that point, the legacy platform is no longer a background dependency - it becomes the only available reference model.

Studies published in IJACSA and Springer show that late discovery of implicit requirements is a leading cause of rework. In legacy replacement programmes, these “requirements” were never made explicit because they were never formally captured.

This is a structural outcome of relying on systems that evolve without parallel investment in shared knowledge.

Water-Scrum-Fall

In 2011, Forrester introduced the term “Water-Scrum-Fall” to describe hybrid delivery models in which agile practices are embedded between upfront planning and downstream release governance. More than a decade later, this pattern persists, and if anything, it has increased.

CIO Dive reported in 2022 that 43% of organisations still use waterfall models, up from 29% in 2019, with compliance, assurance and funding structures cited as the main reasons. KnowledgeHut’s 2025 State of Agile found that agile adoption is now stagnating or reversing in many enterprise environments, with hybrid models becoming the norm.

These regressions are rarely ideological. Most organisations want to be agile. But delivery becomes sequential by structure.

Why Waterfall Reappears by Default

Several factors can pull agile programmes toward waterfall behaviours:

Funding cycles require fixed scope and budget commitments before discovery
Governance models rely on stage gates, rather than continuous assurance
Supplier contracts focus output completion over outcome delivery
Compliance processes are serial in nature, with formal sign-offs and audit trails

In the public sector, the National Audit Office has repeatedly highlighted how legacy estates, inflexible procurement and capacity gaps create barriers to agile working. The State of Digital Government Review 2025 confirms that many central government services still rely on systems more than two decades old, with modernisation constrained by high operational risk and fragile dependencies.

In this environment, teams may adopt agile practices within their sprint cycles, but the programme remains governed by linear constraints, creating hidden waterfalls.

Recognising Hidden Waterfalls Before They Set In

Hidden waterfalls rarely announce themselves. They emerge gradually, often masked by functioning agile rituals. But several indicators can signal that a programme has shifted from iterative delivery to sequential progression:

Sprint goals are increasingly defined by legacy parity rather than user outcomes. Backlog items begin to reference "the old system does X" as the primary acceptance criterion, rather than solving a validated user need.
Discovery stops but requirements keep growing. The team completed a discovery phase early in the programme, but new requirements continue to surface from legacy behaviours that were never formally captured. Each one is treated as an exception rather than evidence of a structural gap.
Release planning compresses into a single milestone. Despite iterative development, the programme converges on a single go-live date with limited rollback options, often driven by contract, funding or political commitments rather than technical readiness.
Testing becomes regression-dominant. The majority of test effort shifts toward proving that the new system reproduces existing behaviour, rather than validating that it meets redefined needs.
Stakeholder confidence depends on sign-off, not evidence. Progress is measured by stage-gate approvals and documentation completeness rather than working software, user feedback or operational metrics.

None of these indicators are necessarily failures in their own right. However, when several appear together, they suggest the programme has structurally reverted to sequential delivery, regardless of the methodology it reports.

Modernisation Approaches

Projects that aim to replace legacy systems in a single release, often called “big bang” delivery, assume considerable risk. These programmes concentrate delivery dependencies, limit rollback options, and make data migration a single-point failure.

Incremental modernisation strategies can offer a more resilient alternative. Patterns such as parallel run, feature toggles, coexistence architectures and the strangler fig pattern allow systems to be evolved rather than replaced outright.

In one survey, 79% of developers said the strangler pattern reduced project risk, primarily because it isolates change and supports rollback. Incremental delivery also aligns better with governance and assurance frameworks - supporting progressive certification, staged user validation and controlled data migration.

In regulated environments, these approaches can reduce disruption and support operational continuity. They also provide decision-makers with clearer evidence of progress and outcomes at each stage.

Balancing Ambition with Legacy Constraints

Preparing for hidden waterfalls is not an argument for replicating legacy systems. It is a call to interrogate them more rigorously, and to distinguish between what must be retained and what can be rethought.

This involves:

Identifying which behaviours are regulatory, contractual or operationally essential
Separating business-critical rules from historical conveniences
Defining the minimum viable increment that preserves service capability while allowing change
Designing systems that support evolution rather than frozen replication

These assessments are less suited to be completed through workshops or documentation. Instead they require early and direct engagement with legacy systems, their data models, codebases, interface behaviours and operational roles.

The Role of AI in Surfacing Constraints

AI-assisted tooling offers practical support in navigating legacy complexity. When applied responsibly, these tools can help teams:

Analyse code to extract business rules and logic paths
Identify unused or redundant code segments
Map dependency chains and integration points
Generate automated tests to capture existing system behaviours

In environments where documentation is sparse and institutional memory has faded, these tools can reduce the time and effort needed to understand legacy systems. Oracle’s whitepaper notes that poor understanding of legacy code is a major cause of data migration failure, an area where AI-driven code analysis can make a measurable difference.

However, it is important to view AI as an enabler, not a decision-maker. Tools can help surface logic and dependency, but they can struggle to decide which behaviours remain relevant or valuable. That task requires domain knowledge, user insight and human judgement.

Preparing for Hidden Waterfalls

Effective preparation involves a combination of technical, governance and delivery decisions:

Map structural constraints early, particularly data, regulatory and legacy integrations
Treat legacy systems as evidence, not default specifications
Select modernisation approaches that allow co-existence and rollback
Align governance and assurance models to tolerate incremental delivery
Use AI tools to reduce manual analysis effort and highlight legacy dependencies

The objective is not necessarily to eliminate all waterfall elements but to make them visible and manageable. Programmes that fail to do this often discover late in delivery that they are operating under assumptions that no longer hold, or that were never articulated to begin with.

Why This Step Is Important

Hidden waterfalls are a predictable outcome of unaddressed structural constraints that agile methods alone are not enough to resolve.

Acknowledging this reality early allows teams to structure programmes that are responsive, transparent and recoverable. It enables more realistic delivery planning, supports operational continuity and improves trust between teams and stakeholders.

This step becomes particularly important when delivery timelines are fixed, data quality is uneven or regulatory scrutiny is high.

Author

Matt Cross is a Lead Business Analyst at Audacia. Matt has a background in leading requirements workshops, defining acceptance criteria for requirements and supporting stakeholders throughout the project lifecycle – on both consultancy and development projects across engineering, data, AI and cloud.

Serverless Architectures: Designing for Scale, Simplicity and Resilience

Audacia — Mon, 19 Jan 2026 08:30:00 +0000

Modern applications face a constant tension between competing architectural demands: systems must scale efficiently, remain highly available, perform well under load and be maintainable without excessive operational overhead.

This blog, adapted from a Tech Talk by Principal Software Engineer, Luke Mitchell, explores cases where serverless architectures can address these requirements by shifting infrastructure management to cloud providers, allowing development teams to focus on building features rather than managing servers.

Scaling Strategies: Horizontal vs Vertical

Understanding scaling approaches provides the foundation for appreciating serverless benefits. Vertical scaling involves adding resources to a single instance - more CPU, RAM or storage. A down-side of this approach is that it is limited by a single point of failure. Meaning when that machine goes down, the entire service becomes unavailable.

Horizontal scaling takes a different approach by adding more instances of the same machine. This design provides built-in fault tolerance because multiple machines handle requests simultaneously. Therefore, if one machine fails, others continue serving traffic. This redundancy makes horizontal scaling more resilient than vertical scaling, though horizontal scaling can include more complexity in orchestration and load distribution.

Case Study 1: From Monolithic Functions to Distributed Processing in Azure

The overhaul of this file processing system demonstrates how serverless patterns transform architecture. The initial implementation, as shown above, used a single Azure Function that continuously ran, grabbing files from an SFTP server, processing each entry sequentially and writing results to a Snowflake database. This design had several limitations: it scaled only vertically, created a single point of failure and left no clear recovery path when errors occurred mid-process.

To mitigate these limitations the system architecture was refactored, as shown above – after which responsibilities were split across multiple components. The initial function now simply reads the file and splits each entry into individual messages on a storage queue. As messages arrive, a second function automatically scales up to process multiple instances in parallel. This distribution transforms sequential processing into concurrent execution, dramatically improving throughput speed.

As well as speed, the switch to a serverless architecture also improves fault tolerance. If a function instance fails mid-processing, the message automatically returns to the queue for retry. Messages that consistently fail move to a poison queue for manual investigation, preventing problematic entries from blocking the entire pipeline. Unlike the original architecture, the need to track processing state within files or implement complex restart logic is eliminated because the queue handles these concerns automatically.

Case Study 2: Serving Static Content at Scale in AWS

Traditional web server architecture requires substantial infrastructure, as can be seen in the diagram above. Firstly, requests made by users are distributed by an application load balancer across EC2 instances deployed in multiple availability zones. Next, auto-scaling groups monitor traffic and adjust instance counts accordingly, adding capacity during peaks and removing it during lulls to control costs. Each virtual machine incurs a cost whether actively serving requests or sitting idle.

The serverless alternative simplifies this substantially. Static files reside in an S3 bucket, with CloudFront serving as the access point. CloudFront operates as a content distribution network with edge locations worldwide. When users request content, they receive it from the nearest edge location rather than travelling back to the origin region. This geographic distribution reduces latency significantly for global audiences.

This serverless approach has benefits for performance, maintainability and scalability:

S3 stores files across multiple availability zones by default and if one zone becomes unavailable, requests route to files in other zones without manual intervention.
CloudFront caches content at edge locations, reducing origin server load and improving response times.
The entire stack scales to handle traffic spikes without configuration changes or capacity planning.

Case Study 3: API Infrastructure Without Servers in AWS

API servers typically follow similar patterns to web servers: virtual machines behind load balancers, deployed across availability zones for resilience. This infrastructure requires ongoing maintenance - operating system patches, image updates, capacity planning and monitoring.

API Gateway provides a serverless alternative, acting as a unified entry point for API traffic. It integrates directly with numerous AWS services: Lambda functions for compute, DynamoDB for database access, SQS for message queuing and SNS for publish-subscribe patterns. This integration flexibility enables varied architectural patterns without managing underlying infrastructure. This makes initial start-up easier than it would be using a virtual machine.

The publish-subscribe model through SNS demonstrates particular power. A single message can fan out to multiple subscribers - perhaps a Lambda function sending notifications to Slack while simultaneously queuing work for asynchronous processing. This pattern enables event-driven architectures where services respond to events without tight coupling between components.

This approach is also accompanied by benefits:

Multi-availability zone deployment happens by default.
The platform automatically handles failover and scaling without explicit configuration.
Updates don't require creating new machine images or coordinating rolling deployments across instances.
The pay-per-use model means costs align directly with actual usage rather than provisioned capacity.

Trade-offs and Considerations

Serverless architectures introduce their own considerations. For example, cold starts, the latency when a function first initialises, can affect user experience through lagging. This can be mitigated at an additional cost through provisioned concurrency (keeping a specified amount of Lambda expressions always running) to keep functions warm. This trade-off matters most for latency-sensitive applications where milliseconds count.

Additionally, cost efficiency should be considered, which depends on scale. Serverless platforms charge per request, making them economical for variable workloads. At extremely high sustained volumes, dedicated infrastructure may become more cost-effective. However, this typically occurs only at the scale of major internet services.

Some use cases still favour traditional servers. Such as long-running processes, which don't map cleanly to function execution models. For example, server-side rendering requires a server to generate HTML dynamically, which S3 and CloudFront cannot provide. Static site generation or pre-rendering can address some of these scenarios, but pure static hosting has SEO limitations without additional tooling.

A learning curve exists for both serverless and non-serverless approaches. To use servers, understanding load balancers, auto-scaling groups and virtual machine maintenance requires expertise. On the other hand, serverless architectures require different knowledge - message queues, function composition and event-driven design. Teams should evaluate their existing skills and strategic direction when choosing approaches.

Conclusion

Serverless architectures deliver meaningful advantages in scalability, performance, fault tolerance and maintainability. By abstracting infrastructure management, they enable teams to focus on application logic rather than operational concerns. While not universal solutions, they provide compelling benefits for most modern applications, particularly those with variable traffic patterns or limited operations resources. The examples demonstrate that serverless patterns often simplify rather than complicate architecture, delivering better results with less overhead.

Watch the Tech Talk

Putting the CD Back into CI/CD: A Guide to Continuous Deployment

Audacia — Mon, 12 Jan 2026 08:30:00 +0000

Putting the CD Back into CI/CD: A Guide to Continuous Deployment
Many organisations talk about CI/CD, but the reality is that most have achieved continuous integration (CI) without continuous deployment (CD). Embracing both CI and CD represents a fundamental shift in how software reaches production and how teams approach risk, quality and delivery.

This blog, adapted from a Tech Talk by Principal Software Engineers Luke Mitchell and Akeel Ahmed, explores two distinct pathways to achieving true continuous deployment: trunk-based development with ephemeral environments, and Git Flow with structured release management. Both approaches can deliver frequent, reliable releases, but they require different technical infrastructure, and cultural and organisational readiness.

The Current State of Continuous Deployment

Continuous integration has become standard practice through automated testing, code reviews and build pipelines. However, the journey from merged code to production often remains batched and infrequent.

The reasons are varied: legacy approval processes inherited from waterfall methodologies, lack of confidence in automated testing, concerns about deployment risk or simply the complexity of managing multiple environments. Yet the benefits of genuine continuous deployment – faster feedback loops, reduced integration risk and the ability to respond rapidly to business needs – make it worth pursuing.

Trunk-Based Development

At the core of trunk-based development is a single principle: the main branch should always be in a deployable state.

The Fundamentals

Unlike Git Flow's multiple long-lived branches, trunk-based development maintains a single main branch. Developers work on short-lived feature branches, typically lasting hours or days rather than weeks, before merging back to main. Each merge triggers an automated pipeline that can deploy directly to production.

This approach demands discipline. Small, focused commits become essential. Code reviews must happen synchronously – within 10 to 15 minutes of raising a pull request. The entire team must prioritise getting code through the pipeline over starting new work.

Ephemeral Environments

One of the most powerful enablers of trunk-based development is the use of ephemeral environments. Rather than maintaining static QA and staging environments where multiple developers' changes intermingle, each feature branch spawns its own temporary environment.

When a developer pushes their branch, the pipeline automatically provisions cloud infrastructure and deploys their changes to an isolated environment. This provides several advantages:

Isolation of changes: Bugs discovered during testing are definitively linked to the development stage.

Parallel development: Developers and testers can work simultaneously without interference, removing bottlenecks from the development and QA processes.

Cost efficiency: Environments are taken down automatically after the code merges to main, ensuring resources are only consumed when needed.

Production parity: Each ephemeral environment can mirror production configuration, reducing environment-specific issues.

The workflow is streamlined: Development happens on the feature branch, code review occurs when the developer opens a pull request, QA testing happens in the ephemeral environment once the PR is approved, and upon successful testing, the code merges to main and the ephemeral environment is deleted.

Release Strategy for Trunk-based Development

Merging to main doesn't necessarily mean immediate production deployment, though it could. Many teams create release candidate branches automatically upon merge to main. These branches can then be deployed to UAT or production based on business requirements.

Teams take different approaches to deployment timing. Some deploy every merge to production immediately – true continuous deployment. Others batch a few tickets together, deploying the most recent release candidate branch that contains all the desired changes. The key principle remains constant: all code in main is production-ready, and the organisation decides when to deploy based on business needs, not technical readiness.

To track what's currently live, many teams maintain a production branch, merging their release candidate branches into it after deployment. This provides a valuable snapshot of the live environment, simplifying rollbacks and hotfixes by maintaining a known good state to return to. Teams requiring additional safeguards sometimes create rollback candidate branches automatically before each production deployment, though this adds complexity that not all teams need.

Feature flags provide an additional layer of deployment control that works with both trunk-based development and Git Flow. They're particularly valuable in trunk-based development, where code deploys to production frequently, by controlling feature visibility independently of code deployment.

The Benefits

Risk mitigation: pinpointing bugs becomes easier in smaller, recent releases.
Early client/user feedback: a clients’ vision can change or become clearer when presented with something concrete – it’s best to know as early as possible.
Reactive to change: small releases reduce the amount of time and difficulty it takes to get feedback and implement changes.

The Cultural Requirements

Trunk-based development requires significant cultural change. It demands trust that developers will maintain quality, that automated tests are comprehensive and that the team will respond quickly to production issues.

It also requires scaling back bureaucratic approval processes. Change Advisory Boards can be antithetical to continuous deployment if every change is scrutinised. The governance must shift from manual approval gates to automated quality gates and rapid response capabilities.

Full team ownership becomes paramount. From junior developers to tech leads, everyone shares responsibility for production stability. This shared accountability, combined with the practice of deploying small changes frequently, reduces risk compared to large, infrequent releases.

Git Flow

Not every organisation can adopt ephemeral environments immediately. Infrastructure constraints, compliance requirements or existing tooling may necessitate static environments. Git Flow provides a structured approach to continuous deployment within these constraints.

The Git Flow Model

Git Flow employs multiple long-lived branches with specific purposes:

Main reflects production and is updated only with tested, stable releases.
Release branches are cut from develop to deploy to production.
Develop serves as the integration branch for ongoing development.
Feature branches are created from develop for new functionality.
Bugfix branches are short-lived branches created from develop or release to fix defects, and are merged back into their source branch once resolved.
Hotfix branches are created from main for urgent production fixes, and merged back into main and develop.

This structure provides clear separation between development, testing and production code. Whilst feature branches can be short-lived with good continuous integration practices, the methodology naturally supports more structured release cycles.

Release Planning and Management

Success with Git Flow depends heavily on release planning and management. Rather than ad-hoc deployments, teams batch related user stories into planned releases. This upfront planning – tagging stories with release identifiers early in the sprint – provides predictability for stakeholders whilst still enabling frequent releases.

The workflow operates in distinct phases:

Development phase: Developers merge feature branches to develop, which automatically deploys to a shared QA environment for testing.

Release preparation: When all features for a release are complete and QA-tested, a release branch is created from develop.

UAT phase: The release branch is deployed to UAT for stakeholder testing. Crucially, no new features are added during this phase – only bug fixes and refinements.

Production deployment: After successful UAT, the release branch deploys to production and merges back to main, providing a live reflection of production in the codebase.

Managing Hotfixes

Git Flow excels at handling production issues whilst development continues. Hotfix branches are created from main, tested independently, and deployed to production without disrupting the develop branch or ongoing releases.

A practical versioning approach helps manage this: if release 1.0 is in production and a bug is discovered, create hotfix branch 1.1, deploy it to production, then merge it back to both main and develop to keep everything aligned.

The Advantages of Structure

Git Flow's structure provides several benefits for teams and stakeholders:

Stability: The main branch always reflects production, reducing confusion about what code is live.

Visibility: Clear branching structure makes it easy to understand what features are in which release.

Control: Product owners and project managers have explicit control over what gets released and when.

Transparency: Every merge, tag and deployment is logged, providing an audit trail for accountability.

This structure particularly benefits larger teams where multiple developers work on the same codebase simultaneously. The isolation between branches provides clearer separation of concerns and reduces the risk of unstable code reaching production.

Choosing Your Approach

In a simple analogy, trunk-based development can be imagined as multiple passengers in different taxis heading to the same destination, whereas Git Flow involves a group of passengers on a bus – going through each checkpoint together. The decision between the two strategies depends on infrastructure capabilities and business requirements.

Trunk-Based Development Fits When:

You have cloud infrastructure that supports ephemeral environments
Your team is comfortable with high deployment frequency
Automated testing provides high confidence
There's organisational trust in the development team
Small, incremental releases align with business needs
You want to minimise the feedback loop between development and production

Git Flow Fits When:

You have static environments that can't easily be replicated
Releases need stakeholder approval or coordination
Compliance requires structured release documentation
Larger teams benefit from clear branch isolation
Business prefers predictable, planned release schedules
You're transitioning from traditional release processes

Neither approach is inherently superior. Both can achieve continuous deployment if implemented well. The key is matching the approach to your context and executing it with discipline.

Making It Work: Practices

Regardless of which strategy you choose, a few practices are essential for successful continuous deployment:

Automated Testing as a Foundation

Quality gates must be automated and comprehensive. Unit tests, integration tests and UI tests should run automatically on every commit. These tests become your confidence in deployment – they must be reliable and fast.

Synchronous Code Reviews

Code reviews can't be allowed to become bottlenecks. Establishing the expectation that pull requests receive attention within 15 minutes keeps code flowing.

Communication and Collaboration

Continuous deployment requires continuous communication. Development teams and testers must collaborate closely, using tools like Slack, Teams or Azure DevOps to stay coordinated. Early feedback loops with product owners and clients help ensure that frequent releases deliver what stakeholders want.

Monitoring and Observability

When deploying frequently, you must know immediately if something goes wrong. Comprehensive monitoring, alerting and logging become essential. The ability to quickly diagnose and resolve production issues provides the confidence to deploy often.

Next Steps

Whether through trunk-based development's simplicity or Git Flow's structured approach – start small. Each increase in deployment frequency teaches lessons about improvements that can be made to testing, automation, monitoring or process.

Moving from continuous integration to genuine continuous deployment represents a significant evolution in development maturity. It requires technical investment in automation and infrastructure, cultural change in how teams approach quality and risk, and organisational trust in development practices.