The public conversation about artificial intelligence often sounds like a competition between models and applications.
Which model is the most intelligent?
Which chatbot gives the best answers?
Which agent can complete the most complicated task?
Which copilot, search engine, or creative tool will attract the most users?
These are important questions, but they describe only the most visible parts of the AI economy.
A model does not train itself. It does not power the machines beneath it, manage its own memory, expose a dependable API, recover from failures, or deliver an application to a user. Every model depends on a much larger industrial system.
NVIDIA CEO Jensen Huang has described this system as a five-layer cake:
Energy → Chips → Infrastructure → Models → Applications
The metaphor is useful because each layer depends on the layer below it. Energy powers chips. Chips need infrastructure that makes their compute usable. Infrastructure enables models to be trained and served. Models power applications. Applications are where economic value becomes visible.
Models receive most of the attention, and applications create the visible experience. But the middle layers determine whether AI can run reliably, efficiently, privately, and at scale.
That middle—especially the runtime and infrastructure layer—is becoming one of the most important parts of the AI system.
The five-layer cake of AI
The full stack can be pictured like this:
┌──────────────────────────────────────────────┐
│ 5. Applications │
│ Agents, copilots, robotics, enterprise AI │
├──────────────────────────────────────────────┤
│ 4. Models │
│ LLMs, VLMs, speech, diffusion, sovereign AI │
├──────────────────────────────────────────────┤
│ 3. Infrastructure │
│ CUDA, ROCm, XLA, MLX, serving, orchestration │
│ Runtime, memory, networking, metrics, deploy │
├──────────────────────────────────────────────┤
│ 2. Chips │
│ GPUs, CPUs, NPUs, TPUs, Apple Silicon, ASICs │
├──────────────────────────────────────────────┤
│ 1. Energy │
│ Power, cooling, thermals, efficiency │
└──────────────────────────────────────────────┘
The boundaries between these layers are not always perfectly clean. Some technologies span multiple layers. A runtime may include hardware-specific libraries, model-execution systems, memory management, serving APIs, and operational controls.
But the framework reveals something that model leaderboards often hide: an improvement at one layer is useful only when the surrounding layers can support it.
A faster chip needs software capable of using it. A more capable model needs enough memory, bandwidth, power, and serving infrastructure. A sophisticated application needs predictable latency, availability, security, and cost.
AI is therefore not merely a software trend. It is a full-stack infrastructure transformation.
Layer 1 — Energy
AI begins as an energy problem.
Before a token can be generated or an image can be classified, electricity must power the processors performing the computation. That power must be generated, transmitted, converted, cooled, and managed.
At data-center scale, energy availability influences where AI infrastructure can be built, how large it can become, and how expensive it is to operate. Cooling systems, power distribution, grid capacity, and thermal management are no longer secondary facilities concerns. They are part of the AI architecture.
The same principle applies to local and edge AI, although the constraints appear differently.
A laptop, phone, robot, vehicle, or industrial device must operate within limits involving battery life, heat, acoustics, physical space, and sustained power consumption. A model that performs well for thirty seconds but causes severe thermal throttling may not be useful for a continuous workload. A local assistant that rapidly drains a battery may not be practical, regardless of benchmark performance.
Energy determines:
- how much computation is available;
- how long it can be sustained;
- where the computation can occur;
- how much it costs;
- and what cooling or thermal design it requires.
Every token, embedding, image, and agent action begins with energy.
Layer 2 — Chips
Chips transform energy into computation.
This layer includes general-purpose CPUs, highly parallel GPUs, neural processing units, tensor processors, language-processing accelerators, and specialized ASICs. It also includes the memory architecture and interconnects that determine how quickly data can move through the system.
Different workloads benefit from different forms of silicon.
CPUs remain important for orchestration, preprocessing, control logic, retrieval, tokenization, and general application work. GPUs excel at the parallel mathematical operations used throughout modern machine learning. Specialized accelerators may optimize particular classes of matrix operations, inference workloads, or power-constrained deployments.
But processor speed alone does not determine AI performance.
Memory capacity determines which models can fit. Memory bandwidth affects how quickly parameters and intermediate values can be accessed. Interconnect speed affects how efficiently work can be distributed across processors or machines. Thermal limits influence sustained throughput. Software compatibility determines whether developers can use the hardware at all.
NVIDIA GPUs dominate much of the data-center AI conversation because of their computational capabilities and the software ecosystem built around them. Apple Silicon illustrates another architectural direction: tightly integrated systems combining CPUs, GPUs, specialized engines, and unified memory for power-efficient local computation.
The key point is broader than either company:
The chip layer gives AI raw compute, but it does not automatically make that compute usable.
A powerful processor without compilers, libraries, execution frameworks, memory tools, and deployment systems is potential energy—not a complete AI platform.
Layer 3 — Infrastructure: the missing middle
Infrastructure is the layer that turns chips into usable systems.
At hyperscale, this layer includes:
- data centers;
- networking and interconnects;
- storage systems;
- cluster schedulers;
- distributed training systems;
- inference servers;
- deployment pipelines;
- monitoring;
- security controls;
- and failure-recovery mechanisms.
At the software-runtime level, it includes:
- hardware programming platforms;
- optimized mathematical libraries;
- model loaders;
- memory and cache management;
- batching;
- token streaming;
- request cancellation;
- timeout handling;
- health and readiness checks;
- metrics and tracing;
- authentication;
- local or network APIs;
- and application integration.
Infrastructure is the missing middle between “I have a chip” and “I have an AI product.”
It answers the practical questions that are often absent from a model demonstration:
How is the model loaded?
What happens when two requests arrive simultaneously?
How is memory reused?
Can the response stream incrementally?
Can a request be cancelled?
What happens when the process is alive but the model is unavailable?
How does an operator measure latency or diagnose a failure?
How is the system upgraded without breaking its consumers?
A notebook that produces one successful response proves that a model can run. It does not prove that the model can function as dependable infrastructure.
CUDA: hardware made programmable
CUDA is the classic example of infrastructure amplifying hardware.
NVIDIA describes CUDA as an accelerated-computing platform and software layer that enables applications to harness GPU computing. Its toolkit provides a development environment for building and optimizing GPU-accelerated applications, while the programming model gives developers a structured way to execute work across CPUs and GPUs.
The importance of CUDA extends beyond a single programming interface.
Around it grew compilers, optimized libraries, debugging tools, communication systems, training frameworks, inference optimizers, deployment patterns, and a large developer ecosystem. Developers did not need to treat every GPU as an isolated piece of hardware. They could build against a platform.
That platform compounded NVIDIA’s hardware advantage.
Powerful GPUs attracted software developers. More software increased the usefulness of the GPUs. Greater usefulness encouraged more adoption, investment, tooling, training, and optimization.
CUDA demonstrated a lasting principle:
A hardware advantage becomes much more powerful when it is paired with a strong software-infrastructure layer.
The same pattern can be seen throughout computing history. Hardware establishes what is theoretically possible. Infrastructure determines what is practically usable.
MLX: local compute also needs infrastructure
MLX offers a newer example from the local-computing direction.
MLX is an array and machine-learning framework designed for Apple Silicon. Its design includes unified memory, lazy computation, dynamic graph construction, and execution across supported CPU and GPU devices. In MLX’s unified-memory model, arrays occupy shared memory and can be operated on by supported devices without explicit transfers between separate CPU and GPU memory spaces.
These characteristics align with the architecture of Apple Silicon and make MLX useful for model experimentation, training, adaptation, and inference on a local machine.
More importantly, MLX demonstrates that AI infrastructure is not exclusively a data-center concern.
A single machine still needs an execution framework. It still needs model-loading conventions, memory management, prompt construction, caching, streaming, request handling, observability, and application-facing interfaces.
Unified memory may simplify data movement, but it does not automatically provide a production API. Lazy execution may improve flexibility, but it does not provide health checks. A model-generation library may produce tokens, but it does not necessarily provide authentication, overload protection, cancellation, deployment automation, or operational metrics.
MLX makes the underlying compute more accessible. A complete runtime layer makes that compute dependable for applications.
CUDA and MLX are not equivalent
CUDA and MLX should not be presented as direct competitors or equivalent platforms.
CUDA is a broad, mature accelerated-computing ecosystem centered on NVIDIA hardware. It spans programming models, compilers, libraries, tooling, and deployment environments across workstations, servers, clusters, and large data centers.
MLX is a newer machine-learning framework specifically designed around Apple Silicon and its architectural characteristics.
Their scopes, histories, ecosystems, and deployment environments are different.
But both illustrate the same foundational idea:
Chips need software infrastructure before models and applications can fully use them.
CUDA illustrates that principle at enormous ecosystem and data-center scale. MLX illustrates it in a local, integrated-computing environment.
Together they help make the infrastructure layer visible.
Layer 4 — Models
Models are the intelligence layer.
This includes large language models, vision-language models, speech systems, embedding models, diffusion models, coding models, rerankers, adapters, fine-tunes, and quantized variants.
Model development receives enormous attention because model capability is easy to compare and demonstrate. Parameter counts, benchmark results, context windows, multimodal features, and reasoning performance provide visible signals of progress.
But the strongest model is not automatically the most appropriate model.
A useful model must fit its environment.
That fit may involve:
- language fit: Does it understand the languages, dialects, and communication styles of its users?
- domain fit: Does it understand the terminology and workflows of medicine, law, engineering, finance, or another field?
- regulatory fit: Can it operate within relevant legal and compliance requirements?
- deployment fit: Can it run on the available hardware and infrastructure?
- cost-performance fit: Is its quality worth its latency, memory, energy, and financial cost?
- privacy fit: Can it operate without exposing sensitive information?
- governance fit: Can its data, behavior, updates, and access be controlled appropriately?
A smaller specialized model may be more useful than a larger general model when it runs faster, remains private, uses local terminology, and can be governed by the organization deploying it.
Sovereign models and sovereign infrastructure
Sovereign AI is often described at the national level: a country’s ability to develop and operate AI using its own infrastructure, data, workforce, and institutions. It is connected to the preservation of local languages, culture, values, and strategic autonomy.
Sovereign models are an important part of that capability. They may be trained or adapted using local languages, public-sector data, domestic expertise, or jurisdiction-specific requirements.
But sovereignty cannot stop at the model file.
A model is not genuinely sovereign when every request must leave the jurisdiction, every update is controlled externally, or the data path depends entirely on infrastructure that the institution cannot inspect or govern.
The sovereignty principle can also be applied more narrowly to companies, universities, research institutions, and communities. They may not seek national technological independence, but they often require control over their models, data, deployment locations, evaluation criteria, and operational policies.
Sovereign AI therefore depends on the layers beneath the model:
- Who owns or controls the infrastructure?
- Where are requests processed?
- Where is data stored?
- Who can inspect the logs?
- Who decides when the model changes?
- Can the system operate privately or locally?
- Can it continue operating if an external provider changes its pricing, policy, or availability?
Sovereign AI is not only about who trains the model. It is also about where the model runs, who controls the data path, and whether the infrastructure supports private, local, or jurisdiction-specific deployment.
Layer 5 — Applications
Applications are the layer where users experience AI and where economic value becomes visible.
This layer includes:
- agents;
- copilots;
- chat assistants;
- retrieval-augmented systems;
- customer-support platforms;
- coding tools;
- clinical workflows;
- financial systems;
- robotics;
- manufacturing systems;
- educational tools;
- and creative applications.
Applications combine models with interfaces, domain logic, data, tools, permissions, workflows, and human decision-making.
A model might generate text, but an application determines why that text is generated, what information the model can access, what actions it may perform, and how its output affects the real world.
This is also where weaknesses in lower layers become visible.
Poor memory management becomes slow responses or crashes. Weak networking becomes unreliable agents. Missing cancellation becomes wasted compute. Inadequate observability becomes difficult incident response. A badly selected model becomes poor user experience. Insufficient power or cooling becomes reduced throughput.
The application layer captures attention, but the lower layers determine reliability, latency, cost, privacy, and scale.
A compelling user interface cannot permanently hide weak infrastructure.
Why runtime infrastructure matters more now
For the first phase of widespread generative-AI adoption, much of the infrastructure complexity was hidden behind cloud APIs.
A developer sent a request to a managed endpoint and received a response. Model loading, hardware scheduling, scaling, memory management, and failure recovery were handled by the provider.
That abstraction accelerated experimentation. It allowed companies to build AI features without operating the underlying compute.
But AI is now spreading into environments where the runtime can no longer remain invisible:
- private deployments;
- local inference;
- edge devices;
- regulated industries;
- sovereign AI programs;
- internal enterprise systems;
- multimodal workflows;
- autonomous and agentic applications;
- and systems combining several models or hardware types.
These environments introduce requirements that cannot always be delegated to a generic external API.
A hospital may need strict data locality. A manufacturer may require low-latency operation even when external connectivity is unavailable. A company may want predictable costs and control over model versions. A developer may want an assistant that runs privately on a workstation. A country may need models and infrastructure aligned with local languages and laws.
In all of these cases, the runtime layer becomes visible.
The runtime decides:
- how models are discovered and loaded;
- how memory and caches are managed;
- how requests are admitted and scheduled;
- how responses stream;
- how cancellation and timeouts behave;
- how failures are normalized;
- how models expose their capabilities;
- how systems report health and readiness;
- how operators observe performance;
- how clients integrate through stable interfaces;
- and how private deployments remain practical.
As AI moves closer to users, devices, institutions, and jurisdictions, runtime infrastructure becomes more visible—and more important.
The full-stack lesson
The five-layer cake makes one thing clear: models do not create value alone.
Energy enables chips.
Chips require infrastructure.
Infrastructure makes models usable.
Models power applications.
Applications create visible economic and social value.
The layers are interdependent.
CUDA made this relationship obvious in the NVIDIA ecosystem. The GPUs mattered enormously, but the programming model, libraries, compilers, tools, and surrounding software ecosystem made those GPUs useful across industries and at scale.
MLX shows the same underlying lesson from a newer local-AI direction. Efficient integrated chips and unified memory create powerful possibilities, but models still need execution frameworks, memory policies, serving layers, operational controls, and application interfaces before they become dependable systems.
The future of AI will not be determined only by who trains the largest model or builds the most popular application.
It will also be shaped by who can supply the energy, design the chips, build the runtimes, operate the infrastructure, govern the models, and connect all five layers into systems that people can trust.
The next wave of AI will not be defined only by who has the biggest model.
It will also be defined by who controls the layers underneath it.
If you to connect with me, let’s connect on LinkedIn or drop me a message—I’d love to explore how I can help with your data to drive your AI success!
Top comments (0)