DEV Community: Prateek Navani

A simple explanation of what a virtual machine actually is

Prateek Navani — Fri, 30 Oct 2026 06:46:13 +0000

A virtual machine is a computer that does not exist, running on a computer that does.
That sounds like a riddle, but it is a fair description. If you have ever opened a cloud server, run Windows on a Mac, or used almost any website, you have touched one. This is the plain-language version, with no jargon left unexplained.

What is a virtual machine?

A virtual machine, or VM, is a computer made entirely of software, running inside a real computer. It has its own operating system, its own share of CPU and memory, its own storage and its own network connection. The real machine is called the host, and the VM is called the guest.
Log into one and you cannot tell the difference. There is a disk to format, an operating system to install, an IP address and a login prompt. It behaves like a physical machine because, from the inside, it believes it is one.
How does one physical computer run many of them?
Through a piece of software called a hypervisor. It sits between the hardware and the VMs, divides up the CPU, memory, storage and network, and keeps each VM separate from the others.
There are two main kinds:
Type 1 (bare metal): the hypervisor runs directly on the server's hardware. This is what data centres use, and VMware ESXi, Microsoft Hyper-V and Xen are examples
Type 2 (hosted): the hypervisor runs as an app on top of a normal operating system. This is what you use on a laptop, and VirtualBox is an example

Why do virtual machines exist at all?

To stop servers from sitting mostly empty, and to keep different systems from interfering with each other.
Before virtualisation, one server often ran one application, and most of its capacity went unused. VMs let many isolated systems share one physical machine. The benefits are practical:
Isolation: a crash or a hack in one VM does not spill into the others
Efficiency: one powerful server does the work of many small ones
Flexibility: each VM can run its own operating system, including a different one from the host
Copyability: you can snapshot a VM, clone it or move it, since the whole machine is stored as a file

What can you actually do with a virtual machine?

Quite a lot, and you may already be doing some of it.
Host a website or an application in the cloud
Test software safely, without risking your main computer
Run Windows on a Mac or Linux machine, or the other way round
Keep development, testing and production environments separate
Rent a cloud GPU, since many cloud GPU offerings are VMs with a GPU attached

What is the difference between a VM and a VPS?

A VM is the technology, and a VPS is a VM sold as a hosting product. That is the whole vps vs vm answer in one line: one describes how it works, the other describes how it is packaged and sold.

VM
VPS
What it is
A software-made computer on a hypervisor
A VM rented as a hosting product
Who uses the term
Engineers, cloud platforms
Hosting providers and their customers
What you usually get
A machine you configure yourself
An OS, fixed resources and root access
Typical billing
Often metered by the second, minute or hour
Often a flat monthly price

Is a cloud instance a VM or a VPS?
Technically, a VM. Cloud providers call them instances, cloud servers or virtual servers, and the name changes from company to company. If you can pick its size, install an operating system and log in as an administrator, you are almost certainly looking at a VM, however it is labelled.

How is a VM different from a container?

A VM virtualises the hardware and runs a full operating system. A container shares the host's operating system and packages only the application and what it needs.

Virtual machine
Container
Runs
A full guest operating system
The app, sharing the host's OS kernel
Typical image size
Several GB, commonly 5 to 30GB
Commonly 50 to 200MB
Start-up
Slower
Faster
Isolation
Stronger, enforced by the hypervisor
Lighter, enforced by the OS
Operating system
Any OS per VM
Must match the host's kernel type

Do you have to choose between them?

No, and many teams use both. A common pattern runs containers inside VMs: the VM provides the strong security boundary, and the containers provide fast, tidy packaging for the application.

What are the downsides of a virtual machine?

Nothing is free, and VMs have costs.
They are heavier. Each VM carries a whole operating system, which uses more memory and storage than a container
They start slower than containers, since an operating system has to boot
They are harder to move between platforms. Moving a VM between different hypervisors can need a format conversion
You look after the operating system. Patching, updates and security on the guest are your job

What do the numbers on a VM plan actually mean?

When you pick a VM, you choose a size. The labels are simpler than they look.
vCPU: a virtual processor, usually one thread of a physical core, handed to your VM by the hypervisor
RAM: the memory set aside for your VM while it runs
Disk: the storage attached to it, which often keeps billing even when the VM is off
Bandwidth: how much data can move in and out
A bigger VM is not automatically a better one. Start with the smallest size that runs your workload comfortably, and resize when the numbers say you need to.

How do you create your first virtual machine?

The steps are similar whether you do it on a laptop or in the cloud.
Decide where it will run: a hosted hypervisor on your laptop, or a cloud provider
Pick a size: vCPUs, memory and disk
Choose an operating system image
Start it and connect, through a console or a secure remote login
When you are finished, shut it down or delete it, so you do not pay for something you forgot
The last step is the one people skip. A forgotten VM is the cheapest mistake to make and one of the most common.

Where this leaves you

A virtual machine is a software computer running inside a real one, kept separate by a hypervisor. A VPS is that same thing sold as hosting, and a container is a lighter cousin that skips the full operating system. Once you hold those three ideas apart, most of the cloud vocabulary stops sounding like a foreign language.

A few quick questions people keep asking about this
Is a virtual machine secure?
It is a strong boundary. Escaping from a VM usually requires a flaw in the hypervisor itself, which is rare. You still need to secure what runs inside the VM, since a badly configured guest is just as exposed as a physical server.
Can I run a virtual machine on my own laptop?
Yes. A hosted hypervisor such as VirtualBox runs on an ordinary laptop. You will need enough memory to share between your own system and the VM, and the more you give it, the slower your main machine feels.
Do I need a VM or a container?
If you need a full operating system, strong isolation or a different OS from the host, use a VM. If you are packaging and shipping an application quickly, a container is often lighter. Plenty of setups use both.

5 signs your AI workload has outgrown on-prem infrastructure

Prateek Navani — Thu, 29 Oct 2026 07:26:49 +0000

Nobody wakes up and decides their server room is too small. It shows up as five smaller problems, and each one looks like someone else's fault: procurement is slow, facilities will not budge, the data science team is "using too much."
The problems are connected, though, and once you see the pattern it is hard to miss. Here are the five signs, with a quick test for each, so you can tell whether you have a few annoying problems or a genuine outgrown setup.

1. Your GPU usage is wildly uneven

If your GPU usage is idle one week and overloaded the next, you have probably sized your hardware for your peak, and your peak is rare. Training runs are bursty. A cluster big enough for the busiest fortnight of the quarter sits mostly empty the rest of the time, and you pay for it either way.
The quick test: look at average GPU utilisation over three months. If it sits well below what you would call sensible, but your team still queues for time during crunch weeks, the shape of your demand does not fit a fixed pool of owned hardware.

2. Hardware delivery taking longer than your project timeline

When lead times are longer than your roadmap, you are planning around the supply chain instead of the work. One 2026 analysis puts data centre GPU lead times at 36 to 52 weeks. Some enterprise buyers report delays measured in many months.
Prices make it worse. Memory costs have kept climbing through this autumn, so a quote can be stale before the purchase order is approved. If your last order arrived after the project it was meant for had already changed, that is this sign.

3. Power and cooling has become the limit instead of budget

This one catches teams by surprise, because the money is approved and the hardware still cannot go in. Modern AI racks can reach 100 kilowatts and above, which air cooling cannot handle, so liquid cooling is now close to a baseline requirement. Most enterprise facilities were never designed for 50 to 100 kilowatt cabinets.
Newer cards push in the same direction. The B200, for example, needs liquid cooling and can push racks past 120 kilowatts. The quick test is simple: ask facilities how much extra power and cooling capacity you can actually add this year. If the honest answer is "very little," your next GPU purchase has no place to go.

4. Models growing faster than your hardware refresh cycle

A cluster is usually bought for three to five years. Models do not wait that long. A 70B model in FP16 already needs about 140GB for its weights alone, and the next step up often means multiple nodes, not a bigger box.
Hardware moves quickly too. Nvidia went from H100 to H200 to B200 in roughly three years. If every new model you want to try needs hardware you do not own, you are being held back by your fleet, and renting the newest card for a few weeks becomes the sensible way to find out whether it is worth owning.

5. Team spending more time running the cluster than using it

Count the hours. Drivers, firmware, job scheduling, failed nodes, storage, on-call rotations: this is real work, and it is invisible in the hardware budget. When your best engineers are babysitting machines, you are paying a senior salary for a task the provider does as a service.
The quick test is to ask your team how much of last month went on keeping the cluster healthy versus training and shipping models. If infrastructure is eating a meaningful share, the cost of running on-prem is higher than your invoice shows.

How do you check all five signs quickly?

Run each one through a single question.
Sign
The one-line test
Uneven usage
Is average utilisation low while crunch weeks still have queues?
Slow delivery
Does hardware arrive after the project it was bought for?
Power and cooling
Can facilities add meaningful capacity this year?
Fast-growing models
Do new models need hardware you do not own?
Operations burden
Is infrastructure eating a real share of engineering time?

Two signs is a warning. Three or more means you have probably outgrown the setup as it stands.

When does on-prem still make more sense?

On-prem deployment makes sense more often than the cloud sales pitch admits. The on premise vs cloud question is not a one-way street, and for steady, heavily used workloads, owning can come out ahead.
One 2026 analysis puts the breakeven for sustained GPU workloads at roughly 12 to 18 months of ownership. It also argues that a team running owned hardware at 85 percent utilisation can outperform a team with three times the cloud allocation running at 40 percent. Strict data residency rules can also push regulated workloads toward infrastructure you control.
The honest summary is that predictable, high-utilisation work often belongs on owned hardware, while bursty and experimental work belongs on rented hardware.

What should you do if you spot three or more signs?

Do not rip everything out. Most organisations end up hybrid, and one 2026 survey found roughly 73% of enterprises now run hybrid cloud estates.
A practical order of operations:
Keep your steady, high-utilisation workloads where they are
Move the bursty ones first, such as experiments and training spikes
Pick one workload as a pilot, measure cost and delivery time, then decide on the next
Revisit the split every six months, since both prices and your workloads will have shifted
**
What does this look like for a real team?**

Here is an illustrative case. Picture a team with eight owned GPUs. Most weeks the cluster runs at about a third of capacity. Twice a quarter, a big fine-tuning push needs three times that, and jobs queue for days.
Ordering more hardware means a long wait for delivery, plus finding space for racks that need liquid cooling the building cannot supply. Meanwhile, two engineers lose a day each week to driver updates and failed nodes.
That team shows four of the five signs at once. The fix is not another eight GPUs. It is keeping the eight for the steady load and renting capacity for the two big pushes a quarter.

Conclusion
Outgrowing on-prem is rarely about one big failure. It is the quiet pile-up of slow deliveries, power limits and engineers doing infrastructure chores. If you recognise three or more of these signs, the answer is usually not a full move, but a hybrid: keep what is steady, rent what is not, and let the numbers tell you where the line sits.

Frequently asked questions
Does moving to the cloud always cost less than staying on-prem?
No. For steady, high-utilisation workloads, owned hardware often wins over a long enough horizon. The cloud wins on flexibility and speed, and that is worth more for bursty or fast-changing work.
Can I fix these signs by buying more hardware?
Only some of them. More GPUs help with capacity, but they do nothing for lead times, power limits or operations burden, and can make those three worse.
What is the smallest first step?
Pick one bursty workload, such as a training run you do each quarter, and run it on rented GPUs. You will learn your real costs and your real delivery speed without touching anything else.

VPC vs VPS: two terms that are often confused

Prateek Navani — Wed, 28 Oct 2026 05:20:45 +0000

Honestly, despite knowing what both these terms are, I myself sometimes get confused between both of them just because of the sound of it. But there is also a genuine confusion between the two terms amongst people when you are new to listening to them.
The names are almost the same, and the two things sit at completely different layers.
Here is the clean version, so you can tell which one you actually need before you pay for the wrong one.

What is the difference between a VPC and a VPS?
A VPS is a server. A VPC is the private network that servers sit in. One is a computer you rent, the other is the network around computers.

VPS
VPC
Stands for
Virtual private server
Virtual private cloud
What it is
One virtual machine with its own OS, CPU, RAM and disk
A private network you define inside a cloud account
What you get
A server you can log into, usually with root access
IP ranges, subnets, routes and firewall rules
Holds other things?
No, it is the thing
Yes, it holds many servers and services
Typical use
A website, an app, a game server
A multi-server setup that must stay private

Why do the two get mixed up so often?
Because both start with "virtual private," and both are sold as private. The word does a lot of work in each case, but it means something different. In a VPS, private means the server's resources are yours. In a VPC, private means the network is separated from other customers' networks.
One more term worth separating: a VM is the technology, and a VPS is a VM sold to you as a product, with an operating system and a monthly price.

What is a VPS, in plain terms?
It is one virtual machine on a shared physical server. You get a fixed slice of CPU, memory and storage that other customers on the same hardware cannot use, and you usually get root access to configure it however you like.
VPS hosting works well for:
A website or an online shop
A small app or API
A game server
A personal VPN or a small database
The price is usually a flat monthly figure, which makes it easy to budget.

What is a VPC, and what does it actually do?
A VPC is a private network that you define inside a public cloud account. You choose the IP range, split it into subnets, set the routes and write the firewall rules.
So if you only remember what is vpc in one line: it is the network, and servers are things you place in it. The main pieces are:
IP range: the private address space your resources live in
Subnets: slices of that range, often split into public and private
Route tables: rules for where traffic goes
Security rules: firewalls that decide who can talk to what
Gateways: the doors to the internet or to other networks
Does putting a server in a VPC make it private?
Not by itself. Whether a server is reachable from the internet depends on its subnet, its public IP and its security rules, not on the VPC label.
It also helps to know that you may already be inside one. As of October 2026, AWS gives every account a default VPC in each region, Google Cloud gives every new project a default network unless an administrator turns that off, and Azure creates a virtual network alongside a new VM if you do not pick an existing one. These defaults are built for convenience, so check them before you assume anything is locked down.

Can you use a VPS and a VPC together?
Yes, and that is how they relate. A VPC is the container, and virtual machines live inside it. A VPS provider usually calls this private networking, and a cloud provider calls it a VPC, but the idea is the same.
A simple example: a web server sits in a public subnet so visitors can reach it, and the database sits in a private subnet with no public address. Only the web server can talk to the database. That structure is hard to build with a single standalone VPS.

How do you decide which one you need?
Start with how many machines you are running.
One server for a site, an app, a game server or a VPN: a VPS
Several servers that must reach each other on private addresses: a VPC, or your VPS provider's private network feature
Separate public and private tiers, or separation required for compliance: a VPC
Many machines for a few hours or days: a cloud setup with a VPC, since it is built for that
Most single-server projects never need a VPC.
Is a VPC more expensive than a VPS?
Not necessarily, but the billing works differently. A VPS is usually a predictable flat monthly price. In a VPC, the network itself is usually free, but you pay for the compute and storage inside it, plus some components such as NAT gateways, VPN connections and data transfer. Check your provider's price list, because the extras are where bills grow.

What about VPN and private cloud?
Two more look-alikes come up in the same conversation.
VPN: an encrypted tunnel that connects you or an office to a network. It is an access method, not a server or a cloud
Private cloud: a whole cloud environment dedicated to one organisation. A VPC is different, since it is a private slice of a shared public cloud
A common setup uses two of these together: the company's apps run in a VPC, and employees connect to it through a VPN.

What mistakes do people make with each one?
A few come up again and again.
With a VPS:
Leaving every port open, since the server is reachable from the internet from day one
Assuming backups are included, when many plans charge for them separately
With a VPC:
Assuming it is private by default, when the subnet and public IP settings decide that
Picking an IP range that overlaps with another network, which can block you from connecting the two later
Being surprised by NAT gateway and data transfer charges on the first bill
Most of these are cheap to avoid if you check them on day one, and expensive to fix after you have real traffic.

Conclusion
The simplest way to keep them apart: a VPS is a machine, and a VPC is the network those machines live in. If you have one machine, you want a VPS. If you have several that must talk privately, you want a VPC. And if you are not sure yet, start with the VPS, since it is easy to grow out of.

Frequently asked questions
Is a VPS the same as a VM?
Almost. A VM is the underlying technology, and a VPS is a VM sold as a hosting product, with an operating system, root access and a monthly price.
Do I need a VPC for a small website?
Usually not. One VPS covers most small sites, and you can add private networking later if you grow into multiple servers.
Can I move from a VPS to a VPC later?
Yes. You are not converting one into the other. You move your application onto servers inside a VPC, and the same software generally runs the same way.

GPU prices just spiked again: here's what that actually means for your AI budget

Prateek Navani — Tue, 27 Oct 2026 05:50:32 +0000

The RTX 5090, a gaming card, with some listings reported an increase again. It was nearly $8,500 which was $1,999 when it was launched.
It is a scary number. Although it is not what lands on your invoice at the end of month. But the scarier part is, GPU’s hourly/monthly rate is not the only number that lands on your invoice when you run AI workloads and rent your compute.
What lands on your invoice is smaller, slower and still worth planning for. Here is what has actually moved, why it moved, and what I would do about it.

Why have GPU prices spiked again?

AI data centres are soaking up DRAM, HBM and GDDR supply, and everything else is being priced around that shortage.
Here is what happened in the first three weeks of October 2026:
Micron said on its 1 October earnings call that DRAM contract prices would rise another 10 to 15 percent this quarter, and that demand would outrun supply through 2027 and into 2028
AMD raised the price of the GDDR6 memory it supplies to board partners from 1 October, its fourth memory-driven adjustment in ten months
Nvidia has reportedly raised some card prices a third time this year, by 20 to 30 percent on certain models
One price tracker had the RTX 5090's lowest new asking price up 64 percent between early August and mid-October, from about $3,000 to about $4,900
Is this the same as the 2021 GPU shortage?
Not really. In 2021, mining demand evaporated once mining stopped paying. This time the demand comes from data centre spending commitments that run for years.
That is my read, not a forecast, but I would not build a budget around a quick drop.

Are cloud GPU rental prices rising too, or only the cards you buy?

They are rising too, but mostly by less than retail cards. Here is what has been reported recently.
What
Reported move
When
RTX 5090, lowest new asking price
About +64% ($3,000 to $4,900)
Early Aug to mid-Oct
Nebius H100, per GPU-hour
$3.85 to $4.50 (about +17%)
From 1 Oct
Nebius H200, per GPU-hour
$4.50 to $5.40 (about +20%)
From 1 Oct
B200 cloud rental, market index
Up 79% to about $8.01 per GPU-hour
Three months to late Sept
AWS H200 instances
+15%
January 2026

These come from different trackers and providers, so treat them as a pattern, not a single market price. The pattern is clear enough: H100 and H200 rentals are rising in the high teens to around 20 percent where providers have announced increases, and the newest cards are rising much faster.
Why do memory-heavy cards move the most?
Because memory is the scarce part. One market index puts H200 rental at roughly 1.76 times the H100 rate, which is almost exactly the ratio of their memory, 141GB against 80GB.
In other words, you are increasingly paying for gigabytes, not just for compute. That changes how you should choose a card, and I will come back to it below.

How will the spike affect AI teams in India?

Mostly through rental rates, and more slowly than the headlines suggest. For gpu prices india teams actually pay, watch rental listings, not retail cards, since most Indian AI teams rent rather than buy.
A few things shape how it reaches you:
Rupee rates follow global moves with a lag. CloudPe has forecast that Indian cloud GPU rental prices could rise 15 to 20 percent over the next 12 to 18 months, which is a conditional forecast, not a promise
Dollar-billed providers add currency movement on top. If your invoice is in USD, a weaker rupee raises your cost even when the hourly rate stays flat
Subsidised compute is priced differently. The IndiaAI Mission reported more than 45,000 GPUs onboarded by June, at roughly ₹67 to ₹92 per GPU-hour. Those rates are set through the programme, not the open market, so I would expect them to move least. Access is approval-gated, though

How do you work out your own exposure in ten minutes?

Take last month's GPU invoice, strip out storage and bandwidth, and look only at the compute line. Multiply it by 1.15 and by 1.20. That gives you a rough range for what the same usage could cost after the increases.
Here is what that looks like in rupees. If you spend ₹1 lakh a month on GPU time, a 20 percent rise adds ₹20,000 a month, or ₹2.4 lakh over a year. Now check how many of those hours were idle. If a third of them were, cutting that waste saves roughly ₹33,000 a month on the same bill, which is more than the rise costs you.
It is a rough sum, but it turns a vague worry into a number you can take to whoever approves the budget.

What should you actually do with your AI budget now?
Five things, in the order I would do them.

Add a buffer: Budget 15 to 20 percent above today's hourly rates for the next year. That is a planning assumption, not a prediction.
Fix idle time first: If a GPU sits idle for a third of the month, switching it off recovers more than a 20 percent rate rise costs. It is the only saving here you control completely.
Check whether you need the memory: Memory-heavy cards carry the steepest increases. If your model fits on a smaller card, do not rent an H200 or B200 out of habit.
Ask for a fixed-rate term quote if your usage is steady. Reserved rates have been climbing too, with one index showing one-year H100 contracts up about 40 percent between October 2025 and March 2026. Locking in costs you flexibility, so only do it for loads you are sure about.
Do not buy hardware unless you must: Retail cards are at spike prices. Buying now means paying a premium you may not get back.

Conclusion

A price spike on a gaming card is a headline. A 15-20% rise in your rental rate is a budget line. Treat the second one as likely, plan a buffer, and spend your first effort on the things you control: idle hours, card size and contract terms.
All the info is as of October 2026, and several of these numbers are moving week by week. Check your own provider's current rate card before you lock anything in.
Frequently asked questions
Will GPU prices come back down soon?
Not on the evidence so far. Micron's own guidance points to tight supply through 2027 and into 2028. I would plan as if today's prices are the floor for a while, and treat any relief as a bonus.
Should I wait for prices to drop before starting my AI project?
Usually no. Waiting means your project sits still while the rental market stays short on capacity. A project that starts now with a 20 percent buffer is better off than one that starts in six months at an unknown price.
Is renting still cheaper than buying after this spike?
For most teams, yes. Retail prices have risen faster than rental rates, which tilts the sums further toward renting. Buying only makes sense if you will keep a card busy almost every day for a year or more.

RTX PRO 6000 for AI Inference: Models, Performance, and Cloud Deployment

Prateek Navani — Mon, 26 Oct 2026 05:29:31 +0000

AI is becoming part of everyday business operations. From customer support chatbots to AI-powered search, document summarization, and image generation, more applications now rely on AI models to deliver results in real time.
Behind these applications is AI inference, the process of using a trained model to generate responses or predictions. As models become larger and applications serve more users, choosing the right GPU becomes increasingly important.
The NVIDIA RTX PRO 6000 Blackwell is one option worth considering. With 96GB of GDDR7 memory and a modern GPU architecture designed for demanding computing workloads, it can handle many AI inference tasks, including running large language models.
But what makes this GPU suitable for inference, which models can it run, and when does renting one make more sense than buying it? Let's take a closer look.

What Is AI Inference?

AI inference happens whenever a trained AI model processes new information and produces an output.
For example, when someone asks a question in an AI chatbot, the model processes the prompt and generates a response. The same process powers image recognition tools, AI writing assistants, recommendation systems, and many other applications.
Some common AI inference workloads include:
Large language models (LLMs): Powering chatbots, AI assistants, and text-generation tools.
Retrieval-augmented generation (RAG): Helping AI applications answer questions using company documents and internal knowledge bases.
Image generation: Creating images from written prompts.
Computer vision: Identifying objects, analyzing images, and processing video.
Business automation: Summarizing documents, extracting information, and handling repetitive tasks.
The GPU you choose can affect how quickly these applications respond, how many users they can serve, and how large a model you can run.

Why Use RTX PRO 6000 for AI Inference?

Not every AI application needs a high-end GPU. However, workloads involving larger models, longer conversations, or multiple simultaneous users can benefit from additional GPU memory and computing power.
The RTX PRO 6000 Blackwell brings several features that make it worth considering for these workloads.

96GB of Memory for Larger AI Models

GPU memory, also known as VRAM, is one of the first things to consider when selecting hardware for AI inference.
A model needs memory to store its weights, but that is only part of the requirement. The inference software, temporary data, and conversation context also consume VRAM.
The RTX PRO 6000 Blackwell comes with 96GB of GDDR7 memory and error-correcting code (ECC). This gives developers room to run many quantized large language models on a single GPU, depending on the model and deployment settings.
Consider a 70-billion-parameter model. At approximately 8-bit precision, its weights alone may require around 70GB of memory. The inference engine and KV cache need additional space, so the complete workload must be evaluated before deployment.
This is where a GPU with 96GB of memory can be useful, particularly for teams that want to run larger models without immediately moving to a multi-GPU setup.

Blackwell Architecture for Modern AI Workloads

The RTX PRO 6000 uses NVIDIA's Blackwell architecture and features fifth-generation Tensor Cores.
These capabilities support modern AI computing techniques, including lower-precision formats such as FP4. When a model and its inference software support these formats, developers may be able to reduce memory requirements and improve inference efficiency.
The actual improvement depends on the model, quantization method, software framework, and workload. It is always a good idea to test your application rather than relying entirely on theoretical performance figures.

Suitable for More Than One Type of AI Application

AI workloads vary considerably. A team might use a GPU to develop a chatbot today and later use the same infrastructure for document processing, image generation, or other machine learning tasks.
The RTX PRO 6000 can be considered for a range of professional AI workloads, including LLM inference, generative AI, and computer vision.
For production applications, however, the right configuration also depends on request volume, response-time requirements, software compatibility, and how consistently the GPU will be used.

Which AI Models Can Run on RTX PRO 6000?
The models you can run depend on their size, numerical precision, context length, and the amount of memory required by the inference engine.
The table below provides a rough estimate of model-weight memory requirements.
Model size
Memory at 16-bit precision
Memory at 8-bit precision
7B parameters
14GB
7GB
13B parameters
26GB
13GB
30B parameters
60GB
30GB
40B parameters
80GB
40GB
70B parameters
140GB
70GB

These are estimates for model weights alone. Actual memory usage will be higher because the GPU also needs space for the inference engine, KV cache, and other runtime operations.

Running Smaller Models: 7B and 13B

Smaller language models are often used for chatbots, text summarization, internal search, classification, and basic AI assistants.
The RTX PRO 6000 has enough memory to provide additional flexibility with these workloads. Depending on the application, that could mean supporting longer prompts, handling more concurrent requests, or experimenting with different model configurations.
However, the GPU you need will depend on the expected workload. If your application only uses a small model with limited traffic, a less expensive GPU may be sufficient.

Running Mid-Sized Models: 30B to 40B

Mid-sized models can be useful when an application needs more capable text generation or reasoning, although performance varies from model to model.
At 8-bit precision, many models in this range can fit comfortably within 96GB of VRAM. At 16-bit precision, memory requirements increase significantly. For example, a 40B model needs approximately 80GB for its weights alone, leaving less room for context and other runtime requirements.
Quantization can help reduce memory consumption, making these models easier to deploy on a single GPU.

Running Large Models: 70B

Running a 70B model is possible on a single RTX PRO 6000 in many quantized configurations, provided the complete workload fits within the available memory.
At 8-bit precision, the weights alone require approximately 70GB. This leaves some memory for the inference engine and KV cache, but the available space may become a limitation with long contexts or multiple concurrent users.
At 16-bit precision, the weights require approximately 140GB, which exceeds the memory capacity of a single RTX PRO 6000.
If your application requires full 16-bit weights, longer context windows, or high concurrency, you may need a larger-memory or multi-GPU configuration.

What Affects AI Inference Performance?

Having enough VRAM is important, but it does not automatically guarantee fast inference.
Several other factors influence how well a model performs in a real application.
Model Quantization

Quantization reduces the precision used to represent model weights. It can make larger models easier to fit into GPU memory and may improve inference speed.
The trade-off is that some quantization methods can affect model quality. The impact depends on the model and technique, so it is worth testing different configurations before deploying one in production.

Context Length
Longer prompts and conversations require additional memory to maintain the information needed to generate responses.
For example, an AI assistant that answers short questions may use considerably less memory than one that analyzes lengthy reports or maintains long conversations.
If your application relies on large context windows, account for KV cache requirements when selecting a GPU.

Number of Simultaneous Users
A model serving one user has different requirements from one handling dozens or hundreds of requests at the same time.
Higher concurrency can improve overall throughput when requests are efficiently batched, but it can also increase memory usage and affect response latency.
Before choosing a GPU, test your application with realistic traffic and measure metrics such as time to first token, tokens generated per second, and total request latency.

The Rest of Your Infrastructure
The GPU is only one part of an AI inference system.
CPU performance, system RAM, storage, networking, and the inference framework can all affect results. A well-balanced configuration helps prevent other components from becoming bottlenecks.
RTX PRO 6000 vs. Other GPUs for AI Inference

The RTX PRO 6000 is not necessarily the right choice for every AI workload. It helps to compare it with alternatives based on your model size and performance requirements.
Use case
GPU to consider
Smaller AI models and lightweight inference
NVIDIA L4
Larger quantized LLMs on a single GPU
NVIDIA RTX PRO 6000 Blackwell
AI inference combined with professional graphics workloads
NVIDIA RTX PRO 6000 Blackwell
Large models requiring high memory bandwidth or multi-GPU scaling
NVIDIA H100 or H200

The NVIDIA L4 may be sufficient for applications that prioritize efficient inference with smaller models. The RTX PRO 6000 is worth considering when additional memory capacity is important, while H100 and H200 configurations may be better suited to workloads that need higher memory bandwidth or multiple GPUs.
The best option depends on the model you intend to run, the number of requests you expect, and your budget.
Whenever possible, benchmark the same model on the configurations you are considering. Real-world results will provide a more useful comparison than specifications alone.

Should You Buy or Rent RTX PRO 6000 for AI Inference?

Once you have identified the right GPU, the next decision is how to access it.
Buying hardware gives you direct control over the system, while renting a cloud GPU can make it easier to start without a large upfront investment.
When Buying Makes Sense

Buying an RTX PRO 6000 may be a good option if your workload runs consistently and you expect to use the GPU for an extended period.
It may be particularly suitable for businesses that already have the necessary infrastructure and technical team to manage their own hardware.
Before purchasing, consider the cost of the GPU, the rest of the server, electricity, cooling, maintenance, and future upgrades. These expenses contribute to the total cost of ownership.

When Renting a Cloud GPU Makes Sense

Cloud GPU rental can be useful when you are developing an AI application, testing a new model, or working with workloads that change over time.
Instead of purchasing hardware upfront, you can access a suitable configuration based on your requirements and the provider's available plans.
Cloud GPU rental can be particularly useful when you want to:
Test an AI model before committing to dedicated hardware.
Launch a new application without investing in a physical GPU.
Scale your resources as demand changes.
Avoid managing physical hardware and its maintenance.
Compare different GPU configurations before making a long-term decision.
The actual savings depend on usage, rental rates, storage, networking, and other charges. For workloads that run continuously, purchasing hardware may be more economical over time.
If you are comparing the costs of both options, our guide to RTX PRO 6000 price in India explains the pricing considerations involved in buying or renting this GPU.

Deploy RTX PRO 6000 for AI Inference on CloudPe
If you want to run AI workloads without purchasing a physical GPU, cloud infrastructure is another option to consider.
CloudPe offers an RTX PRO 6000 Blackwell GPU for workloads such as AI inference, model development, and other supported GPU computing tasks.
Before choosing a configuration, consider the model you plan to deploy, its memory requirements, expected request volume, and the software frameworks you intend to use.
It is also worth checking the available GPU configuration, current pricing, storage options, and deployment details to make sure they match your requirements.
If you are still evaluating whether this GPU fits your budget, start with our guide to RTX PRO 6000 pricing in India. You can then compare the available options based on your workload and expected usage.

NVIDIA data center GPUs comparison table: T4 to B200

Prateek Navani — Mon, 19 Oct 2026 05:45:38 +0000

In 2020 when A100 was launched by Nvidia. If someone asked what GPU to use for inference, the A100 was our default answer. It was not a bad recommendation at the time. It was genuinely one of the best data center GPUs available, and for a few years there was not a clearly better option built specifically for serving models rather than training them.
That stopped being true a while ago, and we stopped recommending the A100 as a default for inference because of it. Let me walk through exactly why, and what changed.

Why was the A100 the default inference recommendation for so long?

Because it was launched in 2020, and for the next few years, nothing purpose-built for inference genuinely outperformed it on a cost and capability basis. The A100 was a versatile, capable workhorse, strong at both training and inference, and in the absence of anything better suited specifically to inference, it became the default answer for both.
Launched in 2020 on the Ampere architecture
Built as a general-purpose data center accelerator, not optimized specifically for either training or inference
Remained the strongest realistic option for years simply because nothing more inference-focused had arrived yet
Was the A100 ever actually built specifically for inference?
No, and this is the part that gets overlooked. The A100 was designed as a broad, capable accelerator meant to handle training and inference reasonably well, at a time before inference workloads had their own dedicated hardware category. It did the job because it was the best available tool, not because it was purpose-built for the job.

What actually changed after 2023?

The L4 launched, and it was built with a fundamentally different goal in mind. Rather than trying to be a strong generalist, it was designed specifically around efficient inference.
Built on the newer Ada Lovelace architecture, with native FP8 support the A100 does not have
24GB of GDDR6 memory, comfortably enough for the model sizes most production inference workloads actually run
A 72W TDP, a fraction of the A100's power draw, without sacrificing the throughput that inference workloads actually need

What made the L4 the right balance for inference specifically?

Technically, it lines up almost perfectly with what most real inference workloads need. Most production inference happens on models somewhere in the 7B to 30B parameter range, not the largest models available, and the L4's memory and bandwidth comfortably cover that range. On the budget side, it does this at a fraction of the A100's hourly cost and power draw, which matters just as much once you are running inference continuously rather than as an occasional workload.

How do the A100 and L4 actually compare once you line them up for inference?

Here is the honest side by side.
Spec
A100 (80GB)
L4
Launch year
2020
2023
Architecture
Ampere
Ada Lovelace
VRAM
80GB HBM2e
24GB GDDR6
Memory bandwidth
Roughly 2,039 GB/s
300 GB/s
TDP
400W
72W
Native FP8 support
No
Yes
Built primarily for
General purpose, training and inference
Inference specifically

Does the A100 lose on every metric, or does it still win somewhere?

It still wins comfortably on raw memory and bandwidth, which matters for genuinely large models or heavy training workloads. If your job actually needs 80GB of VRAM or the bandwidth to train efficiently, the A100 remains a real, capable option. What changed is that most inference workloads never needed that much capacity in the first place, and recommending an A100 by default for those workloads meant paying for headroom that was never going to get used.
👉You may like: L4 vs T4 GPUa

Why does power draw matter this much for an inference recommendation?

Because inference, unlike training, tends to run continuously rather than in defined bursts. A 400W card running around the clock adds up in electricity and cooling costs in a way that a single training run does not, and that cost compounds every single day the workload stays live. The L4's 72W draw is not just a smaller number on a spec sheet. It is a meaningfully lower total cost of ownership for anything running in production long term.

So when does the A100 still make sense today?
There are genuine cases where it remains the right call.
Teams that already own A100 hardware and are extending existing capacity rather than provisioning new infrastructure
Large batch training runs where the extra VRAM and bandwidth are actually being used
Workloads that specifically require close to 80GB of memory on a single card
Situations where a smaller card like the L4 genuinely cannot fit the model in question
Outside of these cases, defaulting to the A100 for a new inference deployment usually means paying for capability the workload will never touch.

What do we recommend instead now?

For most inference workloads in the 7B to 30B range, the L4 is our starting recommendation. For workloads that need more headroom than the L4 comfortably provides but do not require a full A100 or H100, the L40S usually fits that gap well. For genuinely large models or high-throughput production serving, H100 or H200 remain the right call, not because they replace the L4, but because they solve a different problem entirely.
The A100 has not disappeared from the conversation. It has simply moved from being the default answer to being the right answer for a much narrower set of situations than it used to
Cover.

A few quick questions people keep asking about this

Is the A100 now considered obsolete for AI workloads generally? Not obsolete, but clearly positioned differently than it was a few years ago. It remains a capable GPU for training and for workloads that genuinely need its memory and bandwidth. It has simply stopped being the right default for standard inference workloads once purpose-built alternatives arrived.
Does this mean the L4 is always the right choice for inference? Not always, but it is the right starting point for most common inference workloads. Once model size or concurrency needs grow past what the L4 comfortably handles, the L40S or a larger GPU becomes the more appropriate choice.
If we already own A100 hardware, should we replace it with L4s? Not necessarily. If the A100 capacity is already paid for and sitting in your infrastructure, using it for existing inference workloads still makes financial sense. The recommendation shift applies mainly to new deployments and new hardware decisions, not to hardware you already own and are actively using.

Where this leaves you

Recommending a GPU by default because it used to be the best all-around option stops making sense once purpose-built alternatives actually fit the job better. The A100 was the right answer for years because nothing better suited to inference existed yet. Once the L4 arrived with the right balance of memory, efficiency, and cost for the workloads most inference deployments actually run, continuing to recommend the A100 by default stopped being the honest answer, and we changed what we tell people accordingly.

Why we stopped recommending the A100 for AI inference

Prateek Navani — Fri, 16 Oct 2026 05:27:23 +0000

Why was the A100 the default inference recommendation for so long?

What actually changed after 2023?

What made the L4 the right balance for inference specifically?

How do the A100 and L4 actually compare once you line them up for inference?

Does the A100 lose on every metric, or does it still win somewhere?

👉You may like: L4 vs T4 GPU

Why does power draw matter this much for an inference recommendation?

What do we recommend instead now?

A few quick questions people keep asking about this

Where this leaves you

Why "hosted in India" isn't the same as "data sovereign"

Prateek Navani — Thu, 15 Oct 2026 07:52:15 +0000

Next time a cloud vendor tells you "your data stays in India," try this. Ask one follow-up: if a court in another country ordered you to hand over my data, would you have to?

A good vendor answers that straight. A less good one goes back to talking about their Mumbai data centre, which is a different question altogether.

That mix-up is what I want to untangle here. "Where does my data sit" and "who can be made to hand it over" sound like one question. They are two. One is about a building. The other is about who the company answers to. Most compliance checklists only ask the first, and that is how teams end up feeling covered when they are not.

What is the difference between data residency and data sovereignty?

Data residency is about location. It tells you where your data is physically stored. Data sovereignty is about control. It tells you whose laws govern the data, who can legally compel access to it, and who holds the encryption keys.
A server can sit in Mumbai and still be answerable to a law written in another country. That gap is the whole point of this article.
Term
The question it answers
Simple example
Data residency
Where is the data stored?
Your database runs in a Mumbai data centre
Data localisation
Does the law force it to stay here?
RBI requires payment data to be stored only in India
Data sovereignty
Whose laws and control apply to it?
Only Indian courts and Indian-controlled keys can reach it

Why do people mix these up so often?
Because "hosted in India" is easy to say and easy to verify, while control is hard to explain and harder to check. It fits neatly on a pricing page. A sentence about corporate jurisdiction and key management does not.

Can a foreign government still access data stored in an Indian data centre?

In some cases, yes. The US CLOUD Act is the usual example. It applies based on who controls the data, not where the servers are. A provider incorporated in the United States can be required to hand over data it controls, even if that data sits in an Indian region.
Legal commentators have also pointed out that India has no executive agreement under the CLOUD Act. That means there is no treaty route for challenging such a request through Indian courts.
Does this mean using a global cloud in India is unsafe or illegal?
No, and I want to be clear about that. For most workloads, a global provider's Indian region is a perfectly reasonable choice. The concern is specific to sensitive data and regulated sectors, not every app and every dataset. Treat this as a risk to size, not a reason to panic.

What does Indian law actually require today?

Less than people assume, at least at the general level. The DPDP Act uses a negative list model. Personal data can move outside India unless the government restricts a specific destination or category.
The hard rules sit in sector regulation. The DPDP Act also says stricter laws in other sectors continue to apply, so those rules win where they exist.
Sector or data type
Regulator
What is required
Payments
RBI
Payment data must be stored only in India
Insurance
IRDAI
Policyholder data must be kept in India
Securities
SEBI
Indian-hosted or compliant hybrid setups
Aadhaar data vaults
UIDAI
Hosted on MeitY-empanelled cloud
Government data
MeitY
India-hosted infrastructure under the GI Cloud policy
Telecom subscriber data
DoT
Must remain in India

Is MeitY empanelment the same as being sovereign?
No. Empanelment shows a provider has gone through an audit for security and compliance. It is not a sovereignty certification. Treating it as one is the same mistake as treating "hosted in India" as proof of sovereignty, just with an official stamp on it.

What should you ask a cloud provider to check real sovereignty?

Five questions will tell you more than any brochure.
Who owns and controls the company? Look at where it is incorporated and who its parent is, not just where its data centres are
Who holds the encryption keys? Ask whether you can bring your own keys, and whether the provider can ever access them
Where are the people with admin access? Support staff and engineers working from other countries can change the risk picture
What does the contract say about government requests? Check whether the provider commits to notifying you and to pushing back where it legally can
Where do backups, logs and metadata go? Your main database may stay put while supporting data quietly moves elsewhere
Do you need to run this full check for every workload?
No. Match the effort to the data. A public dataset or a test environment does not need the same scrutiny as customer financial records or health data. Sort your workloads by sensitivity first, then apply the full check only where it actually matters.

What does this look like in a real situation?

Picture a fintech startup that stores customer KYC records with a global cloud provider, in its Mumbai region. The data never leaves India, so on paper everything looks fine. Then an enterprise customer sends a security questionnaire asking who can access the data, under which country's laws, and who manages the encryption keys.
If the team cannot answer, the deal stalls. Nothing was technically broken. They simply solved for location and never thought about control.
The fix is rarely dramatic. Usually it means moving key management into the team's own hands, writing down the access policy, and picking a provider who can answer those questions in plain language. Doing that before a customer asks is far cheaper than doing it after.
A few quick questions people keep asking about this

Is hosting in an Indian data centre enough to comply with the DPDP Act? For general personal data, the DPDP Act does not force you to keep everything in India. Residency alone does not cover sector rules, though, so check what your regulator requires on top.
Does choosing an Indian provider automatically give me data sovereignty? Not automatically. Ownership, investors, key management, subprocessors and support locations all still matter. An Indian address on the invoice is a good start, not a finished answer.
Can encryption solve this on its own? It helps a lot, especially when you hold the keys yourself. It does not cover everything, since metadata and key management arrangements can still create exposure. Think of it as one strong layer, not the whole wall.

Where this leaves you

"Hosted in India" is a useful fact, and for many workloads it is all you need. But it describes location, and sovereignty is about control. If your data is regulated, sensitive, or tied to customers who will ask hard questions, look past the data centre address and ask who actually holds the keys and whose laws apply.
This is general information, not legal advice. For anything tied to a specific regulator, a short conversation with a lawyer who knows your sector is worth the time.

Cost-per-token breakdown by GPU: L4 vs A100, L40S and H100

Prateek Navani — Wed, 14 Oct 2026 10:20:58 +0000

Which GPU is "cheapest" for running an LLM? If you are merely going to look at the hourly rate on the pricing page, then it's a pretty straightforward thing, but that is not actually useful. A $3 per hour GPU generating tokens fast can easily end up cheaper than a $0.50 per hour GPU generating them slowly. What actually matters is cost per token, not cost per hour.
Let me walk through how L4, A100, L40S and H100 actually compare once you look at the number that matters, and why the answer is not as simple as "buy the fastest one."

Why does looking at the hourly rate alone give you the wrong answer?

Because the real cost formula depends on throughput.
Cost per million tokens = (price per hour ÷ tokens per second ÷ 3600) × 1,000,000
A GPU with a higher hourly rate but significantly faster token generation can easily come out cheaper per million tokens than a cheaper card that takes much longer to produce the same output.
So the most important factor to consider here is memory bandwidth. It is what decides the token generation speed. During the decode phase, where a model generates output tokens one at a time, the GPU has to read the entire model weight matrix from VRAM for every single token. This makes decode almost entirely memory-bandwidth-bound rather than compute-bound, which is exactly why bandwidth, not raw TFLOPS, tends to predict real-world token throughput so closely.

How do these four GPUs actually compare on memory bandwidth?

Since bandwidth drives decode speed so directly, this is the number worth paying attention to first.
GPU
Memory bandwidth
Memory type
L4
300 GB/s
GDDR6
L40S
Higher than L4, still GDDR6
GDDR6
A100 (80GB)
Roughly 2,039 GB/s
HBM2e
H100
Roughly 3,350 GB/s
HBM3

That bandwidth gap between A100 and H100 maps almost directly onto real throughput. Published benchmarks show the H100 generating roughly 2.15 times more tokens per second than the A100 on a 70B model at batch size 1, tracking closely to their roughly 1.65x bandwidth ratio.
Does that mean H100 always wins on cost per token?
Not always, and this is where it gets genuinely interesting.
On deep spot-market pricing, A100 instances can sometimes undercut H100 so significantly on hourly rate that the A100 ends up cheaper per token for large models, despite being slower, simply because the price gap outweighs the throughput gap
Batching changes the economics dramatically on any GPU. Moving from single-stream requests to a batch of 8 concurrent requests can cut H100 cost per million tokens by roughly 4x, since GPU utilization affects real cost as much as the hardware itself does

What does the actual math look like for a 70B model?

Here is a rough, illustrative comparison based on published benchmark patterns. Real numbers vary by provider, quantization, and framework, so treat this as directional rather than exact.
Scenario
Approximate cost per million output tokens
H100, single-stream request
Roughly $0.40 to $0.45
H100, batch of 8 concurrent requests
Roughly $0.10 to $0.15
A100, on-demand pricing, similar batching
Noticeably higher than H100 at equal batch size, due to slower throughput
A100, deep spot pricing
Can undercut H100 despite slower speed, depending on the size of the discount

The pattern that holds up consistently is that batching and utilization move the needle on cost per token more dramatically than most people expect, often more than the choice of GPU generation itself.

Where does the L4 actually win, and where does it clearly lose?

The L4 was never built to compete with A100 or H100 on large model throughput. Its advantage lives entirely in a different lane.
But when we talk about small models, the Nvidia L4 vs A100 or H100, results will always be in the favor of L4.
For 7B to 13B class models, the L4's much lower hourly rate more than compensates for its lower raw throughput, since these smaller models do not need the bandwidth headroom that makes the A100 worthwhile in the first place. Running a small model on an A100 is usually paying for capacity that workload will never use, which shows up directly as a worse cost-per-token outcome despite the A100's objectively faster hardware.
L4 wins clearly on cost per token for small, latency-tolerant inference workloads
A100 only starts making sense once model size or concurrency genuinely demands its extra bandwidth and memory

Where does the L40S fit between L4 and A100?

Right in the middle, which is exactly the gap it was built to fill. For mid-size models in the 7B to 30B range, the L40S frequently delivers the best cost-per-token of the group, offering meaningfully more throughput than the L4 without the cost premium that comes with A100 or H100 class hardware.

So which GPU should you actually pick to minimize
cost per token?
The right answer depends entirely on model size and how much concurrency you actually need.
Small models (7B to 13B), low to moderate concurrency: L4, almost every time
Mid-size models (7B to 30B) needing more headroom: L40S is usually the sweet spot
Large models (70B and up), production volume: H100, provided you are running with real batching rather than single-stream requests
Large models on a tight budget: A100, particularly worth checking spot pricing before defaulting to H100 by assumption
Always check batching and utilization before assuming a newer GPU is automatically cheaper. A poorly utilized H100 can lose to a well-utilized A100 on cost per token, even though the H100 wins comfortably on every other spec.

Where this leaves you

The hourly price on a GPU listing answers the wrong question. Cost per token depends on throughput, memory bandwidth, batching, and how well you actually utilize the hardware, not just what the card costs to rent for sixty minutes. Match the GPU to your actual model size and concurrency pattern, check batching before assuming single-stream numbers apply, and the cheapest-looking hourly rate is not always the cheapest way to actually generate your tokens.

Frequently asked questions

Is it ever worth paying more per hour for a faster GPU just to lower cost per token?
Often yes, specifically for large models running at real production volume with proper batching. The higher hourly rate is frequently offset, and sometimes more than offset, by the dramatically higher throughput once utilization is genuinely high.
Does batch size matter more than GPU choice for cost per token?
For many workloads, yes. Moving from single-stream to a reasonable batch size can cut cost per token by several times on the same hardware, often a bigger swing than switching to a faster GPU while staying at low utilization.
Should I default to the newest GPU generation to minimize cost per token?
Not automatically. The newest generation usually wins for large models at high utilization, but for small models, an older or smaller card like the L4 frequently wins on cost per token simply because it is not being paid for capacity the workload never uses.

Is the RTX 5090 enough for AI? RTX PRO 6000 Blackwell vs RTX 5090 compared

Prateek Navani — Tue, 13 Oct 2026 14:31:09 +0000

If you are shopping for a GPU to run AI workloads right now, you have probably landed on the same two names everyone else does. The RTX 5090 because it is the flagship consumer card, and the RTX PRO 6000 Blackwell because someone in a forum told you it is what "serious" AI teams actually use.
I have gone through the specs, the pricing, and the real workload limits on both cards, and the honest answer is that neither one is automatically the right pick. It depends entirely on what you are training, how much memory your models actually need, and whether you plan to run this on a desk or in a server rack. Let me walk you through it properly.
Why does this question even come up for AI work specifically?
Because gaming GPUs and AI GPUs get judged on completely different things. For gaming, clock speed and frame rates matter most. For AI, memory capacity and memory bandwidth decide almost everything.
The RTX 5090 is an excellent gaming card. That part is not in question. What is in question is whether its memory setup, built primarily for 4K gaming and content creation, holds up once you point it at real model training or large scale inference.
What actually changed with the RTX 5090 compared to the previous generation?
The RTX 5090 ships with 32GB of GDDR7 VRAM on a 512-bit bus, delivering close to 1.79 TB/s of memory bandwidth. That is a real jump from the RTX 4090's 24GB and roughly 1 TB/s. It also carries 21,760 CUDA cores, a sizeable increase over the previous flagship.
It launched in January 2025 at a $1,999 MSRP. By mid-2026, GDDR7 supply shortages have pushed street prices well past that, often into the $3,000 to $5,000 range depending on the model and availability.
What can the RTX 5090 actually handle for AI work?
Quite a lot, honestly. This is where I think a lot of people underestimate it.
Fine-tuning models in the 7B to 30B parameter range with reasonable batch sizes
Running inference on quantized versions of 70B+ parameter models
Handling high resolution image and video generation pipelines without constant memory errors
Serving as a genuinely capable single-GPU inference box for smaller production workloads
For a solo developer, a researcher, or a small team experimenting with open source models, 32GB covers a surprising amount of ground.
Where does the RTX 5090 start to struggle?
Three places, consistently.
Unquantized 70B+ models: A 70B model in 4-bit quantization needs roughly 35 to 40GB, which is already tighter than the 5090's 32GB. Full precision work on models that size is not realistic on a single card.
No NVLink: The RTX 5090 is a PCIe-only card. Multi-GPU setups do not pool memory automatically, so scaling beyond one card for large training runs gets messy fast.
Licensing: It runs under Nvidia's consumer GeForce EULA, which restricts formal data center deployment. That matters if you are building something you plan to run commercially at scale, not just on your own desk.
What is the RTX PRO 6000 Blackwell, and how different is it really?
Here is the part that surprises people. The RTX PRO 6000 Blackwell uses the exact same GB202 die as the RTX 5090. The difference is almost entirely in memory, reliability features, and licensing, not the underlying silicon.
It ships with 96GB of ECC GDDR7 VRAM, a dedicated NVLink connector on multi-GPU configurations, and full data center EULA compliance. It launched in March 2025 at roughly $8,565 and has climbed to around $13,250 by August 2026, again driven by GDDR7 shortages rather than any change to the card itself.
What does the extra VRAM actually let you do?
This is the number that matters most for anyone doing real model work.
Spec
RTX 5090
RTX PRO 6000 Blackwell
VRAM
32GB GDDR7
96GB ECC GDDR7
CUDA cores
21,760
24,064
Memory bandwidth
~1.79 TB/s
~1.8 TB/s
NVLink support
No
Yes (multi-GPU configs)
ECC memory
No
Yes
Licensing
Consumer EULA
Data center EULA
Launch price
$1,999
$8,565
Mid-2026 street price
$3,000 to $5,000+
Around $13,250

With 96GB, a single RTX PRO 6000 Blackwell can run 70B parameter models at FP8 precision without quantization tricks. Pair two of them with NVLink and you get 192GB of pooled memory, enough for full unquantized 70B inference and genuinely serious fine-tuning work.
RTX 5090 vs RTX PRO 6000 Blackwell: who is each one actually built for?
Once you lay the specs side by side, the split becomes fairly obvious.
RTX 5090 fits solo builders, students, small teams, and anyone prototyping on personal or side project budgets
RTX PRO 6000 Blackwell fits teams running production inference, fine-tuning larger open models, or anyone who needs ECC memory and proper data center compliance
If you are building alone or just experimenting, which one makes sense?
The RTX 5090, almost every time. Unless you specifically need to run unquantized 70B+ models today, 32GB will cover fine-tuning, quantized inference, and generative media work without forcing you into a much larger budget.
If you are running this for a team or a product, which one makes sense?
This is usually where the calculation flips. Once you are serving production traffic, working with datasets that need ECC reliability, or planning to deploy in an actual data center environment, the licensing and memory ceiling on the RTX 5090 start working against you. That is exactly the gap the RTX PRO 6000 Blackwell was built to close.
Is renting a better option than buying either one?
For a lot of teams, yes, at least early on. Cloud pricing for the RTX 5090 currently runs somewhere between $0.29 and $0.85 per GPU hour depending on the provider, with a market median sitting around $0.46 to $0.72. That lets you test whether 32GB is genuinely enough for your workload before committing thousands of dollars to hardware you might outgrow in six months.
If you already know you need 96GB and ECC memory, renting an RTX PRO 6000 Blackwell instance is often more practical than buying one outright, especially with current prices sitting well above launch.
My honest take
If you are still deciding between these two, ask yourself one question first. Do you need to run models above 30 to 40 billion parameters without quantization, or do you need ECC memory and data center compliance right now?
If the answer is no, the RTX 5090 will serve you well and save you a significant amount of money. If the answer is yes, the RTX PRO 6000 Blackwell earns its price tag, and trying to force the 5090 into that role will likely cost you more in wasted time than the price difference between the two cards.
Frequently asked questions
Is the RTX 5090 actually usable for serious AI work, or is it just a gaming card?
It is genuinely usable. The 32GB of VRAM and high bandwidth make it capable for fine-tuning smaller models and running quantized inference on larger ones. It is not a data center card, but it is far from just a gaming GPU.
Why is the RTX PRO 6000 Blackwell so much more expensive if it uses the same core chip?
The price difference comes almost entirely from the memory. 96GB of ECC GDDR7, NVLink support, and data center licensing cost significantly more to produce and certify than the consumer configuration, and GDDR7 shortages have pushed both cards' prices up regardless.
Can I just buy two RTX 5090s instead of one RTX PRO 6000 Blackwell?
Not really in the way people hope. Since the RTX 5090 has no NVLink, two cards do not pool memory the way a dual RTX PRO 6000 Blackwell setup does. You get two separate 32GB pools, not one larger one, which limits what you can actually run across them.

AWS vs Indian cloud providers: data residency and compliance guide

Prateek Navani — Mon, 12 Oct 2026 05:12:20 +0000

Every compliance conversation in India eventually lands on the same question: can we just use AWS, or do we actually need an Indian provider?
The honest answer is "it depends on what you're storing," and most teams do not find that out until an auditor asks for evidence they cannot produce. Here is how to actually think through this decision.
Quick answer: AWS operates fully compliant, MeitY-empanelled data centers in India (Mumbai and Hyderabad), which satisfy most data residency requirements when configured correctly. Indian cloud providers offer the same residency benefits but add full jurisdictional independence, since they are not subject to foreign legal frameworks like the US CLOUD Act. Which one you need depends on the type of data you handle, not on which brand name feels safer.
**
The regulatory landscape you're actually dealing with**

India does not have one single data law. It has a few overlapping ones, and each applies differently depending on your business.
Regulation
What it covers
Where it applies
RBI Storage of Payment Data Direction (2018)
Payment and transaction data
Mandatory for anyone operating a payment system in India
DPDP Act, 2023
Personal data of Indian residents
Applies broadly, across almost every business collecting personal data
MeitY empanelment
Cloud providers approved for government and sensitive workloads
Required for government contracts, recommended for regulated sectors
Sectoral rules (SEBI, IRDAI)
Finance and insurance-specific data
Applies to regulated financial and insurance entities

If you only remember one thing from this table, remember this: RBI's rule is a hard localization requirement for payment data. DPDP is broader but more flexible, since it allows cross-border transfer unless the government specifically restricts a destination country.

What AWS actually offers in India

AWS runs two regions inside India: Mumbai (ap-south-1) and Hyderabad (ap-south-2). Both are fully operational, multi-availability-zone regions, and both have completed MeitY empanelment, which means they are approved for government and many regulated workloads.
This matters more than people assume. A few years ago, "use an Indian cloud provider" and "meet Indian compliance requirements" were treated as the same instruction. That is no longer accurate. AWS's Indian regions can satisfy residency requirements for most data categories, as long as you configure your architecture correctly.
Here's what "configured correctly" actually means in practice:
Data is stored and processed within the Indian region, not just accessed from India
Backups and disaster recovery also stay within Indian regions (Hyderabad backing up Mumbai, for example, not a US or EU region)
Encryption keys are managed within India, not in a foreign KMS region
Analytics and reporting pipelines don't quietly export data to a global warehouse outside India
That last point trips up more teams than any other. Your production database can be perfectly compliant in Mumbai while your analytics team is exporting the same data to a US-hosted dashboard tool without anyone flagging it as a compliance issue.

What Indian cloud providers add on top of that

Indian sovereign cloud providers, like CloudPe, Yotta, ESDS, NIC Cloud, and several others, offer the same regional residency AWS does. The difference is ownership and jurisdiction, not just server location.
This is the nuance most comparisons skip entirely: physical location and legal jurisdiction are not the same thing.
Here's the specific concern:
US-headquartered companies, including AWS, are subject to the US CLOUD Act. That law can compel a US company to hand over data to US authorities, regardless of where that data is physically stored, including a data center sitting in Mumbai. Storing data in India does not automatically remove that exposure if the company operating the infrastructure is US-domiciled.
Indian-owned AWS alternatives in India do not carry that specific exposure, because they are not subject to US jurisdiction in the first place. For most businesses, this distinction is theoretical. For businesses handling government data, defense-adjacent work, or highly sensitive personal data, it is often a contractual requirement, not a nice-to-have.

AWS vs Indian providers, side by side

AWS (India regions)
Indian sovereign cloud
Data residency
Yes, within Mumbai/Hyderabad regions
Yes, India-based by default
MeitY empanelment
Yes, both regions empanelled
Varies by provider, many are empanelled
Jurisdiction
Subject to US law (CLOUD Act)
Fully Indian jurisdiction
Global infrastructure and tooling
Extensive
Narrower, improving over time
Best fit
General enterprise workloads, teams already in the AWS ecosystem
Government data, defense-adjacent work, contracts requiring Indian ownership

Neither column is universally "better." They solve different problems, and the right answer depends entirely on what you're required to prove, not what feels more secure.

How to actually decide for your business

Skip the brand debate and start with your data.

Do you process payment data? RBI's localization rule applies regardless of provider. Use an Indian region, from AWS or an Indian provider, and confirm your disaster recovery setup also stays within India.
Are you a Significant Data Fiduciary under DPDP? If you process personal data of more than roughly two million Indians, you face added obligations, including an India-based data protection officer and independent audits. Your cloud provider's compliance posture becomes part of that audit trail.
Do you have government or defense-adjacent contracts? These often require India-owned infrastructure specifically, not just India-located infrastructure. Check the contract language before assuming a MeitY-empanelled AWS region satisfies it.
Is most of your business general SaaS without regulated data? An India-region deployment on any major cloud provider is usually sufficient, and switching providers purely for perceived compliance safety is rarely worth the migration cost.

Common mistakes worth avoiding

Assuming "India region" and "India compliant" mean the same thing” They don't, until every piece of your pipeline, including backups, analytics, and key management, actually stays inside that region.
Treating this as a one-time decision: DPDP enforcement is still in a phased rollout through 2027. Requirements you meet today may need re-verification as rules mature and get enforced more strictly.
Ignoring the disaster recovery region: This is the single most common gap. Teams localize production data correctly, then quietly replicate backups to a cheaper region outside India without realizing it breaks compliance.
Picking a provider before mapping your data: Provider selection should follow a data mapping exercise, not precede it. You cannot choose the right infrastructure until you know exactly what data you hold and which rule applies to it.

The bottom line

This is not really an "AWS versus Indian providers" decision. It's a data mapping exercise that happens to end with a provider choice.
Map what data you hold, match it against RBI, DPDP, and sector-specific rules, and then pick the infrastructure that satisfies the strictest requirement in that list. Get that order right, and the provider decision becomes straightforward instead of a debate.

Frequently asked questions

Is AWS compliant with Indian data residency laws?
Yes, when configured correctly. AWS's Mumbai and Hyderabad regions are both MeitY-empanelled and can satisfy most Indian data residency requirements, provided data, backups, and key management all stay within those regions.
Do I need an Indian-owned cloud provider for DPDP compliance?
Not necessarily. DPDP compliance is about how you handle personal data, not exclusively about provider ownership. However, certain sectors and government contracts do specifically require Indian-owned infrastructure.
Does storing data in AWS's Indian region protect it from foreign government access?
Not entirely. AWS is a US company and remains subject to the US CLOUD Act, which can compel data disclosure regardless of physical storage location. Indian-owned providers are not subject to this specific law.
What is the RBI data localization requirement?
The RBI's Storage of Payment System Data Direction requires that full, end-to-end payment transaction data for Indian customers be stored exclusively in India. This applies to any business operating a payment system, regardless of size.
Can I use a foreign cloud region for backups if my production data is in India?
For payment data covered under RBI's rule, no. Backups and disaster recovery for that data must also stay within India. For other data categories under DPDP, cross-border transfer is allowed unless the destination country is specifically restricted.

Where to rent an NVIDIA L4 GPU: providers compared

Prateek Navani — Thu, 08 Oct 2026 05:45:23 +0000

The L4 has become the default recommendation for cost-efficient inference, and once you start actually shopping for one, you will notice something odd. The exact same 24GB card shows up priced anywhere from a few cents an hour to well over a dollar, depending purely on where you rent it.
Let me walk through what actually explains that spread, and where to look depending on what you are trying to do.

Why does the exact same L4 cost such wildly different amounts across providers?

Because you are not just paying for the chip. You are paying for the tier of infrastructure and support wrapped around it.
Marketplace and spot platforms: They aggregate spare capacity from many hosts at the lowest prices, with the least guaranteed availability
Specialist GPU clouds: These are built specifically for developers running AI workloads, typically with per-second or per-minute billing and no long commitments
Hyperscalers: They offer the most reliable availability and enterprise SLAs, at a real price premium
Bundled compute platforms: This is where the headline rate includes CPU, RAM, and storage rather than just the GPU alone

Range of Nvidia L4 GPU price actually across legitimate providers
Across currently listed providers, on-demand L4 pricing runs roughly from $0.11 to $1.05 per GPU hour, with a market median commonly sitting between $0.44 and $0.81 depending on which set of providers you compare.
If you are buying the hardware outright instead of renting, expect to pay somewhere around $2,000 to $3,000 for the card itself.
You can also checkout → NVIDIA L4 GPU pricing blog for detailed breakdown.

Marketplace vs spot platforms

This is where the lowest headline numbers show up, sometimes dramatically so.
Marketplace platforms like Vast.ai list L4 access from roughly $0.11 to $0.33 per hour on a regular basis
Some listings briefly show prices even lower than that during periods of oversupply
Availability fluctuates constantly, since these platforms aggregate spare capacity from many independent hosts rather than guaranteeing dedicated infrastructure
But the real question is: Is the cheapest listed price actually usable, or mostly theoretical?
Extremely low headline rates, particularly anything under roughly $0.15 an hour, frequently reflect spot pricing tied to a specific host that may be out of stock by the time you try to launch, or availability limited to a single region. For anything beyond short experimentation, it is worth checking real-time availability rather than trusting the lowest number on a comparison table.

How do dedicated specialist GPU clouds compare?

This tier tends to offer the best balance of price and actual reliability for most developers. Most bill in fine-grained increments, per-second or per-minute, rather than rounding usage up to the nearest hour, though the exact granularity still varies by provider.
Provider
Typical on-demand rate
Billing style
Jarvislabs
Around $0.44/hr
Per-minute
RunPod
Around $0.44 to $0.49/hr
Per-second
TensorDock
From around $0.23/hr
Per-second
Theta EdgeCloud
Around $0.33/hr
Per-second
CloudPe
From ₹35.86/hr (roughly $0.41/hr)
Hourly, pay-as-you-go

These platforms are generally built specifically for AI workloads, with no long-term contracts required and billing that only charges for time the GPU is actually running.

How do hyperscalers compare, and is the higher price actually worth it?

Noticeably more expensive, but for reasons that matter to some buyers more than others.
Google Cloud typically runs around $0.55 to $0.71 per hour, and has the broadest global L4 availability of any major hyperscaler, having adopted the card early
AWS typically runs around $0.70 to $0.80 per hour, generally with solid availability
Microsoft Azure tends to be the most expensive of the three, around $0.65 to $0.70 per hour, backed by enterprise SLAs
The premium here mostly buys reliability and integration, not raw performance. If your infrastructure already lives inside one of these clouds, the convenience of staying there often outweighs the price difference for smaller workloads.

What if you are specifically looking for a provider based in India?

Domestic pricing varies more than you might expect. CloudPe, for instance, lists L4 access from ₹35.86 per hour, which lands right alongside the global specialist cloud tier above rather than at a premium. Other India-based providers price closer to hyperscaler rates for the same card. It is worth comparing any India-based listing directly against both the specialist cloud tier and the hyperscaler tier before committing, rather than assuming a domestic option is automatically cheaper or automatically pricier.

The hidden cost trap

The same headline number can mean two very different things depending on what it actually includes.
Some platforms quote an all-in rate covering GPU, CPU, RAM, and storage together. Others quote a GPU-only rate, then charge CPU and RAM separately on top of that number. Two providers can advertise an identical $0.80 per hour rate, while one of them ends up costing meaningfully more once the full compute stack is added in. Always check whether a quoted rate is GPU-only or fully bundled before comparing it directly against another provider's number.

Things to check beyond price before committing to a provider

It is easy to skip this step when a comparison table makes price look like the only variable that matters.
Real-time stock in your region: Several marketplace and even some specialist listings show as available on paper while sitting waitlisted in practice.
Minimum billing increment. A provider billing per-second or per-minute genuinely differs from one that rounds every session up to a full hour, and that gap adds up fast on short, frequent workloads.
So which provider tier should you actually pick?
Quick experimentation or short test runs: a marketplace platform, accepting some availability risk in exchange for the lowest price
Ongoing development or small-scale production inference: a specialist GPU cloud, for the best balance of price, reliability, and fine-grained, usage-based billing
Enterprise workloads needing SLAs or tight integration with existing cloud infrastructure: a hyperscaler, accepting the price premium for reliability
Data residency requirements specific to India: a domestic provider, but only after comparing its rate carefully against the ranges above

Conclusion

The L4 itself does not change from one provider to the next, but the price and reliability you get absolutely do. Match the provider tier to what you are actually doing: marketplace pricing for quick experiments, specialist clouds for steady development or small production workloads, and hyperscalers when reliability and integration matter more than shaving a few cents off the hourly rate. Check what is actually included in any quoted price before comparing it against another provider's number, and the comparison becomes a lot more honest.

Frequently asked questions

Is it worth using a marketplace platform for anything beyond testing?
For short, tolerant workloads, yes. For anything running continuously in production, the availability risk usually outweighs the savings, and a specialist GPU cloud tends to be the safer choice at only a modest price increase.
Why does Google Cloud have better L4 availability than AWS or Azure?
Google Cloud adopted the L4 earlier than the other major hyperscalers, giving it more time to build out capacity and regional coverage. That head start still shows up as meaningfully broader availability today.
Should I always compare the lowest listed price across providers?
Not without checking what it actually includes. A low headline rate that excludes CPU and RAM, or reflects unreliable spot capacity, can end up costing more or being less usable than a slightly higher, fully bundled rate from a different provider.