Why Your Pipeline Can't Talk to Anything: VPC Fundamentals for Data Engineers

The Diagram Nobody Explained

Somewhere in every data team's shared drive, there's an architecture diagram. Boxes labeled 10.0.1.0/24, arrows crossing between them, a little cloud icon marking "internet," maybe a padlock or two for good measure. On your first week at a new job, someone pulls it up during onboarding, gestures at it for ninety seconds, and says something like "so basically your cluster lives here, and it can talk to the warehouse over here." Everyone nods. You nod too.

And then, months later, you're staring at that same diagram trying to figure out why a job that's supposed to call an external API just... can't. And you realize you never actually learned to read the diagram — you just learned to trust that someone else understood it.

That gap is extremely common among data engineers, for a good reason: most of us are trained to think in DAGs, schemas, and transformations — not in route tables and CIDR blocks. We pick up just enough networking vocabulary to get by: a VPC is "the box our stuff lives in," security groups are "the firewall thing," NAT gateways are "that thing that costs a surprising amount of money." That's usually enough — right up until it isn't, and suddenly you're staring at that diagram with no real mental model to reason from.

This article is the mental model. No cloud console screenshots, no provider-specific quirks yet — just the underlying concepts that every cloud's networking is built on top of. Once these click, AWS and GCP (which we'll compare in Part 2) will feel like two dialects of the same language instead of two different languages entirely.

Let's start with the most basic question of all: what is an IP address, really?

Addresses and Neighborhoods: IP Addresses and CIDR

Every resource that talks over a network — a server, a database, your laptop — needs an address. An IPv4 address is 32 bits total, but nobody writes it as a string of 1s and 0s. Instead it's split into 4 octets (8 bits each), and each octet is written as a decimal number from 0–255: 10.0.0.0.

A single address isn't very useful on its own though — you need a whole range of them for your VPC's resources to draw from. That range is what CIDR notation describes: 10.0.0.0/16.

• 10.0.0.0 — the starting address of the range
• /16 — the prefix length: how many of the 32 bits are "locked" as the network part. Whatever's left over is free to vary — and that's what actually creates your pool of usable addresses.

So for /16: 16 bits are locked, leaving 16 bits free → 2¹⁶ = 65,536 possible addresses, all starting with 10.0..

The bits that stay locked don't have to land on a neat octet boundary — that's just what makes /8, /16, and /24 easy to reason about. Here's how the split shifts as the prefix length grows:

CIDR	Network bits (locked)	Host bits (free)	Available IPs	Example range
/16	16	16	65,536	10.0.0.0 – 10.0.255.255
/24	24	8	256	10.0.0.0 – 10.0.0.255
/28	28	4	16	10.0.0.0 – 10.0.0.15

The pattern: every extra locked bit cuts the available address count in half. That's why the rule of thumb feels backwards at first — a smaller number after the slash means fewer bits are locked, which means more room left for addresses:

One more thing worth flagging early, since it trips people up: a CIDR block only governs private addresses inside your VPC. If a resource needs to be reachable from the public internet, it gets a separate public IP from your cloud provider's own pool — that's not part of your CIDR range at all. We'll come back to that distinction when we talk about public vs. private subnets next.

Zooming out, this is exactly how a VPC is built: you start with one big CIDR block — your VPC, e.g. /16 — and carve it into smaller CIDR blocks — your subnets, e.g. /24 each:

This is exactly how a VPC works. You start with one big CIDR block — your VPC — and carve it into smaller CIDR blocks — your subnets. Which brings us to subnets themselves.

Tool: Use cidr.xyz for CIDR calculations.

Carving Up the Neighborhood: Subnets

If a VPC is a plot of land, subnets are how you zone it. And critically — a subnet lives in one specific location. In AWS, that location is a single Availability Zone. In GCP, it's a single region (we'll dig into why that distinction matters a lot in Part 2).

(One AWS quirk worth knowing up front: every subnet reserves five of its addresses — the first four and the last — for network housekeeping, so a /24 gives you 251 usable IPs, not 256. When a cluster "just fits" in a subnet's worth of addresses, it doesn't.)

Why bother splitting a VPC into subnets at all, instead of just dumping everything into one flat address space? Two reasons:

1. Blast radius and isolation. If your data warehouse's underlying instances live in one subnet and your public-facing ingestion API lives in another, you can write rules that say "nothing from the internet reaches the data subnet, full stop" — regardless of what happens to the ingestion layer.
2. Availability. Spreading subnets (and the resources in them) across multiple physical locations means a single data center hiccup doesn't take down your whole pipeline.

A common pattern you'll see constantly in data platforms: a public subnet for anything that needs to be internet-facing, and one or more private subnets for everything else — your Spark cluster, your orchestration workers, your database. We'll dedicate all of Part 3 to this pattern in detail, because it's genuinely the backbone of most production data architectures.

Notice what's not connected: the internet only ever touches the public subnet. Everything else is reachable only from inside the VPC — which is exactly the point.

For now, the key idea: subnets are how you decide what can be reached from where.

The Postal Service of Your VPC: Routing

Okay — you've got addresses, and you've got subnets. But how does a packet of data actually get from your Spark cluster to that third-party API? It needs directions. That's what a route table is: a list of "if you're trying to reach this address range, send the traffic here" rules.

Every subnet is associated with a route table. Here's roughly what one looks like conceptually:

Destination	Target
10.0.0.0/16 (inside the VPC)	local
0.0.0.0/0 (everything else)	Internet Gateway

That second row is the one that matters most. 0.0.0.0/0 means "anything not covered by a more specific rule" — i.e., the entire internet. Where that row points determines everything about a subnet's personality:

• Points to an Internet Gateway → this is a public subnet. Traffic can flow directly to and from the internet.
• Points to a NAT Gateway → this is a private subnet with outbound-only internet access (more on this in a second).
• Doesn't exist at all → this subnet is fully isolated. Nothing gets in or out via the internet, period.

Remember the opening story's job that couldn't reach the external API? A changed route table is exactly that kind of culprit: one edit silently reclassifies a subnet from "can reach the internet" to "cannot," with zero code changes on your end.

Nobody Memorizes IP Addresses: DNS

Nobody types 10.0.4.17 into anything. We type api.thirdparty.com or warehouse.internal.company.com, and something translates that name into an actual address. That something is DNS (Domain Name System), and inside a VPC it's doing more work than people realize.

Here's the flow when your job tries to reach api.thirdparty.com:

This matters for data engineers specifically because internal DNS is where a lot of "it works from my laptop but not from the cluster" bugs live. Your laptop resolves warehouse.internal.company.com through your company VPN. Your Spark cluster, sitting inside a VPC, resolves it through the VPC's own DNS resolver — and if that resolver isn't configured to know about your internal domains, or if DNS resolution isn't even enabled for the VPC, the name simply won't resolve, even though the network path might be perfectly fine.

If you ever see UnknownHostException or Name or service not known from inside a cluster that clearly can reach the internet, DNS resolution — not routing — is usually the first thing to check.

Going Outside Without Exposing Yourself: NAT

Here's a puzzle: your Spark cluster needs to reach the internet — to pull a Python package, call an external API, whatever. But you don't want it directly reachable from the internet; a compute cluster processing your company's data is not something you want an open door to.

NAT (Network Address Translation) solves exactly this. A NAT Gateway sits in a public subnet and does one job: it lets resources in private subnets initiate outbound connections to the internet, while remaining completely unreachable from the internet itself. Nobody outside can knock on your Spark cluster's door — but your Spark cluster can still walk outside to get what it needs.

This is also, quietly, one of the biggest line items on a lot of data platform cloud bills — NAT gateways charge per gigabyte processed, and data pipelines move a lot of gigabytes. If you've ever wondered why your team's networking cost jumped for no obvious reason, "someone's job started pulling a large dependency on every run through a NAT gateway" is a very common answer.

Putting It All Together

Here's every concept from this article in one picture — the same diagram from the intro, except now you can actually read it:

Four concepts, one flow: DNS turns a name into an address, the route table decides where the packet goes, NAT lets a private resource leave without letting anything in, and the Internet Gateway is the actual door to the outside world.

Why This Actually Matters for Your Pipelines

Let's tie all four concepts back to the kind of failures you'll actually hit as a data engineer:

• "My job can't reach an external API" → check the route table for 0.0.0.0/0. Is it even pointing anywhere?
• "My job works sometimes and times out other times" → could be a NAT Gateway running out of connections/bandwidth under load, a very real thing during large parallel job runs.
• "DNS resolution fails only from the cluster, not my laptop" → VPC DNS settings, not routing.
• "Egress costs spiked out of nowhere" → NAT Gateway data processing charges, often from a dependency install step nobody thought to cache.
• "A security review flagged our data warehouse as publicly reachable" → someone gave its subnet a route to an Internet Gateway when it should've had none at all.

None of these require you to become a network engineer. They just require the mental model we just built: addresses live in blocks, blocks live in subnets, subnets get their personality from their route table, names get resolved by DNS, and NAT lets you go out without letting anything in.

What's Next

In Part 2, we'll take this exact mental model and map it onto AWS and GCP side by side — including the single biggest conceptual difference between them (hint: it's about whether a VPC itself is regional or global, and it changes how you should think about almost everything else).

In Part 3, we'll build out the classic 3-tier VPC — public, private, and data subnets — as a complete, concrete architecture, including how a bastion host fits in as the one narrow, deliberate doorway into an otherwise sealed-off private tier.

For now — next time a pipeline can't reach something it used to reach fine, you've got a checklist to work through instead of a guess.