Networking for Data Engineers: VPCs, PrivateLink, Egress Costs & Cross-Cloud Transfer

Networking for data engineers is the layer everyone assumes the platform team already handled — right up until a pipeline that ran perfectly on a laptop times out in the cloud because a security group blocks the port, a route table points nowhere, or a DNS name resolves to an address nothing can reach. The warehouse, the Kafka cluster, the object-store lake, and the orchestrator do not float in the ether; they live inside virtual private clouds, carved into subnets, fenced by firewalls, and reached only along paths someone deliberately opened. A data engineer who cannot read a route table or explain why a connection hangs is permanently dependent on a networking team that does not understand the data flow — and that dependency is where pipelines go to die.

The second half of the story is money. Compute and storage are the line items everyone watches; data transfer is the one that quietly doubles the bill, because every byte that crosses an availability zone, a region, a cloud, or the public internet is metered — often in both directions, and usually invisibly until finance asks what a NAT gateway charge is. This guide is the working knowledge a senior data engineer is expected to have across both halves: VPC fundamentals (subnets, route tables, NAT, security groups, and endpoints), PrivateLink and private connectivity so the warehouse and Kafka are reachable without ever touching the public internet, the economics of egress costs across the same-AZ / cross-AZ / cross-region / internet ladder, and cross-cloud transfer patterns — peering, colocation, and dedicated interconnects — that keep moving data between AWS, GCP, and Azure from melting the budget. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, sharpen the architecture axis on the system design practice library →, pressure-test the cost trade-offs on the optimization practice library →, and rehearse the movement patterns on the data processing practice library →.

---

On this page

• Why networking matters for data engineers
• VPC fundamentals — subnets, routing, NAT, and endpoints
• PrivateLink and private connectivity
• Egress costs — where the bill comes from
• Cross-cloud transfer — moving data between clouds
• Cheat sheet — networking for data engineers
• Frequently asked questions
• Practice on PipeCode

---

1. Why networking matters for data engineers

Networking is the substrate every pipeline runs on — and egress is the bill nobody watches

The one-sentence invariant: networking for data engineers is the discipline of getting bytes to move between systems that live inside private cloud networks — which means two problems are always in play at once: reachability (can this pipeline actually connect to that warehouse, stream, or bucket, given the subnets, routes, firewalls, and DNS in the way?) and data-transfer cost (which boundary — availability zone, region, cloud, or the public internet — does each byte cross, and therefore what does it cost?) — and a senior data engineer is expected to reason about both without waiting for the networking team, because the pipeline's uptime and half its cloud bill are decided here. Ignore the first problem and pipelines hang mysteriously; ignore the second and the transfer line item silently overtakes compute.

The four axes interviewers actually probe.

• Reachability. Can the pipeline connect at all? This is subnets, route tables, security groups, NACLs, DNS, and endpoints. The senior answer debugs it in a fixed order — is the target's security group open to the source, does the route table have a path, does DNS resolve, is there an endpoint — rather than randomly widening firewall rules until something works. Reaching for 0.0.0.0/0 to "just make it connect" is the tell of someone who does not understand the path.
• Isolation. Does the traffic touch the public internet, and should it? A data plane carrying customer data across the open internet is both a security exposure and, often, a cost. The senior answer names private subnets, PrivateLink, and VPC endpoints as the way to keep service traffic on the cloud's private backbone.
• Data-transfer cost. Which boundary does each byte cross? Ingress is free; egress is priced by distance — same-AZ is cheap, cross-AZ is metered each way, cross-region more, and internet/cross-cloud most of all. The senior answer can name the boundary a byte crosses before it names the price, because the boundary is the cost.
• Throughput and reliability. Will the path carry the volume, and what breaks under load? A single NAT gateway, a peering link, or an interface endpoint has bandwidth and connection limits. The senior answer talks about placement, colocation, and spreading load across AZs as first-class concerns, not afterthoughts.

The 2026 reality — the data plane is a small set of well-worn primitives.

• The VPC is the boundary. Every managed service you use — the warehouse, the streaming cluster, the lake — either lives in your VPC or is reached from it. Understanding subnets, routing, and firewalls is not optional infrastructure trivia; it is how your pipeline reaches its own inputs and outputs.
• Private connectivity is the default expectation. Managed warehouses (Snowflake, Redshift) and message buses (MSK) expose PrivateLink endpoints; object stores expose gateway and interface endpoints. Serving data over the public internet is now the exception that needs justifying, not the norm.
• Egress is the silent line item. Cross-AZ replication in a streaming cluster, cross-region reads from a bucket, and cross-cloud copies are the three usual suspects behind a transfer bill nobody budgeted for. The primitives to control it — colocation, endpoints, compression, and interconnects — are all networking decisions.
• Cross-cloud multiplies everything. A data platform that spans AWS, GCP, and Azure pays internet-grade egress every time data leaves a cloud, so the architecture — what you replicate, what you compress, where compute runs — is dominated by transfer economics.

What interviewers listen for.

• Do you debug reachability in a fixed order (security group → route → DNS → endpoint) instead of widening firewalls at random? — senior signal.
• Do you name the boundary a byte crosses (AZ / region / internet) as where the cost comes from, not "data transfer is cheap"? — required answer.
• Do you reach for private connectivity (private subnets, PrivateLink, endpoints) by default rather than public IPs? — senior signal.
• Do you treat colocating compute with data as the first egress-avoidance move? — required answer.
• Do you know that cross-AZ traffic is metered in both directions, and that a NAT gateway adds its own per-GB charge? — senior signal.

Worked example — the egress boundary ladder every data engineer memorises

Detailed explanation. The single most useful artifact for a cloud-networking interview is a memorised mapping of boundary crossed → who pays and roughly how much. Every cost conversation converges on it: two systems exchanging data pay nothing, a little, or a lot depending entirely on the boundary between them. Build the ladder for a pipeline reading from and writing to systems scattered across a cloud.

• The boundaries. Same host/same-AZ, cross-AZ (same region), cross-region (same cloud), and internet/cross-cloud.
• The asymmetry. Ingress (data into a cloud) is almost always free; egress (data out) is what is metered — and cross-AZ is metered on both ends.
• The rule. Name the boundary first; the price follows from the boundary, and the cheapest architecture is the one that crosses the fewest, cheapest boundaries.

Question. For each pair of endpoints, name the boundary crossed and the rough per-GB cost direction, using illustrative AWS list prices.

Input.

Path	Boundary crossed	Rough cost (illustrative)
Same AZ, private IPs	none (intra-AZ)	~$0.00/GB
Across AZs, same region	cross-AZ	~$0.01/GB each direction
Across regions, same cloud	cross-region	~$0.02/GB
Out to the internet / another cloud	internet egress	~$0.05–$0.09/GB (tiered)

Code.

The egress ladder — memorise the ORDER, not the exact cents (prices change).

  cheapest  same-AZ, private IPs .......... ~free       (colocate here!)
     |      cross-AZ, same region ......... ~$0.01/GB   x2 (charged BOTH ways)
     |      cross-region, same cloud ...... ~$0.02/GB
  priciest  internet / cross-cloud ........ ~$0.05-0.09/GB (tiered, source pays)

Two rules that follow from the ladder:
  1. INGRESS is free; EGRESS is what you pay. Pulling data IN is cheap;
     pushing/serving data OUT (or across a boundary) is the cost.
  2. A NAT gateway adds its OWN per-GB "data processing" charge (~$0.045/GB)
     ON TOP of egress — so private-subnet traffic to the internet is billed twice.

Design consequence: put the two chattiest systems on the SAME side of the
cheapest boundary you can, and never route service traffic through a NAT you
could replace with a VPC endpoint.

Step-by-step explanation.

1. The ladder is ordered by boundary, not by service: two EC2 instances, a broker and a consumer, or a Spark executor and a shuffle partner all pay the same rate for the same boundary. Memorising the four rungs lets you price any path by first asking "which boundary?"
2. Same-AZ private-IP traffic is effectively free, which is why the first move in any cost-sensitive design is colocation: put the producer and consumer in the same AZ so their chatter never climbs the ladder.
3. Cross-AZ is the trap most engineers miss — it is metered on both the sending and receiving side, so a byte crossing an AZ boundary is billed twice. A three-AZ Kafka cluster replicating and serving across AZs can spend more on cross-AZ transfer than on brokers.
4. Cross-region and internet/cross-cloud are progressively more expensive and tiered, and the source side pays. Cross-cloud is just internet egress from the source cloud — there is no special cheaper "cloud-to-cloud" rate on the public path.
5. The NAT gateway charge is the silent multiplier: private-subnet traffic that reaches the internet through a NAT pays the NAT's per-GB processing fee plus the egress, so an S3 read routed through a NAT costs far more than the same read through a (free) gateway endpoint.

Output.

Design move	Boundary it avoids	Effect on the bill
Colocate producer + consumer in one AZ	cross-AZ (×2)	removes the most common hidden charge
S3 gateway endpoint instead of NAT	NAT processing + egress	removes a double charge on bucket reads
Keep compute in the data's region	cross-region	removes ~$0.02/GB on every byte
Avoid unnecessary cross-cloud copies	internet egress	removes the priciest rung entirely

Rule of thumb. Name the boundary before you name the price: same-AZ is ~free, cross-AZ is metered both ways, cross-region more, internet/cross-cloud most — and a NAT adds its own per-GB fee. Design so the chattiest systems sit on the same side of the cheapest boundary you can.

Worked example — what interviewers actually probe

Detailed explanation. The senior networking interview for a data role has a predictable escalation: a connectivity opener ("your job can't reach the warehouse"), then a pivot to cost ("now the transfer bill tripled"). Candidates who debug reachability in a fixed order and can name the boundary behind a cost score highest.

• Ambiguous opener. "Your Spark job can't connect to Snowflake. Where do you look?" — probes reachability method.
• Follow-up 1. "It connects now, but the platform team says it's going over the public internet. Fix it." — probes private connectivity.
• Follow-up 2. "The data-transfer bill tripled this month. Why?" — probes egress literacy.
• Follow-up 3. "You have a Kafka cluster across three AZs. Where's the hidden cost?" — probes cross-AZ.
• Follow-up 4. "You're copying a bucket from AWS to GCP nightly. Make it cheap." — probes cross-cloud.

Question. Draft a senior answer that pre-empts all four follow-ups — a debugging order and a cost model, not a list of random fixes.

Input.

Interview signal	Weak answer	Senior answer
Can't connect	"open the security group to 0.0.0.0/0"	"check SG → route → DNS → endpoint, in that order"
Traffic on the internet	"it works, leave it"	"use PrivateLink / a VPC endpoint; keep it private"
Bill tripled	"data transfer is unpredictable"	"name the boundary: cross-AZ, cross-region, or egress"
Kafka cost	"brokers are expensive"	"cross-AZ replication + fetch; use rack awareness"
Cross-cloud copy	"just re-copy it nightly"	"ship deltas, compress, or use an interconnect"

Code.

Senior networking-for-DE answer template
=========================================

Reachability (fixed order — never random)
  1. Security group  : is the TARGET's inbound rule open to the SOURCE's SG/CIDR + port?
  2. Route table     : does the source subnet have a route to the target (or a NAT/endpoint)?
  3. DNS             : does the hostname resolve, and to a reachable (often private) IP?
  4. Endpoint        : is there a VPC/interface endpoint, or is it (wrongly) going public?

Cost (name the boundary, then the fix)
  - same-AZ    -> ~free  -> colocate the chatty pair here
  - cross-AZ   -> x2     -> AZ-aware clients, rack awareness, fetch-from-follower
  - cross-region -> $$   -> keep compute in the data's region
  - internet/cross-cloud -> $$$ -> ship deltas, compress, or buy an interconnect
  - through a NAT -> +fee -> replace with a VPC endpoint where possible

Step-by-step explanation.

1. The reachability order is the whole point: security group first (the most common cause), then route, then DNS, then endpoint. Fixing them in order finds the real problem instead of masking it by widening a firewall — which often "works" while leaving the actual misconfiguration in place.
2. The private-connectivity pivot shows you treat public internet traffic as a smell: naming PrivateLink or a VPC endpoint signals you keep the data plane on the cloud backbone by default, which is both a security and a cost win.
3. The cost answer is a boundary answer. Weak candidates call transfer "unpredictable"; senior candidates decompose the bill into cross-AZ, cross-region, and egress components, because each has a different, specific fix.
4. The Kafka follow-up is a trap for anyone who has not run a cluster: the hidden cost is cross-AZ replication and cross-AZ consumer fetch, not the brokers themselves — and the fix (rack awareness / fetch-from-follower) is a networking decision.
5. The cross-cloud close shows you know the source cloud pays egress and that the levers are what crosses (deltas, compressed, columnar) and how it crosses (public internet vs a dedicated interconnect) — the two knobs that decide a cross-cloud bill.

Output.

Grading criterion	Weak score	Senior score
Debugs reachability in a fixed order	rare	mandatory
Reaches for private connectivity	occasional	mandatory
Names the boundary behind a cost	rare	senior signal
Knows cross-AZ is the Kafka trap	rare	senior signal
Cross-cloud = ship less, or interconnect	rare	senior signal

Rule of thumb. The senior networking answer is two rehearsed models: a reachability debugging order (security group → route → DNS → endpoint) and a cost model that names the boundary a byte crosses before naming its fix. Deploy both in every infra interview.

Worked example — debugging a pipeline that cannot connect

Detailed explanation. The most common real incident is not a cost surprise but a hang: a pipeline that cannot reach its source. The amateur move is to widen the security group to 0.0.0.0/0 and move on; the senior move is to walk the path in order and find why. Debug a Spark job that times out connecting to a database.

• The symptom. Connection times out (hangs), rather than "connection refused" — a hang usually means a firewall or route is silently dropping packets.
• The order. Security group → route table → DNS → endpoint — each rules out one cause.
• The discipline. Change one thing, test, and never leave a 0.0.0.0/0 rule as the "fix."

Question. A Spark job in a private subnet times out connecting to a Postgres database in another subnet. Diagnose it without opening the firewall to the world.

Input.

Check	Question it answers	Common failure
Security group	is the DB's inbound rule open to the job's SG on 5432?	rule missing or wrong port
Route table	does the job's subnet have a route to the DB's subnet?	no route / wrong target
DNS	does the hostname resolve to the right (private) IP?	resolves to a public IP
Endpoint	is a service reached via an endpoint, or (wrongly) public?	no endpoint, NAT path fails

Code.

# 1. Security group — is the TARGET (db) open to the SOURCE (job) on the port?
#    A timeout (not "refused") usually means a firewall is dropping the packet.
aws ec2 describe-security-groups --group-ids sg-db \
  --query "SecurityGroups[].IpPermissions[?ToPort==\`5432\`]"
# Look for an inbound rule allowing the JOB's security group, e.g.
#   { "FromPort": 5432, "ToPort": 5432, "UserIdGroupPairs": [{"GroupId":"sg-spark"}] }

# 2. Route table — does the job's subnet have a path to the db's subnet/CIDR?
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-spark \
  --query "RouteTables[].Routes"
# Same-VPC subnets share the local route (10.0.0.0/16 -> local); across VPCs you
# need a peering/transit-gateway route, or the packet goes nowhere.

# 3. DNS — does the name resolve, and to a REACHABLE (usually private) address?
dig +short db.internal.example.com
# A private service that resolves to a PUBLIC IP is the classic "works then bills you"
# bug: traffic leaves via the NAT/IGW instead of staying private.

# 4. Endpoint — for AWS services, confirm an endpoint exists (else it needs NAT/IGW).
aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-123 \
  --query "VpcEndpoints[].{svc:ServiceName,state:State}"

Step-by-step explanation.

1. Security group first, because it is the single most common cause: the database's inbound rule must allow the job's security group (or CIDR) on the exact port. A missing or wrong-port rule shows up as a timeout, not a refusal, which is why hangs point here first.
2. If the SG is correct, check the route table: two subnets in the same VPC share the implicit local route, but across VPCs you need a peering or transit-gateway route, and without it the packet is silently discarded. This is the second-most-common cause.
3. DNS is third: the name must resolve, and — critically — to the right address. A private service that resolves to a public IP is the insidious bug where traffic "works" but leaves through the NAT/internet gateway, breaking privacy and adding cost.
4. The endpoint check is fourth: for an AWS service (S3, STS, a Snowflake endpoint), confirm a VPC/interface endpoint exists; without one, the traffic must go through a NAT to the public endpoint, which both costs more and may be blocked.
5. The discipline is one change at a time and never leaving 0.0.0.0/0 behind. Widening the firewall to the world often makes the symptom disappear while leaving the real misconfiguration — and a gaping security hole — in place; the ordered walk finds the actual cause.

Output.

Cause found at step	Correct fix	Wrong fix
Security group	allow the source SG on the exact port	open to 0.0.0.0/0
Route table	add the peering/TGW route	make the instance public
DNS	point the name at the private IP	use the public hostname
Endpoint	add the VPC/interface endpoint	route service traffic via NAT

Rule of thumb. Debug a connection failure in a fixed order — security group, then route, then DNS, then endpoint — changing one thing at a time. A timeout points at a firewall or route; never "fix" it by opening the security group to the world, which hides the real cause and creates a breach.

Senior interview question on networking fundamentals for a data platform

A senior interviewer often opens with: "Your team is standing up a data platform in the cloud — a warehouse, a Kafka cluster, a Spark/ETL fleet, and an object-store lake. Design the networking: which subnets each component lives in, how the ETL fleet reaches the warehouse and the lake privately, how you keep the data plane off the public internet, and where you expect the data-transfer bill to come from — and how you'd hold it down before anyone complains."

Solution Using private subnets, VPC endpoints, AZ colocation, and a boundary-aware cost model

# 1. Placement — data planes in PRIVATE subnets, one AZ per zone, colocated by chatter.
VPC 10.0.0.0/16
  public subnets  (per AZ): NAT gateway, bastion/ingress only  (10.0.0.0/24, 10.0.1.0/24)
  private subnets (per AZ): warehouse, Kafka brokers, Spark, ETL (10.0.10.0/24, 10.0.11.0/24)
  -> Spark executors + the Kafka brokers they consume from live in the SAME AZ
     wherever possible, so shuffle/consume traffic stays intra-AZ (~free).

# 2. Private connectivity — endpoints so service traffic never touches the internet.
S3 (lake)        -> GATEWAY endpoint      (free; bypasses the NAT entirely)
Snowflake/Redshift -> INTERFACE endpoint (PrivateLink) with private-hosted-zone DNS
STS/Secrets/etc. -> INTERFACE endpoints    (so the ETL fleet needs no NAT for AWS APIs)
NAT gateway      -> only for genuine third-party internet calls (kept minimal)

# 3. Terraform — the S3 gateway endpoint that removes NAT cost on lake reads.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]   # associate the private RT
}

# 4. Cost model — where the bill comes from, and the pre-emptive fix for each.
Expected transfer cost          Cause                         Fix
------------------------------  ----------------------------  ---------------------------
cross-AZ Kafka replicate+fetch  brokers/consumers span AZs    rack awareness, AZ-local read
lake reads through the NAT      no S3 endpoint                gateway endpoint (above)
cross-region warehouse loads    compute != data region        keep compute in-region
internet egress on exports      serving/copying out           compress, batch, endpoint/CDN

Step-by-step trace.

Decision	Naive (works, but bleeds)	Designed (this solution)
Component placement	anything in a public subnet	data planes in private subnets
ETL → S3 lake	through a NAT (charged + fee)	S3 gateway endpoint (free, private)
ETL → warehouse	public endpoint over internet	PrivateLink interface endpoint
Kafka consume	cross-AZ by accident	AZ-colocated / rack-aware
Cost posture	discovered on the invoice	modelled per boundary up front
Public exposure	instances with public IPs	no public IPs on the data plane

After the rollout, every data-plane component sits in a private subnet with no public IP; the ETL fleet reaches the lake through a free S3 gateway endpoint and the warehouse through a PrivateLink interface endpoint, so no service traffic traverses the public internet or the NAT; Spark executors consume from brokers in the same AZ so the shuffle and fetch traffic stays intra-AZ; and the transfer bill has been modelled per boundary — cross-AZ, cross-region, egress — with a specific fix pre-applied to each. The NAT gateway exists only for the rare genuine outbound internet call.

Output:

Metric	Naive platform	Designed platform
Data-plane public exposure	public IPs everywhere	none (private subnets)
Lake read path	NAT (egress + processing fee)	gateway endpoint (free)
Warehouse path	public internet	PrivateLink (private)
Cross-AZ Kafka cost	unbounded, accidental	minimized (AZ-aware)
Transfer bill	a surprise on the invoice	modelled and pre-empted
Debuggability	random firewall widening	ordered SG→route→DNS→endpoint

Why this works — concept by concept:

• Private subnets for the data plane — putting the warehouse, brokers, and ETL fleet in private subnets with no public IPs means the only inbound paths are the ones you deliberately open, shrinking both the attack surface and the number of ways traffic can accidentally go (and be billed) over the internet.
• VPC endpoints over NAT — a gateway endpoint for S3 and interface endpoints for AWS/warehouse services keep service traffic on the cloud backbone and off the NAT, removing both the per-GB NAT processing fee and the public-internet exposure in one move.
• AZ colocation — placing the chattiest pairs (Spark executors and the brokers/partitions they read) in the same AZ keeps shuffle and consume traffic intra-AZ and ~free, defusing the cross-AZ charge that is the most common hidden cost in a streaming platform.
• Boundary-aware cost model — decomposing the expected bill into cross-AZ, cross-region, and egress components, each with a pre-applied fix, turns data transfer from an invoice surprise into a designed, bounded line item.
• Cost — free gateway-endpoint lake reads, intra-AZ consume traffic, and in-region compute versus NAT-routed reads, cross-AZ fetch, and cross-region loads. The eliminated cost is the entire class of accidental transfer charges — O(bytes on the cheapest boundary) instead of O(bytes × the priciest boundary they happened to cross).

Design
Topic — design
Design problems on cloud data-platform networking

Practice →

Optimization Topic — optimization Optimization problems on data-transfer cost

Practice →

---

2. VPC fundamentals — subnets, routing, NAT, and endpoints

A VPC is an IP space you carve into subnets; routes, firewalls, and endpoints decide where traffic goes

The mental model in one line: a VPC is a private IP address range (a CIDR block like 10.0.0.0/16) that you slice into subnets pinned to availability zones, where a route table attached to each subnet decides where its traffic can go — an internet gateway gives a subnet public reachability, a NAT gateway gives a private subnet outbound-only internet, security groups are stateful per-instance firewalls that only allow, NACLs are stateless per-subnet firewalls that allow and deny in order, and VPC endpoints let you reach cloud services without leaving the private network at all — so "public subnet" versus "private subnet" is not a setting but simply a consequence of what its route table points at. Get the routing and firewalls right and your data plane is reachable and private; get them wrong and you either cannot connect or you have published a database to the internet.

Subnets, CIDR, and availability zones.

• The VPC CIDR. A private range (10.0.0.0/16 — 65,536 addresses) you subdivide. Subnets are non-overlapping slices (10.0.1.0/24 — 256 addresses) of it.
• A subnet lives in one AZ. Each subnet is pinned to a single availability zone, which is why "spread across AZs" means "create a subnet per AZ" — and why cross-AZ traffic is a real, metered boundary.
• Public vs private is defined by the route table. A subnet is "public" only because its route table sends 0.0.0.0/0 to an internet gateway; a "private" subnet routes 0.0.0.0/0 to a NAT (or nowhere).
• Data planes belong in private subnets. Warehouses, brokers, and ETL workers get no public IP; they reach out through a NAT or, better, through endpoints, and are reached only from within the network.

Routing, internet gateway, and NAT.

• The route table. A list of destination CIDR → target rules. Every VPC has an implicit local route so all subnets can talk to each other; you add routes for the internet, peering, transit gateways, and endpoints.
• Internet gateway (IGW). Gives a subnet bidirectional public reachability — an instance with a public IP in a subnet routed to an IGW is on the internet.
• NAT gateway. Lets private-subnet instances make outbound internet connections (pull a package, call a third-party API) while remaining unreachable from outside — at the cost of a per-GB processing charge.
• The NAT is a cost and a bottleneck. All private-subnet internet traffic funnels through it, so it both bills per GB and caps throughput; endpoints exist partly to keep traffic off it.

Security groups vs NACLs.

• Security group (stateful, instance-level). Attached to an ENI/instance; allow-only rules; stateful, so if you allow an inbound request the response is automatically allowed back out. This is the firewall you use 95% of the time.
• NACL (stateless, subnet-level). Attached to a subnet; ordered allow and deny rules; stateless, so you must explicitly allow both the inbound request and the outbound response (ephemeral ports). A coarse, secondary control.
• The stateful difference matters. Forgetting that a security group is stateful leads to over-configuring return rules; forgetting a NACL is stateless leads to mysteriously blocked responses.
• Reference by group, not by IP. The senior pattern is to allow a security group to reference another security group (e.g. "the DB allows the Spark SG on 5432") rather than IP ranges, so it survives scaling and IP churn.

VPC endpoints.

• Gateway endpoints. For S3 and DynamoDB only; a route-table entry that sends that service's traffic onto the AWS backbone. Free, and it bypasses the NAT — the single biggest easy egress win for a lake.
• Interface endpoints (PrivateLink). For most other services; an ENI with a private IP in your subnet that the service is reached through. Hourly + per-GB cost, but keeps traffic private and off the internet (detailed in section 3).
• Why they matter for data. An ETL fleet that reaches S3, STS, Secrets Manager, and the warehouse through endpoints needs no NAT for those calls at all — cheaper, faster, and private.

The failure modes senior engineers pre-empt.

• 0.0.0.0/0 security-group rules. Opening a data-plane port to the world "to make it connect" exposes it to the internet. Mitigation: reference the source security group and the exact port; never the world.
• One NAT as a bottleneck (and single-AZ cost). A single NAT gateway concentrates all egress and, if it is in another AZ than the caller, adds cross-AZ charges on top. Mitigation: a NAT per AZ, and endpoints to avoid the NAT entirely.
• No S3 gateway endpoint. Lake reads routed through the NAT pay the processing fee for no reason. Mitigation: add the (free) gateway endpoint and route the private subnets to it.

Common interview probes on VPC fundamentals.

• "What makes a subnet public or private?" — its route table: 0.0.0.0/0 to an IGW is public; to a NAT (or nowhere) is private.
• "Security group vs NACL?" — SG is stateful/instance/allow-only; NACL is stateless/subnet/allow+deny ordered.
• "How do private instances reach the internet?" — a NAT gateway (outbound only), or endpoints for cloud services.
• "How do you reach S3 without the internet?" — an S3 gateway endpoint (free), routed from the private subnets.

Worked example — a VPC with public and private subnets and a NAT

Detailed explanation. The canonical data-platform VPC: public subnets holding only the NAT and ingress, private subnets holding the data plane, and route tables that make each "public" or "private." Build it in Terraform so the wiring is explicit.

• Public subnet. Routes 0.0.0.0/0 to the internet gateway; holds the NAT gateway.
• Private subnet. Routes 0.0.0.0/0 to the NAT; holds the warehouse/brokers/ETL.
• The distinction is purely the route. Same VPC, same kind of subnet resource — only the route table differs.

Question. Provision a VPC with one public and one private subnet, an internet gateway, and a NAT, wiring the route tables so the private subnet is outbound-only.

Input.

Resource	Routes 0.0.0.0/0 to	Holds
public subnet	internet gateway	NAT gateway, bastion
private subnet	NAT gateway	warehouse, brokers, ETL
internet gateway	— (attaches to VPC)	—
NAT gateway	— (lives in public subnet)	—

Code.

resource "aws_vpc" "main" { cidr_block = "10.0.0.0/16" }

# Public subnet: its route to the IGW is what makes it "public".
resource "aws_subnet" "public"  { vpc_id = aws_vpc.main.id  cidr_block = "10.0.0.0/24"  availability_zone = "us-east-1a" }
resource "aws_subnet" "private" { vpc_id = aws_vpc.main.id  cidr_block = "10.0.10.0/24" availability_zone = "us-east-1a" }

resource "aws_internet_gateway" "igw" { vpc_id = aws_vpc.main.id }

# A NAT lives IN the public subnet (it needs the IGW) and serves the private one.
resource "aws_eip" "nat" { domain = "vpc" }
resource "aws_nat_gateway" "nat" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public.id           # NAT sits in the PUBLIC subnet
}

# Public route table: 0.0.0.0/0 -> IGW  (this is what "public" means).
resource "aws_route_table" "public" { vpc_id = aws_vpc.main.id }
resource "aws_route" "public_inet" {
  route_table_id = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = aws_internet_gateway.igw.id
}
resource "aws_route_table_association" "pub" {
  subnet_id = aws_subnet.public.id  route_table_id = aws_route_table.public.id
}

# Private route table: 0.0.0.0/0 -> NAT  (outbound-only; unreachable from outside).
resource "aws_route_table" "private" { vpc_id = aws_vpc.main.id }
resource "aws_route" "private_nat" {
  route_table_id = aws_route_table.private.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id = aws_nat_gateway.nat.id
}
resource "aws_route_table_association" "priv" {
  subnet_id = aws_subnet.private.id  route_table_id = aws_route_table.private.id
}

Step-by-step explanation.

1. The VPC declares the 10.0.0.0/16 address space; the two subnets are non-overlapping /24 slices of it, each pinned to an AZ — the subnets themselves are identical resources, so nothing yet makes one public.
2. The internet gateway attaches to the VPC, and the NAT gateway is created inside the public subnet — the NAT needs a route to the IGW to function, which is why it cannot live in the private subnet it serves.
3. The public route table sends 0.0.0.0/0 to the IGW and is associated with the public subnet; this association is the entire definition of "public" — an instance with a public IP here is on the internet.
4. The private route table sends 0.0.0.0/0 to the NAT and is associated with the private subnet, giving those instances outbound-only internet: they can pull packages or call APIs, but nothing outside can initiate a connection to them.
5. The result is the standard posture: the data plane lives in the private subnet, unreachable from the internet, egressing through the NAT only when it must — and the next example removes even that for S3.

Output.

Subnet	Route for 0.0.0.0/0	Reachable from internet?	Can egress?
public	internet gateway	yes (if public IP)	yes
private	NAT gateway	no	outbound only
(private, no NAT route)	none	no	no internet
(S3 via gateway endpoint)	endpoint (added next)	no	S3, privately

Rule of thumb. A subnet is public or private purely because of what its route table points 0.0.0.0/0 at — an internet gateway or a NAT. Put the NAT in a public subnet, put the data plane in a private subnet routed to the NAT, and you have the standard, safe topology.

Worked example — security group vs NACL, stateful vs stateless

Detailed explanation. The most-tested VPC distinction is security group versus NACL. The trap is state: a security group remembers the connection and auto-allows the response; a NACL does not, so you must open the return path yourself. Configure both for a warehouse that accepts connections from an ETL fleet.

• Security group (stateful). Allow inbound 5439 from the ETL security group; the response is automatically allowed back — no outbound rule needed for it.
• NACL (stateless). Allow inbound 5439 and explicitly allow the outbound ephemeral-port range for the response.
• The lesson. Stateful means "one rule per direction of intent"; stateless means "one rule per direction of packet."

Question. Allow an ETL fleet to reach a warehouse on port 5439 using a security group, and show what the equivalent NACL must also open that the security group did not.

Input.

Control	Level	State	Rules needed for one connection
Security group	instance/ENI	stateful	1 inbound (response auto-allowed)
NACL	subnet	stateless	inbound + outbound ephemeral
SG source	reference	—	the ETL security group (not an IP)
NACL range	ephemeral	—	1024–65535 for the return path

Code.

# SECURITY GROUP (stateful): ONE inbound rule; the response is auto-allowed.
resource "aws_security_group" "warehouse" {
  vpc_id = aws_vpc.main.id
  ingress {
    from_port       = 5439
    to_port         = 5439
    protocol        = "tcp"
    security_groups = [aws_security_group.etl.id]   # reference the SOURCE SG, not a CIDR
  }
  # NOTE: no egress rule is needed for the RESPONSE — stateful SGs allow it automatically.
  egress { from_port = 0  to_port = 0  protocol = "-1"  cidr_blocks = ["0.0.0.0/0"] }
}

# NACL (stateless): you must open BOTH the inbound request AND the outbound response.
resource "aws_network_acl_rule" "in_5439" {
  network_acl_id = aws_network_acl.data.id
  rule_number = 100  egress = false  protocol = "tcp"
  rule_action = "allow"  cidr_block = "10.0.11.0/24"  from_port = 5439  to_port = 5439
}
resource "aws_network_acl_rule" "out_ephemeral" {
  network_acl_id = aws_network_acl.data.id
  rule_number = 100  egress = true   protocol = "tcp"
  rule_action = "allow"  cidr_block = "10.0.11.0/24"  from_port = 1024  to_port = 65535  # RESPONSE path
}

Step-by-step explanation.

1. The security group needs a single inbound rule allowing the ETL security group on 5439. Because it is stateful, the warehouse's response back to the ETL worker is automatically permitted — you do not (and should not) write an outbound rule for the reply.
2. Referencing security_groups = [etl.id] instead of a CIDR is the senior move: the rule keeps working as the ETL fleet scales up, down, and churns IPs, because membership — not addresses — defines who is allowed.
3. The NACL, being stateless, treats every packet independently: allowing the inbound request on 5439 is not enough, because the response leaving on an ephemeral port is a separate packet that a stateless firewall will drop unless you explicitly allow the outbound 1024–65535 range.
4. This is exactly the bug that bites people: an over-tight NACL that allows the request but not the ephemeral response produces a connection that establishes and then hangs — indistinguishable at a glance from an application problem.
5. The practical guidance follows: use security groups as the primary control (stateful, expressive, group-referencing) and treat NACLs as a coarse subnet-wide backstop for broad allow/deny — not the place to express fine-grained per-service access.

Output.

Scenario	Security group	NACL
Allow inbound 5439	1 rule	1 rule
Allow the response	automatic (stateful)	must add ephemeral egress
Source is a scaling fleet	reference the SG	must list CIDRs
Deny a specific bad IP	not possible (allow-only)	possible (ordered deny)

Rule of thumb. Use security groups as your real firewall — stateful, instance-level, and best referenced by source security group so return traffic and scaling just work. Reserve NACLs for coarse subnet-wide allow/deny, and remember that being stateless they need the ephemeral return range opened explicitly.

Worked example — an S3 gateway endpoint that bypasses the NAT

Detailed explanation. The highest-leverage, lowest-effort networking change for a data platform is adding an S3 gateway endpoint. Without it, every lake read from a private subnet routes through the NAT — paying the NAT processing fee and egress. The gateway endpoint is free and keeps the traffic on the AWS backbone. Add it.

• Before. Private subnet → NAT → public S3 endpoint: NAT per-GB fee + egress, and a NAT bottleneck.
• After. Private subnet → S3 gateway endpoint → S3 on the backbone: free, private, no NAT.
• The mechanism. A route-table entry for S3's prefix list pointing at the endpoint.

Question. Route a private subnet's S3 traffic through a gateway endpoint so lake reads never touch the NAT, and lock it to the buckets you use.

Input.

Aspect	Through NAT	Through gateway endpoint
Cost	NAT fee + egress	free
Path	public S3 endpoint	AWS backbone
Throughput	NAT-limited	not NAT-limited
Config	default	a route + (optional) policy

Code.

# A GATEWAY endpoint for S3 — free, and it removes S3 traffic from the NAT.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]   # inject the S3 route here

  # Optional but recommended: lock the endpoint to the buckets you actually use.
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = "*"
      Action    = ["s3:GetObject", "s3:PutObject", "s3:ListBucket"]
      Resource  = ["arn:aws:s3:::my-lake", "arn:aws:s3:::my-lake/*"]
    }]
  })
}

# What changed in the private route table (managed by AWS for gateway endpoints):
#   BEFORE:  0.0.0.0/0 -> nat-gateway        (S3 traffic went to the NAT)
#   AFTER:   pl-63a5400a (S3 prefix list) -> vpce-0abc...   (S3 traffic -> endpoint)
#            0.0.0.0/0 -> nat-gateway        (only NON-S3 internet still uses the NAT)

Step-by-step explanation.

1. Declaring a Gateway endpoint for com.amazonaws.<region>.s3 and associating it with the private route table injects a route for S3's managed prefix list that points at the endpoint — a more specific route than 0.0.0.0/0, so S3 traffic now prefers the endpoint.
2. Because the route is more specific, all S3 requests from the private subnet leave via the endpoint on the AWS backbone; only genuine non-S3 internet traffic still falls through to the NAT default route.
3. The gateway endpoint is free — no hourly and no per-GB charge — so this single change removes both the NAT processing fee and the egress on every lake read, which for a data platform is often the largest easy saving available.
4. The endpoint policy is a second lever: restricting Resource to your lake buckets means the endpoint can only be used to reach those buckets, so even a compromised instance cannot exfiltrate to an arbitrary S3 bucket through it.
5. The result is faster (no NAT hop or NAT bandwidth cap), cheaper (free vs metered), and safer (private path plus a bucket allow-list) — the reason "add the S3 gateway endpoint" is the first thing a senior engineer checks on any AWS data platform.

Output.

Metric	Before (NAT)	After (gateway endpoint)
Per-GB cost of lake reads	NAT fee + egress	$0
Path	public internet	AWS backbone
NAT bandwidth pressure	high	S3 removed from it
Blast radius	any S3 bucket	allow-listed buckets

Rule of thumb. Add an S3 (and DynamoDB) gateway endpoint to every private route table — it is free, removes lake traffic from the NAT and the internet, and can be locked to your buckets with an endpoint policy. It is the highest-leverage networking change on an AWS data platform.

Senior interview question on designing a data-platform VPC

A senior interviewer might ask: "Design the VPC for a data platform. Lay out the subnets across AZs, decide what is public versus private and why, connect a private ETL fleet to the internet for third-party APIs and to S3 without a NAT, and set up the firewalls so the warehouse only accepts the ETL fleet — explaining the security-group versus NACL choices and the routing that makes it all work."

Solution Using per-AZ subnets, route tables, endpoints, and group-referencing security groups

# 1. Per-AZ public + private subnets; "public/private" is defined by the route table.
resource "aws_subnet" "public"  { count = 2  vpc_id = aws_vpc.main.id
  cidr_block = cidrsubnet("10.0.0.0/16", 8, count.index)          # 10.0.0.0/24, 10.0.1.0/24
  availability_zone = element(["us-east-1a","us-east-1b"], count.index) }
resource "aws_subnet" "private" { count = 2  vpc_id = aws_vpc.main.id
  cidr_block = cidrsubnet("10.0.0.0/16", 8, count.index + 10)      # 10.0.10.0/24, 10.0.11.0/24
  availability_zone = element(["us-east-1a","us-east-1b"], count.index) }

# 2. A NAT PER AZ (avoid cross-AZ NAT charges + a single-AZ bottleneck).
resource "aws_nat_gateway" "nat" { count = 2
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id }
resource "aws_route" "priv_nat" { count = 2
  route_table_id = aws_route_table.private[count.index].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id = aws_nat_gateway.nat[count.index].id }

# 3. Endpoints so the ETL fleet reaches S3 (free) and AWS APIs privately (no NAT).
resource "aws_vpc_endpoint" "s3" { vpc_id = aws_vpc.main.id
  service_name = "com.amazonaws.us-east-1.s3"  vpc_endpoint_type = "Gateway"
  route_table_ids = aws_route_table.private[*].id }
resource "aws_vpc_endpoint" "sts" { vpc_id = aws_vpc.main.id
  service_name = "com.amazonaws.us-east-1.sts"  vpc_endpoint_type = "Interface"
  subnet_ids = aws_subnet.private[*].id  private_dns_enabled = true }

# 4. Firewalls: warehouse accepts ONLY the ETL SG; SGs reference SGs, not IPs.
resource "aws_security_group" "warehouse" { vpc_id = aws_vpc.main.id
  ingress { from_port = 5439  to_port = 5439  protocol = "tcp"
            security_groups = [aws_security_group.etl.id] } }   # least privilege, scale-proof

Step-by-step trace.

Layer	Component	Purpose
Address space	10.0.0.0/16 VPC, /24 subnets per AZ	non-overlapping, AZ-pinned
Public/private	route table target (IGW vs NAT)	defines exposure
Egress	a NAT per AZ	no cross-AZ NAT charge, no bottleneck
Private service access	S3 gateway + interface endpoints	free/private, no NAT for AWS
Firewall	SG referencing the ETL SG	least-privilege, scale-proof
Backstop	NACL for coarse subnet deny	secondary control

After deployment, the platform has public subnets holding only NATs and ingress and private subnets holding the data plane, one NAT per AZ so egress never crosses an AZ to reach a NAT; the ETL fleet reaches S3 through a free gateway endpoint and AWS APIs (STS, Secrets) through interface endpoints, so it needs the NAT only for genuine third-party calls; and the warehouse's security group admits only the ETL security group on 5439, a rule that survives every scaling event because it references membership, not addresses.

Output:

Metric	Ad-hoc VPC	Designed VPC
Public exposure	instances with public IPs	data plane fully private
NAT topology	one shared (cross-AZ) NAT	one per AZ (no cross-AZ hop)
S3 access	via NAT (fee + egress)	gateway endpoint (free)
AWS API access	via NAT	interface endpoints (private)
Warehouse firewall	CIDR list to maintain	references the ETL SG
Rule durability	breaks on scaling/IP churn	stable across scaling

Why this works — concept by concept:

• Route-table-defined subnets — because "public" and "private" are just what a subnet's route table points 0.0.0.0/0 at, laying out per-AZ public and private subnets and their route tables is the whole exposure design, made explicit and reviewable in code.
• A NAT per AZ — one NAT per availability zone means a private instance always egresses through a NAT in its own AZ, avoiding both the cross-AZ transfer charge of reaching a NAT elsewhere and the single-NAT throughput bottleneck.
• Gateway plus interface endpoints — a free S3 gateway endpoint and interface endpoints for AWS APIs let the ETL fleet do most of its work without the NAT at all, which is cheaper, faster, and private in one decision.
• Group-referencing security groups — allowing the warehouse to admit the ETL security group rather than IP ranges gives least-privilege access that keeps working through every scale-up, scale-down, and IP change, eliminating a whole class of "it broke when we scaled" incidents.
• Cost — free endpoint traffic, in-AZ NAT egress, and no public IPs versus NAT-routed reads, cross-AZ NAT hops, and hand-maintained IP allow-lists. The eliminated cost is both the recurring NAT/egress waste and the operational cost of firewall churn — O(design once) instead of O(fix on every change).

Design
Topic — design
Design problems on VPC and subnet architecture

Practice →

ETL Topic — etl ETL problems on pipeline connectivity and routing

Practice →

---

3. PrivateLink and private connectivity

An interface endpoint puts a service on a private IP in your subnet; DNS points the name at it

The mental model in one line: PrivateLink exposes a service — an AWS API, a managed warehouse like Snowflake, an MSK/Kafka cluster, or a partner's application — as an interface VPC endpoint: an elastic network interface with a private IP inside your own subnet, so traffic to that service is routed over the AWS backbone and never touches the public internet, no internet gateway or NAT involved — and the piece that makes it transparent is DNS, because a private hosted zone (or the endpoint's private DNS) resolves the service's normal hostname to the endpoint's private IP, so your pipeline keeps using the same connection string while the packets quietly stay private. Get the endpoint and its DNS right and the warehouse is reachable, private, and off the internet; get the DNS wrong and the name still resolves to the public address and your "private" traffic leaks out through the NAT.

Interface endpoints vs gateway endpoints.

• Gateway endpoint (S3, DynamoDB only). A route-table entry; free; no ENI. Covered in section 2 — the cheap default for those two services.
• Interface endpoint (PrivateLink, everything else). An ENI with a private IP per AZ; hourly + per-GB cost; used for AWS APIs (STS, Secrets, Kinesis), managed data services, and partner services.
• Why interface endpoints cost more. They provision real network interfaces and carry a per-GB data-processing charge, so you use them where privacy matters, not for high-volume bulk that a gateway endpoint could serve free.
• One ENI per AZ. An interface endpoint places an ENI in each subnet you enable, so callers reach the endpoint in their own AZ — enable it in every AZ your callers run in to avoid a cross-AZ hop.

PrivateLink to a managed warehouse (Snowflake).

• The account URL. Snowflake exposes an AWS PrivateLink endpoint service; SYSTEM$GET_PRIVATELINK_CONFIG returns the service name and the special privatelink account URL to connect to.
• The endpoint. You create an interface VPC endpoint to Snowflake's endpoint-service name, which drops a private-IP ENI into your subnets.
• The DNS. A private hosted zone maps <account>.privatelink.snowflakecomputing.com (and related names) to the endpoint, so clients resolve to the private IP.
• The result. Queries and bulk loads/unloads travel over PrivateLink; the warehouse is never reached over the public internet, satisfying both security review and, often, lower/steadier transfer behaviour.

Private access to Kafka and to a partner service.

• MSK / Kafka. A cluster runs in your VPC (or is reached via multi-VPC connectivity/PrivateLink); brokers are on private IPs, and clients in peered/endpoint-connected VPCs reach them without the internet.
• Consuming a partner's service. If a partner publishes a PrivateLink endpoint service, you create an interface endpoint to it and consume their API privately — no public egress, no IP allow-listing dance.
• Publishing your own. You can put a Network Load Balancer in front of your service and expose it as an endpoint service, so your consumers reach you over PrivateLink — the mirror image.

DNS and endpoint policies — the two things people get wrong.

• Private DNS. With private_dns_enabled (AWS APIs) or a private hosted zone (Snowflake/custom), the service's normal hostname resolves to the endpoint's private IP inside the VPC — this is what makes the switch transparent to clients.
• The leak. If DNS is not overridden, the hostname still resolves publicly and traffic goes out through the NAT/IGW — the endpoint exists but is bypassed. Always verify resolution from inside the VPC.
• Endpoint policies. An interface or gateway endpoint can carry a resource policy restricting what can be reached through it (which buckets, which actions, which accounts), a second layer of least privilege on the private path.

The failure modes senior engineers pre-empt.

• DNS not overridden. The endpoint is created but the hostname resolves to the public IP, so traffic silently uses the internet. Mitigation: enable private DNS / create the private hosted zone and test resolution from a private instance.
• Endpoint enabled in too few AZs. Callers in an AZ without an endpoint ENI cross an AZ to reach one, adding latency and cross-AZ cost. Mitigation: enable the endpoint in every AZ that has callers.
• Over-broad endpoint policy. A default "allow all" endpoint policy lets a compromised instance reach any resource of that service. Mitigation: scope the policy to named buckets/actions/accounts.

Common interview probes on PrivateLink.

• "How do you reach Snowflake without the public internet?" — an interface VPC endpoint to Snowflake's endpoint service plus private-hosted-zone DNS for the privatelink URL.
• "Interface vs gateway endpoint?" — gateway (free, route-based) for S3/DynamoDB; interface (PrivateLink, ENI, priced) for everything else.
• "What breaks most often with PrivateLink?" — DNS: the endpoint exists but the name still resolves publicly.
• "How do you expose your own service privately?" — an NLB behind an endpoint service that consumers reach via interface endpoints.

Worked example — an interface VPC endpoint with an endpoint policy

Detailed explanation. The building block of private connectivity is an interface endpoint. Create one for an AWS API (STS) so an ETL fleet can assume roles without a NAT, and attach a policy scoping what it may do. The same shape applies to any PrivateLink service.

• The endpoint. An interface endpoint to com.amazonaws.<region>.sts, with an ENI per private subnet.
• Private DNS. private_dns_enabled = true so sts.<region>.amazonaws.com resolves to the private IP.
• The policy. Restrict the endpoint to your account's principals/actions.

Question. Give a private ETL fleet access to STS over PrivateLink — no NAT, private DNS, and an endpoint policy that limits it.

Input.

Piece	Value
Service	com.amazonaws.us-east-1.sts
Type	Interface (PrivateLink)
DNS	private_dns_enabled = true
ENIs	one per private subnet (per AZ)
Policy	allow only sts:AssumeRole for your account

Code.

resource "aws_vpc_endpoint" "sts" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.sts"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id     # an ENI in EACH AZ's subnet
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true                          # sts.<region>.amazonaws.com -> private IP

  # Endpoint policy: least privilege ON THE PRIVATE PATH itself.
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = "*"
      Action    = ["sts:AssumeRole"]
      Resource  = "*"
      Condition = { StringEquals = { "aws:PrincipalAccount" = "111122223333" } }
    }]
  })
}

# Verify from a PRIVATE instance that the name now resolves to a PRIVATE IP:
dig +short sts.us-east-1.amazonaws.com
#   10.0.10.42        <- GOOD: private endpoint IP (traffic stays on the backbone)
#   52.94.x.x         <- BAD:  public IP -> private_dns not effective -> going via NAT

Step-by-step explanation.

1. Declaring the interface endpoint for STS provisions an ENI with a private IP in each listed private subnet, so callers in any of those AZs reach STS through an endpoint in their own AZ — no NAT, no internet, no cross-AZ hop.
2. private_dns_enabled = true is the transparency switch: it makes the standard sts.<region>.amazonaws.com hostname resolve, inside the VPC, to the endpoint's private IP, so the ETL code needs no change — it uses the normal endpoint and the packets quietly go private.
3. The dig check from a private instance is the non-negotiable verification: a private IP proves the traffic is on the backbone; a public IP means private DNS is not effective and calls are still leaking through the NAT despite the endpoint existing.
4. The endpoint policy restricts the private path itself — here to sts:AssumeRole for your own account — so even a compromised host cannot use the endpoint to assume roles in an arbitrary external account, a second least-privilege layer beyond IAM.
5. The net effect is that the ETL fleet obtains credentials privately and cheaply, and the presence of the endpoint plus its policy is auditable in code — the pattern you replicate for Secrets Manager, Kinesis, and any partner PrivateLink service.

Output.

Property	Without endpoint	With interface endpoint
Path to STS	NAT → public internet	private ENI on the backbone
DNS resolves to	public IP	private endpoint IP
NAT dependency	yes	none for STS
Access scope	IAM only	IAM + endpoint policy

Rule of thumb. For any non-S3/DynamoDB service you need privately, create an interface endpoint in every caller AZ, enable private DNS so the normal hostname resolves to the private IP, verify resolution with dig from inside the VPC, and scope the endpoint policy. The endpoint without the DNS override is a no-op that still bills you.

Worked example — Snowflake PrivateLink with private-hosted-zone DNS

Detailed explanation. Connecting a warehouse over PrivateLink is the marquee private-connectivity task. For Snowflake it is three moves: get the PrivateLink config, create the interface endpoint to Snowflake's endpoint service, and wire a private hosted zone so the privatelink account URL resolves privately. Do it.

• Get the config. SELECT SYSTEM$GET_PRIVATELINK_CONFIG(); returns the endpoint-service name and the privatelink URLs.
• Create the endpoint. An interface endpoint to that endpoint-service name.
• Wire DNS. A private hosted zone for privatelink.snowflakecomputing.com aliasing to the endpoint.

Question. Make a private ETL fleet connect to Snowflake over PrivateLink, so no query or bulk load traverses the public internet.

Input.

Step	Action
1	SYSTEM$GET_PRIVATELINK_CONFIG → service name + URLs
2	interface endpoint to the Snowflake endpoint service
3	private hosted zone privatelink.snowflakecomputing.com
4	connect to <account>.privatelink.snowflakecomputing.com

Code.

-- 1. In Snowflake (ACCOUNTADMIN): get your account's PrivateLink details.
SELECT SYSTEM$GET_PRIVATELINK_CONFIG();
-- Returns JSON incl.:
--   "privatelink-account-url": "ab12345.us-east-1.privatelink.snowflakecomputing.com"
--   "privatelink-vpce-id"    : "com.amazonaws.vpce.us-east-1.vpce-svc-xxxxxxxx"

-- 2. Interface endpoint to Snowflake's endpoint SERVICE (not an AWS service name).
resource "aws_vpc_endpoint" "snowflake" {
  vpc_id             = aws_vpc.main.id
  service_name       = "com.amazonaws.vpce.us-east-1.vpce-svc-xxxxxxxx"  # from step 1
  vpc_endpoint_type  = "Interface"
  subnet_ids         = aws_subnet.private[*].id
  security_group_ids = [aws_security_group.endpoints.id]
  # NOTE: private_dns_enabled does NOT apply to a partner endpoint service —
  # you must create the private hosted zone yourself (step 3).
}

-- 3. Private hosted zone so the privatelink URL resolves to the endpoint.
resource "aws_route53_zone" "snowflake" {
  name = "privatelink.snowflakecomputing.com"
  vpc { vpc_id = aws_vpc.main.id }                 # PRIVATE zone, VPC-scoped
}
resource "aws_route53_record" "snowflake" {
  zone_id = aws_route53_zone.snowflake.id
  name    = "ab12345.us-east-1.privatelink.snowflakecomputing.com"
  type    = "CNAME"
  ttl     = 300
  records = [aws_vpc_endpoint.snowflake.dns_entry[0]["dns_name"]]   # -> endpoint DNS
}

Step-by-step explanation.

1. SYSTEM$GET_PRIVATELINK_CONFIG() is the source of truth: it returns your account's endpoint-service name (the vpce-svc-...) and the special privatelink-account-url you must connect to — the public account URL will not use PrivateLink even if the endpoint exists.
2. The interface endpoint targets Snowflake's endpoint service name, not an com.amazonaws.<region>.<service> name, because Snowflake is a third-party PrivateLink provider; this drops private-IP ENIs into your subnets pointing at Snowflake's service.
3. Crucially, private_dns_enabled does not work for a partner endpoint service, so you must create the private hosted zone yourself — this is the single most common Snowflake-PrivateLink mistake, where the endpoint is built but DNS is never wired and connections keep going public.
4. The private hosted zone for privatelink.snowflakecomputing.com, scoped to the VPC, plus a record aliasing the privatelink-account-url to the endpoint's DNS name, makes clients inside the VPC resolve the Snowflake hostname to the private endpoint — transparently.
5. Connecting with the privatelink-account-url (not the normal account URL) now routes queries, COPY INTO loads, and unloads over PrivateLink; from a private instance, dig on that URL should return the endpoint's private IP, proving no query touches the public internet.

Output.

Check	Public connection	PrivateLink connection
Account URL used	<acct>.snowflakecomputing.com	<acct>.privatelink.snowflakecomputing.com
Endpoint service	none	interface endpoint to vpce-svc-...
DNS resolves to	Snowflake public IP	endpoint private IP
Query/load path	public internet	AWS backbone (private)

Rule of thumb. Snowflake PrivateLink is three steps — SYSTEM$GET_PRIVATELINK_CONFIG, an interface endpoint to the returned endpoint-service name, and a self-managed private hosted zone for the privatelink URL — and you must connect to the privatelink-account-url, not the normal one. The DNS step is the one everyone forgets; verify it with dig from a private host.

Worked example — an endpoint policy that restricts an S3 gateway endpoint

Detailed explanation. Private connectivity is not only about reachability but about scoping the private path. An S3 gateway endpoint with no policy can reach any bucket; an endpoint policy restricts it to your buckets and even to your account, so the private door only opens to the rooms you own. Lock one down.

• The risk. A wide-open endpoint lets a compromised instance read/write arbitrary S3 buckets (including exfiltration targets).
• The control. An endpoint policy scoping Resource to your lake buckets and aws:PrincipalAccount/s3:ResourceAccount to your account.
• The layering. Endpoint policy + IAM + bucket policy = defence in depth on the private path.

Question. Restrict an S3 gateway endpoint so it can only be used to reach your own account's lake buckets, blocking exfiltration to third-party buckets.

Input.

Guard	Rule
Buckets	only my-lake (and its objects)
Account	only resources in 111122223333
Actions	read/write/list only
Effect	deny everything else through the endpoint

Code.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { # allow your lake buckets only
        Sid = "AllowLake", Effect = "Allow", Principal = "*",
        Action   = ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
        Resource = ["arn:aws:s3:::my-lake", "arn:aws:s3:::my-lake/*"]
      },
      { # hard-deny anything NOT in your account (blocks exfiltration)
        Sid = "DenyOtherAccounts", Effect = "Deny", Principal = "*", Action = "s3:*",
        Resource = "*",
        Condition = { StringNotEquals = { "s3:ResourceAccount" = "111122223333" } }
      }
    ]
  })
}

Step-by-step explanation.

1. The first statement allows only the specific lake bucket and its objects for the read/write/list actions the pipeline needs — so the endpoint's intended use is expressed as an explicit allow, not left to a default.
2. The second statement is the real teeth: an explicit Deny on any S3 resource whose s3:ResourceAccount is not your account. Because an explicit deny always wins, this blocks using the endpoint to reach any bucket in any other account — the classic exfiltration path.
3. Together they make the private door narrow: the endpoint can reach your lake and nothing else, so even a fully compromised instance with broad IAM cannot push data to an attacker-controlled bucket through this endpoint.
4. This composes with IAM and bucket policies as defence in depth: IAM says what the principal may do, the bucket policy says who may touch the bucket, and the endpoint policy says what may be reached through this network path — three independent gates.
5. The senior framing: a private path is not automatically a safe path. Scoping the endpoint policy turns "private connectivity" into "private and least-privilege connectivity," which is what a security review actually asks for.

Output.

Attempted access through endpoint	Verdict
read/write my-lake	allowed
list a different bucket in your account	allowed (if IAM permits)
write to a bucket in another account	denied (exfiltration blocked)
any action on a non-account resource	denied

Rule of thumb. Give every endpoint a policy, not just a route: allow the specific buckets/actions you use and hard-deny resources outside your account with s3:ResourceAccount/aws:PrincipalAccount conditions. A private path without an endpoint policy is still an open exfiltration route.

Senior interview question on private connectivity for a warehouse and lake

A senior interviewer might ask: "Security review says no data-plane traffic may traverse the public internet. Your ETL fleet talks to Snowflake, to S3, to Secrets Manager, and to a partner's API. Make every one of those connections private: which endpoint type for each, how DNS makes the switch transparent to the application, how you verify nothing is leaking to the public path, and how you scope each private path to least privilege."

Solution Using gateway and interface endpoints, private DNS, and scoped endpoint policies

# 1. S3 + DynamoDB -> GATEWAY endpoints (free, route-based, scoped to your buckets).
resource "aws_vpc_endpoint" "s3" {
  vpc_id = aws_vpc.main.id  service_name = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"  route_table_ids = aws_route_table.private[*].id
  policy = jsonencode({ Version = "2012-10-17", Statement = [
    { Effect = "Allow", Principal = "*", Action = ["s3:GetObject","s3:PutObject","s3:ListBucket"],
      Resource = ["arn:aws:s3:::my-lake","arn:aws:s3:::my-lake/*"] } ] })
}

# 2. AWS APIs (Secrets, STS) -> INTERFACE endpoints with private DNS (transparent).
resource "aws_vpc_endpoint" "secrets" {
  vpc_id = aws_vpc.main.id  service_name = "com.amazonaws.us-east-1.secretsmanager"
  vpc_endpoint_type = "Interface"  subnet_ids = aws_subnet.private[*].id
  security_group_ids = [aws_security_group.endpoints.id]  private_dns_enabled = true
}

# 3. Snowflake + the partner API -> INTERFACE endpoints to their ENDPOINT SERVICES
#    + a self-managed private hosted zone (private_dns_enabled does NOT apply here).
resource "aws_vpc_endpoint" "snowflake" {
  vpc_id = aws_vpc.main.id  service_name = "com.amazonaws.vpce.us-east-1.vpce-svc-xxxx"
  vpc_endpoint_type = "Interface"  subnet_ids = aws_subnet.private[*].id
  security_group_ids = [aws_security_group.endpoints.id]
}
resource "aws_route53_zone" "sf" { name = "privatelink.snowflakecomputing.com"
  vpc { vpc_id = aws_vpc.main.id } }

# 4. Verify from a PRIVATE instance that EVERY name resolves to a PRIVATE IP.
for h in my-lake.s3.us-east-1.amazonaws.com \
         secretsmanager.us-east-1.amazonaws.com \
         ab12345.us-east-1.privatelink.snowflakecomputing.com; do
  echo "$h -> $(dig +short "$h" | tail -1)"      # all must be 10.0.x.x (private)
done

Step-by-step trace.

Target	Endpoint type	DNS mechanism	Scope
S3 (lake)	gateway (free)	prefix-list route	bucket-scoped policy
DynamoDB	gateway (free)	prefix-list route	table-scoped policy
Secrets/STS	interface	private_dns_enabled	endpoint policy
Snowflake	interface (endpoint svc)	private hosted zone	SG + account URL
Partner API	interface (endpoint svc)	private hosted zone	endpoint policy

After deployment, S3 and DynamoDB are reached through free gateway endpoints scoped to your buckets/tables; AWS APIs are reached through interface endpoints whose private DNS makes the standard hostnames resolve to private IPs with no code change; Snowflake and the partner API are reached through interface endpoints to their endpoint services with self-managed private hosted zones and the privatelink URLs; and a dig sweep from a private instance proves every hostname resolves to a 10.0.x.x address — nothing traverses the public internet, and each path is scoped to least privilege.

Output:

Metric	Internet-exposed	Fully private
S3 / DynamoDB path	NAT + internet	gateway endpoint (free)
AWS API path	NAT + internet	interface endpoint (private DNS)
Warehouse / partner path	public internet	interface endpoint + private DNS
Leak verification	none	dig proves private IPs
Path scoping	IAM only	IAM + endpoint policies
Security-review result	fails	passes

Why this works — concept by concept:

• Right endpoint per service — free gateway endpoints for S3/DynamoDB and interface endpoints (PrivateLink) for everything else matches each service to the cheapest private mechanism it supports, instead of forcing everything through one costly path.
• Private DNS makes it transparent — enabling private DNS (AWS APIs) or a private hosted zone (Snowflake/partner) resolves the normal hostname to the endpoint's private IP, so applications keep their existing connection strings while the packets go private — the switch is invisible to code.
• dig-verified no-leak — resolving every hostname from a private instance and confirming a 10.0.x.x address is the proof that the endpoints are actually used, catching the number-one failure mode where an endpoint exists but DNS still points public.
• Scoped endpoint policies — attaching a least-privilege policy to each endpoint turns a private path into a private and narrow path, so the connectivity that satisfies the security review also blocks exfiltration through the endpoint.
• Cost — free gateway-endpoint bulk plus modest interface-endpoint fees for control-plane calls, versus NAT processing and internet egress on every connection. The eliminated cost is both the recurring NAT/egress charge and the security risk of a public data plane — private by construction, not by policing.

API integration
Topic — api-integration
API integration problems on private service connectivity

Practice →

Design Topic — design Design problems on private warehouse and lake access

Practice →

---

4. Egress costs — where the bill comes from

Ingress is free; egress is priced by the boundary crossed — and the NAT adds its own fee

The mental model in one line: egress costs are the metered charge for data leaving a location, and the price is set entirely by the boundary the byte crosses — same-AZ is effectively free, cross-AZ is metered on both ends, cross-region costs more, and internet or cross-cloud egress is the most expensive and tiered — while ingress is almost always free, which means the whole discipline of controlling data transfer cost is (1) knowing which boundary each byte crosses, (2) removing the accidental crossings (a NAT-routed S3 read, a cross-AZ Kafka fetch, a cross-region query), and (3) shrinking the unavoidable ones by compressing, batching, and colocating — because on a real data platform, transfer routinely rivals or beats compute on the bill. Name the boundary and you have named the cost; move the workload to the cheaper side of it and you have removed the cost.

The pricing ladder (illustrative AWS list prices; always check current rates).

• Same-AZ, private IPs. ~$0.00/GB — the free tier of the ladder, and the reason colocation is the first cost move.
• Cross-AZ, same region. ~$0.01/GB each direction — so a byte crossing an AZ boundary is effectively billed twice, once leaving and once arriving.
• Cross-region, same cloud. ~$0.02/GB — charged on the source region; the price of putting compute and data in different regions.
• Internet / cross-cloud. ~$0.05–$0.09/GB tiered (cheaper as volume grows, but never free) — paid by the source; cross-cloud is just internet egress from the origin cloud.

The NAT gateway data-processing charge.

• The sneaky fee. A NAT gateway charges ~$0.045/GB processed on top of any egress — so private-subnet traffic to the internet through a NAT is billed for both the NAT processing and the egress.
• The S3 trap. Reading a lake bucket from a private subnet without a gateway endpoint routes through the NAT, paying the NAT fee for traffic that a free gateway endpoint would carry at no charge.
• The fix. Gateway endpoints (S3/DynamoDB) and interface endpoints remove eligible traffic from the NAT entirely, so the NAT only carries genuine third-party internet calls.

Where the bill actually comes from on a data platform.

• Kafka cross-AZ replication and fetch. A 3-AZ cluster replicates partitions across AZs (metered) and, by default, consumers fetch from the leader which is often in another AZ (metered) — frequently the single largest transfer line item.
• Cross-region reads. A job in us-west-2 reading a bucket or warehouse in us-east-1 pays cross-region egress on every byte — often an accident of where the compute was launched.
• Chatty services and re-reads. Microservices exchanging data across AZs, or a pipeline re-reading the same uncompressed data repeatedly, multiply transfer for no analytical gain.
• Serving and exports. Sending query results, files, or a dataset out to the internet or another cloud is direct egress — the visible tip; the AZ/region charges are the hidden mass.

How to cut it.

• Colocate compute with data. Same AZ where possible, same region always — the cheapest byte is the one that never crosses a boundary.
• Use endpoints. Gateway/interface endpoints remove NAT and internet charges for cloud-service traffic.
• Compress and use columnar. Parquet + compression can cut transferred bytes by 5–10×; every avoided byte is avoided egress.
• Batch, cache, and ship deltas. Fewer, larger, incremental transfers beat chatty, repeated, full ones; cache what is re-read.

The failure modes senior engineers pre-empt.

• Assuming intra-region is free. Cross-AZ within a region is not free and is billed both ways. Mitigation: AZ-aware placement and clients; measure cross-AZ explicitly.
• Ignoring the NAT processing charge. Bulk cloud-service traffic through a NAT quietly bills per GB. Mitigation: gateway/interface endpoints; keep the NAT for third-party internet only.
• Per-request cross-region access. A hot path that reads another region on every request pays cross-region egress at request volume. Mitigation: replicate/cache in-region; keep compute with the data.

Common interview probes on egress.

• "Is data transfer within a region free?" — no; cross-AZ is metered both ways, only same-AZ is ~free.
• "Where does a Kafka cluster's transfer cost come from?" — cross-AZ replication and cross-AZ consumer fetch.
• "What's the NAT gateway data-processing charge?" — a ~$0.045/GB fee on top of egress; avoid it with endpoints.
• "How do you cut egress?" — colocate, use endpoints, compress/columnar, batch/cache, ship deltas.

Worked example — an egress worked calculation for cross-AZ Kafka

Detailed explanation. Nothing makes egress concrete like pricing a real flow. Take a Kafka topic ingesting 10 TB/day with typical replication and multi-AZ consumers, and compute the cross-AZ transfer bill — then show what AZ-awareness saves. Use illustrative rates.

• The flow. 10 TB/day produced; replication factor 3 across 3 AZs; several consumer groups fetching from leaders.
• The charge. Cross-AZ replication traffic + cross-AZ consumer fetch, at ~$0.01/GB each way.
• The lever. Rack awareness / fetch-from-follower keeps consumer reads in-AZ.

Question. Estimate the monthly cross-AZ transfer cost of a 10 TB/day Kafka topic with RF=3 and cross-AZ consumers, then the cost after AZ-aware consumption.

Input.

Quantity	Value
Daily produce volume	10 TB (10,000 GB)
Replication factor	3 (across 3 AZs)
Consumer groups (cross-AZ fetch)	3
Cross-AZ rate (illustrative)	~$0.01/GB per direction

Code.

# Cross-AZ Kafka cost (illustrative rates; the ORDER of magnitude is the point).

Produce = 10,000 GB/day.

1) REPLICATION across AZs (RF=3 => 2 follower copies, both cross-AZ):
   replicated cross-AZ bytes/day ~= 10,000 GB x 2 = 20,000 GB
   cross-AZ is billed ~ once per crossing => 20,000 GB x $0.01 ~= $200/day

2) CONSUMER FETCH from leaders (3 groups, default = fetch from LEADER, often cross-AZ):
   ~2/3 of fetches cross an AZ boundary (leader in a different AZ than the consumer)
   fetch cross-AZ bytes/day ~= 10,000 GB x 3 groups x (2/3) ~= 20,000 GB
   => 20,000 GB x $0.01 ~= $200/day

Cross-AZ total ~= $400/day ~= $12,000/month  (often > the broker compute!)

AFTER fetch-from-follower / rack awareness (consumers read an IN-AZ replica):
   replication cross-AZ stays ($200/day) but CONSUMER fetch becomes ~in-AZ (~$0)
   Cross-AZ total ~= $200/day ~= $6,000/month   => ~50% cut, no code change to producers

Step-by-step explanation.

1. Replication is unavoidable cross-AZ traffic: with RF=3 spread across three AZs, each produced byte is copied to two followers in other AZs, so ~2× the produce volume crosses AZ boundaries every day — priced at the cross-AZ rate.
2. Consumer fetch is the second, often larger, component: by default a Kafka consumer fetches from the partition leader, which for a random consumer sits in a different AZ roughly two-thirds of the time, so most consumer traffic also crosses an AZ boundary and is metered.
3. Summing the two puts cross-AZ transfer around $400/day — $12k/month — which on many clusters exceeds the cost of the brokers themselves, the exact "the network costs more than the compute" surprise that makes this an interview favourite.
4. Enabling fetch-from-follower (rack awareness) lets consumers read a replica in their own AZ, collapsing the consumer-fetch component to roughly in-AZ (free) while replication cross-AZ remains — cutting the bill by about half with a config change, not an architecture change.
5. The senior insight is that the biggest transfer costs are usually structural (how replicas and consumers are placed relative to AZ boundaries), so the biggest savings come from placement and client configuration, not from moving less data.

Output.

Component	Default (leader fetch)	AZ-aware (follower fetch)
Replication cross-AZ	~$200/day	~$200/day
Consumer fetch cross-AZ	~$200/day	~$0 (in-AZ)
Monthly cross-AZ total	~$12,000	~$6,000
Change required	—	a client/broker config

Rule of thumb. Price your Kafka transfer by its cross-AZ crossings — replication (RF−1 copies) plus consumer fetch — because it often beats broker compute. Turn on fetch-from-follower / rack awareness so consumers read an in-AZ replica; it roughly halves the bill with a config change and no producer changes.

Worked example — NAT processing charge vs an S3 gateway endpoint

Detailed explanation. The most common accidental egress on AWS is lake reads routed through a NAT. Price the difference against a (free) S3 gateway endpoint for a pipeline that reads 50 TB/month from S3. The gap is pure waste. Compute it.

• Through the NAT. Every GB pays the NAT data-processing charge (~$0.045/GB); S3 traffic in-region is otherwise free, so the NAT fee is the whole cost.
• Through the gateway endpoint. Free — no hourly, no per-GB.
• The delta. The entire NAT charge on that 50 TB is avoidable.

Question. Compute the monthly cost of reading 50 TB/month from S3 through a NAT versus through an S3 gateway endpoint, and state the change.

Input.

Path	Per-GB	Monthly volume	Cost
Private subnet → NAT → S3	~$0.045/GB (NAT)	50 TB	compute below
Private subnet → gateway endpoint → S3	$0.00	50 TB	$0
Data transfer S3↔EC2 in-region	$0.00	—	—
Change	add endpoint + route	—	one-time

Code.

# 50 TB/month of in-region S3 reads: NAT path vs gateway-endpoint path.

50 TB = 51,200 GB.

THROUGH THE NAT:
  NAT data processing = 51,200 GB x $0.045/GB ~= $2,304 / month
  (in-region S3<->EC2 transfer itself is $0, so the NAT fee is the ENTIRE cost —
   pure waste, since a free endpoint carries the same traffic.)
  + NAT hourly (~$0.045/hr ~= $32/mo) — small next to the processing fee.

THROUGH THE S3 GATEWAY ENDPOINT:
  data processing = $0
  hourly          = $0     (gateway endpoints are free)
  TOTAL           = $0 / month for the S3 traffic

SAVING ~= $2,304 / month, for a one-time change:
  add aws_vpc_endpoint "s3" (Gateway) + associate the private route table.

Step-by-step explanation.

1. Reading 50 TB/month from S3 in the same region incurs no S3 data-transfer charge by itself — so when that traffic goes through a NAT, the entire bill is the NAT's per-GB processing fee, roughly $2,300/month of pure overhead.
2. The gateway endpoint carries the identical traffic on the AWS backbone for free — no hourly, no per-GB — so the whole NAT charge is avoidable with no change to throughput or behaviour, only to the route.
3. The change is a one-time addition of a gateway endpoint and a route-table association (from section 2); after it, S3's prefix-list route points at the endpoint and the traffic never reaches the NAT.
4. Beyond the dollars, moving 50 TB off the NAT also relieves NAT bandwidth and connection pressure, which often silently throttles other genuine outbound traffic sharing the same NAT — a reliability win on top of the cost win.
5. This is why "is there an S3 gateway endpoint?" is a reflexive first question on any AWS data platform: it is a free, one-time change that frequently removes four figures a month of accidental NAT charges.

Output.

Metric	NAT path	Gateway endpoint
S3 read data-processing cost	~$2,304/mo	$0
NAT bandwidth pressure	high	S3 removed
Path privacy	public S3 endpoint	AWS backbone
Effort to fix	—	one-time endpoint + route

Rule of thumb. In-region S3 traffic through a NAT is pure waste — the NAT processing fee is the whole bill, and a free gateway endpoint carries the same bytes at no charge. Adding the endpoint is a one-time change that routinely removes thousands a month; audit for it first.

Worked example — deciding placement to avoid cross-region egress

Detailed explanation. The third common leak is cross-region access: compute in one region reading data in another, paying cross-region egress on every byte. The fix is placement — put compute in the data's region, or replicate the data once. Weigh the two for a job reading 20 TB/month from a bucket in another region.

• The leak. Compute in us-west-2 reads a us-east-1 bucket: ~$0.02/GB cross-region on every read.
• Option A. Move the compute to us-east-1 — cross-region cost goes to zero.
• Option B. Replicate the data to us-west-2 once — pay the copy once, then read in-region free.

Question. For a job reading the same 20 TB/month cross-region, compare "move compute in-region" versus "replicate the data once," and pick.

Input.

Option	One-time cost	Recurring cost
Keep cross-region reads	$0	20 TB × $0.02/GB × N reads
Move compute to data's region	~$0 (relaunch)	$0 cross-region
Replicate data once, read in-region	1 × copy egress	$0 cross-region (+ storage)

Code.

# Job reads 20 TB/month, currently cross-region. 20 TB = 20,480 GB.

CURRENT (cross-region reads, say the data is read ~4x/month by the pipeline):
  20,480 GB x $0.02/GB x 4 reads ~= $1,638 / month, forever.

OPTION A — MOVE COMPUTE to the data's region (us-east-1):
  cross-region transfer = $0   (reads are now in-region)
  cost = relaunching the job in-region (~one-time, ~$0 ongoing)
  BEST when the compute can move and nothing else needs the data in us-west-2.

OPTION B — REPLICATE the data ONCE into us-west-2, then read locally:
  one-time copy = 20,480 GB x $0.02/GB ~= $410 (once)
  + extra storage in us-west-2 (~$0.023/GB/mo x 20,480 ~= $471/mo)
  recurring cross-region read = $0
  BEST when OTHER consumers in us-west-2 also need the data (amortise the copy+storage).

Decision: if only this job needs it -> A (move compute).
          if many us-west-2 consumers need it -> B (replicate once).

Step-by-step explanation.

1. The status quo pays cross-region egress on every read: 20 TB read four times a month at ~$0.02/GB is ~$1,600/month, recurring forever — a cost that exists only because compute and data are in different regions.
2. Option A moves the compute to the data's region, taking cross-region transfer to zero for essentially no ongoing cost — the correct answer whenever the compute is free to move and no other region needs the data.
3. Option B replicates the data once (~$410 one-time) and adds ongoing storage in the second region (~$471/month), after which local reads are free — this beats the status quo only when the recurring storage is less than the recurring cross-region reads it removes, i.e. when many consumers share the copy.
4. The comparison is the senior habit: cross-region cost is a placement decision, and the choice between moving compute and replicating data turns on how many consumers need the data locally — one job favours moving compute, a shared dataset favours replicating once.
5. The invariant is that paying cross-region egress on every read forever is almost always the worst option; you either eliminate the boundary (move compute) or pay to cross it once (replicate) and then read locally.

Output.

Approach	Cost profile	Best when
Cross-region reads (status quo)	~$1,638/mo forever	never, if avoidable
Move compute in-region	~$0 ongoing	one job / compute can move
Replicate once	~$410 once + storage	many in-region consumers
(rule)	pay to cross once, not per read	—

Rule of thumb. Never pay cross-region egress on every read. Either move the compute into the data's region (best for a single job) or replicate the data once and read it locally (best when many consumers in the other region share it) — cross a region boundary at most once, never per request.

Senior interview question on diagnosing and cutting a data-transfer bill

A senior interviewer might ask: "Finance flags that data transfer is now your platform's second-largest line item, above storage. You have a multi-AZ Kafka cluster, Spark jobs reading a lake, and a warehouse in another region. Diagnose where the transfer cost is coming from, rank the fixes by leverage, and show the specific changes — with rough numbers — that bring it down without changing what the pipelines deliver."

Solution Using boundary decomposition, endpoints, rack awareness, and in-region placement

# 1. DECOMPOSE the bill by boundary (you cannot fix what you cannot attribute).
#    Turn on cost allocation / transfer breakdown; expect roughly:
#      cross-AZ (Kafka replicate + fetch) .... the big one
#      NAT processing (S3 reads via NAT) ..... pure waste
#      cross-region (warehouse in region B) .. per-read tax
#      internet egress (exports) ............. usually smallest

# 2. RANK fixes by leverage (biggest, cheapest first):
#   a. S3 gateway endpoint      -> removes ALL NAT processing on lake reads (free, 1x change)
#   b. Kafka fetch-from-follower-> ~halves cross-AZ (config, no producer change)
#   c. move Spark into data's AZ-> shuffle/consume become in-AZ (~free)
#   d. warehouse: co-locate or replicate once -> kills per-read cross-region
#   e. compress + Parquet on exports -> shrinks the remaining internet egress 5-10x

# 3. The one-time structural fixes as code.
resource "aws_vpc_endpoint" "s3" {                     # (a) free gateway endpoint
  vpc_id = aws_vpc.main.id  service_name = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"  route_table_ids = aws_route_table.private[*].id
}
# (b) Kafka broker: enable rack awareness so consumers can fetch from an in-AZ follower.
#     broker:  broker.rack=<az-id>       replica.selector.class=RackAwareReplicaSelector
#     consumer: client.rack=<same az-id>

Step-by-step trace.

Cost source	Fix	Rough effect
NAT processing on S3 reads	S3 gateway endpoint	→ $0 (free)
Kafka cross-AZ consumer fetch	fetch-from-follower	~50% of cross-AZ gone
Spark cross-AZ shuffle/consume	AZ-colocate executors	shuffle/consume ~free
Cross-region warehouse reads	co-locate / replicate once	per-read tax → one-time
Internet egress on exports	compress + columnar	5–10× fewer bytes
Attribution	boundary decomposition	targets the leverage

After the changes, lake reads leave the NAT via a free S3 gateway endpoint; Kafka consumers fetch from in-AZ followers so the largest cross-AZ component roughly halves; Spark executors run in the same AZ as the brokers and partitions they read, making shuffle and consume traffic intra-AZ; the cross-region warehouse access is resolved by placement or a one-time replica instead of a per-read tax; and remaining exports are compressed columnar files. The bill is now decomposed, attributed, and each boundary is either eliminated or crossed at most once.

Output:

Metric	Before	After
NAT processing on lake reads	~$2k+/mo	$0 (gateway endpoint)
Kafka cross-AZ transfer	~$12k/mo	~$6k/mo (follower fetch)
Spark cross-AZ	unbounded	~free (AZ-colocated)
Cross-region reads	per-read forever	one-time / eliminated
Export egress	raw volume	5–10× smaller
Cost visibility	one opaque number	attributed per boundary

Why this works — concept by concept:

• Boundary decomposition — attributing the transfer bill to cross-AZ, NAT, cross-region, and internet components turns one opaque number into a ranked target list, so effort goes to the boundary that actually dominates rather than the one that is easiest to see.
• Endpoints remove the NAT tax — a free S3 gateway endpoint carries lake traffic on the backbone at no charge, deleting the NAT processing fee that is the purest waste in the bill with a single one-time change.
• Rack awareness collapses cross-AZ fetch — letting consumers fetch from an in-AZ follower removes the largest structural cross-AZ component of a streaming platform through configuration, without touching producers or the data itself.
• Placement beats per-read tax — colocating compute with data (same AZ for shuffle/consume, same region for the warehouse) or replicating once means a boundary is crossed at most one time instead of on every read, converting a recurring tax into zero or a one-off.
• Cost — free endpoint reads, in-AZ streaming, in-region compute, and compressed exports versus NAT fees, cross-AZ fetch, cross-region per-read charges, and raw-volume egress. The eliminated cost is the entire class of accidental boundary crossings — O(bytes on the cheapest boundary) instead of O(bytes × whatever boundary they drifted across).

Optimization
Topic — optimization
Optimization problems on egress and transfer cost

Practice →

Data processing Topic — data-processing Data processing problems on placement and locality

Practice →

---

5. Cross-cloud transfer — moving data between clouds

Egress is charged at the source cloud; minimize what crosses, or cross on a private interconnect

The mental model in one line: cross-cloud transfer — moving data between AWS, GCP, and Azure — is just internet egress from the source cloud (the destination's ingress is almost always free), so it sits on the most expensive rung of the ladder and the entire discipline is (1) minimize what crosses (ship deltas not full copies, compress and use columnar formats, push compute to the data so only results cross), (2) prefer a private path (a dedicated interconnect plus colocation gives a lower per-GB rate and a private link versus the metered public internet), and (3) respect data gravity (put the consumer where the data already is instead of ping-ponging copies) — because a naive nightly full re-copy across clouds is the single most expensive mistake in a multi-cloud data architecture. The cheapest cross-cloud byte is the one you compute away; the next cheapest crosses compressed, once, on a private link.

Why cross-cloud is the priciest boundary.

• The source pays egress. Data leaving AWS for GCP is billed as AWS internet egress (~$0.05–$0.09/GB tiered); the fact that the destination is another cloud does not make it cheaper.
• Ingress is free. The receiving cloud almost never charges to accept data, so the whole cost is on the sending side — which is why who initiates and what direction the bulk flows matters.
• No magic cloud-to-cloud rate on the public path. Absent a dedicated interconnect, cross-cloud is ordinary internet egress; the "cloud-to-cloud" savings only exist on private interconnects/peering.
• It compounds at volume. A 100 TB dataset re-copied monthly is ~$5k–$9k/month of egress before anyone queries it — which is why minimization dominates the design.

Minimize what crosses.

• Ship deltas, not full copies. Transfer only what changed (CDC, incremental partitions, changed files) instead of re-copying the whole dataset — usually a 10–100× reduction.
• Compress and use columnar. Parquet/ORC + compression shrinks bytes 5–10× versus raw JSON/CSV; every avoided byte is avoided egress.
• Push compute to the data. Aggregate, filter, or pre-join in the source cloud and transfer only the (small) result — move the question to the data, not the data to the question.
• Batch, don't stream, across clouds. A steady cross-cloud stream pays egress continuously; batching lets you compress and dedupe before crossing.

Cross on a private path when volume justifies it.

• Dedicated interconnects. Direct Connect (AWS), Cloud Interconnect (GCP), and ExpressRoute (Azure) give a private link with a lower per-GB rate than internet egress and predictable throughput.
• Colocation / a meet-me room. Placing both clouds' interconnects in the same colocation facility gives a short private cross-connect between them — the classic low-cost, high-throughput cross-cloud path.
• The break-even. Interconnects have fixed monthly costs (port + provider), so they pay off above a volume threshold; below it, compressed internet egress is cheaper.

Respect data gravity.

• Move compute, not data. If the analysis can run where the data lives, run it there and export only results — data gravity says the big thing should stay put.
• Avoid ping-pong. Copying data to another cloud to process and copying results back pays egress twice; design one-directional flows.
• Replicate once, cache, and read local. When a second cloud genuinely needs the data, replicate once (or via a delta feed) and serve local reads, rather than reading cross-cloud per query.

The failure modes senior engineers pre-empt.

• Nightly full re-copies. Re-copying an entire dataset that barely changed is the biggest waste. Mitigation: incremental/delta transfer.
• Uncompressed row formats. Shipping raw CSV/JSON across a cloud boundary pays 5–10× the egress of compressed Parquet. Mitigation: compress + columnar before crossing.
• Two-way sync / ping-pong. Processing in the wrong cloud and shipping results back doubles egress. Mitigation: push compute to the data; one-directional flows.

Common interview probes on cross-cloud.

• "Who pays for cross-cloud transfer?" — the source cloud, as internet egress; ingress is free.
• "How do you make a cross-cloud copy cheap?" — deltas + compression + columnar; push compute to the data.
• "When do you use a dedicated interconnect?" — above a volume break-even; for a private path and lower per-GB.
• "What's the worst cross-cloud anti-pattern?" — nightly full re-copies of barely-changed, uncompressed data.

Worked example — full copy vs delta-plus-compressed cross-cloud

Detailed explanation. The canonical cross-cloud decision is how to keep a dataset in a second cloud in sync. The naive answer re-copies everything nightly; the senior answer ships a compressed delta. Price both for a 50 TB dataset with 2% daily change. Illustrative rates.

• Naive. Nightly full copy of 50 TB (raw) → 50 TB egress every night.
• Senior. Ship only the 2% that changed, as compressed Parquet → a tiny fraction crosses.
• The multiplier. Delta (50×) × compression (5×) ≈ 250× less egress.

Question. Compare monthly egress cost for keeping a 50 TB dataset (2% daily change) synced cross-cloud as nightly full copies versus compressed deltas.

Input.

Approach	Bytes crossing/night	Rate	Monthly cost
Full raw copy	50 TB	~$0.08/GB	compute below
Delta only (2%)	1 TB	~$0.08/GB	compute below
Delta + compressed (5×)	~0.2 TB	~$0.08/GB	compute below

Code.

# Keep a 50 TB dataset synced to another cloud. 2% changes daily. ~$0.08/GB egress.
# 50 TB = 51,200 GB.

A) NIGHTLY FULL COPY (raw):
   51,200 GB/night x 30 = 1,536,000 GB/month
   x $0.08/GB ~= $122,880 / month     <- absurd, and it barely changed.

B) DELTA ONLY (2% changed = ~1,024 GB/night, raw):
   1,024 GB/night x 30 = 30,720 GB/month
   x $0.08/GB ~= $2,458 / month       <- 50x cheaper than full copy.

C) DELTA + COMPRESSED COLUMNAR (Parquet, ~5x):
   ~205 GB/night x 30 = 6,144 GB/month
   x $0.08/GB ~= $492 / month         <- ~250x cheaper than the full copy.

Same freshness (nightly), same data available in the destination cloud —
250x cost difference is ENTIRELY the transfer strategy, not the data.

Step-by-step explanation.

1. The nightly full copy crosses the entire 50 TB every night — 1.5 PB/month — costing ~$123k/month, and almost all of it is re-sending bytes that did not change: the definition of waste.
2. Shipping only the 2% that changed drops the nightly volume to ~1 TB, a 50× reduction, cutting the bill to ~$2,460/month for the same nightly freshness and the same resulting dataset in the destination cloud.
3. Compressing the delta into columnar Parquet shrinks it a further ~5×, to ~205 GB/night, landing at ~$492/month — roughly 250× cheaper than the full copy, again with no change to what the destination can query.
4. The point is that the 250× spread is purely the transfer strategy: the freshness, the data, and the destination experience are identical across A, B, and C — only the number of bytes crossing the priced boundary differs.
5. The senior default follows directly: cross-cloud sync is incremental and compressed by construction, and a full re-copy is reserved for the rare initial seed or a correctness rebuild — never the steady-state mechanism.

Output.

Strategy	Monthly egress	Relative cost
Nightly full raw copy	~$122,880	250×
Delta only (raw)	~$2,458	5×
Delta + compressed columnar	~$492	1× (baseline)
Freshness / data available	identical	—

Rule of thumb. Sync across clouds with compressed, incremental deltas, never nightly full copies — the same freshness can cost 250× less purely by sending only what changed in a columnar, compressed format. Reserve full copies for the initial seed or a rebuild.

Worked example — push compute to the data instead of moving raw rows

Detailed explanation. Sometimes you do not need the data in the other cloud at all — you need an answer. Pushing the aggregation to the source cloud and transferring only the small result is the highest-leverage cross-cloud move. Contrast moving raw rows versus moving a result for a cross-cloud report.

• The naive flow. Copy 10 TB of raw events to the other cloud, then aggregate there.
• The senior flow. Aggregate in the source cloud; transfer only the ~1 GB result.
• The lever. Only the answer crosses the boundary, not the raw input.

Question. A daily report needs a per-region aggregate of 10 TB of events that live in AWS, consumed by a GCP dashboard. Compare moving raw events versus moving the aggregate.

Input.

Approach	Bytes crossing	Where aggregation runs
Move raw events	10 TB/day	GCP (after copy)
Move the aggregate	~1 GB/day	AWS (at the source)
Result size	~1 GB	—
Rate	~$0.08/GB	—

Code.

# Daily per-region report over 10 TB of events in AWS; dashboard in GCP.

A) MOVE RAW, AGGREGATE IN GCP:
   cross-cloud egress = 10,000 GB/day x $0.08 ~= $800/day ~= $24,000/month
   (you shipped 10 TB to compute a 1 GB answer.)

B) PUSH COMPUTE TO THE DATA (aggregate in AWS, ship the result):
   -- run in the SOURCE cloud (Athena/Spark over S3):
   SELECT region, event_date, count(*) AS events, sum(amount_cents) AS revenue
   FROM events WHERE event_date = current_date
   GROUP BY region, event_date;          -- output ~1 GB
   cross-cloud egress = ~1 GB/day x $0.08 ~= $0.08/day ~= ~$2.40/month

Same dashboard, same numbers. ~$24,000/mo -> ~$2.40/mo by moving the QUESTION
to the data instead of the data to the question.

Step-by-step explanation.

1. The naive flow copies all 10 TB of raw events to GCP every day just to compute a small aggregate there — paying ~$800/day of egress to move data whose 99.99% is discarded by the aggregation.
2. The senior flow runs the aggregation in AWS, where the data already lives (Athena, Spark, or the warehouse over S3), and transfers only the ~1 GB result that the dashboard actually needs.
3. Because only the answer crosses the boundary, cross-cloud egress collapses from ~$24k/month to a few dollars — a ~10,000× reduction — with the dashboard receiving identical numbers.
4. This is the "data gravity" principle made concrete: the big, heavy thing (raw events) stays put, and the small, light thing (the query and its result) moves — the opposite of the naive instinct to co-locate the data with the consumer.
5. The generalization is that most cross-cloud consumption is really result consumption: dashboards, reports, and features usually need aggregates or filtered slices, so pushing the filter/aggregate/join to the source and shipping only the output is the default, not an optimization.

Output.

Approach	Cross-cloud bytes/day	Monthly egress
Move raw, aggregate remotely	10 TB	~$24,000
Push compute, move result	~1 GB	~$2.40
Dashboard output	identical	—
Reduction	~10,000×	—

Rule of thumb. When another cloud needs an answer, compute the answer where the data lives and ship only the result — pushing the aggregate/filter/join to the source turns terabytes of cross-cloud egress into gigabytes. Move the question to the data, not the data to the question.

Worked example — dedicated interconnect vs internet egress break-even

Detailed explanation. When cross-cloud volume is genuinely large, a dedicated interconnect (with colocation) beats internet egress on per-GB and gives a private path — but it has fixed monthly costs, so there is a break-even. Compute it for a steady cross-cloud flow.

• Internet egress. ~$0.08/GB, no fixed cost, but scales linearly with volume.
• Interconnect. A lower per-GB (say ~$0.02/GB) plus fixed port/colo costs (say ~$2,000/month).
• The break-even. Volume where fixed + lower per-GB undercuts internet egress.

Question. At what monthly cross-cloud volume does a dedicated interconnect (~$2,000/month fixed + ~$0.02/GB) beat internet egress (~$0.08/GB)?

Input.

Path	Fixed/month	Per-GB	Private?
Internet egress	$0	~$0.08	no
Dedicated interconnect	~$2,000	~$0.02	yes
Difference per GB	—	~$0.06 saved	—
Break-even	fixed ÷ per-GB saving	—	—

Code.

# Interconnect vs internet egress break-even.
# internet:      cost = 0.08 * GB
# interconnect:  cost = 2000 + 0.02 * GB

Set equal:  0.08*GB = 2000 + 0.02*GB
            0.06*GB = 2000
            GB      = 33,333 GB  ~= 33 TB / month  (the break-even)

Below ~33 TB/month : internet egress is cheaper (no fixed cost to amortise).
Above ~33 TB/month : the interconnect wins, and the gap widens with volume:

   100 TB/month:
     internet     = 0.08 * 102,400 ~= $8,192
     interconnect = 2000 + 0.02*102,400 ~= $4,048    -> ~50% cheaper AND private

Bonus (not in the math): the interconnect is a PRIVATE path (no public internet)
and gives predictable throughput — often required for security regardless of price.

Step-by-step explanation.

1. Internet egress is pure variable cost (~$0.08/GB) with no fixed component, so it is cheapest at low volume where there is no fixed cost to justify.
2. The interconnect trades a fixed monthly cost (~$2,000 for port + colocation) for a much lower per-GB rate (~$0.02), so it only makes sense once enough volume crosses to amortise the fixed cost.
3. Setting the two cost equations equal gives a break-even of ~33 TB/month: below it, internet egress wins; above it, the interconnect wins, and the advantage grows with volume because every additional GB saves the ~$0.06 per-GB difference.
4. At 100 TB/month the interconnect is roughly half the cost and provides a private path — so at high volume you get both a cheaper and a more secure link, which is why large cross-cloud flows almost always move onto interconnects.
5. The senior nuance is that the interconnect's private path and predictable throughput are often required by security or SLA regardless of the break-even — so the cost math sets the floor, but compliance can justify it even below the crossover.

Output.

Monthly volume	Internet egress	Interconnect	Winner
10 TB	~$819	~$2,205	internet
33 TB	~$2,700	~$2,700	break-even
100 TB	~$8,192	~$4,048	interconnect
any (security-required)	public path	private path	interconnect

Rule of thumb. Compute the interconnect break-even as fixed cost ÷ per-GB saving (~33 TB/month in this example): below it, compressed internet egress is cheaper; above it, a dedicated interconnect is both cheaper and private. High-volume or security-bound cross-cloud flows belong on an interconnect.

Senior interview question on designing a cross-cloud data sync

A senior interviewer might ask: "You must make a 50 TB dataset that lives and grows in AWS available to analytics teams in GCP, refreshed daily, at minimum cost. Design the sync: what actually crosses the boundary, in what format, how you keep it incremental, whether you push compute to the source, and when you'd move it onto a dedicated interconnect — with the rough economics that justify each choice."

Solution Using delta extraction, compression, source-side compute, and an interconnect above break-even

# 1. What crosses: DELTAS ONLY, compressed columnar — never a full nightly copy.
#    AWS side: extract the day's changed partitions as Parquet+zstd.
#    50 TB dataset, ~2% daily change, ~5x compression -> ~205 GB/night crosses (not 50 TB).

-- 2. PUSH COMPUTE to the source for consumers that need aggregates, not raw rows.
--    Run in AWS (Athena/Spark over S3); ship the small result to GCP.
CREATE TABLE daily_region_rollup AS
SELECT region, event_date, count(*) AS events, sum(amount_cents) AS revenue
FROM events WHERE event_date = current_date GROUP BY region, event_date;   -- ~1 GB out

# 3. TRANSPORT: pick the path by volume (break-even ~33 TB/month here).
#    steady-state delta ~205 GB/night ~= 6 TB/month  -> BELOW break-even
#       => compressed internet egress (Storage Transfer Service / rclone), ~$0.08/GB
#    if a backfill or growth pushes sustained volume > ~33 TB/month
#       => move onto Direct Connect + Cloud Interconnect via a colo (private, ~$0.02/GB)

# 4. DATA GRAVITY: one-directional flow; no ping-pong.
#    AWS (source of truth) --deltas--> GCP (read replica for analytics).
#    Analytics results that must return to AWS are aggregates (small), not raw re-exports.

Step-by-step trace.

Decision	Naive	This design
What crosses	full 50 TB nightly	~205 GB delta (compressed)
Format	raw CSV/JSON	Parquet + zstd
Aggregate consumers	copy raw, aggregate in GCP	aggregate in AWS, ship ~1 GB
Transport	internet egress always	internet now, interconnect above break-even
Direction	two-way ping-pong	one-way AWS→GCP
Steady-state egress	~1.5 PB/month	~6 TB/month

After deployment, only the day's changed partitions cross the boundary, as compressed Parquet, so steady-state cross-cloud volume is ~6 TB/month instead of ~1.5 PB; consumers that need aggregates get them computed in AWS with only ~1 GB results crossing; the transport is compressed internet egress while volume is below the ~33 TB/month break-even, with a documented trigger to move onto a Direct Connect/Cloud Interconnect path if a backfill or growth pushes sustained volume above it; and the flow is strictly one-directional, so nothing is paid twice.

Output:

Metric	Naive cross-cloud sync	Designed sync
Bytes crossing/month	~1.5 PB	~6 TB
Format	raw	compressed columnar
Aggregate egress	10 TB raw	~1 GB result
Monthly egress cost	~$120k+	~$500–$2.5k
Path	public internet	internet now, interconnect above break-even
Direction	two-way	one-directional

Why this works — concept by concept:

• Deltas, not full copies — transferring only the changed partitions turns a 50 TB nightly flow into a ~1 TB one before compression, because re-sending unchanged data is the largest and most avoidable cross-cloud waste.
• Compressed columnar — Parquet plus a strong codec shrinks the delta another ~5×, and since egress is priced per byte, every compression ratio is a direct, linear cost reduction on the priciest boundary.
• Source-side compute — aggregating where the data lives and shipping only results exploits data gravity, so consumers that need answers cost gigabytes instead of terabytes and the raw data never leaves its cloud.
• Break-even-driven transport — staying on compressed internet egress below the interconnect break-even and switching above it means the transport is always the cheaper of the two, with the private interconnect reserved for the volumes (or security requirements) that justify its fixed cost.
• Cost — a compressed one-directional delta feed plus source-side aggregation versus full raw nightly re-copies and remote aggregation. The eliminated cost is roughly two orders of magnitude of egress — O(changed, compressed bytes) instead of O(entire dataset × nights) — for identical destination freshness and query results.

Design
Topic — design
Design problems on cross-cloud and multi-region architecture

Practice →

Optimization
Topic — optimization
Optimization problems on cross-cloud transfer cost

Practice →

---

Cheat sheet — networking for data engineers

• The two problems. Networking for data engineers is always reachability (can the pipeline connect, given subnets, routes, firewalls, DNS, endpoints?) and data-transfer cost (which boundary does each byte cross?). Master both; a pipeline's uptime and half its bill are decided here.
• The egress ladder. same-AZ ~free → cross-AZ ~$0.01/GB each way → cross-region ~$0.02/GB → internet/cross-cloud ~$0.05–0.09/GB tiered. Ingress is free; the source pays egress. Name the boundary and you have named the cost. A NAT gateway adds ~$0.045/GB on top.
• What makes a subnet public/private. Purely its route table: 0.0.0.0/0 → internet gateway is public; 0.0.0.0/0 → NAT (or nowhere) is private. Put NATs in public subnets, data planes in private subnets, one NAT per AZ.
• Security group vs NACL. SG = stateful, instance-level, allow-only, best referenced by source security group (scale-proof). NACL = stateless, subnet-level, ordered allow+deny — needs the ephemeral return range opened explicitly. Use SGs as the real firewall; NACLs as a coarse backstop.
• Endpoints. Gateway endpoints (S3, DynamoDB) are free, route-based, and bypass the NAT — add them to every private route table. Interface endpoints (PrivateLink) are ENIs with private IPs for everything else — priced, but private. Scope both with endpoint policies.
• PrivateLink recipe. Interface endpoint in every caller AZ → enable private DNS (AWS APIs) or a private hosted zone (Snowflake/partner) → connect using the privatelink URL → dig from a private host to prove the name resolves to a 10.0.x.x IP. The DNS override is the step everyone forgets; without it the endpoint is a no-op that still bills you.
• Snowflake PrivateLink. SYSTEM$GET_PRIVATELINK_CONFIG → interface endpoint to the returned endpoint-service name → self-managed private hosted zone for privatelink.snowflakecomputing.com → connect to the privatelink-account-url (not the normal one).
• Kafka cross-AZ is the hidden bill. Cost = replication (RF−1 cross-AZ copies) + consumer fetch (default = from the leader, often cross-AZ). Turn on fetch-from-follower / rack awareness (broker.rack, client.rack, RackAwareReplicaSelector) so consumers read an in-AZ replica — roughly halves cross-AZ with a config change.
• Egress-avoidance checklist. (1) S3 gateway endpoint (kills NAT tax on lake reads). (2) Colocate compute with data — same AZ for shuffle/consume, same region always. (3) Compress + columnar (Parquet) — 5–10× fewer bytes. (4) Ship deltas, not full copies. (5) Cross a region/cloud boundary at most once (replicate/cache), never per read.
• Cross-cloud. Source pays internet egress; ingress is free. Minimize what crosses (deltas + compression + push compute to the data → ship results), prefer a private path (Direct Connect / Cloud Interconnect / ExpressRoute + colocation) above the volume break-even (fixed ÷ per-GB saving), and keep flows one-directional (no ping-pong). Never nightly full re-copies.
• Connectivity debugging order. Security group → route table → DNS → endpoint, changing one thing at a time. A timeout points at a firewall or route. Never "fix" it by opening the SG to 0.0.0.0/0 — that hides the real cause and opens a breach.
• Data-transfer framing. Data transfer routinely rivals compute on a real platform's bill. Decompose it by boundary (cross-AZ / NAT / cross-region / internet), attribute each, and either eliminate the boundary or cross it once — it is a designed line item, not a surprise.

Frequently asked questions

What networking does a data engineer actually need to know?

Enough to answer two questions without waiting for the platform team: can this pipeline reach its source and sink, and what does moving this data cost. Concretely that means VPC fundamentals (subnets, route tables, the difference an internet gateway versus a NAT gateway makes, security groups versus NACLs, and VPC endpoints), private connectivity (PrivateLink and gateway/interface endpoints so the warehouse, lake, and streams are reachable off the public internet), and the economics of data transfer (which boundary — AZ, region, cloud, internet — each byte crosses, and therefore what it costs). You do not need to configure BGP or design a global backbone, but you do need to debug a hung connection in a fixed order (security group → route → DNS → endpoint) and to look at an architecture and predict where the transfer bill will come from. That working knowledge is what separates a data engineer who owns their pipelines from one who is permanently blocked on someone else.

What is the difference between a VPC endpoint and PrivateLink?

They overlap, which is why the terms get muddled. A VPC endpoint is the general concept of reaching a service privately from your VPC, and it comes in two kinds. A gateway endpoint exists only for S3 and DynamoDB: it is a route-table entry that sends that service's traffic onto the AWS backbone, it is free, and it uses no network interface. An interface endpoint — which is what PrivateLink refers to — provisions an elastic network interface with a private IP inside your subnet, and is how you privately reach most AWS APIs, managed data services like Snowflake, and partner services; it carries an hourly and per-GB charge. So "PrivateLink" is specifically the interface-endpoint technology (a private ENI you route to), while "gateway endpoint" is the free, route-based mechanism for the two services that support it. The practical rule: gateway endpoints for S3/DynamoDB (free, add them everywhere), interface endpoints/PrivateLink for everything else that must be private.

Why is my cloud data-transfer bill so high?

Almost always because bytes are crossing more expensive boundaries than you realise. The three usual culprits: cross-AZ traffic (which is metered in both directions, so a multi-AZ Kafka cluster replicating and fetching across AZs can spend more on transfer than on brokers), NAT gateway data-processing charges (a ~$0.045/GB fee on every byte a private subnet sends through a NAT — including S3 reads that a free gateway endpoint would carry at no charge), and cross-region or cross-cloud access (paying egress on every read because compute and data ended up in different regions or clouds). The fix is to decompose the bill by boundary rather than treating "data transfer" as one opaque number: add S3/DynamoDB gateway endpoints to kill the NAT tax, turn on Kafka rack awareness so consumers fetch from an in-AZ replica, colocate compute with data so shuffle and reads stay intra-AZ and in-region, and compress plus ship deltas for anything that must cross. Each boundary has a specific, known fix.

How do I connect to Snowflake or Kafka without using the public internet?

Use private connectivity so the traffic stays on the cloud backbone. For Snowflake on AWS: run SYSTEM$GET_PRIVATELINK_CONFIG to get your account's endpoint-service name and privatelink URL, create an interface VPC endpoint to that endpoint service, then create a private hosted zone for privatelink.snowflakecomputing.com so the name resolves to the endpoint's private IP, and connect using the privatelink-account-url rather than the normal one. For Kafka/MSK: run the cluster in your VPC with brokers on private IPs, and reach it from other VPCs via peering, a transit gateway, or MSK's PrivateLink/multi-VPC connectivity — never a public bootstrap. The step people miss in both cases is DNS: the endpoint can exist while the hostname still resolves to a public address, silently sending your "private" traffic through the NAT. Always verify from a private instance with dig that the service name resolves to a 10.0.x.x private IP; that is the proof the private path is actually being used.

Is cross-AZ traffic really not free?

No — and assuming it is free is one of the most expensive mistakes in cloud data engineering. Only traffic that stays within a single availability zone over private IPs is effectively free. The moment a byte crosses an AZ boundary — even between two subnets in the same region and the same VPC — it is metered, and it is metered on both the sending and receiving side, so a cross-AZ byte is billed roughly twice. This is exactly why a three-AZ Kafka cluster is such a cost trap: it replicates every partition to followers in other AZs (metered), and by default consumers fetch from the leader, which is usually in a different AZ than the consumer (metered again). The defences are placement and configuration: colocate the chattiest producers and consumers in the same AZ, and enable fetch-from-follower / rack awareness so Kafka consumers read an in-AZ replica. Treating cross-AZ as free is how a transfer bill quietly overtakes compute.

How do I move data between AWS, GCP, and Azure cheaply?

Minimize what crosses, compress it, and prefer a private path. Cross-cloud transfer is billed as internet egress from the source cloud (the destination's ingress is free), so it sits on the most expensive rung of the ladder and the whole game is reducing bytes. First, never do nightly full re-copies — ship only the incremental delta (the changed partitions/rows), which is usually a 10–100× reduction. Second, transfer compressed columnar formats (Parquet/ORC), for another 5–10× off. Third, push compute to the data: if the other cloud needs an aggregate or a filtered slice, compute it where the data lives and transfer only the small result rather than the raw input — often a 1,000×+ saving. Fourth, keep flows one-directional to avoid paying egress twice on a ping-pong. Finally, once sustained volume clears the break-even (fixed interconnect cost ÷ per-GB saving), move onto a dedicated interconnect (Direct Connect / Cloud Interconnect / ExpressRoute) with colocation, which is both cheaper per GB and a private path. The cheapest cross-cloud byte is the one you compute away.

Practice on PipeCode

• Sharpen the architecture axis on the system design practice library → for the VPC-layout, private-connectivity, and cross-cloud-topology trade-offs a data-platform networking design has to get right.
• Pressure-test the cost side on the optimization practice library → for the cross-AZ, NAT, cross-region, and egress problems where naming the boundary is the whole answer.
• Rehearse the movement patterns on the data processing practice library → for the placement, locality, delta, and compression decisions that keep transfer off the critical path of a bill.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the VPC, PrivateLink, egress-ladder, and cross-cloud patterns against real graded inputs — connectivity, private access, and data-transfer economics.

Lock in networking-for-data-engineers muscle memory

Docs explain VPCs and PrivateLink. PipeCode drills explain the decision — when a hung pipeline is a `security group` and not the code, when a gateway endpoint deletes a NAT bill, when `cross-AZ` fetch is quietly bigger than your brokers, and when a cross-cloud copy should be a compressed delta instead of a full re-sync. Pipecode.ai is Leetcode for Data Engineering — networking and cost practice tuned for the production trade-offs senior data engineers actually face.

Practice system design problems →
Practice optimization problems →