DEV Community

h6o
h6o

Posted on

When You Use a PGA Subnet on Cloud Run, Traffic to Google APIs Is Treated as Internal

#gcp #cloudrun #networking #vpc

At work, I tried to put a source-IP restriction on a certain Google API key and ran into a phenomenon where the setting just wouldn't take effect no matter what I did. Tracing the cause led me to Cloud Run's network path — specifically, the behavior of Private Google Access (PGA).

Since other people are likely to hit the same thing, I'm writing this down as a memo for myself. I hope it helps anyone in a similar situation.

What happened

What I wanted to do was put a source-IP restriction on a Google API key, to limit the damage in case of a leak. I figured "I'll just allow-list Cloud Run's egress IP," but the setting wouldn't block anything. Cloud NAT's static IP was on the allow list, and yet for some reason it didn't take effect.

Cutting to the conclusion: on a subnet with PGA enabled, traffic destined for Google APIs was not going through Cloud NAT — that was the cause.

What is Private Google Access (PGA)?

PGA (Private Google Access) is a mechanism that lets VMs and services that don't have an external IP reach Google APIs like *.googleapis.com directly through an internal path within the VPC. The benefit is that you can reach Google services without going out to the internet, and it's enabled per subnet.

In Terraform it's enabled with a single line on the subnet.

resource "google_compute_subnetwork" "egress" {
  name                     = "sb-egress-example"
  private_ip_google_access = true   # ← this is PGA
  # ...
}
Enter fullscreen mode Exit fullscreen mode

And Cloud Run attaches to this subnet via Direct VPC Egress.

metadata:
  annotations:
    run.googleapis.com/network-interfaces: '[{"network":"...","subnetwork":".../sb-egress-example"}]'
    run.googleapis.com/vpc-access-egress: all-traffic
Enter fullscreen mode Exit fullscreen mode

vpc-access-egress: all-traffic is the setting that routes all egress through the VPC. So far this looks like a very common configuration.

Traffic to Google APIs does not go through NAT

The important point is the path that Google API-bound traffic takes on a PGA-enabled subnet. As a diagram, it branches like this.

[When PGA is enabled]

Cloud Run ──┬─ *.googleapis.com bound ──▶ goes directly to Google via the internal network (PGA)
            │                              * does not go through Cloud NAT
            │                              * source is an internal IP
            │
            └─ Other internet-bound traffic ──▶ Cloud NAT ──▶ NAT's static external IP
Enter fullscreen mode Exit fullscreen mode

In other words, this is what was happening:

  • General internet-bound traffic goes through Cloud NAT, so the source becomes NAT's static external IP
  • However, *.googleapis.com-bound traffic takes the PGA internal path, so it does not go through NAT
  • As a result, the source as seen from the Google API side is not the NAT external IP but an internal IP

API-key IP restrictions are something that, by their nature, only accept public IPs. So for PGA-routed requests coming in with an internal IP, there was simply no way for an IP restriction to apply. "Can't pin it to an external IP" — this is what that meant.

This behavior is also explicitly stated in the official Cloud NAT documentation.

Note: Traffic sent to Google APIs and services are routed through Private Google Access even if the VM instance initiating the connections uses Public NAT. For more information, see Private Google Access interaction.

― Cloud NAT overview

So it's clearly written as the specification that "even if you're using Public NAT, traffic to Google APIs alone flows via PGA."

For reference, the Cloud NAT side is configured to allocate a static external IP.

resource "google_compute_router_nat" "nat" {
  name                                = "nat-example"
  nat_ip_allocate_option              = "MANUAL_ONLY"   # manually allocate a static IP
  nat_ips                             = [google_compute_address.nat.self_link]
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}
Enter fullscreen mode Exit fullscreen mode

If traffic egresses via NAT, the source becomes this static IP — but the pitfall is that as long as PGA is enabled, Google API-bound traffic alone does not go through this NAT.

Countermeasure: disable PGA if you want IP restrictions to take effect

If you want to pin the source IP toward Google APIs so that API-key IP restrictions take effect, disable PGA on that subnet.

resource "google_compute_subnetwork" "egress" {
  private_ip_google_access = false   # disable PGA
}
Enter fullscreen mode Exit fullscreen mode

With PGA disabled, traffic to Google APIs also goes through Cloud NAT and out to Google from the internet side. The source becomes NAT's static external IP, so finally — in principle — the API-key IP restriction starts working.

You can confirm that the path has switched to NAT by checking Cloud NAT's flow logs for whether *.googleapis.com-bound entries (Google's ASN 15169) appear. When PGA is enabled, Google-bound traffic does not appear here (because it doesn't go through NAT).

Which one to choose

Summarized, the differences are as follows.

Aspect PGA enabled PGA disabled (via NAT)
Reach to Google APIs Directly via internal nw Via the internet (NAT)
Source IP Internal IP (not static) NAT's static external IP
API-key IP restriction Does not work Works
Exposure to the internet Low Egresses via NAT

Looking purely from a security angle, PGA — which "doesn't go to the internet" — looks preferable. But if you want to apply a separate guard like API-key IP restriction, you may have to deliberately choose to go via NAT. That was the lesson this time.

Wrapping up

When you can't pin the source IP for Google API-bound traffic on Cloud Run, a good first thing to suspect is whether PGA is enabled or disabled. I hope this helps anyone stuck in the same place.

Top comments (0)