DEV Community

Cover image for What gcloud start-iap-tunnel is doing while your SSH session hangs
Edgar
Edgar

Posted on Originally published at Medium

What gcloud start-iap-tunnel is doing while your SSH session hangs

#googlecloud #devops #networking #ssh

Originally published in Google Cloud - Community.

I switched Wi-Fi off and on while testing the Mac client I'm building, and an IAP tunnel to a healthy VM took almost three minutes to come back.

I manage VMs in more than ten Google Cloud organizations from a Mac, and for a long time that meant playing gcloud like a piano: an alias per project, a couple of bash scripts to switch accounts, and an export CLOUDSDK_BILLING_QUOTA_PROJECT=... because the quota project kept pointing at some other organization. The scripts didn't always switch everything, so now and then I ran a perfectly good command in the wrong organization. And when I closed the laptop or moved to another network, the SSH session froze and I never knew whether to wait or kill the tunnel.

Google doesn't publish the protocol behind IAP TCP forwarding. The closest thing to a spec is the source of the two clients that use it, gcloud (Python) and IAP Desktop (C#).

One WebSocket per connection 

gcloud compute start-iap-tunnel example-vm 22 --local-host-port=localhost:2222 --zone=europe-west1-b
Enter fullscreen mode Exit fullscreen mode

This only opens port 2222 on your Mac. When ssh connects to it, gcloud opens a WebSocket to tunnel.cloudproxy.app, and IAP opens a plain TCP connection to port 22 of the VM from an address in 35.235.240.0/20, so the VM never sees your IP. Open three terminals through that one port and you get three WebSockets and three connections to the VM.

Your Google access token rides along as an ordinary bearer header. If it's the wrong token, or the account lacks roles/iap.tunnelResourceAccessor, the WebSocket still opens and only then closes with code 4033, after ssh has already connected to port 2222.

Inside the WebSocket

Every message is a binary frame with a two-byte type:

1   CONNECT_SUCCESS_SID     session ID, kept for reconnects
2   RECONNECT_SUCCESS_ACK   bytes the server had when the old socket died
4   DATA                    up to 16,384 bytes of your stream
7   ACK                     running total of bytes received
10  LONG_CLOSE              close code and reason
Enter fullscreen mode Exit fullscreen mode

The sender keeps every DATA frame until the other side's ACK total passes it. gcloud sends a standalone ACK once it owes 32 KiB, IAP Desktop once it owes 1 MiB. Up to that much can sit queued for a resend.

When the network drops

If the WebSocket drops without a close frame, the client reconnects to the same session. It calls /v4/reconnect with the session ID and the number of bytes it has received. The server replies with its own count, the client resends what's missing, and the TCP connection to the VM carries on.

gcloud retries for up to fifteen minutes, starting 1.5 seconds apart and backing off to 20 seconds, and gives each attempt up to a minute to open. IAP Desktop stops after two reconnects.

In my Wi-Fi test the relay refused one reconnect with 4077, "Failed to takeover the existing session", then left the next two open for a minute each without any reply. A client can recover sooner by giving up on the reconnect and opening a fresh session, but that costs the SSH session: a new relay session is a new TCP connection to port 22. For a tunnel that carries a database forward, I'd rather drop it and reopen in seconds than stare at a frozen terminal for three minutes. An interactive shell is the one case where I'd still wait. My client does the same: a bastion forward drops a stalled reconnect and opens a fresh session, an SSH tab keeps retrying and redials when macOS says the network is back.

What 4033, 4003 and 4047 mean

  • 4033: IAP won't let you in. With ten organizations on one laptop, for me this was almost always a correct command under the wrong account, so I run gcloud config list account before I open the IAM page.
  • 4003: IAP reached your network but not the port. Usually a missing firewall rule for 35.235.240.0/20, or nothing listens on the VM.
  • 4047: wrong instance, zone or project.
  • 1006 and most others: a dropped connection. Reconnect.

I build Ostgate, a paid macOS app that talks to this relay directly for SSH, SFTP, RDP and tunnels without gcloud. It isn't affiliated with Google: https://ostgate.app

Top comments (0)