RDP to a Windows VM over IAP: serial port 4, NLA and a certificate nobody checks

Originally published in Google Cloud - Community.

Google's page on connecting to Windows VMs has a section called "Verify the RDP certificate". You read serial port 1 from the VM's first boot, find a line like the one below, and compare the thumbprint with the certificate your RDP client is shown.

GCEInstanceSetup: RDP certificate details: Subject: CN=rdp-test, Thumbprint: <40 hex characters>

I tried it on a Windows Server 2025 VM in an organization I work with, a month after it was created. Serial port 1 held 27,597 bytes, all from the last boot, and not one GCEInstanceSetup line. The other place the thumbprint can live, the guest attribute hostkeys/rdp, answered HTTP 412, "Constraint constraints/compute.disableGuestAttributesAccess violated". The VM presented its certificate over the tunnel without trouble (TLS 1.3, NLA), and neither the serial log nor the guest attributes gave me anything to compare it to. The serial API returns only the most recent 1 MB per port, and only from a running VM.

RDP over IAP is two jobs with different permissions

The data goes through a tunnel, and the password comes back by a route that never touches it:

gcloud compute start-iap-tunnel win-01 3389 \
    --local-host-port=localhost:13389 \
    --zone=europe-west1-b --project=acme-prod

gcloud compute reset-windows-password win-01 \
    --user=dana --zone=europe-west1-b --project=acme-prod

The RDP client then connects to localhost:13389. The tunnel needs an ingress rule from 35.235.240.0/20 to tcp:3389, and roles/iap.tunnelResourceAccessor for whoever connects (without the role it closes with 4033). The password command needs something else. Google's docs call it "write access to the Compute Engine project", which comes down to compute.instances.setMetadata and compute.instances.getSerialPortOutput, and IAP-secured Tunnel User has neither. Someone with only that role can open the tunnel and has no password to sign in with, so the reset has to come from whoever holds Compute Instance Admin on the VM.

How the Windows password reaches you

Google documents the steps for scripting it yourself, and IAP Desktop and my own client follow them:

1. The client makes a throwaway RSA-2048 key pair and keeps the private half in memory.
2. It adds an entry to the metadata key windows-keys with userName, email, the public key's modulus and exponent, and expireOn about five minutes ahead. It reads the current metadata first and writes it all back with the new entry, because the API's metadata update replaces the whole set.
3. The guest agent on the VM sees the entry, creates the account or resets its password, encrypts the password with that public key (RSA-OAEP) and prints a JSON line with encryptedPassword and the same modulus to serial port 4.
4. The client polls serial port 4, takes the line whose modulus matches its key, and decrypts.

Google's Go sample decrypts with OAEP over SHA-1. The entry can also carry hashFunction: "sha256", which the agent honours; my client asks for SHA-256 and falls back to SHA-1 for older agents.

Every reset replaces the previous password, so if a colleague ran reset-windows-password after you, your copy is dead and nothing tells you. The agent also needs a moment after it prints the line, and signing in straight away can fail once. My client polls port 4 thirty times, two seconds apart, before it gives up.

In an October test run my own client did that to me. Its Set Windows Password item reset admin whatever Windows user the VM was set to, and each run quietly killed the password I'd saved for that VM, so the next automatic sign-in failed until I saved it again.

What NLA does inside the IAP tunnel

Your RDP client connects to 127.0.0.1. IAP checks who you are and routes the stream to the VM you named, but it gives the client no proof that the far end is that VM. The name the client used, localhost, never matches the certificate subject, and Windows' RDP certificate is self-signed, so there's no CA to trust either. IAP Desktop connects without server authentication and so does my client. FreeRDP says so in its log: [DANGER] Certificate not checked.

CredSSP, the protocol behind NLA, is what ties your sign-in to the connection. During the exchange the client encrypts the server's TLS public key (from CredSSP version 5, a hash of it) into the authentication. A box in the middle that terminates TLS with its own key can't relay your sign-in to the real VM, because the key inside won't match.

A fake endpoint can still collect the NTLM response and crack it offline, and only a certificate check would catch that. A VM you sign in to also receives your delegated credentials, which Restricted Admin mode avoids by keeping the password off the VM.

If NLA is off and the server answers with TLS-only or standard RDP security, the password goes to the unchecked endpoint inside the Client Info PDU. From your machine to Google it still rides in the relay's TLS WebSocket, and from IAP to the VM it crosses Google's network to the internal address, so an attacker would need a position on that last stretch or an impostor VM with the same name. I keep NLA on and turn it off only for a VM I trust.

My client asks only for CredSSP (HYBRID_EX or HYBRID) and refuses a server that picks anything else, with no retry; a real Windows VM has always picked HYBRID when it was offered, and the log line after connect reads rdp security negotiated=HYBRID_EX nla=on. With NLA off on the client and required on the VM, FreeRDP stops with FREERDP_ERROR_CONNECT_HYBRID_REQUIRED_BY_SERVER.

With Restricted Admin mode on, my test Windows Server let NLA succeed, started the session, then showed "Account restrictions are preventing this user from signing in" and logged the session off. There was no connect error at all. If you see that message right after the desktop appears, turn Restricted Admin off and connect again.

Checking the RDP certificate thumbprint anyway

If you want Google's anchor, read it on the first boot, before anything restarts the VM:

gcloud compute instances get-serial-port-output win-01 --port=1 \
    --zone=europe-west1-b --project=acme-prod | grep "RDP certificate"

Google's instance_setup.ps1 writes that line once, from the first-boot setup script, and then deletes the script. The docs warn that the Thumbprint field "might be empty", and the script leaves it empty when it finds no certificate for CN=<hostname>. The Subject in the line is read from the WinRM certificate instead of the RDP one (a bug in the script), so compare only the thumbprint.

The same script puts the thumbprint into the guest attribute hostkeys/rdp too, which doesn't roll out of a buffer:

gcloud compute instances get-guest-attributes win-01 --query-path=hostkeys/ \
    --zone=europe-west1-b --project=acme-prod

It only writes it when the VM has enable-guest-attributes=TRUE in its metadata at first boot, so that has to be set at creation, and an org policy can block reading it, as it did above. I've seen this in the script source and not yet on a live VM, so test it on one VM before you rely on it. Reading either value needs compute.instances.getSerialPortOutput or compute.instances.getGuestAttributes, which Viewer, Compute Viewer and Compute Instance Admin have and IAP-secured Tunnel User doesn't. A mismatch can mean an attacker, or a certificate Windows has replaced since the first boot, and nothing tells you which.

The docs also give a way that doesn't depend on boot history: run PowerShell from the Special Administrative Console and read the certificate Windows holds right now.

Get-ChildItem 'Cert:\LocalMachine\Remote Desktop\' |
    Where-Object { $_.Subject -like "CN=$env:COMPUTERNAME*" -and $_.NotAfter -gt $(Get-Date) -and $_.HasPrivateKey } |
    Select-Object Subject, Thumbprint | Format-List

SAC sits on serial port 2 of the interactive serial console, which is off by default. You turn it on with the metadata key serial-port-enable=TRUE (that needs compute.instances.setMetadata), and opening a command channel in SAC asks for a Windows account. A different org policy blocks it, compute.disableSerialPortAccess, so an organization that blocks guest attributes may still allow this. It is the only one of the three that shows the current certificate rather than the one from the first boot.

Errors over the tunnel: 0x204, 4003 and 4010

Windows App on a Mac shows error code 0x204 when it can't reach a Remote Desktop server at the address you gave it. Over IAP that's the path: the VM still booting, the firewall rule, the tunnel role, or a client pointed at a port where no start-iap-tunnel is listening.

A missing firewall rule fails slowly. When I measured it, IAP took about 31 seconds to close a connection to a port with no rule with 4003, while an open port answered in about 1.1 seconds. IAP Desktop's default connect timeout is 30 seconds. A client that gives up first reports its own timeout, and you never see the 4003 that would have pointed at the firewall. If RDP hangs for half a minute and then fails, check the 3389 rule before you reset anyone's password.

RDP sessions in my client died whenever I scrolled fast, with close code 4010. The pump that copied the local socket into the relay sent frames concurrently, and a burst of scroll input could reorder them. The VM got a TLS stream with bytes out of order, answered with a TCP reset, and IAP closed the session. Reading the socket, waiting for the send to finish, and only then reading again fixed it.

---

I build Ostgate, a paid macOS app that does the password and RDP steps above without gcloud, and it leaves the certificate unchecked for the reasons above. It isn't affiliated with Google: https://ostgate.app