We Chased the 9 People Intercepting Our App's TLS. It Was Google.

Originally published at blog.whynext.app.

Certificate pinning killed our app once. On April 5, 2026, we had pinned the SPKI of our leaf certificate. ACM auto-renewed the certificate, the leaf public key changed, and in that instant every user's API calls were blocked. We had no remote kill switch, so we had to push a hotfix build back through the stores. After that, the repo carried a rule: "Do not add certificate pinning."

Two months later, we built pinning again. This time we pinned the SPKIs of the four Amazon Root CAs instead of the leaf. Roots do not change when ACM renews, so the same accident cannot happen twice. We also added a kill switch we can flip from remote config. On June 29, we remotely enabled report-only mode, which reports mismatches without blocking anything.

Two days later, there was an issue waiting in Sentry.

[CertPinning] SPKI mismatch (report-only)
74 events · 9 users · 2026-06-29

Two hashes we never sent

Opening an event showed the two SPKIs of the certificate the app had actually received.

mode: "reportOnly"
host: "web.pronouncekorean.com"
expected_pin_count: 4
served_spki: [
  "sha256/9hqPsoMiyQMwLCoRPk6FoCYmOsPiGqzQqUcpuZIfvgs=",
  "sha256/r9mjYco6rQO8YkTqr/XXGsQlDUuQqz2mGr67S0imt7M="
]

I pulled the real server chain with openssl s_client and compared. Not the leaf our server sends, not the intermediate, not the root - none of them matched either hash. These 9 people never received our server's real certificate at all. Someone was terminating TLS in the middle, re-signing with their own certificate, and handing that to the app.

The fact that served_spki had only two entries was a clue too. On the normal path, the chain the app reports contains three certificates, all the way up to the root. Only two (leaf plus intermediate) meant this report came from a different branch in the code.

This is where I drew my first conclusion. To be precise, the AI agent drew it and I found it plausible. "This is not a bug to fix. Report-only pinning is working exactly as designed and catching interception happening on other people's networks. Corporate or school proxies, antivirus web protection, captive portals, that sort of thing. We cannot control any of it, so mute it in Sentry and just watch it as a metric."

Since we were in there anyway, a suggestion came along with it: add a backup pin. I picked that.

A backup pin would not remove a single one of these events

Right after the work on adding the pin started, something felt inconsistent, so I asked back.

Hold on, shouldn't a mismatch be impossible in the first place?

That one line collapsed the direction we had just chosen. The agent re-read the Android probe code and corrected itself.

val verified = verifiedChainSpki(offered, host)
if (verified == null) {
    // Trust store validation failed = interception/forgery -> report without any pin comparison
    report(host, offered.mapNotNull(::spkiSha256))
    return
}
// Pin comparison only happens for a chain that passed validation
if (verified.none(pins::contains)) { report(host, verified) }

The probe has two steps. First it validates the chain against the OS trust store, and only a chain that passes gets compared against the pin list. Our 74 events had already failed at the first step. With verified == null, the code reports immediately without ever running the pin comparison. Backup pins only matter at the second step, the case where the chain validates but the pin does not match.

Conclusion: adding Starfield G2 to the pins will not reduce this issue by a single event.

That was also the moment it became clear that a normal user cannot produce this report, because the live chain matches our pins exactly. So by definition these 9 people are not normal users. There was no bug to fix here. What we had to find out was who these 9 were.

Ignore it and we can never turn it on

I could not accept the "ignore it and monitor" recommendation either.

We have to fix this properly. We need to actually turn this on later, and if it stays like this we never can.

The end goal of this pinning work is enforce mode. The gate for deciding whether we can turn enforce on was already written in the design doc: two weeks with a real-user mismatch rate below 0.5%. If we leave the mismatches unexplained, these 74 events keep polluting the gate. Muting them makes the 74 disappear from the dashboard, but the gate stays polluted exactly as before.

So I dug into the data more. As soon as I pulled the tag distribution, something strange came up.

All 74 events came from the same device, and that device does not exist

The device tag on all 74 events was 100% a single value: OnePlus8Pro. But the fingerprint of that device made no sense.

Field	Event value	Real OnePlus 8 Pro
archs	includes x86_64, x86	arm64 only (Snapdragon 865)
Screen	288 x 448 px, 106 dpi	1440 x 3168, 513 dpi
CPU	2 cores, frequency 0	8 cores
simulator	false	-

Consumer ARM phones do not advertise an x86 ABI. There is no Android phone in the world with a 288x448 screen at 106 dpi. This was an x86 virtual device with only its Build.MODEL disguised as a real device name. It was even claiming simulator: false to pass as physical hardware.

The 9 user IDs were not people either. All 9 existed only on June 29, and in that single day they ran all three versions: 1.7.0, 1.8.0, and 1.8.1. That is not a person, that is a pipeline. The region on all 74 events was the US, but the city was null on every one of them. That is what traffic from a data center range looks like.

At this point I asked: so who is analyzing our app? The agent put security vendors' automated APK analysis sandboxes at number one, followed by app intelligence companies and APK mirror sites. Automated store review came in at number four.

The fourth-ranked theory became first with three IPs

Number four bothered me. So I asked.

Could it be Google Play Console or the App Store?

The answer that came back was firm.

Google Play pre-launch report / Test Lab - almost certainly not. The decisive reason: Google does not MITM your HTTPS. Play pre-launch uses the standard trust store and receives your real certificate as is. Store crawlers do not intercept and re-sign traffic. That is what an analysis sandbox does.

As pure reasoning, the argument is hard to fault. And it was wrong.

What was missing was IP evidence. Sentry scrubs IPs as PII, so user.ip was null on every event, and the original IP of a past event cannot be recovered. To identify the source we needed IPs, and the tool we were looking at did not have them.

What we did have was the production API's access log. It recorded X-Forwarded-For and User-Agent on every request, and our app's User-Agent was unique: Dart/3.12 (dart:io). And June 29 was still inside the 7-day log retention window.

First I pulled the event timestamps of all 74 mismatches from Sentry and grouped them into three clusters: 02:12-02:20, 11:30-11:41, and 17:32-17:40 (UTC). Then I queried CloudWatch for the IPs of requests that came in with Dart/3.12 during those three windows.

02:12-02:20  ...  66.249.84.132, 66.249.84.141
11:30-11:41  ...  66.102.7.69
17:32-17:40  ...  66.249.84.141

66.249.x.x is the range Googlebot uses, and 66.102.x.x is Google too. All three windows had a Google IP in them. In two of the three, the only app request in that window came from a Google IP. The remaining window also had an IP that looked like a real user, but the Google IP was in there as well.

Three minutes later, the agent corrected itself.

Correction. Earlier I said "Google does not MITM, so this is probably not them," but the IP evidence overturns that guess. This is Google. Measured IPs are stronger evidence than reasoning.

What was pulling our app apart was Google's routine automated scan, the one every app on Play gets. Not an attacker, not a competitor, not a security vendor. That scan environment instruments traffic. That is why, to our probe, the chain looked unverifiable.

So what did we fix?

We left the pinning alone and changed where the probe applies. We decided not to turn pinning on at all in scanner environments.

That left the question of how to recognize a scanner. It was already clear we could not trust simulator: false, so we needed a signal that cannot be spoofed. CPU architecture cannot be hidden.

// Decisive signal: a consumer ARM phone never advertises an x86 ABI.
// Google's scanner can spoof Build.MODEL with a real device name (observed: "OnePlus8Pro" on x86),
// but it cannot hide the fact that it is x86_64.
if (supportedAbis.any((abi) => abi.contains('x86') || abi.contains('i686'))) {
  return true;
}

On top of that we also check for Cuttlefish/GCE hardware names and test-keys fingerprints. The detector is deliberately biased toward answering "this is an emulator." A false positive just means we skip pinning in that one environment, and an attacker running the app on an emulator can patch the app to disable client-side pinning anyway, so we lose nothing. A false negative is far worse, because it traps a scanner behind a block screen.

Here is why that matters. If we had simply turned enforce on, this is what would have happened. Google's pre-launch crawler hits our "insecure connection" block screen. That gets recorded as a failure in the Play Console pre-launch report. Nothing is wrong with the code, and yet a red line shows up on the store review screen.

What stayed with me

Sentry scrubs IPs as PII. That is the right default. But because of it, the one field that could identify what was going on was gone from the event. In the end we cross-referenced two things, the app's User-Agent and the timestamps of the events, and recovered from the access log what Sentry had thrown away. Missing from one tool does not mean missing everywhere.

The sentence "Google does not intercept HTTPS" was a smooth piece of reasoning. It worked through the purpose of a store crawler, the behavior of the standard trust store, and the difference from a sandbox, and pushed the theory down to fourth place. Then three IPs flipped the conclusion in three minutes. Ask an AI about principles and you get principles back. Whether those principles match reality is something only the logs can tell you.

If we had muted these 74 events in Sentry, the dashboard would have been clean, and on the day we turned enforce on we would have gotten a red line in store review without knowing why.

We still have not turned enforce on. The gate requires two weeks with a real-user mismatch rate below 0.5%. Now that the scanner is filtered out, we have to count those two weeks again from the start.