In September a customer emailed me. They had bought a Pro licence for pyobfus, my Python obfuscator, and activation kept failing with this:
License verification failed and no valid cache available: Access denied
I first treated it as a problem with their machine. It wasn't. Online activation was failing for everyone, on every platform, on every release up to that point. The last successful online verification in our records was on 8 July. The email arrived on 12 September.
I can't tell you exactly when it started, because I have no record of when the edge started rejecting the requests. What I can tell you is why it took two months to notice, and that part is entirely on me.
What was actually happening
The licence client used urllib.request without setting a User-Agent, so every request went out with urllib's default, Python-urllib/3.x.
The licence server is a Cloudflare Worker. Cloudflare's edge was rejecting that User-Agent before the request reached the Worker, with an HTTP 403 and a plain-text body:
error code: 1010
Cloudflare documents 1010 as a block based on the browser signature. In the requests I tested, changing only the User-Agent changed the result:
| User-Agent sent | Result |
|---|---|
Python-urllib/3.12 |
403, error code: 1010
|
python-requests/2.x |
Worker's own JSON reply |
| a browser string | Worker's own JSON reply |
| no User-Agent header at all | Worker's own JSON reply |
So this was not a "block everything that isn't a browser" rule. The signature my client sent was blocked, and the others I tried were not.
Why nobody saw it
Two things on my side turned an edge block into two months of silent failure.
The error message threw away the only clue. The client expected JSON. When it got a plain-text 403 it couldn't parse the body, so it fell back to a hard-coded string: "Access denied". That reads like "your key is wrong". A customer who sees it is likely to blame themselves or give up rather than write in.
Nothing was watching. There was no probe and no alert. The only monitoring was "a customer writes in".
Once I started digging, I found older problems in the same code.
- I had never checked whether paid licences had ever completed an online verification. Some had not. An empty device list doesn't prove someone never used the product, since the offline path doesn't touch the server, but it was a question I should have been asking all along.
- The device fingerprint included
platform.release(), so a minor OS update gave the same machine a new identity. The local cache stopped matching and re-registering used up another of the three device slots, and customers had no way to free one. It also useduuid.getnode(), which returns a random number when Python can't read a hardware address, so a new Python process could identify the same machine differently. I had also failed to follow up on an earlier report of exactly this. - A server revocation could be swallowed by the offline-cache fallback. The client treated the server's rejection as a generic failure, fell back to the cache and reported the licence as valid, with the word "revoked" still in the message.
What I changed
Send a User-Agent that says who you are. The client's User-Agent now starts with pyobfus-license/<version>. If your urllib requests pass through a content delivery network (CDN), set a User-Agent explicitly (url, payload and __version__ come from your own code, and you still send the request with urllib.request.urlopen):
import urllib.request
req = urllib.request.Request(
url,
data=payload,
headers={
"Content-Type": "application/json",
"User-Agent": f"yourtool/{__version__}",
},
)
Say what actually failed.
The client now treats a 403 without the Worker's JSON error as an infrastructure failure. The message says the request was blocked before it reached the licence server, that the licence may still be valid, and gives the exact offline command to use instead.
Watch the real path, with a key that doesn't exist.
A scheduled GitHub Actions workflow runs twice a day. It first calls the production endpoint through the same client function customers use, with the same headers and User-Agent, sending a well-formed licence key that was never issued. A plain-text edge block, a 404 whose body doesn't match the expected unknown-key response, or a connection error fails the run, and GitHub emails me about the failure.
This avoids changing customer records or device lists and keeps real licence keys out of CI secrets. I tested it three ways before trusting it: green against the fixed client, red when I put the old User-Agent back, red when the endpoint was unreachable.
That wasn't enough. The first version only checked the client's error message, and the client reports every 404 the same way, so a missing route or a proxy's 404 page would have passed as healthy. A reviewer caught it while I was writing this post. The probe now repeats the request and checks that the 404 body matches the expected unknown-key JSON response, and I added a fourth test: a 404 page from a different site has to turn it red. It still only covers the rejection path. It does not prove that a valid key can activate.
Decide which failures allow a cache fallback.
The old client rejected its cache when the device fingerprint changed, yet accepted it after a generic server rejection, which is exactly backwards. Now a signed cache survives a drifting device identity, and an online answer that explicitly says revoked or expired refuses the cache fallback. The device ID is now a random value generated once and kept in ~/.pyobfus/device_id, so it survives OS updates, new virtualenvs and containers rebuilt with the same home directory. On the server, a fourth device now replaces the least recently verified one instead of being refused.
Both the device ID and the local cache can be copied, and there has always been an offline registration command. The old fingerprint check caused false lockouts without reliably preventing copying. The server keeps three recent device records, which is not a hard limit on offline use, and revocation takes effect when the client receives an online rejection.
The part a release can't fix
A fix in a new version only reaches people who upgrade. Every copy already installed keeps sending Python-urllib and keeps failing, and a customer who has given up may never see the release notes.
So two things went out with the fix. The new error message names the offline path, which bypasses this network block because it never contacts the server (replace <KEY> with your licence key):
pyobfus-license register <KEY> --no-verify
And I wrote to the customers I knew were affected the day I found the cause, before the fix was released, including one who had never complained.
If you ship anything that phones home
- Set an explicit User-Agent. The default one is a shared signature, and you don't control how the CDN in front of you treats it.
- Never let a fallback message replace the evidence. If the response isn't in the shape you expected, say so and show the status code, without echoing keys or personal data.
- Probe production with an input that has no side effects. A made-up ID that the server should reject is often enough for the rejection path. Break the probe on purpose once to see it go red, and check that the alert reaches you.
- Check whether paid licences have ever completed an online verification. Support tickets won't show you everyone who gave up.
The fixes shipped in pyobfus 0.5.26 on 13 September, and online activation has worked for new purchases since. As of 11 October the monitor's only red run was a bug in the workflow itself, not an outage. This gives me a way to catch this kind of failure without waiting for a customer to report it.
I used an AI assistant to help organise this post. I checked the technical claims against the code, and the incident timeline against my own records.
Top comments (0)