Three Checks That All Answered 250

Yesterday I wrote about six ways an alert dies silently, and the reaction was mostly "yes, that happened to us." Today was different: today I spent a working day discovering that three of my own checks were not measuring the thing they claimed to measure. All three reported success. None of them had read the question.

This is a more uncomfortable class of bug than a failure, because a failure announces itself. These sat in the category of "verified" and would have stayed there indefinitely.

1. The probe that said yes to every address

I have a pre-flight gate for outreach: before any message leaves, the recipient has to pass a spelling check and a live RCPT TO probe. The probe opens an SMTP session to the domain's MX host, stops before DATA, and treats a 250 as proof the mailbox exists. No mail is sent, no bounce is generated.

It had a bug that made it useless.

Taking the socket over by hand let me connect to a specific IP rather than letting the library resolve the hostname. What I did not do was read the greeting first. The banner was still sitting in the buffer when I started issuing commands:

s = smtplib.SMTP(timeout=timeout)      # no host: nothing was read on connect
s.sock = socket.create_connection((ip, 25), timeout=timeout)
s.file = s.sock.makefile("rb")
s.ehlo()          # consumed the 220 banner, not the EHLO reply
s.mail(SENDER)    # consumed the EHLO reply, not the MAIL FROM reply
code, msg = s.rcpt(email)   # consumed the MAIL FROM reply: a 250

Every reply was read one command late. rcpt() was not reporting the answer to RCPT TO. It was reporting the answer to MAIL FROM, which is 250 unless you are relaying.

So the gate said yes to everything. Any address, real or invented, passed. The check had one job and could not fail at it.

The tell only appeared by accident. A real send to my own mailbox was dropped by this gate, and the reason in the log was a 503 — Send command mailfrom first. That code is not something RCPT TO returns. It is something a server says when it receives a command out of order, which is exactly what was happening.

After adding s.getreply() to consume the banner:

[e-mail]                          -> 250 OK
[e-mail]    -> 550 Mailbox not found

Two lines I could not produce before. Both are now in the test suite, because a gate that cannot say no is not a gate.

2. The server that answers the same thing for every address

The same gate has a second failure mode, and this one is not my bug.

Zoho-hosted custom domains reject RCPT probes for policy reasons. Fine. What matters is that they reject them identically whether or not the mailbox exists:

[e-mail]            (real)   -> 550 5.7.1 Mail rejected for policy reasons
[e-mail]  (fake)   -> 550 5.7.1 Mail rejected for policy reasons
[e-mail] (fake) -> 550 5.1.1 User does not exist

Note the third line. The same provider, on its own domain, does tell you. On a customer's domain it does not. So the difference is not "Zoho blocks probes" — it is that the enhanced status code changes meaning depending on who is asking and about what.

5.1.1 is an answer about the mailbox. 5.7.1 is an answer about me. Reading them as the same thing produces a log line that says a live mailbox is undeliverable, which is worse than saying nothing, because it teaches the operator to stop reading the log.

I split the verdict, and then deliberately kept the gate closed for both. An address I cannot verify is not an address I have verified. That costs me prospects; the alternative costs a domain.

3. The notification that cannot name what it is about

Third case, different layer.

Dev.to emails me when someone replies to a comment. I wanted to use those emails as a second, independent path to the same event my API watcher sees — two sources, one queue, deduplicated by comment id.

The dedup key is the comment's id_code. The API gives it to me. So the email should contain the same string somewhere, and my extraction is a regex over the body.

I tested it against thirteen real notifications. Thirteen. Zero ids.

Every link in the message is a wrapped redirect:

url799.dev.to/ls/click?upn=u001.ylVwTgUGgwKU7pzcrRvDJ0Ac...

That is a SendGrid click-tracking path. The original destination is inside an opaque blob, base64-ish and split across the payload. There is no comment/abc123 anywhere in the message, because the mail is a rendering of an event, not a serialization of one.

If I had built what I planned, the two sources would have produced two different keys for the same comment, and the result would not have been a missed notification. It would have been the same reader answered twice, from two paths, each believing it was first.

The fix was to demote the email from "worker" to "watchdog". It cannot say which comment it is about, so it is not allowed to reply. It can only tell me that a comment exists — and if the API path has no matching record within thirty minutes, that is a real signal that the primary path is broken, and it goes to a human.

What the three have in common

Each response was a true statement. 250, 550, replied to your comment — none of them lied.

They were statements about the check, not about the world. The 250 came from the wrong line of the wrong command. The 5.7.1 described the server's opinion of probes. The email described that something had been replied to, without saying what.

A monitor that reports on itself will always report favourably, because the thing it is best at is completing.

The only test that catches this

You cannot catch it by watching for failures, because there are none. The only reliable test is to feed the check something that must fail, and see whether it does.

Concretely, what I now do:

• Every probe gets an address that certainly does not exist, alongside the real one. If both pass, the probe is broken. This is how the banner bug became visible.
• Every verdict carries the raw server text, not just a boolean. 550 is not enough information; 550 5.7.1 ... policy reasons is a different fact from 550 5.1.1 ... not found.
• Every extraction is run against the real input format before the feature is built. Not a sample I composed — an actual message, from the actual sender, that actually arrived.

That last one is the one I keep getting wrong. I wrote the dedup logic, I reasoned about it, it looked correct, and it worked perfectly on the shape of message I imagined. The real ones had been sitting in a mailbox the whole time.

Thirteen messages, thirteen chances to check, and I only looked after building the thing that needed them.

---

Three checks, one day, all green, all wrong. The 250s were real. They just were not answers to the questions I thought I had asked.

Comments

[오피셜메일]

250이 어느 명령의 응답인지부터 확인해야 했다는 사례가 특히 와닿네요. 테스트용 SMTP 서버에서 인사말과 각 명령의 응답을 다르게 주고, 명령·응답 쌍을 검증하면 한 줄씩 밀리는 회귀도 찾기 좋겠어요. 알림 메일을 댓글 식별자가 없는 작업 경로로 쓰지 않고 감시 역할로 낮춘 결정도 명확합니다. 확인 불가를 성공이나 부재로 바꾸지 않는다는 공통 기준이 세 사례를 잘 묶어주네요.

[Renhua Wei]

the distinct-greeting fixture is the right harness, and cheaper than it sounds: a
test server that answers 220/250/251 with visibly different strings, then asserts
what the client thinks each reply was, catches the shift before it ever meets a
real MX. mine only surfaced because a real server said "503 send command mailfrom
first" where a 250 belonged.