DEV Community

Taylor Wang
Taylor Wang

Posted on

The Agent Returned a JSON Client. I Spent 48 Hours Parsing HTML

#python #testing #agents #debugging

Have you ever merged an agent-written HTTP client because the types looked clean and the comments sounded sure? I did that, then I spent forty-eight hours blaming a parser that was doing exactly what I asked. The stack traces were precise, which made the lie feel like honest engineering instead of a missing header. The upstream had never set application/json on the path I was calling, and the body was HTML.

This is a field-notes writeup from a messy weekend, not a ranking of coding products. I wanted a probe I could rerun whenever an agent claimed a network response contract. If you steal one artifact from these notes, steal the mismatch log and the failing exit code. The rest of this post is commentary, plus the commands I wish I had typed first.

Agentic threads this week keep circling the same failure mode I hit in the client. The model assumes the world matches the prompt, then it writes retries around that assumption. Status 200 is not a schema, and a fluent answer is not a header dump. Why do we decorate the parser before we print Content-Type?

Hour 0: a client that looked finished

The agent handed me a small Python module with retries, a typed dataclass, and a cheerful docstring. I asked whether the response was JSON, and it said yes without printing a single response header. I still ran the generated client against a local stub, and the stub returned JSON because I had written the stub myself. That is the first trap, and it is painfully ordinary: the agent and I had agreed on a fantasy upstream.

Here is the shape it emitted. Treat it as a reconstructed example, not as a proven client from production.

# example_client.py — reconstructed hypothesis, not a verified client
from dataclasses import dataclass
import json
import urllib.request

@dataclass
class Payload:
    id: str
    status: str

def fetch_status(url: str) -> Payload:
    with urllib.request.urlopen(url, timeout=5) as response:
        body = json.loads(response.read().decode("utf-8"))
    return Payload(id=body["id"], status=body["status"])
Enter fullscreen mode Exit fullscreen mode

Looks harmless, right? It stays harmless until the bytes are a login page. json.loads then becomes a very loud way to say you never checked the contract.

Hours 0–8: what I tried anyway

I did the usual human dance, and none of it printed the header that would have ended the incident.

  1. I added stricter JSON decoding flags, as if strict=True could invent an object from markup.
  2. I wrapped json.loads in a retry loop, which retried the same HTML document with great confidence.
  3. I logged body[:200], saw <!DOCTYPE html>, and kept hunting for a decoder bug because I wanted JSON.
  4. I asked the agent to fix the parser, and it added a second decoder plus a fallback regex.

That last step is the expensive one. The agent optimized the wrong layer because I never showed it the headers from a real GET. I was debugging tone, not transport.

Hours 8–24: what actually broke

The path existed. The status was 200. The body was an HTML login interstitial from a reverse proxy I had forgotten about. My parser was not broken, and the dataclass was not wrong. The contract was fictional, which is worse, because every later patch made the fiction look more engineered.

Three facts would have ended the incident before lunch, and I wrote them on a sticky note:

  • HTTP status is not a schema, even when the number is 200.
  • A reverse proxy can serve a login page on the same path your OpenAPI file still calls a resource.
  • Content-Type plus a tiny key check beats another decoder, every single time I have tried it.

I needed a command that treats the agent's claim as a hypothesis. Chat is a draft. A probe is evidence.

Hours 24–32: the probe I now run first

The artifact is a local upstream double plus a checker. You can run it without the public internet, which matters when you are tired and tempted to trust a story. The double can lie on purpose. That is the feature, not a bug.

A tiny upstream that can lie 

# fake_upstream.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os

MODE = os.environ.get("UPSTREAM_MODE", "html")

class Handler(BaseHTTPRequestHandler):
    def log_message(self, format, *args):
        return

    def do_GET(self):
        if self.path != "/v1/status":
            self.send_error(404)
            return
        if MODE == "json":
            body = json.dumps({"id": "abc", "status": "ok"}).encode()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        body = b"<!DOCTYPE html><html><body>login</body></html>"
        self.send_response(200)
        self.send_header("Content-Type", "text/html; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8077), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

The mismatch log 

# assumption_probe.py
import json
import sys
import urllib.error
import urllib.request

CLAIMED_TYPE = "application/json"
CLAIMED_KEYS = ("id", "status")
URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8077/v1/status"

def main() -> int:
    request = urllib.request.Request(URL, method="GET")
    try:
        with urllib.request.urlopen(request, timeout=5) as response:
            status = response.status
            content_type = response.headers.get("Content-Type", "")
            raw = response.read()
    except urllib.error.HTTPError as exc:
        status = exc.code
        content_type = exc.headers.get("Content-Type", "") if exc.headers else ""
        raw = exc.read()

    preview = raw[:180].decode("utf-8", errors="replace")
    type_ok = CLAIMED_TYPE in content_type.lower()
    keys_ok = False
    if type_ok:
        try:
            parsed = json.loads(raw.decode("utf-8"))
            keys_ok = all(k in parsed for k in CLAIMED_KEYS)
        except json.JSONDecodeError:
            keys_ok = False

    print(f"url={URL}")
    print(f"status={status}")
    print(f"content_type={content_type!r}")
    print(f"claimed_type={CLAIMED_TYPE!r} match={type_ok}")
    print(f"claimed_keys={CLAIMED_KEYS} match={keys_ok}")
    print(f"preview={preview!r}")
    return 0 if (status == 200 and type_ok and keys_ok) else 2

if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Commands I actually type 

# terminal 1: start the lying upstream
UPSTREAM_MODE=html python fake_upstream.py

# terminal 2: treat the agent claim as a hypothesis
python assumption_probe.py http://127.0.0.1:8077/v1/status
echo $?
Enter fullscreen mode Exit fullscreen mode

When UPSTREAM_MODE=html, the probe exits 2 and prints text/html. That exit code is the whole point of the weekend. I do not want another story from the model. I want a number a job can fail on before anyone rewrites JSON code.

Switch the mode and rerun the same checker without editing the client:

# stop the previous server, then:
UPSTREAM_MODE=json python fake_upstream.py
python assumption_probe.py http://127.0.0.1:8077/v1/status
echo $?
Enter fullscreen mode Exit fullscreen mode

Now the generated client is allowed to exist. Not earlier, and not because the docstring sounded sure.

Hours 32–48: why I reran the probe off my laptop

My laptop lies in boring ways that do not show up in agent chat. It has a warmed DNS cache, a leftover cookie jar, and a proxy variable from a previous incident. An agent running in the same shell inherits all of that mess, then reports that the endpoint looks fine. I wanted a second box that did not know my .bashrc.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I reran the same probe with MonkeyCode's free model access and free server option so the checker started from a boring environment instead of my haunted laptop. I am not going to invent model names, quotas, hardware, or duration I cannot show you. The useful split is simpler than any of that: draft the client in one place, then execute the probe in a shell you can actually describe.

If you need that split, the free server option is a convenient second box for the same commands. After the probe was red, I stopped asking for another decoder and started asking for headers.

Decision table I keep in the notes

Agent claim Probe I run Pass if Fail means
"It returns JSON" Read Content-Type application/json is present You are parsing a document
"Status 200 means the schema is fine" Decode body only after the type check Claimed keys exist after json.loads Proxy page, empty body, or HTML
"Retries will help" Fingerprint the preview across attempts The preview actually changes You are retrying the same HTML
"Works on my machine" Rerun the probe in a clean shell Same exit code on both machines Laptop cache became the test

I print this table in the pull request when the agent wrote the client. Reviewers can argue about retries later. They should not have to argue about text/html.

What I would repeat next time

I would pin the claim in the review, not in a chat transcript that nobody will reread.

  • Paste probe output, including content_type and a short preview, next to the client diff.
  • Refuse extra decoder code until the content type matches the claim in the docstring.
  • Run the probe once locally and once on a clean server, then compare exit codes only.
  • Delete the stub that always returns JSON, or keep a mode that lies on purpose so the probe can fail.
  • Ask the agent for the curl equivalent first: curl -sI is a smaller lie surface than a generated class.

Would I still use an agent to draft the client after this weekend? Yes, after the probe is red or green. The draft is cheap. The unverified contract is not cheap, and it burns the kind of hours that feel like real debugging.

Limitations

This probe does not validate auth refresh, pagination, streaming, or idempotency keys. It does not prove that id means what your product thinks it means. A JSON object with the right keys can still be a mock planted by yesterday's stub. If your upstream needs mTLS or a signed cookie dance, the local double will not save you, and you should not pretend it will.

The clean-server rerun only helps when the failure is environmental. It will not catch a bad API shape. It will not catch a field that changed meaning without changing type. I also do not use this as a load test, a latency benchmark, or a security review.

Who should not use this approach

Skip it if you already have contract tests generated from a real OpenAPI document and a staging cluster you trust. Skip it if the agent is not allowed to talk to the network at all, because then you need fixtures, not urlopen. Skip it if you need throughput numbers; this is a content-type check, not a benchmark and not a capacity plan.

If you are on-call and the parser is throwing, run the probe before you ask anyone to rewrite JSON code. Forty-eight hours is a long time to spend decorating a wrong assumption with better types.

Top comments (0)