DEV Community: Mikhail Golikov

Green unit tests are a comfort blanket

Mikhail Golikov — Sat, 20 Jun 2026 11:31:59 +0000

My converter has a test suite. It was green. Every example I could think of went in, the right pytest file came out, and I shipped it to PyPI as 1.0. Weeks later it sits there marked Production/Stable.

Then I pointed a fuzzer at it, and it stopped being so quiet.

The blind spot in example tests

Unit tests check the inputs you thought of. That is their whole nature: you sit down, you imagine how the code gets used, you write an example for each case. The cases you imagine tend to be the cases the code already handles, because you wrote the code and the test in the same afternoon, with the same blind spots.

A converter makes that worse. postman2pytest takes a Postman collection, a JSON file, and turns it into a runnable pytest suite. The input is not something I control. It is a file a stranger exports from a tool, edits by hand, truncates over a flaky download, or generates from a script that had a bug. The real test suite for a parser is the set of files you did not write.

So I stopped writing examples and let a machine write them for me.

Letting Hypothesis do the imagining

Hypothesis is property-based testing for Python. Instead of one example, you describe the shape of the input and assert a property that must hold for all of them. It then generates hundreds of cases, including the mean ones you would never type out.

For a parser the property is simple to state. Feed it any well-formed JSON, whatever its shape. It should do exactly one of two things: return a list of parsed tests, or raise a clear error that tells the user what is wrong. It must never fall over with a raw stack trace.

import json
from hypothesis import given, strategies as st

json_values = st.recursive(
    st.none() | st.booleans() | st.integers() | st.text(),
    lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)

@given(value=json_values)
def test_parser_is_graceful_on_any_json(value, tmp_path):
    path = tmp_path / "in.json"
    path.write_text(json.dumps(value))
    try:
        result = parse_collection(path)
    except ValueError:
        return  # the one allowed failure: a clear, typed error
    assert isinstance(result, list)

Then I ran it.

What it found

The parser handled every real Postman collection the same as before. The fuzzer never touched valid input. It broke my handling of garbage, in five distinct shapes.

A collection whose top level was a list instead of an object. A collection whose info block was a string. An item that was a bare number where the code expected a dict. A folder nested deep enough that the recursion bottomed out. And a payload so deeply nested that json.loads itself blew the stack before my code ran at all.

In each case the user would have seen an AttributeError, a TypeError, or a RecursionError, with an internal stack trace and no idea what to fix. None of those is a useful message. The tool knew the input was wrong. It just had no manners about saying so.

The fix is a contract, not a patch

Plugging the five holes was the boring part. The contract underneath them was the real work: valid collection in, pytest suite out; anything else, one clear ValueError that names the problem. Degrade, do not crash.

That turned into a few isinstance checks at the boundaries, a depth limit on folder recursion, and wrapping the top-level json.loads so a stack-busting document becomes a readable "this file is nested too deeply to parse" instead of a traceback. The same review on my other parser, secure-log2test, which reads Kibana log exports, turned up the same family of gaps in the same afternoon. Same shape of bug, same shape of fix.

What I actually learned

"Production/Stable" is a statement about the API, not about the input. It never promised to survive a corrupt file, and I had let people assume it did.

And finding bugs with a fuzzer is the fuzzer doing its job. Point Hypothesis at a parser and it finds nothing? You probably did not let it generate anything nasty enough. The bugs are the evidence the method works.

If your tool reads a file someone else made, your unit tests are a comfort blanket. Spend twenty lines on a property test before you trust them. The input you did not write is the one that finds you in production.

postman2pytest and secure-log2test are MIT-licensed on PyPI. The fuzz tests are in each repo if you want a template to copy.

Keep the LLM out of your chatbot tests

Mikhail Golikov — Thu, 04 Jun 2026 22:10:03 +0000

It is a warm evening. You are home from work, you crack the window open, and the room fills with that low gold light you get for about twenty minutes a day. You have changed into the soft clothes. There is a cold drink sweating in your hand. You drop onto the sofa, pick up the remote, and hit play on the show you have been saving all week.

It does not play. Your account "needs attention." And the only thing now standing between you and that perfect evening is a long, heartfelt conversation with a support bot.

So that bot had better not lose the plot on turn three. Somebody has to build it, and somebody has to test it, and the moment you test a multi-turn conversation you hit a fork in the road.

TLDR

An LLM that grades your chatbot's answers is a second non-deterministic system. Now two of them have to agree before CI goes green, and they will not always agree.
Most of what you actually want to assert about a conversation is boring and checkable: did the bot ask for the city after it got the name, did it keep the slot, did it return the right reply on turn three.
pytest-conversational gives you a small Conversation object, a bot adapter you write, and pytest fixtures that keep turn order and per-turn state. No model on the test side.
It is alpha. Single-turn and multi-turn assertions work today; a YAML scenario format and async adapters are on the roadmap, not in the box yet.

Two ways people test chatbots, and why both hurt

When you sit down to test a conversational interface, you usually land in one of two camps.

The first camp is a pile of requests.post calls. You fire a message at the bot, read the JSON, and write an assertion against it. This works for one turn. By turn four you are hand-rolling the history payload, threading state through local variables, and copy-pasting the same setup into every test. When something breaks, the failure message tells you a dict did not equal another dict, and you go spelunking to find out which turn went wrong.

The second camp is a full conversational-testing framework that owns your whole setup. It handles multi-turn flows, but it pins you to one platform and one way of describing a bot. The day your bot moves behind a different transport, or you want to test a plain Python function instead of an HTTP service, you are fighting the framework.

There is a third option that keeps coming up, and it is the one I want to push back on: have a language model read the bot's reply and decide whether it was good. It looks appealing because it handles paraphrase. The bot said "sure thing, what city?" instead of "got it, what city?" and the grader shrugs and passes it.

The problem is what you just built. Your test suite now contains two non-deterministic systems, and CI only goes green when both of them agree. The model under test drifts when you change a prompt. The grader model drifts when the provider ships a new version. A test that passed on Monday fails on Thursday and nothing in your code changed. You also pay latency and tokens on every run, and you have given up the one thing a test should give you: a clear, repeatable yes or no.

A colleague who leads engineering on a chatbot platform put it well when we were going back and forth on this online. Keep the test side deterministic. Let the bot be as clever as it likes; the thing checking the bot should be dumb and predictable.

The pivot: make the test side boring on purpose

Here is the shift. Most of what you care about in a conversation is not "was the answer phrased nicely." It is structural:

After the user gives a name, does the bot ask for the next slot?
Does the bot remember the name three turns later?
On an out-of-scope message, does it fall back instead of inventing an answer?
Does the reply contain the order number, match a known pattern, or land in a known set of options?

All of that is checkable without a model. You need something that keeps turn order, holds per-conversation state, and prints a readable transcript when an assertion fails. That is the whole job pytest-conversational is trying to do, and nothing more.

You bring the bot. The plugin wires it into pytest.

What it looks like

Install it:

pip install pytest-conversational

Python 3.10 and up.

A bot is any callable that takes the user text and the conversation, and returns a string. The simplest test:

def my_bot(text, convo):
    if "hello" in text.lower():
        return "hi"
    return "sorry, did not get that"


def test_greeting(conversation_factory):
    convo = conversation_factory(bot=my_bot)
    convo.say("hello there")
    assert convo.last.bot == "hi"

conversation_factory is a fixture. You hand it a bot, you get a fresh Conversation back, isolated from every other test.

Multi-turn state, without globals

This is where the single-turn camp suffers. A bot adapter can read convo.state and convo.turns, so slot filling stays inside the conversation instead of leaking into your test body:

def slot_filling_bot(text, convo):
    slots = convo.state.setdefault("slots", {})
    if "name" not in slots:
        slots["name"] = text
        return "got it, what city?"
    if "city" not in slots:
        slots["city"] = text
        return f"hello {slots['name']} from {slots['city']}"
    return "done"


def test_two_slot_flow(conversation_factory):
    convo = conversation_factory(bot=slot_filling_bot)
    convo.say("Mikhail")
    convo.say("Hove")
    assert convo.state["slots"] == {"name": "Mikhail", "city": "Hove"}
    assert convo.last.bot == "hello Mikhail from Hove"

The test reads top to bottom like the conversation it describes. When it fails, you call convo.transcript() and see every turn, not a diff of two dictionaries.

A multi-turn test catching a bot that drops the name on the final turn. The failure message shows exactly what the bot said versus what the test expected.

Matchers for the fuzzy bits

Exact string equality is fine until it is not. Some replies legitimately vary, and you want to assert a shape rather than a literal. The expect module covers the common cases and puts the actual reply in the failure message, so pytest output shows what the bot said versus what you wanted:

from pytest_conversational import expect

def test_replies(conversation_factory):
    convo = conversation_factory(bot=my_bot)
    convo.say("hi")

    expect.contains(convo.last.bot, "hello")
    expect.regex(convo.last.bot, r"^hello\s+\w+")
    expect.one_of(convo.last.bot, ["hello there", "hi there", "hey"])

contains is a case-insensitive substring check by default. regex runs re.search and hands back the match object so you can inspect captured groups. one_of checks the reply against a list of alternatives, with an exact mode and a substring mode. None of this needs a model to decide anything. The rules are yours and they do not drift.

Testing a bot that lives behind HTTP

If your bot is a deployed service, you do not have to write a transport by hand. There is a bundled webhook adapter:

pip install pytest-conversational[http]

from pytest_conversational import Conversation
from pytest_conversational.adapters import http_webhook


def test_remote_bot():
    bot = http_webhook("https://my-bot.example.com/webhook", timeout=3.0)
    convo = Conversation(bot=bot)
    convo.say("hello")
    assert "hi" in convo.last.bot.lower()

The default contract is a POST of {"user": text, "history": [[u, b], ...]} and a 200 with {"reply": "..."}. If your endpoint speaks a different shape, pass request_builder and response_parser callbacks.

One thing worth slowing down on. The webhook URL goes straight to httpx as written. If a test ever feeds it a URL that came from fixture data, a config file, or anything you did not type yourself, the adapter will dutifully hit it. That includes 127.0.0.1, the cloud metadata address 169.254.169.254, and anything on 10.x.x.x inside your network. A test reaching the metadata endpoint is not a hypothetical; it is how a fair number of SSRF incidents start. Pin the URL to a literal in the test, or run it through your own allowlist before it gets near the adapter.

Where this fits, and where it does not

Determinism on the test side is the right call for structure, state, routing, and fallbacks. That is the bulk of what breaks in a conversational product, and it is exactly the part a model-as-judge handles worst.

It is not the right tool for judging open-ended generation quality. If your actual question is "is this a good, helpful, on-brand paragraph," a rule cannot answer that, and you should not pretend it can. Use human review or an evaluation harness for that, and keep it out of the gate that decides whether your build is broken. The two jobs are different. Conflating them is how you end up with a flaky CI that nobody trusts.

Honest status: the plugin is alpha, with a v1.0 release targeted for June 2026. Single-turn and multi-turn assertions, the matchers, and the HTTP adapter all work now. A scenario format you can load from YAML or plain-text fixtures is on the roadmap, and so is async adapter support for coroutine-based bots. Those are not shipped, and I would rather say so than have you pip install and go looking for them.

Try it

pip install pytest-conversational

Point it at a bot you already have, write one multi-turn test, and see whether the failure transcript tells you more than your current setup does. The code, issues, and roadmap are on GitHub: https://github.com/golikovichev/pytest-conversational

If you do try it, the assertions you wish existed are the most useful issues to open. That is what shapes the scenario format before v1.0.

Your Kibana logs are full of test cases. Here is a CLI that extracts them, with auth scrubbed by default.

Mikhail Golikov — Fri, 15 May 2026 09:45:15 +0000

Every sprint we export a JSON dump from Kibana, scroll through hundreds of log entries, and tell ourselves we will turn them into test cases later.

Later never comes.

The logs contain real API calls. Real endpoints, real payloads, real status codes from production. It is the closest thing to a specification of how the system actually behaves. And almost none of it ever becomes an automated test, because converting it manually takes longer than the sprint.

I got tired of later. I wrote secure-log2test, a CLI that reads a Kibana JSON export and generates a ready-to-run pytest file. One command. Working tests.

There is one constraint that shaped the whole design: no data leaves your machine. No LLM API calls. No cloud. Everything runs locally.

Why the privacy constraint matters

The obvious alternative to building a tool is asking an LLM to write the tests from your logs. It would probably work. Right up until someone on the security team noticed that production logs full of PII and internal API structures were being sent to an external service.

In enterprise environments that conversation ends badly. So I made it impossible: the core has no network calls at all.

But the bigger privacy story is what happens to secrets inside the log entries themselves. Production logs leak Authorization: Bearer ... headers all the time. They leak Cookie values, X-API-Key values, and increasingly they leak request bodies that contain password, refresh_token, client_secret, or whatever the team called their auth field this week. If the generated tests carry those values, the regression suite becomes a credential dump on disk, ready to be accidentally committed.

secure-log2test scrubs three layers before the test file is written:

A static list of well-known auth headers: authorization, proxy-authorization, proxy-authenticate, cookie, set-cookie, x-api-key, x-auth-token, x-csrf-token, x-access-token, refresh-token, id-token, x-amz-security-token, authentication.
A regex pattern (auth|token|secret|key|session|cookie|credential|bearer|password|passwd) that catches custom header names project teams invent.
The same logic walks JSON request bodies recursively. So {"password": "..."}, {"client_secret": "..."}, OAuth {"refresh_token": "..."} all get replaced with ***REDACTED*** at parse time.

The marker is a placeholder. You inject real credentials at run time through environment variables or fixtures.

How it works

pip install secure-log2test

secure-log2test data/sample_kibana_export.json --output tests_generated.py
pytest tests_generated.py -v

A sample export ships with the repo (data/sample_kibana_export.json), so you can see real output without setting up a Kibana instance first.

Input is the Kibana JSON hits.hits[*]._source format you get from any saved search:

{
  "hits": {
    "hits": [
      {
        "_source": {
          "url": "/api/v1/orders",
          "method": "POST",
          "status": 201,
          "headers": {"Authorization": "Bearer abc.xyz", "Content-Type": "application/json"},
          "body": {"item_id": 42, "password": "hunter2"}
        }
      }
    ]
  }
}

Output is a pytest file with one test function per log entry:

def test_post_api_v1_orders_1():
    response = requests.request(
        method="POST",
        url=BASE_URL + "/api/v1/orders",
        headers={"Authorization": "***REDACTED***", "Content-Type": "application/json"},
        json={"item_id": 42, "password": "***REDACTED***"},
        timeout=10,
    )
    assert response.status_code == 201

Both the Authorization header value and the password field inside the body are redacted. The original log dict is not mutated.

The architecture: two stages

Parse (secure_log2test/core/parser.py). Pydantic v2 validates and normalises each entry. Records with missing fields fall back to safe defaults rather than crashing. Malformed entries are dropped with a warning, not silently swallowed. Redaction runs as a Pydantic field_validator so it cannot be skipped accidentally.

Generate (secure_log2test/core/generator.py). The validated list goes through a Jinja2 template (templates/test_module.py.j2) and lands as a .py file. The template is the only place that knows what pytest looks like. Want a different output (httpx instead of requests, unittest instead of pytest, k6 scenarios)? You replace the template. The parser stays untouched.

This split is the part I am most happy with. The redaction logic is unit-testable in isolation. The output format is a config file. Anyone can fork the template and emit something else from the same parsed entries.

The user-feedback loop that improved v1.0.1

The first PyPI release shipped at v1.0.0 on May 11th. Within hours an external user fed it a Grafana Loki export with Cyrillic content from a Russian backend. The parser opened the file without an explicit encoding argument. On Linux this works (utf-8 by default). On Windows the same call raises UnicodeDecodeError because Windows defaults to cp1252.

Bug filed, reproduced, fixed within a day. v1.0.1 went out on the 13th with explicit encoding="utf-8-sig" on the file open. I added a regression test that simulates the cp1252 environment so the same bug cannot come back.

What I learned: every framework that handles user-provided input needs an adversarial encoding test. The happy path is easy. The bug lived in the gap between "what my dev machine does by default" and "what a Windows shell does by default."

What the tests cover

59 tests as of v1.1.0, across parser and generator:

Valid input, malformed records, missing fields, empty exports.
Header redaction with the static list. Header redaction by regex pattern. Custom team-specific headers like X-Custom-Token caught by pattern.
Body redaction walker: password fields, OAuth refresh tokens, nested dicts, lists of dicts, non-dict pass-through.
Float duration coercion (Kibana sometimes outputs 134.0 instead of 134).
Template rendering, payload serialisation, test naming.
CLI argument handling, file output path creation.
A CI smoke test that runs the CLI end-to-end on the sample export and parses the generated Python with ast.parse.

CI runs on Python 3.10, 3.11, 3.12, and 3.13 via GitHub Actions.

Honest limitations

Kibana / Elasticsearch JSON export shape only. Grafana Loki Explore exports are tracked in issue #4.
Single-file input. Multi-file batch mode is on the roadmap.
Output format: pytest only. JSON / CSV for downstream pipelines are tracked in issue #5.
Loads the full file into memory. Not designed for multi-GB exports.
Does not infer sequences or dependencies between requests.
Does not replace manual test design. It accelerates the first pass.

The generated tests are a starting point. You review them, set the base URL via environment variable, add setup or teardown where needed. But you start from working, runnable code rather than a blank file and a pile of log entries.

Where it goes next

v1.1 will add response body assertions and optional schema match (issue #1). v1.2 will allow custom redaction rules via config file (issue #2). Two good first issue slots are open right now if you want to grab one.

Get it

pip install secure-log2test

Repo (MIT licence): github.com/golikovichev/secure-log2test

If your Kibana export shape is different from what the parser expects, open an issue with a redacted sample. The parser is the easy part to extend.

Postman and pytest Are Living in Parallel Universes. Here's a Bridge.

Mikhail Golikov — Fri, 24 Apr 2026 14:13:35 +0000

You have a Postman collection with 40 requests. Organized into folders. With test scripts that check status codes. You spent time on this. It's good.

You also have a CI pipeline that has never heard of Postman and doesn't plan to.

These two things have coexisted peacefully for months because nobody wants to be the person who manually rewrites 40 requests as pytest functions. There's also Newman, but Newman runs tests, it doesn't generate code you can read, modify, or version properly.

So the collection documents the API. The CI tests the API. They describe the same system and have never met.

I built postman2pytest to introduce them.

One Command

pip install postman2pytest

postman2pytest \
  --collection my_api.postman_collection.json \
  --out tests/test_api.py

BASE_URL=https://staging.example.com pytest tests/test_api.py -v

The output is plain Python you can read and edit and commit to version control. No framework lock-in. No runtime wrapper. Just generated Python code.

What the Output Looks Like

Given a Postman collection with a Users folder containing POST /api/v1/users with a test script asserting status 201:

def test_users_post_create_user():
    """POST ENV_base_url/api/v1/users (users)"""
    url = f"{BASE_URL}/api/v1/users"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ.get('token', '')}",
    }
    body = json.loads('{"name": "John Doe", "email": "john@example.com"}')
    response = requests.post(url, headers=headers, json=body)
    assert response.status_code == 201, (
        f"Expected 201, got {response.status_code}: {response.text[:200]}"
    )

A few things worth noticing:

Folder names end up in the function name. Create user inside Users becomes test_users_post_create_user. If you have 40 requests and three folders called List, you'll thank this later.

Postman variables become environment variables. {{base_url}} becomes the BASE_URL env var. {{token}} in an Authorization header becomes os.environ.get('token', '') in an f-string. The generated tests are environment-aware by default.

Status codes come from your existing test scripts. If you wrote pm.response.to.have.status(201) in Postman, the generated test asserts exactly 201. No test script defaults to 200.

Disabled headers stay disabled. You toggled them off in Postman for a reason.

The Architecture

Two stages, cleanly separated.

Parse (core/parser.py): reads the Postman JSON and produces a flat list of ParsedRequest objects, validated with Pydantic v2. Nested folders are flattened recursively. Malformed items are skipped with a warning; the rest of the collection still generates.

class ParsedRequest(BaseModel):
    name: str
    method: str
    url: str
    headers: dict[str, str]
    body: str | None
    expected_status: int
    folder: str | None

Generate (core/generator.py): takes the flat list and renders a Jinja2 template. The tricky part is variable substitution: {{base_url}}/api/v1/users needs to become f"{BASE_URL}/api/v1/users" in Python, and Bearer {{token}} in a header needs to become f"Bearer {os.environ.get('token', '')}". Two custom Jinja2 filters handle this: strip_base_url for URLs, render_header_value for header values.

The split is deliberate. You can use the parser independently to generate a different output format. The template is the only thing that knows what pytest looks like.

What It Doesn't Do (Yet)

Postman environments (the .postman_environment.json file)
OAuth 2.0 flows
Pre-request scripts
Response body assertions

These are all solvable. v1.0 is small enough to be trustworthy. I'd rather you use it and tell me what's missing than promise features I haven't built.

36 Tests, Because Eating Your Own Dogfood Matters

pip install postman2pytest pytest
pytest tests/ -v  # 36 passed

CI runs on Python 3.10, 3.11, and 3.12 via GitHub Actions.

Why Not Just Use Newman?

Newman runs your Postman tests. That's useful. But it doesn't generate code; it generates a report. When the test fails in CI, Newman tells you it failed. pytest tells you it failed, shows you the diff, lets you add fixtures, parametrize the case, integrate with your existing test infrastructure.

If your team runs pytest for unit tests, integration tests, and contract tests, having your API smoke tests in the same runner means a single command, one report, integrated into the existing CI step.

GitHub: github.com/golikovichev/postman2pytest
PyPI: pypi.org/project/postman2pytest

If you hit a collection format this doesn't handle, open an issue.