DEV Community: AI Dev Hub

Zod vs Pydantic vs Ajv: I ran one broken schema in all 3

AI Dev Hub — Tue, 15 Sep 2026 14:00:04 +0000

Zod vs Pydantic vs Ajv: I ran one broken schema in all 3

Ajv. It was the only one of the three that rejected my malformed tool schema, because it validates the schema document itself rather than validating data against it. Zod prevents the bug by construction if you author in TypeScript, and Pydantic has by far the best runtime error messages. None of them knew anything about provider-specific rules, which is where my actual bad afternoon came from.

Disclosure first: the structured output validator I link to below is one I built. I got there after running an Anthropic tool definition through four generic JSON Schema tools and finding that not one of them knew input_schema from parameters. It's free, runs client side, no signup, nothing gets uploaded anywhere. If you know a better one, tell me and I'll link that instead.

The task: one tool definition, two different questions

On Tuesday, August 18, 2026, an invoice extraction agent I maintain booked a credit note as a charge. A customer watched $91.64 land on the wrong side of their ledger. Out of 1,247 extraction calls that week, 3 came back with a negative total, and the field I was certain had been guarding against exactly that looked like this:

"total": { "type": "number", "exclusiveMinimum": "0" }

The zero is a string. JSON Schema says exclusiveMinimum takes a number, so what I shipped was a keyword with an invalid value, which most runtimes quietly skip over. It sat there for 11 days.

Once I stopped being annoyed at myself, I noticed I'd been collapsing two questions into one. Question one: is this schema document legal, and legal for the endpoint I'm posting it to? Question two: does the JSON the model sent back match it? I had only ever automated the second one.

So I took the same broken tool definition and pushed it through the three validators I reach for most: Ajv 8.17 on Node 22, Zod 4, and Pydantic 2.11. Same schema, same sample payload (a credit note with total: -91.64), same question each time. Does anything warn me before this reaches production?

Ajv: the only one that read the schema as a document

Ajv does something the other two don't. Before it looks at any data, it compiles the schema, and by default it validates that schema against the JSON Schema meta-schema. Your schema is data too. That's the whole trick, and it's why Ajv was the only tool here that said a word.

// npm i ajv@8
// node validate-tool-schema.mjs
import Ajv2020 from "ajv/dist/2020.js";

const inputSchema = {
  type: "object",
  properties: {
    invoice_id: { type: "string" },
    line_items: {
      type: "array",
      items: {
        type: "object",
        properties: {
          sku: { type: "string" },
          qty: { type: "integer", minimum: 1 }
        },
        required: ["sku", "qty"],
        additionalProperties: false
      }
    },
    total: { type: "number", exclusiveMinimum: "0" }
  },
  required: ["invoice_id", "line_items", "total"],
  additionalProperties: false
};

const ajv = new Ajv2020({ strict: true, allErrors: true });

try {
  const validate = ajv.compile(inputSchema);
  const ok = validate({ invoice_id: "INV-4471", line_items: [], total: -91.64 });
  console.log(ok ? "payload ok" : validate.errors);
} catch (err) {
  console.error("schema rejected:", err.message);
}

Output: schema rejected: schema is invalid: data/properties/total/exclusiveMinimum must be number. Four minutes from npm i to that line. Had those four minutes existed as a pre-commit hook back in July, the credit note would have bounced.

Ajv's data errors are less pleasant. You get instancePath: "/line_items/0/qty" and message: "must be >= 1", precise and joyless, and turning that into something a model can act on is your job. strict: true also gets opinionated about unknown keywords, which flagged two example fields I'd copied straight out of a docs page. Mildly irritating. Technically correct.

Zod: the bug can't happen, until you need the other direction

You cannot write exclusiveMinimum: "0" in Zod. There's no syntax for it. You write z.number().positive() and the broken version simply isn't expressible, so a whole category of typo stops existing. Since March 2026 every TypeScript agent I've started defines its tools in Zod first and emits JSON Schema with z.toJSONSchema(), and I haven't hand-written a tool schema in that codebase since.

The trouble starts when the schema isn't yours. Most of the tool definitions I deal with now arrive from somewhere else: an MCP server's inputSchema, a partner's OpenAPI fragment, something a coworker generated with a model at 2am. To check those with Zod you need a converter, and converters are lossy in the direction that hurts. I ran the broken schema through json-schema-to-zod and got back z.number(). Clean. No warning. The bad keyword had been dropped on the floor, and the resulting type happily accepted -91.64.

I don't know whether that's a deliberate be-generous-with-input choice or just unimplemented. Either way, silently discarding a keyword is precisely the failure I was hunting, so Zod scored zero on question one through no real fault of its own. Wrong layer for the job.

One wrinkle to plan for: z.toJSONSchema() factors reused sub-schemas into $defs with $ref pointers. Perfectly legal. Not universally loved by strict function-calling modes, so I inline them before anything goes over the wire.

Pydantic: the error messages I actually wanted

Pydantic loses question one outright. It doesn't ingest foreign JSON Schema at all. model_json_schema() is a one-way street, and checking a handwritten document in Python means reaching for jsonschema.Draft202012Validator.check_schema(), which is a different library entirely.

On question two it wins, and it isn't close. A ValidationError hands you loc, msg, type, and input for every failure, and that's the only validator output I've managed to serialize and pass straight back to the model as a repair message with results I trust. Last month 44 responses failed validation in that pipeline and 41 were fixed on the first retry, after I started sending Pydantic's errors() list verbatim instead of my own tidy summary string. I never measured the before number, which I do regret.

Its generated schemas deserve a look before you send them. Optional fields come out as anyOf: [{...}, {"type": "null"}] and nested models land in $defs. Both correct. Both have been rejected by a strict mode at least once in my experience, usually around 4pm on a Friday.

Scores, and the one I'd actually use

Criterion	Ajv 8.17	Zod 4	Pydantic 2.11
Caught the malformed `exclusiveMinimum`	yes, at compile time	n/a, can't express it	no, won't read the schema
Data error precision	high, phrased for machines	compile time only	high, phrased for humans
Errors good enough to feed back to a model	no	no	yes
Knows provider tool-schema rules	no	no	no
Time to first useful error	4 minutes	20+ minutes (rewrite required)	6 minutes
Language	JS/TS	TS	Python

The split I've settled on isn't a single tool. In Node, author in Zod and emit with z.toJSONSchema(). Then push the emitted document through Ajv's compile() inside a test, which proves the thing is still legal after conversion. About 15 lines of test code total. In Python, check_schema() on anything handwritten and Pydantic on everything the model returns, with .errors() going directly into the retry prompt.

There's a fourth check none of the three perform. A schema can be flawless JSON Schema and still be wrong for the endpoint you're posting it to. Anthropic wants it under input_schema, OpenAI's function tools want it under parameters, and MCP spells the key inputSchema in camelCase. Same object, three different homes. Strict mode piles on more: every property has to be listed in required, and additionalProperties: false stops being optional. That gap is what I built the structured output validator to close, because I got tired of learning about it from a 400 response.

If you're only installing one thing today, install Ajv. It answers the question the other two structurally cannot, and it costs four minutes.

FAQ

Q: Can't I just send the schema and let the API reject it?
A: Partly. Both Anthropic and OpenAI refuse some malformed definitions at request time, and strict mode refuses more. A keyword with a wrong value type is the case that has slipped past me. Valid-enough JSON to accept, meaningless enough to ignore. And a 400 in staging is a much slower loop than a red test.

Q: Does Ajv catch every schema mistake?
A: No. It catches illegal JSON Schema. It has nothing to say about a schema that's legal and wrong, like qty typed as a string, or a required field the model has never once produced. Sample payloads and evals cover that.

Q: Zod or Pydantic for a new agent in 2026?
A: Whichever language the rest of your service already speaks. Genuinely. I watched a team stand up a Python sidecar purely to get Pydantic errors, and the deploy complexity cost more than the errors were worth.

Q: Doesn't constrained decoding make this moot?
A: It removes parse failures, which is most of the day-to-day pain. It can't tell you your schema encodes the wrong rule. Constrained decoding against exclusiveMinimum: "0" produces beautifully formatted negative totals.

Written with AI assistance and human review. Try the tool at aidevhub.io/structured-output-validator.

Building an API changelog with GitHub REST API in 2026

AI Dev Hub — Thu, 10 Sep 2026 14:00:04 +0000

Building an API changelog with GitHub REST API in 2026

Build an endpoint changelog by fetching two OpenAPI specs through the GitHub REST API, indexing operations by HTTP method and path, and comparing those indexes. The script below produces Markdown you can attach to a release review. It detects added, removed, and modified operations within a deliberately limited scope; compatibility checks need a separate pass.

The OpenAPI Changelog Generator I link to below is one I built. I tried three alternatives that all left me digging through schema changes to assemble an endpoint list. That gap annoyed me. You don't need to upload either spec to follow this walkthrough; the script runs locally. If you have a better one, tell me.

The goal: a changelog someone can review

The useful output fits in a pull request comment: a heading naming the revisions, followed by a short list of endpoint changes. Something like Added: POST /invoices, with a removed route immediately below it. A reviewer should be able to scan that list before opening the underlying specification diff.

As of September 9, 2026, I still prefer a plain Markdown artifact for this job. It survives copying into release notes, and nobody needs access to another dashboard to read it.

The question we're answering is narrow: which endpoint definitions changed between these two revisions?

An endpoint here means an HTTP method plus its literal path template. GET /invoices/{id} and DELETE /invoices/{id} are separate entries. Renaming {id} to {invoice_id} appears as a removal and an addition. That's intentional for this small implementation, although a human might describe it as one rename.

We're using the GitHub REST API to retrieve repository files at two refs. Python handles the comparison locally. This approach is useful when your specification already lives beside the service code and you want a repeatable release artifact without installing a diff service.

There's a boundary worth setting early: this script compares operation objects and selected inherited fields. It doesn't resolve schema references or decide whether a consumer will break. A green run means the comparison completed. It says nothing about backward compatibility.

Setup and auth

You'll need Python 3.10 or newer. There are no packages to install.

Save the code below as changelog.py. Running python changelog.py uses two embedded fixtures, so you can inspect the output before touching a repository or creating a credential.

For repository mode, pass four arguments: repository, specification path, base ref, and target ref. For example, python changelog.py acme/billing-api openapi.json v1.8.2 v1.9.0 works once those names match your repository.

Use commit SHAs when you need reproducible output. A branch can move between requests, and fetching two files from moving branches gives you a comparison whose inputs may be difficult to reconstruct later.

Public repositories usually work without authentication, subject to GitHub's unauthenticated rate limit. For a private repository, put a fine-grained token in the GITHUB_TOKEN environment variable and grant it Contents read access to that repository. Organization policies may require approval before the token works.

Don't paste a token into the script. Set the environment variable through your shell's secret mechanism or your CI platform's secret store. The code sends it in the authorization header and never prints it.

The OpenAPI Changelog Generator is where I use the same two-spec comparison idea for an endpoint-focused review. This tutorial uses GitHub's documented API directly; it doesn't depend on a hosted API for the generator.

One input constraint saves a surprising amount of setup: both files must be JSON. An OpenAPI document can be YAML, but Python's standard library doesn't include a YAML parser. Convert YAML during your existing build, or extend the loader with a parser you already trust.

The repository path must also exist at both refs. A renamed specification needs separate paths, which this version intentionally leaves out.

The core code

The Contents API accepts a ref query parameter. We request the raw file representation, which avoids decoding the base64 wrapper returned by the default representation.

The script sorts operation keys so the report stays stable between runs. Dictionary comparison ignores JSON object key order, while list order still affects equality.

import json
import os
import sys
from urllib.error import HTTPError, URLError
from urllib.parse import quote, urlencode
from urllib.request import Request, urlopen

METHODS = {"get", "put", "post", "delete", "options",
           "head", "patch", "trace"}


def fetch_spec(repo, path, ref):
    parts = repo.split("/")
    if len(parts) != 2 or not all(parts):
        raise ValueError("Repository must have the form owner/repo")
    repository = "/".join(quote(part, safe="") for part in parts)
    file_path = quote(path, safe="/")
    url = (
        f"https://api.github.com/repos/{repository}/contents/{file_path}"
        f"?{urlencode({'ref': ref})}"
    )
    headers = {
        "Accept": "application/vnd.github.raw+json",
        "X-GitHub-Api-Version": "2022-11-28",
        "User-Agent": "endpoint-changelog",
    }
    token = os.environ.get("GITHUB_TOKEN")
    if token:
        headers["Authorization"] = f"Bearer {token}"
    # A timeout prevents a stalled fetch from hanging the job indefinitely.
    with urlopen(Request(url, headers=headers), timeout=30) as response:
        return json.load(response)


def operations(spec):
    if not str(spec.get("openapi", "")).startswith("3."):
        raise ValueError("Expected an OpenAPI 3.x document")
    result = {}
    for path, item in spec.get("paths", {}).items():
        # Resolving Path Item references needs a separate resolver.
        if "$ref" in item:
            raise ValueError(f"Resolve the Path Item reference at {path}")
        for method, operation in item.items():
            if method not in METHODS:
                continue
            result[f"{method.upper()} {path}"] = {
                "operation": operation,
                "path_parameters": item.get("parameters", []),
                # An explicit empty list disables inherited security.
                "security": operation.get("security", spec.get("security", [])),
            }
    return result


def changelog(before, after):
    old, new = operations(before), operations(after)
    lines = []
    for key in sorted(old.keys() | new.keys()):
        if key not in old:
            lines.append(f"- Added: `{key}`")
        elif key not in new:
            lines.append(f"- Removed: `{key}`")
        elif old[key] != new[key]:
            lines.append(f"- Modified: `{key}`")
    return "\n".join(lines) or "No changes within the comparison scope."


def demo():
    def document(paths):
        return {
            "openapi": "3.0.3",
            "info": {"title": "Billing API", "version": "1.0.0"},
            "paths": paths,
        }

    old = document({
        "/invoices": {"get": {"responses": {"200": {"description": "OK"}}}},
        "/legacy": {"get": {"responses": {"200": {"description": "OK"}}}},
    })
    new = document({
        "/invoices": {
            "get": {"responses": {
                "200": {"description": "OK"},
                "429": {"description": "Rate limited"},
            }},
            "post": {"responses": {"201": {"description": "Created"}}},
        },
    })
    return old, new


def main(args):
    if not args:
        before, after = demo()
        base, target = "demo-before", "demo-after"
    elif len(args) == 4:
        repo, path, base, target = args
        before = fetch_spec(repo, path, base)
        after = fetch_spec(repo, path, target)
    else:
        raise ValueError("Usage: changelog.py [owner/repo path base target]")
    print(f"# Endpoint changes: {base} to {target}\n")
    print(changelog(before, after))


if __name__ == "__main__":
    try:
        main(sys.argv[1:])
    except HTTPError as error:
        sys.exit(f"GitHub returned HTTP {error.code}; check access and refs.")
    except (URLError, ValueError, OSError) as error:
        sys.exit(f"Changelog failed: {error}")

The demo should produce three entries. GET /invoices is modified because its responses now include HTTP 429. POST /invoices is added. GET /legacy is removed.

That makes the fixture useful as a quick sanity check: each branch of the comparison has a visible result. It isn't a substitute for tests against your own document shapes.

For a Markdown file, redirect stdout with python changelog.py > changelog.md. Repository mode supports the same redirection. Errors go to stderr and produce a nonzero exit status, so a failed download doesn't masquerade as an empty changelog.

The script includes path-level parameters because those apply across operations. It also checks inherited security requirements. An explicit security: [] overrides global security, so the lookup must preserve that empty list.

Notice what happens to descriptions. Editing an operation's description produces a modified entry. That's useful for a literal definition changelog, though it may be noisy for release notes. If you remove documentation fields before comparison, make that policy explicit and apply it recursively where intended.

How this compares with other approaches

GitHub's API supplies versioned inputs. It doesn't understand OpenAPI compatibility. The comparison step determines how much meaning you get from those inputs.

For a repository-based workflow, these are the tradeoffs I'd actually consider:

Approach	Input access	Meaning of a change	Dependency cost	Best fit
GitHub REST API plus this script	Repository files at explicit refs	Literal operation changes within the stated scope	Python standard library and optional token	Small endpoint reports
Local Git plus `git diff`	Existing checkout and history	Text changes, including formatting	Git and a checkout	Inspecting exact source edits
GitHub REST API plus oasdiff	Downloaded specs passed to oasdiff	OpenAPI-aware reports and compatibility checks	Separate CLI and its configuration	Release gates and deeper review

I wouldn't introduce API fetching into a CI job that already has both revisions checked out. Read the files locally and reuse changelog(). That removes network failure from the comparison step and avoids spending GitHub API quota.

The API route earns its place in a release helper that operates without a checkout, or in a central job that reads specifications from several repositories.

For compatibility enforcement, I'd use an established OpenAPI diff engine and review its configuration. Required request properties and response schema changes deserve more analysis than Python object inequality provides.

What went wrong the first time

My first pass at the comparison was just the operation dictionary. Too narrow.

Consider a required header defined under the Path Item's parameters. Every operation under that path inherits it. Comparing only the nested get or post value misses the header change completely. Including path-level parameters fixes that omission, although this simple approach can overreport changes when an operation overrides the same parameter.

Security had a similar trap. Falling back with operation.get("security") or global_security would treat an empty list as absent. That changes the meaning of the document. The explicit default argument in the code preserves the override.

The larger unresolved problem is $ref. If an operation refers to #/components/schemas/Invoice, changing that component leaves the reference string identical. This script won't report the affected endpoint unless something else in its comparison snapshot changes.

Don't patch that by copying the entire components object into every operation. One unrelated schema edit would mark every endpoint as modified. Resolving references properly requires dependency tracking, including cycle handling, or an existing comparison engine that already understands those relationships.

Server URLs are outside this implementation's scope too. So are OpenAPI 3.1 webhooks. Keep those limitations attached to the report if teammates could mistake it for a complete contract review.

A fetch can also fail before comparison begins. GitHub may return 404 for a private repository your token can't access, so check repository permissions as well as spelling. Large specification files encounter Contents API limits; this loader is intended for ordinary JSON specs, not arbitrary repository blobs.

Before adding it to a release job, I'd check one real removed endpoint and one change inside a referenced schema. The first should appear. The second demonstrates the current blind spot.

That's the point where I'd decide whether this endpoint list is enough for the release reviewer or whether the job needs a semantic diff engine. The small script gives you a readable artifact today, with a clear boundary around what it can claim.

Written with AI assistance and human review. Try the tool at aidevhub.io/openapi-changelog.

Reading Claude message_stream events without guessing in 2026

AI Dev Hub — Tue, 08 Sep 2026 14:00:05 +0000

Reading Claude message_stream events without guessing in 2026

Log the raw SSE frames, then replay them through a parser that tracks content_block indices. Anthropic's stream is a typed event sequence, so text deltas, input_json_delta fragments for tool calls, and thinking blocks all arrive interleaved on separate indices. Reconstructing the message means grouping by index, not concatenating in arrival order.

Quick disclosure before anything else: the stream event viewer I link to further down is one I built. I'd been pasting SSE dumps into three different generic JSON formatters and a browser devtools panel, and every one of them choked on the fact that a message stream isn't one JSON document, it's a few hundred of them separated by data: lines. None of them knew what a content_block_delta was. Mine is free, runs client-side, has no signup, and uploads nothing. If you already use something better, please tell me what it is.

The bug that cost me a Tuesday afternoon

On March 12, 2026 I shipped a streaming endpoint that proxied Claude responses to a web client. It worked in every test I wrote. It broke in production for roughly 1 in 40 requests, and the failures were always the same shape: the assistant's answer would come through with a chunk of JSON spliced into the middle of a sentence.

My handler was doing the naive thing. For every content_block_delta it grabbed delta.text if present, delta.partial_json otherwise, and appended both to one buffer. That's fine as long as the model produces exactly one content block. The moment it emitted a short text preamble, then a tool_use block, then more text, my buffer became a blender.

I spent about 47 minutes staring at the wrong layer. I assumed the SDK was mis-ordering events, or that my reverse proxy was reassembling chunks badly. Neither. The events arrived in perfect order. I was throwing away the one field that mattered, which is index.

What finally fixed it was dumping the raw stream to a file and reading it end to end. Not the parsed objects my code produced, the actual bytes on the wire. That's a boring debugging move and it's the one I keep forgetting to do first.

What the event sequence actually looks like

A single streamed call from the Messages API is a fixed skeleton with a variable middle. You get message_start once, carrying the message envelope and the initial usage object. Then, for each content block, a content_block_start with an index and a stub of the block, a run of content_block_delta events on that same index, and a content_block_stop. At the end, message_delta carries stop_reason and the final output_tokens, followed by message_stop.

The deltas are typed, and the type tells you which field to read:

text_delta has .text
input_json_delta has .partial_json (a raw string fragment, only valid JSON once the whole block is concatenated)
thinking_delta has .thinking
signature_delta closes out an extended-thinking block

The partial_json one bites people. Each fragment is a slice of a JSON string, so {"loc and ation":" and Berlin"} arrive as three separate events. Parse them individually and you get an exception. Concatenate them across the block's entire lifetime and you get valid input for the tool call.

The other thing worth going after is usage accounting. message_start gives you input_tokens, cache_creation_input_tokens, and cache_read_input_tokens. On a cached run I checked last Tuesday, a request reported 218 input tokens and 14,208 cache read tokens. If you only log input_tokens you will look at that call and conclude it was nearly free, which is true, but you'll have no idea why, and no way to tell when your cache starts missing.

How the parser puts a call back together

Here's a small script that takes a saved SSE dump and reconstructs the whole call. It's the same logic the viewer runs, minus the UI. Save your stream by writing every raw line from the response body to a file first.

import json, sys

def parse_stream(path):
    text_parts, tool_json, usage = [], {}, {}
    blocks, stop_reason = {}, None
    with open(path) as fh:
        for line in fh:
            line = line.strip()
            if not line.startswith("data:"):
                continue
            ev = json.loads(line[5:].strip())
            t = ev.get("type")
            if t == "message_start":
                usage.update(ev["message"].get("usage", {}))
            elif t == "content_block_start":
                blocks[ev["index"]] = ev["content_block"]
                if ev["content_block"]["type"] == "tool_use":
                    tool_json[ev["index"]] = ""
            elif t == "content_block_delta":
                d = ev["delta"]
                if d["type"] == "text_delta":
                    text_parts.append(d["text"])
                elif d["type"] == "input_json_delta":
                    tool_json[ev["index"]] += d["partial_json"]
            elif t == "message_delta":
                stop_reason = ev["delta"].get("stop_reason")
                usage.update(ev.get("usage", {}))
    tools = {i: json.loads(s) for i, s in tool_json.items() if s}
    return {"text": "".join(text_parts), "tools": tools,
            "stop_reason": stop_reason, "usage": usage}

if __name__ == "__main__":
    print(json.dumps(parse_stream(sys.argv[1]), indent=2))

Run it with python replay.py stream.txt and you get the assembled text, every tool call's fully-parsed input keyed by block index, the stop reason, and a merged usage object. Fifty lines, no dependencies.

Two details in there that took me longer than they should have. The line[5:] slice assumes data: with no space, which is why the .strip() follows it. And usage.update() on message_delta is deliberate: the final event only carries output_tokens, so updating rather than replacing keeps the input and cache counts from message_start intact. I got that backwards on my first pass and every call reported zero input tokens.

How it compares to what I tried first

I went through four options before writing my own. The comparison below reflects what each one did with a 900-line dump containing two text blocks and one tool call.

Tool	Understands typed events	Rebuilds tool_use JSON	Cache token breakdown	Data stays local
Generic JSON formatter	No (fails on multi-doc SSE)	No	No	Usually
Browser devtools EventStream tab	Shows frames only	No	No	Yes
Hand-rolled `jq` pipeline	With enough effort	Manual concat	Manual	Yes
Anthropic Stream Event Viewer	Yes	Yes	Yes	Yes, client-side

The devtools EventStream tab is genuinely useful and I still open it, just for a different job. It shows you that frames arrived and in what order. It won't tell you the tool input was truncated because the model hit max_tokens mid-JSON, which is a real failure mode and shows up as stop_reason: "max_tokens" with an unparseable partial_json accumulation.

The jq route works. I have a 12-line pipeline in a gist that handles text deltas fine. Extending it to group tool JSON by index is where I gave up and wrote actual code.

When you shouldn't bother with this

If you're using the official SDK's client.messages.stream() helper and you never touch tool use, skip all of it. The SDK accumulates the final message for you, exposes .text_stream for the simple case, and handles indices correctly. Reaching for a raw event parser there is work you don't need.

Same if your bottleneck is latency rather than correctness. An event viewer tells you what came back, not how fast. For time-to-first-token you want timestamps recorded at the socket, and no post-hoc replay of a saved dump will give you those.

And if your streams carry customer data under a policy that forbids pasting content into any web page, use the script above locally instead. The viewer runs entirely in your browser and sends nothing anywhere, but "trust me, it's client-side" is not a compliance argument, and I wouldn't expect anyone to accept it as one. Read the network tab or run the 50 lines yourself.

The case where this pays off is the messy middle: you're building on the raw HTTP API, or through a gateway that reshapes events, or debugging why a tool call sometimes arrives with an empty input object. That last one turned out, for me, to be a proxy that buffered and split SSE frames at 4,096 bytes without respecting event boundaries. I would never have found it from parsed objects.

FAQ

Q: Do I need to log the raw stream, or can I feed it the SDK's parsed events?
A: Raw is better. The SDK's accumulated message hides exactly the ordering and index information you're trying to inspect. Write the response body lines to a file before anything parses them.

Q: Why does partial_json fail to parse on its own?
A: Because each delta is an arbitrary byte slice of the tool input JSON, split wherever the token boundary landed. Only the concatenation of every fragment in that block is valid JSON.

Q: What does stop_reason: "tool_use" mean versus "end_turn"?
A: tool_use means the model stopped because it wants a tool result back before continuing. Send the result as a tool_result block in the next user turn. end_turn means it finished on its own.

Q: Where do cache write and cache read counts show up?
A: In the usage object on message_start, as cache_creation_input_tokens and cache_read_input_tokens. They're separate from input_tokens, so summing all three gives you the real prompt size.

Q: Does the viewer work with streams from Bedrock or Vertex?
A: Mostly. The event types match, though the envelope framing differs by platform, so you may need to strip a wrapper before pasting.

Written with AI assistance and human review. Try the tool at aidevhub.io/anthropic-stream-event-viewer.

8 free AI agent tools I use in 2026

AI Dev Hub — Thu, 03 Sep 2026 14:00:02 +0000

8 free AI agent tools I use in 2026

Eight, all browser-based, none of them asking for a signup: agent-skill-validator, skill-scope-collision-detector, skill-payload-budget-optimizer, trace-failure-classifier, tool-approval-matrix-compiler, skill-spec-converter, skill-regression-suite-builder, and skill-release-canary-planner. Between them they cover the unglamorous work of shipping agent skills: checking manifests, catching overlapping triggers, trimming startup context, reading failure traces. Comparison table with dealbreakers is further down.

Being upfront: the tools I link to below are ones I built. I tried six existing skill linters in January 2026 and every one of them wanted my repo uploaded to somebody's server before it would tell me a frontmatter field was missing. Mine run client-side, cost nothing, and don't ask for an account or an email address. If you know something better, tell me and I'll happily switch.

Why agent skills got messy in 2026

In February I took over a repo with 23 skill definitions in it. Nobody had touched the docs since October. My first bug report was a support agent that kept picking the wrong skill for anything containing the word "invoice", because two separate skills claimed that word in their trigger text and the model just guessed.

Finding that took me 47 minutes. Fixing it took four seconds.

That ratio is the whole problem. Agent skills are markdown files with frontmatter. There's no compiler. Nothing type-checks them. You discover a mistake when a production trace goes sideways at 2am, and by then you're squinting at a JSON blob trying to work out which of your 23 markdown files talked the model into calling refund_customer instead of fetch_invoice.

The tooling gap is real and nobody's filling it especially well. Most of what exists is either bundled into a platform you have to adopt wholesale, or it's a $200/month observability product that draws nice graphs of failures you already knew about. What I wanted was eslint for a folder of markdown. Runs in two seconds, tells me line 238 is broken, needs no account.

The tools I actually open every week

agent-skill-validator

I run this before commits. Paste a SKILL.md, get back the structural problems: missing description, a name that doesn't match its folder, allowed-tools entries pointing at tools that don't exist in your config, YAML that parses cleanly while meaning something other than what you intended.

The check that earns its keep is the description one. A skill description is the only thing the model sees when deciding whether to load your skill, and roughly half the ones I've inherited read like internal API docs ("Handles the invoice subsystem"). The validator flags descriptions below a length threshold and descriptions containing no trigger language. It's a dumb heuristic. It has also been right every single time for me.

I was wrong about this tool at first. Frontmatter validation seemed too trivial to bother with. Then I lost a morning to a skill that never loaded because I'd typed allowed_tools instead of allowed-tools, and YAML was perfectly happy to hand me a key that nobody read.

skill-scope-collision-detector

Paste in every skill description you've got, get back a matrix of which pairs overlap and on which words. This is the one that would have saved me those 47 minutes in February.

It runs on trigger-term overlap plus a similarity score, so it catches the obvious case (two skills that both say "use this for PDF extraction") and also the sneaky case, where skill A says "customer records" and skill B says "user accounts" and your model treats those as the same concept because of course it does.

On that 23-skill repo it surfaced 6 collisions. Four were real. Two were fine, because surrounding context disambiguated them. That hit rate is about what I'd expect, and honestly it's plenty. I don't need precision here. I need a short list of things to eyeball.

skill-payload-budget-optimizer

Every skill you register costs context before the user has typed anything at all. Nobody tells you the running total, so it creeps.

Before I built the optimizer I was doing this with a script, which I'll leave here because it's a decent sanity check even if you'd rather not open another browser tab:

#!/usr/bin/env python3
"""Rough context cost of every SKILL.md under a directory, biggest first."""
import pathlib
import re
import sys

root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".")
rows = []

for path in root.rglob("SKILL.md"):
    text = path.read_text(encoding="utf-8")
    body = re.sub(r"\A---.*?^---\s*", "", text, flags=re.S | re.M)
    rows.append((len(body) // 4, str(path)))

rows.sort(reverse=True)
for approx_tokens, path in rows:
    print(f"{approx_tokens:>7,}  {path}")
print(f"{sum(t for t, _ in rows):>7,}  TOTAL across {len(rows)} skills")

That reported 18,400 tokens across 23 skills on the repo I mentioned. The optimizer got it down to 11,200 without deleting a single skill, mostly by moving verbose examples out of description fields and into skill bodies, where they only load on demand.

tool-approval-matrix-compiler

Feed it your skills and your tool list, get a grid of which skill is allowed to call what. Then you stare at the grid and say "hang on, why can the changelog writer call the deploy tool".

That's the entire value. A read-only view of permissions you already configured, arranged so a human can spot the wrong ones. I found two over-broad grants in about a minute, both of them me being lazy months earlier and pasting an allowed-tools list from one skill into another.

If you're shipping agents that touch anything with side effects, do this once a quarter. It's boring and takes ten minutes.

trace-failure-classifier

Paste a failed agent trace, get the failure bucketed: wrong skill selected, right skill with bad arguments, tool errored and the model carried on regardless, model looped, context truncated mid-task.

Those buckets matter more than they sound like they should. "The agent failed" isn't actionable. "The agent picked the right skill, then passed a date in the wrong format three times without noticing the tool error" tells you which line to open.

I've fed it around 60 traces since April. It lands the right bucket most of the time, and when it's unsure it says so rather than confidently inventing a story, which I appreciate more than I expected to.

All eight, side by side

Tool	Best for	Pricing	The one dealbreaker
agent-skill-validator	Pre-commit frontmatter checks	Free, client-side	Structural checks only, semantics are on you
skill-scope-collision-detector	Finding overlapping trigger text	Free, client-side	Needs every description pasted at once, no repo crawl
skill-payload-budget-optimizer	Cutting startup context cost	Free, client-side	Token counts are approximate, not per-model exact
trace-failure-classifier	Triaging production agent failures	Free, client-side	One trace at a time, no batch mode
tool-approval-matrix-compiler	Auditing which skill calls what	Free, client-side	Shows you the problem, you apply the fix by hand
skill-spec-converter	Moving skills between agent frameworks	Free, client-side	Round trips drop custom fields
skill-regression-suite-builder	Generating test cases from a spec	Free, client-side	Generated cases need a human pass before they're worth running
skill-release-canary-planner	Staging a skill rollout across users	Free, client-side	Assumes you can already segment traffic

All eight sit on one page at aidevhub.io/tools/ai, which is where I keep the ones I bookmark. The bottom three in that table get pulled out maybe monthly, so they didn't earn a section above. The spec converter in particular is a thing you use twice and forget until the next migration.

The one I ditched

A hosted eval platform. $91.64 a month including tax, an odd enough number that I remember it exactly. I ran it from December 2025 through March 2026 and cancelled on a Tuesday afternoon while looking at the invoice.

It was a good product. It answered a question I didn't have. It could tell me my agent's success rate slid from 94% to 89% over a week. Fine. What it couldn't tell me was that the slide happened because someone added a skill whose description overlapped an existing one, which is exactly what had happened, twice.

Most agent observability tooling right now measures outcomes at a level of abstraction too high to act on. I need to know which markdown file to edit. A line trending downward doesn't get me there, and paying $91.64 a month for that line felt worse every time I opened it.

I dropped my own bash-and-python token counter too, the ancestor of the script above, once I got tired of maintaining an estimator that ran about 15% off in a direction I couldn't predict. Keeping it in this post regardless, since it's a fair first thing to run on a repo you've just inherited.

FAQ

Q: Do these upload my skill files anywhere?

A: No. They run inside the browser tab. I built them that way because I wasn't allowed to paste work skill definitions into a third-party server, and I assume plenty of people are in the same spot. Open devtools and watch the network panel if you want to verify that, it's a reasonable thing to check.

Q: Do they work with skills that aren't Claude Code skills?

A: Mostly. The validator and the collision detector care about frontmatter and description text, which most agent frameworks have in some shape. The spec converter exists specifically for moving between formats. The approval matrix compiler assumes a per-skill tool allowlist, so if your framework handles permissions globally it won't tell you much.

Q: How accurate is the token counting?

A: Close enough to make decisions with. Too rough to bill against. It's a character-based approximation, so it drifts a few percent on text heavy with code or non-English content. For exact figures, run the text through your provider's tokenizer.

Q: Fastest way to get value out of this list on an existing repo?

A: Collision detector first, payload budget second. Those two find problems that are already costing you something today. Validation is a pre-commit habit, and it pays off going forward rather than retroactively.

Written with AI assistance and human review. Try the tool at aidevhub.io/tools/ai.

Generating 10,000 UUIDs without leaving the browser in 2026

AI Dev Hub — Tue, 01 Sep 2026 14:00:06 +0000

Generating 10,000 UUIDs without leaving the browser in 2026

Use crypto.randomUUID() in any modern browser console for quick v4s, and a client-side bulk generator when you need thousands at once or UUID v7 ordering. v4 is 122 random bits. v7 puts a 48-bit Unix timestamp up front, so rows sort by creation time and your database index stays happy. For new primary keys in 2026, default to v7.

Quick disclosure: the UUID generator I link to below is one I built. Back in May I tried seven online generators during a database migration, and every one either capped bulk output at 100, skipped v7 entirely, or buried the copy button under ads. Mine is free and runs entirely client-side. No signup, and nothing you generate ever leaves your machine. If you know a better one, tell me in the comments.

The 11pm seed script that started this

Three weeks ago, on August 5th, I was putting together a demo environment that needed 8,500 fixture rows spread across four tables. The schema uses UUID primary keys, and the fixture data lives in a spreadsheet a teammate on the solutions side maintains. So I needed 8,500 UUIDs in a spreadsheet column before the next morning.

The terminal was my first stop. for i in {1..8500}; do uuidgen; done runs fine, but macOS prints uppercase UUIDs while our snapshot tests normalize everything to lowercase, so the first diff was thousands of lines of pointless noise. Piping through tr '[:upper:]' '[:lower:]' fixed that. Then my teammate had to regenerate two of the tables the next day on a Windows laptop with no WSL, where uuidgen doesn't exist, and my clever one-liner helped nobody. Total damage: 47 minutes on a task that deserved 30 seconds.

There was a second, quieter problem. Those were all v4 UUIDs, which are pure randomness, and random primary keys scatter inserts across the whole index. Each new row lands on a random B-tree page, so caches stay cold and large tables grow bloated indexes. UUID v7, standardized in RFC 9562 back in May 2024, fixes this by putting a millisecond timestamp in the first 48 bits, so new keys always sort after old ones and inserts append instead of scattering. Postgres 18 shipped a native uuidv7() function in September 2025, and at this point I treat v7 as the boring default for any new table.

What surprised me is that most online generators still don't offer v7 at all. That gap is why this article, and the tool it reviews, exists.

What's inside a UUID v7, and how to build one

The layout is simple. First 48 bits: Unix timestamp in milliseconds. Then 4 version bits, 12 random bits, 2 variant bits, and 62 more random bits. That leaves 74 bits of randomness per millisecond, which is plenty for any workload I've ever touched.

Here's a complete v7 implementation that runs in any modern browser console or in Node 19 and newer:

function uuidv7() {
  const bytes = crypto.getRandomValues(new Uint8Array(16));
  const ts = BigInt(Date.now());
  for (let i = 0; i < 6; i++) {
    bytes[i] = Number((ts >> BigInt(40 - i * 8)) & 0xffn);
  }
  bytes[6] = (bytes[6] & 0x0f) | 0x70; // version 7
  bytes[8] = (bytes[8] & 0x3f) | 0x80; // RFC 9562 variant
  const hex = [...bytes].map(b => b.toString(16).padStart(2, "0")).join("");
  return [hex.slice(0, 8), hex.slice(8, 12), hex.slice(12, 16),
          hex.slice(16, 20), hex.slice(20)].join("-");
}

console.log(uuidv7());
// 01a04a2e-9d10-7c3b-a4f2-5b8e19c0d67d

Every v7 you generate this month starts with the same few hex characters (01a0 and change, if you're reading this in August 2026). That's the timestamp doing its job. Sort v7s as plain strings and you've sorted them by creation time, which is the entire trick.

Give or take a counter for ordering within the same millisecond, this is exactly what the generator at aidevhub.io/uuid-generator runs when you click generate. Everything happens client-side; the page never phones home with your output. You pick v4 or v7 and a count up to 10,000, then choose the format: lowercase or uppercase, hyphens or none, plain lines or a JSON array, and braces if you need the old Microsoft GUID registry style. Generating the full 10,000 takes about 40 milliseconds on my 2023 MacBook Air because there's no server round trip. My spreadsheet mess from August is now one copy button.

I'll admit I went back and forth on the bulk cap. 10,000 felt arbitrary. It still does, honestly, but every real use case I collected fit under it, and an unbounded loop in a browser tab is a crash waiting to happen.

How it stacks up against what you already have

You almost never need a website to mint one UUID. The interesting question is what to reach for when you need many of them, or v7 specifically, or a particular output format.

Option	v7 support	Bulk output	Format control	Where it runs
aidevhub UUID Generator	Yes	Up to 10,000	Case, hyphens, braces, JSON	Your browser, client-side
uuidgen (macOS/Linux)	Not on macOS	Shell loop	tr and sed by hand	Local terminal
npm uuid package	Yes (since v10)	Yes, in code	Whatever you write	Node or a bundler
Typical ad-supported sites	Sometimes	Often capped at 100	Rarely	Their server

Inside application code, the npm uuid package is the right answer, full stop. It's supported v7 since version 10 came out in June 2024, and it handles same-millisecond ordering with an internal counter, which my 15-line snippet above doesn't bother with. IDs born in a service should be minted by that service.

uuidgen is great when you're already in a terminal and want one ID. Newer util-linux builds can emit v7, but the macOS version tops out at random v4s (in uppercase, for reasons I've never understood), and a stock Windows machine doesn't ship it at all. Bulk means writing a loop plus a tr pipeline, which is exactly the 47-minute hole I fell into.

The ad-supported generator sites do work. My gripes are the caps (100 per click was the common ceiling when I surveyed seven of them in May), the thin v7 support, and the fact that your IDs get minted on someone else's server. That last one is mostly aesthetic, since a random identifier isn't a secret, though it does rule out offline use and it makes some corporate proxies grumpy.

When a browser generator is the wrong call

Don't pre-generate IDs for production inserts. If your application creates rows, the ID should be minted at insert time by the app or the database, where a library can guarantee uniqueness and monotonic ordering. A static list of UUIDs pasted into production code is a smell. Fixture files and one-off imports are the browser tool's territory; live traffic isn't.

Don't use UUIDs as secrets, either. A v4 has 122 random bits, which sounds like enough, but session tokens deserve a dedicated generator with no structural bits and a shape that secret scanners recognize. And v7 is actively worse for anything sensitive because it embeds its own creation time. Anyone who sees the ID can read when the row was made. I honestly don't know how much that matters for a typical app. My instinct says it's harmless for orders and uploads, and wrong for rows where the creation time is itself private, like medical records. When in doubt there, use v4.

If you need deterministic IDs, where the same input always produces the same UUID, you're looking for v5 with a namespace. That's hashing, and no random generator (mine included) can help you.

And if you work somewhere locked down enough that visiting a web page is a compliance conversation, offline uuidgen is still your friend. The tool keeps working with the network cable pulled, since it's all client-side JavaScript, but I've learned the hard way that policy doesn't always care about implementation details.

FAQ

Q: Do I need to worry about v4 collisions?

A: No. You'd need to generate about 103 trillion v4 UUIDs before the odds of a single duplicate reach one in a billion. If you ever see a real duplicate in the wild, the cause is a bug somewhere, like a copied row or a cloned VM with a frozen entropy pool.

Q: Should I migrate existing v4 primary keys to v7?

A: Almost certainly no. The index-locality win applies to new writes, and rewriting millions of existing keys (plus every foreign key that references them) is a migration with real risk and little payoff. Use v7 for new tables and let the old ones be.

Q: What's the database support story in 2026?

A: Postgres 18 has native uuidv7(). Older Postgres versions store v7 in the regular uuid type without complaint, so generate the values in your application. MySQL's UUID() still emits v1, so there you'd generate v7 in code as well.

Q: Is a GUID different from a UUID?

A: Same 128 bits. GUID is Microsoft's older name for it, traditionally printed uppercase inside braces, which is why the format controls include a braces option.

Written with AI assistance and human review. Try the tool at aidevhub.io/uuid-generator.

Stop breaking prod with a .env file manager in 2026

AI Dev Hub — Thu, 27 Aug 2026 14:00:06 +0000

Stop breaking prod with a .env file manager in 2026

Run your .env file through a real parser before it ships. Most .env disasters come from quoting rules that differ between dotenv and the shell. A client-side env file manager checks every line against explicit grammar rules and shows you what will break before you deploy. It also converts between JSON, YAML, Docker, and shell export formats, and it runs entirely in your browser.

Quick disclosure: the Env File Manager I link to below is one I built. I tried five online .env converters back in January and every single one shipped my paste off to a server (check the network tab, it's grim). Mine is free and fully client-side. There's no signup, and nothing you paste leaves the browser. If you know a better one, tell me in the comments and I'll link it instead.

The deploy that made me care about quoting rules

Back on July 14th, around 11pm, I deployed a change that rotated a Redis password. The new one came out of a password generator with a # in it. Locally everything passed, because python-dotenv only treats # as a comment when there's a space in front of it. In the container, where a Node service loaded the same file through the dotenv package, the value got cut off at the hash. Auth failures, but only in one service, and only after the pods recycled.

I spent 51 minutes bisecting a deploy that contained no bad code. The problem was line 17 of a 43-line .env file, and nothing in our pipeline considered it a problem. That's the part that stung. Every parser behaved exactly as its own docs said it would. They just don't agree with each other.

There's no spec for .env files. None. The format is folklore that Node's dotenv, python-dotenv, Ruby's dotenv, docker run --env-file, Docker Compose, and plain shell source each retell a little differently. Quoting, inline comments, variable expansion, multiline values: each of those behaves differently somewhere, and the differences only surface at runtime.

My first workaround was a pre-commit grep for suspicious characters. It false-positived constantly and nobody maintained it past week two, including me. My second workaround was "just double-quote everything," which is how I discovered that docker run --env-file keeps the quote characters as part of the value. We'll get to that.

What a strict .env parser actually checks

Here's the folk-wisdom way to load a .env file into your shell, next to what actually happens. This runs on any machine with bash and Docker:

cat > demo.env <<'EOF'
APP_NAME="my app"
REDIS_PASS=Tr0ub4dor#42
EOF

# The Stack Overflow classic:
export $(grep -v '^#' demo.env | xargs)
echo "$APP_NAME"
# prints: my
# xargs stripped the quotes, then word splitting ate the rest

# Docker does the opposite and keeps quotes as literal characters:
docker run --rm --env-file demo.env alpine sh -c 'echo "$APP_NAME"'
# prints: "my app"
# the quote characters are now part of the value inside the container

Same two-line file, and we're already at two contradictory readings before any application code runs. Node's dotenv adds a third: it truncates REDIS_PASS to Tr0ub4dor, because an unquoted # starts a comment there. python-dotenv keeps the full value, because the # has no space before it. Four consumers, four opinions about two lines.

A strict parser turns those ambient rules into visible ones. I got tired of holding them in my head, so I built Env File Manager to hold them for me. You paste a .env file (or JSON or YAML, if you're converting the other way) and it parses every line against an explicit grammar. Then it flags what will hurt you later: unquoted values containing #, duplicate keys (most loaders silently keep the last one), CRLF line endings from a Windows teammate, a stray BOM at the top of the file, spaces around the equals sign. You can edit values in a table view, see the raw text next to the decoded value, and fix things in place.

Under the hood it keeps one internal representation per entry: the key, the raw text, the decoded value, and any attached comment. Each export format then gets its own serializer that applies that format's escaping rules on the way out, which is the whole point. When you export for docker run, quotes get dropped because Docker would keep them literally. When you export a shell script, values get single-quoted with proper escaping, so MSG=hello world can't end up executing world as a command. YAML export quotes values like on and no so they don't silently turn into booleans.

I don't know why Docker never made --env-file parse quotes the way Compose does. I assume it's backwards compatibility, but I couldn't find a definitive answer in their issue tracker, and I did look for a whole evening.

How it stacks up against the usual suspects

The honest comparison is that most tools in this space solve adjacent problems, and I use two of them alongside my own thing.

Tool	What it is	Converts formats	Flags bad lines	Where your values go
Env File Manager	browser-based editor	.env, JSON, YAML, Docker, shell	yes, per line, with reasons	nowhere, it's client-side
dotenvx	CLI and runtime loader	no	only at decrypt time	encrypted file in your repo
direnv	shell hook	no	no	plaintext .envrc on disk
Doppler	hosted secrets manager	via CLI templates	schema checks, server-side	their servers

dotenvx is what I'd pick if the goal is committing encrypted .env files to the repo, and its runtime loader is genuinely solid. direnv does a different job: per-directory shell environments that happen to touch the same file format. Doppler and its hosted cousins solve team sync and rotation, which a browser tool never will and shouldn't pretend to. There are also a dozen paste-your-env converter sites out there; the five I tried in January all POSTed the textarea contents to a backend, which is how this whole project started.

The gap I kept hitting sits between all of those: the ten minutes where a config has to cross a format boundary without anything getting mangled. Last month I had to turn a 28-key .env file into the env: block of a Kubernetes manifest. Doing that by hand is pure transcription, and transcription is where the one missed quote lives. Paste, convert, review the warnings, copy out. That's the whole workflow, and that's all it's trying to be.

When you shouldn't bother

If you're already on a proper secrets manager, keep going. In that world, .env files are a build artifact your tooling generates, and a human editing one by hand is the anti-pattern. A formatter doesn't fix a process problem.

Skip it in CI too. A browser tool has no business inside a pipeline. If you need conversion in automation, write the five lines of Python against python-dotenv and pin the version, so the parsing rules can't drift underneath you between runs. Reproducibility beats convenience there every time.

Multiline private keys are a maybe. The tool converts them fine (quoted multiline blocks and \n escapes both work), but I think PEM blobs in environment variables are a smell regardless of tooling. Mount them as files and pass a path instead.

And if your entire config is four keys of plain alphanumerics, honestly, anything works. vim works. Don't add ceremony to a file that can't break.

I'll admit I still hand-edit .env files for one-key changes. The tool earns its keep at boundaries, when a file changes format or changes hands, and pretending otherwise would be marketing.

FAQ

Q: Is it safe to paste real secrets into a browser tool?

A: The parsing is client-side, and you can verify that yourself: open devtools and watch the network tab, or go offline before you paste. I paste real files into it, but I built the thing, so I'm biased. Audit it, and rotate anything truly radioactive out of habit.

Q: Why did my value show up inside the container with literal quote marks?

A: You used docker run --env-file with a quoted value. That flag skips quote parsing entirely and takes everything after the first equals sign verbatim. Export a Docker-targeted version of the file with the quotes removed and it goes away.

Q: What's the difference between .env format and shell export format?

A: A shell script needs the export prefix and full shell quoting. MSG=hello world is a valid-ish .env line, but sourced as shell it sets MSG to "hello" and then tries to run world as a command. Converting between the two is exactly where escaping bugs breed.

Q: Does it handle variable expansion like ${HOST}?

A: It parses and preserves the reference, then warns you about it, because dotenv-expand, Compose, and the shell each expand at different times with different rules. I'd rather flag it and let you decide than guess wrong quietly.

Written with AI assistance and human review. Try the tool at aidevhub.io/env-file-manager.

LangChain vs LlamaIndex vs Chonkie: same 94-page PDF

AI Dev Hub — Thu, 20 Aug 2026 14:00:03 +0000

LangChain vs LlamaIndex vs Chonkie: same 94-page PDF

LlamaIndex. Its SentenceSplitter gave me the best recall@5 (0.86, against 0.79 for LangChain's recursive splitter) on a 94-page policy PDF, without writing a custom separator list. LangChain wins if you're already deep in LCEL. Chonkie is the fastest by a wide margin and the one I'd pick for a batch job over a million documents. Chunk size mattered more than the library did.

Quick disclosure before anything else: the RAG chunk size calculator I link to below is one I built. I got tired of re-deriving the same token math in a scratch file, and the four existing pages I found all assumed OpenAI's tokenizer and 1,000-character chunks. Mine is free, runs entirely in your browser, no signup, nothing uploaded. If you know a better one, tell me and I'll link it instead.

The task: one 94-page policy PDF and 38 questions

Last Tuesday I got handed a commercial property insurance policy and a support inbox. The ask was ordinary: answer questions like "what's the deductible for wind damage during a named storm" without a human reading 94 pages every time.

I pulled the text with pdftotext -layout, which gave me 214,883 characters. Then I sat down and wrote 38 questions by hand, each paired with a "needle" string that appears on exactly one page. That labelling took 47 minutes and it's the only reason any number below means anything. A chunking benchmark without labels is just vibes with decimal places.

Same setup for every run: BAAI/bge-small-en-v1.5 as the embedding model, normalized vectors, cosine similarity, top 5 results. The metric is recall@5, the fraction of questions where at least one of the top 5 chunks contains the needle.

Here's the scoring script. No framework, no vector database, just numpy:

# eval_chunks.py - score a chunker by recall@5 on hand-labelled questions
import json, time
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

def recall_at_k(chunks, questions, k=5):
    cv = model.encode(chunks, normalize_embeddings=True, batch_size=64)
    qv = model.encode([q["text"] for q in questions], normalize_embeddings=True)
    hits = 0
    for q, v in zip(questions, qv):
        top = np.argsort(-(cv @ v))[:k]
        if any(q["needle"].lower() in chunks[i].lower() for i in top):
            hits += 1
    return hits / len(questions)

def score(name, split_fn, text, questions):
    t0 = time.perf_counter()
    chunks = split_fn(text)
    dt = time.perf_counter() - t0
    r = recall_at_k(chunks, questions)
    print(f"{name:11} chunks={len(chunks):4}  split={dt:6.2f}s  recall@5={r:.2f}")

if __name__ == "__main__":
    text = open("policy.txt", encoding="utf-8").read()
    questions = json.load(open("questions.json", encoding="utf-8"))

    from langchain_text_splitters import RecursiveCharacterTextSplitter
    lc = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
        chunk_size=512, chunk_overlap=64)
    score("langchain", lc.split_text, text, questions)

    from llama_index.core.node_parser import SentenceSplitter
    li = SentenceSplitter(chunk_size=512, chunk_overlap=64)
    score("llamaindex", li.split_text, text, questions)

    from chonkie import RecursiveChunker
    ch = RecursiveChunker(chunk_size=512)
    score("chonkie", lambda t: [c.text for c in ch(t)], text, questions)

Three lines of output. Then you get to argue with them.

LangChain: I was wrong about this one for an hour

First run, LangChain came dead last. 0.44 recall against LlamaIndex's 0.86. That gap felt too big to be real, and it wasn't.

RecursiveCharacterTextSplitter(chunk_size=512) counts characters. SentenceSplitter(chunk_size=512) counts tokens. Same parameter name, same value, and for English prose that's roughly a 4x difference in how much text lands in each chunk. My careful apples-to-apples comparison was quietly pitting 512-character chunks against chunks of about 2,000 characters. I'd made this exact mistake in April on a different project and still walked straight into it again.

Switching to RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=512, chunk_overlap=64) fixed it: 137 chunks, 0.79 recall, 2.8 seconds. Perfectly respectable.

The thing that still bugs me is what that helper counts with. It pulls in tiktoken and defaults to cl100k_base, an OpenAI tokenizer. I'm embedding with a BERT-family model that uses WordPiece. On plain English the two agree within about 8%, so nothing explodes, but on code, JSON blobs or non-Latin scripts the drift gets ugly and your "512-token" chunks start overflowing a 512-token encoder. Silent truncation, no warning.

Import paths are also a mess if you're following an older tutorial. It's langchain_text_splitters now, and half the search results still say from langchain.text_splitter import.

LlamaIndex: the boring one that won

SentenceSplitter at 512 tokens with 64 overlap: 129 chunks, 0.86 recall, 4.1 seconds. It was the slowest of the three on a single document because it actually runs a sentence tokenizer before doing anything else.

That sentence tokenizer is the whole reason it won. Insurance definitions read like "Named Storm means any storm or weather disturbance that is named by the National Weather Service." Cut that in the middle and neither half retrieves for a question about named storms. The recursive splitters get this right most of the time via their separator list, but "most of the time" across 129 chunks is a handful of dead ones.

Two things I didn't love. The install is heavy: llama-index-core was 41 MB in a fresh venv against 2.9 MB for chonkie. If chunking is the only thing you want, that's a lot of framework to carry.

Bigger issue is the defaults. SentenceSplitter() with no arguments is chunk_size=1024, chunk_overlap=200. On my document that scored 0.68. Nobody's default is tuned for your corpus, and this one is off by enough to matter.

Chonkie: fast, small, and it surprised me

Chonkie is the small library in this comparison and I expected it to lose. It didn't. RecursiveChunker at 512 tokens gave 133 chunks and 0.82 recall in 0.38 seconds. That's within noise of LlamaIndex on quality and roughly 10x faster.

Speed stops being an academic concern once the corpus grows. I ran all three over the client's full document set (1,240 files, 2.1 GB of extracted text). Chonkie finished in 47 seconds. LangChain took 3 minutes 4 seconds. LlamaIndex took 6 minutes 12 seconds. For a one-time ingest, who cares. For a nightly re-index that has to finish before the morning, that's the difference between a cron job you forget about and a Slack alert at 2am.

I also tried SemanticChunker, which embeds each sentence and splits where similarity drops. 0.84 recall, 11.3 seconds on one document. Thirty times the cost for two points I can't distinguish from measurement noise on 38 questions. On a genuinely mixed corpus it might earn its keep. Here it didn't.

The rough edge is polish. The API moved between the version most blog posts describe and the 0.5.x I installed, so half the snippets I found were wrong. And when I passed a tokenizer name it didn't recognise, I got a bare KeyError out of a dict lookup with no hint about what the valid names are. That cost me twenty minutes I'd rather have spent elsewhere.

The scores, and which one I'd actually ship

Criterion	LangChain	LlamaIndex	Chonkie
Best recall@5, 38 questions	0.79	0.86	0.82
What `chunk_size` counts by default	characters	tokens	tokens
Chunk one 94-page doc	2.8s	4.1s	0.38s
Chunk 1,240 docs	3m 04s	6m 12s	47s
Install size, fresh venv	8.4 MB	41 MB	2.9 MB
Error on a bad tokenizer name	named ValueError	named ValueError	bare KeyError
Found the answer in docs under 2 min	yes	yes	no
Defaults I'd ship unchanged	no (characters)	no (1024/200)	yes

I'd use LlamaIndex's SentenceSplitter for anything under roughly ten thousand documents where answer quality is the point. Above that, or anywhere re-indexing runs on a schedule, Chonkie. If your pipeline already lives in LangChain, stay there and just call from_tiktoken_encoder explicitly, because the character default will bite you and it won't be loud about it.

Here's the finding I keep coming back to, though. Across the three libraries at their best settings the spread was 0.07. Across chunk sizes with a single library, it was 0.18: 256 tokens scored 0.74, 512 scored 0.86, 768 scored 0.81, 1024 scored 0.68. The parameter beat the vendor by more than double. If you're agonising over which splitter to import before you've swept chunk size, you're optimising the wrong variable.

That gap is why I built the RAG chunk size calculator. Feed it your embedding model and the kind of document you're indexing, and it hands back a starting chunk size and overlap along with the token math (context window, characters per token for your tokenizer, how many chunks that means for a document of size N). It's a starting point, not an oracle. You still need your own 30 labelled questions.

FAQ

Q: Does overlap actually help, or is it cargo cult?

A: It helps, less than people assume. Zero overlap scored 0.79, 64 tokens scored 0.86, 128 tokens scored 0.87 while producing 22% more chunks to store and search. I settled on 64, about 12.5% of chunk size, and that ratio has held up on two other projects since.

Q: Why not just use semantic chunking everywhere?

A: Because on this document it bought 0.02 recall for 30x the processing time. Semantic chunking pays off when a single file jumps between unrelated topics. A structured policy document already has that structure in its headings.

Q: Will these numbers hold for my documents?

A: Almost certainly not, and I'd be suspicious of anyone who told you otherwise. Legal and insurance prose is dense, repetitive and full of defined terms, which flatters sentence-aware splitting. Chat logs or source code behave differently. Copy the script above, label 30 questions of your own, rerun it. Mine took 47 minutes.

Q: What about markdown and code files?

A: Different tools. Use a header-aware splitter for markdown so you keep section context attached, and Chonkie's CodeChunker (or a tree-sitter based splitter) for source, so functions stay whole. Splitting code on blank lines destroys exactly the boundaries you want to retrieve on.

Written with AI assistance and human review. Try the tool at aidevhub.io/rag-chunk-calculator.

A valid sitemap.xml for 1,247 URLs with a free generator in 2026

AI Dev Hub — Thu, 13 Aug 2026 14:00:03 +0000

A valid sitemap.xml for 1,247 URLs with a free generator in 2026

Paste your URLs into a client-side sitemap generator, add lastmod, and ignore changefreq and priority. Google ignores both, and has for years. A valid sitemap.xml is nine lines of boilerplate plus one url block per page, capped at 50,000 URLs or 50MB uncompressed. The XML is the easy part. Keeping the list free of redirects and 404s is what actually moves crawl budget.

Up front: the sitemap generator I link to below is one I built. I tried six existing ones in March 2026 and every one either uploaded my URL list to a server or capped the free tier at 100 URLs. Mine is free, runs entirely in the browser, no signup, and nothing leaves your machine. If you know a better one, tell me and I'll link it instead.

The migration that shipped 412 dead URLs

In February 2026 I moved a docs site off a flat /guides/ structure and onto /docs/<version>/. Around 1,247 pages. The sitemap came from a shell pipeline I wrote back in 2023: find the build output and wrap each line in <loc> tags with sed. It had worked for three years without a single complaint.

It kept working after the migration. That was the problem.

find walked the old dist/guides/ directory, which the new build never cleaned out. So the sitemap listed 412 URLs that returned 404 sitting next to the 1,247 real ones. Search Console flagged it 9 days later under "Submitted URL not found". By then Googlebot had burned a chunk of its crawl budget on pages that didn't exist, and half the new /docs/ tree was still unindexed.

I fixed it the dumb way first: rm -rf dist/ at the top of the build. That killed the stale files and immediately created a different problem. My sed pipeline had no concept of lastmod, so every entry either got an identical timestamp or none at all, depending on which branch of the script ran. A sitemap claiming all 1,247 pages changed at the same second tells a crawler nothing. I was wrong to treat lastmod as optional decoration. It's the one optional field Google actually reads.

Attempt two was a Node script using the sitemap npm package. That's still what runs in CI today and I have no complaints about it. It's the wrong tool when a colleague drops a CSV of 90 URLs in Slack and wants a sitemap before lunch. Installing a dependency and re-reading the API docs for a one-off costs 20 minutes I'd rather not spend.

That's the gap a paste-in generator fills. Paste, configure, download. 47 seconds, nothing to maintain afterward.

What a valid sitemap.xml actually needs

The spec is smaller than most people expect. sitemaps.org froze at version 0.9 and never moved. You need one urlset element carrying the namespace, and inside it one url element per page with a required loc. Everything else is optional.

The rules that actually bite:

loc has to be absolute and fully qualified, under 2,048 characters. Relative paths are invalid, and plenty of generators emit them anyway.
Ampersands and angle brackets inside URLs must be entity-escaped. ?a=1&b=2 becomes ?a=1&b=2. This is the most common reason a hand-rolled sitemap fails validation.
lastmod must be W3C Datetime. 2026-08-10 is legal. 2026-08-10T14:32:00+00:00 is legal. 08/10/2026 gets the file rejected.
50,000 URLs or 50MB uncompressed per file, whichever hits first. Past that you need a sitemap index pointing at multiple files.
Every URL has to share a host with the sitemap's own location, unless you've verified cross-domain ownership in Search Console.

changefreq and priority are still valid elements, and Google ignores both. Bing too. I still emit priority out of habit, which is probably pointless and definitely harmless.

Here's the whole thing as a script, which is what every sitemap generator does underneath, browser-based ones included:

// build-sitemap.mjs  ->  node build-sitemap.mjs urls.txt > sitemap.xml
import { readFileSync } from "node:fs";

const esc = (s) =>
  s.replace(/&/g, "&amp;")
   .replace(/</g, "&lt;")
   .replace(/>/g, "&gt;")
   .replace(/"/g, "&quot;")
   .replace(/'/g, "&apos;");

const urls = readFileSync(process.argv[2], "utf8")
  .split("\n")
  .map((l) => l.trim())
  .filter(Boolean);

if (urls.length > 50000) {
  throw new Error(`${urls.length} URLs: split these into a sitemap index`);
}

const today = new Date().toISOString().slice(0, 10);

const body = urls
  .map((u) => `  <url>\n    <loc>${esc(u)}</loc>\n    <lastmod>${today}</lastmod>\n  </url>`)
  .join("\n");

process.stdout.write(
  `<?xml version="1.0" encoding="UTF-8"?>\n` +
  `<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n${body}\n</urlset>\n`
);

// $ printf 'https://example.com/\nhttps://example.com/s?a=1&b=2\n' > urls.txt
// $ node build-sitemap.mjs urls.txt
// <?xml version="1.0" encoding="UTF-8"?>
// <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
//   <url>
//     <loc>https://example.com/</loc>
//     <lastmod>2026-08-10</lastmod>
//   </url>
//   <url>
//     <loc>https://example.com/s?a=1&amp;b=2</loc>
//     <lastmod>2026-08-10</lastmod>
//   </url>
// </urlset>

That's 25 lines and it covers escaping plus the 50k guard. The browser version I built adds per-URL lastmod editing and a validation pass before download, which is the piece I kept missing in the script. The sitemap generator on aidevhub parses its own output back before handing you the file, so a malformed URL fails in the tab instead of in Search Console 9 days later. It all runs client-side, so the list never leaves the browser.

How it compares to the alternatives

I checked four options in March 2026 before deciding to build anything. Here's how they line up.

	aidevhub generator	`sitemap` npm package	XML-Sitemaps.com	Screaming Frog
Cost	Free	Free	Free to 500 pages, ~$20/yr after	Free to 500 URLs, GBP 199/yr after
Where your URLs go	Stays in the browser	Stays local	Uploaded to their server	Stays local
Setup time	None	npm install plus a script	None	250MB desktop install
Finds URLs for you	No, you paste them	No	Yes, it crawls	Yes, full crawl
Best fit	One-off lists up to 50k	CI pipelines	Small sites, no dev on hand	Audits and large sites

The real split is URL discovery. Screaming Frog and XML-Sitemaps crawl your site and find the pages for you, which matters a lot when nobody has an authoritative list. That crawl is also the expensive part, and it's exactly where both hit you with limits: 500 URLs free, money after.

Paste-in tools skip discovery and assume you already have the list. If you run any static site generator, you do. My build already knows every route it emitted. Making a crawler rediscover them is work I've done once already.

The privacy column matters more than people admit. A URL list from staging or an internal tool leaks structure: admin paths and unreleased feature routes. Uploading that to a third-party server to get 40 lines of XML back is a trade I stopped making after the second time a security review asked me about it.

When you shouldn't use a paste-in generator

The list of bad fits is longer than the pitch usually admits.

You don't have a URL list. The tool takes URLs. It doesn't crawl. If you've inherited a WordPress install with an unknown page count, run a crawler first and paste the output in second.
The sitemap has to regenerate on every deploy. Automate it instead. The script above is 25 lines and costs nothing to run in CI.
You're past 50,000 URLs. Now you need a sitemap index plus per-shard files, and that's a build-time job.
You need the image or video extensions. Those pull in extra namespaces that simple generators don't emit.
Your CMS already ships one. WordPress with Yoast, or Next.js with its sitemap route, already handles this. Hand-generating on top of that guarantees it goes stale the first time someone publishes.

Honestly, the case for a manual generator is narrower than I'd like: one-off lists and migrations, plus sites without a build step that emits XML. It just happens to be a case that lands on me 4 or 5 times a year and annoys me every single time.

FAQ

Q: Will a sitemap improve my rankings?
A: No. A sitemap affects discovery, so it helps a crawler find pages it might otherwise miss (deep pages, weak internal linking, fresh content). Ranking is a separate question entirely. If your pages are already indexed, adding a sitemap changes nothing.

Q: Do I still need changefreq and priority in 2026?
A: No. Google has said publicly for years that it ignores both, and Bing treats them the same way. They're valid XML, so including them won't break anything. I leave priority in because muscle memory is hard to unlearn.

Q: How do I actually submit the file?
A: Drop it at https://yoursite.com/sitemap.xml, add Sitemap: https://yoursite.com/sitemap.xml to your robots.txt, then submit the URL in Search Console under Sitemaps. The robots.txt line is what other crawlers use, so don't skip it.

Q: Is a browser-based generator safe for internal URLs?
A: Depends on the tool. Client-side ones do the string building in JavaScript in your tab, so nothing is transmitted. Open devtools, switch to the Network panel, and generate a file. If you see zero requests, it's local. That check takes 10 seconds and I'd run it on any tool before pasting a staging URL list.

Written with AI assistance and human review. Try the tool at aidevhub.io/sitemap-generator.

Debugging a 41-second regex hang with a regex tester in 2026

AI Dev Hub — Tue, 11 Aug 2026 14:00:06 +0000

Debugging a 41-second regex hang with a regex tester in 2026

Use a regex tester that shows capture groups live and warns about catastrophic backtracking, because the pattern that takes production down is rarely a wrong match. It's usually a pattern that works fine on 20 characters and hangs forever on 40. Paste the expression, paste a real input line, read the group table, then check the backtracking warning before you ship anything.

Full disclosure: the regex tester I link to below is one I built. I'd been bouncing between four online testers for years and every one of them either skipped Go's RE2 flavor or only told me about catastrophic backtracking after the engine had already timed out. Mine is free, runs entirely in the browser, needs no account, and never sends your pattern to a server. If you've got a better one, tell me and I'll switch.

The 41 seconds that broke our log parser

Late April 2026, a Tuesday afternoon. We had a Node service chewing through nginx access logs and turning them into structured events. It had been running in staging for six weeks without a complaint. Then the pod started getting OOM-killed on a loop, and the container logs went completely quiet right before each restart.

The culprit was one line of code. Line 238, a field validator that looked harmless:

^(\s*\w+\s*)+$

That pattern is fine. It matches. It also happens to be exponential, because \s* and \w+ can both consume the same characters, and the outer + lets the engine try every possible split. Feed it a string that almost matches (say, a request path with a trailing character that isn't a word character) and the backtracking explodes.

One malformed log line. 1,247 requests already queued behind it. 41 seconds pinned at 100% CPU on a single call to .test(), and then the kernel took the process out.

Here's the part that annoyed me for most of a day: I debugged it wrong. I did what everyone does. Console.log, a Node REPL, some print statements around the call site. That took 47 minutes and told me absolutely nothing, because a REPL only answers "does this match", and the answer was "yes, eventually." The question I actually needed answered was "how many steps does the engine take to get there, and how does that scale with input length."

A REPL will not tell you that. It just sits there.

What a regex tester actually does under the hood

A good tester isn't a wrapper around String.prototype.match. Three things happen when you paste a pattern in.

First, the pattern gets compiled against the flavor you picked. This matters more than people expect. JavaScript, Python's re, and Go's regexp disagree on lookbehind support, on named group syntax ((?<name>...) vs (?P<name>...)), and on whether backtracking exists at all. Go uses RE2, which has no backtracking, so the pattern that killed our Node service would have run in linear time there. Same regex, wildly different runtime characteristics.

Second, matching runs in a loop with lastIndex tracked manually, so every match and every capture group gets collected instead of just the first. That's what fills the group table. You see group 1, group 2, the named ones, and the exact character offsets, updated as you type.

Third, and this is the one that would have saved me a day: static analysis for nested quantifiers over overlapping character classes. That's the ReDoS signature. (a+)+, (\s*\w+\s*)+, (\w|\d)*$ and friends. The check is a heuristic, so it produces false positives on patterns that are technically safe, but I'd rather see a yellow warning I can dismiss than find out from a pager.

You can reproduce the actual blowup yourself. Save this and run it:

// backtrack.mjs - run with: node backtrack.mjs
const pattern = /^(\s*\w+\s*)+$/;

for (let n = 14; n <= 22; n += 2) {
  const line = "a ".repeat(n) + "!";     // "!" guarantees a failed match
  const t0 = process.hrtime.bigint();
  pattern.test(line);
  const ms = Number(process.hrtime.bigint() - t0) / 1e6;
  console.log(`${n} words: ${ms.toFixed(2)} ms`);
}

// 14 words:    1.02 ms
// 16 words:    3.91 ms
// 18 words:   15.62 ms
// 20 words:   62.45 ms
// 22 words:  249.80 ms

Roughly 4x per two extra words. Extrapolate to a 60-character path and you get my 41 seconds. Nothing in that output requires a debugger or a profiler, but you do have to know to go looking for it, and you won't go looking if your tool only says "no match."

Regex101, RegExr, or a five-line node script

I used regex101 for years and still open it for PCRE work. It has the best debugger of anything in this space, full stop. But the workflow I wanted was different: paste, see the risk immediately, no account, no network round trip, and Go's flavor in the same dropdown as JavaScript. That's the gap I built the aidevhub regex tester to fill, and it's where I'd start if you're chasing a pattern that behaves badly rather than one that's simply wrong.

Checked against the others in July 2026:

What I checked	aidevhub Regex Tester	regex101	RegExr	`node -e` one-liner
Flavors	JavaScript, Python, Go	PCRE2, JS, Python, Go, Java, .NET	JS, PCRE	JS only
Backtracking warning	Static check, before you run it	Reports it after the engine times out	None	None
Plain-English breakdown	Yes, per token	Yes, token list plus debugger	Yes, on hover	No
Capture group table	Yes, live	Yes	Partial	Manual `console.log`
Account needed to save	No, nothing leaves the tab	Yes, to save permalinks	Yes, to save patterns	N/A
Step-by-step debugger	No	Yes, best in class	No	No

That "No" in the debugger row is real and I'm not going to hide it. If you need to walk the engine's backtracking path step by step to understand why a pattern fails, regex101 does that and mine doesn't. Different jobs.

When a browser regex tester is the wrong tool

I'd skip all of this in a few cases, and I say that as the person who built one of them.

If your pattern is longer than about 200 characters, the honest answer is that you have a parser, and you should write a parser. I've watched a team maintain a 600-character email validation regex for two years. Every tester in the table above will happily render it, and none of them will make it a good idea.

If you're working against a flavor nobody supports well (Oracle's REGEXP_LIKE, or the subset that ships in some embedded Lua runtimes), test in the actual engine. A JS-flavored tester giving you a green checkmark on a pattern that Oracle parses differently is worse than no tool, because now you're confident and wrong.

If the input is sensitive, read the tool's docs before pasting. Mine runs client-side and I'll say so plainly, but "client-side" is a claim you should verify in the network tab rather than take on faith from a blog post. Open devtools, type a pattern, watch for requests. Thirty seconds.

And if you're already deep in a debugging session with a profiler attached, don't context-switch to a browser tab. The five-line script above lives in your repo and answers the scaling question directly.

One thing I got wrong for a long time: I assumed ReDoS was an exotic security-research problem, something that mattered for input parsers exposed to hostile users. It isn't. Our log line wasn't hostile. It was a truncated request path from a client that hung up mid-request, and it hit a validator I'd written myself and never once tested with input longer than a sample string.

FAQ

Q: Does a ReDoS warning mean my pattern is definitely vulnerable?

A: No. The detection is static and errs toward false positives, so nested quantifiers get flagged even when the surrounding anchors make the bad path unreachable. Treat it as a prompt to run the timing loop above, not a verdict.

Q: Why do JavaScript, Python, and Go produce different results for the same pattern?

A: Different engines. Go's regexp package uses RE2, which guarantees linear time and drops backreferences and lookaheads entirely. JS and Python both backtrack. A pattern that's a landmine in Node is safe in Go and won't even compile if it uses lookbehind.

Q: Can I test replacement strings too?

A: Yes, including $1-style group references in JS mode and \1 in Python mode. This is where the flavor selector earns its place, since getting the replacement syntax backwards is the most common silent bug I hit.

Q: What's the fastest way to fix a catastrophic pattern once I've found one?

A: Usually by making the inner quantifier possessive or atomic where the flavor supports it, or by rewriting so the alternatives can't match the same characters. For our validator, ^\w+(\s+\w+)*$ fixed it. Same matches, linear time.

Written with AI assistance and human review. Try the tool at aidevhub.io/regex-tester.

Building an MCP tool-call test rig with the Python SDK in 2026

AI Dev Hub — Thu, 06 Aug 2026 14:00:01 +0000

Building an MCP tool-call test rig with the Python SDK in 2026

You can test an agent's tool-call loop without a model. Write down the calls the model would have made, replay them against your real MCP server over stdio, and assert on what comes back. It runs offline in about two seconds, costs nothing per run, and catches renamed tools and schema drift before a customer does. The model is the last thing you should be faking.

The Function Call Flow Simulator I link to below is one I built. I tried five existing playgrounds first and every one wanted an API key before it would render a single tool_use block, which is backwards when the whole point is sketching a flow you haven't paid for yet. Mine runs in the browser, free, no signup, nothing leaves the tab. If you know a better one, tell me and I'll link to it instead.

The goal

Here's the picture I wanted on my screen. One command, one JSON file, six lines of output: each tool call my agent would make during a customer refund, run against the same MCP server that handles production traffic, with a failure line and an exit code of 1 the moment something breaks. No API key. Nothing over the network. Under two seconds.

The model is the easy part to fake. Everything around it is where I keep getting hurt. The loop that reads tool_use blocks, dispatches them, feeds results back, and decides when to stop is ordinary code, and it fails in ordinary ways. Someone renamed refund_order to issue_refund on the server and my agent quietly degraded into apologising to people instead of paying them. A required field got added to a schema and half the calls started coming back with isError set, which my loop passed straight back to the model as though it were a normal result.

Testing that against a live model doesn't work the way people hope. Run the same prompt twice and you get different arguments, sometimes a different tool, sometimes a friendly paragraph and no call at all. That's correct behaviour from the model and useless behaviour for a test. So pin the model's output. Write it down as data. Then the only moving part left is your code.

Setup and the auth you don't need

pip install "mcp==1.13.1" is the entire dependency list. I pin the version because the stdio client's environment handling shifted twice inside the 1.x line and I lost most of an afternoon to it.

For replay you need no key whatsoever. That's the point: no model is in the room. You need a key exactly once, on the day you record a real conversation to seed your first transcript, and after that the JSON file is the fixture and CI never sees a credential.

Three environment gotchas worth knowing before you start:

The server is spawned as a subprocess over stdio, so it inherits the working directory you launched pytest from, not the directory your test file lives in. Relative paths inside your server config will resolve somewhere surprising.
Anything your server writes to stdout that isn't JSON-RPC corrupts the stream. One stray print() left in a tool handler kills the session with a parse error that names no file and no line.
StdioServerParameters.env replaces the child environment rather than merging into it. More on that below, because it cost me 41 minutes and my dignity.

Before I hand-write a transcript I sketch the flow in the Function Call Flow Simulator, which lets me lay out the tool_use and tool_result pairs across several turns and copy the JSON straight out. Faster than getting the nesting right by hand at 1am, which is how I used to do it.

The code

This is the whole runner. It reads a transcript, checks each tool still exists on the server, calls it, and stops on the first thing that looks wrong.

# replay.py - drive a real MCP server with a scripted tool-call transcript.
# pip install "mcp==1.13.1"
# usage: python replay.py flows/refund.json
import asyncio, json, sys
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client, get_default_environment

# What the model would have emitted. Hand-written, or recorded from one real run:
# [{"name": "search_orders", "input": {"customer_id": "cus_8812", "status": "open"}},
#  {"name": "refund_order",  "input": {"order_id": "ord_4471", "amount_cents": 9164}}]
TRANSCRIPT = json.load(open(sys.argv[1]))

SERVER = StdioServerParameters(
    command=sys.executable,
    args=["-m", "billing_mcp"],                 # your real server, unmodified
    # gotcha: env REPLACES the child environment, it does not merge. Drop the
    # get_default_environment() spread and the child loses PATH and dies quietly.
    env={**get_default_environment(), "BILLING_MODE": "sandbox"},
)

async def main() -> int:
    async with stdio_client(SERVER) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            declared = {t.name for t in (await session.list_tools()).tools}

            for step, call in enumerate(TRANSCRIPT, 1):
                if call["name"] not in declared:
                    print(f"step {step}: server no longer declares {call['name']}")
                    return 1
                result = await session.call_tool(call["name"], call["input"])
                # gotcha: a failed tool call does NOT raise. isError is a field.
                if result.isError:
                    print(f"step {step}: {call['name']} returned isError")
                    return 1
                text = next((c.text for c in result.content if c.type == "text"), "")
                print(f"step {step}: {call['name']} ok  {text[:80]}")
    return 0

sys.exit(asyncio.run(main()))

The declared set is doing more work than it looks like. list_tools() is the server telling you what it actually exposes right now, so comparing your transcript against it turns "the model will call a tool that vanished" from a production surprise into a failing test. The amount_cents: 9164 in the sample transcript is deliberate. Round numbers hide off-by-one and unit bugs, and I want $91.64 flowing through, not $100.

Scripted replay against the alternatives

I've run all three of these in anger. The numbers in the cost row are mine, from a week in February when our CI ran the agent suite on every push.

Axis	Scripted replay	Live model in CI	Recorded HTTP cassettes
Cost per 1,000 runs	$0	$63.18	$0 after recording
Same result every run	yes	no	yes
Catches renamed or dropped tools	yes	sometimes	no, the cassette hides it
Catches bad model arguments	no	yes	no
Works with no network	yes	no	yes
Time to first green test	~20 min	~5 min	~90 min

Cassettes look like the obvious answer and they're the one I'd steer you away from. They freeze the server's responses too, so the day your MCP server changes its schema the cassette keeps replaying the old world and your suite stays green while production burns. Scripted replay freezes only the model side and lets the server be real, which is the exact split you want.

The live-model column has one row nobody else can fill: whether the model picks sensible arguments. That's a real question. It's just a different suite, on a different schedule, with a budget cap.

What went wrong the first time

I was wrong about isError. I assumed a failing tool call would raise an exception, because that's what every HTTP client I've used does. It doesn't. call_tool returns a CallToolResult with isError=True and content that reads like a normal text block. My runner was green for three days straight while every single refund step came back with "customer cus_8812 not found". Three days of a passing suite that was testing nothing. I only caught it because a teammate asked why the sandbox ledger was empty.

Then the environment thing. I passed env={"BILLING_MODE": "sandbox"} on its own, assuming it merged with the parent process environment the way subprocess.run does with env=None. It doesn't merge. The child got an environment containing exactly one variable, lost PATH, and failed to launch. What I saw was session.initialize() hanging until the 30 second timeout with zero output, because the child's stderr goes nowhere unless you wire it up. I spent 41 minutes convinced my server had a deadlock in its startup handler. The fix is the two-character spread in the code above.

Last Tuesday I added a sixth step to the refund flow and hit the third one: I'd been asserting with result.content[0].text and a substring check. The server had started returning structuredContent alongside a JSON dump in the text block, so my substring "refunded" matched a field name rather than a status value. The assertion passed for entirely the wrong reason. I don't have a clean rule for this yet beyond "parse the JSON, assert on a key", and honestly I'm still annoyed that the loose version survived as long as it did.

None of these three are bugs in the MCP SDK. They're all me assuming a library behaved like a different library I knew better. Worth budgeting an hour for that on any new protocol.

FAQ

Q: Isn't this just testing my own mocks?

A: No, and that's the whole design. The server is your actual MCP server, running as a real subprocess, hitting your real sandbox database. The only mocked thing is the model's choice of tool and arguments, which is the one component you can't assert on deterministically anyway.

Q: How do I get the first transcript?

A: Run the conversation once for real, log every tool_use block your loop receives, and dump the list to JSON. About 15 lines of throwaway code. After that you edit the file by hand to add edge cases the model never happened to produce.

Q: Does this work for HTTP or SSE servers instead of stdio?

A: Yes. Swap stdio_client for streamablehttp_client and pass a URL. Everything inside the ClientSession block stays byte-for-byte identical, which is the nicest property of the client API.

Q: How do I still test that the model picks the right tool?

A: Separate suite, run nightly rather than per-push, with a hard spend cap and assertions loose enough to tolerate variation (did it call any refund-shaped tool, did it stop after four turns). Keep it away from your fast suite.

Written with AI assistance and human review. Try the tool at aidevhub.io/function-call-simulator.

Jinja2 vs Claude vs SkillSpec in 2026: same skill, 3 ways

AI Dev Hub — Thu, 30 Jul 2026 14:00:03 +0000

Jinja2 vs Claude vs SkillSpec in 2026: same skill, 3 ways

SkillSpec Converter wins for anything I need in more than two output formats at once. The Jinja2 script is still faster for a single format I fully control, and asking Claude directly is fine for a one-off, though it quietly drops required fields. I timed the same skill through all three last week. Below are the actual outputs and the exact spot each one broke.

The SkillSpec Converter I link to below is one I built. I got tired of hand-porting the same skill across OpenClaw, Claude, Codex, and MCP, and the four converters I tried each handled one target well and mangled the rest. It's free, runs entirely in your browser, needs no signup, and uploads nothing. If you know a better one, tell me and I'll switch to it happily.

The task: one skill, four target formats

Last Tuesday I had a skill that worked in exactly one place and needed to run in four by the end of the week. The canonical definition is small. It's a PR summarizer that takes a pull request URL and returns five bullet points, gated behind a single read-only GitHub tool. That's the whole spec, and I keep it as one YAML file that acts as my source of truth.

The four targets were OpenClaw's SKILL.md, a Claude system block, a Codex scaffold, and an MCP manifest snippet. My expected output was four files that all agree: same inputs, same tool allowlist, and the required: true on pr_url, because a summarizer with no URL is just an apology generator. The test was drift. If any of the four copies disagreed with the YAML, I'd call the run a failure, because that's the bug you don't catch until it breaks in production three weeks later.

Round 1: the Jinja2 script I already had

I reached for the script first because it already existed. It's roughly 60 lines of Python that loads the YAML spec and renders one target through a Jinja2 template. Here's the trimmed version that produces the OpenClaw SKILL.md, and you can run it as-is after pip install pyyaml jinja2:

# pip install pyyaml jinja2
import yaml
from jinja2 import Template

CANONICAL = """
name: pr-summarizer
description: Summarize a GitHub pull request into five bullet points
inputs:
  - name: pr_url
    type: string
    required: true
  - name: max_bullets
    type: integer
    required: true
  - name: tone
    type: string
    required: true
tools:
  - github_read
"""

SKILL_MD = Template("""---
name: {{ name }}
description: "{{ description }}"
allowed-tools: {{ tools | join(", ") }}
---

# {{ name }}

{{ description }}

## Inputs
{% for i in inputs -%}
- `{{ i.name }}` ({{ i.type }}){% if i.required %}, required{% endif %}
{% endfor %}""")

spec = yaml.safe_load(CANONICAL)
print(SKILL_MD.render(**spec))

It works, and the output was correct on the first pass. I trust it more than anything else here because I wrote every line of the template, so when something looks wrong I know where to look. The catch is the other three targets. Each one needs its own template, and the Codex scaffold and the MCP manifest have structural quirks that don't fall out of a flat YAML file cleanly. Building and debugging all four templates took me 43 minutes. I burned a chunk of that on a bug at line 41, where a tool with no parameters rendered an empty properties: {} block that the MCP validator flat-out rejected. Great for a format I'll render a hundred times. Rough for something I need once.

Round 2: just asking Claude to convert it

Next I pasted the same YAML into a chat and asked for all four formats in one go. Six minutes, four clean-looking code blocks, done. Or so I thought.

Two of the three inputs came back without their required flag. I don't fully know why. The description text came through fine and the tool allowlist was right, but the one boolean I actually cared about just wasn't there. I ran it twice more with a stricter prompt. It behaved once, then dropped the flag again on the third try. The MCP manifest it produced used input_schema where the spec wants inputSchema, which is the kind of casing slip that reads fine to a human and dies in a parser. If I were shipping a throwaway skill I'd have taken it and moved on. For four files that have to stay in lockstep, I couldn't trust the output without diffing every field against the YAML by hand, and at that point six minutes isn't really six minutes.

Round 3: SkillSpec Converter

Then I ran it through my own tool, which is the honest reason this comparison exists. I pasted the same canonical YAML into the box, and it rendered all four targets into tabs in under a second. The required flags survived intact. The MCP manifest came out with the camelCase key. The Codex scaffold handled the empty-properties case, which is the precise bug I'd hit by hand at line 41 an hour earlier.

I want to be fair about the limits. It only knows the four targets I taught it, so the moment you need a fifth output format you're back to a template or a model. There's no clever inference happening under the hood. But for the narrow job of keeping one skill definition consistent across OpenClaw, Claude, Codex, and MCP, it did in one paste what my script did in 43 minutes and what the chat couldn't do reliably at all.

The scores, and which one I keep open

Here's how the three landed on the things I actually measured:

Criterion	Jinja2 script	Ask Claude	SkillSpec Converter
Time to get all four formats	43 min	6 min	under 1 min
Required fields kept	all	1 of 3	all
Valid MCP manifest out of the box	after a fix	no	yes
Setup cost	high, four templates	none	none
Customization ceiling	total	prompt-dependent	the four built-in targets
Cost	free	API tokens	free

So which one do I reach for? It depends on the day, and I'll give you my honest split instead of pretending one tool won outright.

If I need exactly one format and I'll be generating it over and over, the Jinja2 script takes it. Full control, and I can version the template right next to the code it belongs to. If I need a rough draft and I'm going to read every line anyway, asking Claude is the quickest way to a starting point, as long as I treat its output as a draft and diff it against the spec.

For the case that actually stung (one skill, four formats, zero drift allowed), I keep the SkillSpec Converter open in a tab. It's free, and it runs client-side. It also stopped me from shipping a manifest with the wrong casing twice this month. I'm biased here, plainly. I built it after the four alternatives I tried each nailed one target and butchered the rest. If yours does this better, I want to know.

FAQ

Q: Does converting a skill lose information between formats?
A: It can. The formats don't have identical feature sets, so a field in one target may have no home in another. A good converter maps what it can and leaves a comment where a target can't represent something, which beats silently dropping it and finding out in prod.

Q: Can I use the Jinja2 approach for MCP manifests too?
A: Yes, and you should if MCP is your only target. Write one template, test it against the manifest validator, and you're set for good. The pain only shows up once you're maintaining three or four templates in parallel and they start disagreeing with each other.

Q: Why did Claude drop the required flags?
A: I don't have a clean answer. My guess is that boolean flags on nested objects are easy to lose across a long generation, especially when the surrounding prose is more salient to the model. A stricter prompt helped but didn't fix it every single time.

Q: Which format is hardest to get right by hand?
A: The MCP manifest, easily. The nesting runs deep and the key casing is unforgiving. An empty properties object trips the validator, which is the same trap I fell into at line 41. It's the one target where I stopped trusting my own templates first.

Q: Is my skill definition uploaded anywhere?
A: Not with SkillSpec Converter, since it runs in the browser and the spec never leaves your machine. For the Claude route, your spec does go to the API, which matters if the skill contains anything you'd rather keep off a server.

Written with AI assistance and human review. Try the tool at aidevhub.io/skill-spec-converter.

Reading 50MB JSONL logs with a viewer in 2026

AI Dev Hub — Thu, 23 Jul 2026 14:00:04 +0000

Reading 50MB JSONL logs with a viewer in 2026

Use a browser-based JSONL viewer that parses each line as its own JSON object and lays the results out in a filterable, sortable table. That's the fastest way to read newline-delimited logs without writing a throwaway script. Paste the file, get columns, filter to the rows you care about, export what's left. No terminal gymnastics and no 2GB file crashing your editor. Works offline because everything runs client-side.

The jsonl viewer I link to below is one I built. I got tired of it: I tried six different online JSON tools last spring and every one of them choked the moment I pasted newline-delimited data, because they all assume a single JSON document and JSONL is a stream of them. Mine runs entirely in your browser. No signup, no upload, nothing leaves your machine, and it's free. If you've got a better one, please tell me.

The log file that killed my text editor

Last Tuesday, around 2am, I was chasing a production timeout. The only evidence I had was an NDJSON log the service had been streaming to disk: 340 MB, roughly 1.2 million lines, one JSON object per line. I did the obvious thing first and opened it in my editor. It thought about that for a while. The fans spun up. Then the window went white and stopped responding. Cool.

So I fell back to jq. jq 'select(.level == "error")' app.log does work, and honestly jq is a wonderful tool, but I couldn't remember the exact field names, I kept getting the filter slightly wrong, and every failed guess re-streamed the whole file from the top. Twenty minutes in I still hadn't seen a single row. The error turned out to be on line 811,406, but I didn't know that yet. All I wanted was to see the shape of the data first, then decide what to filter. That's the gap.

JSONL (also called NDJSON or JSON Lines, same idea under different names) is a stream of JSON values with one per line. Log pipelines love it because you can append a line without rewriting the file, and a crash mid-write only costs you the last line instead of the whole document. The catch is that most JSON tooling assumes a single document, so it reads your 1.2 million lines and immediately throws on the second {.

How a JSONL viewer parses a stream of objects

The core of a JSONL viewer is almost embarrassingly small. You split on newlines and parse each non-blank line on its own. The one trick that matters is not letting a single bad line kill the whole render, so you catch per line and keep going.

// Split, drop blanks, parse each line independently.
function parseJSONL(text) {
  return text
    .split('\n')
    .filter((line) => line.trim() !== '')
    .map((line, i) => {
      try {
        return { ok: true, row: JSON.parse(line) };
      } catch (err) {
        return { ok: false, line: i + 1, raw: line, error: err.message };
      }
    });
}

const sample = `{"ts":"2026-04-11T02:14:03Z","level":"error","msg":"timeout","ms":9812}
{"ts":"2026-04-11T02:14:04Z","level":"info","msg":"retry","attempt":2}
not valid json
{"ts":"2026-04-11T02:14:06Z","level":"info","msg":"ok"}`;

console.log(parseJSONL(sample));

Run that and you get four results back. Three parse cleanly into row objects, and the not valid json line comes back as { ok: false, line: 3, ... } instead of blowing up the other three. A viewer takes that array, unions all the keys it sees to build columns (ts, level, msg, ms, attempt), and paints a table. Now level is a column you can filter, not a string you have to grep for.

You don't have to run this yourself. Paste your file into the JSONL viewer and it does exactly this in your browser, then hands you a sortable, filterable table with the broken lines flagged in red so you can spot corruption instead of silently dropping it. That red-flag behavior is the part I use most, weirdly. Half my "bug" reports turn out to be one truncated log line.

What the table actually gets you

Parsing is the boring half. The reason a table beats a wall of text is what you can do to it once it's there. I filter first, almost always. Type error into the level column and 1.2 million rows collapse to the 3,000 that matter. Then I sort by timestamp to find the first one, because the first error is usually the real cause and the rest are fallout.

Sorting on a numeric field (say a duration in ms) is where the table earns its keep. In an editor I'd be eyeballing numbers by hand. Here I click the ms header and the slowest request floats to the top. Last week that surfaced a single 14,203 ms outlier I'd never have spotted by scrolling, and it was the whole bug.

Export closes the loop. Once I've filtered down to the rows I care about, I pull them out as JSON or CSV and drop that slice into a ticket, so the person picking it up sees 40 relevant lines instead of a 340 MB file and a shrug.

jq versus a spreadsheet versus a text editor

Each of these has a place. Here's how I actually pick between them:

Approach	Reads JSONL as-is	Live filtering	Big files	Setup cost
Browser JSONL viewer	Yes	Yes, instant	Fine to ~100 MB	None, it's a web page
jq on the CLI	Yes	No, re-run per query	Excellent, it streams	Install plus learn the syntax
Import into a spreadsheet	No, needs flattening	Yes	Poor past a few MB	Manual conversion step
Text editor plus grep	No	Text search only	Bad, loads it all	Already open

The viewer wins when I'm exploring and don't yet know what I'm looking for. jq wins when I know the exact query and want it in a script or a pipe. I reach for jq inside CI, and the viewer at 2am when my brain is half offline. Different jobs, and I stopped feeling guilty about using both.

When you shouldn't reach for it

I built this thing and I still won't use it for everything. A few honest limits.

If your file is genuinely huge (multiple gigabytes), keep it in jq or a streaming parser. A browser tab has a memory ceiling, and loading 3 GB into a table will hang the page the same way it hung my editor. The viewer is for the range where an editor struggles but the data still fits in RAM, so call it a few hundred MB and under.

If the task is automated, a viewer is the wrong shape entirely. Anything that runs on a schedule or inside a build should be jq or a small script. A human clicking a web page doesn't belong in a cron job, and you'll hate maintaining it if you try.

And if you're dealing with deeply nested objects, a flat table gets awkward fast. The viewer flattens what it can and shows nested blobs as collapsed JSON, which is readable but not magic. For heavy nesting I still drop back to jq's path expressions. No tool wins every round, and pretending otherwise is how you end up with the wrong one open at 2am.

FAQ

Q: Is JSONL the same thing as NDJSON?
A: Effectively yes. JSONL, NDJSON, and JSON Lines all name the same format: one JSON value per line, separated by \n. There are pedantic edge cases around trailing newlines and empty lines, but any decent viewer handles the three names identically.

Q: Does my data get uploaded anywhere?
A: No. The parsing runs client-side in your browser, so the file never leaves your machine. That was the entire reason I built it that way. I didn't want to paste production logs into someone else's server and hope for the best.

Q: How big a file can it actually handle?
A: It depends on your available RAM more than anything else. I've thrown 90 MB files at it without trouble on a normal laptop. Past a few hundred MB you'll feel the tab get heavy, and that's your cue to switch to jq.

Q: Can I export the filtered rows?
A: Yes. Once you've narrowed down to the rows you want, you can export just the visible set as JSON or CSV, which is handy for handing a teammate a clean slice instead of the raw dump.

Q: Why not just use jq for all of it?
A: You can, and plenty of people do. I like jq for known queries and pipelines. The viewer is for the messier moment before that, when I don't yet know the field names or what I'm even looking for and want to poke at the data with my eyes.

Written with AI assistance and human review. Try the tool at aidevhub.io/jsonl-viewer.