DEV Community: Arthur031221

I tested my Docling JSON checker against real documents and found my own bug

Arthur031221 — Thu, 01 Oct 2026 06:07:03 +0000

Docling turns a PDF, DOCX, or scanned page into a structured JSON document. Two open issues against the project caught my attention: one where a text item's source span runs past the end of its own text after dehyphenation, another where a table cell's content bleeds into the column next to it. In both cases the conversion still returns a valid JSON file, so nothing downstream raises an error. A search index or a RAG pipeline just gets a quietly wrong chunk.

What it does

docling-guard is a small Python CLI with no dependencies. It reads a Docling JSON export and checks two things: that every provenance span fits inside the text it claims to describe, and that table cells do not collide or overlap on the page grid. A second command, compare, diffs two exports and reports any drop in text or table count after a Docling upgrade. It does not run OCR or load model weights, it only inspects the JSON Docling already produced.

Testing it for real

The first release shipped with one synthetic fixture, a short document built by hand to demonstrate the checks. That proves the code runs, not that the checks are useful on real documents.

This week I ran it against the 15 real Docling exports in Docling's own test suite: arXiv papers, a German newspaper interview, a technical handbook, right-to-left documents, and tables. It reported errors in 9 of 15 documents, which looked alarming until I read the findings closely.

Almost all were the same false positive. Docling strips list markers like "b." from a text item's visible text field but keeps the full string, markers included, in a separate orig field. The provenance span is measured against orig, not text. My checker compared it against text, so nearly every enumerated list item in the corpus tripped the check.

Fixing that dropped the count from 9 documents to 3. Those 3 show a different, real pattern: a multi-part text item whose combined span runs 2 characters past its own text, the exact dehyphenation overshoot described in the issue that got me started on this.

What is in the repo now

The three fixtures that caught the bug are committed under tests/data/real/, with source and license noted in SOURCES.md. A script, measure_real_documents.sh, reproduces the full 15-document count. The README now states the real numbers instead of only the synthetic one.

What is rough

It only checks the shape Docling exports today: a texts array and a tables array with the fields this version of Docling uses. A large export, over 64 MiB, is rejected outright rather than streamed. And the table overlap check only compares horizontal coordinates, so it can warn on a legitimate unusual layout.

docling-guard is MIT licensed and free to use.

https://github.com/Arthur031221/docling-guard

cardsmith: offline flashcards from your own PDFs, checked against the source

Arthur031221 — Thu, 01 Oct 2026 00:07:02 +0000

I study from long PDFs and slide decks often enough that flashcards would help, but making them by hand from a chapter of reading takes long enough that I usually skip the step and reread instead. Rereading is a weaker way to study than active recall, so I built cardsmith to close that gap.

What it does

Point cardsmith at a PDF, a PPTX, or a plain text file. It splits the document into chunks of roughly 120 to 900 words, sends each chunk to a local model through Ollama, and asks the model to write flashcards using only facts in that chunk. Every card comes back with a verbatim source_quote, the exact sentence the card is based on.

That is the part I care most about. Most AI flashcard generators hand you cards with no way to tell whether the model made something up. cardsmith checks whether the quote it returned is an actual substring of the source chunk, and if it is not, the card is flagged in the preview before you save anything. You can still edit or delete any card before it goes into your deck.

Cards live in a local SQLite database with a plain SM-2 scheduler, the same algorithm Anki started from. Four grade buttons record how well you knew each card and schedule the next review. When you want to study in Anki itself, export to a real .apkg file with a genanki-built deck.

What I measured

On a 3,248 word public domain biology chapter, cardsmith generated 152 cards in about 13 minutes on a MacBook Air. The mechanical quote check matched 142 of 152 cards to an exact substring of the source text. I then hand rated a 40 card sample against the source and found 37 factually accurate, 1 inaccurate, and 2 unusable, for 92.5 percent accuracy. The full method and raw cards are in the repository's eval directory.

What is rough

Card quality depends on the 4B model I default to, which occasionally paraphrases a quote instead of copying it, which is why the grounding check exists rather than trusting the model's claim. There is no OCR, so a scanned PDF with no text layer will not work without running it through an OCR tool first. It is single user and single machine by design, with no sync and no account.

Try it

ollama pull qwen3:4b
uvx --from git+https://github.com/Arthur031221/cardsmith cardsmith

The repository is at https://github.com/Arthur031221/cardsmith, MIT licensed. I would welcome feedback, especially on the grounding check and on what would make the accuracy number more trustworthy.

Ollama silently truncated my context window. A scanner for local LLM setups

Arthur031221 — Wed, 30 Sep 2026 19:53:14 +0000

On my MacBook Air, Ollama 0.34.4 served qwen3:1.7b with a 4,096-token window. The model supports 40,960. I sent a 7,000-token prompt and got HTTP 200 back, but the model had only seen the last 2,050 tokens. There was no error and no warning.

That was the start of llm-doctor, a health check for local LLM setups.

The problems it looks for

Running models locally leaves a lot of quiet mess behind.

Every runtime keeps its own copy of the same GGUF. Ollama hides it behind a sha256 blob name, so you cannot see that LM Studio holds the same 20 GB file.
Quantizers fix chat templates after release, and your old download keeps the broken one. Tool calls then fail in ways that look like the model being bad.
Ollama defaults to a small context window, and the OpenAI-compatible endpoint that coding agents use cannot raise it per request.
Leaving Ollama means downloading everything again, because its blobs have no names.

Using it

uv tool install git+https://github.com/Arthur031221/llm-doctor

llm-doctor                                   # scan every model store
llm-doctor fix                               # preview dedupe and cleanup
llm-doctor fix --yes                         # apply it
llm-doctor unbundle --to llama-server        # expose Ollama models to llama.cpp
llm-doctor endpoint http://localhost:11434   # agent-readiness probe

The GIF above runs against a throwaway home directory where Ollama and LM Studio hold the same GGUF, plus one orphan blob. The scan reports both, and fix --yes replaces the LM Studio copy with a symlink to the Ollama blob and deletes the orphan. The demo fixture reclaims 140 MB, and every change is logged to ~/.llm-doctor/fix-log.jsonl.

The endpoint probe is the part I use most. It checks context length, tool calls, parallel tool calls, streaming tool calls, JSON schema output and think tags, then prints what to change for each failure. For the Ollama case above it reported 4,096 effective tokens of 40,960 advertised and marked the context check as failed.

Limits

It needs Python 3.10 or newer. It is not on PyPI yet, so install from GitHub as shown. I tested mostly on macOS with Ollama, so Linux and Windows reports are the most useful feedback I can get. If you run a store or runtime it does not detect, open an issue with the folder layout.

The code is at https://github.com/Arthur031221/llm-doctor under the MIT license.

Eleven model snapshots retire on October 23. Find them in your repo before your users do

Arthur031221 — Wed, 30 Sep 2026 15:07:02 +0000

OpenAI shuts down eleven model snapshots on 2026-10-23, among them gpt-4-turbo, gpt-4o-2024-05-13, o3-mini and o4-mini. The GPT-5 and o3 snapshots follow on 2026-12-11. Anthropic and Google retire models on their own schedules too. Most codebases find out when requests start returning 404.

Finding what retires

I wrote modelshift to answer two questions from the terminal. Which model IDs in my repo are going away, and does the replacement still behave the same?

npx github:Arthur031221/modelshift scan examples/sample-app

The scan walks the repository and lists each model ID with its file and line, its provider, its lifecycle status and the recommended replacement. The status comes from a bundled registry built from the providers' deprecation pages. The exit code is 1 when something retires within the window, so it also works as a CI check.

I ran it over four popular, unrelated open-source repositories. It found 131 distinct model IDs that are retiring or already retired, referenced 1,649 times across 296 files.

Rewriting the IDs

modelshift fix examples/sample-app

fix prints a diff that swaps each ID for its replacement and drops parameters the new model rejects. Claude Opus 4.7 and later, for example, return a 400 when a request sets temperature, top_p or top_k. Add --write to apply the change, or --pr to open a pull request.

Checking the replacement

Swapping the string is the easy part. The hard part is knowing whether the new model still returns valid JSON, still calls your tools with the same arguments, refuses more often, or takes twice as long.

modelshift replay --from qwen3:1.7b --to qwen3:4b --prompts demo/prompts.jsonl --no-think

replay runs your prompts on both models and compares JSON validity, refusals, tool calls, length, latency and text similarity, then writes an HTML report. It works against any OpenAI-compatible endpoint, including a local Ollama server, so you can try it without API keys. In the bundled example, a weather tool call that the small model made was missing from the larger model's answer, which is the kind of change a string swap would never show.

Limits

It needs Node 20 or newer and is not on npm yet, so run it with npx from GitHub. The registry is maintained by hand from provider pages, so a wrong date is the most likely bug. If you find one, please open an issue.

The code is at https://github.com/Arthur031221/modelshift under the MIT license.

Your coding agent may have saved your API keys in plain text. Here is how to find and remove them

Arthur031221 — Wed, 30 Sep 2026 10:25:17 +0000

Coding agents read .env files as a matter of course, and the tools write what the agent saw to disk: prompt histories, session transcripts, local databases. An API key that passed through a session can stay there in plain text for months.

Where the keys go

Coding agents read .env files as a matter of course. Every tool then stores what it saw in its own format. Claude Code writes JSONL files under ~/.claude/projects. Codex uses ~/.codex/sessions. Cursor keeps a SQLite database. Cline, Gemini CLI, OpenCode and Aider each keep their own copy. These files sit there for months, and folder sync and backups copy them around.

Nothing I found cleaned up what had already leaked, and nothing stopped the next agent from doing the same, so I wrote a tool for it.

What agentleaks does

It is a single Go binary with three commands.

scan walks the history of 13 tools and checks it against 64 rules. The output is a table with the provider, the tool, the file, the line or database record, and a masked preview of the key, so the report itself never prints a full secret.

agentleaks

fix redacts the keys in place. The replacement text contains no quote or backslash, so a JSON string stays a JSON string, and every JSONL record is re-validated after rewriting. SQLite rows are updated in a transaction. The original file is copied to ~/.agentleaks/backups first, and the modification time is preserved so tools that sort sessions by time do not reshuffle.

agentleaks fix          # dry run
agentleaks fix --yes    # apply

guard installs hooks into the agents' own config so the next read of .env or ~/.aws/credentials is refused.

agentleaks guard

The demo above runs against a throwaway home directory with randomly generated fake keys, not real history.

Install

go install github.com/Arthur031221/agentleaks/cmd/agentleaks@latest

Static binaries for macOS, Linux and Windows are attached to each release on GitHub. There are no runtime dependencies, and nothing leaves your machine unless you run the opt-in verify command.

What is rough

Guard coverage is uneven, because each tool's hook system exposes different events. It also does not scan git history, for which gitleaks and trufflehog are the right tools.

The code and the rule list are at https://github.com/Arthur031221/agentleaks. If you use a tool or a key format I do not cover yet, tell me and I will add a rule for it.