DEV Community: OptiRefine

Your RAG pipeline is bad at docs because your chunker splits code fences

OptiRefine — Fri, 28 Aug 2026 02:36:09 +0000

Here is the chunking code from more or less every RAG tutorial published in the last two years:

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
chunks = splitter.split_documents(docs)

It works fine on prose. Run it over a documentation site and it quietly does four destructive
things, none of which throw an error, and all of which show up later as "the assistant gives
vague answers about our API."

I spent a while building a docs-specific ingestion pipeline and most of the work turned out to be
in these four places. None of it is glamorous. All of it moved retrieval quality more than
swapping embedding models did.

1. It cuts code fences in half

A 1,000-character window lands mid-code-block constantly. You get one chunk ending in:

```python
def authenticate(client_id, client_secret):
    resp = requests.post(TOKEN_URL, data={

and the next chunk starting with the rest of the dict and a closing fence that now has no opener.

Two things go wrong. The obvious one is that neither chunk contains a runnable example, so the
model retrieves half a function and confabulates the other half. The subtler one is that a
half-open fence poisons everything downstream — every markdown renderer, and most models,
treat the remainder of that chunk as code. Your carefully written prose about token expiry is now
inside a Python block as far as the model is concerned.

The fix is that code blocks and tables can only split at line and row boundaries, and if a
split does happen the fence has to be reopened with its language tag and the table header has to
be repeated. And overlap must never bisect a fence — the overlap window is where this bug hides,
because the chunk itself looks fine and only the neighbour is broken.

2. It throws away where the chunk came from

This is the one that costs the most retrieval quality for the least effort to fix.

A sliding window gives you a chunk that reads:

You must include the state parameter and verify it on return. Tokens expire after 3600
seconds.

Embed that and ask "how long do OAuth tokens last?" and it may or may not come back, because
nothing in the text says OAuth, or authentication, or which product this is. The words that would
have matched are in an <h2> four hundred characters up the page.

Documentation is a tree and chunks should inherit their path. Every chunk in a docs pipeline
should open with its heading path:

Authentication > Authorization flows > OAuth 2.0

### OAuth 2.0

You must include the `state` parameter and verify it on return...

Now the chunk is independently answerable. It carries its own context, it embeds near the
question people actually ask, and when you show sources in the UI you have a breadcrumb to
display instead of a bare URL.

Keep the un-prefixed version in a separate field too (rawText), because when you feed retrieved
chunks to the model for generation you often want the clean body without the breadcrumb noise
repeated ten times in the prompt.

3. `chunk_size=1000` isn't 1000 of anything you care about

RecursiveCharacterTextSplitter counts characters. Your embedding model has a token limit.
The ratio between them varies enormously with content:

Content	Roughly chars per token
English prose	~4.0
Dense technical prose	~3.5
Code	~2.5
Minified JSON / config blobs	~2.0

So a 1,000-character chunk is ~250 tokens of prose but can be ~500 tokens of code. If you sized
your window against a model limit using a character count, your code chunks are silently getting
truncated at the embedding step, and truncation at the embedding step is invisible — no error, no
warning, just a vector for the first 60% of the chunk.

Count real BPE tokens with the actual tokenizer (tiktoken / gpt-tokenizer, cl100k_base or
o200k_base depending on your model). It is one dependency and it removes a whole class of
"why is retrieval worse on the API reference pages" investigations.

4. It has no idea your docs site has four versions of every page

Documentation sites are duplicate factories. Versioned docs (/v2/, /v3/, /latest/),
locale variants, print views, and framework-generated index pages mean a naive crawl of a
500-page docs site can yield 1,800 chunks where 600 would do.

Exact-hash dedupe catches byte-identical pages and gets you maybe half of it. The rest needs
near-duplicate detection — SimHash with banded lookup is cheap and works well here.

But there's a trap. Naive near-duplicate detection will happily collapse:

Install on Linux — Run ./configure && make && make install.

Install on Windows — Run ./configure && make && make install.

Identical bodies, completely different answers to "how do I install this on Windows?" So
similarity matching has to be scoped to the heading: two chunks are only candidates for
deduplication if their heading path matches too.

Doing all four without writing a crawler

I ended up packaging this as an Apify actor —
Docs-to-RAG Pipeline Builder — because I
was rebuilding the same thing for every project and the crawl half is more annoying than it looks.

Minimal run:

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxCrawlPages": 500
}

That returns chunks that already respect all four rules above. Each one looks like:

{
  "id": "9f2c1a77b0e34d15",
  "url": "https://docs.example.com/api/authentication",
  "anchorUrl": "https://docs.example.com/api/authentication#oauth-2-0",
  "pageTitle": "Authentication",
  "breadcrumb": ["Authentication", "Authorization flows"],
  "heading": "OAuth 2.0",
  "headingLevel": 3,
  "chunkIndex": 2,
  "chunkCount": 5,
  "text": "Authentication > Authorization flows > OAuth 2.0\n\n### OAuth 2.0\n\n...",
  "rawText": "### OAuth 2.0\n\n...",
  "tokenCount": 612,
  "contentHash": "1b0f...",
  "generator": "mkdocs-material",
  "extractor": "profile:mkdocs-material",
  "crawledAt": "2026-08-24T10:14:22.104Z"
}

Note anchorUrl. Deep-linking a citation to the exact heading rather than the page top is a
two-line change that makes a support bot feel dramatically more trustworthy, and almost nobody
does it.

Two design decisions worth stealing even if you build your own

Crawl in two passes, not one. Rendering every page in Chromium is slow and expensive;
rendering none of them breaks on every client-side-rendered docs site. So: fetch over plain HTTP
first, score the extracted content, and escalate only the pages that fail the gate — JS shells,
suspiciously thin pages — to a browser. renderingMode: "auto" with escalateBelowWords: 60
does this. On a typical docs site the large majority of pages never touch a browser.

Let extraction strategies compete instead of picking one. Docs sites are built by generators,
and generators have known DOM shapes. So run several extractors — an explicit
mainContentSelector if the user gave one, a profile matched to the detected generator (MkDocs,
Docusaurus, Sphinx, Starlight, VitePress and friends), a general readability-style extractor, and
a link-density heuristic falling back to <body> — score all of their outputs, and take the
winner. A single hardcoded selector is a silent failure waiting for the day the docs site
upgrades its theme; it doesn't error, it just starts returning navigation sidebars as content.

The run report tells you which extractor won on which pages, which is how you find out your
selector went stale.

Embeddings without an API bill

embeddingProvider has three settings, and two of them cost nothing:

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "embeddingProvider": "cloudflare-worker",
  "workerUrl": "https://docs-to-rag-worker.you.workers.dev",
  "workerApiKey": "<API_TOKEN>"
}

You deploy a small Worker to your own Cloudflare account and it runs @cf/baai/bge-m3 (1024
dims) on Workers AI. The free tier covers roughly 9 million tokens a day — a 500-page site
chunked at 800 tokens uses about 0.5% of one day's allowance. Set embeddingProvider: "local"
instead and it runs Xenova/all-MiniLM-L6-v2 (384 dims) on-device with no network calls after
the model downloads.

Both are meaningfully worse than a frontier embedding model on hard retrieval. Both are entirely
adequate for "answer questions about our docs", and being free changes how willing you are to
re-embed after changing your chunking — which you will, several times.

The artifacts nobody asks for and everybody ends up using

Alongside the dataset it writes to the key-value store:

chunks.jsonl — the embedding-ready records
corpus.md — the whole site as one readable markdown file
llms.txt and llms-full.txt — the llms.txt site index format
run-report.json — extractor wins, escalations, failures

corpus.md is the one I use most and expected least. When retrieval gives a wrong answer, being
able to grep the entire corpus in one file tells you in five seconds whether the content was
missing from the crawl or present-but-not-retrieved. Those two failures look identical from the
chat UI and have completely different fixes.

Measure it on your own corpus, because I can't measure it on yours

Retrieval quality claims that aren't measured on your data aren't worth much, and mine aren't
either. The good news is that this particular A/B is cheap to run, and free if you use local
embeddings.

Build the same corpus twice:

Baseline — RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) over raw HTML converted to text
Structure-aware — heading-recursive chunking, breadcrumb prefixes on, dedupe on

Then write 20–30 questions you already know the answer to. This is the part people skip and
it's the only part that matters. Take them from your actual support inbox or your team's Slack —
questions real people asked, not questions you invented while looking at the docs, which are
biased toward the phrasing the docs already use.

Three numbers to compare:

Chunk count. The dedupe delta is usually the first thing that jumps out on a versioned docs site. Fewer chunks at equal coverage is a straight win on embedding cost and retrieval noise.
Hit rate @5 — for each question, is the chunk containing the answer in the top five? This is the number that predicts whether your assistant feels good to use.
Whole-example rate. Of the retrieved chunks containing code, how many contain a complete, runnable block rather than a fragment? This is the one the sliding window loses badly, and it's worth counting separately because hit-rate alone hides it.

If structure-aware chunking doesn't beat the baseline on your corpus, that is a real and useful
finding — it probably means your docs are flatter than you thought, and your effort belongs in
the retriever or in reranking instead. Measure before you commit to any of this.

Where I'd push back on myself

Structure-aware chunking is not free. It's a lot more code than eight lines of LangChain, and on
a corpus that isn't structured — support-ticket exports, transcripts, PDFs of scanned contracts —
none of it helps, because there are no headings to be aware of. The sliding window is genuinely
the right default there.

The claim is narrower than "sliding windows are bad": documentation is one of the most
structured corpora you will ever index, and throwing that structure away at the chunking step is
the single most common own-goal in RAG pipelines over docs. If you're indexing something else,
ignore most of this and go tune your retriever instead.

The actor is Docs-to-RAG Pipeline Builder
(MIT-licensed, pay-per-event, runs with a single startUrls entry). If you'd rather build your
own, the four rules at the top of this post are the ones I'd implement first — in that order.

A short-form video script pipeline that runs on a schedule and needs no LLM API key

OptiRefine — Fri, 28 Aug 2026 02:33:36 +0000

Every "automate your faceless YouTube channel" tutorial follows the same shape. Pull an RSS feed,
for loop over the items, POST each one to an LLM with a prompt that says "write a 40-second
TikTok script," write the response to a file. Forty lines. Works on the demo.

Then you schedule it and the parts nobody wrote about start showing up:

The model returns prose with no segment structure roughly one time in six, and your downstream parser explodes.
Half the RSS items are three-sentence stubs that link to a paywall. There is nothing to write a script about, but the model cheerfully writes one anyway, from nothing.
Your scripts run 55 seconds when you asked for 40, because nobody counted words against a speaking rate.
You pick three categories, and the first one eats the entire item budget before the other two get a look in.
You're paying per token for outputs you throw away.

I ended up building this as an Apify actor
(TikTok & YouTube Shorts Script Generator),
and the interesting part turned out not to be the generation. It was the five problems above. Here
is how each one gets solved, whether you use the actor or write your own.

The output has to be structured, or none of it composes

A script is not a paragraph. It's a timed sequence with distinct jobs per segment, and if your
pipeline treats it as a blob you can't do anything downstream — no b-roll matching, no caption
timing, no per-segment retakes.

So the output shape is a list of segments, each labelled:

{
  "category": "tech",
  "topic": "A hidden phone setting",
  "duration": "40 seconds",
  "platform": "TikTok/YouTube Shorts",
  "wordCount": 96,
  "segments": [
    {
      "label": "HOOK",
      "start": 0, "end": 3,
      "durationSec": 3,
      "speech": "Your phone has been throttling itself since the day you bought it.",
      "visual": "Close-up of hand pulling down settings panel, harsh overhead light"
    },
    {
      "label": "BODY",
      "start": 3, "end": 33,
      "speech": "...",
      "visual": "..."
    },
    {
      "label": "CTA",
      "start": 33, "end": 40,
      "speech": "...",
      "visual": "..."
    }
  ],
  "status": "ok",
  "generatedAt": "2026-08-27T14:02:11.884Z"
}

Two things about that shape are load-bearing.

speech and visual are separate fields. Not one field with stage directions inline. The
moment you want to feed narration to TTS, or match stock footage to a scene description, or
generate captions, you need the spoken words alone with no parenthetical noise in them. Models
will merrily write (cut to close-up) in the middle of the narration if you let them, and then
your voiceover says "cut to close-up" out loud.

start and end exist before there's any audio. The plan is a plan. When you do generate
audio, you get actualStart and actualEnd measured from the file — and the gap between planned
and actual is the most useful debugging number in the whole pipeline. A hook that planned for 3
seconds and measured 5.2 is a hook that will not survive the scroll.

The hook is a per-niche problem, not a prompt problem

This is where most generic pipelines produce unusable output, and it's not fixable by asking
harder in the prompt.

A tech short and a beauty short have structurally different openings. Tech opens on a
counterintuitive claim — your phone has been throttling itself. Beauty opens on a visual
result, before/after, no claim at all. News opens on the stakes. Gaming opens mid-action, usually
mid-sentence. Kids' content opens on a question the viewer can answer out loud.

One prompt cannot do all of those well, which is why single-prompt pipelines produce the
recognisable slop voice — a generic "Did you know that..." opener bolted onto every topic
regardless of niche.

So category is a first-class input, not a topic string. Each of
tech, trends, beauty, fashion, sports, gaming, news, learning, kid friendly,
music, general carries its own persona, hook strategy and segment structure.

RSS mode, which is the version you'd schedule:

{
  "mode": "rss",
  "categories": ["tech", "gaming"],
  "maxItems": 10,
  "duration": "40 seconds"
}

Direct mode, for when you have your own source of topics — a spreadsheet, a scraper, your own
backlog:

{
  "mode": "direct",
  "items": [
    {
      "category": "tech",
      "topic": "A hidden phone setting",
      "data": "Most phones ship with a battery saver that is off by default. Enabling it..."
    }
  ]
}

Direct mode is the one that matters if you're building something bigger. It makes this a stage
in a pipeline rather than the whole pipeline — anything that can produce {category, topic, data}
can feed it.

The multi-category budget trap

Ask for ["tech", "gaming", "news"] with maxItems: 10 and the naive implementation walks the
tech feed first, produces ten tech scripts, and stops. You asked for three categories and got one.

The fix is to divide the budget across categories and interleave collection rather than
draining feeds in order. It's an obvious bug once you've seen it and an easy one to ship, because
it looks completely fine when you test with a single category.

Thin sources produce hallucinated scripts

An RSS item with a 40-character summary and a paywalled link contains no information. Feed it to a
model with "write a script about this" and the model will write one — confidently, and entirely
from its own priors. That's a fabrication with your channel's name on it.

The gate is boring and it works: require a summary of at least ~500 characters, or 200+
characters of extracted body text, and skip the item otherwise. Thin and paywalled pages get
dropped rather than embellished.

Whatever you build, put a content-length floor somewhere in it. It is the difference between a
pipeline that summarises and one that invents.

Failed generations are a first-class output, not an exception

Small models return unparseable output some of the time. Not often, but often enough that at
scheduled scale you will get some every week. The wrong responses are to retry forever, to crash
the run, or to silently drop the item.

The right one is to route it: unparseable generations land in a separate failed-generations
dataset with "status": "unparseable", a reason, and the raw text that came back — and they
aren't billed, because you didn't get a script.

That last part is worth designing for even in your own build. If you're paying per token, you're
paying for garbage output at the same rate as good output, and you have no incentive signal
telling you the prompt is degrading. Separating billable success from non-billable failure makes
prompt regressions visible instead of just expensive.

The failed-generations dataset is also the best prompt-debugging corpus you'll get. Read twenty
of them and you'll usually find one specific phrasing in one category that reliably derails the
model.

Generation without an API key

The actor runs an embedded Llama 3.1 8B by default. No OpenAI key, no Anthropic key, no
separate billing relationship — you run it and it generates.

That is not because an 8B model is better. It isn't. It's that short-form scripts are close to the
ideal task for a small model: ~100 words of output, rigid structure, strong per-category
scaffolding around the prompt. Almost all of the quality here comes from the structure, and very
little from raw model capability. When the scaffolding does most of the work, model size stops
being the bottleneck.

If you disagree, the escape hatch is documented: deploy the worker in worker/ to your own
Cloudflare account and pass workerUrl and workerSecret as a pair — a URL without a secret
is rejected, and the built-in generator's own secret is never forwarded to a custom worker.

Voiceover, if you want it

Add an ElevenLabs key and each segment gets an audioUrl plus measured timings:

{
  "mode": "rss",
  "categories": ["tech"],
  "maxItems": 5,
  "elevenLabsApiKey": "<your key>",
  "voiceId": "21m00Tcm4TlvDq8ikWAM",
  "ttsModelId": "eleven_turbo_v2_5",
  "ttsConcurrency": 2
}

Two things to know. ElevenLabs bills your account directly, not through Apify — the run log
estimates the character count up front so you can see the damage before it happens. And
ttsConcurrency defaults to 2 because the free tier will rate-limit you above that; raise it only
if you're on a paid plan.

What to expect from the output, honestly

An 8B model producing 100 words against a rigid template gets you a solid first draft and not a
finished script. In practice the parts that hold up and the parts you rewrite are consistent:

Usually fine as-is: the segment structure and timing, the visual cues (they're generic but
they're a real shot list, which is more than a blank page), and the CTA.

Usually needs a pass: the hook. It's the highest-leverage seven words in the whole video and
it's the thing a small model is weakest at, because a good hook depends on knowing what your
specific audience already believes. Expect to rewrite most of them. That's fine — rewriting one
line beats writing five.

Watch for: claims stated more confidently than the source supports. The content-length gate
stops the worst of it, but a summary that says "researchers suggest" can come out the other end
as "researchers proved." Read the speech fields against the sourceUrl before you record
anything factual.

The right mental model is that this replaces the blank page and the timing math, not the editorial
judgement. If you generate ten and keep three, it's working correctly.

Putting it on a schedule

The whole point is that you don't run this by hand. In Apify, a schedule is a cron expression on
the actor — 0 7 * * * for a daily 7am batch — and the output accumulates in a dataset you can
pull from anywhere:

curl -s "https://api.apify.com/v2/datasets/$DATASET_ID/items?clean=true&format=json" \
  | jq -r '.[] | select(.status == "ok")
           | "\(.topic)\n" + (.segments[] | "  [\(.label)] \(.speech)")'

A sensible daily loop looks like: generate 10 in the morning → you skim and keep 3 → those 3 go to
a shot list → record or assemble → publish. The pipeline's job is to make the skim cheap. It is
not to publish unattended, and I'd argue strongly against wiring the output directly to an upload
API. The value is in cutting the blank-page cost of the first draft, not in removing the human
from the loop.

Limits worth knowing before you build on it

maxItems caps at 50 per run. Shared inference allowance; run more often rather than bigger.
generationConcurrency defaults to 5 (max 20), ttsConcurrency to 2 (max 10).
There's a run deadline (runDeadlineSeconds, default 210) and a per-item timeout. Long batches with TTS on will hit them — split the work rather than raising both to the ceiling.
RSS quality is the ceiling on output quality. Use feedOverrides to point at feeds that publish real summaries. Garbage in still applies, and no amount of prompt work fixes a feed that publishes headlines only.

The actor is
TikTok & YouTube Shorts Script Generator
— pay-per-event, and unparseable generations aren't charged.

If you're building your own instead: the four things worth copying, in order, are the
segment-level output shape, the per-category hook strategy, the content-length gate on the source,
and the separate failure dataset. The model call is the least interesting part of the whole thing.

After CHAINDROP, I stopped trusting npm audit. Here’s the dependency check I run instead.

OptiRefine — Fri, 28 Aug 2026 02:31:22 +0000

On 4 August 2026, the CHAINDROP wave of the Shai-Hulud worm compromised over 400 npm packages
with a combined 1.3 billion monthly downloads. keyv alone accounts for around 600 million of
those. flat-cache, cacheable-request, cache-manager — none of them packages you chose.
All of them packages you have.

The worm stole credentials for AI providers, AWS, GCP, Azure, and GitHub, obfuscated itself with
control-flow flattening and Base91 string encoding, and pulled its command-and-control address
out of an Ethereum smart contract so takedowns wouldn't stick.

By the time you read this, that specific campaign is contained. The pattern is not. This is the
third round of the same worm family, and each round has worked the same way: compromise a
maintainer account, publish a patch version, wait for everyone's ^ range to pick it up.

So here's the uncomfortable thing about the tool most of us reach for first.

`npm audit` answers the wrong question

npm audit tells you which of your dependencies have published advisories. That is a useful
question. It is not the question these attacks punish you for.

A malicious version published forty minutes ago has no advisory. It has no CVE. It will pass
npm audit cleanly until a researcher files it — which, in the CHAINDROP case, took hours
during which the package was being installed continuously.

What would have flagged those packages ahead of time is duller and more boring:

Maintainer count of one. A single compromised account is a single point of failure, and most of the worm's entry points were solo-maintained packages.
Long silence, then a sudden patch release. A package that hadn't shipped in fourteen months suddenly cutting a patch version at 3am is the signal.
Transitive depth. The packages that did the damage were four and five levels down, where nobody looks.
Deprecation. Deprecated packages still installed in production are unowned attack surface.

None of that is a vulnerability. All of it is risk. And there is no single command that gives it
to you across a whole package-lock.json.

That is the gap I kept hitting, so I built something to fill it.

The check I actually run

The data already exists, publicly, in four places:

npm registry / PyPI — versions, licenses, maintainers, deprecation flags
OSV.dev — vulnerability advisories, properly version-scoped
deps.dev — the resolved dependency graph, including transitives
pypistats / npm downloads — adoption signal

The annoying part isn't getting any one of them. It's getting all four, for 300 packages, joined
on the same key, without writing four rate-limit handlers and a cache.

I wrapped that into an Apify actor —
npm, PyPI & crates.io Package Intelligence —
that takes a list of packages and returns one row each: version, license, maintainer count,
download volume, OSV advisories, deps.dev dependency counts, and a 0–100 health score.

The bulk endpoint takes up to 50 packages per call, which is what makes this practical against a
real lock file rather than a curiosity you run on one package.

Step 1 — get your actual dependency list

Not your package.json. Your lock file, because that's where the transitives live:

# npm — every package in the resolved tree, deduplicated
npm ls --all --json \
  | jq -r '[.. | objects | select(has("version")) | .name?] | unique | .[]' \
  | grep -v '^null$' > packages.txt

wc -l packages.txt

For Python:

pip list --format=json | jq -r '.[].name' > packages.txt

On a mid-sized Node service this is usually somewhere between 400 and 1,200 lines. That number
alone is worth sitting with for a second.

Step 2 — score them in bulk

Input to the actor is deliberately boring:

{
  "packages": ["keyv", "flat-cache", "cacheable-request", "express", "lodash"],
  "ecosystem": "npm",
  "includeDependencies": true
}

Package names can carry an ecosystem prefix, so a polyglot monorepo goes in as one run rather
than three:

{
  "packages": ["npm:keyv", "pypi:requests", "crates:serde"],
  "includeDependencies": true
}

Run it from the API and pipe the dataset straight into jq:

jq -Rn '{packages: [inputs], includeDependencies: true}' < packages.txt > input.json

curl -s -X POST \
  "https://api.apify.com/v2/acts/optirefine~package-intelligence/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  --data-binary @input.json \
  > health.json

The scoring endpoints are pay-per-event and priced per package; the raw lookup endpoints
(metadata, vulns, deps, downloads) are free at 60/min and 2,000/day, so you can prototype the
whole pipeline before spending anything. Packages that don't exist return 404 without charging,
which matters more than it sounds when you feed it a lock file containing internal scoped
packages.

Step 3 — sort by the thing that actually predicts trouble

jq -r '
  map(select(.maintainerCount == 1 and .downloads > 1000000))
  | sort_by(.healthScore)
  | .[]
  | [.name, .healthScore, .maintainerCount, .downloads, .lastPublished]
  | @tsv
' health.json | column -t

That query — high download volume, single maintainer, low health score — is the CHAINDROP
shape. It is the list I'd want on a screen before approving any dependency bump.

The four numbers worth writing down

When you run this against your own tree, four counts tell you almost everything. I'd write them
in a comment on the PR that introduces the check, so you have a baseline to compare against in
six months:

Total packages in the resolved tree. Not direct dependencies — everything. This is the number that makes people quiet in a room.
How many are single-maintainer with over a million weekly downloads. Your CHAINDROP-shaped surface. This is the list a human should actually read.
How many are deprecated but still installed. Unowned code running in production. Every one of these is either a migration ticket or a decision to accept the risk on purpose — and it should be one of those two, not neither.
How many haven't published in over a year. Not inherently bad — some packages are simply finished. But combined with (2), it's the set where a sudden patch release should make you look rather than merge.

Track those four over time rather than chasing an aggregate score. The absolute numbers vary
enormously by ecosystem and project age, so your own trend is the only meaningful comparison.
What you want to see is (2) and (3) going down while (1) goes up, which is what a team actively
managing its surface looks like.

How the health score is built (and why I kept it dumb)

The score is a documented v1 heuristic, not a model. It reads on these signals:

Release recency — how long since the last publish
Maintainer count — one is a risk, not a virtue
Deprecation status — flagged hard
Vulnerability load — OSV advisories resolved against the current version, not every version ever published
Dependency health — the shape of what it pulls in

That last point about OSV is the one I'd defend hardest. A lot of tooling reports every advisory
ever filed against a package name, which makes well-maintained packages that patched promptly
look worse than abandoned ones that never had a researcher look at them. Scoring against the
resolved current version fixes that inversion.

The score is deliberately auditable. Every input is a public field you can go and check yourself.
If you disagree with the weighting, the raw fields are all in the same row and you can build your
own — which I'd encourage, because the right weighting depends on your threat model, not mine.

What it does not do: it doesn't detect malicious code. Nothing in this pipeline reads a
package's source. It measures the conditions under which a compromise is more likely to go
unnoticed. Those are different claims, and I'd rather be clear about which one I'm making.

Wire it into CI, or it won't happen

The version that actually changes behaviour runs on pull requests that touch the lock file:

# .github/workflows/dependency-health.yml
name: Dependency health
on:
  pull_request:
    paths: ['package-lock.json', 'requirements.txt', 'poetry.lock']

jobs:
  health:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 2 }

      - name: Diff newly added packages
        id: added
        run: |
          git diff HEAD^ HEAD -- package-lock.json \
            | grep -oP '^\+\s+"node_modules/\K[^"]+' \
            | sort -u > added.txt
          echo "count=$(wc -l < added.txt)" >> "$GITHUB_OUTPUT"

      - name: Score them
        if: steps.added.outputs.count != '0'
        env:
          APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}
        run: ./scripts/score-packages.sh added.txt

One deliberate design choice worth copying: only score newly added dependencies, not version
bumps of existing ones. Scoring the whole tree on every PR produces a wall of noise that people
learn to click through within a week, and a check everyone ignores is worse than no check.

Pair it with the boring controls that would have actually stopped CHAINDROP:

A soak period before adopting updates — pin exact versions and let new releases age 48–72h
npm 12+, which blocks preinstall hooks by default
2FA enforced on every npm account you own
Post-incident: rotate GitHub tokens, npm credentials, and cloud keys from any machine that installed during the window, and grep your repos for unauthorized commits (CHAINDROP's were labelled chore: update config)

The part I'd argue about

Health scoring is a lagging measure dressed as a leading one. A single-maintainer package with a
low score has been that way for years without being compromised, and most of them never will be.
Treat the score as a prioritisation aid for where to spend review attention, not as a gate
that blocks merges. I've watched teams turn a score threshold into a hard CI failure and then
spend the next month adding exceptions until the check meant nothing.

The useful version of this is smaller than it sounds: a weekly list of the fifteen riskiest
things in your tree, looked at by a human for ten minutes.

The actor is here if it's useful:
npm, PyPI & crates.io Package Intelligence.
Free endpoints for the raw registry, vulnerability and dependency data; pay-per-event for the
health scoring, so a one-off audit of a few hundred packages costs about the price of a coffee.

If you run it against your own tree, I'd like to hear what your single-maintainer count came out
at — and whether the number surprised you. That's the one I've never seen anyone guess correctly
in advance.

Sources

How AuraWatch Finds Security Bugs Your Linter Will Never Catch

OptiRefine — Tue, 05 May 2026 20:14:10 +0000

How AuraWatch Finds Security Bugs Your Linter Will Never Catch

Most security tools for developers fall into one of two camps: static linters that pattern-match on surface-level code, or AI-powered scanners that guess based on training data. Both have the same fundamental problem — they can't reason about your code. They can tell you that eval() exists, but they can't tell you whether user input actually reaches it.

AuraWatch is built differently. It uses a six-engine deterministic analysis pipeline that constructs a full semantic model of your code and reasons about it mathematically. No guessing. No false positives from pattern matching. Provable results.

AuraWatch launches on the VS Code Marketplace on May 15th, 2026.

Here's how it works under the hood.

The Problem With Existing Tools

Consider this code:

app.post('/search', async (req, res) => {
    const { term } = req.body;
    const results  = await db.query(`SELECT * FROM products WHERE name = '${term}'`);
    res.json(results);
});

A regex-based linter might flag the template literal inside query() if it's lucky. But what about this:

app.post('/search', async (req, res) => {
    const term    = req.body.term;
    const cleaned = term.trim();
    const results = await db.query(`SELECT * FROM products WHERE name = '${cleaned}'`);
    res.json(results);
});

The data moved through an intermediate variable. Most linters miss this entirely because they're not tracking data flow — they're just looking for patterns. AuraWatch catches both because it builds a complete picture of how data moves through your code.

The Six-Engine Pipeline

Every scan runs six independent engines in parallel. Each engine looks at the code from a different angle. Their results are merged and deduplicated before being returned.

Engine 1 — LST Analyzer (Lossless Semantic Tree)

The foundation of AuraWatch is a Lossless Semantic Tree, built using ts-morph for JavaScript/TypeScript and LibCST for Python. Unlike a standard AST, an LST preserves every token in the original source including whitespace, comments, and formatting. This matters because transformers need to reconstruct the original code faithfully when applying fixes.

The LST engine runs nine rule-based analyzers over the tree:

eval() calls
dangerouslySetInnerHTML without DOMPurify
Hardcoded credentials in variable declarations
Hardcoded credentials in object literal properties (catches { password: 'secret' } inside database config objects — a pattern most tools miss)
SQL injection via template literals and string concatenation
child_process.exec() with dynamic arguments
Math.random() used to generate security-sensitive values
Unguarded database connection calls (null dereference)
Deep property chains without optional chaining

Each analyzer returns a structured vulnerability object with a unique ID, CWE classification, line number, severity, and a flag indicating whether a deterministic auto-fix is available.

Engine 2 — Type-Aware Taint Analysis

This is where AuraWatch goes beyond linting. The taint engine performs inter-procedural data flow analysis to track user-controlled data from its source to a dangerous sink.

Sources are any access to HTTP request data:

req.body.username
req.query.search
req.params.id
req.headers['x-custom']

Sinks are functions where tainted data causes harm:

eval()          // CWE-95: Code Injection
exec()          // CWE-78: OS Command Injection
db.query()      // CWE-89: SQL Injection
fs.readFile()   // CWE-22: Path Traversal
res.redirect()  // CWE-601: Open Redirect
innerHTML       // CWE-79: XSS

The engine works in three passes:

Source collection — walks all variable declarations and assignments to identify variables initialized from req.* values, including destructured patterns like const { username, password } = req.body
Taint propagation — marks any variable that flows from a tainted source as tainted itself
Sink matching — checks whether any argument to a known sink contains a tainted variable

When a tainted variable reaches a sink, AuraWatch reports which specific variables carried the taint and exactly where the data entered the program.

Engine 3 — JS-SMT Symbolic Constraint Solver

SMT (Satisfiability Modulo Theories) solving is traditionally the domain of formal verification — not developer tooling. AuraWatch brings a lightweight SMT constraint solver directly into the VS Code extension pipeline.

The engine maintains a range map for every integer variable it encounters, tracking the possible values each variable can take given the conditional guards in scope:

// After seeing: if (count > 0)
// The engine knows: count.min = 1

// After seeing: if (count < 10)
// The engine knows: count.max = 9

// Combined: count ∈ [1, 9]

With this range information, the engine can prove:

Division by zero:

function paginate(items, pageSize) {
    return items.length / pageSize; // SMT proves pageSize can be 0
}

Contradictory conditions (dead code):

if (score > 100) { ... }
if (score < 50)  { ... } // SMT proves these are mutually exclusive

Always-true auth bypasses:

if (isAdmin || true) { // SMT proves this always executes
    deleteAllUsers();
}

For Python, AuraWatch uses the Z3 SMT solver directly via its Python bindings, enabling full logical formula construction and satisfiability checking. The JavaScript engine uses a custom lightweight interval arithmetic solver that runs in-process without any native dependencies.

Engine 4 — Async Flow and Prototype Pollution Tracker

This engine handles two distinct but related categories of JavaScript-specific vulnerabilities.

Prototype pollution occurs when user-controlled data is used as an object key in an assignment:

// User sends: { "__proto__": { "isAdmin": true } }
function merge(target, source) {
    for (const key in source) {
        target[key] = source[key]; // key = "__proto__"
    }
}

If key equals __proto__, constructor, or prototype, this mutates the base Object prototype — affecting every object in the process. AuraWatch detects computed property assignments where the key is user-controlled, direct __proto__ mutations, and Object.assign() calls with untrusted sources.

Unhandled promise rejection crashes Node.js processes in production (since Node 15). The engine identifies promise chains that use .then() without .catch() at the expression statement level — a common oversight when converting callback-based code to async.

Engine 5 — Hallucination Defense

This engine is specifically designed for the AI-generated code era. Large language models frequently hallucinate npm package names — inventing packages that don't exist on the npm registry. When a developer installs an AI-suggested package and the real package doesn't exist, a malicious actor can publish a package with that exact name. This is a supply chain attack vector that barely existed five years ago.

AuraWatch maintains a curated registry of ~200 well-known legitimate packages. For every require() call and import statement it finds, it checks:

Is the package in the known-good registry?
Does the package name match a known typosquatting pattern?
If a package.json is provided in the scan request, is the package listed as a dependency?

Typosquatting patterns are matched against a curated list of confusable names:

lodas  → lodash
expres → express
axioss → axios
helmt  → helmet

A package flagged as potentially hallucinated gets a CRITICAL severity finding — because installing a non-existent package that gets claimed by an attacker is code execution on install via the postinstall lifecycle hook.

Engine 6 — Framework-Specific Sink Maps

Generic rules miss framework-specific vulnerabilities. This engine knows the dangerous patterns specific to Express, Fastify, Next.js, and React.

Express:

res.render(userInput) — Server-Side Template Injection when the template name is user-controlled
res.redirect(userInput) — Open Redirect when the destination URL comes from user input
res.send(\${userInput}) — Reflected XSS when user data appears in an HTML response

Next.js:

context.query values spread directly into page props in getServerSideProps without validation

Fastify:

reply.send(request.body) — echoing raw request body in responses that may be rendered as HTML

The framework is auto-detected from import statements — no configuration required.

Deterministic Auto-Fixing

When AuraWatch finds a vulnerability with a known safe transformation, it can fix it automatically using the same LST infrastructure it uses for analysis.

Fixers work as CST Transformers — they traverse the syntax tree and replace unsafe nodes with safe equivalents while preserving all surrounding formatting and comments:

Vulnerability	Auto-Fix
Hardcoded secret	Replaced with `process.env.VARIABLE_NAME \
{% raw %}`Math.random()` in security context	Replaced with `crypto.randomBytes(4).readUInt32BE(0)`
`dangerouslySetInnerHTML`	Wrapped with `DOMPurify.sanitize()`, import added
`eval(x)` (Python)	Replaced with `ast.literal_eval(x)`
`pickle.loads(x)` (Python)	Replaced with `json.loads(x)`
`yaml.load(x)` (Python)	Replaced with `yaml.safe_load(x)`
`subprocess.run(shell=True)` (Python)	`shell=True` argument removed
Unguarded async result	Property accesses converted to optional chaining (`?.`)

Fixes are presented as a unified diff in VS Code's native diff viewer before anything is written to disk. The user reviews the patch and explicitly accepts or rejects it — AuraWatch never modifies your files without confirmation.

Architecture: Two Engines, One Interface

AuraWatch runs as two separate Cloud Run services behind a single VS Code extension:

Python Engine (LibCST + Z3)

Handles .py files
Uses LibCST for lossless CST analysis
Uses Z3 for formal SMT solving
10 LST analyzers, 6 auto-fixers

JS/TS Engine (ts-morph + custom SMT)

Handles .js, .ts, .jsx, .tsx files
Uses ts-morph for full TypeScript-aware AST analysis
Custom interval arithmetic SMT solver
6 base analyzers + 4 advanced engines

The VS Code extension automatically routes each scan to the correct engine based on the active file's language. The API key, billing, and auth system are unified — one account works across both engines.

Authentication uses magic links — no passwords. When a user signs in, a time-limited token is written to Firestore, emailed to the user, and validated server-side when clicked. The resulting API key is stored in VS Code's native SecretStorage API (the OS keychain) — never in plaintext settings files.

What's Next

The roadmap includes:

GitHub Actions integration — run the full analysis pipeline as a CI check on every pull request
Inline diagnostics — surface findings as VS Code squiggles directly in the editor without opening the chat panel
Java and Go support — extending the LST engine to additional languages
Cross-function taint tracking — following data flow across function boundaries and module imports

Launch

AuraWatch launches on the VS Code Marketplace on May 15th, 2026. Follow along on Twitter/X and LinkedIn to get notified on launch day. If you work in security or DevSecOps and want to give early feedback, reach out — every piece of feedback shapes what gets built next.

The Code Review Comment I Left 47 Times

OptiRefine — Tue, 28 Apr 2026 00:33:22 +0000

I counted once. Not proud of it.

Forty-seven times across different PRs, different engineers, different companies — I left some version of the same comment. It usually looked like this:

"Hey — this nested loop is going to hurt you at scale. You're doing O(n²) work here. Can you use a set for the inner lookup instead?"

Sometimes the author got it immediately and fixed it in ten minutes. Sometimes I'd spend forty-five minutes in a thread explaining what O(n²) actually means in practice, with benchmarks, with examples, with a rewrite pasted directly into the comment. Sometimes the fix went in. Sometimes a slightly different version of the same pattern showed up in the next PR from the same person.

And every single time, I thought the same thing: this should not require a human.

The Thing Nobody Talks About in Code Review

Code review has this mythology around it. We talk about it like it's this high-value, high-skill practice where senior engineers pass down wisdom to junior ones. And sometimes it is. The architectural discussions, the "have you considered what happens when this service goes down" conversations, the stuff that requires genuine experience and judgment — that's worth a human's time.

But a massive chunk of code review isn't that. It's pattern matching. It's the same ten categories of problem appearing in slightly different costumes, over and over, in every codebase, at every company. Nested loops. Hardcoded credentials. Unused variables. Functions so tangled with conditionals that you need a flowchart to test them. Files left open without context managers. List comprehensions written as manual loops for no reason.

These aren't judgment calls. They have objectively correct answers. And yet we route them through humans anyway, which means they get caught inconsistently, they get caught late, and they generate the kind of review comments that quietly frustrate junior engineers who feel like they're failing when really they just weren't told the rules.

I got tired of being part of that system. So I built something to replace the parts of it that shouldn't need me.

What I Actually Built

OptiScan is a static analysis engine. You paste Python code in. It parses the code into a concrete syntax tree — not a string, not a token stream, an actual typed tree where every node is a specific kind of thing with specific kinds of children. Then it walks that tree and runs a set of analysis passes, each of which is looking for something specific.

The output is not a suggestion. It's a rewrite.

When the engine finds a nested loop doing pair comparisons — the O(n²) pattern I've left forty-seven comments about — it doesn't tell you it's bad. It replaces it. You get back working code that does the same thing in O(n) time using a HashSet for constant-time membership checks. You can copy it directly. You can run the benchmark in the browser to see the difference in milliseconds.

That's the thing I wanted to get right. A linter that tells you your code is slow and then leaves you to figure out the fix is only half useful. The fix is the point.

Let Me Show You What I Mean

Here's the function that ships as the default example:

def find_pairs(arr, target):
    result = []
    for i in range(len(arr)):
        for j in range(i+1, len(arr)):
            if arr[i] + arr[j] == target:
                result.append((arr[i], arr[j]))
    return result

dataset = list(range(500))
pairs = find_pairs(dataset, 150)

Clean enough. Readable. Does what it says. And for a dataset of 500 elements it probably runs fine on your laptop, which is exactly why it makes it to production.

But feed it 50,000 elements and suddenly you're doing 1.25 billion comparison operations. Feed it a million elements and you've brought a service to its knees over a function that looked completely harmless in review.

The AST engine parses this, detects the outer loop, detects the inner For node as the only direct child of the outer loop body, confirms there's an append call buried inside, identifies the collection name from the range(len(arr)) pattern, and rewrites:

arr_set = set(arr)
for num in arr:
    complement = target - num
    if complement in arr_set:
        result.append((num, complement))

One pass. O(1) set lookup. The benchmark in the browser — running both versions via Pyodide, a Python interpreter compiled to WebAssembly — shows this version running between 20 and 40 times faster on the default dataset. Not because I said so. Because you can run it yourself and watch the numbers.

The Security Part That Kept Surprising People

I added a security auditor because hardcoded secrets in source code is the other comment I've left too many times. But when I started building it I ran into something that made me realise most linters are only doing half the job.

The obvious case is this:

api_key = "sk-abc123def456"

Everyone knows that's bad. Most linters catch it. But what about this:

api_key = 1223

That's a real thing people do. Numeric API keys, numeric tokens, numeric "passwords" that are just IDs they didn't want to hardcode as strings for some reason. Most static analysis tools only check for string literals in sensitive variable assignments because that's the obvious case. They're checking the wrong layer — they're looking at the value type they expect rather than the variable name pattern combined with any literal at all.

The OptiScan security auditor checks both. It matches the variable name against a list of sensitive keywords — api_key, token, secret, password, credentials, stripe_key, twilio_token, twenty-odd others — and then checks whether the assigned value is any literal type: string, integer, float, all of it. If your variable name looks like it should be a secret, it shouldn't be assigned a literal. Ever. In any form.

It also catches eval() and exec() calls, subprocess with shell=True, AWS Access Key IDs embedded in string literals via regex, and a handful of dangerous module imports. The report comes back with line numbers and specific remediation instructions, not just a flag.

Cyclomatic Complexity Is One of Those Metrics That Sounds Boring Until You See It On Your Code

I'll be honest — I almost didn't add this. Cyclomatic complexity feels like the kind of thing you put in an enterprise tool so project managers have something to put in a quarterly report.

Then I ran it on some of my own old code.

The score is simple: start at 1, add 1 for every branch point. Every if. Every for. Every while. Every except. Every and and or in a boolean expression. Functions under 5 are LOW — easy to reason about, easy to write tests for. 6–10 is MEDIUM — manageable but worth a look. 11 and above is HIGH — that function is doing too many things and the test surface is enormous.

The thing about cyclomatic complexity is that it's not really about complexity. It's about testability. A function with a score of 14 has 14 independent paths through it. That's 14 test cases minimum for full branch coverage. In practice you're probably testing 3 of them and hoping the others behave. The score makes that problem visible in a way that reading the code doesn't always do.

Dead Code and Why It Accumulates

Every codebase I've worked in for more than a year has dead code in it. Functions that were replaced but not removed. Variables assigned in an early version of a feature that got refactored away. Imports for libraries that no longer get called.

It accumulates because removing it feels risky and finding it is annoying. You're never quite sure if something is actually unused or if you just can't see where it's used. So you leave it. Then the next engineer leaves it. Then it's been there for three years and nobody knows what it does or whether removing it will break something.

The dead code detector uses libcst's scope provider — actual scope analysis, not just text search — to track which names are defined and which names are referenced in the same module. Defined but never referenced means dead. It surfaces unused functions with their names, unused variable assignments, and unreachable code after return statements. Not heuristics. Not grep. Scope analysis.

On Building This Alone

I want to say something about the solo build experience because I think it's relevant to the tool itself.

When you're building something without a team, you have to be extremely honest about where the value actually is. You can't ship six half-finished features and call it a product. You have to pick the thing that is genuinely useful and make that thing work properly.

For me that meant the analysis had to be deterministic. Not "pretty good most of the time." Deterministic. If the security auditor says there's a hardcoded secret, there's a hardcoded secret. If the complexity engine says this is O(n²), it's O(n²). If the rewriter produces code, that code has to actually run and be correct.

That constraint forced some decisions that made the engine better. Using libcst instead of a language model for analysis. Testing the transformer output before marking anything as successfully converted — the bug where my transformer was silently deleting variable declarations because it marked a conversion as successful before checking if the conversion actually worked taught me that lesson very directly.

The engine is real. It has rough edges I'm filing down. But what it does, it does correctly.

Where It Goes From Here

GitHub PR integration is the one I'm working toward most urgently. Paste a pull request URL, get back an analysis of every Python file changed in the diff. That's the version that fits into a real engineering workflow without anyone having to change how they work.

After that: more language support, team workspaces so organisations can share scan history and audit reports, and a webhook mode so you can trigger scans automatically on push and route results to wherever your team already lives.

Try It

optirefine.qzz.io

Free tier available. No credit card. Paste something you've been staring at for too long and see what comes back.

If the engine misses something it should catch, or produces a rewrite that's wrong, I want to know. The best way to improve static analysis tooling is to throw real-world code at it — the messy, inconsistent, underdocumented code that actually exists in production, not the clean examples from tutorials.

That's the code that matters. That's what it's built for.

Solo-built. Actively maintained. Feedback welcome.

DEV Community: OptiRefine

Your RAG pipeline is bad at docs because your chunker splits code fences

1. It cuts code fences in half

2. It throws away where the chunk came from

3. chunk_size=1000 isn't 1000 of anything you care about

4. It has no idea your docs site has four versions of every page

Doing all four without writing a crawler

Two design decisions worth stealing even if you build your own

Embeddings without an API bill

The artifacts nobody asks for and everybody ends up using

Measure it on your own corpus, because I can't measure it on yours

Where I'd push back on myself

A short-form video script pipeline that runs on a schedule and needs no LLM API key

The output has to be structured, or none of it composes

The hook is a per-niche problem, not a prompt problem

The multi-category budget trap

Thin sources produce hallucinated scripts

Failed generations are a first-class output, not an exception

Generation without an API key

Voiceover, if you want it

What to expect from the output, honestly

Putting it on a schedule

Limits worth knowing before you build on it

After CHAINDROP, I stopped trusting npm audit. Here’s the dependency check I run instead.

npm audit answers the wrong question

The check I actually run

Step 1 — get your actual dependency list

Step 2 — score them in bulk

Step 3 — sort by the thing that actually predicts trouble

The four numbers worth writing down

How the health score is built (and why I kept it dumb)

Wire it into CI, or it won't happen

The part I'd argue about

Sources

How AuraWatch Finds Security Bugs Your Linter Will Never Catch

How AuraWatch Finds Security Bugs Your Linter Will Never Catch

The Problem With Existing Tools

The Six-Engine Pipeline

Engine 1 — LST Analyzer (Lossless Semantic Tree)

Engine 2 — Type-Aware Taint Analysis

Engine 3 — JS-SMT Symbolic Constraint Solver

Engine 4 — Async Flow and Prototype Pollution Tracker

Engine 5 — Hallucination Defense

Engine 6 — Framework-Specific Sink Maps

Deterministic Auto-Fixing

Architecture: Two Engines, One Interface

What's Next

Launch

The Code Review Comment I Left 47 Times

The Thing Nobody Talks About in Code Review

What I Actually Built

Let Me Show You What I Mean

The Security Part That Kept Surprising People

Cyclomatic Complexity Is One of Those Metrics That Sounds Boring Until You See It On Your Code

Dead Code and Why It Accumulates

On Building This Alone

Where It Goes From Here

Try It

3. `chunk_size=1000` isn't 1000 of anything you care about

`npm audit` answers the wrong question