DEV Community: Devil's Dev

RAG Pipeline Utils v3.8: Testing RAG Answers in TypeScript

Devil's Dev — Fri, 06 Nov 2026 10:07:03 +0000

v3.8.0 is about testing RAG in TypeScript. The answer matchers now compile with types, and a new guide walks through the whole setup with Vitest, from a unit test to a test set that fails CI when answers get worse. Nothing breaks for existing code.

npm install @devilsdev/rag-pipeline-utils@^3.8.0

The matchers, with types

@devilsdev/rag-pipeline-utils/testing has had three matchers for a while: toBeGroundedIn (every sentence of the answer is supported by its sources, numbers and negation included), toCite and toContainNoPII. They worked in JavaScript. In TypeScript, expect(result).toBeGroundedIn() didn't compile, because nothing told TypeScript the matchers existed. Worse, the declaration our own Testing page suggested failed under Vitest 5 with "All declarations of 'Assertion' must have identical type parameters".

3.8.0 exports their signatures as RagMatchers. Add them to Vitest's Matchers once, in a declaration file:

import "vitest";
import type { RagMatchers } from "@devilsdev/rag-pipeline-utils/testing";

declare module "vitest" {
  interface Matchers<R, T> extends RagMatchers<R> {}
}

Now the matchers type-check like any other assertion. A misspelled matcher is an error, and so is a wrong argument:

expect(answer).toCite(123)
Argument of type 'number' is not assignable to parameter of type 'string | string[]'.

Jest works the same way, with jest.Matchers<R>; the Testing page has both.

A guide: Testing RAG in TypeScript

RAG code rarely fails loudly: someone changes the chunk size, the prompt or the model, everything still runs, and an answer quietly says thirty days where the document says twenty. The new guide sets up a project that catches that, in three layers:

Unit tests with the typed matchers, on an offline test pipeline: deterministic, no API key.
Answer checks with checkAnswer, which return the exact sentence the sources don't support, for answers from any framework.
A test set scored by rag-pipeline eval, with minimums or a baseline from your main branch, failing the build when scores drop.

test("a wrong number is caught", async () => {
  const sources = [
    {
      pageContent:
        "Full-time employees receive twenty days of paid vacation per year.",
    },
  ];
  const answer =
    "Full-time employees receive thirty days of paid vacation per year.";

  expect(answer).not.toBeGroundedIn(sources);

  const check = await checkAnswer({ answer, sources });
  expect(check.grounded).toBe(false);
  expect(check.unsupported).toEqual([answer]);
});

The guide is built from a runnable project, examples/testing-typescript. A test keeps the two identical. Before each release, a check installs the packed package with the latest Vitest and TypeScript, then type-checks the project, runs its tests and runs its evaluation, so the code in the guide works as written. Read the guide.

Also

Releases announce themselves. This post reached you through it: publishing a release now posts to GitHub Discussions, dev.to, Bluesky and Mastodon, built from the release notes and this blog post. Every post passes checks first: each number must appear in the files it came from, and each link must be live.
Fewer development dependencies. Eight unused OpenTelemetry SDK packages and three modules nothing loaded are gone, 237 packages in all. npm audit findings in development tooling fell from 64 to 14, none high. What you install from npm had none and still has none.

The scoped packages, @devilsdev/rag-mcp-server and create-rag-pipeline are released at 3.8.0 too. Full details: CHANGELOG.

Originally published on the rag-pipeline-utils blog.

Meet rag-pipeline-utils: RAG for Node.js that tells you when your LLM is making things up

Devil's Dev — Fri, 06 Nov 2026 07:53:38 +0000

Your RAG chatbot just told an employee they get thirty vacation days. The handbook it retrieved says twenty. Nothing crashed, no test failed, and the answer reads perfectly well. That's how retrieval-augmented generation usually breaks: quietly.

I maintain @devilsdev/rag-pipeline-utils, an open-source RAG library for Node.js built around one idea: you should be able to check every answer against the sources it came from, cheaply, in your app and in CI. This post is a quick tour.

npm install @devilsdev/rag-pipeline-utils

Node.js 22+, with CommonJS, ESM and TypeScript types included.

Catch the made-up sentence

You don't have to build your pipeline with this library to use it. Bring an answer from anywhere (LangChain, Mastra, the Vercel AI SDK, LlamaIndex or your own code) and pass it in with the sources it was given:

import { checkAnswer } from "@devilsdev/rag-pipeline-utils";

const check = await checkAnswer({
  answer:
    "Full-time employees receive thirty days of paid vacation per year. " +
    "Remote work is allowed two days per week.",
  sources: [
    {
      pageContent:
        "Full-time employees receive twenty days of paid vacation per year.",
    },
    {
      pageContent:
        "Remote work is allowed two days per week with manager approval.",
    },
  ],
});

console.log(check.grounded); // false
console.log(check.groundedness); // 0.5
console.log(check.unsupported); // ["Full-time employees receive thirty days of paid vacation per year."]

It checks each sentence against the sources: the words, but also the numbers and the negations, which is where answers tend to go wrong. It runs locally with no API key and makes no model calls, so the same input gives the same verdict every time, offline, in tests.

If you're worried about correct paraphrases being flagged, pass a judge and your LLM is asked about the sentences that failed on wording alone, and nothing else.

A pipeline in a dozen lines

If you do want the library to run the pipeline, createRagPipeline() wires an embedder, a retriever, an optional reranker and an LLM. This one runs offline, with the built-in TF-IDF embedder and an in-memory store:

import {
  createRagPipeline,
  LocalEmbedder,
  MemoryRetriever,
} from "@devilsdev/rag-pipeline-utils";

const docs = [
  { id: "vacation", content: "Full-time employees get twenty vacation days." },
  { id: "remote", content: "Remote work is allowed two days per week." },
];

const embedder = new LocalEmbedder();
embedder.train(docs.map((doc) => doc.content));
const retriever = new MemoryRetriever();
await retriever.store(
  docs.map((doc) => ({ ...doc, vector: embedder.embed(doc.content) })),
);

const pipeline = createRagPipeline({ embedder, retriever });
const result = await pipeline.run({ query: "How many vacation days?" });

console.log(result.results[0].id); // "vacation"

Swap in OpenAI, Anthropic, Cohere or Ollama for the model, and pgvector, Qdrant, Pinecone or Weaviate for the store, when you're ready. Anything with an embed() or generate() method works too, including Vercel AI SDK models.

Prefer to start from a working app? npm create rag-pipeline@latest scaffolds one for Next.js with the AI SDK, Mastra, or Cloudflare Workers.

Test answers like any other value

RAG regressions don't throw. Someone changes the chunk size, the prompt, the model or the embedder, everything still runs, and the answers drift. So the library ships offline test helpers and matchers for Jest and Vitest:

import { expect, test } from "vitest"; // or Jest's global expect
import {
  createTestPipeline,
  ragMatchers,
} from "@devilsdev/rag-pipeline-utils/testing";

expect.extend(ragMatchers);

test("answers vacation questions from the handbook", async () => {
  const { pipeline } = await createTestPipeline();
  const result = await pipeline.run({
    query: "How many vacation days do I get?",
  });

  expect(result).toBeGroundedIn(); // every sentence supported by result.results
  expect(result).toCite("vacation");
  expect(result).toContainNoPII();
});

For whole test sets there's a CLI: rag-pipeline eval scores a dataset and fails when answers get worse than a saved baseline, and the rag-eval-action GitHub Action runs it on every pull request and posts the comparison as a comment.

What else is in the box

Framework integrations. groundingMiddleware adds retrieval and grounding to any Vercel AI SDK model without changing the rest of your code; there are scorers and guardrails for Mastra.
Guardrails. Prompt-injection detection and PII redaction, with published precision and recall, plus access-control filtering before anything reaches the model.
Production details. Hybrid search, reranking, token budgets, timeouts and cancellation, retries, caching, and OpenTelemetry (or Langfuse) traces with the evaluation scores attached.
The edge. A /edge build for Cloudflare Workers, Vercel Edge and Deno.
Claude, via MCP. @devilsdev/rag-mcp-server lets Claude search a folder of your documents and check its own answers against them.
Take only what you need. The main package has everything; smaller packages like @devilsdev/rag-eval (about 6 MB installed) are there if install size matters.

Try it

The fastest way in is the answer check on StackBlitz: paste an answer of your own, with no install or API key.

Documentation
GitHub (issues and ideas welcome, and a star helps other developers find it)
npm

I'd love to hear how you test your RAG apps today, and what you'd want a library like this to catch. Tell me in the comments.

RAG Pipeline Utils v3.7.1: Security Fixes From a Code-Scanning Review, and a New npm Page

Devil's Dev — Fri, 06 Nov 2026 07:44:06 +0000

v3.7.1 is a patch release with no breaking changes. It fixes a slow pattern in QueryPlanner that a long query could use to tie up a server, and two ways a crafted __proto__ key could write to Object.prototype in the CLI. It also gives the npm page a new README. Upgrading is recommended if you pass user queries to QueryPlanner.

npm install @devilsdev/rag-pipeline-utils@^3.7.1

How these were found

Main had 52 open code-scanning alerts from CodeQL and OpenSSF Scorecard, most of them old. We went through every one of them against the code:

timed each slow-pattern alert;
reproduced each prototype-pollution alert;
checked whether the flagged code is actually shipped in the published package.

The result:

19 are fixed in this release: the code issues below, plus Docker image pins and tighter workflow permissions.
27 were false alarms, and each was dismissed on GitHub with its reason. Examples: modules that aren't in the published package, test fixtures with attack strings, and a random session label that was never a secret.
The 6 still open are repository settings (code review, branch protection, fuzzing) or an example that has no lockfile on purpose.

`QueryPlanner` and long runs of spaces

CodeQL flagged QueryPlanner's ; pattern as slow on long input. Timing the whole plan() call found a worse one it hadn't flagged:

The pattern for "…, and how …" clauses was cubic in the length of a run of spaces: 4,000 spaces took 21 seconds, and 20,000 didn't finish within a minute.
The "vs" and comparison patterns were quadratic.

QueryPlanner is exported from the main entry and from /edge, and it's given user queries, so one long query could hold a server's CPU.

The planner now reads each run of whitespace as one space before matching, and the clause boundaries can split a run only one way. On every query in the test suite and the edge cases we added, it splits exactly as before, and 200,000 characters of any shape we tried plan in under 100 ms. The one visible change: a sub-query has single spaces where the query had runs of whitespace.

`proto` in the CLI

rag-pipeline plugin install. A plugin whose package.json declared "stage": "__proto__" passed the stage check, and registering it wrote to Object.prototype in the CLI process. Only the stage names the installer knows are accepted now.
rag-pipeline config set. The key path was checked, but a __proto__ key inside the JSON value was merged into Object.prototype. Such keys are skipped now, and the merge only goes into the config's own objects.

Both fixes come with regression tests that fail on 3.7.0.

A new README on npm

The npm page now speaks to developers meeting the library for the first time. It opens with the problem the library solves, an answer that says thirty vacation days when the source says twenty, and shows checkAnswer catching exactly that sentence. Then a pipeline in a dozen lines, and what else is in the box. Both examples run in CI, so they stay correct.

Also

The eval Markdown report escapes backslashes, so an error message containing \| no longer breaks a table cell.
Docker base images are pinned by digest, and Dependabot keeps those digests current.
The Security and Supply Chain workflows are read-only by default.

The full list is in the CHANGELOG.

Originally published on the rag-pipeline-utils blog.

DEV Community: Devil's Dev

RAG Pipeline Utils v3.8: Testing RAG Answers in TypeScript

The matchers, with types

A guide: Testing RAG in TypeScript

Also

Meet rag-pipeline-utils: RAG for Node.js that tells you when your LLM is making things up

Catch the made-up sentence

A pipeline in a dozen lines

Test answers like any other value

What else is in the box

Try it

RAG Pipeline Utils v3.7.1: Security Fixes From a Code-Scanning Review, and a New npm Page

How these were found

QueryPlanner and long runs of spaces

__proto__ in the CLI

A new README on npm

Also

`QueryPlanner` and long runs of spaces

`proto` in the CLI