DEV Community: Sid Probstein

Get Every Email to Your Domain in One Gmail Inbox, Free, with Cloudflare

Sid Probstein — Fri, 14 Aug 2026 18:29:26 +0000

You own a domain. Your website lives on it. With Cloudflare's free plan you can make anything@yourdomain.com - any address, invented on the spot - land in your regular Gmail inbox. No Google Workspace subscription, no mail server, no changing registrars. About 30 minutes, $0.

What you get:

A catch-all: every address at your domain forwards to one inbox.
Free disposable addresses: give netflix@yourdomain.com to Netflix and bank@yourdomain.com to your bank. When spam arrives addressed to one of them, you know who leaked your address.
Your website, untouched.

One limitation: this is receive-only. You can reply as you@yourdomain.com from free Gmail via "Send mail as", but the mail goes out through Google's servers without your domain's blessing, so some recipients see "via gmail.com" and strict spam filters may object. If sending from the domain matters, that's what Google Workspace is for. For receiving, read on.

You need: a domain (registered anywhere; it stays there), a free Cloudflare account, and an inbox to receive the mail.

Step 1: Add your domain to Cloudflare

Cloudflare scans your existing DNS and imports what it finds. Review this list carefully - the scan is good but not guaranteed complete, and whatever it misses stops resolving after the switch. Check what your DNS really says from any terminal:

dig +short A yourdomain.com
dig +short MX yourdomain.com
dig +short NS yourdomain.com

If the MX query returns nothing, you have no existing email service and this migration is pure upside. If it returns something, understand what that mail service is before proceeding.

Step 2: The gotcha - where does your DNS actually live?

That NS query tells you who currently answers DNS for your domain, and it is not always your registrar.

In my case the nameservers pointed at NS1 (*.nsone.net) - Netlify DNS. My registrar just held the registration; the DNS zone lived in Netlify, from clicking "use Netlify DNS" when connecting my site years earlier. The A records Cloudflare imported were Netlify load balancer IPs, which Netlify rotates. Had I switched nameservers and kept those records, my site would have broken on the next rotation.

The fix: point at names, not IPs. For a Netlify site (same idea for Vercel, GitHub Pages, and friends):

Find your site's default subdomain in the Netlify dashboard (your-site.netlify.app).
In Cloudflare's DNS page, delete the imported A records.
Add two CNAMEs, both DNS only (click the orange cloud so it turns grey):
- www → your-site.netlify.app
- @ (apex) → your-site.netlify.app (Cloudflare flattens apex CNAMEs automatically)

DNS-only matters. Your host already runs a CDN and manages your HTTPS certificate; stacking Cloudflare's proxy in front invites redirect loops and cert renewal failures for zero benefit. Grey cloud means Cloudflare only answers DNS. Email Routing doesn't use the proxy, so it works either way.

Step 3: Switch nameservers at your registrar

Cloudflare assigns you two nameservers, like ada.ns.cloudflare.com and bob.ns.cloudflare.com. At your registrar, replace the current nameservers with those two. The registrar keeps the registration and the renewal bill; it just stops answering DNS questions.

Propagation is faster than its reputation - the change hits the .com registry within minutes:

dig +noall +authority NS yourdomain.com @a.gtld-servers.net

When that shows the Cloudflare nameservers, click Check nameservers now on your domain's Cloudflare overview page instead of waiting for its periodic poll. Then load your website. If you did Step 2 right, nothing changed.

Step 4: Turn on Email Routing

In the Cloudflare dashboard for your domain: Email → Email Routing → Get started.

Add a destination address - your Gmail. Click the link in the verification email Cloudflare sends.
On Routing rules, enable the Catch-all with action "Send to an email" pointed at your verified destination.
Accept the DNS records Cloudflare offers to add: three MX records (route1/2/3.mx.cloudflare.net) that direct your domain's mail to Cloudflare, plus SPF and DKIM records that keep forwarded mail from being spam-flagged. The odd-looking MX priorities are normal.

That single catch-all rule is the whole feature.

Step 5: Test it

dig +short MX yourdomain.com
# 36 route3.mx.cloudflare.net.
# 95 route2.mx.cloudflare.net.
# 98 route1.mx.cloudflare.net.

Send a message from an account other than the destination inbox (Gmail hides mail you send to yourself) to an address you just made up: whatever@yourdomain.com. It should arrive within seconds. Check spam the first time and hit "Not spam" if needed.

Caveats

Receive-only, as covered above. Email Routing forwards; it doesn't send or store.
Catch-alls attract spam eventually - dictionary-attack spam to random@yourdomain.com all lands on you. Gmail's filtering handles it, and the per-service trick turns the bug into a feature: add a Cloudflare rule to drop any address that gets burned.
Your old DNS host's features die at the switch - web redirects, parked pages, registrar email forwarding. That's the point, but inventory what you use before you flip.

Half an hour, zero dollars, and every address at your domain lands in one inbox.

Enterprise source integration as configuration: inside SWIRL's SearchProviders

Sid Probstein — Fri, 07 Aug 2026 03:51:36 +0000

The expensive part of federated search was never the query fan-out.
It's the N integrations behind it, each with its own API, auth, query
syntax, and response format. SWIRL's answer is to make each integration a configuration object called a SearchProvider, and the 66-second video above walks through one in the admin tool.

The anatomy, per provider:

Connection: which connector to use (HTTP GET/POST returning JSON, M365 Graph, SQL, Elastic DSL, ...), the endpoint, and the authenticator that supplies each user's credentials at query time.
Query: a template that maps SWIRL's query into the source's own syntax, plus query processors that adapt it automatically (synonyms, rewrites, source-specific quirks).
Results: JSONPath-style mappings that normalize each source's response into a common result shape: title, body, author, date, URL. This is what makes cross-source ranking honest; every result competes in the same schema, re-ranked by the same three-pass pipeline.

Because providers are rows, not code, the workflow is: activate a
preloaded provider (M365, Box, ServiceNow, Elastic, PostgreSQL,
BigQuery, and dozens more ship in the box), add credentials, search.
Change a mapping and the next query uses it; there is no deploy step.

Example provider JSON: https://github.com/swirlai/swirl-search/tree/main/SearchProviders

Guide: https://docs.swirlaiconnect.com/SP-Guide.html

Replaced Salesforce and HubSpot with Postgres, a Spreadsheet, and an AI Agent

Sid Probstein — Tue, 04 Aug 2026 16:00:51 +0000

We have used two real CRMs.

First Salesforce, because our founding sales person said it was essential. Then HubSpot, because our founding marketing person said it was essential. Both people were right that tracking the pipeline is essential... but really, neither system earned its keep at our scale.

What the data actually looked like, once it escaped, was a spreadsheet: three columns of company names labeled Engaged Prospects, Legal, and Partners/Channels. >100 organizations, no statuses, no activity history, no next steps.

A full CRM had twice proven to be overkill; the spreadsheet was underkill.

So I replaced both with a mini-CRM built in an afternoon with Claude Cowork, using tools we already run: local PostgreSQL, the spreadsheet itself, and SWIRL for search. Here is the full build, including the bug we shipped and caught.

Step 1: From three columns to a schema

The agent read the spreadsheet with openpyxl and proposed a schema built for the actual job, which is tracking activity, not just listing names:

CREATE TABLE organizations (
    id          SERIAL PRIMARY KEY,
    name        TEXT NOT NULL,
    category    TEXT NOT NULL CHECK (category IN ('engaged_prospect', 'legal', 'partner')),
    status      TEXT NOT NULL DEFAULT 'active',   -- free-form: active, stalled, won, dead...
    notes       TEXT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (name, category)
);

CREATE TABLE contacts (
    id      SERIAL PRIMARY KEY,
    org_id  INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    name    TEXT NOT NULL,
    title   TEXT,
    email   TEXT,
    phone   TEXT
);

CREATE TABLE activities (
    id            SERIAL PRIMARY KEY,
    org_id        INT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
    activity_date DATE NOT NULL DEFAULT CURRENT_DATE,
    activity_type TEXT NOT NULL DEFAULT 'note',   -- call, email, meeting, demo...
    summary       TEXT NOT NULL,
    next_step     TEXT,
    next_step_due DATE
);

Two design decisions that paid off immediately:

Activities are append-only history, not a mutable "last touched" field. You never lose the story of an account.
Status is free-form text. We started with a suggested vocabulary, but the first real update was "dead", which was not on the list. A CHECK constraint on status would have turned a one-word instruction into a schema migration. Constrain the things that break joins (category); leave vocabulary to the humans.

Two views do most of the daily work:

-- Latest activity per organization
CREATE VIEW org_latest_activity AS
SELECT o.id, o.name, o.category, o.status,
       a.activity_date AS last_activity_date,
       a.activity_type AS last_activity_type,
       a.summary AS last_activity_summary,
       a.next_step, a.next_step_due
FROM organizations o
LEFT JOIN LATERAL (
    SELECT * FROM activities a
    WHERE a.org_id = o.id
    ORDER BY a.activity_date DESC, a.id DESC
    LIMIT 1
) a ON true;

-- The daily to-do list
CREATE VIEW followups_due AS
SELECT o.name, o.category, a.next_step, a.next_step_due
FROM activities a
JOIN organizations o ON o.id = a.org_id
WHERE a.next_step IS NOT NULL
  AND (a.next_step_due IS NULL OR a.next_step_due <= CURRENT_DATE)
ORDER BY a.next_step_due NULLS LAST;

An idempotent loader script pulled every name out of the three spreadsheet columns and inserted it with the right category. ON CONFLICT (name, category) DO NOTHING means re-running it after the spreadsheet grows is always safe.

Step 2: Make it searchable with SWIRL

We run SWIRL for federated search, so the obvious next move was a SearchProvider that folds the CRM into the same search box as everything else.

First, a denormalized view so a single ILIKE sweep can hit everything worth matching:

CREATE VIEW crm_search AS
SELECT o.id, o.name, o.category, o.status,
       COALESCE(o.notes, '') AS notes,
       la.last_activity_date, la.last_activity_type,
       COALESCE(la.last_activity_summary, '') AS last_activity_summary,
       COALESCE(la.next_step, '') AS next_step,
       la.next_step_due,
       COALESCE((SELECT string_agg(
                     c.name || COALESCE(' (' || c.title || ')', '')
                            || COALESCE(' <' || c.email || '>', ''), '; ')
                 FROM contacts c WHERE c.org_id = o.id), '') AS contacts,
       o.updated_at::date AS updated
FROM organizations o
LEFT JOIN org_latest_activity la ON la.id = o.id;

Then the provider. SWIRL's PostgreSQL connector takes a query template with mapped fields:

{
    "name": "Mini-CRM - PostgreSQL (SQL)",
    "connector": "PostgreSQL",
    "url": "localhost:5432:crm:<db-user>:<db-password>",
    "query_template": "select {fields} from {table} where {field1} ilike '%{query_string}%' or {field2} ilike '%{query_string}%' or {field3} ilike '%{query_string}%' or {field4} ilike '%{query_string}%' or {field5} ilike '%{query_string}%' or {field6} ilike '%{query_string}%'",
    "query_mappings": "fields=*,sort_by_date=updated,table=crm_search,field1=name,field2=notes,field3=last_activity_summary,field4=contacts,field5=category,field6=next_step",
    "result_processors": [
        "MappingResultProcessor",
        "CosineRelevancyResultProcessor"
    ],
    "result_mappings": "title='{name} ({category} / {status})',body='Last activity: {last_activity_summary} ({last_activity_type}, {last_activity_date}). Next step: {next_step} (due: {next_step_due}). Contacts: {contacts}. Notes: {notes}',date_published=updated"
}

Searching a company name, a status ("won"), a category ("partner"), or a phrase from a call summary all just work, ranked by SWIRL's relevancy pipeline alongside every other source.

The result_mappings line went through one iteration worth mentioning. The stock SQL-provider pattern is result_mappings: "DATASET", which collapses all rows into a single result carrying a table payload. Fine for analytics, wrong for a CRM: you want each account as its own result with its status and next step visible. Switching to explicit template mappings gives every org its own card:

Acme Manufacturing (engaged_prospect / active)
Last activity: intro call re search POC (call, 2026-07-15).
Next step: send scoping doc (due: 2026-07-25). ...

Mapping date_published=updated also makes date-sorting real instead of "unknown".

One more detail for the LLM era: SWIRL providers can carry query_instructions in their config, which is handed to an LLM when the source is queried through SWIRL's assistant or MCP server. Ours documents the full schema, the enumerated category/status values, and five SQL templates (activity history, due follow-ups, pipeline counts). That turns "what needs attention this week?" into real SQL against followups_due instead of a keyword guess.

Step 3: The update loop nobody hates

The first plan for data entry was conversational: the agent walks through each account and asks what happened. That died on contact with reality after one answer. Dictating 100+ updates one at a time is miserable.

The fix: the spreadsheet stays the editing surface. The agent added a second sheet, "Tracking", one row per organization, pre-filled from the database:

Name | Category | Status | Activity Date | Activity Type | Activity Summary | Next Step | Next Step Due | Notes

Edit any cells, save, run the sync:

Status or Notes differ from the DB: the org is updated.
Activity Summary is filled in and differs from the org's latest activity: a new activity row is inserted. History accumulates; the spreadsheet only ever shows the latest.
New name with a category: org created.
The script never deletes anything, and running it twice in a row is a no-op.

After any bulk change made directly in SQL, the sheet is regenerated from the database so the two never fight.

For bulk updates, plain English turned out to beat both surfaces. "Mark everything dead except these accounts; add this new deal as won via partner X" became one reviewed transaction. The agent applies it, prints the resulting pipeline counts, and refreshes the sheet.

The bug: names are not keys

The sync script's first version keyed organizations by name. One company in our data legitimately exists in two categories (it is both a service firm and a channel partner), which is exactly why the table's unique constraint is (name, category).

The script's in-memory state dict silently kept only one of the two rows, and its UPDATE ... WHERE name = X hit both. Net effect: syncing the sheet resurrected a row that a bulk update had just marked dead.

It surfaced immediately for one reason: the sync prints a change report, and we expected zero. A refresh-then-sync cycle should be a perfect no-op, and it reported "1 org updated" instead. That single unexpected line of output was the whole detection mechanism. The fix was mechanical (key by name plus category, scope every UPDATE and INSERT the same way), verified by re-running until the no-op was real.

If your loader is idempotent, "re-run it and demand zero changes" is the cheapest integration test you will ever write.

What we ended up with

A Postgres database with full activity history, one psql away.
A spreadsheet that is now a UI, not a database.
CRM accounts as first-class results in our federated search, next step and status on the card.
An LLM-queryable source: schema-aware SQL through SWIRL's MCP server.
A natural-language admin loop for bulk operations, with SQL you can read before it runs.

Total schema: three tables, three views, zero ORM, and two canceled CRM subscriptions. Claude Cowork wrote the schema, the loaders, the provider, and the bug; the change report caught the bug; the humans just answered questions and edited cells.

Would this scale to a 50-seat sales team? No, and it is not trying to. For a founder-led pipeline of a hundred accounts, the boring stack is hard to beat: every piece is inspectable, every update is a SQL statement you can read, and the search box already knew where to look.

Taught Claude Code to Install, Configure, and Debug SWIRL Enterprise Search Engine

Sid Probstein — Mon, 03 Aug 2026 14:49:47 +0000

Evaluating enterprise search software has a fixed cost that has nothing to do with the software: stand up a server, read the docs, connect a source, debug the connector, configure an LLM. By the time you see a real answer from your own data, a week is gone.

We built a Claude Code plugin that compresses that to about an hour, and along the way learned something more interesting than the time savings: a plugin skill is an executable runbook, and the best way to write one is to watch where the agent researches and move that knowledge upstream.

What it is

swirl-claude-plugin is free and Apache-2.0. It gives Claude Code eight guided workflows for SWIRL, a federated search + RAG engine:

claude plugin marketplace add swirlai/swirl-claude-plugin
claude plugin install swirl@swirl --scope user

Start a new Claude Code session and you have:

Command	What it does
`/swirl:start`	Assesses where you are and routes you
`/swirl:install`	Docker or local install, verified with a live search
`/swirl:provider`	Connect a source using 25+ built-in connectors
`/swirl:connector`	Develop a custom connector
`/swirl:rag`	Connect an LLM, configure ranking models, stand up cited answers
`/swirl:migrate`	Community-to-Enterprise moves and version upgrades
`/swirl:mcp`	Wire SWIRL's MCP server into Claude
`/swirl:troubleshoot`	Diagnose from logs; file a support ticket when diagnosis needs help

Why "an hour" is honest

SWIRL is federated. It queries SharePoint, OneDrive, Box, databases, and web APIs where they live, re-ranks results with local models, and feeds the best passages to the LLM you configure. Nothing is copied, ingested, or indexed; there is no vector database to build.

That is why the evaluation is fast: the usual multi-day step (moving your data into the search engine) does not exist. Claude installs the stack, connects two or three sources, and you are looking at ranked, cited answers from your own documents.

Skills are runbooks, not code

There is no code in the plugin. Each command is a skill: a markdown file whose frontmatter description routes the request and whose body loads focused operational knowledge into Claude's context. The install skill knows the real Docker quick start for each edition and insists on verifying with a live search before declaring success. The troubleshoot skill embodies a support rule we enforce on ourselves: the user's report is data; find the log line before proposing a fix. The skills also encode what NOT to do: never print API keys, never paste secrets into chat, never restart a stack someone is demoing on.

The development loop that actually improves it

Here is the pattern we now use for every release, with a real example.

During a walkthrough, a user asked Claude to "enable the cross-encoder and use larger embeddings instead of spaCy." Claude did what a capable agent does with missing knowledge: it researched. A dozen commands into settings files and compiled modules, the user steered it: ranking models in SWIRL are configured as AI Providers, in the database, like every other model choice. Once pointed there, Claude flipped the reader role to mxbai-embed-large served by the stack's own Ollama sidecar, verified the cross-encoder was already active in each result's explain output, and confirmed the change took effect with no restart.

That transcript became the next release. The rag skill now states it outright: ranking models are AI Providers; do not hunt through settings or source. The cross-encoder ships on by default, so confirm it rather than configure it. The embedding model is whatever provider holds the reader role. Changes apply at query time.

The general form: watch where your agent researches, then move that knowledge into the skill. Research is the tax an agent pays for what the skill does not say. Every walkthrough transcript is a free curriculum; the diff between "what the agent had to discover" and "what it should have known" is exactly the next version of the skill.

If you are building your own plugin

Three practices that will save you real time:

Give the plugin an explicit identity. Ship a .claude-plugin/plugin.json with the plugin's name and version, and keep component definitions (skills, commands) in exactly one place. Ambiguity here surfaces on the user's machine, not yours.
Test the exact path a stranger will run, from a clean environment. HOME=$(mktemp -d) claude plugin marketplace add you/your-repo && claude plugin install ..., then verify the user-visible surface: the actual command names in a session, not just "installed." Manifest validators check syntax, not experience.
Write the description for routing. The frontmatter description is what decides whether a user's phrasing reaches your skill. When a real user's words missed, we added their words to the description.

The loop closes with MCP

The part we like most: /swirl:mcp has Claude wire SWIRL's MCP server into itself. The same assistant that installed your knowledge layer then queries it, with permissions enforced server-side per user. Your agents get governed access to enterprise data; the data never moves.

Try it

Plugin: claude plugin marketplace add swirlai/swirl-claude-plugin
Docs: docs.swirlaiconnect.com/claude-plugin
One-page PDF: datasheet, no registration
SWIRL Community is open source; want an Enterprise license to evaluate? Work email in, license out; no human will contact you.

Issues and PRs welcome on the repo.

Turning Slackbot into an Enterprise Research Agent with MCP

Sid Probstein — Tue, 28 Jul 2026 15:04:03 +0000

We connected a Slack agent to our federated search engine over the Model Context Protocol, then asked it a question only our OneDrive could answer. It searched, read the document, and replied in Slack with the coverage limits in a table and a citation that opens the exact sentence in SharePoint.

This post covers the architecture, the wiring, and two gotchas that cost us real debugging time.

The architecture

Three pieces, all standard:

A Slack agent that speaks MCP as a client. Slack's agent platform lets a bot declare MCP servers in its manifest and connect to them at runtime.
SWIRL's MCP server. SWIRL is a federated search + RAG engine; its MCP server exposes search, search_rag (grounded answers with citations), read_document, score_document, list_providers, and chat. It's a standalone process that calls a running SWIRL deployment over HTTP, so licensing, throttling, and per-user permissions are enforced by SWIRL exactly as for any API client.
The sources. SharePoint, OneDrive, Box, databases, web APIs; SWIRL queries them where they live. Nothing is copied into the bot, the model, or a vector database.

The wiring

On the Slack side, the bot's manifest declares the MCP server and the bot connects with a slash command. On the SWIRL side, the server runs in HTTP transport mode:

SWIRL_MCP_TRANSPORT=http SWIRL_MCP_PORT=8675 \
SWIRL_MCP_TOKEN=<api-key> \
python -m swirl_mcp

For the demo we exposed it through a tunnel with auth disabled; do not do that in production. The production path is the server's OAuth 2.1 resource-server mode (SWIRL_MCP_AUTH=oidc): the MCP host runs PKCE against your IdP, the server validates each bearer JWT (issuer, audience, JWKS signature), and SWIRL maps the token to the real calling user, so every search is permission-trimmed per caller.

What a query looks like

Asked "Search OneDrive for SWIRL's insurance policy", the bot:

calls list_providers to find the OneDrive source id;
calls search scoped to that provider;
calls search_rag for a grounded answer over the results.

The reply carries the policy limits as a table, flags that professional services are excluded from that policy, and includes a text_fragment_url citation: a deep link that opens the source document scrolled to the exact quoted passage.

Two gotchas worth knowing

Timeout mismatch. Slack gives an MCP tool call about 60 seconds; SWIRL's search_rag polls up to 90 seconds for the RAG answer, sized for a cold local model. On a slow model the Slack call dies first and the bot reports a failure while the answer lands seconds later. Fix: fast model for Slack-facing RAG, or lower SWIRL_MCP_RAG_POLL_TIMEOUT so failures are honest.

Tool descriptions are UX. The bot's first act in the demo is describing SWIRL's tools, in its own words, accurately. That's not the bot being clever; it's the MCP server's tool descriptions being written for an LLM audience. If your agent misuses a tool, fix the description before touching the agent.

The takeaway

Agents don't need your data ingested into them; they need a governed search layer. The bot never held our documents. It held a connection to a server that could search them, as us, with permissions enforced and citations attached.

The same MCP server works from Claude Desktop and Claude Code. There's a Claude Code plugin that stands up the whole stack: swirl-claude-plugin, and the 2-minute demo video is here: [LINK-TO-VIDEO].

Docs: docs.swirlaiconnect.com

SWIRL 5 is GA: knowledge authority for enterprise AI

Sid Probstein — Wed, 22 Jul 2026 15:04:12 +0000

SWIRL 5 is generally available! I want to use this post to explain what it is at an engineering level, because the one-line pitch ("the knowledge authority layer for enterprise AI") does not tell you how it works or where the hard parts are. I would rather show you the machine.

Quick disclosure: I wrote the original version of SWIRL and I run the company. So read this as the person who built it explaining the design, not as a neutral survey. I have tried to be honest about the limits, including where a hand-tuned stack matches us.

The thing that changed

A year ago, "federated search across your systems, then re-rank the results" was a product you could sell. Today it is table stakes. MCP turned retrieval into a commodity: any model can reach any system through a connector, and every serious stack ships a re-ranker. If your pitch is "we retrieve and we re-rank," a technical evaluator will point at four other tools that do the same thing by lunch.

So the interesting question moved. It is no longer "can you find the document." Everything finds the document. It is "which of the versions you found is the one my organization actually stands behind." In a real enterprise corpus the answer to any given query does not exist once. It exists as a draft, three redlines, a copy someone saved to their desktop, and the executed final, all sitting in different systems, all semantically near-identical. Retrieval returns all of them. The model picks one, confidently, and it has no idea which one carries authority.

That is a governance problem wearing a search costume, and it is what SWIRL 5 is built to solve.

Canonical version election

The core primitive is version election. When SWIRL federates a query and gets back a cluster of near-identical documents, it does not just hand the pile to the model. It:

Clusters the versions it found across every source.
Scores each on signals that actually correlate with authority: source authority (an executed contract in iManage outranks a draft on someone's OneDrive), naming ("Executed", "Final", version numbers), and recency.
Elects one canonical version, and exposes its reasoning so a human can see why.

On top of that, teams can pin a canonical result for a query directly. Once pinned, every later search and every agent calling SWIRL gets the endorsed answer, full stop. Election is the automatic path; pinning is the human override. Both produce the same thing: a single answer the organization has stood behind, not the model's best guess.

This is the piece the frontier models cannot do for themselves. Claude and Copilot are extremely good at drafting and summarizing. Neither has any way to know which of your nine versions is ratified, because that fact does not live in the documents. It lives in your organization.

Ranking: three passes, run locally, no vector database

Under the election sits the ranking pipeline. It runs in three passes, and both models run locally in your tenant. Nothing goes over the wire to a third-party ranking service, and there is no vector database to build, secure, or keep in sync.

Keyword + BM25. Lexical first. Quoted phrases and exact terms are honored as written. In enterprise and legal content this matters: a defined term or a specific clause has to match exactly, not approximately.
Embedding re-rank. E5-Large-V2 embeddings with title-aware chunking, fused with the lexical scores using reciprocal rank fusion. Semantic recall, without letting it steamroll the exact matches from pass one.
Cross-encoder. MS-MARCO cross-encoder reads the query and each candidate document together, as a pair, and scores real relevance rather than vector similarity. It is the expensive pass, which is exactly why it runs last, only on the candidates that survived the first two.

The "no vector database" part is not a slogan, it is a deployment property. There is no second copy of your content sitting in an index that your information-governance policy never contemplated. The documents stay in iManage, Box, SharePoint, wherever they already live. Only the ranking happens in SWIRL. For a regulated buyer, removing that second copy removes an entire category of risk.

If you want the independent version of this argument, Meta's XetHub benchmarked keyword-only, vector-only, and hybrid re-rank, and hybrid won. Their post was literally titled "you don't need a vector database."

Assembly: a bounded prompt, not a context dump

Retrieval and ranking decide what is relevant. Assembly decides what the model actually sees, and that is where the token bill and a lot of the accuracy live.

SWIRL treats the prompt as a hard budget, around 3,000 tokens, and fills it deliberately: at most 10 sources, only those scoring above a relevance threshold, a topic matcher that scores the spans inside each source so only the passages that answer the query go in, per-source truncation to fit, and version de-duplication that drops the superseded copies before assembly. One bounded call, not the refine or tree-summarize modes that call the model once per chunk and multiply both tokens and latency.

The counterintuitive result, which we measured against LangChain and LlamaIndex defaults and checked against their source: in a versioned corpus, sending less produces a better answer. More context means more near-identical duplicates, and the model gets less certain, not more. We wrote that study up separately if you want the numbers and the honest caveats.

Grounding checks

Every generated answer is checked against the sources it cited. Claims that are not supported by the retrieved passages are flagged rather than shipped silently. This is not a hallucination cure, nothing is, but it changes the failure mode from "confident and wrong and unmarked" to "flagged for a human." In an enterprise setting that distinction is the whole game.

How it fits your stack

SWIRL 5 is headless and API-first, with a first-class MCP server. Any AI work surface calls it over MCP or REST and gets back ranked, permissioned, canonical answers with citations attached. It runs in your tenant, honors each source system's existing permissions on every query, and works with the model you choose, hosted Claude or GPT, Copilot, or a local model on your own hardware. The agent angle is the one I would watch: a human reading a superseded policy usually catches it, an agent does not pause, so serving agents the approved answer instead of a raw retrieval is a safety property, not a nicety.

Where it is hard, honestly

Version election is heuristic. Source authority, naming, and recency get you a long way, but naming conventions are messy and "most recent" is not always "operative." Pinning exists precisely because the automatic signal is not always enough. We show the reasoning so humans can correct it.
A carefully hand-tuned vanilla RAG can match our per-call token count. Low k, a good reranker, small chunks, plus your own de-dup layer. Our value is delivering that discipline by default and adding cross-version de-duplication that off-the-shelf stacks do not, not magic per-token compression.
Grounding checks reduce unsupported claims, they do not eliminate them. Treat the flag as a prompt for review, not a guarantee.

Try it

If you are putting AI on your own data, that is exactly the case this is built for. It is generally available now at swirlaiconnect.com, and I am happy to run it against a slice of your own systems so you can see the ranking, the citations, and the permission boundary on your data rather than a demo corpus. I built it, so you would be talking to the person who wrote the code.

How many tokens does your RAG stack actually send to the LLM?

Sid Probstein — Tue, 07 Jul 2026 13:13:00 +0000

The token bill for a RAG system is not set by your vector database. It's set one step later, by how you assemble the context you hand the LLM. Retrieval finds candidates; assembly decides how many of them, how much of each, and across how many LLM calls. That's where the money is spent.

So "vector DB vs. framework X vs. SWIRL" is the wrong axis. The real comparison is between context-assembly strategies. Here's an honest, source-checked look at how many tokens each common approach sends, and where SWIRL 5 actually costs less.

The honest headline first

SWIRL doesn't win by using smaller chunks. Anyone can lower top_k. On a single lean query, a minimal config like LlamaIndex's default (top_k=2) sends fewer raw tokens than SWIRL. If someone tells you SWIRL "always uses the fewest tokens," a technical evaluator will disprove it in five minutes.

SWIRL's advantage is structural, and it shows up exactly where enterprise content lives, in corpora full of document versions:

It never spends tokens on duplicate or superseded versions. Stock top-k returns whatever is nearest in embedding space, which in a versioned corpus means several near-identical copies. SWIRL collapses them to one canonical document before the LLM sees them.
It always answers in one bounded call. The multi-document synthesis modes people reach for when they want quality (LangChain refine/map_reduce, LlamaIndex refine/tree_summarize) multiply LLM calls, and refine multiplies tokens super-linearly. SWIRL never pays that tax.

What the common defaults actually do

Naive vector-DB RAG (Pinecone / Weaviate / Qdrant tutorials)

The pattern is: embed the query → retrieve top-k chunks → stuff them all into one prompt. Typical documented defaults:

Chunk size ~512 tokens (the common "start here"; the band is 256-1024)
Chunk overlap 10-20%
top_k 3-5
No de-duplication. None of the vendor quickstarts add a dedup or diversity step. Overlap alone guarantees adjacent chunks share text, and multiple versions of a document sit in the same embedding neighborhood, so top-k routinely returns redundant context and the pipeline sends all of it.

Input tokens ≈ k × chunk_tokens + overhead. At k=5 and ~1,000-token chunks, that's ~5,000 tokens, a large fraction of it redundant.

Sources: Pinecone, Weaviate, Qdrant.

LangChain

Verified from source:

Retriever default k=4 (similarity_search(k=4) in vectorstores/base.py)
RecursiveCharacterTextSplitter default 4,000 chars / 200 overlap (≈1,000 tokens) (text_splitters/base.py)

The combine-documents chains differ enormously in cost:

Chain	LLM calls (N docs)	Token behavior
`stuff`	1	all docs in one prompt
`map_reduce`	N + ≥1	one call per doc, then reduce
`refine`	N, sequential	re-sends the growing answer each step → super-linear tokens, no parallelism
`map_rerank`	N	one doc per call

refine is the trap: each step re-transmits the accumulating answer plus the next document, so the running answer is re-sent N-1 times and grows as it goes.

LlamaIndex

Verified from source (llama-index-core):

chunk_size 1,024 tokens, similarity_top_k 2, default response mode compact (constants.py, factory.py)
No default de-duplication. Node postprocessors are opt-in and the default similarity cutoff is off, so two versions of a doc in top-k both go to the LLM.

Response modes (docs):

Mode	LLM calls	Token behavior
`compact` (default)	~1	packs nodes into as few prompts as fit
`refine`	N	one call per node, re-sends evolving answer
`tree_summarize`	>1, recursive	summarize groups, then summaries-of-summaries
`accumulate`	N	query each node separately

compact with top_k=2 is genuinely lean, but that's a recall trade-off, and it still sends duplicate versions.

What SWIRL 5 does (verified in source)

SWIRL treats the LLM prompt as a hard budget and fills it deliberately:

Mechanism	Behavior
Hard prompt budget	RAG prompt capped at ~3,000 tokens (`SWIRL_RAG_TOK_DEFAULT`); assembly stops when full
Source cap + relevance gate	≤ 10 sources, only those scoring ≥ 0.8 are eligible
Scored semantic chunks	a BM25 topic-matcher scores spans within each source; on overflow SWIRL narrows to those scored spans, not the whole chunk
Truncation	per-source token-by-token truncation to fit the budget
Markup	relevant spans wrapped in `<SW-IMPORTANT>…</SW-IMPORTANT>` plus a compact per-source metadata header
De-dup before the LLM	version-cluster alternates dropped; if a doc is pinned, only the canonical is kept, so N versions collapse to 1
Single call	one stuff-style synthesis call (worst case +1 JSON repair)

Net: ~3,000 input tokens, one call, zero redundant-version tokens, a predictable ceiling that doesn't grow with document size or corpus size.

Putting numbers on it

Scenario: a query over a corpus where the relevant policy exists in 5 versions, plus 3 other relevant documents, a realistic enterprise shape. Total LLM input tokens per query (summed across calls):

Reading it honestly:

vs. a deliberately minimal stuff config (LlamaIndex compact, top_k=2): SWIRL is comparable per call, but that config buys its low count with poor recall and still ships duplicate versions.
vs. typical stuff RAG (k=5, ~1k chunks): SWIRL is ~40-45% fewer input tokens, and it removes the redundant-version half entirely.
vs. the quality-oriented multi-call modes (refine, tree_summarize, map_reduce): SWIRL is a 2-3× reduction in input tokens, and a larger reduction in output tokens, since those modes generate once per call.

Where the advantage is real, and where it isn't

Real and hard to get off-the-shelf: cross-version de-duplication. None of these frameworks do it by default. In a versioned corpus it's the difference-maker, and it compounds: the more versions, the more SWIRL saves.
Real: one bounded call vs. N-call synthesis; a predictable cost ceiling.
Honest caveat: a hand-tuned vanilla RAG (low k, a good reranker, small chunks, stuff, plus your own dedup layer) can match SWIRL's per-call token count. SWIRL's value is delivering that discipline by default, and adding version de-dup the others lack. It isn't magic per-token compression.

Takeaway

If you're answering questions over a corpus with real document versioning, the tokens you waste aren't in chunk size - they're in sending the LLM the same document five times and in synthesis modes that call the model once per chunk. SWIRL 5's design removes both by default. Measure your own stack the same way: total input and output tokens per query, summed across every LLM call. That number, not top_k, is your bill.

Links + Snippets Not Enough for RAG

Sid Probstein — Mon, 06 Jul 2026 12:00:00 +0000

Several people posting lately about how RAG + search is not enough:

"All the model gets is a list of links and snippets. It's not enough to make sense of most real business questions."

Agreed. If your pipeline stuffs ten search snippets into a prompt and hopes, you get confident mush. A snippet is a pointer to evidence, not the evidence. The fix is a step most RAG setups skip: page fetch plus a reader. Three parts.

1. Fetch the page, not the snippet

The search result is a pointer. So before generating, SWIRL fetches the actual page or document behind each top hit. Now the pipeline is working from the full source, not the 200 characters a search API happened to return.

2. Read it with a reader LLM

This is the part that matters. Some model reads each fetched page against the question and pulls out the passages that actually answer it, marking and scoring them. The chaff never reaches the expensive answering model.

Do extraction before generation, not instead of it.

3. Budget the context

Real pages do not fit neatly into a context window, so the reader either truncates to the highest-signal passages or marks them in place, best first, until the budget is spent. The answering model gets curated, ranked evidence instead of a pile of chunks competing for attention.

The result is the difference between "here are some links" and an answer you can act on. Same retrieval, far better grounding, because something actually read the sources before the model spoke.

That take is exactly why the reader step exists. Naive RAG earns its bad reputation. This is how you avoid it.

(And don't get me started about not having to put it all in a vector database first!)

Making RAG admit when it's guessing: source-grounded hallucination checks

Sid Probstein — Wed, 01 Jul 2026 15:12:48 +0000

The failure mode that scares me most in RAG isn't a wrong answer. It's a confident wrong answer with three citations that don't actually say what the answer claims.

So in SWIRL 5 I stopped trusting the model to police itself and added a check that runs after generation.

The flow:

Generate the answer with its citations, as usual.
Split the answer into atomic claims — roughly one assertion per sentence.
For each claim, pull the specific spans from the retrieved passages the model cited.
Run an entailment check: does the cited text actually support this claim, contradict it, or neither?
Any claim that isn't supported gets flagged in the UI, inline, before the user reads a word of it.

The interesting part wasn't the entailment model; it was everything around it.

Claim segmentation is harder than it sounds. Naive sentence splitting produces claims that are unverifiable on their own because the subject lives two sentences up.

Citations lie by omission. A model will cite a document that's topically relevant but doesn't contain the specific number it just quoted. The whole point of the check is to catch exactly that gap.

Latency budget. An honesty layer nobody waits for is an honesty layer nobody ships. SWIRL 5 batches and optionally caches passage embeddings and more.

The result isn't "SWIRL never hallucinates." Nothing can promise that. The result is: when it's on thin ice, it tells you, and it points at the exact sentence.

That's the version of trustworthy I can actually build.

SWIRL Community 4.5 Update

Sid Probstein — Sun, 28 Jun 2026 16:30:28 +0000

I wrote about SWIRL here last summer. Time for an update.

SWIRL Community 4.5 is out. It is the open-source, Apache-2.0 build of SWIRL: federated search and RAG across your apps, running on your own machine, no vector database required. Three things in this release are worth your attention.

1. Point RAG at any LLM

The big one. Community now lets you configure AI Providers and send your RAG queries to any model you want: OpenAI, Anthropic, Azure OpenAI, or a fully local model behind Ollama or vLLM. Pick the provider, set the model, done.

ai_provider:
  name: anthropic      # openai | azure | ollama | vllm
  model: your-model
  api_key: ${LLM_KEY}

This matters for two reasons. RAG quality is now your choice, not ours, so you can run a stronger model when accuracy counts. And you can keep the whole pipeline local, with retrieval and generation both inside your network. This was an Enterprise feature; it is in Community as of 4.5.

2. Ask RAG a question directly

RAG now accepts a natural-language question as its input. Before, you ran a search and generated over the results; now you can hand SWIRL a plain question and it does retrieval and generation in one step. Question in, cited answer out, which means a lot less glue code to wire SWIRL into a chat box or an app.

3. Updated Galaxy

Community now includes the new Galaxy 5 UI, including a search history widget + dashboard, cleaned up admin interface, built-in activity analytics and more.

That is the release. Free, self-hosted, and now model-agnostic.

The knowledge-authority layer: what your agents can't get from the outside

Sid Probstein — Wed, 17 Jun 2026 13:28:55 +0000

Every enterprise AI conversation right now starts in the same place: "connect the model to our data." Then it stalls in the same place: which data, copied where, governed by whom.

Let me make an argument that runs against the current default - and then show the architecture it implies.

The default is a second copy of your data

The standard RAG recipe is: crawl your sources, chunk them, embed them, and load the vectors into a database. Now your model can retrieve. It also means you have a second copy of your content living in an index you have to secure, keep in sync, and explain to whoever owns compliance. You've recreated every permission boundary by hand, and you'll eventually get one wrong.

For a lot of teams that copy is simply not allowed. Regulated content, client-confidential material, anything privileged - copying it into a vendor store is exposure you don't get paid to take on.

You probably don't need the vector database

Here's the part people don't want to hear. Meta's XetHub team benchmarked three retrieval strategies: keyword-only (BM25), vector-only, and hybrid (keyword to pull candidates, then re-rank). Keyword-only came last. Vector-only did better.

Hybrid won - and their conclusion was blunt: "No vector database necessary."

That matches what we see in production. Vector similarity is a great high-precision filter, not a great first pass. Lead with exact matches and quoted terms, then let embeddings and a cross-encoder re-rank what's left.

What "make your LLM better" actually means

It's not a slogan; it's a pipeline. In SWIRL, relevance is three passes, and both models run locally:

Federate and match. Query every connected source in parallel - keyword + BM25 - and honor quoted phrases and exact terms first.
Embedding re-rank. Re-rank candidates with E5-large-v2, using title-aware chunking and hybrid keyword+vector fusion (RRF). No vector database to build or secure.
Cross-encoder re-rank. An MS-MARCO cross-encoder reads the query and document together and scores real relevance, not vector distance.

Feed that to your LLM - whatever model you've chosen, including an on-prem one - and the answer gets better, because the context got better. Same model, sharper input.

The layer no model supplies from the outside

The stack is settling: foundation models orchestrate, MCP is the retrieval interface, the chat UI is a commodity. The piece none of them provide from outside your walls is knowledge authority - which document is official, which clause your org actually uses, which answer carries approval.

So we made it a first-class layer. SWIRL 5 exposes an MCP server. Any agent - Claude, Copilot, ChatGPT, your own - calls SWIRL and gets ranked, permissioned, organization-approved answers. A team pins the canonical result for a query once; every agent gets it after that. And no copy of your data leaves your tenant.

Why this shape

Three properties fall out of it, and they're the whole reason to build it this way:

Private by architecture. Data stays in place; permissions are enforced live; there's no second index to govern.
The answer, not a guess. Cross-encoder ranking plus canonical answers means people and agents get the result the org trusts.
The safe on-ramp to AI. Headless and MCP-native, deployed in your tenant - the lowest-risk way to give agents enterprise reach.

If you're wiring agents into enterprise data and the "just copy everything into a vector store" step is making your security team twitch, there's another shape available. SWIRL 5 goes GA July 15; the preview is open if you want to point it at your own stack. Either way - I'd genuinely like to hear how you're handling the authority problem, because I don't think the industry has it figured out yet.

Sid Probstein is the creator of SWIRL and CEO of SWIRL AI.

Moving Docker images between repos with crane (after imagetools wasted my afternoon)

Sid Probstein — Thu, 11 Jun 2026 17:55:15 +0000

I had a freshly-built, multi-arch dev image (linux/amd64 + linux/arm64) and one job: promote it into a private partner repo on Docker Hub, plus stamp a dated tag so :latest is always traceable back to a real build. Cross-repo. Should be five minutes.

It was not five minutes.

The problem

The obvious tool is docker buildx imagetools create... it's built for copying manifests between tags. So I reached for it. And it sat there. Then it 400'd. Then I retried, and it hung. Cross-repo blob copies on Docker Hub are reproducibly flaky with imagetools, and I burned the better part of an hour confirming that before I went looking for something else.

The fallback most people reach for next is worse: pull the image down, retag it, push it to the new repo. That round-trips the entire image — every layer, every arch - through your laptop's daemon and disk, only to push the same bits back up. And if you're not careful with how you tag, you flatten a multi-arch index down to whatever single arch your machine happens to be. No thanks.

Why crane

crane (from Google's go-containerregistry) does the copy registry-to-registry. It never pulls the image to your machine.

It preserves the multi-arch manifest. crane cp copies the whole image index by digest. Both arches come along. Nothing gets flattened.
It actually works cross-repo on Docker Hub. Where imagetools 400'd and hung, crane did the server-to-server copy in seconds.
No daemon, no disk. It talks to the registries directly. Your laptop just orchestrates.

flowchart LR
    subgraph slow["pull → tag → push"]
        A[source repo<br/>you/app:dev] --> L[your daemon + disk<br/>whole image] --> B[partner repo<br/>you/partner-app:latest]
    end
    subgraph crane["crane cp"]
        C[source repo<br/>you/app:dev] -->|manifest + blobs<br/>by digest| D[partner repo<br/>you/partner-app:latest]
    end

Run it as a container, no install

docker run --rm gcr.io/go-containerregistry/crane:debug <crane-args>

That's the whole installation story. Nothing on the host.

Auth issues (MacOS)

crane in the container reads ~/.docker/config.json. On macOS with Docker Desktop, your login isn't in that file — it's credsStore: osxkeychain, sitting in the macOS keychain. So if you naively mount ~/.docker/config.json into the container, crane sees no usable credential and hands you UNAUTHORIZED.

The fix: pull the credential out of the keychain on the host, write it into a temporary inline config, mount that, and delete it the moment you're done. Never print it, never commit it.

TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
cat > "$TMP/mkcfg.py" <<'PY'
import json, sys, base64
d = json.load(sys.stdin)
auth = base64.b64encode((d["Username"] + ":" + d["Secret"]).encode()).decode()
json.dump({"auths": {"https://index.docker.io/v1/": {"auth": auth}}}, open(sys.argv[1], "w"))
PY
echo "https://index.docker.io/v1/" | docker-credential-osxkeychain get | python3 "$TMP/mkcfg.py" "$TMP/config.json"

CRANE() { docker run --rm -v "$TMP/config.json":/root/.docker/config.json:ro \
            gcr.io/go-containerregistry/crane:debug "$@"; }

The trap ... EXIT cleans up the temp config when your shell exits, so the credential doesn't linger.

On Linux (or anywhere docker login writes inline creds), skip all of that and mount ~/.docker/config.json directly. And obvious-but-worth-saying: you need a read/write login on the destination. A read-only token can't push.

Promote the image

# copy SRC -> DST, full multi-arch manifest, server to server:
CRANE cp you/app:dev \
         you/partner-app:latest

# also stamp a dated, immutable tag so :latest is always traceable to a build:
CRANE cp you/app:dev \
         you/partner-app:dev-2026-06-11

Two copies, both registry-side, both done before you can refill your coffee.

Verification

Don't trust that the copy worked. Prove it. The destination digest must equal the source digest — same bits, same manifest:

CRANE digest you/app:dev                 # source
CRANE digest you/partner-app:latest      # must equal source
CRANE digest you/partner-app:dev-2026-06-11   # must equal source

If those three lines match, the right image landed under both tags. If they don't, you copied the wrong thing — better to find out here than in a partner's deployment.

Bonus

CRANE ls you/partner-app                          # list tags
CRANE manifest you/app:dev                         # full manifest JSON, see the arches
CRANE tag you/app@sha256:<digest> newtag           # add a tag to an existing digest
CRANE copy <SRC> <DST>                             # alias of cp

One caveat worth knowing: tag deletion isn't a crane operation. Use the Docker Hub UI or the API for that (and it needs an admin-scoped token — a read/write PAT won't delete).

TL;DR

Reach for crane any time you're moving an image between registries or repos and you care about the manifest arriving intact — promotions, mirrors, cross-org handoffs. It skips the daemon, skips the disk, and it doesn't fall over on Docker Hub cross-repo copies the way imagetools does. And whatever you do, build the crane digest source-equals-dest check into the workflow. It costs two seconds.