DEV Community: Sid Probstein

Making RAG admit when it's guessing: source-grounded hallucination checks

Sid Probstein — Wed, 01 Jul 2026 15:12:48 +0000

The failure mode that scares me most in RAG isn't a wrong answer. It's a confident wrong answer with three citations that don't actually say what the answer claims.

So in SWIRL 5 I stopped trusting the model to police itself and added a check that runs after generation.

The flow:

Generate the answer with its citations, as usual.
Split the answer into atomic claims — roughly one assertion per sentence.
For each claim, pull the specific spans from the retrieved passages the model cited.
Run an entailment check: does the cited text actually support this claim, contradict it, or neither?
Any claim that isn't supported gets flagged in the UI, inline, before the user reads a word of it.

The interesting part wasn't the entailment model; it was everything around it.

Claim segmentation is harder than it sounds. Naive sentence splitting produces claims that are unverifiable on their own because the subject lives two sentences up.

Citations lie by omission. A model will cite a document that's topically relevant but doesn't contain the specific number it just quoted. The whole point of the check is to catch exactly that gap.

Latency budget. An honesty layer nobody waits for is an honesty layer nobody ships. SWIRL 5 batches and optionally caches passage embeddings and more.

The result isn't "SWIRL never hallucinates." Nothing can promise that. The result is: when it's on thin ice, it tells you, and it points at the exact sentence.

That's the version of trustworthy I can actually build.

SWIRL Community 4.5 Update

Sid Probstein — Sun, 28 Jun 2026 16:30:28 +0000

I wrote about SWIRL here last summer. Time for an update.

SWIRL Community 4.5 is out. It is the open-source, Apache-2.0 build of SWIRL: federated search and RAG across your apps, running on your own machine, no vector database required. Three things in this release are worth your attention.

1. Point RAG at any LLM

The big one. Community now lets you configure AI Providers and send your RAG queries to any model you want: OpenAI, Anthropic, Azure OpenAI, or a fully local model behind Ollama or vLLM. Pick the provider, set the model, done.

ai_provider:
  name: anthropic      # openai | azure | ollama | vllm
  model: your-model
  api_key: ${LLM_KEY}

This matters for two reasons. RAG quality is now your choice, not ours, so you can run a stronger model when accuracy counts. And you can keep the whole pipeline local, with retrieval and generation both inside your network. This was an Enterprise feature; it is in Community as of 4.5.

2. Ask RAG a question directly

RAG now accepts a natural-language question as its input. Before, you ran a search and generated over the results; now you can hand SWIRL a plain question and it does retrieval and generation in one step. Question in, cited answer out, which means a lot less glue code to wire SWIRL into a chat box or an app.

3. Updated Galaxy

Community now includes the new Galaxy 5 UI, including a search history widget + dashboard, cleaned up admin interface, built-in activity analytics and more.

That is the release. Free, self-hosted, and now model-agnostic.

The knowledge-authority layer: what your agents can't get from the outside

Sid Probstein — Wed, 17 Jun 2026 13:28:55 +0000

Every enterprise AI conversation right now starts in the same place: "connect the model to our data." Then it stalls in the same place: which data, copied where, governed by whom.

Let me make an argument that runs against the current default - and then show the architecture it implies.

The default is a second copy of your data

The standard RAG recipe is: crawl your sources, chunk them, embed them, and load the vectors into a database. Now your model can retrieve. It also means you have a second copy of your content living in an index you have to secure, keep in sync, and explain to whoever owns compliance. You've recreated every permission boundary by hand, and you'll eventually get one wrong.

For a lot of teams that copy is simply not allowed. Regulated content, client-confidential material, anything privileged - copying it into a vendor store is exposure you don't get paid to take on.

You probably don't need the vector database

Here's the part people don't want to hear. Meta's XetHub team benchmarked three retrieval strategies: keyword-only (BM25), vector-only, and hybrid (keyword to pull candidates, then re-rank). Keyword-only came last. Vector-only did better.

Hybrid won - and their conclusion was blunt: "No vector database necessary."

That matches what we see in production. Vector similarity is a great high-precision filter, not a great first pass. Lead with exact matches and quoted terms, then let embeddings and a cross-encoder re-rank what's left.

What "make your LLM better" actually means

It's not a slogan; it's a pipeline. In SWIRL, relevance is three passes, and both models run locally:

Federate and match. Query every connected source in parallel - keyword + BM25 - and honor quoted phrases and exact terms first.
Embedding re-rank. Re-rank candidates with E5-large-v2, using title-aware chunking and hybrid keyword+vector fusion (RRF). No vector database to build or secure.
Cross-encoder re-rank. An MS-MARCO cross-encoder reads the query and document together and scores real relevance, not vector distance.

Feed that to your LLM - whatever model you've chosen, including an on-prem one - and the answer gets better, because the context got better. Same model, sharper input.

The layer no model supplies from the outside

The stack is settling: foundation models orchestrate, MCP is the retrieval interface, the chat UI is a commodity. The piece none of them provide from outside your walls is knowledge authority - which document is official, which clause your org actually uses, which answer carries approval.

So we made it a first-class layer. SWIRL 5 exposes an MCP server. Any agent - Claude, Copilot, ChatGPT, your own - calls SWIRL and gets ranked, permissioned, organization-approved answers. A team pins the canonical result for a query once; every agent gets it after that. And no copy of your data leaves your tenant.

Why this shape

Three properties fall out of it, and they're the whole reason to build it this way:

Private by architecture. Data stays in place; permissions are enforced live; there's no second index to govern.
The answer, not a guess. Cross-encoder ranking plus canonical answers means people and agents get the result the org trusts.
The safe on-ramp to AI. Headless and MCP-native, deployed in your tenant - the lowest-risk way to give agents enterprise reach.

If you're wiring agents into enterprise data and the "just copy everything into a vector store" step is making your security team twitch, there's another shape available. SWIRL 5 goes GA July 15; the preview is open if you want to point it at your own stack. Either way - I'd genuinely like to hear how you're handling the authority problem, because I don't think the industry has it figured out yet.

Sid Probstein is the creator of SWIRL and CEO of SWIRL AI.

Moving Docker images between repos with crane (after imagetools wasted my afternoon)

Sid Probstein — Thu, 11 Jun 2026 17:55:15 +0000

I had a freshly-built, multi-arch dev image (linux/amd64 + linux/arm64) and one job: promote it into a private partner repo on Docker Hub, plus stamp a dated tag so :latest is always traceable back to a real build. Cross-repo. Should be five minutes.

It was not five minutes.

The problem

The obvious tool is docker buildx imagetools create... it's built for copying manifests between tags. So I reached for it. And it sat there. Then it 400'd. Then I retried, and it hung. Cross-repo blob copies on Docker Hub are reproducibly flaky with imagetools, and I burned the better part of an hour confirming that before I went looking for something else.

The fallback most people reach for next is worse: pull the image down, retag it, push it to the new repo. That round-trips the entire image — every layer, every arch - through your laptop's daemon and disk, only to push the same bits back up. And if you're not careful with how you tag, you flatten a multi-arch index down to whatever single arch your machine happens to be. No thanks.

Why crane

crane (from Google's go-containerregistry) does the copy registry-to-registry. It never pulls the image to your machine.

It preserves the multi-arch manifest. crane cp copies the whole image index by digest. Both arches come along. Nothing gets flattened.
It actually works cross-repo on Docker Hub. Where imagetools 400'd and hung, crane did the server-to-server copy in seconds.
No daemon, no disk. It talks to the registries directly. Your laptop just orchestrates.

flowchart LR
    subgraph slow["pull → tag → push"]
        A[source repo<br/>you/app:dev] --> L[your daemon + disk<br/>whole image] --> B[partner repo<br/>you/partner-app:latest]
    end
    subgraph crane["crane cp"]
        C[source repo<br/>you/app:dev] -->|manifest + blobs<br/>by digest| D[partner repo<br/>you/partner-app:latest]
    end

Run it as a container, no install

docker run --rm gcr.io/go-containerregistry/crane:debug <crane-args>

That's the whole installation story. Nothing on the host.

Auth issues (MacOS)

crane in the container reads ~/.docker/config.json. On macOS with Docker Desktop, your login isn't in that file — it's credsStore: osxkeychain, sitting in the macOS keychain. So if you naively mount ~/.docker/config.json into the container, crane sees no usable credential and hands you UNAUTHORIZED.

The fix: pull the credential out of the keychain on the host, write it into a temporary inline config, mount that, and delete it the moment you're done. Never print it, never commit it.

TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
cat > "$TMP/mkcfg.py" <<'PY'
import json, sys, base64
d = json.load(sys.stdin)
auth = base64.b64encode((d["Username"] + ":" + d["Secret"]).encode()).decode()
json.dump({"auths": {"https://index.docker.io/v1/": {"auth": auth}}}, open(sys.argv[1], "w"))
PY
echo "https://index.docker.io/v1/" | docker-credential-osxkeychain get | python3 "$TMP/mkcfg.py" "$TMP/config.json"

CRANE() { docker run --rm -v "$TMP/config.json":/root/.docker/config.json:ro \
            gcr.io/go-containerregistry/crane:debug "$@"; }

The trap ... EXIT cleans up the temp config when your shell exits, so the credential doesn't linger.

On Linux (or anywhere docker login writes inline creds), skip all of that and mount ~/.docker/config.json directly. And obvious-but-worth-saying: you need a read/write login on the destination. A read-only token can't push.

Promote the image

# copy SRC -> DST, full multi-arch manifest, server to server:
CRANE cp you/app:dev \
         you/partner-app:latest

# also stamp a dated, immutable tag so :latest is always traceable to a build:
CRANE cp you/app:dev \
         you/partner-app:dev-2026-06-11

Two copies, both registry-side, both done before you can refill your coffee.

Verification

Don't trust that the copy worked. Prove it. The destination digest must equal the source digest — same bits, same manifest:

CRANE digest you/app:dev                 # source
CRANE digest you/partner-app:latest      # must equal source
CRANE digest you/partner-app:dev-2026-06-11   # must equal source

If those three lines match, the right image landed under both tags. If they don't, you copied the wrong thing — better to find out here than in a partner's deployment.

Bonus

CRANE ls you/partner-app                          # list tags
CRANE manifest you/app:dev                         # full manifest JSON, see the arches
CRANE tag you/app@sha256:<digest> newtag           # add a tag to an existing digest
CRANE copy <SRC> <DST>                             # alias of cp

One caveat worth knowing: tag deletion isn't a crane operation. Use the Docker Hub UI or the API for that (and it needs an admin-scoped token — a read/write PAT won't delete).

TL;DR

Reach for crane any time you're moving an image between registries or repos and you care about the manifest arriving intact — promotions, mirrors, cross-org handoffs. It skips the daemon, skips the disk, and it doesn't fall over on Docker Hub cross-repo copies the way imagetools does. And whatever you do, build the crane digest source-equals-dest check into the workflow. It costs two seconds.

Running PyTorch fork-safe in Celery on macOS

Sid Probstein — Mon, 01 Jun 2026 13:23:46 +0000

If you've ever seen this in your Celery logs:

Process 'ForkPoolWorker-7' pid:32839 exited with 'signal 11 (SIGSEGV)'
billiard.exceptions.WorkerLostError:
    Worker exited prematurely: signal 11 (SIGSEGV) Job: 0.

...and the macOS crash report buries the real message in a JSON blob:

"asi": {
  "CoreFoundation": ["*** multi-threaded process forked ***"],
  "libsystem_c.dylib": ["crashed on child side of fork pre-exec"]
}

...you've hit one of the classic fork-after-init traps. Here's what's going on and how to actually fix it.

The one-line fix (if you're in a hurry)

Set these env vars before the Celery worker's MainProcess imports anything heavy:

# <your-project>/celery.py ... the very first thing the worker imports
import os

for var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS",
            "NUMEXPR_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ.setdefault(var, "1")

os.environ.setdefault("OBJC_DISABLE_INITIALIZE_FORK_SAFETY", "YES")

The first set forces every BLAS library to single-threaded mode. VECLIB_MAXIMUM_THREADS is the one most people forget; it covers Apple's Accelerate framework, which is what PyTorch uses by default on Apple Silicon. The last one tells the Objective-C runtime to skip its fork-safety abort.

Why this happens

PyTorch's nn.Linear on macOS arm64 calls into Apple Accelerate, which does its parallel matmuls via libdispatch (Grand Central Dispatch).

The first BLAS call lazily spins up a pool of libdispatch worker queues in the calling process.

If that "calling process" is your Celery worker's MainProcess (say, because something during boot does a tiny matmul: spaCy preload, an embedding warmup, anything that imports numpy and runs a real op), those queues now live in the parent.

When the prefork pool then fork()s a child, the child inherits broken queue handles. The next BLAS call from inside the child dereferences a stale pointer and you get the SIGSEGV.

The stack trace in the crash report makes it unambiguous:

0: _dispatch_apply_with_attr_f      (libdispatch)
1: dispatch_apply_with_attr         (libdispatch)
3: cblas_sgemm                      (Accelerate)
5: at::native::cpublas::gemm        (libtorch_cpu)
6: at::native::addmm_impl_cpu_      (libtorch_cpu)
7: at::native::linear               (libtorch_cpu)
8: torch::autograd::THPVariable_linear

What doesn't work

"Just lazy-load the model in the child." Even if you defer from_pretrained until you're inside a forked child, that first call still hits Accelerate BLAS, and the dispatch queues your child inherited from the parent are already broken.

"Just bypass sentence_transformers.CrossEncoder.predict() and use bare-torch." Same story. Whether you go through CrossEncoder or call AutoModelForSequenceClassification directly, the SIGSEGV is one frame down inside linear().

"Just don't import torch at the top of the module." Necessary but not sufficient. In our case, removing import torch from ai_provider.py was real progress, but then we discovered litellm transitively pulls torch the first time you call it. Every "warmup" preload that touched litellm still poisoned the parent. You have to audit every code path that runs before the first fork.

The defensive pattern that does work

Defer heavy imports. Don't import torch at module top in anything that's part of the Celery autodiscovery chain. Push it into the function that needs it:

# Bad ... taints anyone who imports this module
import torch

def rerank(query, documents):
    with torch.no_grad():
        ...

# Good ... torch only loads in workers that actually rerank
def rerank(query, documents):
    import torch
    with torch.no_grad():
        ...

Gate "warmup" preloads off the Celery worker. Preloading models at startup makes sense for an ASGI server like Daphne. It is actively harmful in a forking Celery worker, because the warmup runs in MainProcess:

class MyAppConfig(AppConfig):
    def ready(self):
        is_celery_worker = "celery" in sys.argv and "worker" in sys.argv
        if not is_celery_worker:
            self._preload_cross_encoder()

What about Linux / Docker?

Yes, this affects Linux too, just less dramatically.

OpenBLAS and MKL both spin up thread pools on first use that don't survive fork; the typical Linux failure mode is a hang or a deadlock rather than a SIGSEGV.

The good news: the same *_NUM_THREADS=1 env vars are the fix.

VECLIB_MAXIMUM_THREADS and OBJC_DISABLE_INITIALIZE_FORK_SAFETY are no-ops on Linux, so the snippet above is portable. The deferred-import and gate-off-warmup patterns apply unchanged.

The AI agent stack that’s quietly taking over enterprise workflows

Sid Probstein — Sat, 03 May 2025 02:58:27 +0000

Accenture, IBM, and AWS are all placing bets on Crew AI. Why? Because it makes building and deploying real AI agents possible.

With Crew AI, teams are spinning up agents that:

Launch predictive marketing campaigns
Automate financial back-office ops
Optimize inventory and logistics
And tackle 100+ other enterprise use cases

But here’s the catch: agents are only as good as the data they can reach. That’s where SWIRL comes in.

By pairing Crew AI with SWIRL, you get more than just agents—you get enterprise-ready, data-rich workflows that scale. No custom plumbing. No brittle integrations.

With Crew AI + SWIRL, your agents can:

Connect to 100+ enterprise data sources out-of-the-box
Fetch the most relevant structured/unstructured data across silos
Respect row-level permissions with real enterprise auth
Summarize and answer with your LLM of choice
Plug in easily via zero-code connectors

Want to see this in action?

Message me for a demo or check the open source edition here: https://github.com/swirlai/swirl-search

It was already indexed!

Sid Probstein — Sun, 13 Aug 2023 16:48:52 +0000

I recently had the pleasure of chatting with @dmitrykan on his Vector Podcast. Check it out: https://dmitry-kan.medium.com/vector-podcast-with-sid-probstein-search-in-siloed-data-with-swirl-f2b9595a2715

We talked about quite a few things, including:

The challenges of enterprise search in the post-cloud era
How cross-silo search is particularlyt tricky because of entitlements (aka permissions) across silos
Zero-code configuration of connectors in Swirl, where JSON path and developer API doc get the job done
How large language models contextually re-rank disparate search results

There was an interesting twist at the end of the call. Dmitry uses a service called Clearword to transcribe recordings. Dmitry asked: “how quickly can you index the transcript and search it with Swirl?”

Here is my answer: It was already indexed!

Since there is no audio and it goes by quickly, let me explain ... Clearword emailed the transcript to both of us shortly after recording ended. It was indexed by Microsoft Outlook within seconds of arriving in my inbox.

To verify this, I copied some text from the middle of the transcript and pasted it into Swirl, which returned the link to the email message with the transcript and the phrase I searched for.

That simple truth - that the average enterprise is awash in search forms - is the entire reason metasearch is such a game-changing approach. Instead of making yet another repository, Swirl sends queries to existing search APIs and re-ranks the results from everything. It saves users a huge amount of time without a major IT project.

Want to see for yourself? git it going with 2 commands via Docker here: https://github.com/swirlai/swirl-search

Swirl 2.5 released

Sid Probstein — Wed, 09 Aug 2023 14:37:03 +0000

I am delighted to announce availability of Swirl 2.5!

This version focused on performance. Configured with 12 SearchProviders, Swirl 2.5 supports ~15 queries/second on a Standard F16s v2 server (16 vcpus, 32 GiB memory) with a median response time of ~3 seconds.

Version 2.5 also includes SearchProviders for HubSpot contact, company, and deal records, plus improvements to the Galaxy search UI (shown above).

Check out the Release notes for full details: https://github.com/swirlai/swirl-search/releases/tag/v2.5.0

What is Swirl? A new open source metasearch engine; it queries anything with an API then uses spaCy to re-rank the unified results without copying any data! Includes zero-code configs for Apache Solr, ChatGPT, Elastic Search, OpenSearch, PostgreSQL, Google BigQuery, RequestsGet, Google PSE, NLResearch.com, Miro, Microsoft 365, HubSpot, Atlassian, YouTrack, GitHub & more!

I wrote a metasearch engine called Swirl

Sid Probstein — Thu, 03 Aug 2023 18:48:40 +0000

Hi all! Swirl sends queries to existing search engines, unifies the results and re-ranks them all using large language models. It solves cross-silo information access and search problems in a fraction of the time and effort required to copy, ingest and index data.

Here's a brief video intro: https://youtu.be/sfsBYyu6qDQ

Swirl was written in python atop the django/celery/redis stack with a choice of Sqlite3 or PostgreSQL back-ends. The source code is available under the Apache 2.0 license. The distribution includes zero-code configs for Apache Solr, ChatGPT, Elastic Search, OpenSearch, PostgreSQL, Google BigQuery, RequestsGet, Google PSE, NLResearch.com, Miro, Microsoft 365, Atlassian, YouTrack, GitHub, HubSpot & more. Plug-in your access tokens or use Microsoft 365 to login and users can stop searching and start Swirling!

Links:

We are seeking contributions and feedback from developers working on all kinds of search solutions... thanks!