I Wanted a RAG System That Could Show Its Work, So I Built Alchimista

Most RAG demos stop at the answer.Alchimista

You upload a document, ask a question, get a plausible response, maybe see a few source chunks, and that's where the story ends.

But I kept coming back to a different problem:

What happens when the answer itself isn't enough?

What if I need to know which evidence was used, what happened to sensitive data before processing, which policy was active, whether two tenants could ever see each other's documents, and what the system actually did rather than what its architecture diagram claims it should do?

That's where Alchimista started.

The idea

Alchimista is a self-hosted document intelligence and AI governance system built around a fairly simple principle:

An AI answer should leave evidence behind.

The basic path looks like this:

document → privacy processing → ingestion → retrieval → cited answer → decision evidence → audit

That sounds straightforward. Making all of those boundaries agree with each other was considerably less straightforward.

RAG wasn't the difficult part

Retrieving chunks and generating an answer is only one piece of the system.

The more interesting questions appeared around it.

A document belongs to a tenant. Retrieval has to respect that boundary.

An answer needs citations. Those citations need to point back to actual indexed evidence.

Sensitive information may need to be detected or pseudonymized before it reaches an external model.

If privacy processing fails, silently continuing with the original text is not an acceptable fallback.

And if the system records an AI decision, the audit trail shouldn't accidentally become another place where sensitive data leaks.

So Alchimista gradually became less of a "RAG application" and more of an infrastructure layer around the complete evidence path.

Privacy as part of the pipeline

Alchimista currently has several privacy modes.

At the strictest level, sensitive values are pseudonymized before chunking and embedding, meaning the normal retrieval index never receives their cleartext representation.

Reversible mappings live separately in an AES-256-GCM encrypted vault.

More importantly, enabled privacy policies fail closed. If the privacy service is unavailable or produces an invalid result, processing stops instead of quietly bypassing the protection.

That behavior matters more to me than adding another checkbox labelled "privacy".

Local first

An earlier version of the project leaned heavily on cloud infrastructure.

That worked, but eventually I wanted the core system to be independently runnable and inspectable.

The current baseline needs:

• Docker with Compose
• Python 3.11+
• PostgreSQL

No GCP project, external vector database, Auth0 tenant, object store, or paid service is required.

The cloud integrations still exist as optional adapters, but they are no longer prerequisites for proving that the system works.

Starting it locally is intentionally boring:

git clone https://github.com/Daniele-Cangi/Alchimista.git
cd Alchimista
python scripts/init_local_env.py
docker compose up --detach --build --wait

Then the application is available locally with Documents, Ask, Privacy, Audit, Governance, and System in the same interface.

I wanted a proof, not just a demo

One thing I've become increasingly skeptical of in my own projects is a green UI that proves very little about the system underneath it.

So Alchimista has an end-to-end self-hosted smoke test.

It doesn't just check that the containers started.

It exercises ingestion, privacy detection, pseudonymization and restoration, protected persistence, SQL retrieval, citations, tenant isolation, AI decision evidence, audit metadata, the encrypted vault, cleanup behavior, and persistence across a PostgreSQL restart.

The successful end state is deliberately unexciting:

SELF_HOSTED_SMOKE_OK

But that line means much more to me than a screenshot of a chatbot answering a document question.

Some deliberate limitations

There are also things I don't want Alchimista to pretend.

It implements technical controls; it does not magically certify GDPR, the EU AI Act, or any other regulation.

PII detection cannot promise to identify every possible sensitive value.

The default local embedding path is designed to make the complete system independently runnable and test retrieval mechanics. It isn't a claim that a deterministic offline embedder is the best semantic retrieval system available.

And stronger pseudonymization can affect retrieval quality, which means strict privacy modes have to be evaluated against the actual domain rather than assumed to be free.

I think being explicit about those boundaries makes the system more useful, not less.

What's next

Alchimista is now at the point where I'm less interested in adding features for the sake of feature count and more interested in attacking the boundaries.

What breaks under larger document collections?

How much retrieval quality is lost under different privacy strategies?

Which pieces of AI decision evidence are genuinely useful during an investigation months later?

Where should the trust boundary actually sit when an external model is introduced?

Those are the experiments I'm interested in next.

This is also my first proper post on DEV, so I'll probably use this space less as a release feed and more as a place to document those experiments — including the ones that don't work.

Alchimista is open source and available on GitHub.

Feedback on the architecture, privacy model, or places you'd try to break it is very welcome.