DEV Community: gibs-dev

We built an open Neo4j expert dataset — here's what we learned

gibs-dev — Mon, 02 Mar 2026 23:34:04 +0000

We're building GibsGraph, an open-source tool that lets you query any Neo4j graph in plain English — or build new graphs from unstructured text. To generate good Cypher, the agent needs real Neo4j expertise. Not LLM training data. Actual documentation, patterns, and best practices.

So we built a curated expert dataset from scratch. 920 records. 5 categories. Fully bundled as JSONL — no setup needed.

Here's what we learned along the way.

What's in the dataset

We parsed the official Neo4j documentation — the Cypher manual, modeling guides, knowledge base articles — into structured records:

Type	Count	Source
Cypher examples	446	Official docs, parsed from AsciiDoc
Best practices	318	Knowledge base articles, modeling guides
Cypher functions	133	Cypher manual function reference
Cypher clauses	36	Cypher manual clause reference
Modeling patterns	23	Data modeling docs + curated additions

Each record has a source_file tracing back to the original documentation, an authority_level (1 = official docs, 2 = curated), and for some records a quality_tier that controls whether they get loaded at runtime.

After quality filtering, 849 records make it into the live system.

The automated audit

Trusting your own data is dangerous. So we built a 4-tier audit script that verifies everything it can mechanically:

Tier 1 — Completeness: Every record has its required fields, valid enums, real source paths.

Tier 2 — Cypher validation: All 1,131 Cypher snippets checked for balanced syntax, string interpolation, and pseudocode detection.

Tier 3 — Cross-reference: Every function and clause name checked against the official Neo4j 5.x reference. We hardcoded 126 built-in functions, plus known APOC and GDS entries.

Tier 4 — Duplicates: Within-file duplicate detection, naming convention checks (PascalCase node labels, UPPER_SNAKE relationship types).

What the audit caught

First run results:

Records: 956 | Cypher snippets: 1,131

Verified:     3,604 checks passed
Flagged:      33 issues
Human review: 341 records

7 real failures:

"Introduction" — a parser artifact that got classified as a Cypher function. Not a function. It's the intro paragraph of the functions docs page.
allReduce — flagged as unrecognized. Turns out this is a valid Neo4j 5.x predicate function, but our reference list was missing it. The audit caught a gap in our own reference data.
Unbalanced Cypher in LET clause — the LET clause examples had syntax issues. LET is a newer GQL-conformance addition and the docs examples are still maturing.
A best practice with an unclosed bracket — "Query to kill transactions" had unbalanced Cypher in its example.
An APOC example with unbalanced strings — apoc.load.jsonParams example had mismatched quotes.

Every one of these is a real data quality issue that would have degraded the agent's Cypher generation.

What we can't auto-verify

The 23 modeling patterns and 318 best practices contain domain advice: "When to use this pattern", "What the anti-pattern is", "Why this practice matters." No script can tell you if a modeling recommendation is sound. That takes a human who writes Cypher professionally.
a
This is the honest gap. The mechanical data is 98.8% clean. The advice data needs expert eyes.

The dataset is open

Everything is MIT-licensed and on GitHub:

Raw data: src/gibsgraph/data/*.jsonl
Audit script: data/ascripts/audit_expert_data.py
Review CSVs: data/review_modeling_patterns.csv and data/review_best_practices.csv

If you work with Neo4j and want to help verify the modeling patterns and best practices, we've set up review CSVs with empty review_status and reviewer_notes columns. Even reviewing 10 records helps.

Join the discussion on GitHub

We'd rather ship 200 verified records than 900 unverified ones.

Written with AI assistance, reviewed and published by https://gibs.dev.

Thank you Claude (L)

I built a RAG system where hallucinations aren't acceptable. Here's what actually worked.

gibs-dev — Wed, 18 Feb 2026 16:46:04 +0000

I built a regulatory compliance API for EU law in 5 weeks of evenings. Here's what I learned about RAG, citations, and why nobody panics until it's too late.

I work in construction during the day. At night, I build software.

Five weeks ago, I started building Gibs — an API that answers questions about EU regulations (DORA, AI Act, GDPR) and returns the answer with a direct citation to the specific EUR-Lex article. Grounded answers with source citations. No "this is not legal advice" disclaimer walls. Just: question in, cited answer out.

This post is about the technical decisions, the things that broke, and what I learned about building a RAG system where correctness actually matters.

The problem

The EU AI Act entered into force in August 2024, but the high-risk AI system rules (Annex III) don't apply until August 2026. DORA (Digital Operational Resilience Act) is already live since January 2025. GDPR has been around since 2018 but people still can't get straight answers about Article 22.

If you're a fintech building an AI feature, or a municipality deploying a chatbot, you need to know: What are my obligations? Which articles apply? What's the deadline?

Today, the answer is: hire a lawyer, or spend three days reading EUR-Lex. I wanted to build a third option.

Architecture: the boring version

User question
  → Classifier (which regulation?)
  → Qdrant vector search (relevant chunks)
  → Reranker (sort by relevance)
  → LLM synthesis (answer + citations)
  → Response with EUR-Lex article references

Nothing revolutionary. Standard RAG pipeline. The interesting parts are in the details.

Lesson 1: Chunking legal text is harder than you think

EU regulations aren't blog posts. A single DORA article can reference three other articles, two delegated acts, and an implementing technical standard. If you chunk naively by paragraph, you lose the cross-references that make the answer correct.

I ended up building a custom parser that:

Preserves article boundaries (never splits mid-article)
Attaches metadata: regulation name, article number, section, EUR-Lex URL
Handles delegated acts as separate documents with parent references
Builds a cross-reference graph — over 3,600 edges mapping which articles reference which other articles, across regulations

The graph matters because legal text is relational. DORA Article 19 references Article 20, which references a delegated act, which supplements the base regulation. If your retrieval only returns Article 19, the answer is incomplete. The graph lets the pipeline follow those edges and pull in related chunks automatically.

The DORA corpus alone is 641 chunks across 12 delegated acts. Getting the metadata right took longer than building the retrieval pipeline.

Lesson 2: Classification routing matters more than embedding quality

My first eval scored 79.9% on DORA questions. Terrible.

The problem wasn't the embeddings or the LLM. It was routing. A question like "What are the RTS requirements for threat-led penetration testing?" doesn't contain the word "DORA" — so my classifier sent it to the AI Act collection and retrieved completely wrong chunks.

The fix was expanding the classifier's keyword patterns:

# Before: only matched "DORA" literally
# After:
DORA_PATTERNS = [
    r'DORA', r'digital\s+operational\s+resilience',
    r'ICT\s+(risk|incident)', r'TLPT',
    r'financial\s+entit', r'RTS\s+on\s+(ict|incident|tlpt)',
    r'delegated\s+act'
]

That single change took accuracy from 79.9% to 90.8%. The lesson: in domain-specific RAG, routing is your biggest lever. Spend your time there, not on prompt engineering.

Lesson 3: Eval is everything

I built a golden dataset of 140 questions across DORA, AI Act, and GDPR. Each question has:

Expected answer keywords (answer_must_contain)
Expected source articles (sources_must_contain)
Whether the system should abstain (question outside scope)

The eval runner scores every response automatically and breaks down accuracy by category: objective, cross_reference, delegated_act, adversarial, real_user, negative, and should_abstain.

This caught bugs I never would have found manually. For example: my scorer was case-sensitive, so "Article 6" matched but "article 6" didn't. Tiny bug, 3% accuracy hit.

python eval_runner.py --regulation dora --delay 7000 --retry

If you're building any RAG system, build eval first. I'm serious. Before you optimize anything, know how to measure it.

Lesson 4: Citations are the product

I could have built a chatbot that answers compliance questions. There are dozens of those. What makes Gibs different is that every claim in the response maps to a specific article, and you can click through to EUR-Lex and verify it.

This is non-negotiable for compliance use cases. A lawyer will never trust "According to EU regulations, you need to do X." They will trust "According to Article 6(1) of Regulation (EU) 2024/1689, deployers of high-risk AI systems shall..." with a link.

The synthesis prompt forces the model to cite specific articles and cross-reference the source chunks. If a claim can't be grounded in the retrieved text, the system says so instead of hallucinating.

Lesson 5: Abstention is a feature

90% of compliance questions are in-scope. The other 10% are questions like "What's the best CRM for startups?" — and if your system confidently answers those with made-up regulatory citations, you've lost all trust.

Gibs has an abstention threshold. If the retrieved chunks aren't relevant enough, the system says "This question is outside the scope of the indexed regulations." My abstention accuracy is 96.7% — meaning it almost never hallucinates an answer for out-of-scope questions.

The stack

Vector DB: Qdrant (one collection per regulation — strict isolation)
Embeddings + Reranker: Cohere embed-v3 + rerank-v3.5
Synthesis: LLM via API (multi-pass: decompose → retrieve → synthesize → verify)
Framework: Python + FastAPI + LangGraph
Hosting: Self-hosted Docker on a mini PC with 8GB RAM
Auth: API key + Stripe for billing
SDKs: Python and TypeScript published on PyPI/npm
MCP Server: For AI assistant integration (Cursor, Windsurf, etc.)

Total monthly infrastructure cost: roughly what you'd spend on a nice dinner.

What I'd do differently

Start with eval, not with the product. I built the pipeline first and eval second. Should have been the other way around.
Chunk smaller, retrieve more. My initial chunks were too large. Smaller chunks with more retrieval and better reranking would have been better from the start.
Don't underestimate metadata. The EUR-Lex URL, the article number, the regulation name — that metadata is what makes the citations work. It took 40% of my time and it was worth every hour.

Current status

DORA: 90.8% accuracy, all 12 delegated acts indexed
AI Act: 88% accuracy, all articles + annexes + recitals
python and npm built
mcp built and listed
GDPR: Live, eval in progress
API: Live at api.gibs.dev
Free tier: 50 requests/month, no credit card

I'm looking for beta users — especially compliance consultants, fintechs dealing with DORA, or anyone building AI systems that need EU AI Act classification.

If you want to try it: gibs.dev

If you want to see the API docs: docs.gibs.dev

If you want to talk about RAG for legal text, citations, or building dev tools while working a day job in construction — I'm in the comments.

Gibs is a developer tool for regulatory research — not a substitute for qualified legal advice.

Built by one person, evenings and weekends, with a lot of coffee and a somewhat irresponsible sleep schedule.