DEV Community: Himadri

We nearly published a 2x regression that didn't exist

Himadri — Mon, 24 Aug 2026 07:02:47 +0000

We were about to ship a performance write-up claiming three queries had gotten 2-4x slower. All three numbers were wrong, and the reason is one every benchmark author should worry about: the same binary, on the same machine, running the same query, gave different answers depending on what time of day we ran it.

The measurement that didn't add up

While checking a change to Samyama Graph, we timed LDBC SNB Interactive query IC9 on one commit, one dataset, one 16-core workstation, four hours apart:

when	commit	IC9 median
morning	`b2a3557`	2,822 ms
evening	`b2a3557`	4,912 ms

Same binary. 1.74x slower, four hours later. ps showed the benchmark pinned at 100% of one core and nothing else on the box above 9%. No other process explains that gap.

So we ran it again — same binary, minutes apart this time, nothing changed in between:

run	IC9 median
1	2,011 ms
2	2,490 ms

24% apart. Back to back.

Why we almost missed it

We had a real change on main to compare against a baseline on an older commit. Baseline was measured in the morning; the new code was measured that evening. Read naively, the evening numbers said IC10 had gotten 2x slower, IC11 4x, IC12 3x.

None of those regressions existed. The host itself had drifted between the two sittings — thermal throttling, frequency scaling, background load, whatever the cause, the machine at 6pm was not the machine at 9am. We re-ran everything back to back, on one sitting, and the picture reversed:

query	before (`b2a3557`)	after (`main`)	ratio
IC9	4,912 ms	2,490 ms	1.97x faster
IC10	340 ms	352 ms	~1.0x
IC11	382 ms	400 ms	~1.0x
IC12	428 ms	483 ms	~1.1x — inside the 24% band

The change was a genuine ~2x win on IC9 and a wash everywhere else. The "2-4x regressions" were the host, not the code.

We only caught it because a 4x regression from a small change looked implausible enough to re-measure. A smaller, real regression sitting inside that same noise band would not have raised anyone's eyebrows, and would have shipped.

What this means for anyone reading benchmark numbers

Two things, and they cut in opposite directions:

A regression can hide inside the noise. If your benchmark's run-to-run variance is 24%, any real regression under ~25% is statistically invisible next to it. A nightly CI gate set to fire on a 25% threshold would page on noise some nights and miss a real 20% regression every other night.

An improvement can be manufactured by timing alone. Measure a baseline in the morning and a new version in the evening on a drifting host, and you can produce whatever ratio you want without touching a line of code. This is true whether the drift is accidental or not — which is exactly why "before" and "after" have to come from one sitting.

Neither of these is specific to our engine, our benchmark suite, or Rust. It's a property of measuring wall-clock time on a shared physical machine, and it will bite any project that compares two numbers taken hours apart and calls the difference a result.

What we changed

A calibration line in every run. Each benchmark now times a short, fixed, CPU-bound operation at the start and the end, and reports it alongside load average and mean core frequency. Two runs whose calibration numbers disagree were taken on hosts running at different speeds, whatever the milliseconds of the actual queries say. If a single run's own opening and closing calibration differ by more than 10%, the run says so — its numbers aren't even internally comparable to each other.

A rule for before/after claims. A performance comparison needs both numbers from one back-to-back session on one machine — not "the same machine," the same sitting. Quote the ratio, not the absolute milliseconds, when the point is an improvement: a ratio measured back to back survives a slow host; an absolute number doesn't survive being read next to one taken elsewhere.

Cross-engine ratios get the same treatment. If a "we're 3x faster than X" claim has our number from one session and the competitor's from another, the ratio isn't a ratio — it's two unrelated numbers divided by each other. Same machine, same sitting, or the comparison doesn't get published.

The full account, including the four numbers that started this, is in the original issue: samyama-ai/samyama-graph#529. The benchmark suite and the LDBC SNB reproducer that surfaced this are in the repository — cargo bench --bench ldbc_benchmark, documented in docs/BENCHMARKS.md.

We'd rather publish this than the regression that wasn't real.

Twelve questions, six knowledge graphs, and not one query written by hand

Himadri — Thu, 20 Aug 2026 12:27:23 +0000

Stuart Broad has dismissed David Warner 20 times.

That is not a fact you can retrieve by embedding cricket commentary and searching for something similar. It is a count() over a typed edge. Ask a vector index and you get paragraphs about their rivalry — the number is nowhere, because the number was never written down anywhere. It only exists once you model dismissals as relationships and count them.

I have been trying to work out where that line falls: which questions want a vector index, and which want a graph. The most direct way to find out was to give a model six knowledge graphs and ask it things.

The setup

Six MCP servers, each in front of a different graph: bank model risk, cricket, drug interactions, football, Indian Supreme Court judgments, and biological pathways. Claude Code on the other end.

Before asking anything, I fixed the rules so I could not quietly help it along:

You are running a live, recorded demo with 6 MCP servers connected.
RULES for every question I ask next:

- Call exactly ONE MCP tool on the matching server, then stop.
- Answer in ONE short line per row, top 3 only.
- No preamble, no "let me...", no explanation, no follow-up offers. Just the result.
- Never restart or reconfigure anything. Never ask me to confirm.
Reply "Ready." and nothing else.

One tool call per question. No retries, no second attempts, no picking a better phrasing after seeing a bad answer. Whatever came back is what is in the video.

I never write a query. Claude picks the server, writes the Cypher, and the engine answers.

What came back

Bank model risk — "what feeds the CCAR stress test?"

Credit Scorecard · Unsecured Personal #00 (Model) → CCAR PPNR · CRE #15
Credit Scorecard · Unsecured Personal #00 (Model) → CCAR PPNR · CRE #27
CCAR PPNR · Project Finance #01 (Model) → CCAR 2026

This is lineage — a path through models, not a lookup. The interesting part is the shape of the answer: which upstream models reach a regulatory submission, and by which route. There is no document that states this. It is a traversal.

Pharmacology — "heaviest side-effect burden"

Pregabalin — 839 side effects
Aripiprazole — 827
Citalopram — 823

Biology — "most connected protein hubs"

TP53 — 739 partners
RPS27A — 717
EGFR — 502

Degree centrality, in other words. TP53 coming top is exactly what anyone in the field would expect, which is the point — it is a sanity check that the graph is loaded correctly and the question was understood.

Indian Supreme Court — "most productive judges"

Dipak Misra — 104 cases
T. S. Thakur — 81
Rohinton F. Nariman — 74

Six domains, one session, one tool call each. The model switches between banking and cricket and pharmacology mid-conversation without being told which server to use — it picks from the tool descriptions.

The part that is actually interesting

Every one of these is a counting or path question over typed relationships. Not one of them is a similarity question.

That is the split I have landed on, and it is less about which technology is better than about what shape the question has:

"What is this like?" — a vector index. Fuzzy, semantic, tolerant of phrasing.
"How many, via what route, ranked by what?" — a graph. Exact, structural, and gives the same answer twice.

RAG systems overwhelmingly do the first, then struggle when a user asks the second — because "who has dismissed Warner most" retrieves match reports, and the model counts them badly.

The honest caveat: there is no vector search anywhere in this video. Samyama Graph has HNSW vector search in the same engine, and none of these twelve questions use it. It is all traversal. The genuinely interesting case is the hybrid — retrieve by embedding, then expand the graph around the hits, or traverse first and embed the neighbourhood — and I have only tried the second.

Two things I would not put in a launch post

A cold query is slow. In the standalone cricket case study, the first traversal takes 3.1 seconds and the second takes 0.68. Same shape of query, same data. That is a cold cache, and it is the first number anyone will see.

One answer is a data-modelling trap. Asked for World Cup titles, the football graph returns Brazil 5, United States 4, Italy 4. If you know football you have already spotted it — that is men's and women's competitions counted together. The engine answered exactly what was asked; the graph does not distinguish the two tournaments. Which is a decent illustration of the real failure mode of this whole approach: the model will faithfully answer a question your schema quietly misunderstood, and it will sound completely confident doing it.

Reproducing it

All eleven case studies are built from public datasets and run end to end — cricket from Cricsheet (CC BY 4.0), drug interactions, pathways, football, legal judgments, and the rest:

https://github.com/samyama-ai/samyama-graph/tree/main/case_studies

The cricket graph is 36,619 nodes and 1,392,017 edges, which is small enough to load on a laptop and large enough that the traversals are not trivial.

Disclosure

I work on Samyama Graph — an open-source graph-vector database written in Rust, with OpenCypher queries and HNSW vector search in one engine, 14 graph algorithms, Apache-2.0. So I am not a neutral party on the engine.

I am much less certain about the retrieval question, and that is the bit I would like to argue about: for hybrid graph-plus-vector retrieval feeding an LLM, is it better to traverse first and embed the neighbourhood, or retrieve by embedding and expand the graph around the hits?

I have only tried the first. If you have tried the second, I would like to hear how it went.

https://github.com/samyama-ai/samyama-graph/tree/main/case_studies

Your GraphRAG stack is two databases. It should be one.

Himadri — Tue, 21 Jul 2026 11:00:00 +0000

Most GraphRAG systems today are built with at least two storage layers.

One system stores vectors.

Another system stores relationships.

The vector database answers questions like:

Which chunks, entities, documents, or concepts are semantically close to this query?

The graph database answers questions like:

How are these entities connected, and what paths explain the relationship?

That architecture works, but it creates an important problem:

The join between semantic retrieval and graph reasoning happens outside the database.

Usually, that join happens in application code, orchestration code, or an agent workflow. The application retrieves vector matches, sends those IDs to a graph database, performs a traversal, gets more nodes, and then asks an LLM to assemble the final answer.

It works.

But it is not ideal.

Because once the join moves outside the database, the query planner loses control.

The system can no longer optimize vector search, graph traversal, filtering, ranking, and graph algorithms together. Each layer only sees part of the problem.

That is the architectural problem we are exploring with Samyama Graph.

Samyama Graph is a Rust-native graph-vector database designed for GraphRAG, knowledge graphs, AI agent memory, and large-scale relationship analytics.

It brings together:

OpenCypher-style graph querying
Vector search
Graph algorithms
Redis-compatible access
A single-binary Rust runtime

The idea is simple:

Graph traversal and vector search should not always live in separate systems. For many GraphRAG workloads, they belong in the same engine.

Why GraphRAG needs more than vector search

Vector search is very good at semantic similarity.

It can find text, entities, or documents that “feel close” to a query. That is extremely useful for RAG.

But vector search alone does not understand relationships.

For example, if a user asks:

Which clinical trials are connected to a drug, a pathway, and a condition through supporting biomedical literature?

A vector search system may retrieve relevant documents. But it does not naturally answer:

Which drug is connected to which condition?
Which pathway links them?
Which paper supports the relationship?
Which clinical trial is related?
Which path through the knowledge graph explains the answer?

Those are graph questions.

A graph can represent entities and relationships directly:

Drug -> targets -> Protein
Protein -> participates_in -> Pathway
Pathway -> associated_with -> Condition
Condition -> studied_in -> ClinicalTrial
Paper -> supports -> Relationship

This is where GraphRAG becomes more powerful than basic RAG.

Instead of retrieving only similar text, the system can retrieve relevant entities and then reason over relationships.

The common GraphRAG architecture

A common GraphRAG architecture looks like this:

User query
   ↓
Embedding model
   ↓
Vector database
   ↓
Application code / agent logic
   ↓
Graph database
   ↓
Application code / reranking
   ↓
LLM response

This is practical, but it adds complexity.

The application has to decide:

How many vector results to fetch
Which IDs to send into the graph
How far to traverse
Which paths matter
How to combine graph scores and vector scores
How to avoid duplicated, irrelevant, or weakly connected context
How to explain why the final context was selected

Over time, the orchestration layer becomes a custom query engine.

But it is not really a query engine. It usually has no cost model, no unified optimizer, and no deep understanding of the data layout across both retrieval modes.

That is the design smell.

If the application is doing the join between vector search and graph traversal, the database is no longer solving the full retrieval problem.

A different design bet

Samyama Graph takes a different approach.

Instead of treating graph and vector as separate stores, it explores what happens when both are part of the same database engine.

Samyama Graph is open source on GitHub:
https://github.com/samyama-ai/samyama-graph

The goal is to support workflows where developers can combine:

semantic retrieval
graph pattern matching
relationship traversal
graph algorithms
structured filtering

inside one system.

In practice, this matters because GraphRAG is rarely just “find the nearest chunks.”

A useful GraphRAG query often looks more like:

Find semantically relevant concepts, then expand through trusted relationships, filter by entity type, rank by graph structure, and return an explainable path.

That is a graph-vector problem.

Not just a vector problem.

Why one engine can be useful

Keeping graph traversal and vector search closer together has several advantages.

1. Less orchestration code

When vector and graph live separately, a lot of glue code is needed.

Developers have to write custom logic to move IDs, scores, metadata, and filters between systems.

A single engine can reduce that surface area.

2. Better explainability

GraphRAG systems should not only return an answer.

They should help explain why that answer was retrieved.

Graphs are naturally explainable because they can show paths:

Question
 -> matched entity
 -> related concept
 -> supporting source
 -> final answer context

This is especially important in domains like healthcare, finance, cybersecurity, compliance, and enterprise knowledge management.

3. Better fit for relationship-heavy data

Some data is naturally connected:

biomedical knowledge
clinical trials
fraud networks
infrastructure dependencies
software architecture
supply chain relationships
enterprise knowledge graphs
agent memory

For these workloads, relationships are not optional metadata. They are the core of the problem.

4. Cleaner developer experience

A single graph-vector database can make the stack easier to reason about.

Instead of asking:

Which database owns this part of retrieval?

developers can ask:

What graph-vector query should I run?

Why Rust?

Samyama Graph is written in Rust.

Rust is a good fit for database infrastructure because it gives strong control over memory, performance, and concurrency without requiring a garbage-collected runtime.

For a graph-vector database, that matters.

Graph workloads can be memory-intensive. Vector search can be compute-heavy. Combining both in one engine requires careful attention to performance and predictable execution.

Rust gives us a strong foundation for that direction.

What Samyama Graph supports today

Samyama Graph currently focuses on:

OpenCypher-style graph querying
HNSW-based vector search
Graph algorithms
Redis-compatible protocol access
Single-binary deployment
Docker-based local setup
Knowledge graph and GraphRAG-style workloads

It is not trying to be every database at once.

The goal is to be useful for developers who need connected-data reasoning and semantic retrieval in the same system.

Where it falls short today

Samyama Graph is still growing, and we want to be transparent about that.

A few important limitations today:

OpenCypher support is not yet complete
The product and ecosystem are still evolving
Examples, integrations, and tutorials are being expanded
Some larger benchmark-style claims need more public reproducibility support

We believe open-source infrastructure earns trust when it is clear about both strengths and current gaps.

What can you build with Samyama Graph?

Samyama Graph is useful when your application needs both connected-data reasoning and semantic retrieval.

Examples include:

GraphRAG systems that combine vector search with graph traversal
Knowledge graph applications for enterprise, research, healthcare, and operations data
AI agent memory where entities, tools, actions, and context are stored as a graph
Biomedical and clinical graphs across papers, trials, pathways, drugs, and conditions
Fraud and investigation graphs for relationship discovery and pattern analysis
Infrastructure and dependency graphs for impact analysis and root-cause exploration
Large-scale graph analytics using built-in graph algorithms

Try it locally

You can run Samyama Graph with Docker:

docker run -d -p 6379:6379 -p 8080:8080 ghcr.io/samyama-ai/samyama-graph:latest

Then connect with a Redis client:

redis-cli -p 6379

Create a simple graph:

GRAPH.QUERY mydb "CREATE (a:Person {name: 'Alice'})-[:KNOWS]->(b:Person {name: 'Bob'})"

Query it:

GRAPH.QUERY mydb "MATCH (a)-[:KNOWS]->(b) RETURN a.name, b.name"

Why we are building this

We believe the next generation of AI applications will need more than text retrieval.

They will need:

semantic search
structured relationships
graph traversal
explainable paths
memory over entities and events
ranking across both similarity and structure

That is why we are building Samyama Graph.

Not as “just another vector database.”

Not as “just another graph database.”

But as a graph-vector database for developers building GraphRAG, knowledge graph, and AI infrastructure systems.

Open source

Samyama Graph is open source on GitHub:

https://github.com/samyama-ai/samyama-graph

If this project is useful to your GraphRAG, knowledge graph, or AI infrastructure work, a GitHub star helps more developers discover it.

We would also welcome feedback, issues, examples, and contributions from developers working on graph databases, vector search, Rust infrastructure, RAG, and AI agents.