DEV Community: Samyama.AI

We nearly published a 2x regression that didn't exist

Himadri — Mon, 24 Aug 2026 07:02:47 +0000

We were about to ship a performance write-up claiming three queries had gotten 2-4x slower. All three numbers were wrong, and the reason is one every benchmark author should worry about: the same binary, on the same machine, running the same query, gave different answers depending on what time of day we ran it.

The measurement that didn't add up

While checking a change to Samyama Graph, we timed LDBC SNB Interactive query IC9 on one commit, one dataset, one 16-core workstation, four hours apart:

when	commit	IC9 median
morning	`b2a3557`	2,822 ms
evening	`b2a3557`	4,912 ms

Same binary. 1.74x slower, four hours later. ps showed the benchmark pinned at 100% of one core and nothing else on the box above 9%. No other process explains that gap.

So we ran it again — same binary, minutes apart this time, nothing changed in between:

run	IC9 median
1	2,011 ms
2	2,490 ms

24% apart. Back to back.

Why we almost missed it

We had a real change on main to compare against a baseline on an older commit. Baseline was measured in the morning; the new code was measured that evening. Read naively, the evening numbers said IC10 had gotten 2x slower, IC11 4x, IC12 3x.

None of those regressions existed. The host itself had drifted between the two sittings — thermal throttling, frequency scaling, background load, whatever the cause, the machine at 6pm was not the machine at 9am. We re-ran everything back to back, on one sitting, and the picture reversed:

query	before (`b2a3557`)	after (`main`)	ratio
IC9	4,912 ms	2,490 ms	1.97x faster
IC10	340 ms	352 ms	~1.0x
IC11	382 ms	400 ms	~1.0x
IC12	428 ms	483 ms	~1.1x — inside the 24% band

The change was a genuine ~2x win on IC9 and a wash everywhere else. The "2-4x regressions" were the host, not the code.

We only caught it because a 4x regression from a small change looked implausible enough to re-measure. A smaller, real regression sitting inside that same noise band would not have raised anyone's eyebrows, and would have shipped.

What this means for anyone reading benchmark numbers

Two things, and they cut in opposite directions:

A regression can hide inside the noise. If your benchmark's run-to-run variance is 24%, any real regression under ~25% is statistically invisible next to it. A nightly CI gate set to fire on a 25% threshold would page on noise some nights and miss a real 20% regression every other night.

An improvement can be manufactured by timing alone. Measure a baseline in the morning and a new version in the evening on a drifting host, and you can produce whatever ratio you want without touching a line of code. This is true whether the drift is accidental or not — which is exactly why "before" and "after" have to come from one sitting.

Neither of these is specific to our engine, our benchmark suite, or Rust. It's a property of measuring wall-clock time on a shared physical machine, and it will bite any project that compares two numbers taken hours apart and calls the difference a result.

What we changed

A calibration line in every run. Each benchmark now times a short, fixed, CPU-bound operation at the start and the end, and reports it alongside load average and mean core frequency. Two runs whose calibration numbers disagree were taken on hosts running at different speeds, whatever the milliseconds of the actual queries say. If a single run's own opening and closing calibration differ by more than 10%, the run says so — its numbers aren't even internally comparable to each other.

A rule for before/after claims. A performance comparison needs both numbers from one back-to-back session on one machine — not "the same machine," the same sitting. Quote the ratio, not the absolute milliseconds, when the point is an improvement: a ratio measured back to back survives a slow host; an absolute number doesn't survive being read next to one taken elsewhere.

Cross-engine ratios get the same treatment. If a "we're 3x faster than X" claim has our number from one session and the competitor's from another, the ratio isn't a ratio — it's two unrelated numbers divided by each other. Same machine, same sitting, or the comparison doesn't get published.

The full account, including the four numbers that started this, is in the original issue: samyama-ai/samyama-graph#529. The benchmark suite and the LDBC SNB reproducer that surfaced this are in the repository — cargo bench --bench ldbc_benchmark, documented in docs/BENCHMARKS.md.

We'd rather publish this than the regression that wasn't real.

Your GraphRAG stack is two databases. It should be one.

Himadri — Tue, 21 Jul 2026 11:00:00 +0000

Most GraphRAG systems today are built with at least two storage layers.

One system stores vectors.

Another system stores relationships.

The vector database answers questions like:

Which chunks, entities, documents, or concepts are semantically close to this query?

The graph database answers questions like:

How are these entities connected, and what paths explain the relationship?

That architecture works, but it creates an important problem:

The join between semantic retrieval and graph reasoning happens outside the database.

Usually, that join happens in application code, orchestration code, or an agent workflow. The application retrieves vector matches, sends those IDs to a graph database, performs a traversal, gets more nodes, and then asks an LLM to assemble the final answer.

It works.

But it is not ideal.

Because once the join moves outside the database, the query planner loses control.

The system can no longer optimize vector search, graph traversal, filtering, ranking, and graph algorithms together. Each layer only sees part of the problem.

That is the architectural problem we are exploring with Samyama Graph.

Samyama Graph is a Rust-native graph-vector database designed for GraphRAG, knowledge graphs, AI agent memory, and large-scale relationship analytics.

It brings together:

OpenCypher-style graph querying
Vector search
Graph algorithms
Redis-compatible access
A single-binary Rust runtime

The idea is simple:

Graph traversal and vector search should not always live in separate systems. For many GraphRAG workloads, they belong in the same engine.

Why GraphRAG needs more than vector search

Vector search is very good at semantic similarity.

It can find text, entities, or documents that “feel close” to a query. That is extremely useful for RAG.

But vector search alone does not understand relationships.

For example, if a user asks:

Which clinical trials are connected to a drug, a pathway, and a condition through supporting biomedical literature?

A vector search system may retrieve relevant documents. But it does not naturally answer:

Which drug is connected to which condition?
Which pathway links them?
Which paper supports the relationship?
Which clinical trial is related?
Which path through the knowledge graph explains the answer?

Those are graph questions.

A graph can represent entities and relationships directly:

Drug -> targets -> Protein
Protein -> participates_in -> Pathway
Pathway -> associated_with -> Condition
Condition -> studied_in -> ClinicalTrial
Paper -> supports -> Relationship

This is where GraphRAG becomes more powerful than basic RAG.

Instead of retrieving only similar text, the system can retrieve relevant entities and then reason over relationships.

The common GraphRAG architecture

A common GraphRAG architecture looks like this:

User query
   ↓
Embedding model
   ↓
Vector database
   ↓
Application code / agent logic
   ↓
Graph database
   ↓
Application code / reranking
   ↓
LLM response

This is practical, but it adds complexity.

The application has to decide:

How many vector results to fetch
Which IDs to send into the graph
How far to traverse
Which paths matter
How to combine graph scores and vector scores
How to avoid duplicated, irrelevant, or weakly connected context
How to explain why the final context was selected

Over time, the orchestration layer becomes a custom query engine.

But it is not really a query engine. It usually has no cost model, no unified optimizer, and no deep understanding of the data layout across both retrieval modes.

That is the design smell.

If the application is doing the join between vector search and graph traversal, the database is no longer solving the full retrieval problem.

A different design bet

Samyama Graph takes a different approach.

Instead of treating graph and vector as separate stores, it explores what happens when both are part of the same database engine.

Samyama Graph is open source on GitHub:
https://github.com/samyama-ai/samyama-graph

The goal is to support workflows where developers can combine:

semantic retrieval
graph pattern matching
relationship traversal
graph algorithms
structured filtering

inside one system.

In practice, this matters because GraphRAG is rarely just “find the nearest chunks.”

A useful GraphRAG query often looks more like:

Find semantically relevant concepts, then expand through trusted relationships, filter by entity type, rank by graph structure, and return an explainable path.

That is a graph-vector problem.

Not just a vector problem.

Why one engine can be useful

Keeping graph traversal and vector search closer together has several advantages.

1. Less orchestration code

When vector and graph live separately, a lot of glue code is needed.

Developers have to write custom logic to move IDs, scores, metadata, and filters between systems.

A single engine can reduce that surface area.

2. Better explainability

GraphRAG systems should not only return an answer.

They should help explain why that answer was retrieved.

Graphs are naturally explainable because they can show paths:

Question
 -> matched entity
 -> related concept
 -> supporting source
 -> final answer context

This is especially important in domains like healthcare, finance, cybersecurity, compliance, and enterprise knowledge management.

3. Better fit for relationship-heavy data

Some data is naturally connected:

biomedical knowledge
clinical trials
fraud networks
infrastructure dependencies
software architecture
supply chain relationships
enterprise knowledge graphs
agent memory

For these workloads, relationships are not optional metadata. They are the core of the problem.

4. Cleaner developer experience

A single graph-vector database can make the stack easier to reason about.

Instead of asking:

Which database owns this part of retrieval?

developers can ask:

What graph-vector query should I run?

Why Rust?

Samyama Graph is written in Rust.

Rust is a good fit for database infrastructure because it gives strong control over memory, performance, and concurrency without requiring a garbage-collected runtime.

For a graph-vector database, that matters.

Graph workloads can be memory-intensive. Vector search can be compute-heavy. Combining both in one engine requires careful attention to performance and predictable execution.

Rust gives us a strong foundation for that direction.

What Samyama Graph supports today

Samyama Graph currently focuses on:

OpenCypher-style graph querying
HNSW-based vector search
Graph algorithms
Redis-compatible protocol access
Single-binary deployment
Docker-based local setup
Knowledge graph and GraphRAG-style workloads

It is not trying to be every database at once.

The goal is to be useful for developers who need connected-data reasoning and semantic retrieval in the same system.

Where it falls short today

Samyama Graph is still growing, and we want to be transparent about that.

A few important limitations today:

OpenCypher support is not yet complete
The product and ecosystem are still evolving
Examples, integrations, and tutorials are being expanded
Some larger benchmark-style claims need more public reproducibility support

We believe open-source infrastructure earns trust when it is clear about both strengths and current gaps.

What can you build with Samyama Graph?

Samyama Graph is useful when your application needs both connected-data reasoning and semantic retrieval.

Examples include:

GraphRAG systems that combine vector search with graph traversal
Knowledge graph applications for enterprise, research, healthcare, and operations data
AI agent memory where entities, tools, actions, and context are stored as a graph
Biomedical and clinical graphs across papers, trials, pathways, drugs, and conditions
Fraud and investigation graphs for relationship discovery and pattern analysis
Infrastructure and dependency graphs for impact analysis and root-cause exploration
Large-scale graph analytics using built-in graph algorithms

Try it locally

You can run Samyama Graph with Docker:

docker run -d -p 6379:6379 -p 8080:8080 ghcr.io/samyama-ai/samyama-graph:latest

Then connect with a Redis client:

redis-cli -p 6379

Create a simple graph:

GRAPH.QUERY mydb "CREATE (a:Person {name: 'Alice'})-[:KNOWS]->(b:Person {name: 'Bob'})"

Query it:

GRAPH.QUERY mydb "MATCH (a)-[:KNOWS]->(b) RETURN a.name, b.name"

Why we are building this

We believe the next generation of AI applications will need more than text retrieval.

They will need:

semantic search
structured relationships
graph traversal
explainable paths
memory over entities and events
ranking across both similarity and structure

That is why we are building Samyama Graph.

Not as “just another vector database.”

Not as “just another graph database.”

But as a graph-vector database for developers building GraphRAG, knowledge graph, and AI infrastructure systems.

Open source

Samyama Graph is open source on GitHub:

https://github.com/samyama-ai/samyama-graph

If this project is useful to your GraphRAG, knowledge graph, or AI infrastructure work, a GitHub star helps more developers discover it.

We would also welcome feedback, issues, examples, and contributions from developers working on graph databases, vector search, Rust infrastructure, RAG, and AI agents.