I Built a Graph-Based System to Understand Why a Pull Request Might Be Risky
Tags: "#webdev" "#opensource" "#ai" "#programming"
Most code review tools focus on what changed in a pull request.
That makes sense. A PR changes a few files, introduces some new logic, modifies existing behavior, and reviewers try to figure out whether everything still works.
But there's another question I've been thinking about:
What if the risky part of a pull request isn't actually in the code it changes, but in the history and decisions surrounding that code?
Maybe a similar issue was fixed a few months ago. Maybe a previous architectural decision explains why a particular implementation exists. Maybe an incident in a related module is worth looking at before approving the change.
That idea led me to build Provenance for Graph Hacks.
It's a project I'm developing to explore how repository history, incidents, architectural decisions, and relationships between code can help reviewers understand potential risks in a pull request.
It's still a work in progress, and I've learned quite a bit from trying to make the idea work beyond a simple demo.
The problem I wanted to explore
Imagine opening a pull request that changes a small part of a large codebase.
The diff looks reasonable. The tests pass. Nothing immediately stands out.
However, the modified module might be connected to a component that caused a production issue in the past. Or the code might depend on an architectural decision that isn't obvious from the implementation.
A normal diff doesn't automatically bring that context together.
Of course, an experienced developer can investigate the history manually. Git logs, issue discussions, previous pull requests, and documentation all help.
The problem is that connecting all of this information takes time, especially in a repository that has been evolving for years.
I wanted to experiment with a system that could connect these pieces of information and make relevant historical context easier to discover during code review.
The goal isn't to replace developers or automatically decide whether a PR should be merged. It's to give reviewers another source of evidence before they make that decision.
Why a graph?
My first important design choice was to represent repository knowledge as a graph.
A repository contains more than files and commits. It also contains relationships between people, changes, incidents, decisions, and components.
For example, a simplified graph might look like this:
"Pull Request → Changes → Module → Related Incident"
Or:
"Architectural Decision → Affects → Component"
These relationships can help connect a proposed change with information that might otherwise remain scattered across the repository's history.
Instead of treating every piece of information as an isolated document, the graph provides a way to explore how different pieces of repository knowledge relate to one another.
I'm using FalkorDB for graph storage and queries in the project.
The important part isn't simply storing nodes and edges. It's making sure the relationships are meaningful, the evidence can be traced back to its source, and historical information is used at the correct point in time.
How the system is structured
I've been building the project in stages rather than trying to implement everything at once.
At a high level, the workflow looks like this:
- Ingestion: Collect repository information and prepare it for processing.
- Knowledge extraction: Identify relevant incidents, decisions, and other contextual information from available sources.
- Graph construction: Represent the information and its relationships in FalkorDB.
- Risk analysis: Examine a pull request against the available repository context.
- Evaluation: Compare the system's predictions with known outcomes and inspect where it gets things wrong.
The project also includes a web application and supporting scripts for evaluation and development.
One challenge is that these stages depend on one another. A risk-scoring algorithm can't reliably use an architectural decision that was never extracted or stored correctly in the first place.
That's why I've been trying to keep ingestion, graph queries, analysis, and evaluation separate enough to test independently.
The difficult part: making risk scores meaningful
At first glance, risk scoring sounds straightforward.
Find related incidents, look at the affected modules, combine the available evidence, and calculate a score.
The real problem is deciding what to do when evidence is missing.
Suppose a pull request has a relevant historical incident but no verified architectural decision attached to it. Should the absence of a decision automatically make the PR look safer?
Not necessarily.
Missing information isn't the same thing as evidence that everything is safe.
I updated the scoring logic to account for the evidence channels that are actually available rather than automatically treating missing channels as zero-risk signals. I also tightened temporal filtering so that the analysis uses information available at the relevant historical cutoff.
That second part matters more than it might seem.
If I'm evaluating an old pull request, the system shouldn't use an incident that happened after that pull request was merged. Doing so would make the system appear to predict something using information it couldn't have known at the time.
This is a classic data leakage problem, and it can make an evaluation look much better than the underlying system really is.
I don't want a risk score that looks impressive because it has access to the future.
I want to know what the system could reasonably have concluded from the evidence available at that point.
Evaluating it against real pull requests
I added an evaluation harness and tested the risk-analysis pipeline against a set of 21 historical pull requests from "vitejs/vite".
The evaluation uses historical merge timestamps and applies time-based cutoffs to reduce future-information leakage. The PRs are processed through the risk-analysis pipeline rather than relying on manually assigned scores.
The first evaluation was disappointing.
The system detected none of the risky examples in the benchmark. Recall was 0%, and accuracy was around 52.38%.
That was a useful result, even if it wasn't the result I wanted.
It showed that having a graph database, a risk formula, and a working pipeline doesn't automatically mean the system is useful.
After investigating the evaluation and scoring behavior, I made several changes, including correcting historical timestamps, handling missing evidence channels, improving temporal filtering, and aligning the evaluation threshold with the project's risk bands.
The subsequent evaluation reported:
Metric| Result
Precision| 87.5%
Recall| 70%
Accuracy| 81.0%
True positives| 7
False positives| 1
False negatives| 3
On this 21-PR benchmark, the system detected 7 of the 10 examples labelled risky. It also produced one false positive among 11 examples labelled normal.
That's a meaningful improvement over the initial result, but I'm being careful about how I interpret it.
These numbers describe a small evaluation set, not proven production performance.
The scoring and threshold were adjusted using this benchmark, so the results aren't an independent measure of generalization. The three missed risky PRs also matter. A system designed to surface potential risks shouldn't hide its failures behind a single accuracy number.
The next step is to test against additional, independently selected examples and investigate the remaining false negatives without changing labels or hardcoding special cases.
The challenges that don't show up in a demo
A few things have made this project more interesting than simply connecting a model to a graph database.
- Extracting reliable knowledge
A repository contains a lot of information, but not all of it is a verified architectural decision or a meaningful incident.
Extraction can fail because of model availability, provider limits, or ambiguous source material. Even when extraction succeeds, the resulting information needs to be checked and connected to the correct entities.
A graph full of unreliable relationships doesn't solve the problem. It just makes unreliable information easier to navigate.
- Missing evidence
Some pull requests have rich historical context. Others don't.
The system needs to distinguish between a genuinely low-risk result and a result where there simply isn't enough evidence to make a confident assessment.
That's still an area where I want to improve the project's treatment of uncertainty.
- Temporal correctness
Historical analysis only makes sense if the system respects time.
A relationship can be valid today without having been known when an older PR was reviewed. Getting this right requires more than filtering the visible nodes; the underlying queries and relationships must also respect the cutoff.
- Evaluating the system honestly
It's tempting to keep adjusting a threshold until the metrics look good.
But a better score isn't necessarily a better system. A useful evaluation needs fixed labels, transparent methodology, appropriate baselines, and tests that expose failures instead of hiding them.
Working on this project has made that distinction much more concrete for me.
What's next?
The next planned stage is focused on making the repository's historical context more useful to developers.
I'm working toward three areas:
- A repository time slider: Explore the graph at a historical cutoff and see what information was available at that point.
- Contributor ownership and bus-factor analysis: Estimate how concentrated code ownership is across contributors and identify modules that depend heavily on a small number of people.
- UI and workflow improvements: Make the application easier to navigate, with better loading states, error handling, and clearer explanations of the evidence behind a result.
These features still need implementation and verification. I don't want to describe them as finished before they're actually working.
I'm also keeping external GitHub PR comments disabled while I continue validating the analysis locally. I would rather understand the system's limitations before letting it act on real pull requests.
What I'm taking away from this project
One thing I've realized while building Provenance for Graph Hacks is that the interesting part of an AI-assisted developer tool isn't necessarily the model.
The surrounding engineering matters just as much.
Where does the information come from? Can I trace a conclusion back to its evidence? What happens when the data is incomplete? Can I reproduce the result? And does the evaluation measure what I actually care about?
Those questions have influenced the project more than simply trying to add another AI feature.
The project isn't finished, and the current evaluation has clear limitations. But I now have a concrete pipeline, a graph-based representation of repository context, and an evaluation process that exposes problems I can work on.
For me, that's a more useful milestone than a polished demo that only works on its best example.
I'm continuing to build and test it, and I'll share more as the next stage takes shape.
If you've worked on code intelligence, graph-based systems, or risk analysis for pull requests, I'd genuinely like to hear your thoughts. How would you represent historical decisions and incidents so that a reviewer can understand not just the score, but the evidence behind it?
Thanks for reading.
Project: Provenance for Graph Hacks
Note: Evaluation figures above are from the current 21-PR benchmark and should not be interpreted as production accuracy.
Top comments (0)