Ever picked up a GitHub issue in a repo you don't know? The first hours go into one question: where does the fix even live?
I'm building PatchPath for Graph Hacks (WeMakeDevs x FalkorDB) to answer that. It parses a Python repo into a knowledge graph inside FalkorDB - files, symbols, calls, imports, tests - then walks the graph from the issue text to the files and symbols most likely to change. No LLM in the retrieval path. The graph does the work.
This post is not about the idea. It's about what happened when I measured it honestly for a full day.
The setup
First thing I did at the start of the build window: froze a benchmark. 12 real, recently fixed GitHub issues across packaging, requests and click, with the actual merged fixes as the answer key. Every change after that got measured against the same 12 cases. No re-picking, no quiet edits.
41% unknown to 5% unknown
The first analyzer run stored 11,506 call edges for pypa/packaging. 4,747 of them - 41% - were UNRESOLVED, meaning the graph couldn't say who was being called.
Seven versions later: 541 UNRESOLVED, 5%.
The honest breakdown of how:
- Most of the drop was calling things what they are. Calls to builtins, stdlib and third-party code aren't unknowns, they're known-external. Version 0.2.0 added an EXTERNAL class for them.
- The rest was real resolution work. EXACT edges went from 6,102 to 6,720: package re-exports (0.3.0), pytest fixture binding (0.4.0), parameter annotations (0.5.0), builtin-typed locals (0.6.0), external factories like re.match (0.7.0), re-export walk-ups through submodules (0.7.1).
- My favorite commit of the day deleted 7 edges. 0.7.0 found 7 EXACT edges that were confident and wrong, and removed them. A smaller true number beats a bigger fake one.
The benchmark barely moved. That's the point.
File-level hits went 7/12 to 8/12 in the morning, then stayed flat for six straight runs while the analyzer changed five more times.
If the score had climbed every run, you'd know I was tuning to my own test. Flat is what honest iteration looks like: the graph got more truthful, and the retrieval score refused to flatter me.
Where it still fails
- requests #6711: the issue text names no code at all. There is nothing to match. Accepted miss.
- Three test-only fixes count as headline misses. The tool finds all three through a separate test-file oracle (3/3), reported separately, never merged into the headline.
- Fresh repos it had never seen: 4/4 on itsdangerous, 1/4 on jinja. The jinja misses are compiler-internals issues whose text shares no vocabulary with the fixed code. Written down in the repo, not hidden.
The graph earns its place
The load-bearing step is a bounded caller expansion:
MATCH p=(o:Symbol)-[:CALLS*1..3]->(s:Symbol {qualname: $q})
Measured today on the real packaging graph: three hops deep, over 1,250 paths, median 1.02 ms. Fifteen timed runs, warm-up excluded, numbers committed to the repo. Single machine, no throughput claim - but the graph step adds no visible latency inside the loop.
The AI part
I built this with an AI assistant running most of the iteration loops. I chose the concept, set the constraints, froze the benchmarks, and checked every number before it went into a README. The hackathon asks for AI use to be disclosed, and honestly it's the most interesting part: the workflow - measure, change one thing, measure again, delete what's wrong - is what produced something I'd put my name on.
What's not done
The demo video is still being made. Deployment is local-first; there is no hosted version. The symbol-level hit is still a heuristic, and the misses above are real.
Repo, benchmark data, miss breakdowns and the latency table: github.com/MaybeSomeone-arc18/patchpath
Build window ends Oct 18. If you've triaged issues in an unfamiliar repo - what do you wish you knew before you started?
Top comments (0)