Why a PR walkthrough can pass validation and still be hard to read

I made an HTML guide for reading pull requests. When I shared it with coworkers, the visual feedback was encouraging, but I still felt that parts were hard to follow. One person wanted to see the explanation before the code pane, which led to a layout switch.

That feedback points to a limit of validation: source references can all be valid while the reader still lacks the context needed to understand a change.

The implementation discussed here is my project, PR Tour. A coding agent writes the reading order, explanations, and definition mappings into a manifest. A Python builder reads the repository and produces the HTML. The builder can check the source material much more directly than it can check the explanation.

Make every excerpt belong to a snapshot

The manifest records the PR URL and base and head commits. The builder resolves the commits, computes their merge base, and collects the changed files between that merge base and the head. The left and right sides of the guide come from those snapshots.

This makes a reference reproducible. It also imposes a limit: if the PR changes afterward, the saved HTML still describes the old snapshot. A working link to a PR does not make the embedded code current.

The useful invariant is that an excerpt and its line numbers refer to the same source revision. Whether the explanation correctly describes that revision remains a separate review question.

Reconstruct both sides of the diff

A normal diff omits unchanged regions. For supported text files, the builder adds that context back as expandable rows, then checks the result against the source from Git.

It checks two properties on each side:

• The line numbers form the complete expected sequence.
• The reconstructed line text equals the original source text.

Checking the numbering alone would leave room for incorrect text at a valid position. Checking text alone would miss errors in the line labels that notes rely on. Both are needed for a trustworthy source view.

The builder also requires every changed file to appear in a reading step, including files it cannot render as a text diff. Binary and non-UTF-8 files get a notice directing the reader to GitHub.

That is file coverage, not explanation coverage. A step can mention a file and still omit an important failure path. The same file can also appear in multiple steps, because following a behavior may require returning to it later.

Validate link placement without claiming symbol resolution

Definition previews use authored mappings. The builder verifies that the target exists, its source range is valid, and the clickable text matches the source. An explicit text link must be unambiguous or include its column. Overlapping links are rejected.

There is a small cross-language detail here: Python string positions count Unicode code points, while JavaScript string slicing counts UTF-16 code units. The builder converts positions before rendering them in the browser:

start = len(line[:column].encode("utf-16-le")) // 2
end = start + len(link_text.encode("utf-16-le")) // 2

For example, in "😀send", Python places the s at index 1. JavaScript places it at index 2. Sending the Python index directly to JavaScript would put the link at the wrong position. The conversion concerns string indexing, not visual character width.

Even with correct offsets, the chosen definition can be conceptually wrong. Two objects can have the same name. A working preview proves that a source range exists; it does not prove that the author mapped the reference to the relevant object.

Keep evidence near the explanation

The reader can open selected definitions and inspect the diff beside the explanation. Notes point to the lines being discussed. The layout switch lets readers choose which pane comes first.

These are navigation choices, not evidence that the guide improves comprehension. The public Starlette example has five changed files, eight reading steps, and thirteen definition previews. Those counts describe its structure; they do not measure how well it teaches the change.

The output is one HTML file containing the source excerpts, explanations, styles, and scripts. Sharing it therefore shares the included source code too. The public example uses public Starlette code and includes its attribution.

Test the explanation with a reader

A next evaluation could ask a reviewer to explain a transition between two steps, identify the definition that justifies a claim, or point out a missing failure case. That would test something a line-range validator cannot.

No review-time benchmark or controlled comprehension study has been performed for this release. The upstream Starlette test suite was not run as part of generating the example, either. Building the guide should not be confused with testing the underlying change.

The unresolved question for me is how to find the missing connection in an otherwise complete-looking guide. Valid source references give the reader evidence to inspect. The explanation still has to earn their trust.

Implementation references

• Builder at the version discussed here: snapshot resolution in build, source reconstruction in changed_files, and link checks in collect_links and append_link.
• Public example manifest: the authored reading steps and definition mappings.

Writing disclosure: Codex drafted this article from my project, feedback I shared, and its source code. Codex checked the implementation descriptions against the pinned version linked above.