How can AI code review tools be evaluated reliably for real-world pull requests?

I am evaluating AI-assisted code review systems and have found that published benchmarks use substantially different methodologies. Some replay historical bugs, some evaluate complete pull requests, some inject synthetic defects into real code, and others use human or LLM-based judgments of review comments.

I am trying to design a fair evaluation for systems that review real-world pull requests.

The main issues I am running into are:

• Dataset construction: Should the test set consist primarily of historical production bugs, clean pull requests, synthetic defects, or a combination?
• Ground truth: How should a benchmark define the complete set of valid findings when the original human review may have missed defects?
• Precision and recall: Should findings be matched individually, or should a PR count as correct if the system identifies at least one valid issue?
• False positives: How should speculative, stylistic, duplicate, or technically correct but non-actionable comments affect the score?
• Severity: Should critical security or correctness defects receive more weight than low-severity maintainability issues?
• Human review: What level of expert annotation or adjudication is necessary to establish reliable ground truth?
• Repository context: Should systems receive the full repository, issue description, tests, commit history, and search capabilities, or should all systems be restricted to the diff?
• Languages and projects: How should results be normalized across programming languages, repositories, domains, and different patch sizes?
• Reproducibility: What artifacts and experimental details should be published so another team can independently reproduce the evaluation?

For example, a system that produces 20 comments and correctly identifies 10 defects is quite different from one that produces 5 comments and correctly identifies 4, even if their aggregate scores happen to be similar. Likewise, a benchmark based only on human review comments may penalize a system for finding a legitimate defect that the original reviewers missed.

I am looking for a methodology that measures useful defect detection on real pull requests, rather than simply similarity to historical review comments.

What benchmark design and scoring procedure would provide the most statistically and methodologically defensible comparison of AI code review systems, including dataset construction, ground-truth validation, finding-level precision/recall, severity weighting, repository context, human adjudication, and reproducibility?