Claude Sonnet described 24 diffs and kept to the code on almost all of them. Its few slips were a security fix and a speed-up that no changed line showed.
Rarely, and the few it adds are the ones a reviewer is most likely to believe. We handed Claude Sonnet 24 diffs and asked it for a pull request description, title included, for each one. From the request alone, 21 of the 24 came out right. Two of its three misses were claims: it called an HTML change a cross-site scripting fix and a set lookup a speed-up, though no changed line said either. With our written rules loaded it got 22 right, so on this test the rules moved Sonnet by one diff. Claude Haiku went from 17 to 19.
Read the full report on AISkills402: https://aiskills402.com/blog/llm-pr-description-claims-diff
Top comments (0)