Claude Sonnet turned 23 sets of commits into release notes. Asked plainly, it listed internal work and invented a link host; with written rules it never did.
Not reliably. When it misses, it says more than the commits justify: it keeps work that no user will ever notice, and it supplies details nobody gave it. We gave Claude Sonnet 23 sets of commits and asked for release notes. With nothing but the request, 15 of the 23 entries were right. With a written rule set loaded, all 23 were. Six of its eight plain-request misses were entries about internal work, and in another it made up the host for a compare link. Claude Haiku went from 13 to 22.
Read the full report on AISkills402: https://aiskills402.com/blog/llm-changelog-lists-internal-commits
Top comments (0)