Sonnet and Haiku each got 19 of 24 social posting API questions right alone and missed the same five. A skill fixed them, though its first wording broke two.
Partly. Asked 24 short questions about posting to six services, Claude Sonnet and Claude Haiku each answered 19 correctly with no help, and they failed the same five. The five were not random. Several were things the services changed or never wrote down. Their plan would have been sound on how the protocols work and wrong on a few facts: one paywall from this year, one rule that runs against intuition, and Pinterest behaviour that no help page states. With a skill's text added, Sonnet reached 24 and Haiku 23, but the first wording of that skill also made Sonnet fail two questions it had answered correctly alone.
Read the full report on AISkills402: https://aiskills402.com/blog/social-api-facts-models-get-wrong?utm_source=devto&utm_medium=social
Top comments (0)