Claude Sonnet and Haiku wrote MCP tool definitions for twelve API operations; we sent 96 calls through each schema. Without our skill, 23 of 24 passed.
Yes, and the result surprised us. We wrote twelve short pieces of API documentation, and for each operation Claude Sonnet and Claude Haiku wrote the MCP tool definition: a name, a description, an inputSchema and annotations. Every schema was compiled with a JSON Schema validator and then given the calls the documentation allows and the calls it forbids, 96 in all. Without any instructions from us, Haiku passed all twelve operations and Sonnet eleven. Sonnet's one miss was not in the schema at all. Our skill for this job brought both models to twelve, and that is the whole difference we could measure.
Read the full report on AISkills402: https://aiskills402.com/blog/llm-mcp-tool-schema-little-help
Top comments (0)