ShipStation Global built Deckhand, an AI IT help desk agent that lives inside Slack: it answers questions from a curated knowledge base, reads screenshots of error messages, and files Jira tickets — all on Amazon Bedrock AgentCore, with no servers to manage. The results sound spectacular: 97.5% of channel queries resolved, 1,600 hours saved a year.
But here's the catch the article digs into: the two sources describe that same 97.5% figure differently. AWS calls it a resolution rate. Builder Nick Ciubotariu calls it the proportion of messages Deckhand replied to. A reply is not a resolution.
Key points:
- Curated, not scraped: The knowledge base came from years of Slack history, mined with "Recursive Language Models" for threads where a working fix was visible — not every discussion treated as equally useful. Preparing the knowledge source matters as much as the agent.
- Answer, escalate, yield: Deckhand reads error screenshots; when it can't solve something, a one-click action converts the conversation into a detailed Jira ticket. And it learned when to stay silent — recognizing when a human has taken over and backing off.
- The metric deserves scrutiny: AWS says 97.5% of channel queries resolved in the first three months, 96% positive feedback. Ciubotariu says 97.5% of messages replied to. Positive feedback did climb from 82% at launch to 96%, a third of questions arrived after hours (Deckhand answered 94% of those), and 409 of 1,221 conversations became tickets. Honest vendor metrics, but not independently verified.
- Read the numbers, keep the wording: The article argues the 97.5% figure should stay tied to each source's exact wording until it's independently verified. A response rate can't capture whether the system appropriately yields a conversation to a person.
Go deeper
The full analysis — including the knowledge-preparation lessons and what "replied vs. resolved" means for your own support-bot metrics — is on Agentic Affairs:
How ShipStation Global built Deckhand, its Slack AI help desk
Originally published on Agentic Affairs (https://agentic-affairs.com)
Top comments (0)