DEV Community

Cover image for AWS Service Reference Information MCP Server: how up-to-date, scoped, reference data helps deliver better answers.
AWS Community Builders profile image Paul SANTUS
Paul SANTUS for AWS Community Builders

Posted on • Edited on

1 1

AWS Service Reference Information MCP Server: how up-to-date, scoped, reference data helps deliver better answers.

#aws #genai #iam #mcp

In my previous post, I introduced AWS Service Reference Information MCP Server, a Model Context Protocol server exposing AWS Authorization Reference data to LLMs.

To illustrate how up-to-date, scoped, reference data helps deliver better answers, I took a random question from re:Post, and asked Claude:

Improved answers

Is there a way to restrict "ModifyInstanceAttribute" to specific condition or resource?

You can see both Claude answers, without the MCP server support, and with the server. There are a few notable points:

  • Interestingly, in the conversation without the MCP server, Claude does a better job at listing global condition keys. It's likely because global condition keys are absent from AWS Authorization Reference data (dear AWS, this is a feature request!!): the LLM is somehow dragged away from this data, in favour of data returned by the MCP server, which it is instructed to trust.
  • When asked to give a full list of available condition keys, without the MCP server Claude finds only 26 condition keys, and with the server it is able to list 38 of them, categorized by the resource (ec2 instance, volume or security group).
  • For last question (Can ec2:VolumeID be used when calling ec2:ModifyInstanceAttribute ?), without the MCP server Claude gives a plainly wrong answer, while the LLM on stamina with the MCP server will give a right answer! (Don't trust me on it, check on the reference page for the EC2 service)

Answer grounded on current data

Let's try something else.

What are the conditions key for bedrock:Rerank API?

Yes, there's a trick here. Rerank is quite a new API. We have to give Claude some credit for its honest answer.

I'm not familiar with the specific conditions or key requirements for the bedrock API. This appears to be related to Amazon Bedrock's retrieval and ranking capabilities, but my knowledge cutoff is October 2024, and I don't have detailed information about this specific API.

Since my most valuable (and painful) lesson learnt from Engineering school is the ability to say a plain and simple "I don't know, let me check", I'm grateful that Claude doesn't make anything up here.

With the MCP server, Claude is now able to come up with a pretty good answer.

One more thing..

In my tests, I noticed that, unless the prompt warns 'him' not to do so, Claude's "desire" to be thorough can lead 'him' to perform "N+1" queries.

In programming, N+1 queries are a bad behaviour / design where the application will not perform a join to eagerly fetch "library + books details", for instance, but will perform one call to get the list of books, then one call per book to get the book details.

When you pay on the input/output token basis, N+1 can quickly become not only a performance but also a cost issue.

Just like bad developers who don't have a good command of their ORM library, unconscious AI engineers will easily get trapped into N+1!

Top comments (0)