DEV Community

AWS Community Builders profile image Jérôme GUYON
Jérôme GUYON for AWS Community Builders

Posted on

Build your own AWS CLI service (yes, really)

#awscli

I keep building internal tools and small utilities for my teams, and every single time, I face the same frustrating questions: if I ship a custom tool, I'm asking people to use something new, with a new set of credentials, and to read documentation they'll inevitably ignore.

That's a lot of possible friction for tools that are supposed to reduce friction 🙃.

Here's the thing that changed my approach: every developer or ops engineer I'm working with already spends their day in the AWS CLI.

They already have working credentials configured, they are fluent in --profile for switching between accounts, --query for JMESPath filtering, ...

What I could plug my internal APIs directly into the tool they already use every day?

Turns out botocore, the library underneath both the AWS CLI and boto3, is entirely model-driven : Every single AWS service is defined as a JSON file that describes its operations, request and response shapes, and botocore generates the client class at runtime.

And you can provide your own JSON model 💪 !

I built a working example to demonstrate the pattern from end to end that wraps the public API (the AWS Community directory of Heroes and Community Builders) into what looks and feels like a native AWS CLI service.

The underlying technique applies to any REST API you want to expose.

aws aws-community list-community-members --program HERO --category "Serverless Hero" --output table
Enter fullscreen mode Exit fullscreen mode

SigV4 authentication, standard CLI flags, table/JSON/YAML output, JMESPath queries, profile switching. All of this works with zero client code written on my side.

The pattern in four pieces. The architecture looks like this:

AWS CLI / boto3  →  API Gateway (IAM auth)  →  Lambda  →  your backend
Enter fullscreen mode Exit fullscreen mode
  1. API Gateway with AWS_IAM authorization handles authentication for you. Every incoming request must carry a valid SigV4 signature, which means your team authenticates with the same IAM credentials they already use for every other AWS operation.
  2. A Lambda function (or any API Gateway integration) receives the request body and does the actual work. In my demo it calls an external API, but yours could query DynamoDB, read from RDS, hit an internal microservice, or compute something on the fly.
  3. A service model (service-2.json) describes your operations, their inputs, their outputs, and the data types involved. This is how botocore knows what CLI commands to expose, what parameters each command accepts, and how to serialize the request over the wire.
  4. An endpoints file (endpoints.json) tells botocore which hostname to send requests to for each region. Without this file, users would have to pass --endpoint-url manually on every invocation.

Let me show you how to build one.

Once you understand the structure, adapting it to your own API is mostly a matter of replacing the Lambda logic and renaming operations in the service model.

Step 1: Deploy the infrastructure. Pick whichever region you prefer. The project uses a single CloudFormation template with an inline Lambda for the demo.

aws cloudformation deploy \
  --template-file cloudformation/template.yaml \
  --stack-name AWSCommunityService \
  --capabilities CAPABILITY_IAM
Enter fullscreen mode Exit fullscreen mode

Step 2: Retrieve the full API Gateway endpoint from the stack outputs.

API_ENDPOINT=$(aws cloudformation describe-stacks \
  --stack-name AWSCommunityService \
  --query 'Stacks[0].Outputs[?OutputKey==`Endpoint`].OutputValue' \
  --output text)  

echo $API_ENDPOINT
# https://a1b2c3d4e5.execute-api.eu-west-1.amazonaws.com/Prod
Enter fullscreen mode Exit fullscreen mode

Step 3: Inject the hostname into your endpoints file so botocore knows where to route requests.

# Extract just the hostname from the full URL  
API_HOST=$(echo $API_ENDPOINT | sed 's|https://||' | cut -d'/' -f1)  

sed -i.bak "s/REPLACE_WITH_ENDPOINT/$API_HOST/" models/endpoints.json && rm models/endpoints.json.bak
Enter fullscreen mode Exit fullscreen mode

Step 4: Set the AWS_DATA_PATH environment variable to tell botocore where your model files live.

export AWS_DATA_PATH=./models/
Enter fullscreen mode Exit fullscreen mode

Step 5: Try it out.

$ aws aws-community list-programs
{
    "Programs": [
        {
            "ProgramName": "COMMUNITY_BUILDER",
            "Categories": ["AI Engineering", "Cloud Operations", "Data", ...]
        },
        {
            "ProgramName": "HERO",
            "Categories": ["AI Hero", "Community Hero", "Container Hero", ...]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode 
$ aws aws-community list-community-members \
    --program HERO --max-results 5 \
    --output table \
    --query 'CommunityMembers[].{Name:Name,Category:Category,Country:Country}'

------------------------------------------------------------
|                   ListCommunityMembers                   |
+------------------+----------+----------------------------+
|     Category     | Country  |           Name             |
+------------------+----------+----------------------------+
|  Serverless Hero |  CA      |  Darryl Ruggles            |
|  AI Hero         |  AR      |  Ricardo Ceci              |
|  AI Hero         |  AR      |  Matias Kreder [AWS Hero]  |
|  AI Hero         |  IT      |  Damiano Giorgi            |
|  Serverless Hero |  SE      |  Gunnar Grosch             |
+------------------+----------+----------------------------+
Enter fullscreen mode Exit fullscreen mode

Et voilà! The --output table formatting, the --query JMESPath expression, the --next-token pagination mechanism, they all work exactly as they would with any official AWS service.

It works in Python too. The same model file that powers the CLI also gives you a fully functional boto3 client:

import boto3  

acs = boto3.client("aws-community")  

response = acs.list_community_members(Program="HERO", MaxResults=10)  
for member in response["CommunityMembers"]:  
    print(f"{member['Name']} - {member['Category']}")
Enter fullscreen mode Exit fullscreen mode

You get input validation against the model, the Stubber for mocking in tests, and pagination support through NextToken.

How to write the service model. This is the heart of the pattern. The service-2.json file contains everything botocore needs to generate your client. Here's a minimal example showing a single operation:

{
  "metadata": {
    "protocol": "rest-json",
    "signingName": "execute-api",
    "endpointPrefix": "my-service"
  },
  "operations": {
    "GetThing": {
      "http": { "method": "POST", "requestUri": "/Prod/GetThing" },
      "input": { "shape": "GetThingRequest" },
      "output": { "shape": "GetThingResponse" }
    }
  },
  "shapes": {
    "GetThingRequest": {
      "type": "structure",
      "required": ["ThingId"],
      "members": {
        "ThingId": { "shape": "String" }
      }
    },
    "GetThingResponse": {
      "type": "structure",
      "members": {
        "Name": { "shape": "String" },
        "Status": { "shape": "String" }
      }
    },
    "String": { "type": "string" }
  }
}
Enter fullscreen mode Exit fullscreen mode

Three metadata fields require your attention. The protocol field should be "rest-json" for a typical JSON API behind API Gateway. The signingName must be "execute-api" so that SigV4 signs the request with the correct service scope for API Gateway (change this only if you're running a custom authorizer). The endpointPrefix is the name botocore uses both to look up the hostname in endpoints.json and to determine the CLI command name, so "my-service" means your users will type aws my-service get-thing.

The shapes section defines your data structures using the same primitives that every official AWS service uses: structure for objects, list for arrays, string, integer, boolean, and timestamp for scalar types. You can add min/max constraints on integers, pattern on strings, and mark fields as required. If you want to see how a complex model looks, browse any service under botocore/data/ in the botocore GitHub repository, there are literally hundreds of production examples to learn from.

How to write the endpoints file. This tells botocore which hostname corresponds to your service in each region:

{
  "version": 3,
  "partitions": [{
    "defaults": {
      "hostname": "{service}.{region}.{dnsSuffix}",
      "protocols": ["https"],
      "signatureVersions": ["v4"]
    },
    "dnsSuffix": "amazonaws.com",
    "partition": "my-partition",
    "partitionName": "My Service",
    "regionRegex": "^(us|eu|ap)\\-\\w+\\-\\d+$",
    "regions": {
      "eu-west-1": { "description": "EU (Ireland)" }
    },
    "services": {
      "my-service": {
        "endpoints": {
          "eu-west-1": {
            "hostname": "YOUR_API_ID.execute-api.eu-west-1.amazonaws.com"
          }
        }
      }
    }
  }]
}
Enter fullscreen mode Exit fullscreen mode

If you deploy your API Gateway in multiple regions, add an entry for each region under the endpoints object. Each entry points to a different API Gateway deployment, giving you region-aware routing with no client-side logic.

Why AWS_DATA_PATH instead of aws configure add-model? The add-model command is tempting because it feels official, but it only copies service-2.json to ~/.aws/models/. It completely ignores endpoints.json, and botocore only reads custom endpoint definitions from directories listed in AWS_DATA_PATH. So if you use add-model alone, your users have to append --endpoint-url https://xxxxx.execute-api.eu-west-1.amazonaws.com to every command.

Things to know:

  • Access control through IAM. Anyone calling your service needs execute-api:Invoke permission on the API Gateway resource ARN. You manage access with the same IAM policies, roles, and permission boundaries you already use for every other AWS service in your organization.
  • No shell tab completion. Custom services don't register themselves with the CLI's auto-completer. Your users can run aws my-service help to discover available operations, but pressing Tab after aws my-service won't suggest anything.
  • The model format is technically undocumented. AWS has never published a formal specification for service-2.json. In practice, the format has remained stable for years and the hundreds of models bundled with botocore provide more than enough reference material to work from confidently.
  • Pagination requires an extra file. Botocore's built-in auto-paginator (the one behind --page-size and automatic result aggregation) requires a separate paginators-1.json model file. Without it, your users paginate manually by passing --next-token from one call to the next.
  • Onboarding a new user takes 30 seconds. Share the models/ folder (via git), have the user set one environment variable, and they're ready to go.
  • The full project is on GitHub.

— Jerome

Top comments (1)

Collapse
 
dvddpl profile image
Davide de Paolis AWS Community Builders •

That's a lot of possible friction for tools that are supposed to reduce friction

absolutely! and this is where a lot of platform teams fail.

very smart solution and interesting read. thanks for sharing