DEV Community: Peter Koech

Running Our AWS Deployments From Slack With Amazon Q (and the Things Nobody Tells You)

Peter Koech — Fri, 11 Sep 2026 09:47:33 +0000

For the last couple of years, most of the day-to-day deployment work on one of the platforms I look after has happened inside a single private Slack channel. Nobody opens the CodePipeline console to kick off a staging build. Nobody SSHs anywhere to restart a container. A developer types something like this and gets on with their day:

@Amazon Q run start-pipeline-dev2
@Amazon Q run switch-branch-dev1 feature/my-branch
@Amazon Q run cont-restart api-cluster api-stg5

The service behind this is Amazon Q Developer in chat applications, which most of us still call by its older name, AWS Chatbot. (AWS has renamed it more than once. The console currently lists it under "ChatOps for AWS". I'll mostly say Chatbot here because that's what everyone on the team says.)

I recently wrote an internal manual on how our setup actually works, because too much of it lived in my head and in one colleague's Slack history. This post is a cleaned-up version of that document with names and identifiers replaced. If you're thinking about ChatOps on AWS, or you inherited a Chatbot setup and want to understand it, this should save you some digging.

What it actually is

Chatbot is a thin, authenticated proxy between a Slack channel and the AWS SDK. That's the whole mental model. It takes a message, resolves it to an API call, checks whether the call is allowed, makes the call under an IAM role, and posts results back. There is no agent running in your account, no secrets stored in Slack, no database of commands.

Two independent flows are worth keeping separate in your head:

Commands are synchronous. Slack → Chatbot → AWS service. You type it, it runs.

Notifications are asynchronous. AWS service → SNS topic → Chatbot → Slack. Pipeline stage changes, CloudWatch alarms and so on get published to an SNS topic that the channel configuration is subscribed to. Chatbot formats the payload and drops it in the channel.

Once you internalise that the second path is just an SNS subscription, a lot of troubleshooting gets easier. When notifications stop arriving, it's almost always the subscription or the topic, not Chatbot itself.

The message pipeline, step by step

When someone mentions the bot in the channel, this happens:

Slack forwards the message event to the Chatbot endpoint via the registered Slack App.
Chatbot looks up the alias. start-pipeline-prod expands to something like codepipeline start-pipeline-execution --name myapp-prod --region us-east-1.
Chatbot evaluates the action against both the IAM role it runs under and the guardrail policies attached to the channel. Fail either one and you get an access denied message in Slack.
The SDK call is made directly against the service. For standard commands there's no Lambda in the middle.
State changes flow back through SNS as described above. One thing that surprised a few people on the team: Chatbot doesn't keep command history. If you need to know who stopped the staging pipeline on Tuesday, the answer is in CloudTrail, filtered by the Chatbot role. Slack's own history helps too, but CloudTrail is the record.

Two permission layers, and why you want both

This is the part I'd read twice if you're setting this up.

The IAM role is the execution boundary. Chatbot assumes it for every API call. It defines the ceiling of everything that could possibly happen through any channel using that role. Ours was created from the "policies from a template" option in the console and then trimmed.

The guardrail policies are the channel boundary. They're attached to a specific channel configuration, and they only ever restrict. They never grant. The effective permission set is the intersection of the two.

Why bother with the second layer? Because you can share one IAM role across several channels and give each channel a different effective scope. Our production channel and a wider engineering channel can point at the same role while only one of them is allowed to touch prod pipelines. It also means a Slack user who gets added to the wrong channel by accident can't do anything the guardrail doesn't already allow for that channel.

The cost is that "access denied" now has two possible causes. When it happens, go to CloudTrail, find the denied action, and check which layer is missing it. Don't guess.

Aliases are what make it usable

Chatbot accepts raw SDK commands:

@Amazon Q codepipeline start-pipeline-execution --name myapp-prod --region eu-west-1

Nobody wants to type that. Aliases map a short name to the full command, and you can pass parameters through at runtime. @Amazon Q alias list shows everything registered, with the expansions.

When a single SDK call isn't enough: Lambda aliases

Some things can't be expressed as one API call. Branch switching is the obvious example. To repoint a pipeline's source stage at a different branch you have to GetPipeline, edit the source stage in the returned definition, and UpdatePipeline with the modified document. That's a small Lambda function.

Chatbot handles this fine. The alias just becomes lambda invoke --function-name branch-switch-fn, and the function does the multi-step work and returns a message.

The detect-changes family works the same way. It flips PollForSourceChanges on the source stage so a pipeline stops auto-triggering on push. We use it when a staging environment needs to be frozen for QA while people keep merging into the branch. Set it to false, test, set it back to true.

Wiring an existing Lambda into Chatbot

This bit trips people up, so here's the order that works.

Fix the Lambda's resource-based policy: The function has to trust the Chatbot service principal explicitly. Having lambda:InvokeFunction on the IAM role is not enough.

aws lambda get-policy --function-name FUNCTION_NAME --region eu-west-1

If chatbot.amazonaws.com isn't in there:

aws lambda add-permission \
  --function-name FUNCTION_NAME \
  --statement-id AllowChatbot \
  --action lambda:InvokeFunction \
  --principal chatbot.amazonaws.com \
  --region eu-west-1

2. Update the guardrail: In the channel configuration, make sure the guardrail policy includes lambda:InvokeFunction scoped to the function ARN. Without this, the role can permit it all day and Chatbot will still refuse.

*3. Register the alias in Slack:
*

@Amazon Q alias add switch-branch-dev1 lambda invoke --function-name branch-switch-fn --region eu-west-1

4. Test it and check the CloudWatch log group: for the function to confirm the invocation actually arrived. If the command fails silently, it's step 1 nine times out of ten.

Things that have bitten us

I'll finish with the troubleshooting table from the manual, because it's the part I look at most.

Bot doesn't respond at all: The Slack App authorisation expired or the OAuth token was revoked. Reconnect the Slack client in the Chatbot console.
Access denied: Check CloudTrail for the exact denied action, then figure out which of the two permission layers is missing it.
Pipeline notifications stopped: Someone deleted the SNS subscription or recreated the topic. Re-subscribe the channel configuration.
get-logs gives an expired link: The pre-signed S3 URL lives for ten minutes. Our failure notifications include a pre-filled get-logs command for exactly this reason. Run it as soon as you see the red message.
Lambda alias fails silently: Resource-based policy is missing the Chatbot principal. See above.
Branch switch does nothing: Usually a typo in the branch name, occasionally an expired GitHub token inside the function. The Lambda logs will tell you.
detect-changes false didn't stop auto-triggering: Confirm with GetPipeline that the flag actually changed. Run it again and check with pipeline-status.

Was it worth it?

Yes, with one caveat. The setup took an afternoon. The value came from being disciplined about aliases and guardrails, and from writing down where everything lives, since the Chatbot console, the IAM role, the SNS topic, and the Slack channel ID are spread across four different places and none of them link to each other.

If you run a small team where developers regularly need to deploy to shared environments, this removes a surprising amount of friction and a surprising number of interruptions. Just decide early that CloudTrail is your audit log and the guardrail is your safety net, and you'll be fine.

Happy to answer questions in the comments if you're setting something similar up.

Amazon S3 + Cloudflare Cloud Connector magic

Peter Koech — Sat, 25 Jan 2025 09:06:30 +0000

Securely host your static website in Amazon S3 using Cloudflare's CloudConnector

Peter Koech for AWS Community Builders ・ Jan 25

#aws #s3 #cloudflarechallenge #cloudcomputing

Securely host your static app in Amazon S3 using Cloudflare's CloudConnector

Peter Koech — Sat, 25 Jan 2025 09:00:06 +0000

I am going to demonstrate how to use Amazon S3 to host your objects or a static website and use Cloudflare as a Content Distribution Network (CDN).

Amazon S3 is an industry-leading object storage service offering unrivaled scalability, performance, and availability.

Cloudflare is one of the leading stand-alone Content Distribution Network (CDN), Amazon CloudFront is AWS's service that offers CDN and a host of other powerful features.

Normally users would use Amazon S3 and CloudFront to achieve the demo below, but some organizations have deployed Cloudflare as a dedicated CDN service in their infrastructure

Amazon S3

We will create a simple static website: http://stellar-landing.s3-website-us-east-1.amazonaws.com

It's usually a breeze using AWS CLI or SDK since you only run a couple of commands to get up and running. As you may be aware, every service in AWS can be interacted with through an API call.

So let me demonstrate, how we can set up stellar-landing bucket using AWS CLI.

Create the bucket using AWS CLI

aws s3 mb s3://stellar-landing
Access the console and enable public access for the bucket, allowing you to run the subsequent commands.
On your terminal, create a json file, we can name it "public-read.json" and inside it, you can place this policy which tells the bucket to allow everyone on the internet denoted by principal "*" to GET objects


{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"PublicReadGetObject",
      "Effect":"Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource":"arn:aws:s3:::stellar-landing/*"
    }
  ]
}

We need to run a put-bucket-policy that updates the bucket's policy to allow for accessibility to the public.

aws s3api put-bucket-policy --bucket stellar-landing --policy file://public-read.json
To configure the bucket for static web hosting, you can use the put-bucket-website command:

aws s3 website s3://stellar-landing/ --index-document index.html --error-document error.html

As you can see, AWS CLI only has three commands to get a static site hosted. I can demonstrate how to do the same via the Console (point and click)

Create the bucket using AWS Console

Log into the console, then search for Amazon S3, click on it and the click on Create Bucket. Give it a name and leave the other options as is.

Go to the permissions tab, then uncheck the checkbox for block all public access

Attach the bucket policy to enable GetObjet requests for all users (principals)

Go to the properties tab; you will need to make two key change, under Static website hosting select on Enable then under Hosting type select Host a static website

Afterwards, you'll notice an endpoint provisioned for your public bucket, it follows this pattern; the name of the bucket stellar-landing followed by s3-website-us-east-1.amazonaws.com thus the final link Link

CloudFlare CDN

Now that we have our bucket up and running, before Cloudflare's connector service, it was nearly impossible to use CloudFlare as a CDN for Amazon S3 buckets to be hosted in CloudFlare, most methods were too complex as detailed in this link by Jaroslav Link

Then came Cloud Connector Link from CloudFlare, a service that enables us to route matching incoming traffic from our static website hosted in Amazon S3 bucket Link.

Steps to enable and configure Cloud Connector

Step 1

Step 2

Enter your bucket URL where your traffic can be directed to the URL via CNAME, select your bucket URL from your Amazon S3 console, it should be stellar-landing.s3-website-us-east-1.amazonaws.com

Step 3

Configure your Cloud Connector, give it a name such as stellar-landing-cdn under If incoming requests match Select custom filter expression
Under When incoming requests match; Select Field as URI Full, then under Operator select Contains, then for Value enter your preferred friendly URL that you wish your bucket to be referred to e.g https://stellar-landing.nation.africa

** Step 4**

Still in Cloudflare under DNS menu on the left, select Records
Then click on Add record
For **Type* select CNAME, then under Name provide an ideal name for your Amazon S3 hosted website e.g stellar-landing and finally under Target, enter your S3 bucket endpoint stellar-landing.s3-website-us-east-1.amazonaws.com
To enable SSL for your new endpoint on Cloudflare, click the Proxy status toggle; it should change to orange. This indicates that requests will now be proxied securely through CloudFlare.

You can now securely access your statically hosted website via this link Link i.e https://stellar-landing.nation.africa as opposed to non secure Amazon S3 endpoint: http://stellar-landing.s3-website-us-east-1.amazonaws.com/

PS: The template for the static website is owned by @ajlkn Link

Scaling down the storage of a MySQL RDS database with zero downtime using AWS Blue/Green Deployment

Peter Koech — Tue, 21 Jan 2025 21:29:24 +0000

As a DevOps engineer, you will find yourself optimizing and tinkering the RDS databases powering your workloads, you probably know the struggle with updating databases especially if you have heavy workloads depending on them. The risk of experiencing downtime during production is high. Traditional database management often leads to a catch-22 situation where you want to update your database for better performance but fear database downtime.

Well, worry no more! In this article, we will explore the AWS Blue/Green Deployment method that allows you to change your Relational Database Service (RDS) with zero downtime. Yes, you read that right: ZERO downtime, and it works like magic!

From explaining the Blue/Green Deployment strategy to providing a step-by-step guide on how you can utilize this in AWS, this article covers everything you need to know when updating your RDS database instance without compromising availability.

So, let's dive straight in!

Preparation

You need to identify a time when your dev team will not be performing any releases and also an ideal time when you have the least traffic on your workloads.

It's a good DevOps practice to notify the larger team of what time you schedule to perform such changes.

I will demonstrate downscaling DB's instance storage volume in this case while using Blue/Green deployment with zero downtime.

The same exercise can be achieved using either

AWS RDS Console
AWS CLI
AWS SDK

For our exercise, we will use the AWS RDS Console
NB Before performing Blue/Green deployments for either a MySQL or Mariad-based RDS instance, please enable automated backups link

Step 1

For our demo, we will scale the storage space from 400 GB to 200 GB. Our database cluster is made up of a master and 3 replicas as shown below.

Select the master database from the cluster
Click on actions then click on Create Blue/Green Deployment from the drop-down menu to open the "Blue" screen

Step 2

We need to create a copy of the production environment, here referred to as the Blue environment to a staging environment referred to as Green

Give your Blue/Green deployment an identifier name, remember the identifier is case-insensitive, but is stored as all lowercase.

Step 3

The next step is configuring the Blue/Green deployment by specifying the ideal database engine and the parameter group for your Green environment.

For our case, since we are interested in downsizing the storage size, we will use the same engine version and parameter group as the Blue environment (Production DB)

Step 4

This step entails configuring your Green instance database type; for our case, we will use use the same instance as Blue.

Step 5

Our main task is focussing on downsizing the allocated storage for the Green deployment.

The configuration update on AWS console will be done on one screen and will be replicated across the entire database cluster.

Remember we are scaling down storage for the master database and it's 3 replicas as shown in the figures below.

Side-by-side view of the Blue and Green master Deployments highlighting the updates.

Side-by-side view of the Blue/Green database replicas highlighting the changes made

Once you confirm all is looking good, a few things to note at this stage is that the cost of the Green deployment will be shown before you hit the create button.

Blue/Green will take anything between a few minutes to hours, this is dictated by factors such as the size of the database, how busy the workloads running off the database are, and the number of replicas.

For our case, it took about 5 hours for the Green deployment to be ready and it worked like magic.

Zero downtime, only a bit of lag on the Primary cluster (Blue) environment.

A lot of the magic happens under the hood, AWS handles renaming the Green deployment's DB cluster naming convention to match Blue environment, including updating the Endpoints for the Master and Replicas.

You won't need to update your application's configuration, the most I did is notifying the dev team via Slack about this exercise.