DEV Community: Engin ALTAY

Complete Guide - Deploying Production-Ready MongoDB Replica Set on AWS

Engin ALTAY — Tue, 21 Jan 2025 13:33:38 +0000

MongoDB is a leading NoSQL database trusted by organizations for its flexibility, scalability, and high availability. This guide walks you through deploying a production-ready MongoDB replica set on AWS using a self-hosted, Dockerized setup. By leveraging AWS’s robust infrastructure, such as EC2 instances, paired with cost-saving strategies like Savings Plans or Reserved Instances, you can achieve a highly available and cost-optimized database solution.

We’ll also compare this self-hosted approach to managed services like MongoDB Atlas, showcasing how you can significantly reduce costs while retaining full control over your database environment. To ensure robust security, we’ll enable authentication and implement proper role-based authorization, safeguarding your data against unauthorized access.

Whether you’re a DevOps professional or a database administrator, this guide provides practical insights and detailed steps for deploying, securing, and optimizing MongoDB replica sets in an AWS environment.

Preparation Phase: Setting Up MongoDB Replica Set Deployment on AWS

To ensure a smooth deployment of your MongoDB replica set, we will use the latest stable Debian release, Bookworm, for compatibility with the latest MongoDB version. Additionally, the Docker engine must be installed and operational. Proper internal hostnames must also be assigned to your EC2 instances for seamless replica set initialization.

Step 1 - Verify the Server Environment

Kernel Information:

uname -a

Ensure the system is running the Debian Bookworm kernel.

Docker Installation:

docker info

Confirm that Docker is installed and functioning correctly.

Resource Availability:

htop

Check the system's resource usage, including CPU and memory, to ensure adequate capacity.

Disk Space:

df -h

Verify that sufficient disk space is available for MongoDB data storage.

Step 2 (Optional) - Expand Filesystem

We'll use persistent EBS volume attached to EC2 instances to store MongoDB data. It'll be easier to manage separate EBS volume to backup and restore in case of need.

If you’ve increased the size of a persistent disk attached to your EC2 instance, grow the filesystem to utilize the expanded space:

xfs_growfs /dev/xvdp

Step 3 - Assign Internal Hostnames

Internal hostnames are critical for MongoDB replica set configuration. Assign a unique, descriptive hostname to each machine:

hostnamectl set-hostname <hostname>.internal.company.net

Replace "hostname" with an appropriate identifier for each instance, such as mongo1, mongo2, and mongo3.

Overview of Authentication-Enabled MongoDB Replica Set

Setting up an authentication-enabled MongoDB replica set involves securing both internal communication between replica set members and external client connections. This is achieved through the following configurations:

Internal Authentication:
Members of the replica set use a keyfile for secure communication. This ensures that only trusted nodes can join and exchange data within the replica set.
Client Authentication with Role-Based Access Control (RBAC):
External clients, such as the MongoDB shell or applications, must authenticate using valid credentials. Access is managed through RBAC, which assigns specific roles and permissions to each user.

This dual-layered security setup ensures robust protection for your MongoDB replica set, safeguarding both internal operations and client interactions.

Deploying MongoDB Replica Set with Keyfile Access Control

Setting up a MongoDB replica set with keyfile-based authentication involves generating a secure keyfile, preparing Docker volumes to persist data and configurations, and setting appropriate file permissions. Additionally, a dedicated MongoDB user must be created for managing the necessary files securely.

Step 1 - Generate the Keyfile

Keyfile authentication ensures secure communication between replica set members. Each mongod instance uses the keyfile as a shared password to authenticate with other members. Only nodes with the correct keyfile can join the replica set.

To generate a secure keyfile:

openssl rand -base64 756 > prod-mongo.pem
chmod 400 prod-mongo.pem

openssl rand -base64 756: Generates a 756-byte pseudo-random string encoded in Base64, creating a secure keyfile.
chmod 400: Restricts file access to the owner.

Step 2 - Create MongoDB Linux User and Group

On each EC2 instance, create a dedicated Linux user and group with consistent UID and GID for MongoDB:

groupadd -r mongodb && useradd -r -u 999 -g mongodb mongodb

This ensures MongoDB processes, running with UID 999 in Docker containers, have proper access permissions.

Step 3 - Copy the Keyfile to All Replica Set Members

Distribute the prod-mongo.pem keyfile to all servers in the replica set. Ensure file permissions and ownership are correctly set:

chmod 400 prod-mongo.pem  
chown 999:999 prod-mongo.pem

It is required to store the keyfile in the mongo-key Docker volume for use by the containers.

Step 4 - Create Docker Volumes for Persistent Storage

Prepare Docker volumes to persistently store MongoDB data, configuration files, logs, and the keyfile for each container:

docker volume create mongo-data  
docker volume create mongo-config  
docker volume create mongo-log  
docker volume create mongo-key

Copy the prod-mongo.pem keyfile into the mongo-key volume.

Step 5 - Configure Logging

Alongside security measures, effective logging is essential for monitoring and troubleshooting. Create a mongod.conf file and store it in the mongo-config Docker volume at /var/lib/docker/volumes/mongo-config/.

# mongod.conf

storage:  
  dbPath: /data/db  

systemLog:  
  destination: file  
  logAppend: true  
  path: /var/log/mongodb/mongod.log  

net:  
  port: 27017  
  bindIp: 0.0.0.0  

processManagement:  
  timeZoneInfo: /usr/share/zoneinfo

Step 6 - Start MongoDB Containers

Start each replica set member as a Docker container, ensuring the --auth and --keyFile options are enabled for secure access and communication.

Run the following command on each EC2 instance:

docker run \
--name aws-mongodb-prod \
-h aws-mongodb-prod \
--restart unless-stopped \
-v mongo-data:/data/db \
-v mongo-config:/etc/mongo-config \
-v mongo-key:/opt \
-v mongo-log:/var/log/mongodb \
-p 27017:27017 \
-d mongo:8.0.4 -f /etc/mongo-config/mongod.conf --keyFile /opt/prod-mongokey.pem --replSet awsProdRepl --auth

Initiating the MongoDB Replica Set

After deploying MongoDB as standalone instances using Docker, the replica set is not yet active because no replica set configuration has been provided. To enable high availability and replication, you must configure and initiate the replica set.

Step 1 - Access the MongoDB Shell

First, exec into the MongoDB container to access the mongo shell:

docker exec -it aws-mongodb-prod bash

Once inside the container, start the MongoDB shell:

mongosh

Step 2 - Initiate the Replica Set

Now that you have access to the mongo shell, you can initiate the replica set by passing the appropriate configuration. Replace "hostname" with the actual internal hostnames of your replica set members:

rs.initiate({  
   _id: "awsProdRepl",  
   members: [  
      { _id: 0, host: "<hostname>:27017" },  
      { _id: 1, host: "<hostname>:27017" },  
      { _id: 2, host: "<hostname>:27017" }  
   ]  
})

If successful, the output should include:

{ "ok" : 1 }

Important Notes:

Replica Set Name: The _id field in the configuration must match the replica set name you provided in the Docker --replSet argument (e.g., awsProdRepl).
Hostnames: Ensure the host field contains the correct hostnames of each replica set member, along with the MongoDB port (27017).
Ordering: The _id values in the members array define the priority of the nodes. Typically, the primary node has the lowest _id.

Step 3 - Verify the Replica Set

After initiating the replica set, you can verify its status using the following command in the mongo shell:

rs.status()

This will display details about the replica set, including the state of each member.

🎉 Congratulations!
You have successfully deployed and configured a MongoDB replica set with authentication and keyfile-based access control. Your MongoDB deployment is now highly available and ready for production.

Post-Deployment Actions: Creating Privileged & Restricted Users

After successfully initiating the MongoDB replica set with keyfile-based internal authentication, the next step is to implement user access control. This ensures secure access to your MongoDB deployment, especially for clients like the Mongo shell or application APIs. Below are the steps to create both privileged and restricted users.

Step 1 - Access Mongo Shell

Start by accessing the Mongo shell from within the Docker container:

docker exec -it aws-mongodb-prod bash  
mongosh

Step 2 - Create a Privileged MongoDB User

For managing MongoDB cluster operations, you'll need a privileged user.

To create a privileged user, switch to the admin database and execute the following command:

use admin  

db.createUser(  
  {  
    user: "<user>",  
    pwd: "<password>",  
    roles: [ { role: "root", db: "admin" } ]  
  }  
)

Replace and with the desired username and password. This user will now have full administrative privileges on the MongoDB cluster

Step 3 - Create a Restricted User for Applications and APIs

Next, create a restricted user for use by applications or APIs that need limited access. In this example, the user will have read and write access only to the user_events database.

use user_events  

db.createUser(  
  {  
    user: "prodUser",  
    pwd: "<password>",  
    roles: [ { role: "readWrite", db: "user_events" } ]  
  }  
)

Replace with the desired password for the prodUser. This user will only have read and write access to the user_events database.

Step 4 - Verify User Access

To verify that the prodUser has the correct access, authenticate using the newly created user:

use user_events  
db.auth("prodUser", "<password>")

If successful, you should see the output:

🎉 Congratulations!
You have successfully created the prodUser with restricted access and the privileged user with full administrative rights. Your MongoDB deployment now has proper user access controls, ensuring secure operations and access for different use cases.

Conclusion

Deploying a production-ready MongoDB replica set on AWS using Docker provides a robust, secure, and cost-effective solution compared to managed database services like MongoDB Atlas or AWS DocumentDB. By leveraging AWS EC2 instances and keyfile-based authentication, you achieve high availability with granular control over the deployment. This approach ensures both internal security between replica set members and external security for client connections using role-based access control.

With thoughtful configuration, including persistent Docker volumes, optimized logging, and properly defined user roles, the solution is tailored to meet enterprise requirements. The use of AWS Savings Plans or Reserved Instances further reduces operational costs, offering significant savings over managed database services while maintaining full control over your infrastructure.

This deployment model combines flexibility, performance, and security, making it an excellent choice for businesses looking to balance cost efficiency and operational excellence in their database infrastructure. By following this guide, you now have the tools to set up a scalable, secure, and cost-optimized MongoDB replica set on AWS.

References:

Powering AWS Fargate with IaC - AWS CloudFormation

Engin ALTAY — Mon, 11 Mar 2024 09:01:56 +0000

Today's tech world, there's a clear truth that your organization needs to be agile as well as your workloads need to run smooth. Especially in containers area, there are plenty of methods to deploy your containerized workloads to your environment.

In this post, I'd like to mention powering AWS Fargate - a serverless compute that run containers without needing to manage your infrastructure, with Infrastructure as Code AWS CloudFormation - to provision your fargate workloads, including load balancing and rolling-update deployment features.

I assume that you already have that in use or familiar with tech stack that I mentioned below.

Imagine the following case:

You have created your ECS cluster with Fargate option,
You already have provisioned internet-facing ELB,
You already have VPC, subnets and security group for your application.

But in the continuation, you need to:

Provision your AWS Fargate workloads,
Attach security group,
Expose as a ECS service,
Associate with ELB, create your ELB listener routing rule and more.

All these steps become unmanageable and tedious after number of your application, environment (dev,test,staging,prod) and workload grows.

To make this agile and automated, we'll leverage AWS CloudFormation, combining with GitLab CI/CD.

Step 1 - Building our CloudFormation template

Using AWS CloudFormation to provision and update our resources in AWS environment helps us in a way to centralize and track our each change.

We need to define each resource definitions to our CloudFormation template. This is the core component we'll work on it.

deploy-fargate.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: An example CloudFormation template for Fargate.
Parameters:
  VPC:
    Type: String
    Default: <VPC_ID_HERE>
  SubnetPublicA:
    Type: String
    Default: <PUBLIC_SUBNET_A>
  SubnetPublicB:
    Type: String
    Default: <PUBLIC_SUBNET_B>
  SubnetPublicC:
    Type: String
    Default: <PUBLIC_SUBNET_C>
  Image:
    Type: String
    Default: <ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/nginx:latest
  ClusterName:
    Type: String
    Description: ECS_CLUSTER_NAME here
    Default: <ECS_CLUSTER_NAME>   
  ServiceName:
    Type: String
    Description: ECS_SERVICE_NAME here
    Default: "API_NAME-prod-svc"
  TaskDefinitionName: 
    Type: String
    Description: Task Definition Name
    Default: "API_NAME-prod-fargate"
  ContainerPort:
    Type: Number
    Default: 3000
  ContainerSecurityGroup:
    Type: String
    Description: api-container-sec-rules    
    Default: <SECURITY_GROUP_ID>
  ELBListenerArn:
    Type: String
    Default: <ELB_LISTENER_ARN>


Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      # Name of the task definition.
      Family: !Ref TaskDefinitionName
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      # 1024 (1 vCPU) - Available memory values: 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB
      Cpu: 1024
      # 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB - Available cpu values: 1024 (1 vCPU)
      Memory: 3GB
      # "The ARN of the task execution role that containers in this task can assume. All containers in this task are granted the permissions that are specified in this role."
      ExecutionRoleArn: !GetAtt ExecutionRole.Arn
      # "The (ARN) of an IAM role that grants containers in the task permission to call AWS APIs on your behalf."
      TaskRoleArn: !Ref TaskRole
      ContainerDefinitions:
        - Name: API_NAME
          Image: !Ref Image
          Cpu: 0
          Essential: true
          PortMappings:
            - ContainerPort: !Ref ContainerPort
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: ecs              

  LogGroup:
    Type: AWS::Logs::LogGroup   
    Properties:
      LogGroupName: !Join  ['', [/ecs/, !Ref TaskDefinitionName]]
      RetentionInDays: 14

  # A role needed by ECS
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ['', [!Ref ServiceName, "ECSExecutionRole"]]
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
        -'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'

  # A role for the containers
  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ['', [!Ref ServiceName, "ECSTaskRole"]]
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
        - 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'


  Service:
    Type: AWS::ECS::Service
    DependsOn:
      - LoadBalancerListenerRule    
    Properties: 
      ServiceName: !Ref ServiceName
      Cluster: !Ref ClusterName
      TaskDefinition: !Ref TaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: 1
      HealthCheckGracePeriodSeconds: 120
      CapacityProviderStrategy:
        - CapacityProvider: FARGATE_SPOT
          Base: 0
          Weight: 1      
      NetworkConfiguration: 
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          Subnets:
            - !Ref SubnetPublicA
            - !Ref SubnetPublicB
            - !Ref SubnetPublicC
          SecurityGroups:
            - !Ref ContainerSecurityGroup
      LoadBalancers:
        - ContainerName: API_NAME
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref TargetGroup

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: /API_NAME/health
      HealthCheckTimeoutSeconds: 5
      UnhealthyThresholdCount: 2
      HealthyThresholdCount: 3
      TargetType: ip
      Name: !Ref ServiceName
      Port: !Ref ContainerPort
      Protocol: HTTP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 30  #default 300 seconds
      VpcId: !Ref VPC

  LambdaDescribeELBListenerPriority:
    Type: 'Custom::LambdaDescribeELBListenerPriority'
    Properties:
      ServiceToken: 'arn:aws:lambda:eu-central-1:<ACCOUNT_ID>:function:DescribeELBListener'

  LoadBalancerListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    #DependsOn: GetListenerRulesLambdaFunction
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref TargetGroup
      Conditions:
        - Field: host-header
          HostHeaderConfig:
            Values:
              - "api.example.com"
        - Field: path-pattern
          PathPatternConfig:
            Values:
              - "/API_NAME*"
      ListenerArn: !Ref ELBListenerArn
      Priority: !GetAtt LambdaDescribeELBListenerPriority.NextPriorityValue

Outputs:
  NextPriorityValue:
    Value: !GetAtt LambdaDescribeELBListenerPriority.NextPriorityValue

In this template, a few sections need to be mentioned to clarify the case we are dealing with.

In the Parameters section, provide some constants that we already created before such as VPC Id, Subnets, ECS Cluster Name, Security Group, ELB Listener etc.
You can also create these from scratch but in my case they all have created before.
As a Capacity Provider, FARGATE_SPOT is used, but you can change it to FARGATE as your needs or you can benefit combining both.
As a placeholder, API_NAME is used. Replace API_NAME with application name that needed to be run on AWS Fargate.
We did not declare auto scale actions for Fargate service. Desired count set to 1. Planning to mention auto scale policy for Fargate in the next post.
Declared Custom Resource LambdaDescribeELBListenerPriority which describes the ELB Listener and finds next available priority number to create listener routing rule.

That custom resource is a bit headache, I expect CloudFormation to handle automatically finding next available ELB Listener priority number and put my rule to there. But it does not. It expects you to provide priority number. In a development lifecycle and CI/CD perspective, it's impossible to know which priority number is free and set it before running CloudFormation template. Therefore, we write a simple lambda function that describes ELB Listener, takes max priority number and adds +1 to create available priority number.

I provide the related lambda function below.

DescribeELBListener

import boto3
import json
import urllib3

http = urllib3.PoolManager()
SUCCESS = "SUCCESS"
FAILED = "FAILED"

def lambda_handler(event, context):
    elbv2 = boto3.client('elbv2')

    # Get all listener rules for the provided ARN
    response = elbv2.describe_rules(ListenerArn='<ELB_LISTENER_ARN>')

    # Filter out rules with non-numeric priorities
    filtered_rules = [rule for rule in response['Rules'] if str(rule['Priority']).isdigit()]

    # Find the maximum priority among the remaining rules
    max_priority = max(filtered_rules, key=lambda x: x['Priority'])['Priority']

    # Prepare the CloudFormation (CF) stack event response payload    
    responseValue = int(max_priority) + 1
    responseData = {'NextPriorityValue': responseValue}

    send(event, context, SUCCESS, responseData)

def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None):
    responseUrl = event['ResponseURL']

    responseBody = {
        'Status': responseStatus,
        'Reason': reason or f'See the details in CloudWatch Log Stream: {context.log_stream_name}',
        'PhysicalResourceId': physicalResourceId or context.log_stream_name,
        'StackId': event['StackId'],
        'RequestId': event['RequestId'],
        'LogicalResourceId': event['LogicalResourceId'],
        'NoEcho': noEcho,
        'Data': responseData
    }

    json_responseBody = json.dumps(responseBody, default=str)

    headers = {
        'content-type': '',
        'content-length': str(len(json_responseBody))
    }
    try:
        response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody)
        print("Status code:", response.status)

    except Exception as e:
        print("send(..) failed executing http.request(..):", e)

Step 2 - Prepare CI/CD Pipeline - GitLab

Now we need to prepare CI/CD side to run our CloudFormation template.

Never use your own credentials to access AWS and run CloudFormation template from your local environment.
Here, we use centralized private GitLab to run CloudFormation stack and provided AWS access by following least-privileged permission policy to access resources.
Pay attention to "image": ".dkr.ecr.eu-central-1.amazonaws.com/nginx:latest" section. Latest tag will be replaced with our respectful container image tag during CI/CD job.

Building GitLab CI/CD Pipeline

To automate and run our IaC template, we leverage version control system, so each our iteration is trackable and easy to apply changes.

Below fully ready .gitlab-ci.yml file that includes build & IaC deployment stages.

variables:
  API: nginx
  REGISTRY: "<your_aws_account_number_here>.dkr.ecr.eu-central-1.amazonaws.com"
  ECS_CLUSTER_NAME: "YOUR_ECS_CLUSTER_NAME"
  ECS_SERVICE_NAME: "${API}-prod-svc"
  ECS_TASK_FAMILY: "${API}-prod-fargate"
  CF_STACK_NAME: '${API}-cf-template-${CI_PIPELINE_IID}'  

stages:
  - build
  - deploy
  - update

before_script:
  - echo "Build Name:" "$CI_JOB_NAME"
  - echo "Branch:" "$CI_COMMIT_REF_NAME"
  - echo "Build Stage:" "$CI_JOB_STAGE"


build:
  stage: build
  script:
    - $(aws ecr get-login --no-include-email --region eu-central-1)
    - VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
    - echo $VER    
    - docker build -t ${API} .
    - docker tag ${API} ${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
    - docker push ${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
  environment:
    name: prod


deploy_cloudformation:
  stage: deploy
  when: manual
  image:
    name: amazon/aws-cli:latest
    entrypoint: ['']
  rules:
    - if: $API != "null" && $CI_COMMIT_BRANCH == "master"
  script:
    - echo "Deploying your IaC CloudFormation..."
    - yum install jq -y
    - jq -Version
    - VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
    - echo $VER
    - echo "API Name ----> ${API} <----"
    - echo "ECS FARGATE Cluster is = ${ECS_CLUSTER_NAME}"
    - sed -i 's/API_NAME/'"${API}"'/g' deploy-fargate.yaml #replace API_NAME placeholder with the container that we want to run on AWS Fargate.
    - cat deploy-fargate.yaml
    - |
      aws cloudformation create-stack \
        --stack-name $CF_STACK_NAME \
        --template-body file://deploy-fargate.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --parameters \
      ParameterKey=Image,ParameterValue=${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
    - echo "Visit https://api.example.com to see changes"
  needs:
    - job: build
      optional: true
  tags:
    - gitlab-dind-runner
  environment:
    name: prod
    url: https://api.example.com

With this, in every CI/CD pipeline we run, build our container image, tag with respectful CI/CD pipeline id, then inject it to CloudFormation template to run it on ECS Fargate.

Updating ECS service - rolling update

In the final step, we need to update ECS service with our updated task revision. To do this, we need to run cloudformation update-stack instead of create-stack. Below pipeline will trigger a rolling update for ECS service.

update_cloudformation:
  stage: update
  when: manual
  image:
    name: amazon/aws-cli:latest
    entrypoint: ['']
  rules:
    - if: $API != "null" && $CI_COMMIT_BRANCH == "master"
  script:
    - echo "Deploying your IaC CloudFormation..."
    - yum install jq -y
    - jq -Version
    - VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
    - echo $VER
    - echo "API Name ----> ${API} <----"
    - echo "ECS FARGATE Cluster is = ${ECS_CLUSTER_NAME}"
    - sed -i 's/API_NAME/'"${API}"'/g' deploy-fargate.yaml #replace API_NAME placeholder with the container that we want to run on AWS Fargate.
    - cat deploy-fargate.yaml
    - |
      aws cloudformation update-stack \
        --stack-name $CF_STACK_NAME \
        --template-body file://deploy-fargate.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --parameters \
      ParameterKey=Image,ParameterValue=${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
    - echo "Visit https://api.example.com to see changes"
  needs:
    - job: build
      optional: true
  tags:
    - gitlab-dind-runner
  environment:
    name: prod
    url: https://api.example.com

That's it! Now you are able to deploy your containerized application with zero downtime, and in an automated way.

In this post, I wanted to mention how to automate deployment process of your container to Amazon ECS Fargate using AWS CloudFormation & GitLab CI/CD.

References:
https://docs.gitlab.com/runner/
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ECS.html

Bring AWS Notifications Into Your Slack Channel

Engin ALTAY — Mon, 28 Nov 2022 12:13:26 +0000

Today’s software development lifecycle world requires responding to notifications immediately; such as incidents, application deployments, security events and so on.

ChatOps — The Slack Way

Nowadays, most of the DevOps teams rely on Slack to collaborate with team members and the system they manage. In the notifications, teams might need to switch among Slack, email, text messages and phone calls at the end of the day. In addition to context switching, merging the data from all those different sources is inefficient and time consuming for teams who monitor and interact with their AWS resources.

Meet AWS Chatbot. Interactive agent that makes it easier to monitor and interact with your Amazon Web Services (AWS) resources from your team’s Slack channels. By integrating AWS Chatbot with Slack, DevOps teams can receive real-time notifications, view incident details, and response incident quickly without need to cycle among other tools.

“With AWS Chatbot, all notifications centrally managed within Slack that our teams already use every day. We’ve configured to receive various notifications such as network & system alerts, application deployments, performance monitoring and more directly into related Slack channels. Teams can take action immediately without needing to switch from where they’re already working. This speed up our response time and overall development agility.” — Engin Altay, SRE, Foreks Digital.

Integrating AWS Chatbot with Slack

Let’s move on AWS Console to begin integrating AWS Chatbot with Slack channel.

Create an Amazon SNS Topic

Amazon Simple Notification Service (SNS) is a fully managed Pub/Sub messaging service that can send notifications two-ways, A2A and A2P.

To create Amazon SNS topic, navigate to AWS Console, go SNS dashboard and create Standart type of topic.

Setting up AWS Chatbot

AWS Chatbot is an AWS service that enables ChatOps for teams. AWS Chatbot processes notifications from Amazon Simple Notification Service (Amazon SNS), and forwards them to chat rooms so teams can monitor and respond to AWS related events in their Slack channel.

AWS Chatbot supports Amazon Chime or Slack chat clients.

Navigate to AWS Console, go to AWS Chatbot and configure new client.

When you click configure, you’ll be redirected to Slack workspace.

Click Add to Slack, and you’ll be asked for required permissions for AWS Chatbot to interact with your Slack channel.

Rename bot user as you wish, which will be shown in your Slack channel.

Then, add bot user to Slack by running the following command in related channel.

/invite @aws-bot

AWS Chatbot — Configure the Slack Channel Settings

AWS Chatbot requires your Slack channel ID to send notifications to specified channel.

In Slack, get channel ID by right clicking the channel and copy the link. Channel ID is the string at the end of the URL.

Then, AWS Chatbot requires IAM permissions to perform actions.

From the policy templates, select Notification permissions. It’s enough to get metric from Amazon CloudWatch.

Lastly, AWS Chatbot needs SNS topic you previously created to send notifications from AWS services to your chat client.

Click Configure, and that’s it! Successfully integrated AWS Chatbot with Slack workspace.

To verify, go to SNS dashboard and see subscription status is confirmed for AWS Chatbot.

After all integration and configuration steps, I’d like to show a real world example that one of our alarm from AWS CloudWatch displayed as rich messages with graphs in our Slack channel.

Leveraging AWS Chatbot makes organizations easily manage, centralize, and monitor AWS notifications into their daily chat rooms. With AWS Chatbot, teams can collaborate and respond to events faster.

Thank you for taking the time to read my article. I appreciate you sharing your thoughts and feedback.

Resources

Automated Deployment to Amazon ECS Fargate

Engin ALTAY — Tue, 18 Oct 2022 11:17:37 +0000

Today's software development lifecycle world requires rapid and uninterrupted way to publish your product. Especially in containers area, there are plenty of methods to deploy your containerized application to your environment.

In this post, I'd like to mention how to deploy your containerized application to Amazon ECS Fargate - a serverless option that run containers without needing to manage your infrastructure, in an automated way using AWS CLI and GitLab CI/CD.

Before begin, below are the required environment. So, I assume that you already have that in use or familiar with tech stack that I mentioned below.

Imagine you have created your ECS cluster, then deployed your first release of containerized application to ECS Fargate. But in the continuation, most probably it'll be required to automate deployment of your new release of container.

To do this, We'll leverage AWS CLI, combining with GitLab CI/CD.

Step 1 - Create your ECS task definition

We need to create ECS task definition which is similar to definitions that refers docker related commands of your containerized application.

myapp-ecs-task-definition.json

{
    "family": "myapp-preprod-fargate",
    "executionRoleArn": "arn:aws:iam::<aws_account_id>:role/ecsTaskExecutionRole",
    "taskRoleArn": "arn:aws:iam::<aws_account_id>:role/ecsTaskExecutionRole",
    "cpu": "1 vCPU",
    "memory": "4GB",
    "networkMode": "awsvpc",
    "containerDefinitions": [
        {
            "name": "myapp-preprod",
            "image": "<aws_account_id>.dkr.ecr.eu-central-1.amazonaws.com/myapp:latest",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/myapp-preprod-fargate",
                    "awslogs-region": "eu-central-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "portMappings": [
                {
                    "protocol": "tcp",
                    "containerPort": 3000
                }
            ],
            "cpu": 0,
            "essential": true
        }
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ]
}

Pay attention to "image": ".dkr.ecr.eu-central-1.amazonaws.com/myapp:latest"

Here we'll replace "latest" tag with our respectful container image version tag.

Step 2 - Prepare your gitlab-ci.yml

Now we need to prepare CI/CD side, which will be used to automate deployment of your updated containerized application to ECS Fargate.

ecs-gitlab-ci.yml is a fully ready .gitlab-ci.yml file that includes build & deployment stages. We'll go through the Amazon ECS Fargate related section.

Setting unique container image version tag

Below command will replace image tag from latest to respectful tag which is *environment name preprod and pipeline number. *

sed -i 's/latest/'"${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}"'/g' myapp-ecs-task-definition.json

After running command above, container image will look as below.

<aws_account_id>.dkr.ecr.eu-central-1.amazonaws.com/myapp:preprod-1

With this, in every CI/CD pipeline we run, unique container image tag will be injected to ECS task definition file.

Getting ECS task revision number

Now create new task definition and get revision number by registering our updated task definition file.

export TASK_REVISION=$(aws ecs register-task-definition \
--family ${ECS_TASK_FAMILY} \
--cli-input-json file://myapp-ecs-task-definition.json \
--region eu-central-1 | jq --raw-output '.taskDefinition.revision')

echo "Registered ECS Task Definition = " $TASK_REVISION

After running command above, ECS task definition file with unique revision number will be created.

aws ecs register-task-definition command will basically creates new task definition file by incrementing revision number by one. And will look as below.

echo "Updated Task Definition = " $ECS_TASK_FAMILY:$TASK_REVISION
myapp-preprod-fargate:1

Updating ECS service - rolling update

In the final step, we need to update ECS service with our updated task revision. Below command will trigger a rolling update for ECS service.

UPDATE_ECS_SERVICE=$(aws ecs update-service \
--cluster $ECS_CLUSTER_NAME \
--service $ECS_SERVICE_NAME \
--task-definition $ECS_TASK_FAMILY:$TASK_REVISION \
--desired-count 1 \
--region eu-central-1 | jq --raw-output '.service.serviceName')

echo "Deployment of $UPDATE_ECS_SERVICE has been completed"

That's it! Now you are able to deploy your containerized application with zero downtime, and in an automated way.

In this post, I wanted to mention how to automate deployment process of your container to Amazon ECS Fargate using AWS CLI & GitLab CI/CD.

DEV Community: Engin ALTAY

Complete Guide - Deploying Production-Ready MongoDB Replica Set on AWS

Preparation Phase: Setting Up MongoDB Replica Set Deployment on AWS

Step 1 - Verify the Server Environment

Step 2 (Optional) - Expand Filesystem

Step 3 - Assign Internal Hostnames

Overview of Authentication-Enabled MongoDB Replica Set

Deploying MongoDB Replica Set with Keyfile Access Control

Step 1 - Generate the Keyfile

Step 2 - Create MongoDB Linux User and Group

Step 3 - Copy the Keyfile to All Replica Set Members

Step 4 - Create Docker Volumes for Persistent Storage

Step 5 - Configure Logging

Step 6 - Start MongoDB Containers

Initiating the MongoDB Replica Set

Step 1 - Access the MongoDB Shell

Step 2 - Initiate the Replica Set

Important Notes:

Step 3 - Verify the Replica Set

Post-Deployment Actions: Creating Privileged & Restricted Users

Step 1 - Access Mongo Shell

Step 2 - Create a Privileged MongoDB User

Step 3 - Create a Restricted User for Applications and APIs

Step 4 - Verify User Access

Conclusion

References:

MongoDB Official Documentation

AWS Resources

Docker Resources

Powering AWS Fargate with IaC - AWS CloudFormation

Step 1 - Building our CloudFormation template

Step 2 - Prepare CI/CD Pipeline - GitLab

Updating ECS service - rolling update

Bring AWS Notifications Into Your Slack Channel

ChatOps — The Slack Way

Integrating AWS Chatbot with Slack

Create an Amazon SNS Topic

Setting up AWS Chatbot

AWS Chatbot — Configure the Slack Channel Settings

Automated Deployment to Amazon ECS Fargate

Step 1 - Create your ECS task definition

Step 2 - Prepare your gitlab-ci.yml

Setting unique container image version tag

Getting ECS task revision number

Updating ECS service - rolling update