DEV Community: Nikolai Main

Using Grafana & Prometheus

Nikolai Main — Sat, 02 Nov 2024 15:11:01 +0000

Seeing as Grafana and Prometheus go hand in hand, I'll include them in the same post and keep each section relatively short. I'll provide a brief overview of each tool along with some basic getting started instructions. The finer details can, of course, be found in their respective documentation. Additionally, I'll briefly cover setting up alerts with Grafana and Slack.

Prometheus

Prometheus is a monitoring and alerting tool providing a wealth of insights for almost any system. It records time-series data, or simply put a series of changes to a given metric over a given time period. Common measurements include request times, active connections, resource usage etc.

Prometheus' main components are:

Prometheus Server: Responsible for 'scraping' time series data.
Client Libraries: Used in instrumenting application code.
Alert Manager: Alert handling

Prometheus supports machine based metrics as well as dynamic service oriented architectures. Since each prometheus server is standalone you can rely on it to diagnose issues with your other systems that may experience outages.

Using Prometheus

Collecting metrics is done via process called 'scraping'. A metric source, or 'instance', usually corressponds to a single process. Once you link an instance to your prometheus server it will begin collecting data based on a configuration you define in a yaml file.

An example Prometheus configuration could look like:

global:

  scrape_interval: 15s

  evaluation_interval: 15s

alerting:

  alertmanagers:

    - static_configs:

        - targets:

           - alertmanager:9093

rule_files:

   - rules.yml

scrape_configs:

  - job_name: "prometheus"

    static_configs:

      - targets: ["localhost:9090"]

  - job_name: node_exporter

    static_configs:

      - targets: ["localhost:9100"]

Notes:

Global Config
- Contains settings like scrape_interval, evaluation interval etc..
Alerting
- States where alerts are sent for further handling.
Rules
- Files containing the metric thresholds that trigger alerts.
Scrape Configs
- A list of every location from which metrics are gathered.

There are several other configuration settings that can be defined however I won't cover them in this post.

Metric Types

Prometheus supports four different metric types:

Counter: A single value that can only increase or be reset to zero. A simple metric that can be used for counting network requests.
Gauge: A single value that can both increase and decrease. Commonly used for counting pods in a K8 cluster or events in an SQS queue.
Histogram: Collects data in several quantiles, or 'buckets', defined by the user. Each value recorded into the histogram will increment all buckets where the value is less than or equal to the value of the bucket. For example, You have the buckets: 0.1, 0.5, 1 and 5. If a value of 0.3 is recorded buckets 0.5, 1 and 5 will be increased. From here you can deduce the distribution of values by subtracting the value of the previous bucket from the current:
- 0.1 has a value of 0 and has no preceding bucket , Therefore it has no recorded values.
- 0.5 has a value of 1 and the preceding bucket has a value of 0, totalling 1.
- 1 has a value of 1 and the preceding bucket a value of 1, totalling 0.
- 5 has a value of 1, preceding bucket also 1, totalling 0. As such, 100% of all values fall under the 0.5 quantile.

Now an example with a single value doesn't really demonstrate a histogram's utility, but take an organization required to serve 95% of all network requests within 300ms. You could set up a bucket with a value of 0.3 and an alert to trigger if the number of requests in that percentile drops below 0.95, enabling you to investigate the issue and notify relevant parties.

Summary: Similar function to histograms but data isn't collected in buckets and instead quantiles are created and estimated by the prometheus server.

General Usage

Configure metric collection locations.
1. If using Kubernetes, Prometheus can automatically scrape clusters to retrieve resource type metrics.
2. If you wish to collect application specific metrics like HTTP request failures, You need to confugure instrumentation within the application using the relevant metrics library.
Define scrape config and any other parameters (as described above) in the prometheus.yml file.
Access your prometheus dashboard (localhost:9090 by default). From here you can run queries and view the data as simple graphs.

Prometheus also provides the ability to create alerts however, The process of creating and handling alerts is much more intuitive and streamlined in Grafana.

Grafana

Grafana is a monitioring and visulization tool that can ingest data from a wide range of sources like MySQL, MongoDB, Postgres etc.. to Kubernetes, AWS, Docker and seemingly everything else.

Grafana offers several features.

Panels and Visualisation: These are composed of a query - a specific metric from a data source - and a visualization. Grafana offers several types of visualization from standard line time-series graphs to heatmaps.
Dashboards: These are a set of one or more panels that provide a quick overview of the information related to a specific metric or data source.
Alerting: Define alerts based on metrics from your data sources and send alerts via email or to other messaging solutions like Slack.
Querying: Since Grafana can ingest data from several different sources, many of which use different querying languages, They offer a powerful query engine that allows you to create custom, complex queries.

Security

Authentication, Authorization

Firstly there are 3 types of roles in Grafana:

Viewer: Has read-only access to dashboards and panels.
Editior: Can create, edit and delete dashboards. Also has access to annotations and alerting.
Admin: Full access to Grafana instance. All of the above and user base control.

Grafana provides the ability to configure basic authentication and authorization within the app. You can create individual users with custom userpass credentials

Alternatively you can use an existing federation or implement one of several oAuth providers like Google or Github.

If you have certain teams within your organization that should only see data relevant to them you can create groups to enable this.

Best Practices

If you are running other services on the same server where Grafana is running it's best to incorporate other measures to secure your applications.

Configure Grafana to only allow data from trusted sources.
Restrict Grafana's communication with other services running on the server.
Configure a proxy server to filter all network traffic.
Those with the viewer role can query any data source in an organization so it's important to be selective with the data you expose.
Disallow anonymous access.

Basic Usage

Start the Grafana server. Assuming you're self hosting and not using Grafana cloud, Access the console at localhost:3000
Add a new data source. If you're using Grafana for the first time you should have a panel with some getting started buttons. Otherwise click the menu on the left hand side > connections > data sources
1. Enter connection details. If using prometheus on default settings enter http://localhost:9090.
Create a dashboard
1. In the query search box find the relevant metric.
2. Click run queries and your data should show up in the panel above.
Once done click save.

Alerting

As mentioned above, Grafana also provides alerting capabilities similar to Prometheus. The process however, is much more simple as it can all be done in Grafana and doesn't require the configuration of another component. I'll explain briefly how to send alerts to Slack:

Slack

Create a Slack workspace.
Go to Slack API and create an app. (https://api.slack.com/)
Click on incoming webhooks and enable them.
1. Click add a new webhook, select your workspace and relevant channel.
2. Copy the webhook url.

Grafana

Go to the Grafana console and from the menu on the left and select alerting > Contact points
1. Create a contact point > Select slack under integrations > Enter webhook URL.
From the same menu, Alerting > alert rules. Create a new alert rule.
1. Enter a name
2. Define query and alert condition
  1. Select the metric you wish to create an alert for.
  2. Create an expression. For example, your metric value crossing a certain theshold.
  3. Set it as the alert condition.
3. Evaluation behavior.
  1. If no rule folders exist, create one.
  2. Similarly, if no evaluation groups exist, create one.
  3. Set the pending period. For the purpose of this set it to none.
4. Configure labels and notifications.
  1. For simplicity just select the Slack channel as the contact point.
  2. Don't bother with labels for now.
5. Enter a summary and description for the alert if necessary.

Now trigger your alert and you should see a notification in your Slack channel.

Final Notes

Keeping track of all types of metrics within your infrastructure, applications, pipeline, etc., is a vital practice in ensuring your entire system stays healthy. As such, the pairing of Prometheus and Grafana provides an excellent solution to this problem. Prometheus excels in collecting and storing time-series data, offering powerful querying capabilities, while Grafana enhances this by providing intuitive and customizable dashboards for visualizing these metrics.

Understanding Kubernetes

Nikolai Main — Mon, 28 Oct 2024 15:09:28 +0000

This turned out to be a relatively lengthy post, but it represents the culmination of my recent study of Kubernetes, where I aimed to gain a comprehensive understanding of its core concepts and features. Throughout my devops research, I came to realise that prioritizing security considerations before any deployment is a more practical and effective approach to implementing technology. As a result, my study focused on two main areas: the fundamental structural components of Kubernetes and the essential security practices associated with it.

Due to resource constraints I was limited to how much of Kubernetes I could explore. However, I used two tools provided me enough access to learn it at a moderate level. These included:

Minikube: A lightweight Kubernetes implementation that creates a virtual machine on any OS and deploys a simple cluster containing only one node.
Kind: A tool for running local Kubernetes clusters using Docker container "nodes."

With Minikube, you are limited to a single cluster, which, for my purposes, was not an issue, as it is more than capable of running a simple three-tier application for reference.

In contrast, Kind offers more flexibility, allowing you to create multiple clusters and control various aspects of them

Nonetheless, this is my account of Kubernetes at a basic yet sufficient level, highlighting it's essential concepts, features, and tools.

Kubernetes Overview

Kubernetes is a fleet container management system that automatically manages the state of a multi-container system. Some key features include:

Service Discovery and Load Balancing: Kubernetes can expose a container based on its DNS or IP. It can also automatically divert traffic to a different container if one is particularly busy.
Storage Orchestration: Automatically mounts a storage option of your choice.
Automated Rollouts and Rollbacks: Kubernetes will automatically meet your desired state in a controlled manner.
Automatic Bin Packing: You can specify your CPU and memory requirements, and Kubernetes will automatically allocate the necessary resources to each container.
Self-Healing: Kubernetes automatically restarts containers that have been shut down.
Secret Management: Manages sensitive information such as passwords and tokens.
Batch Execution: Kubernetes can manage batch and CI workloads.
Horizontal Scaling: Easily scale applications up or down based on demand.
IPv4/IPv6: Allocation of IPv4/IPv6 addresses to pods and services.

Infrastructure

Clusters

Kubernetes coordinates a highly available cluster of computers that work together as a single unit. The Kubernetes cluster architecture consists of:

Control Plane: This coordinates the cluster and handles tasks such as scheduling applications, maintaining the desired state, scaling applications, and rolling out updates. This is governed by the Master Node, which contains several components:
- API Server: The front end for the control plane, handling all REST commands.
- Scheduler: Assigns work to nodes based on resource availability.
- Controller Manager: Manages controllers that regulate the state of the cluster.
- etcd: A distributed key-value store that holds the cluster's state and configuration.
Node Workers: A node is a VM or physical machine that serves as a worker in a cluster. Each node has a kubelet, which is an agent that handles communication between the node and the control plane.
Pods: Within each node, there exists one or more Pods that run application workloads. Each pod can contain multiple containers.

In production workloads, it's recommended to have at least three nodes to ensure you don't experience any downtime if a node goes down. If one node fails, a control plane instance is lost, so having redundant nodes is a sensible choice.

I've come to learn, that as production workloads really start to scale, it's also sensible to have multiple control planes for the same reasons.

Services

Services provide the ability to connect to a deployment. They are the abstraction that allows pods to die and replicate in Kubernetes without impacting your application. Some types of services include:

ClusterIP: The default service type, exposing the service on a cluster-internal IP. Used for internal communication within the cluster.
NodePort: Exposes the service on a static port on each node's IP address, allowing external access.
LoadBalancer: Creates an external load balancer to distribute traffic to the service.
ExternalName: Maps the service to an external DNS name.

Jobs

A job is a resource that creates one or more pods for the purpose of completing a task and then gets destroyed. Kubernetes offer 3 types of job:

Single-Completion Job: Self-explanatory, A single pod is created to complete a task.
Parallel Job: Allows several pods to run concurrently.
Cron Job: Runs jobs on a scheduled basis.

Namespaces

Namespaces are a means of creating a virtually separated section of a Kubernetes cluster. This is useful for:

Isolating Resources: Several teams may work on the same cluster and would want to keep their resources separate from other teams, preventing possible conflicts.
Environment Separation: Allocate space for development, testing, and production.
Access Control: Define who can access and manage resources.
Organizational Clarity: Keep resources organized and easy to manage.

Deployments

A deployment is a higher-level abstraction that manages the lifecycle of a set of replicas of a pod. In other words, it manages the desired state of a given application, defining how many replicas are needed to ensure constant uptime and efficient healing.

Some key features of deployments include:

Declarative Definition: You can declare the desired state of your deployment in a Deployment configuration. Kubernetes then automatically tries to match that state.
Rolling Updates: You can update your applications without experiencing any downtime. Kubernetes replaces old pods with new ones, ensuring that some instances are always available.
Rollback: If an update fails, you can easily revert to a stable version, as Kubernetes keeps track of deployment history.
Scaling: It's very easy to make changes to deployment requirements. Kubernetes will add or destroy pods automatically to match the desired number of replicas.
Self-Healing: If a pod fails or is destroyed, the deployment controller automatically creates a new pod to replace it, maintaining the replica count.

A simple deployment may look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: example-container
        image: example-image

Notes:

kind specifies the type of resource, in this case a deployment.
metadata - standard metadata section
spec - contains information regarding your desired state like the number of desired replicas, container image etc..

Scaling

Kubernetes offers two types of scaling: Horizontal and Vertical. Both of which are common terms in cloud architecture.

Horizontal scaling refers to the addition or removal of pods to handle load requirements.
Vertical scaling refers to the scaling of individual resources. So allocating more or less CPU/memory to existing pods. Kubernetes does not automatically handle this type of scaling.

Autoscaling functionality can be defined as a resource, A horizontal scaler, for example, might look like this:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-deployment
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

You can also manually scale a deployment using the command:

kubectl scale <deployment> --replicas=<num>

Rolling Updates

To maintain uptime, Kubernetes provides the ability to update pods by scheduling the creation of new, updated pods and waiting for those to deploy before destroying the old pods. Rolling updates allow the following actions:

Promote an app from one environment to another.
Roll back to previous versions.
Implement CI/CD with zero downtime.

Updates are triggered by any change to your deployment like the changing of application image which can be done with the following command:

kubectl set image <deployment_name> <image>

Once an update is underway there are several useful commands that can check the status of the update, pause the update or even revert to previous versions:

kubectl rollout status <deployment_name>
kubectl rollout undo <deployment_name>
kubectl rollout history deployment/<deployment_name>

Config

There are several ways to set environment variables in Kubernetes: Dockerfile, Kubernetes YAML, Kubernetes ConfigMaps, and Kubernetes Secrets.

A benefit of using ConfigMaps and Secrets is that they can be reused across containers. Both ConfigMaps and Secrets are API objects that store key-value pairs, but Secrets are used to store confidential/sensitive information.

ConfigMaps can be configured as either environment variables or volumes. If you make a change to a ConfigMap configured as an environment variable the change isn't reflected until the relevant pods are manually refreshed using kubectl rollout. If configured as volume however, the pod recognizes the changes almost immediately.

Immutable ConfigMaps are a suitable option for configurations that are expected to be constant and not change over time. Marking a ConfigMap as immutable provides performance benefits, as the kubelet in a node does not need to watch for changes.

Storage

Volumes

A volume is a directory that allows data to be stored beyond the lifecycle of a container but not beyond a pod. When a pod is destroyed, so is the volume. This is useful if you wish to share data between containers within a pod.

Persistent Volumes

Persistent Volumes (PV) are pieces of storage in the cluster that have been either manually or dynamically provisioned using Storage Classes, allowing data to outlive the lifecycle of individual pods. PVs are cluster resources, similar to nodes.

Persistent volumes are created when a user makes a Persistent Volume Claim (PVC). These 'claims' specify size, access modes, and other requirements and are used by pods to request storage resources.

PVs support different access modes:

ReadWriteOnce (RWO): Can be mounted as read-write by a single node.
ReadOnlyMany (ROX): The volume can be mounted as read-only by many nodes.
ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes.

Security

Pod Security Standards

There are three tiers of security allowance in Kubernetes:

Privileged: Pods can run with any privilege.
Baseline: Provides several sets of security improvements but does not restrict functionality.
Restricted: Enforces several of the baseline rules halting deployment if they are in violation.

In the restricted tier, specific privileges and configurations are enforced, meaning that if a pod violates these rules, it will not be allowed to deploy. In contrast, the baseline tier provides recommendations, and while it may notify you of potential security improvements, it does not prevent deployment.

To enforce pod security, the following command is used on the relevant namespace:

kubectl annotate namespace example-namespace
  pod-security.kubernetes.io/enforce=restricted

This command sets the Pod Security Admission controller to enforce the restricted level of security for all pods created in the specified namespace.

Security at the Namespace Level

The main way security is enforced at the namespace level is through RBAC (Role-Based Access Control). RBAC is controlled and applied with the following:

Roles: Define a set of permissions for resources in a namespace.
RoleBindings: Associate a user or group with a role.

Example of Roles and RoleBindings

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: example-namespace
  name: example-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: example-rolebinding
  namespace: example-namespace
subjects:
- kind: User
  name: example-user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: example-role
  apiGroup: rbac.authorization.k8s.io

Seccomp

Seccomp is built into the Linux kernel and stands for Secure Computing Mode. It provides a mechanism to limit the attack surface of applications by allowing only a predefined set of system calls.

What are System Calls?

System calls are the primary means of communication between user-space applications and the Linux kernel. When a process is run with a seccomp profile, it can only execute system calls that are allowed by that profile.

Some common system calls include:

File Operations: open(), read(), write(), close().
Process Management: fork(), exec(), wait().
Networking: socket(), bind(), connect().

Seccomp Modes

Seccomp offers two modes:

Strict Mode: Only a limited set of system calls are allowed.
Filtered Mode: More flexible, allowing the use of BPF (Berkeley Packet Filter) to define complex rules for system call filtering.

You can run a node with default profiling by running the kubelet with the --default-seccomp flag. The default profile provides a strong set of security defaults while maintaining the functionality of the workload.

Policies

Policies are configurations that manage other configurations or runtime behaviors. These can be used to manage network traffic, resource allocation, or consumption.

They can be applied using one of the following methods:

Admission Controllers
- Admission controllers run in the API server and can validate or mutate API requests. They intercept requests to the API prior to the persistence of the object but after the request is authenticated and authorized. They limit requests to create, delete, and modify objects.
Validating Admission Policies
- VAPs allow the creation of configurable validation checks using the declarative language CEL (Common Expression Language). For example, they can be used to deny the use of the 'latest' image tag.
Dynamic Admission Control
- Dynamic Admission Controllers run outside the API server as separate applications that receive webhook requests to perform validation or mutation of API requests. They can perform complex checks, including those that require other cluster resources or external data.
Kubelet Configurations
- Some kubelet configurations can act as policies, such as PID limits and reservations or Node Resource Managers.

Useful Commands

It quickly became very repetitive typing kubectl out so frequently so I sought a means of shortening my call to it and came across the ability to create/provide aliases for processes.

Creating an alias

PowerShell:

New-Alias <desired-alias> <command>

Bash:

alias desired-alias="command"

Useful kubectl Commands

kubectl cluster-info Prints info relating to the current or named cluster.
kubectl expose deployment <name> --type=<service_type> Creates a service.
kubectl get <resource> Lists all resources of a given type.
kubectl describe <resource> Shows details of a resource or group of resources.
kubectl create <resource> <name> --image=<image> Creates a deployment.
kubectl apply <path_to_yaml_file> Executes a deployment definition.
kubectl edit <resource_type> <resource> Edits a resource.
kubectl port-forward <type/name> host:container Forwards a port.
kubectl logs <pod_name> Gets logs from a container in a pod.
kubectl exec -ti <pod_name> <command> Executes a commands within a container.
- -t (TTY) Allocates a pseudo-TTY, allowing interaction.
- -i (interactive) Keeps the standard input open.
kubectl label <resource_type> <resource> <label> Adds a label to a resource.

Final Notes

As of this post, aside from the Minikube examples, I have yet to deploy my own application on Kubernetes. However, based on what I have learned over the past few months, I can see how straightforward the process can be. I now understand why Kubernetes is such a popular tool for modern application deployment, given its automated node management, self-healing capabilities, and robust networking features. Additionally, I see how easilyKubernetes integrates with CI/CD tools and Infrastructure as Code (IaC) practices and as such I plan to deploy a previous project on Kubernetes.

Static Hosting on S3 w/ Cloudfront Distribution.

Nikolai Main — Thu, 24 Oct 2024 09:49:24 +0000

AWS offers a free and easy solution for hosting static sites. Possible uses for a static site include personal blogs, portfolio sites, or even as a disaster recovery option - If your main service goes down, you can automatically route users to a static site that explains the situation.

The process of creating a static site is straightforward and can be integrated with CloudFront, a caching service that speeds up retrieval by storing website data at edge locations closer to your users. Additionally, Amazon's free TLS certificate service, ACM, can be implemented to provide HTTPS validation for the site.

Components Used

Route 53 - DNS service
Cloudfront - Caching service
Certificate manager - Provides HTTPS certificates
S3 - Bucket storage.
Lambda (Optional) - Create a Lambda function to invalidate the cloudfront cache each time you make changes to your S3 bucket

Data flow

User makes a request to yourdomainname.com
Route53 resolves the request and passes it to CloudFront.
CloudFront reroutes any HTTP traffic to HTTPS and returns tls certificate to validate.
If nothing is cached in CloudFront - Or your caching settings require it - The S3 bucket is queried and retrieves the website.
Content is sent to user.

Creation Process

S3

Create new S3 bucket.
Turn 'block public access' off.
Under object ownership enable ACLs.
Go to your bucket’s properties and scroll all the way to the bottom > Enable static hosting.
- Make sure you set your index document properly, ensuring it’s named correctly and isn't nested in any folders.
When uploading your website, make sure to enable public access for all necessary files.

Route 53

Register a new domain with Route53
Create hosted zone
- Name the hosted zone with your domain name.
- Ensure the allocated ns (name server) records match the registrars.

Domain Registrar

Hosted Zone

ACM

Request certificate from ACM - When requesting a certificate from ACM, make sure your region is set to N. Virginia (us-east-1). This is important because Amazon's administrative infrastructure is primarily located in this region, and certain services, like CloudFront, require certificates to be issued from us-east-1.
Validate your certificate by creating records in your Route 53 hosted zone.

Cloudfront

Create CloudFront distribution
Link S3 bucket - Use as website endpoint
Redirect HTTP to HTTPS.
Link the certificate you created in ACM.
Once created go into settings and add alternate name yourdomainname.com

Finally go back to Route53 and create an A record aliased to your cloudfront distribution.

Lambda (Optional)

Create a lambda function to automatically invalidate (Refresh) your cloudfront cache whenever there is a change in your S3 bucket. This is useful regardless of what you use the static site for but particularly so if youre hosting a blog where you’re regularly changing the site.

Create lambda function with your your S3 bucket as the trigger. More specifically ‘all object create events’ and ‘all object delete events’
Go to IAM and find the role created and provide it access to S3 and Cloudfront.

const AWS = require('aws-sdk');
const cloudfront = new AWS.CloudFront();
const DISTRIBUTION\_ID = '<yourcloudfrontdistributionid';
exports.handler = async (event) => {
    console.log('Event:', JSON.stringify(event, null, 2));
    const params = {
        DistributionId: DISTRIBUTION\_ID,
        InvalidationBatch: {
            Paths: {
                Quantity: 1,
                Items: \['/\*'\], // Invalidates all object in CloudFront
            },
            CallerReference: \`${Date.now()}\`,
        },
    };
    try {
        const data = await cloudfront.createInvalidation(params).promise();
        console.log('Invalidation request sent:', data);
        return {
            statusCode: 200,
            body: JSON.stringify({
                message: 'Invalidation request submitted successfully.',
                invalidationId: data.Invalidation.Id
            }),
        };
    } catch (err) {
        console.error('Error invalidating cache:', err);
        return {
            statusCode: 500,
            body: JSON.stringify({
                message: 'Error invalidating cache.',
                error: err.message
            }),
        };
    }
};

Final Notes

Overall, Setting up a static site on AWS using S3 provides a simple, robust and efficient hosting solution. This configuration ensures high availability and fast content delivery through CloudFront's global network, while ACM secures your site with HTTPS. Automating cache invalidation with a Lambda function keeps your content up-to-date without manual intervention, making it ideal for frequently updated sites. This may not be the most complex of set-ups in AWS, but a useful one nonetheless.

Building a Full-Stack Application with Docker Compose

Nikolai Main — Sat, 19 Oct 2024 09:25:59 +0000

In this project, I created a full-stack container setup using Docker Compose, an orchestration tool that allows you to define multi-container deployments using YAML files. This not only highlighted Docker Compose's utility but also introduced me to Express and React.

I defined a frontend application built with React, a backend API with Express.js, and a PostgreSQL database. While I haven't fully grasped how this setup is beneficial in a production environment, it's brilliant for developing an application.

For context, The application is a company directory that provides create, read, update, and delete functionality for editing personnel, departments, and locations.

Tech Used

Docker
Postgres
Javascript
- Express.js
- React

Useful Commands

General:

docker logs <container name> Display current logs for a given container.
-f: "follow" print logs as they are received.

Compose Specific:

docker compose up/down Run/Stop the Docker compose file
-v: used with compose down - destroys volume as well
--build used with compose up - rebuilds containers
docker compose watch build containers in 'watch mode'

Database

I first created a database with PostgreSQL which turned out to be a relatively straightforward process.

Relevant notes with regards to the volume section:

The first line defines a volume - a location to store persistent data. If a volume with the name db_data doesn't exist, It will be created and any data written to /var/lib/postgresql/data - the location within the container that stores PostgreSQL data - is then written to the db_data volume.
The second line defines the location of an sql.init file, which contains a set of initial queries to populate the database.


  db:

    image: postgres:latest

    container_name: my-postgres-db

    environment:

      POSTGRES_USER: myuser

      POSTGRES_PASSWORD: mypassword

      POSTGRES_DB: mydatabase

    ports:

      - "5433:5432"

    volumes:

      - db_data:/var/lib/postgresql/data

      - ./database/init:/docker-entrypoint-initdb.d/

To make sure everything loaded correctly, I used docker logs my-postgres-db to check the container logs. Once I saw that there were no errors, I logged into the database with psql to verify everything was there and ran a few SELECT queries to confirm.

Backend

Next, I created the backend with Express.js. Although it's a relatively new topic for me, it was easy enough to pick up at a basic level. My Compose service looked like this:

Relevant notes:

The context section of the service tells Docker where to find the Dockerfile necessary in building the container image.
I used Compose's watch feature, which is similar to nodemon for Node.js applications - It automatically rebuilds the container when code changes are detected, which is very useful in development.
The two actions defined in the watch section check for any changes to my index.js file which contains all of my http methods and my package.json file to check if any new modules are installed.


    build:

      context: ./api

    container_name: my-express-app

    ports:

      - "4000:4000"

    depends_on:

      - db

    develop:

      watch:

        - action: rebuild

          path: ./api

          target: index.js

          ignore:

            - node_modules/

        - action: rebuild

          path: package.json

The Dockerfile for the Express.js app looks like this. In essence, it grabs all the relevant content from my directory, installs necessary dependencies, and, once running, executes node index.js to start the Express.js server.

FROM node:18

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 3000

CMD ["node", "index.js"]

With the database and a means to communicate with it set up, I began creating the ~12 methods required to make the application function correctly. To aid in testing the API's functionality, I opened two command lines: one to run curl commands and another to view the container logs.

Frontend

Having defined the backend and the database, all that was left was the frontend. I used React for this, mainly beacuse I had heard of it, and was aware of how common it is in modern frontend development, but had never experimented with it. Very quickly I realized how easy the whole process is and how useful it is for development. Running npx create-react-app my-app gets you up and running with a basic template and a server within minutes.

Initially, I planned to define the whole environment with Docker, but it was just as convenient to keep using React separately. Had I used Docker however, I would have defined a similar compose service to my API where I used watch statements to automate change refresh.

Nevertheless, I created the frontend - a very simple UI with a series of controls to switch between several tables in the database, as well as provide CRUD functionality.

The Dockerfile for my frontend looks like this:

Relevant notes:

This Dockerfile consists of two stages. First, the build stage, where the application image is compiled with the necessary components. Second, creating the image that will host the application with Nginx.
The reason for doing it in two stages is that it results in a smaller final image, as it only contains the server configuration and static files.

FROM node:18 AS build

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

RUN npm run build

# Stage 2: Serve the application with Nginx

FROM nginx:alpine

COPY --from=build /app/build /usr/share/nginx/html

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

For completion sake, I did add the frontend to my Compose file, allowing the full-stack application to be deployed with a single command. Due to network constraints, I never went as far as making this accessible from the internet, but it is an equally simple process.


    build:

      context: ./frontend

    container_name: frontend-app

    ports:

      - "80:80"

    depends_on:

      - api

Final Notes

While I can't see this setup being too useful in a production environment, it's certainly useful for development. Having the ability to spin up and tear down your entire development environment with a single command is incredibly useful. I'm yet to really dip my feet into Kubernetes, but from my limited research, I imagine Kubernetes is a much better option in providing production environments for multi-container deployments.

Understanding Docker

Nikolai Main — Wed, 09 Oct 2024 19:03:05 +0000

Docker is a powerful containerization service that enables the running of multiple applications on the same operating system, each in logically isolated environments. This isolation allows applications to operate independently without interfering with one another, ensuring that they have their own dependencies, libraries, and configurations. This architecture provides developers with numerous benefits, including:

Key Benefits of Docker

Consistency: Applications run in a pre-defined environment specified in your image configuration, maintaining uniformity across development, testing, and production stages. This significantly reduces the "it works on my machine" problem.
Scalability: Applications can be easily scaled by adding or removing containers as needed. In modern microservices architecture, this is particularly beneficial, as individual components can scale independently based on demand.
CI/CD Integration: The ease of stopping and starting containers facilitates rapid deployment and seamless integration into Continuous Integration/Continuous Deployment (CI/CD) pipelines, allowing for faster iterations and updates to applications.

Docker provides several features covering image creation, storage options, and container orchestration, all of which will be discussed in this post.

Image Creation

A Dockerfile contains the instructions that tell Docker how to build your container. An example of a Dockerfile may look like this:
FROM node:18-alpine WORKDIR /appCOPY . . RUN CMD EXPOSE 3000

Breakdown of the Dockerfile

FROM node:18-alpine: This defines the base image for your application. It uses a Node.js image with Node.js pre-installed, version 18, based on the Alpine Linux distribution, which provides a smaller image size.
WORKDIR /app: This sets the working directory inside the container to /app. If the directory does not exist, Docker will create it.
COPY . .: This command copies all files from your current directory on the host into the /app directory in the container.
RUN : This runs any necessary commands during the container build process, such as installing dependencies or compiling assets.
CMD : This defines the command that will be executed when the container starts.
EXPOSE 3000: This declares that the application will be listening on port 3000, allowing other services or users to connect to it.

Building the Docker Image

To build the Docker image, use the following command:

docker build -t .

-t: This tags your image with a name, making it easier to reference later.
.: This tells Docker to look for the Dockerfile in the current directory.

Running the Docker Container

To run the container, use the following command:

docker run -dp 127.0.0.1:3000:3000

-d: This runs the container in detached mode, allowing it to run in the background.
-p: This creates a port mapping between the host and the container in the format HOST:CONTAINER, publishing the container’s port 3000 to 127.0.0.1:3000 on the host.

Stopping and Removing Containers

Once you are finished with your application, you can either shut it down from the Docker UI or use the following commands:

docker ps: Lists all running containers (note the ID of the container you wish to stop).
docker stop : Stops the specified container.
docker rm : Removes the specified container.

Storage

Due to the ephemeral nature of containers, all data within a container is lost once it is shut down. This can be problematic for applications that require persistent data, such as databases. Docker offers two types of storage options to address this issue:

Volumes

Volumes are fully managed by Docker and are isolated from the host filesystem, making them inaccessible from the host unless explicitly mounted. They also provide better performance than bind mounts.Creating and mounting a volume is straightforward and involves two commands:

docker volume create docker run -dp --mount type=volume,src=,target=

Breaking Down the Docker Run Command

type: Can be volume or bind.
src: The name of the volume you created.
target: Where the volume will be mounted. Any data written to this path will be saved in the volume.

Bind Mounts

Mounting a bind is also simple. The following command is used:

docker run -dp --mount type=bind,src=,target=

In this command, the mount type is “bind,” and the source is a directory path instead of a volume name.As changes are made in the target destination, those changes will be recorded almost instantly in the source destination/bind directory. This functionality can be paired with development tools like Nodemon to listen for changes in the development environment and automatically restart the server it’s hosting.

Orchestration

Another feature Docker offers is container orchestration, which manages multi-container applications. In modern development, it’s common to see applications split into several “microservices,” which helps to decouple an otherwise large and cumbersome application.

Benefits of Microservices Architecture

No Single Point of Failure: If one component of your application fails, the rest can continue running, enhancing the overall reliability of the application.
Better Organization: Microservices help organize your infrastructure better, making it easier to manage and scale individual components.

Docker Compose

Docker provides orchestration capabilities through Docker Compose, which uses YAML files to define your containers and any storage options. A typical Compose file might look like this:

version: '3.8'
services:
  app:
    image: node:18-alpine
    command: sh -c "yarn install && yarn run dev"
    ports:
      - 127.0.0.1:3000:3000
    working_dir: /app
    volumes:
      - ./:/app
    environment:
      MYSQL_HOST: mysql
      MYSQL_USER: root
      MYSQL_PASSWORD: secret
      MYSQL_DB: todos

  mysql:
    image: mysql:8.0
    volumes:
      - todo-mysql-data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: todos

volumes:
  todo-mysql-data:

Breaking Down the Compose File

services: These are the containers you wish to deploy. In the example above, two containers are declared: one for the application and one for the database.
volumes: This defines the persistent storage used to capture database changes.

Once you have you’ve defined your compose file, you start every thing with:
docker compose up
And similarly, shutting everything down with:
docker compose down

Additional Features of Docker Compose

Watch: Similar to bind mounts, Compose will ‘listen’ for changes in your application and update it as you make those changes.

services:
  web:
    build: .
    command: npm start
    develop:
      watch:
        - action: sync
          path: ./web
          target: /src/web
          ignore:
            - node_modules/
        - action: rebuild
          path: package.json        path: package.json

In the example above, whenever a change is detected in the /web directory, Compose write the change to the target directory at /src/web. Once everything has been copied the application is updated.

Secrets: To adhere to security best practices, you should never reveal secrets in plain text. Compose allows you to specify a secrets file and reference it instead of explicitly stating sensitive information in your configuration.

Final Notes

Docker is a powerful tool that simplifies the development, deployment, and management of applications through containerization. By leveraging Docker's features, developers can create consistent environments, scale applications efficiently, and integrate seamlessly into CI/CD workflows. Whether you're working on a small project or a large-scale microservices architecture, Docker provides the tools you need to streamline your development process.

Optimizing AWS Infrastructure Deployment: Terraform, Sentinel, and CI/CD Best Practices

Nikolai Main — Sun, 06 Oct 2024 11:00:39 +0000

This project follows on from a previous post where I built AWS infrastructure solely in the AWS console. In it i cover the following topics:

Centralized Terraform state management
Terraform code validation with Sentinel
CI/CD Pipeline deployment
AWS Infrastructure

Less focus is placed on actual application design but may be covered in a later post.

Project Overview

In my initial project, I spent about an hour building the infrastructure and quickly realized how easy it is to make even minor mistakes that can lead to system failures. This often resulted in spending an additional 10 minutes here and there, sifting through each component to identify the error.

Recognizing this challenge in a relatively small project made me acutely aware of the potential headaches that could arise when managing larger systems.

To address this issue, I turned to Terraform. I dedicated a similar amount of time — approximately 1-2 hours — to define my infrastructure. However, the benefits were substantial: instead of spending 1-2 hours each time I needed to deploy, I can now get my entire infrastructure up and running in about 10 minutes, with a comparable teardown time.

This improvement effectively reduced my deployment time by approximately 50 minutes. Additionally, I can confidently assert that my application and infrastructure are secure, thanks to the comprehensive scans conducted prior to deployment:

Infrastructure Validation: My infrastructure is validated and checked with Sentinel in my cloud workspace. If any misconfigurations—such as poor naming and tagging conventions, overly permissive IAM policies, or insecure VPC designs—are present in my Infrastructure as Code (IaC), the run will fail, and I will be notified of the necessary changes.
Application Security Scans: For my application, I utilize GitLab's built-in suite of security tools to scan for code and dependency vulnerabilities, as well as exposed secrets. If GitLab isn’t an option, there are several other security scanning tools available, such as CodeQL, SonarQube, and Trivy. Once the application image is built, it undergoes an additional scan with Trivy to ensure its security.

Infrastructure Overview

Frontend Infrastructure (Repo 1)

ECR (Elastic Container Registry)
ECS (Elastic Container Service)
Application Load Balancer

Backend Infrastructure (Repo 2)

VPC (Virtual Private Cloud)
RDS (Relational Database Service)
API Gateway
AWS Lambda
Secrets Manager

Security Checks and Scans

Pipeline Scans

Secret Detection
SAST (Static Application Security Testing) Scanning
Dependency Scanning
SCA (Software Composition Analysis) Scanning

Sentinel Scans

Appropriate IAM Permissions
General Configuration Checks
VPC Traffic Flows

Deployment Workflow Overview

Workflow 1 (Backend Configuration)

Backend code is pushed to GitLab.
Terraform run triggered in cloud workspace.
Sentinel policies check code for misconfigurations
- VPC: Naming conventions and private subnet config
- Security Groups: Only allowing traffic over necessary ports
- Lambda: IAM permissions and general config
- RDS: Check for encryption, Public Accessibility and Default credentials.
- Secrets Manager: Checks for secret rotation and read replicas
Upon validation infrastructure can be applied. Note relevant outputs.
- RDS Endpoint + Secret Name are needed for Lamdba to work in this project. (I later came back to this and retrieved those outputs dynamically from within the Lambda function)

Example Sentinel Policy - VPC Checks

import "tfplan/v2" as tfplan
import "tfrun" as run
import "strings"

// Define variables

messages = \[\]
resource = "VPC"

// Define main function
checks = func() {
  if run.is\_destroy == true {
    return true
  }

  // Retrieve resource info
  vpc = filter tfplan.resource\_changes as \_, rc {
    rc.mode is "managed" and
    rc.type is "aws\_vpc"
  }
  subnet = filter tfplan.resource\_changes as \_, rc {
    rc.mode is "managed" and
    rc.type is "aws\_subnet"
  }

  // Checking if resource exists.
  if length(vpc) == 0 {
    append(messages, "No vpc found.")
  }
  if length(subnet) == 0 {
    append(messages, "No subnets found.")
  }

  // Iterate over subnets
  for subnet as address, subnet {
    // Check number of available addresses
    if int(strings.split(subnet.change.after.cidr\_block, "/")\[1\]) < 24{
      append(messages, (subnet.address + " CIDR prefix too large. Must be at least 24."))
    }
    if(strings.has\_prefix(subnet.address, "aws\_subnet.private")){

      // Check subnet CIDR block
      if subnet.change.after.cidr\_block == "0.0.0.0/0"{
        append(messages, "Subnet not private. Edit CIDR block")
      }

      // Check if subnet has a public IP enabled.
      if subnet.change.after.map\_public\_ip\_on\_launch == true{
       append(messages, "Subnet not private. Public IP enabled")
      }
    }
  }

  // Run VPC checks
  for vpc as address, vpc {

    // Check if requires\_compatibilities is set and includes "FARGATE"
    requires\_name = vpc.change.after.tags else \[\]

    // Check VPC name/tags
    if length(requires\_name) == 0 or requires\_name.Name == "main-vpc"{
      append(messages, "VPC must follow proper naming conventions. Current name: " + requires\_name.Name)
    }
  }

  // Checking if any error messages have been produced
  // If messages is empty, the policy returns True and passes.
  if length(messages) != 0 {
    print(resource + " misconfigurations:")
    counter = 1
   for messages as message{
     print(string(counter) + ". " + message)
      counter += 1
    }
    return false
  }
  return true
}

// Main rule
main = rule {
   checks()
}

Workflow 2 (Frontend Configuration)

Application code is developed on local machine and pushed to Gitlab.
Pipeline is trigged (More details below)
- Scan application code
- Build image, scan and push to ECR
- Retrieve relevant outputs from backend infrastructure
- Create TF_vars file and push back to GitLab
2nd Terraform workspace triggered by push to repo w/ tag
Similar plan > sentinel scan > apply process takes place.

GitLab Pipeline

Stage 1: Test - SAST, Dependency, Secrets etc..

image: docker:latest
services:
- docker:dind
variables:
  DOCKER\_HOST: tcp://docker:2375/
  DOCKER\_DRIVER: overlay2
  REPO\_NAME: gitlab-cicd

// Declaring the required GitLab scans.
include:
  - template: Jobs/Dependency-Scanning.gitlab-ci.yml
  - template: Jobs/SAST.gitlab-ci.yml
  - template: Jobs/Secret-Detection.gitlab-ci.yml

// All included templates run during 'test' stage.
stages:
  - test
  - build-image
  - fetch-terraform-outputs
  - update-terraform

Stage 2: Build, Scan, Push

build:
  stage: build-image
  before\_script:
  - apk add --no-cache aws-cli
  - apk add --no-cache curl
  script:

  // Building Docker image
  - echo "Building Docker image..."
  - docker build -t $REPO\_NAME:latest .

  // Scanning Docker image with Trivy
  - echo "Running Trivy scan on Docker image"
  - curl -sSL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh
    | sh -
  - export PATH=$PATH:$(pwd)/bin
  - trivy image --exit-code 0 --severity HIGH,CRITICAL $REPO\_NAME:latest || true
  - trivy image --format json --output trivy-results.json $REPO\_NAME:latest

  # Retrieving ECR repo credentials
  - echo "Logging in to Amazon ECR..."
  - aws ecr get-login-password --region $AWS\_DEFAULT\_REGION | docker login --username
    AWS --password-stdin $AWS\_ACCOUNT\_ID.dkr.ecr.$AWS\_DEFAULT\_REGION.amazonaws.com

  # Pushing Docker image to ECR
  - echo "Pushing Docker image to ECR..."
  - TIMESTAMP=$(date +%Y%m%d%H%M%S)
  - IMAGE\_TAG="$REPO\_NAME:$TIMESTAMP"
  - docker tag $REPO\_NAME:latest $AWS\_ACCOUNT\_ID.dkr.ecr.$AWS\_DEFAULT\_REGION.amazonaws.com/$IMAGE\_TAG
  - docker push $AWS\_ACCOUNT\_ID.dkr.ecr.$AWS\_DEFAULT\_REGION.amazonaws.com/$IMAGE\_TAG
  - echo "TF\_VAR\_image\_uri=$AWS\_ACCOUNT\_ID.dkr.ecr.$AWS\_DEFAULT\_REGION.amazonaws.com/$IMAGE\_TAG"
    >> build.env
  artifacts:
    paths:
    - build.env

Stage 3: Fetch TF outputs

fetch-terraform-outputs:
  stage: fetch-terraform-outputs
  image: alpine:latest
  script:
  - apk add --no-cache curl jq
  - echo "Creating variables for specific outputs..."

  // Retrieving outputs via Terraform Cloud API
  - "curl -s -X GET \\\\\\n  \\"https://app.terraform.io/api/v2/workspaces/${HCP\_WORKSPACE\_ID}/current-state-version-outputs\\"
    \\\\\\n -H \\"Authorization: Bearer ${HCP\_TOKEN}\\" \\\\\\n  -H
    'Content-Type: application/vnd.api+json' | \\\\\\njq -r '.data\[\] | select(.attributes.name

    // Saving outputs as environment variables to be passed to the next stage.
    | test(\\"public\_subnet\_ids|alb-sg-id|container-sg-id|vpc\_id\\")) | \\n  if .attributes.name
    == \\"public\_subnet\_ids\\" then\\n    \\"PUBLIC\_SUBNET\_IDS=\\\\(.attributes.value)\\"\\n
    \\ elif .attributes.name == \\"alb-sg-id\\" then\\n    \\"ALB\_SG\_ID=\\\\(.attributes.value)\\"\\n
    \\ elif .attributes.name == \\"container-sg-id\\" then\\n    \\"CONTAINER\_SG\_ID=\\\\(.attributes.value)\\"\\n
    \\ elif .attributes.name == \\"vpc\_id\\" then\\n    \\"VPC\_ID=\\\\(.attributes.value)\\"\\n
    \\ else\\n    empty\\n  end' > terraform\_outputs.env\\n"
  artifacts:
    reports:
      dotenv: terraform\_outputs.env

Stage 4: Update main.tf

update-terraform:
  stage: update-terraform
  image: alpine:latest
  dependencies:
  - build
  - fetch-terraform-outputs
  before\_script:
  - apk add --no-cache git
  - git config --global user.email "${USER\_EMAIL}"
  - git config --global user.name "${USER\_NAME}"
  script:
  - echo "Contents of current directory:"
  - ls -la
  - echo "Contents of build.env:"
  - cat build.env || echo "build.env not found"
  - echo "Contents of terraform\_outputs.env:"
  - cat terraform\_outputs.env || echo "terraform\_outputs.env not found"
  - export $(cat build.env | xargs)
  - export $(cat terraform\_outputs.env | xargs)
  - echo "Cloning repository..."
  - git clone https://<username>:${GITLAB\_PAT}@gitlab.com/<project_id>/<repo.git> || exit
    1
  - cd Test

  // Create TF\_vars file
  - echo "Creating/Updating TF\_vars file..."
  - |
    cat << EOF > terraform.tfvars
    image\_uri = "${TF\_VAR\_image\_uri}"
    public\_subnet\_ids = ${PUBLIC\_SUBNET\_IDS}
    alb\_sg\_id = "${ALB\_SG\_ID}"
    container\_sg\_id = "${CONTAINER\_SG\_ID}"
    vpc\_id = "${VPC\_ID}"
    EOF

  // Commit and push TF\_vars to repo
  - git add terraform.tfvars
  - git commit -m "Update image URI and Terraform outputs in TF\_vars \[ci skip\]" ||
    echo "No changes to commit"
  - TAG\_NAME="$(date +%Y.%m.%d-%H%M%S)"
  - echo "Creating a new tag $TAG\_NAME"

  // Creating a tag to trigger TF cloud only from pushes from this pipeline.
  // 'ci skip' tells the repo not to run the pipeline again on this push. 
  - git tag -a $TAG\_NAME -m "Release version $TAG\_NAME \[ci skip\]"
  - git push origin HEAD:main --tags || exit 1

Final Notes

In conclusion, I now have an end-to-end deployment solution that ensures my application is both secure and robust. This streamlined process has significantly reduced my mean time to deployment, allowing me to reallocate time and resources to other areas.

By identifying potential issues much earlier in the deployment process, I can mitigate risks that previously led to delays and unnecessary costs. This proactive approach not only enhances the overall efficiency of our development cycle but also improves the quality of our releases.

Looking ahead, I plan to involve additional security testing, Incorporate testing and production environments and integrate a monitoring tool such as Grafana.

Connecting to an RDS instance via Bastion Host

Nikolai Main — Fri, 27 Sep 2024 12:11:14 +0000

In keeping with security best practices, databases should always remain private and isolated from the internet. However, for development purposes, accessing your database - such as an RDS instance - can be necessary. Connecting through a Bastion Host provides a secure way to establish this connection.

The tools required for this are:

An AWS account.
Powershell (or other command line shell)
MySQL Workbench (or other visual database interaction software)

Create a VPC

Create a new VPC with both a private and public subnet and while in the VPC dashboard create 2 security groups:

One for your EC2 instance: Allow inbound on port 80 (SSH) from your IP.
One for your RDS instance: Allow 3306 (MySQL) from the EC2 security group. Go back into the EC2 group and allow outbound on 3306 to the RDS security group.

Create an EC2 instance

Free tier options are sufficient here.

Make sure you create a new key pair if you dont already have one and download the .pem file.
Place it in your public subnet, assign the security group you created earlier and ensure assign public ip is ticked.
Take note of the DNS thats created.

Create a new RDS instance

Place it in your private subnet and assign it the RDS security group you created earlier.
Once created take note of the DB endpoint and the provided credentials (once shown you wont be able to view them again.

Connect to your Database

Open a new powershell and use the following command to create a port forwarding session, Replacing each value with your own. Once logged in keep this shell open.
```
ssh -i <path-to-your-key.pem> -L 3306:<rds-endpoint>:3306 <ec2-user@<ec2-public-dns>
```
Open MySQL Workbench and create a new connection. Enter localhost as the hostname and 3306 as the port. Enter the database credentials and click ok. Click on your newly created connection and you should be in.

By following these steps you have securely connected to your private RDS instance through a Bastion Host. This method keeps your database isolated from the internet, aligning with security best practices, while still providing the access needed for development purposes.

Modern Directory Architecture: AWS Serverless with Containerized Frontend

Nikolai Main — Thu, 26 Sep 2024 10:39:20 +0000

The purpose of this project was to create a solution for a hypothetical company looking to host their directory application in the cloud. If needed to be private Cognito could be implemented to provide authentication flow (Covered in another post). The solution consists of the following:

Frontend hosted in ECS containers behind an Application Load Balancer.
Backend hosted in RDS (MySQL).
API Gateway + Lambda to facilitate communication between the frontend and backend
AWS Secrets Manager to store DB credentials
Optionally connect to Route53 & Amazon Certificate Manager (DNS & HTTPS flow covered in a another post)

HLD of proposed architecture.

Application Flow

User makes request the Application Load Balancer and subsequently routes to one of the containers.
Users sends query requests through the application which gets passed to API gateway and subsequently Lambda.
Lambda queries Secrets Manager to retrieve database secrets and shortly after queries the database to complete the user request.
Data is sent back to the container/user.

Components

VPC

Create a VPC with 2 private subnets.
Create an Interface Endpoint for Secrets Manager

Security Groups

There's likely an optimal order of creating these but make all of them and add the rules after.

Lambda

Inbound: 443 from secrets endpoint
Outbound: 443 to Secrets Interface Endpoint SG, 3306 to RDS SG

Containers

Inbound: 443 and/or 80 from ALB SG
Outbound: All (or restrict to your needs)

ALB

Inbound: 443 & 80 from 0.0.0.0 or defined range
Outbound: 443 & 80 to containers SG

Interface Endpoint

Inbound: 443 from Lambda SG
Outbound: None required

RDS

Inbound: 3306 from Lambda SG
Outbound: None

Lambda

Due to the number of requests my application required, I created a single lambda_function (single lambda_handler and several imported sub-functions). As such, my lambda_function folder looked something like this:

company_requests/
├── pymysql/
├── PyMySQL-1.1.1.dist-info/
├── lambda_function.py
├── deleteEmployee.py
├── getAll.py
├── getAllDepartments.py
├── getAllLocations.py
...

Within your development environment (or within Lambda), Create a folder with all of your required Python functions for their associated requests and be sure to include any dependencies. Zip the folder (I used 7zip) and then upload it to Lambda.
Heres a brief snippet of my lambda_function.py:

def lambda_handler(event, context):

  execution_start_time = time.time()

  # Retrieving secrets from the function defined above.
  secret = get_secret()
  username = secret["username"]
  password = secret["password"]
  db_host = "<your-rds-endpoint"
  db_name = "<your-table-name"

  creds = {"user":username, "pass":password, "db_host":db_host, "db_name":db_name}

  # Determine which function to call based on the HTTP method and path
  http_method = event.get('httpMethod')
  resource = event.get('resource')

  # Personnel operations
  if resource == "/all" and http_method =="GET":
      return get_all(event, creds, execution_start_time)
  elif resource == "/personnel" and http_method == "GET":
      return get_all_personnel(event, creds, execution_start_time)
  ...

Function to retrieve secrets from Secrets Manager (also in lambda_function.py.

import boto
import json

def get_secret():
     secret_name = <your_secret_name>
     region_name = <your_region>

     # Create a Secrets Manager client
     session = boto3.session.Session()
     client = session.client(
         service_name='secretsmanager',
         region_name=region_name
     )

     try:
         get_secret_value_response = client.get_secret_value(
             SecretId=secret_name
         )
     except ClientError as e:
         # Handle exceptions
         raise e

     # Decrypts secret using the associated KMS key.
     secret = get_secret_value_response['SecretString']
     return json.loads(secret)

Once you've created your Lambda function, Go to IAM and assign it the following policies:
1. RDS read access - Creating an inline policy can allow for stronger enforcement of least privilege.
2. EC2 Create Network Interface - Needed for placing Lambda inside your VPC.
3. Secrets Manager read secret - Again create an inline policy only allowing access to the necessary secret.
Go into your Lambda function’s network configuration and place it inside the same subnet/s of your VPC that hold your RDS instances. Attach the security group you created earlier.
You may need to increase the functions timeout to ensure it has time to complete requests. This can be done in general configuration.

API Gateway

Create a new API gateway with all of the required resource paths and methods. My resource tree looked something like: / /all /departments /locations /personnel -GET -POST ...
When creating each resource ensure Lambda integration is ticked on all of them and link your Lambda function.
Make sure to enable CORS on each resource path too
Once done, Deploy your api.

ECR

This step assumes you have already containerised your application (Steps on how to do so are covered in a different post).

Create a new repository and follow the provided push steps
Once uploaded take note of the image URI.

ECS

Create a task definition using the URI of the image you uploaded
- Ensure you select the same ports used in your container.
Create a service
1. Select the task definition you just made, Give the service a name and specify 2 tasks.
2. Open the networking tab and select your VPC. Select your private subnets and assign the appropriate security group.
3. Open the load balancing tab and create a new ALB with the containers as targets.
4. After creating go to the load balancer settings in EC2 and assign the appropriate security group.
5. If implementing ACM for HTTPS specify the certificate here and add HTTPS traffic to the container and ALB security groups.

RDS

Create a new DB with whichever engine you require and place it in your private subnets.
Assign it the appropriate security group.

Secrets Manager

Create a new secret store and save your RDS DB credentials there.

After having to create around 12 API resource paths, not to mention all of the other componets, I realised how useful infrastructure as code can be in situation like this so in later posts I will cover defining this project (and likely others) in Terraform.

Although this project achieves the intended outcome, I'm sure there are several tweaks that can be made to make it more streamline and improve security so if this post happens to catch any attention and you have a suggestion please do share.