DEV Community: Roy Ra

Setting up Datadog APM on EKS + Fargate

Roy Ra — Fri, 09 Sep 2022 16:33:17 +0000

In this post, I will go through the steps required to integrate Datadog APM with EKS using Fargate as computing option.

The process is very similar to Setting up Datadog APM on ECS, Fargate(Spring Boot).

The datadog agent container must run as a sidecar of each of your application's container.

Installing datadog agent

We will use helm to install datadog agent into Kubernetes cluster.

First, run the command below to add datadog's repository to helm.

helm repo add datadog https://helm.datadoghq.com
helm repo update

After that, we have to edit values.yaml file, which will be used as a configuration file used by datadog agent running in the cluster. The fields I overrided(edited) are as below.

registry: Apply the value of AWS in the comment.
datadog.apiKey: Value of the API key that you created in Datadog.
datadog.apm.portEnabled: Set to true.

Finally, let's run the command below to finish installation.

helm install datadog-agent -f values.yaml datadog/datadog

Configuring APM

To integrate Datadog APM with your application, you must make sure that your application uses datadog agent when running itself. For instance, I have a Spring boot application (JVM), and corresponding dockerfile looks as below.

There is 1 datadog agent installed on your EKS cluster, and another datadog agent running as a process of your application to collect metrics.

FROM openjdk:11-jdk AS builder
WORKDIR application
ARG JAR_FILE=build/libs/MY_APP.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM openjdk:11-jdk
WORKDIR application
RUN  apt-get update \
  && apt-get install -y wget \
  && rm -rf /var/lib/apt/lists/*
RUN wget -O dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'

COPY --from=builder application/application.jar ./application.jar
RUN true
COPY --from=builder application/dependencies/ ./
RUN true
COPY --from=builder application/spring-boot-loader/ ./
RUN true
COPY --from=builder application/snapshot-dependencies/ ./
RUN true
COPY --from=builder application/application/ ./
RUN true

ENV TZ Asia/Seoul

ENTRYPOINT ["java", "-javaagent:dd-java-agent.jar", "-Ddd.profiling.enabled=true", "-XX:FlightRecorderOptions=stackdepth=256", "org.springframework.boot.loader.JarLauncher"]

In the example above, dd-java-agent.jar uses port 8126 of container running datadog agent to send metrics to datadog.

Now all we have to do is to add another container spec to Kubernetes deployment file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: planit-deployment
  namespace: planit
  labels:
    app: planit
spec:
  replicas: 3
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: planit
  template:
    metadata:
      labels:
        app: planit
    spec:
      containers:
        - name: "CONTAINER_NAME"
          image: "YOUR_IMAGE_NAME"
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
              protocol: TCP
        - name: datadog-agent 
          image: datadog/agent:latest
          ports:
            - containerPort: 8126
              name: traceport
              protocol: TCP
          env:
            - name: DD_KUBERNETES_KUBELET_NODENAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: DD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_API_KEY
            - name: DD_SITE
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_SITE
            - name: DD_EKS_FARGATE
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_EKS_FARGATE
            - name: DD_APM_ENABLED
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_APM_ENABLED
            - name: DD_ENV
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_ENV
            - name: DD_SERVICE
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_SERVICE
            - name: DD_VERSION
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_VERSION
            - name: DD_PROFILING_ENABLED
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_PROFILING_ENABLED
            - name: DD_APM_IGNORE_RESOURCES
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_APM_IGNORE_RESOURCES
            - name: DD_LOGS_INJECTION
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: DD_LOGS_INJECTION

datadog-secret is a Kubernetes secret object I created to store configuration values related to Datadog.

After deploying new Kubernetes deployment, you should be able to see APM working well as below.

Monitoring EKS cluster with Prometheus and Grafana

Roy Ra — Tue, 06 Sep 2022 09:03:04 +0000

In this post, I will go through the minimum steps to configure Prometheus and Grafana in the existing EKS cluster, which is most likely used for monitoring and alerts.

Configuring Node Groups

Creating new node group

Assuming that there are no pre-existing node groups, let's create one.

First, let's attach IAM role to Kubernetes service account using eksctl.



eksctl create iamserviceaccount \
  --name aws-node \
  --namespace kube-system \
  --cluster $EKS_CLUSTER_NAME \
  --role-name "AmazonEKSVPCCNIRole" \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy \
  --override-existing-serviceaccounts \
  --approve

Now let's head over to AWS Management Console -> EKS -> Your cluster -> Compute -> Add node group.

We HAVE to use EC2 for Prometheus and Grafana, since they will both need volumes mounted to them.

When creating node group, we have to attach an IAM role to EC2 worker nodes. For easy demonstration, I created a new IAM role and attached policies as below.

When filling all the information required to create a node group, just make sure to apply values as below.

EC2 instance type: m5.large
Subnets: Your private subnets within the VPC that EKS cluster exists.

You can leave all the other values as it is.

To confirm that your EC2 worker nodes are properly running, run the command below. The result should say that 2 pods are running.



$ kubectl delete pods -n kube-system -l k8s-app=aws-node
NAME             READY   STATUS    RESTARTS   AGE
aws-node-6szct   1/1     Running   0          4s
aws-node-nz5cd   1/1     Running   0          8s

aws-ebs-csi-driver

Prometheus and Grafana needs persistent storage attached to them, which is also called PV(Persistent Volume) in terms of Kubernetes.

For stateful workloads to use Amazon EBS volumes as PV, we have to add aws-ebs-csi-driver into the cluster.

Associating IAM role to Service account

Before we add aws-ebs-csi-driver, we first need to create an IAM role, and associate it with Kubernetes service account.

Let's use an example policy file, which you can download using the command below.



curl -sSL -o ebs-csi-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/docs/example-iam-policy.json

Now let's create a new IAM policy with that file.



export EBS_CSI_POLICY_NAME=AmazonEBSCSIPolicy
aws iam create-policy \
--region $AWS_REGION \
--policy-name $EBS_CSI_POLICY_NAME \
--policy-document file://ebs-csi-policy.json
export EBS_CSI_POLICY_ARN=$(aws --region ap-northeast-2 iam list-policies --query 'Policies[?PolicyName==`'$EBS_CSI_POLICY_NAME'`].Arn' --output text)
echo $EBS_CSI_POLICY_ARN
# arn:aws:iam::12345678:policy/AmazonEBSCSIPolicy

After that, let's attach the new policy to Kubernetes service account.



eksctl create iamserviceaccount \
  --cluster $EKS_CLUSTER_NAME \
  --name ebs-csi-controller-irsa \
  --namespace kube-system \
  --attach-policy-arn $EBS_CSI_POLICY_ARN \
  --override-existing-serviceaccounts --approve

And now, we're ready to install aws-ebs-csi-driver!

Installing aws-ebs-csi-driver

Assuming that helm is installed, let's add new helm repository as below.



helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo update

After adding new helm repository, let's install aws-ebs-csi-driver with below command using helm.



helm upgrade --install aws-ebs-csi-driver \
  --version=1.2.4 \
  --namespace kube-system \
  --set serviceAccount.controller.create=false \
  --set serviceAccount.snapshot.create=false \
  --set enableVolumeScheduling=true \
  --set enableVolumeResizing=true \
  --set enableVolumeSnapshot=true \
  --set serviceAccount.snapshot.name=ebs-csi-controller-irsa \
  --set serviceAccount.controller.name=ebs-csi-controller-irsa \
  aws-ebs-csi-driver/aws-ebs-csi-driver

Configuring Prometheus

First, let's create a new Kubernetes namespace where all Prometheus-related resources will reside in.



kubectl create ns prometheus

We will install Prometheus using helm, so let's add the Prometheus repository to helm and install it running the command below.



# Add helm repositories for Prometheus
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Install Prometheus
helm upgrade -i prometheus prometheus-community/prometheus \
  --namespace prometheus \
  --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

Few minutes after installing Prometheus, we can see all the resources created as below.

To test if they are working, let's use kubectl.



kubectl port-forward -n prometheus deploy/prometheus-server 8081:9090 &

Now let's launch web browser, and go to localhost:8081.

The below web page should be seen if Prometheus has been successfully installed.

Configuring Grafana

As the same as Prometheus, we will create a dedicated Kubernetes namespace for Grafana, and install it using helm.

First let's create a new namespace.



kubectl create ns grafana

We also need a manifest file, which will be used to configure Grafana. Below is an example of the file, called grafana.yaml.



# grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.prometheus.svc.cluster.local
        access: proxy
        isDefault: true

Now let's install Grafana using helm.



# Add helm repository for Grafana
helm repo add grafana https://grafana.github.io/helm-charts

# Install Grafana
helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClass="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='YOUR_PASSWORD' \
    --values grafana.yaml \
    --set service.type=NodePort

Before exposing Grafana to the world, let's see how the Kubernetes service running Grafana is defined.

We can see that the target port is 3000, which is the port used by pods running Grafana.

So we have to attach a new Security group to EC2 worker nodes, allowing inbound requests for port 3000.

We can do this by creating a new Security group, and attaching it to EC2 worker nodes in the EC2 console of AWS Management Console.

After applying new security group to EC2 worker nodes, let's define a new Kubernetes ingress, which will provision an ALB.

To make Kubernetes ingress to create an ALB, we have to install aws-load-balancer-controller first.

Let's see how ingress looks like.



apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: grafana
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: grafana-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: ${PUBLIC_SUBNET_IDs}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: ${ACM_CERT_ARN}
    alb.ingress.kubernetes.io/security-groups: ${ALB_SECURITY_GROUP_ID}
    alb.ingress.kubernetes.io/healthcheck-port: "3000"
    alb.ingress.kubernetes.io/healthcheck-path: /api/health
spec:
  ingressClassName: alb
  rules:
    - host: ${YOUR_ROUTE53_DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 80

After applying new ingress and having new ALB ready, we can head over to ${YOUR_ROUTE53_DOMAIN} and see that Grafana is ready as below.

That's it! In this post, we created a new node group which will be used by Prometheus and Grafana, and installed and configured both Prometheus and Grafana.

Hope this post helps you!

After importing Grafana dashboard using 3119, I got a new dashboard as below.

Using ArgoCD on EKS + Fargate

Roy Ra — Sat, 03 Sep 2022 05:23:53 +0000

In this post, I will describe how to use ArgoCD on existing EKS cluster.

I will not go through all the steps required to configure ArgoCD, but some troubles I encountered.

This documentation from ArgoCD provides all the information about how to install ArgoCD into Kubernetes cluster, and sync(deploy) applications.

But I encountered some troubles when trying to access the ArgoCD API Server on EKS which is using Fargate.

Since Fargate pods are only allowed to be launched within private subnet, we have to use Kubernetes Ingress to route requests from outside world into actual pods. And with AWS Load-Balancer-Controller add-on, when we create a Kubernetes Ingress, it will provision an AWS Application Load Balancer on behalf, accepting all the requests in the frontier.

You can install AWS Load-Balancer-Controller add-on following this documentation.

When we install both ArgoCD and AWS Load-Balancer-Controller add-on into EKS Cluster, we will fail to connect to ArgoCD API Server. This is because the pods running ArgoCD API Server are exposed using port 8080, and security group attached to pods do not accept inbound requests from 8080 by default. This results in ALB's health check failure on its target groups.

So we have to define new security group for ArgoCD API Server pods, and apply it.

To do this, you can simply follow this documentation.

The new security group to attach to pods needs to meet requirements below.

Allow inbound request on port 53(TCP) from security group of the EKS cluster.
Allow inbound request on port 53(UDP) from security group of the EKS cluster.
Allow inbound request on port 8080(TCP).

Let's create new security group, and apply the yaml file below.

# argocd-pod-sg-policy.yaml
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: argo-sgp
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-server
  securityGroups:
    groupIds:
      - ${NEW_CREATED_SECURITY_GROUP_ID}
      - ${EKS_CLUSTER_SECURITY_GROUP_ID}

Let's apply the yaml file above and create new SecurityGroupPolicy. Also, this new SecurityGroupPolicy doesn't apply to already-running pods, so we have to restart all ArgoCD API Server pods.

kubectl delete --all pod -n argocd

Lastly, let's define a Kubernetes Ingress, which will eventually provision a new AWS Application Load Balancer.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-ingress
  namespace: argocd
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: argocd-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: ${PUBLIC_SUBNET_IDS}
    alb.ingress.kubernetes.io/certificate-arn: ${ACM_CERT_ARN}
    alb.ingress.kubernetes.io/security-groups: ${ALB_SECURITY_GROUP}
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/conditions.argogrpc: |
      [{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
spec:
  ingressClassName: alb
  rules:
  - host: dev-argocd.planit-study.com
    http:
      paths:
      - path: /
        backend:
          service:
            name: argogrpc
            port:
              number: 443
        pathType: Prefix
      - path: /
        backend:
          service:
            name: argocd-server
            port:
              number: 443
        pathType: Prefix

On the yaml file above, we used service named argogrpc.
ArgoCD API Server runs on both gRPC and HTTP/HTTPS.
gRPC is used to serve requests from ArgoCD CLI, and HTTP/HTTPS is used for UI(web) requests.

It is ideal to create new service only serving gRPC requests, so let's create one, which is also in the ArgoCD documentation.

apiVersion: v1
kind: Service
metadata:
  annotations:
    alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
  labels:
    app.kubernetes.io/name: argocd-server
  name: argogrpc
  namespace: argocd
spec:
  ports:
  - name: "443"
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
  sessionAffinity: None
  type: NodePort

Now we can successfully access ArgoCD API Server via both CLI and Web(UI)! And that's it! Hope this post helps you :)

Deploying simple application to EKS on Fargate.

Roy Ra — Fri, 26 Aug 2022 11:17:00 +0000

In this post, I'll describe how to deploy a server application on EKS using Fargate. I'll also go through all the problems I encountered while deploying it. This is my first time using EKS!

Step (1) - Configuring credentials and environment

Below are the required tools that we need to successfully deploy EKS in this post.
- AWS CLI v2: Install following this documentation.
- kubectl: Install version 1.23.6, since we will be deploying using kubernetes version 1.21



  curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.22.6/2022-03-09/bin/darwin/amd64/kubectl
  cd ~/.kube
  chmod +x ./kubectl
  mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && 
  export PATH=$HOME/bin:$PATH
  echo 'export PATH=$PATH:$HOME/bin' >> ~/.bash_profile
  kubectl version --short --client

eksctl: Follow this documentation to install eksctl.

After installing all 3 tools above, create an IAM user with programming access enabled to use in your local terminal, and do some tasks. In my case, I created a temporary IAM user and attached AdministratorAccess policy to it.

We will also use KMS, which provides easy ways to configure and manage encryption keys used by various AWS services.



aws kms create-alias --alias-name alias/demo-eks --target-key-id $(aws kms create-key --query KeyMetadata.Arn --output text)

We will use the created CMK's ARN later on, so let's store it as an environment variable.



export MASTER_ARN=$(aws kms describe-key --key-id alias/demo-eks --query KeyMetadata.Arn --output text)
echo $MASTER_ARN > master_arn.txt

Step (2) - Configuring VPC

Now we're going to create a new VPC, which will be used by EKS cluster that we will create later on.
eks-vpc-3az.yaml is a file which will be used by CloudFormation to create resources below.
- VPC.
- 3 public subnets.
- 3 private subnets.
- 1 internet gateway.
- 3 NAT gateways.
- 3 public route tables, and 3 private route tables for each subnet, respectively.
- Security groups for EKS Cluster.
Let's create this VPC using the command below.



  aws cloudformation deploy \
    --stack-name "demo-eks-vpc"
    --template-file "eks-vpc-3az.yaml"
    --capabilities CAPABILITY_NAMED_IAM

For about 5 minutes later, we can see that a brand new VPC has been created in AWS Management Console.
Navigating to CloudFormation page, after selecting the "demo-eks-vpc" stack, we can see the outputs as below image in the output tab.

(3) Provisioning EKS Cluster using eksctl

Setting environment variables

The ID of VPC that we created in step (2) and it's subnet IDs will be used frequently in this step, so lets' run a script as below to use it as environment variables.



#VPC ID
export vpc_ID=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=demo-eks-vpc | jq -r '.Vpcs[].VpcId')
echo $vpc_ID

#Subnet ID, CIDR, Subnet Name
aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)'
echo $vpc_ID > vpc_subnet.txt
aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' >> vpc_subnet.txt
cat vpc_subnet.txt

# Store VPC ID, Subnet IDs as environment variables.
export PublicSubnet01=$(aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' | awk '/demo-eks-vpc-PublicSubnet01/{print $1}')
export PublicSubnet02=$(aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' | awk '/demo-eks-vpc-PublicSubnet02/{print $1}')
export PublicSubnet03=$(aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' | awk '/demo-eks-vpc-PublicSubnet03/{print $1}')
export PrivateSubnet01=$(aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' | awk '/demo-eks-vpc-PrivateSubnet01/{print $1}')
export PrivateSubnet02=$(aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' | awk '/demo-eks-vpc-PrivateSubnet02/{print $1}')
export PrivateSubnet03=$(aws ec2 describe-subnets --filter Name=vpc-id,Values=$vpc_ID | jq -r '.Subnets[]|.SubnetId+" "+.CidrBlock+" "+(.Tags[]|select(.Key=="Name").Value)' | awk '/demo-eks-vpc-PrivateSubnet03/{print $1}')
echo "export vpc_ID=${vpc_ID}" | tee -a ~/.bash_profile
echo "export PublicSubnet01=${PublicSubnet01}" | tee -a ~/.bash_profile
echo "export PublicSubnet02=${PublicSubnet02}" | tee -a ~/.bash_profile
echo "export PublicSubnet03=${PublicSubnet03}" | tee -a ~/.bash_profile
echo "export PrivateSubnet01=${PrivateSubnet01}" | tee -a ~/.bash_profile
echo "export PrivateSubnet02=${PrivateSubnet02}" | tee -a ~/.bash_profile
echo "export PrivateSubnet03=${PrivateSubnet03}" | tee -a ~/.bash_profile
source ~/.bash_profile

After running all the commands above, VPC ID and subnet IDs will be stored in environment variables. Also there will be a file called vpc_subnet.txt, and it will look as below.



vpc-0e88a2ed7a32c0336
subnet-02b5356084f4355cb 10.11.16.0/20 demo-eks-vpc-PublicSubnet02
subnet-0ea280f1567234a3b 10.11.253.0/24 demo-eks-vpc-TGWSubnet03
subnet-0a79d22a3acf610bf 10.11.32.0/20 demo-eks-vpc-PublicSubnet03
subnet-0c96f4c64e724524b 10.11.251.0/24 demo-eks-vpc-TGWSubnet01
subnet-0d5d255e8542cf405 10.11.64.0/20 demo-eks-vpc-PrivateSubnet02
subnet-0c3c37774542ac95c 10.11.252.0/24 demo-eks-vpc-TGWSubnet02
subnet-07f97eaa984b5ced2 10.11.0.0/20 demo-eks-vpc-PublicSubnet01
subnet-0ebb353a91c36ab17 10.11.48.0/20 demo-eks-vpc-PrivateSubnet01
subnet-0e3c3abe52dbe07e8 10.11.80.0/20 demo-eks-vpc-PrivateSubnet03

Let's also store AWS region code.



export AWS_REGION=ap-northeast-2 # in my case

Creating EKS Cluster

Cluster's name and version of Kubernetes will be used by yaml files, so let's store it as environment variables.



export ekscluster_name="demo-eks"
export eks_version="1.21"

To create EKS cluster, we will use eks-cluster-3az.yaml.
We will configure nodes to use Fargate instead of EC2, and accordin to this documentation, Fargate needs to be provisioned within private subnets.
Let's run the command below, to fill environment variables with actual values in eks-cluster-3az.yaml



cat << EOF > eks-cluster-3az.yaml
# A simple example of ClusterConfig object:
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${ekscluster_name}
  region: ${AWS_REGION}
  version: "${eks_version}"

vpc:
  id: ${vpc_ID}
  subnets:
    private:
      PrivateSubnet01:
        az: ${AWS_REGION}a
        id: ${PrivateSubnet01}
      PrivateSubnet02:
        az: ${AWS_REGION}b
        id: ${PrivateSubnet02}
      PrivateSubnet03:
        az: ${AWS_REGION}c
        id: ${PrivateSubnet03}

secretsEncryption:
  keyARN: ${MASTER_ARN}

fargateProfiles:
  - name: demo-dev-fp
    selectors:
      - namespace: demo-dev
      - namespace: kube-system
    subnets:
      - ${PrivateSubnet01}
      - ${PrivateSubnet02}
      - ${PrivateSubnet03}

cloudWatch:
  clusterLogging:
    enableTypes:
      ["api", "audit", "authenticator", "controllerManager", "scheduler"]

EOF

Fargate profile is needed to run resources on Fargate.(fargateProfiles). Any resources that meet criteria defined in fargateProfiles.selectors will run on Fargate.
In the example above, we used demo-dev, the namespace that application-related resources will use, and kube-system.
kube-system will be used to provision additional resources such as aws-load-balancer-controller which we will install later.
If we rule out kube-system namespace, we will get an error as below when installing aws-load-balancer-controller.
ingress 0/2 nodes are available: 2 node(s) had taint {eks.amazonaws.com/compute-type: fargate}, that the pod didn't tolerate.

Now lets' run the command below to create cluster!



eksctl create cluster --config-file=eks-cluster-3az.yaml

Connecting to EKS cluster

When we run kubectl get svc command, we will get an output like below, which means that EKS cluster is ready!



# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   43m

Value of PORT(S) might be different, but don't worry.

(4) Deploying application!

First, let's create a Kubernetes namespace, where we will create resources for our application.

In the above when we created faragte profile, we put namespace: demo-dev in the fargateProfiles.selectors field.



kubectl create ns demo-dev

Installing AWS Load Balancer Controller add-on

AWS Load Balancer Controller manages AWS Elastic Load Balancers(ELB) used by Kubernetes cluster. This controller provides:
- Provision new AWS ALB when Kubernetes Ingress is created.
- Provision new AWS NLB when Kubernetes LoadBalancer is created.
Before installing AWS Load Balancer Controller, we must integrate OIDC provider first.
Configuring OIDC ID Provider(IdP) to EKS cluster allows you to use AWS IAM roles for Kubernetes service accounts, and this requires an IAM OIDC provider in the cluster. Let's run the command below to integrate OIDC into the cluster.



eksctl utils associate-iam-oidc-provider --cluster demo-eks --approve
# 2022-08-23 16:04:39 [ℹ]  will create IAM Open ID Connect provider for cluster "demo-eks" in "ap-northeast-2"
# 2022-08-23 16:04:40 [✔]  created IAM Open ID Connect provider for cluster "demo-eks" in "ap-northeast-2"

Now we're ready to install AWS Load Balancer Controller add-on! Install by following steps described in this documentation.

Deploying application

In my case, I was using an application which used port number 8080 by default. I didn't want to change default this port number, so I created a simple yaml file(deployment.yaml) for Kubernetes Deployment, and also created a yaml file for Kubernetes Service as below.



apiVersion: v1
kind: Service
metadata:
  name: demo-dev-svc
  namespace: demo-dev
spec:
  selector:
    app: demo
  ports:
   -  protocol: TCP
      port: 80
      targetPort: 8080
  type: NodePort

As you can see, type of service is NodePort, and the type should be NodePort when using Fargate on EKS.
Also, the service will receive requests from port number 80, and forward the request to pods using pod's port number 8080.
Lastly, let's create a Kubernetes Ingress, which will eventually create an AWS ALB with the help of AWS Load Balancer Controller add-on.



apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo-dev-ingress
namespace: demo-dev
annotations:
  alb.ingress.kubernetes.io/load-balancer-name: demo-dev-lb
  alb.ingress.kubernetes.io/scheme: internet-facing
  alb.ingress.kubernetes.io/target-type: ip
  alb.ingress.kubernetes.io/subnets: ${PUBLIC_SUBNET_IDS}
  alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
  alb.ingress.kubernetes.io/certificate-arn: ${ACM_CERT_ARN}
  alb.ingress.kubernetes.io/security-groups: ${SECURITY_GROUP_FOR_ALB}
  alb.ingress.kubernetes.io/healthcheck-port: "8080"
  alb.ingress.kubernetes.io/healthcheck-path: /actuator/health
  alb.ingress.kubernetes.io/success-codes: "200"
spec:
ingressClassName: alb
rules:
  - host: alb-url.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: demo-dev-svc
              port:
                number: 80

In the spec, we will make ALB to forward all request to demo-dev-svc Kubernetes service.
You can find about annotations in this document.
Note that alb.ingress.kubernetes.io/target-type should be set to ip when using Fargate.
Also the ALB provisioned when creating this Ingress should receive traffic from the internet, so it should be placed in public subnets(alb.ingress.kubernetes.io/subnets: ${PUBLIC_SUBNET_IDS}), and it should be internet-facing(alb.ingress.kubernetes.io/scheme: internet-facing).
Now when we apply new Ingress to Kubernetes, new ALB will be created with appropriate listener rule configured.
But when we view the health check of ALB on target groups, it will fail, saying 504 gateway timeout.
since default security group attached to Fargate pods do not allow inbound requests on port 8080.

Resolving issue: 504 Gateway Timeout

This is because default security group attached to Fargate pods does not allow inbound requests on port 8080.
So we need to configure custom security groups to Fargate pods, which is described in this documentation.
According to the documentation above, new security group must:
- Allow inbound request on port 53(TCP) from security group of the EKS cluster.
- Allow inbound request on port 53(UDP) from security group of the EKS cluster.
Let's create a new security group, and apply the below yaml file.



apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
name: dev-sgp
namespace: demo-dev
spec:
podSelector:
  matchLabels:
    app: demo
securityGroups:
  groupIds:
    - ${NEW_CREATED_SECURITY_GROUP_ID}
    - ${EKS_CLUSTER_SECURITY_GROUP_ID}

When we apply the yaml file, we must restart all pods since the new security group will not effect already running pods.
Now when we check the ALB health check, we can see that health checks on target groups are successfully done, and all targets are healthy.

Wrapping up

That's it! In this post, I didn't want to focus on any Kubernetes fundamentals, but wanted to share how I troubleshooted issues that I encountered in my deployment process.
Also, there are tons of documents to see when we encounter some issues when deploying Kubernetes resources. I wanted to narrow down these documents to what we really have to see, which is why I embedded so many links to documentations in this post.

Extra

When we log in to AWS Management Console with root account and visit EKS service page, we will see a warning as below.



Your current user or role does not have access to Kubernetes objects on this EKS cluster This may be due to the current user or role not having Kubernetes RBAC permissions to describe cluster resources or not having an entry in the cluster’s auth config map.

If you want to view your resources on your EKS Cluster, run the command below.



kubectl edit configmap aws-auth -n kube-system

And add below fields to the same depth as mapRoles.



mapUsers: |
  - userarn: arn:aws:iam::[account_id]:root
  groups:
  - system:masters

After save and quit, you will be able to view all resources within your EKS cluster.

Using DynamoDB on Spring Boot (feat.Kotlin)

Roy Ra — Fri, 22 Jul 2022 04:46:28 +0000

In this post, I will talk about how to use Amazon DynamoDB with Spring Boot application made with Kotlin.

The codes written here are all stored in this Github repository.

Situation

Here we will think about a situation where we have to store posts created. Below is the example scheme of data that we will store in DynamoDB.

{
    "post_id": "123a",
    "user_id": "roy",
    "title": "Title of this post",
    "content": "Content of this post",
    "created_at": "2022-07-16T14:07:31.000Z"
}

And below are our application's requirements.

Get a specific post using post_id.
Search posts with title and sort it using created_at.
Search posts by written by a specific user(user_id), and sort it using created_at.

To meet the application's requirements, we can configure this DynamoDB table as below.

Primary Key: post_id(partition key)
Global Secondary Indexes:
- post_user_id_created_at_idx: HASH: user_id, RANGE: created_at
- post_title_created_at_idx: HASH: title, RANGE: created_at

Setup

To demonstrate DynamoDB without actually using it, we will run a docker container with a DynamoDB image.

(1) Clone this repository to your local machine and change directory into it.

(2) Run docker container using below commands.

# Run
docker compose -f docker-compose.yml up -d  
# Remove
docker compose -f docker-compose.yml down

When you run the "Run" command, it will run a docker container acting as DynamoDB in localhost:54000.

(3) Configure DynamoDB table with scripts/create-dynamodb-table.sh

# Add permission
chmod +x ./scripts/create-dynamodb-table.sh
# Execute
./scripts/create-dynamodb-table.sh

This script will configure the table needed.

create-dynamodb-table.sh contains all the commands required to create a DynamoDB table with aws-cli, and all global secondary indexs are defined in scripts/gsi.json file.
create-dynamodb-table.sh reads information about global secondary indexes from gsi.json, and configures it.

Configuring DynamoDB in Spring Boot Application

Adding dependencies

First we have to add required dependencies to our code.
Assuming using Gradle, let's add two dependencies as below.

dependencies {
    //..
    implementation("com.amazonaws:aws-java-sdk-dynamodb:1.12.258")
    implementation("io.github.boostchicken:spring-data-dynamodb:5.2.5")
}

spring-data-dynamodb is not an official library maintained by AWS nor Spring team. It originated from michaellavelle/spring-data-dynamodb, and was forked and maintained at derjust/spring-data-dynamodb to support Spring Boot versions up to 2.1.x. Now it is being maintained at boostchicken/spring-data-dynamodb, supporting Spring Boot versions up to 2.2.x.

DynamoDBConfig.kt

DynamoDBConfig class defines configurations to use DynamoDB.

Configuration
@EnableDynamoDBRepositories(basePackages = ["com.example.post.domain"])
class DynamoDBConfig(
    @Value("\${amazon.dynamodb.endpoint}") private val endpoint: String,
    @Value("\${amazon.aws.accessKey}") private val accessKey: String,
    @Value("\${amazon.aws.secretKey}") private val secretKey: String,
    @Value("\${amazon.aws.region}") private val region: String
) {

    @Primary
    @Bean
    fun dynamoDBMapper(amazonDynamoDB: AmazonDynamoDB): DynamoDBMapper {
        return DynamoDBMapper(amazonDynamoDB, DynamoDBMapperConfig.DEFAULT)
    }

    @Bean
    fun amazonDynamoDB(): AmazonDynamoDB {
        val awsCredentials = BasicAWSCredentials(accessKey, secretKey)
        val awsCredentialsProvider = AWSStaticCredentialsProvider(awsCredentials)
        val endpointConfiguration = AwsClientBuilder.EndpointConfiguration(endpoint, region)
        return AmazonDynamoDBClientBuilder.standard()
            .withCredentials(awsCredentialsProvider)
            .withEndpointConfiguration(endpointConfiguration)
            .build()
    }

    @Bean
    fun awsCredentials() = BasicAWSCredentials(accessKey, secretKey)
}

As you can see, we are configuring AWS credentials, and registering AmazonDynamoDB as a Spring Bean.

Post.kt

Post class represents the entities which will be saved in posts DynamoDB table. Let's define field, add appropriate annotations based on requirements.

@DynamoDBTable(tableName = "posts")
class Post(
    @field:DynamoDBHashKey
    @field:DynamoDBAttribute(attributeName = "post_id")
    val id: String = UUID.randomUUID().toString(),

    @field:DynamoDBAttribute(attributeName = "user_id")
    @field:DynamoDBIndexHashKey(globalSecondaryIndexName = "post_user_id_created_at_idx")
    val userId: String,

    @field:DynamoDBAttribute(attributeName = "title")
    @field:DynamoDBIndexHashKey(globalSecondaryIndexName = "post_title_created_at_idx")
    val title: String,

    @field:DynamoDBAttribute(attributeName = "content")
    val content: String,

    @field:DynamoDBAttribute(attributeName = "created_at")
    @field:DynamoDBTyped(DynamoDBMapperFieldModel.DynamoDBAttributeType.S)
    @field:DynamoDBIndexRangeKey(globalSecondaryIndexNames = ["post_user_id_created_at_idx", "post_title_created_at_idx"])
    val createdAt: LocalDateTime = now()
)

PostRepository.kt

PostRepository interface is where you declare spring-data-jpa-styled methods to query items from DynamoDB table.

@EnableScan
interface PostRepository : CrudRepository<Post, String> {
    fun findByUserIdOrderByCreatedAtAsc(userId: String): List<Post>
    fun findByTitleOrderByCreatedAtDesc(title: String): List<Post>
}

Testing read, write and resolving issues

Above code works fine when we try to insert new item into DynamoDB table using PostRepository.save(). However, when we call PostRepository.findByUserIdOrderByCreatedAtAsc(), it throws an error saying:

java.lang.NoSuchMethodException: com.example.post.domain.Post.<init>()

This means that we have to add default constructor for Post class, so let's simply add default values for each properties to implement this.

@DynamoDBTable(tableName = "posts")
class Post(
    @field:DynamoDBHashKey
    @field:DynamoDBAttribute(attributeName = "post_id")
    val id: String = UUID.randomUUID().toString(),

    @field:DynamoDBAttribute(attributeName = "user_id")
    @field:DynamoDBIndexHashKey(globalSecondaryIndexName = "post_user_id_created_at_idx")
    val userId: String = "",

    @field:DynamoDBAttribute(attributeName = "title")
    @field:DynamoDBIndexHashKey(globalSecondaryIndexName = "post_title_created_at_idx")
    val title: String = "",

    @field:DynamoDBAttribute(attributeName = "content")
    val content: String = "",

    @field:DynamoDBAttribute(attributeName = "created_at")
    @field:DynamoDBTyped(DynamoDBMapperFieldModel.DynamoDBAttributeType.S)
    @field:DynamoDBIndexRangeKey(globalSecondaryIndexNames = ["post_user_id_created_at_idx", "post_title_created_at_idx"])
    val createdAt: LocalDateTime = now()
)

After that when we invoke the repository method again, we get different error saying:

java.lang.NullPointerException: null
    at com.amazonaws.services.dynamodbv2.datamodeling.StandardBeanProperties$MethodReflect.set(StandardBeanProperties.java:133) ~[aws-java-sdk-dynamodb-1.12.258.jar:na]

This error occurs because spring-data-dynamodb first creates Post instance using default constructor, and sets each values using setters. Since every properties of Post class is declared as val, there are no setters created. Let's simply declare all properties with var instead of val.

@DynamoDBTable(tableName = "posts")
class Post(
    @field:DynamoDBHashKey
    @field:DynamoDBAttribute(attributeName = "post_id")
    var id: String = UUID.randomUUID().toString(),

    @field:DynamoDBAttribute(attributeName = "user_id")
    @field:DynamoDBIndexHashKey(globalSecondaryIndexName = "post_user_id_created_at_idx")
    var userId: String = "",

    @field:DynamoDBAttribute(attributeName = "title")
    @field:DynamoDBIndexHashKey(globalSecondaryIndexName = "post_title_created_at_idx")
    var title: String = "",

    @field:DynamoDBAttribute(attributeName = "content")
    var content: String = "",

    @field:DynamoDBAttribute(attributeName = "created_at")
    @field:DynamoDBTyped(DynamoDBMapperFieldModel.DynamoDBAttributeType.S)
    @field:DynamoDBIndexRangeKey(globalSecondaryIndexNames = ["post_user_id_created_at_idx", "post_title_created_at_idx"])
    var createdAt: LocalDateTime = now()
)

Now, when we call the repository method, we get different error saying:

java.lang.IllegalArgumentException: argument type mismatch

While this error message is quite unkind, if we think carefully, the only property that has different type from Kotlin code and DynamoDB is createdAt. This field's Kotlin type is LocalDateTime, while DynamoDB attribute type is S, indicating string.

So let's simply remove the @field:DynamoDBTyped(DynamoDBMapperFieldModel.DynamoDBAttributeType.S) annotation from createdAt property, and invoke the repository method again.

Now we get another error, saying:

com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException: Post[created_at]; only scalar (B, N, or S) type allowed for key

Why is this happening? We previously defined that createdAt's field type is S using @DynamoDBTyped annotation, but error message is indicating that we should define DynamoDB type for createdAt!

The problem is that while Date can be automatically converted to S in spring-data-dynamodb, LocalDateTime cannot. So we have to declare a converter for this field which is responsible of conversion between Date and LocalDateTime.

Getting back to DynamoDBConfig class, let's add this converter.

@Configuration
@EnableDynamoDBRepositories(basePackages = ["com.example.post.domain"])
class DynamoDBConfig(
    @Value("\${amazon.dynamodb.endpoint}") private val endpoint: String,
    @Value("\${amazon.aws.accessKey}") private val accessKey: String,
    @Value("\${amazon.aws.secretKey}") private val secretKey: String,
    @Value("\${amazon.aws.region}") private val region: String
) {

    companion object {
        class LocalDateTimeConverter : DynamoDBTypeConverter<Date, LocalDateTime> {
            override fun convert(source: LocalDateTime): Date {
                return Date.from(source.toInstant(ZoneOffset.UTC))
            }

            override fun unconvert(source: Date): LocalDateTime {
                return source.toInstant().atZone(TimeZone.getDefault().toZoneId()).toLocalDateTime()
            }
        }
    }

    // Spring Beans..
}

After that, we have to make createdAt property to use this converter.

@DynamoDBTable(tableName = "posts")
class Post(
    // Other properties..

    @field:DynamoDBAttribute(attributeName = "created_at")
    @field:DynamoDBTypeConverted(converter = DynamoDBConfig.Companion.LocalDateTimeConverter::class)
    @field:DynamoDBIndexRangeKey(globalSecondaryIndexNames = ["post_user_id_created_at_idx", "post_title_created_at_idx"])
    var createdAt: LocalDateTime = now()
)

Boom! Now we can use repository methods successfully!

Wrapping up

It's been a quite long journey, resolving all the new errors upcoming. But this is the fun of developing, isn't it?

By the way if we have to configure primary key of DynamoDB table using both partition key and sort key, we need to configure domain class(in this example, Post class) more tha n just adding properties and applying @DynamoDBHashKey and @DynamoDBRangeKey annotation for each property. I will discuss how to implement this in the later post.

Beginner's guide to AWS Solutions Architect Associate(SAA) certification.

Roy Ra — Wed, 13 Jul 2022 16:06:00 +0000

In this post, I would like to share a little bit of story about how I prepared for AWS Solutions Architect Associate exam from a beginner's perspective.

The exam that I took (SAA-C02) is retiring on August 29, 2022.

Preparing

This exam exists to verify your knowledge about general services provided by AWS, and how and when to use these services in various situations. As you can find many blog posts or videos explaining what kinds of questions appear in this exam, I will not discuss it here.

I prepared for this exam only using the suggested learning path provided by CloudAcademy. The lectures covered many services provided by AWS, mostly focused on the services that are most likely to appear in the exam.

Going straightforward, I will discuss the benefits and drawbacks of preparing for this exam using CloudAcademy.

Benefits of preparing using CloudAcademy

All lectures cover one or more services provided by AWS, but also discuss enough, meaningful context. For example, if we have to learn about multi-tiered architecture, the lecture covers why we need it, various situations of when to implement it, and how to implement it using AWS services.
Each category(security, management, etc.) has hands on lab sessions, which allows you to actually use some of the services covered in the session, and having hands-on experience really helps you keep those services in mind much longer.
Each category has quizzes, which asks you about the services learned. It really helps to keep the knowledges in your mind.
The last and best is that at the very end, there are example questions provided by CloudAcademy team, which is really similar from questions that appear in the SAA exam. Also, there seems to be 300+ number of questions, so you can retake the exam as much as you want, and know your vulnerabilities.

Drawbacks of preparing using CloudAcademy

There was only one drawback of using CloudAcademy, and it was that the questions didn't cover some of the services that appeared in my exam. For example, the SAA exam I took had some questions about Step Functions, Rekognition, Transcribe, etc.

Taking the exam

I took the exam from home, and there was a human test proctor for me, directing a lot of things such as "clean up all the things under your desk" and more. It took almost 30 minutes for me to get my environment ready to take the exam.

I got two warnings to not cover my mouth during the exam, and everything else went smooth. I didn't need extra 30 minutes for the exam, and I solved and submitted my answers in almost 90 minutes.

I think the best factor that speeded up my time is that I took the exam questions provided by CloudAcademy for almost 10+ times.

Wrapping up

All the processes of preparing for SAA really helped me a lot about expanding my knowledge of AWS, and how to reduce cost, design various systems while maintaining high availability.

The exam didn't require that much of memorization, I just needed to be fully aware about what each service provides, how it gives benefits to customers.

Setting up Datadog APM on ECS, Fargate(Spring Boot)

Roy Ra — Tue, 07 Jun 2022 01:35:53 +0000

Hello this is Roy, and today I want to write a short blog post about enabling Datadog APM at Spring Boot application running on Fargate, along with ECS.

APM helps you deep dive into your application, allowing you to see all the metrics and what kind of tasks were executed(also in which order) to handle a request.

For example, if you were to use APM on server applications, which is also in this case, you can see how much time it took to execute queries to database, how long it took to process associated codes, and also the latency time, which is a time indicating how long it took for the server to process the request made from a client.

For container metrics, you can reference this nice documentation from Datadog itself.

As I was using Amazon ECS with Fargate, I was confused if I had to see APM configuration for ECS or for Fargate.

After reading all the relevant docs, I ended up noticing that there were three essential steps to enable Datadog APM on my environment.

Download dd-java-agent.jar
Assign this downloaded jar file to -javaagent flag when running the application jar file.
Set appropriate environment variables for Datadog.

For step 3, you can configure it in your task definition file.

In my case, my previous Dockerfile for containerizing my Spring Boot application looked as below.

FROM openjdk:11-jdk AS builder
WORKDIR application
ARG JAR_FILE=build/libs/MY_APPLICATION.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM openjdk:11-jdk
WORKDIR application
COPY --from=builder application/dependencies/ ./
RUN true
COPY --from=builder application/spring-boot-loader/ ./
RUN true
COPY --from=builder application/snapshot-dependencies/ ./
RUN true
COPY --from=builder application/application/ ./
RUN true
ENV TZ Asia/Seoul
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

As I was wondering where I was supposed to download the dd-java-agent.jar file and assign it to -javaagent flag, I found that Dockerfile was the appropriate place in my case.

So my Dockerfile changed as below, and as you can see it downloads dd-java-agent.jar file, and assigns it to -javaagent flag when running my application's jar file.

FROM openjdk:11-jdk AS builder
WORKDIR application
ARG JAR_FILE=build/libs/MY_APPLICATION.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM openjdk:11-jdk
WORKDIR application
RUN  apt-get update \
  && apt-get install -y wget \
  && rm -rf /var/lib/apt/lists/*
RUN wget -O dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'

COPY --from=builder application/application.jar ./application.jar
RUN true
COPY --from=builder application/dependencies/ ./
RUN true
COPY --from=builder application/spring-boot-loader/ ./
RUN true
COPY --from=builder application/snapshot-dependencies/ ./
RUN true
COPY --from=builder application/application/ ./
RUN true

ENV TZ Asia/Seoul

ENTRYPOINT ["java", "-javaagent:dd-java-agent.jar", "-Ddd.profiling.enabled=true", "-XX:FlightRecorderOptions=stackdepth=256", "org.springframework.boot.loader.JarLauncher"]

And that's it! After deploying new tasks, I was able to see my application on Datadog APM tab, and was able to see all the events made, all the details and process time of it, how long it took to process a request and much more.

I spent around 3 days configuring this, and I am hoping that this blog will help you if you are in the same situation as I am.

Thank you, and any feedbacks or comments are always appreciated.

Configuring Minimum-Secure Web Application with Amazon VPC, EC2, and RDS

Roy Ra — Mon, 18 Apr 2022 12:35:15 +0000

There are several key points that we should consider when making web applications on the cloud.(on-premises too.)

Security
High Availability
Scalability
Durability
And much more..

In this post, we will typically focus on Security, by making a simple infrastructure for web application on AWS.

The EC2 instance inside the public subnet is a web server, accepting traffics from the internet. RDS instance in the private subnet is attended to be only used by the EC2 instance in the public subnet. It only needs to be able to connect to EC2, but not the internet. So we will create a private subnet and put RDS instance inside it, preventing access from the internet at the root.

We will proceed with this implementation in the following order.

Creating VPC and subnets (public, private)
More VPC configurations
Creating and configuring EC2 instance as a web server application
Creating RDS instance
Validating implementation

Creating VPC and subnets (public, private)

First and foremost, we must create a VPC where our AWS resources will reside in. So let's create a VPC named Demo-VPC, with IPv4 CIDR set as 10.0.0.0/16.
After that, we have to create two subnets of that VPC, where one is public and the other is private. For the public subnet, let's set IPv4 CIDR as 10.0.1.0/24, and for the private subnet, let's set IPv4 CIDR as 10.0.2.0/24.
We also have to create another subnet in another availability zone within the same region than two other subnets we previously created. We have to create this subnet since RDS requires two different subnets created in different availability zones. Let's set its IPv4 CIDR as 10.0.3.0/24.

More VPC configurations

For our VPC to connect with internet, we have to create and attach an Internet Gateway to our VPC. Simply head over to VPC console, and create and attach the Internet Gateway to the VPC.
After that, we have to edit Route Tables for this VPC, to allow all traffics outside of this VPC to be delivered over the internet. Go to VPC console, Route tables section and let's add another rule like below.
- Destination: 0.0.0.0/0
- Target: Internet Gateway we created. (In the form of igw-XX)

Creating and configuring EC2 instance as an web server application

Now we have to create an EC2 instance which will reside in the public subnet, so that it can allow and communicate with traffic over internet.
Choose the correct VPC and public subnet we created, and for the Security Group, we should add two inbound rules.
- Type: SSH, Source: 0.0.0.0/0: To allow ssh access to EC2 instance. It is obviously not recommended to allow ssh access from everywhere, but we will set it as 0.0.0.0/0 to simplify the process.
- Type: Custom TCP, Port: 8080, Source: 0.0.0.0/0: Our sample server application will use port 8080, so we will pretend as if port 8080 is for HTTP or HTTPS.

Take a note of the Security Group ID created, since we have to use it later.

Creating RDS instance

Before creating RDS, we have to create another Security Group for it. Inbound rule to add is as below.
- Type: MySQL/Aurora, Source: sg-xxx(Security Group ID we created for EC2 instance.)
On the configuration page, we have to check that this RDS will reside in private subnet.
- VPC: VPC we created.
- Public Access: No
- VPC Security Group: Security Group we just created for RDS.
After creating the instance, navigate to RDS -> Databases and select the instance we just created. We can see the Endpoint & Port section. Note that in Security section, Publicly accessible is marked as No.

Validating implementation

Now that we have implemented every requirements we first saw, it's time to validate it.
First, let's see if we can access the database from our local machine, using the specified RDS endpoint assigned by AWS.

As we can see, it failed since RDS resides in the private subnet of VPC, blocking all traffics originated from the internet.
Now let's see if our server application on EC2 is working as expected. I have created two endpoints in this sample code, one for creating a user and the other for reading user information by an ID.
As in the two screenshots below, it is proven that EC2 instance can communicate with RDS instance.
Creating User API - this will perform INSERT query on RDS.
Reading User API - this will perform SELECT query on RDS.

Wrapping up, we successfully implemented a web application which meets the minimum security criteria. It is always important to think in the perspective of security, when designing and implementing software architecture, and blocking access from the internet at the root using private subnets in VPC can be the first step to implement it.

Thank you for viewing this article. If you have any questions or feedbacks, please leave a comment or contact me!

Consuming SQS Messages with Lambda(TypeScript)

Roy Ra — Mon, 04 Apr 2022 07:41:46 +0000

Amazon SQS(Simple Queue Service) is a fully managed message queuing service by AWS that enables you to lower the degree of coupling between services(ex. mircoservices), distributed systems, and serverless applications.

There are a lot of ways to produce message to SQS, but in this post we will focus on consuming(receiving) these messages with a lambda function which is written in TypeScript.

Let's assume we already have our SQS ready, and we only need to write the lambda code to consume the messages in this queue.

Below are the technical stacks we will use.

Serverless
TypeScript
AWS Lambda

We will go through this implementation in three sections,

Creating IAM User for managing Lambda.
Writing some codes.
Deploying, and connecting with SQS!

Creating IAM User for managing Lambda.

We can easily deploy Lambda functions using Serverless framework. We'll see the deployment script later, but let's focus on the IAM user who will manage the Lambda function we make here.

Serverless deploys our Lambda function, which is zipped and stored to S3, and all of the logs created by the Lambda function will remain in CloudWatch.

So we need permissions for below AWS Services.

Lambda
S3
CloudWatch

But since this Lambda function will receive and consume messages from SQS, this IAM user also needs a permission to receive message from SQS, which is sqs:ReceiveMessage.

Writing some codes!

Let's start coding the Lambda function.
Here, we want our Lambda function to receive message from SQS, and send that message to a specific channel in our Slack workspace.

In case you don't know how to make slack apps, you can create a Slack App and get OAuth Token and clientSecret by following this great article here.

Now that you have your OAuth Token and clientSecret, let's start coding!

First we have to add dependencies below to our fresh new project.

"dependencies": {
    "@slack/bolt": "^3.11.0",
    "aws-lambda": "^1.0.7",
    //...
}

Now we have to write a handler function, which will handle the event(receiving message).

const OAUTH_TOKEN = "slack app oauth token";
const SIGNING_SECRET = "slack app signing secret";
const CHANNEL = "slack channel name including #"

const consumeSqsMessage: Handler = async (event: SQSEvent) => {
  const app = new App({
    token: OAUTH_TOKEN,
    signingSecret: SIGNING_SECRET,
  });
  const message = event.Records[0]?.body;
  await sendMessageToSlack(app, message);
};

const sendMessageToSlack = async (app: App, message: string) => {
  try {
    await app.client.chat.postMessage({
      token: OAUTH_TOKEN,
      channel: CHANNEL,
      text: message,
    });
    console.log("Sent!");
  } catch (error) {
    console.log("Failed to send message. error: ", { error });
  }
};

export { consumeSqsMessage };

Deploying, and connecting with SQS!

Now that we have our Lambda function ready, lets' take a look at the serverless deployment script.

service: <ServiceName>
plugins:
  - serverless-plugin-typescript

provider:
  name: aws
  runtime: nodejs14.x
  state: active
  region: <aws region>

functions:
  notifySQSMessageToSlack:
    handler: src/functions.consumeSqsMessage
    timeout: 30

Note that the timeout value of Lambda functions should be lower than or equal to the visibility timeout of SQS.

Now that we have our Serverless Deployment script ready, we can deploy this to AWS Lambda using this script with Github Actions.

Finally, all we have to do is configure SQS to handle this Lambda function whenever a new message comes in.
We can do this very easily, just go to SQS Console and find Lambda triggers tab below! Click Configure Lambda function trigger, and we'll see the Lambda function we just deployed.

That's it! We just made a lambda function which consumes messages from SQS, and sends it to a Slack channel.

Thank you for viewing this article. If you have any questions or feedbacks, please leave a comment or contact me!