How I deployed my first project for my devops portfolio: EC2 health check and monitoring and Conclusion

So this is the final part where I implement the monitoring of my instance.

Aim of my monitoring

I wanted to monitor my instances disk usage, cpu and ram.

Thoughts and implementations of monitoring

At first I thought of using garafana and prometheus for monitoring as I implemented it in my company. Even though I don't have much idea but very basics of how to collect metrics and view it in grafana dashboard. Then expose the service via guest user to the www. I prepared node exporter into my dockerfiles in Database pod and both app pods and even one for the EC2 but then something struck my mind. WHY??? Why do I have to go this far to monitor. All I want is simple monitor the aim CPU, RAM and Disk usage that is all. Then I rolled back to unix commands and that was the answer.

Commands I settled for

df -h | tee test.txt
top -bn 1 | awk 'NR >= 1 && NR < 6 {print}' | tee test.txt
mpstat -T | tee test.txt

Yes I know top command shows me the cpu but I just kept the mpstat just because I felt like having 2 cpu metrics that I can make an estimated average of it.

Challenge is to send myself the metrics

I choose one by EMail, SMS, and in Discord channel. First is I made a separate python script exclusively for discord. I made a virtual env using venv module and pip installed discord-webhook and created a webhook in my server and used the url and successfully sent the message.

When I first sent the message it was not exactly like this as I improved it once time went by.

Then I came across second challenge SMS and EMAIL

At first I thought of having a personal smtp server and send it but then I found out it is blocked by all ISPS across the world to prevent spams then I thought of a service that does the very same thing SNS. I watched the web and created a basic SNS topic from dashboard and created a mail subscription and an SMS and published messages and wallah as it is working I quickly terraformed it.
Second challenge is to now publish a message from the EC2 instance.
I came across this website and wrote the initial script in python with the access keys in it for the testing and it was working fine. Then I created a role manually and assigned it to the instance and then tried it without the access keys and it was working fine again. The Snag is that when I did it using terraform it wasn't reflecting in the EC2 instance. Thing is after making the role I couldn't assign the role in EC2 from the console. So I went on the hunt why it was not working via terraform and working if created manually via dashboard. Upon inspection of both the roles side by side I saw something different from the dashboard created one with terraform.

I saw instance-profile/RoleName and terrafrom created role/RoleName at the ARN. Now I understood that I merely created normal role profile, but I need to assign an instance-profile to it and hence the hunt begun and I modified my terraform for it and it worked like a charm this time.

I made a custom policy using aws official policy generator.

main.tf

# ----------------------IAM ROLES & POLICIES ----------------

# AWS IAM role creation
data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

# AWS IAM role creation completed
resource "aws_iam_role" "sns-trigger-ec2" {
  name               = "EC2snsPublish"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

# Policy creation in json
data "aws_iam_policy_document" "sns-publish-policy" {
  statement {
    effect = "Allow"
    actions = ["sns:Publish"]
    resources = [
          "arn:aws:sns:ap-south-1:058264131778:ec2-health-topic-email",
          "arn:aws:sns:ap-south-1:058264131778:ec2-health-topic-sms"
        ]
    }
}

# Policy creation getting in json
resource "aws_iam_policy" "sns-publish-policy" {
  name        = "sns-publish-policy"
  description = "SNS Publish only policy"
  policy      = data.aws_iam_policy_document.sns-publish-policy.json
}

# Attach the newly created policy to the newly created IAM role
resource "aws_iam_role_policy_attachment" "sns-policy-attach" {
  role       = aws_iam_role.sns-trigger-ec2.name
  policy_arn = aws_iam_policy.sns-publish-policy.arn
}

# Create an instance profile for the IAM role
resource "aws_iam_instance_profile" "sns-policy-attach" {
  name = "ec2_instance_profile"
  role = aws_iam_role.sns-trigger-ec2.name
}

# -----------------------------------------------------------

# SNS Topic creation and customization
resource "aws_sns_topic" "user_updates_email" {
  name = "ec2-health-topic-email"
  display_name = "EC2 Health topic email"
}

#resource "aws_sns_topic" "user_updates_sms" {
#  name = "ec2-health-topic-sms"
#  display_name = "EC2 Health topic sms"
#}

# Creating subscription
resource "aws_sns_topic_subscription" "user_update_email"{
    topic_arn = aws_sns_topic.user_updates_email.arn
    protocol = "email"
    endpoint = "[e-mail]"
}

#resource "aws_sns_topic_subscription" "user_update_sms"{
#    topic_arn = aws_sns_topic.user_updates_sms.arn
#    protocol = "sms"
#    endpoint = "+91-------"
#}

Final part

I created a python script which creates the body and using webhook to send the message to discord channel of mine and one EMAIL to me via SNS and all run using virtual env and cronjobed it.

healthupdate.py

from os import system as sys
import boto3
from discord_webhook import DiscordWebhook

# Utility class
class Utility:

    commands = {
      "diskformat" : "df -h | tee test.txt",
      "cpuram" : "top -bn 1 | awk 'NR >= 1 && NR < 6 {print}' | tee test.txt",
      "cpu" : "mpstat -T | tee test.txt",
      "removetemp" : "rm temp.txt test.txt"
    };


    filename = [
      "temp.txt",
      "test.txt"
    ];

    def getmessage(command):
      sys(command);

      with open(Utility.filename[1], 'r') as file:
        data = file.read();

      with open(Utility.filename[0], 'w') as file:
        file.write("```

");
        file.write(data);
        file.write("

```");

      with open(Utility.filename[0], 'r') as file:
        data = file.read();

      sys(Utility.commands["removetemp"]);

      return data;

    def __init__(self):
      pass;

# aws SNS
class SNS:
    def __init__ (self, region):
      self.sns_client = boto3.client('sns', region_name = region);

    def publishmail(self, message, subject):
      self.sns_client.publish(TopicArn="arn:aws:sns:ap-south-1:058264131778:ec2-health-topic-email", Message = message, Subject = subject);

    def publishsms(self, message):
      self.sns_client.publish(TopicArn="arn:aws:sns:ap-south-1:058264131778:ec2-health-topic-sms", Message = message);

# Discord Webhook
class Discord:
    def __init__ (self, DWebhook):
      self.webhookurl = DWebhook;

    def trigger(self, message, uname):
      self.uname = uname;

      self.webhook = DiscordWebhook(url = self.webhookurl, username = self.uname, content = message)
      self.webhook.execute();


body = "";
subject = "Daily instance health check";

message = Utility.getmessage(Utility.commands["diskformat"]);
body += "**Disk Space data**\n" + message + '\n';

message = Utility.getmessage(Utility.commands["cpuram"]);
body += "**Top command output**\n" + message + '\n';

message = Utility.getmessage(Utility.commands["cpu"]);
body += "**CPU data mpstat output**\n" + message + '\n';


# Discord webhook triggers
webhook = Discord("https://discord.com/api/webhooks/1400674337966522399/gfdgdfgdgdfg----------yjfLwc");
webhook.trigger("# Daily instance health check\n\n" + body, "Doctor CPU");

# aws sns triggers
sns = SNS('ap-south-1');
sns.publishmail(body, subject);

I ended up removing SMS as EMAIL and discord is enough to have my eyes on it.

FAQ

In this project of mine I used DB as pod not RDS. That must have been question of many.
I used it as a pod because I wanted to understand cross pod communication as a developers perspective and also RDS is expensive for me currently. That is the reason why I didn't have the DB running in the Instance itself and choose the pod.

My Architecture evolution screenshots

Conclusion

Past 1.5 years of me planning and developing the project and its architectures was a journey to me. My experience across projects in my office, different architectures I changed my deployment architecture multiple times and each iteration was more improved one and many times I scrapped old architectures as I aimed to be able to set up the production as fast as possible and as automated as possible with minimal intervention of mine and if want new addition I can expand it without any problem.
I understood one thing while deploying and creating architecture. It is never about what top end tools we use or what top trending services we need. It all boils down to needs. What and how the budget or the application will be running maybe a few change in app to be compatible with this style of deployment hence a lot of communication between teams. Sometimes a legacy tool can help a lot with automation.
I could be wrong about many things here since this is my first company and my first experience in this field but after all I went through in office and here I came to this conclusion for now.