DEV Community: Akhil Ghatiki

AWS ECS Task deployment failed alert using Amazon EventBridge — Part 2 (Terraform code)

Akhil Ghatiki — Tue, 25 Oct 2022 16:33:34 +0000

Please read Part 1 for more context before you continue with this blog post

The Part 1 talks about event pattern rule to filter events of a task failure. In this blog post, lets take a look at the terraform code for this implementation

This blog post assumes you are quite familiar with the Terraform and you have some hands on experience with it.


 terraform
resource "aws_cloudwatch_event_rule" "ecs_task_failure" {
  count   = length(local.ecs_services_list)
  name          = "${local.ecs_services_list[count.index]}-ecs-task-failed"
  description   = "Rule to monitor failures in ecs tasks"
  event_pattern = <<PATTERN
                  {
                    "source": ["aws.ecs"],
                    "detail-type": ["ECS Task State Change"],
                    "detail": {
                      "group": ["service:service-name"],
                      "stoppedReason": [{
                        "anything-but": {
                          "prefix": "Scaling activity initiated by (deployment"
                        }
                      }],
                      "lastStatus": ["STOPPED"]
                    }
                  }
                  PATTERN
  is_enabled    = true
}

The above snippet creates the event rule as per the pattern we discussed in Part 1.

And you can create the target to this event rule as below.



resource "aws_cloudwatch_event_target" "cloudwatch_alarms" {
  arn       = <<your sns arn>>
  target_id = "service-name-ecs-task-failed-event-target"
  rule      = aws_cloudwatch_event_rule.ecs_task_failure.name
  input     = "{\"Subject\":\"ALARM: ECS task failed - service-name\",\"AlarmDescription\":\"ECS task failed\"}"
}

Now, you can have your lambda triggered for the events coming from the SNS and the lambda can alert the teams (any communication medium that you use)

God Speed !!

AWS ECS Task deployment failed alert using Amazon EventBridge

Akhil Ghatiki — Mon, 17 Oct 2022 12:56:18 +0000

Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances.

We have recently decided to move our deployments to ECS. As, the uncle Ben said — With great power, comes great responsibility — setting up the monitoring and alerts for this new implementation was quite interesting and this blog post deep dives into one specific alarm that we configured i.e., How do you get alerted when an ECS task fails during the deployment ?

We had microsoft teams webhook in place, to which the alerts have to be sent when a failure / error occurs. This alert dump can be any other application in your case.

We have leveraged Amazon EventBridge rules for creating alerts when ever an ECS task fails. This gives us the opportunity to check what went wrong. Below is a high level block diagram of the implementation.

And yes, I know what your eyes are looking for — The event rule pattern. Here is the pattern we used.

{
  "detail": {
    "group": ["service:<<name of your service>>"],
    "lastStatus": ["STOPPED"],
    "stoppedReason": [{
      "anything-but": {
        "prefix": "Scaling activity initiated by (deployment"
      }
    }]
  },
  "detail-type": ["ECS Task State Change"],
  "source": ["aws.ecs"]
}

gist : https://gist.github.com/akhil-ghatiki/1251a54da8eaaca3c20f5322f5106319.js

You can use any pattern for your rule. In the above example, we are filtering it based on the group key whose value is the service name. So, any event with the service name will be sent to the SNS. Below is a sample event that can help you with other attributes you might need for your rule pattern.

{
  "version": "0",
  "id": "XXXXXXXXXXXXX",
  "detail-type": "ECS Task State Change",
  "source": "aws.ecs",
  "account": "XXXXXXXXX",
  "time": "2022-10-10T10:06:29Z",
  "region": "XXXXXXXX",
  "resources": [
    "arn:aws:ecs:XXXXXXXX:XXXXXXXXX:task/XXXXXXXXXX/XXXXXX"
  ],
  "detail": {
    "attachments": [
      {
        "id": "XXXXXXXXXXXXX",
        "type": "sdi",
        "status": "DELETED",
        "details": []
      },
      {
        "id": "XXXXXXXXXXXXX",
        "type": "eni",
        "status": "DELETED",
        "details": [
          {
            "name": "subnetId",
            "value": "XXXXXXXX"
          },
          {
            "name": "networkInterfaceId",
            "value": "XXXXXXXX"
          },
          {
            "name": "macAddress",
            "value": "XXXXXXXX"
          },
          {
            "name": "privateDnsName",
            "value": "XXXXXXXX"
          },
          {
            "name": "privateIPv4Address",
            "value": "XXXXXXXX"
          }
        ]
      },
      {
        "id": "XXXXXXXXXXXXX",
        "type": "elb",
        "status": "DELETED",
        "details": []
      }
    ],
    "attributes": [
      {
        "name": "ecs.cpu-architecture",
        "value": "XXXXXXXX"
      }
    ],
    "availabilityZone": "XXXXXXXXa",
    "clusterArn": "arn:aws:ecs:XXXXXXXX:XXXXXXXXX:cluster/XXXXXXXXXX",
    "connectivity": "CONNECTED",
    "connectivityAt": "2022-10-10T10:05:38.02Z",
    "containers": [
      {
        "containerArn": "arn:aws:ecs:XXXXXXXX:XXXXXXXXX:container/XXXXXXXXXX/XXXXXX/XXXXXXXXX",
        "lastStatus": "STOPPED",
        "name": "XXXXXXX",
        "image": "XXXXXXXX.dkr.ecr.XXXXXXXX.amazonaws.com/XXXXXXXX/XXXXXXX:XXXXXX",
        "runtimeId": "XXXXXX-1979092248",
        "taskArn": "arn:aws:ecs:XXXXXXXX:XXXXXXXXX:task/XXXXXXXXXX/XXXXXX",
        "networkInterfaces": [
          {
            "attachmentId": "XXXXXXXXX",
            "privateIpv4Address": "XXXXXXX"
          }
        ],
        "cpu": "0"
      }
    ],
    "cpu": "512",
    "createdAt": "2022-10-10T10:05:34.301Z",
    "desiredStatus": "STOPPED",
    "enableExecuteCommand": false,
    "ephemeralStorage": {
      "sizeInGiB": 20
    },
    "executionStoppedAt": "2022-10-10T10:05:46.497Z",
    "group": "service:XXXXXXX",
    "launchType": "FARGATE",
    "lastStatus": "STOPPED",
    "memory": "1024",
    "overrides": {
      "containerOverrides": [
        {
          "name": "XXXXXXX"
        }
      ]
    },
    "platformVersion": "1.4.0",
    "startedBy": "ecs-svc/XXXXXX",
    "stoppingAt": "2022-10-10T10:05:56.53Z",
    "stoppedAt": "2022-10-10T10:06:29.152Z",
    "stoppedReason": "CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref \"XXXXXXXX.dkr.ecr.XXXXXXXX.amazonaws.com/XXXXXXXX/XXXXXXX:XXXXXX\": XXXXXXXX.dkr.ecr.XXXXXXXX.amazonaws.com/XXXXXXXX/XXXXXXX:XXXXXX: not found",
    "stopCode": "TaskFailedToStart",
    "taskArn": "arn:aws:ecs:XXXXXXXX:XXXXXXXXX:task/XXXXXXXXXX/XXXXXX",
    "taskDefinitionArn": "arn:aws:ecs:XXXXXXXX:XXXXXXXXX:task-definition/XXXXXXX:XXX",
    "updatedAt": "2022-10-10T10:06:29.152Z",
    "version": 4
  }
}

gist : https://gist.github.com/akhil-ghatiki/e2654a551d6989ed0cb652318357f20b.js

God Speed !!

One point stop for Installing SonarQube server in an AWS EC2 instance

Akhil Ghatiki — Sun, 29 May 2022 15:38:55 +0000

Writing clean code is an art. It is a pathway towards software craftsman /craftswoman ship. But sometimes not every one is well equipped with this art. As the saying goes it takes 10,000 hours to master a skill. So, we need something that will enforce the developers of the team to write the code in a cleaner and secure way.

SonarQube is an amazing tool that helps in this regard. It helps us detect the code smells, potential bugs and security vulnerabilities in your code. It can integrate with your existing workflow to enable continuous code inspection across your project branches and pull requests.

SonarQube can be run in you local machine or as a docker container or can be hosted in a server. When the requirement is for having it for the whole team and planning to integrate in the CI pipeline, you have the only option of hosting it in the server. In this write out we will be walking through hosting it in an AWS EC2 instance.

First things first:

The reader is expected to have a very basic understanding of cloud, servers, CI/CD, CLI , database.Familiarity with AWS ecosystem is good to have. Trying to keep it as simple as possible even if the reader is new to these terms.

Spawning up an AWS EC2 instance:

Create a AWS EC2 instance. Just putting out the screen shot of configuration as the intention of this write out is not inclined more towards creating EC2 instance.

Important note : Be careful while choosing the Amazon Machine Image(AMI). Choose Amazon Linux AMI 2018.03.0 (ami-005956c5f0f757d37). This image includes PostgreSql which we are going to use in further steps.

Download the “.pem file” while creating the instance and save it in your machine. This is required to ssh into the instance in the further steps.
Installing SonarQube server in the above created instance:

SSHing into the instance :

Connect to the EC2 instance using a secure shell. Thanks to the jumpbox server that lets us do this.

ssh -i <<path to your .pem file>> ec2-user@<<ip address of your EC2>>

Setup PostgreSql : We have to setup a database for sonarqube to save the report analysis. This helps in maintaining the report versions as well. The supported databases include Oracle, Microsoft SQL server and PostgreSQL. We will be going ahead with PostgreSQL here.

sudo yum install postgre96 postgresql96-server

Note : Install the version higher than 9.3. The default version that gets installed if you do not mention the version in command will be 9.3 and this is not supported by SonarQube.

If you are going ahead with different AMI other than the above mentioned, checkout the supporting postgresql versions in this link.

The below commands are self explanatory. Run them in this order:

sudo service postgresql96 initdb //creating a new postgre cluster

sudo service postgresql96 start  // starting the postgre sevice

sudo passwd postgres  // changing the password of the default server

su — postgres // login with the new password

psql // start the shell

Create a user and database for sonar:

CREATE USER sonar WITH ENCRYPTED PASSWORD ‘sonar_password’;

CREATE DATABASE sonarqube;

GRANT ALL PRIVILEGES ON DATABASE sonarqube to sonar

Now we need to edit the authentication modes of postgresql.

Open the file in below path in vim :

sudo vi /var/lib/pgsql96/data/pg_hba.conf

Change the modes as shown below: (Highlighted in block letters)

TYPE DATABASE USER ADDRESS METHOD
# “local” is for Unix domain socket connections only
local all sonar md5

local all all peer

# IPv4 local connections:

host all all 127.0.0.1/32 md5

# IPv6 local connections:

host all all ::1/128 md5

Restart the postgreSQL server to have the new authentication modes applied.

sudo service postgresql96 restart
Installing OpenJDK11

We need JDK 11 or higher to run SonarQube 7.9 or more. The AWS EC2 instance comes with Amazon’s corretto distribution of java. But better stick to openJDK.

Download the zip in the instance:

curl -O https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz

Extract the zip:

tar zxvf openjdk-11.0.1_linux-x64_bin.tar.gz

Move the folder to your required location (This optional)

sudo mv jdk-11.0.1 /usr/local/

Change the access of the JDK folder:

sudo chmod -R 755 /usr/local/jdk-11.0.1

Add java home path:

export JAVA_HOME=/usr/local/jdk-11.0.1export PATH=$JAVA_HOME/bin:$PATH

check the java version:

java -version This should show up the installed java version in the instance
Installing the SonarQube

We are going to download the binaries and use them to install.

Note: Please be careful about the edition that you are going to install. Sonarqube comes in community, developer and enterprise editions.

We are going to use community edition in this write out. More details can be found here in this link. Right click on the respective download button and choose copy link location if you aim to install other edition.

Download the binaries:

wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-8.3.1.34397.zip

Unzip the binaries:

unzip sonarqube-8.3.1.34397.zip

Move the file path to desired location:

sudo mv -v sonarqube-8.3.1.34397/* /opt/sonarqube

Create a group for sonar users:

sudo groupadd sonar

Create a user:

sudo useradd -c “Sonar System User” -d /opt/sonarqube -g sonar -s /bin/bash sonar

Activating the user with password:

sudo passwd sonar

Add sonar user to EC2 user:

sudo usermod -a -G sonar ec2-user

Exit from ec2-user bash and reconnect to the server to load new group for ec2-user

Change the ownership of all the sonar files to the sonar user:

sudo chown -R sonar:sonar /opt/sonarqube

Change file access privileges:

sudo chmod -R 775 /opt/sonarqube

Configuring the SonarQube server:

Set run as user:

sudo vi /opt/sonarqube/bin/linux-x86–64/sonar.sh

Find the line RUN_AS_USER, uncomment it by removing the pound sign and enter sonar user as the value below

RUN_AS_USER=sonar

Modify the sonar.properties to add the details of postgreSQL:

sudo vi /opt/sonarqube/conf/sonar.properties

Add jdbc user name and password:

sonar.jdbc.username=sonar sonar.jdbc.password=sonar_password

Uncomment Postgres driver property.Remove current schema param if you are not using a custom schema for SonarQube database

sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube

set a path for dedicated volume with fast io for Elasticsearch data storage

sonar.path.data=/path/to/fast/io/volume/data sonar.path.temp=/path/to/fast/io/volume/temp

Start the server:

$/opt/sonarqube/bin/linux-x86–64/sonar.sh start

If you want to log the server logs in the terminal go for:

$/opt/sonarqube/bin/linux-x86–64/sonar.sh console

Yikes !! Your sonar should be up now. You can access the sonarQube UI at

http://<<EC2 instance public ip>>:9000/sonarqube

Considerations:

We have installed the PostgreSQL within the instance. This kind off makes it a stateful and introduces a single point of failure of the database if the EC2 instance goes down. Consider using AWS RDS to to configure in place of internal DB like we did here. This will help in persisting even if the instance goes down. Once the new instance connects back to db in AWS RDS, you have everything back.
Conclusion:

Tried to cover all the mistakes which were encountered while installing SonarQube in EC2. Hope this helps someone not to recommit them!

God Speed!!!

References used to write this:

https://stackoverflow.com/questions/44658510/setting-up-sonarqube-on-aws-using-ec2

https://medium.com/@mkaschke/how-to-install-sonarqube-on-aws-ec2-20e3a2fec634

https://medium.com/@harith.sankalpa/a-complete-guide-to-install-sonarqube-server-in-amazon-linux-ed1fee227d81

https://docs.sonarqube.org/latest/setup/install-server/

This article is originally published on https://akhil-ghatiki.github.io/#/sonar-aws

Code smell Katas — Graceful exception handling

Akhil Ghatiki — Sat, 28 May 2022 17:17:23 +0000

In computer programming, a code smell is any characteristic in the source code of a program that possibly indicates a deeper problem. Determining what is and is not a code smell is subjective, and varies by language, developer, and development methodology. — wikipedia

Code smells are often symptoms of poor design or implementation choices.

Let’s take up a Code Smell kata in this blog post.

Consider a business scenario which has to make a HTTP call to a service to update a document if it exists or create new document if it doesn’t exist.

Below class has only one public method createOrupdateDocument and all other methods are private and have no visibility to other classes.

public class CodeSmellHttpClient {
  public static final Logger LOG = LoggerFactory.getLogger(CodeSmellHttpClient.class);
  private final String baseUrl = "https://some.service.com/documents/";

  private HttpClient httpClient = HttpClients.createDefault();

  private String bearerToken = "some bearer token";

  public void createOrUpdateDocument(String requestBody, String documentId) {
    if (this.isDocumentExistingWithId(documentId)) {
      this.updateDocument(requestBody, documentId);
    } else {
      this.createDocument(requestBody);
    }
  }

  private void createDocument(String requestBody) {
    try {
      HttpPost httpPostRequest = new HttpPost(baseUrl);
      StringEntity requestEntity = new StringEntity(requestBody, ContentType.APPLICATION_JSON);

      httpPostRequest.setEntity(requestEntity);
      httpPostRequest.addHeader(getAuthorizationHeader());

      HttpResponse response = httpClient.execute(httpPostRequest);
      LOG.info(response.toString());
    } catch (IOException e) {
      LOG.error(e.getMessage());
      e.printStackTrace();
    }
  }

  private void updateDocument(String requestBody, String documentId) {
    final String updateDocumentUrl = baseUrl.concat(documentId);

    HttpPatch httpPatchRequest = new HttpPatch(updateDocumentUrl);
    StringEntity requestEntity = new StringEntity(requestBody, ContentType.APPLICATION_JSON);

    httpPatchRequest.setEntity(requestEntity);
    httpPatchRequest.setHeader(getAuthorizationHeader());
    try {
      HttpResponse response = httpClient.execute(httpPatchRequest);
      LOG.info(response.toString());
    } catch (IOException e) {
      LOG.error(e.getMessage());
      e.printStackTrace();
    }
  }

  private boolean isDocumentExistingWithId(String documentId) {
    final String getTemplateUrl = baseUrl.concat(documentId);

    HttpGet httpGetRequest = new HttpGet(getTemplateUrl);
    httpGetRequest.setHeader(getAuthorizationHeader());
    HttpResponse response = null;

    try {
      response = httpClient.execute(httpGetRequest);
      LOG.info(response.toString());
    } catch (IOException e) {
      LOG.error(e.getMessage());
      e.printStackTrace();
    }

    if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
      LOG.info(documentId + "already exists");
      return true;
    }

    LOG.info(documentId + "does not exists");
    return false;

  }

  private Header getAuthorizationHeader() {
    return new BasicHeader(HttpHeaders.AUTHORIZATION,
        "Bearer " + bearerToken);
  }

}

Above code : https://gist.github.com/akhil-ghatiki/98339ea96a6a39876aa3745f9cf185cb#file-codesmell1-java

Let’s deviate a bit from code smells for now and take a look at above class. Why should this class take care about creating HttpClient ? How can you test this class for unit testing ? What can you test for unit testing ?

Consider moving the creation of HttpClient into a different class and passing it to this class as constructor parameter like below or as an autowired bean if you are using Spring framework.

private HttpClient httpClient;

public CodeSmellHttpClient(HttpClient httpClient, String bearerToken) {
  this.httpClient = httpClient;
  this.bearerToken = bearerToken;
}

This will help you in mocking the creation of HttpClient and verifying the calls of httpClient.execute(httpGetRequest), httpClient.execute(httpPostRequest),httpClient.execute(httpPatchRequest) respectively when get, post and patch requests are made through the private methods.

ProTip: If one is following test driven development, one will not end up into most of the code smells like in above implementation. Test driven development forces you to write cleaner code in many ways. All hail TDD !

Now, just look into isDocumentExistingWithId(String documentId) method in above code, take a minute and try listing out the code smells in it.

What happens if there is an exception ?

Everything works smooth in a happy scenario here. But in case of exception, the method catches the exception and logs it, and then the if statement after the catch is checking the status code in response. One is not very far away form seeing a huge red log lines of NullPointerException in the console when an exception occurs.

Alright, now lets make some changes to tackle the exception scenario where there is an exception in this method. How about changing the method to below ? — moving the if condition into the try block.

private boolean isDocumentExistingWithId(String documentId) {  final String getTemplateUrl = baseUrl.concat(documentId);

  HttpGet httpGetRequest = new HttpGet(getTemplateUrl);
  httpGetRequest.setHeader(getAuthorizationHeader());
  HttpResponse response = null;

  try {
    response = httpClient.execute(httpGetRequest);
    LOG.info(response.toString());    if(response.getStatusLine().getStatusCode()==HttpStatus.SC_OK){
      LOG.info(documentId + "already exists");
      return true;
    }
  } catch (IOException e) {
    LOG.error(e.getMessage());
    e.printStackTrace();
  }
  LOG.info(documentId + "does not exists");
  return false;
}

This seems to mitigate the issue we were discussing a while ago. But wait, did we look into the caller method of this method ? Have a look into createOrUpdateDocument(String requestBody, String documentId) in the code example. (Sorry for making you scroll the page so many times :p)

The createDocument(newRequest) gets executed because we are returning false when the exception is caught in the isDocumentExistingWithId(String documentId) and this is invoking the createDocument method. We don’t want that to happen.

Take a minute now and think how can this be handled ! Don’t scroll down to answer before giving it a thought.

.
If throwing the exception is resonating in the corridors of your brain, yes, you are right !

Throw the exception in isDocumentExistingWithId(String documentId) instead of handling it.Let the caller method decide what to do with that exception. This is what I call handling the exception gracefully. Treat the exception like first class citizens in your implementations. Don’t ignore them. Functional programming does amazing job in treating exceptions as first class citizens. Will not discuss more about functional programming as that is way beyond the scope of this blog post.

ProTip: Treat exceptions like first class citizens in your implementations.

Below is the implementation of throwing the exception as discussed.

public class CodeSmellHttpClient {

  public static final Logger LOG = LoggerFactory.getLogger(CodeSmellHttpClient.class);
  private final String baseUrl = "https://some.service.com/documents/";

  private HttpClient httpClient;

  private String bearerToken = "some bearer token";

  public CodeSmellHttpClient(HttpClient httpClient, String bearerToken) {
    this.httpClient = httpClient;
    this.bearerToken = bearerToken;
  }

  public void createOrUpdateDocument(String requestBody, String documentId) {
    boolean isDocumentExisting;
    try {
      isDocumentExisting = isDocumentExistingWithId(documentId);
    } catch (IOException e) {
      LOG.error("Unable to get template details", e);
      throw new RuntimeException("Unable to get template details", e);
    }
    if (isDocumentExisting) {
      this.updateDocument(requestBody, documentId);
    } else {
      this.createDocument(requestBody);
    }
  }

  private void createDocument(String requestBody) {

    try {

      HttpPost httpPostRequest = new HttpPost(baseUrl);
      StringEntity requestEntity = new StringEntity(requestBody, ContentType.APPLICATION_JSON);

      httpPostRequest.setEntity(requestEntity);
      httpPostRequest.addHeader(getAuthorizationHeader());

      HttpResponse response = httpClient.execute(httpPostRequest);
      LOG.info(response.toString());
    } catch (IOException e) {
      LOG.error(e.getMessage());
      e.printStackTrace();
    }
  }

  private void updateDocument(String requestBody, String documentId) {
    final String updateDocumentUrl = baseUrl.concat(documentId);

    HttpPatch httpPatchRequest = new HttpPatch(updateDocumentUrl);
    StringEntity requestEntity = new StringEntity(requestBody, ContentType.APPLICATION_JSON);

    httpPatchRequest.setEntity(requestEntity);
    httpPatchRequest.setHeader(getAuthorizationHeader());
    try {
      HttpResponse response = httpClient.execute(httpPatchRequest);
      LOG.info(response.toString());
    } catch (IOException e) {
      LOG.error(e.getMessage());
      e.printStackTrace();
    }
  }

  private boolean isDocumentExistingWithId(String documentId) throws IOException {
    final String getTemplateUrl = baseUrl.concat(documentId);

    HttpGet httpGetRequest = new HttpGet(getTemplateUrl);
    httpGetRequest.setHeader(getAuthorizationHeader());

    HttpResponse response = httpClient.execute(httpGetRequest);
    LOG.info(response.toString());

    if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
      LOG.info(documentId + "already exists");
      return true;
    }
    LOG.info(documentId + "does not exists");
    return false;
  }

  private Header getAuthorizationHeader() {
    return new BasicHeader(HttpHeaders.AUTHORIZATION,
        "Bearer " + bearerToken);
  }

}

Above code : https://gist.github.com/akhil-ghatiki/37594c5852e7293a2d97b2e0028aa6b3#file-codesmell2-java

Treat the exceptions good and they will treat you better.

I have intentionally left few code smells unaddressed. Drop them in the comments if you figure out any.

This article is originally published on: https://akhil-ghatiki.github.io/#/code-smell-katas-graceful-exception-handling

Debugging the performance issue of a distributed system

Akhil Ghatiki — Sat, 30 Apr 2022 05:00:55 +0000

Debugging the performance of a distributed system is always a pain. But on the other side of this coin, you get a lot of learning, knowing the system better and this helps in building an architectural mindset.

Recently, I have spent close to 2 weeks on debugging a performance issue of a distributed system our team was developing. I am sharing my learning from this period in this blog post.

First things first:

I assume the reader has some knowledge of micro services, distributed systems and how critical would a performance issue of a distributed could be. From here on, “system” or “DS” represents a distributed system.

Do not go gentle into that good night….. :

Read this poem by Dylan Thomas here (Its my favourite poem). But yeah, coming back to debugging, I would say DO GO gentle into “this” good night. Don’t just jump into the implementation and crave for the fix of the issue. This is the first mistake one can do. Without analyzing the existing implementation of the system, the developer just dives to fix it. When you know the system better and if you have a clear understanding of the performance issue, you can have a slight guess about where the issue could be and what could be the bottleneck of this system. So, first, analyze the existing system.

Capture the metrics of the existing system:

This is crucial. Unless you know how well/bad the existing system is performing, you can’t proceed further. Leverage the power of logs and profiles of the system to capture the metrics. These metrics could be about the I/O operations, thread pool profiling, publishing data to another system or subscribing data from another system or could be as simple as a small API call. Keep a note of memory usage and CPU usage of the system. It would be great if you are using a tool like Splunk to get insights from your logs. Dashboards built using the logs of a DS are helpful when you are dealing with this type of issue. They drastically reduce the time you put in. Invest time in developing dashboards (Ex: Splunk Dashboards) while developing itself. They will save a lot of time later. Proceed further once you are quite clear about the existing performance.

Are you messing with default configurations ?

When you are dealing with a distributed system, at times we configure a few parts of this system with default configurations. Let’s take a small use case. If you are using Kafka or RabbitMQ as a messaging broker in your system, in general, we go ahead with a few default configurations. For example, some configurations like fetch.min. bytes, max.poll.records, fetch.max.wait.ms etc of Kafka should be left to default configurations unless you are very sure that it is not your system that is the bottleneck for the performance. Default configurations are default for a reason, they are not just random figures. The team which developed Kafka might have tested it rigorously and came up with these defaults. So, think twice before changing the defaults.

Chalk out the architecture:

This helps a lot. Doesn’t your office have whiteboards? No big deal, grab a paper and sketch your DS. Try to include every minute detail of the system. The key aspects would be drawing out, which part of the system is communicating with which part. Include all the API calls happening, the messaging brokers if any. The more detailed your sketch is, the less time you are going to spend solving the issue. From this sketch mark out the points which could be affecting your system.

It’s not always the game of instance count:

Yes, when capturing the metrics of a DS. Stick on to one instance of the system . Spawning up multiple instances of the system to capture metrics might mislead the decision. Performance should always be talked per instance. So, fix the problem per instance before you scale up the count.

Divide and Conquer:

Here lies the key aspect of debugging the performance of a DS. Divide the whole system into smaller chunks (of course in your mind. That is why I asked to chalk it out in the previous section). Now, decommission one part at a time in your DS either by bringing down the service or just commenting out the code. Capture the metrics after that. Loop this in until you narrow down the problem to a particular part of your system. Now, conquer the issue.

God Speed !!!!