DEV Community: Gabriela Menocal

Simple Guide for analyzing web traffic for troubleshooting purposes

Gabriela Menocal — Wed, 29 Mar 2023 01:34:55 +0000

There are many tools out there that can help you to troubleshoot from free to paid tools which can help you identify where the problem is an help remediate, some examples are listed below:

Networking and traffic inspection tools:

Windows OS:

TCPView*
PsPing v2.1* *Important: from windows internals tools. Installation is required

Windows, Unix based, Mac OS:

WireShark
IPerf
Fiddler
Netstat
Nslookup
Solarwinds
Paessler (PRTG)
Much more

Every software and tool has its own pros and cons, and their selection will depend on scope, reason from troubleshooting and budget. In this article I will focus on a few of them that might help to get more concrete information where the problem is, whether you are in a cloud, hybrid and on-prem environment.

In one of my previous articles I discussed a little bit of the IPerf tool, which I believe is a great tool for testing connectivity and throughput between two points. Wireshark on the other hand I would say it is a network engineering tool par excellence, and there are many guides, documentation and videos that are super helpful so I decided that I will leave this tool out from my article for now. But I highly recommend using and getting familiar with both whether you are troubleshooting, learning concepts or only getting used to your service patterns, software or application's web traffic.
Before getting into the topic, I think is important to remind REST API codes (HTTP Status codes), so please review the reference below:

So in Summary, any 40X code is client problem, 50X code server problem, 30X redirections and 20X everything is alright :).

I've been trying different tools from paid to open source, in Linux and Windows environments. In this post, I decided to share some of the tools I've been using recently, which some use cases and examples.

PRGT:

While I was looking for a network traffic analyzer, I found Paessler, it was mentioned on most of the websites where tools are listed, and because I've never used it before I decided to try it.

It was convenient that they offer a trial option when you don't have a windows machine (which was my case), so having a sandbox was really easy and convenient in order to try it for the first time. Something t notice is that the downloadable version for Windows machine gives you a trial for 30 days in comparison with the hosted version which gives you 10 days.

Netstat:

This utility is largely used in Linux and Windows systems and will help you determine where the problem might be located or guide you through the troubleshooting process, some commands that I believe are good to keep under the sleeves are:

-netstat the simple command will show the transport protocol (Layer 4) state of the connection (ESTABLISHED, CLOSE_WAIT, FIN_WAIT, etc.) and foreign address

netstat –a –o –p tcp

-Other useful commands:

- 1. netstat –f –p tcp | findstr ESTABLISHED
- 2. netstat -na –o
- 4. netstat –a| findstr <_e.g. port number_>
- 5. netstat –a -o| findstr <_e.g.port number_>

where:
a => active connections
o => PID (Process ID)
p => protocol, ex. UDP, TCP.

TCPView

TCPView is a Windows program that will show you detailed listings of all TCP and UDP endpoints on your system, including the local and remote addresses and state of TCP connections.
It gives us a graphical interface, easy to understand and manage lightweight, and also can help you to check UDP and TCP traffic

References:

https://learn.microsoft.com/en-us/sysinternals/downloads/tcpview

How to choose the appropriate storage type in AWS ?

Gabriela Menocal — Thu, 21 Apr 2022 22:28:47 +0000

Choosing an appropriate storage is one of the most important decisions while either creating an app, migration or simply creating something completely new and unique.

Nowadays, we have multiple options of storage types to choose from, but when deciding you will need to consider the purpose for it. For example, will it be for mobile applications, archiving, sharing content, who is going to use the application and so on…

In this article I am not hoping to give the answer to all doubts related to this, but share a little bit of my experience on the field and share some tips that have helped me when I need to design about it.

First let’s briefly describe the most common file storages types and some of their main features of each of them:

File Storage	Block Storage	Object Storage
No Metadata	Limited Metadata	Metadata
High IO performance	Hierarchical System	Unique identifier
Write/Read efficiency	Limit in Scalability and performance.	Better for Read only applications and Distribution
Greater Storage efficiency	Protocols: NFS/CIFS, SMB, etc.	Protocols: HTTP, HTTPS, REST operation (GET, PUT, DELETE, etc.)

And some other factor in data management should be identified for our type of data: Structured or Unstructured data. The majority of the data we use and access everyday is normally unstructured, such as videos, images, emails, spreadsheets, word processing files, etc.

This is a big and deep topic, however this is out of the scope of this blog, but because I know there are people like me who would like to go further on this, I found these articles, which I believe are a great start:

So, going back to our main topic, as a rule of thumb, if the data can be displayed in a row, columns and/or SQL DB without or minimal modification, it means that most likely you are working with structured data.

Please do not forget that normally unstructured data requires more storage than structured data, therefore choosing the right storage is crucial to keep a good performance for the apps and services.

Now that we've discussed different data types, the next question arises, how can we use this knowledge to choose the right storage when using AWS?

In order to connect the two points, I would like to describe a little bit of the storage types offered by AWS. If we go to the AWS official documentation page we will notice that they offered the following (by the time this blog was written):

I highly recommend go to the official documentation to understand a little bit more each of these storages types, or read some of these blogs written by other members of AWS Community Builders, for example check these:

There are many more article explaining to this and different types of topics and services, please take a look in the channel
AWS Community Builders Channel.

So, I mapped the two points in the following table, which will give you a general idea of the type of data for which the storage is better suited for:

Storage Type	Block Storage	File Storage	Object Storage	Volume Based Storage	Tape Storage
Amazon S3	⛔	⛔	😃	⛔	⛔
AWS Backup	😃	😃	😃	😃	😃
Amazon EBS	😃	⛔	⛔	⛔	⛔
Amazon EFS	⛔	😃	⛔	⛔	⛔
AWS Elastic Disaster Recovery	NA	NA	NA	NA	NA
Amazon FSx	⛔	😃	⛔	⛔	⛔
Amazon S3 Glacier	⛔	⛔	😃	⛔	⛔
AWS Snow Family	NA	NA	NA	NA	NA
AWS Storage Gateway	⛔	😃	⛔	😃	😃

Please use as a reference for table 1:

😃 => This is the storage type.
⛔ => It is not suitable for it.

And I also mapped the Storage features with AWS Storage offering, to give you a general idea and it can be used as a reference:

Disclaimer: Not everything is written in stone, so please make sure to use this tables as reference and general guidelines only.

Storage Type	Metadata	High IO Performance	Distribution	Scalability	Data Modification
Amazon S3	😃🙌	😐	😃🙌	😃🙌	😃
AWS Backup	😃	😃	😃	😃	😃
Amazon EBS	⛔	😃🙌	⛔	😐	😃🙌
Amazon EFS	😐	😃	😃	😃	😃
AWS Elastic Disaster Recovery	NA	NA	NA	NA	NA
Amazon FSx	😐	😃🙌 (Lustre)	😃	😃	😃
Amazon S3 Glacier	😃🙌	⛔	😃	😃🙌	😐
AWS Snow Family	NA	NA	NA	NA	NA
AWS Storage Gateway	😃	😃	😃	😃	😃

For anyone who is curious about it, this is the Link for the Lustre Project: https://www.lustre.org/.

Please use as reference for table 2:

😐 => Possible but with some caveats or there are some limitations or restrictions.
😃 => It will work but there might be other options.
🙌 => This is what it was built for.
⛔ => it is not suitable for it.

I hope you find this post useful and please if you have any comments, do not hesitate in letting me know.

How to measure the real time network traffic ‘from’ and ‘to’ your EC2 instances

Gabriela Menocal — Sun, 13 Mar 2022 04:49:43 +0000

``When working with traffic and time sensitive applications and workflows, it is always a good practice to measure the actual available bandwidth for your application.
This will help you in various ways, some of them are listed below:

Define the baseline of the actual bandwidth available.
Identify if there are any bottlenecks, and for troubleshooting purposes.
The Data will help business decisions
Increase and improve the service and customer satisfaction.

Fortunately Amazon offers an extensive list of official resources documentation, knowledge base articles, blogs, videos among others, while using with its cloud products and services.
I am not intend going into a detailed description about the use of each of them but rather sharing my experience how those tools have helped me, in conjunction with open sources tools, to solve problems and come up with better solutions for my customers by gathering information needed to take decisions.

One of the tools, which I believe most of the networks engineers have used at some point of their career, is IPERF. which as per their website described: “iPerf3 is a tool for active measurements of the maximum achievable bandwidth on IP networks”.

If you ever search for tutorials about this tool, you will notice that the majority (based on my own search experience) only describe how to use it for TCP traffic.

Wait! Before continuing, I am assuming that you have a basic knowledge of network concepts such as model OSI, particularly Transport Layer. But if this is not the case, I found these well explained articles that will help you understand a little bit more.

1.The OSI model explained and how to easily remember its 7 layers.

What is the OSI Model?

Lets continue from where I left, in my previous experience using IPERF3 it was especially to verify that the available bandwidth is what I was told, for the simple reason the application was sensible to the bandwidth availability.

For this blog I will measure both TCP and UDP Traffic.

The Network topology I used is shown below:

For this tests, I used IPERF3 version.

Running the Iperf server's commands on the EC2 instance in AWS.
Running the Iperf client's command on my laptop.

Server's side command (AWS): iperf -S

Client's side command (Laptop):iperf -C 'Server IP'

So far, this is something you will normally find as an examples while looking in how to use Iperf, but something is not as common is measuring UDP traffic, the main difference is you need to add the tag: -u and -b (bandwidth) on the client side as is shown below:

__UDP => __Client's side command (Laptop): iperf -C 'Server IP'-u -b 1M

where '1M' represents 1 Mbps

You can also add the tag '-p', in the below example I am sending two paralell connections to the server and the bandwith limit is applied separately to each stream.

`
Please review the official documentaion link, for further information IPERF3 Documentation

I hope this article help is useful for you!!

Appropriate instance type depending on your workflow in AWS

Gabriela Menocal — Thu, 08 Jul 2021 01:20:14 +0000

Creating and working with Cloud Providers is increasingly popular (almost a norm). Different workflows and applications can be running and be accessible everywhere without the need for physical infrastructure. This is true whether you are testing, doing migrations or creating new workflows.

This post aims to share some tips, ideas and suggestions on how to choose an appropriate virtual machine in AWS with an EC2 instance.

A few questions should be answered before landing on an instance type (these questions are a baseline, you can tailor them, or include more, based on the workflow you are working on):

What is the main purpose of this workflow?
Do I need to run this instance temporarily or permanently?
What applications will be running in the instance?
Is the traffic steady? Will it increase over time?
How resilient do I need the solution to be?
Will it be a centralized or distributed workflow?
Do I have a limit in the budget?

In my experience, these questions are usually enough to choose an appropriate instance type (sometimes you even need only a subset), however, on occasion, more questions should be added as needed.

AWS offers an extended catalog of instance types which are classified based on the use case, for example, Compute Optimized or Memory Optimized. (AWS instance types). This classification allows us to easily decide which one needs to be used.

Some other sources of information can be found here:

These links are a great resource to find and sort the characteristics and instance types based on categories, technical specification, additional capabilities among others.

When installing any software in a machine, you will need to check whether the machine satisfies the minimum system requirements.

Here are examples of hardware specifications for two different applications;

Table I: Application 1

Table II: Application 2

From the above images, you can see the minimum hardware requirements for the given applications, so you can “map” those requirements with your instances types:

AWS presents a matrix with different configuration parameters, per family, and per size. For example, the following image shows characteristics such as vCPU and GiB, among others. The vCPUs are the virtual CPUs (please refer to AWS vCPU). Long history short you can assume the following: 1xvCPU = 1xCore

So, from Table 2: Application 2, the minimum instance types needed will be m5.2xlarge and up, because this instance has 8xvCPUs = 8xCores. required for the application.

I have learned that when something is listed with the words “Up to...”; in all likelihood you will work with a burstable instance, which means the specified bandwidth will be sporadic and not constant at any given time. Hence if you need to be constant and guaranteed, these instances would not be your best choice.

Regarding the physical processor speed, such as >= 2.4 GHz from from Table 2: Application 2, will require some knowledge about the relationship between Mbps and MHz, which is not part of the scope for this post, because this is comparing the two of them is a slightly more complex subject, however, you can take the following formula as a reference, which clearly states the correlation between them:

number of cycles per second * bits transferred per cycle = number of bits per second

This will give you the amount of data that can be processed based on the speed of the processor (measured in MHz). When moving to the cloud we will need to take in consideration two more things: IOPS and Throughput

IOPS is related to the number of read and write instructions/operations per second that can be processed
Throughput is the number of bits that can be read and written per second.

This means you will need to take a look at the EBS Bandwidth column and Network Bandwidth column, to make sure that you not only comply with the number of cores but also with the appropriate write and read speeds required for your application.

Also, the way AWS measures their EBS performance by IOPS, and also, as per AWS documentation “Amazon EBS allows you to create storage volumes and attach them to Amazon EC2 instances..” For more detailed information please refer to the following links:

Amazon EBS documentation

You can check that AWS described and classified its volume in two categories, SSD and HDD and listed their capabilities. For the complete tables and up to date features please check AWS EBS Features

I highlighted the IOPS and Throughput as a reference. Also, please notice that depending on the instance type and family the features and characteristics might be slightly different, however make sure that you should be able to identify and locate what you are looking for while choosing the instance types.

And the last point I would like to mention is the Memory given in the GiB unit, which is equivalent to the physical memory. So from the Application 1’s table the 4GB RAM => 4GiB Memory described in AWS documentation.

Key Points:

Understand the system requirements of your application
Avoid using ‘burstable’ instances in case a required throughput is needed at any given time
Understand the right IOPS and Throughput required for your deployment.
Also, the good news whether you decide to change your instance type, AWS offers this flexibility as well, please refers to Change instances Types