DEV Community: Abhishek Ray

Surviving Your First On-Call Shift: 5 Essential Tips

Abhishek Ray — Wed, 21 Aug 2024 04:25:57 +0000

Joining the on-call rotation for the first time can be daunting. With the right preparation and mindset, you can handle it like a pro.

Here are five essential tips to help you survive your first on-call shift:

1. Shadow and Prepare

Before taking on full responsibility, take part in at least one shadow rotation:

Observe an experienced colleague during their on-call shift (shadow)
Have a senior engineer watch and guide you through a shift (reverse shadow)
Use these experiences to familiarize yourself with common issues and procedures

As you shadow, start preparing your environment:

Install and test all necessary tools
Verify access to production systems and monitoring tools

2. Understand Your Responsibilities

Know exactly what qualifies as an on-call issue and what can wait until business hours:

Familiarize yourself with your team's escalation policy
Understand the severity levels of different types of incidents
Know when to wake up your manager or escalate to other teams
Don't hesitate to ask for help if you're unsure about something

3. Master Your Tools

Efficiency during incidents depends on your ability to use your tools effectively:

Practice using monitoring and alerting systems
Understand how to use your ticketing system for tracking issues
Be proficient with communication tools for team coordination
Know how to access and navigate production systems safely

4. Document Everything

Thorough documentation is crucial for effective incident management and knowledge sharing:

Log every action you take during an incident
Record the outcomes of your actions
Use a dedicated tool or notebook for documentation
Include enough detail for others to understand what happened and what you did

5. Learn and Improve

Every on-call shift is an opportunity to grow and enhance your skills:

After each shift, reflect on the incidents you handled
Identify areas where you can improve
Seek feedback from more experienced team members
Contribute to improving runbooks and processes based on your experiences

Remember, everyone starts somewhere. Your team is there to support you, so don't be afraid to ask for help when you need it.

Good luck, and may your pager stay silent!

AWS RDS Proxy Deep Dive: What is it and when to use it

Abhishek Ray — Wed, 16 Dec 2020 18:42:00 +0000

Introduction

AWS Relational Database Service (RDS) is a managed database service that was launched almost 10 years ago. Over the last 10 years, application patterns have evolved which has led to various challenges when it comes to interacting with the database:

Applications, especially serverless applications, have a large number of open DB connections. RDS only allows a limited number of open connections.
Custom failure handling code can become hard to manage and error-prone. Handling database failovers is one such example of handling failures.

AWS RDS proxy is a fully-managed database proxy for Amazon RDS. RDS proxy works by pooling and sharing DB connections and thus makes applications more scalable as well as resilient to database failures. They key benefits of using RDS proxy are:

Connection pooling and sharing: Improves application performance by reducing the number of open database connections
RDS proxy helps improve application availability during failure scenarios such as a database failover.
RDS Proxy gives you the choice to use IAM authentication for connecting to the database, thus removing the need for database credentials in the application code.

RDS proxy is currently available for Aurora MySQL, Aurora PostgreSQL, RDS MySQL and RDS PostgreSQL.

Key Concepts

Connection Pooling

Connection pooling is an optimization that enables applications to share and re-use database connections, thus reducing the load on the database itself. Opening and closing a new database connection is CPU-intensive whereas additional memory is needed for each open connection. Connection pooling also removes the need to worry about database connections in the application code.

Each database transaction uses one underlying database connection which can be reused once the transaction has finished. This transaction-level reuse is called connection multiplexing (or connection reuse).

In connection multiplexing, database connections are shared between client connections which helps minimize the resource overhead on the database server.

Pinning

In some cases, RDS proxy can’t safely reuse a database connection outside of the current session. In such scenarios, the same connection is used for the session until the session ends. This behavior is called pinning.

AWS recommends trying to avoid pinning as much as possible since it makes it harder to share connections and thus reduces the benefits of using RDS proxy.

Some reasons why a connection might be pinned are:

Change of session variable
Change of configuration parameter

More details about pinning can be found here.

Failure Handling

A common failure scenario is a database failover, where the original database instance becomes unavailable and is replaced by another one. A failover can happen for various reasons:

Planned maintenance such as a database upgrade
A problem with the database instance itself

RDS proxy can improve application availability in such a situation by waiting for the new database instance to be functional and maintaining any requests received from the application during this time. The end result is that the application is more resilient to issues with the underlying database.

AWS claims that RDS proxy improves failover times by 30-60%. These numbers seem to hold up based on tests like this.

Security benefits

RDS proxy can provide an additional layer of security between the application and the underlying database. AWS recommends enforcing IAM authentication while connecting to the proxy as that eliminates the need to specify database credentials anywhere in the code.

Pricing

Amazon RDS Proxy is priced per vCPU per hour for each database instance for which it is enabled. The price depends on the RDS instance type used by your database. The larger the database instance, the more you end up paying.
Partial hours are billed in one-second increments with a 10-minute minimum charge.

Additional information can be found here.

Monitoring

RDS proxy can be monitored by using Amazon CloudWatch. CloudWatch is well integrated with RDS proxy and provides useful metrics that can be used to understanding the performance and behavior of the proxy.

Some key metrics to keep an eye are:

DatabaseConnections: Number of database connections to the backend database
DatabaseConnectionsCurrentlyBorrowed: Number of connections currently being used by your application. Important to set an alarm on this metric.
DatabaseConnectionsCurrentlySessionPinned: Number of connections in the pinned state. This number should ideally be as low as possible to maximize RDS proxy performance.

Shortcomings

Some key limitations for RDS proxy are:

RDS proxy must be in the same VPC as the database instance. The proxy cannot be publicly accessible even if the database instance is.
RDS proxy cannot be used with a self-managed EC2-instance based database.
RDS proxy cannot be used for Aurora Serverless yet
A proxy can only be associated with 1 Database instance

Conclusion

Amazon RDS proxy is a database proxy that helps improve application availability and performance. It is particularly helpful for applications that have the following requirements:

Unpredictable workloads
Frequently open and close database connections
Higher availability during transient database failures

AWS EC2 Instance Comparison: M5 vs R5 vs C5

Abhishek Ray — Mon, 02 Sep 2019 04:08:27 +0000

Originally published at https://www.learnaws.org.

AWS has a lot of different instance types and it can be difficult to decide which instance type would be the best for a particular use-case. In this article, we will compare three of the most popular instance types (M5, R5 and C5) AWS offers across various dimensions such as their specs, their performance as well the cost.

Comparing M5, R5 and C5 instance families

M5 Instances

Amazon EC2 M5 Instances offer a balance of compute, memory and networking resources and thus they are ideal for a broad range of workloads.

R5 Instances

Amazon EC2 R5 instances are memory-optimized instances. R5 instances are well suited for memory intensive applications such as high-performance databases, distributed web scale in-memory caches, mid-size in-memory databases, real time big data analytics, and other enterprise applications.

C5 Instances

Amazon EC2 C5 instances are optimized for compute. C5 instances offer the lowest price per vCPU in the Amazon EC2 family and are ideal for running advanced compute-intensive workloads.

Performance

We compared the performance between these instance families by running a bunch of tests on the following instances:

m5.xlarge
r5.xlarge
c5.xlarge

Instance Type	Number of cores	Memory (GB)
m5.xlarge	4	16
r5.xlarge	4	32
c5.xlarge	4	8

Processor performance

We used Geekbench to compare the performance of these instance types. Geekbench is a processor benchmarking program. It runs a series of tests on a processor and times how long the processor takes to complete the tasks. The faster the CPU, the higher the score.

All three instance types use the Intel Xeon Platinum 8000 series (Skylake-SP) processor. However, one key difference is the CPU clock speed. Both the M5 and R5 instances families have a core Turbo CPU clock speed of up to 3.1 GHz whereas the C5 instances have a clock speed of up to 3.4 GHz.

As expected, the c5 instances perform the best on the Geekbench tests.

Instance Type	Single-core score	Multi-core score
m5.xlarge	3905	8697
r5.xlarge	4112	9132
c5.xlarge	4215	9415

Memory performance

We used Stream benchmark to compare the memory performance across these instance types. Stream is used to measure the sustained memory bandwidth.

In this test, the r5 instances performed the best as expected. R5 instances are memory-optimized and have the most amount of memory as compared to M5 and C5. The M5 instances have more memory available than the C5 instances and have better memory performance than the C5 instances.

m5.xlarge


-------------------------------------------------------------

Function    Best Rate MB/s  Avg time     Min time     Max time

Copy:           27275.6     0.006025     0.005866     0.006193

Scale:          27054.6     0.006070     0.005914     0.006232

Add:            28308.0     0.008854     0.008478     0.009080

Triad:          28298.5     0.008879     0.008481     0.009268

-------------------------------------------------------------

r5.xlarge


-------------------------------------------------------------

Function    Best Rate MB/s  Avg time     Min time     Max time

Copy:           28324.3     0.005796     0.005649     0.005914

Scale:          27793.0     0.005834     0.005757     0.005905

Add:            28616.2     0.008505     0.008387     0.008691

Triad:          28704.4     0.008515     0.008361     0.008679

-------------------------------------------------------------

c5.xlarge

-------------------------------------------------------------

Function    Best Rate MB/s  Avg time     Min time     Max time

Copy:           26445.8     0.006110     0.006050     0.006173

Scale:          26306.9     0.006143     0.006082     0.006235

Add:            27072.4     0.008962     0.008865     0.009093

Triad:          27002.7     0.008980     0.008888     0.009053

-------------------------------------------------------------

Disk performance

We also ran simple read & write benchmarks to test the performance of the EBS volumes on each of these instance types.

The R5 instances performed the best on the disk performance benchmarks.

m5.xlarge

Write

dd bs=16k count=102400 oflag=direct if=/dev/zero of=test_data

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 77.5204 s, 21.6 MB/s

Read


dd bs=16K count=102400 iflag=direct if=test_data of=/dev/null

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 57.9766 s, 28.9 MB/s

r5.xlarge

Write

dd bs=16k count=102400 oflag=direct if=/dev/zero of=test_data

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 58.0189 s, 28.9 MB/s

Read

dd bs=16K count=102400 iflag=direct if=test_data of=/dev/null

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 40.6274 s, 41.3 MB/s

c5.xlarge

Write

dd bs=16k count=102400 oflag=direct if=/dev/zero of=test_data

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 63.4356 s, 26.4 MB/s

Read

dd bs=16K count=102400 iflag=direct if=test_data of=/dev/null

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 46.5171 s, 36.1 MB/s

Pricing

We compared the prices between these instance families by looking at the price per hour for each instance type.

C5 instances are the cheapest amongst the three instance families whereas the R5 instances are the most expensive. R5 instances are almost 50% more expensive than C5 instances.

Price/core for Linux on-demand instances:

Instance Type	Price / hour
m5.xlarge	$0.192
r5.xlarge	$0.252
c5.xlarge	$0.170

Conclusion

The new C5 instances offer the best processor performance as well as price and should be the go-to instance type for most use-cases.

If the use-case requires a lot of memory, then the choice becomes either between M5 and R5. Critical applications such as in-memory databases and caches should use the R5 instances since they are optimized for such use-cases.

Applications such as batch processing systems are probably better suited for M5 instances since they need the additional memory but are not as performance-critical.