DEV Community: Anupa Supul

How Does AWS Protect Sensitive Data? Understanding AWS KMS, Encryption & Key Management

Anupa Supul — Wed, 11 Nov 2026 05:43:07 +0000

Imagine you're building a web application and deploying it on AWS.

Your application stores customer information, database credentials, uploaded documents, and other sensitive data.

Now consider something important:

What happens if someone gains access to the underlying storage? Can they read your data?

This is where encryption becomes essential.

AWS provides several services and mechanisms to protect sensitive information. One of the most important is AWS Key Management Service (AWS KMS).

In this article, I'll explore how encryption works in AWS, what KMS does, how envelope encryption protects data, and how services such as Amazon S3 and AWS Secrets Manager use encryption in real-world applications.

Figure: Understanding AWS data protection through encryption, key management, and secure application architecture.

1. What Is Encryption?

Encryption is the process of converting readable information (plaintext) into an unreadable format (ciphertext) using a cryptographic algorithm and key.

For example:

Original Data (Plaintext)
          |
          v
      Encryption
          |
          v
    Encrypted Data
     (Ciphertext)
          |
          v
      Decryption
          |
          v
 Original Data Again

Without the appropriate decryption key, an unauthorized person should not be able to read the protected data.

Encryption is commonly used to protect:

Customer information
Database records
Uploaded documents
Application credentials
Payment-related information
Communication between applications

In cloud environments, encryption is generally considered in two important situations.

Encryption at rest

Encryption at rest protects data while it is stored.

Examples:

Files stored in Amazon S3
Database storage in Amazon RDS
Volumes attached to Amazon EC2
Backups and snapshots

Even if someone gains access to the underlying storage media, encrypted data remains protected without the required decryption capability.

Encryption in transit

Encryption in transit protects data while it travels between systems.

For example:

User Browser
     |
     | HTTPS / TLS
     |
     v
Application Load Balancer
     |
     | HTTPS (if configured)
     |
     v
Backend Application

HTTPS uses TLS to establish an encrypted connection between communicating parties.

Remember: Encryption at rest protects stored data, while encryption in transit protects data moving across a network.

2. What Is AWS KMS?

AWS Key Management Service (KMS) is a managed service that allows you to create and control cryptographic keys used to protect data.

Instead of building your own key management infrastructure, AWS KMS provides a managed way to create, manage, and use cryptographic keys.

KMS can integrate with many AWS services, including:

Amazon S3
Amazon EBS
Amazon RDS
AWS Lambda
AWS Secrets Manager
Amazon EFS

For example, when you enable SSE-KMS encryption on an S3 bucket, Amazon S3 can use a KMS key to protect the objects stored in that bucket.

The important thing is that KMS manages cryptographic keys. It does not replace the storage service where your actual application data lives.

3. Understanding AWS KMS Keys

A KMS key is a logical representation of a cryptographic key managed by AWS KMS.

There are several key ownership and management options.

AWS owned keys

These are managed by AWS and used across AWS services in supported scenarios.

AWS manages the keys.
Customers cannot view or manage them directly.
There is no separate customer-facing key management charge.

AWS managed keys

These are created and managed by AWS services in your account.

For example, an AWS service might use an AWS managed KMS key to encrypt data.

AWS manages the key lifecycle.
You can view the key in your account.
You have limited control compared with customer managed keys.

Customer managed keys

These are keys that you create and manage in AWS KMS.

You have greater control over:

Key policies
IAM permissions
Key rotation options
Key disabling and deletion
Grants and access management

For example, an organization may create a customer managed KMS key specifically for encrypting sensitive customer documents.

This provides greater control over how the key is used.

4. How Does AWS KMS Encryption Work?

Let's imagine an application storing customer documents in Amazon S3.

When using KMS-based encryption, the application or AWS service can request cryptographic operations from KMS.

A simplified flow looks like this:

       Application
            |
            v
       Amazon S3
            |
            v
         AWS KMS
            |
            v
      KMS Key Used
       for Encryption
            |
            v
     Encrypted Object
       Stored in S3

However, there is an important concept behind this process: Envelope Encryption.

5. What Is Envelope Encryption?

Envelope encryption is a method of protecting data by encrypting the actual data with one key and protecting that key with another key.

This is particularly useful when applications need to encrypt large amounts of data.

Let's understand it step by step.

AWS KMS generates a data key.
The application uses the plaintext data key to encrypt the actual data.
The encrypted data key is stored alongside the encrypted data.
The plaintext data key is removed from memory when no longer needed.
When the data needs to be decrypted, the encrypted data key is sent to KMS.
KMS decrypts the data key, allowing the application to decrypt the original data.

Architecture:

              AWS KMS
                 |
                 v
        Generate Data Key
                 |
          +------+------+
          |             |
          v             v
    Plaintext Key   Encrypted Key
          |             |
          v             |
    Encrypt Data         |
          |             |
          v             v
     Encrypted Data + Encrypted Data Key
                 |
                 v
             Storage

Why is this useful?

Instead of sending large amounts of application data directly to KMS for encryption, the application can use a data key locally to encrypt the data.

KMS protects the data key.

This approach improves scalability and is widely used in cloud encryption systems.

6. How Does Amazon S3 Protect Stored Data?

Amazon S3 supports multiple server-side encryption options.

Let's look at the important ones.

SSE-S3

Server-Side Encryption with Amazon S3 managed keys.

Amazon S3 manages the encryption keys.
Encryption and decryption are handled automatically by S3.
Suitable for many standard object-storage use cases.

SSE-KMS

Server-Side Encryption with AWS KMS keys.

Uses AWS KMS to protect encryption keys.
Supports customer managed keys and AWS managed keys.
Provides additional control through KMS policies and permissions.
Can provide audit visibility through AWS logging services.

This is particularly useful when organizations need more control over encryption key access.

SSE-C

Server-Side Encryption with Customer-Provided Keys.

The customer supplies the encryption key with each relevant request.
Amazon S3 performs the encryption and decryption.
The customer is responsible for securely managing the key.

For most applications, SSE-S3 or SSE-KMS is easier to manage than SSE-C.

Quick comparison

Feature	SSE-S3	SSE-KMS	SSE-C
Key management	Amazon S3	AWS KMS	Customer
Encryption	Server-side	Server-side	Server-side
Customer key control	Limited	Greater control	Customer-managed
KMS integration	No customer KMS key required	Yes	No

Important: S3 automatically encrypts new object uploads with SSE-S3 by default. You can configure SSE-KMS when you need KMS-based key management and additional access controls.

7. How Does AWS Secrets Manager Protect Sensitive Information?

Encryption is not only about files and databases. Applications also need to protect sensitive credentials.

Imagine your backend needs to connect to MongoDB or an RDS database.

A dangerous approach would be storing credentials directly inside your source code:

const dbPassword = "my-secret-password";

If this code is pushed to GitHub, your credentials could be exposed.

Instead, AWS provides AWS Secrets Manager.

Secrets Manager helps you:

Store database credentials and API keys.
Retrieve secrets securely through authorized applications.
Rotate supported secrets automatically when configured.
Integrate with AWS KMS for encryption at rest.

A simplified architecture:

        Backend Application
                |
                v
             IAM Role
                |
                v
       AWS Secrets Manager
                |
                v
             AWS KMS
                |
                v
       Encrypted Secret

For example, an ECS task can use its task role to retrieve a specific database secret from Secrets Manager.

KMS helps protect the secret at rest, while IAM controls whether the application is authorized to retrieve it.

Remember the difference:

IAM controls who can access a secret.
Secrets Manager stores and manages the secret.
KMS protects encryption keys used to encrypt the secret.

8. What Happens If Someone Tries to Access an Encrypted Resource?

Encryption alone does not guarantee authorization.

AWS also uses IAM policies, KMS key policies, and other access-control mechanisms to determine whether a request is permitted.

For example, imagine a developer has permission to read an encrypted S3 object but does not have the required KMS permissions.

The developer may still be unable to decrypt and read that object.

For SSE-KMS, access generally requires the appropriate S3 permissions and KMS permissions, subject to the applicable key policy, IAM policy, grants, and other authorization rules.

This gives organizations another layer of access control.

9. Real-World AWS Architecture Example

Let's bring everything together using a simple application architecture.

Imagine a React frontend and Node.js backend deployed on AWS.

             User
               |
               v
          CloudFront
               |
               v
         React Frontend
               |
               v
        Application API
               |
          +----+-----+
          |          |
          v          v
      Amazon S3   AWS Secrets
       Documents   Manager
          |          |
          v          v
       AWS KMS    AWS KMS
          |          |
          v          v
      Encrypted   Encrypted
       Objects     Secrets
          |
          v
       Storage

      HTTPS / TLS protects
      data in transit

In this architecture:

CloudFront delivers the frontend securely.
HTTPS/TLS protects communication in transit.
Amazon S3 stores uploaded documents.
AWS KMS protects encryption keys used for S3 encryption.
Secrets Manager stores sensitive application credentials.
IAM roles control which services can access these resources.

This is a simplified example, but it demonstrates how different AWS services work together to protect application data.

10. What Should You Remember for the AWS SAA-C03 Exam?

Here are some important points:

Encryption at rest: Protects stored data.
Encryption in transit: Protects data moving across networks.
AWS KMS: Managed service for creating and controlling cryptographic keys.
Customer managed key: Gives customers greater control over key management.
AWS managed key: Managed by AWS on behalf of a customer in their account.
Envelope encryption: Encrypts data using a data key and protects that data key using another key.
SSE-S3: Amazon S3 manages the encryption keys.
SSE-KMS: Uses AWS KMS for key management.
SSE-C: Customer provides and manages the encryption key.
AWS Secrets Manager: Stores and manages sensitive information such as credentials.
IAM: Controls permissions to access AWS resources and use KMS keys.

One more important exam concept:

Disabling a KMS key prevents it from being used for cryptographic operations. Deleting a KMS key is a separate operation and can permanently make encrypted data unrecoverable if no usable copy of the key remains.

Final Thoughts

When I started learning cloud computing, I mainly focused on deploying applications and connecting AWS services.

But understanding encryption made me realize that a production-ready application needs more than just working infrastructure.

It needs proper access control, secure communication, encryption, and responsible key management.

AWS KMS, Amazon S3, IAM, and Secrets Manager work together to provide these security capabilities.

My biggest takeaway from this topic is simple:

Protecting data is not just about encrypting it. It's also about controlling who can decrypt it and how encryption keys are managed.

I'm continuing my AWS learning journey and documenting the concepts that help me understand how real-world cloud applications are built and secured.

How Does AWS Control Who Can Access What? Understanding IAM Policies, Roles & Least Privilege

Anupa Supul — Thu, 29 Oct 2026 05:34:21 +0000

When I started learning AWS, I realized something important: creating cloud resources is only part of the job. We also need to control who can access those resources, what they can do, and which AWS services can communicate with each other.

Imagine deploying a MERN application on AWS. Your backend needs to read files from S3, retrieve database credentials, and write application logs. Should we give it full access to our AWS account?

Definitely not.

This is where AWS Identity and Access Management (IAM) becomes essential.

In this article, I'll explore IAM users, groups, roles, policies, trust policies, permission boundaries, and the principle of least privilege using practical examples.

Figure: Understanding AWS IAM through identities, permissions, and a real-world application example.

1. What Is AWS IAM?

AWS IAM is a service that helps control access to AWS resources.

It answers three fundamental questions:

Who is making the request?
What actions are they allowed to perform?
Which resources can they access?

For example, a developer might be allowed to view EC2 instances but not delete production databases. Meanwhile, an application running on Amazon ECS might need permission to read a specific file from S3.

IAM lets us define these permissions instead of granting everyone unrestricted access.

2. Understanding the Main IAM Components

IAM users

An IAM user represents an identity with long-term credentials, when those credentials are configured.

For example, a developer may have an IAM user that can access specific development resources.

However, for people, AWS recommends federation and temporary credentials where practical rather than creating long-term IAM user credentials for everyday access.

Important: Avoid using the AWS root user for routine tasks. Protect it carefully and use it only when root-level access is specifically required.

IAM groups

An IAM group allows you to manage permissions for multiple IAM users together.

For example:

Developers — access to development resources
Auditors — read-only access for reviewing resources
Administrators — broader administrative permissions

Groups simplify permission management, but they are not a replacement for carefully designed policies.

IAM roles

An IAM role is an identity that provides a set of permissions and can be assumed by an authorized principal.

Roles are especially useful when AWS services need to communicate securely.

Imagine an application running on Amazon EC2 that needs to read a file from S3.

Instead of placing AWS access keys inside the application code, we can attach an appropriate IAM role to the EC2 instance. The application can then obtain temporary credentials through the AWS-provided credential mechanism.

This reduces the need to store long-term credentials in the application.

IAM roles are also used with ECS tasks, Lambda functions, and many other AWS services.

IAM policies

An IAM policy is a document that defines permissions.

A policy can specify:

Effect: Allow or Deny
Action: The operation being requested
Resource: The AWS resource involved
Condition: Optional requirements that must be satisfied

For example, an application might need permission to download objects from one S3 bucket without being allowed to delete them.

A simplified policy could look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-app-documents/*"
    }
  ]
}

This allows the specified GetObject operation on objects in the named bucket. It does not, by itself, grant permission to upload or delete objects.

Replace the example bucket name with your actual resource when implementing a policy.

3. Identity-Based vs. Resource-Based Policies

AWS supports different ways of defining access.

Identity-based policies attach to identities such as users, groups, and roles. They describe what those identities can do.

Resource-based policies attach to supported resources and describe which principals can access them.

For example, an IAM role may have permission to read objects from an S3 bucket, while the bucket policy may also restrict which principals are allowed to access that bucket.

Both sides can matter. An Allow in one policy does not automatically override an applicable explicit Deny or other access restrictions.

4. What Is a Trust Policy?

This is one of the most important distinctions when learning IAM roles.

An IAM role has two separate questions to answer:

Trust policy: Who is allowed to assume this role?
Permissions policy: What can the role do after it is assumed?

Imagine an ECS task that needs to access an S3 bucket.

The task role's trust policy establishes that the appropriate ECS tasks can assume the role. Its permissions policy grants only the required S3 actions.

These policies solve different problems. Allowing a service to assume a role does not automatically give that role permission to access S3.

5. How IAM Works in a Real Application

Let's use a simplified architecture for a web application deployed on AWS.

                Developer
                    |
                    v
              AWS Console / CLI
                    |
                    v
               IAM Identity
                    |
            IAM Policies
                    |
                    v
             AWS Resources


          Application Running on ECS
                    |
                    v
               ECS Task Role
                    |
            Permissions Policy
                    |
          +---------+----------+
          |                    |
          v                    v
      Amazon S3        AWS Secrets Manager

The developer accesses AWS using an authorized identity. The application accesses other AWS services through its task role.

These are separate identities with separate permissions.

For example:

A developer can deploy a new application version.
An ECS task can read specific S3 objects.
The task can retrieve a particular secret if its role is authorized to do so.
Neither identity automatically needs full administrator access.

This is the basic idea behind secure access control in cloud applications.

6. What Is the Principle of Least Privilege?

The principle of least privilege means granting only the permissions necessary to perform a task.

Suppose your application needs to read files from S3.

A poor approach would be to grant s3:* on every bucket in the account.

A better approach would be to grant only the necessary S3 actions on the specific bucket or objects the application needs.

Broad permission	Least-privilege alternative
All S3 actions	Only required S3 actions
All resources	Specific resource ARNs where supported
Permanent credentials	Temporary credentials where practical
Administrator access for every developer	Role-based access suited to each responsibility

Least privilege is not simply about writing smaller policies. It also means reviewing permissions regularly, removing unused access, and considering what could happen if an identity were compromised.

7. What Are Permission Boundaries?

Permission boundaries help set the maximum permissions that an IAM user or role can receive through identity-based policies.

Think of them as a permissions ceiling.

For example, an organization might allow a development team to create roles for its applications but require that those roles remain within a defined permissions boundary.

A boundary does not grant permissions by itself. The identity still needs applicable policies that allow the requested actions.

Also, a permissions boundary is not a universal restriction on every possible access path. Resource-based policies and other AWS authorization rules must be considered when evaluating effective access.

8. IAM Roles vs. Hardcoded Credentials

Consider two approaches for an application that accesses AWS services.

Approach A: Hardcoded credentials

const accessKey = "YOUR_ACCESS_KEY";
const secretKey = "YOUR_SECRET_KEY";

This is unsafe. Credentials can accidentally enter Git repositories, logs, build artifacts, or shared files.

Approach B: IAM role and temporary credentials

Configure the application's execution environment with the appropriate IAM role. Use the AWS SDK's default credential provider chain to obtain credentials.

For example, with AWS SDK for JavaScript v3:

import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3";

const s3 = new S3Client({
  region: "ap-south-1"
});

const response = await s3.send(
  new ListObjectsV2Command({
    Bucket: "my-app-documents"
  })
);

console.log(response.Contents);

The SDK can obtain credentials from the supported runtime credential provider chain, including the role credentials provided in an appropriately configured AWS environment.

The example still requires permission for s3:ListBucket on the bucket. If the application only needs to download a known object, grant s3:GetObject on that object instead.

Remember: Using an IAM role does not automatically make an application secure. The role must have the right trust relationship and narrowly scoped permissions.

9. IAM and AWS Secrets Manager

IAM and Secrets Manager work together, but they have different responsibilities.

IAM controls whether an identity can perform an action.
AWS Secrets Manager stores and manages sensitive values such as database credentials and API keys.

For example, an ECS task can use its task role to retrieve a particular database secret from Secrets Manager. The task role should not be allowed to retrieve every secret in the account unless that broader access is genuinely required.

Avoid embedding secrets in source code, Docker images, or GitHub repositories.

10. Common IAM Mistakes to Avoid

Here are a few mistakes worth watching for when building AWS projects:

Using the root user for everyday development.
Giving applications administrator permissions.
Hardcoding AWS access keys into source code.
Confusing a role's trust policy with its permissions policy.
Assuming a permissions boundary grants access.
Forgetting that an explicit Deny can override an Allow.
Granting access to every resource when only one resource is required.
Leaving unused users, access keys, or permissions in place.

IAM policies should be tested and reviewed carefully, particularly before making changes in production.

11. What Should You Remember for the AWS SAA-C03 Exam?

These are useful distinctions to remember:

IAM user: An identity that can have long-term credentials.
IAM group: Organizes IAM users for permission management.
IAM role: An assumable identity commonly used for temporary credentials.
IAM policy: Defines permissions.
Trust policy: Defines who can assume a role.
Permissions boundary: Sets a maximum permission boundary for an IAM identity.
Least privilege: Grants only the access needed.
Explicit Deny: Overrides an applicable Allow.
Instance profile: A mechanism used to pass an IAM role to an EC2 instance.
ECS task role: Grants permissions to application code running in an ECS task.
ECS task execution role: Grants permissions needed by the ECS agent to perform tasks such as pulling images and sending logs, depending on configuration.

That last distinction matters: the task role is for the application; the execution role is for ECS task startup and management operations.

Final Thoughts

Learning IAM changed how I think about deploying applications on AWS. It is not enough for an application to work — it must also have the right permissions and nothing more than it needs.

Whether you're deploying a Lambda function, running containers on ECS, or building a larger cloud architecture, IAM is one of the foundations of secure AWS development.

My biggest takeaway is simple: don't ask only whether an application can access a resource. Ask whether it should have that access, and whether the permission can be made more specific.

That's the mindset behind least privilege.

I'm continuing my AWS learning journey and documenting the concepts that help me understand how real cloud systems are built and secured.

What Happens When a Serverless Workflow Has 10 Steps? Understanding AWS Step Functions

Anupa Supul — Thu, 29 Oct 2026 05:21:22 +0000

In my previous articles, I explored AWS Lambda, API Gateway, DynamoDB, SQS, SNS, and EventBridge.

Each service solves a specific problem. But as I learned more about serverless architecture, I started thinking about another challenge.

What happens when an application needs to coordinate many tasks, handle failures, and decide what to do next?

Imagine an online shopping application.

When a customer places an order, the system might need to validate the order, process payment, update inventory, arrange delivery, and send a confirmation.

Some steps must happen in a particular order. Some can run in parallel. Others might need to be retried if they fail.

Managing all this logic inside one large Lambda function can become difficult.

This is where AWS Step Functions becomes useful.

1. What Is AWS Step Functions?

AWS Step Functions is a workflow orchestration service that helps coordinate multiple tasks into a defined workflow.

It uses state machines to describe how a process moves from one step to another.

A workflow can invoke Lambda functions, integrate with supported AWS services, make decisions, run tasks in parallel, and handle errors.

Instead of manually managing every transition in your application code, you define the workflow and its behavior.

Think of it this way:

Lambda executes a task.
Step Functions coordinates the tasks.

2. How Does It Work?

Consider a simplified order-processing workflow.

Customer Places Order
         |
         v
    Validate Order
         |
         v
    Process Payment
         |
         v
   Update Inventory
         |
         v
   Send Confirmation
         |
         v
    Order Complete

Each step performs a particular responsibility. Step Functions manages the transitions between those steps.

If payment fails, for example, the workflow can follow a defined error-handling path rather than blindly continuing to update inventory.

This makes complex business processes easier to understand and maintain.

3. Understanding State Machines

A state machine defines the states in a workflow and how execution moves between them.

In Step Functions, common state types include:

Task: Performs work, such as invoking Lambda.
Choice: Selects a path based on a condition.
Parallel: Executes branches concurrently.
Wait: Pauses execution for a specified period.
Succeed: Marks successful completion.
Fail: Marks unsuccessful completion.
Pass: Passes data through or provides a simple transformation.

For example, after payment processing, the workflow could check the result.

       Process Payment
              |
              v
       Payment Successful?
           /       \
         Yes        No
          |          |
          v          v
 Update Inventory  Handle Failure
          |
          v
    Confirm Order

The workflow is not just a sequence of functions. It can include decisions, parallel tasks, waiting periods, and error-handling paths.

4. What Happens When a Task Fails?

Failures are normal in distributed systems. An external API might be temporarily unavailable, or a service might return an error.

Step Functions lets you define retry and error-handling behavior for supported workflow states.

For example, a task can retry a temporary failure before moving to a fallback path.

       Process Payment
              |
              v
          Task Failed
              |
              v
         Retry Task
              |
        +-----+-----+
        |           |
      Success     Failed Again
        |           |
        v           v
   Continue      Handle Error

You can configure retry policies with parameters such as the error types to retry, the number of attempts, and the delay between attempts.

You can also configure catch behavior to move to another state when an error matches the configured conditions.

However, a retry does not guarantee that an operation is safe to repeat. If a payment request succeeds but the response is lost, repeating it could create a duplicate charge unless the payment operation supports appropriate idempotency.

Important lesson: Workflow orchestration helps manage failures, but your application still needs correct error handling and idempotent operations.

5. A Real-World Architecture

Let's connect Step Functions with the AWS services we've already explored.

An illustrative order-processing workflow using API Gateway, Lambda, Step Functions, and supporting AWS services.

Here is a simplified architecture diagram in text form:

             React Frontend
                    |
                    v
               API Gateway
                    |
                    v
                AWS Lambda
                    |
                    v
            AWS Step Functions
                    |
             Validate Order
                    |
                    v
             Process Payment
                    |
                    v
          +---------+---------+
          |                   |
          v                   v
    Update Inventory    Record Order Status
          |                   |
          +---------+---------+
                    |
                    v
             Send Notification
                    |
                    v
              Order Complete

This is a conceptual architecture; the exact integration can vary depending on the application.

Step Functions can invoke Lambda functions or integrate directly with supported AWS services. For longer-running or asynchronous tasks, a workflow can also wait for an external task to complete using supported integration patterns.

For example, you could use:

API Gateway to receive the request.
Lambda to validate and initiate the workflow.
Step Functions to coordinate order processing.
DynamoDB to store order data.
SNS to distribute notifications.
CloudWatch to monitor workflow activity and failures.

The benefit is separation of responsibilities. Each component focuses on its job, while the workflow defines how the overall process fits together.

6. Standard vs. Express Workflows

AWS Step Functions offers two workflow types: Standard and Express.

Feature	Standard	Express
Typical use	Long-running, durable workflows	High-volume, short-duration workflows
Maximum duration	Up to one year	Up to five minutes
Execution model	Exactly-once workflow execution model, with integration-specific considerations	At-least-once or at-most-once execution, depending on the mode
Common use cases	Order processing, approvals, business workflows	High-volume event processing, data transformation

The execution model matters. Even when a Standard workflow has an exactly-once execution model, an external side effect may still need idempotency, especially when the workflow retries an integration or an external system's response is uncertain.

Choose the workflow type based on the duration, throughput, reliability, and execution requirements of the process.

7. When Should You Use AWS Step Functions?

Step Functions is useful when:

A process contains multiple dependent tasks.
You need conditional logic or parallel execution.
You want clearly defined retry and error-handling behavior.
You need to coordinate multiple AWS services.
You want a visible execution history to help troubleshoot workflows.

It may be unnecessary for a simple API that only performs one short operation. A single Lambda function may be enough for that.

The goal is not to use Step Functions everywhere. It is to use it when workflow complexity makes orchestration worthwhile.

8. My Main Takeaways

As I continue learning AWS, I'm realizing that building cloud applications involves more than understanding individual services. It also means understanding how those services work together to implement real business processes.

My mental model is simple:

Lambda performs individual tasks.
Step Functions coordinates workflows.
SQS queues work for asynchronous processing.
SNS distributes notifications.
EventBridge routes events based on rules.
DynamoDB stores application data.
CloudWatch helps monitor the system.

Each service has a different responsibility, and choosing the right combination depends on the problem you're solving.

For me, AWS Step Functions is an important piece of the puzzle because it helps turn a collection of individual cloud services into a coordinated workflow.

I'm continuing to document my AWS learning journey one concept at a time. If you're learning AWS too, I'd love to hear what you're building and which services you're exploring.

How Do Large Applications Process Work in the Background? Understanding Amazon SQS, SNS & EventBridge

Anupa Supul — Tue, 20 Oct 2026 05:08:16 +0000

When I started learning AWS, I focused mainly on deploying applications, building APIs, and connecting databases.

But then I started thinking about something interesting.

What happens when an application needs to perform several tasks without making the user wait for everything to finish?

Imagine ordering a product from an online store. After placing the order, the system might need to update inventory, send a confirmation email, generate an invoice, and notify the delivery team.

Should the customer wait until every single task is completed?

Probably not.

This is where event-driven architecture and messaging services become useful.

In this article, I'll explore three AWS services that help applications communicate and process work more efficiently:

Amazon SQS — queues messages for processing.
Amazon SNS — distributes messages to multiple subscribers.
Amazon EventBridge — routes events to targets based on rules.

1. Why Do Applications Need Background Processing?

Consider a simple application where every operation happens inside one request.

Customer places an order
          |
          v
      Save order
          |
          v
   Update inventory
          |
          v
   Send confirmation
          |
          v
   Generate invoice
          |
          v
   Return response

If each operation runs sequentially, the customer may need to wait for all of them. If an external email service is slow, the entire request could take longer.

It also creates tight coupling: a problem in one dependency can affect the main request.

Instead, we can separate the immediate operation from background work.

Customer places an order
          |
          v
       Save order
          |
          v
   Queue background work
          |
          v
  Return response to customer
          |
          v
   Process tasks separately

Now the application can respond after completing the necessary synchronous work, while other operations happen independently.

This is one of the main ideas behind asynchronous processing.

2. Amazon SQS — Let Messages Wait in a Queue

Amazon Simple Queue Service (SQS) is a managed message queuing service.

Think of it as a waiting line for tasks. A producer sends a message to the queue, and a consumer retrieves and processes it.

For example:

Order API
    |
    v
Amazon SQS Queue
    |
    v
Order Processing Worker
    |
    v
Update Inventory

The API doesn't need to wait for the worker to finish processing the message.

Why is SQS useful?

It helps decouple application components.
Messages can wait until consumers are ready to process them.
It helps absorb temporary traffic spikes.
Failed messages can be retried.
Dead-letter queues can isolate messages that repeatedly fail.

SQS offers Standard queues and FIFO queues. Standard queues provide high throughput with at-least-once delivery, so consumers should be designed to handle duplicate messages. FIFO queues support ordering within message groups and deduplication features, with their own throughput and configuration considerations.

One important concept is the visibility timeout. When a consumer receives a message, the message becomes temporarily invisible to other consumers. If processing succeeds, the consumer deletes it. If the message isn't deleted before the visibility timeout expires, it can become available again.

That is why reliable message processing needs error handling and idempotency.

3. Amazon SNS — Send One Message to Multiple Subscribers

Amazon Simple Notification Service (SNS) is a managed publish/subscribe messaging service.

Instead of placing work in a queue for one consumer pattern, SNS can distribute a published message to multiple subscribers.

Imagine an order has been placed. Several systems need to react:

A notification service sends a confirmation.
An analytics service records the event.
Another service starts a fulfilment workflow.

SNS can help distribute the notification to these subscribers.

              Order Service
                    |
                    v
                 Amazon SNS
                    |
          +---------+---------+
          |         |         |
          v         v         v
        SQS A     Lambda    Email
      Queue       Function  Endpoint

SNS supports different subscription types, including SQS queues, Lambda functions, HTTP/S endpoints, and email, depending on the configuration and use case.

Why is SNS useful?

SNS is a good fit when multiple subscribers need to receive a notification about the same event.

For example, an order service can publish an OrderPlaced notification without needing to know the internal implementation of every subscriber.

That reduces direct dependencies between services.

A key distinction: SNS distributes messages; it is not a general-purpose durable work queue in the same way SQS is. Combining SNS with SQS can be useful when multiple consumers need their own queues and independent processing.

4. Amazon EventBridge — Route Events Based on Rules

Amazon EventBridge is a serverless event bus service that routes events to targets using rules.

An event describes something that happened, such as:

An order being placed.
A payment succeeding.
A file being uploaded to Amazon S3.
A scheduled task reaching its execution time.

EventBridge can inspect an event and route it to the targets whose rules match.

For example:

       OrderPlaced Event
               |
               v
       Amazon EventBridge
               |
        +------+------+
        |             |
        v             v
   Order Processor  Analytics
       Lambda        Target

You can define rules that match event attributes and send matching events to supported targets.

EventBridge is particularly useful for reacting to events from AWS services, your own applications, and supported SaaS integrations.

It also supports scheduled and time-based event patterns through its scheduling capabilities.

When should you use EventBridge?

Use it when you want to route events based on what happened, without tightly connecting the event producer to every downstream service.

For example, an application could react differently to OrderPlaced, PaymentFailed, and OrderCancelled events.

5. SQS vs SNS vs EventBridge

These services can work together, but they solve different problems.

Feature	Amazon SQS	Amazon SNS	Amazon EventBridge
Main purpose	Queue messages for processing	Distribute messages to subscribers	Route events using rules
Core pattern	Queue and consumer	Publish/subscribe	Event bus and targets
Typical use	Background jobs	Notifications to multiple subscribers	Reacting to application or AWS events
Message handling	Consumers poll or use supported integrations	Pushes notifications to subscribers	Routes matching events to targets
Useful feature	Visibility timeout and dead-letter queues	Multiple subscribers and filtering	Event pattern matching and routing

Here's the simple mental model I use:

SQS: "Process this work when you're ready."
SNS: "Send this notification to these subscribers."
EventBridge: "When this event happens, route it to the appropriate target."

6. How They Work Together in a Real Application

Let's return to our online store.

After a customer places an order, the API saves the order and publishes an event. Different services can then handle different responsibilities.

An e-commerce application can use queues for background processing, notifications for multiple subscribers, and event rules to route events.

Here is a simplified architecture diagram in text form:

React Frontend
      |
      v
API Gateway
      |
      v
AWS Lambda
      |
      v
Save Order in DynamoDB
      |
      v
Publish OrderPlaced Event
      |
      v
Amazon EventBridge
      |
      +-------------------+
      |                   |
      v                   v
Order Processing       Analytics
      |
      v
Amazon SQS
      |
      v
Background Worker
      |
      v
Update Inventory / Fulfil Order

A separate notification workflow could use SNS to distribute an order notification to multiple subscribers.

The exact architecture depends on the requirements. For example, an SQS queue can be an EventBridge target, allowing the event to be routed to a durable queue for later processing.

This architecture separates the order API from slower background tasks. It also lets different components scale and fail independently to a degree, provided that retries, error handling, and downstream dependencies are designed correctly.

What happens when the order is placed?

The React frontend sends a request to API Gateway.
Lambda validates the request and saves the order in DynamoDB.
The application publishes an order event.
EventBridge evaluates its rules and routes matching events to configured targets.
Background processing runs independently, potentially through SQS and a worker.
SNS can distribute notifications to subscribers when needed.

One important production consideration: don't assume that saving an order and publishing an event are automatically one atomic operation. If the database write succeeds but event publishing fails, the system may need a reliable publishing strategy, such as the transactional outbox pattern, to avoid losing important events.

7. Reliability: Retries, Dead-Letter Queues, and Duplicate Messages

Asynchronous systems are powerful, but they introduce their own challenges.

What happens if a worker crashes while processing a message? What if a message is delivered more than once?

These are normal distributed-system problems.

Retries

Temporary failures may be resolved by retrying the operation. However, retries should be configured carefully to avoid overwhelming a failing dependency.

Dead-letter queues

A dead-letter queue (DLQ) can hold messages that exceed the configured retry or receive-count threshold. This gives you a way to investigate and recover messages that repeatedly fail.

Idempotency

An operation is idempotent when repeating it produces the same intended result rather than performing the effect multiple times.

For example, if a worker receives the same order-processing message twice, it should not deduct inventory twice unintentionally.

SQS Standard queues can deliver duplicate messages, and other event-driven workflows can also retry delivery. Design consumers with this possibility in mind.

Monitoring

CloudWatch metrics, logs, and alarms can help identify growing queue backlogs, processing failures, throttling, and other problems.

The goal isn't to assume messages will never fail. It's to design the application so failures can be detected and handled safely.

8. When Should You Use Each Service?

Choose the service based on the communication problem you're trying to solve.

Use Amazon SQS when:

Work needs to wait until a consumer is ready.
You want to separate request handling from background processing.
You need to buffer traffic spikes.
Consumers need to retry processing failed messages.

Use Amazon SNS when:

One message needs to reach multiple subscribers.
You want a publish/subscribe communication pattern.
Multiple systems need to react to the same notification.

Use Amazon EventBridge when:

Events need to be routed based on their content.
AWS services or applications generate events.
Different event types need different processing workflows.
You want event-driven integration without hardcoding every destination into the producer.

These are common patterns, not strict boundaries. Real architectures often combine these services.

9. My Main Takeaways

As I continue learning AWS, I'm realizing that building a reliable application involves more than deploying a backend and connecting a database.

It also means understanding how different components communicate, how failures are handled, and how background work can be processed without slowing down the user experience.

Here is the mental model I use:

Amazon SQS queues work for consumers.
Amazon SNS distributes messages to subscribers.
Amazon EventBridge routes events using rules.
AWS Lambda can execute event-driven processing.
Amazon DynamoDB can store application data.
Amazon CloudWatch helps monitor and troubleshoot the system.

The key lesson is that these services solve different communication problems. Choosing the right one depends on whether you need a queue, a notification mechanism, or event-based routing.

And in production, retries, duplicate handling, permissions, monitoring, and cost still matter.

I'm documenting my AWS learning journey one concept at a time. If you're learning cloud computing too, I'd love to hear what you're building and which AWS services you're exploring.

Can You Build a Backend Without Managing Servers? Understanding Amazon API Gateway, AWS Lambda & DynamoDB

Anupa Supul — Tue, 20 Oct 2026 04:56:12 +0000

When I first started learning AWS, I thought deploying a backend meant creating an EC2 instance, installing Node.js, configuring a web server, and keeping everything running.

That approach works. But what if you could build an API without managing the underlying servers yourself?

That is where serverless architecture comes in.

In this article, I’ll explore three AWS services that work well together to build serverless backends:

Amazon API Gateway — receives and manages API requests.
AWS Lambda — executes backend code.
Amazon DynamoDB — stores and retrieves data.

We’ll also look at how these services connect, explore a small Node.js example, and discuss when serverless makes sense in a real application.

1. What Does Serverless Actually Mean?

Serverless does not mean there are no servers.

Servers still exist. The difference is that AWS manages much of the underlying infrastructure, including server provisioning and infrastructure availability for these services.

Instead of managing a server yourself, you focus more on writing application code and defining how the services should work together.

For example, imagine you are building a task management application.

Users should be able to:

Create a task.
View their tasks.
Update a task.
Delete a task.

With a traditional backend, you might deploy a Node.js application on EC2 or ECS.

With a serverless approach, you could use API Gateway, Lambda, and DynamoDB.

The basic flow looks like this:

User / React Frontend
         |
         v
   API Gateway
         |
         v
      Lambda
         |
         v
    DynamoDB
         |
         v
    JSON Response

Each service has a specific responsibility. This separation makes the architecture easier to understand and lets you scale different parts of the application according to their needs.

2. Amazon API Gateway — The Entry Point for Your API

Imagine your React frontend needs to request a list of tasks from your backend.

It might send a request such as:

GET /tasks

But where should that request go?

Amazon API Gateway provides an API entry point that clients can call. It can route requests to backend services such as Lambda functions.

It also supports capabilities such as authorization, request validation, throttling, and monitoring integrations, depending on the API type and configuration.

For example, you might define these routes:

HTTP Method	Endpoint	Purpose
GET	`/tasks`	Retrieve tasks
GET	`/tasks/{id}`	Retrieve one task
POST	`/tasks`	Create a task
PUT	`/tasks/{id}`	Update a task
DELETE	`/tasks/{id}`	Delete a task

Each route can be connected to a suitable Lambda function.

For example:

GET /tasks
    |
    v
getTasks Lambda

POST /tasks
    |
    v
createTask Lambda

API Gateway can also help protect an API from excessive requests through throttling and integrate with supported authorization mechanisms.

One important point: API Gateway is not your database, and it does not automatically make every API secure. Authentication, authorization, validation, and appropriate access policies still need to be configured.

3. AWS Lambda — Run Backend Code Without Managing Servers

Now let's look at the service that executes our backend logic.

AWS Lambda runs code in response to events. An API Gateway request can trigger a Lambda function, which processes the request and returns a response.

You don't need to manage an EC2 instance just to keep this function available.

For example, a createTask Lambda function might:

Receive the incoming request.
Read the task details.
Validate the input.
Store the task in DynamoDB.
Return a response to the client.

A simplified flow looks like this:

API Gateway
     |
     v
Lambda Function
     |
     +--> Validate request
     |
     +--> Execute business logic
     |
     +--> Save data in DynamoDB
     |
     v
Return JSON response

Lambda supports multiple runtimes, including Node.js and Python.

A simple Node.js Lambda example

Here is a minimal Lambda handler that returns a JSON response.

export const handler = async (event) => {
  return {
    statusCode: 200,
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      message: "Hello from AWS Lambda!",
      method: event.requestContext?.http?.method,
    }),
  };
};

This example assumes an API Gateway HTTP API using its payload format that exposes the HTTP method under requestContext.http.method.

When API Gateway invokes the function, Lambda returns a response that API Gateway can send back to the client.

This is only a demonstration. A production API would also need suitable error handling, input validation, authentication, and logging.

What happens when traffic increases?

Suppose your application normally receives 20 requests per minute, but suddenly receives thousands.

Lambda can scale the number of concurrent function executions in response to incoming demand, subject to account quotas, concurrency settings, and other service limits.

You don't manually create more EC2 instances just to handle every increase in traffic.

However, scaling is not unlimited or instantaneous in every situation. Concurrency quotas, downstream database capacity, throttling, and cold starts can affect performance.

That's why serverless applications still require thoughtful design.

4. Amazon DynamoDB — Store Application Data

Our backend now has an entry point and a way to execute code. But where should the application store its data?

For this example, we'll use Amazon DynamoDB.

DynamoDB is a managed NoSQL database service that supports key-value and document data models.

Unlike a traditional relational database, DynamoDB does not require you to design every relationship around SQL tables and joins. Instead, you design tables and indexes around the access patterns your application needs.

Imagine storing tasks with fields such as:

{
  "userId": "user-123",
  "taskId": "task-001",
  "title": "Learn AWS Lambda",
  "completed": false
}

A DynamoDB table could use:

userId as the partition key.
taskId as the sort key.

This key design makes it possible to retrieve tasks belonging to a particular user using a query on that user's partition key.

The actual design should depend on the application's access patterns. If you need to search tasks by status, due date, or another attribute, you may need additional indexes or a different key design.

Why choose DynamoDB?

Some useful characteristics include:

Managed infrastructure.
Flexible scaling options.
Low-latency access for suitable key-based access patterns.
Integration with Lambda and other AWS services.
On-demand capacity options for workloads where usage is difficult to predict.

But DynamoDB is not automatically the best database for every application.

If your application relies heavily on relational joins, complex SQL queries, or relational constraints, Amazon RDS or Amazon Aurora may be more suitable.

The important lesson is to choose a database based on your data model and access patterns, not simply because it is serverless.

5. Connecting the Architecture

Let's put the three services together.

Architecture overview: the frontend sends an API request, API Gateway routes it to Lambda, and Lambda interacts with DynamoDB before returning a response.

Here is the same architecture as a text diagram:

                 USER
                  |
                  v
            REACT FRONTEND
                  |
             HTTPS Request
                  |
                  v
           AMAZON API GATEWAY
                  |
           Route to Lambda
                  |
                  v
             AWS LAMBDA
                  |
          Validate and Process
                  |
                  v
             AMAZON DYNAMODB
                  |
             Query / Write
                  |
                  v
             AWS LAMBDA
                  |
            JSON Response
                  |
                  v
            API GATEWAY
                  |
                  v
            REACT FRONTEND

Let's walk through a request to create a task.

The user fills out a form in the React application.
The frontend sends a POST /tasks request.
API Gateway receives the request and invokes the configured Lambda function.
Lambda validates the input and executes the business logic.
Lambda writes the task to DynamoDB using the AWS SDK.
Lambda returns a response.
API Gateway sends the response to the frontend.
React updates the interface to show the result.

This architecture separates the frontend, API entry point, application logic, and database responsibilities.

For a small application, it can be a straightforward way to build and deploy a backend. As the application grows, you'll need to think more carefully about permissions, observability, data access patterns, and failure handling.

6. A Practical Node.js Example: Saving a Task to DynamoDB

Let's make the example a little more realistic.

Suppose we have a DynamoDB table named Tasks with userId as its partition key and taskId as its sort key.

The Lambda function below creates a task using the AWS SDK for JavaScript v3.

Install the required package in your project:

npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

Create a file named index.mjs:

import { randomUUID } from "node:crypto";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
  DynamoDBDocumentClient,
  PutCommand,
} from "@aws-sdk/lib-dynamodb";

const client = new DynamoDBClient({});
const dynamoDB = DynamoDBDocumentClient.from(client);

const TABLE_NAME = process.env.TABLE_NAME;

export const handler = async (event) => {
  try {
    if (!TABLE_NAME) {
      throw new Error("TABLE_NAME is not configured");
    }

    const body = JSON.parse(event.body || "{}");
    const { userId, title } = body;

    if (
      typeof userId !== "string" ||
      !userId.trim() ||
      typeof title !== "string" ||
      !title.trim()
    ) {
      return {
        statusCode: 400,
        body: JSON.stringify({
          message: "userId and title are required",
        }),
      };
    }

    const task = {
      userId: userId.trim(),
      taskId: randomUUID(),
      title: title.trim(),
      completed: false,
      createdAt: new Date().toISOString(),
    };

    await dynamoDB.send(
      new PutCommand({
        TableName: TABLE_NAME,
        Item: task,
        ConditionExpression:
          "attribute_not_exists(userId) AND attribute_not_exists(taskId)",
      })
    );

    return {
      statusCode: 201,
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(task),
    };
  } catch (error) {
    console.error("Failed to create task:", error);

    return {
      statusCode: 500,
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        message: "Unable to create task",
      }),
    };
  }
};

This example demonstrates the basic flow, but there are a few important details to understand.

First, the table name comes from an environment variable. Configure TABLE_NAME in the Lambda function's environment settings rather than hardcoding the table name throughout your code.

Second, the function uses the AWS SDK's default credential provider chain. When deployed to Lambda, it can obtain temporary credentials from the function's execution role. You generally should not put permanent AWS access keys in your application code.

Third, the conditional expression prevents overwriting an item with the same composite primary key. Because this example generates a UUID for each new task, a collision is extremely unlikely, but the condition still makes the intended behavior explicit.

Fourth, the example is simplified. It accepts a userId from the request body for demonstration. In a real multi-user application, derive the user's identity from a verified authentication mechanism rather than trusting a client-supplied user ID. Otherwise, a user might attempt to create or access another user's data.

Also, the catch block returns a generic error to the client while logging the underlying error for troubleshooting. Production applications should distinguish expected client errors from unexpected server failures and avoid exposing internal details.

Important: configure IAM permissions

The Lambda execution role needs permission to write to the intended DynamoDB table.

A tightly scoped policy should grant only the required action, such as dynamodb:PutItem, on the specific table ARN. If the function later needs to query or update tasks, grant only those additional actions.

Avoid giving the function broad permissions such as dynamodb:* on every table in the account.

Important: configure the DynamoDB table

The table must exist before this function can use it. Its primary key must match the attributes in the item:

Partition key: userId (String).
Sort key: taskId (String).

Configure the TABLE_NAME environment variable with the table's actual name.

This code is a handler example, not a complete deployment tutorial. You still need to create the table, configure the Lambda runtime and execution role, connect the function to API Gateway, and test the request and response formats.

7. What About Authentication and Security?

A serverless backend still needs security. AWS managing the infrastructure does not automatically secure your application's business logic or data.

Here are some services and controls that may be useful.

AWS IAM

IAM controls which AWS resources and actions an identity can access.

For this architecture, the Lambda execution role should grant only the permissions the function needs, such as writing to one DynamoDB table.

Amazon Cognito or another authentication provider

If users need to sign in, you need an authentication mechanism.

Amazon Cognito can provide user sign-up, sign-in, and token-based authentication capabilities. API Gateway can be configured to validate supported authorization tokens, depending on the API type and setup.

Authentication identifies the caller. Your application must still enforce authorization: a signed-in user should not automatically have access to every other user's tasks.

AWS Secrets Manager

If the application needs external API credentials or other secrets, Secrets Manager can store and manage them.

DynamoDB access through the Lambda execution role generally does not require a database password. Avoid adding unnecessary credentials simply because other backend architectures use them.

Amazon CloudWatch

CloudWatch can collect Lambda logs, metrics, and alarms. These help you investigate errors, monitor latency, and detect unusual invocation or throttling patterns.

A simple security model

Frontend
   |
   v
API Gateway
   |
   +--> Validate authentication
   |
   v
Lambda Execution Role
   |
   +--> Only required DynamoDB actions
   |
   v
DynamoDB Table

The exact configuration depends on the application's requirements, but the core principle stays the same: grant only the permissions required, and verify who is allowed to access each user's data.

8. What Happens When Traffic Increases?

One reason serverless architecture is attractive is that it can reduce the amount of infrastructure management required as demand changes.

However, each component has its own limits and scaling behavior.

API Gateway

API Gateway can handle API traffic and supports configurable throttling and quotas. Your API may still receive throttling responses if configured limits or service quotas are exceeded.

Lambda

Lambda can increase concurrent executions as demand grows, within applicable concurrency quotas and account limits.

But more concurrent Lambda executions can also increase pressure on downstream services. If a function triggers expensive database operations, scaling the function alone will not solve the underlying bottleneck.

DynamoDB

DynamoDB offers on-demand and provisioned capacity modes. The appropriate choice depends on traffic patterns, predictability, and cost requirements.

Your key design and indexes also matter. A poorly designed access pattern can create hot partitions or inefficient queries even when the database itself is managed.

What about cold starts?

A cold start can occur when Lambda needs to initialize a new execution environment before processing a request. This may add latency.

The effect depends on factors such as runtime, initialization work, dependencies, memory configuration, and the execution environment.

For many applications, occasional cold starts are acceptable. For latency-sensitive workloads, measure actual performance and evaluate options such as reducing initialization work or using provisioned concurrency where appropriate.

Serverless makes infrastructure management easier in many cases, but it doesn't remove the need for performance testing.

9. Serverless Is Not Always Cheaper

This is an important lesson: serverless is a deployment model, not a guarantee of lower cost.

With Lambda, costs can depend on request volume, execution duration, configured memory, and other applicable charges.

API Gateway pricing depends on the API type and usage. DynamoDB costs depend on the capacity mode, requests, storage, and additional features.

For a small application with occasional requests, a serverless design can be cost-effective because you don't need to maintain an always-running application server for every request.

For a high-volume application with predictable, sustained workloads, another architecture might be more economical. You need to compare actual usage and current pricing rather than assume one model always wins.

Remember to consider supporting services, logging volume, data transfer, provisioned capacity, and other configuration choices when estimating costs.

10. Lambda vs EC2 vs ECS/Fargate

Since I've been learning EC2, Docker, Amazon ECR, ECS, and Fargate, I find it useful to compare these approaches.

Feature	AWS Lambda	EC2	ECS with Fargate
Infrastructure management	AWS manages the execution infrastructure	You manage the instances and operating system	AWS manages the underlying compute infrastructure
Application model	Event-driven functions	Applications or services on virtual machines	Containerized applications
Scaling	Based on invocations and concurrency settings	Configure scaling for instances and workloads	Configure service/task scaling
Long-running processes	Subject to function execution limits	Suitable	Suitable, subject to task and service configuration
Runtime control	Supported Lambda runtimes and execution model	High control	Container-level control
Operational effort	Often lower for suitable workloads	Often higher	Moderate, depending on the application
Good fit	APIs, event handlers, scheduled tasks	Custom server environments and broad workloads	Containerized APIs, workers, and services

Lambda is especially useful when work can be broken into event-driven executions.

EC2 is useful when you need control over the virtual machine and its environment.

ECS with Fargate is useful when you want to run containers without managing the underlying EC2 instances yourself.

These services are not mutually exclusive. A real application can use Lambda for event-driven operations and ECS/Fargate for a long-running API or worker.

Choose based on the workload, operational requirements, performance expectations, and cost—not just which service sounds newer.

11. What I Would Consider Before Deploying to Production

Before deploying a serverless API for real users, I would check the following:

[ ] Configure API routes and appropriate authorization.
[ ] Validate request bodies and query parameters.
[ ] Derive user identity from verified authentication rather than trusting client-supplied identity fields.
[ ] Use a least-privilege Lambda execution role.
[ ] Design DynamoDB keys and indexes around real access patterns.
[ ] Configure environment variables and manage any required secrets safely.
[ ] Add useful logs, metrics, alarms, and tracing where appropriate.
[ ] Handle errors and retries carefully.
[ ] Make operations safe to retry where duplicate processing is possible.
[ ] Set suitable timeouts, concurrency limits, and throttling.
[ ] Test under realistic traffic and estimate costs.
[ ] Deploy through a repeatable infrastructure-as-code and CI/CD process.

One detail worth emphasizing is idempotency. Distributed systems can retry requests or events after failures. If a repeated operation would create duplicate records, charge a customer twice, or trigger the same action repeatedly, the application needs a strategy to prevent that.

For asynchronous workflows, services such as Amazon SQS can help decouple components. But queue-based processing also requires careful retry handling, dead-letter queues where appropriate, and idempotent consumers.

12. When Should You Use This Architecture?

A serverless API built with API Gateway, Lambda, and DynamoDB is worth considering when:

You are building a small or medium-sized API.
Traffic is variable or difficult to predict.
You want to reduce server administration.
Your application fits DynamoDB's data model and access patterns.
You want to integrate backend operations with events from other AWS services.
You prefer managed services for the initial deployment.

It may be less suitable when:

Your application depends heavily on relational queries and complex joins.
You need sustained, long-running processing beyond Lambda's execution limits.
You require very predictable low latency and have not accounted for initialization behavior.
Your workload has steady high utilization and another compute model may offer better economics.
You need operating-system or runtime control that the chosen serverless service does not provide.

These are trade-offs, not absolute rules. The best architecture depends on the actual requirements.

13. My Main Takeaways

After learning about API Gateway, Lambda, and DynamoDB, I think the biggest benefit of serverless is the change in focus.

Instead of spending as much time managing servers, you can spend more time designing APIs, writing business logic, securing data access, and understanding how application components interact.

Here is the mental model I use:

API Gateway receives and routes API requests.
Lambda executes backend logic.
DynamoDB stores and retrieves data.
IAM controls AWS permissions.
Cognito or another identity provider can support authentication.
CloudWatch helps monitor and troubleshoot the system.
SQS can decouple asynchronous work when the application needs it.

But serverless does not mean architecture disappears. You still need to design for security, failures, retries, performance, and cost.

That is what makes learning AWS interesting: the goal isn't simply to learn what each service does. It's to understand how to combine the services to solve a real problem.

What's next?

I'm continuing my AWS learning journey by exploring how these services fit into real application architectures, from containerized backends to serverless APIs and automated deployments.

If you're also learning AWS, I'd love to hear what you're building and which services you're exploring.

Stop Hardcoding Secrets: How Production Applications Manage Credentials on AWS

Anupa Supul — Mon, 12 Oct 2026 05:50:35 +0000

Imagine you're building a Node.js backend using Express and MongoDB.

During development, you might have a .env file containing:

PORT=5000
MONGO_URI=mongodb+srv://...
JWT_SECRET=...
API_KEY=...

Your application works perfectly.

But now imagine pushing the project to GitHub, deploying it through a CI/CD pipeline, and running it on AWS.

Where should those secrets live?

Should we put them directly inside the source code? Add them to the Docker image? Store them in GitHub? Or let every developer manage their own copies?

These questions become important when an application moves from a local development environment to production.

In this article, we'll explore how AWS Secrets Manager, Systems Manager Parameter Store, and IAM help us manage application configuration and credentials more securely.

1. Why Hardcoding Secrets Is Dangerous

Let's start with a simple example.

const dbPassword = "my-super-secret-password";
const jwtSecret = "my-jwt-secret";

This may look convenient, but it creates a serious security problem.

Anyone who can access the source code may also be able to access the credentials.

If we push these values to a public repository, the secrets could be exposed. Even a private repository can have too many people or automated systems with access.

There is another problem: what happens when a password changes?

We may need to modify the code, rebuild the Docker image, and deploy the application again just to update a credential.

Application code and sensitive configuration should be managed separately.

That is the principle behind secure secrets management.

2. What About the `.env` File?

A .env file is useful during local development.

For example:

PORT=5000
MONGO_URI=your_database_connection_string
JWT_SECRET=your_jwt_secret

Your Node.js application can load these values using a package such as dotenv.

require("dotenv").config();

const port = process.env.PORT;
const mongoUri = process.env.MONGO_URI;

This keeps configuration out of the source code itself.

However, a .env file is not automatically a secure secrets management system.

For local development, we should keep it out of Git using .gitignore.

.env
.env.*
!.env.example

An .env.example file can document the required variable names without containing real credentials.

For production, we need a controlled way to provide those values to our running application.

That's where AWS services come in.

3. AWS Secrets Manager — Store Sensitive Information

AWS Secrets Manager is designed to store, retrieve, and manage sensitive information such as database credentials, API keys, and other application secrets.

For example, we could store a secret containing:

Secret name: production/my-backend

MONGO_URI: ...
JWT_SECRET: ...
THIRD_PARTY_API_KEY: ...

The application retrieves the required secret when it needs it, provided it has the appropriate IAM permissions.

A simplified architecture looks like this:

Developer
    |
    v
AWS Secrets Manager
    |
    | Authorized access
    v
Application on ECS / Fargate
    |
    v
Connect to Database / External API

The important point is that the sensitive values do not need to be hardcoded into our application source code or baked into the Docker image.

Secrets Manager also supports secret rotation workflows, including integrations and rotation configurations for supported services.

Rotation still needs to be configured correctly; storing a secret in Secrets Manager does not automatically mean every secret will rotate itself.

4. Systems Manager Parameter Store — Manage Configuration

Not every configuration value is a secret.

Consider these examples:

APP_ENV=production
LOG_LEVEL=info
MAX_RETRIES=3
API_BASE_URL=https://api.example.com

These values may need to change between development, testing, and production, but they are not necessarily confidential.

AWS Systems Manager Parameter Store provides a central place to store application configuration and parameters.

It supports different parameter types, including:

String — ordinary configuration values.
StringList — comma-separated string values.
SecureString — sensitive values encrypted using AWS KMS.

For example:

/my-app/production/LOG_LEVEL
/my-app/production/API_BASE_URL
/my-app/production/DB_PASSWORD

Parameter Store can help organize configuration by application and environment.

A SecureString can be useful for encrypted parameters, while Secrets Manager is often a better fit when managed secret lifecycle and rotation features are important.

The choice depends on the requirements rather than one service being universally better.

5. Secrets Manager vs Parameter Store

This is where many developers get confused.

Both services can store sensitive configuration, but their strengths differ.

Feature	Secrets Manager	Parameter Store
Main purpose	Secret lifecycle management	Configuration and parameter management
Database credentials	Yes	Can store encrypted values
Encrypted values	Yes	Yes, using `SecureString`
Secret rotation	Built-in rotation capabilities and integrations	Usually requires a separate rotation workflow
Ordinary configuration	Possible	A natural fit
Cost considerations	Generally charged for secrets and API usage	Standard parameters have a free tier; advanced features can incur charges

For example, I might choose Secrets Manager for database credentials that require a managed rotation workflow and Parameter Store for ordinary application settings.

Both can be useful in the same application.

6. IAM — Who Is Allowed to Read the Secrets?

Storing secrets securely is only half of the problem.

We also need to decide who or what is allowed to access them.

This is where AWS Identity and Access Management (IAM) comes in.

IAM controls access to AWS resources through identities, policies, and roles.

Imagine our Node.js backend is running as an ECS task on Fargate.

We want it to retrieve one database secret from Secrets Manager.

We should not give the application unrestricted access to every secret in the AWS account.

Instead, we can grant the task permission to retrieve only the required secret.

ECS Task
   |
   v
Task Role
   |
   v
IAM Policy
   |
   v
Allowed Secret Only

This follows the principle of least privilege: give each component only the permissions it needs.

For ECS, there are two roles worth distinguishing:

Task execution role: Used by ECS/Fargate for operations such as pulling private images from ECR and sending logs to supported destinations.
Task role: Provides AWS permissions to the application code running inside the container.

If the application itself calls Secrets Manager, its task role generally needs the appropriate permission.

If ECS retrieves secrets and injects them into a container as environment variables, the task execution role needs the relevant permissions instead.

Getting this distinction right helps avoid unnecessary permissions and deployment failures.

7. How Does This Work with Docker and ECS?

Let's connect secrets management to the previous article about Docker, ECR, ECS, and Fargate.

A common mistake is putting production credentials directly into the Dockerfile.

For example, this is a bad idea:

ENV JWT_SECRET=my-production-secret

Why?

Because the value can become part of the image configuration and may be exposed to anyone who can inspect the image.

Instead, the image should contain the application, while production configuration is supplied separately.

                 Docker Build
                     |
                     v
              Application Image
                     |
                     v
                 Amazon ECR
                     |
                     v
                ECS / Fargate
                     |
                     v
           Runtime Configuration
                     |
           +---------+----------+
           |                    |
           v                    v
     Secrets Manager       Parameter Store

For example, an ECS task definition can reference a secret stored in Secrets Manager and make it available to the container at startup.

The application can then read the value through its environment.

One important detail: when ECS injects a secret into an environment variable, the value is retrieved when the container starts. If the secret changes later, an already-running container does not automatically receive the updated value. A new task deployment or restart may be needed.

Also, environment variables are convenient, but they can be exposed through debugging tools, process inspection, or logs if handled carelessly. They should never be printed unnecessarily.

8. Example: A Node.js Backend on AWS

Imagine we deploy a Node.js application using:

GitHub Actions for CI/CD.
Docker for packaging.
Amazon ECR for storing the image.
Amazon ECS with Fargate for running the backend.
Amazon RDS for the database.
AWS Secrets Manager for database credentials.
IAM for controlling access.

The application flow looks like this:

Users
  |
  v
Application Load Balancer
  |
  v
ECS / Fargate
  |
  +-------> Secrets Manager
  |          Database Credentials
  |
  +-------> Amazon RDS
              Application Data

The backend needs permission to retrieve the secret and network access to connect to the database.

The database should normally be placed in private subnets, with security group rules that allow connections only from the appropriate application tasks.

Secrets Manager handles secret storage, IAM controls authorization, and the VPC and security groups control network access.

These are separate security layers working together.

9. What About GitHub Actions and CI/CD?

Our previous article explained how GitHub Actions can build Docker images, push them to ECR, and deploy them to ECS.

The pipeline also needs secure access to AWS.

A recommended approach is to configure OpenID Connect (OIDC) so GitHub Actions can assume an IAM role and obtain temporary AWS credentials.

GitHub Actions
       |
       v
      OIDC
       |
       v
    AWS IAM Role
       |
       v
     Amazon ECR
       |
       v
     Amazon ECS

The workflow role should have only the permissions required for its deployment responsibilities.

This is different from the IAM task role used by the application after deployment.

We should also avoid printing secret values in CI logs or passing credentials through command-line arguments unnecessarily.

The goal is to secure both the deployment process and the application runtime.

10. What Should We Choose?

Here is a practical way to decide.

Use Secrets Manager when:

You need centralized storage for sensitive credentials.
You need secret rotation capabilities.
Your application needs controlled access to database passwords or API keys.

Use Parameter Store when:

You need centralized application configuration.
You want to organize settings by environment.
You need encrypted parameters and don't require the full Secrets Manager lifecycle features.

Use IAM when:

You need to control which users, services, or workloads can access AWS resources.
You want applications to use roles instead of hardcoded AWS access keys.
You want to enforce least-privilege permissions.

These services are complementary, not direct replacements for one another.

Quick Cheat Sheet

Tool	Main Responsibility
`.env`	Local application configuration
`.gitignore`	Prevents local environment files from being committed
AWS Secrets Manager	Stores and manages sensitive secrets
Systems Manager Parameter Store	Stores configuration and parameters
`SecureString`	Encrypted Parameter Store value
IAM Role	Grants AWS permissions to an identity or workload
ECS Task Role	Permissions used by application code
ECS Task Execution Role	Permissions used by ECS/Fargate for task startup and related operations
GitHub OIDC	Provides a way for GitHub Actions to obtain temporary AWS credentials
AWS KMS	Manages encryption keys used by supported AWS services

Final Thoughts

Moving an application to AWS is not just about getting it to run.

We also need to think about where its credentials live, who can access them, how they are updated, and what happens when those credentials change.

A local .env file can be useful during development, but production applications need a more controlled approach.

Secrets Manager, Parameter Store, and IAM help separate application code from sensitive configuration and enforce controlled access.

The key lesson is simple:

Don't hardcode secrets. Don't give every component unrestricted access. Keep configuration, permissions, and application code separate.

As I continue learning AWS, I'm finding that production engineering is not only about deploying applications. It's also about making sure they remain secure, maintainable, and reliable after deployment.

What Happens When You Push Code to Production? Understanding AWS CI/CD with GitHub Actions, Docker, ECR & ECS

Anupa Supul — Mon, 12 Oct 2026 05:43:14 +0000

Imagine you have finished building a Node.js backend. Everything works on your laptop, and you are ready to deploy it.

Traditionally, you might connect to a server, pull the latest code, install dependencies, restart the application, and hope nothing breaks.

But what happens when you need to do this every week? Every day? Or several times a day?

Manually repeating those steps becomes slow, error-prone, and difficult to manage.

This is where CI/CD becomes important.

After learning Docker, Amazon ECR, ECS, and Fargate, the next question is how to connect them into an automated deployment pipeline.

Let's follow a simple example: deploying a Node.js application to AWS whenever new code is pushed to GitHub.

1. First, What Is CI/CD?

CI/CD is a set of practices that helps developers build, test, and deliver software more reliably.

Continuous Integration (CI) means regularly integrating code changes and automatically checking whether the application still works.

Continuous Delivery (CD) means keeping the application ready for release. With Continuous Deployment, changes that pass the required checks can be deployed automatically to production.

Think about a team working on the same backend.

One developer fixes a login bug. Another adds a new API endpoint. Someone else updates the database integration.

Without automation, every change needs manual checking and deployment.

With CI/CD, a pipeline can automatically test the changes, build the application, package it into a Docker image, and deploy a new version.

The basic idea is:

Write code → Test it → Build it → Deploy it

2. The Complete AWS CI/CD Architecture

Here is the architecture we will use throughout this article.

                 DEVELOPER
                     |
                     v
                   GitHub
                     |
                     v
              GitHub Actions
                     |
              +------+------+
              |             |
              v             v
          Install         Run Tests
         Dependencies       |
              |             |
              +------+------+
                     |
                     v
               Build Docker Image
                     |
                     v
                Amazon ECR
             (Image Registry)
                     |
                     v
                Amazon ECS
                     |
                     v
                AWS Fargate
                     |
                     v
               Running Tasks
                     |
                     v
               Production App

This pipeline separates the deployment process into clear responsibilities.

GitHub stores the source code.
GitHub Actions automates the workflow.
Docker packages the application.
Amazon ECR stores the container image.
Amazon ECS manages the deployment and running tasks.
AWS Fargate runs the containers without requiring us to manage the underlying EC2 servers.

Each component solves a different problem.

3. Step One: Push Code to GitHub

Let's say we update our backend.

We add a new endpoint, fix a bug, or improve an existing feature.

After committing the changes, we push them to GitHub.

Developer
    |
    v
Local Code Changes
    |
    v
Git Commit
    |
    v
Git Push
    |
    v
GitHub Repository

GitHub becomes the starting point for our deployment workflow.

But pushing code to GitHub does not automatically mean that the code is safe to deploy. We still need to check it.

That is the responsibility of the next stage.

4. Step Two: GitHub Actions Runs the Pipeline

GitHub Actions is a CI/CD automation platform built into GitHub.

We can define a workflow that runs when someone pushes code to a particular branch or opens a pull request.

For example:

Code pushed to main
        |
        v
GitHub Actions starts
        |
        v
Install dependencies
        |
        v
Run automated tests
        |
        v
Continue if checks pass

A workflow is defined in a YAML file inside the repository, usually under:

.github/workflows/

A simplified workflow might look like this:

name: Backend CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

This example focuses on continuous integration. It checks the code and runs tests, but it does not yet build or deploy the Docker image.

If the tests fail, we can configure the pipeline to stop before deployment.

That is important because automation should prevent broken code from reaching production, not just deploy faster.

5. Step Three: Build the Docker Image

Once the required checks pass, the next stage is packaging the application.

This is where Docker connects with the pipeline.

Source Code
     |
     v
Dockerfile
     |
     v
Docker Build
     |
     v
Docker Image

The Docker image contains the application code, its dependencies, and the instructions needed to run it.

For example, a Node.js backend might use a Dockerfile like this:

FROM node:22

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .

EXPOSE 3000

CMD ["npm", "start"]

This is a simplified example. A production Dockerfile should also consider security, image size, dependency management, and which files should be excluded using .dockerignore.

The important point is that the pipeline can build a new image whenever the application changes.

Instead of manually preparing a server for every release, we produce a consistent application package.

6. Step Four: Push the Image to Amazon ECR

Now we have a Docker image. Where should it go?

Amazon Elastic Container Registry (ECR) is AWS's managed container image registry.

GitHub Actions can authenticate with AWS, build the image, tag it, and push it to an ECR repository.

GitHub Actions
      |
      v
Docker Image
      |
      v
Amazon ECR
      |
      v
Versioned Image

For example, we might tag images using the Git commit SHA:

my-backend:a1b2c3d
my-backend:d4e5f6a

Each tag identifies a particular build. This makes it easier to identify which version is running and to deploy a previous version when needed.

For production, avoid relying only on the mutable latest tag. Use explicit version tags and, where appropriate, immutable image digests.

One important security detail: GitHub Actions needs permission to access ECR. A recommended approach is to use AWS IAM roles with GitHub Actions OpenID Connect (OIDC), so the workflow can obtain temporary AWS credentials without storing long-lived AWS access keys in GitHub secrets.

7. Step Five: Deploy the New Image to Amazon ECS

The image is now stored in ECR, but it is not running yet.

We need ECS to deploy it.

Amazon ECS manages our container workloads. An ECS task definition describes how a container should run, including its image, CPU, memory, ports, and configuration.

When we deploy a new application version, the pipeline can register or reference an updated task definition and ask the ECS service to deploy it.

New Docker Image
       |
       v
Amazon ECR
       |
       v
Updated Task Definition
       |
       v
Amazon ECS Service
       |
       v
New Running Tasks

ECS then manages the deployment according to the service's configuration.

If we use Fargate, AWS manages the underlying compute infrastructure for the tasks.

We still need to configure the task's networking, IAM permissions, resource requirements, logging, and other application settings.

The benefit is that we no longer need to manually log into an EC2 instance to install the application and restart the process after every release.

8. Step Six: How Does the New Version Reach Users?

Imagine our backend currently has three running tasks:

ECS SERVICE — VERSION 1

Task 1  ── Version 1
Task 2  ── Version 1
Task 3  ── Version 1

Now we deploy Version 2.

ECS can use a rolling deployment to gradually replace the old tasks with new ones, according to the service's deployment settings.

DEPLOYMENT IN PROGRESS

Task 1  ── Version 2
Task 2  ── Version 1
Task 3  ── Version 1

Then:

Task 1  ── Version 2
Task 2  ── Version 2
Task 3  ── Version 1

Finally:

DEPLOYMENT COMPLETE

Task 1  ── Version 2
Task 2  ── Version 2
Task 3  ── Version 2

An Application Load Balancer can route requests to healthy registered targets while the deployment progresses.

However, a rolling deployment does not automatically guarantee zero downtime. That depends on settings such as health checks, minimum healthy capacity, maximum deployment capacity, and whether the new application version is compatible with existing clients and data.

This is why deployment configuration matters just as much as the pipeline itself.

9. What If the New Deployment Fails?

Suppose Version 2 contains a bug.

The application starts, but an API endpoint returns errors.

A reliable deployment process should not assume that a successful Docker build means the application is healthy.

We need checks at different stages:

Automated tests before building or deploying.
Container startup and health checks.
ECS service and deployment status.
Application logs and metrics.
Post-deployment smoke tests that verify important endpoints.

ECS can replace unhealthy tasks, but replacing a task is not the same as detecting every application bug. A task might be running while its business logic is broken.

For deployments that fail, ECS deployment circuit breaker settings can help detect failures and optionally roll back to a previous deployment. The pipeline can also stop or trigger further action when post-deployment checks fail.

The goal is to detect problems quickly and reduce the impact on users.

10. CI/CD Is More Than Just Deployment

A common misconception is that CI/CD simply means automatically uploading code to a server.

A good pipeline should help us deliver changes safely and consistently.

A more complete workflow might look like this:

Developer Pushes Code
          |
          v
   Automated Tests
          |
          v
   Code / Security Checks
          |
          v
    Build Docker Image
          |
          v
     Push Image to ECR
          |
          v
    Deploy to ECS
          |
          v
   Health / Smoke Tests
          |
          v
      Production

Depending on the project, we might also include dependency vulnerability scanning, infrastructure validation, manual approval before production, and notifications when a deployment succeeds or fails.

Not every project needs the same pipeline. The right setup depends on the application's risk, team size, compliance requirements, and deployment frequency.

11. What About Secrets and Permissions?

Automation needs access to AWS resources, but giving a pipeline unrestricted access is dangerous.

For example, GitHub Actions may need to push images to ECR and update an ECS service.

It should not automatically receive permission to modify every resource in the AWS account.

A safer design uses:

IAM: Grants only the permissions the workflow needs.
GitHub OIDC: Allows GitHub Actions to assume an AWS role using temporary credentials.
Secrets Manager or Parameter Store: Stores application secrets and configuration where appropriate.
ECS task roles: Allow the running application to access specific AWS services.
Security Groups and VPC networking: Restrict network communication between the load balancer, application tasks, and database.

It is also important to distinguish the deployment pipeline's permissions from the permissions of the running application. They serve different purposes and should be configured separately.

12. The Complete Deployment Architecture

Let's connect the pieces.

                    DEVELOPER
                        |
                        v
                      GitHub
                        |
                        v
                 GitHub Actions
                        |
              +---------+---------+
              |                   |
              v                   v
          Run Tests          Build Image
                                  |
                                  v
                             Amazon ECR
                                  |
                                  v
                             Amazon ECS
                                  |
                                  v
                             AWS Fargate
                                  |
                                  v
                          Running ECS Tasks
                                  |
                                  v
                                  ALB
                                  |
                                  v
                               USERS

This diagram shows the deployment pipeline and the running application, but they are two different flows.

Deployment flow:

Code → GitHub Actions → Docker → ECR → ECS

Application request flow:

Users → ALB → ECS Tasks → RDS / S3 / Other Services

In a complete web application, Route 53 and CloudFront can sit in front of the application, while networking, security, monitoring, and scaling support the system.

The CI/CD pipeline changes how we deliver application versions. It doesn't replace the architecture that serves user requests.

13. GitHub Actions vs AWS CodePipeline

One question worth asking is whether we must use AWS CodePipeline.

No.

GitHub Actions can handle CI/CD workflows directly from a GitHub repository. It can run tests, build Docker images, push them to ECR, and trigger ECS deployments.

AWS CodePipeline is another option for orchestrating release workflows using AWS services.

The choice depends on the existing development workflow and operational requirements.

GitHub Actions	AWS CodePipeline
Integrated with GitHub repositories	Integrated with AWS deployment workflows
Uses workflow YAML files	Uses pipeline stages and actions
Can build and deploy to AWS	Can orchestrate AWS and supported external actions
Convenient when source code is already on GitHub	Useful when a team wants an AWS-centered pipeline

Neither is automatically the better choice. A team should consider its existing tools, access controls, deployment process, and maintenance needs.

The Mental Model I Keep Now

Before learning CI/CD, my deployment mental model was:

Write Code
    |
    v
Deploy Manually
    |
    v
Restart Application

Now I think about the process as:

Code
 |
 v
GitHub
 |
 v
Automated Checks
 |
 v
Docker Image
 |
 v
Amazon ECR
 |
 v
Amazon ECS
 |
 v
AWS Fargate
 |
 v
Production

Each stage has a responsibility.

GitHub stores the code. GitHub Actions automates the workflow. Docker packages the application. ECR stores the image. ECS manages the deployment. Fargate runs the containers.

That is the connection between development and production.

Quick Cheat Sheet

Technology	Main Responsibility
GitHub	Source code management
GitHub Actions	CI/CD workflow automation
CI	Integrate changes and run checks
CD	Prepare or deliver changes to production
Docker	Package the application
ECR	Store container images
ECS	Manage container workloads and deployments
Task Definition	Describes how a task should run
ECS Service	Maintains desired tasks and manages deployments
Fargate	Runs containers without managing EC2 hosts
ALB	Routes requests to healthy targets
CloudWatch	Logs, metrics, dashboards, and alarms
IAM + OIDC	Control pipeline access to AWS

Final Thoughts

The biggest lesson for me is that deploying an application shouldn't depend on someone remembering a long list of manual steps.

A well-designed CI/CD pipeline makes those steps repeatable, testable, and easier to monitor.

It also connects the tools developers use every day with the AWS services that run their applications in production.

Docker taught me how to package an application. ECR gave that package a place to live. ECS and Fargate showed me how to run it on AWS.

CI/CD connects all of them into a repeatable delivery process.

And that's an important step toward understanding not just cloud infrastructure, but how modern software is delivered and operated.

The next time you push code to GitHub, think about everything that could happen automatically before that code reaches your users.

Code → Test → Build → Package → Deploy → Verify.

I Stopped Managing Servers Manually — Understanding Docker, Amazon ECR, ECS & Fargate

Anupa Supul — Tue, 15 Sep 2026 04:59:23 +0000

I Stopped Managing Servers Manually — Understanding Docker, Amazon ECR, ECS & Fargate

I had already learned how a web application can be built on AWS using EC2, Load Balancers, Auto Scaling, RDS, S3, SQS, and other services.

But there was still one question I kept coming back to:

What if I don't want to manage the server itself?

With EC2, I am still responsible for things like the operating system, installed software, runtime environment, and server configuration.

But modern applications are increasingly built around containers.

That led me to the next part of my AWS learning journey:

Docker → Amazon ECR → Amazon ECS → AWS Fargate

And once I understood how these four pieces connect, container deployment on AWS started making much more sense.

First, What Problem Are Containers Solving?

Imagine I build a Node.js application on my laptop.

It works perfectly.

Node.js version: 22
Express: installed
Dependencies: installed
Environment: configured
Application: running

Then I give the application to another developer.

Suddenly:

“It works on my machine.”

Maybe they have a different Node.js version.

Maybe a dependency is missing.

Maybe an environment variable isn't configured.

Maybe the operating system behaves differently.

This is one of the problems containers help solve.

Instead of only shipping our application code, we can package the application together with the environment it needs to run.

That's where Docker comes in.

1. Docker — Package the Application

Docker allows us to package an application into a container image.

For a Node.js backend, the image can contain:

Node.js
Application code
Dependencies
Configuration needed to run

A simplified flow looks like this:

Application Code
       │
       ▼
   Dockerfile
       │
       ▼
 Docker Image
       │
       ▼
   Container

The important distinction is:

Image = packaged blueprint

Container = running instance of that image

For example:

Docker Image
     │
     ├── Container 1
     ├── Container 2
     └── Container 3

This makes it much easier to run the same application consistently across different environments.

2. Dockerfile — The Instructions for the Image

A Docker image is usually created from a Dockerfile.

A simplified Node.js example could look like:

FROM node:22

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

EXPOSE 3000

CMD ["npm", "start"]

I don't need to manually install Node.js on every server.

The Docker image already defines what the application needs.

So instead of saying:

“Please configure a server exactly like my development machine.”

I can say:

“Run this container.”

That is a much more repeatable deployment model.

3. But Where Does the Docker Image Go?

Now we have a Docker image.

But AWS needs somewhere to store it.

This is where Amazon Elastic Container Registry (ECR) comes in.

ECR is AWS's managed container image registry.

The flow becomes:

Developer
    │
    ▼
Dockerfile
    │
    ▼
Docker Build
    │
    ▼
Docker Image
    │
    ▼
Amazon ECR

Think of ECR as a private warehouse for container images.

For example, we might have:

my-backend
my-worker
my-payment-service

inside an ECR repository.

When we build a new version of the application, we can push a new image to ECR.

4. ECR Doesn't Run Our Application

This distinction is important.

ECR stores the image.

It doesn't run the application.

For example:

ECR
 │
 │ stores
 ▼
Docker Image

We still need a service that can take that image and actually run it.

That's where Amazon ECS comes in.

5. Amazon ECS — Running Containers on AWS

Amazon Elastic Container Service, or ECS, is a container orchestration service.

In simple terms:

ECS manages running containers for our application.

The architecture now starts looking like:

Code
  │
  ▼
Docker Image
  │
  ▼
ECR
  │
  ▼
ECS
  │
  ▼
Running Container

But ECS introduces a few concepts that are worth understanding.

6. ECS Cluster

An ECS Cluster is a logical grouping of resources used to run our ECS workloads.

Think of it as the environment where our services run.

For example:

Production Cluster
       │
       ├── Backend Service
       ├── Worker Service
       └── Payment Service

A cluster itself isn't simply “one server.”

It's more useful to think of it as a logical home for our container workloads.

7. ECS Task Definition

Now ECS needs to know how our container should run.

That's where a Task Definition comes in.

It can describe things such as:

Which Docker image?
How much CPU?
How much memory?
Which port?
Environment variables?
IAM permissions?
Logging configuration?

For example:

Task Definition
       │
       ├── Image → ECR
       ├── CPU
       ├── Memory
       ├── Port
       └── Environment

This is similar to giving ECS a set of instructions:

“This is the container I want you to run, and this is how it should run.”

8. ECS Service — Keeping Containers Running

A task definition tells ECS how to run a container.

But what if I want three copies of my backend running continuously?

That's where an ECS Service comes in.

For example:

ECS Service
     │
     ├── Task 1
     ├── Task 2
     └── Task 3

The service maintains the desired number of tasks.

If a running task fails, ECS can replace it.

This is one of the important differences between simply running a container and having a container orchestration system manage it.

9. Where Does the Container Actually Run?

This is where AWS Fargate becomes interesting.

ECS can run containers using different compute options.

One option is managing the underlying EC2 instances yourself.

Another option is AWS Fargate.

Fargate is a serverless compute engine for containers.

With Fargate, AWS manages the underlying infrastructure required to run the containers.

So instead of thinking:

ECS
 ↓
EC2
 ↓
Operating System
 ↓
Container

we can use:

ECS
 ↓
Fargate
 ↓
Container

We still define the container's CPU, memory, networking, and other configuration, but we don't have to manage the underlying EC2 servers ourselves.

10. ECS + Fargate Architecture

Now our application starts looking like this:

                         USERS
                           │
                           ▼
                       Route 53
                           │
                           ▼
                      CloudFront
                           │
                           ▼
                          ALB
                           │
                           ▼
                     ECS Service
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
          Fargate       Fargate      Fargate
           Task          Task          Task
              │            │            │
              └────────────┼────────────┘
                           │
                           ▼
                           RDS

The ALB distributes requests between the running ECS tasks.

If one task becomes unhealthy, the service can replace it.

If traffic increases, we can increase the desired number of tasks.

11. What Happens When We Deploy a New Version?

This is where the whole model becomes really powerful.

Suppose we update our Node.js application.

We make some changes:

Version 1
    ↓
Code changes
    ↓
Version 2

We then build a new Docker image.

Source Code
     │
     ▼
Docker Build
     │
     ▼
New Docker Image
     │
     ▼
Amazon ECR

ECS can then deploy the new version.

The simplified flow becomes:

Developer
    │
    ▼
GitHub
    │
    ▼
Build Docker Image
    │
    ▼
Amazon ECR
    │
    ▼
ECS Service
    │
    ▼
New Tasks
    │
    ▼
Production

This is the foundation for modern container-based CI/CD pipelines.

12. What About the Database?

Containers are designed to be replaceable.

That means we generally shouldn't treat the container itself as the permanent place for important application data.

Instead:

ECS / Fargate
      │
      ├── Application
      │
      ▼
     RDS
      │
      ▼
Persistent Data

If a container disappears and ECS starts a new one, the application data remains in RDS.

The same principle applies to files.

Instead of storing uploaded files inside the container:

Container ❌

we can use:

S3 ✅

So:

Application
    │
    ├── Database → RDS
    ├── Files → S3
    ├── Cache → ElastiCache
    └── Jobs → SQS

This separation is extremely important when designing scalable applications.

13. Security Still Matters

Moving from EC2 to containers doesn't mean security disappears.

We still need to think about:

IAM

ECS tasks can use IAM roles to access AWS services without hardcoding AWS credentials into the application.

Security Groups

We can control network communication between:

ALB
 ↓
ECS
 ↓
RDS

Secrets Manager

Sensitive values such as database credentials and API secrets can be stored securely instead of being hardcoded into the image.

VPC

Our ECS tasks can run inside private subnets while the ALB remains publicly reachable.

So the architecture can still follow a secure pattern:

Internet
    │
    ▼
   ALB
    │
    ▼
Private ECS Tasks
    │
    ├── RDS
    ├── S3
    └── Other AWS Services

14. What Happens When Traffic Increases?

Let's say our application normally runs three tasks:

ECS Service

Task 1
Task 2
Task 3

Suddenly, traffic increases.

We can configure ECS Service Auto Scaling to increase the number of running tasks.

Normal Traffic

Task 1
Task 2
Task 3

Then:

High Traffic

Task 1
Task 2
Task 3
Task 4
Task 5
Task 6

The ALB distributes traffic across the available healthy tasks.

This gives us a scalable container architecture without manually logging into servers and starting applications ourselves.

15. ECS vs EC2 — What's the Difference?

This was one of the comparisons that helped me understand the purpose of ECS.

EC2	ECS + Fargate
Manage virtual servers	Manage containers
OS management required	AWS manages underlying infrastructure with Fargate
Install runtime manually	Runtime packaged in container
Server-focused	Container-focused
More infrastructure control	Less infrastructure management
Scaling servers	Scaling container tasks

Neither is automatically “better.”

The right choice depends on the application's requirements, team experience, control requirements, and operational needs.

16. ECS vs Lambda

Another useful comparison is ECS vs Lambda.

Lambda is excellent for event-driven, short-lived workloads where we don't want to manage servers or containers directly.

ECS is useful when we want more control over a containerized application and its runtime.

A simplified comparison:

ECS	Lambda
Container-based	Function-based
Long-running services are common	Event-driven execution is common
More runtime control	Less infrastructure management
Good for containerized APIs/workers	Good for many event-driven workloads

Again, neither service replaces the other.

The architecture should determine the choice.

17. The Complete Flow

Now we can connect everything together.

                         USER
                           │
                           ▼
                       Route 53
                           │
                           ▼
                      CloudFront
                           │
                           ▼
                          ALB
                           │
                           ▼
                    ECS Service
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
          Fargate       Fargate      Fargate
           Task          Task          Task
              │            │            │
              └────────────┼────────────┘
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
            RDS           S3        ElastiCache
         Database      File Storage      Cache
                           │
                           │
                           ▼
                          SQS
                           │
                           ▼
                     Worker Tasks

And the deployment side looks like:

Developer
    │
    ▼
GitHub
    │
    ▼
CI/CD Pipeline
    │
    ▼
Docker Build
    │
    ▼
Amazon ECR
    │
    ▼
Amazon ECS
    │
    ▼
Fargate
    │
    ▼
Production

Now we have two important flows:

Application flow

User → ALB → ECS → AWS Services

Deployment flow

Code → Docker → ECR → ECS → Fargate

That distinction made the whole architecture much easier for me to understand.

The Mental Model I Keep Now

I don't think of ECS, ECR, and Fargate as three unrelated AWS services anymore.

I think of them as a chain:

Docker
   │
   │ Packages
   ▼
ECR
   │
   │ Stores
   ▼
ECS
   │
   │ Manages
   ▼
Fargate
   │
   │ Runs
   ▼
Container

In simple words:

Docker → Package my application

ECR → Store my image

ECS → Manage my containers

Fargate → Run those containers without managing the underlying servers

That mental model is much easier to remember than four separate definitions.

What Changed My Perspective?

Before learning containers, my mental model was mostly:

Code
 ↓
EC2
 ↓
Run application

Now it looks more like:

Code
 ↓
Docker
 ↓
Container Image
 ↓
ECR
 ↓
ECS
 ↓
Fargate
 ↓
Production

And when we connect that with the architecture from my previous AWS article:

Users
  ↓
Route 53
  ↓
CloudFront
  ↓
ALB
  ↓
ECS / Fargate
  ↓
RDS / S3 / SQS / ElastiCache

AWS starts looking less like a collection of services and more like a complete platform for running applications.

Quick Cheat Sheet

Technology / Service	Main Job
Docker	Packages applications into containers
Dockerfile	Defines how an image is built
Docker Image	Packaged application blueprint
Container	Running instance of an image
ECR	Stores container images
ECS	Orchestrates containers
Task Definition	Defines how containers should run
ECS Service	Maintains desired running tasks
Fargate	Runs containers without managing servers
ALB	Distributes traffic
RDS	Stores relational data
S3	Stores files and objects
SQS	Handles asynchronous jobs
ElastiCache	Caches frequently accessed data
IAM	Controls permissions
VPC	Provides network isolation

Final Thoughts

The biggest shift in my AWS learning wasn't memorizing more services.

It was learning to ask:

“What problem is this service solving, and where does it fit in the architecture?”

Once I started thinking that way, AWS stopped feeling like a huge list of unrelated services.

Instead, it started looking like a set of building blocks.

And when those building blocks are combined properly, we can build applications that are scalable, secure, highly available, and easier to operate.

That, for me, is where learning AWS started becoming much more interesting.

How a Real Website Works on AWS — Complete AWS Architecture

Anupa Supul — Fri, 04 Sep 2026 09:16:17 +0000

Let’s imagine we are building a MERN web application with a React frontend, Node.js/Express backend, database, file uploads, authentication, and thousands of users.

This is what the architecture could look like.

The Big Picture

                         USERS
                           │
                           ▼
                       Route 53
                           │
                           ▼
                      CloudFront
                       /       \
                      /         \
                     ▼           ▼
                  S3             ALB
             React Frontend      │
                                 ▼
                           Auto Scaling
                                 │
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
                   EC2          EC2          EC2
                    │            │            │
                    └────────────┼────────────┘
                                 │
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
                   RDS       ElastiCache      SQS
                Database        Cache        Queue
                    │                         │
                    │                         ▼
                    │                       Worker
                    │
                    ▼
                   S3
              File Storage

At first, this diagram can look complicated.

But the important thing is that each service has a specific responsibility.

Instead of thinking about AWS as one giant system, think of it as a collection of specialized building blocks.

1. What Happens When You Open a Website?

Let's say a user types:

www.mywebsite.com

into their browser.

The request doesn't simply jump directly to an EC2 server.

It goes through several layers.

First, Route 53 helps the browser find where the domain should go. The request can then reach CloudFront, AWS's content delivery network.

CloudFront decides where the request should be served from.

If the user requests a React file such as JavaScript, CSS, or an image, CloudFront can retrieve it from Amazon S3.

If the request is something like:

GET /api/users

CloudFront can send it toward the backend through an Application Load Balancer (ALB).

So even before our backend receives the request, AWS has already handled DNS, content delivery, and traffic routing.

2. Route 53 — Finding Our Website

Amazon Route 53 is AWS's DNS service.

DNS is basically the system that translates a human-friendly domain name into a destination.

For example:

www.mywebsite.com
        ↓
     DNS lookup
        ↓
   CloudFront

Without DNS, users would need to remember an IP address instead of a domain name.

Route 53 therefore acts like the phone book of our website.

It doesn't run our frontend or backend. Its job is mainly to help users find the correct destination.

3. CloudFront — Bringing Content Closer to Users

Once Route 53 points the request toward CloudFront, CloudFront can handle the delivery of content.

CloudFront is a CDN (Content Delivery Network).

Imagine our application is hosted in a region far away from a user. Sending every image, JavaScript file, and CSS file all the way from our origin would add unnecessary latency.

CloudFront solves this by caching content at locations around the world called edge locations.

So a simplified flow looks like:

User
  │
  ▼
CloudFront Edge Location
  │
  ├── Cached content → Return immediately
  │
  └── Not cached → Ask the origin

This is especially useful for static content such as:

React JavaScript files
CSS
Images
Videos
Documents

It can also sit in front of our API and work with services such as AWS WAF.

4. React Frontend — Where Users Interact

Our React application is the part users actually see.

The frontend contains things like:

Login page
Dashboard
Profile
Products
Buttons
Forms
Images

Because a React production build consists largely of static files, we don't necessarily need an EC2 server just to serve those files.

We can store the built frontend in Amazon S3.

For example:

React Project
     │
     ▼
npm run build
     │
     ▼
HTML + CSS + JS
     │
     ▼
Amazon S3
     │
     ▼
CloudFront
     │
     ▼
Users

This is one of the first things that changed how I looked at AWS.

Not everything needs a traditional server.

5. Application Load Balancer — Sending Users to the Right Server

Now suppose the frontend calls our backend:

POST /api/login
GET /api/products
POST /api/orders

We don't want every user connecting directly to one EC2 instance.

Instead, requests can go through an Application Load Balancer (ALB).

                  ALB
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
       EC2        EC2        EC2

The ALB distributes incoming requests across healthy backend instances.

It also performs health checks.

For example, if one EC2 instance stops responding:

EC2 #1 → Healthy
EC2 #2 → Healthy
EC2 #3 → Unhealthy

The ALB can stop sending new requests to the unhealthy instance.

This is much more reliable than depending on a single server.

6. EC2 + Auto Scaling — Running the Backend

Our Node.js and Express backend can run on Amazon EC2.

For example:

React
  │
  ▼
API Request
  │
  ▼
ALB
  │
  ├── EC2 #1
  ├── EC2 #2
  └── EC2 #3

But what happens when traffic suddenly increases?

This is where EC2 Auto Scaling becomes important.

During normal traffic:

2 EC2 instances

During heavy traffic:

4 EC2 instances

During extremely high traffic:

6 EC2 instances

The exact numbers depend on how we configure the scaling policy.

The important idea is that Auto Scaling adjusts computing capacity based on demand, while the ALB distributes traffic between the available instances.

These two services solve different problems:

Service	Main Responsibility
ALB	Distributes requests
Auto Scaling	Adjusts number of EC2 instances
EC2	Runs our application

7. RDS — Where Our Application Data Lives

Our backend needs somewhere to store permanent data.

For example:

Users
Orders
Products
Payments
Messages
Profiles

This is where Amazon RDS can be used.

Instead of installing and maintaining a database manually on an EC2 server, RDS provides a managed relational database environment.

The backend might perform:

React
  ↓
ALB
  ↓
EC2
  ↓
RDS

For example, when a user creates an account:

User submits signup form
        ↓
React sends API request
        ↓
Backend receives request
        ↓
Backend validates data
        ↓
Backend writes user to RDS

The database should normally be placed in private subnets, rather than being directly exposed to the public internet.

8. S3 — Not Just for the Frontend

One thing that can easily confuse beginners is S3.

S3 is object storage.

It can store much more than React files.

Our application might allow users to upload:

Profile pictures
PDF files
Documents
Product images
Videos
Reports

Instead of putting these large files inside the database, we can store them in S3.

The database can then store information such as:

User ID
File name
S3 object key
Upload date

So we can think of it like:

RDS → Application data
S3  → Files / Objects

This separation makes the architecture easier to scale.

9. SQS — When Everything Doesn't Need to Happen Immediately

Some operations don't need to happen while the user is waiting.

Imagine a user uploads a video.

Our application might need to:

Save the file
Generate thumbnails
Process the video
Send an email
Update another system

Doing everything inside the original request could make the application slow.

Instead, we can use Amazon SQS.

User
 │
 ▼
Backend
 │
 ▼
SQS Queue
 │
 ▼
Worker
 │
 ├── Process file
 ├── Send email
 └── Perform background task

The backend can put a message into the queue and respond to the user.

A worker can process the job separately.

This creates decoupling between different parts of the system.

10. ElastiCache — Making Frequent Requests Faster

Not every piece of data needs to be fetched from the database every time.

Imagine thousands of users are requesting the same popular information.

Instead of repeatedly doing:

EC2 → RDS → Return data

we can cache frequently accessed data.

EC2
 │
 ▼
ElastiCache
 │
 ├── Data found → Return quickly
 │
 └── Data missing → Query RDS

ElastiCache can provide an in-memory cache using technologies such as Redis/Valkey or Memcached.

The important idea is:

Database = persistent source of data

Cache = fast temporary copy of frequently used data

Caching is an optimization, so it isn't required for every application.

11. Security — Protecting the Architecture

Now we have a working architecture, but putting everything on the internet would be a terrible idea.

Security needs to exist at multiple layers.

IAM

AWS IAM controls who or what can access AWS resources.

For example, instead of putting AWS access keys directly inside our Node.js code, we can give an EC2 instance an appropriate IAM role.

EC2
 │
 ▼
IAM Role
 │
 ▼
Allowed AWS Services

The application gets only the permissions it actually needs.

Security Groups

Security Groups act like virtual firewalls around AWS resources.

For example:

Internet
   │
   ▼
  ALB
   │
   ▼
  EC2
   │
   ▼
  RDS

We can configure rules such as:

Internet → ALB       ✅
ALB → EC2             ✅
EC2 → RDS             ✅
Internet → RDS        ❌

This is much safer than exposing the database directly to the internet.

AWS WAF

AWS WAF protects web applications by inspecting incoming HTTP/HTTPS requests.

It can help block things such as malicious or unwanted web requests based on configured rules.

A simplified security flow becomes:

Internet
   │
   ▼
  WAF
   │
   ▼
CloudFront
   │
   ▼
  ALB
   │
   ▼
  EC2

Secrets Manager

Our application also needs sensitive information such as:

Database password
API keys
Application secrets

These shouldn't simply be written directly inside source code.

AWS Secrets Manager provides a safer place to store and retrieve secrets.

12. VPC — The Network Behind Everything

All of these services don't just randomly exist on the internet.

Our backend architecture can be placed inside an Amazon VPC (Virtual Private Cloud).

A simple design could look like:

                    VPC
                     │
          ┌──────────┴──────────┐
          │                     │
    Public Subnets        Private Subnets
          │                     │
         ALB              EC2 / RDS / Cache

The public and private subnet design helps control which resources can be reached from the internet.

For a production architecture, we would normally distribute resources across multiple Availability Zones to improve availability.

13. Monitoring — Knowing What Is Happening

Building the infrastructure isn't enough.

We also need to know:

Is the application healthy?

Are servers running out of resources?

Are requests failing?

Is traffic increasing?

This is where Amazon CloudWatch becomes useful.

CloudWatch can provide metrics, logs, dashboards, and alarms.

For example:

CPU usage > threshold
        ↓
CloudWatch Alarm
        ↓
Auto Scaling / Notification

For auditing AWS API activity, CloudTrail records account and API activity.

So:

CloudWatch → Monitoring & Logs
CloudTrail  → AWS Activity & Auditing

14. HTTPS — Encrypting Communication

When users access:

https://www.mywebsite.com

their communication should be encrypted.

AWS Certificate Manager (ACM) can provide and manage TLS certificates for supported AWS services.

So instead of sending sensitive information over plain HTTP, we use HTTPS.

This is especially important for:

Login credentials
Personal information
Payments
API requests
Session information

15. Putting Everything Together

Now the individual services should make much more sense.

The complete request path can be summarized like this:

                         USER
                           │
                           ▼
                       Route 53
                           │
                           ▼
                      CloudFront
                       /       \
                      /         \
                     ▼           ▼
                  S3             ALB
             React Frontend      │
                                 ▼
                           Auto Scaling
                                 │
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
                   EC2          EC2          EC2
                    │            │            │
                    └────────────┼────────────┘
                                 │
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
                   RDS       ElastiCache      SQS
                Database        Cache        Queue
                    │                         │
                    │                         ▼
                    │                       Worker
                    │
                    ▼
                   S3
              File Storage

And around this architecture we have:

IAM
Security Groups
WAF
Secrets Manager
ACM
CloudWatch
CloudTrail
VPC

These aren't just extra AWS services added to make the diagram look impressive.

Each one solves a different operational problem.

The Most Important Thing I Learned

When I first started learning AWS, I looked at services individually.

EC2 = Server
S3 = Storage
RDS = Database
Route 53 = DNS
CloudFront = CDN

That is useful for getting started.

But real cloud architecture is about understanding how those services work together.

A real application isn't just:

User → EC2

It is closer to:

User
 ↓
DNS
 ↓
CDN
 ↓
Load Balancer
 ↓
Application Servers
 ↓
Database / Cache / Queue / Storage

And surrounding all of that are networking, security, monitoring, and scaling mechanisms.

That's when AWS started making much more sense to me.

Quick Cheat Sheet

Requirement	AWS Service
Domain / DNS	Route 53
CDN	CloudFront
React static files	S3
Backend	EC2
Load balancing	ALB
Automatic scaling	Auto Scaling
Relational database	RDS
File storage	S3
Background jobs	SQS
Caching	ElastiCache
Permissions	IAM
Network isolation	VPC
Firewall rules	Security Groups
Web protection	WAF
Secrets	Secrets Manager
HTTPS certificates	ACM
Monitoring	CloudWatch
AWS activity auditing	CloudTrail

Final Thoughts

The biggest shift in my AWS learning wasn't memorizing more services.

It was learning to ask:

“What problem is this service solving, and where does it fit in the architecture?”

Once I started thinking that way, AWS stopped feeling like a huge list of unrelated services.

Instead, it started looking like a set of building blocks.

And when those building blocks are combined properly, we can build applications that are scalable, secure, highly available, and easier to operate.

That, for me, is where learning AWS started becoming much more interesting.

Where Does My Website Actually Store Its Data? Understanding Amazon RDS

Anupa Supul — Mon, 17 Aug 2026 04:56:29 +0000

After learning how to run applications on EC2, I started thinking about something I had never really questioned before.

Where does all the data actually go?

When a user creates an account, where is that account stored?

When someone places an order, where does the order information live?

When I update my profile, how does the application remember that change the next time I log in?

As a MERN developer, I already understood the idea of a database.

I had worked with MongoDB and Mongoose.

But AWS made me think about the bigger question:

How do you run and manage a database reliably in the cloud?

That's when I started learning about Amazon RDS.

The Problem I Started With

Imagine I deploy my backend on an EC2 instance.

My application is running perfectly.

A user signs up.

Their information needs to be saved somewhere.

So my application needs a database.

The basic idea looks like this:

User
  │
  ▼
Backend
  │
  ▼
Database

At first, this seems simple.

But then I started thinking about everything that comes with managing a real database.

Someone has to:

Install the database
Configure it
Manage updates
Take backups
Monitor it
Handle failures
Protect it
Recover it when something goes wrong

That's a lot of responsibility.

And this is where managed database services become useful.

So What Is Amazon RDS?

The simplest way I understand Amazon RDS is:

Amazon RDS is a managed relational database service from AWS.

Instead of manually setting up and maintaining a relational database server yourself, AWS manages much of the underlying database infrastructure for you.

RDS supports database engines such as:

MySQL
PostgreSQL
MariaDB
Oracle
SQL Server
Amazon Aurora

So instead of spending most of my time managing the database server itself, I can focus more on my application and the data.

Running a Database Yourself vs RDS

This comparison helped me understand the difference.

If I Manage Everything Myself

Imagine I launch an EC2 instance and install MySQL manually.

EC2
 │
 ├── Operating System
 ├── MySQL
 ├── Configuration
 ├── Updates
 ├── Backups
 └── Monitoring

Now I'm responsible for all of it.

With Amazon RDS

With RDS, the architecture becomes much simpler:

Application
     │
     ▼
Amazon RDS
     │
     ▼
Relational Database

AWS handles much of the underlying infrastructure.

That doesn't mean I have no responsibility.

I still need to think about:

Database design
Queries
Users and permissions
Security
Application configuration
Cost

But AWS takes care of many infrastructure-level tasks.

A Real-World Example

Imagine I build an online store.

A customer creates an account.

Then they add a product to their cart.

Finally, they place an order.

The application needs to store information such as:

Customer
    │
    ├── Account
    │
    ├── Orders
    │
    ├── Products
    │
    └── Inventory

A relational database is useful for this kind of structured data.

The application architecture could look like:

User
  │
  ▼
Frontend
  │
  ▼
Backend API
  │
  ▼
Amazon RDS

The frontend doesn't directly communicate with the database.

The backend handles that communication.

Why Shouldn't Users Connect Directly to the Database?

This was another thing that became clear to me while learning AWS.

Imagine this:

Internet Users
      │
      ▼
   Database

That would be a terrible architecture.

You don't want random internet users connecting directly to your database.

Instead, we want:

Internet Users
      │
      ▼
   Frontend
      │
      ▼
  Backend API
      │
      ▼
  Amazon RDS

The backend controls how the application communicates with the database.

Security Groups can also be configured so the database only accepts connections from the backend resources that actually need access.

That creates a much safer architecture.

What Happens If the Database Fails?

This is where managing databases gets interesting.

Imagine my application has thousands of users.

Everything is working.

Then suddenly, the database server experiences a failure.

What happens to the application?

If I manually manage the database, I have to think about how to recover it.

RDS provides features designed to improve database availability and recovery.

One important feature is Multi-AZ deployment.

Multi-AZ

With Multi-AZ, AWS can maintain a standby database instance in another Availability Zone.

A simplified example:

                 AWS Region
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
 Availability Zone A      Availability Zone B
          │                     │
          ▼                     ▼
     Primary RDS            Standby RDS

The application normally communicates with the primary database.

If a failure occurs, AWS can fail over to the standby.

What I found interesting here is that high availability isn't simply about having a backup file.

It's about designing the system so it can continue operating when something goes wrong.

What About Backups?

Another thing I started appreciating about managed databases is backups.

Imagine I accidentally delete important data.

Without a backup strategy, recovering it could become a nightmare.

RDS provides automated backup capabilities and supports database snapshots.

So instead of thinking:

"I'll configure backups later."

I can make backups part of the infrastructure from the beginning.

That was another important mindset shift for me.

What If My Application Gets Bigger?

Let's say my application starts with:

100 users

Then it grows to:

10,000 users

And eventually:

1,000,000 users

The database now has much more work to handle.

Eventually, database scaling becomes its own engineering problem.

Depending on the workload, there are different ways to improve database performance and capacity.

For example:

Increasing the database instance size
Using read replicas
Adding caching
Optimizing queries
Choosing the right database architecture

So scaling isn't only about EC2.

The database needs to scale too.

RDS With Everything I've Learned So Far

This was probably the most useful part for me.

I've already learned about:

Route 53
CloudFront
Elastic Load Balancer
EC2
Auto Scaling
Security Groups

Now I can finally see where the database fits into the bigger picture.

A simplified AWS architecture could look like this:

Users
  │
  ▼
Route 53
  │
  ▼
CloudFront
  │
  ▼
Load Balancer
  │
  ▼
Auto Scaling Group
  │
  ▼
EC2 Instances
  │
  ▼
Amazon RDS

Each service has a different responsibility.

Route 53 helps users find the application.

CloudFront helps deliver content efficiently.

Load Balancer distributes incoming traffic.

Auto Scaling adjusts the number of EC2 instances.

EC2 runs the application.

RDS stores the relational data.

Seeing all these pieces together made AWS feel much less like a list of services and more like an actual system.

One Thing I Finally Understood

Before learning AWS, I mostly thought about databases from the application developer's perspective.

I would:

Create Database
      ↓
Connect Backend
      ↓
Write Queries
      ↓
Done

But production systems are different.

Now I have to ask:

What happens if it fails?
        ↓
How are backups handled?
        ↓
Who can access it?
        ↓
Can it scale?
        ↓
How do I keep it available?

Those questions become increasingly important as an application grows.

The Bigger Picture

After learning EC2, ELB, Auto Scaling, and RDS, I started seeing cloud architecture differently.

It's not about memorizing:

"EC2 does this."

"RDS does that."

It's about understanding how the pieces work together.

For example:

                    Users
                      │
                      ▼
                  Route 53
                      │
                      ▼
                 CloudFront
                      │
                      ▼
              Load Balancer
                      │
                      ▼
              Auto Scaling
                 │      │
                 ▼      ▼
               EC2     EC2
                 │      │
                 └──┬───┘
                    │
                    ▼
                Amazon RDS

Now the architecture actually tells a story.

Users arrive.

Traffic is routed.

Content is delivered.

Requests are distributed.

Servers scale.

The application runs.

Data is stored.

That's when AWS started making much more sense to me.

What Changed My Perspective

Before learning AWS, I thought about databases mainly as something my application connects to.

Now I understand that the database itself is an important part of the architecture.

Questions like:

What happens if the database fails?

How do I protect it?

How do I back it up?

How will it handle more users?

How do I make it highly available?

These aren't just database questions.

They're system design questions.

And that's probably the biggest thing RDS taught me.

Final Thoughts

When I first started learning AWS, EC2 was exciting because I could actually see my application running on a cloud server.

But RDS taught me something different.

Running an application is only one part of the system.

The application also needs somewhere reliable to store the information that makes it useful.

And managing that database involves much more than simply creating a database and connecting to it.

That's where managed services like Amazon RDS become valuable.

For me, the biggest lesson was simple:

A database isn't just where data lives. It's part of the architecture that needs to be designed for security, availability, backups, and growth.

The more I learn AWS, the more I realize that building software isn't only about making the application work.

It's about making sure the entire system can keep working when real users start depending on it.

🚀 What I'm Learning Next

The more AWS services I learn, the more I realize that each one solves a different piece of the same puzzle.

EC2 taught me where applications can run.

ELB taught me how traffic can be distributed.

Auto Scaling taught me how applications can adapt to demand.

And RDS taught me where the application's important data can live reliably.

The next step is understanding how these pieces work together in a real application.

I'm continuing to document my AWS learning journey by taking the concepts I learn and explaining them in the simplest way I can.

If you're learning AWS too, I hope this made Amazon RDS a little easier to understand.

Happy learning! 🚀

Why Doesn't My Website Crash When Thousands of People Visit? Understanding Amazon EC2

Anupa Supul — Fri, 31 Jul 2026 10:16:11 +0000

When I first started learning AWS, I assumed that if a website became popular, the solution was simple.

Buy a bigger server.

More CPU.

More RAM.

More storage.

Problem solved.

But then I started thinking about websites like Netflix, Amazon, and Instagram.

Millions of people visit them every day.

Sometimes millions of users are online at exactly the same time.

Surely they don't keep buying one giant server every time traffic increases.

So how do they handle sudden spikes in visitors without crashing?

That question introduced me to Amazon EC2 Auto Scaling.

The Problem

Imagine I build my own portfolio website.

Normally, around 100 people visit it each day.

Everything works perfectly.

Then one day, someone with a huge social media following shares my website.

Within minutes...

10,000 people try to open it.

What happens?

If I only have one EC2 instance, that single server now has to handle every request.

Soon it becomes overloaded.

Pages load slowly.

Some users get errors.

Eventually...

The website crashes.

flowchart LR

Users --> EC2

style EC2 fill:#ff9999

Clearly, one server isn't enough.

My First Thought

My first solution was:

Just buy a bigger server.

For example:

Upgrade from

2 GB RAM

16 GB RAM

This is called Vertical Scaling.

flowchart LR

SmallEC2["Small EC2"]

-->LargeEC2["Bigger EC2"]

This works...

But only up to a point.

Eventually, even the biggest server has limits.

AWS Had a Better Idea

Instead of buying one huge server...

Why not create more servers whenever they're needed?

That's exactly what Auto Scaling does.

Instead of this:

flowchart LR

Users --> OneServer

AWS can do this:

flowchart LR

Users

-->ELB

ELB-->EC21

ELB-->EC22

ELB-->EC23

ELB-->EC24

Now the workload is shared across multiple EC2 instances.

Each server handles only part of the traffic.

What Is Amazon EC2 Auto Scaling?

Amazon EC2 Auto Scaling automatically launches or removes EC2 instances depending on your application's traffic.

Simply put:

More visitors ➜ More servers
Fewer visitors ➜ Fewer servers

No manual work required.

Scale Out

Suppose your website suddenly becomes popular.

Traffic increases.

Auto Scaling detects the increase.

It automatically launches additional EC2 instances.

flowchart LR

Traffic

-->AutoScaling

AutoScaling

-->EC21

AutoScaling

-->EC22

AutoScaling

-->EC23

AutoScaling

-->EC24

This process is called Scale Out.

Scale In

Now imagine it's midnight.

Most users leave.

You no longer need four servers.

Keeping them running would waste money.

Auto Scaling automatically terminates the extra EC2 instances.

flowchart LR

TrafficDrops

-->AutoScaling

AutoScaling

-->EC21

This is called Scale In.

You only pay for what you actually use.

Auto Scaling Group

Auto Scaling doesn't create random servers.

Instead, it manages them inside something called an Auto Scaling Group (ASG).

An ASG follows rules that you define.

For example:

Minimum Instances → 2
Desired Instances → 3
Maximum Instances → 10

If traffic increases, AWS can grow from:

3 → 4 → 5 → 6 → ... → 10

If traffic decreases, it can shrink back down.

Everything happens automatically.

Auto Scaling + Elastic Load Balancer

One thing finally clicked for me.

Auto Scaling creates servers.

But...

How do users know which server to use?

That's where the Elastic Load Balancer (ELB) comes in.

flowchart LR

Users

-->ELB

ELB-->EC21

ELB-->EC22

ELB-->EC23

ELB-->EC24

The Load Balancer distributes traffic.

Auto Scaling creates or removes servers.

Together, they make applications highly available.

Real-World Examples

Auto Scaling is useful almost everywhere.

Imagine:

Black Friday shopping
Concert ticket sales
Online gaming events
Netflix releasing a new season
Viral social media applications

Traffic suddenly increases.

Instead of crashing...

AWS automatically adds more servers.

Once traffic drops, it removes the extra servers.

Why This Saves Money

Without Auto Scaling, you might keep ten servers running all day.

Even when only one server is needed.

That's expensive.

With Auto Scaling:

Busy hours:

8 Servers

Quiet hours:

2 Servers

AWS adjusts automatically.

You only pay for the servers you actually need.

Benefits of Auto Scaling

Handles sudden traffic spikes

Improves application availability

Prevents server overload

Automatically adds or removes EC2 instances

Reduces infrastructure costs

Works seamlessly with Elastic Load Balancers

What Changed My Perspective

Before learning Auto Scaling, I thought handling more users meant buying a more powerful server.

Now I understand that's only one approach.

Sometimes the better solution isn't:

A bigger server.

It's:

More servers working together.

That small mindset shift completely changed how I think about scalable applications.

Final Thoughts

When I first heard the term Amazon EC2 Auto Scaling, I imagined it was just another AWS feature.

Now I realize it's one of the key reasons modern cloud applications stay online during sudden traffic spikes.

Instead of guessing how many servers you'll need...

AWS watches your traffic for you.

When demand increases, it automatically adds more servers.

When demand decreases, it removes the extras.

That's one of the biggest advantages of cloud computing.

You don't have to build for your busiest day.

AWS scales with you.

Key Takeaways

Auto Scaling automatically launches and removes EC2 instances.
Scale Out adds more servers during high traffic.
Scale In removes servers when traffic decreases.
Auto Scaling Groups manage EC2 instances using rules.
Elastic Load Balancer distributes traffic across all instances.
Together, ELB and Auto Scaling build highly available applications.

*Thanks for reading! *

How Does AWS Know Which Website I'm Trying to Visit? Understanding Amazon Route 53

Anupa Supul — Fri, 31 Jul 2026 09:57:57 +0000

When I first started learning web development, opening a website felt almost magical.

I would type something like:

www.amazon.com

Press Enter.

A few seconds later, the website appeared.

I never stopped to think about what actually happened.

How did my computer know where Amazon's servers were?

How did it find the correct server among millions connected to the internet?

And why don't we type an IP address like:

54.239.28.85

instead of simply typing:

amazon.com

Those questions eventually led me to one of the first networking services in AWS:

Amazon Route 53

The Question That Made Me Curious

Imagine I build my own MERN application.

I deploy it on an EC2 instance.

AWS gives me a public IP address like:

18.142.120.54

Technically, people can visit my application using that IP.

But let's be honest...

Nobody wants to remember numbers like that.

It's much easier to visit:

myportfolio.com

myapp.com

That made me wonder:

How does my computer know that "myapp.com" actually means "18.142.120.54"?

The Phone Contact Analogy

This example made everything click for me.

Imagine your best friend has this phone number.

+94 71 123 4567

Do you memorize the number?

Probably not.

Instead, you save it as:

John

Now whenever you want to call him, you simply tap:

John

Your phone automatically finds the correct phone number.

You don't need to remember it.

The internet works almost the same way.

Instead of remembering:

18.142.120.54

we simply remember:

mywebsite.com

Something translates that easy-to-read name into the server's IP address.

That "something" is called DNS (Domain Name System).

What Is Amazon Route 53?

Amazon Route 53 is AWS's managed DNS service.

Its job is simple:

Convert human-friendly domain names into IP addresses that computers understand.

Instead of this:

18.142.120.54

We simply type:

mywebsite.com

Route 53 performs the translation and sends us to the correct server.

What Actually Happens?

Let's say I open:

www.myportfolio.com

Here's what happens behind the scenes.

flowchart LR

A[User Types myportfolio.com]

-->B[Browser]

-->C[Amazon Route53 DNS Lookup]

-->D[Returns IP Address]

-->E[EC2 Server]

-->F[Website Loads]

The entire process usually takes only milliseconds.

A Real AWS Example

Suppose I host my MERN application on an EC2 instance in Singapore.

Its public IP address is:

18.xxx.xxx.xxx

Without Route 53, users would have to visit:

18.xxx.xxx.xxx

With Route 53:

flowchart LR

A[myportfolio.com]

-->B[Amazon Route53]

-->C[18.xxx.xxx.xxx]

-->D[EC2 Instance]

Much easier.

Why Is It Called Route 53?

One thing that caught my attention was the number 53.

It's actually named after Port 53, the standard network port used for DNS communication.

A small detail, but one I found interesting.

Route 53 Does More Than DNS

Initially, I thought Route 53 only converted domain names into IP addresses.

But it can also:

Route traffic globally
Perform Health Checks
Fail over to healthy resources
Route traffic intelligently
Improve application availability

Where Route 53 Fits in AWS

As I learned more AWS services, I realized Route 53 is usually the very first AWS service involved when someone opens a website.

flowchart LR

User

-->Route53

-->CloudFront

-->LoadBalancer

-->EC2

-->Database

Every service has its own responsibility.

Route 53 simply helps users find the right destination.

Why Route 53 Matters

Without DNS:

Users must remember IP addresses.

With Route 53:

Easy-to-remember domain names

Highly available DNS

Faster routing

Health monitoring

Traffic routing policies

Works with almost every AWS service

What Changed My Perspective

Before learning AWS, I thought typing a website name directly connected me to the server.

Now I understand there's an important translation happening first.

Someone has to convert:

mywebsite.com

into

18.xxx.xxx.xxx

That's exactly what DNS does.

And in AWS, that job belongs to Amazon Route 53.

Final Thoughts

When I first heard the name Route 53, I had no idea what it actually did.

Now I think of it much more simply.

It's like the contact list on my phone.

I don't remember phone numbers.

I remember names.

Similarly, my browser doesn't magically know where a website lives.

Route 53 translates a human-friendly domain name into the server's actual address.

Sometimes the most important AWS services are the ones quietly working behind the scenes.

And Route 53 is definitely one of them.

Key Takeaways

Route 53 is AWS's managed DNS service.
It converts domain names into IP addresses.
It makes websites easier to access.
It supports health checks and intelligent routing.
It's usually the first AWS service involved when visiting a website.

*Thanks for reading! *