DEV Community: Ramkumar Jayakumar

Getting the Actual Client IP When Using Application Load Balancer (ALB) in AWS Lambda

Ramkumar Jayakumar — Sat, 20 Jul 2024 12:37:40 +0000

When I was new to AWS, I faced an interesting challenge while working on a task to digitally sign a document, which required the client's IP as part of the e-signature. Initially, I was thrilled when the implementation seemed to work perfectly the first time. However, my excitement was short-lived. During testing, I noticed that the same IP address was being returned, even when I accessed the application from different machines. It was then that I realized the IP address I was receiving was not the actual client IP but the IP of the load balancer.

This discovery led me down a path of investigation and learning. I had to dig deeper to understand what was happening and how to retrieve the real client IP. In this blog, I will share my experience and provide a comprehensive guide on how to achieve this using AWS Lambda and Python, ensuring you can accurately capture the client’s IP address when using an Application Load Balancer (ALB).

Understanding the Challenge

When a client makes a request to your application through an ALB, the load balancer acts as an intermediary. Consequently, the IP address your application sees is that of the ALB, not the client's. To address this, ALB includes the client's IP in the X-Forwarded-For HTTP header. This header can contain multiple IP addresses in case the request has passed through multiple proxies.

Here's what we need to handle:

Extract the Client IP: Retrieve and parse the X-Forwarded-For header.
Handle Multiple IPs: Ensure that we get the correct client IP even when multiple proxies are involved.

Security Consideration

The X-Forwarded-For header should be used with caution due to potential security risks. The entries can only be considered trustworthy if added by systems that are properly secured within the network. This ensures that the client IPs are not tampered with and are reliable.

Choosing the Right Tools

AWS Lambda and Python

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. Python, with its simplicity and readability, is an excellent choice for handling this task within a Lambda function.

Key Components

AWS Lambda Function: The core function that processes incoming requests.
Application Load Balancer (ALB): The load balancer that forwards requests to the Lambda function.

Implementation Details

Setting Up AWS Lambda with ALB

First, ensure your Lambda function is set up and integrated with an ALB. Follow AWS's official guide if needed: Using Lambda functions as targets for Application Load Balancer.

Lambda Function Code

Let's dive into the Python code for the Lambda function. This function will extract the client's IP address from the X-Forwarded-For header.

import json

def lambda_handler(event, context):
    # Extract the 'X-Forwarded-For' header
    x_forwarded_for = event['headers'].get('x-forwarded-for')

    if x_forwarded_for:
        # The first IP in the list is the client's IP
        client_ip = x_forwarded_for.split(',')[0]
    else:
        # Fallback if header is not present
        client_ip = event['requestContext']['identity']['sourceIp']

    # Log the client IP
    print(f"Client IP: {client_ip}")

    # Respond with the client IP
    return {
        'statusCode': 200,
        'body': json.dumps({'client_ip': client_ip})
    }

Explanation

Extract the Header: Retrieve the X-Forwarded-For header from the incoming request.
Parse the Header: Take the first IP, which represents the client's original IP.
Fallback Mechanism: Use the source IP from the request context if the header is not present.
Logging and Response: Log and return the client's IP for verification.

Example Request and Response

Request:

{
    "headers": {
        "x-forwarded-for": "203.0.113.195, 70.41.3.18, 150.172.238.178"
    },
    "requestContext": {
        "identity": {
            "sourceIp": "70.41.3.18"
        }
    }
}

Response:

{
    "client_ip": "203.0.113.195"
}

Conclusion

Identifying the actual client IP in an AWS Lambda function behind an ALB requires careful handling of the X-Forwarded-For header. This approach ensures accurate IP logging and enhances the application's ability to personalize and secure user interactions.

Refrences

Coding Efficiency: Designing a High-Performance Data Import System with Excel Integration

Ramkumar Jayakumar — Wed, 20 Mar 2024 10:20:24 +0000

Introduction: Bridging the Data Gap - Efficiently Importing Data from Other Systems

In today's digital world, our applications often need to work together. Imagine you have a customer relationship management (CRM) system that stores all your customer information, but you also have a separate system for tracking sales and orders. To get a complete picture of your business, you might need to import data (like customer details) from one system into the other. This process of bringing data into a system from an external source is called a data import.

Data imports can be crucial for making informed decisions. For example, by combining your customer information with your sales data, you can analyze trends and identify your most valuable customers. But how do you actually move this data between systems?

There are several ways to approach data migration, each with its own advantages and limitations. This blog post will explore how you can design a data import system using a familiar tool: Microsoft Excel.

We'll delve into strategies for handling large datasets efficiently, ensuring data accuracy, and creating a user-friendly experience for the import process. By leveraging the right techniques, you can unlock the power of Excel import and bridge the data gap between your systems seamlessly.

Navigating the Maze of Data Migration

Moving data between different systems can feel like navigating a maze. There are several approaches you can take, each with its own pros and cons. Let's explore some common methods for data import:

Manual Creation: Imagine having to type in all your customer information one by one from a spreadsheet into your CRM system. This is manual data creation, and while it works for very small datasets, it's time-consuming, error-prone, and not scalable for large amounts of data.
Custom Scripts: For more complex data transfers, you might consider writing custom scripts or programs. This approach offers a lot of flexibility, but they lack the efficiency and can be time-consuming to develop and maintain.
API Integration: Many applications offer APIs (Application Programming Interfaces) that allow them to talk to each other. Imagine an API as a special translator between your CRM system and your sales data system. By using the API, you can build a connection that lets you import data automatically. However, using APIs often requires development effort for both sides and may not be available for all systems.
Excel Import: Simple and ubiquitous, but raises concerns about scalability and data integrity, especially for high volumes.

Excel Import: Reimagined for Efficiency

While Excel import is a familiar and accessible solution for bringing data into your system, it can face challenges when dealing with large datasets or complex data validation needs. Imagine you have a massive customer list with thousands of entries in an Excel spreadsheet. Traditionally, importing this data might take a long time or lead to errors if the data isn't formatted correctly.

Here's where we can reimagine Excel import for efficiency:

High-Volume Processing: By leveraging specialized techniques, we can optimize the import process to handle large datasets without sacrificing performance or system stability. This might involve splitting the data into smaller chunks for processing or using efficient algorithms for data validation.
Robust Validation and Error Handling: Data accuracy is crucial. Our system can perform thorough checks on your Excel data to ensure it meets the requirements of your target system. For example, it can verify data types (like numbers or dates) and identify any inconsistencies or missing information. This helps catch errors early on and prevents inaccurate data from entering your system.
Streamlined Workflow: The data import process shouldn't be a hassle. We can design a user-friendly interface that guides you through each step, from configuring the import to uploading your Excel file. This makes it easy for anyone, even beginners, to import data efficiently.

By incorporating these features, we can transform Excel import from a simple data transfer tool into a robust and efficient system. Imagine being able to import your large customer list with confidence, knowing the system will handle the process smoothly and ensure the accuracy of your data.

System Design

Let's delve into the design section for an efficient data import system using Excel, addressing key concerns such as:

1. Dynamic Field Mapping: Empowering User Flexibility

One of the key challenges in data import systems is accommodating the diverse structure of data from different sources. This is where dynamic field mapping plays a pivotal role. Here's a closer look at how it works:

User-Driven Configuration: Instead of relying on fixed mappings, users are empowered to configure the import process based on their specific needs. Upon initiating a data import task, the system prompts users to define mappings between the columns in the Excel file and corresponding fields in the destination system.
Flexible Mapping Interface: To ensure user-friendly interaction, the system provides an intuitive interface for mapping configuration. Users are presented with a list of available fields in the destination system, allowing them to easily match them with the columns in the Excel file. Additionally, the interface supports dynamic suggestions and auto-completion to streamline the mapping process.
Customizable Mapping Profiles: Recognizing that users may need to import data from multiple sources with varying structures, the system supports the creation of customizable mapping profiles. Users can define and save multiple mapping configurations, each tailored to a specific source or import scenario. This flexibility allows for efficient handling of diverse data formats without the need for repetitive mapping tasks.
Validation and Error Handling: To ensure data accuracy and integrity, the system performs validation checks during the mapping configuration phase. This includes verifying the consistency of data types, detecting potential mismatches between source and destination fields, and flagging any inconsistencies or conflicts for user review. Comprehensive error handling mechanisms are in place to guide users in resolving mapping issues effectively.
To ensure seamless configuration and mapping between Excel data and the database, a structured approach leveraging MySQL is adopted. Here's how the database schema facilitates the process:

1.Import Configuration Table: This table stores various import configurations available in the system, enabling users to manage different import scenarios effectively.

CREATE TABLE `import_configuration` (
  `import_configuration_id` int UNSIGNED NOT NULL,
  `configuration_name` varchar(55) NOT NULL,
  `created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `is_deleted` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`import_configuration_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;

2.Configuration Mapping Table: This table establishes mappings between Excel columns and database fields for each import configuration. It defines column titles, mandatory fields, sort order, and default values, facilitating accurate data mapping.

CREATE TABLE import_mapping_columns (

  import_mapping_column_id int UNSIGNED NOT NULL,

  import_configuration_id int UNSIGNED NOT NULL,

  import_column_name varchar(55) DEFAULT NULL,

  import_column_title varchar(100) NOT NULL,

  is_mandatory enum('0','1') NOT NULL,

  sort_order int NOT NULL,

  default_value json DEFAULT NULL,

  is_deleted tinyint(1) NOT NULL DEFAULT '0',

  PRIMARY KEY (import_mapping_column_id),

  FOREIGN KEY (import_configuration_id) REFERENCES import_configuration(import_configuration_id)

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;

2.File Upload Phase: Streamlining Data Entry

The file upload phase initiates the import process, ensuring seamless integration of Excel data into the system. Here's a concise overview of this crucial phase:

Configuration Preparation: Users begin by configuring the import process, selecting the appropriate import configuration from the predefined options. This step ensures that the system knows how to interpret the incoming Excel data.
Excel Upload: Once configured, users upload the Excel file containing the data to be imported. The system validates the file format and size to ensure compatibility and prevent data loss.
Storage in S3: The uploaded file is securely stored in an Amazon S3 bucket, providing scalability, reliability, and easy access for subsequent processing steps.
Background Task Creation: Concurrently, the system creates a background task or job to manage the import process efficiently. This task orchestrates the various steps involved in reading, validating, and importing the Excel data into the system.

3. Excel Reading and Temporary Record Creation: Ensuring Data Integrity

In the Excel Reading and Temporary Record Creation phase, the system processes the uploaded Excel file, validates its contents, and creates temporary records for further validation. Here's a detailed breakdown of this crucial step:

CRON Picks up the Background Task: The scheduled task responsible for managing the import process retrieves the Excel file from the designated Amazon S3 bucket.
Updating Task Status: To prevent duplicate processing, the system updates the status of the background task to "in progress." This prevents multiple instances from picking up the same task simultaneously, maintaining data integrity.
Heading Validation: The system verifies the headers of the Excel file to ensure they match the expected format defined in the import configuration. Any discrepancies are flagged for user attention, preventing data misinterpretation.
Excel Reading: Using specialized libraries or tools, the system reads the data from the Excel file, extracting each row as a separate dataset for further processing. This step ensures accurate data extraction while handling large volumes efficiently.
Row Validation: Each row of data undergoes validation against predefined rules and constraints specified in the import configuration. Mandatory fields, data types, and format checks are performed to identify any inconsistencies or errors. Use hashmaps or some md5 hashing mechanisms for getting duplicate entries within the excel file validation.
Splitting into Chunks: To optimize performance and manage memory usage, the dataset is split into manageable chunks for bulk insertion into the database. This ensures efficient processing, especially with large datasets.
Bulk Insertion: Validated data is bulk-inserted into temporary tables within the database, segregating valid records from exceptions. This step lays the groundwork for further validation and refinement of the imported data.
Within System Validations: Concurrently, the system performs additional validations within the database environment, such as duplicate checks and cross-referencing against existing records. This ensures data consistency and integrity at every stage of the import process.
Temporary Records Creation: Validated records are stored as temporary entries within the database, providing a snapshot of the imported data for subsequent review and refinement. Exception records are also retained for further investigation and correction.

By meticulously executing the Excel Reading and Temporary Record Creation phase, the system establishes a solid foundation for data integrity, enabling accurate and reliable integration of external data into the system.

    public function map($row): array

    {

        $this->trimValues($row);

    $row['failed_columns'] = array();
    $row['validation_message'] = array();
    $row['import_status'] = 1;

    $this-&gt;setDefaultValues($row);
    $this-&gt;validate($row)

    $row['import_status'] = count($row['failed_columns']) &gt; 0 ? 0 : 1;
    $row['failed_columns'] = json_encode($row['failed_columns']);
    $row['validation_message'] = !empty($row['validation_message']) ? implode(", ", $row['validation_message']) : null;

    return $row;
}

// remove white spaces and non-breaking spaces

    protected function trimValues(&$row): void

    {

        $row = array_map(function ($value) {

            // For numeric values accept zero.

            if ($value == 0) {

                return $value;

            }

            return !empty(trim($value)) ? trim(str_replace("\xc2\xa0", '', $value)) : NULL;

        }, $row);

    }

public function registerEvents(): array

    {

        return [

            AfterImport::class => function () {

                $this->updateImportDetails();

                $this->updateTimesheetImportCount();

            },

        ];

    }

// Insert the chunked Data

public function array(array $array)

{

    ImportDetails::insert($array);

    return $array;

}

    public function chunkSize(): int

    {

        return $this->chunk_size;

    }

Verification & Real Record Creation: Validating and Finalizing Imported Data

After the temporary records are created from the Excel data, the system proceeds to verify and finalize the imported data. This phase ensures that only accurate and validated records are permanently stored in the system. Here's a detailed explanation of this crucial step:

Validate Imported Records: All temporary records generated from the Excel data undergo thorough validation checks. This includes verifying data consistency, ensuring adherence to business rules, and detecting any anomalies or discrepancies.
Fix Exception Records: If any validation errors are identified during the verification process, the system prompts users to review and rectify the exception records. Users can correct data inaccuracies or provide missing information to ensure completeness and accuracy.
Background Task for Real Record Generation: Once the imported data passes validation, a background task is initiated to generate real records from the validated entries. This task orchestrates the finalization process, ensuring data integrity before permanent storage.
CRON Picks Up the Task: The scheduled task manager periodically picks up the task for real record creation, ensuring timely processing and minimal delay in data integration.
Validate for Duplicate Records: Before creating real records in the database, the system performs a final check to ensure that no duplicate entries exist. This prevents redundancy and maintains data consistency within the system.
Construct Insertion Query: Based on the validated temporary records, the system constructs an optimized insertion query using efficient bulk-insert techniques. This minimizes database overhead and maximizes performance during the final data insertion step.
Bulk Insertion of Real Records: The validated and deduplicated records are bulk-inserted into the system's database tables, finalizing the data import process. This step ensures that only accurate and verified data is permanently stored for use within the application.

By meticulously verifying and finalizing imported data, the system ensures data accuracy, integrity, and consistency, laying the groundwork for effective data-driven decision-making and business operations. This phase marks the culmination of the data import process, ensuring that the system is equipped with reliable and trustworthy information for optimal performance and functionality.

Conclusion

In the dynamic landscape of data management, the seamless integration of external data into our systems is crucial. Throughout this journey, we've delved into the intricacies of designing a high-performance data import system, focusing on the versatility of Excel and the efficiency of our approach.

From configuring import settings to finalizing real records, each phase of the data import process is pivotal in ensuring accuracy, integrity, and efficiency. Our commitment to optimization extends beyond functionality to include minimizing database hits and connections, while also striving to maintain O(n) time complexity.

Through dynamic field mapping and a structured database schema, users can tailor import configurations to their needs, fostering flexibility and customization. This empowers organizations to navigate the complexities of data migration with confidence.

During file upload, we prioritize compatibility and security, laying the groundwork for efficient processing. Subsequent phases, from Excel reading to temporary record creation and verification, ensure meticulous validation with minimal database interactions.

As we conclude, it's clear that a well-designed data import system enhances operational efficiency and underpins informed decision-making. By embracing innovative solutions and optimization strategies, we navigate the complexities of data migration, driving growth and innovation in the digital age while ensuring minimal database overhead and optimal performance. Together, let's continue harnessing the power of efficient data integration to unlock new possibilities and opportunities.
Feel free to post your ideas as to how you would process the data imports.

General Docker Troubleshooting, Best Practices & Where to Go From Here

Ramkumar Jayakumar — Fri, 19 Jan 2024 13:00:00 +0000

Tackling Common Issues: No Space and Sluggish Containers

Addressing common issues is crucial for maintaining a seamless Docker experience. Let's highlight the importance and break down each of the upcoming sections:

No Space Issue:

Image Cleanup:

To remove unused images:

 docker rmi <img1-id> <img2-id> <img3-id> ..

Execute system prune:
```
 docker system prune
```
This command intelligently removes unnecessary data, including stopped containers and obsolete layers.

Container Performance:

1.Monitoring Container Stats:

Use the docker stats command:
```
 docker stats <name/id>
```
Provides real-time snapshots of your container's performance.

2.Inspecting Container Details:

Explore detailed information in JSON format:

 docker inspect <container name/id>

For less detailed output:

  docker inspect <container name/id> | less

3.Viewing Running Processes:

   docker top <container name/id>

Reveals running processes inside the container.

Docker Best Practices

Adhering to best practices is paramount for ensuring security and stability within your Docker environment:

Image Security:
- Use verified images for enhanced security.
- Leverage Container Image Scanners like Clair, Trivy, and Dagda for unverified images.
Avoid Using Latest:
- Specify image versions to maintain stability.
User Privileges:
- Utilize non-root users within containers for increased security.

Where to Go Next

Docker Compose:

Multi-Container Apps:
- Docker excels for single-application deployment.
- For apps with multiple components, use Docker Compose to define containers and their relationships in a single Compose Manifest.
Getting Started:
- Start containers with:
```
 docker-compose up
```

Kubernetes:

Scaling for Production:
- Docker eases single-system container deployment.
- Kubernetes addresses challenges in managing hundreds of thousands of containers in production.
Kubernetes Challenges:
- Overcoming difficulties in linking docker networks across hosts.
- Managing containers across multiple hosts.
- Lack of built-in solutions for container migration between hosts.
- Production concerns like load balancing and securing traffic are challenging with Docker client alone.
Container Orchestrations:
- Solutions like VMware’s vCenter and Rundeck utilize orchestrators for easy scaling, container movement, and traffic routing.

Planet-Scale Container Orchestrator - Kubernetes:

Kubernetes automates deployment, scaling, and management of containerized applications.
It's designed for distributed systems, running components across multiple machines.
Enables auto-scaling and URL-based traffic routing, making it a platform of platforms.

Conclusion:

Navigating Docker troubleshooting, adopting best practices, and exploring advanced orchestration tools are essential steps in mastering containerization. Whether you choose Docker Compose for multi-container simplicity or venture into the vast landscape of Kubernetes, each path unlocks new possibilities for deploying and managing your applications.

References:

Docker Documentation. Docker system prune.
Docker Documentation. Manage data in containers.
Clair. Vulnerability Static Analysis for Containers.
Trivy. A Simple and Comprehensive Vulnerability Scanner for Containers.
Dagda. A tool to perform static analysis of known vulnerabilities, trojans, viruses, malware & other malicious threats in Docker images/containers.
Docker Compose Documentation. Overview of Docker Compose.
Kubernetes Documentation. Kubernetes Basics.
VMware. vCenter Server.
Rundeck Documentation. Introduction to Rundeck.

Create Containers and Interaction in Docker

Ramkumar Jayakumar — Wed, 17 Jan 2024 13:00:00 +0000

Building Containers: The Detailed Way

Containers in Docker are born from container images—a compressed, pre-packaged file system containing your app, its environment, and configuration instructions. Let's embark on the journey of container creation, starting with the meticulous approach.

Create Container Using 'docker container create'

Understanding Images:
- Images are compressed file systems with instructions on how to start your app.
- Docker Hub serves as a default repository for images.
Command to Create Container:

   docker container create --help
   docker container create hello-world:linux

The create command doesn't start the container; it merely generates it.

Listing Containers:

   docker ps
   docker ps --all

The status "0 exit code (exited)" indicates successful execution.

Starting the Container:

   docker container start <container_id>

Check logs using docker logs <container_id>.

Choosing the detailed way may be preferable in scenarios where you want granular control over container creation, particularly when specific configurations or setups are required.

Shortcut: 'docker run' Command

On the flip side, the 'docker run' command offers a more efficient and convenient way to create and run containers:

Single Command Magic:

   docker run <image_name>

This command combines create, start, and attach actions.

Note:
- Use docker ps to list containers and retrieve IDs.

Experimenting with both methods is encouraged. The 'docker run' command is often favored for its simplicity, making it an excellent choice for quick development iterations.

Dockerfile and Image Creation

Dockerfile Basics

Introducing Dockerfile basics is essential for efficient image creation. Let's explain each component succinctly:

Example Dockerfile:

   FROM ubuntu:latest
   LABEL maintainer="Your Name"
   USER nobody
   COPY . /app
   RUN apt-get update && apt-get install -y curl bash
   USER nobody
   ENTRYPOINT ["curl"]

Explanation:
- FROM ubuntu:latest: Specifies the base image as the latest version of Ubuntu.
- LABEL maintainer="Your Name": Adds metadata to the image, specifying the maintainer.
- USER nobody: Sets the default user for subsequent commands to 'nobody'.
- COPY . /app: Copies the contents of the current directory to the '/app' directory in the image.
- RUN apt-get update && apt-get install -y curl bash: Updates package lists and installs 'curl' and 'bash'.
- USER nobody: Switches back to the 'nobody' user.
- ENTRYPOINT ["curl"]: Defines the default executable for the container as 'curl'.

Building an Image:

   docker build -t <image-name> .

Use -t to tag an image.

Intermediary Images:
- Docker generates intermediate images for each line in the Dockerfile.
- The final image is squashed and tagged.
Running the Image:

   docker run <image-name>

Removing Images

To maintain a clean and efficient system, it's crucial to remove unused images:

To remove an image:

 docker rmi <image_name>

Use docker images to list images.

Advanced Container Operations

Container Naming

Organizing advanced container operations under subheadings for better structure:

Assign a name to your container:

  docker run -d --name <container_name> <image_name>

Port Binding

Bind ports for network access:

  docker run -d --name <server_name> -p 5001:5000 <image_name>

Access the server at http://localhost:5001.

Volume Mounting

Preserve data outside containers using volumes:

  docker run --rm --entrypoint sh -v /tmp/container:/tmp ubuntu -c "echo 'Hello There.' > /tmp/file && cat /tmp/file"

Map a folder on your machine to a folder in the container.

Container Registries

Understanding Registries:
- Container image registries track and store container images.
- Docker Hub is the default registry.
Pushing Images to Docker Hub:

   docker login
   docker tag <image_to_rename> username/new_image_name
   docker push <image-name>

Deleting Images in Docker Hub:
- Delete images through the Docker Hub settings in the browser.

Conclusion

Mastering container creation and interaction forms the backbone of Docker. Whether you choose the detailed or shortcut approach, understanding the nuances empowers you to efficiently manage and deploy your applications.

References:

Anatomy of Docker

Ramkumar Jayakumar — Mon, 15 Jan 2024 13:00:00 +0000

In this section, we delve into the intricate components that form the backbone of Docker. Understanding the anatomy of Docker is essential for harnessing its true power in containerization.

Container Components:

Docker operates on the fusion of Linux namespaces and control groups, providing a unique and isolated environment for applications. Let's simplify this complex structure:

Linux Namespace:
- A Linux kernel feature providing different "views" of your system to applications.
- Offers a unique and isolated instance of system resources, allowing independent operation of processes or containers.
Linux Control Group (cgroup):
- Another Linux kernel feature restricting how much hardware each process can use.
- Utilized for monitoring and restricting CPU, network, disk bandwidth, and memory consumption.
- Assigning disk quotas (not supported by Docker).

Note: While Docker is native to Linux and some newer versions of Windows, containers can run on any OS. Docker is compatible with Windows Server, and with the introduction of Windows Subsystem for Linux (WSL) from Windows 10 Anniversary Edition onwards, performance on Windows has significantly improved.

Linux's Eight Kernels:

Let's briefly explore the eight namespaces in the Linux kernel that provide isolation for processes, networking, interprocess communication, and more:

PID Namespace (pid):
- Isolates the process ID number space, ensuring processes in different namespaces have distinct PIDs.
Network Namespace (net):
- Provides isolation in terms of network resources.
- Enables each namespace to have its own network interfaces, routing tables, and firewall rules.
Mount Namespace (mnt):
- Isolates the set of mount points seen by a group of processes.
- Ensures each namespace has its own filesystem hierarchy.
IPC Namespace (ipc):
- Isolates interprocess communication resources.
- Each namespace has its own System V IPC, message queues, and semaphore arrays.
UTS Namespace (uts):
- Isolates system identifiers like hostname and NIS domain name.
- Enables each namespace to have its own hostname and domain name.
User Namespace (user):
- Provides isolation for user and group IDs.
- Allows different namespaces to have different user and group mappings.
Cgroup Namespace (cgroup):
- Introduced in Linux kernel 5.10.
- Enables isolation for control groups, allowing each namespace to have its own set of cgroups.
Time Namespace (time):
- Introduced in Linux kernel 5.6.
- Ensures each namespace has its own perception of time.

Docker Advantages:

Dockerfiles:

Dockerfiles simplify configuring and packaging apps and their environments.

Users instruct Docker on container configuration through Dockerfiles.
Docker utilizes these Dockerfiles to package apps and their environments into images.

Example:



# Dockerfile example (more details in the next blog)

FROM ubuntu:latest

RUN apt-get update && apt-get install -y nginx

CMD ["nginx", "-g", "daemon off;"]

Sharing Images:

Docker Hub, a global repository, facilitates easy image sharing.
Users can utilize alternative registries like AWS Elastic Container Registry (ECR), GCP Google Container Registry (GCR), Quay.io, Harbor, Azure Container Registry (ACR) etc., and share images effortlessly.

Docker CLI:

Docker CLI makes starting apps in containers a breeze.
No worries about UID mappings, network interfaces, or Docker configuration files.

Docker Limitations:

Docker relies on Linux namespaces and control groups, making it technically native to Linux and some newer versions of Windows.
Containers can run on any OS, but their images are bound to their parent OS. Workarounds exist for these limitations.

Docker Alternatives:

Container Runtime Interface (CRI):

Kubernetes introduced CRI, enabling developers to create their container runtimes.
Examples: CRI-O, RUNC, Firecracker from AWS.
Podman by RedHat for highly secure, rootless containers.

Docker Machine:

Docker Machine serves as a workaround for Mac and Windows users, allowing them to run Docker on their systems. However, if you are using Linux, there are specific distributions optimized to support container runtimes more efficiently.

Docker Machine for Mac/Windows:

Designed as a workaround for non-Linux environments.
Utilizes VirtualBox to create virtual machines running Docker.
Offers a solution for those operating systems lacking native Docker support.

Linux Distributions Optimized for Containers:

When using Docker on Linux, consider distributions optimized for container runtimes. Examples include:

Ubuntu: A widely used and well-supported Linux distribution with good compatibility for Docker.
Alpine Linux: Known for its lightweight nature, making it an excellent choice for containerized applications.
Fedora CoreOS: A minimal OS designed explicitly for running containerized applications at scale.

Performance Considerations:

Docker Machine on Mac and Windows may exhibit slower performance due to the virtualization layer and dependency on VirtualBox.
On Linux, choosing a distribution optimized for containers can enhance Docker's performance and resource utilization.
When selecting a Linux distribution for Docker, it's essential to consider factors like compatibility, resource efficiency, and the level of support provided for container runtimes. Each distribution may have its strengths, so choose the one that aligns best with your specific requirements.

Docker Desktop:

A more refined solution introduced in 2016.
Smaller and faster VM running on Apple's native hypervisor or Hyper-V on Windows.
Improved GUI for configuration, mounting volumes, and exposing network ports.
Addresses the limitations of Docker Machine.

Conclusion:

Understanding Docker's anatomy is crucial for harnessing its power. From Linux namespaces to Dockerfiles, we've peeled back the layers. Stay tuned for the next blog, where we'll dissect Dockerfiles and dive deeper into crafting containerized environments.

References:

Why Docker?

Ramkumar Jayakumar — Sat, 13 Jan 2024 12:55:40 +0000

Have you ever found yourself frustrated with the "It works on my machine" dilemma during your software development journey? Despite careful testing in various environments, your application misbehaves elsewhere, falling victim to the common challenges of software engineering.

The Landscape of Solutions:

Configuration Management:
Tools like Chef, Ansible, and Puppet tackle the issue by enabling developers to write code in configuration languages. This code describes precisely what machines need to run your app.
Drawback: Requires in-depth knowledge of underlying OS and hardware for tools like Chef, Puppet, and Ansible.
Virtual Machines as Code (Vagrant):
HashiCorp’s Vagrant empowers developers to script entire virtual machines for their apps.
Drawback: Involves understanding the virtual machine's hardware requirements and often configuring the OS before app installation. Heavy, relatively slow, and demands inconvenient configurations.

A Paradigm Shift: Docker

Why Docker?
Amidst these challenges, Docker emerges as a game-changer. It empowers developers to package their applications into images that seamlessly run on containers. These containers act as virtualized operating systems, configured with just enough resources to run your app.

Virtual Machines vs Containers:
Virtual Machines:

Run on a platform called Hypervisor.
Hypervisor emulates hardware operations, translating them to real hardware operations on hosts.
Takes up a lot of space.
Requires OS installation and configuration.
Can run multiple apps simultaneously but can't directly interact with the host securely.

Containers:

Run on container runtimes.
Runtimes work with the OS to allocate hardware and copy files, running like any other app on the system.
Do not emulate hardware, don't need to "boot up," and use the same hardware and OS.
Do not require OS installation, enabling quick startup.
Take up less space, allowing more apps to run.
Run only one app at a time by design but can interact with the host, posing security challenges (mostly resolved).

The Docker Advantage:
Docker streamlines the development process by creating images from lightweight configuration files, detailing everything your app needs to run. Unlike traditional virtual machines, containers hide hardware details, ensuring consistent app performance across different environments.

Portability and Efficiency:
Build your app once, and if the machine can run Docker, your app will run consistently, irrespective of its location. Docker brings efficiency to the development process, enabling you to build, deploy, and scale your applications quickly, safely, and cost-effectively.

In a nutshell, Docker leverages images and containers to offer a universal solution—run your apps anywhere with consistent behavior. Say goodbye to the chaos of environment discrepancies, and embrace the power to build, deploy, and scale your applications effortlessly.

Next Steps:

In this beginner's guide to Docker, we'll unravel the world of containerization, exploring how Docker empowers developers to craft, share, and run applications seamlessly. Let's decode Docker and unlock its potential for your development endeavors.

References:

Persistent Data Grids in Angular: A Comprehensive Guide

Ramkumar Jayakumar — Wed, 20 Dec 2023 10:15:57 +0000

Ever wondered how web applications keep your search data and grid configurations intact as you navigate through pages? This seamless experience is thanks to data grid implementations—a cornerstone of modern web development.

Data grids are essential tools, revolutionizing the presentation and interaction with vast sets of information. Picture online banking platforms allowing effortless transaction filtering or project management tools dynamically updating tasks without a page refresh. These real-world scenarios underscore the power of data grids, offering users intuitive controls for customization, sorting, and real-time updates.

This blog explores the intricacies of achieving persistent storage for data grids in Angular, eliminating the need for manual data synchronization. Join us as we delve into ensuring a seamless user experience where apps effortlessly retain grid configurations and search data.

Understanding the Challenge

Before delving into the technical details, let's briefly outline the key elements of an application's data grids. Typically, a data grid consists of an external filter that searches a database and a grid that displays the retrieved records. Two critical pieces of information need to be preserved:

Search Data: Retaining all the searches made in external filters whenever the page is visited.
Grid Configuration: Storing options like selected columns, column width, and column order, which should persist across visits.

The next important thing is how long this data must be stored. Deciding when to retain or discard the data depends on the context. Search data is often considered temporary, while grid configurations are expected to persist across user sessions.
So, the search data should be stored for a particular session against a data grid, while grid configurations are stored against a data grid and the user ID.

Choosing the Right Storage Option

Now that we've identified what data to store and when, let's explore the options available to achieve persistent storage. Angular offers an array of options for persistence, each with its own strengths and quirks:

LocalStorage: Persistent even after closing the tab, ideal for large data and simple settings. Performance would be a concern for large chunks of data since we will be parsing and stringifying the large data.
Example: User preferences like language or theme persist across logins.
SessionStorage: Similar to LocalStorage, but data vanishes when the browser window closes.
Example: Shopping cart contents during a single shopping session.
IndexedDB: A more powerful local database with richer object storage capabilities. While IndexedDB offers powerful storage capabilities, its complexity and overhead, encompassing factors like error handling challenges, potential performance limitations, and reduced compatibility across browsers and modes, make it a less straightforward choice for simple persistent data grid storage.
Example: Offline data persistence for complex applications.
Cookies: small data packages sent between browser and server, useful for tracking user sessions, but their privacy concerns, technical limitations, and potential negative impact on the user experience make them a less ideal choice.
Example: Remembering login credentials for returning users.
Angular Routes: Embed data directly in the URL, making it bookmarkable and SEO-friendly. Can be used for simple configurations
Example: Sharing specific content configurations with a link.
/analytics/user-report?start_date=2023-10-01&end_date=2023-12-19 (shows specific date range for analysis)
Angular Services: Services act as data managers, making your app cleaner, more organized, and easier to work with. Its key advantages are code reusability, abstraction, and synchronization. Data will be lost upon page refresh/ tab closure.

Remember: The best choice depends on your application's needs. Consider data size, persistence duration, security implications, and potential limitations.

Storage Options Implementation

In my implementation, I've opted for a hybrid approach, utilizing both local storage and a backend database to store different types of data. Here's a glimpse of the structure used for storing grid configurations.



{
    "grid_name": {
        "external_filters": {
            "name": null,
            "customer": [ 1234, 1235],
            "email": null,
            "some_boolean_filter": false,
            "created_at": "2023-12-13T18:30:00.000Z"
        },
        "columns_config": [
            // Individual Column Configurations
            {
                "field": "name",
                "width": 120,
                "title": "Name",
                "orderIndex": 1
            },
            {
                "field": "amount",
                "width": 100,
                "title": "Amount(₹)",
                "cssClass": "text-end",
                "orderIndex": 3
            },
            {
                "field": "email",
                "width": 100,
                "title": "Email",
                "hidden": true,
                "orderIndex": 2
            }

        ], 
        "state": {
            // Grid State Information
            "skip": 0,
            "take": 10,
            "sort": []
        }
    }
}

Saving Grid Configurations

Functional Breakdown

Let's get to the exact details as to how and when we save the data. First, we need to break down each of the operations into separate functions so that we can reuse them. We will have the following functions:

saveGridSettings: This function builds a data structure containing column configurations, grid state, and external filter data. It then saves this data to LocalStorage.



saveGridSettings() {
    const external_filters = this.searchForm.value;
    const grid_config = {
        contractor_list: {
            state: this.grid_state,
            external_filters: external_filters,
            columns_config: this.columnsConfig
        }
    };
    this.persistentStorageService.setGridSettings('grid_name', grid_config);
}

setGridSettings(grid_key: string, gridConfig: any): void {
localStorage.setItem(grid_key, JSON.stringify(gridConfig));
}


- **saveColumnConfigSettings**: This function saves the user-specific column configuration for the current grid to the backend via an API call. It's triggered when the "Apply" button is clicked.

saveColumnConfigSettings(event: any) {
    const input_data = {
        grid: 'grid_name',
        columns_config: event
    };
    this.saveGridSettings();

    this.persistentStorageService.saveColumnConfigSettings(input_data).subscribe({
        next: (response: any) => {
            this.toast.showSuccess(response.message);
            this.getColumnConfigSettings();
            this.contractor_list_grid_loading = false;
        },
        error: (error: any) => {
            this.toast.showError(error.message);
        }
    });
}


- **getColumnConfigSettings**: This function retrieves the user-specific column configuration for the current grid from the backend. It's used during component initialization.

getColumnConfigSettings() {
    const input_data = {
        grid: 'grid_name'
    };

    this.persistentStorageService.getColumnConfigSettings(input_data).subscribe({
        next: (response: any) => {
            if (response && response['columns_config'].length !== 0) {
                this.mapGridSettings(response['columns_config']);
            } else {
                this.initialGridSettings();
            }
        },
        error: (error: any) => {
            this.toast.showError(error.message);
            this.initialGridSettings();
        }
    });
}


- **initializeGridRememberSettings**: This function checks for existing grid settings in LocalStorage. If unavailable, it fetches the user's column configuration via API or uses the default configuration if none exists.

ngOnInit(): void {
    // Other initializations
    this.initializeGridRememberSettings();
}

initializeGridRememberSettings() {
    if (!!this.savedState) {
        this.mapGridSettings();
    } else {
        this.getColumnConfigSettings();
    }
}


### Application Flow

![Persistent data grids in angular workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jkk20kcat00bfuzl2hdq.png)

- **User interaction**: When the user applies filters or modifies the grid layout, the updated state (including external filters, column configurations, and grid state) is saved to LocalStorage with the help of state change events.

- **Local Storage cache**: This cached data ensures quick restoration of configurations when the user revisits the page.

- **API calls**: Saving of user-specific column configurations to the backend happens only when the user explicitly confirms with the "Apply" button, triggering the saveColumnConfigSettings function.

- **Component initialization**: During initial page load, the initializeGridRememberSettings function checks for LocalStorage data. If unavailable, it retrieves the user's configuration via API or uses the default configuration.

Overall, this approach leverages LocalStorage for efficient caching and utilizes the backend for persistent, user-specific configuration storage.

## Conclusion

Choosing the right approach for persistent storage in Angular data grids involves careful consideration of your application's requirements. In my case, LocalStorage with a backend database for grid configurations provided a flexible and scalable solution.

However, LocalStorage is not without drawbacks. Performance concerns arise due to the parsing and stringification processes involved. Instead, leveraging an Angular service to store grid data can optimize performance by avoiding these processes. This comes at the cost of losing data upon tab closure. Ultimately, the choice depends on your specific needs.

To further enhance this solution, consider maintaining a queue in LocalStorage with a maximum capacity of 10 data grids. This minimizes storage size by evicting older entries, assuming users navigating beyond 10 pages are unlikely to revisit those specific searches.

Remember, thorough analysis and planning before diving into code is crucial for ensuring a robust and efficient solution for persistent data grids in your Angular application. Happy coding!

## Further Resources

- What are data grids? [Introducing In-Memory Data Grid — Hazelcast IMDG | by Dina Bogdan | The Startup | Medium](https://medium.com/swlh/introducing-in-memory-data-grid-hazelcast-imdg-1f2af2be5344)

- For a deeper dive into Angular Routes, see [Angular Tutorial](https://www.mygreatlearning.com/blog/angular-routing-tutorial/).

- Learn more about LocalStorage and SessionStorage [Beginner's Guide to Web Storage](https://levelup.gitconnected.com/beginners-guide-to-web-storage-a88b6e5b522e)

- A Developer's Guide to HTTP Cookies: [cookies - Mozilla | MDN](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/cookies)

- Introducing IndexedDB: [IndexedDB API](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API)