DEV Community: jiisanda🙆‍♂️

Docker Rootless: high security and high performance

jiisanda🙆‍♂️ — Tue, 11 Feb 2025 20:24:24 +0000

On installing docker on Ubuntu or Debian system using the following command…

$ curl -sSL https://get.docker.com/ | sh

We might get the following message at the end of the command that says…

To run Docker as a non-privileged user, consider setting up the
Docker daemon in rootless mode for your user:

    dockerd-rootless-setuptool.sh install

Visit https://docs.docker.com/go/rootless/ to learn about rootless mode.

What does that mean?

By default, Docker requires root privileges to run, as it interacts with system-level components such as Docker daemon and system resources like networking and storage. However, running Docker as root can pose security risks.

To mitigate these risks and allow non-privileged users to run Docker without granting them root access, Docker introduced a feature called “rootless mode”. In rootless mode, the Docker daemon and containers run entirely within the user’s own namespace, without needing root privileges. In this article, we will explore how Rootless mode works, setting up, advantages, limitations and use cases.

🤔 How Rootless mode work?

Docker rootless fundamentally changes the traditional way of running the both Docker daemon and containers as an unprivileged user. This is how it is done:

First, rootless mode uses user namespace, which is a Linux kernel feature that lets unprivileged users create containers. Inside these namespaces, a user can have privileges that look like root permissions, but these privileges are actually restricted to the namespace. This is the same as having admin rights in a sandbox - we can do administrative tasks, but only within that confined space.

The rootless mode operates by running the Docker daemon as a regular user process instead of root. This daemon process creates a user namespace where it appears to have root privileges, but these privileges don’t extend outside the namespace.

Network management becomes interesting in rootless mode. For port binding below 1024 (traditionally requiring root), rootless mode can use rootlesskit, which provides an unprivileged port mapping feature. This lets containers bind to privileged ports through user-space root privileges.

One important technical detail is how subuid and subgid mapping work. The system needs to map the user’s UID inside the containers to a range of UIDs on the host system. This mapping is configured in /etc/subuid and /etc/subgid, allowing the user namespace to work properly while maintaining security isolation.

🟡 Prerequisites

Before setting up Docker in Rootless mode, there are several prerequisites to consider:

You must install newuidmap and newgidmap on the host. These commands are provided by the uidmap package.
For Debian / Ubuntu:

$ sudo apt-get install uidmap

Check Current User’s UID/GID: /etc/subuid and /etc/subgid contains a significant number of subordinate UIDs/GIDs for the user. Let’s verify that:

$ id -u
1001
$ whoami
jiisanda
$ grep ^$(whoami): /etc/subuid
jiisanda:231072:65536
$ grep ^$(whoami): /etc/subgid
jiisanda:231072:65536

By verifying these steps, you ensure that the user running Docker in Rootless mode has a sufficient number of subordinate UIDs/GIDs allocated for proper operation.

Install the dbus-user-session package: On Ubuntu, installing the dbus-user-session package is recommended for proper functioning of certain system services and components. This package provides the D-Bus session bus for user sessions, which can be crucial for running services like Docker.

To install dbus-user-session using the following command…

$ sudo apt-get install dbus-user-session

📥 Install

If the system-wide Docker daemon is already running, consider disabling it:

$ sudo systemctl disable –now docker.service docker.socket
$ sudo rm /var/run/docker.sock

If installed Docker 20.10 or later with RPM/DEB packages, it’s a pretty simple set up. You should have docker-rootless-setuptools.sh in /usr/bin.

Run dockerd-rootless-setuptool.sh install as a non-root user to set up the daemon:

$ dockerd-rootless-setuptool.sh install
[INFO] Creating /home/testuser/.config/systemd/user/docker.service
...
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger testuser`

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

$ export PATH=/usr/bin:$PATH
$ export DOCKER_HOST=unix:///run/user/1000/docker.sock

If dockerd-rootless-setuptool.sh is not present, you may need to install the docker-ce-rootless-extras package manually, as…

$ sudo apt-get install -y docker-ce-rootless-extras

✅ Best Practices for using Docker Rootless

Ensure Your System Meets Requirements
- Use a modern Linux kernel (5.11+ recommended for better performance).
- Install necessary dependencies like uidmap, slirp4netns, fuse-overlayfs.
- Run dockerd-rootless-setuptool.sh check to verify system compatibility.
Use OverlayFS for Better Performance
- By default, fuse-overlayfs is used in rootless mode.
- If your system supports it, enable OverlayFS (--storage-driver=overlay2) for better disk performance.
Optimize Networking for Performance
- Rootless mode uses slirp4netns for user-space networking, which can be slower.
- If possible, use Host networking (--network=host) when running containers that require better performance.
- Consider VPNKit or RootlessKit for improved networking options.
Use Systemd for Better Stability
- Start Docker Rootless with systemd instead of running it manually, this ensures Docker restarts automatically after a system reboot:
```
$ systemctl --user enable docker
$ systemctl --user start docker
```
Limit Resource Usage
- Since Rootless Docker runs as a user process, it doesn’t have system-wide privileges.
- Use cgroups v2 to set resource limits for CPU, memory, and I/O.
Example:
```
$ docker run --memory=512m --cpus=1 my-container
```
Enable Logging & Debugging
- Check logs to troubleshoot issues, after running the command below run docker info to verify that Docker is running in rootless mode:
```
$ journalctl --user -u docker.service
```

🚀 Use cases for Docker Rootless

Multi-User Environments (Universities, Research Labs)
- Allows different users to run containers without needing root privileges.
- Prevents one user’s container from affecting another user’s work.
Secure Development Environments
- Developers can run Docker without risking system security.
- Ideal for isolated testing and running containers on personal machines.
CI/CD Pipelines
- Useful for CI/CD environments where security is a priority.
- No need to grant root access to build agents running Docker.
Shared Hosting & Workstations
- Hosting providers can allow customers to run Docker containers without exposing the entire system.
- Perfect for corporate workstations where security policies prohibit root access.
Running Containers in Restricted Environments
- If you don’t have sudo access (e.g., cloud-hosted VMs or corporate laptops), rootless mode allows you to use Docker safely.

🚨 When NOT to Use Docker Rootless

❌ High-Performance Production Environments
- Rootless networking (slirp4netns) is slower than rootful networking. Storage performance is slightly lower without direct disk access.
❌ If You Need Privileged Containers
- Rootless Docker cannot run privileged containers (--privileged flag won’t work). Containers cannot access low-level hardware (e.g., GPUs, USB devices).
❌ If You Need System-Wide Networking
- Rootless mode does not support port bindings under 1024 (e.g., you can’t expose 80:80). Workarounds exist (iptables tricks), but it’s not as flexible as rootful Docker.

Conclusion

Docker Rootless is a great choice for secure, non-privileged environments like development, CI/CD, and multi-user systems. However, for production workloads requiring high performance, rootful Docker is still the better option.

AWS Network Fundamentals for EC2 instance!

jiisanda🙆‍♂️ — Sun, 29 Dec 2024 10:33:32 +0000

Have you ever launched an EC2 instance by following a tutorial, blindly inputting values without fully understanding their significance? While it might work at first glance, the magic behind EC2 lies in its networking fundamentals—Virtual Private Clouds (VPCs), subnets, gateways, route tables, and security groups all working together to ensure your instance is both functional and secure.

In this article, we’ll go beyond surface-level configurations. You’ll gain a clear understanding of the core networking concepts that power AWS EC2. We’ll explore how to create and configure VPCs, set up public and private subnets with appropriate CIDR ranges, manage gateways for connectivity, and secure your network using security groups and Network ACLs.

To make it even more engaging, we’ll include step-by-step YouTube embeds to show how to configure each component in the AWS Management Console. By the end, you won’t just know how to set up an EC2 instance—you’ll know why each step matters and how it impacts your cloud architecture.

Let’s dive in and master the backbone of AWS EC2 networking!

At the starting point we just have a AWS Cloud, so just go to https://aws.amazon.com and create your free account. After that we have something called VPCs (Virtual Private Cloud).

VPC

A VPC is a virtual private cloud, which works like a private network to isolate the resources within it. The solution described in these topics uses an AWS service called Amazon VPC.

It's like a fence around a bunch of resources, separating all of the resources within that fence from another VPC.

Here, while creating a VPC, we specified a CIDR of 10.0.0.0/16. But what exactly is a CIDR range, and how do /24 or /16 differ?

What is a CIDR Range?

CIDR (Classless Inter-Domain Routing) is a method used to allocate IP addresses efficiently. Instead of using fixed blocks of IPs, CIDR allows for flexible assignment of IP ranges. A CIDR block is expressed as an IP address, followed by a slash (/) and a number indicating how many bits are used for the network prefix.

For example, in the CIDR 10.0.0.0/16:

10.0.0.0 is the base IP address.
/16 means that the first 16 bits are reserved for the network portion, leaving the remaining bits for defining individual hosts.

CIDR Block Size and Address Allocation

The number after the slash determines how many IP addresses are available in the range:

/16: Reserves the first 16 bits for the network. This allows for 2^(32-16) = 65,536 IP addresses, which is suitable for large subnets. Example: 10.0.0.0 to 10.0.255.255.
/24: Reserves the first 24 bits, leaving 8 bits for hosts. This provides 2^(32-24) = 256 IP addresses, which is common for smaller manageable subnets (e.g., public and private subnets). Example: 10.0.0.0 to 10.0.0.255.

Now within this VPC, we have isolated networks, with different CIDR ranges called subnets.

Subnets

A subnet is a defined set of network IP addresses that are used to increase the security and efficiency of network communications. You can think of them like postal codes, used for routing packages from one location to another. The Subnets list in the Amazon VPC console displays subnet IDs and also their associated VPC IDs, route tables, and network ACLs. You need to provide at least two subnets in different availability zones to create a VPC connection.

A subnet is a range of IP addresses in our VPC. We can create AWS resources, such as EC2 instances, in specific subnets.

Each subnet must reside within one Availability Zones. By launching AWS resources in different Availability zones, we can protect our application from failure of single zone.

Normally, for pet projects we can create 2 subnets, a public subnet and a private subnets, with different Availability zone.

Public Subnet: The subnet has a direct route to an internet gateway (discussed further). Resources in a public cloud can access the public internet.
Private Subnet: The subnet does not have a direct route to an internet gateway. Resources in a private subnet require a NAT gateway (discussed further) to access the public internet.

So, following is what we have now: a VPC with two subnets, PublicSubnet and PrivateSubnet. While creating these subnets, we assigned:

PublicSubnet: IPv4 CIDR block 10.0.0.0/24
PrivateSubnet: IPv4 CIDR block 10.0.1.0/24

Significance of Subnet CIDR Blocks

The CIDR blocks define the range of IP addresses available within each subnet. Here's the breakdown:

PublicSubnet (10.0.0.0/24)
- This subnet can host resources that require internet access.
- The /24 CIDR block means there are 256 available IPs (2^8) in this subnet, ranging from 10.0.0.0 to 10.0.0.255.
- AWS reserves 5 IP addresses from this range for internal use, leaving 251 usable IPs for EC2 instances and other resources.
PrivateSubnet (10.0.1.0/24)
- This subnet is meant for resources that don’t need direct internet access, such as databases or internal services.
- The /24 CIDR block provides another 256 IPs, ranging from 10.0.1.0 to 10.0.1.255, with 251 usable IPs after AWS reservations.

By assigning distinct CIDR blocks to each subnet, we ensure that:

Resources in the PublicSubnet and PrivateSubnet don’t overlap or conflict.
We maintain clear separation for routing and security purposes.

Recap of Our Current Setup

We now have:

A VPC with a CIDR range of 10.0.0.0/16, giving us up to 65,536 IP addresses.
Two subnets within this VPC:
- A PublicSubnet (10.0.0.0/24) for internet-facing resources.
- A PrivateSubnet (10.0.1.0/24) for internal resources.

Now we are going to launch an EC2 instance into our public subnet.

Launching Public EC2

So, we launched the instance with basic configurations: naming it, selecting the AMI, choosing the instance type, and creating a key-pair for secure access. Then, in the Network Settings, we selected the VPC and public subnet we created earlier. To ensure our instance could communicate with the internet, we enabled public IP generation.

Next, we configured the Security Group, which acts as a virtual firewall for our instance to control incoming and outgoing traffic. We allowed SSH traffic (port 22) in the Security Group, which enables us to connect to the instance using our key-pair securely from our local machine.

But Wait, Was That Enough?

No! We encountered an error while trying to connect to our EC2 instance. This happened because our VPC doesn’t yet have an Internet Gateway (IGW). An IGW is essential to allow internet communication for resources within a public subnet. Without it, even instances with public IPs cannot connect to the internet.

What's Next?

Now, let’s resolve this issue!
We’ll add an Internet Gateway to our VPC to enable connectivity for our PublicSubnet.

Gateway

A gateway connects your VPC to another network. Example, An internet gateway is a VPC component that allows communication between instances in your VPC and the internet.

The Internet Gateway acts as a doorway, allowing traffic to flow between the internet and the instances in our public subnet. However, simply attaching the Internet Gateway to our VPC isn’t enough—we also need to provide a route for traffic to reach the gateway.

In the video, we successfully created an Internet Gateway and attached it to our VPC. Note: A VPC can only have one Internet Gateway at a time.

But Wait, Why Can’t We Connect Yet?

Even though the Internet Gateway is attached, the public subnet doesn’t know how to send traffic to the gateway. This happens because the Route Table associated with the subnet doesn’t include a route to the Internet Gateway. Without this, traffic to and from the internet has no defined path, leaving our instance unreachable.

What's Next?

Let’s fix this by updating the Route Table associated with our PublicSubnet. We’ll add a route to the Internet Gateway, enabling traffic to flow between the internet and our EC2 instance seamlessly.

Route Table

A route table contains a set of rules, called routes, that are used to determine where network traffic is directed. You can view the route table in the Amazon VPC console at https://console.aws.amazon.com/vpc/. The VPC details display the route table that the VPC is using. You can also see Route tables listed in the Amazon VPC console.

Think of a route table as a GPS system for your network traffic. It helps direct traffic to its intended destination, whether it’s within the VPC or outside to the internet.

In the VPC dashboard, under the "Route Table" section, you’ll notice that your VPC comes with a default Route Table—referred to as the main Route Table. This is used by all unassociated subnets by default.

The local route (10.0.0.0/16) in this table allows communication between resources within the VPC, such as the public and private subnets. However, it doesn’t direct traffic outside the VPC.

Creating Separate Route Tables

By default, both public and private subnets use the main Route Table. To ensure proper isolation and functionality:

Create a dedicated Route Table for the public subnet.
Create another Route Table for the private subnet.

This separation ensures public resources can communicate with the internet, while private resources remain secure and isolated.

Associating and Adding Routes

After creating the Route Tables, associate each table with its corresponding subnet—PublicSubnet and PrivateSubnet.

For the PublicSubnet Route Table:

Add a route with the following details:
- Destination: 0.0.0.0/0 (all internet traffic).
- Target: Internet Gateway (IGW).

Save the changes, and now your public subnet has internet connectivity!

Success!

And just like that, our EC2 instance is connected to the internet. Now, we can SSH into it, update packages, and configure it further directly from the shell. Let’s launch an EC2 instance into our private subnet.

Private Instance and NAT Gateways

So, for the private subnet, we created another EC2 instance. While doing so, we selected the same VPC and ensured the instance was launched in the private subnet. We successfully connected to the private instance using SSH from the public instance, thanks to the route table configurations. Also, don’t forget to have your .pem file in the public instance with the correct permissions (chmod 400).

However, once inside the private instance, when we tried to update the packages, we encountered an issue—our private instance couldn’t access the internet. Why? Because the private subnet is isolated from direct internet access for security reasons.

Here’s where the NAT Gateway comes into play.

A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances.

Think of it as your instance requesting updates or connecting to an external API—your NAT Gateway makes the request on behalf of your private instance, retrieves the response, and passes it back to the instance. This ensures your private subnet remains shielded from any unsolicited access.

Here are a few suggestions to refine and enhance your explanation about NAT Gateway and its role in enabling secure internet access for private subnets:

After creating our NAT Gateway in the public subnet, we configured the private subnet’s route table to direct outbound traffic to the NAT Gateway. Here’s how it works:

Why the NAT Gateway is in the Public Subnet:
- The NAT Gateway itself needs internet access to route outbound traffic from the private subnet. By placing it in the public subnet, it can utilize the internet gateway already configured for public internet access.
Routing the Private Subnet through the NAT Gateway:
- We updated the Private Route Table to include a new route:
  - Destination: 0.0.0.0/0 (all internet traffic)
  - Target: The NAT Gateway ID
- This ensures all outbound traffic from the private subnet destined for the internet is routed via the NAT Gateway.
Testing the Connection:
- After updating the route table, we SSH’d into the private instance using the public instance as a jump host.
- We then successfully ran package updates (sudo yum update -y or sudo apt update), confirming that the private instance could access the internet.

NACLs and Security Group

A network access control list (ACL) allows or denies specific inbound or outbound traffic at the subnet level. You can use the default network ACL for your VPC, or you can create a custom network ACL for your VPC with rules that are similar to the rules for your security groups in order to add an additional layer of security to your VPC.

A security group is a set of rules that controls the network access to the resources it is associated with. Access is permitted only to and from the components defined in the security group's inbound and outbound rules. If no rules are defined, the security group prevents all access. You can view security groups from several different consoles, depending on which resource that a particular security group applies to. You can see all the security groups and their settings in one place in the VPC console.

The network is configured securely, and we understand the significance of each component in making this work.

Summary of What We've Accomplished:

VPC Creation: We built a secure Virtual Private Cloud with a CIDR range of 10.0.0.0/16, capable of hosting up to 65,536 IPs.
Subnet Configuration: Divided the VPC into two subnets:
- PublicSubnet for internet-facing resources (10.0.0.0/24).
- PrivateSubnet for internal resources (10.0.1.0/24).
Internet Gateway Setup: Attached an Internet Gateway to our VPC to allow internet traffic for public resources.
Route Table Management:
- Configured a custom Route Table for the PublicSubnet with a route to the Internet Gateway for internet connectivity.
- Left the PrivateSubnet secured without a direct route to the internet.

Understanding the networking fundamentals is a game-changer when working with AWS. It transforms the process of launching an EC2 instance from a simple "trial and error" into a deliberate and well-architected solution. By mastering these principles, you're not just deploying applications—you’re building robust, scalable, and secure cloud architectures.

Let me know if you'd like to dive deeper into any of these concepts or explore additional AWS networking components like NAT Gateways, VPNs, or security best practices!

References

Reverse Proxy and Load Balancing: Do we need both?

jiisanda🙆‍♂️ — Thu, 10 Oct 2024 08:11:54 +0000

In this guide, I’ll walk you through configuring Nginx as both a reverse proxy and a load balancer, along with handling SSL termination for secure client-server communication. This setup can be used widely to enhance performance, scalability for websites and APIs.

Understanding Reverse Proxy and Load Balancing:

Though both terminologies sound similar, reverse proxy acts as an intermediate between clients and servers. It forwards client requests to servers and relays the server’s responses back to the clients.

While Load Balancer distributes incoming client requests across multiple backend servers, ensuring the load is spread evenly. It monitors server health to route traffic away from any server that becomes unavailable.

Do we need both ?

Yes, in some cases. Most production environments benefit from using both.

Load balancer helps to eliminate single point of failure, making API more reliable by allowing the API to be deployed in multiple backend servers, this intern improves user experience by reducing number of errors responses for clients, as it routes requests to healthy servers.

Load balancer also does health checks, which is when it sends separate health check requests, in frequent intervals and determines a server is healthy based on specified type of response like 200 or OK.

A reverse proxy hides the details of our backend servers from clients. It can also provide security against DDoS attacks, blacklist certain IP addresses, and prevent overloading.

SSL encryption and decryption is CPU intensive, offloading these tasks to a reverse proxy reduces the load on your backend servers and improves overall performance.

Configuring Nginx as a Reverse Proxy

Nginx provides multiple protocols for proxying requests, including HTTP, FastGCI, SGCI, and uwsgi. We will be using HTTP:

The proxy_pass directive allows Nginx to forward client requests to backend servers. For example,

location /api/app1 {
    proxy_pass http://backed_server;
}

In this configuration, a request to /api/app1/hello will be provided into http://backend_server/api/app1/hello. You can define multiple location blocks to handle different types of requests.

Setting up Nginx for Load Balancing

By using upstream directive load balancing can be set up. You can setup multiple servers in a backend pool and apply different algorithms to distribute client request

upstream backend_pool {
    server host1.example.com;
    server host2.exmaple.com;
    server host3.exmaple.com;
}

Here nginx forwards requests to any of the servers in the backend_pool. By default Nginx uses a round-robin method, distributing requests evenly across servers.

Upstream Balancing Algorithms

Round Robin (default) as above
Least Connections:

Sends request to server with the fewest active connections, which is useful when servers are slower than others.

upstream backend_pool {
    least_conn;
    server host1.example.com;
    server host2.example.com;
}

IP Hash

Ensures requests from the same client IP always go to the same server unless not available, important for session persistence. Here either the first three octets of IPv4 address or whole IPv6 address is used to calculate the hash value.

upstream backend_pool {
    ip_hash;
    server host1.example.com;
    server host2.example.com;
}

Generic Hash

To which server to send a request to is determined from a user-defined key, which can be text, string, variable or combination. Example, a key can be paired, source IP address and port or a URI.

upstream backend_pool {
    hash $request_uri consistent;
    server host1.example.com;
    server host2.example.com;
}

SSL Termination

To setup HTTPS in a config file or Nginx, to allow SSL encryption and decryption reducing the load on our backend servers.

server {
    listen 443 ssl;
    server_name www.example.com;

    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        ssl_ciphers         HIGH:!aNULL:!MD5;

    location /api/hello {
        proxy_pass http://backend_pool;
    }
}

This configuration listens on port 443 for HTTPS traffic, handles SSL encryption/decryption, and provides requests to backend_pool.

Note:

While Nginx does not provide support for advanced health check, it allows basic health probing using max_fails and fail_timeout parameters.

upstream backend_pool {
    server host1.example.com max_fails=2 fail_timeout=5s;
    server host2.example.com;
}

In this configuration, if host1 fails twice within a 5-second window, Nginx will stop routing requests to it until it recovers.

Conclusion

Now, we understand the load balancing and load balancing support in Nginx. To configure both, we can create a nginx.conf file in the /etc/nginx directory, and add the configurations as below.

events {}

http {
    # define the upstream for load balancing
    upstream backend {
        server host1.example.com:8080,
        server host2.example.com:8080 max_fails=2 fail_timeout=5s;
    }

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10ms;

    server {
        listen 443 ssl;
        server_name www.example.com;
        keepalive_timeout 70;

        ssl_certificate         /etc/nginx/ssl/server.crt;
            ssl_certificate_key     /etc/nginx/ssl/server.key;
        ssl_protocols           TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
                ssl_ciphers         HIGH:!aNULL:!MD5;

        location /service/api1 {
            proxy_pass http://backend;
        }

        location /service/api2 {
            proxy_pass http://backend;
        }
    }
}

UUID or ULID: Awesomeness of Unique Identifiers!

jiisanda🙆‍♂️ — Tue, 19 Mar 2024 15:51:13 +0000

Welcome! In this article we are going to have the showdown of two prominent choices of unique identifiers: UUID and ULID! 😲

In the landscape of software development, the task of generating unique identifiers has always been a crucial challenge. Whether it's managing database keys, tracking events in distributed systems, or ensuring session security, the choice of identifier can significantly impact the efficiency and performance of your application. In this showdown both parties present their strengths, factors that could make you choose one over the other, a glimpse of there implementation, and weaknesses. So sit back, get some popcorns 🍿, and get ready for a showdown that will empower you to make an informed decision for your next project.

Stage Setup

In software development, unique identifiers play a crucial role in ensuring data integrity, system scalability, and security. They act as unique markers for various entities within a system, like database records, distributed events, and user sessions.

Traditional auto-incrementing IDs, while simple, can become problematic at scale, leading to performance issues, collision risks, and data leakage.

if need explanation for any of the problems do comment happy to answer...

So, the two powerful alternatives in the world of unique identifiers: UUIDs and ULIDs!

UUIDs are the OGs of uniqueness, with the standardized format and 128-bit punch, these chads says, "I'm globally unique, baby!" 😎. As they boast their universal uniqueness across the cosmos💫!

But then comes ULIDs, the new kids who brings the whole new steps in the game! UILDs ain't just uniqueness, but also lexicographically sorted, baby! With the blend of timestamp sweetness and randomness, ULIDs slide into your codebase like smooth saxophone 🎷 on a summer night.

So what's the fuss, you ask? Well, UUIDs bring the tried-and-true reliability, perfect for when you need global uniqueness. But ULIDs? They are all about time based sorting, making them the go to identifiers where chronological order is the thing.

UUIDs (Universally Unique Identifiers)

UUIDs are different from sequential ids. RFC_4122 says,

UUIDs are of a fixed size (128 bits) which is reasonably small compared to other alternatives. This lends itself well to sorting, ordering, and hashing of all sorts, storing in databases, simple allocation, and ease of programming in general.

Layout and Byte order of UUID

timestamp

60 bit value
UUID1: represented by UTC (Since, 00:00:00.00 15th Oct 1582)
UUID3 and 5: timestamp (50 bits) is constructed from name (See Algo below)
UUID4: timestamp is randomly or pseudo-randomly generated. (see algo)

clock-sequence (14 bit)

UUID1: clock sequence is used to help avoid duplicates that arises when clock is set backwards in time (or) if the node ID changes.
UUID 3 and 5: 14-bit value constructed from a name (see Algo below)
UUID4: randomly or pseudo-randomly generated. (see algo)

node (48-bit)

UUID1: node field is an IEEE MAC address, usually host address.
UUID 3 and 5: Constructed from name (see algo)
UUID4: randomly or pseudo-randomly generated (see algo)

RFC_4122: Basic Algorithm sates an algorithm for generating UUIDs if they do not need to be generated frequently, but there were some issues. And hence different versions of UUIDs was implemented.

Let's look briefly at each and peek at it's Python implementation...

UUID1 (MAC Address + timestamp)

UUID1 concatenates the 48-bit MAC Address of the "node" (computer generating the UUID), with a 60-bit timestamp. The Python implementation is as follows:

def uuid1(node=None, clock_seq=None) -> UUID:
    """Generate a UUID from a host ID, sequence number, and the current time.
    If 'node' is not given, getnode() is used to obtain the hardware
    address.  If 'clock_seq' is given, it is used as the sequence number;
    otherwise a random 14-bit sequence number is chosen."""

    # some code

    time_low = timestamp & 0xffffffff
    time_mid = (timestamp >> 32) & 0xffff
    time_hi_version = (timestamp >> 48) & 0xfff
    clock_seq_low = clock_seq & 0xff
    clock_seq_hi_variant = (clock_seq >> 8) & 0x3f

    returns UUID(fields = (time_low, time_mid, time_hi_version,
                        clock_seq_hi_variant, clock_seq_low, node), version=1)

UUID 3 and 5

The version 3 and 5 are name-based UUIDs. For example, some name spaces are domain name system, URLs, or reserved words in programming languages. Some potential python specific name space ids are as follows:

NAMESPACE_DNS = UUID('6ba7b810-9dad-11d1-80b4-00c04fd430c8')
NAMESPACE_URL = UUID('6ba7b811-9dad-11d1-80b4-00c04fd430c8')
NAMESPACE_OID = UUID('6ba7b812-9dad-11d1-80b4-00c04fd430c8')
NAMESPACE_X500 = UUID('6ba7b814-9dad-11d1-80b4-00c04fd430c8')

def uuid3(namespace, name):
    """Generate a UUID from the MD5 hash of a namespace UUID and a name."""

    # some code    

    digest = md5(
        namespace.bytes + name,
        usedforsecurity=False
    ).digest()
    return UUID(bytes=digest[:16], version=3)

def uuid5(namespace, name):
    """Generate a UUID from the SHA-1 hash of a namespace UUID and a name."""

    # some code

    hash = sha1(namespace.bytes + name).digest()
    return UUID(bytes=hash[:16], version=5)

UUID4

The version 4 is meant for generating UUIDs from truly-random or pseudo-random numbers

def uuid4():
    """Generate a random UUID."""
    return UUID(bytes=os.urandom(16), version=4)

Now if none of bytes, fields is given then class UUID() will generate a TypeError saying one of the hex, bytes, fields, or int argument must be given.

This was all about UUIDs, if want to know more about UUID do look at RFC 4122...

ULIDs (Universally Unique Lexicographically Sortable Identifiers)

A ULID is 128 bit compatible with UUIDs, we can be generate 1.21e+24 unique ULIDs per second. These are as the name suggest lexicographically sortable. ULIDs are case sensitive, and no special character so URL safe.

Layout

In general the structure of a ULID is as follow

 01AN4Z07BY      79KA1307SR9X4MV3

|----------|    |----------------|
 Timestamp          Randomness
   48bits             80bits

timestamp

48 bit integer
UNIX-time in milliseconds
Won't run out of space 'til the year 10889 AD.

randomness

80 bits
Cryptographically secure source of randomness, if possible.

Sorting and Encoding and Montonocity

The left-most character must be sorted first, and the right-most character sorted last (lexical order). The default ASCII character set is used. For encoding Crockford's Base32 is used as shown below. This alphabet excludes I, L, O and U to avoid confusion and abuse.

0123456789ABCDEFGHJKMNPQRSTVWXYZ

While generating a UUID within same millisecond, it can provide some guarantees regarding some order. So if same millisecond is detected, the random component is incremented by 1 bit in the least significant bit position.

Usage

You usually would create a new :class:ULID object by calling the default constructor with no argument. In that case it will fill the timestamp part with the current datetime. And to encode the object it is usually converted to string.

You can create ULIDs, using different property passing as arguments. It can be generated using timestamp, or from uuid, from hex or byte, from string, or from datetime.

Advantages of ULID over UUIDs

shorter string representation (26 characters in ULIDs vs 36 in UUIDs)
Sortability for efficient ordering and retrieval.
Potential performance benefits in certain scenarios like
- In databases that uses the sorted indexes, ULIDs can potentially improve query performance because they leverage the existing sorting order of the index.
- When working with time-series data, ULIDs (which often includes a timestamp component) can be stored and retrieved in chronological order without additional sorting.

Choosing the Right Champion

Now that we've explored both UUIDs and ULIDs, let's help you pick the champion for your next project!

Here's is a quick comparison:

Feature	UUIDs	ULIDs
Uniqueness	Guaranteed	Guaranteed
Sortability	No	Yes (Lexicographically)
String Length	36 character	26 character
Performance	Generally Good	Potentially better with sorted indexes/time-series data

When to choose ULIDs

Sortability is essential: ULIDs excel when you need to efficiently sort or filter your identifiers.
Performance optimization matters: In scenarios with sorted indexes or time-series data, ULIDs can potentially offer performance benefits.
Compactness is desired: The shorter string length of ULIDs can be a space-saving advantage.

When to use UUIDs

Focus on guaranteed uniqueness: If the absolute certainty of no collision is paramount, UUIDs are the established choice.
Sorting isn't a priority: If order doesn't matter for your identifiers, UUIDs function perfectly well.

Ultimately, the best choice depends on your specific project requirements. Weigh the importance of uniqueness, sortability, performance, and string length to make an informed decision.

Example

Let's a look over how can you generate UUID and ULID in python...

# Generate a ULID
from ulid import ULID

ulid = ULID().generate()
print(f"ULID: {ulid}")  # Example output: 01HQCK8PK2T23Q13VVS03K47F9E

# Generate a UUID (version 4 - random)
import uuid

uuid = uuid.uuid4()
print(f"UUID: {uuid}")  # Example output: 123e4567-e89b-12d3-a456-426614174000

Conclusion

This article has explored the strength and weaknesses of two potential contenders in the unique identifier arena: UUIDs and ULIDs.

Key Takeaways

Both UUIDs and ULIDs guarantee uniqueness, a crucial aspect for data integrity and security.
UUIDs reigns supreme when prioritizing absolute uniqueness and don't require sorting capabilities.
ULIDs shine when sortability and potentially improved performance are key consideration, thanks to their lexicographically sorting and timestamp component.
Their compact string representation (26 characters) offers a space-saving advantage compared to UUIDs (36 characters)

Sidecar Pattern

jiisanda🙆‍♂️ — Sat, 28 Oct 2023 21:08:55 +0000

👋 Introduction

Have you ever played the game CS? 🎮 In CS, there are two teams: Terrorists and Counter-Terrorists, right? Let’s talk about the Terrorists team. In this team, there's a player with the bomb, while the rest of the players support the player with the bomb to plant them. It's like a well-coordinated mission where the main player has their trusted companion who assists them in planting it. 💣💥

Or perhaps you've heard about sidekicks? Yeah! Like the sidekicks of superheroes. 🦸‍♂️ Just as a superhero has their trusted companion who assists them in various tasks and enhances their capabilities, a sidecar is a helper container that runs alongside a main application container in a software environment. It provides additional services, resources, or functionality to the main application, much like a sidekick complements the abilities of a superhero, making it more effective in their mission. 🦸‍♀️🚀

The sidecar pattern is similar to teamwork in action! 🤝 It's a single-node pattern composed of two containers. 🚢 The first container is the application container 📦 – without it, the application wouldn't exist. Alongside the application container, there's a trusty sidecar container 🛠️. The role of the sidecar is to enhance and empower the application container, just like a sidekick bolsters a superhero! 🦸‍♂️💥

These application and sidecar containers work in harmony, scheduled on the same machine, which is referred to as a 'pod.' 🌟 They share a multitude of resources, a hostname, and a network. The general sidecar pattern can be depicted as follows:

📂🚗🏢 An Example Sidecar - Microservices

In many software applications and services, various peripheral functionalities such as monitoring, logging, configuration management, and networking services are required. These functionalities can be implemented as separate components or services.
Historically, developers would often integrate these services directly into the main application codebase, especially if they were written in the same programming language. This approach allowed for efficient code sharing and resource utilization, as these components could run in the same process as the main application. However, this tight integration also meant that these components were not well isolated, and any issues or outages in one component could potentially impact other components or even the entire application. Consequently, there was a high level of interdependence between these components and the core application.

The problem arises when these applications are built using different languages. For instance, Application 1 is built with Python, while the 2nd application is built using Rust. Each application will have its own implementations of caching, authentication, and logging.

One potential solution to this problem is to develop separate API services for these functionalities, as illustrated in the figure.

Does it have any problem?
When the application is decomposed into microservices, it becomes possible to employ different programming languages and technologies for individual services. While this approach provides greater flexibility, it comes with the drawback of each component having its own set of dependencies and the need for language-specific libraries to interact with the underlying platform and shared resources of the parent application. Furthermore, deploying these components as separate services can introduce latency into the application. The management of code and dependencies for these language-specific interfaces can also introduce significant complexity, particularly concerning aspects like hosting, deployment, and overall system management.

💡 Solution

We will use a sidecar pattern.

A sidecar service is not necessarily part of the application, but it is connected to it. It goes where the parent application goes. These are supporting processes or services that are deployed with the primary application. For each instance of the application, an instance of the sidecar is deployed and hosted alongside it.

🌟 Advantages of using a sidecar pattern:

The sidecar can access the same resources as the primary application. It can effectively monitor system resources utilized by both itself and parent applications. Resource Access.
The sidecar pattern is commonly employed with the containers and is often referred to as a sidecar container.
A sidecar can be used to enhance functionality by attaching it as a separate process in the same host.
A sidecar operates autonomously from the primary application, both in terms of the runtime environment and programming language, which eliminates the need to develop separate sidecar for each language.

🕐🤔 When can this pattern can be used:

When building a PaaS with Sidecars

Managing a dynamic configuration
When a component or feature must be co-located on the same host as the application.
A service that shares the overall lifecycle of your main application, but can be independently updated.
A component is owned by a different organization.

🚫❌ But not suitable when:

Communication between a parent application and sidecar services includes some overhead. i.e., where inter-process communication is important.
For small applications, the cost of deploying a sidecar service for each instance is not worth the isolation.
When the service needs to scale differently from the main applications. If so, it's better to deploy the feature as a separate service.

🔗 Further Reading and References

How does python-dotenv simplify Configuration Management?

jiisanda🙆‍♂️ — Fri, 21 Jul 2023 14:08:50 +0000

Navigating through the python projects, we can quickly realize that handling configurations can be like exploring mazes with numerous paths to choose from. In this article, we’ll embark on a thrilling journey and delve into the world of environment variables in Python, with particular focus on the realm of .env files.

First, let’s understand what configuration management is. I just finished a Netflix series, related to space travel, so picture this. You are a captain of your spaceship, commanding a crew of space travelers. Your spaceship is equipped with a range of cool gadgets and futuristic technologies, each with its own set of settings and values. The settings might include the ship’s speed, destination coordinates, communication frequencies, and more.

Now as you venture through the vastness of the cosmos, you encounter different planets, each with its unique challenges and environments. On some planets, you need to fly at a slower speed to avoid debris, while on others you might need to crank up the communication frequencies to establish contact with alien civilization.

In spaceship jargon, these settings are called configuration variables. And here’s where python-dotenv comes into play - it’s like having a magical map that stores all the settings separately, making your life as captain so much easier!

With python-dotenv, you can create a special treasure chest called .env, where you write down all the configuration variables for each planet. For example,

SPEED=10
DESTINATION_X=234
DESTINATION_Y=567
COMMUNICATION_FREQ=200

Now, whenever you approach a new planet, you consult your treasure chest and quickly update your spaceship’s settings without tinkering with the spaceship’s control panel directly. It’s like casting the spell to magically adjust the ship’s speed and communication frequencies for each destination!

The best part? Since the .env treasure chest is hidden safely in your spaceship’s storage room, you can keep all your top-secret settings secure from prying eyes - even from other spaceship crew who might be curious about your awesome adventures.

So with python-dotenv, you become the master of configuration management, effortlessly adapting to different planets and situations, all while keeping your spaceship’s secret safe and sound. 🙆‍♂️🚀✨

Okay Enough! Let’s jump into some technical stuff, understanding how to use python-dotenv and behind the scenes…

🤔 What is python-dotenv?

In docs, python-dotenv defines itself as

Python-dotenv reads key-value pairs from a .env file and can set them as environment variables. It helps in the development of applications following the 12-factor principles.

Let’s understand this first, key-value pairs are a way of organizing data, where each piece of data is associated with a unique key. 12-factor principles, are a set of best practices for building modern scalable, maintainable software applications. One of these principles involves storing configuration in the environment, which means using environment variables to manage configuration instead of hardcoding them into the code.

🏃‍♂️💨 Getting Started with python-dotenv

Install python-dotenv using

pip install python-dotenv

Make sure you activate your virtual environment, if not activated.
This will add python-dotenv to your application, to make it load the configuration from the .env file when it is present (eg., in development).

To configure the development environment, add a .env in the root directory of your project:

.
├── .env
└── main.py

The syntax for writing in the '.env' file for python-dotenv is similar to Bash.

# Spaceship Config
DOMAIN=dev.to
CAPTAIN_EMAIL=captain@${DOMAIN}
DESINATION_PLANET=${DOMAIN}/priplanus
SPEED=10
DESTINATION_X=234
DESTINATION_Y=567
COMMUNICATION_FREQ=200

Also make sure to add .env file to .gitignore, especially if it contains secrets like passwords.
How to load/use these environment variables in main.py file:

main.py

import os
from dotenv import load_dotenv

load_dotenv() # takes environment variables from .env

SPEED = os.getenv(“SPEED”) # or use os.environ(“SPEED”)

This is how you would be able to use environment variables in a Python Project. Easy isn’t it? Now let’s discuss what happens when you call load_dotenv().

🔌📥 load_dotenv()

Disclaimer: Source of all the data I am to provide is from GitHub of python-dotenv.

When you call load_dotenv() from the dotenv package, it can optionally accept an argument called dotenv_path: Optional[StrPath] or stream: Optional[IO[str]], which allows you to specify the path to a custom .env file for loading environment variables. If you don't provide the dotenv_path or stream, the function uses find_dotenv() to search for the .env file in increasingly higher folders within the project. Once it finds the file, it loads the environment variables from it.

The find_dotenv() function returns the path to the .env file if it is found, or an empty string if it doesn't find the file.

Now, we have the file path, so load_dotenv() calls the class DotEnv, which does the main part of loading all the variables found as environment variables.

So, DotEnv has a method called set_as_environment_variables, which loads the current dotenv as a system variable. Here is how it is implemented.

Once find_dotenv() locates the .env file, load_dotenv() proceeds to call the DotEnv class to perform the crucial task of loading all the variables found in the .env file as environment variables.

The DotEnv class contains a method named set_as_environment_variables, which handles the process of setting the values from the .env file as system-level environment variables. This means that after invoking this method, the variables defined in the .env file become accessible as environment variables throughout the entire Python application.

Here is how it is implemented:

01 def set_as_environment_variables(self) -> bool:
02         if not self.dict():
03             return False
04 
05         for k, v in self.dict().items():
06             if k in os.environ and not self.override:
07                 continue
08             if v is not None:
09                 os.environ[k] = v
10 
11         return True

Here, dict() is a method in the DotEnv class, which returns dotenv as a dictionary. After checking all the conditions, in line 09 of the above listing, the environment variables are set.

Here, self.overside: bool, is by default set to True. If in load_dotenv, you add an argument as load_dotenv(override=False), then, the already present environment variable will not be updated.

And this is how python-dotenv works, behind the scenes, for setting the environment variables. There are more features of python-dotenv like:

Load configuration without altering the environment
Parse configuration as a stream
Load .env files in IPython

Go through the docs and the GitHub, to see how these features work.

⌚👋👋 Outro

And now, armed with the power of python-dotenv, you, the spaceship captain, navigated the cosmos with ease, effortlessly adjusting to the unique challenges of each planet.

Happy coding, see you soon in the next article. 🚀✨🌌

Follow me on GitHub, LinkedIn, and Subscribe my YouTube.