DEV Community: Mohamad Ashraful Islam

AsyncIO Task Runner (Coro Runner): Simplifying Concurrent Python Task Management

Mohamad Ashraful Islam — Fri, 05 Dec 2025 05:36:19 +0000

Introduction

Managing asynchronous tasks in Python can be complex, especially when you need fine-grained control over concurrency limits and task scheduling. Enter Coro Runner (async-coro-runner), a lightweight Python utility that makes concurrent asynchronous task management straightforward and efficient.

In this post, we'll explore what makes Coro Runner unique, how to use it, and why it might be the perfect solution for your async workload management needs.

What is Coro Runner?

Coro Runner is a Python library built on top of Python's native asyncio module that provides a simple yet powerful interface for managing concurrent asynchronous tasks. It's designed to simplify the execution of multiple async tasks with configurable concurrency limits, all within a single-threaded environment.

The library is particularly useful when you need to:

Execute multiple async tasks concurrently with controlled parallelism
Manage task queues with different priorities
Avoid overwhelming resources with too many concurrent operations
Schedule tasks dynamically from anywhere in your application

Key Features

1. Configurable Concurrency

Define exactly how many tasks should run simultaneously, preventing resource exhaustion while maximizing throughput.

2. Simple API

Add tasks from anywhere in your codebase with a straightforward interface that doesn't require deep asyncio knowledge.

3. Worker Queue System

Multiple queues can be configured with their own priority levels, allowing sophisticated task management.

4. Built on asyncio

Leverages Python's battle-tested asyncio module (introduced in Python 3.4), ensuring stability and compatibility.

Installation

Getting started is as simple as:

pip install coro-runner

Quick Start Guide

Here's a basic example to get you started:

from coro_runner import CoroRunner

# Initialize the runner with a concurrency limit of 10
runner = CoroRunner(concurrency=10)

# Define an async task
async def process_data(item_id, **kwargs):
    # Your async processing logic here
    await some_async_operation(item_id)
    return f"Processed {item_id}"

# Add tasks from anywhere in your code
runner.add_task(process_data, args=[1], kwargs={"priority": "high"})
runner.add_task(process_data, args=[2], kwargs={"priority": "low"})
runner.add_task(process_data, args=[3], kwargs={"priority": "high"})

Important: Your task function must be an async function (defined with async def).

Real-World Use Case: FastAPI Integration

One of the most powerful applications of Coro Runner is in web applications. Here's an example using FastAPI to handle background tasks:

from fastapi import FastAPI
from coro_runner import CoroRunner

app = FastAPI()

# Initialize runner once at startup
runner = CoroRunner(concurrency=10)

async def background_task(task_id: int):
    # Simulate some async work
    await asyncio.sleep(1)
    print(f"Task {task_id} completed")

@app.get("/fire-task")
async def trigger_tasks(count: int = 25):
    # Schedule multiple tasks
    for i in range(count):
        runner.add_task(background_task, args=[i])

    return {"message": f"Scheduled {count} tasks"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

Why Choose Coro Runner?

Compared to Raw asyncio

While Python's asyncio is powerful, managing task concurrency often requires boilerplate code. Coro Runner abstracts this complexity:

Without Coro Runner:

# Managing concurrency manually with asyncio
semaphore = asyncio.Semaphore(10)

async def limited_task(task_func, *args):
    async with semaphore:
        return await task_func(*args)

# Need to manage the semaphore for every task...

With Coro Runner:

# Simple and clean
runner = CoroRunner(concurrency=10)
runner.add_task(your_task, args=[1, 2, 3])

Declare Once, Use Everywhere

A significant advantage is that you can initialize the runner once and call it from anywhere in your application:

# config.py
from coro_runner import CoroRunner
runner = CoroRunner(concurrency=20)

# module_a.py
from config import runner
runner.add_task(task_a, args=[1])

# module_b.py
from config import runner
runner.add_task(task_b, args=[2])

Advanced Features (Coming Soon)

The Coro Runner roadmap includes exciting enhancements:

Monitoring Tool Integration: Real-time task monitoring and analytics
Low-Level API: Advanced features like callbacks, acknowledgments, and error handling
Robust Logging: Detailed execution tracking for debugging and optimization

Use Cases

Coro Runner shines in scenarios such as:

Web Scraping: Fetch multiple URLs concurrently while respecting rate limits
Data Processing: Process large datasets with controlled parallelism
API Integration: Call multiple external APIs without overwhelming them
Background Jobs: Queue and execute background tasks in web applications
Batch Operations: Perform bulk operations with controlled concurrency

Example: Controlled Web Scraping

import aiohttp
from coro_runner import CoroRunner

runner = CoroRunner(concurrency=5)  # Only 5 concurrent requests

async def fetch_url(url: str):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()

# Schedule 100 URLs but only 5 will run at once
urls = [f"https://api.example.com/data/{i}" for i in range(100)]
for url in urls:
    runner.add_task(fetch_url, args=[url])

Development and Contribution

Coro Runner is actively developed and welcomes contributions. The project uses Uv for dependency management and includes:

Comprehensive test suite with pytest
GitHub Actions for CI/CD
Docker support for isolated testing
Example applications demonstrating real-world usage

To contribute:

# Clone the repository
git clone https://github.com/iashraful/async-coro-runner.git
cd async-coro-runner

# Install dependencies
uv sync

# Run tests
uv run pytest -s

Project Links

GitHub Repository: iashraful/async-coro-runner
PyPI Package: coro-runner
Documentation: Full Documentation
Author: Ashraful Islam

Requirements

Python 3.12 or later
uv (for development)

Conclusion

Coro Runner provides a clean, simple abstraction over asyncio's task management, making it easier to build high-performance concurrent applications in Python. Whether you're building web scrapers, API integrations, or background job processors, Coro Runner helps you manage concurrency without the complexity.

The library's philosophy is clear: provide powerful concurrency control with the simplest possible API. With its growing feature set and active development, Coro Runner is positioned to become an essential tool for Python developers working with asynchronous code.

Give it a try in your next project, and experience the simplicity of managed async task execution!

Getting Help

Have questions or running into issues? Here's how to get help:

Issues: GitHub Issues
Discussions: Engage with the community on GitHub
Documentation: Check the docs folder for detailed guides

Stay Updated

⭐ Star the repository on GitHub to stay updated with the latest features and improvements!

Have you used Coro Runner in your projects? Share your experience in the comments below!

Building a Rock-Solid Home Server Backup Strategy - My 3-2-1 Approach

Mohamad Ashraful Islam — Tue, 05 Aug 2025 17:24:55 +0000

When you’re running a home server that stores your personal files, photos, media, and even services like Pi-hole or Jellyfin, a proper backup strategy isn’t optional—it’s essential. Data loss can happen due to hardware failure, accidental deletion, or even corruption. That’s why I follow a proven and practical method known as the 3-2-1 backup strategy.

In this blog post, I’ll walk you through how I’ve implemented this strategy using a combination of external SSD storage and network-based backups over SSH.

Originally Posted on Ashraful's Blog

What is the 3-2-1 Backup Strategy?

The 3-2-1 rule is a simple yet powerful backup principle:

3 copies of your data (1 primary + 2 backups)
2 different types of media (e.g., SSD, HDD, cloud, etc.)
1 copy stored offsite

This strategy reduces the risk of total data loss by spreading your backups across multiple locations and types of storage.

1. Primary Storage: My Home Server

My home server is the heart of my digital setup. It hosts:

A media server (Jellyfin)
Personal files and documents
Photos and videos (via Immich)
Various self-hosted services (like Pi-hole, Portainer, etc.)

This is where the primary copy of all my data lives.

2. Local Backup: External SSD with Rsync Script

To maintain a local backup, I’ve connected an external SSD directly to my home server.

The SSD is mounted and automatically recognized on boot.
I back up everything important to this drive using a Bash script powered by rsync.
A cron job or systemd timer can automate the execution of this script regularly.

Here’s a sample of the SSD backup script:

#!/bin/bash

SOURCE="/data"
DEST="/mnt/backup_ssd"

echo "Starting backup to external SSD..."
mkdir -p "$DEST"
rsync -azP --delete "$SOURCE/" "$DEST/"
echo "Backup to SSD completed."

Here is the actual script of mine. Github Repo

This makes sure the SSD always has the latest version of my data and also removes deleted files for consistency.

3. Network Backup: Remote Sync Over SSH with Variables

To fulfill the offsite part of the 3-2-1 strategy, I’ve also implemented a remote backup over my local network.

Here’s how it works:

I use a custom Bash script that uses rsync over SSH.
It pulls variables like source, destination, remote user, and host from the script, making it easy to reuse or tweak.

Here’s a sample of the SSH-based backup script:

#!/bin/bash

SOURCE="/data"
REMOTE_USER="user"
REMOTE_HOST="192.168.1.10"
REMOTE_PATH="/mnt/backup/data"

echo "Starting remote network backup..."
rsync -azP -e ssh "$SOURCE/" "$REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH"
echo "Remote backup completed."

This ensures I’m keeping a secondary backup in a different physical location, even if it's just across the network.

Why This Setup Works for Me

Redundancy: With multiple local and remote backups, I have peace of mind even if one disk fails.
Speed: The SSD makes for quick restores, while SSH syncs avoid re-copying everything unnecessarily.
Privacy: Everything stays in my own network—no cloud involved.

Final Thoughts

Implementing a backup strategy might seem tedious at first, but trust me—it’s worth every bit of effort. Data loss can be painful and costly, especially when it comes to irreplaceable personal content.

By following the 3-2-1 strategy and using a mix of external SSDs and network-based backups, I’ve built a system that’s simple, reliable, and tailored to my setup. If you’re running a home server, I highly recommend investing time in your own backup plan—your future self will thank you.

Setting Up Pi-hole in Docker with Proper DNS Configuration

Mohamad Ashraful Islam — Tue, 05 Aug 2025 05:55:28 +0000

Originally Posted on Ashraful's Blog

This guide walks through

Installing Pi-hole with Docker Compose
Ensuring port 53 is available
Disabling the system's DNS resolver if necessary
Configuring your router to use Pi-hole
Setting up local DNS entries (if you need)

📦 Step 1: Install Pi-hole Using Docker Compose

Create a docker-compose.yml file:

version: "3"

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    restart: unless-stopped
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "8080:80"
      - "8443:443"
    environment:
      TZ: "Asia/Dhaka"
      WEBPASSWORD: "changeme"
    volumes:
      - "./etc-pihole/:/etc/pihole/"
      - "./etc-dnsmasq.d/:/etc/dnsmasq.d/"
    dns:
      - 127.0.0.1
      - 1.1.1.1
    cap_add:
      - NET_ADMIN

Then run:

docker compose up -d

🚪 Step 2: Make Sure Port 53 Is Available

Port 53 is critical for DNS. Run this to check if it's already in use:

sudo lsof -i :53

If you see something like systemd-resolved or named, you need to free that port.

🛑 Step 3: Disable the System DNS Resolver (If Needed)

On most Linux distros, systemd-resolved binds to port 53 by default.

To disable it:

sudo systemctl disable --now systemd-resolved

Also replace the symlink for /etc/resolv.conf:

sudo rm /etc/resolv.conf
echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf

(You can also add a fallback like 8.8.8.8 if needed.)

🌐 Step 4: Configure Your Router to Use Pi-hole

To apply ad blocking to your whole network:

Log into your router admin page
Find the DHCP/DNS settings
Set Primary DNS to the IP of your Pi-hole server (e.g., 192.168.1.10)
Remove or override any secondary DNS that bypasses Pi-hole (like 8.8.8.8)

Now all devices will query Pi-hole for DNS.

🧭 Step 5: Configure Local DNS (And Why)

✅ Why Configure Local DNS?

Setting up local DNS allows you to:

Access devices by name (nas.local, printer.lan, etc.)
Avoid typing IP addresses manually
Make services on your home server feel more like the cloud

🔧 How to Do It

Go to Pi-hole Admin Panel → Local DNS → DNS Records
Add entries like:

movie.ashraful.dev → 192.168.10.10
nas.lan → 192.168.10.20

Test from any device:

ping movie.ashraful.dev

If it resolves, you're all set!

✅ Summary

Pi-hole in Docker is clean and powerful — just make sure port 53 is free.
Disable systemd-resolved if it blocks port 53.
Set your router’s DNS to point to Pi-hole to enable network-wide filtering.
Use local DNS to make your network smarter and easier to use.

Enjoy ad-free browsing across your entire network! 🧠🛡️

Fixing DNS Resolution After Disabling systemd-resolved for Pi-hole

Mohamad Ashraful Islam — Tue, 05 Aug 2025 04:49:15 +0000

However, after doing this, I ran into a strange issue: some Python programs (specifically those using dnspython) could no longer resolve domain names. Here's how I diagnosed and fixed the problem.

Originally Posted on Ashraful's Blog

🔍 The Problem

To route DNS through Pi-hole, I disabled systemd-resolved:

sudo systemctl disable --now systemd-resolved

My home server then used Pi-hole's IP (e.g., 192.168.10.10) as its DNS server. Most things worked fine — ping, curl, and dig had no issues.

But some Python code using dnspython threw an error like:

dns.resolver.NoNameservers: All nameservers failed to answer the query

Or even:

FileNotFoundError: [Errno 2] No such file or directory: '/etc/resolv.conf'

🧠 What’s Going On?

Many programs — including dnspython — read DNS server information directly from /etc/resolv.conf.

When systemd-resolved is active, /etc/resolv.conf is often a symlink to a dynamically generated file like:

/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

But once systemd-resolved is disabled, this symlink points to a non-existent file. Programs depending on it will fail to resolve any domains.

✅ The Fix

We need to replace the broken symlink with a static resolv.conf file that directly specifies our Pi-hole DNS.

1. Remove the broken symlink

sudo rm /etc/resolv.conf

2. Create a new static file

sudo nano /etc/resolv.conf

Paste your Pi-hole DNS IP (replace with your actual IP):

nameserver 192.168.10.10

Optionally, add a fallback:

nameserver 192.168.10.10
nameserver 8.8.8.8

Save and close the file.

3. (Optional) Prevent overwrites

To ensure no other service modifies it (e.g., NetworkManager):

sudo chattr +i /etc/resolv.conf

To remove the protection later:

sudo chattr -i /etc/resolv.conf

🧪 Test DNS Resolution

Using nslookup:

nslookup google.com

Using Python:

import dns.resolver
result = dns.resolver.resolve("google.com")
print(result[0])

If everything is working, you'll get a valid IP response.

🧵 Summary

Because I'm using Pi-hole as my network guardian, I wanted to disable any system-level DNS services that could bypass it. Disabling systemd-resolved was necessary — but it broke /etc/resolv.conf, which some programs still depend on.

By creating a manual resolv.conf that points to Pi-hole, I ensured full DNS functionality while keeping all traffic filtered and protected.

Happy tinkering! 🛠️🧠🔒

The Untold Story of My Home Server - A Journey Through Self-Hosted Services

Mohamad Ashraful Islam — Sun, 03 Aug 2025 05:17:15 +0000

Originally Posted on Ashraful's Blog

Why a Home Server?

Before diving into the services, let’s talk about why I set up a home server. Cloud services like Netflix, Google Drive, or Evernote are convenient, but they come with recurring costs, privacy concerns, and dependency on someone else’s infrastructure. A home server lets me host my own media, back up my photos, manage my network, and even run custom apps—all on my terms. My setup runs on a (mini pc bought from Amazon) Ubuntu machine with Docker, making it easy to manage and scale. Here’s a look at the services that make it tick.

1. Immich: My Photo and Video Sanctuary

Immich is my go-to for backing up and managing photos and videos. Think Google Photos, but self-hosted, private, and free.

Why I Chose It: Immich offers automatic backups, face recognition, and a clean mobile app, all without sending my personal memories to the cloud. It’s perfect for keeping my family’s photos secure.
How It Works: I installed Immich via Docker, and it syncs photos and videos from my phone automatically over Wi-Fi. It organizes them by date, location, or people (using AI-based tagging), and I can access them from any device via a web interface or app.
My Experience: Immich has saved me from losing precious memories—like my kid’s first birthday video—while keeping them off Big Tech’s servers. The setup took some tweaking to get the AI features running smoothly, but it’s been worth it.

2. Jellyfin: My Personal Netflix

Jellyfin is the crown jewel of my home server, turning it into a media streaming powerhouse. This open-source media server lets me stream my collection of movies, TV shows, music, and even audiobooks to any device—my TV, phone, or laptop—without a subscription.

Why I Chose It: Unlike Plex, which has a freemium model with paid features, Jellyfin is 100% free and open-source. It gives me full control over my media library, with no cloud dependency.
How It Works: I store my media files (MP4s, MP3s, etc.) on the server’s hard drive. Jellyfin organizes them with metadata, album art, and subtitles, creating a sleek interface that rivals commercial streaming platforms. I access it via a web browser or Jellyfin’s apps on my devices.
My Experience: Setting up Jellyfin was a breeze with Docker. I love binge-watching my favorite shows on my Roku, knowing it’s all hosted at home. Plus, I can share my library with family without worrying about data leaks.

3. Cloudreve: My Personal Cloud Storage

Cloudreve is my Google Drive alternative, offering a self-hosted cloud storage solution for files, documents, and more.

Why I Chose It: Cloudreve is lightweight, supports multiple storage backends (like local drives or S3), and has a user-friendly interface. It’s perfect for accessing files from anywhere without relying on third-party cloud providers.
How It Works: I set up Cloudreve to store files on my server’s hard drive. It provides a web interface and WebDAV support, so I can upload, download, or share files securely. I also use it to back up important documents.
My Experience: Cloudreve has been a game-changer for sharing large files with friends or accessing work documents remotely. The setup was straightforward, though I had to configure SSL for secure access.

4. Pi-Hole: The Ad-Blocking Network Guardian

Pi-Hole is my network-wide ad and tracker blocker, making my internet experience faster and cleaner.

Why I Chose It: Pi-Hole blocks ads at the DNS level, meaning no pop-ups or video ads on any device connected to my network. It’s also a great way to monitor network activity.
How It Works: Running in a Docker container, Pi-Hole acts as my network’s DNS server. It filters out requests to known ad and tracker domains, replacing them with nothing. I pointed my router’s DNS settings to Pi-Hole, and it works seamlessly.
My Experience: The difference is night and day—web pages load faster, and YouTube is ad-free on my smart TV. The dashboard showing blocked domains is oddly satisfying to check.

5. Portainer: Keeping My Containers in Check

Portainer is my go-to tool for managing Docker containers, making it easy to monitor and control all the services on my server.

Why I Chose It: Portainer’s web-based interface simplifies Docker management, especially for someone like me who prefers a GUI over command-line tinkering.
How It Works: Portainer connects to my Docker environment and lets me start, stop, or update containers, check logs, and monitor resource usage. It’s like a control panel for my server’s services.
My Experience: Portainer has saved me countless hours of debugging. I can see at a glance if Jellyfin or Immich is acting up and restart containers with a click.

6. Nginx Proxy Manager: Simplifying Reverse Proxies

Nginx Proxy Manager makes it easy to manage reverse proxies, allowing secure access to my services with custom domains and SSL certificates.

Why I Chose It: Setting up reverse proxies manually with Nginx configs is a hassle. Nginx Proxy Manager’s graphical interface lets me configure proxies and SSL (via Let’s Encrypt) in minutes.
How It Works: I use it to route traffic to my services (e.g., jellyfin.mydomain.com to Jellyfin’s port). It handles SSL termination, ensuring secure connections.
My Experience: This was a lifesaver for accessing my services remotely without exposing ports to the internet. The setup was intuitive, and I love the clean dashboard.

7. Cockpit: Remote Management Made Easy

Cockpit is my tool for remotely managing my Ubuntu server, giving me a web-based interface to monitor and tweak system settings.

Why I Chose It: Cockpit is lightweight and integrates well with Ubuntu, offering a one-stop shop for system stats, updates, and user management.
How It Works: Running in a Docker container, Cockpit provides a dashboard to check CPU, memory, disk usage, and logs. I can also manage services or reboot the server remotely.
My Experience: Cockpit is my safety net when I’m away from home. I once fixed a stuck service while on vacation, all from my phone’s browser.

8. My Custom Note and Expense Tracking System: A Personal Touch

The final piece of my home server puzzle is a note and expense tracking system I built myself, tailored to my needs.

Why I Built It: I wanted a simple, private way to track daily notes and expenses without relying on apps like Notion or Mint, which store data in the cloud.
How It Works: Written in [insert language, e.g., Python with Flask or Node.js], it’s a web app running in a Docker container. It has a minimalist interface for adding notes (with tags and search) and logging expenses (with categories and monthly summaries). Data is stored in a local SQLite database.
My Experience: Building this was a labor of love. It’s not perfect, but it’s mine. I use it daily to jot down ideas or track spending, and it’s satisfying to know it’s fully under my control.

The Bigger Picture: Why This Matters

Running these services on my home server has been transformative. Jellyfin and Immich keep my media and memories safe and accessible. Cloudreve and my custom app handle my files and personal data without third-party snooping. Pi-Hole, Portainer, Nginx Proxy Manager, and Cockpit keep everything running smoothly and securely. Together, they’ve saved me money, boosted my privacy, and taught me valuable skills.

But beyond the tech, this setup is about empowerment. It’s about saying “no” to walled gardens and “yes” to owning your digital life. My server isn’t just a machine—it’s a statement that I can build, manage, and control my own corner of the internet.

Getting Started with Your Own Home Server

Inspired to start your own home server? Here’s a quick roadmap:

Hardware: Start small with a Raspberry Pi, an old PC, or a dedicated NAS.
OS: Use Ubuntu Server or a NAS-focused OS like TrueNAS for ease of use.
Docker: Learn Docker to simplify service deployment (Portainer makes this easier).
Services: Pick one or two services (like Jellyfin or Pi-Hole) and expand from there.
Community: Join communities like r/homelab or r/selfhosted on Reddit, or follow home server discussions on X for tips and inspiration.

Conclusion: The Heart of My Digital Home

My home server is more than a collection of services; it’s a reflection of my desire for independence, creativity, and control. From streaming movies with Jellyfin to tracking expenses with my custom app, every service tells a story of problem-solving and discovery. If you’re curious about home servers, I encourage you to dive in. Start small, experiment, and join the growing community of self-hosters. What’s your home server story? Let me know in the comments or on X!

Did you know docker-compose only takes environment variables from`.env` only?

Mohamad Ashraful Islam — Sun, 11 Jun 2023 05:25:45 +0000

Recently I have encountered an issue with docker compose. I have separated the environment files and declared todocker-compose.yaml like following,

    env_file:
      - .dev_env

When I ran docker compose up, it was not working on the compose itself but the container. because of environment variables. Then I found following solution for this.

docker compose --env-file .dev_env --env-file .prod_env up

Windows is not that bad for software development

Mohamad Ashraful Islam — Fri, 01 Jul 2022 17:34:06 +0000

Today it's not about virtualbox/vmware. It's a new Windows 10 feature. They called it WSL(Windows Subsystem for Linux)

Originally Posted on HERE

What is WSL?

Already told Windows Subsystem for Linux. Windows made sure to install Linux operating system inside the Windows environment without extra virtualization software(Virtual Box, VmWare, ...). So, How they did that?? That's burning question. Today I'll not go that way. But one thing I can tell that is, they slightly modified the kernel to fit with windows. Isn't it awesome?? YES! I am really exited to share about it.

What we are going to cover today?

Prerequisites
Enable WSL.
Install Ubuntu (22.04 Maybe).
Set WSL Resource.
Install Docker on Ubuntu.
VSCode WSL development.
Intellij IDE WSL development.

Prerequisites

You must be running Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11.

To check your Windows version and build number, select Windows logo key + R, type winver, select OK.

Enabling WSL

You can enable from GUI as well as from Powershell. I always prefer the hard way(command line). So, Run the following commands on Powershell.

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

Enabling the Virtualization Technology

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Now you need to restart the machine.

Install WSL Update

Downthe the update installer from here and Run the installer.

Set WSL2 as default WSL version.

wsl --set-default-version 2

Update existing distribution (if any)

wsl --set-version <distribution name> 2

Install Ubuntu (22.04)

There are several ways to install the OS in WSL. You can download from the Windows Store. This is the hassel free process. Once installed you will be able to login using your given password. Don't afraid, there is no UI. Only command line like Ubuntu Server :)

Now open Powershell again,

wsl -l -v

You will get something like following,

Set WSL Resource

Yes. You can still have chance to modify your resource allocation. Otherwise your windows will decide to do so. The process is pretty simple. You need to create a file to a specific location.

Open the file,

notepad "$env:USERPROFILE\.wslconfig"

Add these lines to the editor and save it.

[wsl2]
memory=4GB  
processors=4

You can read the microsoft's document for more configuration.

Now login to the ubuntu again, with Powershell/CMD whatever shell you have.

wsl

Install Docker on WSL(Ubuntu)

It's not mandatory. So, don't push yourself into Docker if your are not a Docker guy.

Installing Docker is same guide as ubuntu. You can follow the official Docker docs OR blindly follow me. Run the following commands on the Ubuntu's terminal not Powershell.

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

$ sudo apt update
$ sudo apt install \
   ca-certificates \
   curl \
   gnupg \
   lsb-release

Add Docker’s official GPG key:

$ sudo mkdir -p /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Use the following command to set up the repository

$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install Docker Engine

$ sudo apt update
$ sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Start the docker and check

$ sudo service docker start
$ sudo service docker status

Test your docker installation

$ sudo docker run hello-world

Output:

So, Docker is working.

Incase any Network issue from WSL

$ echo -e "[network]\ngenerateResolvConf = false" | sudo tee -a /etc/wsl.conf
$ sudo unlink /etc/resolv.conf
$ echo nameserver 1.1.1.1 | sudo tee /etc/resolv.conf

IPTables related issues

I found some people have iptables related errors. So I recommend the following command and choose legacy

$ update-alternatives --config iptables

Setting up VSCode

Actually there is nothing to setup. Trust me and do the following things,

You just need to install an extension. You can get it from here.
Create a project on WSL(ubuntu)
On the bottom left corner you will see remote development button.
Select New WSL Window
Open the project from WSL(Ubuntu)

Now it's all yours.

Setting up Intellij IDE (Pycharm in my case.)

Using Intellij IDE (Pycharm) codes stays on the windows machine.
Interpreter(Environment) will be just shared from WSL. The step by step guide is here.

Conclusion

I've talked too much today. But it made me to do so. But, Why I am writing this post?

In short, After 7-8 years I have been trying windows 10. I am so pleased to see the WSL and how it works. So, I thought people may like it too. Thank you.

FastAPI Streaming Response

Mohamad Ashraful Islam — Thu, 30 Jun 2022 05:24:31 +0000

What is a streaming response?

Streaming Response basically stream the data. So, how this happen?? Let's say you have a good amount of data. For example, 10MB text data. How you will send data through API? You might get timeout, other network issues for downloading the such a data from server. So, Streaming response come in first place to resolve the issue.

How it works?

It's really simple. Think how your downloader works, chunk by chunk. So, the Streaming response is. Your 10MB will be downloaded chunk by chunk. In the technical language we call it multipart.

Streaming Response in FastAPI

from typing import Generator
from starlette.responses import StreamingResponse
from fastapi import status, HTTPException

# A simple method to open the file and get the data
def get_data_from_file(file_path: str) -> Generator:
    with open(file=file_path, mode="rb") as file_like:
        yield file_like.read()

# Now response the API
async def get_image_file(path: str):
    try:
        file_contents = get_image_from_file(file_path=path)
        response = StreamingResponse(
            content=file_contents,
            status_code=status.HTTP_200_OK,
            media_type="text/html",
        )
        return response
    except FileNotFoundError:
        raise HTTPException(detail="File not found.", status_code=status.HTTP_404_NOT_FOUND)

You just use the function get_image_file and you'll get your desired Streaming response.

Why I am writing about it?

Because there is a twist about it. Starlette(The mother of FastAPI) encountered a bug about async generator. To working around the issue in FastAPI it created another issue. When you use sync generator for serving file or stream the response it because really slow.
Bug link here.

Moral of the story We should use the async generator for serving/streaming the API. Implementation is here,

from typing import Generator

# Just use the async function you already have. :)
async def get_data_from_file(file_path: str) -> Generator:
    with open(file=file_path, mode="rb") as file_like:
        yield file_like.read()

That's it. Now I am good to go.

Introduction to Scylla DB

Mohamad Ashraful Islam — Wed, 22 Dec 2021 07:24:17 +0000

Originally Posted on Ashraful's Blog

What is Scylla DB?

Scylla is a realtime NoSQL database, written into C++ without changing the low level API of Apache Cassandra. So, it's an drop-in replacement for Apache Cassandra.

Features

NoSQL Database
Open Source
Good Performance
Consistency
Availability
Scalability
Drop-in replacement for Apache Cassandra

Architecture

Scylla is a database that scales out and up. Scylla adopted much of its distributed scale-out design from the Apache Cassandra project (which adopted distribution concepts from Amazon Dynamo and data modeling concepts from Google BigTable).

Node

A node is the basic unit of organization for a Scylla database. It is comprised of the Scylla database server software running on a computer server.

Cluster

A Scylla cluster is consist of multiple nodes or Scylla instances. As we already know Scylla has built in flavor of distributed database system, So, Scylla visualized the nodes on a hash ring where multiple ring is connected to the hash.

Keyspace

A Scylla keyspace is a collection of tables with attributes that define how data is replicated on nodes. In general a keyspace is just like database in RDBMS.

Performance

According to Scylla's documentation, Scylla has much better performance than Cassandra. A full post based on performance. Here, Scylla vs Cassandra Performance Benchmark

Replication Factor

Scylla replicates data according to replication factor(RF). Basically number of RF is equal to the number of nodes in a cluster. But we can configure according to our need. For example, RF = 3 and number of nodes is 5. In this configuration scylla will write to two nodes per request out of three nodes.

Consistency Level

The Consistency Level (CL) determines how many replicas in a cluster must acknowledge a read or write operation before it is considered successful.

Some of the most common Consistency Levels used are:

ANY – A write must be written to at least one replica in the cluster. A read waits for a response from at least one replica. It provides the highest availability with the lowest consistency.
QUORUM – When a majority of the replicas respond, the request is honored. If RF=3, then 2 replicas respond. QUORUM can be calculated using the formula (n/2 +1) where n is the Replication Factor.
ONE – If one replica responds; the request is honored.
LOCAL_ONE – At least one replica in the local data center responds.
LOCAL_QUORUM – A quorum of replicas in the local datacenter responds.
EACH_QUORUM – (unsupported for reads) – A quorum of replicas in ALL datacenters must be written to.
ALL – A write must be written to all replicas in the cluster, a read waits for a response from all replicas. Provides the lowest availability with the highest consistency.

RF vs CL

Replication Factor(RF) and Consistency Level(CL) are too much related to performance and high availability. If CL is ALL then cluster is highly consistent but it might have unavailability. For example, if one node is down then the acknowledgement will not received so, it'll fail. Because we already configured to CL = ALL.

The following image has RF=3 and CL=1. It means during read/write servers will replicate to 3 nodes but only one acknowledgement is required to pass the request. So, it's highly available. If more than one nodes down data will not fail.

Conclusion

To summarize, these are the main points we covered:

Scylla has a ring-type architecture
It’s a distributed, highly available, high performance, low maintenance, highly scalable NoSQL database
In Scylla all nodes are created equal, there are no master/slave nodes
Data is automatically distributed and replicated on the cluster according to the replication strategy
Scylla supports multiple data centers

Python Unit Testing

Mohamad Ashraful Islam — Sun, 13 Jun 2021 16:47:49 +0000

Originally Posted on Ashraful's Blog

What is Testing❓

Testing is basically checking the features are okay or not and finding bugs on the system. Basically, there are many types of testing we do with software. Today we will talk about the most famous unit testing process. Let's keep going.

What is a unit testing❓

A unit test is a way of testing a unit - the smallest piece of code that can be logically isolated in a system. In most programming languages, that is a function, a subroutine, a method or property.

Python's builtin unittest 💥

Let's try some functions and their unit tests

# test_add.py
# A very basic function for adding two numbers
def add(a: int, b: int) -> int:
    return a + b

# Writing Unit Test
import unittest

class TryingTheAwesomeUnitTest(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(5, 7), 12)

if __name__ == '__main__':
    unittest.main()

Save the file as test_add.py and run the file python3 test_add.py and see the following output.

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Introduction to PyTest 🚀

pytest: helps you write better programs

Pytest is another testing library for Python. Let's dig into it.

Installation💡

pip install pytest

# test_2_add.py

# The same old function
def add(a: int, b: int) -> int:
    return a + b

def test_add():
    assert add(3, 5) == 9 # I want see the fail response

Run the tests🐛

Just type pytest on the directory where you have saved the file.

$ pytest

Result🙈

============================================ test session starts =============================================
platform darwin -- Python 3.8.1, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/ashraful/Public/scripts
collected 1 item

test_2_add.py F                                                                                        [100%]

================================================== FAILURES ==================================================
__________________________________________________ test_add __________________________________________________

    def test_add():
>       assert add(3, 5) == 9 # I want see the fail response
E       assert 8 == 9
E        +  where 8 = add(3, 5)

test_2_add.py:8: AssertionError
========================================== short test summary info ===========================================
FAILED test_2_add.py::test_add - assert 8 == 9
============================================= 1 failed in 0.04s ==============================================

Tips: Don't forget to put the filename test_ as prefix otherwise pytest can't detect the file. Whatever test_ as prefix is mandatory convention for test case.

Introduction to API Gateway

Mohamad Ashraful Islam — Sun, 07 Feb 2021 10:09:25 +0000

Originally posted on Ashraful's Blog

💡 What is an API Gateway❓

An API gateway is an API management tool that sits between a client and a collection of backend services.

An API gateway acts as a reverse proxy to accept all application programming interface (API) calls, aggregate the various services required to fulfill them, and return the appropriate result.
In short: An API Gateway is the abstraction layer between client and microservices.

🔥 When to use an API Gateway❓

Let's imagine that you are building a native mobile application for android and ios. In that case, you have two teams to develop the apps. On the backend side, you have separate services like Auth, Product, Cart, Order, etc. And you perfectly deploy the services. Let's assume the following IPs are assigned to services.

Auth     192.168.1.10
Product  192.168.1.11
Cart     192.168.1.12
Order    192.168.1.13

Now you tell your app team to call API according to the services' IP/Domain. Okay Good. Your business is good.

After some days you realized to separate the Order and Payment service. Your backend team finished the job. And now the painful part is releasing the app(Android, iOS) and make sure that everyone gets the update. Both teams need to work just to change the API URL. Not only that, problems like this can arise at any time. So, here is a simple solution is, "API Gateway".

🎉 How an API Gateway is solving your problem❓

We just talk about the problem. Now let's talk about the solution. We just need to check the request URI and pass the request to the respected service. For example:

# Something like this.
if request_uri == '/api/login/':
   pass_the_request(auth_service)

There are lots of API gateways that are already written for multiple purposes. For a small project, I'd like to use Nginx's reverse proxy. We'll talk about it later.

Horizontal vs Vertical Scaling

Mohamad Ashraful Islam — Tue, 22 Sep 2020 14:45:31 +0000

💡 Idea Behind the Scaling

Suppose I have a business application and I want to access it from anywhere over the internet, so I can make money. So, how I supposed to do that? I know you are brilliant and you have the answer. Yes!! I will rent a machine from a cloud provider and host it traditionally.
After some days my app becomes very popular and so many people are using it. Now the real problem begins. People are facing downtime and I am losing customers 🤦. So, I asked one of my friends and he told me to scale up the machine to solve the problem.

💥 What is Scalability?

Scalability is able to handle more requests with a bigger machine or adding more machines. There are two types of scaling,

Vertical Scaling
Horizontal Scaling

Vertical Scaling
In one sentence, Buying a bigger machine called vertical scaling. I mean a single machine serves all the requests.

Horizontal Scaling
Adding more than one machine is called horizontal scaling. I mean you have already a machine and you add more machines with it to serve the requests.

🍻 Pros and Cons

Horizontal	Vertical
1. Load balancer required	1. N/A
2. Call over the network (Little Slower)	2. Interprocess communication (Faster)
3. No problem until all the servers crash	3. Single point of failure
4. Data Inconsistency	4. Consistent
5. Nicely scale up according to need	5. Hardware limits at some point.

🚀 Conclusion

So, now the big question is Which one should we use ❓ My answer is Both. This actually depends on the system you are designing. Both the scaling system have some good sides. You just need to think about how much big a system you are going to build and how many requests you need to serve in each second and is it consistent to use this model. That it. You'll have your answer.