DEV Community: Dhananjay Haridas

What is GitHub CLI?

Dhananjay Haridas — Wed, 03 Jun 2026 12:15:29 +0000

Introduction

Most developers interact with GitHub through a web browser. We open GitHub, create repositories, review pull requests, manage issues, and create releases through the website.

But what if you could perform all those actions directly from your terminal?

That is exactly what GitHub CLI, commonly known as gh, allows you to do.

GitHub CLI is an official command-line tool developed by GitHub that enables developers to work with GitHub without constantly switching between their code editor, terminal, and browser.

By the end of this guide, you will understand:

What GitHub CLI is
Why it exists
How to install it
How authentication works
Common commands every developer should know
Real-world workflows
Best practices and tips

What is GitHub CLI?

GitHub CLI is a command-line application that lets you interact with GitHub from your terminal.

Instead of visiting:

https://github.com

and clicking through menus, you can perform GitHub operations using commands.

For example:

gh repo create

creates a repository.

gh pr create

creates a pull request.

gh issue create

creates an issue.

gh release create

creates a release.

Why Was GitHub CLI Created?

Before GitHub CLI existed, developers typically had three options:

Option 1: Use the GitHub Website

Pros:

Easy to understand
Visual interface

Cons:

Constant context switching
Slower for repetitive tasks

Option 2: Use Git Commands

Git commands help manage source control:

git commit
git push
git pull
git branch

However, Git itself does not manage:

Pull Requests
Issues
Releases
Discussions

These are GitHub features.

Option 3: Use GitHub APIs

GitHub provides APIs.

Example:

curl \
  -H "Authorization: Bearer TOKEN" \
  https://api.github.com/repos/user/project/issues

This works but can become complex.

GitHub CLI bridges this gap.

It provides a simple developer-friendly interface over GitHub APIs.

Git vs GitHub CLI

Many beginners confuse Git and GitHub CLI.

Git	GitHub CLI
Version control system	GitHub management tool
Works locally	Communicates with GitHub
`git push`	`gh pr create`
`git branch`	`gh issue create`
`git merge`	`gh release create`

Think of it like this:

Git = Manage code history

GitHub CLI = Manage GitHub platform features

Installing GitHub CLI

Ubuntu

sudo apt update
sudo apt install gh

Verify:

gh --version

Example output:

gh version 2.70.0

macOS

Using Homebrew:

brew install gh

Windows

Using Winget:

winget install GitHub.cli

First-Time Authentication

After installation:

gh auth login

You will see options such as:

? What account do you want to log into?
GitHub.com

Choose:

GitHub.com

Then:

HTTPS

Then:

Login with browser

GitHub CLI will generate a code.

Example:

First copy your one-time code:
ABCD-EFGH

Open the browser and authenticate.

Check Authentication Status

gh auth status

Example:

Logged in to github.com

Creating a Repository

Create a repository directly from the terminal.

gh repo create my-project

Interactive mode will ask:

Public or Private?

Choose your preference.

Cloning a Repository

Traditional Git:

git clone https://github.com/user/project.git

GitHub CLI:

gh repo clone user/project

Example:

gh repo clone octocat/demo

Viewing Repository Information

gh repo view

Example output:

Repository:
user/project

Description:
My project

Stars:
120

Creating Issues

Issues are used to track bugs and feature requests.

Create one:

gh issue create

Interactive prompts appear.

You can also provide everything directly:

gh issue create \
  --title "Login Bug" \
  --body "Users cannot log in"

Listing Issues

gh issue list

Example:

#23 Login Bug
#24 Add Dark Mode
#25 Improve Performance

Viewing a Specific Issue

gh issue view 23

Creating Pull Requests

One of the most powerful features.

After pushing a branch:

git push origin feature/login

Create a PR:

gh pr create

GitHub CLI asks:

Title?
Description?
Base branch?

Then creates the pull request.

Listing Pull Requests

gh pr list

Example:

#45 Add Login Feature
#46 Fix Database Error

Viewing Pull Requests

gh pr view 45

Checking Out a Pull Request

Instead of manually fetching branches:

gh pr checkout 45

This automatically switches to the PR branch.

Reviewing Pull Requests

Approve:

gh pr review 45 --approve

Request changes:

gh pr review 45 --request-changes

Comment:

gh pr review 45 --comment

Merging Pull Requests

gh pr merge 45

Options:

gh pr merge 45 --merge

gh pr merge 45 --squash

gh pr merge 45 --rebase

Working with Releases

A release is a published version of software.

Example:

v1.0.0
v1.1.0
v2.0.0

Create a Release

gh release create v1.0.0

Add Release Notes

gh release create v1.0.0 \
  --notes "Initial release"

Upload Files

gh release create v1.0.0 \
  dist/app.tar.gz

This is likely what happened in your project.

Your Makefile probably executes something similar to:

gh release create \
  v1.0.0 \
  dist/registry-data-v1.0.0.tar.gz

Without GitHub CLI installed, that step fails.

Viewing Releases

gh release list

Example:

v1.0.0
v1.1.0
v2.0.0

Download Release Assets

gh release download v1.0.0

Working with GitHub Actions

View workflows:

gh workflow list

View workflow runs:

gh run list

Inspect a run:

gh run view

Watch live logs:

gh run watch

Working with Gists

Create a gist:

gh gist create notes.txt

List gists:

gh gist list

Useful Commands Every Developer Should Know

Authentication

gh auth login
gh auth status
gh auth logout

Repositories

gh repo create
gh repo clone
gh repo view

Issues

gh issue create
gh issue list
gh issue view

Pull Requests

gh pr create
gh pr list
gh pr view
gh pr checkout
gh pr merge

Releases

gh release create
gh release list
gh release view

Real-World Example Workflow

Imagine you are implementing a feature.

Step 1

Create a branch:

git checkout -b feature/payment

Step 2

Write code.

Step 3

Commit changes:

git add .
git commit -m "Add payment support"

Step 4

Push:

git push origin feature/payment

Step 5

Create PR:

gh pr create

Step 6

Review PR:

gh pr view

Step 7

Merge:

gh pr merge

Step 8

Create Release:

gh release create v1.0.0

Entire workflow completed without opening GitHub in a browser.

Advantages of GitHub CLI

Faster Workflow

No browser switching.

Automation Friendly

Perfect for scripts and CI/CD pipelines.

Consistent Interface

Everything from one terminal.

Developer Productivity

Reduces repetitive work.

Official Tool

Maintained by GitHub.

When Should You Use GitHub CLI?

Use GitHub CLI if you:

Work with GitHub daily
Create pull requests frequently
Manage releases
Handle issues regularly
Build automation scripts
Work in DevOps or platform engineering

Common Beginner Mistakes

Forgetting Authentication

gh auth login

must be completed first.

Confusing Git with GitHub CLI

Remember:

Git manages code history.

GitHub CLI manages GitHub features.

Missing Permissions

Your GitHub account must have permissions for the repository.

Not Updating GitHub CLI

Keep it updated:

sudo apt update
sudo apt upgrade gh

Summary

GitHub CLI (gh) is GitHub's official command-line tool that allows developers to manage repositories, issues, pull requests, releases, workflows, and many other GitHub features directly from the terminal.

For a junior developer, learning GitHub CLI provides several benefits:

Faster development workflows
Less dependence on the browser
Easier automation
Better understanding of GitHub operations
Improved productivity in professional software teams

If you're already comfortable with commands like:

git add
git commit
git push

then learning:

gh pr create
gh issue create
gh release create

is the next natural step toward becoming a more efficient developer.

Fuzzy Text Search in PostgreSQL: How Trigrams Make Computers "Almost Right"

Dhananjay Haridas — Wed, 03 Jun 2026 08:39:43 +0000

A deep dive into one of the most clever tricks in database engineering - explained so simply, your grandmother could follow along.

The Problem: Computers Are Too Exact

Imagine you're searching for a restaurant called "Maharaja Palace" on an app. You type "Maharja Palce" - two small typos. A traditional search engine looks at your input and says:

"No results found."

Zero. Nothing. Because the text doesn't exactly match.

This is the fundamental problem with how computers search for text. They compare strings character by character, and if even one letter is out of place, the match fails. Humans, on the other hand, are remarkably good at understanding that "Maharja" probably means "Maharaja". We do it instinctively, without even thinking.

Fuzzy search is the art of teaching computers to think a little more like humans - to find things that are close enough, not just exactly right.

And in PostgreSQL, the most powerful tool for this is called Trigram Search, powered by an extension called pg_trgm.

Part 1: What Is a Trigram? (The Building Block of Everything)

Before we dive into how PostgreSQL searches, we need to understand one simple concept: the trigram.

A trigram is just a group of 3 consecutive characters from a word.

That's it. Three letters in a row.

Let's break the word "cat" into trigrams:

Word: "cat"
PostgreSQL pads the word with spaces for edge handling: "  cat  "

Trigrams:
  "  c"
  " ca"
  "cat"
  "at "
  "t  "

💡 PostgreSQL secretly adds two spaces before and one space after every word. This is to give the beginning and end of words their own identity, so "cat" and "concatenate" don't blur together.

Let's try a longer word: "elephant"

Word: "  elephant "

Trigrams:
  "  e"
  " el"
  "ele"
  "lep"
  "eph"
  "pha"
  "han"
  "ant"
  "nt "

Every word gets chopped into these little 3-character windows, sliding one character at a time across the word from left to right.

Think of it like a magnifying glass that's 3 characters wide, slowly sliding across a word and taking a snapshot at every position.

Part 2: Why Trigrams? Why Not 2 or 4 Letters?

This is a fair question. Why exactly three?

Bigrams (2 letters): Too short. The bigram "ca" appears in "cat", "car", "can", "castle", "scar", "occasion"... it's so common it doesn't carry much meaning. Matching would produce too many false positives.
4-grams or longer: Too specific. A 4-gram like "elep" only really appears in "elephant" and close relatives. If you make a typo in that 4-gram, you lose the match entirely. Too brittle.
Trigrams (3 letters): The sweet spot. Common enough to appear across related words, specific enough to carry meaning. A single typo only destroys a few trigrams, leaving most of the others intact to still find a match.

It's the Goldilocks zone of string matching.

Part 3: The Magic - How Similarity Is Calculated

Here's where the real cleverness lives.

Once every word is broken into trigrams, PostgreSQL measures how similar two words are by counting how many trigrams they share.

The formula is:

Similarity = (Number of shared trigrams) / (Total unique trigrams across both words)

This gives a number between 0 (completely different) and 1 (identical).

Let's work through a real example

Say we're comparing "night" and "knight".

Step 1: Generate trigrams for "night"

"  n", " ni", "nig", "igh", "ght", "ht "

→ 6 trigrams

Step 2: Generate trigrams for "knight"

"  k", " kn", "kni", "nig", "igh", "ght", "ht "

→ 7 trigrams

Step 3: Find the shared trigrams

Shared: "nig", "igh", "ght", "ht "

→ 4 shared trigrams

Step 4: Calculate similarity

Total unique trigrams = 6 + 7 - 4 = 9   (subtract shared ones to avoid counting them twice)
Shared = 4

Similarity = 4 / 9 = 0.44

So "night" and "knight" are about 44% similar according to trigrams. That seems low - they sound the same! But trigrams are measuring character patterns, not pronunciation. The extra "kn" at the start of "knight" introduces several new unique trigrams that "night" doesn't have.

Now let's try a typo: "niight" vs "night"

Trigrams for "niight":

"  n", " ni", "nii", "iig", "igh", "ght", "ht "

Trigrams for "night":

"  n", " ni", "nig", "igh", "ght", "ht "

Shared: " n", " ni", "igh", "ght", "ht " → 5 shared

Total unique: 7 + 6 - 5 = 8

Similarity = 5/8 = 0.625

Despite the typo ("niight"), the similarity is still 62.5% - high enough that a search would still find the match. This is the magic of trigrams.

Part 4: The Threshold - Deciding What "Close Enough" Means

Now that we can calculate similarity as a number, we need to decide: how similar is similar enough?

PostgreSQL uses a default threshold of 0.3 (30%). If two strings share at least 30% of their trigrams, they're considered a match.

You can think of it like a teacher grading papers:

Score ≥ 0.3 → Pass (it's a match)
Score < 0.3 → Fail (not a match)

You can adjust this threshold depending on how strict or lenient you want your search to be:

-- Make it stricter (only very close matches)
SET pg_trgm.similarity_threshold = 0.5;

-- Make it more lenient (catch more distant matches, but more noise)
SET pg_trgm.similarity_threshold = 0.2;

Practical guidance:
| Threshold | Best For |
|-----------|----------|
| 0.1 – 0.2 | Very forgiving search, names from other languages |
| 0.3 | General purpose (default, works well most of the time) |
| 0.5 | Names, product codes where near-exact match is needed |
| 0.7+ | Almost exact matching only |

Part 5: Under the Hood - How PostgreSQL Actually Executes the Search

Understanding the math is one thing. But how does PostgreSQL actually run this efficiently when you have millions of rows? Recalculating trigrams for every row on every query would be painfully slow.

The answer is: the GIN Index.

What is a GIN Index?

GIN stands for Generalized Inverted Index. The name sounds intimidating, but the concept is simple.

Think of it like the index at the back of a book. Instead of reading every page to find where "photosynthesis" is mentioned, you flip to the back, find "photosynthesis" alphabetically, and it tells you exactly which pages contain it.

A GIN index does the same thing for trigrams:

GIN Index Structure (simplified):

" ca" → [row 5, row 12, row 89, row 203]
" co" → [row 3, row 7, row 44]
"cat" → [row 5, row 89]
"cof" → [row 3]
"ofe" → [row 3]
...

For every possible trigram, the index stores a list of which rows contain that trigram. When you search for "coffee", PostgreSQL:

Breaks "coffee" into its trigrams: " co", "cof", "off", "ffe", "fee", "ee "
Looks up each trigram in the GIN index
Finds rows that contain several of these trigrams (intersection)
Calculates the exact similarity score only for those candidate rows
Returns results above the threshold

Instead of scanning millions of rows, it scans a short list of candidates. This is what makes trigram search fast - often hundreds of times faster than a brute-force scan.

Visualizing the Query Flow

Your Query: "SELECT * WHERE name % 'cofee'"
                    │
                    ▼
         Break 'cofee' into trigrams
         [" co", "cof", "ofe", "fee", "ee "]
                    │
                    ▼
         Look up each trigram in GIN index
         → Candidate rows: [3, 7, 44, 89, 102]
                    │
                    ▼
         Calculate similarity() for each candidate
         → Row 3: 0.71 ✓  (above threshold)
         → Row 7: 0.18 ✗
         → Row 44: 0.65 ✓
         → Row 89: 0.22 ✗
         → Row 102: 0.80 ✓
                    │
                    ▼
         Return rows 3, 44, 102 - ordered by score

Without the GIN index, step 2 would require checking every row. With it, only a handful of candidates get the expensive similarity calculation.

Part 6: Setting It Up in PostgreSQL

Enough theory - here's how you actually use it.

Step 1: Enable the Extension

CREATE EXTENSION IF NOT EXISTS pg_trgm;

This is a one-time setup per database. The extension comes bundled with PostgreSQL - no installation needed.

Step 2: Create Your Table and Index

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL
);

-- Create a GIN index on the name column
CREATE INDEX idx_products_name_trgm
    ON products
    USING GIN (name gin_trgm_ops);

The gin_trgm_ops part tells PostgreSQL: "use trigram logic for this index".

Step 3: Query Using Similarity

-- The % operator: returns rows above the similarity threshold
SELECT * FROM products
WHERE name % 'cofee maker';

-- The similarity() function: returns the score, lets you sort by closeness
SELECT name, similarity(name, 'cofee maker') AS score
FROM products
WHERE name % 'cofee maker'
ORDER BY score DESC;

Sample output:

name               | score
-------------------+-------
Coffee Maker Pro   | 0.82
Coffee Maker Mini  | 0.78
Coffee Mug Set     | 0.31

Step 4: Tune the Threshold (Optional)

-- See the current threshold
SHOW pg_trgm.similarity_threshold;

-- Change it for this session
SET pg_trgm.similarity_threshold = 0.4;

Part 7: Real-World Use Cases

1. Search Autocomplete / "Did you mean?"

-- User typed "postgress" - find the closest match
SELECT name, similarity(name, 'postgress') AS score
FROM technologies
WHERE name % 'postgress'
ORDER BY score DESC
LIMIT 5;

2. Deduplication - Finding Near-Duplicate Records

-- Find customer records that might be the same person
SELECT a.name, b.name, similarity(a.name, b.name) AS score
FROM customers a
JOIN customers b ON a.id < b.id
WHERE similarity(a.name, b.name) > 0.7
ORDER BY score DESC;

3. Product Search with Typo Tolerance

-- Search even with misspellings
SELECT id, name, price,
       similarity(name, $1) AS relevance
FROM products
WHERE name % $1
   OR description % $1
ORDER BY relevance DESC
LIMIT 20;

4. Combining with Full-Text Search for Best Results

-- Use full-text search for speed + trigrams for typo tolerance
SELECT name, ts_rank(tsv, query) AS text_rank,
       similarity(name, 'databse') AS fuzzy_rank
FROM articles,
     plainto_tsquery('english', 'database') query
WHERE tsv @@ query
   OR name % 'databse'
ORDER BY (ts_rank(tsv, query) + similarity(name, 'databse')) DESC;

Part 8: The `word_similarity` and `strict_word_similarity` Functions

PostgreSQL's pg_trgm also provides two variants of the similarity function that handle multi-word strings differently.

Function	What It Does
`similarity(a, b)`	Compares the full strings as a whole
`word_similarity(a, b)`	Checks if `a` is similar to any word in `b`
`strict_word_similarity(a, b)`	Like `word_similarity` but requires whole-word boundaries

Example:

SELECT similarity('word', 'two words');
-- Returns: 0.18 (compares full strings)

SELECT word_similarity('word', 'two words');
-- Returns: 0.8 (finds that 'word' closely matches 'words')

This is invaluable when users search for a single product word inside a longer product name.

Part 9: Performance Tips

Always Create the GIN Index Before Querying Large Tables

Without the index, PostgreSQL does a full sequential scan - checking every row. On a table with 10 million rows, this can take minutes. With the index, the same query can complete in milliseconds.

-- Check if your query is using the index
EXPLAIN ANALYZE
SELECT * FROM products WHERE name % 'laptop';

Look for Index Scan using idx_products_name_trgm in the output. If you see Seq Scan instead, your index isn't being used - check that it was created correctly.

Index Maintenance

GIN indexes can become bloated over time with many inserts and updates. Periodically run:

VACUUM ANALYZE products;

Consider a `GIST` Index for Certain Workloads

-- Alternative index type
CREATE INDEX idx_products_name_gist
    ON products
    USING GIST (name gist_trgm_ops);

Index Type	Build Speed	Query Speed	Update Speed	Choose When
GIN	Slower	Faster	Slower	Read-heavy workloads
GiST	Faster	Slower	Faster	Write-heavy workloads

Part 10: Limitations - When Trigrams Don't Work Well

Trigrams are powerful but not magic. They struggle in a few situations:

Short Strings (Less Than 3 Characters)

A 1 or 2 character string can't form a meaningful set of trigrams. Searching for "AI" or "DB" will produce poor results.

-- Workaround: Fall back to ILIKE for very short queries
SELECT * FROM tags
WHERE CASE
    WHEN length('AI') < 3 THEN name ILIKE '%AI%'
    ELSE name % 'AI'
END;

Very Different Languages

Trigrams work on character patterns. For languages with different scripts (Arabic, Chinese, Thai), the trigram approach still works but may need different tokenization strategies.

Phonetic Similarity

Trigrams match visual similarity, not sound similarity. "Night" and "knight" sound identical but have low trigram similarity because they look different. For phonetic matching, consider the fuzzystrmatch extension with soundex or metaphone.

CREATE EXTENSION fuzzystrmatch;

SELECT * FROM names
WHERE metaphone(name, 10) = metaphone('knight', 10);
-- This would match "night" and "knight"

Summary: The Trigram Story in One Paragraph

Every word in your database gets sliced into overlapping 3-character chunks called trigrams. These trigrams are stored in a GIN index - like a giant lookup table mapping each trigram to the rows containing it. When you search for a word, PostgreSQL slices that word into trigrams too, quickly finds candidate rows via the index, then calculates a similarity score (0 to 1) based on how many trigrams the two words share. If that score passes a threshold (default 0.3), it's considered a match. The result: fast, typo-tolerant, human-friendly search - baked right into your database, no external search engine required.

Quick Reference Cheat Sheet

-- Setup
CREATE EXTENSION pg_trgm;
CREATE INDEX idx ON table USING GIN (column gin_trgm_ops);

-- Basic search
SELECT * FROM table WHERE column % 'search term';

-- With score
SELECT *, similarity(column, 'search term') AS score
FROM table WHERE column % 'search term'
ORDER BY score DESC;

-- Adjust threshold
SET pg_trgm.similarity_threshold = 0.3;

-- Word-level similarity
SELECT word_similarity('search', 'long string with search inside');

-- Check index usage
EXPLAIN ANALYZE SELECT * FROM table WHERE column % 'term';

Fuzzy search is one of those features that, once you add it to your application, users can't imagine living without. That tolerant, forgiving search experience - the one that doesn't punish you for a stray keystroke - is powered by these elegant little 3-character windows sliding across your data, quietly doing the math.

Scaling Celery-Based Application in Production

Dhananjay Haridas — Sun, 26 May 2024 14:19:23 +0000

This documentation covers how to scale a Celery-based application for document extraction and comparison using FastAPI, Celery, and Redis. The guide includes steps for task splitting, configuring task dependencies, and scaling individual tasks.

Introduction
Task Definitions
Orchestrating Tasks with Parallel Processing
FastAPI Integration
Scaling Celery Workers
Using Dedicated Queues for Each Task Type
Autoscaling
Distributed Task Execution
Monitoring and Management
Load Balancing and High Availability
Summary

Introduction

This guide provides a detailed explanation of how to scale a Celery-based application that performs document extraction and comparison. It covers breaking down the tasks, orchestrating them for parallel processing, and scaling the application to handle increased loads in a production environment.

Task Definitions

Define the tasks for fetching, extracting, and comparing documents:

# tasks.py

from celery_config import celery_app
import logging

logger = logging.getLogger(__name__)

@celery_app.task
def fetch_documents_task(blob_path):
    try:
        documents = fetch_documents(blob_path)  # Replace with your actual fetch logic
        return documents  # Assume this returns a list of document paths or contents
    except Exception as e:
        logger.error(f"Error fetching documents: {e}")
        raise

@celery_app.task
def extract_data_task(document):
    try:
        extracted_data = extract_data(document)  # Replace with your actual extraction logic
        return extracted_data
    except Exception as e:
        logger.error(f"Error extracting data: {e}")
        raise

@celery_app.task
def compare_data_task(extracted_data_list):
    try:
        comparison_results = compare_data(extracted_data_list)  # Replace with your actual comparison logic
        return comparison_results
    except Exception as e:
        logger.error(f"Error comparing data: {e}")
        raise

Orchestrating Tasks with Parallel Processing

Use a combination of chains and groups to handle dependencies and parallel processing:

# main.py or workflow.py

from celery import chain, group
from tasks import fetch_documents_task, extract_data_task, compare_data_task

def process_documents(blob_path):
    # Step 1: Fetch documents
    fetch_task = fetch_documents_task.s(blob_path)

    # Step 2: Extract data from each document in parallel
    extract_tasks = fetch_task | group(extract_data_task.s(doc) for doc in fetch_task.get())

    # Step 3: Compare the extracted data
    compare_task = compare_data_task.s()

    # Combine the workflow into a single chain
    workflow = chain(fetch_task, extract_tasks, compare_task)
    result = workflow.apply_async()
    return result

FastAPI Integration

Integrate the workflow with a FastAPI endpoint:

# main.py

from fastapi import FastAPI
from workflow import process_documents  # Import your workflow function
from celery_config import celery_app

app = FastAPI()

@app.post("/process/")
async def process_endpoint(blob_path: str):
    result = process_documents(blob_path)
    return {"task_id": result.id}

@app.get("/status/{task_id}")
async def get_status(task_id: str):
    result = celery_app.AsyncResult(task_id)
    if result.state == 'PENDING':
        return {"status": "Pending..."}
    elif result.state == 'SUCCESS':
        return {"status": "Completed", "result": result.result}
    elif result.state == 'FAILURE':
        return {"status": "Failed", "result": str(result.result)}
    else:
        return {"status": result.state}

Scaling Celery Workers

Increasing the Number of Workers

Start multiple Celery worker processes:

celery -A celery_config worker --loglevel=info --concurrency=4

To scale further, start more workers:

celery -A celery_config worker --loglevel=info --concurrency=4
celery -A celery_config worker --loglevel=info --concurrency=4

Distributed Workers

Run workers on different machines by pointing them to the same message broker:

celery -A celery_config worker --loglevel=info --concurrency=4 -Q fetch_queue
celery -A celery_config worker --loglevel=info --concurrency=8 -Q extract_queue
celery -A celery_config worker --loglevel=info --concurrency=2 -Q compare_queue

Using Dedicated Queues for Each Task Type

Defining Queues

Configure Celery to define multiple queues:

# celery_config.py

from celery import Celery

celery_app = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')

celery_app.conf.task_queues = (
    Queue('fetch_queue', routing_key='fetch.#'),
    Queue('extract_queue', routing_key='extract.#'),
    Queue('compare_queue', routing_key='compare.#'),
)

celery_app.conf.task_routes = {
    'tasks.fetch_documents_task': {'queue': 'fetch_queue', 'routing_key': 'fetch.documents'},
    'tasks.extract_data_task': {'queue': 'extract_queue', 'routing_key': 'extract.data'},
    'tasks.compare_data_task': {'queue': 'compare_queue', 'routing_key': 'compare.data'},
}

Starting Workers for Specific Queues

celery -A celery_config worker --loglevel=info --concurrency=4 -Q fetch_queue
celery -A celery_config worker --loglevel=info --concurrency=8 -Q extract_queue
celery -A celery_config worker --loglevel=info --concurrency=2 -Q compare_queue

Autoscaling

Enable autoscaling to dynamically adjust the number of worker processes:

celery -A celery_config worker --loglevel=info --autoscale=10,3

--autoscale=10,3: Scales between 3 and 10 worker processes based on load.

Distributed Task Execution

Distribute Celery workers across multiple machines:

Example Setup

Machine 1 (Message Broker and Backend):
- Run Redis as your broker and backend.

Machine 2 (Worker Node):

Start Celery workers:

 celery -A celery_config worker --loglevel=info --concurrency=4 -Q fetch_queue

Machine 3 (Worker Node):

Start Celery workers:

 celery -A celery_config worker --loglevel=info --concurrency=8 -Q extract_queue

Machine 4 (Worker Node):

Start Celery workers:

 celery -A celery_config worker --loglevel=info --concurrency=2 -Q compare_queue

Monitoring and Management

Use monitoring tools like Flower, Prometheus, and Grafana to monitor Celery tasks:

Flower

Start Flower to monitor Celery workers:

celery -A celery_config flower

Load Balancing and High Availability

Implement load balancing for high availability and fault tolerance:

Example Load Balancer Setup

Use HAProxy or another load balancer to distribute requests across multiple Redis instances.

Summary

Scale Workers: Increase the number of Celery workers to handle more tasks concurrently.
Dedicated Queues: Use different queues for different types of tasks and scale them independently.
Autoscaling: Enable autoscaling to dynamically adjust the number of worker processes based on load.
Distributed Execution: Distribute workers across multiple machines to improve scalability and fault tolerance.
Monitoring: Use monitoring tools to keep track of the performance and health of your Celery workers.
Load Balancing: Implement load balancing for high availability and fault tolerance.

By following these strategies, you can effectively scale your Celery-based application to handle increased loads and ensure reliable task execution in a production environment.

MongoDB Shell Commands - A Comprehensive Guide

Dhananjay Haridas — Sat, 13 Apr 2024 22:04:49 +0000

Step 1: Start MongoDB Shell

Open your terminal.
Run the mongo command to start the MongoDB shell:

mongo

Step 2: Switch to a Database

To switch to a specific database, use the use command followed by the name of the database:

use mydatabase

Replace mydatabase with the name of your database.

Step 3: Show Databases

To show all databases available on the MongoDB server, run the following command:

show databases

This command will list all databases, including empty databases.

Step 4: Show Collections

After switching to a database, you can list all collections within that database using the show collections command:

show collections

This command will display all collections present in the current database.

Step 5: Show Data from a Collection

To display all documents (data) in a collection, use the db.collectionName.find() method. Replace collectionName with the name of your collection:

db.collectionName.find()

For example, to display all documents in a collection named users:

db.users.find()

You can also apply query criteria to filter the results. For example, to find documents where the age field is greater than 18:

db.collectionName.find({ age: { $gt: 18 } })

Step 6: Insert Documents into a Collection

To insert documents into a collection, use the db.collectionName.insertOne() or db.collectionName.insertMany() methods.

For example, to insert a single document into a collection named users:

db.users.insertOne({ name: "John", age: 30 })

To insert multiple documents into a collection:

db.users.insertMany([
    { name: "Alice", age: 25 },
    { name: "Bob", age: 35 }
])

Step 7: Update Documents in a Collection

To update documents in a collection, use the db.collectionName.updateOne() or db.collectionName.updateMany() methods.

For example, to update a single document in the users collection:

db.users.updateOne({ name: "John" }, { $set: { age: 31 } })

To update multiple documents:

db.users.updateMany({ age: { $lt: 30 } }, { $set: { category: "Young" } })

Step 8: Delete Documents from a Collection

To delete documents from a collection, use the db.collectionName.deleteOne() or db.collectionName.deleteMany() methods.

For example, to delete a single document from the users collection:

db.users.deleteOne({ name: "John" })

To delete multiple documents:

db.users.deleteMany({ category: "Young" })

Step 9: Exit MongoDB Shell

To exit the MongoDB shell, type exit and press Enter:

exit

These commands cover the basic CRUD operations (Create, Read, Update, Delete) in MongoDB from the shell. Adjust the database, collection names, and query criteria as needed for your specific use case.

MongoDB Installation Guide for Windows Subsystem for Linux (WSL)

Dhananjay Haridas — Sat, 13 Apr 2024 21:55:17 +0000

This guide will assist you in installing MongoDB using the Windows Subsystem for Linux (WSL) via the Command Line.

Install MongoDB - Community Edition

Launch a terminal (the Ubuntu app) and navigate to the root of the Ubuntu File System by typing cd ~.
Copy and paste the following command into the terminal to import the MongoDB public GPG Key:

   sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5

Next, add the MongoDB package to your sources list by pasting this line into the terminal:

   echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list

Note: Ensure you are on Ubuntu Xenial. We'll update this for Zesty LTS when available.

Refresh your local package database with the command:

   sudo apt-get update

Install MongoDB by running the following command:

   sudo apt-get install -y mongodb-org

This will install the latest stable version (MongoDB 3.6 at the time of writing). Refer to the provided link if you wish to install a different version.

Resolve Unmet Dependencies Error

If you encounter an error regarding unmet dependencies, execute the following steps:

Download libssl1.1_1.1.1f-1ubuntu2_amd64.deb:

   wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb

Install the downloaded package using dpkg:

   sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb

Create Data and DB Directories

Ensure you are still in the root of the Ubuntu FS by typing cd ~. Confirm with pwd, which should display /home/<user>/.
Create the necessary directories by typing:

   sudo mkdir -p data/db

This command creates a data directory with a db subdirectory. The -p flag ensures the parent directory is created if it doesn't exist.

Start mongod Server and mongo Shell

Open a new terminal window and run the following command to start the MongoDB server:

   sudo mongod --dbpath ~/data/db

You should see status messages, with the last line indicating that the server is waiting for connections on port 27017.

In another WSL window, type mongo to launch the MongoDB shell.
You'll see a connection notification in the first terminal window and the prompt in the new window should change to > indicating you're in the mongo shell.
Refer to this cheat sheet for guidance on using the mongo shell.
To exit the shell, press Ctrl + C. You'll receive a confirmation message before returning to the command line.

Troubleshooting

For further assistance, consult the MongoDB Install Docs or feel free to reach out directly via message or email.

credis -
github
stackoverflow

Windows Subsystem for Linux (WSL) Installation and Configuration Guide

Dhananjay Haridas — Sat, 06 Apr 2024 08:37:39 +0000

This comprehensive guide offers detailed instructions on setting up and configuring Ubuntu within the Windows Subsystem for Linux (WSL). WSL is a compatibility layer provided by Microsoft that enables users to run native Linux command-line tools directly on Windows 10 and Windows 11. By integrating a Linux kernel interface, WSL bridges the gap between the Windows and Linux ecosystems, providing users with the flexibility to leverage the strengths of both operating systems without the need for dual-booting or virtual machines.

Why Use WSL?

Seamless Integration: With WSL, users can seamlessly integrate Linux tools and utilities into their Windows workflow, enhancing productivity and development capabilities.
Compatibility: WSL ensures compatibility with a wide range of Linux software, allowing users to run popular applications, programming languages, and development environments natively on Windows.
Resource Efficiency: Unlike traditional virtual machines, WSL offers lightweight resource utilization, minimizing overhead and maximizing system performance.
Development Environment: WSL provides developers with a familiar Linux environment for software development, enabling them to utilize command-line tools, package managers, and development frameworks seamlessly.

Installing WSL (Windows Subsystem for Linux)

Open PowerShell as Administrator: Right-click the Start menu, then select "Windows PowerShell (Admin)".
Install WSL: Run the following command in PowerShell:

   wsl --install

This command will enable the WSL feature on your system.

Switching to WSL 2

Set Default WSL Version to 2: After installing WSL, switch to WSL 2 by running:

   wsl --set-default-version 2

This ensures that any new Linux distributions installed will use WSL 2.

Managing Linux Distributions

List Available Linux Distributions: To see all available Linux distributions in the Microsoft Store, use:

   wsl --list --online

This command will display a list of Linux distributions that you can install.

Install Ubuntu: If you choose Ubuntu, you can install it directly using:

   wsl --install -d Ubuntu

This command will download and set up Ubuntu on your system.

Check Installed Distributions: To see which Linux distributions are installed and their versions, run:

   wsl -l -v

This will list all installed distributions along with their respective versions.

Configuring Ubuntu

Set Default Distribution: If Ubuntu is not your default distribution, you can set it using:

   wsl --setdefault Ubuntu

This will make Ubuntu the default distribution to launch when using WSL.

Configure Ubuntu: After setting Ubuntu as default, update its package lists by running:

   sudo apt update

This ensures that you have the latest package information.

This Markdown document provides a step-by-step guide for installing, managing, and configuring WSL with Ubuntu on your Windows system.