DEV Community: KARTHIK NARAYAN

Adding an Index in Django Without Locking Your Database Table

KARTHIK NARAYAN — Mon, 29 Jul 2024 10:56:31 +0000

Adding an index to a table in Django is usually easy peasy. Just use makemigrations and migrate and you're done, right? But what if I told you that doing it the usual way locks your table until the index is done being created? Yikes! That's a problem, especially if your table is huge or your app is live.

The Usual Way (That Locks Your Table)

Normally, you'd do something like this to add an index:

from django.db import migrations, models

class Migration(migrations.Migration):

    dependencies = [
        ('api', '0004_alter_user_price'),
    ]

    operations = [
        migrations.AddIndex(
            model_name='user',
            index=models.Index(fields=['email'], name='api_user_email_a7eefd_idx'),
        ),
    ]

When you run this migration, it generates SQL like this:

BEGIN;
CREATE INDEX "api_user_email_a7eefd_idx" ON "api_user" ("email");
COMMIT;

This locks your table until the index is created. Not cool!

Nightmare Experience

Migrated a Production table in traditional way, which locked a table for 7 minutes 😢.

The Solution: Background Indexing

Lucky for us, modern databases like PostgreSQL, SQL Server, and Oracle can create indexes in the background, so they don't lock the table. But Django doesn't use this feature out of the box. No worries, though, we've got a trick: SeparateDatabaseAndState.

Step-by-Step Guide

Create an Traditional Migration File

   python manage.py makemigrations app_name

Create an Empty Migration File

Run this command to create an empty migration file:

   python manage.py makemigrations --empty --name custom_migration_name app_name

Add SeparateDatabaseAndState

Open the new migration file and add SeparateDatabaseAndState like this:

   from django.db import migrations, models

   class Migration(migrations.Migration):

       dependencies = [
           ('app_name', 'prev_migration_name'),
       ]

       operations = [
           migrations.SeparateDatabaseAndState(
               state_operations=[],
               database_operations=[],
           ),
       ]

Copy Migration Operations and SQL From Traditional File

From migration file generated by Django, copy the operations and paste them in state_operations.
Use sqlmigrate command to generate SQL Statement for the traditional migration file.

   python manage.py sqlmigrate app_name migration_name

Update new migration file and delete traditional migration file,

   from django.db import migrations, models

   class Migration(migrations.Migration):

       dependencies = [
           ('api', '0004_alter_user_price'),
       ]

       operations = [
           migrations.SeparateDatabaseAndState(
               state_operations=[
                   migrations.AddIndex(
                       model_name='user',
                       index=models.Index(fields=['email'], name='api_user_email_a7eefd_idx'),
                   ),
               ],
               database_operations=[
                   migrations.RunSQL(
                       sql="""
                       BEGIN;
                       CREATE INDEX "api_user_email_a7eefd_idx" ON "api_user" ("email");
                       COMMIT;
                       """
                   ),
               ],
           ),
       ]

Update SQL with background indexing and Add Reverse SQL

Now, let's actually add the background indexing. Update the database_operations like this:

   from django.db import migrations, models

   class Migration(migrations.Migration):

       dependencies = [
           ('api', '0004_alter_user_price'),
       ]

       operations = [
           migrations.SeparateDatabaseAndState(
               state_operations=[
                   migrations.AddIndex(
                       model_name='user',
                       index=models.Index(fields=['email'], name='api_user_email_a7eefd_idx'),
                   ),
               ],
               database_operations=[
                   migrations.RunSQL(
                       sql="""
                       CREATE INDEX CONCURRENTLY "api_user_email_a7eefd_idx" ON "api_user" ("email");
                       """,
                       reverse_sql="DROP INDEX "api_user_email_a7eefd_idx",
                   ),
               ],
           ),
       ]

The magic word here is CONCURRENTLY. It tells PostgreSQL to create the index without locking the table. If you're using SQL Server or Oracle, you'd use:

SQL Server: CREATE INDEX index_name ON table_name (column_name) WITH (ONLINE = ON);
Oracle: CREATE INDEX index_name ON table_name (column_name) ONLINE;

That's It!

By using SeparateDatabaseAndState, you can add indexes without locking your tables. No more downtime or slow queries during index creation. Your users (and your database) will thank you!

Documentation Reference.

Happy coding👨‍💻🐍!

Git Credibility: How to Steal the Spotlight (and Credit) in Collaborative Coding.

KARTHIK NARAYAN — Sun, 01 Oct 2023 13:54:21 +0000

Coding with your colleagues can be a rollercoaster ride of collaboration and shared caffeine-induced hysteria. Sometimes, you find yourself in a situation where you've practically co-authored a piece of code, but the official credit seems to be slipping through your fingers. Fear not, dear programmers, for we have uncovered the secret to rewriting commit history and claiming your well-deserved 15 milliseconds of coding fame!

The Case of the Vanishing Commit Credit

Imagine you and your colleague, Bob, have been burning the midnight oil to conquer a particularly pesky coding challenge. You've been sharing debugging insights, writing lines of code together, and practically finishing each other's Git commands. As the exhilarating moment approaches to immortalize your shared efforts with a Git commit, you realize your colleague might walk away with all the commit credit! Shock! Horror!. For me, it's a nightmare.

Unearthing the Git-gredient

Enter the mystical world of Git commit messages, where you can wield the power of co-authorship! With a single command, you can ensure that both you and Bob get the recognition you deserve. It's like writing your names on the digital chalkboard of history, for everyone to see.

1. Pair Programming Commit

When you and your colleague are pair programming and making the commit together on one machine, you can use the "Co-authored-by" trailer in the very first commit message. Here's an example of how to structure the commit message:

git commit -m "update done

Co-authored-by: Your Name <your@email.com>
Co-authored-by: Your Colleague's Name <colleague@email.com>"

This way, you're attributing both authors right from the start.

Or

2. The Co-authored-by Concoction

First, Bob creates a commit with his changes using the classic:

git commit -m "Implemented feature ABC"

Now, it's your turn to shine. To make sure your contributions don't go unnoticed, you'll craft a commit message with an added twist:

git commit -m "Implemented feature ABC

Co-authored-by: Your Name <your@email.com>

You've added a "Co-authored-by" trailer, and in it, you've bestowed upon yourself and Bob the honor you so richly deserve. Replace "Your Name" and "your@email.com" with your actual details and watch as the magic unfolds.

The Big Reveal

With the combined forces of your commits, you're ready to unleash your code upon the world! Don't forget to push your changes to the remote repository:

git push origin your-branch

The Final Act

As the code reviewers scroll through the commit history, they'll see your names side by side, like a dynamic duo of coding comedy. Your colleagues will admire your teamwork, your commitment to sharing the credit, and your excellent sense of humor (after all, you did read this blog, right?).

Setting Up Django for Production Using Gunicorn on Linux.

KARTHIK NARAYAN — Sat, 30 Sep 2023 18:21:22 +0000

Fun Fact: Why do Java developers wear glasses? Because they don't see sharp!

Django is a popular web framework for building web applications in Python. While the built-in development server is handy for testing your Django application during development, it's not suitable for production use. For production, you should use a production-ready web server like Gunicorn and manage it using systemd.

Prerequisites

Python 3 installed on your server.
A Django project ready for production deployment.
Virtual environment.

1: Install Gunicorn:-

Inside your virtual environment, install Django and Gunicorn:

pip install gunicorn

2: Configure Gunicorn :-

Create a Gunicorn configuration file for your Django project. In your project directory, create a file named gunicorn_config.py and add the following content, customizing it to your project:

import multiprocessing

bind = "127.0.0.1:8000"  # Replace with your desired IP and port
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "gthread"
threads = 2
timeout = 60

2: Test Gunicorn :-

Test Gunicorn to ensure it can serve your Django application. Run the following command from your project directory, replacing myproject with your Django project's name:

gunicorn -c gunicorn_config.py myproject.wsgi

If Gunicorn starts without errors, it's working correctly. You can stop it by pressing Ctrl + C.
Boom! Gunicorn is now part of our family.

3: Create a systemd Service :-

Create a systemd service file to manage the Gunicorn process. Use a text editor to create a file named myproject_gunicorn.service in the /etc/systemd/system/ directory:

sudo nano /etc/systemd/system/myproject_gunicorn.service

Add the following content to the file, adjusting the paths and configuration as needed:

[Unit]
Description=Gunicorn daemon for myproject
After=network.target

[Service]
User=your_username
Group=your_group
WorkingDirectory=/path/to/your/project
ExecStart=/path/to/your/virtualenv/bin/gunicorn -c gunicorn_config.py myproject.wsgi
Restart=always

[Install]
WantedBy=multi-user.target

Replace your_username, your_group, /path/to/your/project, and /path/to/your/virtualenv with your actual information.

Save and exit the text editor.

4: Enable and Start the systemd Service :-

Enable the systemd service and start Gunicorn:

sudo systemctl enable myproject_gunicorn
sudo systemctl start myproject_gunicorn

Check the status of the Gunicorn service to make sure it's running without errors:

sudo systemctl status myproject_gunicorn

If everything is configured correctly, your Django application should now be running in production using Gunicorn and managed by systemd. You can access it through the specified IP and port. Make sure to configure your web server (e.g., Nginx or Apache) as a reverse proxy to forward requests to Gunicorn for better security and performance in a production environment.

Until next time, keep coding and keep smiling, because development is all about making the magic happen! ✨
Happy coding!!!

Setting up Nginx to serve static content on any Linux-based server.

KARTHIK NARAYAN — Sat, 30 Sep 2023 17:45:42 +0000

Fun Fact: It works on my machine!

NGINX :

Welcome, brave adventurer, to the whimsical world of web hosting! Today, we embark on a quest to install and configure the legendary Nginx web server on our trusty Linux machine. Prepare yourself for an epic journey filled with code and commands.

1: Update System Packages:-

Before we dive into the magical realm of Nginx, we must ensure our Ubuntu system is up-to-date. Imagine your system's package list as a treasure map, and you want the latest X mark the spot.

sudo apt update

2: Installing Nginx :-

Now, the time has come to summon the great Nginx onto your server. With a swift command, it shall be done.

sudo apt install nginx

Boom! Nginx is now part of your server's fellowship

3: Activate and Auto-Start NGINX :-

let's start Nginx with a few commands that'll make our server up and running.

sudo systemctl start nginx
sudo systemctl enable nginx

You can check nginx status using the following command,
sudo systemctl status nginx

4: Alter Firewall :-

If you've got a fire-breathing dragon (firewall) guarding your server, let's give her some food (traffic).

sudo ufw allow 'Nginx Full'

5: Test Nginx :-

Open a web browser and enter your server's public IP address or domain name. You should see the default Nginx welcome page. This indicates that Nginx is correctly configured and running on your server.

6: Server Configuration :-

By default, Nginx serves content from the /var/www/html directory. If you want to host a website or application, you'll need to configure Nginx to serve your content. This is typically done by creating server blocks (similar to virtual hosts).

Now let's look at how to serve static content,

A.Create the sites-available Configuration File:

Open your terminal and use a text editor to create a new Nginx configuration file for your site in the sites-available directory:

sudo nano /etc/nginx/sites-available/example.com

B.Add the Configuration for Hosting Static Files:

Add the following configuration to the file, adjusting it as needed for your specific setup:

server {
    listen 80;
    server_name example.com www.example.com;

    # Define the root directory for your static files
    root /var/www/mywebsite/static;

    # Configure access to static files
    location / {
        try_files $uri $uri/ =404;
    }

    # Additional configuration options can be added here

    # Define access log and error log locations
    access_log /var/log/nginx/example.com.access.log;
    error_log /var/log/nginx/example.com.error.log;

    # Add any other server-specific configurations here

}

Here,

server_name specifies the domain names associated with your site.
root points to the directory where your static files are stored.
The location/ block is used to configure how Nginx handles requests for static files. It uses the try_files directive to check if the requested file exists and serves it, or returns a 404 error if not found,

Save the configuration file and exit the text editor.

C.Create the Symlink in sites-enabled:

Now that you've created the sites-available configuration, you need to create a symbolic link (symlink) in the sites-enabled directory to activate this configuration.

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

7: Final check and Reload :-

Before applying the new configuration, you should test it for syntax errors,

sudo nginx -t

Reload Nginx to apply the new changes,

sudo systemctl reload nginx

Until next time, keep coding and keep smiling, because development is all about making the magic happen! ✨
Happy coding!!!

*args and **kwargs explained in simple way

KARTHIK NARAYAN — Wed, 16 Aug 2023 06:27:05 +0000

let's break down *args and **kwargs using some fun and relatable examples!

1: *args - The Ice Cream Sundae of Arguments

Imagine you're at an ice cream parlour, and you want to create your ultimate sundae with as many toppings as you like. Well, *args is like those endless toppings. You don't know how many you'll choose, and you don't need to specify them all upfront.

def make_sundae(base_flavor, *toppings):
    sundae = f"A {base_flavor} sundae with:"
    for topping in toppings:
        sundae += f" {topping}"
    return sundae

print(make_sundae("vanilla", "chocolate sauce", "sprinkles", "whipped cream"))

Output:

A vanilla sundae with: chocolate sauce sprinkles whipped cream

In this example, the *toppings parameter collects all the extra toppings you provide and bundles them into a tuple. It's like making a sundae with any number of toppings without knowing exactly how many.

2: **kwargs - The Party Planner of Arguments

Now, let's plan a party using **kwargs. You're the host, and you want everything to be just right, from decorations to music. **kwargs is like having a checklist of items with their specific details.

def plan_party(**details):
    plan = "Party Details:"
    for key, value in details.items():
        plan += f"\n- {key}: {value}"
    return plan

print(plan_party(theme="tropical", guests=50, music="DJ Bob", snacks="finger foods"))

Output:

Party Details:
- theme: tropical
- guests: 50
- music: DJ Bob
- snacks: finger foods

In this example, the **details parameter collects the named arguments and their values into a dictionary. It's like planning a party with a bunch of different aspects, each with its specific information.

3: Combining `*args` and `**kwargs` - The Deadly-Act Show

Now, let's bring out the big guns and combine *args and **kwargs in a single function. Imagine you're ordering a customizable burger with various toppings, and you also want to note down specific preferences.

def custom_burger(base, *toppings, **preferences):
    burger = f"You ordered a {base} burger with:"
    for topping in toppings:
        burger += f" {topping}"
    burger += "\nPreferences:"
    for key, value in preferences.items():
        burger += f"\n- {key}: {value}"
    return burger

print(custom_burger("beef", "cheese", "bacon", dressing="mayo", sides="fries"))

Output:

You ordered a beef burger with: cheese bacon
Preferences:
- dressing: mayo
- sides: fries

In this example, you're using both *args and **kwargs in the same function. The *toppings collects extra toppings as a tuple, while **preferences collects named preferences as a dictionary. It's like crafting a burger with various toppings and specifying your unique preferences.

And there you have it! *args and **kwargs are your dynamic duos for handling flexible function arguments in Python. Just like creating custom sundaes, planning parties, or building the perfect burger, these concepts let you handle a variety of inputs with flair and finesse.
Until next time, keep coding and keep smiling, because Python is all about making the magic happen! 🐍✨
Happy coding!!!

How to connect multiple databases to Django project

KARTHIK NARAYAN — Sun, 28 May 2023 06:45:54 +0000

Django, a popular Python web framework, provides excellent support for working with databases. By default, Django supports connecting to a single database, but there are scenarios where you may need to connect multiple databases to your Django project. In this blog post, we will explore how to establish connections to multiple databases and perform database operations seamlessly in Django.

Configuring Multiple Databases:
To connect multiple databases in Django, you need to define their configurations in your project's settings file (settings.py). Within the settings file, locate the DATABASES dictionary, which holds the database configurations. By default, it contains a single entry named 'default'. To add more databases, you can define additional entries with unique names. For example:

`DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'default_db',
        'USER': 'your_username',
        'PASSWORD': 'your_password',
        'HOST': 'localhost',
        'PORT': '3306',
    },
    'second_db': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'second_db',
        'USER': 'your_username',
        'PASSWORD': 'your_password',
        'HOST': 'localhost',
        'PORT': '5432',
    },
}`

In the above example, we added a second entry named 'second_db' with its respective configuration parameters. Adjust the values according to your specific databases.

Database Routing:
After configuring the databases, Django needs to know which database to use for different models or database operations. Django provides a feature called database routing that allows you to specify the database to use based on certain conditions. To configure database routing, you need to create a new Python module in your project (e.g., database_router.py) and define the routing logic.

class DatabaseRouter:
    def db_for_read(self, model, **hints):
        if model._meta.app_label == 'app_label':
            return 'second_db'
        return None

    def db_for_write(self, model, **hints):
        if model._meta.app_label == 'app_label':
            return 'second_db'
        return None

    def allow_relation(self, obj1, obj2, **hints):
        if obj1._meta.app_label == 'app_label' or obj2._meta.app_label == 'app_label':
            return True
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label == 'app_label':
            return db == 'second_db'
        return None

In the above example, we check if the model belongs to the specified app label and return the name of the database. You can customize this logic based on your project's requirements.

Using Multiple Databases:
Now that you have configured the databases and created the routing logic, you can start using multiple databases in your Django project. To perform database operations on a specific database, you can use the using keyword argument with Django's database-related functions. For example:

from django.db import connections

def fetch_data_from_multiple_databases():
    default_db_cursor = connections['default'].cursor()
    second_db_cursor = connections['second_db'].cursor()

    default_db_cursor.execute('SELECT * FROM table_name')
    default_data = default_db_cursor.fetchall()

    second_db_cursor.execute('SELECT * FROM table_name')
    second_data = second_db_cursor.fetchall()

    return default_data, second_data

In the above we accessed the database connections using the connections object.
Happy coding 🥰

DEV Community: KARTHIK NARAYAN

Adding an Index in Django Without Locking Your Database Table

The Usual Way (That Locks Your Table)

Nightmare Experience

The Solution: Background Indexing

Step-by-Step Guide

That's It!

Git Credibility: How to Steal the Spotlight (and Credit) in Collaborative Coding.

The Case of the Vanishing Commit Credit

Unearthing the Git-gredient

1. Pair Programming Commit

Or

2. The Co-authored-by Concoction

The Big Reveal

The Final Act

Setting Up Django for Production Using Gunicorn on Linux.

Prerequisites

1: Install Gunicorn:-

2: Configure Gunicorn :-

2: Test Gunicorn :-

3: Create a systemd Service :-

4: Enable and Start the systemd Service :-

Setting up Nginx to serve static content on any Linux-based server.

NGINX :

1: Update System Packages:-

2: Installing Nginx :-

3: Activate and Auto-Start NGINX :-

4: Alter Firewall :-

5: Test Nginx :-

6: Server Configuration :-

A.Create the sites-available Configuration File:

B.Add the Configuration for Hosting Static Files:

C.Create the Symlink in sites-enabled:

7: Final check and Reload :-

*args and **kwargs explained in simple way

1: *args - The Ice Cream Sundae of Arguments

2: **kwargs - The Party Planner of Arguments

3: Combining *args and **kwargs - The Deadly-Act Show

How to connect multiple databases to Django project

3: Combining `*args` and `**kwargs` - The Deadly-Act Show