DEV Community: Vivek

Inbox in Discord

Vivek — Fri, 06 Jun 2025 06:58:00 +0000

Vivek

Jun 5 '25

Inbox Innovation with Postmark: Presenting InboxNinja 🥷

#devchallenge #postmarkchallenge #webdev #api

Comments 2

2 min read

Inbox Innovation with Postmark: Presenting InboxNinja 🥷

Vivek — Thu, 05 Jun 2025 07:50:41 +0000

This is a submission for the Postmark Challenge: Inbox Innovators.

Ever wished your incoming support, careers, or operations emails could instantly land in your team’s Discord channels so you can notify the right people, ask for input, and triage issues collaboratively?

Email workflows are often rigid and slow. InboxNinja transforms that experience by giving teams a real-time, customizable email relay bot that plugs directly into their Discord workspace.

What I Built

InboxNinja is a real-time, self-hosted email relay bot designed for operational teams. It receives emails via Postmark, processes them through a webhook, summarizes the content, and neatly delivers it into specific Discord channels. Messages arrive organized in threads, with attachments and action buttons for instant response.

It turns inboxes from static, disconnected workflows into real-time, actionable conversations inside Discord.

Demo

Code Repository

vivekthedev / inboxninja

a real-time, self-hosted email relay bot

📨 InboxNinja

A powerful, real-time email-to-Discord relay system for operational teams.
This application uses Postmark's Inbound Email Webhooks to receive emails for different departments (Careers, Support, Legal, Operations, Partnerships), summarizes the content and forwards it to the appropriate Discord channel.

Built for productivity, team alignment, and instant issue management. 🚀

🎯 Features

✅ Real-time email forwarding via Postmark Webhook
✅ Automated summarization using Gemini / LLM
✅ Auto-routing to relevant Discord channels based on the email address
✅ Supports attachments and file relaying
✅ Interactive buttons for support teams to respond quickly
✅ Modular and clean code with Quart + Discord.py

⚙️ Tech Stack

Postmark Inbound Webhooks
Discord.py
Quart (Async Flask Alternative)
Google Gemini / Summarizer
Python Asyncio
dotenv for secure config

🏗️ Project Architecture


/project-root
├── app.py                # Main application — runs both Discord bot & Quart server
├── bot.py                # Discord bot button views
├── gemini.py             #

…

View on GitHub

How I Built It

Postmark Inbound Webhooks for core email parsing and forwarding
Postmark API to retrieve complete message data via Message ID
Discord.py to build the Discord bot, send messages, handle buttons, and manage threads
Quart as the asynchronous web server for webhook handling

How it Works

The app runs two asynchronous services. A Quart web server handles webhooks, and a Discord bot manages messages and actions.

Workflow Overview

A new email arrives at Postmark.
Postmark's inbound webhook forwards the email payload to the /webhook endpoint.
The server extracts key details like sender info, subject, body text, and attachments.
It determines the appropriate Discord channel based on the email address.
The message is summarized using Gemini AI. Attachments are decoded and a clean message template is prepared.
The bot posts this into the right channel, starting a dedicated thread per sender.
Action buttons like “Show Full Email” and “Mark as Resolved” are attached.

Challenges

Managing both the asynchronous Quart server and Discord bot within a single process required careful event loop handling. Async conflicts, especially with asyncio.run(), demanded thoughtful debugging and examples from public repositories.

Handling attachments was another tricky area. Postmark delivers them as base64 strings, which needed decoding and repackaging as Discord File objects while preserving file metadata.

Discord.py’s documentation around custom button views was limited. After exploring community examples and third-party resources, I was able to implement this functionality effectively.

Conclusion

I am proud of how this project came together. It is functional, cleanly built, and addresses a genuine workflow problem for modern operational teams. In future, I am planning to build a lightweight priority detection system for support emails and using Postmark's Transactional Email service for reply

SecureLens 🔍: An LLM you can trust.

Vivek — Mon, 05 May 2025 01:41:11 +0000

This is a submission for the Permit.io Authorization Challenge: AI Access Control

What I Built

Presenting SecureLens 🔍: An internal company copilot tool to improve your productivity when dealing with several projects, clients, or internal company functions. SecureLens saves you the time for manually accessing crucial data in your day to day tasks. The best part, you can trust your copilot since all the risk of accidental data leaks and unauthorized AI outputs is handled for you using Permit.io.

Architecture

Permissions

Example Users

[
{"username": "alicej", "password":"2025DEVChallenge", "role":"employee"}
{"username": "bobsmith", "password":"2025DEVChallenge", "role":"manager"}
{"username": "charliek", "password":"2025DEVChallenge", "role":"admin"}
]

Demo

ABAC Rules

Employees can ivew PII data for only those ctomer for which they are accountable.

Project Repo

vivekthedev / securelens

📊 Secure AI Copilot with Fine-Grained Authorization

📌 Overview

An internal AI-powered assistant (Copilot) for enterprise environments that interacts with business-critical data systems — CRM, HR, and Financial databases — while enforcing granular, dynamic, and secure data access controls at every stage.

📊 Data Schema

Tables:

users (id, name, email, phone, role)
customers (id, name, email, phone, budget, address, assigned_emp_id)
hr_data (id, emp_id, designation, salary, benefits, department)
financial_data (id, vendor_name, amount, txn_date, notes)

📌 Why This Project

AI copilots are becoming integral to internal enterprise tools
Traditional RBAC isn’t enough for AI-generated dynamic content
We demonstrate secure AI data operations combining RBAC, ABAC, RAG filtering, and AI response enforcement — enterprise-ready and compliance-friendly

View on GitHub

My Journey

While going through the challenge description, a thought of Copilot tool came to my mind that can use Permit IO capability to serve data to only Authorised users and wouldn't it be better to have a Multi agent pipeline that takes the prompt, filters it, check the permission and return the response in a beautiful SQL table.

In any company, people spend a lot of time just searching, some technical ones could write their own queries but for the non-technical ones, they don't have this luxury and can only view the data with filters and conditions provided to them via a UI. This is where SecureLens come it, with its multi-agent capability and Authorization checks, it gives anyone the exact data you need. No risky queries, no unauthorized access, no wasted time.

In the beginning, the idea began with a document based tool which check the document's metadata and give output to the user if they are authorized. Later, I expanded the scope to include a few more agents to clean the prompt and access data from a SQL database using a dedicated agent.

I started with React, FastAPI and PydanticAI but due to time constraints and a little frontend knowledge. I decided to use Chainlit. Chainlit handles all the frontend part which let me focus more on Agents and Permissions.

In order to create agents, I used PydanticAI and Google Gemini LLM. With all the details ironed out. I began building my Policies and Agents.

Authorization for AI Applications with Permit.io

Permit IO turned out to be easier to work with then I initially thought. With easy to use UI creating policies was just a work of few clicks. Permit IO has three core concepts:

Roles: which are the actors
Resource: which are the entities on which the action needs to be performed
Action: the actual action to be performed by the actor

All of this can only be checked using Permit's check function. Every resources or attributes goes into this funciton and it returns True if the permission is granted or false otherwise. One other feature of Permit which I think was undercelebrated was defining resource as Parent-Child, which lets me easily define permission for parent and it get applied to the child automatically by a specific role.

Lastly, I used PermitIO's sync_user function to easily add all my users from the DB to the Permits workspace in a few seconds. One limitation I came across was that ABAC and ReBAC features currently work only within Docker container environments. It would be great to see native SDK support for these in local development in the future.

Conclusion

SecureLens is built to solve a growing problem of access control so that enterprises can integrate AI copilots into their system without risking data leaks and unathorized access. By participating and building in this hackathon I learned crucial concepts about AI access control, working with permissions and building agent.

Exporting Google NotebookLM Notes Is a Mess — So I Wrote a Python Script for It

Vivek — Sun, 27 Apr 2025 07:11:02 +0000

Google’s NotebookLM is a handy tool for interacting with your documents and taking notes from them. But if you’ve tried exporting your saved notes, you’ll quickly run into a frustrating problem.

There’s no export option in the app itself. You can’t easily back up your notes, search through them offline, or move them to another tool. The only official way to get your notes is by downloading them all at once — but this comes with a big downside: it dumps everything together into a single file. This ruins the structure of individual notes and strips away important formatting, making the notes messy and harder to work with.

I faced this issue myself, so I decided to fix it.

🛠️ I Created a Python Script to Fix This

I built a simple Python script that grabs your exported NotebookLM notes and neatly restructures them. It separates each note cleanly, preserves the original formatting as much as possible, and makes it easier to search or convert them into any format you want.

📌 How My Script Solves It

Since NotebookLM doesn’t offer any public API, I had to build the solution differently.

The script works by:

Parsing the raw HTML structure from the downloaded notes.
Converting each note into clean, readable Markdown.
Exporting them as individual PDF files.

This way, each note stays separate, well-formatted, and easy to search or archive.

⚙️ How to Work With the Script

Open the NotebookLM Notebook whose notes you want to export.
In the Studio section, you’ll find an option to Convert all notes to source.

Clicking this option will create a new file in your Sources section.

Next: Get the Source of this file to feed it into the script.

The script uses BeautifulSoup4 to traverse through the source and converts it to Markdown.

📋 How to Grab the HTML Source Code:

Press F12 or right-click on the screen and select Inspect. This will open DevTools.
In the top-left corner of the DevTools window, click on Select an element in the page to inspect it.
Hover over the elements in the Sources section to find the HTML code of your notes.

You’ll see all the notes code placed inside a <div> tag with the class elements-container.

This is the only manual step in the process.

Right-click on that <div> element.
Choose Copy > Copy element.
Paste it into a text file named notes.txt. (You can rename the file, but you’ll have to update the script accordingly.)

📦 Setting Up the Script

You’ll need:

Python 3.10+
Install dependencies:

  pip install beautifulsoup4==4.13.4 markdown_pdf==1.7

Download the script and save it in the same folder as notes.txt.

🚀 Running the Script

Open your terminal and run:

python export_note.py

Let the script do its thing.

Once finished, you’ll find all your notes exported as separate PDF files.

📄 What You’ll Get

The exported PDF files preserve the Markdown formatting — including:

Headings
Bullet lists
Code blocks
Bold Text

One of the output PDF files from my notes looked like this:

If you’ve been stuck with the same problem, I hope this saves you some time and frustration.
It’s always fun turning annoying gaps in tools into working solutions.

If you face any problems, feel free to reach out — DMs are open!

Presenting PageMyCV - Resume Website in one Click (using Pulumi ESC)

Vivek — Sun, 06 Apr 2025 16:04:51 +0000

This is a submission for the Pulumi Deploy and Document Challenge: Shhh, It's a Secret!

Managing secrets like API Keys and Configuration in any application is not just crucial from security point of view but also essential for boosting maintainability of the application. These secrets can grow to a huge number as your application extend. This is the reason you need a reliable Secret Management Service like Pulumi ESC.

In this article you will learn the importance of managing application secrets and I will introduce you to my project PageMyCV which uses Pulumi ESC to store and retrieve API Keys and Configurations.

Project Repo

vivekthedev / pagemycv

PageMyCV

PageMyCV is a Flask-based web application that transforms a PDF resume into a professional-looking online portfolio page. Users can upload their resume, and the app automatically extracts key details using Google GenAI and hosts the final webpage via AWS S3 and CloudFront.

Features

Upload a PDF resume via the web interface
Extract structured information using Google GenAI
Generate a clean HTML portfolio using Jinja templates
Automatically host the portfolio using AWS S3 and CloudFront
Uses Pulumi ESC for managing secure environment variables

Tech Stack

Python + Flask – Web server and routing
Fitz (PyMuPDF) – PDF text extraction
Google GenAI – Resume data extraction
Pulumi ESC SDK – Environment configuration
AWS S3 + CloudFront – Static site hosting
Boto3 – AWS SDK for Python

Prerequisites

Python 3.8 or higher
Google GenAI API Key
AWS Account with:
- S3 Bucket
- CloudFront distribution
Pulumi ESC configured with the required environment values

Setting

…

View on GitHub

Live Link

Try for yourself

Working Demo

My Journey

The idea of making this application struck me when I was creating my own Developer site using Pelican. I noticed I am copying and pasting a lot of stuff from my Resume which felt repetitive and inefficient and decided to build an application which takes my resume, extract all the key details and renders a HTML page which I can host on Github Pages or any other static web page hosting service. And that’s how PageMyCV was born.

I’ve tried to make this as simple as possible. All the user needs is a resume, regardless of the industry. The app picks up on the common details that are usually present across most fields and builds the page from there.

How it Works

When a user visits the web interface, they are prompted to upload their resume in PDF format. Once submitted, the app reads the file using fitz (PyMuPDF), a lightweight and efficient library for parsing PDF documents. This step extracts all textual content from the uploaded resume.

The raw text is then passed to Google’s GenAI API. We use a structured function calling mechanism to instruct the model on exactly what data we want extracted—such as the user's name, email, education, work history, technical skills, project details, and notable achievements. The model returns this data in a structured JSON format, which the app then uses to populate a pre-designed HTML template.

After the HTML page is rendered, it's uploaded to an AWS S3 bucket. We use the boto3 library to handle this interaction programmatically. Once the file is successfully stored in S3, it is served to the public via a CloudFront distribution. The user is then redirected to their unique CloudFront URL where their new online portfolio is instantly available.

Earlier, the project used Pydotenv to manage secrets, which made it so much inefficient. What if I want to rotate API_KEYs? Change Bucket Name or Use to different CloudFront Domain?

Pain Points of Traditional Approach

Managing configuration manually quickly becomes cumbersome. Storing sensitive values like API keys in plain text opens the door to accidental leaks. The developer might accidently commit .env files and your Configuration are publically exposed.

Using Pulumi ESC

Tools like Pulumi ESC are built to solve these exact problems. They allow you to securely manage secrets. A user can easily manage several configurtion in the same application just by switching environment which removes the need to manually changing model names to check the response when building AI applications.

To begin, you’ll need to install the ESC CLI on your machine. This CLI allows you to create, view, and update your environments with ease. Once installed, authentication is handled through a single environment variable: PULUMI_ACCESS_TOKEN. This token gives your CLI access to your Pulumi organization and its managed environments.

What used to be a clutter of 4–5 separate environment variables that had to be manually exported or managed in .env files, is now reduced to just one.

Resources

In Pulumi ESC all the magic happens in esc environments. Each environment holds a separate set of variables required by your application.
To create a new environment run the following in terminal:

esc env init default/myapp

This will create a new environment named myapp in default namespace. Since you can manage multiple environment you can type the below command to list all the environments:

> esc env ls
default/myapp

In order to create a new variable in this enviroment run the following command:

> esc env set default/myapp varNmae varValue

Note that the above command would only set the variable but to make it available you need to edit the conf file using:

esc env edit default/myapp

Running the above command would open a YAML file. Add the following text at the end of values block to make your variables accessible in the applicaiton.

environmentVariables:
  varName: ${varName}

To test this you can run the following command:

> esc env run default/myapp -- python -c "import os; print(os.getenv('varName'))"

varValue

In the above command you ran the python code with myapp environment. esc handled all the environment variables set up for you. In the above output you can see "varValue" was returned by our Python code.

This is how Pulumi ESC made my environment management part easy. In my application I have stored MODEL, API_KEY, BUCKET, AND CLOUDFRONT configuration which was easily managed by Pulumi ESC.

Takeaway

Using Pulumi ESC helped me move away from scattered .env files to a cleaner, safer setup. With one access token, I could securely manage secrets across multiple environments.

Effortless Database Migrations: Why Alembic is Your Python Must-Have

Vivek — Fri, 23 Feb 2024 10:31:58 +0000

SQL databases operate by defining specific fields, data types, and relationships across various models. These characteristics make SQL databases ideal for tasks like data science and record management. In Python-based web applications, SQL databases are commonly chosen, especially with Object Relational Manager (ORM) tools like SQLAlchemy. These tools facilitate the creation, querying, and modification of SQL models using Python.

However, altering a production database after it’s been defined can be difficult. Frameworks such as Django provide their own database migration systems to simplify and streamline this process. It introduces various checks to ensure database updates are performed smoothly. But, Python Frameworks such as Flask and FastAPI do not come with their own database migration system, this is where Alembic comes into play. Alembic is a database Migration system for usage with the SQLAlchemy Database Toolkit.

It allows you to make changes to your databases structure in a secure way using only Python without writing a single SQL statement. Additionally, with Alembic you can rollback to the previous version of your database definition with just a few commands. Alembic make migrations in Database really easy with intuitive syntax. It make easier to evolve our database with our application.

In this tutorial, you will learn to use Alembic in a FastAPI application. We will cover most important features like upgrade, downgrade, addcolumn and dropcolumn to enhance our FastAPI application. For this tutorial, we have already created a sample application in FastAPI which has a CRUD functionality on a book table having the following fields.

id: int  
title: string  
rating: integer  
price: float  
published: boolean

Setting Up the starter code

Cloning Application from GitHub

In order to setup the starter application, you have to first clone the application in your local directory using this command:

git clone https://github.com/vivekthedev/blog-tuts.git

Creating virtual environment

Once you have the application on your system, create a python virtual environment in order to isolote the dependencies from the system site packages. You can create a virtual environment using the following command:

python -m venv env

After running the above command, run env\Scripts\activate if you are on Windows else run source env/bin/activate if you are on Mac/Linux in order to activate your virtual environment.

Installing Dependencies

Once you activate the virtual envrironment you can install dependencies present in requirements.txt file by running pip install -r requirements.txt command.

Running and Testing Starter Application

At this point you are ready to run and test the application. In the same terminal, run uvicorn main:app --reload to start the application on the default address of http://127.0.0.1:8000/ , visit http://127.0.0.1:8000/docs to test your application on the Swagger UI.

What are Database Migrations?

As mentioned earlier, SQL databases have a strict structure with predefined rows and columns, which makes altering the database more challenging. For application developers, this process may require writing SQL statements, ensuring constraints like NOT NULL or UNIQUE are met, and specifying DEFAULT values. Moreover, if issues arise during the update process, reverting to the previous database state can be difficult, especially if TRANSACTION command aren’t explicitly defined when executing SQL statements.

Alembic solves this problem by providing an easier and intuitive way to evolve your database without writing any line of SQL code. Additionally, working with Alembic is much easier compared to manually changing database structure through SQL or via GUI application.

Each database update is stored in a Python file, providing flexibility to customize as needed. Reverting to a specific database state is as easy as executing a simple command, making the process more manageable and efficient.

Working with Alembic

To start working with Alembic, you first need to install Alembic using the following command. Also make sure to run the command in the same virtual environment as you FastAPI starter application:

pip install alembic

Once you install Alembic, you now have a alembic Command Line Tool to configure your database. To integrate Alembic with your project run the following command in the same terminal:

alembic init alembic-conf

The above command creates a directory named alembic-conf in your project directory and generates the following files:

alembic-conf/versions/ : This directory stores various versions of your database, you will use this directory in order to examine or update various states of your database.
alembic-conf/env.py : This file is used by alembic to run migrations, it defines how the migrations should be made on the database.
alembic-conf/script.py.mako : It is a template file which is used to create new database versions in the versions directory. In other words every file/migration in versions directory is created using this template file.
alembic-conf/README : Contains information about the projects or the alembic. You can use it as changelog or writing something informative about the project.
alembic.ini : This is the configuration file where you define you various setting in order to configure your migration process.

Configuring Database with Alembic

To enable Alembic to make changes to your Database, it requires knowledge of the location of your database service, which is defined by the SQLALCHEMY_DATABASE_URL. To do this, locate the database.pyfile within your project and copy the value sqlite:///books.sqlite3 . Then, navigate to Line 63 of the alembic.ini file and paste this value just after the variable sqlalchemy.url. Make sure to NOT enclose it in quotes .

Now open alembic-conf/env.py file, and populate it with the following code:

import sys

import models

sys.path.append("..")

After the above imports find the target_metadata variable and set it to models.Base.metadata .

Once you set up the above two configurations successfully, you are now ready to create you alembic revisions in order to make changes in your database.

Creating Alembic Revisions

For your first Alembic revision, you will create a new column in your Books table named Description which should be of type String. In order to create a revision file, open the terminal and type the following command.

alembic revision -m "create book description"

After running the above command, you will see that a new file with the name “<hash_value>_create_book_description.py” has been created. This file will contain our revision logic which we will define in upgrade() function in the file.

Open the newly created file and in the upgrade() function type the following code:

def upgrade() -> None:  
    op.add_column('books', sa.Column('description', sa.String(), server_default="A short Description"), nullable=False)

In the provided code, you’ve created a new revision that, when applied, adds a new column named ‘description’ of type string with a default value of “A short description”. It’s considered good practice to define the downgrade() function at the same time as the upgrade() function to ensure a smooth downgrading process in the future.

To implement this, in the downgrade() function, we will reverse the actions performed in the upgrade() function, which in this case involves removing the ‘description’ column from the ‘books’ table. To achieve this, type the following code inside the downgrade() function:

def downgrade() -> None:  
  op.drop_column('books', 'description')

This code will instruct Alembic to remove the ‘description’ column from the ‘books’ table when the downgrade process is initiated.

In this file, you can see another variable revision which contains the hash value of the revision file. Copy this value as it will be useful when applying the revisions to the database.

Applying Revisions

We have successfully created revision to make changes in the database, in order to apply these changes run the following command in your terminal:

alembic upgrade <hash_value>

Note that “<hash_value>” should be replaced with the hash value of your revision file, which you copied in the previous section. When you execute this command, you will receive confirmation messages indicating that your revisions have been applied successfully:

INFO  [alembic.runtime.migration] Context impl SQLiteImpl.  
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.  
INFO  [alembic.runtime.migration] Running upgrade  -> e9cbb68b0547, create book description

To verify that the changes have been applied to our database, we can run the application using the following command:

uvicorn main:app --reload

After that, proceed to http://127.0.0.1:8000/ to view the books stored in our database. However, you may notice that the models do not display the newly created ‘description’ field for the book. This is because FastAPI fetches all model information from the models.py file. To ensure FastAPI fetches the ‘description’ field as well, you need to add a new line in the models.py file. Open the models.py file and append this line at the end of the model:

class Books(Base):  
    __tablename__ = "books"  
    id = Column(Integer, primary_key=True, index=True)  
    title = Column(String)  
    price = Column(Float)  
    author = Column(String)  
    rating = Column(Integer)  
    published = Column(Boolean)  
    description = Column(String) # <- This line

Now open the http://127.0.0.1:8000/, you will see the following output:

You can also use sqlite3 command line tool or SQL Viewer tool in order to inspect changes in your database.

With that, you have successfully created and applied your very first migration using Alembic. This functionality proves to be highly useful for a production-level application, as it allows for seamless database schema modifications.

For your next migration, the objective is to create a new table called ‘reviews’. This next part will serve as a learning opportunity to understand how to create a new table using Alembic Operations. Eventually, we will remove this table to demonstrate the downgrade functionality of Alembic, showcasing its ability to rollback changes.

Creating Table with Alembic

In order to create a new revision, open the terminal and type the following following command:

alembic revision -m "create reviews table"

A new file with the name <hash_value>_create_reviews_table should now be generated in the “alembic-conf/versions” directory. Open this file, and you’ll notice that it contains a variable called “down_revision” with a value corresponding to the hash value of the previous migration file. This setup makes it easy to track our migrations and ensures that Alembic can keep the upgrade and downgrade processes in sync.

Upgrade and Downgrade function

Now let’s create Operations for creating a new table. Open the newly created migration file and populate the upgrade function with the following code:

def upgrade() -> None:  
    op.create_table(  
        "reviews",  
        sa.Column("id", sa.Integer(), primary_key=True, index=True),  
        sa.Column("author", sa.String(50), nullable=False),  
        sa.Column("content", sa.String(200)),  
        sa.Column("timestamp", sa.TIMESTAMP, server_default=sa.func.now()),  
    )

In the above code, we are creating a new table named reviews

which has the following columns:

id : An Integer type field for primary_key data of each row.
author : This column stores the author of the review and is defined as a String(50) type. It means that the column value can’t exceed the length of 50 characters.
content : Another String field which stores the actual review content. This column cannot exceed the character length of 200.
timestamp : A field with default value of the current timestamp which signifies the date time when a particular review was created.

As stated above, we should always write the downgrade function in order to revert the changes done by upgrade function. Write the following code in the downgrade function:


def downgrade() -> None:  
    op.drop_table('reviews')

The above code is pretty self-explainatory, we are just removing the table ‘reviews’ that we created in upgrade function.

Applying the Migration

To apply the migration, first copy the hash value from the migration file and then open your terminal and run the following command:

alembic upgrade <hash_value>

Once you’ve executed the provided command, you’ll receive a confirmation message in your terminal indicating that the latest migration has been successfully applied, resulting in the creation of the new “reviews” table. However, as our FastAPI application currently lacks any endpoints that query the “reviews” table, you won’t be able to observe these changes directly from the application.

To inspect the changes, we’ll utilize an SQL Viewer Tool. Simply open the SQL Viewer Tool in your browser. Upon loading the page, you’ll notice a “Drop Here” text. Click on it and navigate to your project directory, where you’ll find the database file named ‘books.sqlite3’. Select this database file, and the SQL tool will read all the tables within it. You’ll then be able to view all the tables available in the dropdown menu within the tool’s interface. This allows you to inspect the newly created “reviews” table and any other tables present in your database.

In the above image you can a reviews table which currently has no rows as we haven’t populated it with data.

Downgrade with Alembic

Downgrading with Alembic is straightforward. Simply enter the downgrade command along with the identifier of the desired state in the terminal. Alembic will automatically handle all the necessary steps to revert your application to the specified state.

Alembic supports alembic downgrade <hashvalue> syntax, but the more common (and easier) way to downgrade the application involves relative identifiers using the -N and +N syntax. Let’s do it practically to understand how it works. Open your terminal and type the following command:

alembic downgrade -1

The above command instructs Alembic to take the database to a state one step previous from the current state. In other words it reverts the last applied migration to the database. Naturally, when you put -2 instead of -1 it reverts the last two migration and so on.

After executing the command, you’ll receive a confirmation message in the terminal indicating that the downgrade operation ran successfully. To verify this, you can open the SQLite Viewer Tool and upload the database again. Upon inspection, you’ll notice that the dropdown menu displaying all the tables no longer includes the ‘reviews’ table. This confirms that the database has indeed been reverted back to its previous state.

You can also see in the above image that there is no review table in the database. Make sure to reupload the database to the viewer tool in order for them to reflect in your browser.

Conclusion

Alembic is a really powerful tool to add database migration using SQLAlchemy. It make migration process more intuitve, easy and secure. Developers can easily manage and rollback to a different state in the database with ease. Alembic also supports operation like Batch processing, raw sql execution, adding constraints and much more. You can read about all the operations provided by Alembic in the Documentation.

In this tutorial you learnt about Alembic and how you can use it in your python projects to easily manage your database. You learnt upgrading and downgrading in Alembic, creating columns, tables and and reverting changes. With these functionalities Alembic becomes a must-have when working with Python/SQLAlchemy project in order to easily manage your database.

Hope you like the tutorial, in case you have any question, shoot them in the comments. I will try my best to answer them.

Are Tuples really immutable? #PyTip03

Vivek — Sat, 20 Aug 2022 08:15:28 +0000

Tuple in Python are immutable Data Structures. Unlike sets, dicts and lists, the values in a tuple cannot be changed once initialized.

But is it really the case for tuples? See, tuples are immutable for the contents they hold ** that means the **references of the objects inside the tuple cannot be changed. But what if we change the contents of the without changing the object references.

To see it in action look at the below code

>>> p = 7
>>> id(7)
2917349484976
>>>p = 8
>>>id(p)
2917349485008

In the above code when we changed the value of an integer p the id of the object changed, that means the variable p started to point any other memory location. Now look at the below code.

>>> a = [1,2,3]
>>> id(a)
2917355980416
>>> a.append(4)
>>> id(a)
2917355980416
>>> a += [3,4,5]
>>> id(a)
2917355980416

In the above code we can see that even if we modify our list the id of the list reamains same, which means our variable a still points to the same memory location. We can exploit this behaviour to make our tuple immutable.

Type the below code to make a mutable tuple.

>>> tup = (1,2,[3])
>>> tup
(1, 2, [3])
>>> tup[2]
[3]
>>> tup[2].append(4)
>>> tup
(1, 2, [3, 4])

You can see that since the list object held its id even after we appended 4 to it, so the tuple had no problem mutating itself. The int data structure in Python changes its reference once its value is changes that is why we cannot change integer value in a tuple.

>>> tup[0] = 8
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Thanks for reading, Hope you liked it.

Iterable Unpacking in Python #PyTip02

Vivek — Fri, 05 Aug 2022 17:59:00 +0000

Python provides an awesome variety of sequential data structures. The most common of them are list and tuple. These sequences are sometimes used to store records, which later needs to be unpacked.

When unpacking the values from a sequence you may encounter error like-ValueError: too many values to unpack. In this short post I will share a short PyTip to avoid this error.

Problem

Create a file or run the below code in the Python REPL.

primes = [2, 3, 5, 7, 11]
first, second, third = primes

The above code will produce the following error.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: too many values to unpack (expected 3)

The above error states that to unpack your iterable, your iterable size must be three.

Solution

To solve this problem PEP 3132 was proposed. Now, write the following code in Python REPL to see Extended Iterable Unpacking in action.

>>> primes = [2, 3, 5, 7, 11]
>>> first, second, third, *_ = primes
>>> print(first, second, third, _)
2 3 5 [7, 11]

In the above output you can see that *_ syntax caught all the excess values from the iterable. The _ variable could be anything of your choice, it doesn't have to be an underscore. Additionally, "*_" may be used anywhere in the statement. Consider the example below to get the idea.

>>> primes = [2, 3, 5, 7, 11]
>>> first, *_ , last = primes
>>> print(first, last, _)
2 11 [3, 5, 7]

I find this feature extreamly useful and hope you find it useful too. Thanks for the reading today's PyTip.

How to make immutable dictionaries in Python #PyTip01

Vivek — Thu, 04 Aug 2022 05:20:09 +0000

Python is known for its batteries included capability, it has solutions for every data structure requirements. Dictionaries in Python are very useful when storing a record as Key/Value pairs.
Although dictionaries are awesome, they are mutable that means their values can be changed after their intialization.

In this short post you will learn how you can use MappingProxyType to make your dictionaries immutable.

Problem

Below is the example of how you would use dictionaries in Python.

>>> person = {"id" : 735616, "first_name": "John", "last_name":"Doe"}
>>> person["id"]
735616
>>> person["first_name"] + ' ' + person["last_name"]
'John Doe'
>>> person["id"] = 000000
>>> person["id"]
0

In the above code we can see that our person record was changed and we have no way to block this change. This is where MappingProxyType comes in.

Solution

MappingProxyType is a read-only wrapper that provides a proxy to the underlying data structure. MappingProxyType do not support item assignment which makes our data immutable.

>>> from types import MappingProxyType
>>> im_person = MappingProxyType({"id":736533, "first_name":"John", "last_name":"Doe"})
>>> im_person["id"]
736533
>>> im_person["id"] = 87
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'mappingproxy' object does not support item assignment

In the above example you can see that when we attempted to change the value of id, MappingProxyType immediately throws TypeError, thus making our dictionary immutable. Please note that MappingProxyType accepts any data type not just dictionaries. You can also use str and list.

Caveat

Before using MappingProxyType, you must know that it only provides a Proxy to the underlying data structure which means if the underlying data is changed then MappingProxyType will reflect those changes.

>>> im_person = MappingProxyType(person)
>>> im_person
mappingproxy({'id': 0, 'first_name': 'John', 'last_name': 'Doe'})
>>> person
{'id': 0, 'first_name': 'John', 'last_name': 'Doe'}
>>> person["id"] = 00
>>> im_person
mappingproxy({'id': 0, 'first_name': 'John', 'last_name': 'Doe'})
>>>

As you can see in the above code that the MappingProxyType changed when the underlying data dictionary was changed.

Conclusion

In this short article you learned about types.MappingProxyType,
they are useful when you want to make sure that your Key/Value pairs remains immutable. Hope you liked it.

Amazing Functools Features in Python

Vivek — Sat, 02 Apr 2022 16:35:57 +0000

I was recently reading Django’s Source Code, and I came across the @wraps decorator, which led me to the functools docs, where I discovered some fantastic functools features. That discovery led to the creation of this article.

This tutorial will teach you how to use some fantastic functools methods to make your life simpler.

What is `functools`

functools is a Python built-in module that contains Higher Order functions that can interact with other functions. A complete functools documentation may be found here.
Let’s see some decorators in action.

lru_cache

When invoking a function with the same arguments, this decorator in the functools module saves n number of function calls in cache, which saves a lot of time.

Assume for the sake of demonstration that we have a very large function that takes a long time to execute. The function a_heavy_operation() takes 3 seconds to execute in this example.

import time

start = time.time()
def a_heavy_operation():
    time.sleep(3)
    return 11 + 22


print(a_heavy_operation())
print(a_heavy_operation())

print(time.time() - start)

# Output
# 33
# 33
# 6.024240255355835

It takes about 6 seconds to run the above code. To the above function, we’ll add lru cache.

import time
from functools import lru_cache

start = time.time()


@lru_cache()
def a_heavy_operation():
    time.sleep(3)
    return 11 + 22


print(a_heavy_operation())
print(a_heavy_operation())

print(time.time() - start)

# Output
# 33
# 33
# 3.0158064365386963

Take a look at how using lru cache made our code run faster. Python saved the function’s cache and retrieved the cached value, reducing our execution time.

wraps

Wraps is used in functools to keep the function details. When we decorate a function, the function’s information is gone. We utilise the @wraps decorator on the decorator wrapper function to prevent this.

Take a look at this code to see what I mean.

from functools import lru_cache


def my_decorator(func):
    def log(*args, **kwargs):
        print("Running ")
        return func(*args, *kwargs)

    return log


@my_decorator
def add(a, b):
    """my beautiful doc"""
    return a + b

Run the above code in -i mode using, python -i file.py
Let's see what we have:

>>> add(1,2)
Running 
3
>>> add(3,4)
Running 
7
>>> add.__name__
log
>>> add.__doc__
>>>

We can see that our decorator is operating properly in the previous example, since it is consistently “Running” on each run. However, our function’s information has been lost, and it is unable to return the name or the docstring.

We have @wraps to help us with this problem. Make the changes below to the code.

from functools import wraps

def my_decorator(func):
    @wraps(func)
    def log(*args, **kwargs):
        print("Running ")
        return func(*args, *kwargs)

    return log


@my_decorator
def add(a, b):
    """my beautiful doc"""
    return a + b

Now again run the code using python -i file.py

>>> add(1,2) 
Running 
3       
>>> add.__name__
'add'
>>> add.__doc__
'my beautiful doc'
>>>

Voila! The function information is now saved in our function.

singledispatch

To create a generic function, singledispatch is utilised. Generic functions are those that perform the same operation on a variety of data types.

Assume I want to create a function that returns the first value from an iterable of several data types.


def return_first_element(data):
    if isinstance(data, list):
        print(data[0])
    elif isinstance(data, str):
        print(data.split()[0])
    elif isinstance(data, dict):
        print(list(data.values())[0] )
    else:
        print(print(data))

Now run python -i file.py to run the code in interactive mode.

>>> return_first_element({"Age":20, "Height": 180})
20
>>> return_first_element("Hello Mr Python")
Hello
>>> return_first_element([12,432,563])      
12
>>>

Our function is effective, but it isn’t clean. Using if/elif/else statements to create generic functions is not recommended in Python. So, what’s the solution? singledispatch, of course.

Let’s make a few modifications to our code.

from functools import singledispatch

@singledispatch
def return_first_el(data):
    return data


@return_first_el.register(list)
def _(data):
    return data[0]


@return_first_el.register(dict)
def _(data):
    return list(data.values())[0]


@return_first_el.register(str)
def _(data):
    return data.split()[0]

To check the results, run the code again in interactive mode with python -i file.py.

>>> return_first_el({"Age":20, "Height": 180}) 
20
>>> return_first_el("Hello Mr Python")             
'Hello'
>>> return_first_el([124, 765, 897])   
124
>>> return_first_el({12,31,1})       
{1, 12, 31}

Look how our return_first_el function acted as a fallback function when no data type matched for ‘set’.

Look at how much cleaner our code is now; the singledispatch made it easier to add more data types, and each datatype now gets its own place where we can perform further operations on the data.

total_ordering

The total_ordering decorator saves a ton of time in Object Oriented Progrmming.
Consider this example, the below class declares a class Man with name and age property and (=) __eq__ and (<) __lt__ dunder methods.

class Man:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __eq__(self, o):
        return self.age == o.age

    def __lt__(self, o):
        return self.age < o.age

Let's try to run the code to see what we have.

>>> obj = Man("Vivek", 20)
>>> obj2 = Man("Alex", 24) 
>>> obj = obj
>>> obj == obj2
False
>>> obj < obj2
True
>>> obj >= obj2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '>=' not supported between instances of 'Man' and 'Man'

Our code worked for (==) and (<), but it didn’t work when we used an operator that wasn’t defined in the class. Given that we create at least one operator dunder method and eq method, @total_ordering generates the,>,=,>=, and more comparison operators for our class.

Let’s add our decorator just above the class.

from functools import total_ordering

@total_ordering
class Man:
.....
.....

Now again run thee code in interactive mode to see the results

>>> o = Man("Vivek", 20) 
>>> b = Man("Alex", 24) 
>>> o == b
False
>>> o >= b  
False
>>> o <= b
True

Take a look at how total ordering generated our class’s comparison operators.

Conclusion

I hope you found this post useful; I strongly advise you to study the documentation in order to fully comprehend the internal mechanics of these higher level functions. If you enjoyed this, please consider following me on Twitter, where I share stuff related to Python, Web development, and open source software. I’ll see you there.

How to configure Django/PostgreSQL environment using Docker

Vivek — Thu, 03 Mar 2022 17:26:56 +0000

In this step-by-step article you will learn how you can set up a local development environment with Django and with PostgreSQL service using Docker.

What is Docker?

Docker is a way to create an isolated environment for your software, with the growing number of variables within a software such as operating systems, versions, dependencies there is need of virtualization so that a software can run on multiple systems without any problem. This is where Docker comes in, Docker virtualizes the application layer of the OS so that every application can run on this environment, The virtual environment is referred to as Docker Container. So with Docker we only need a configuration file and Voila you can launch your software on any system, this configuration file is called Docker Image.

Why Docker?

But why do I need to use Docker to create a virtual environment? Python already has venv andpipenv. The answer is that these virtual environments can only create environments for Python packages, not environments for non-Python packages such as PostgreSQL. In this tutorial, we will use pipenv to track Python dependencies.

Docker: Installation

Docker is available for Windows, Mac and Linux you can download it from the official site. After downloading and installing the setup file you can confirm your installation by running docker --version on your terminal/command prompt, which will output something like this.

Docker version 20.10.12, build e91ed57

Setting up Python Project

After installing Docker, create a virtual environment for your Python packages using pipenv. Install pipenv by running the below command. We have to use pipenv to package our Python dependencies.

pip install pipenv

Create a new directory anywhere on your PC. Set up your Python project here and open a command prompt in this folder. In the example below, we have created a folder named test and moved our command line to this folder.

Run the following command to create a Virtual Environment and install django using pipenv.

pipenv install django

You should see a Pipfile and 'Pipfile.lock' file in the directory. To enter this virtual environment, simply run pipenv shell. Then you should see something like this:

The (test-xxxxxxx) shows that you are in the virtual environment.

Now run django-admin startproject newproj to create a new Django project in this environment. There is a dot (.) at the endwhich tells django that it should create a new project in the current directory.

Now run python manage.py startapp newapp to create a new Django app in this project. In order for this application to be discoverable in your project, you must add it to the INSTALLED_APPS setting in your newproj/settings.py file.

...

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'newapp',
]
...

Before jumping into docker make sure that everything is working by running python manage.py runserver and goto, http://127.0.0.1:8000/
This screen should open up, this means your installation is working.

This is how our project looks like at this point:

D:\WORK\PY\TEST
│   db.sqlite3
│   manage.py
│   Pipfile
│   Pipfile.lock
│
├───newapp
│   │   admin.py
│   │   apps.py
│   │   models.py
│   │   tests.py
│   │   views.py
│   │   __init__.py
│   │
│   └───migrations
│           __init__.py
│
└───newproj
        asgi.py
        settings.py
        urls.py
        wsgi.py
        __init__.py

At this point, you're ready to set up your Docker image, but before that, make sure you're NOT in the pipenv environment. Run exit to exit the environment.

Set Up Docker

What is Docker Image:

A Docker image is like a blueprint and tells Docker how to build an application, where its dependencies are located, and the commands to run to build the application. Now create a "Dockerfile" in the same directory where your Pipfile and Pipfile.lock are located and write the following command to the file:

FROM python:3.9

ENV PYTHONDONTWRITEBYTECODE 1

WORKDIR /project

COPY Pipfile Pipfile.lock /project/
RUN pip install pipenv && pipenv install --system

COPY . /project/

This is a Docker image. Let's go through each line and see what it means.

FROM python:3.9 > FROM command is used to get the base image from Docker Hub. Since we are creating a Django project, we need a base Python image as Django is built on top of Python. If you are using Jenkins written in Java, you need to import openjdk. Replace 3.9 with something else and you can install any version of Python.

ENV PYTHONDONTWRITEBYTECODE 1 > Environment variable that tells Docker not to write cache files (.pyc), which are Python bytecode files, usually stored in the __pycache__ folder.

WORKDIR /project > WORKDIR is used to set the working directory of the Django project where project files will be stored.

COPY Pipfile Pipfile.lock /project/ > The project's dependencies are stored in the Pipfile, so you need to copy them to your working directory (/project/ in this case).

pip install pipenv && pipenv install system > This command is self-explanatory and uses pipenv to install all dependencies on your system.

COPY . /project/ > Instruct Docker to copy all files and folders (source code) to a WORKDIR named /project/.
Now it is time to build out Docker Image, on the terminal run the following command.

Now we have to build our docker image using the Dockerfile, from the following command

docker build .

Here's what Docker does after running the above program:

Creating a Linux Environment
Install Python as an environment from Docker Hub
Install all dependencies on the system
Copy all code from the directory to the Linux directory

To check this, VS Code provide a very useful Docker inspection tool to view the Linux directory structure of a Docker container. If you have a Docker container you can see this.

You have now created your Docker image. Now we need a docker-compose.yml file to control how the container is started.

Make a new file with name docker-compose.yml and write the following:

version: '3.8'
services:
    web:
      build: .
      command: python /project/manage.py runserver 0.0.0.0:8000
      volumes:
        - .:/project
      ports:
        - 8000:8000

Let's analyze the above file line by line. The first line tells us which Docker version we want to use.

Next, we define the services we want to run in the container. In this case, we want to use only one web service. It then tells Docker how to create the container and tells it to literally start the Django server by looking at the current directory (.) and running the runserver command. Volumes is Docker's data persistence mechanism and is used to keep Docker filesystem data synchronized with local directories. 'ports' specifies the ports that this particular web service can use.

docker-compose.yml job finished. Now run the docker-compose up command to start the service.

Now goto, http://127.0.0.1:8000/ if the page is working then congratulations, you just started your first Docker Container.

You can see in the below picture that we can now inspect our project files in the Docker Linux Container using VS Code

Adding PostgreSQL

Django comes with a built-in SQLite database, which is not recommended for production applications. For production, you need a powerful database like PostgreSQL. In this next part we are just going to do that.

To use PostgreSQL, you need to:

Install psycopg2 so that Python can talk with PostgreSQL.
Create a PostgreSQL service in Docker with the required environment variables.
Use volumes to persist data even if service is interrupted
Edit the settings.py file in your Django project to change DATABASE type.

So, let's begin. To install psycopg2, you must first stop the docker container using the docker-compose down command.

To install the PostgreSQL adapter for Django "psycopg2", enter the command pipenv install psycopg2. We are installing the adapter using 'pipenv' rather than 'pip' to make it usable in a docker environment that installs all dependencies for pipfile.lock.

After the installation is complete, you need to create a service for PostgreSQL in the dockercompose.yml file. Open the file and type:

version: '3.8'
services:
  web:
    build: .
    command: python /project/manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/project
    ports:
      - 8000:8000
    depends_on:
      - db

  db:
    image: postgres
    environment:
      POSTGRES_PASSWORD: postgres

    volumes:
      - postgres_data:/var/lib/postgresql/data/
volumes:
  postgres_data:

In the above file here is what we have done,

We created a new service called db, this service will pull PostgreSQL image from the Docker Hub,
We setup environment PostgresSQL password which we will use in our settings.py file
volumes are the location where all the data is stored
volumes outside the db service indentation tells the docker that we want to persist the postgres_data locally even when we stop our docker container.
we also made changes in web service by adding depends_on which tells docker that to start the db service before web service.

Now the only thing that is left to do is to tell our Django project that we have a postgreSQL db service and we want to use this database not SQLite database.

Open newproj/setting.py file and make the following changes in the DATABASES settings (line 77)

....
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'postgres',
        'USER': 'postgres',
        'PASSWORD':'postgres',
        'HOST':'db',
        'PORT':5432
    }
}
....

First we have to change our ENGINE to 'django.db.backends.postgresql', the NAME and USER settings can be of your choice, in the PASSWORD we have to put the password we defined in the docker-compose.yml, the HOST is your service name which is db and in the PORT we have to provide 5432 which is PostgreSQL default port.

All done, now let's start our docker container using the following command

docker-compose up -d --build

Note that we use -d flag which stands for detach mode it will be useful when we want to execute some commands in the same terminal and --build flag is used because we made changes in our Pipenv file that is why we are telling Docker to build the Container from scratch.

After running the above command goto http://127.0.0.1:8000/ , if that opens up then that means your db and web service are working.

Execute commands from docker

You may have already noticed that you can't run commands like migrate and createsuperuser using the command line, because you're getting the error "Could not resolve hostname 'db' to address: unknown host".

PostgreSQL databases can only be used in Docker containers, so all database-related commands must be run in the container.

Migrate Database

Go to the same terminal and run the following command, where we run the previous command to start our container.

docker-compose exec web python manage.py migrate

Create Super User

To test if our database is working correctly, let's create our database and login the user from the localhost

docker-compose exec web python manage.py createsuperuser

Type the Username, Email and Password and go to Django Admin to see if our Admin Site is working.

If this window opens up then congratulation you have successfully created a Django/PostgreSQL environment using Docker

Conclusion

Docker is far more vast then this, if you want to learn about Docker in detail then I would recommend the following resources.

GitHub Code Repository for the above article is available here.

If you liked the article then also consider checking me out on Twitter where I post stuff like this in under 280 characters daily.

If you are facing any problem in following up with this article then please feel free to ask in the comments or on Twitter

How to make Python Script that organises files

Vivek — Thu, 24 Feb 2022 16:09:52 +0000

In this tutorial, you'll learn how to create a Python script that organises your folders by file extension.

The prerequisites of this article is that you should be comfortable with Basic Python programming and how to use modules in Python.

The downloads folder in my PC is second Recycle Bin to me.
Here is what it looks like.

There are 112 files in that folder, and I am not very good at moving each file one by one after validating the file type. Let's use Python to automate the task.

What to build?

A script or GUI programme that asks for the folder that needs to be organised and then moves all of the files according to their file types. For instance, if I have files like these:

presentation.pptx
index.html
cover.jpeg
header.png
project.docx
file.dll

After running the script the folder structure should be:

Organized/
     |
     |_ Documents/
     |     |_ presentation.pptx
     |     |_ project.docx
     |
     |_ Code/
     |     |_ index.html
     |
     |_ Images/
     |     |_ cover.jpeg
     |     |_ header.png
     |
     |_ Others/
           |_ file.dll

Looks so clean now. Each file type has its own dedicated folder, the lookup has become much easier.

The Planning

Before touching the keyboard first we must design our logic, here is what we need to do:

Ask for the folder that needs to be organized.
Retrieves all filepaths from the folder.
Check the filetype.
Move the file according to its extension.

Looks good to me, now lets code:

Step 1 : Tell File Extension to the script

First lets create a dictionary that tells our script the folder where each type of file will go.

filetypes = {
    "Images": ["png", "jpg", "jfif", "webp", "jpeg", "bmp", "tiff", "gif", "raw", "psd"],
    "Documents": ["doc", "docx", "ppt", "pptx", "xls", "xlsx", "pdf", "txt"],
    "Videos": ["mp4", "mpeg", "mkv", "srt"],
    "Code": ["html", "css", "js", "py", "cpp", "c"],
    "Audio": ["mp3", "wav", "ogg"],
    "Compressed": ["zip", "tar", "rar"],
}

Step 2 : Ask the Directory Path

I am getting the Directory name using a GUI dialog box, that will be convenient. Tkinter has this amazing askdirectory function which we can use here.

from tkinter import filedialog
p = filedialog.askdirectory()

When we run the above cade our dialog box is showing up and p contains the path of the directory which we selected. Okay so we are done with asking the directory name.

Step 3 : Getting all the files

Now we have to get all the files from the given directory, for this I am using Pathlib, a built-in module in Python that makes working with Paths super easy. But first we have to create a folder Organized like we planned above.

from Pathlib import Path
PATH = Path(p)
dest = PATH / "Organized"
dest.mkdir(exist_ok=True)

The above code take the path p provided by the tkinter file dialog box and sends it to the Path(), we make a new folder "Oganized" only if it is not already there, we do it by using exist_ok=True.

Now the iteration part,

files = []
for i in PATH.iterdir():
    if i.is_file():
        files.append(i)

The above code iterates on every directory and files in the PATH, and if it is a file then saves it in the files list. (this is why we used Pathlib)

Step 3 : Check Type -> Move File

Now the only thing left to do is checking the file type then creating a folder for the file (not creating if it already exists) then move the file but move the file into Others folder if the file extension is unknown.

for file in files:
        # done flag tells that if the file matched to a dictionary value or not
        done = 0
        # iterate over the keys and check if the file belong to the particular key
        for k in filetypes.keys():
            # Check if the file extension is in the values of the key
            if file.suffix[1:] in filetypes[k]:
                done = 1
                # make a new folder with `key name` and move the file there
                destf = dest / f"{k}"
                destf.mkdir(exist_ok=True)
                shutil.move(str(file.resolve()), str(destf))

        if done != 1:
            # if the file was not present in the dictionary the make Others folder  and move the file there
            destf = dest / "Others"
            destf.mkdir(exist_ok=True)
            shutil.move(str(file.resolve()), str(destf))

The above code take each file from the files list and then checks if the extension is present in the dictionary or not.
If the extension is in the dictionary, make a folder for the file type and move the file into that folder. If the file's extension isn't found in the folder, move it to the 'Others' folder.

Time to run the script : The moment of truth

I ran the script in my downloads folder, look how the script asked for the directory and within seconds created Organized folder which contains all the files.

What more can be done?

The script is working fine but here are few things you can experiment with the script:

Use argparse for Command Line Input
Improve the performance, by removing the second loop and creating a new search method.
Show information about the files found and progress bar using tqdm

Try extending the script and make sure to create a PR on the repository.

Here is the script repository link

Thanks for sticking till the end, it means a lot.
If you liked the article then also consider checking me out on Twitter where I post stuff about python, javascript and open source daily.