DEV Community: Karishma Shukla

Why Markdown Is The Secret To Better AI

Karishma Shukla — Thu, 08 Jan 2026 15:28:23 +0000

The status quo of web scraping is broken for AI. For a decade, web extraction was a war over CSS selectors and DOM structures. We wrote brittle scrapers that broke the moment a div turned into a span.

But as we enter 2026, the bottleneck isn't getting the data : it’s the quality and density of that data when it hits an LLM’s context window.

If you are still feeding raw HTML to your RAG pipelines or AI agents, you are paying a "tax" in tokens, latency, and hallucinations.

Here is why Markdown is the missing link in the modern AI data stack.

The Token Tax: HTML is 90% Noise

Large Language Models don't read web pages; they process tokens. A standard e-commerce product page can easily reach 150KB of HTML, which translates to roughly 40,000+ tokens.

When you convert that same page to clean, semantic Markdown:

Size drops by 95%: You go from 40,000 tokens to ~2,000.
Cost Efficiency: You can process 20x more pages for the same API cost.
Signal-to-Noise Ratio (SNR): You strip away <script>, <style>, and nested <div> that forces the model's attention mechanism to work harder for less signal.

Data Format	Avg. Tokens per Page	Estimated Cost (GPT-4o)	Cost Efficiency
Raw HTML	45,000	$0.1125	Baseline
Clean Markdown	1,800	$0.0045	96% Reduction

Note: These estimates are based on current 2026 pricing for GPT-4o at $2.50 per 1M input tokens. By distilling HTML into Markdown, you're effectively increasing your context window by 25x for the same price.*

Structural Bias: LLMs are Native Markdown Speakers

LLMs are trained on the internet, which means they are trained on GitHub, StackOverflow, and Technical Documentation. The "Common Crawl" of high-quality reasoning data is written in Markdown.

Markdown provides semantic hierarchy that HTML obscures:

Headers (#, ##): Explicitly define the "parent-child" relationship of ideas.
Tables (|): Allow models to perform "columnar reasoning" (e.g., comparing prices across rows) without getting lost in <tr> and <td> nesting.
Bullet Points (-): Signal distinct entities or steps in a process.

When a model sees a Markdown header, it understands it as a context anchor. In raw HTML, that same header is just another node in a 50-level deep DOM tree.

RAG Accuracy: The "Chunking" Problem

Most RAG pipelines use "Naive Chunking"- splitting text every 500 characters.

The HTML Failure: A split might happen in the middle of a <table> tag, effectively destroying the data's meaning for the vector database.
The Markdown Solution: Markdown allows for Semantic Chunking. You can split data at the # or ## boundaries. This ensures that every chunk in your vector store is a coherent, self-contained unit of information.

Technical Insight: "Header-Aware Chunking" in Markdown-based RAG pipelines has been shown to improve retrieval accuracy by 40% to 60% because the embeddings capture the contextual intent of the section rather than just random word proximity.*

The Path Forward: Data is the New Code

We are moving toward a future where the "Browser" is just an OS for AI Agents.

The goal of data extraction in 2026 isn't just to "have" the data - it's to make it usable for the machines that will process it. High-density, structured Markdown is the only way to make LLMs smarter, faster, and cheaper to run.

We are building the future of AI-native extraction to bridge the gap between the messy web and the clean context windows your models deserve.

Ready to turn the web into your personal database?
Get Started For Free!

Join the Community

We are building the future of no-code, AI-native extraction.

Open Source No-Code Playwright Alternative For Web Scraping

Karishma Shukla — Thu, 28 Aug 2025 14:31:04 +0000

Selenium walked so automation could run.
Playwright made things faster, cleaner, and more developer-friendly.

Now, meet Maxun, the no-code sibling of Selenium and Playwright.

“Maxun is the kind of tool that makes Selenium cry in a corner. Show and go scraping is a dream.”

That’s what someone recently wrote on a Medium post about us.
And honestly…that sums it up better than I ever could.

Why we built Maxun

Traditional scraping is powerful but tedious.

CSS selectors break every time the UI shifts.
Writing code for lists, pagination, or screenshots takes longer than it should.
Sharing scrapers with teammates or non-devs is painful.

We Thought

What if you could build and run scrapers the way you browse a site?
So we built Maxun.

What makes Maxun different?

🖱️ Point-and-click scraping → Select data visually in your browser.
🆓 Free & open-source → Run it locally, forever. No paywalls.
☁️ Self-host or Cloud → Use our cloud for scale + managed infra, or host it yourself.
🔌 Integrations → Google Sheets, Airtable, CSV, JSON, API, Webhooks — data where you need it.
🛡️ Resilient → Retrain your scrapers in ~2 minutes without writing any code.

No prompt engineering.
No “agentic” magic that breaks silently.
Just scraping that works.

Why not just use an LLM?

You can. Custom AI agents or DIY Playwright + GPT can work…sometimes.

But scraping isn’t just about “visit this site and click that button.”
It’s about:

Handling pagination.
Exporting clean data.
Scheduling runs.
Running hundreds of robots without losing your sanity.

That’s why Maxun exists.
Think of it as Selenium for everyone - with a UI and batteries included.

Try it out 👇
GitHub → https://github.com/getmaxun/maxun
Website → https://maxun.dev

Free to self-host, or try Cloud if you want managed scaling.

What Selenium did for automation, Maxun aims to do for data extraction.
Only this time, it’s no-code, open-source, and free to start.

Would love your thoughts - is no-code Selenium what you’ve been waiting for?
Try it, star it, fork it - and let me know what you build! 🚀

Maxun: Open Source No-Code Web Data Extraction Platform⚡️

Karishma Shukla — Wed, 30 Oct 2024 07:49:58 +0000

👋 Hey Everybody
We are thrilled to open-source Maxun today.
Maxun is an open-source no-code web data extraction platform. It lets you build custom robots for data scraping in just a few clicks.
Github : https://github.com/getmaxun/maxun

🤔 But...why another web scraper? We have so many of them!

It’s surprising that in 2024, scraping data is still more complicated than actually using it. The last time I tried existing providers, I realized they mostly cater to technical users or big corporations with too much $ in the bank.
One provider didn’t even respond to my support request, which cost me valuable business days as I had no bandwidth to write scrapers.

💡 Then, I prototyped a small solution, open-sourced it, and discovered that quite a few people had similar experiences. So, I decided to build a scraper that:

Requires ZERO coding skills
Bypasses geolocation restrictions, captchas, and anti-bot measures
Offers a simple API
Gives me complete control over my data
Adapts to website layout changes
Just keeps getting better over time

We have a lot of exciting things coming up: Maxun Cloud, Extract Behind Login, Deep Data Extraction Workflows, Integrations +++

Our mission is simple: democratize access to information on the internet.

❤️ Hope you loved it.

🫂 Please join our community:
Github : https://github.com/getmaxun/maxun
Discord : https://discord.gg/5GbPjBUkws
Twitter : https://x.com/maxun_io

❤️ Would love to hear more about your use-cases & feedbacks.

Cheers,
Team Maxun

Monkeying Around with Python: A Guide to Monkey Patching

Karishma Shukla — Mon, 19 Feb 2024 16:32:48 +0000

What is Monkey Patching in Python?

Imagine modifying a car's engine while it's running. Monkey patching works similarly, allowing you to dynamically alter or extend a class or module's behavior at runtime. It involves changing or adding methods or attributes to existing modules or classes.

Extending a Class

class Cat:
  def meow(self):
    return "Meowww!"

# Monkey patch to extend functionality
def ragdoll_meow(self):
  return "Miaowww!"

original_meow = Cat.meow
Cat.meow = ragdoll_meow

ragdoll = Cat()
print(ragdoll.meow())  # Output: Miaowww!

Here, we replaced the meow() method with a breed-specific one for Ragdolls. Remember, this only affects instances created after patching.

When to use Monkey Patching?

While alternatives like Subclassing, Decorators, Dependency Injection, and Wrapper Classes offer structured ways to modify or extend functionality, there are situations where monkey patching becomes a preferred choice:

Fixing third-party bugs: Patch the problematic function with your fix without modifying the original library.
Testing and mocking: Isolate specific components by patching their behavior, creating controlled testing environments.
Experimentation: Try out different functionalities without permanent code changes.
Debugging: Temporarily replace problematic code to pinpoint the issue's source.

Real World Use Cases

1. Fixing a Library Bug
Imagine a library function you rely on has a minor bug. Instead of waiting for a fix, you can monkey patch it with your temporary solution:

# Original buggy function
def calculate_area(length, width):
  return length * width  # Missing multiplication by 2

# Monkey patch the bug
def fixed_calculate_area(length, width):
  return length * width * 2

original_area = calculate_area
calculate_area = fixed_calculate_area

print(calculate_area(5, 3))  # Output: 30 (correctly calculated)

2. Mocking for Testing
Mocking external dependencies during testing helps isolate your code and ensures its functionality even without external systems running. Here's how you might mock a network call:

# Original function fetching data
def get_data_from_api():
  # Makes an actual API call

# Monkey patch to return mock data
def mock_get_data_from_api():
  return {"data": "mocked"}

original_data_func = get_data_from_api
get_data_from_api = mock_get_data_from_api

2.1 Using Python's patch()
The unittest.mock module has patch() that allows you to temporarily replace a target with a mock object.

from unittest.mock import patch

@patch("module.function")
def test_function(mock_function):
    # Your test logic here

Problems to Watch Out For

While monkey patching is powerful, it comes with some problems.

Debugging complexity: Altered code can make debugging trickier, as the original behavior is hidden.
Unintended consequences: Changes might affect unforeseen parts of your application.
Version control challenges: Tracking and managing monkey-patched code separately becomes necessary.

Conclusion

Monkey patching offers flexibility at runtime in Python development. Remember, even monkeys need bananas in moderation. Patch wisely only if required. Keep your code sane! 🐒

Simplify Complex SQL Queries with Common Table Expressions (CTEs)

Karishma Shukla — Mon, 21 Aug 2023 18:00:39 +0000

What are Common Table Expressions?

Common Table Expressions (CTEs) are a valuable feature in SQL that lets you create temporary result sets within a query. They simplify complex queries, enhance code readability, and improve query performance. CTEs are initiated using WITH keyword.

Fig: CTE Syntax. Image from MariaDB

When to use CTEs?

CTEs are particularly useful to:

Break down complex operations into simpler steps
Handle hierarchical data structures
Implement pagination for large result sets
Streamline complex aggregation tasks
Have reusable code if you need the same logic at multiple places
Improve code readability and maintainability if your query involves subqueries, multiple joins, or intricate filtering conditions

Types of CTEs

Broadly CTEs can be classified into:

Non-recursive (Simple) CTEs
Recursive CTEs

1. Simple Common Table Expressions

Non-recursive CTEs are straightforward and do not involve self-reference. They are useful for simplifying complex queries, aggregations, and transformations by breaking them into smaller, more manageable steps.

Example: Total Salary by Department

WITH department_salary AS (
  SELECT department_id, SUM(salary) AS total_salary
  FROM employees
  GROUP BY department_id
)
SELECT * FROM department_salary;

Here, the CTE department_salary calculates the total salary for each department by using the SUM and GROUP BY functions. The main query then fetches the results from the CTE.

2. Recursive Table Expressions

Recursive CTEs are used to work with hierarchical or recursive data structures. They allow a query to reference its own output, enabling operations like traversing a tree structure or finding paths in a graph.

Example: Organization Hierarchy

Suppose we have a table named employees with columns employee_id, name, and manager_id, where manager_id refers to the employee_id of the employee's manager.

WITH RECURSIVE org_hierarchy AS (
  SELECT employee_id, name, manager_id, 1 AS level
  FROM employees
  WHERE manager_id IS NULL  -- Root level employees (managers)

  UNION ALL

  SELECT e.employee_id, e.name, e.manager_id, oh.level + 1
  FROM employees AS e
  JOIN org_hierarchy AS oh ON e.manager_id = oh.employee_id
)
SELECT * FROM org_hierarchy;

In this example, we define a recursive CTE named org_hierarchy. The initial query retrieves root-level employees (managers) by selecting those with a NULL manager_id. The recursive part of the CTE uses the UNION ALL clause to join the employees table with the CTE itself, connecting employees to their respective managers using the manager_id.

The recursive CTE is structured as follows:

The anchor query selects the root-level employees (managers) and assigns them a level of 1.
The recursive query selects employees who report to the managers found in the previous iteration, incrementing the level by 1.
The final query retrieves the entire organizational hierarchy, including employees and their respective levels within the hierarchy.

Yes, recursive CTEs are confusing. I myself struggle a lot with them. It takes a long time to understand when to use them and why. 🙃

Conclusion

In conclusion, Common Table Expressions (CTEs) are powerful for enhancing the readability, maintainability, and efficiency of complex queries.

Find me on GitHub, Twitter

How to Improve Performance of Your Database by Indexing Large Tables

Karishma Shukla — Mon, 07 Aug 2023 14:24:20 +0000

What is Database Indexing?

Database indexing is a technique that makes searching and retrieving data from a database faster. It is like creating a quick guide for finding information in a large book. It helps speed up searches and makes finding things easier.

Indexing speeds up SELECT queries and WHERE clauses. On the other hand slows down INSERT and UPDATE queries.

Fig: Database Index Data Structure

Why Indexing?

Imagine you have a database of books, and you want to find all the books that have the word "programming" in the title. Without an index, the database would have to scan every row in the table to find the books that match the search criteria. This could take a long time, especially if there are a lot of books in the table.

However, if you create an index on the title column, the database can quickly find the rows that match the search criteria. The index is a separate data structure that stores the values of the title column in sorted order. The database can use the index to quickly find the rows that contain the word "programming" in the title.

Indexing A Table With 50 Million Rows

For this example, we will create a database pg-million in PostgreSQL containing table customers with columns: first_name, last_name, mobile_no, country.

Insert 50 million rows of random data

CREATE TABLE customers(first_name VARCHAR(50), last_name VARCHAR(50), mobile_no INTEGER, country VARCHAR(50))

INSERT INTO customers (first_name, last_name, mobile_no, country)
SELECT substr(md5(random()::text), 1, 10),
       substr(md5(random()::text), 1, 10),
       (random() * 70 + 10)::integer,
       (CASE WHEN random() < 0.5 THEN 'India' ELSE 'United Kingdom' END)
FROM generate_series(1, 50000000);

Create an index on `country` column

We create an index on country column to have a well-organized list that lets us quickly locate all the customers from a particular country without searching through the entire list.

CREATE INDEX idx_partial_country ON customers (country) WHERE country IN ('India', 'United Kingdom')

Time to create index: 2m 2s

For this example, we are using partial indexes. A partial index is created based on a condition that filters rows for specific values. This allows the database to index and optimize only the relevant rows, reducing the index size and improving query performance for those specific values.

Note: The syntax for creating indexes and types of indexes differs among different databases. You should use appropriate syntax and index type depending on your database and use-case.

Measuring Query Execution Time Before and After Indexing

Consider the following query

SELECT * FROM customers WHERE country='United Kingdom';

Query Execution Time without index: 41836.270 ms

Query Execution Time with index: 24254.644 ms

Improvement in query execution time ~42.03%

(For better understanding you can find all the code here

How Well Are The Indexes Performing?

It is important to gain insights into index effectiveness. A few helpful metrics include:

Index Usage Statistics: Monitor the usage of indexes to understand which indexes are actively contributing to query performance. (Ex: Track the size of indexes, as larger indexes may impact disk space and I/O performance)
Query Performance Metrics: Monitor query execution times and response times for queries that involve indexed columns. (Ex: A sudden increase in query execution time may indicate index-related issues.)
Index Maintenance Metrics: Regularly assess the health of indexes and their impact on database operations. (Ex: Track index bloat, which occurs when indexes become inefficient due to excessive insertions, updates, or deletions.)

When To Use Indexing?

Frequent Search Queries: Use indexing when you frequently search for specific data in a large dataset. It helps to find the desired information quickly.
Performance Improvement: Indexing can improve the speed of data retrieval operations, especially for complex queries, by avoiding scanning the entire dataset.
Large Data Volumes: Indexing is used when dealing with sizable amounts of data, as it helps maintain efficient query performance even as the dataset grows.

When To Not Use Indexing?

Frequent Write Operations: Avoid excessive indexing if your database experiences frequent insert, update, or delete operations, as indexes can slow down these write operations and consume additional storage space. Indexes should not be used on the columns that are frequently manipulated.
Small Datasets: For relatively small datasets, indexing may not provide significant performance gains and can introduce unnecessary overhead. In such cases, the benefits may not outweigh the costs.

Conclusion

If you are looking for ways to improve the performance of your database, then database indexing is a good place to start. By creating indexes on the columns that are frequently used in queries, you can significantly improve the performance of your database and make your queries faster. However, it is important to weigh the benefits and drawbacks of indexing before making a decision.

Find me on GitHub, Twitter

Build Resilient Systems with Idempotent APIs

Karishma Shukla — Fri, 28 Jul 2023 15:21:47 +0000

Networks are unreliable but our systems cannot be.

What is Idempotency?

Idempotency is a property of API design that ensures that making the same request multiple times produces the same result as making it once. In other words, no matter how many times an idempotent API endpoint is invoked with the same set of parameters, the outcome remains unchanged after the first successful request.

In the context of API designing, idempotency is crucial to prevent unintended side effects and ensure the predictability and reliability of the API. It allows clients to safely retry requests without causing any data duplication, overwriting, or other unwanted effects.

Idempotent API for file upload. Image by author

Idempotency of an API is determined by the changes it makes to the system, not the response it provides.

To better understand the above statement, consider this example:

Consider an API endpoint that is designed to sign up a new user account in a web application. If this API is idempotent, it means that no matter how many times the API is called with the same input data (e.g., the same email and password), it will create the user account only once, and any subsequent invocations will have no further effect.
The API may return a successful response (e.g., status code 200) for the first request and subsequent requests, indicating that the user account already exists, but the system state remains unchanged.
The idempotency is evaluated based on the side effect of creating the user account, not the response message.

Real World Use Cases

Payment Processing: Idempotent APIs prevent double charging when processing payments, ensuring consistent and accurate billing.
Order Processing: Idempotent APIs in e-commerce platforms avoid duplicate orders or unintended changes to order status.
File Uploads: Idempotent APIs for file uploads prevent unnecessary duplication, ensuring files are stored only once, even during retries or network issues.
Subscription Management: Idempotent APIs handle subscription requests without creating duplicate subscriptions or unwanted changes to user preferences.
Distributed Systems: Idempotent APIs in distributed systems maintain consistency and handle failures gracefully, enabling safe retries without data inconsistencies.

How to Implement Idempotency in API Design?

Assign unique identifiers: Use UUIDs or other unique identifiers for each request to track and identify requests.
Idempotent HTTP methods: Design APIs using idempotent HTTP methods like GET, PUT, and DELETE. These methods ensure that multiple identical requests have the same effect as a single request.
Expiration time for idempotency keys: Set a reasonable expiration time for idempotency keys to ensure they are valid only for a certain period.
Response codes and headers: Utilize appropriate HTTP status codes (e.g., 200, 201, 204) and headers (e.g., ETag, Last-Modified) to indicate idempotency and successful processing.

HTTP Methods and Idempotency

Idempotent methods are those that can be safely repeated multiple times without changing the result beyond the initial operation.

The HTTP methods that are idempotent are GET, HEAD, PUT, and DELETE.
POST is not idempotent. This is because each time you make a POST request, it creates a new resource on the server, leading to a different result with each request. Subsequent POST requests will create additional resources, altering the state of the server, making it non-idempotent.

What are Idempotency Keys?

Before making an API call, the client requests a random ID from the server, which acts as the idempotency key.
The client includes this key in all future requests to the server. The server stores the key and request details in its database.
When the server receives a request, it checks if it has already processed the request using the idempotency key.
If it has, the server ignores the request. If not, it processes the request and removes the idempotency key, ensuring processing is done only once.

Example

In the example below, the server generates an idempotency key and returns it to the client in the response header (Idempotency-Key). The client must include this key in all subsequent requests. The server checks if it has already processed the request using the idempotency key and ensures exactly-once processing.

Note - This example is not intended for production use; instead, it serves to illustrate the fundamental concept of idempotent keys.

Node.js (Express)

const express = require('express');
const app = express();
const { v4: uuidv4 } = require('uuid');

const idempotencyKeys = new Set();

app.use(express.json());

app.post('/api/resource', (req, res) => {
  const idempotencyKey = req.header('Idempotency-Key');
  if (idempotencyKeys.has(idempotencyKey)) {
    return res.status(200).json({ message: 'Request already processed' });
  }

  const resourceId = uuidv4();
  // ... add logic to create the resource

  idempotencyKeys.add(idempotencyKey);

  return res.status(201).json({ resource_id: resourceId });
});

app.listen(3000, () => {
  console.log('Server started on port 9000');
});

Python (Flask):

from flask import Flask, request, jsonify
import uuid

app = Flask(__name__)
idempotency_keys = set()

@app.route('/api/resource', methods=['POST'])
def create_resource():
    idempotency_key = request.headers.get('Idempotency-Key')
    if idempotency_key in idempotency_keys:
        return jsonify({'message': 'Request already processed'}), 200

    resource_id = str(uuid.uuid4())
    # ... add logic to create the resource

    idempotency_keys.add(idempotency_key)

    return jsonify({'resource_id': resource_id}), 201

if __name__ == '__main__':
    app.run()

Conclusion

Idempotent APIs play a crucial role in ensuring the reliability, consistency, and efficiency of a system.

Find me on GitHub, and Twitter

A Practical Guide to Metaprogramming in Python

Karishma Shukla — Sun, 16 Jul 2023 18:25:22 +0000

What is metaprogramming?

Metaprogramming is a programming technique where a program can modify or generate code at runtime. It allows developers to write code that can analyze, modify, or create other code.

In other words, metaprogramming is a way of writing programs that manipulate programs.

Metaprogramming is supported in a lot languages including Python, JavaScript, Ruby, Clojure, Julia and Java (even though it is considered to be a static and verbose language!)

Fig: Relationship between structural and process concepts of metaprogramming. Image from Research Gate

Where is metaprogramming used?

Metaprogramming is useful for a lot of use cases. Some of them include

Frameworks and Libraries
Code Generation
Template Engines

Metaprogramming in Python

Metaprogramming in Python allows you to write code that can manipulate code itself, creating new code, modifying existing code, or analyzing code structures. Python provides several mechanisms for metaprogramming. Let's explore each of them with code examples:

1. Decorators

Decorators are functions that modify the behavior of other functions or classes. They wrap the target function or class and provide additional functionality. Decorators use the @ symbol and can be applied to functions or classes. Here's an example:

def uppercase_decorator(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return result.upper()
    return wrapper

@uppercase_decorator
def greet(name):
    return f"Hello, {name}!"

print(greet("Karishma"))

Output

HELLO, KARISHMA!

In the above example, the uppercase_decorator modifies the behavior of the greet function by converting its return value to uppercase.

2. Metaclasses

Metaclasses allow you to define the behavior of classes. They act as the blueprint for creating classes and can modify class creation and behavior. Here's an example:

class MetaClass(type):
    def __new__(cls, name, bases, attrs):
        uppercase_attrs = {key.upper(): value for key, value in attrs.items() if not key.startswith('__')}
        return super().__new__(cls, name, bases, uppercase_attrs)

class MyClass(metaclass=MetaClass):
    message = "Hello, World!"

print(MyClass.MESSAGE)

Output

Hello, World!

In the above example, the MetaClass metaclass modifies the attributes of the MyClass class by converting them to uppercase.

3. Function and Class Decorators

Besides using decorators with the @ symbol, you can also apply decorators using the function or class syntax. Here's an example:

class CubeCalculator:
    def __init__(self, function):
        self.function = function

    def __call__(self, *args, **kwargs):

        # before function
        result = self.function(*args, **kwargs)

        # after function
        return result

# adding class decorator to the function
@CubeCalculator
def get_cube(n):
    print("Input number:", n)
    return n * n * n

print("Cube of number:", get_cube(100))

Output

Input number: 100
Cube of number: 1000000

In the above example, we define a class decorator CubeCalculator that wraps the function get_cube to perform additional actions before and after its execution, and then applies the decorator to the get_cube function. When get_cube is called with an input number, it prints the input number, calculates its cube, and returns the result.

4. Dynamic Code Generation

Python allows you to generate code dynamically using techniques such as eval() or exec(). This can be useful for generating code based on certain conditions or dynamically creating functions or classes. Here's an example:

name = "Karishma"
age = 2

code = f'def greet():\n    print("Name: {name}")\n    print("Age: {age}")'

exec(code)
greet()

Output

Name: Karishma
Age: 2

In the above example, the code string is dynamically generated and executed using exec() to define the greet function.

Most common in-built keywords and functions for metaprogramming in Python

In Python, the following keywords and concepts are commonly associated with metaprogramming and provide the foundation for metaprogramming:

getattr(), dir(), hasattr(): These functions allow you to dynamically get and set attributes of an object at runtime. The built-in inspect module is responsible for an important concept of metaprogramming called introspection in which the program simply looks at and reports on itself.
exec(), eval() and compile(): These functions enable the execution of dynamically generated code strings or evaluation of expressions.
__getattr__() and __setattr__(): These special methods can be defined in classes to handle attribute access and assignment dynamically.
__metaclass__: This special attribute can be used in a class definition to specify a metaclass for that class.

Conclusion

These are some of the ways to achieve metaprogramming in Python. Each technique provides flexibility in manipulating code structures and behavior, enabling you to create more dynamic and flexible applications.

Find me on GitHub, Twitter

How Python uses Garbage Collection for Efficient Memory Management

Karishma Shukla — Fri, 07 Jul 2023 12:29:14 +0000

What are variables in Python?

A variable in Python is usually assumed to be a label of a value. Instead, a variable references an object that holds a value.

In Python, variables are references.

How are objects stored in Python?

An object can be defined as a block of memory with a specific value supporting specific type of operations.

In Python, everything is an object.

A Python object is stored in memory with names (not variables) and references

Name - Just a label for an object. An object can have multiple names.
References - A name referring an object.

Every object consists of : reference count, type, value.

How variables are stored in memory. Image by author.

References Introduction

The following example assigns a number with value 10 to num variable

num = 10

Under the hood, Python creates a new integer object of type int in the memory. The variable num references to that memory address

To find the memory address of an object referenced by a variable we can use the built-in id() function.

The id() function returns memory address as a base-10 number. We will convert it into hexadecimal using in-built hex() function.

print(hex(id(num)))

--> 0x7ffdb446d448

Hex representation of a reference’s memory address. Image by author.

Passing arguments in Python functions

In Python unlike other languages, there is no such thing as pass by value or pass by reference.

Instead, Python has the concept of pass by assignment or pass by object reference.

When a function is called with an argument, a new reference to the object is created and assigned to the parameter variable in the function. The parameter variable becomes a new reference to the same object in memory, not a copy of the object itself. Any modifications made to the object within the function will affect the original object outside the function.

The value of the reference (the memory address) is passed to the function, not the value of the object itself.

Example : The parameter is immutable

Immutable objects include built-in data types like int, float, complex, bool, strings, bytes and tuples.

def f(name):
    name = 'John'

new_name = 'Mary'
f(new_name)
print(new_name)

Output:

Mary

In the above example, both name and new_name point to Mary at the same time. But when name = ‘John‘, a new object is recreated with the value of John and name continues pointing to it, while new_name still points to Mary. Hence the value of new_name does not change.

Example : The parameter is mutable

Mutable objects include list, dict and set.

def f(students):
    students.append(3)

students = [0,1,2]
f(students)
print(students)

Output:

[0,1,2,3]

In the example above, as students is a list, changing the value of students will also change value of all variables that point to it. Hence students becomes [0,1,2,3]

Garbage Collection

Garbage collection in Python refers to the automatic process of reclaiming memory occupied by objects that are no longer in use. It is a mechanism that manages the allocation and deallocation of memory in Python.

Python uses a garbage collector to automatically detect and remove objects that are no longer referenced or reachable by the program. When an object is no longer needed, the garbage collector identifies it as garbage and frees up the memory occupied by that object.

The two strategies used for garbage collection are

reference counting
generational garbage collection

1. Reference Counting

It keeps track of the number of references to each object, and when the count reaches zero, indicating that no references to the object exist, the object is considered garbage and the memory is reclaimed.

To get the reference count of an object, we can use the built in ctypes module.

import ctypes

def count_references(address):
    """
    Count the number of references to the object at the given address.
    """
    return ctypes.c_long.from_address(address).value

students = 15
print(count_references(id(students)))

# Step 1
toppers = students 
print(count_references(id(students)))

# Step 2
toppers = 2
print(count_references(id(students))) 

# Step 3
students = 1
print(count_references(id(students)))

Step 1: reference count of students = 2. Image by author.

Step 2: reference count of students = 1. Image by author.

Step 3: The number of references of the integer object with value of 15 will be 0. Image by author.

But reference counting cannot solve the problem of cyclical reference.

What is cyclical reference?

A cyclical reference, also known as a reference cycle or circular reference, occurs in Python when a group of objects reference each other in a way that forms a closed loop, preventing them from being garbage collected. This can lead to memory leaks as the objects involved are not eligible for automatic memory reclamation since their reference counts never reach zero.

Basic example of cyclical reference:

x = []
x.append(x)
print(x)

In the above example x is referring to itself, which makes it a cyclical reference.

To solve this problem Python uses Generational Garbage Collection.

2. Generational Garbage Collection

Generational Garbage Collection uses a trace-based garbage collection technique.

Trace-based garbage collection is a technique used in some garbage collection algorithms to identify and collect unreachable objects. It works by tracing the execution of a program and identifying live objects based on their accessibility from root references.

Generational Garbage Collection divides objects into different generations based on their age, with the assumption that most objects become garbage relatively quickly after they are created.

The main idea behind Generational Garbage Collection is that younger objects are more likely to become garbage than older objects. Python's garbage collector focuses its efforts on the younger generations, performing frequent garbage collection on them. Older generations are garbage collected less frequently since they are expected to contain objects that have survived multiple collections and are less likely to become garbage.

Generational Garbage Collection helps address the problem of cyclical references by periodically examining objects in different generations and collecting those that are no longer reachable. It detects and breaks cyclical references by identifying unreachable objects through a process known as "mark and sweep."

Generational Garbage Collection thus ensures:

no memory leaks
proper utilization of system resources
efficient garbage collection

Programmatically interact with Python’s garbage collector

In the example below, we create two classes Students and Boys referencing each other and perform garbage collection using in-built gc module (Garbage Collector interface).

You should never disable the garbage collector unless required.

import gc
import ctypes

def count_references(address):
    """
    Count the number of references to the object at the given address.
    """
    return ctypes.c_long.from_address(address).value

def object_exists(obj_id):
    """
    Return True if the object with the given id exists.
    """
    for obj in gc.get_objects():
        if id(obj) == obj_id:
            return True
    return False

class Students:
    def __init__(self):
        self.boys = Boys(self)
        print(f'Students: {hex(id(self))}, Boys: {hex(id(self.boys))}')

class Boys:
    def __init__(self, students):
        self.students = students
        print(f'Boys: {hex(id(self))}, Students: {hex(id(self.students))}')

gc.disable()

students = Students()

students_id = id(students)
boys_id = id(students.boys)

print(f'Number of references to students: {count_references(students_id)}') # 2

print(f'Number of references to boys: {count_references(boys_id)}') # 1

print(f'Does students exist? {object_exists(students_id)}') # True
print(f'Does boys exist? {object_exists(boys_id)}') # True

students = None

print(f'Number of references to students: {count_references(students_id)}') # 1

print(f'Number of references to boys: {count_references(boys_id)}') # 1

print(f'Does students exist? {object_exists(students_id)}') # True
print(f'Does boys exist? {object_exists(boys_id)}') # True

print('Collecting garbage...')
gc.collect()

print(f'Does students exist? {object_exists(students_id)}') # False
print(f'Does boys exist? {object_exists(boys_id)}') # False

print(f'Number of references to students: {count_references(students_id)}') # 0

print(f'Number of references to boys: {count_references(boys_id)}') # 0

Output:

Boys: 0x1e18b68c6d0, Students: 0x1e18b698510
Students: 0x1e18b698510, Boys: 0x1e18b68c6d0
Number of references to students: 2
Number of references to boys: 1
Does students exist? True
Does boys exist? True
Number of references to students: 1
Number of references to boys: 1
Does students exist? True
Does boys exist? True
Collecting garbage...
Does students exist? False
Does boys exist? False
Number of references to students: 0
Number of references to boys: 0

Conclusion

Garbage collection in Python helps manage memory efficiently, automatically freeing up resources and preventing memory leaks, so developers can focus on writing code without explicitly managing memory deallocation.

Find me on GitHub, and Twitter

'any' vs 'unknown' in TypeScript

Karishma Shukla — Mon, 05 Jul 2021 19:50:00 +0000

When you start learning TypeScript, you will come across two types - "any" and "unknown"

any - The any type allows us to assign literally “any” particular value to that variable, simulating what we know as plain JavaScript.

unknown - The unknown type is the type-safe counterpart of any. Anything is assignable to unknown, but unknown isn't assignable to anything but itself and any without a type assertion or a control flow based narrowing.

Let's understand with an example.

let age: number;
let userAge: any;

userAge = 'This is some age';
userAge = 20;

age = userAge;

This code works! 🎉
Type of userAge is any so it can be assigned any value - string, number etc.

let age: number;
let userAge: unknown;

userAge = 'This is some age';
userAge = 20;

age = userAge;

The statement age=userAge gives an error.
The type is unknown so what is the problem here?
To assign an unknown value to a value with a fixed type, we have to do some quick type check!

let age: number;
let userAge: unknown;

userAge = 'This is some age';
userAge = 20;

if(typeof userAge === 'number') {
  age = userAge;
}

And now this works too! 🎉

When to use what?

You shouldn't use either of them. But if you really have to then unknown is a better choice if you know what you want to do with that value eventually.
I don't recommend using any - it takes away the actual essence of TypeScript!

DEV Community: Karishma Shukla

Why Markdown Is The Secret To Better AI

The Token Tax: HTML is 90% Noise

Structural Bias: LLMs are Native Markdown Speakers

RAG Accuracy: The "Chunking" Problem

The Path Forward: Data is the New Code

Join the Community

Open Source No-Code Playwright Alternative For Web Scraping

Why we built Maxun

We Thought

What makes Maxun different?

Why not just use an LLM?

Maxun: Open Source No-Code Web Data Extraction Platform⚡️

🤔 But...why another web scraper? We have so many of them!

Monkeying Around with Python: A Guide to Monkey Patching

What is Monkey Patching in Python?

Extending a Class

When to use Monkey Patching?

Real World Use Cases

Problems to Watch Out For

Conclusion

Simplify Complex SQL Queries with Common Table Expressions (CTEs)

What are Common Table Expressions?

When to use CTEs?

Types of CTEs

1. Simple Common Table Expressions

Example: Total Salary by Department

2. Recursive Table Expressions

Example: Organization Hierarchy

Conclusion

How to Improve Performance of Your Database by Indexing Large Tables

What is Database Indexing?

Why Indexing?

Indexing A Table With 50 Million Rows

Insert 50 million rows of random data

Create an index on country column

Measuring Query Execution Time Before and After Indexing

How Well Are The Indexes Performing?

When To Use Indexing?

When To Not Use Indexing?

Conclusion

Build Resilient Systems with Idempotent APIs

What is Idempotency?

Real World Use Cases

How to Implement Idempotency in API Design?

HTTP Methods and Idempotency

What are Idempotency Keys?

Example

Conclusion

A Practical Guide to Metaprogramming in Python

What is metaprogramming?

Where is metaprogramming used?

Metaprogramming in Python

1. Decorators

2. Metaclasses

3. Function and Class Decorators

4. Dynamic Code Generation

Most common in-built keywords and functions for metaprogramming in Python

Conclusion

How Python uses Garbage Collection for Efficient Memory Management

What are variables in Python?

How are objects stored in Python?

References Introduction

Passing arguments in Python functions

Garbage Collection

1. Reference Counting

What is cyclical reference?

2. Generational Garbage Collection

Programmatically interact with Python’s garbage collector

Conclusion

'any' vs 'unknown' in TypeScript

When to use what?

Create an index on `country` column