As a best-selling author, I invite you to explore my books on Amazon. Don't forget to follow me on Medium and show your support. Thank you! Your support means the world!
Let me tell you something about Python that took me years to fully appreciate. You can write code that writes code. Not in some abstract, philosophical sense, but literally. You can create classes while your program runs. You can intercept every attribute access and decide what happens. You can build functions from strings generated from configuration files. This is metaprogramming, and it is the superpower that makes frameworks like Django ORM, SQLAlchemy, and Pydantic work. I will walk you through eight techniques, with code you can run and modify. I learned these by breaking things and fixing them, so I will share the mistakes I made too.
Let me start with metaclasses. A metaclass is the class of a class. Just like an instance is created from a class, a class is created from a metaclass. By default, Python uses type as the metaclass. You can create your own metaclass and inject behavior every time a new class is defined.
I remember the first time I needed this. I had ten model classes, and I kept forgetting to add a created_at timestamp. I could have copied the same line into every class, but that is boring and error prone. Instead, I wrote a metaclass that automatically adds the timestamp attribute.
import time
class TimestampMeta(type):
def __new__(mcs, name, bases, namespace):
namespace['_created_at'] = time.time()
return super().__new__(mcs, name, bases, namespace)
class BaseModel(metaclass=TimestampMeta):
pass
class User(BaseModel):
def __init__(self, name):
self.name = name
u = User("Alice")
print(u._created_at) # 1640000000.123
Notice that _created_at is set when the class is defined, not when an instance is created. That means all instances of User share the same timestamp? Yes, because it is a class attribute. If you need per-instance timestamps, you need to hook into __init__ instead. But the metaclass runs before any instance exists, so you can also inject methods or modify existing ones.
Let me modify the previous example to rename all methods automatically:
class RenameMeta(type):
def __new__(mcs, name, bases, namespace):
new_namespace = {}
for key, value in namespace.items():
if callable(value) and not key.startswith('__'):
new_namespace['auto_' + key] = value
else:
new_namespace[key] = value
return super().__new__(mcs, name, bases, new_namespace)
class Service(metaclass=RenameMeta):
def run(self):
return "running"
s = Service()
print(s.auto_run()) # "running"
This is useful when you build plugins or DSLs that require all public methods to follow a naming convention. You can also add validation: raise an error if a class does not implement a required method.
Now, descriptors. Descriptors control how attributes are accessed and set on an instance. Properties are built-in descriptors, but you can create your own. The magic methods are __get__, __set__, and __delete__.
I once needed to ensure that certain attributes were always positive integers. I could have written a setter in every class, but that duplicates logic. Instead, I created a descriptor that validates on assignment.
class PositiveInteger:
def __init__(self):
self.data = {}
def __get__(self, obj, objtype=None):
if obj is None:
return self
return self.data.get(id(obj), None)
def __set__(self, obj, value):
if not isinstance(value, int) or value <= 0:
raise ValueError(f"{value} is not a positive integer")
self.data[id(obj)] = value
def __delete__(self, obj):
del self.data[id(obj)]
class ShoppingCart:
quantity = PositiveInteger()
price = PositiveInteger()
def __init__(self, quantity, price):
self.quantity = quantity
self.price = price
cart = ShoppingCart(3, 1500)
cart.quantity = 5 # OK
# cart.quantity = -1 # ValueError
You may wonder why I stored values in a dictionary keyed by id(obj). Because the descriptor is a class attribute shared by all instances. If I stored value directly, every instance would overwrite the same storage. Using id(obj) gives each instance its own slot. This is exactly how properties work under the hood.
I used this pattern later to build lazy-loaded attributes that compute once and cache the result.
class LazyProperty:
def __init__(self, func):
self.func = func
self.name = func.__name__
self.cache = {}
def __get__(self, obj, objtype=None):
if obj is None:
return self
if id(obj) not in self.cache:
self.cache[id(obj)] = self.func(obj)
return self.cache[id(obj)]
class Report:
def __init__(self, data):
self.data = data
@LazyProperty
def summary(self):
print("Computing summary (expensive)...")
return sum(self.data) * 2
r = Report([1, 2, 3])
print(r.summary) # computes and prints "Computing...", returns 12
print(r.summary) # returns 12, no print
Now, __getattr__ and __setattr__. These hooks allow you to intercept attribute access on instances. __getattr__ is called only when the attribute is not found through normal means. __getattribute__ is called for every attribute access, including existing ones. Use __getattr__ for dynamic proxies.
I used this to create a wrapper around a remote API. The wrapper does not know the method names in advance; it sends whatever method you call as a JSON request.
class RemoteAPI:
def __init__(self, base_url):
self.base_url = base_url
def __getattr__(self, name):
def method(*args, **kwargs):
print(f"Sending request to {self.base_url}/{name}")
# Simulate a real request
return {"method": name, "args": args, "kwargs": kwargs}
return method
api = RemoteAPI("https://api.example.com")
result = api.getUser(id=42)
print(result)
But be careful with recursion. If __getattr__ tries to access self.something that does not exist, it will call __getattr__ again infinitely. Always use object.__getattribute__ for default behavior.
class SafeProxy:
def __getattr__(self, name):
if name.startswith('_'):
raise AttributeError(name)
# fallback to default
return object.__getattribute__(self, name)
Now, dynamic code generation with exec and compile. This is the most dangerous but also the most powerful technique. You can build functions from strings defined in configuration files or databases.
I wrote a validation system once where the rules came from a YAML file. I needed to generate functions that check constraints without hardcoding them.
def make_validator(rules):
code_lines = ["def validator(value):"]
for rule in rules:
if rule['type'] == 'min_length':
code_lines.append(f" if len(value) < {rule['value']}:")
code_lines.append(f" raise ValueError('Too short')")
if rule['type'] == 'regex':
code_lines.append(f" import re")
code_lines.append(f" if not re.match(r'{rule['pattern']}', value):")
code_lines.append(f" raise ValueError('Invalid format')")
code_lines.append(" return value")
code = "\n".join(code_lines)
namespace = {}
exec(code, namespace)
return namespace['validator']
rules = [
{'type': 'min_length', 'value': 3},
{'type': 'regex', 'pattern': '^[a-zA-Z]+$'},
]
validator = make_validator(rules)
print(validator("abc")) # "abc"
# validator("ab") # raises ValueError
Always limit what exec can access. Supply an empty dictionary as the namespace to prevent the generated code from modifying your real variables. If you need to allow imports, whitelist them carefully.
The inspect module lets you examine functions and classes at runtime. You can extract argument names, defaults, annotations, and even source code. I used it to build an automatic input sanitizer that reads type hints.
import inspect
def typed(func):
sig = inspect.signature(func)
hints = func.__annotations__
def wrapper(*args, **kwargs):
bound = sig.bind(*args, **kwargs)
bound.apply_defaults()
for name, value in bound.arguments.items():
if name in hints:
expected = hints[name]
if not isinstance(value, expected):
raise TypeError(f"{name} must be {expected.__name__}, got {type(value).__name__}")
result = func(*args, **kwargs)
if 'return' in hints:
expected = hints['return']
if not isinstance(result, expected):
raise TypeError(f"Return must be {expected.__name__}, got {type(result).__name__}")
return result
return wrapper
@typed
def divide(a: int, b: int) -> float:
return a / b
print(divide(10, 3)) # 3.3333
# divide("10", 3) # TypeError
This is similar to how modern libraries enforce types, but with less overhead. You can extend it to coerce values, log calls, or cache results.
Now, dynamic class creation with type. Instead of writing class definitions in your source code, you can call type(name, bases, dict) to create a class at runtime. This is how ORMs like SQLAlchemy create models from table definitions.
I built a small configuration-driven data model system. The user provides a list of fields, and I generate a class with those fields, a constructor, and a readable representation.
def create_model(name, fields):
annotations = {}
init_lines = ["def __init__(self, {}):".format(
", ".join(f"{fname}: {ftype.__name__ if hasattr(ftype, '__name__') else ftype}" for fname, ftype, _ in fields)
)]
for fname, _, default in fields:
if default is not None:
init_lines.append(f" self.{fname} = {fname}")
else:
init_lines.append(f" self.{fname} = {fname}")
annotations[fname] = _
init_code = "\n".join(init_lines)
namespace = {}
exec(init_code, namespace)
def __repr__(self):
return f"{name}({', '.join(f'{k}={v!r}' for k,v in self.__dict__.items())})"
cls = type(name, (object,), {
'__annotations__': annotations,
'__init__': namespace['__init__'],
'__repr__': __repr__,
})
return cls
Person = create_model("Person", [
("name", str, None),
("age", int, None),
("active", bool, True)
])
p = Person(name="Bob", age=30, active=True)
print(p) # Person(name='Bob', age=30, active=True)
The exec inside create_model is safe because the namespace is empty and we control the code string completely. But you must be careful with user-supplied field names that could contain malicious strings.
Now, the ast module. Parsing Python source into an Abstract Syntax Tree lets you analyze and transform code before it runs. You can build static analyzers, linters, or simple optimizers.
I once wanted to precompute constant expressions in a large config file to speed up startup. I used ast.NodeTransformer to fold arithmetic expressions.
import ast
class Folder(ast.NodeTransformer):
def visit_BinOp(self, node):
self.generic_visit(node)
if isinstance(node.left, ast.Constant) and isinstance(node.right, ast.Constant):
if isinstance(node.op, ast.Add):
return ast.Constant(node.left.value + node.right.value)
elif isinstance(node.op, ast.Mult):
return ast.Constant(node.left.value * node.right.value)
return node
def compile_with_folding(source):
tree = ast.parse(source)
folded = Folder().visit(tree)
ast.fix_missing_locations(folded)
code = compile(folded, '<string>', 'exec')
ns = {}
exec(code, ns)
return ns
ns = compile_with_folding("x = 2 + 3 * 4")
print(ns['x']) # 14, computed at compile time
You can see the original 2 + 3 * 4 becomes 14 before execution. The ast module is also used by tools like Black and Pylint to understand your code without running it.
Finally, context managers and contextvars. Sometimes you need to pass a global-like state across function calls without using globals. contextvars is like a thread-local variable but working with async code too. Combined with contextmanager, you can create scoped resources.
I used this for a logging system where each request has a unique ID. Instead of passing the ID through every function, I set it in a context variable at the beginning of the request.
from contextvars import ContextVar
from contextlib import contextmanager
request_id = ContextVar('request_id', default=None)
@contextmanager
def request_context(rid):
token = request_id.set(rid)
try:
yield
finally:
request_id.reset(token)
def log(message):
rid = request_id.get()
if rid:
print(f"[{rid}] {message}")
else:
print(message)
with request_context("abc123"):
log("Processing payment") # [abc123] Processing payment
log("Outside context") # Outside context
Notice I did not use yield from or complicated setup. The contextmanager decorator turns a generator function into a context manager. The ContextVar provides a clean way to store per-context data.
These eight techniques have saved me from writing thousands of lines of repetitive code. Each one solves a different problem. Metaclasses for class creation hooks. Descriptors for attribute access control. __getattr__ for dynamic proxies. exec for generating functions from config. inspect for runtime introspection. type for dynamic classes. ast for code analysis. Context managers and contextvars for scoped state.
Start small. Pick one technique and try it in a side project. Add a metaclass that logs every class creation. Or create a descriptor that caps string length. You will see how Python bends to your will without sacrificing readability.
I once broke my entire test suite by misusing __getattr__ (infinite recursion, as I mentioned). Now I always add a guard. I also learned to never use exec on user input without sandboxing. But within controlled boundaries, these tools are safe and liberating.
If you write a framework, a library, or even a complex application, you will eventually need to generate code dynamically or intercept behavior you did not plan for. These patterns are the foundation of Python's flexibility. They are not magic tricks; they are deliberate design choices built into the language.
I hope you take the time to experiment with each one. Write a script that generates an entire ORM from a schema. Or build a proxy that mocks a web service. The code examples I gave are simple enough to run, but they are starting points. Modify them, break them, fix them. That is how you learn.
Remember, the goal is not to use metaprogramming everywhere. The goal is to know when it is the cleanest solution. For most code, plain functions and classes are enough. But when you face a problem that demands flexibility at runtime, these techniques will be your tools.
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | Java Elite Dev | Golang Elite Dev | Python Elite Dev | JS Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
Top comments (0)