DEV Community: michal puzanov

Python Dotenv Library: Advanced Topics (Part 2)

michal puzanov — Tue, 04 Aug 2026 10:04:30 +0000

This article continues the examination of advanced challenges associated with the Python Dotenv library, with a particular emphasis on production-oriented scenarios. Part 1 introduced production-grade architecture, advanced syntax, and typing considerations; this second part extends that discussion by examining additional practical dimensions.

The material is intended for a broad technical audience. Readers with varying levels of programming experience should be able to follow the discussion, although prior familiarity with Python is advantageous.

This article is organised around two principal topics.

Programmatic and alternative control
Security and best practices

Programmatic and alternative control

This section examines alternatives that may be employed during development. For example, variables can be loaded into a standard Python dictionary rather than injected into process-wide environment state; configuration can also be streamed from memory rather than read from a physical file. Finally, we consider the Dotenv command-line interface (CLI).

Isolated loading with dotenv_values()

In some cases, it is preferable to load a .env file directly into a standard Python dictionary. In practice, the return type is an OrderedDict, which can be accessed in the same way as a regular dictionary while preserving insertion order.

The principal motivation is to reduce global side effects and keep the process environment clean. This approach is particularly useful for testing and for multi-tenant applications that must pass distinct configurations to different modules.

The .env file

USER_ROLES='["admin", "editor", "viewer"]'

The Python code

import ast
from dotenv import dotenv_values

# Load environment variables from the .env file into a ordered dictionary
production_config = dotenv_values()

# Safely parse the string into a list object using ast.literal_eval
roles = ast.literal_eval(production_config.get("USER_ROLES"))
print(f"roles: {roles} (Type: {type(roles).__name__})")

# Print the type of the config object
print(f"config Type: {type(production_config).__name__}")

roles: ['admin', 'editor', 'viewer'] (Type: list)
config Type: OrderedDict

Stream loading

In this approach, a physical file is not read from disk; instead, a raw string or in-memory stream is supplied directly to load_dotenv().

This pattern is valuable in modern cloud environments (for example, AWS Lambda, Google Cloud Functions, and microservice platforms), where configuration strings may be retrieved dynamically from secure APIs or secret-management systems at runtime. In such deployments, no physical .env file may be present on disk.

The Python code

import io
import os
from dotenv import load_dotenv

# Pretend this string came from a secure cloud database or vault
secret_config_stream = "API_KEY=super_secret_token\nTIMEOUT=60"

# Load directly from the text stream
load_dotenv(stream=io.StringIO(secret_config_stream))

print(os.environ.get("API_KEY"))  # super_secret_token

super_secret_token

A combination of both methods may also be used: a .env file and an in-memory stream.

The .env file

SERVER_PORT=8080
MAX_CONNECTIONS=5

The Python code

import io
import os
from dotenv import load_dotenv

# Pretend this string came from a secure cloud database or vault
secret_config_stream = "API_KEY=super_secret_token\nTIMEOUT=60"

# Load .env file and directly from the text stream
load_dotenv(".env", stream=io.StringIO(secret_config_stream))

print("variables from .env file:")
print(os.environ.get("SERVER_PORT"))  # 8080 (example value from .env file)
print(os.environ.get("TIMEOUT"))  # 60 (from secret_config_stream)
print("variable from secret_config_stream:")
print(os.environ.get("API_KEY"))  # super_secret_token

variables from .env file:
8080
60
variable from secret_config_stream:
super_secret_token

The package-provided command-line tool (CLI)

Dotenv installs a command-line utility into the active Python environment.

This utility allows developers to execute terminal commands or scripts within the context of a specified environment file, without embedding configuration-loading logic directly in application code.

Installation using pip

pip install python-dotenv[cli]

The .env file

APP_ENV="development"
DB_HOST="localhost"

The Python code file test.py

import os

print("variables from .env file:")
print(os.environ.get("APP_ENV"))  # development (example value from .env file)
print(os.environ.get("DB_HOST"))  # localhost (example value from .env file)

Bash terminal

dotenv -f .env run python test.py

variables from .env file:
development
localhost

Working with multiline values

Handling multiline values is a common edge case in Python configuration workflows.

When developers attempt to store complex multiline data, such as RSA private keys, SSL certificates, or raw JSON payloads, Dotenv may parse the content incorrectly if formatting is not explicit. Consider the following example.

The .env file

# This will fail or only read the first line
PRIVATE_KEY=-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0Y...
...more lines...
-----END RSA PRIVATE KEY-----

This fails because Dotenv interprets each new line as a separate variable assignment. It reads the first line as a key-value pair and then misinterprets the remaining lines, which can lead to truncation or corruption.

Solution 1: Double quotes and literal newlines

A robust approach is to wrap the entire value in double quotes and preserve the physical line breaks.

The .env file

MY_PRIVATE_RSA_KEY="-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0Y...
...everything inside quotes...
-----END RSA PRIVATE KEY-----"

The Python code

import os
from dotenv import load_dotenv

load_dotenv()

private_key = os.environ.get("MY_PRIVATE_RSA_KEY")
print(private_key)

"-----BEGIN RSA PRIVATE KEY-----

What happened? The output is still wrong! The issue is while Dotenv natively parses literal multiline breaks locally, this pattern breaks immediately in the real world when passed through container configurations, CI/CD pipelines, or deployment platforms. Docker, Kubernetes, Vercel, and GitHub Actions generally do not support literal newlines inside environment values and will choke, truncate, or drop the variable.

Solution 2: Use the \n escape character - better solution

This will work all the time, but we would need to use a single-line .env style; line breaks may be represented using escaped \n characters.

The .env file

MY_ANOTHER_RSA_KEY="-----BEGIN RSA PRIVATE\nKEY-----\nMIIEowIBAAKCAQEA0Y...\n...everything inside quotes...\n-----END RSA

The Python code

import os
from dotenv import load_dotenv

load_dotenv()

private_key = os.environ.get("MY_ANOTHER_RSA_KEY")

print(private_key)

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0Y...
...everything inside quotes...
-----END RSA PRIVATE KEY-----

Using single and double quotes

The .env file

MY_SINGLE_QUOTE_RSA_KEY='-----BEGIN RSA PRIVATE\nKEY-----\nMIIEowIBAAKCAQEA0Y...\n...everything inside quotes...\n-----END RSA PRIVATE KEY-----'

The Python code

import os
from dotenv import load_dotenv

load_dotenv()

private_key = os.environ.get("MY_SINGLE_QUOTE_RSA_KEY")
print(private_key)

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0Y...
...everything inside quotes...
-----END RSA PRIVATE KEY-----

Security and best practices

This section introduces production-grade configuration validation using the widely adopted libraries Pydantic and Marshmallow, and then addresses import race conditions.

Validation

By default, python-dotenv loads values into os.environ as raw strings. If a key is missing or incorrectly formatted, Python typically raises an error only when that variable is used later at runtime. This behaviour can produce silent misconfiguration.

A fail-fast architecture mitigates this risk: validate all required configuration values and types immediately during application start-up.

The following .env settings are used in both validation examples.

The .env file

DB_PORT_VALIDATOR=6666
DEBUG_VALIDATOR=true
HTTP_URL_VALIDATOR=http://admin:SuperSecret123@localhost:5432/production_db
JWT_SECRET_KEY_VALIDATOR=complex_secret_key_123
DB_NAME_VALIDATOR=production_db

Strict validation using Pydantic

Pydantic is a strong choice because it integrates directly with environment variables through pydantic-settings and supports explicit, type-safe validation at application boundaries.

Install the package

pip install pydantic-settings

The Python code

from dotenv import load_dotenv
from pydantic import HttpUrl, SecretStr, Field
from pydantic_settings import BaseSettings, SettingsConfigDict

# 1. Load the raw strings into the environment
load_dotenv()

# 2. Define a strict schema with Pydantic
class ProductionConfig(BaseSettings):
    # Automatically casts string "5432" to int, fails if it's text
    DB_PORT_VALIDATOR: int = Field(default=5432, ge=1, le=65535) 

    # Enforces true boolean casting
    DEBUG_VALIDATOR: bool = False 

    # Validates proper URL structure
    HTTP_URL_VALIDATOR: HttpUrl 

    # Wraps sensitive keys so they don't accidentally leak in logs/print statements
    JWT_SECRET_KEY_VALIDATOR: SecretStr 

    # Read directly from os.environ
    model_config = SettingsConfigDict()

# 3. Validate at boot
try:
    production_config = ProductionConfig()
    print("✅ Configuration is valid. Booting application...")
except Exception as e:
    print(f"❌ CRITICAL: Configuration validation failed:\n{e}")

print(f"DB_PORT_VALIDATOR: {production_config.DB_PORT_VALIDATOR}, Type: {type(production_config.DB_PORT_VALIDATOR).__name__}")
print(f"DEBUG_VALIDATOR: {production_config.DEBUG_VALIDATOR}, Type: {type(production_config.DEBUG_VALIDATOR).__name__}")
print(f"HTTP_URL_VALIDATOR: {production_config.HTTP_URL_VALIDATOR}, Type: {type(production_config.HTTP_URL_VALIDATOR).__name__}")
print(f"JWT_SECRET_KEY_VALIDATOR: {production_config.JWT_SECRET_KEY_VALIDATOR}, Type: {type(production_config.JWT_SECRET_KEY_VALIDATOR).__name__}")
print(f"Number of keys in production_config: {len(production_config.model_dump())}")

✅ Configuration is valid. Booting application...
DB_PORT_VALIDATOR: 6666, Type: int
DEBUG_VALIDATOR: True, Type: bool
HTTP_URL_VALIDATOR: http://admin:SuperSecret123@localhost:5432/production_db, Type: HttpUrl
JWT_SECRET_KEY_VALIDATOR: **********, Type: SecretStr
Number of keys in production_config: 4

Strict validation using Marshmallow

This example uses a validation library different from Pydantic. Marshmallow is an effective alternative for schema validation over dictionary-like input. Consequently, we use dotenv_values() rather than loading values directly into the process environment.

The Python code

from dotenv import dotenv_values, find_dotenv
from marshmallow import Schema, fields, validate, ValidationError, EXCLUDE

# 1. Load .env from current working directory
raw_config = dotenv_values()

# 2. Define the Marshmallow validation schema
class ProductionConfig(Schema):
    class Meta:
        unknown = EXCLUDE

    DB_PORT_VALIDATOR = fields.Int(required=True, validate=validate.Range(min=1, max=65535))
    DEBUG_VALIDATOR = fields.Bool(load_default=False)
    HTTP_URL_VALIDATOR = fields.URL(required=True)
    JWT_SECRET_KEY_VALIDATOR = fields.Str(required=True, validate=validate.Length(min=16))

# 3. Validate and parse the data
try:
    production_config = ProductionConfig().load(raw_config)
    print("✅ Marshmallow validation passed.")
except ValidationError as err:
    print(f"❌ Configuration Error: {err.messages}")

print(f"DB_PORT_VALIDATOR: {production_config['DB_PORT_VALIDATOR']}, Type: {type(production_config['DB_PORT_VALIDATOR']).__name__}")
print(f"DEBUG_VALIDATOR: {production_config['DEBUG_VALIDATOR']}, Type: {type(production_config['DEBUG_VALIDATOR']).__name__}")
print(f"HTTP_URL_VALIDATOR: {production_config['HTTP_URL_VALIDATOR']}, Type: {type(production_config['HTTP_URL_VALIDATOR']).__name__}")
print(f"JWT_SECRET_KEY_VALIDATOR: {production_config['JWT_SECRET_KEY_VALIDATOR']}, Type: {type(production_config['JWT_SECRET_KEY_VALIDATOR']).__name__}")
print(f"Number of keys in config: {len(production_config)}")

✅ Marshmallow validation passed.
DB_PORT_VALIDATOR: 6666, Type: int
DEBUG_VALIDATOR: True, Type: bool
HTTP_URL_VALIDATOR: http://admin:SuperSecret123@localhost:5432/production_db, Type: str
JWT_SECRET_KEY_VALIDATOR: complex_secret_key_123, Type: str
Number of keys in config: 4

The examples illustrate several implementation differences between Pydantic settings and Marshmallow schemas. In Pydantic settings, model_config = SettingsConfigDict() is used to control configuration behaviour. In Marshmallow, equivalent behaviour is configured through the Meta subclass. In this example, unknown = EXCLUDE instructs Marshmallow to ignore input keys that are not explicitly declared in the schema.

Issues with automated loading (import race condition)

A common anti-pattern is to load environment variables in a module that is itself imported by other modules.

This can create a race condition in which dependent modules read configuration values before load_dotenv() has executed. The result is a class of failures commonly described as import race conditions.

In Python, when a module is imported, top-level code in that file executes immediately and typically only once per interpreter session. If modules depend on environment variables but load_dotenv() has not yet run, the application may read None or empty values.

The wrong architecture

The Python code

# app/database.py
import os
# The developer assumes this is fine because it is near the variable usage
DB_CONN = os.environ.get("DATABASE_URL")

# main.py
from app.database import DB_CONN  # 1. Python executes database.py now; DB_CONN becomes None.
from dotenv import load_dotenv

load_dotenv()                     # 2. Too late: environment loading occurs after database.py executed.

If the application attempts to use DB_CONN, it may fail with a connection error or a NoneType-related exception because initialisation order is incorrect.

The solution

To ensure deterministic behaviour, establish a strict architectural rule: call load_dotenv() explicitly at the top of the application entry point (for example, main.py, wsgi.py, or asgi.py), before importing local application modules.

The Python code

# main.py
from dotenv import load_dotenv

# 1. Load environment variables before anything else
load_dotenv()

# 2. It is now safe to import application components
from app.database import DB_CONN
from app.server import start_server

if __name__ == "__main__":
    start_server()

Conclusion and recommendations

This article has examined advanced operational patterns for Dotenv in production-oriented Python systems. The discussion demonstrated that configuration management should be treated as a first-class engineering concern rather than a peripheral implementation detail. In particular, isolated loading with dotenv_values(), stream-based ingestion, and CLI-based execution each provide context-dependent advantages when controlling configuration scope and reducing unintended global side effects.

The analysis of multiline values highlighted a frequent source of configuration defects and showed that explicit quoting and escape strategy are essential for reliability when handling cryptographic material or other structured secrets. Furthermore, the validation examples using Pydantic and Marshmallow illustrated the practical value of fail-fast design: configuration should be parsed, typed, and verified at application start-up so that invalid state is rejected before business logic executes. Finally, the import race condition example reinforced the importance of deterministic initialisation order, with load_dotenv() executed at the entry point prior to importing dependent modules.

Here are some recommendations which I found helpful for Dotenv.

Adopt a fail-fast configuration policy and validate all critical settings during application bootstrap.
Prefer typed validation frameworks, such as Pydantic or Marshmallow, for production workloads where correctness and observability are required.
Use dotenv_values() when local, non-global configuration scoping is needed, particularly in tests and multi-tenant execution contexts.
Use stream loading for cloud-native deployments where configuration is retrieved dynamically from secret managers or secure APIs.
Standardise multiline secret formatting conventions across teams to avoid truncation and parsing ambiguities.
Enforce a strict import order in entry-point modules: call load_dotenv() before importing application components that consume environment variables.
Where operationally appropriate, use the Dotenv CLI to separate runtime configuration concerns from application source code.

Did this help you transition to Dotenv? Let me know in the comments below, and don't forget to drop a like if you enjoyed the read! Thank you.

Python Dotenv Library: Advanced Topics (Part 1)

michal puzanov — Tue, 04 Aug 2026 09:59:29 +0000

This article examines advanced challenges associated with the Python Dotenv library and presents practical strategies for addressing them. Although introductory usage is relatively straightforward, production-oriented scenarios often require additional design considerations. Typical examples include managing multiple .env files and maintaining separate configurations for development and production environments. These concerns motivate a more systematic discussion of advanced usage patterns.

The material is written for a broad technical audience. Readers with different levels of programming experience can follow the discussion, although familiarity with Python is beneficial.

This article is organised around two principal topics.

Production-grade architecture
Advanced syntax and typing

Production-grade architecture

This section analyses architectural considerations relevant to using Dotenv in production-focused applications.

Using the .env template pattern

Dotenv was designed primarily to manage sensitive configuration values, such as database credentials and API keys. Although it can also store non-sensitive settings, secure secret management remains its primary purpose. Consequently, the real .env file should never be committed to version-control platforms such as GitHub, GitLab, or Bitbucket, because repository access may expose credentials to unauthorised parties.

A common practical question is how collaborators can know which variables must be defined. The standard solution is to maintain a template file that documents required keys and expected formats.

The real .env file

Stored only on the local machine (for example, excluded through .gitignore).

The real .env file

DATABASE_URL=postgresql://admin:super_secret_password_123@localhost:5432/mydb
STRIPE_API_KEY=sk_live_51Nx...real_secret_key...
DEBUG=True

The template .env.example file

Safe to commit to source control.

The template .env.example

# This is a template. Copy this to a file named '.env' and fill in your real values.
DATABASE_URL=postgresql://username:password@localhost:5432/database_name
STRIPE_API_KEY=your_stripe_test_key_here
DEBUG=False

Multi-environment workflows

A multi-environment workflow enables a Python application to select configuration automatically based on its runtime context (for example, local development, staging, or production). Instead of manually editing a single .env file during deployment, separate files are maintained for each environment, and a selector variable, commonly APP_ENV, determines which file is loaded. The following example demonstrates this pattern in practice.

Step 1: Create environment-specific files

env.development - for local development

DATABASE_URL=postgresql://localhost/dev_db
DEBUG=True

.env.production - for production deployment

DATABASE_URL=postgresql://secure-cloud-cluster/prod_db
DEBUG=False

Step 2: Dynamic script loading

In the application entry point (for example, main.py or config.py), read the APP_ENV system variable before loading python-dotenv. If APP_ENV is not defined, the script falls back to the default value "development".

import os
from pathlib import Path
from dotenv import load_dotenv

# 1. Determine which environment we are in (defaults to 'development')
app_env = os.environ.get("APP_ENV", "development").lower()

# 2. Construct the filename (e.g., '.env.development' or '.env.production')
env_filename = f".env.{app_env}"
base_dir = Path.cwd()
env_path = base_dir / env_filename

# 3. Check if the file exists, then load it
if env_path.exists():
    print(f"Loading configuration from: {env_filename}")
    load_dotenv(dotenv_path=env_path)
else:
    print(f"Warning: Config file {env_filename} not found in {base_dir}. Using system defaults.")

# 4. Access your variables as normal
database_url = os.environ.get("DATABASE_URL")
print(f"Database URL in use: {database_url}")

Loading configuration from: .env.development
Database URL in use: postgresql://localhost/dev_db

Hierarchical loading and override behaviour

By default, Dotenv does not overwrite variables that already exist in the process environment. In cloud-oriented deployments, however, such as Docker or Kubernetes, explicit override control is often required. This behaviour is managed through the override parameter. The examples below reuse the .env.development and .env.production files introduced earlier.

Default behaviour (override=False)

import os
from pathlib import Path
from dotenv import load_dotenv

# 1. Construct the filename (env files are in current working directory)
env_filename = [".env.development", ".env.production"]  
base_dir = Path.cwd()

for filename in env_filename:
    env_path = base_dir / filename
    # 2. Check if the file exists, then load it
    if env_path.exists():
        print(f"Loading configuration from: {filename}")
        # 3. Load the .env file with no overrides (existing environment variables will not be overwritten)
        load_dotenv(dotenv_path=env_path)
    else:
        print(f"Warning: Config file {filename} not found in {base_dir}. Using system defaults.")

# 4. Access your variables as normal
database_url = os.environ.get("DATABASE_URL")
print(f"Database URL in use: {database_url}")

Loading configuration from: .env.development
Loading configuration from: .env.production
Database URL in use: postgresql://localhost/dev_db

In this case, the value was not overridden. The application still resolves the value defined in .env.development.

Advanced behaviour (override=True)

import os
from pathlib import Path
from dotenv import load_dotenv

# 1. Construct the filename (env files are in current working directory)
env_filename = [".env.development", ".env.production"]  
base_dir = Path.cwd()

for filename in env_filename:
    env_path = base_dir / filename
    # 2. Check if the file exists, then load it
    if env_path.exists():
        print(f"Loading configuration from: {filename}")
        # 3. Load the .env file with overrides (existing environment variables will be overwritten)
        load_dotenv(dotenv_path=env_path, override=True)
    else:
        print(f"Warning: Config file {filename} not found in {base_dir}. Using system defaults.")

# 4. Access your variables as normal
database_url = os.environ.get("DATABASE_URL")
print(f"Database URL in use: {database_url}")

Loading configuration from: .env.development
Loading configuration from: .env.production
Database URL in use: postgresql://secure-cloud-cluster/prod_db

In this configuration, the value is overridden. The final value is taken from .env.production.

Hierarchy problem

In many real-world systems, local .env values should never override secrets provided by production infrastructure (for example, Docker, Kubernetes, or managed secret stores). A practical mitigation is to adopt an explicit naming convention that distinguishes locally overrideable values from infrastructure-controlled values.

Naming strategy (intent-based prefixes)

Use LOCAL_ prefixes for values intended only for local development.
Use SYS_ or K8S_ prefixes for values injected and managed by infrastructure.

# In .env (Local development)
LOCAL_MOCK_PAYMENTS=True

# In Kubernetes Deployment (Production)
SYS_LIVE_PAYMENT_ROUTER=https://stripe.com

Advanced syntax and typing

This section focuses on writing cleaner configuration files using key-value composition patterns and addressing a fundamental limitation: environment variables are always read as strings.

Variable expansion (interpolation)

Repeated literals in .env files reduce maintainability. Dotenv supports variable expansion through the ${VARIABLE_NAME} syntax, which allows dependent values to be composed from reusable primitives.

Problem: If a core database value changes, multiple duplicated strings may require manual updates.
Solution: Define core components once and construct composite values dynamically.

The .env file

DB_USER=postgres_admin
DB_PASS=SuperSecret123
DB_HOST=localhost
DB_PORT=5432
DB_NAME=production_db

# Construct the full connection string using expansion
LOCAL_DATABASE_URL=postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}/${DB_NAME}

The Python code

To enable this behaviour in Python, ensure interpolation is active (it is enabled by default in load_dotenv).

import os
from dotenv import load_dotenv

load_dotenv()
# Returns the fully constructed URL automatically
print(os.environ.get("LOCAL_DATABASE_URL"))

postgresql://postgres_admin:SuperSecret123@localhost:5432/production_db

The string-to-boolean pitfall

One of the most common issues in environment-based configuration is that all environment values are strings at read time. This behaviour frequently causes logical errors when developers expect textual values such as "true" or "false" to behave as native booleans. Although this topic was introduced in the previous article, it warrants emphasis because it remains a frequent source of bugs.

The .env file

DEBUG=true
ENABLE_SIGNUPS=false

The Python code

import os
# os.environ.get returns the STRING "true"
if os.environ.get("DEBUG"): 
    print("Debug is active!")  # This will run.
# os.environ.get returns the STRING "false"
if os.environ.get("ENABLE_SIGNUPS"): 
    print("Signups are enabled!")  # This will run!

Debug is active!
Signups are enabled!

This behaviour follows Python's truth-value rules for strings: any non-empty string evaluates to True, whereas only an empty string evaluates to False.

The Python code

value1 = "true"
value2 = "false"
value3 = "1"
value4 = "0"
value5 = ""

print(bool(value1), bool(value2), bool(value3), bool(value4), bool(value5))

True True True True False

A robust approach is to use a utility that converts string representations of booleans into native boolean types.
In this example, JSON parsing is used for deterministic conversion.

The Python code

import os
import json
from dotenv import load_dotenv

# we need to override the existing environment variables to ensure that the values from the .env file are used
load_dotenv(override=True)

# json.loads converts "true" to True, and "false" to False
is_debug = json.loads(os.getenv("DEBUG", "false").lower())
enable_signups = json.loads(os.getenv("ENABLE_SIGNUPS", "false").lower())

print(f"Debug mode: {is_debug} (Type: {type(is_debug).__name__})")
print(f"Signups enabled: {enable_signups} (Type: {type(enable_signups).__name__})")

Debug mode: True (Type: bool)
Signups enabled: False (Type: bool)

Python integration with Pydantic

Manually converting every environment string into types such as int, float, or bool becomes difficult to maintain at scale. The pydantic-settings package automates parsing and validation, improving reliability and reducing configuration boilerplate. In the following example, we reuse the previous variables and add a floating-point timeout value.

This package is not included in the Python standard library and must be installed separately.

Install the package with pip

pip install pydantic-settings

The .env file

DEBUG=true
ENABLE_SIGNUPS=false
TIMEOUT_LIMIT=4.5

The Python code

import os
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    # Tell Pydantic to read from the .env file
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    # Pydantic will automatically look for these keys and cast them
    DEBUG: bool = False
    ENABLE_SIGNUPS: bool = False
    TIMEOUT_LIMIT: float = 30.0  # Default timeout limit in seconds

    # Initialize and validate everything at application startup
try:
    config = Settings()

    is_debug = config.DEBUG
    enable_signups = config.ENABLE_SIGNUPS
    timeout_limit = config.TIMEOUT_LIMIT

    timeout_limit_env = os.getenv("TIMEOUT_LIMIT")

    print(f"Debug mode: {is_debug} (Type: {type(is_debug).__name__})")
    print(f"Signups enabled: {enable_signups} (Type: {type(enable_signups).__name__})")
    print(f"Timeout limit: {timeout_limit} seconds (Type: {type(timeout_limit).__name__})")
    print(f"Timeout limit from environment variable: {timeout_limit_env} (Type: {type(timeout_limit_env).__name__})")
except Exception as e:
    print(f"Configuration validation failed: {e}")

Debug mode: True (Type: bool)
Signups enabled: False (Type: bool)
Timeout limit: 4.5 seconds (Type: float)
Timeout limit from environment variable: 4.5 (Type: str)

As shown above, values are also accessible through standard Python environment access patterns, while Pydantic simultaneously provides structured parsing, validation, and default handling.

The Python code - validation example

import os
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    # Tell Pydantic to read from the .env file
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    # Pydantic will automatically look for these keys and cast them
    DEBUG: bool = False
    ENABLE_SIGNUPS: bool = False
    TIMEOUT_LIMIT: int = 30  # Default timeout limit in seconds

    # Initialize and validate everything at application startup
try:
    config = Settings()

    is_debug = config.DEBUG
    enable_signups = config.ENABLE_SIGNUPS
    timeout_limit = config.TIMEOUT_LIMIT

    print(f"Debug mode: {is_debug} (Type: {type(is_debug).__name__})")
    print(f"Signups enabled: {enable_signups} (Type: {type(enable_signups).__name__})")
    print(f"Timeout limit: {timeout_limit} seconds (Type: {type(timeout_limit).__name__})")
except Exception as e:
    print(f"Configuration validation failed: {e}")

Configuration validation failed: 1 validation error for Settings
TIMEOUT_LIMIT
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='4.5', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/int_parsing

The Python code - default value example


import os
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    # Tell Pydantic to read from the .env file
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    # Pydantic will automatically look for these keys and cast them
    DEBUG: bool = False
    ENABLE_SIGNUPS: bool = False
    TTTIMEOUT_LIMIT: float = 30.0  # Default timeout limit in seconds

    # Initialize and validate everything at application startup
try:
    config = Settings()

    is_debug = config.DEBUG
    enable_signups = config.ENABLE_SIGNUPS
    timeout_limit = config.TTTIMEOUT_LIMIT

    print(f"Debug mode: {is_debug} (Type: {type(is_debug).__name__})")
    print(f"Signups enabled: {enable_signups} (Type: {type(enable_signups).__name__})")
    print(f"Timeout limit: {timeout_limit} seconds (Type: {type(timeout_limit).__name__})")
except Exception as e:
    print(f"Configuration validation failed: {e}")

Debug mode: True (Type: bool)

Signups enabled: False (Type: bool)

Timeout limit: 30.0 seconds (Type: float)

Conclusion and Recommendation

This article examined advanced usage patterns for Python Dotenv in production-oriented contexts. The discussion covered secure template-based workflows, multi-environment configuration loading, override semantics, variable interpolation, and safe type conversion strategies. Together, these practices improve both configuration reliability and operational security.

From an engineering perspective, the key insight is that environment management should be treated as part of system architecture rather than as a minor implementation detail. Poorly structured configuration logic can produce subtle failures, especially when applications move from local development to containerised or cloud environments.

The following recommendations are strongly advised for maintainable and secure deployment pipelines:

Maintain a strict separation between secrets and templates by committing only .env.example and excluding real .env files from version control.
Adopt explicit environment-specific files (for example, .env.development and .env.production) and select them dynamically through a control variable such as APP_ENV.
Use override behaviour deliberately and document precedence rules to prevent local values from unintentionally replacing infrastructure-managed secrets.
Apply naming conventions (for example, LOCAL_, SYS_, K8S_) to clarify variable ownership and reduce configuration ambiguity.
Avoid direct truthiness checks for string-based environment values; always perform explicit type conversion.
For larger applications, prefer schema-driven configuration with pydantic-settings to centralise parsing, validation, and default handling.

In summary, advanced Dotenv usage is most effective when combined with disciplined configuration design. Organisations that standardise these patterns can reduce runtime errors, strengthen secret hygiene, and improve reproducibility across development, staging, and production environments.

Did this help you transition to Dotenv? Let me know in the comments below, and don't forget to drop a like if you enjoyed the read! Thank you.

Python Dotenv Library Introduction

michal puzanov — Tue, 04 Aug 2026 09:56:58 +0000

What is Dotenv

Dotenv is a lightweight software library for Python and Node.js used by developers to manage application settings, secrets, and credentials outside of their source code. It automatically loads configuration options from a simple text file (usually named .env) and injects them directly into your application's runtime memory. This means Dotenv does not write variables to a permanent system file; it reads key-value pairs from your local text file and injects them into the runtime memory (RAM) of your active application process. In this article, we focus on the Python version.

How does it work

Source File: It reads your project configuration from a text file (usually named .env)
Destination: It loads these variables into the operating system's process environment. In Python, they are accessed via os.environ.
Lifespan: The variables only exist while your application is actively running. Once you stop the server or program, the variables are cleared from memory. This is a significant improvement over storing variables in system files.

Dotenv Python installation via Terminal

The correct package name is python-dotenv; the original project name dotenv has been deprecated.

Using PyPI

pip install python-dotenv

Using a specific Python 3 version

pip3 install python-dotenv

Using Poetry

poetry add python-dotenv

Using UV

uv pip install python-dotenv

Verify the installation

We are going to create a quick test script to ensure it is installed correctly.

import os
from dotenv import load_dotenv

# Load the variables from your .env file
load_dotenv()

# Verify by checking your environment path or a specific variable
print("dotenv loaded successfully!")

dotenv loaded successfully!

Using Dotenv in Python code

Dotenv stores key-value records, and it can automatically recognise string, int, and float variable types. That does not mean you cannot store other types like bool, tuple, list, or dict, but you need to parse key-value pairs and transform them into specific formats. We are going to look at all examples.

String, Integer and Float variables

The .env file

We write variables in the standard way, as we do in Python code.

APP_ENV="development"
DB_HOST="localhost"
SERVER_PORT=8080
MAX_CONNECTIONS=5
TIMEOUT_LIMIT=4.5
TAX_RATE=0.15

#### The Python code ```python import os from dotenv import load_dotenv # Load variables into memory load_dotenv() # 1. Fetching Strings (No conversion needed) env_name = os.getenv("APP_ENV") print(f"Environment: {env_name} (Type: {type(env_name).__name__})") # 2. Fetching Integers (Convert using int) port = int(os.getenv("SERVER_PORT")) print(f"Port: {port} (Type: {type(port).__name__})") # 3. Fetching Floats (Convert using float) timeout = float(os.getenv("TIMEOUT_LIMIT")) print(f"Timeout: {timeout} (Type: {type(timeout).__name__})") ``` Environment: development (Type: str) Port: 8080 (Type: int) Timeout: 4.5 (Type: float) ### Boolean variables We can store boolean values (True/False) in a .env file, but just like numbers, Dotenv reads them strictly as strings ("true" or "false"). We have to convert them to actual booleans in your Python code. The issue is, you cannot use the standard bool() function because Python evaluates any non-empty string as True, even if the value is "False". ```python value1 = "true" value2 = "false" value3 = "1" value4 = "0" value5 = "" print(bool(value1), bool(value2), bool(value3), bool(value4), bool(value5)) ``` True True True True False #### The .env file

DEBUG=true
ENABLE_SIGNUPS=false

The Python code (2 ways to parse)

Method A: string comparison

Compare the incoming string directly against "true".

import os
from dotenv import load_dotenv

load_dotenv()

# Lowercase the string to prevent casing issues
is_debug = os.getenv("DEBUG", "false").lower() == "true"
enable_signups = os.getenv("ENABLE_SIGNUPS", "false").lower() == "true"

print(f"Debug mode: {is_debug} (Type: {type(is_debug).__name__})")
print(f"Enable signups: {enable_signups} (Type: {type(enable_signups).__name__})")

Debug mode: True (Type: bool)
Enable signups: False (Type: bool)

Method B: JSON parsing (best for strict validation)

Because "true" and "false" are valid JSON tokens, you can use Python's built-in json.loads() to convert them perfectly.

import os
import json
from dotenv import load_dotenv

load_dotenv()

# json.loads converts "true" to True, and "false" to False
is_debug = json.loads(os.getenv("DEBUG", "false").lower())
enable_signups = json.loads(os.getenv("ENABLE_SIGNUPS", "false").lower())

print(f"Debug mode: {is_debug} (Type: {type(is_debug).__name__})")
print(f"Signups enabled: {enable_signups} (Type: {type(enable_signups).__name__})")

Debug mode: True (Type: bool)
Signups enabled: False (Type: bool)

Tuple variables

We have several possibilities for how to store and parse tuple variables in Dotenv.

Method A: comma-separated values

Store the items separated by commas. You do not need to include parentheses in the .env file.

The .env file

ALLOWED_COLORS="red,green,blue"
DATABASE_COORDINATES="45.12,-122.33"

#### The Python code ```python import os from dotenv import load_dotenv load_dotenv() # Parsing a tuple of strings colors = tuple(os.getenv("ALLOWED_COLORS").split(",")) print(f"colors: {colors} (Type: {type(colors).__name__})") # Parsing a tuple of floats (using a list comprehension) coords = tuple(float(x) for x in os.getenv("DATABASE_COORDINATES").split(",")) print(f"coords: {coords} (Type: {type(coords).__name__})") ``` colors: ('red', 'green', 'blue') (Type: tuple) coords: (45.12, -122.33) (Type: tuple) #### Method B: Literal Evaluation using ast library (best for explicit tuple syntax) - The .env file

ADMIN_IDS="(1001, 1002, 1003)"

The Python code

import os
import ast
from dotenv import load_dotenv

load_dotenv()

# Safely evaluate the string as a Python tuple object
admin_ids = ast.literal_eval(os.getenv("ADMIN_IDS"))

print(f"admin_ids: {admin_ids} (Type: {type(admin_ids).__name__})")

admin_ids: (1001, 1002, 1003) (Type: tuple)

List variables

We can use exactly the same approaches for list variables.

Method A: comma-separated values

Store the items separated by commas. You do not need to include parentheses in the .env file.

The .env file

FRUITS="apple,banana,cherry"
PRIME_NUMBERS="2,3,5,7"

The Python code

Using .split(",") automatically creates a Python list object. You do not need to wrap it in anything extra.

import os
from dotenv import load_dotenv

load_dotenv()

# 1. List of strings
fruit_list = os.getenv("FRUITS").split(",")
print(f"fruit_list: {fruit_list} (Type: {type(fruit_list).__name__})")

# 2. List of integers (using list comprehension)
primes = [int(x) for x in os.getenv("PRIME_NUMBERS").split(",")]
print(f"primes: {primes} (Type: {type(primes).__name__})")

fruit_list: ['apple', 'banana', 'cherry'] (Type: list)
primes: [2, 3, 5, 7] (Type: list)

Method B: Literal evaluation using the ast library (best for explicit list syntax)

The .env file

USER_ROLES='["admin", "editor", "viewer"]'

The Python code

Use ast.literal_eval() or json.loads(). Both will read the brackets and instantly convert the string into a native Python list.

import os
import ast
import json
from dotenv import load_dotenv

load_dotenv()

# Safely parse the string into a list object using ast.literal_eval
roles = ast.literal_eval(os.getenv("USER_ROLES"))
print(f"roles: {roles} (Type: {type(roles).__name__})")

# Safely parse the string into a list object using json.loads
roles_json = json.loads(os.getenv("USER_ROLES"))
print(f"roles_json: {roles_json} (Type: {type(roles_json).__name__})")

roles: ['admin', 'editor', 'viewer'] (Type: list)
roles_json: ['admin', 'editor', 'viewer'] (Type: list)

Dict variables

Let us look at three methods for storing dictionaries in a Dotenv environment.

Method A: string parsing using JSON

Store the dictionary as a string in the .env file.

The .env file for JSON - the dictionary must be in JSON format

APP_CONFIG_JSON='{"theme": "dark", "retries": 3, "debug": true}'

The Python code

import os
import json
from dotenv import load_dotenv

load_dotenv()

# Parse the string into a real Python dictionary
config = json.loads(os.getenv("APP_CONFIG_JSON"))

print(f"config: {config} (Type: {type(config).__name__})")
print(f"Theme: {type(config['theme']).__name__}, Retries: {type(config['retries']).__name__}, Debug: {type(config['debug']).__name__}")

config: {'theme': 'dark', 'retries': 3, 'debug': True} (Type: dict)
Theme: str, Retries: int, Debug: bool

Method B: string parsing using Ast library

The .env file for Ast - the dictionary is in native Python format

APP_CONFIG='{"theme": "dark", "retries": 3, "debug": True}'

The Python code

import os
import ast
from dotenv import load_dotenv

load_dotenv()

# Safely parse the Python-formatted string into a dict
config = ast.literal_eval(os.getenv("APP_CONFIG"))

print(f"config: {config} (Type: {type(config).__name__})")
print(f"Theme: {type(config['theme']).__name__}, Retries: {type(config['retries']).__name__}, Debug: {type(config['debug']).__name__}")

config: {'theme': 'dark', 'retries': 3, 'debug': True} (Type: dict)
Theme: str, Retries: int, Debug: bool

Method C: using a prefix-based flat structure - best used for simple configurations.

The .env file

DB_HOST="localhost"
DB_PORT=5432
DB_USER="admin"

The Python code

import os
from dotenv import load_dotenv

load_dotenv()

# Build the dictionary manually
database_config = {
    "host": os.getenv("DB_HOST"),
    "port": int(os.getenv("DB_PORT", 5432)), # Cast to int safely
    "user": os.getenv("DB_USER")
}

print(f"database_config: {database_config} (Type: {type(database_config).__name__})")
print(f"Host: {type(database_config['host']).__name__}, Port: {type(database_config['port']).__name__}, User: {type(database_config['user']).__name__}")

database_config: {'host': 'localhost', 'port': 5432, 'user': 'admin'} (Type: dict)

Host: str, Port: int, User: str

Conclusion and recommendations

In this article, we covered the core idea of Dotenv: keep configuration and secrets outside your source code, then load them safely at runtime.

Key takeaways:

Dotenv reads everything as strings, so explicit parsing is required for int, float, bool, tuple, list, and dict values.
For structured values, json.loads() is a strong default when your .env value is valid JSON.
ast.literal_eval() is useful for Python-literal formats, but use it only with trusted input.
Prefix-based flat keys (like DB_HOST, DB_PORT) are often the simplest and most maintainable pattern for real projects.

Recommended practice for production:

Keep .env files out of version control (.gitignore).
Store only environment-specific configuration in .env, not hardcoded values in application code.
Define clear defaults and validation in your Python startup logic to avoid runtime surprises.

With these patterns, Dotenv becomes a clean, safe, and scalable way to manage app configuration across local development, staging, and production environments.

Did this help you transition to Dotenv? Let me know in the comments below, and don't forget to drop a like if you enjoyed the read! Thank you.

Python SciPy Library

michal puzanov — Tue, 04 Aug 2026 09:52:38 +0000

SciPy (Scientific Python) is a free and open-source Python library for scientific, mathematical, and technical computing. It provides a broad collection of high-level numerical algorithms and utility functions designed to support complex computation efficiently.

This notebook offers an introductory overview of SciPy and aims to provide foundational orientation for learners who are beginning to use the library.

SciPy is widely used in both academic research and industry. It is applied by data scientists, machine learning engineers, financial analysts, physicists, and mechanical and electrical engineers to model real-world systems and analyse empirical data.

Key Characteristics

Built on NumPy: SciPy is constructed directly on top of NumPy and uses NumPy arrays as its core data structure.
High Performance: Although SciPy is used through Python, many core algorithms are implemented in optimised low-level languages such as C, C++, and Fortran.
Core Ecosystem Component: SciPy forms a central part of the scientific Python ecosystem and integrates naturally with libraries such as pandas, Matplotlib, and scikit-learn.

Core Features and Capabilities

SciPy is organised into specialised subpackages, each addressing a specific mathematical or engineering domain. Major subpackages include:

scipy.optimize: Function minimisation and maximisation, curve fitting, and root finding.
scipy.integrate: Numerical integration and ordinary differential equation (ODE) solvers.
scipy.stats: Probability distributions, descriptive statistics, and hypothesis testing (for example, t-tests).
scipy.linalg: Advanced linear algebra, including matrix decompositions (LU, QR, SVD) and eigenvalue solvers.
scipy.signal and scipy.fft: Digital signal processing, filtering, convolution, and Fast Fourier Transform methods.
scipy.interpolate: Estimation of intermediate values between known data points, including spline and linear interpolation.
scipy.sparse: Memory-efficient sparse matrix structures and related solvers.

SciPy vs NumPy

Core focus:
- NumPy - array data structures and fundamental array operations.
- SciPy - advanced scientific and engineering algorithms.
Functionality:
- NumPy - indexing, sorting, reshaping, and basic linear algebra.
- SciPy - optimalisation, signal processing, integration, and statistical analysis
Relatioship:
- NumPy - fondational library
- SciPy - built directly on top of NumPy

Practical Examples

Installation

Using pip

pip install scipy

Using conda

conda install scipy

Using poetry

poetry add scipy

Verifying the Installation

from scipy import __version__ as scipy_version

# Check SciPy version
print(f"SciPy version: {scipy_version}")

SciPy version: 1.18.0

Constants and Special Functions

from scipy import constants, special

Mathematical and Physical Constants

print("Pi:", constants.pi)
print("Golden ratio:", constants.golden_ratio)
print("Speed of light (m/s):", constants.speed_of_light)

Pi: 3.141592653589793
Golden ratio: 1.618033988749895
Speed of light (m/s): 299792458.0

Representative Special Functions

print("Factorial(5):", special.factorial(5))
print("Combinations C(6, 2):", special.comb(6, 2))
print("Gamma(5):", special.gamma(5))
print("Bessel J0 at x=1:", special.jv(0, 1.0))
print("Error function erf(1):", special.erf(1.0))

Factorial(5): 120.0
Combinations C(6, 2): 15.0
Gamma(5): 24.0
Bessel J0 at x=1: 0.7651976865579666
Error function erf(1): 0.8427007929497148

Linear abgebra

from scipy import linalg
import numpy as np

a = np.array([[3.0, 2.0], [1.0, 4.0]])
b = np.array([[7.0], [5.0]])

Solving Linear Systems

solution = linalg.solve(a, b)

print("Solution of the linear system Ax = b:")
print(solution)

Solution of the linear system Ax = b:
[[1.8]
 [0.8]]

Matrix Properties

print("Determinant of matrix a:", linalg.det(a))
print("Inverse of matrix a:")
print(linalg.inv(a))

Determinant of matrix a: 10.0
Inverse of matrix a:
[[ 0.4 -0.2]
 [-0.1  0.3]]

Matrix Decompositions

p, l, u = linalg.lu(a)

print("LU decomposition of matrix a:")
print("P matrix:")
print(p)
print("\nL matrix:")
print(l)
print("\nU matrix:")
print(u)

u_matrix, s, vh = linalg.svd(a)

print("\nSingular Value Decomposition of matrix a:")
print("U matrix:")
print(u_matrix)
print("\nSingular values:")
print(s)
print("\nV^H matrix:")
print(vh)

LU decomposition of matrix a:
P matrix:
[[1. 0.]
 [0. 1.]]

L matrix:
[[1.         0.        ]
 [0.33333333 1.        ]]

U matrix:
[[3.         2.        ]
 [0.         3.33333333]]

Singular Value Decomposition of matrix a:
U matrix:
[[-0.64074744 -0.76775173]
 [-0.76775173  0.64074744]]

Singular values:
[5.11667274 1.95439508]

V^H matrix:
[[-0.52573111 -0.85065081]
 [-0.85065081  0.52573111]]

Optimisation

from scipy import optimize

def objective_function(x):
    return (x - 3) ** 2 + 2

Scalar Function Minimisation

min_result = optimize.minimize_scalar(objective_function)

print("\nOptimisation result:")
print("Optimal x:", min_result.x)
print("Objective function value at optimal x:", round(min_result.fun, 4))

Optimisation result:
Optimal x: 3.0
Objective function value at optimal x: 2.0

Root Finding

root_result = optimize.root_scalar(lambda x: x ** 3 - 2, bracket=[0, 2])

print("Cube root of 2:", round(root_result.root, 6))

Cube root of 2: 1.259921

Curve Fitting

x_data = np.array([0, 1, 2, 3, 4], dtype=float)
y_data = np.array([1.1, 2.9, 7.2, 12.8, 21.1], dtype=float)
fit_params, _ = optimize.curve_fit(lambda x, a, b: a * x ** 2 + b, x_data, y_data)

print("Fitted parameters (a, b):", fit_params)

Fitted parameters (a, b): [1.22931034 1.64413793]

Numerical Integration

from scipy import integrate

Numerical Integration of x^2 from 0 to 1

integral_result, error_estimate = integrate.quad(lambda x: x ** 2, 0, 1)
print("Definite integral of exp(-x^2) from 0 to 1:")
print("Integral result:", round(integral_result, 6))
print("Error estimate:", error_estimate)

Definite integral of exp(-x^2) from 0 to 1:
Integral result: 0.333333
Error estimate: 3.700743415417189e-15

Trapezoidal Integration for Sampled Data

x = np.linspace(0, np.pi, 5)
y = np.sin(x)
trap_result = integrate.trapezoid(y, x)

print("Trapezoid result for sin(x):", trap_result)

Trapezoid result for sin(x): 1.8961188979370398

Solving a Simple ODE: y' = -0.5y

ode_solution = integrate.solve_ivp(
    lambda t, values: -0.5 * values,
    t_span=(0, 5),
    y0=[2.0],
    t_eval=np.linspace(0, 5, 6),
)

print("ODE t values:", ode_solution.t)
print("ODE y values:", ode_solution.y[0])

ODE t values: [0. 1. 2. 3. 4. 5.]
ODE y values: [2.         1.21305368 0.73534013 0.44651426 0.2706912  0.16434549]

Interpolation Methods

from scipy import interpolate

x = np.array([0, 1, 2, 3, 4], dtype=float)
y = np.array([0, 1, 4, 9, 16], dtype=float)
x_new = np.linspace(0, 4, 10)

Piecewise Interpolation

linear_interp = interpolate.interp1d(x, y)
cubic_interp = interpolate.interp1d(x, y, kind='cubic')

print("Linear interpolation:", linear_interp(x_new))
print("Cubic interpolation:", cubic_interp(x_new))

Linear interpolation: [ 0.          0.44444444  0.88888889  2.          3.33333333  5.11111111
  7.33333333  9.77777778 12.88888889 16.        ]
Cubic interpolation: [ 0.          0.19753086  0.79012346  1.77777778  3.16049383  4.9382716
  7.11111111  9.67901235 12.64197531 16.        ]

Cubic Spline Interpolation

spline = interpolate.CubicSpline(x, y)

print("Spline value at x=2.5:", spline(2.5))

Spline value at x=2.5: 6.25

Statistical Analysis

from scipy import stats

sample = np.array([1.2, 2.3, 2.9, 3.5, 4.1, 5.0, 5.8, 6.2, 7.1, 8.0])

Descriptive Statistics

description = stats.describe(sample)

print("count:", description.nobs)
print("min:", description.minmax[0])
print("max:", description.minmax[1])
print("mean:", description.mean)
print("variance:", description.variance)

count: 10
min: 1.2
max: 8.0
mean: 4.61
variance: 4.796555555555555

Hypothesis Testing

t_stat, p_value = stats.ttest_1samp(sample, popmean=5.0)

print("One-sample t-test:", (t_stat, p_value))

One-sample t-test: (np.float64(-0.5631185935851243), np.float64(0.5871066468209352))

Probability Distribution Utilities

print("Normal CDF at 1.96:", stats.norm.cdf(1.96))
print("95th percentile of normal distribution:", stats.norm.ppf(0.95))

Normal CDF at 1.96: 0.9750021048517795
95th percentile of normal distribution: 1.6448536269514722

Correlation Analysis

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])

correlation, corr_p_value = stats.pearsonr(x, y)

print("Pearson correlation:", correlation)
print("Correlation p-value:", corr_p_value)

Pearson correlation: 0.7745966692414834
Correlation p-value: 0.1240270626575546

Fast Fourier Transform (FFT)

from scipy import fft

time_step = 0.1
wave = np.array([0, 1, 0, -1, 0, 1, 0, -1], dtype=float)

Forward FFT

frequency_domain = fft.fft(wave)
frequencies = fft.fftfreq(wave.size, d=time_step)

print("Frequencies:", frequencies)
print("FFT coefficients:", np.round(frequency_domain, 3))

Frequencies: [ 0.    1.25  2.5   3.75 -5.   -3.75 -2.5  -1.25]
FFT coefficients: [0.-0.j 0.+0.j 0.-4.j 0.+0.j 0.-0.j 0.-0.j 0.+4.j 0.-0.j]

Inverse FFT

reconstructed = fft.ifft(frequency_domain)

print("Recovered signal:", reconstructed.real)

Recovered signal: [ 0.  1.  0. -1.  0.  1.  0. -1.]

Signal Processing

from scipy import signal

time = np.linspace(0, 1, 200, endpoint=False)
waveform = np.sin(2 * np.pi * 5 * time) + 0.5 * np.sin(2 * np.pi * 20 * time)

Low-pass Butterworth Filtering

b, a = signal.butter(4, 0.15)
filtered = signal.filtfilt(b, a, waveform)

print("Original first 5 samples:", np.round(waveform[:5], 3))
print("Filtered first 5 samples:", np.round(filtered[:5], 3))

Original first 5 samples: [0.    0.45  0.785 0.93  0.882]
Filtered first 5 samples: [-0.008  0.175  0.346  0.493  0.614]

Peak Detection

peaks, properties = signal.find_peaks(waveform, height=1.0)

print("Peak indices:", peaks[:5])
print("Peak heights:", np.round(properties["peak_heights"][:5], 3))

Peak indices: [ 12  52  92 132 172]
Peak heights: [1.427 1.427 1.427 1.427 1.427]

Cross-correlation

correlation = signal.correlate([1, 2, 3], [0, 1, 0.5], mode="full")
print("\nCross-correlation:", correlation)

Cross-correlation: [0.5 2.  3.5 3.  0. ]

Sparse Matrix Computation

from scipy import sparse

dense = np.array([
    [0, 0, 3, 0],
    [4, 0, 0, 0],
    [0, 0, 5, 7],
    [0, 2, 0, 0],
])

Constructing Sparse Formats

csr = sparse.csr_matrix(dense)
coo = sparse.coo_matrix(dense)

print("CSR matrix:\n", csr)
print("\nNon-zero count:", csr.nnz)

CSR matrix:
 <Compressed Sparse Row sparse matrix of dtype 'int64'
    with 5 stored elements and shape (4, 4)>
  Coords    Values
  (0, 2)    3
  (1, 0)    4
  (2, 2)    5
  (2, 3)    7
  (3, 1)    2

Non-zero count: 5

Sparse Matrix Operations

vector = np.array([1, 2, 3, 4])

product = csr @ vector
print("Sparse matrix-vector product:", product)

identity = sparse.eye(4, format="csr")
print("Identity sparse matrix:\n", identity.toarray())

Sparse matrix-vector product: [ 9  4 43  4]
Identity sparse matrix:
 [[1. 0. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 0. 1.]]

Spatial Algorithms

from scipy import spatial

points = np.array([
    [0.0, 0.0],
    [1.0, 1.0],
    [2.0, 0.5],
    [0.5, 2.0],
    [1.5, 1.8],
])

KD-tree Nearest-neighbour Search

tree = spatial.KDTree(points)
distance, index = tree.query([0.9, 1.1])

print("Nearest point index:", index)
print("Nearest point:", points[index])
print("Distance:", distance)

Nearest point index: 1
Nearest point: [1. 1.]
Distance: 0.14142135623730953

Pairwise Distance Computation

distance_matrix = spatial.distance.cdist(points[:2], points[2:])

print("Distance matrix:\n", np.round(distance_matrix, 3))

Distance matrix:
 [[2.062 2.062 2.343]
 [1.118 1.118 0.943]]

Convex Hull Construction

hull = spatial.ConvexHull(points)

print("Convex hull vertex indices:", hull.vertices)

Convex hull vertex indices: [0 2 4 3]

Image Processing

from scipy import ndimage

test_image = np.arange(16, dtype=float).reshape(4, 4)

Filtering and Geometric Transforms

blurred = ndimage.gaussian_filter(test_image, sigma=1.0)
edges = ndimage.sobel(test_image, axis=0)
rotated = ndimage.rotate(test_image, 45, reshape=False)

print("Original image:\n", test_image)
print("\nBlurred image:\n", np.round(blurred, 2))
print("\nVertical Sobel edges:\n", edges)
print("\nRotated image:\n", np.round(rotated, 2))

Original image:
 [[ 0.  1.  2.  3.]
 [ 4.  5.  6.  7.]
 [ 8.  9. 10. 11.]
 [12. 13. 14. 15.]]

Blurred image:
 [[ 2.13  2.77  3.64  4.28]
 [ 4.68  5.32  6.19  6.83]
 [ 8.17  8.81  9.68 10.32]
 [10.72 11.36 12.23 12.87]]

Vertical Sobel edges:
 [[16. 16. 16. 16.]
 [32. 32. 32. 32.]
 [32. 32. 32. 32.]
 [16. 16. 16. 16.]]

Rotated image:
 [[ 0.    2.32  5.92  0.  ]
 [ 0.78  4.43  8.27 12.06]
 [ 2.94  6.73 10.57 14.22]
 [ 0.    9.08 12.68  0.  ]]

Connected-component Labelling

mask = np.array([[1, 0, 0], [1, 1, 0], [0, 0, 1]])

labels, num_features = ndimage.label(mask)

print("Labeled components:\n", labels)
print("Number of features:", num_features)

Labeled components:
 [[1 0 0]
 [1 1 0]
 [0 0 2]]
Number of features: 2

Clustering Methods

from scipy.cluster import hierarchy, vq

data = np.array([
    [1.0, 1.0],
    [1.2, 0.9],
    [3.0, 3.1],
    [3.2, 2.9],
    [8.0, 8.0],
])

Hierarchical Clustering

linkage_matrix = hierarchy.linkage(data, method="ward")
cluster_labels = hierarchy.fcluster(linkage_matrix, t=2.5, criterion="distance")

print("Cluster labels from hierarchy:", cluster_labels)

Cluster labels from hierarchy: [1 1 2 2 3]

K-means-style Vector Quantisation

centroids, distortion = vq.kmeans(data, 2)
codes, distances = vq.vq(data, centroids)

print("Centroids:\n", np.round(centroids, 3))
print("Distortion:", distortion)
print("Assigned cluster codes:", codes)
print("Distances to centroids:", np.round(distances, 3))

Centroids:

 [[2.1   1.975]

 [8.    8.   ]]

Distortion: 1.1499690274985555

Assigned cluster codes: [0 0 0 0 1]

Distances to centroids: [1.47  1.402 1.441 1.437 0.   ]

Conclusion

SciPy provides a coherent and mature toolkit for scientific computing in Python, extending NumPy with specialised algorithms for optimisation, integration, interpolation, statistics, signal processing, sparse matrices, spatial analysis, image processing, and clustering. Across the examples in this notebook, a consistent workflow emerges: represent data as arrays, select an appropriate SciPy submodule, and apply high-level numerical routines through clear and reproducible code.

In practical settings, SciPy is most effective when used as part of the broader scientific Python ecosystem. NumPy supports efficient array operations, while libraries such as Matplotlib and pandas complement SciPy through visualisation and structured data analysis. Collectively, these tools support reliable problem solving in research and professional environments where numerical accuracy, computational performance, and maintainability are all essential.

For further study, a productive next step is to select one domain-relevant submodule, for example scipy.optimize, scipy.stats, or scipy.signal, and examine its API in greater depth using real datasets and systematic parameter tuning. This focused approach develops both practical intuition and methodological confidence, helping to translate foundational examples into robust analytical workflows.

Python Pandas Library

michal puzanov — Tue, 04 Aug 2026 09:49:33 +0000

Pandas is an open-source library for data analysis and manipulation in Python. It provides fast, flexible and expressive data structures for working with relational and labelled data. Originally developed by Wes McKinney in 2008, it has become a foundational tool in modern data science and serves as a highly programmable analogue to spreadsheet software.

Key characteristics

NumPy foundation: Built on top of NumPy, it inherits highly optimised, array-based computational performance.
Label-driven alignment: Data are automatically aligned according to explicit row and column labels, thereby improving the reliability of calculations involving partially mismatched datasets.
Heterogeneous typing: Unlike strict numerical arrays, Pandas can accommodate mixed data types, including integers, strings, floats and booleans, within a single tabular structure.
Missing-data resilience: It provides native support for detecting, representing and handling missing values, such as NaN.

Core data structures

Series: A one-dimensional labelled array capable of holding any data type. In practical terms, it resembles a single column in a spreadsheet.
DataFrame: A two-dimensional tabular data structure with labelled rows and columns. It may be regarded as a collection of Series sharing a common index, analogous to a table in SQL or a worksheet in Excel.

Core features and capabilities

Robust input/output parsing: Pandas supports efficient reading and writing across multiple formats, including CSV, Excel, SQL databases, JSON and Parquet.
Advanced data cleaning: Built-in methods enable users to identify, filter and remove duplicates, and to impute missing values.
Flexible wrangling and reshaping: The library facilitates pivoting, melting, slicing and subsetting operations based on conditional logic.
High-performance merging: Relational operations such as inner, outer, left and right joins, as well as concatenation, can be executed in concise code.
Split-apply-combine (GroupBy): Data may be grouped by specified criteria and summarised using aggregate operations such as sums, averages and custom calculations.
Time-series functionality: Pandas provides specialised tools for handling dates, converting time zones, generating date ranges and resampling time-stamped data.
Integrated plotting: It offers Matplotlib-based plotting utilities, enabling the rapid visualisation of tabular data with the .plot() interface.

Time-series functionality and integrated plotting are not covered in this article.

Practical examples

Installation

Using pip:

pip install pandas

Using conda:

conda install pandas

Using poetry:

poetry add pandas

Verify the installation

The following script may be used to confirm that pandas has been installed correctly.

import pandas as pd

print(pd.__version__)

3.0.5

Creating DataFrame

from dictionary

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'city': ['NYC', 'LA', 'Chicago']
})

print(df)

      name  age     city
0    Alice   25      NYC
1      Bob   30       LA
2  Charlie   35  Chicago

from dictionary using NumPy

import numpy as np

df_dict = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': np.linspace(25, 35, num=3, dtype=int),
    'city': ['NYC', 'LA', 'Chicago']
})

print(df_dict)

      name  age     city
0    Alice   25      NYC
1      Bob   30       LA
2  Charlie   35  Chicago

from list of lists

df_list = pd.DataFrame(
    [['Alice', 25, 'NYC' ], ['Bob', 30, 'LA'], ['Charlie', 35, 'Chicago']], 
    columns=['name', 'age', 'city']
)

print(df_list)

      name  age     city
0    Alice   25      NYC
1      Bob   30       LA
2  Charlie   35  Chicago

from ndarray

df_ndarray = pd.DataFrame(
    np.array([['Alice', 25, 'NYC'], ['Bob', 30, 'LA'], ['Charlie', 35, 'Chicago']]),
    columns=['name', 'age', 'city']
)

print(df_ndarray)

      name age     city
0    Alice  25      NYC
1      Bob  30       LA
2  Charlie  35  Chicago

Data inspection and information

basic info

print(f"Shape: {df.shape}")  # (rows, columns)
print(f"Column names: {df.columns.tolist()}")
print(f"Data types:\n{df.dtypes}\n")
print(f"DataFrame Info:\n{df.info()}\n")

Shape: (3, 3)
Column names: ['name', 'age', 'city']
Data types:
name      str
age     int64
city      str
dtype: object

<class 'pandas.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   name    3 non-null      str  
 1   age     3 non-null      int64
 2   city    3 non-null      str  
dtypes: int64(1), str(2)
memory usage: 204.0 bytes
DataFrame Info:
None

head and tail

print(f"First default number of rows:\n{df.head()}\n")
print(f"First 2 rows:\n{df.head(2)}\n")
print(f"Last default number of rows:\n{df.tail()}\n")
print(f"Last 2 rows:\n{df.tail(2)}")

First default number of rows:
      name  age     city
0    Alice   25      NYC
1      Bob   30       LA
2  Charlie   35  Chicago

First 2 rows:
    name  age city
0  Alice   25  NYC
1    Bob   30   LA

Last default number of rows:
      name  age     city
0    Alice   25      NYC
1      Bob   30       LA
2  Charlie   35  Chicago

Last 2 rows:
      name  age     city
1      Bob   30       LA
2  Charlie   35  Chicago

descriptive statistics for numerical columns

print(df.describe())

        age
count   3.0
mean   30.0
std     5.0
min    25.0
25%    27.5
50%    30.0
75%    32.5
max    35.0

descriptive statistics for all columns with transpose layout

print(df.describe(include='all').T)

     count unique    top freq  mean  std   min   25%   50%   75%   max
name     3      3  Alice    1   NaN  NaN   NaN   NaN   NaN   NaN   NaN
age    3.0    NaN    NaN  NaN  30.0  5.0  25.0  27.5  30.0  32.5  35.0
city     3      3    NYC    1   NaN  NaN   NaN   NaN   NaN   NaN   NaN

unique and missing values

print(f"Unique values in 'name':\n {df['name'].unique()}\n")
print(f"Null values:\n {df.isnull().sum()}\n")

Unique values in 'name':
 <StringArray>
['Alice', 'Bob', 'Charlie']
Length: 3, dtype: str

Null values:
 name    0
age     0
city    0
dtype: int64

Selecting data by columns and rows

select column - returns Series

print(df['name'])

0      Alice
1        Bob
2    Charlie
Name: name, dtype: str

select multiple columns

print(df[['name', 'age']])

      name  age
0    Alice   25
1      Bob   30
2  Charlie   35

select by position (iloc)

print("Select first 2 rows, first 2 columns (iloc):")
print(df.iloc[0:2, 0:2])

Select first 2 rows, first 2 columns (iloc):
    name  age
0  Alice   25
1    Bob   30

select by label (loc)

print("Select by label using loc:")
print(df.loc[0])  # First row
print(df.loc[0:1, ['name', 'age']])  # First 2 rows, specific columns

Select by label using loc:
name    Alice
age        25
city      NYC
Name: 0, dtype: object
    name  age
0  Alice   25
1    Bob   30

Filtering and conditional selection

simple filter

print(df[df['age'] > 25])

      name  age     city
1      Bob   30       LA
2  Charlie   35  Chicago

multiple conditions with AND

print("Age > 25 AND city == 'LA':")
print(df[(df['age'] > 25) & (df['city'] == 'LA')])

Age > 25 AND city == 'LA':
  name  age city
1  Bob   30   LA

multiple conditions with OR

print("Age == 25 OR city == 'Chicago':")
print(df[(df['age'] == 25) | (df['city'] == 'Chicago')])

Age == 25 OR city == 'Chicago':
      name  age     city
0    Alice   25      NYC
2  Charlie   35  Chicago

multiple values using list

print("City in ['NYC', 'Chicago']:")
print(df[df['city'].isin(['NYC', 'Chicago'])])

City in ['NYC', 'Chicago']:
      name  age     city
0    Alice   25      NYC
2  Charlie   35  Chicago

string filtering

print("City starts with 'L':")
print(df[df['city'].str.startswith('L')])

City starts with 'L':
  name  age city
1  Bob   30   LA

filter using query method

print(df.query('age > 25'))

      name  age     city
1      Bob   30       LA
2  Charlie   35  Chicago

Adding and modifying columns

It is good practice to avoid modifying the original DataFrame when adding, deleting or altering data. For this reason, a deep copy of the DataFrame is used in the examples that follow.

hard copy dataframe

df_copy = df.copy()

print(f"Copy of Original DataFrame: \n{df_copy}\n")

Copy of Original DataFrame: 
      name  age     city
0    Alice   25      NYC
1      Bob   30       LA
2  Charlie   35  Chicago

add new column

df_copy['age_next_year'] = df_copy['age'] + 1

print(df_copy)

      name  age     city  age_next_year
0    Alice   25      NYC             26
1      Bob   30       LA             31
2  Charlie   35  Chicago             36

modify existing column with lambda function

df_copy['age_group'] = df_copy['age'].apply(lambda x: 'Young' if x < 30 else 'Older')

print(df_copy)

      name  age     city  age_next_year age_group
0    Alice   25      NYC             26     Young
1      Bob   30       LA             31     Older
2  Charlie   35  Chicago             36     Older

conditional assignment (using np.where)

df_copy['status'] = np.where(df_copy['age'] > 25, 'Senior', 'Junior')

print(df_copy)

      name  age     city  age_next_year age_group  status
0    Alice   25      NYC             26     Young  Junior
1      Bob   30       LA             31     Older  Senior
2  Charlie   35  Chicago             36     Older  Senior

rename columns

df_copy = df_copy.rename(columns={'name': 'full_name', 'city': 'location'})
print(df_copy)

df_copy.rename(columns={'full_name': 'name', 'location': 'city'}, inplace=True)
print(df_copy)

Rename columns:
  full_name  age location  age_next_year age_group  status
0     Alice   25      NYC             26     Young  Junior
1       Bob   30       LA             31     Older  Senior
2   Charlie   35  Chicago             36     Older  Senior
Rename columns back using inplace:
      name  age     city  age_next_year age_group  status
0    Alice   25      NYC             26     Young  Junior
1      Bob   30       LA             31     Older  Senior
2  Charlie   35  Chicago             36     Older  Senior

drop columns

df_dropped = df_copy.drop(columns=['city'])

print(df_dropped)

      name  age  age_next_year age_group  status
0    Alice   25             26     Young  Junior
1      Bob   30             31     Older  Senior
2  Charlie   35             36     Older  Senior

Data cleaning and missing values

sample dataframe with missing values

df_missing = pd.DataFrame({
    'name': ['Alice', None, 'Charlie', None],
    'age': [25, 30, np.nan, 40],
    'city': ['NYC', None, 'Chicago', 'LA']
})

print("DataFrame with missing values:")
print(df_missing)

DataFrame with missing values:
      name   age     city
0    Alice  25.0      NYC
1      NaN  30.0      NaN
2  Charlie   NaN  Chicago
3      NaN  40.0       LA

check for missing values using info

print(df_missing.info())

<class 'pandas.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   name    2 non-null      str    
 1   age     3 non-null      float64
 2   city    3 non-null      str    
dtypes: float64(1), str(2)
memory usage: 228.0 bytes
None

There are four records in the sample DataFrame. The 'name' column contains two non-null values, while the 'age' and 'city' columns contain three non-null values each.

print(f"Check for missing values:\n{df_missing.isnull()}\n")
print(f"Check for missing values:\n{df_missing.isnull().sum()}\n")

Check for missing values:
    name    age   city
0  False  False  False
1   True  False   True
2  False   True  False
3   True  False  False

Check for missing values:
name    2
age     1
city    1
dtype: int64

drop rows with missing values

# Drop rows with any missing values
print(f"{df_missing.dropna()}\n")

# Drop rows where specific column is missing
print(df_missing.dropna(subset=['age']))

Drop rows with missing values:
    name   age city
0  Alice  25.0  NYC

Drop rows where 'age' is missing:
    name   age city
0  Alice  25.0  NYC
1    NaN  30.0  NaN
3    NaN  40.0   LA

fill missing values

# Fill missing values with 0
print(df_missing.fillna(0))
print()

# Fill missing values with forward fill
print(df_missing.ffill())
print()

# Fill with mean
df_missing['age'] = df_missing['age'].fillna(df_missing['age'].mean())
print(df_missing)
print()

      name   age     city
0    Alice  25.0      NYC
1        0  30.0        0
2  Charlie   0.0  Chicago
3        0  40.0       LA

      name   age     city
0    Alice  25.0      NYC
1    Alice  30.0      NYC
2  Charlie  30.0  Chicago
3  Charlie  40.0       LA

      name        age     city
0    Alice  25.000000      NYC
1      NaN  30.000000      NaN
2  Charlie  31.666667  Chicago
3      NaN  40.000000       LA

Grouping and aggregation

sample dataframe

df_sales = pd.DataFrame({
    'department': ['Sales', 'Sales', 'IT', 'IT', 'HR', 'HR'],
    'employee': ['Alice', 'Bob', 'Charlie', 'David', 'Eve', 'Frank'],
    'salary': [50000, 55000, 75000, 75000, 45000, 48000],
    'bonus': [5000, 6000, 10000, 12000, 3000, 2000]
})
print(df_sales)

  department employee  salary  bonus
0      Sales    Alice   50000   5000
1      Sales      Bob   55000   6000
2         IT  Charlie   75000  10000
3         IT    David   75000  12000
4         HR      Eve   45000   3000
5         HR    Frank   48000   2000

group by single column

print(df_sales.groupby('department')['salary'].mean())

department
HR       46500.0
IT       75000.0
Sales    52500.0
Name: salary, dtype: float64

group by and aggregate multiple columns

agg_result = df_sales.groupby('department').agg({
    'salary': ['mean', 'sum', 'count'],
    'bonus': 'mean'
})
print(agg_result)

             salary                  bonus
               mean     sum count     mean
department                                
HR          46500.0   93000     2   2500.0
IT          75000.0  150000     2  11000.0
Sales       52500.0  105000     2   5500.0

custom aggregation names

print(df_sales.groupby('department').agg(
    avg_salary=('salary', 'mean'),
    total_bonus=('bonus', 'sum'),
    count=('employee', 'count')
))

            avg_salary  total_bonus  count
department                                
HR             46500.0         5000      2
IT             75000.0        22000      2
Sales          52500.0        11000      2

group by multiple columns

df_sales['year'] = [2023, 2023, 2024, 2024, 2023, 2024]

print(df_sales.groupby(['department', 'year'])['salary'].mean())

department  year
HR          2023    45000.0
            2024    48000.0
IT          2024    75000.0
Sales       2023    52500.0
Name: salary, dtype: float64

Sorting and ranking

sort by single column ascending

print(df_sales.sort_values('salary'))

  department employee  salary  bonus  year
4         HR      Eve   45000   3000  2023
5         HR    Frank   48000   2000  2024
0      Sales    Alice   50000   5000  2023
1      Sales      Bob   55000   6000  2023
3         IT    David   75000  12000  2024
2         IT  Charlie   75000  10000  2024

sort by single column descending

print(df_sales.sort_values('salary', ascending=False))

  department employee  salary  bonus  year
3         IT    David   75000  12000  2024
2         IT  Charlie   75000  10000  2024
1      Sales      Bob   55000   6000  2023
0      Sales    Alice   50000   5000  2023
5         HR    Frank   48000   2000  2024
4         HR      Eve   45000   3000  2023

sort by multiple columns

print(df_sales.sort_values(['department', 'salary']))

  department employee  salary  bonus  year
4         HR      Eve   45000   3000  2023
5         HR    Frank   48000   2000  2024
2         IT  Charlie   75000  10000  2024
3         IT    David   75000  12000  2024
0      Sales    Alice   50000   5000  2023
1      Sales      Bob   55000   6000  2023

sort by index

# create a new DataFrame with 'employee' as the index
df_index = df_sales.set_index('employee')

print(df_index.sort_index())

         department  salary  bonus  year
employee                                
Alice         Sales   50000   5000  2023
Bob           Sales   55000   6000  2023
Charlie          IT   75000  10000  2024
David            IT   75000  12000  2024
Eve              HR   45000   3000  2023
Frank            HR   48000   2000  2024

ranking

df_sales['salary_rank'] = df_sales['salary'].rank(ascending=False)

print(df_sales[['employee', 'salary', 'salary_rank']])

  employee  salary  salary_rank
0    Alice   50000          4.0
1      Bob   55000          3.0
2  Charlie   75000          1.5
3    David   75000          1.5
4      Eve   45000          6.0
5    Frank   48000          5.0

dense ranking

df_sales['salary_dense_rank'] = df_sales['salary'].rank(method='dense', ascending=False)

print(df_sales[['employee', 'salary', 'salary_dense_rank']])

  employee  salary  salary_dense_rank
0    Alice   50000                3.0
1      Bob   55000                2.0
2  Charlie   75000                1.0
3    David   75000                1.0
4      Eve   45000                5.0
5    Frank   48000                4.0

Merging and joining DataFrames

sample dataframes df1, df2 and df3

df1 = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'department': ['Sales', 'IT', 'HR']
})

df2 = pd.DataFrame({
    'id': [1, 2, 3],
    'salary': [50000, 70000, 45000],
    'bonus': [5000, 10000, 3000]
})

print(df1)
print()
print(df2)

   id     name department
0   1    Alice      Sales
1   2      Bob         IT
2   3  Charlie         HR

   id  salary  bonus
0   1   50000   5000
1   2   70000  10000
2   3   45000   3000

inner join (intersection)

print(pd.merge(df1, df2, on='id'))

   id     name department  salary  bonus
0   1    Alice      Sales   50000   5000
1   2      Bob         IT   70000  10000
2   3  Charlie         HR   45000   3000

left join

df3 = pd.DataFrame({
    'id': [1, 2, 4],
    'salary': [50000, 70000, 80000]
})

print(pd.merge(df1, df3, on='id', how='left'))

   id     name department   salary
0   1    Alice      Sales  50000.0
1   2      Bob         IT  70000.0
2   3  Charlie         HR      NaN

right join

print(pd.merge(df1, df3, on='id', how='right'))

   id   name department  salary
0   1  Alice      Sales   50000
1   2    Bob         IT   70000
2   4    NaN        NaN   80000

outer join (union)

print(pd.merge(df1, df3, on='id', how='outer'))

   id     name department   salary
0   1    Alice      Sales  50000.0
1   2      Bob         IT  70000.0
2   3  Charlie         HR      NaN
3   4      NaN        NaN  80000.0

concatenate vertically

print(pd.concat([df1, df1], ignore_index=True))

   id     name department
0   1    Alice      Sales
1   2      Bob         IT
2   3  Charlie         HR
3   1    Alice      Sales
4   2      Bob         IT
5   3  Charlie         HR

concatenate horizontally

print(pd.concat([df1, df2], axis=1))

   id     name department  id  salary  bonus
0   1    Alice      Sales   1   50000   5000
1   2      Bob         IT   2   70000  10000
2   3  Charlie         HR   3   45000   3000

Reshaping data using pivot tables

sample dataframe1

df_pivot = pd.DataFrame({
    'product': ['A', 'A', 'B', 'B', 'C', 'C'],
    'month': ['Jan', 'Feb', 'Jan', 'Feb', 'Jan', 'Feb'],
    'sales': [100, 150, 200, 250, 300, 350]
})

print(df_pivot)

  product month  sales
0       A   Jan    100
1       A   Feb    150
2       B   Jan    200
3       B   Feb    250
4       C   Jan    300
5       C   Feb    350

pivot table

pivot = df_pivot.pivot(index='product', columns='month', values='sales')

print(pivot)

month    Feb  Jan
product          
A        150  100
B        250  200
C        350  300

sample dataframe2

df_pivot2 = pd.DataFrame({
    'category': ['A', 'A', 'A', 'B', 'B', 'B'],
    'region': ['North', 'South', 'North', 'South', 'North', 'South'],
    'sales': [100, 150, 120, 200, 250, 300]
})

print(df_pivot2)

  category region  sales
0        A  North    100
1        A  South    150
2        A  North    120
3        B  South    200
4        B  North    250
5        B  South    300

pivot with aggregation

pivot_agg = df_pivot2.pivot_table(
    index='category', 
    columns='region', 
    values='sales', 
    aggfunc='sum'
)

print(pivot_agg)

region    North  South
category              
A           220    150
B           250    500

unstack

df_stacked = df_pivot.set_index(['product', 'month'])['sales']
df_unstacked = df_stacked.unstack()

print(df_stacked)
print()
print(df_unstacked)

product  month
A        Jan      100
         Feb      150
B        Jan      200
         Feb      250
C        Jan      300
         Feb      350
Name: sales, dtype: int64

month    Feb  Jan
product          
A        150  100
B        250  200
C        350  300

melt (unpivot)

df_melted = pd.melt(df_unstacked.reset_index(), id_vars=['product'], var_name='variable', value_name='value')

print(df_melted)

  product variable  value
0       A      Feb    150
1       B      Feb    250
2       C      Feb    350
3       A      Jan    100
4       B      Jan    200
5       C      Jan    300

String operations

sample dataframe

df_str = pd.DataFrame({
    'name': ['alice smith', 'bob jones', 'charlie brown'],
    'email': ['alice@email.com', 'bob@email.com', 'charlie@email.com']
})

print(df_str)

            name              email
0    alice smith    alice@email.com
1      bob jones      bob@email.com
2  charlie brown  charlie@email.com

string upper, title case

print(df_str['name'].str.upper())
print()

print(df_str['name'].str.title())
print()

0      ALICE SMITH
1        BOB JONES
2    CHARLIE BROWN
Name: name, dtype: str

0      Alice Smith
1        Bob Jones
2    Charlie Brown
Name: name, dtype: str

string lenght

print(df_str['name'].str.len())

0    11
1     9
2    13
Name: name, dtype: int64

string contains

print(df_str[df_str['name'].str.contains('alice')])

          name            email
0  alice smith  alice@email.com

extract part of string

print(df_str['name'].str.split(' ').str[0])
print()
print(df_str['name'].str.split(' ').str[1])

0      alice
1        bob
2    charlie
Name: name, dtype: object

0    smith
1    jones
2    brown
Name: name, dtype: object

replace part of string

print(df_str['name'].str.replace('smith', 'Smith'))

0      alice Smith
1        bob jones
2    charlie brown
Name: name, dtype: str

extract digits from string

df_codes = pd.DataFrame({'code': ['A123', 'B456', 'C789']})
print(df_codes)
print()

df_codes['code'] = df_codes['code'].str.replace(r'\D', '', regex=True)
print(df_codes)

get domain from email address

print(df_str['email'].str.extract(r'@([a-z]+)\.com'))

       0
0  email
1  email
2  email

Datetime operations

sample dataframe

df_dates = pd.DataFrame({
    'date': ['2024-01-15', '2024-02-20', '2024-03-10'],
    'value': [100, 150, 200]
})

print(df_dates)
print(df_dates.dtypes)

         date  value
0  2024-01-15    100
1  2024-02-20    150
2  2024-03-10    200
date       str
value    int64
dtype: object

convert to datetime type

df_dates['date'] = pd.to_datetime(df_dates['date'])

print(df_dates)
print(df_dates.dtypes)

        date  value
0 2024-01-15    100
1 2024-02-20    150
2 2024-03-10    200
date     datetime64[us]
value             int64
dtype: object

extract components

df_dates['year'] = df_dates['date'].dt.year
df_dates['month'] = df_dates['date'].dt.month
df_dates['day'] = df_dates['date'].dt.day
df_dates['day_name'] = df_dates['date'].dt.day_name()

print(df_dates)

        date  value  year  month  day day_name
0 2024-01-15    100  2024      1   15   Monday
1 2024-02-20    150  2024      2   20  Tuesday
2 2024-03-10    200  2024      3   10   Sunday

date add 10 days

print(df_dates['date'] + pd.Timedelta(days=10))

0   2024-01-25
1   2024-03-01
2   2024-03-20
Name: date, dtype: datetime64[us]

date range 5 days

date_range = pd.date_range(start='2024-01-01', periods=5, freq='D')

print(date_range)

DatetimeIndex(['2024-01-01', '2024-01-02', '2024-01-03', '2024-01-04',
               '2024-01-05'],
              dtype='datetime64[us]', freq='D')

filter by date

start_date = pd.to_datetime('2024-02-01')
end_date = pd.to_datetime('2024-03-01')
df_filtered = df_dates[(df_dates['date'] >= start_date) & (df_dates['date'] <= end_date)]

print(df_filtered)

        date  value  year  month  day day_name
1 2024-02-20    150  2024      2   20  Tuesday

Mathematical and statistical operations

sample dataframe

df_math = pd.DataFrame({
    'A': [10, 20, 30, 40, 50],
    'B': [5, 10, 15, 20, 25],
    'C': [1, 2, 3, 4, 5]
})

print(df_math)

    A   B  C
0  10   5  1
1  20  10  2
2  30  15  3
3  40  20  4
4  50  25  5

basic statistics

print(f"Mean: {df_math['A'].mean()}")
print(f"Median: {df_math['A'].median()}")
print(f"Std Dev: {df_math['A'].std()}")
print(f"Min: {df_math['A'].min()}")
print(f"Max: {df_math['A'].max()}")
print(f"Sum: {df_math['A'].sum()}")

Mean: 30.0
Median: 30.0
Std Dev: 15.811388300841896
Min: 10
Max: 50
Sum: 150

percentiles

print(f"25th percentile: {df_math['A'].quantile(0.25)}")
print(f"75th percentile: {df_math['A'].quantile(0.75)}")

25th percentile: 20.0
75th percentile: 40.0

correlation

print(df_math.corr())

     A    B    C
A  1.0  1.0  1.0
B  1.0  1.0  1.0
C  1.0  1.0  1.0

covariance

print(df_math.cov())

       A      B     C
A  250.0  125.0  25.0
B  125.0   62.5  12.5
C   25.0   12.5   2.5

arithmetic operations

df_math['A_add_B'] = df_math['A'] + df_math['B']
df_math['A_subtract_B'] = df_math['A'] - df_math['B']
df_math['A_squared'] = df_math['A'] ** 2

print(df_math)

    A   B  C  A_add_B  A_subtract_B  A_squared
0  10   5  1       15             5        100
1  20  10  2       30            10        400
2  30  15  3       45            15        900
3  40  20  4       60            20       1600
4  50  25  5       75            25       2500

apply numpy function

print(df_math['A'].apply(np.sqrt))

0    3.162278
1    4.472136
2    5.477226
3    6.324555
4    7.071068
Name: A, dtype: float64

cumulative operations

print(f"Cumulative sum of A:\n{df_math['A'].cumsum()}\n")
print(f"Cumulative product of A:\n{df_math['A'].cumprod()}\n")

Cumulative sum of A:
0     10
1     30
2     60
3    100
4    150
Name: A, dtype: int64

Cumulative product of A:
0          10
1         200
2        6000
3      240000
4    12000000
Name: A, dtype: int64

Useful tips and best practices

Create a copy to avoid accidental overwriting

df_copy = df.copy()

use inplace carefully

df_inplace = pd.DataFrame({'A': [1, 2, 3]})
df_inplace.sort_values('A', inplace=True)

Vectorised operations are generally faster than apply/loops

import time
df_large = pd.DataFrame({'A': range(10000)})

# Slow: using apply
start = time.time()
result_slow = df_large['A'].apply(lambda x: x * 2)
time_slow = time.time() - start

# Fast: vectorized
start = time.time()
result_fast = df_large['A'] * 2
time_fast = time.time() - start

print(f"  Apply time: {time_slow:.6f}s")
print(f"  Vectorized time: {time_fast:.6f}s")

✓ Vectorized operations are faster
  Apply time: 0.010587s
  Vectorized time: 0.000676s

use groupby instead of loops

df_grouped = pd.DataFrame({
    'group': ['A', 'A', 'B', 'B'],
    'value': [1, 2, 3, 4]
})
result = df_grouped.groupby('group')['value'].sum()

print(result)

✓ Use groupby for group operations (not loops)
group
A    3
B    7
Name: value, dtype: int64

For memory efficiency, categorical values may be preferable for repeated string data

df_mem = pd.DataFrame({'category': ['A', 'B', 'A', 'B'] * 1000})

print(f"✓ Use categorical for repeated string values")
print(f"  Object dtype: {df_mem['category'].memory_usage(deep=True)} bytes")

df_mem['category'] = df_mem['category'].astype('category')
print(f"  Categorical: {df_mem['category'].memory_usage(deep=True)} bytes")

✓ Use categorical for repeated string values
  Object dtype: 200132 bytes
  Categorical: 4232 bytes

chaining operations

result = (df
  .query('age > 25')
  .copy()
  .assign(age_group='Senior')
  .sort_values('age', ascending=False)
)

print(result)

      name  age     city age_group

2  Charlie   35  Chicago    Senior

1      Bob   30       LA    Senior

Conclusion

Pandas is one of the most practical and widely used libraries for data analysis in Python. This tutorial has introduced the core ideas behind creating and inspecting DataFrames, selecting and filtering data, cleaning missing values, grouping observations, and combining datasets from different sources.

A major strength of Pandas lies in its ability to make data manipulation both readable and efficient. With a small set of intuitive methods, it becomes straightforward to transform raw data into a structure suitable for exploration, reporting, or further analysis. These capabilities are especially valuable in real-world workflows where datasets are often messy, incomplete, or spread across multiple files.

To build confidence with Pandas, regular practice is essential. Readers are encouraged to experiment with the methods shown here on their own datasets, compare different approaches for the same task, and gradually explore more advanced features such as time-series handling, custom aggregations, and multi-index operations. In this way, the foundations introduced in this article can grow into a deeper and more flexible understanding of data analysis in Python.

Did this article help you? Let me know in the comments below, and don’t forget to drop a like if you enjoyed the read! Thank you.

Python NumPy Library

michal puzanov — Tue, 04 Aug 2026 09:45:14 +0000

NumPy (Numerical Python) is a foundational open-source Python library for numerical and mathematical computation.

It introduces the N-dimensional array (ndarray), a high-performance data structure for storing and manipulating large datasets efficiently. NumPy forms the computational foundation of the Python data-science ecosystem; major libraries such as Pandas, SciPy, scikit-learn, and TensorFlow build directly upon it.

This tutorial is designed to provide a concise yet practical overview of NumPy and to support day-to-day technical work through clear, task-oriented examples.

Key characteristics

High performance: NumPy operations are implemented in highly optimised C, enabling many numerical workloads to run substantially faster than equivalent operations on standard Python lists.
Vectorisation: NumPy reduces reliance on explicit Python loops by applying operations across entire arrays in a single expression.
Memory efficiency: NumPy arrays store homogeneous data in contiguous memory blocks, typically reducing memory overhead relative to Python lists.

Core features and capabilities

NumPy provides a broad suite of tools for numerical computation, including:

Multidimensional arrays: Creation and manipulation of 1D vectors, 2D matrices, and higher-dimensional structures.
Broadcasting: Arithmetic operations between arrays of different, but compatible, shapes.
Linear algebra: Built-in routines for matrix multiplication, determinants, inverses, and systems of linear equations.
Random number generation: Utilities for generating random samples from common statistical distributions.
Mathematical functions: Fast element-wise operations for trigonometric, logarithmic, exponential, and statistical calculations (for example, mean, median, and standard deviation).

Python lists vs NumPy ndarrays

Python lists can store heterogeneous data types (for example, strings, integers, and objects) in a single container. This flexibility is useful, but lists are comparatively inefficient for numerical computation.
NumPy ndarrays are homogeneous (all elements share the same data type). This design supports efficient vectorised operations and improved memory efficiency.

Practical examples

Installation

Using pip

pip install numpy

Using conda

conda install numpy

Using poetry

poetry add numpy

Verify the installation

The following test script can be used to confirm that NumPy has been installed correctly.

import numpy as np

# Check NumPy version
print(f"NumPy version: {np.__version__}")

NumPy version: 2.5.1

`Ndarray` creation

This section demonstrates several standard methods for creating NumPy arrays.

One-dimensional ndarray

arr1d = np.array([1, 2, 3, 4, 5])
print("From list:", arr1d)

From list: [1 2 3 4 5]

Two-dimensional ndarray

arr2d= np.array([[1, 2, 3], [4, 5, 6]])
print("\n2D array:\n", arr2d)

2D array:
 [[1 2 3]
 [4 5 6]]

Zero-filled and one-filled arrays

zeros = np.zeros(5)
print("\nZeros:", zeros)

ones = np.ones((3, 3))
print("\nOnes:\n", ones)

Zeros: [0. 0. 0. 0. 0.]

Ones:
 [[1. 1. 1.]
 [1. 1. 1.]
 [1. 1. 1.]]

Range-based and evenly spaced sequences

range_arr = np.arange(0, 10, 2)
print("\nRange (0 to 10, step 2):", range_arr)

linspace_arr = np.linspace(0, 10, 5)
print("\nLinspace (0 to 10, 5 points):", linspace_arr)

Range (0 to 10, step 2): [0 2 4 6 8]

Linspace (0 to 10, 5 points): [ 0.   2.5  5.   7.5 10. ]

Identity and uninitialised arrays

# create 2D identity matrix
identity = np.eye(3)
print("\nIdentity matrix:\n", identity)

# create 2D array with random garbage values, it is faster than random.rand() and random.randn()
empty = np.empty((2, 2))
print("\nEmpty array shape:", empty.shape)

Identity matrix:
 [[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]

Empty array shape: (2, 2)

Randomly generated arrays

# create 2D array with random values between 0 and 1
random_arr = np.random.rand(3, 3)
print("\nRandom array (0-1):\n", random_arr)

# create 2D array with random integers between 1 and 10
random_int = np.random.randint(1, 10, size=(2, 3))
print("\nRandom integers (1-10):\n", random_int)

Random array (0-1):
 [[0.55874991 0.81386435 0.31782834]
 [0.39704509 0.89016825 0.82541621]
 [0.10668708 0.15977588 0.65121931]]

Random integers (1-10):
 [[2 2 9]
 [8 4 4]]

`Ndarray` properties and attributes

Reference array

arr = np.array([[1, 2, 3], [4, 5, 6]])

Array properties

print("Shape:", arr.shape)
print("Dimensions:", arr.ndim)
print("Size (total elements):", arr.size)
print("Data type:", arr.dtype)
print("Item size (bytes):", arr.itemsize)
print("Strides:", arr.strides)

Shape: (2, 3)
Dimensions: 2
Size (total elements): 6
Data type: int64
Item size (bytes): 8
Strides: (24, 8)

Reshaping arrays

reshaped = arr.reshape(3, 2)
print("\nReshaped to (3, 2):\n", reshaped)

Reshaped to (3, 2):
 [[1 2]
 [3 4]
 [5 6]]

Flattening arrays

flattened = arr.flatten()
print("\nFlattened:", flattened)

Flattened: [1 2 3 4 5 6]

Copy versus view

arr_copy = arr.copy()
arr_view = arr.view()
arr_copy[0, 0] = 999
print("\nOriginal:", arr[0, 0])
print("Copy modified:", arr_copy[0, 0])
arr_view[0, 0] = 888
print("Original after view modified:", arr[0, 0])
print("View modified:", arr_view[0, 0])

Original: 1
Copy modified: 999
Original after view modified: 888
View modified: 888

A view is typically faster than creating a copy, but it shares underlying data with the original ndarray. Consequently, modifying values through a view also modifies the original array. Views are particularly useful when adjusting shape or data-type representations without duplicating data.

Changing the shape of a view

# 1. Create a flat 1D original array
original = np.array([10, 20, 30, 40, 50, 60])

# 2. Create a view and change its dimensions to a 2x3 matrix
matrix_view = original.reshape(2, 3)

# 3. Check the shapes
print("Original Shape:", original.shape)
print("View Shape:    ", matrix_view.shape)
print("\nOriginal Array:\n", original)
print("\nMatrix View:\n", matrix_view)

Original Shape: (6,)
View Shape:     (2, 3)

Original Array:
 [10 20 30 40 50 60]

Matrix View:
 [[10 20 30]
 [40 50 60]]

Changing the data type representation of a view

# 1. Create a flat 1D original array
original = np.array([10, 20, 30, 40, 50, 60])

# 2. View the exact same memory bytes as 16-bit integers
# Because 16-bit is half the size of 64-bit, each number splits into four!
matrix_view = original.view(np.int16)

# 3. Check the shapes
print("Original dtype:", original.dtype)
print("View dtype:    ", matrix_view.dtype)
print("\nOriginal Array:\n", original)
print("\nMatrix View:\n", matrix_view)

Original dtype: int64
View dtype:     int16

Original Array:
 [10 20 30 40 50 60]

Matrix View:
 [10  0  0  0 20  0  0  0 30  0  0  0 40  0  0  0 50  0  0  0 60  0  0  0]

Indexing, slicing, and `where` conditions

Reference arrays

arr = np.arange(20)
arr_2d = np.arange(24).reshape(4, 6)

print("Original 1D:", arr)
print("\n2D array:\n", arr_2d)

Original 1D: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]

2D array:
 [[ 0  1  2  3  4  5]
 [ 6  7  8  9 10 11]
 [12 13 14 15 16 17]
 [18 19 20 21 22 23]]

Basic indexing

print("\nElement at index 5:", arr[5])
print("Element at [0, 2]:", arr_2d[0, 2])
print("First row:", arr_2d[0])
print("Last column:", arr_2d[:, -1])

Element at index 5: 5
Element at [0, 2]: 2
First row: [0 1 2 3 4 5]
Last column: [ 5 11 17 23]

Boolean indexing

mask = (arr > 10) & (arr < 15)
print("\nArr > 10 and < 15:", arr[mask])

Arr > 10 and < 15: [11 12 13 14]

Fancy indexing through explicit index selection

indices = [0, 5, 10, 15]
print("arr[[0, 5, 10, 15]]:", arr[indices])

arr[[0, 5, 10, 15]]: [ 0  5 10 15]

Slicing operations

print("\narr[5:10]:", arr[5:10])
print("arr[::2]:", arr[::2])  # Every 2nd element
print("arr[::-1]:", arr[::-1])  # Reversed

arr[5:10]: [5 6 7 8 9]
arr[::2]: [ 0  2  4  6  8 10 12 14 16 18]
arr[::-1]: [19 18 17 16 15 14 13 12 11 10  9  8  7  6  5  4  3  2  1  0]

Two-dimensional slicing

print("\narr_2d[1:3, 2:5]:\n", arr_2d[1:3, 2:5])
print("\narr_2d[:, 1]:", arr_2d[:, 1])  # All rows, column 1

arr_2d[1:3, 2:5]:
 [[ 8  9 10]
 [14 15 16]]

arr_2d[:, 1]: [ 1  7 13 19]

Conditional selection with where

result = np.where(arr > 10, arr, 0)
print("\nWhere arr > 10:", result)

Where arr > 10: [ 0  0  0  0  0  0  0  0  0  0  0 11 12 13 14 15 16 17 18 19]

Arithmetic and mathematical operations

Unlike Python lists, NumPy applies operations across entire ndarrays.

Reference arrays

a = np.array([1, 2, 3, 4, 5])
b = np.array([10, 20, 30, 40, 50])

Basic arithmetic operations

print("a + b:", a + b)
print("a - b:", a - b)
print("a * b:", a * b)
print("b / a:", b / a)
print("a ** 2:", a ** 2)

a + b: [11 22 33 44 55]
a - b: [ -9 -18 -27 -36 -45]
a * b: [ 10  40  90 160 250]
b / a: [10. 10. 10. 10. 10.]
a ** 2: [ 1  4  9 16 25]

Universal functions

print("\nSquare root:", np.sqrt(a))
print("Absolute value:", np.abs(np.array([-1, -2, 3])))
print("Exponential:", np.exp(np.array([1, 2, 3])))
print("Logarithm:", np.log(np.array([1, 2.718, 10])))

Square root: [1.         1.41421356 1.73205081 2.         2.23606798]
Absolute value: [1 2 3]
Exponential: [ 2.71828183  7.3890561  20.08553692]
Logarithm: [0.         0.99989632 2.30258509]

Trigonometric functions

angles = np.array([0, np.pi/4, np.pi/2, np.pi])

print("\nSine:", np.sin(angles))
print("Cosine:", np.cos(angles))
print("Tangent:", np.tan(angles))

Sine: [0.00000000e+00 7.07106781e-01 1.00000000e+00 1.22464680e-16]
Cosine: [ 1.00000000e+00  7.07106781e-01  6.12323400e-17 -1.00000000e+00]
Tangent: [ 0.00000000e+00  1.00000000e+00  1.63312394e+16 -1.22464680e-16]

Rounding functions

decimals = np.array([1.234, 5.678, 2.567])

print("\nCeiling:", np.ceil(decimals))
print("Floor:", np.floor(decimals))
print("Round:", np.round(decimals, 2))

Ceiling: [2. 6. 3.]
Floor: [1. 5. 2.]
Round: [1.23 5.68 2.57]

Statistical functions

Reference arrays

arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
arr_2d = np.arange(1, 13).reshape(3, 4)

Basic descriptive statistics

print("Sum:", np.sum(arr))
print("Mean:", np.mean(arr))
print("Median:", np.median(arr))
print("Std Dev:", np.std(arr))
print("Variance:", np.var(arr))

Sum: 55
Mean: 5.5
Median: 5.5
Std Dev: 2.8722813232690143
Variance: 8.25

Axis-wise statistics (2D example)

print("\n2D array:\n", arr_2d)
print("\nSum along axis 0 (columns):", np.sum(arr_2d, axis=0))
print("Sum along axis 1 (rows):", np.sum(arr_2d, axis=1))
print("Mean along axis 0:", np.mean(arr_2d, axis=0))
print("Mean along axis 1:", np.mean(arr_2d, axis=1))

2D array:
 [[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]

Sum along axis 0 (columns): [15 18 21 24]
Sum along axis 1 (rows): [10 26 42]
Mean along axis 0: [5. 6. 7. 8.]
Mean along axis 1: [ 2.5  6.5 10.5]

Minimum and maximum functions

print("\nMin:", np.min(arr))
print("Max:", np.max(arr))
print("Argmin (index):", np.argmin(arr))
print("Argmax (index):", np.argmax(arr))

Min: 1
Max: 10
Argmin (index): 0
Argmax (index): 9

Percentiles

print("\n25th percentile:", np.percentile(arr, 25))
print("50th percentile (median):", np.percentile(arr, 50))
print("75th percentile:", np.percentile(arr, 75))

25th percentile: 3.25
50th percentile (median): 5.5
75th percentile: 7.75

Cumulative operations

print("\nCumulative sum:", np.cumsum(arr[:5]))
print("Cumulative product:", np.cumprod(np.array([1, 2, 3, 4])))
print("Cumulative max:", np.maximum.accumulate(np.array([1, 3, 2, 5, 4])))

Cumulative sum: [ 1  3  6 10 15]
Cumulative product: [ 1  2  6 24]
Cumulative max: [1 3 3 5 5]

Array manipulation

Reference arrays

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
c = np.array([[6, 7, 8, ], [9, 10, 11]])

Concatenation

concat = np.concatenate([a, b])
print("Concatenate:", concat)

Concatenate: [1 2 3 4 5 6]

stack, hstack, and vstack

stacked = np.stack([a, b])
print("\nStack:\n", stacked)

# horizontal
hstacked = np.hstack([a, b])
print("\nHStack:", hstacked)

# vertical
vstacked = np.vstack([[a], [b]])
print("\nVStack:\n", vstacked)

Stack:
 [[1 2 3]
 [4 5 6]]

HStack: [1 2 3 4 5 6]

VStack:
 [[1 2 3]
 [4 5 6]]

Splitting arrays

arr = np.arange(10)
split_result = np.array_split(arr, 3)
print("\nArray_split into 3 parts:")
for i, part in enumerate(split_result):
    print(f"  Part {i}: {part}")

Array_split into 3 parts:
  Part 0: [0 1 2 3]
  Part 1: [4 5 6]
  Part 2: [7 8 9]

Transposition

print("\nOriginal:\n", c)
print("Transposed:\n", c.T)

Original:
 [[ 6  7  8]
 [ 9 10 11]]
Transposed:
 [[ 6  9]
 [ 7 10]
 [ 8 11]]

Unique values

arr_with_dupes = np.array([1, 2, 2, 3, 3, 3, 4])
print("\nUnique values:", np.unique(arr_with_dupes))

Unique values: [1 2 3 4]

Sorting

arr_unsorted = np.array([3, 1, 4, 1, 5, 9, 2, 6])
print("Sorted:", np.sort(arr_unsorted))
print("Argsort (indices):", np.argsort(arr_unsorted))

Sorted: [1 1 2 3 4 5 6 9]
Argsort (indices): [1 3 6 0 2 4 7 5]

Linear algebra

Dot product and matrix multiplication

# 1d ndarrays
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

dot_product = np.dot(a, b)
print("Dot product:", dot_product)  # 1*4 + 2*5 + 3*6 = 32

# 2d ndarrays
mat_a = np.array([[1, 5], [3, 4]])
mat_b = np.array([[5, 6], [7, 8]])

matrix_product = np.dot(mat_a, mat_b)
print("\nMatrix product:\n", matrix_product)

# Using the @ operator for matrix multiplication
matrix_product_operator = mat_a @ mat_b
print("\nMatrix product using @ operator:\n", matrix_product_operator)

Dot product: 32

Matrix product:
 [[40 46]
 [43 50]]

Matrix product using @ operator:
 [[40 46]
 [43 50]]

Trace (sum of diagonal elements)

print("Trace:", np.trace(mat_a))

Trace: 5

linalg: linear algebra submodule

# Determinant
det = np.linalg.det(mat_a)
print("\nDeterminant:", det)

# Inverse
inv = np.linalg.inv(mat_a)
print("\nInverse:\n", inv)

# Eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(mat_a)
print("\nEigenvalues:", eigenvalues)
print("Eigenvectors:\n", eigenvectors)

# Rank
print("\nRank:", np.linalg.matrix_rank(mat_a))

# Norm
print("\nNorm (default):", np.linalg.norm(a))
print("Norm (L2):", np.linalg.norm(a, ord=2))
print("Norm (L1):", np.linalg.norm(a, ord=1))

Determinant: -11.000000000000002

Inverse:
 [[-0.36363636  0.45454545]
 [ 0.27272727 -0.09090909]]

Eigenvalues: [-1.65331193+0.j  6.65331193+0.j]
Eigenvectors:
 [[-0.88333068+0.j -0.66249905+0.j]
 [ 0.46875037+0.j -0.74906275+0.j]]

Rank: 2

Norm (default): 3.7416573867739413
Norm (L2): 3.7416573867739413
Norm (L1): 6.0

Broadcasting

Broadcasting enables operations on ndarrays with different shapes, provided that their dimensions are compatible.

Array and scalar broadcasting

arr = np.array([1, 2, 3, 4, 5])
result = arr + 10
print("Array + scalar:", result)

Array + scalar: [11 12 13 14 15]

One-dimensional and two-dimensional ndarray broadcasting

arr_1d = np.array([1, 2, 3])
arr_2d = np.array([[10], [20], [30]])

result = arr_1d + arr_2d
print("\n1D + 2D (broadcasting):")
print("Shape (3,) + (3, 1) = (3, 3)")
print(result)

1D + 2D (broadcasting):
Shape (3,) + (3, 1) = (3, 3)
[[11 12 13]
 [21 22 23]
 [31 32 33]]

Operations across dimensions

matrix = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
column = np.array([10, 20, 30])

print("\nSubtract column from matrix:")
print(matrix - column)

Subtract column from matrix:
[[ -9 -18 -27]
 [ -6 -15 -24]
 [ -3 -12 -21]]

Broadcasting rules:

If arrays have different ranks, pad the smaller shape with leading dimensions of size 1.
Check that each aligned dimension is compatible (equal, or one of them is 1).
Any dimension of size 1 is conceptually stretched to match the corresponding larger dimension.

Broadcasting rules: examples

Shape (5,) broadcasts with (3, 5) -> (3, 5)
Shape (3, 1) broadcasts with (3, 4) -> (3, 4)
Shape (1, 5) broadcasts with (3, 5) -> (3, 5)

Random number generation

Set seed for reproducibility

np.random.seed(1000)  # For reproducibility

Uniform distribution on [0, 1)

uniform = np.random.rand(5)
print("Uniform [0, 1):", uniform)

Uniform [0, 1): [0.65358959 0.11500694 0.95028286 0.4821914  0.87247454]

Random integers

ints = np.random.randint(1, 10, size=5)
print("Random integers [1, 10):", ints)

Random integers [1, 10): [9 5 5 5 3]

Normal (Gaussian) distribution

normal = np.random.randn(5)
print("Normal distribution:", normal)

Normal distribution: [ 0.57363145 -0.74841131 -0.4122031  -0.07400906 -0.92893693]

Normal distribution with custom mean and standard deviation

custom_normal = np.random.normal(loc=100, scale=15, size=5)
print("\nNormal (μ=100, σ=15):", custom_normal)

Normal (μ=100, σ=15): [120.85092205 117.92603993 110.61013587 114.8944316  102.09195908]

Exponential distribution

exponential = np.random.exponential(scale=2.0, size=5)
print("Exponential (λ=0.5):", exponential)

Exponential (λ=0.5): [4.69093541 0.02095277 0.1549649  0.56109307 0.28613573]

Random choice from an ndarray

arr = np.arange(10)
choices = np.random.choice(arr, size=5, replace=False)
print("\nRandom choice (no replace):", choices)

Random choice (no replace): [4 2 8 0 3]

In-place shuffling

arr = np.arange(10)
np.random.shuffle(arr)
print("Shuffled:", arr)

Shuffled: [4 9 5 1 3 6 2 0 8 7]

Shuffling with a copied permutation

arr = np.arange(10)
shuffled = np.random.permutation(arr)
print("Permutation:", shuffled)

Permutation: [2 8 7 1 4 0 5 6 9 3]

Binomial distribution

binomial = np.random.binomial(n=10, p=0.5, size=5)
print("\nBinomial (n=10, p=0.5):", binomial)

Binomial (n=10, p=0.5): [6 4 5 5 5]

File input/output and import/export

Setup code

import os
import tempfile
import numpy as np

# Create sample array
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

# Create temp directory for demo
temp_dir = tempfile.mkdtemp()

print(f"original ndarray: {arr}")
print(f"Temporary directory created at: {temp_dir}")

original ndarray: [[1 2 3]
 [4 5 6]
 [7 8 9]]
Temporary directory created at: /tmp/tmpr0nzny9i

Save in .npy format (binary)

npy_path = os.path.join(temp_dir, 'array.npy')
np.save(npy_path, arr)
print(f"Saved .npy file to {npy_path}")

Saved .npy file to /tmp/tmpr0nzny9i/array.npy

Load .npy file

loaded_arr = np.load(npy_path)
print("Loaded from .npy:\n", loaded_arr)

Loaded from .npy:
 [[1 2 3]
 [4 5 6]
 [7 8 9]]

Save multiple arrays as .npz (compressed)

npz_path = os.path.join(temp_dir, 'arrays.npz')
arr2 = np.array([10, 20, 30, 40])
np.savez(npz_path, array1=arr, array2=arr2)
print(f"\nSaved .npz file to {npz_path}")

Saved .npz file to /tmp/tmpr0nzny9i/arrays.npz

Load .npz file

loaded = np.load(npz_path)
print("Loaded from .npz:")
print("  array1:\n", loaded['array1'])
print("  array2:", loaded['array2'])

Loaded from .npz:
  array1:
 [[1 2 3]
 [4 5 6]
 [7 8 9]]
  array2: [10 20 30 40]

Save as a text file (CSV-like format)

txt_path = os.path.join(temp_dir, 'array.txt')
np.savetxt(txt_path, arr, delimiter=',', fmt='%d')
print(f"\nSaved text file to {txt_path}")

Saved text file to /tmp/tmpr0nzny9i/array.txt

Load from a text file

loaded_txt = np.loadtxt(txt_path, delimiter=',')
print("Loaded from text:\n", loaded_txt)

Loaded from text:
 [[1. 2. 3.]
 [4. 5. 6.]
 [7. 8. 9.]]

Useful functions and advanced techniques

Apply a function to each element

arr = np.array([1, 2, 3, 4, 5])
squared = np.vectorize(lambda x: x**2)(arr)
print("Vectorized function (square):", squared)

Vectorized function (square): [ 1  4  9 16 25]

Piecewise operations

arr = np.array([1, 2, 3, 4, 5])
result = np.piecewise(arr, [arr < 3, arr >= 3], [lambda x: x**2, lambda x: x*10])
print("\nPiecewise (x<3: x², x≥3: 10x):", result)

Piecewise (x<3: x², x≥3: 10x): [ 1  4 30 40 50]

Apply operations along an axis

matrix = np.array([[1, 2, 3], [4, 5, 6]])
sums0 = np.apply_along_axis(np.sum, axis=0, arr=matrix)
sums1 = np.apply_along_axis(np.sum, axis=1, arr=matrix)

print("\nApply sum along axis 0:", sums0)
print("Apply sum along axis 1:", sums1)

Apply sum along axis 0: [5 7 9]
Apply sum along axis 1: [ 6 15]

Repeat and tile

arr = np.array([1, 2, 3])
print("\nRepeat (each element 2 times):", np.repeat(arr, 2))
print("Tile (whole array 2 times):", np.tile(arr, 2))

Repeat (each element 2 times): [1 1 2 2 3 3]
Tile (whole array 2 times): [1 2 3 1 2 3]

Reduction operations

arr = np.array([1, 2, 3, 4, 5])
result = np.add.reduce(arr)  # Sum
print("\nReduce with add (sum):", result)

Reduce with add (sum): 15

searchsorted (binary search)

sorted_arr = np.array([1, 3, 5, 7, 9])
indices = np.searchsorted(sorted_arr, [2, 4, 6, 8])
print("\nSearchsorted indices:", indices)

Searchsorted indices: [1 2 3 4]

Extract diagonal elements

matrix = np.arange(9).reshape(3, 3)
diagonal0 = np.diag(matrix, k=0)  # Main diagonal
diagonal1 = np.diag(matrix, k=1)  # Diagonal above main
print("\nDiagonal of matrix:\n", matrix)
print("Diagonal elements:", diagonal0)
print("Diagonal above main:", diagonal1)

Diagonal of matrix:
 [[0 1 2]
 [3 4 5]
 [6 7 8]]
Diagonal elements: [0 4 8]
Diagonal above main: [1 5]

Create a diagonal matrix

diag_matrix = np.diag([1, 2, 3])
print("\nDiagonal matrix from [1, 2, 3]:\n", diag_matrix)

Diagonal matrix from [1, 2, 3]:
 [[1 0 0]
 [0 2 0]
 [0 0 3]]

Count and display non-zero values

arr = np.array([0, 1, 0, 2, 3, 0])
print("\nNonzero count:", np.count_nonzero(arr))
print("Nonzero indices:", np.nonzero(arr))

Nonzero count: 3
Nonzero indices: (array([1, 3, 4]),)

Tips, techniques, and performance

Setup code

import time

Avoid Python loops by using vectorisation

arr = np.arange(1_000_000)

# Slow: Python loop
start = time.time()
result = np.array([x**2 for x in arr])
loop_time = time.time() - start
print(f"Python loop: {loop_time:.6f} seconds")

# Fast: NumPy vectorization
start = time.time()
result = arr ** 2
vectorized_time = time.time() - start
print(f"NumPy vectorized: {vectorized_time:.6f} seconds")
print(f"Speedup: {loop_time/vectorized_time:.1f}x faster\n")

=== Performance: Vectorization ===
Python loop: 0.238411 seconds
NumPy vectorized: 0.001625 seconds
Speedup: 146.7x faster

Use in-place operations where appropriate

arr = np.arange(5)
print("Original:", arr)
arr += 10  # In-place (more memory efficient)
print("After += 10:", arr)

Original: [0 1 2 3 4]
After += 10: [10 11 12 13 14]

Data types and memory usage

arr_float64 = np.arange(1000, dtype=np.float64)
arr_float32 = np.arange(1000, dtype=np.float32)
arr_int32 = np.arange(1000, dtype=np.int32)

print(f"Float64: {arr_float64.nbytes} bytes")
print(f"Float32: {arr_float32.nbytes} bytes")
print(f"Int32: {arr_int32.nbytes} bytes")

Float64: 8000 bytes
Float32: 4000 bytes
Int32: 4000 bytes

Memory efficiency: views versus copies

original = np.arange(10)
view = original[:]  # This is a view, shares memory
copy = original[:].copy()  # This is a copy

print(f"View shares memory: {view.base is original}")
print(f"Copy doesn't share memory: {copy.base is original}")

View shares memory: True
Copy doesn't share memory: False

Useful diagnostics for debugging

arr = np.random.randn(3, 4, 5)

print(f"Shape: {arr.shape}")
print(f"Ndim: {arr.ndim}")
print(f"Dtype: {arr.dtype}")
print(f"Size: {arr.size}")
print(f"Memory: {arr.nbytes} bytes")

Shape: (3, 4, 5)
Ndim: 3
Dtype: float64
Size: 60
Memory: 480 bytes

Check for NaN and infinite values

arr = np.array([1, 2, np.nan, 4, np.inf, -np.inf])

print(f"Array: {arr}")
print(f"Has NaN: {np.isnan(arr).any()}")
print(f"Has Inf: {np.isinf(arr).any()}")
print(f"Is finite: {np.isfinite(arr)}")

Array: [  1.   2.  nan   4.  inf -inf]
Has NaN: True
Has Inf: True
Is finite: [ True  True False  True False False]

Type casting

arr = np.array([1.5, 2.7, 3.2])

print(f"Original (float): {arr}")
print(f"As int: {arr.astype(int)}")
print(f"As str: {arr.astype(str)}")

Original (float): [1.5 2.7 3.2]

As int: [1 2 3]

As str: ['1.5' '2.7' '3.2']

Conclusion

NumPy is a core component of scientific computing in Python. This tutorial has outlined how NumPy arrays differ from Python lists, how they can be created and inspected, and how indexing, arithmetic, and statistical operations can be performed efficiently.

A key strength of NumPy lies in its speed and vectorised programming model. Rather than relying on explicit Python loops for every calculation, practitioners can apply concise operations to complete datasets. This capability makes NumPy an essential tool for data analysis, machine learning, and numerical modelling.

To develop proficiency, practise regularly with arrays of different shapes, slicing patterns, and reshaping strategies. Comparing NumPy workflows with equivalent pure-Python approaches is particularly useful for understanding performance and expressiveness benefits. These foundations also support more advanced work with libraries such as Pandas, SciPy, and TensorFlow.

This article has presented a concise, practice-oriented reference for fundamental NumPy workflows. For continued development, readers are encouraged to extend these examples to domain-specific datasets and to evaluate computational trade-offs in realistic analytical pipelines.

Did this article help you? Let me know in the comments below, and don't forget to drop a like if you enjoyed the read! Thank you.