DEV Community: Muhammad Ikramullah Khan

Advanced Git: The Commands That Save Your Career

Muhammad Ikramullah Khan — Sat, 09 May 2026 16:20:00 +0000

You just committed your API keys to GitHub. All of them. AWS keys, database passwords, Stripe API secrets. Everything.

You pushed before you realized. The commit is on GitHub. Public repository. Anyone can see it.

You delete the commit locally. Push again. But it's too late. The keys are in the Git history. Forever. Anyone can checkout that old commit and see your secrets.

You're panicking. Do you delete the entire repository? Start over? Hope nobody noticed?

Or this happens. You've been working on a feature for three days. Made 15 commits. Then you realize you've been committing to main instead of a feature branch. Your production code now has half-finished, untested features mixed in.

Or this one. You accidentally deleted a file. Committed the deletion. Pushed it. Two days later, you need that file back. But it's gone. Or is it?

These situations feel impossible. Like you broke Git permanently. Like you need to start the project over.

You don't. Git has commands for these exact scenarios. Commands that rewrite history. Recover deleted code. Remove secrets from every commit. Undo mistakes that seem permanent.

These are the advanced Git commands. The ones you hope you never need. But when you need them, they save everything.

Let me show you.

The Three Ways to Undo: Reset vs Revert vs Rebase

Git has three main ways to undo commits. They do different things.

Git Reset: Rewind Time

What it does: Moves the branch pointer backward, like the commits never happened.

When to use: Undo commits that you haven't pushed yet (local only).

Three modes:

1. Soft reset (keep changes, uncommit):

git reset --soft HEAD~1

This uncommits the last commit but keeps all changes staged. Like you never ran git commit.

Use case: Wrong commit message. Reset, recommit with correct message.

# You committed with bad message
git commit -m "fixed stuff"

# Reset (keeps changes staged)
git reset --soft HEAD~1

# Commit again with good message
git commit -m "Fix pagination bug in product scraper"

2. Mixed reset (keep changes, unstage):

git reset HEAD~1

(Default mode, same as --mixed)

This uncommits AND unstages, but keeps changes in your working directory. Like you never ran git add or git commit.

Use case: Committed too early. Want to modify code before committing.

# You committed but forgot to add error handling
git commit -m "Add login feature"

# Reset (keeps changes, unstages them)
git reset HEAD~1

# Add error handling
echo "try/except" >> login.py

# Now commit everything together
git add .
git commit -m "Add login feature with error handling"

3. Hard reset (delete everything):

git reset --hard HEAD~1

This deletes the commit AND all changes. Gone. Like the work never happened.

Use case: Completely undo bad work.

# You tried a feature, it doesn't work, delete everything
git reset --hard HEAD~1

# All changes from that commit are gone

⚠️ Warning: --hard is destructive. You lose uncommitted work.

Git Revert: Create Opposite Commit

What it does: Creates a new commit that undoes a previous commit.

When to use: Undo commits that you already pushed (public commits).

Example:

# You have these commits:
# C: Add pagination
# B: Add error handling
# A: Initial commit

# Pagination introduced a bug
# Undo commit C (keep B and A)
git revert C

This creates commit D that removes everything commit C added.

Why not reset? Reset rewrites history. If you already pushed, teammates have the old history. Reset causes chaos.

Revert creates new history on top. Everyone can pull it cleanly.

Example:

# See commit history
git log --oneline

# Output:
# d4e5f6g Add pagination
# c3d4e5f Add error handling
# b2c3d4e Initial commit

# Revert the pagination commit
git revert d4e5f6g

# New commit created that undoes pagination
git log --oneline

# Output:
# a1a1a1a Revert "Add pagination"
# d4e5f6g Add pagination
# c3d4e5f Add error handling
# b2c3d4e Initial commit

The pagination code is gone, but the history shows you added it and then reverted it. Transparent.

Git Rebase: Rewrite History

What it does: Rewrites commits to change their history, order, or content.

When to use: Clean up messy commit history before pushing.

Example:

# You have messy commits:
# E: Fix typo
# D: Fix another typo
# C: Actually add the feature
# B: WIP progress
# A: Start feature

# Rebase to combine into one clean commit
git rebase -i HEAD~5

This opens an editor showing all 5 commits. You can:

Squash (combine commits)
Reword (change commit message)
Reorder (change sequence)
Drop (delete commits)

We'll cover this in detail later.

Quick Reference

Command	Changes History?	Use When	Safety
`reset --soft`	Yes	Recommit with better message	Safe (keeps changes)
`reset --mixed`	Yes	Modify before committing	Safe (keeps changes)
`reset --hard`	Yes	Delete bad work completely	DANGEROUS (loses work)
`revert`	No	Undo public commits	Safe
`rebase`	Yes	Clean up before pushing	Medium (rewrite history)

Golden rule: Never rewrite history that you've pushed and others have pulled. Use revert instead.

Recovering Deleted Commits

You ran git reset --hard and deleted commits. Then realized you needed them.

Good news: Git keeps deleted commits for ~30 days. You can recover them.

The Reflog: Git's Safety Net

Git logs every HEAD movement in the reflog (reference log).

git reflog

Output:

a1b2c3d HEAD@{0}: reset: moving to HEAD~3
d4e5f6g HEAD@{1}: commit: Add pagination
c3d4e5f HEAD@{2}: commit: Add error handling
b2c3d4e HEAD@{3}: commit: Initial feature

Every action is logged. Even the reset that deleted commits.

Recovering a Deleted Commit

Scenario: You reset and deleted commits. You want them back.

# See what you deleted
git reflog

# Find the commit you want (before the reset)
# Let's say it's d4e5f6g

# Create a branch at that commit
git branch recovered d4e5f6g

# Or reset back to it
git reset --hard d4e5f6g

Your "deleted" commits are back.

Real Example

# You're working on a feature
git commit -m "Implement login"
git commit -m "Add validation"
git commit -m "Add error handling"

# You realize you're on main, not a feature branch
# Panic. Delete the commits.
git reset --hard HEAD~3

# Oh no, you needed those commits!
# Check reflog
git reflog

# Output:
# a1a1a1a HEAD@{0}: reset: moving to HEAD~3
# b2b2b2b HEAD@{1}: commit: Add error handling
# c3c3c3c HEAD@{2}: commit: Add validation
# d4d4d4d HEAD@{3}: commit: Implement login

# Create feature branch with those commits
git checkout -b feature/login d4d4d4d

# Or, get the exact commit hash
git cherry-pick d4d4d4d c3c3c3c b2b2b2b

You recovered everything.

How Long Does Reflog Keep Deleted Commits?

Default: 90 days for reachable commits, 30 days for unreachable (deleted) commits.

After that, Git garbage collects them. They're truly gone.

Lesson: If you deleted something, recover it within 30 days.

Removing Secrets from History

You committed API keys. Pushed to GitHub. Now you need to remove them from every commit in history.

Why Deleting the Commit Isn't Enough

# You committed secrets
git commit -m "Add API integration" 
# File contains: API_KEY = "sk_live_abc123xyz"

# You pushed to GitHub
git push

# You realize the mistake
# You remove the key and commit again
sed -i 's/API_KEY = .*/API_KEY = os.getenv("API_KEY")/' config.py
git commit -m "Remove hardcoded API key"
git push

Problem: The key is still in the history. Anyone can do:

git checkout <previous-commit>
cat config.py
# They see: API_KEY = "sk_live_abc123xyz"

The secret is in the Git database forever (unless you rewrite history).

Step 1: Rotate the Secret Immediately

Before fixing Git, make the exposed secret useless.

For API keys:

Log into the service (AWS, Stripe, etc.)
Revoke the exposed key
Generate a new key
Update your local config (not in Git)

Now the exposed key is worthless. Even if someone finds it in Git history, it won't work.

Step 2: Remove Secret from Git History

Use git filter-branch or the newer git filter-repo.

Install git-filter-repo:

# Mac
brew install git-filter-repo

# Linux
pip install git-filter-repo

# Windows
pip install git-filter-repo

Remove a file from all history:

# Remove config.py from all commits
git filter-repo --path config.py --invert-paths

This rewrites every commit, removing config.py as if it never existed.

Remove specific text from all files:

# Remove the API key pattern from all files
git filter-repo --replace-text <(echo 'sk_live_abc123xyz==>REMOVED')

This finds sk_live_abc123xyz in every file in every commit and replaces it with REMOVED.

Step 3: Force Push

# This rewrites GitHub history
git push --force-with-lease origin main

--force-with-lease is safer than --force. It fails if someone else pushed while you were rewriting.

Step 4: Tell Collaborators

Everyone who cloned the repo needs to re-clone:

# Don't pull, re-clone
rm -rf project-name
git clone https://github.com/you/project-name.git

Pulling won't work. The history is different.

Alternative: BFG Repo-Cleaner

BFG is faster than filter-repo for large repos.

Install:

# Mac
brew install bfg

# Or download jar
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar

Remove passwords:

# Create passwords.txt with secrets to remove
echo "sk_live_abc123xyz" > passwords.txt
echo "password123" >> passwords.txt

# Run BFG
bfg --replace-text passwords.txt

# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push
git push --force origin main

BFG is 10-100x faster than filter-branch on large repositories.

Prevention: Use Environment Variables

Never commit secrets. Use environment variables instead.

# Bad (hardcoded)
API_KEY = "sk_live_abc123xyz"

# Good (environment variable)
import os
API_KEY = os.getenv("API_KEY")

Create .env file (gitignored):

# .env
API_KEY=sk_live_abc123xyz
DATABASE_URL=postgres://user:pass@localhost/db

Add .env to .gitignore:

.env
.env.*
!.env.example

Create .env.example (committed):

# .env.example
API_KEY=your_key_here
DATABASE_URL=your_database_url_here

Teammates copy .env.example to .env and fill in their own secrets.

Load in Python:

from dotenv import load_dotenv
import os

load_dotenv()

API_KEY = os.getenv("API_KEY")

Secrets never touch Git.

Git Stash: Temporary Storage

You're working on a feature. Halfway done. Your boss says: "Fix this production bug NOW."

You can't commit (code doesn't work yet). You can't switch branches (changes would come with you).

Solution: Stash.

Basic Stashing

# You're working on feature
echo "half-finished code" >> feature.py

# Emergency! Need to fix bug
# Save work temporarily
git stash

# Your working directory is clean
git status
# Output: nothing to commit, working tree clean

# Switch to main, fix bug
git checkout main
# (fix bug, commit, push)

# Back to feature branch
git checkout feature/my-feature

# Restore your work
git stash pop

Your half-finished code is back. Like you never left.

Stash with Description

# Stash with message
git stash save "WIP: half-done pagination feature"

# List stashes
git stash list

# Output:
# stash@{0}: On feature/pagination: WIP: half-done pagination feature
# stash@{1}: On main: WIP: fixing typo

Multiple Stashes

# Stash current work
git stash save "Feature A progress"

# Work on something else
echo "other work" >> other.py

# Stash that too
git stash save "Feature B progress"

# List all stashes
git stash list

# Apply specific stash
git stash apply stash@{1}  # Applies Feature A

# Pop removes from stash list, apply keeps it

Stashing Untracked Files

By default, stash only saves tracked files (files Git knows about).

# Stash including untracked files
git stash -u

# Or
git stash --include-untracked

Viewing Stash Contents

# See what's in a stash
git stash show

# See detailed diff
git stash show -p

Dropping Stashes

# Remove specific stash
git stash drop stash@{0}

# Remove all stashes
git stash clear

Real Example: Context Switching

# Working on scraper feature
vim scraper.py
# (edit, edit, edit... not done yet)

# Boss: "Production is down! Fix now!"
git stash save "Half-done scraper refactor"

# Fix production
git checkout main
vim hotfix.py
git add hotfix.py
git commit -m "Fix production crash"
git push

# Back to feature
git checkout feature/scraper-refactor
git stash pop

# Continue where you left off
vim scraper.py

Stash lets you context-switch without losing work or committing garbage.

Interactive Rebase: Rewriting History

You made 10 commits while building a feature. Half are typo fixes. Messages say "WIP" and "test". You want to clean this up before pushing.

Interactive rebase lets you:

Combine (squash) commits
Reword messages
Reorder commits
Delete commits
Edit commits

Basic Interactive Rebase

# Rebase last 5 commits
git rebase -i HEAD~5

This opens an editor:

pick a1b2c3d Implement login
pick d4e5f6g Fix typo
pick g7h8i9j Actually fix typo
pick j1k2l3m Add validation
pick m4n5o6p WIP

# Commands:
# p, pick = use commit
# r, reword = use commit, but edit message
# e, edit = use commit, but stop to amend
# s, squash = use commit, meld into previous
# f, fixup = like squash, discard commit message
# d, drop = remove commit

Example 1: Squash Typo Fixes

pick a1b2c3d Implement login
fixup d4e5f6g Fix typo
fixup g7h8i9j Actually fix typo
pick j1k2l3m Add validation
fixup m4n5o6p WIP

This combines commits:

Login implementation + 2 typo fixes = 1 commit
Validation + WIP = 1 commit

Result: 2 clean commits instead of 5 messy ones.

Example 2: Reword Messages

pick a1b2c3d Implement login
reword d4e5f6g Fix typo
pick g7h8i9j Add validation

Git will pause at d4e5f6g and let you change the message to something better like "Add input validation for login form".

Example 3: Reorder Commits

pick g7h8i9j Add validation
pick a1b2c3d Implement login
pick d4e5f6g Fix typo

Commits will be reordered. Validation comes first, then login, then typo fix.

⚠️ Careful: Reordering can cause conflicts if commits depend on each other.

Example 4: Drop Commits

pick a1b2c3d Implement login
drop d4e5f6g Debug commit (remove this)
pick g7h8i9j Add validation

The debug commit disappears from history.

Real Example: Cleaning Feature Branch

Before rebase:

git log --oneline

# Output:
# m4n5o6p Fix another typo
# j1k2l3m Fix typo
# g7h8i9j Add tests
# d4e5f6g WIP: still working
# a1b2c3d Implement pagination

Rebase:

git rebase -i HEAD~5

In editor:

pick a1b2c3d Implement pagination
fixup d4e5f6g WIP: still working
reword g7h8i9j Add tests
fixup j1k2l3m Fix typo
fixup m4n5o6p Fix another typo

After rebase:

git log --oneline

# Output:
# b2b2b2b Add comprehensive tests for pagination
# a1a1a1a Implement pagination with automatic page detection

Two clean commits. Ready to push.

When to Use Interactive Rebase

Good times:

Before pushing a feature branch
Cleaning up messy local history
Combining "fix typo" commits
Rewriting bad commit messages

Bad times:

After pushing (if others pulled it)
On the main branch
On shared branches

Rule: Only rebase commits you haven't pushed.

Cherry-Picking: Copying Commits

You have a commit on one branch. You want it on another branch. Cherry-pick copies it.

Basic Cherry-Pick

# On branch A, you made a bugfix
git checkout branch-a
git commit -m "Fix critical bug"
# Commit hash: d4e5f6g

# Switch to branch B
git checkout branch-b

# Copy the bugfix commit to branch B
git cherry-pick d4e5f6g

Branch B now has that bugfix, as if you made the commit there.

Real Example: Hotfix to Multiple Branches

Scenario: You have three branches:

main (production)
develop (integration)
feature/new-ui (in progress)

You fixed a critical bug on main. Need it on all branches.

# Fix on main
git checkout main
vim bugfix.py
git commit -m "Fix security vulnerability"
# Commit: a1b2c3d

# Apply to develop
git checkout develop
git cherry-pick a1b2c3d

# Apply to feature branch
git checkout feature/new-ui
git cherry-pick a1b2c3d

All three branches have the fix.

Cherry-Picking Multiple Commits

# Pick commits a1b2c3d, d4e5f6g, and g7h8i9j
git cherry-pick a1b2c3d d4e5f6g g7h8i9j

# Pick a range
git cherry-pick a1b2c3d..g7h8i9j

Handling Cherry-Pick Conflicts

Sometimes the commit doesn't apply cleanly.

git cherry-pick d4e5f6g

# Output:
# CONFLICT (content): Merge conflict in scraper.py
# error: could not apply d4e5f6g... Fix bug

Resolve like a merge conflict:

# Edit scraper.py, fix conflicts
vim scraper.py

# Mark resolved
git add scraper.py

# Continue cherry-pick
git cherry-pick --continue

Or abort:

git cherry-pick --abort

When to Use Cherry-Pick

Good uses:

Applying hotfixes to multiple branches
Copying specific features between branches
Recovering a commit from a deleted branch

Bad uses:

Copying many commits (use merge or rebase instead)
As a replacement for proper branching

Git Bisect: Finding the Bug-Introducing Commit

Your code worked last week. Now it's broken. You have 100 commits since then. Which one broke it?

Bisect does binary search through commits to find the bug.

How Bisect Works

Mark current commit as "bad" (has the bug)
Mark an old commit as "good" (doesn't have the bug)
Git checks out the middle commit
You test: is the bug present?
Mark it good or bad
Git narrows the range
Repeat until you find the exact commit that introduced the bug

Using Bisect

# Start bisect
git bisect start

# Current commit is bad (has bug)
git bisect bad

# Commit from last week was good
git bisect good a1b2c3d

Git checks out a commit in the middle.

# Test the code
python scraper.py

# If bug is present:
git bisect bad

# If bug is NOT present:
git bisect good

Git checks out another commit. Repeat testing.

Eventually:

d4e5f6g is the first bad commit
commit d4e5f6g
Author: John Doe
Date:   Mon Mar 18 14:32:10 2024

    Refactor pagination logic

:100644 100644 a1a1a1a b2b2b2b M    scraper.py

Found it! Commit d4e5f6g introduced the bug.

# Exit bisect
git bisect reset

Automated Bisect

If you have a test script:

# test.sh returns 0 if good, 1 if bad
#!/bin/bash
python scraper.py
# Exit 0 if works, exit 1 if broken

# Run bisect automatically
git bisect start
git bisect bad
git bisect good a1b2c3d
git bisect run ./test.sh

Git runs the script on each commit. Finds the bug automatically.

Real Example

# Pagination worked last Friday
# Today (Monday) it's broken
# 50 commits since Friday

git bisect start
git bisect bad  # Current commit is bad

# Find Friday's commit
git log --since="5 days ago" --oneline

# Mark Friday's last commit as good
git bisect good f1f1f1f

# Git checks out middle commit
# Test:
python scraper.py --test-pagination
# Broken

git bisect bad

# Git checks out another
# Test:
python scraper.py --test-pagination
# Works!

git bisect good

# Continue until:
# a7a7a7a is the first bad commit
# "Optimize page detection logic"

# Found the commit that broke pagination
git bisect reset

# Fix the bug in that commit's code
git revert a7a7a7a

Bisect saved you from manually checking 50 commits.

Amending Commits

You committed. Then immediately realized you forgot something.

Amend Last Commit

# You committed
git commit -m "Add login feature"

# Forgot to add a file
echo "validation code" >> validate.py

# Add it to the last commit
git add validate.py
git commit --amend --no-edit

The file is now part of the last commit. Commit message unchanged.

Amend Commit Message

# Bad message
git commit -m "stuff"

# Fix it
git commit --amend -m "Implement user authentication with JWT tokens"

Amend and Edit Message

git commit --amend

Opens editor. You can change message and review what's being committed.

⚠️ Warning About Amending

Only amend commits you haven't pushed.

If you already pushed and then amend, you'll need to force push:

git push --force

This rewrites history on GitHub. Bad for collaboration.

Better: Just make a new commit fixing the mistake.

Git Reflog: Complete History

Reflog records every HEAD movement. It's your safety net.

Viewing Reflog

git reflog

Output:

a1b2c3d HEAD@{0}: commit: Add feature
d4e5f6g HEAD@{1}: checkout: moving from main to feature
g7h8i9j HEAD@{2}: commit: Fix bug
j1k2l3m HEAD@{3}: reset: moving to HEAD~1

Every checkout, commit, reset, merge, rebase... everything.

Using Reflog to Undo

Scenario: You ran git reset --hard and deleted 5 commits. You want them back.

# Check reflog
git reflog

# Find where you were before reset
# HEAD@{1}: commit: Last good commit

# Go back to that state
git reset --hard HEAD@{1}

Your commits are restored.

Reflog for Specific Branch

git reflog show feature/login

Shows all movements on that branch.

Reflog Expiration

Reflog keeps entries for:

90 days (reachable commits)
30 days (unreachable/deleted commits)

After that, Git garbage collects them.

Check what will be deleted:

git reflog expire --expire-unreachable=now --all --dry-run

Advanced Reset Techniques

Reset to Specific Commit

# Go back 3 commits
git reset --hard HEAD~3

# Go to specific commit
git reset --hard a1b2c3d

# Go to state 5 moves ago in reflog
git reset --hard HEAD@{5}

Reset Specific File

Undo changes to one file without affecting others.

# Unstage a file
git reset HEAD scraper.py

# Restore file to last commit
git checkout HEAD scraper.py

# Or (newer syntax)
git restore scraper.py

Reset Remote Branch

Your local main is ahead of GitHub. You want to match GitHub exactly.

# Fetch latest
git fetch origin

# Reset local to match remote
git reset --hard origin/main

All local commits ahead of remote are gone.

Handling "Detached HEAD" State

You checkout an old commit. Git says: "You are in 'detached HEAD' state."

What this means: You're looking at an old commit. Any new commits won't be on a branch.

Getting Out of Detached HEAD

Option 1: Create a branch here

# You're at an old commit
git checkout a1b2c3d

# Detached HEAD warning appears

# Create branch to save this state
git checkout -b recover-old-feature

Now you're on a branch. Can commit normally.

Option 2: Go back to a branch

# Return to main
git checkout main

You're back on a branch.

Using Detached HEAD Safely

Detached HEAD is useful for exploring old code:

# Look at code from last month
git log --since="1 month ago"
git checkout <commit-from-last-month>

# Explore, test, read
cat old_file.py

# Return to present
git checkout main

Nothing breaks. It's just exploring.

Force Push (And When NOT To)

Force push overwrites remote history with local history.

When Force Push is Needed

After rewriting local history:

# You rebased or amended
git rebase -i HEAD~5

# Push fails (history changed)
git push
# Error: rejected, tip of branch is behind

# Force push (overwrites remote)
git push --force

Safer Force Push

# Use --force-with-lease instead of --force
git push --force-with-lease

This fails if someone else pushed while you were working. Prevents overwriting their commits.

When NOT to Force Push

❌ Never force push to:

main or master branch
Shared branches with active collaborators
Public branches others depend on

✅ Safe to force push:

Your own feature branches
Branches only you work on
Before creating a pull request (cleaning up)

Protecting Main Branch

On GitHub: Settings → Branches → Add rule

Branch name: main
✅ Restrict who can push to matching branches
✅ No force pushes

Now main is protected. Nobody can force push to it.

Git Aliases: Shortcuts for Long Commands

Save time with aliases.

Creating Aliases

# Short log
git config --global alias.lg "log --oneline --graph --all"

# Now use:
git lg

# Status
git config --global alias.st status

# Undo last commit (soft)
git config --global alias.undo "reset --soft HEAD~1"

Useful Aliases

# Pretty log
git config --global alias.lg "log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit"

# Show unstaged changes
git config --global alias.df "diff"

# Show staged changes
git config --global alias.dfs "diff --staged"

# List aliases
git config --global alias.aliases "config --get-regexp alias"

# Amend without editing message
git config --global alias.amend "commit --amend --no-edit"

# Delete merged branches
git config --global alias.cleanup "!git branch --merged | grep -v '\\*\\|main\\|develop' | xargs -n 1 git branch -d"

View All Aliases

git aliases

Or:

git config --global --get-regexp alias

Quick Reference: Advanced Commands

Undoing

# Undo last commit (keep changes)
git reset --soft HEAD~1

# Undo last commit (discard changes)
git reset --hard HEAD~1

# Undo pushed commit (create opposite)
git revert <commit>

Stashing

# Save work temporarily
git stash

# Save with message
git stash save "WIP feature"

# List stashes
git stash list

# Apply and remove from stash
git stash pop

# Apply but keep in stash
git stash apply

Rewriting History

# Interactive rebase (last 5 commits)
git rebase -i HEAD~5

# Amend last commit
git commit --amend

# Amend without editing message
git commit --amend --no-edit

Recovery

# See all HEAD movements
git reflog

# Recover deleted commit
git reset --hard <commit>

# Find what introduced a bug
git bisect start

Cherry-Picking

# Copy commit to current branch
git cherry-pick <commit>

# Copy multiple commits
git cherry-pick <commit1> <commit2>

Force Operations

# Force push (dangerous)
git push --force

# Safer force push
git push --force-with-lease

# Reset to remote state
git reset --hard origin/main

Emergency Scenarios

Scenario 1: Committed Secrets

# 1. Rotate the secret immediately
# 2. Remove from history
git filter-repo --replace-text <(echo 'SECRET_KEY==>REMOVED')

# 3. Force push
git push --force-with-lease

# 4. Tell team to re-clone

Scenario 2: Accidentally Deleted Branch

# Find last commit on deleted branch
git reflog

# Recreate branch
git branch recovered-branch <commit>

Scenario 3: Committed to Wrong Branch

# Create branch with commits
git branch feature-branch

# Remove commits from current branch
git reset --hard HEAD~3

# Switch to feature branch
git checkout feature-branch

Scenario 4: Messed Up Rebase

# Abort rebase
git rebase --abort

# Or find pre-rebase state in reflog
git reflog
git reset --hard HEAD@{before-rebase}

Scenario 5: Need Old Version of File

# Get file from specific commit
git show <commit>:path/to/file.py > file.py

# Or checkout just that file
git checkout <commit> -- path/to/file.py

Summary

Advanced Git saves you from disasters.

Undoing commits:

reset --soft - Uncommit, keep changes staged
reset --mixed - Uncommit, keep changes unstaged
reset --hard - Delete everything (dangerous)
revert - Create opposite commit (safe for pushed commits)
rebase -i - Rewrite history (before pushing)

Recovery:

reflog - See all actions, recover deleted commits
bisect - Find bug-introducing commit
cherry-pick - Copy commits between branches

Temporary storage:

stash - Save work temporarily, switch context

Fixing mistakes:

commit --amend - Fix last commit
filter-repo - Remove secrets from history
force-with-lease - Safer force push

Golden rules:

Never rewrite pushed history (unless emergency)
Always rotate secrets before removing from Git
Use reflog when you think you lost commits
Stash before switching context
Test before force pushing

When things go wrong:

Don't panic
Check git reflog
Find the state before you broke it
Reset or create branch there
Commits are rarely truly lost

Git's safety nets (reflog, stash, revert) mean you can experiment confidently. Even "permanent" mistakes are usually recoverable.

You now know Git from basics to advanced. You can collaborate, fix mistakes, and recover from disasters.

The Complete Git Journey

Blog 1: Local Git basics (commits, history)
Blog 2: GitHub (cloud backup, portfolio)
Blog 3: Branches (parallel development)
Blog 4: Collaboration (teams, open source)
Blog 5: Advanced commands (fixing everything)

You've learned:

Version control fundamentals
Cloud backup and sync
Safe experimentation with branches
Working with others without chaos
Recovering from any mistake

What's next?

Use Git every day. Make mistakes. Fix them with these commands. The more you use Git, the more natural it becomes.

Git isn't just a tool. It's a safety net. It's a time machine. It's collaboration infrastructure.

Master Git, and you'll never lose code again.

Resources:

Oh Shit, Git!: https://ohshitgit.com/
Git Flight Rules: https://github.com/k88hudson/git-flight-rules
Pro Git Book: https://git-scm.com/book/en/v2
Learn Git Branching: https://learngitbranching.js.org/

Collaboration: Working on Code Without Killing Each Other

Muhammad Ikramullah Khan — Fri, 01 May 2026 09:12:15 +0000

You find a scraper on GitHub that does exactly what you need. Almost. It scrapes product prices but doesn't handle pagination. You know how to add pagination. You could copy the code, modify it, and use your own version.

But that's wasteful. The original developer maintains the project. Fixes bugs. Adds features. If you copy it, you're on your own. You have to manually copy their updates into your version forever.

Better idea: add pagination to their project. Contribute it back. Now everyone benefits. They get pagination. You get their future updates automatically. Win-win.

Or maybe this happens. You're working with a teammate on the same scraper. You both edit scraper.py. You finish first. Commit. Push. Done.

Your teammate finishes. Tries to push. Git rejects it. "Your branch is behind. Pull first." They pull. Merge conflict. Your changes clash with theirs. Same function. Different implementations.

Now you're both stuck. Who's code wins? How do you merge without breaking both versions? Do you text each other? Email? Video call to figure out whose change stays?

This is where collaboration workflow saves you.

Git and GitHub have built-in tools for working with other people. Forking projects. Proposing changes. Reviewing code. Resolving conflicts. Tracking issues. The tools exist. You just need to learn the pattern.

Let me show you how to collaborate without chaos.

The Two Collaboration Scenarios

You'll encounter two main situations.

Scenario 1: Contributing to Someone Else's Project

You don't have permission to push to their repo. You can't just commit and push like it's yours.

The workflow:

Fork (make your own copy on GitHub)
Clone your fork
Create a branch
Make changes
Push to your fork
Create pull request to original repo
Wait for maintainer to review and merge

This is how open source works.

Scenario 2: Team Project (Shared Access)

Everyone has permission to push. But you need rules so people don't overwrite each other.

The workflow:

Clone the shared repo
Always pull before starting work
Create a branch for your feature
Make changes
Push your branch
Create pull request
Teammate reviews
Merge after approval
Everyone pulls the updated main

This is how teams work.

Both scenarios use the same Git features (branches, pull requests). The difference is permissions.

Forking: Making Your Own Copy

Forking creates a copy of someone's repo under your GitHub account.

When to Fork

Contributing to open source projects
Customizing a project for your needs
Experimenting with changes before proposing them
Learning from a project by modifying it

How to Fork

Let's fork a real project. We'll use a simple scraper template.

Step 1: Find a Project

Go to: https://github.com/example-user/simple-scraper

(For this example, find any small scraping project you want to contribute to)

Step 2: Click Fork

Top-right corner: "Fork" button
Choose your account (where to fork it)
Optional: Change the name
Click "Create fork"

You now have a copy: https://github.com/YOUR-USERNAME/simple-scraper

This is yours. You can push to it. Modify it. Break it. The original is untouched.

Cloning Your Fork

# Clone YOUR fork, not the original
git clone https://github.com/YOUR-USERNAME/simple-scraper.git
cd simple-scraper

# Check the remote
git remote -v

Output:

origin  https://github.com/YOUR-USERNAME/simple-scraper.git (fetch)
origin  https://github.com/YOUR-USERNAME/simple-scraper.git (push)

origin points to your fork.

Adding the Original as Upstream

You'll want to pull updates from the original project.

# Add the original repo as 'upstream'
git remote add upstream https://github.com/example-user/simple-scraper.git

# Verify
git remote -v

Output:

origin    https://github.com/YOUR-USERNAME/simple-scraper.git (fetch)
origin    https://github.com/YOUR-USERNAME/simple-scraper.git (push)
upstream  https://github.com/example-user/simple-scraper.git (fetch)
upstream  https://github.com/example-user/simple-scraper.git (push)

Now you have two remotes:

origin - your fork (you can push here)
upstream - original repo (you can only pull from here)

Contributing to Open Source

Let's add a feature to the project we forked.

Step 1: Create a Feature Branch

# Make sure you're on main
git checkout main

# Pull latest from original project
git pull upstream main

# Create feature branch
git checkout -b add-pagination

Step 2: Make Your Changes

# Edit scraper.py
# Add pagination functionality

def scrape_all_pages(base_url):
    """Scrape all pages with pagination"""
    all_products = []
    page = 1

    while True:
        url = f"{base_url}?page={page}"
        products = scrape_products(url)

        if not products:
            break

        all_products.extend(products)
        page += 1

    return all_products

Step 3: Test Your Changes

# Run the scraper
python scraper.py

# Make sure it works
# Test edge cases

Step 4: Commit Your Changes

git add scraper.py
git commit -m "Add pagination support for multi-page scraping"

Good commit message format:

First line: Brief summary (50 chars or less)
Optional: Blank line + detailed explanation

git commit -m "Add pagination support for multi-page scraping

- Implemented scrape_all_pages() function
- Handles pagination automatically
- Stops when no more products found
- Tested with 10+ page product listings"

Step 5: Push to Your Fork

# Push to YOUR fork (origin), not the original (upstream)
git push -u origin add-pagination

Step 6: Create a Pull Request

On GitHub:

Go to your fork: https://github.com/YOUR-USERNAME/simple-scraper
You'll see: "add-pagination had recent pushes" with "Compare & pull request" button
Click "Compare & pull request"

Pull Request Form:

Base repository: example-user/simple-scraper (original)
Base branch: main
Head repository: YOUR-USERNAME/simple-scraper (your fork)
Compare branch: add-pagination

Title: "Add pagination support"

Description:

## Summary
Adds pagination support for scraping multi-page product listings.

## Changes
- New function `scrape_all_pages()` that handles pagination
- Automatically iterates through pages until no products found
- Backward compatible (existing code still works)

## Testing
- Tested with 10+ page listings
- Tested with single page (no regression)
- Tested with invalid URLs (proper error handling)

## Example Usage

python
products = scrape_all_pages('https://example.com/products')
print(f'Scraped {len(products)} products across multiple pages')


Fixes #15

shell

The Fixes #15 references an issue number. If there's an open issue requesting this feature, link it.

Click "Create pull request"

Step 7: Wait for Review

The maintainer will:

Review your code
Test it
Ask questions or request changes
Approve and merge, or decline

If they request changes:

# Make the requested changes
# Edit files

# Commit
git add .
git commit -m "Address review feedback: improve error handling"

# Push to same branch
git push

The pull request updates automatically. No need to create a new one.

If they merge:

Your code is now in the original project! Congratulations, you contributed to open source.

Cleanup:

# Switch back to main
git checkout main

# Pull the merged changes from original
git pull upstream main

# Delete your feature branch
git branch -d add-pagination

# Update your fork's main on GitHub
git push origin main

Team Collaboration Workflow

You're working with teammates on a shared repo. Everyone has push access.

The Golden Rules

Never push directly to main (use branches + pull requests)
Always pull before starting work
Create a branch for every feature/fix
Get approval before merging (code review)
Pull main before merging your branch

Daily Team Workflow

Morning routine:

# Update your local main
git checkout main
git pull origin main

# Create today's feature branch
git checkout -b feature/user-login

During the day:

# Make changes
# (edit files)

# Commit regularly
git add .
git commit -m "Implement login validation"

# Push to GitHub (backup + share with team)
git push -u origin feature/user-login

When feature is done:

# Make sure main hasn't changed
git checkout main
git pull origin main

# Merge main into your feature branch (get latest changes)
git checkout feature/user-login
git merge main

# Resolve any conflicts
# (if main changed while you worked)

# Push updated branch
git push

# Create pull request on GitHub
# Wait for teammate review
# Merge after approval

After your PR is merged:

# Update local main
git checkout main
git pull origin main

# Delete feature branch
git branch -d feature/user-login
git push origin --delete feature/user-login

Handling Simultaneous Work

Scenario: You and a teammate both work on different features simultaneously.

You:

git checkout -b feature/add-export
# (work on export feature)
git commit -m "Add CSV export"
git push -u origin feature/add-export
# (create PR, get approved, merge)

Teammate (at the same time):

git checkout -b feature/add-filters
# (work on filters)
git commit -m "Add product filters"
git push -u origin feature/add-filters
# (create PR)

Your teammate's PR now needs updating:

# Your export feature merged first
# Teammate needs to update their branch

git checkout feature/add-filters
git pull origin main  # Get your merged export feature
# (resolve any conflicts if both touched same files)
git push
# (PR updates, gets approved, merges)

This is why "pull before merge" matters.

Code Review Best Practices

Pull requests aren't just for merging. They're for reviewing code.

As the Author (Creating PR)

Write a clear description:

## What this PR does
Adds ability to export scraped data to CSV format.

## Why we need it
Users requested CSV export for importing to Excel.

## How to test
1. Run scraper: python scraper.py
2. Check data/export.csv exists
3. Open in Excel, verify formatting

## Screenshots
(if UI changes)

## Checklist
- [x] Code works locally
- [x] Added tests
- [x] Updated README
- [x] No merge conflicts

Keep PRs small:

One feature per PR
Easier to review
Faster to merge
Less likely to have conflicts

Respond to feedback graciously:

"Good catch, I'll fix that"
"I didn't consider that case, adding now"
"Great suggestion, implemented"

As the Reviewer

What to look for:

Does it work? (test the code locally)
Does it break anything? (run existing tests)
Is it readable? (can you understand it?)
Is it secure? (no passwords, proper input validation)
Is it efficient? (no obvious performance issues)

How to comment:

Good feedback:

Line 23: Consider adding error handling here for when the URL is invalid.

Suggestion:
try:
    response = requests.get(url)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    logger.error(f"Failed to fetch {url}: {e}")
    return None

Bad feedback:

this is bad

Be specific. Be constructive. Suggest solutions.

Approving or requesting changes:

On GitHub:

"Comment" - add feedback without blocking
"Approve" - looks good, ready to merge
"Request changes" - needs fixes before merging

Reviewing Your Own Code

Even solo, review your own PRs:

Create PR
Look at the diff (what changed)
Read it like you didn't write it
Catch mistakes you missed
Merge when satisfied

This habit catches bugs before production.

Handling Merge Conflicts in Teams

Two people edited the same line. Git can't auto-merge.

Example Conflict

Person A edits scraper.py:

def scrape_product(url):
    response = requests.get(url, timeout=30)

Person B edits the same line:

def scrape_product(url):
    response = requests.get(url, timeout=60)

Both push to different branches. Person A merges first. Person B tries to merge.

Conflict!

Resolving the Conflict

Person B:

# Pull latest main
git checkout main
git pull origin main

# Try to merge into feature branch
git checkout feature/increase-timeout
git merge main

Output:

Auto-merging scraper.py
CONFLICT (content): Merge conflict in scraper.py
Automatic merge failed; fix conflicts and then commit the result.

Open scraper.py:

def scrape_product(url):
<<<<<<< HEAD
    response = requests.get(url, timeout=60)
=======
    response = requests.get(url, timeout=30)
>>>>>>> main

Person B's choices:

Keep their change (60): Maybe they tested and 60 is better
Keep Person A's change (30): Maybe 30 is the agreed standard
Choose different value: Maybe 45 is a compromise
Ask Person A: "Hey, I increased timeout to 60. You set it to 30. Which should we use?"

After deciding (let's say keep 60):

def scrape_product(url):
    response = requests.get(url, timeout=60)

Remove the markers. Save file.

# Mark conflict as resolved
git add scraper.py

# Complete the merge
git commit -m "Merge main, kept 60s timeout after discussion with Person A"

# Push updated branch
git push

The PR updates. Reviewer sees the conflict was resolved. Merges when satisfied.

.gitignore: What NOT to Commit

Some files shouldn't be in Git.

Why .gitignore Matters

Bad things that happen when you commit the wrong files:

API keys get exposed (security breach)
Repo becomes huge (data files, databases)
Merge conflicts on generated files (compiled code, caches)
Teammate's environment breaks (OS-specific files)

Creating .gitignore

# In your project root
touch .gitignore

Essential patterns for Python projects:

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
env/
venv/
ENV/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# Environment
.env
.venv
config.py

# IDE
.vscode/
.idea/
*.swp
*.swo
*~

# OS
.DS_Store
Thumbs.db

# Data files
*.csv
*.json
*.xlsx
data/
scraped_data/

# Logs
*.log
logs/

# Database
*.db
*.sqlite3

Commit .gitignore:

git add .gitignore
git commit -m "Add .gitignore for Python project"

Common Patterns

Ignore all files of a type:

*.csv
*.log
*.db

Ignore a directory:

data/
logs/

Ignore all except one file:

# Ignore all .env files
.env*

# Except the example
!.env.example

Ignore files in root only:

/config.py  # Only root config.py, not src/config.py

Testing .gitignore

# Check what Git sees
git status

# If ignored files still show up:
# They were already tracked before .gitignore

# Remove from Git (keep on disk)
git rm --cached filename.csv

# Commit the removal
git commit -m "Stop tracking data files"

.gitignore for Scrapers

# Virtual environment
venv/
env/

# Environment variables (API keys!)
.env
config.py

# Scraped data
data/
*.csv
*.json
*.xlsx

# Logs
*.log
logs/

# Database
*.db
*.sqlite3

# Temporary files
temp/
cache/

# Screenshots (if using Selenium)
screenshots/

# Keep example config
!config.example.py

GitHub Issues: Tracking Work

Issues are like to-do lists for your project.

Creating an Issue

On GitHub → Issues → New issue

Title: "Add pagination support"

Description:

## Problem
Scraper only gets first page of results. Multi-page listings are incomplete.

## Proposed Solution
Add pagination support to scrape_products() function.

## Example
Site: https://example.com/products
Current: Gets 20 products (page 1)
Desired: Gets all 200 products (10 pages)

## Acceptance Criteria
- [ ] Scrapes all pages automatically
- [ ] Stops when no more products
- [ ] Handles pagination URLs (?page=N)
- [ ] Works with existing code

Labels: enhancement, good first issue

Assignee: Yourself or teammate

Click "Submit new issue"

Working on an Issue

# Reference issue in branch name
git checkout -b fix-pagination-15

# Reference issue in commits
git commit -m "Add pagination support (fixes #15)"

# Reference issue in PR
# (GitHub auto-links when you mention #15)

When the PR merges, the issue closes automatically.

Using Issues for Bug Reports

Title: "Scraper crashes on products without prices"

Description:

## Bug Description
Scraper crashes with `AttributeError: NoneType` when product has no price.

## Steps to Reproduce
1. Run: `python scraper.py`
2. URL: https://example.com/products/out-of-stock
3. Error occurs on line 45

## Expected Behavior
Should skip products without prices or set price to 0.

## Actual Behavior

AttributeError: 'NoneType' object has no attribute 'text'
File "scraper.py", line 45
price = price_elem.text


## Environment
- Python 3.9
- Ubuntu 20.04
- Requirements.txt versions

Labels: bug, high priority

This gives teammates everything they need to reproduce and fix.

GitHub Projects: Organizing Work

Projects are kanban boards for issues and PRs.

Creating a Project

On GitHub → Projects → New project

Name: "Scraper Roadmap"
Template: "Board"
Create

Default columns:

Todo
In Progress
Done

Add custom columns:

Backlog (ideas for later)
Review (PRs waiting for review)
Testing (needs testing before deploy)

Adding Issues to Project

Drag issues between columns:

Backlog: "Add authentication support"
Todo: "Add pagination" (next to work on)
In Progress: "Fix price parsing bug" (actively working)
Review: "Add CSV export" (PR created, waiting review)
Done: "Initial scraper setup" (merged and deployed)

Team Workflow with Projects

Weekly planning:

Review Backlog
Move important issues to Todo
Assign to team members

Daily:

Move your issue to In Progress when you start
Create PR when done
Move to Review
Teammate reviews
Merge and move to Done

Visual progress tracking. Everyone sees what's being worked on.

Forking vs Branching: When to Use Which

Use Forking When:

Contributing to projects you don't have push access to
Open source contributions
Experimenting with major changes to someone else's project
Creating your own version of a project

Pattern: Fork → Clone fork → Branch → PR to original

Use Branching When:

Working on a project you have push access to
Team projects
Your own projects
Any project where you're a collaborator

Pattern: Clone → Branch → PR to same repo

Same Repo, Collaborative:

# Everyone clones the same repo
git clone https://github.com/team/project.git

# Everyone creates branches
git checkout -b feature/my-feature

# Everyone creates PRs to main branch
# (same repo, different branches)

Fork-Based, External Contribution:

# Fork on GitHub
# Clone your fork
git clone https://github.com/YOU/project.git

# Create branch
git checkout -b feature/my-contribution

# PR to original repo
# (from your fork to their main)

Real Example: Contributing to a Scrapy Project

Let's contribute to a real open source project.

Step 1: Find a Project

Search GitHub: "scrapy spider" or "web scraper python"

Let's say you find: scrapy-examples/ecommerce-spider

Step 2: Fork It

Click "Fork" on GitHub.

Step 3: Clone Your Fork

git clone https://github.com/YOUR-USERNAME/ecommerce-spider.git
cd ecommerce-spider

# Add upstream
git remote add upstream https://github.com/scrapy-examples/ecommerce-spider.git

Step 4: Find Something to Contribute

Check Issues:

Look for good first issue label
Or help wanted

Let's say Issue #42: "Add support for product reviews"

Step 5: Create Feature Branch

git checkout -b add-product-reviews

Step 6: Implement the Feature

# spiders/products.py

class ProductSpider(scrapy.Spider):
    name = 'products'

    def parse(self, response):
        # Existing product parsing
        yield {
            'name': response.css('.product-name::text').get(),
            'price': response.css('.price::text').get(),
        }

        # New: Follow review links
        review_url = response.css('.reviews-link::attr(href)').get()
        if review_url:
            yield response.follow(review_url, self.parse_reviews)

    def parse_reviews(self, response):
        """Parse product reviews"""
        for review in response.css('.review'):
            yield {
                'product_url': response.url,
                'rating': review.css('.rating::attr(data-rating)').get(),
                'text': review.css('.review-text::text').get(),
                'author': review.css('.author::text').get(),
                'date': review.css('.date::text').get(),
            }

Step 7: Test Thoroughly

# Run the spider
scrapy crawl products -o output.json

# Check the output
cat output.json

# Make sure reviews are included

Step 8: Update Documentation

# README.md

## Features

- Scrapes product names and prices
- Follows pagination automatically
- **NEW:** Scrapes product reviews with ratings

## Output

The spider outputs JSON with product data and reviews:

json
[
{
"name": "Laptop Pro 15",
"price": "$1299.99"
},
{
"product_url": "https://example.com/laptop-pro-15",
"rating": "5",
"text": "Excellent laptop!",
"author": "John D.",
"date": "2024-03-15"
}
]

shell

Step 9: Commit and Push

git add .
git commit -m "Add product review scraping support (fixes #42)

- Implemented parse_reviews() method
- Follows review links from product pages
- Extracts rating, text, author, date
- Tested with 50+ products
- Updated README with review output format"

git push -u origin add-product-reviews

Step 10: Create Pull Request

On GitHub:

Title: "Add product review scraping support"

Description:

Closes #42

## Summary
Adds ability to scrape product reviews in addition to product data.

## Changes
- New `parse_reviews()` method
- Follows review links from product pages
- Extracts rating, review text, author, and date
- Updated README with examples

## Testing
- Tested on 50+ products with reviews
- Handles products without reviews (gracefully skips)
- Verified output format

## Output Example

json
{
"product_url": "https://example.com/laptop",
"rating": "5",
"text": "Great product",
"author": "Jane S.",
"date": "2024-03-15"
}


## Checklist
- [x] Code tested locally
- [x] Documentation updated
- [x] No breaking changes
- [x] Follows project coding style

shell

Step 11: Respond to Review

Maintainer comments: "Can you add error handling for missing review dates?"

# Make changes
# Edit the code

git add .
git commit -m "Add error handling for missing review dates"
git push

PR updates automatically.

Step 12: PR Gets Merged

Congratulations! You contributed to open source.

Cleanup:

git checkout main
git pull upstream main
git branch -d add-product-reviews
git push origin main

Your contribution is now part of the project forever.

Team Communication

Git tracks code. Communication tracks decisions.

Good Commit Messages

Bad:

updated stuff
fixed bug
changes

Good:

Add pagination support for multi-page scraping

- Implemented scrape_all_pages() function
- Automatically detects last page
- Tested with 20+ page listings

Fixes #15

Format:

Short summary (50 chars or less)

Detailed explanation (wrap at 72 chars):
- What you changed
- Why you changed it
- How to test it

Fixes #issue-number

Pull Request Etiquette

Do:

Explain what and why
Link related issues
Respond to feedback promptly
Test your code
Update documentation

Don't:

Create massive PRs (500+ line changes)
Ignore review feedback
Merge without approval
Force push after review started
Get defensive about criticism

Code Review Comments

As reviewer, be kind:

❌ "This is wrong"
✅ "Consider handling the case where URL is None"

❌ "Bad code"
✅ "This could be optimized by caching the results"

❌ "Didn't you read the docs?"
✅ "The library actually has a built-in method for this: requests.get(url, timeout=30)"

As author, be receptive:

❌ "My code is fine"
✅ "Good point, I'll add that check"

❌ "That's not important"
✅ "I see your concern. How about this approach instead?"

Advanced Team Workflows

Protected Branches

Prevent direct pushes to main.

On GitHub → Settings → Branches → Add rule:

Branch name: main
✅ Require pull request before merging
✅ Require approvals: 1
✅ Require status checks to pass
Save

Now nobody can push directly to main. All changes go through PRs.

Required Reviews

Settings → Branches → main:

✅ Require approvals: 2 (for teams)
✅ Dismiss stale reviews when new commits pushed

Forces code review before merging.

Auto-Delete Branches

Settings → General:

✅ Automatically delete head branches

When PR merges, branch auto-deletes. Keeps repo clean.

Branch Naming Conventions

Team standard:

feature/description - new features
fix/description - bug fixes
docs/description - documentation
refactor/description - code cleanup
test/description - adding tests

Everyone follows the same pattern. Easy to understand at a glance.

Handling Large Teams

Code Owners

Create .github/CODEOWNERS:

# Every PR needs approval from these owners

# Default owners for everything
* @team-lead @senior-dev

# Specific owners for parts
/scrapers/ @scraping-team
/database/ @backend-team
/docs/ @documentation-team

GitHub auto-assigns reviewers based on files changed.

Draft Pull Requests

Create PR that isn't ready yet:

Create PR
Select "Draft pull request"
Keep working
Mark "Ready for review" when done

Shows progress without spamming reviewers.

PR Templates

Create .github/pull_request_template.md:

## Summary
<!-- What does this PR do? -->

## Changes
<!-- List main changes -->

## Testing
<!-- How was this tested? -->

## Checklist
- [ ] Code works locally
- [ ] Tests pass
- [ ] Documentation updated
- [ ] No merge conflicts

Every new PR pre-fills with this template.

Summary

Collaboration is about process, not just tools.

Contributing to open source:

Fork the repo
Clone your fork
Create feature branch
Make changes
Push to your fork
Create PR to original
Respond to feedback
Celebrate when merged

Working in teams:

Clone shared repo
Always pull before starting
Create feature branch
Make changes
Push branch
Create PR
Get review approval
Merge
Delete branch
Pull updated main

Key tools:

Forks (your copy of someone's project)
Branches (parallel development)
Pull requests (propose changes)
Code review (quality control)
Issues (track work)
Projects (organize work)
.gitignore (exclude files)

Best practices:

Small, focused PRs
Clear commit messages
Responsive to feedback
Test before pushing
Review your own code
Pull before merging
Communicate decisions

Collaboration rules:

Never push directly to main
Always use branches
Always get review approval
Pull before starting work
Keep PRs small and focused

Git and GitHub handle the mechanics. You handle the communication. Both matter equally.

Next up: Blog 5 - "Advanced Git: The Commands That Save Your Career"

The final blog in the series. We'll cover the scary stuff: recovering deleted commits, rewriting history (carefully), handling committed secrets, git reset vs revert vs rebase, stashing changes, cherry-picking commits, and the advanced commands that fix seemingly impossible situations.

Resources:

GitHub Skills: https://skills.github.com
Open source guide: https://opensource.guide/how-to-contribute/
Conventional commits: https://www.conventionalcommits.org/
Code review best practices: https://google.github.io/eng-practices/review/

Branches: How to Break Things Without Breaking Things

Muhammad Ikramullah Khan — Wed, 22 Apr 2026 18:11:15 +0000

You have a working scraper. It scrapes 1,000 products daily. No errors. It's been running smoothly for two weeks.

Your boss asks: "Can we add image downloading? Just test it first, don't break the working scraper."

You have a problem. You need to modify the scraper to add images. But if you edit the working code directly, you might break it. Then nothing works. Your daily scraping stops. Your boss is unhappy.

You could make a copy of the entire project folder. scraper_v1, scraper_v2, scraper_with_images. Work on the copy. If it works, copy the changes back to the original. If it breaks, delete the copy.

This gets messy fast. Which folder has the latest changes? Did you edit both? Which one should you deploy?

This is exactly why Git has branches.

A branch is a parallel version of your code. You create a branch called "add-images". Work on it. Break things. Fix them. Test everything. When it works perfectly, you merge it back into your main code.

Your main code stays untouched. Running smoothly. While you experiment in a separate branch. If your experiment fails, just delete the branch. The main code never knew it happened.

Branches sound complicated. They're not. Think of them as save files in a video game. You have your main save (the one you care about). You create a test save to try risky strategies. If it works, great. If not, load your main save and try again.

Let me show you.

What Branches Actually Are (The Simple Version)

Forget everything you've heard about directed acyclic graphs and commit trees.

A branch is just a label pointing to a commit.

That's it.

You've been using a branch this whole time. It's called main (or master in older Git).

When you make commits, they go on the main branch. It's the default. The one that exists automatically.

Creating a new branch:

git branch feature-images

This creates a new label called feature-images pointing to the same commit as main.

Now you have two branches:

main - your working code
feature-images - a copy where you'll add images

When you switch to the new branch and make commits, only that branch moves forward. main stays where it was.

Think of it like this:

Before branching:
main → commit1 → commit2 → commit3

After creating branch:
main → commit1 → commit2 → commit3
feature-images → commit1 → commit2 → commit3

After committing on feature-images:
main → commit1 → commit2 → commit3
feature-images → commit1 → commit2 → commit3 → commit4 → commit5

main hasn't changed. feature-images has new commits. They exist in parallel.

Later, you can merge feature-images back into main. Then main gets commits 4 and 5 too.

Your First Branch

Let's create a branch and work on it.

Step 1: Check Current Branch

git branch

Output:

* main

The * shows which branch you're on. You're on main.

Step 2: Create New Branch

git branch add-error-handling

Check branches again:

git branch

Output:

  add-error-handling
* main

You created a branch but you're still on main.

Step 3: Switch to New Branch

git checkout add-error-handling

Output:

Switched to branch 'add-error-handling'

Check where you are:

git branch

Output:

* add-error-handling
  main

The * moved. You're now on the new branch.

Shortcut: Create and switch in one command:

git checkout -b add-error-handling

This creates the branch AND switches to it. Use this instead of separate commands.

Step 4: Make Changes on the Branch

# Edit your code
echo "try:" >> scraper.py
echo "    # scraping code" >> scraper.py
echo "except Exception as e:" >> scraper.py
echo "    print(f'Error: {e}')" >> scraper.py

# Commit on this branch
git add scraper.py
git commit -m "Added error handling to scraper"

Step 5: Switch Back to Main

git checkout main

Check your file:

cat scraper.py

The error handling code is gone! Your file looks like it did before.

This is the magic. The changes exist on add-error-handling branch. They don't exist on main. You can switch back and forth.

Step 6: View Branches with Commits

git log --oneline --all --graph

Output:

* d4e5f6g (add-error-handling) Added error handling to scraper
| 
* c3d4e5f (HEAD -> main, origin/main) Last commit on main
* b2c3d4e Previous commit
* a1b2c3d Initial commit

You can see the branches diverged. add-error-handling has one extra commit.

Merging Branches

Your feature works. You tested it on the branch. Time to bring it into main.

Step 1: Switch to Main

Always merge INTO the branch you're on. You want the changes IN main, so switch to main first.

git checkout main

Step 2: Merge the Feature Branch

git merge add-error-handling

Output:

Updating c3d4e5f..d4e5f6g
Fast-forward
 scraper.py | 4 ++++
 1 file changed, 4 insertions(+)

"Fast-forward" means: main just moved forward to where add-error-handling was. No conflicts. Clean merge.

Check your file:

cat scraper.py

The error handling code is now in main!

Step 3: Delete the Feature Branch (Optional)

You don't need it anymore. The changes are in main.

git branch -d add-error-handling

Output:

Deleted branch add-error-handling (was d4e5f6g).

Check branches:

git branch

Output:

* main

Back to one branch. Clean.

The Feature Branch Workflow

This is how professionals work with branches.

Pattern:

# Start on main with latest code
git checkout main
git pull

# Create feature branch
git checkout -b feature-name

# Work on the feature (multiple commits okay)
git add .
git commit -m "Implemented X"
git add .
git commit -m "Fixed bug in X"
git add .
git commit -m "Added tests for X"

# Switch back to main
git checkout main

# Merge the feature
git merge feature-name

# Delete the branch
git branch -d feature-name

# Push to GitHub
git push

Real Example: Adding Login to Scraper

# Start fresh
git checkout main
git pull

# Create login branch
git checkout -b add-login

# Implement login (first attempt)
echo "def login(username, password):" >> scraper.py
echo "    # login logic" >> scraper.py
git add scraper.py
git commit -m "Added basic login function"

# Test it, find bugs, fix them
echo "    # improved login logic" >> scraper.py
git add scraper.py
git commit -m "Fixed login bug with special characters"

# Add error handling
echo "    # error handling" >> scraper.py
git add scraper.py
git commit -m "Added error handling to login"

# Feature is done, merge it
git checkout main
git merge add-login

# Clean up
git branch -d add-login

# Push to GitHub
git push

You made three commits on the branch. They all merged into main together. Your main branch history shows a clean progression.

Handling Merge Conflicts (The Scary Part)

Sometimes Git can't merge automatically. This happens when:

You changed the same line in both branches
You deleted a file in one branch, edited it in another
Two branches modified the same section differently

Creating a Conflict (Intentionally)

Let's simulate a conflict to learn how to fix it.

On main branch:

git checkout main

# Edit scraper.py
echo "print('Version A')" > scraper.py

git add scraper.py
git commit -m "Main branch changes"

Create and switch to new branch:

# Branch from BEFORE the main change
git checkout HEAD~1
git checkout -b feature-conflict

# Edit the SAME LINE differently
echo "print('Version B')" > scraper.py

git add scraper.py
git commit -m "Feature branch changes"

Try to merge:

git checkout main
git merge feature-conflict

Output:

Auto-merging scraper.py
CONFLICT (content): Merge conflict in scraper.py
Automatic merge failed; fix conflicts and then commit the result.

Git can't decide which version to keep. You have to choose.

Resolving the Conflict

Open scraper.py:

<<<<<<< HEAD
print('Version A')
=======
print('Version B')
>>>>>>> feature-conflict

What this means:

<<<<<<< HEAD - Start of your current branch's version (main)
======= - Separator
>>>>>>> feature-conflict - End of the other branch's version

You need to:

Decide which version to keep (or combine both)
Remove the conflict markers
Save the file
Commit the resolution

Option 1: Keep Version A

print('Version A')

Option 2: Keep Version B

print('Version B')

Option 3: Keep Both

print('Version A')
print('Version B')

Option 4: Write Something New

print('Combined version C')

Let's keep Version A:

# Edit file to just have:
print('Version A')

# Save and close

# Tell Git the conflict is resolved
git add scraper.py

# Commit the merge
git commit -m "Resolved merge conflict, kept Version A"

Done! The merge is complete.

Avoiding Conflicts

Best practices:

Pull before branching - Start with latest code
Work on different files - Multiple people editing different parts
Merge often - Don't let branches drift apart for weeks
Communicate with team - "I'm editing scraper.py today"

Most conflicts are easy to resolve. Don't panic when you see one.

Visualizing Branches

Understanding branches visually helps.

Simple Branch Example

Initial state:
main → A → B → C

Create branch 'feature':
main    → A → B → C
feature → A → B → C (same commits)

Commit on feature:
main    → A → B → C
feature → A → B → C → D → E

Merge feature into main:
main → A → B → C → D → E
(feature branch can be deleted)

Divergent Branches

Initial:
main → A → B → C

Create branch:
main    → A → B → C
feature → A → B → C

Commit on both:
main    → A → B → C → D
feature → A → B → C → E → F

Merge:
main → A → B → C → D → (merge commit)
                  ↓      ↑
feature → A → B → C → E → F

When branches diverge, merging creates a "merge commit" that combines both histories.

Viewing Your Branch Graph

git log --graph --oneline --all

Output:

* d4e5f6g (HEAD -> main) Merge feature into main
|\  
| * c3d4e5f (feature) Feature commit 2
| * b2c3d4e Feature commit 1
* | a1b2c3d Main commit after branching
|/  
* 9a8b7c6 Commit before branching

This shows how branches split and merge.

Working with Branches on GitHub

Branches work on GitHub too.

Pushing a Branch to GitHub

# Create and work on branch locally
git checkout -b add-pagination
echo "# pagination code" >> scraper.py
git add scraper.py
git commit -m "Added pagination support"

# Push branch to GitHub
git push -u origin add-pagination

The -u flag sets up tracking. After this, you can just use git push without arguments.

On GitHub: Visit your repo. You'll see a banner: "add-pagination had recent pushes."

Pulling Branch Changes from GitHub

Someone else pushed changes to a branch. You want them.

# See all branches (including remote ones)
git branch -a

# Switch to the remote branch
git checkout add-pagination

# Pull latest changes
git pull

Deleting Remote Branches

# Delete local branch
git branch -d add-pagination

# Delete branch on GitHub
git push origin --delete add-pagination

Pull Requests (The GitHub Way to Merge)

Pull requests (PRs) are GitHub's interface for merging branches.

Why use pull requests instead of merging locally?

Code review - Others can review your changes before merging
Discussion - Comment on specific lines, ask questions
Testing - Automated tests run on the branch
History - See what was discussed during merge
Safety - Main branch is protected, only approved PRs merge

Creating a Pull Request

Step 1: Push Your Branch

git checkout -b improve-error-messages
echo "print('Better error message')" >> scraper.py
git add scraper.py
git commit -m "Improved error messages for clarity"
git push -u origin improve-error-messages

Step 2: Create PR on GitHub

Go to your repo on GitHub
You'll see: "Compare & pull request" button
Click it
Title: "Improve error messages"
Description: "Changed error messages to be more descriptive and helpful"
Click "Create pull request"

Step 3: Review and Merge

For your own projects:

Review the changes (see the diff)
Click "Merge pull request"
Click "Confirm merge"
Optionally: Delete the branch

The changes are now in main on GitHub.

Step 4: Update Local Main

git checkout main
git pull

Your local main now has the merged changes.

Why This Workflow Matters

In teams, this is how everyone works:

You create a branch
You push it to GitHub
You create a pull request
Teammates review your code
They approve (or request changes)
You merge when approved
Everyone pulls the updated main

Even solo, PRs give you:

A place to document why you made changes
A chance to review your own code
A clean merge history

Branch Naming Conventions

Good branch names describe what you're working on.

Good Names

Features:

feature/add-login
feature/image-download
add-pagination

Bug fixes:

fix/crash-on-invalid-url
bugfix/price-parsing
fix-memory-leak

Improvements:

improve/error-handling
refactor/scraper-logic
optimize-database-queries

Experiments:

experiment/new-parser
test/alternative-approach
spike/selenium-integration

Bad Names

branch1, branch2, new-branch
test, temp, asdf
johns-branch, fixes
my-changes, updates

Use descriptive names. Future you will thank past you.

Multiple Branches Simultaneously

You can have many branches for different features.

Example: Working on Three Features

# Feature 1: Login
git checkout -b add-login
# (work on login)
git commit -m "Implemented login"

# Feature 2: Image download (pause login work)
git checkout main
git checkout -b add-images
# (work on images)
git commit -m "Added image downloading"

# Feature 3: Error handling (pause images)
git checkout main
git checkout -b improve-errors
# (work on errors)
git commit -m "Better error messages"

You now have three branches:

git branch

Output:

  add-images
  add-login
* improve-errors
  main

Finish error handling first:

git checkout main
git merge improve-errors
git branch -d improve-errors
git push

Back to images:

git checkout add-images
# (continue work)

Switch context anytime:

# Jump between features as needed
git checkout add-login    # Work on login
git checkout add-images   # Work on images
git checkout main         # Check production code

Each branch is independent. Changes in one don't affect others.

Recovering from Branch Mistakes

Mistake 1: Committed to Wrong Branch

You meant to create a branch but committed to main by accident.

Fix:

# Create branch from current position
git branch feature-branch

# Reset main to previous commit
git reset --hard HEAD~1

# Switch to new branch (has your commit)
git checkout feature-branch

Your commit is now on the feature branch, not main.

Mistake 2: Want to Undo Branch Merge

You merged a branch but it broke things.

Fix:

# See recent commits
git log --oneline

# Find the commit BEFORE the merge
# Let's say it's abc123

# Reset to before merge
git reset --hard abc123

The merge is undone. The branch still exists if you want to fix it and merge again.

Mistake 3: Deleted Branch Too Early

You deleted a branch but needed it again.

Fix:

# Find the last commit on that branch
git reflog

# You'll see something like:
# d4e5f6g HEAD@{3}: commit: Last commit on deleted branch

# Recreate the branch at that commit
git branch recovered-branch d4e5f6g

The branch is back!

Branch Strategies

Different workflows for different needs.

Solo Developer

Simple strategy:

main - production code, always working
Feature branches - one per feature
Merge when feature is done
Delete branch after merge

git checkout -b add-feature
# (work)
git checkout main
git merge add-feature
git branch -d add-feature

Small Team (2-5 people)

Feature branch strategy:

main - production code
Feature branches - one per developer per feature
Pull requests for review
Merge after approval

git checkout -b johns-login-feature
git push -u origin johns-login-feature
# (create PR on GitHub)
# (teammate reviews)
# (merge via GitHub)

Larger Projects

Git Flow strategy:

main - production releases only
develop - integration branch
feature/* - feature branches
release/* - release preparation
hotfix/* - emergency fixes

More complex. Not needed for most projects.

Real Example: Building a Scraper with Branches

Let's build a real scraper using proper branch workflow.

Starting Point

# Clone your repo
git clone https://github.com/yourusername/ecommerce-scraper.git
cd ecommerce-scraper

# Check main branch
git checkout main

Feature 1: Basic Scraping

# Create feature branch
git checkout -b feature/basic-scraping

# Create scraper
cat > scraper.py << 'EOF'
import requests
from bs4 import BeautifulSoup

def scrape_products(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    products = []
    for item in soup.select('.product'):
        products.append({
            'name': item.select_one('.name').text,
            'price': item.select_one('.price').text
        })

    return products

if __name__ == '__main__':
    products = scrape_products('https://example.com/products')
    print(f'Found {len(products)} products')
EOF

# Commit
git add scraper.py
git commit -m "Implemented basic product scraping"

# Push and create PR
git push -u origin feature/basic-scraping

On GitHub: Create pull request, review, merge.

Locally:

git checkout main
git pull
git branch -d feature/basic-scraping

Feature 2: Add Data Saving

# New branch from updated main
git checkout main
git pull
git checkout -b feature/save-data

# Add saving functionality
cat >> scraper.py << 'EOF'

import json

def save_products(products, filename='products.json'):
    with open(filename, 'w') as f:
        json.dump(products, f, indent=2)
EOF

# Update main function
# (edit scraper.py to call save_products)

# Commit
git add scraper.py
git commit -m "Added JSON data saving"

# Push and PR
git push -u origin feature/save-data

Merge via GitHub. Update local:

git checkout main
git pull
git branch -d feature/save-data

Feature 3: Error Handling

git checkout main
git pull
git checkout -b feature/error-handling

# Add error handling
# (edit scraper.py)

git add scraper.py
git commit -m "Added comprehensive error handling"
git push -u origin feature/error-handling

Merge via GitHub.

Feature 4: Pagination (Work in Progress)

git checkout main
git pull
git checkout -b feature/pagination

# Start working on pagination
# (edit scraper.py)

git add scraper.py
git commit -m "WIP: Started pagination implementation"

# Realize you need to fix a bug in main first
git checkout main
git checkout -b hotfix/price-parsing

# Fix the bug
# (edit scraper.py)

git add scraper.py
git commit -m "Fixed price parsing for decimal values"
git push -u origin hotfix/price-parsing

Merge hotfix immediately via GitHub.

# Back to pagination
git checkout feature/pagination

# Merge the hotfix into your feature branch
git merge main

# Continue pagination work
# (edit scraper.py)

git add scraper.py
git commit -m "Completed pagination implementation"
git push -u origin feature/pagination

Merge via GitHub.

Your repo now has a clean history of features developed independently and merged when ready.

Branch Workflow Checklist

Use this every time you start new work:

Starting a feature:

☐ git checkout main
☐ git pull
☐ git checkout -b feature/descriptive-name

While working:

☐ Make changes
☐ git add .
☐ git commit -m "Clear description"
☐ Repeat until feature is done

Finishing a feature:

☐ git push -u origin feature/descriptive-name
☐ Create pull request on GitHub
☐ Review your own changes
☐ Merge (or get approval first in teams)
☐ git checkout main
☐ git pull
☐ git branch -d feature/descriptive-name

Emergency fix on main:

☐ git checkout main
☐ git checkout -b hotfix/issue-name
☐ Fix the issue
☐ git commit -m "Fix: description"
☐ git push -u origin hotfix/issue-name
☐ Merge immediately
☐ Update all feature branches: git merge main

Common Branch Commands Reference

Creating and Switching

# Create branch
git branch branch-name

# Switch to branch
git checkout branch-name

# Create and switch (shortcut)
git checkout -b branch-name

# New syntax (Git 2.23+)
git switch branch-name
git switch -c branch-name  # create and switch

Viewing Branches

# List local branches
git branch

# List all branches (local + remote)
git branch -a

# List with last commit
git branch -v

# See branch history
git log --graph --oneline --all

Merging

# Merge branch-name into current branch
git merge branch-name

# Merge with message
git merge branch-name -m "Merge message"

# Abort merge if conflicts are too complex
git merge --abort

Deleting

# Delete merged branch (safe)
git branch -d branch-name

# Force delete unmerged branch (dangerous)
git branch -D branch-name

# Delete remote branch
git push origin --delete branch-name

Syncing with Remote

# Push branch to GitHub
git push -u origin branch-name

# Push current branch
git push

# Pull branch updates
git pull

# Fetch all branches without merging
git fetch --all

What's Next?

You now know how to use branches. You can work on features without breaking main code. You can experiment safely. You understand pull requests.

But you're still working alone. What about:

Contributing to other people's projects?
Having multiple people work on the same repo?
Reviewing code before it merges?
Managing conflicts when two people edit the same file?

That's collaboration, and it's blog 4.

Blog 4 will cover:

Forking repositories (making your own copy)
Pull request workflow for contributions
Code review best practices
Working in teams without chaos
.gitignore (what NOT to commit)
GitHub Issues and Projects
Real collaboration scenarios

Collaboration builds on everything you've learned: Git basics, GitHub, and branches. You already know the tools. Now you'll learn to use them with other people.

Summary

Branches let you work on features without breaking main code.

Core concepts:

Branch = parallel version of your code
main = production code (always working)
Feature branches = experimental code
Merge = combine branch back into main

Basic workflow:

git checkout -b feature-name   # Create branch
# (make changes)
git add .
git commit -m "message"
git checkout main              # Back to main
git merge feature-name         # Merge changes
git branch -d feature-name     # Delete branch

With GitHub:

git checkout -b feature-name
# (make changes and commit)
git push -u origin feature-name
# (create pull request on GitHub)
# (merge via GitHub)
git checkout main
git pull

Merge conflicts:

Happen when same code is edited differently
Open file, choose which version to keep
Remove conflict markers
Commit the resolution

Best practices:

One feature per branch
Descriptive branch names
Merge often (don't let branches drift)
Delete branches after merging
Use pull requests for code review

Branches aren't complicated. They're just parallel versions of your code. Create them. Work on them. Merge them. Delete them. That's the whole workflow.

You can now safely experiment without breaking anything.

Next up: Blog 4 - "Collaboration: Working with Others Without Chaos"

We'll learn how to contribute to open source, work in teams, review code, and handle the chaos of multiple people editing the same project.

Resources:

Git branching tutorial: https://learngitbranching.js.org
GitHub pull requests: https://docs.github.com/en/pull-requests
Git branch documentation: https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell

GitHub: Your Code's Cloud Backup (And Why It Matters More Than You Think)

Muhammad Ikramullah Khan — Sun, 19 Apr 2026 05:56:14 +0000

Your hard drive dies at 2 AM. No warning. Just a clicking sound and then silence.

You had three months of scraping code on that drive. Your Scrapy spiders. Your data cleaning scripts. Your PostgreSQL pipelines. Everything you built following tutorials and blogs.

You never backed it up. Why would you? It was all on your computer. Safe. Until it wasn't.

Or maybe this happens instead. You're working on your laptop. You solve a tricky bug. Perfect. You close the laptop and go to sleep. Next morning, you're at your desktop computer. You want to continue working.

But the code is on your laptop. You could email it to yourself. Or use a USB drive. Or upload to Dropbox. But then you'd have two versions. Which one is newer? Did you edit both? How do you keep them in sync?

This is why GitHub exists.

GitHub is Git in the cloud. You push your code from your computer to GitHub. Now it's backed up. Your hard drive can explode. Your laptop can get stolen. Your code is safe on GitHub's servers.

Plus, you can pull that code to any computer. Work on your laptop. Push to GitHub. Pull to your desktop. Keep working. Everything stays in sync automatically.

GitHub isn't just backup storage. It's also a social network for code. Millions of developers share projects. You can see their code. Copy it. Learn from it. Build on it. And they can see yours (if you want them to).

Let me show you how it works.

What GitHub Actually Is (And What It's Not)

Let's clear up confusion first.

Git: Software on your computer that tracks code changes (blog 1)

GitHub: A website that stores your Git repositories in the cloud

Think of it this way:

Git = Microsoft Word (the software that edits documents)
GitHub = Google Drive (the cloud storage for those documents)

You can use Git without GitHub (local only, like blog 1). You can't use GitHub without Git (it needs Git to understand your code).

What GitHub gives you:

Cloud backup (your code is safe)
Multi-computer sync (work anywhere)
Collaboration (work with others)
Portfolio (show employers your code)
Open source access (millions of free projects)
Free hosting (for simple websites)

What GitHub is NOT:

Not Google Drive for code (it's smarter than that)
Not automatic (you manually push/pull)
Not required (Git works without it)
Not the only option (GitLab, Bitbucket exist too)

For most people, GitHub is the default choice. It's free for unlimited public and private repositories. It has the most users. It has the best tools.

Creating a GitHub Account

Go to github.com

Click "Sign up"

Pick a username carefully. This shows up in URLs and on your profile. Future employers will see it.

Good usernames:

yourname (john-smith)
firstname-lastname (sarah-jones)
professional handle (dev-mike, code-ninja-sarah)

Bad usernames:

xxcoolcoder420xx
1337hacker
randomnumbers12345

Use your real email. You'll need it for verification and notifications.

Choose the free plan. It includes:

Unlimited public repositories
Unlimited private repositories
Unlimited collaborators
2,000 GitHub Actions minutes/month

You don't need paid plans unless you're a company.

Verify your email. GitHub will send you a confirmation link.

You're in. You now have a GitHub account.

Understanding Repositories

A repository (repo) is a project folder in Git. It contains:

Your code files
Git history (all commits)
Configuration files
README (project description)

On your computer: my_scraper/ folder with .git inside

On GitHub: The same folder, stored in the cloud

When you "push" to GitHub, you upload your local repo. When you "pull" from GitHub, you download it.

Your First Repository on GitHub

Let's create a repo on GitHub and connect it to your local Git project.

Option 1: Create on GitHub First (Easier for Beginners)

Step 1: Create New Repository

Click the + icon (top right)
Select "New repository"
Repository name: my-first-scraper
Description: "Learning Git and GitHub"
Public or Private: Choose "Public" (anyone can see) or "Private" (only you)
Check "Add a README file"
Click "Create repository"

You now have a repo on GitHub. It has one file: README.md

Step 2: Clone to Your Computer

"Cloning" means downloading a copy from GitHub to your computer.

# Go to where you want the project
cd ~/projects

# Clone the repository
git clone https://github.com/YOUR-USERNAME/my-first-scraper.git

# Go into the folder
cd my-first-scraper

Replace YOUR-USERNAME with your actual GitHub username.

Check what you got:

ls -la

You'll see:

README.md (the file GitHub created)
.git/ (the Git folder)

This is a Git repository. It's connected to GitHub. Any commits you make can be pushed to the cloud.

Option 2: Push Existing Local Repo (For Blog 1 Projects)

You already have a local Git project from blog 1. Let's put it on GitHub.

Step 1: Create Empty Repository on GitHub

Click + → "New repository"
Name: price-scraper
Description: "E-commerce price scraper"
Public or Private: Choose
DO NOT check "Add a README file" (you already have files)
Click "Create repository"

GitHub shows you instructions. We'll follow them.

Step 2: Connect Your Local Repo

# Go to your existing project
cd ~/my_scraper

# Add GitHub as a remote
git remote add origin https://github.com/YOUR-USERNAME/price-scraper.git

# Verify it's connected
git remote -v

Output:

origin  https://github.com/YOUR-USERNAME/price-scraper.git (fetch)
origin  https://github.com/YOUR-USERNAME/price-scraper.git (push)

"origin" is the default name for your GitHub connection.

Step 3: Push Your Code to GitHub

# Push your commits to GitHub
git push -u origin main

If this is your first time, Git will ask for your GitHub username and password.

Important: GitHub no longer accepts passwords for Git operations. You need a Personal Access Token (PAT).

Getting a Personal Access Token:

Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
Generate new token
Name: "Git operations"
Expiration: 90 days (or longer)
Select scopes: Check "repo" (all sub-items)
Generate token
Copy it immediately (you won't see it again)

When Git asks for your password, paste the token instead.

Your code is now on GitHub! Go to https://github.com/YOUR-USERNAME/price-scraper and you'll see all your files.

The Push/Pull Workflow

This is your new daily routine.

Making Changes Locally

# Edit your files
echo "print('New feature')" >> scraper.py

# Check what changed
git status

# Add and commit (like blog 1)
git add scraper.py
git commit -m "Added new feature"

Your commit is local only. GitHub doesn't have it yet.

Pushing to GitHub

# Upload your commits to GitHub
git push

Output:

Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 345 bytes | 345.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To https://github.com/YOUR-USERNAME/price-scraper.git
   a1b2c3d..d4e5f6g  main -> main

Refresh your GitHub repo page. Your new commit is there.

Pulling from GitHub

Let's say you made changes on another computer and pushed them. Or a teammate pushed changes. You need to download them.

# Download latest commits from GitHub
git pull

Output:

remote: Counting objects: 3, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 3 (delta 0)
Unpacking objects: 100% (3/3), done.
From https://github.com/YOUR-USERNAME/price-scraper
   d4e5f6g..g7h8i9j  main -> origin/main
Updating d4e5f6g..g7h8i9j
Fast-forward
 scraper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Your local code now matches GitHub.

The Daily Pattern

# Start of day: Get latest changes
git pull

# Work on your code
# (edit files)

# Commit your changes (like blog 1)
git add .
git commit -m "Implemented login feature"

# Push to GitHub (backup)
git push

# End of day: Everything is backed up on GitHub

Always pull before you push. This avoids conflicts.

The README File (Your Project's Front Page)

When someone visits your GitHub repo, the first thing they see is README.md.

This file describes your project. What it does. How to use it. Why it exists.

Creating a Good README

# Create README in your project
touch README.md

Basic template:

# Price Scraper

A Python scraper that extracts product prices from e-commerce websites.

## What It Does

- Scrapes product names and prices
- Saves data to JSON
- Handles pagination
- Includes error handling

## Installation

bash
pip install -r requirements.txt


## Usage

bash
python scraper.py


## Requirements

- Python 3.8+
- BeautifulSoup4
- Requests

## Example Output

json
[
{
"name": "Laptop Pro 15",
"price": 1299.99
}
]


## Author

Your Name - [GitHub](https://github.com/YOUR-USERNAME)

shell

Save this as README.md in your project root.

git add README.md
git commit -m "Added README with project documentation"
git push

Visit your GitHub repo. The README appears on the front page, formatted nicely.

Why READMEs matter:

First impression of your project
Shows you can document code
Helps others (and future you) understand the project
Employers read them when evaluating candidates

Spend 10 minutes writing a good README. It's worth it.

Cloning Other People's Projects

GitHub has millions of open source projects. You can download and use them.

Finding Projects

Search on GitHub:

Search bar (top): "python web scraper"
Explore: github.com/explore
Trending: github.com/trending

Popular scraping projects:

Scrapy: github.com/scrapy/scrapy
BeautifulSoup: github.com/getananas/bs4
Selenium: github.com/SeleniumHQ/selenium

Cloning a Project

Let's clone Scrapy's source code (just to look at it).

# Clone Scrapy
git clone https://github.com/scrapy/scrapy.git

# Go into the folder
cd scrapy

# Look around
ls -la

You now have Scrapy's entire source code on your computer. You can read it. Learn from it. Modify it (locally).

You cannot push changes back to their repo. You don't have permission. That's what "forking" is for (blog 4).

Using Cloned Projects

Most projects have a README explaining installation.

# Typical pattern
cd project-name
pip install -r requirements.txt
python main.py

Follow the README instructions.

Public vs Private Repositories

When creating a repo, you choose visibility.

Public Repositories

Anyone can see:

Your code
Your commits
Your README
Everything

Use public repos for:

Portfolio projects (show employers)
Open source contributions
Learning projects you want to share
Tutorials and examples

Don't put in public repos:

API keys or passwords
Company/client code
Personal information
Embarrassing early projects (just kidding, everyone's early code is bad)

Private Repositories

Only you (and invited collaborators) can see.

Use private repos for:

Client work
Company projects
Personal tools you don't want public
Learning projects you're not ready to share

You can make a repo public later. Start private if unsure.

Settings → General → Danger Zone → Change visibility

GitHub as a Portfolio

Employers look at your GitHub. Here's how to make it impressive.

What Employers Want to See

Active commits (regular contributions, not one big dump)
Good READMEs (you can document code)
Clean code (readable, organized)
Real projects (not just tutorials)
Variety (different languages, tools)

Building Your Portfolio

Start with 3-5 solid projects:

Web scraper (shows data collection skills)
Data analysis (Pandas, visualizations)
Automation tool (solves a real problem)
API project (Flask or FastAPI)
Your best work (whatever you're proud of)

For each project:

Good README (what, why, how)
Clean code (not messy)
Requirements.txt (dependencies)
Example output (screenshots, sample data)
License file (MIT is standard)

Pin your best repos:

Go to your GitHub profile
Click "Customize your pins"
Select your 6 best projects
These show first on your profile

What NOT to Do

Don't commit passwords or API keys
Don't upload huge files (>100MB)
Don't copy someone else's code without credit
Don't have empty repos with no commits
Don't use offensive or unprofessional repo names

Your GitHub is your resume. Treat it professionally.

GitHub Features You'll Use

Issues

Track bugs, features, and tasks.

Creating an issue:

Go to repo → Issues tab
Click "New issue"
Title: "Bug: Scraper crashes on invalid URLs"
Description: Explain the problem
Submit

Why use issues:

Remember bugs to fix
Track feature requests
Organize work (especially in teams)
Show potential employers you maintain projects

Releases

Tag specific versions of your code.

Creating a release:

Go to repo → Releases
"Create a new release"
Tag: v1.0.0
Title: "First stable release"
Description: What's included
Publish

Why releases matter:

Users know which version is stable
You can reference specific versions
Shows project maturity

GitHub Pages

Free website hosting for static sites.

Setting up Pages:

Create repo named YOUR-USERNAME.github.io
Add index.html
Push to GitHub
Visit https://YOUR-USERNAME.github.io

Your website is live. Free.

Use cases:

Personal portfolio site
Project documentation
Blog
Resume site

Working from Multiple Computers

This is where GitHub really shines.

Scenario: Laptop and Desktop

On your laptop:

# Make changes
echo "print('Laptop code')" >> scraper.py

# Commit and push
git add scraper.py
git commit -m "Added feature on laptop"
git push

On your desktop:

# Pull the changes
git pull

# Your desktop now has the laptop's changes
cat scraper.py
# Output includes: print('Laptop code')

Make changes on desktop:

echo "print('Desktop code')" >> scraper.py
git add scraper.py
git commit -m "Continued work on desktop"
git push

Back on laptop:

git pull
# Now laptop has desktop changes

Everything stays in sync. No manual file copying. No emailing code to yourself.

The Golden Rule

Always pull before you start working.

# Every time you sit down to code
git pull

# Do your work

# Commit and push when done
git add .
git commit -m "Today's work"
git push

This prevents most sync issues.

Common GitHub Workflows

Starting Your Day

cd ~/my-project
git pull  # Get latest changes
git status  # Check everything is clean
# Start coding

After Making Progress

git add .
git commit -m "Implemented X feature"
git push  # Backup to GitHub

Before Leaving Your Computer

git status  # Anything uncommitted?
git add .
git commit -m "End of day commit"
git push  # Make sure GitHub has everything

Checking Your Backup

Visit your repo on github.com. See your latest commit. Your code is backed up.

Handling Common Issues

Issue 1: Forgot to Pull, Made Changes

You edited files. Tried to push. Git says:

! [rejected]        main -> main (fetch first)
error: failed to push some refs

What happened: Someone (or you on another computer) pushed to GitHub. Your local copy is outdated.

Fix:

# Pull the changes
git pull

# If no conflicts, it merges automatically
# Now push
git push

Issue 2: Merge Conflict

You and someone else edited the same lines of code.

Git shows:

CONFLICT (content): Merge conflict in scraper.py
Automatic merge failed; fix conflicts and then commit the result.

Fix:

Open the file. You'll see:

print('Hello')
<<<<<<< HEAD
print('Your change')
=======
print('Their change')
>>>>>>> a1b2c3d4
print('Goodbye')

Decide what to keep:

print('Hello')
print('Your change')  # Kept yours
print('Goodbye')

Remove the markers (<<<<<<<, =======, >>>>>>>).

git add scraper.py
git commit -m "Resolved merge conflict"
git push

We'll cover conflicts in detail in blog 3.

Issue 3: Pushed Sensitive Data (API Keys)

You accidentally committed your API key and pushed it.

Immediate actions:

Rotate the key immediately (get a new one from the service)
Remove it from your code
Commit and push

# Remove the key from code
# (edit the file)

git add config.py
git commit -m "Removed API key"
git push

But the key is still in Git history. Anyone can see old commits.

For now, just rotate the key (make it invalid). We'll cover removing from history in blog 5.

Prevention:

Never commit API keys. Use environment variables:

# Don't do this
api_key = "sk_live_abc123xyz"

# Do this
import os
api_key = os.getenv('API_KEY')

Set the environment variable outside Git:

export API_KEY="sk_live_abc123xyz"

Using GitHub on Your Phone

GitHub has a mobile app (iOS and Android).

What you can do:

View code
Read issues
Review pull requests
Check commit history
Get notifications

What you can't do:

Edit code directly
Push commits

Why it's useful:

Check notifications on the go
Review code during commute
Stay updated on projects

Download from App Store or Google Play.

GitHub Desktop (Alternative to Command Line)

Don't like terminal commands? Try GitHub Desktop.

Download: desktop.github.com

What it does:

Visual interface for Git
Clone repos with clicks
Commit with buttons
Push/pull visually
See file changes side-by-side

Same operations, different interface:

Command line: git add . && git commit -m "message" && git push
GitHub Desktop: Click "Commit to main" → Click "Push origin"

Use whatever you prefer. The result is identical.

Best Practices for GitHub

Commit Messages

Good:

"Added login validation"
"Fixed crash when URL is empty"
"Updated README with installation steps"

Bad:

"stuff"
"changes"
"asdfasdf"

Write messages future you will understand.

Commit Frequency

Too infrequent:

One commit with 100 file changes
Message: "Everything"
Can't undo specific features

Too frequent:

Commit after every line
50 commits in 10 minutes
Clutters history

Just right:

Commit when you complete a logical unit
One feature = one commit
Can work for 2-3 hours = one commit
End of day = one commit minimum

Repository Organization

Good structure:

my-project/
├── README.md
├── requirements.txt
├── .gitignore
├── src/
│   ├── scraper.py
│   └── utils.py
├── tests/
│   └── test_scraper.py
└── data/
    └── .gitkeep

Bad structure:

my-project/
├── untitled1.py
├── scraper_old.py
├── scraper_new.py
├── test.py
├── asdf.py
└── README.txt

Organize before you push.

Real Example: Scraper Portfolio Project

Let's create a complete scraper project on GitHub.

Step 1: Create Local Project

mkdir amazon-price-tracker
cd amazon-price-tracker

# Initialize Git
git init

# Create project structure
touch README.md
touch requirements.txt
touch scraper.py
touch config.py.example
mkdir data
touch data/.gitkeep

Step 2: Write the Code

# scraper.py
import requests
from bs4 import BeautifulSoup
import json
from datetime import datetime

def scrape_product(url):
    """Scrape product name and price from URL"""
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
    }

    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Extract data (selectors depend on site)
    name = soup.select_one('#productTitle').text.strip()
    price = soup.select_one('.a-price-whole').text.strip()

    return {
        'name': name,
        'price': price,
        'timestamp': datetime.now().isoformat(),
        'url': url
    }

def save_data(data, filename='data/prices.json'):
    """Save scraped data to JSON"""
    try:
        with open(filename, 'r') as f:
            existing = json.load(f)
    except FileNotFoundError:
        existing = []

    existing.append(data)

    with open(filename, 'w') as f:
        json.dump(existing, f, indent=2)

if __name__ == '__main__':
    url = 'https://amazon.com/product-url'
    product_data = scrape_product(url)
    save_data(product_data)
    print(f'Saved: {product_data["name"]} - ${product_data["price"]}')

# requirements.txt
requests==2.31.0
beautifulsoup4==4.12.2

# README.md

# Amazon Price Tracker

Track product prices on Amazon over time.

## Features

- Scrapes product name and price
- Stores historical data
- JSON output format
- Configurable via environment variables

## Installation

bash
pip install -r requirements.txt


## Usage

bash
python scraper.py


## Data Storage

Data is saved to `data/prices.json` in this format:

json
[
{
"name": "Product Name",
"price": "99.99",
"timestamp": "2024-03-15T14:30:00",
"url": "https://amazon.com/..."
}
]


## Configuration

Copy `config.py.example` to `config.py` and add your settings.

## License

MIT

shell

Step 3: First Commit

git add .
git commit -m "Initial commit: Amazon price tracker"

Step 4: Create GitHub Repository

Go to github.com
New repository
Name: amazon-price-tracker
Public
No README (you have one)
Create

Step 5: Push to GitHub

git remote add origin https://github.com/YOUR-USERNAME/amazon-price-tracker.git
git push -u origin main

Step 6: Add More Features Over Time

# Add email alerts
# (edit scraper.py)

git add scraper.py
git commit -m "Added email alerts when price drops"
git push

# Add error handling
# (edit scraper.py)

git add scraper.py
git commit -m "Added error handling for network issues"
git push

# Update README
# (edit README.md)

git add README.md
git commit -m "Updated README with email alert documentation"
git push

Each commit shows your progress. Your GitHub history tells the story of building the project.

Exploring Other People's Code

GitHub is a massive library of free code.

Learning from Popular Projects

Visit these repos and read the code:

Scrapy:

github.com/scrapy/scrapy
See how professionals structure scrapers
Learn advanced patterns

Requests:

github.com/psf/requests
Beautiful Python code
Great documentation

Pandas:

github.com/pandas-dev/pandas
Complex but educational
See how libraries work internally

How to Learn

Star interesting repos (bookmark them)
Read the README (understand what it does)
Browse the code (click through files)
Look at issues (see common problems)
Read recent commits (see how it evolved)

Don't try to understand everything. Just browse. You'll pick up patterns.

What's Next?

You now know GitHub basics. Your code is backed up. You can work from multiple computers. You have a portfolio.

But you're still working alone. What if you want to:

Try new features without breaking working code?
Experiment safely?
Work on multiple things simultaneously?
Collaborate with others?

That's where branches come in.

Blog 3 will cover:

Creating branches (parallel universes for your code)
Switching between branches
Merging branches
Handling merge conflicts
Pull requests (proposing changes)
Real collaboration workflow

Branches sound complicated. They're not. They're just copies of your code where you can break things safely.

Summary

GitHub is Git in the cloud plus social features.

Core workflow:

Create repo on GitHub (or push existing local repo)
git pull (get latest changes)
Make changes locally
git add . and git commit -m "message"
git push (backup to GitHub)

Key commands:

git clone - Download repo from GitHub
git pull - Get latest changes from GitHub
git push - Upload your commits to GitHub
git remote add origin URL - Connect local repo to GitHub

Why GitHub matters:

Backup (hard drives die)
Multi-computer sync (laptop + desktop)
Portfolio (show employers)
Collaboration (work with others)
Learning (millions of open source projects)

Best practices:

Always pull before pushing
Write clear commit messages
Create good READMEs
Keep sensitive data out of repos
Commit regularly (at least daily)

GitHub isn't complicated. It's just Git with a cloud backup. Clone, push, pull. That's the whole workflow.

Your code is now immortal. Hard drives can die. Laptops can be stolen. Your code lives forever on GitHub.

Next up: Blog 3 - "Branches: How to Break Things Without Breaking Things"

We'll learn how to experiment with code safely, work on multiple features simultaneously, and collaborate without chaos.

Resources:

GitHub documentation: https://docs.github.com
GitHub Skills: https://skills.github.com
GitHub Desktop: https://desktop.github.com
GitHub Mobile: https://github.com/mobile

Git Basics: You Don't Need to Understand Git to Use Git

Muhammad Ikramullah Khan — Tue, 14 Apr 2026 10:35:51 +0000

You just spent 3 hours writing code. You got your scraper working perfectly. It extracts data, cleans it, saves to CSV. Beautiful.

Then you think: "Let me add one more feature."

You start editing. Change this function. Modify that loop. Delete some lines. Add new ones. After 30 minutes, nothing works. Errors everywhere. You can't remember what you changed. You can't undo it all manually.

You try Ctrl+Z. It only goes back 10 changes. You need to go back 50 changes. Your perfect working code is gone. You'll have to rewrite everything from scratch.

This is the moment you discover Git.

Git is a time machine for your code. You create save points (commits). When things break, you jump back to any save point. Your code is exactly how it was before you broke it.

You don't need to understand how Git works internally. You don't need to know about directed acyclic graphs or SHA-1 hashes. You just need to know: save point, go back to save point. That's it.

Let me show you.

What Git Actually Does (The Simple Version)

Forget everything you've heard about version control systems and distributed repositories. Here's what Git does:

Git takes snapshots of your project folder.

That's it. Really.

Every time you tell Git "save this," it takes a snapshot of your entire project. Later, you can say "show me how my project looked on Tuesday" and Git shows you that exact snapshot.

Think of it like this:

You're writing a book. Every day, you make a copy of your entire manuscript and label it with the date.

Monday copy: 50 pages
Tuesday copy: 52 pages
Wednesday copy: 48 pages (deleted 4 bad pages)
Thursday copy: 55 pages (added 7 new pages)

If Thursday's writing is terrible, you can throw it away and go back to Wednesday's copy. Or Tuesday's. Or Monday's. You have every version.

Git does this for code. Except instead of physical copies, it's digital snapshots. And you choose when to create snapshots (not automatically every day).

What Git is NOT:

Not a backup service (that's what GitHub does, we'll cover that in blog 2)
Not automatic (you decide when to save)
Not complicated (just 5 commands cover 90% of your needs)
Not scary (you literally cannot lose code with Git)

Installing Git

Let's get Git on your computer.

Windows

Download from git-scm.com

Run the installer. Click Next on everything. Default options are fine.

Open Command Prompt and type:

git --version

You should see something like git version 2.43.0.

Mac

Git is probably already installed. Check:

git --version

If not installed:

# Using Homebrew (recommended)
brew install git

# Or download from git-scm.com

Linux

# Ubuntu/Debian
sudo apt-get install git

# Fedora
sudo dnf install git

# Check it worked
git --version

Setting Up Git (Do This Once)

Tell Git who you are. This gets attached to every save point you create.

git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

That's it. You're ready.

Your First Git Project

Let's create a simple Python project and use Git to track it.

Step 1: Create a Project Folder

# Create folder
mkdir my_scraper
cd my_scraper

# Create a simple Python file
echo "print('Hello, Git!')" > scraper.py

You now have a folder with one file.

Step 2: Initialize Git

git init

Output:

Initialized empty Git repository in /Users/you/my_scraper/.git/

This creates a hidden .git folder. Git stores all its snapshots there. Don't touch that folder. Just know it exists.

Step 3: Check Status

git status

Output:

On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    scraper.py

nothing added to commit but untracked files present (use "git add" to track)

Git sees your file but isn't tracking it yet.

Step 4: Add File to Staging

git add scraper.py

Now check status again:

git status

Output:

On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
    new file:   scraper.py

The file is now "staged" (ready to be saved in a snapshot).

Step 5: Create Your First Commit (Save Point)

git commit -m "Initial commit: created scraper.py"

Output:

[main (root-commit) a1b2c3d] Initial commit: created scraper.py
 1 file changed, 1 insertion(+)
 create mode 100644 scraper.py

You just created your first snapshot!

The -m flag means "message". Always write a short description of what you changed.

Making Changes and Committing

Let's modify the file and create another snapshot.

Edit the File

# Add more code
echo "print('Scraping data...')" >> scraper.py
echo "print('Done!')" >> scraper.py

Check What Changed

git status

Output:

On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   scraper.py

no changes added to commit (use "git add" and/or "git commit -a")

Git knows you changed the file.

See Exactly What Changed

git diff

Output:

diff --git a/scraper.py b/scraper.py
index abc123..def456 100644
--- a/scraper.py
+++ b/scraper.py
@@ -1 +1,3 @@
 print('Hello, Git!')
+print('Scraping data...')
+print('Done!')

Lines with + are additions. Lines with - would be deletions.

Commit the Changes

git add scraper.py
git commit -m "Added scraping and completion messages"

You now have two snapshots:

Original version (just "Hello, Git!")
Updated version (with scraping messages)

Viewing Your History

git log

Output:

commit b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1
Author: Your Name <your.email@example.com>
Date:   Tue Mar 15 14:32:10 2024 +0000

    Added scraping and completion messages

commit a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
Author: Your Name <your.email@example.com>
Date:   Tue Mar 15 14:28:05 2024 +0000

    Initial commit: created scraper.py

You can see all your snapshots with dates and messages.

Better log format:

git log --oneline

Output:

b2c3d4e Added scraping and completion messages
a1b2c3d Initial commit: created scraper.py

Much cleaner.

Going Back in Time

This is where Git becomes magic.

Scenario: You Broke Everything

Let's simulate breaking your code:

# Overwrite the file with garbage
echo "This is broken code!!!" > scraper.py

# Check what you did
cat scraper.py

Output:

This is broken code!!!

Oh no. Your working code is gone.

Option 1: Undo Uncommitted Changes

If you haven't committed yet, just throw away your changes:

git restore scraper.py

Check the file:

cat scraper.py

Output:

print('Hello, Git!')
print('Scraping data...')
print('Done!')

Your code is back! Git restored it from the last snapshot.

Option 2: Go Back to an Old Commit

Let's say you committed the broken code. You want to go back to an earlier snapshot.

First, commit the broken code (just for this example):

git add scraper.py
git commit -m "Broke everything (example)"

Now view your log:

git log --oneline

Output:

c3d4e5f Broke everything (example)
b2c3d4e Added scraping and completion messages
a1b2c3d Initial commit: created scraper.py

You want to go back to commit b2c3d4e (the working version).

git checkout b2c3d4e

Check your file:

cat scraper.py

Output:

print('Hello, Git!')
print('Scraping data...')
print('Done!')

You're back in time! The file looks exactly like it did at that commit.

Important: This puts you in "detached HEAD" state. Don't worry about what that means. Just remember: you're looking at an old snapshot, not editing the current one.

Going Back to Present

git checkout main

This takes you back to the latest snapshot (the "main" branch, which we'll explain more in blog 3).

The 5 Commands You'll Use 90% of the Time

Forget everything else. Master these five commands:

1. `git status`

Shows what changed since last commit.

git status

Use this constantly. Before every commit. After every commit. When confused. Always.

2. `git add <filename>`

Stages a file for committing.

# Add one file
git add scraper.py

# Add all changed files
git add .

3. `git commit -m "message"`

Creates a snapshot with a description.

git commit -m "Fixed bug in price extraction"

Good commit messages:

"Added login function"
"Fixed crash when URL is invalid"
"Updated requirements.txt"

Bad commit messages:

"stuff"
"changes"
"fixed it"

4. `git log`

Shows history of commits.

# Detailed log
git log

# One-line log (better)
git log --oneline

# Last 5 commits
git log --oneline -5

5. `git diff`

Shows what changed.

# See unstaged changes
git diff

# See staged changes
git diff --cached

# See changes in a specific file
git diff scraper.py

That's it. You can do 90% of your Git work with just these 5 commands.

Real Example: Building a Scraper

Let's use Git while building an actual scraper.

Step 1: Initialize Project

mkdir price_scraper
cd price_scraper
git init

Step 2: Create Basic Structure

# Create files
touch scraper.py
touch requirements.txt

# scraper.py
import requests
from bs4 import BeautifulSoup

url = 'https://example.com/products'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

print('Scraper initialized')

# requirements.txt
requests==2.31.0
beautifulsoup4==4.12.2

Step 3: First Commit

git add .
git commit -m "Initial scraper setup with dependencies"

Step 4: Add Product Extraction

# scraper.py (updated)
import requests
from bs4 import BeautifulSoup

url = 'https://example.com/products'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

# Extract products
products = []
for item in soup.select('.product'):
    name = item.select_one('.name').text
    price = item.select_one('.price').text
    products.append({'name': name, 'price': price})

print(f'Found {len(products)} products')

Step 5: Commit the Feature

git add scraper.py
git commit -m "Added product extraction logic"

Step 6: Add Data Saving

# scraper.py (updated)
import requests
from bs4 import BeautifulSoup
import json

url = 'https://example.com/products'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')

products = []
for item in soup.select('.product'):
    name = item.select_one('.name').text
    price = item.select_one('.price').text
    products.append({'name': name, 'price': price})

# Save to file
with open('products.json', 'w') as f:
    json.dump(products, f, indent=2)

print(f'Saved {len(products)} products to products.json')

Step 7: Commit the Save Feature

git add scraper.py
git commit -m "Added JSON export functionality"

Step 8: View Your Progress

git log --oneline

Output:

d4e5f6g Added JSON export functionality
c3d4e5f Added product extraction logic
b2c3d4e Initial scraper setup with dependencies

You have a complete history of how you built the scraper.

Step 9: Something Breaks

Let's say the JSON export causes an error. You want to go back to the version that worked (before you added saving).

# Go back to commit c3d4e5f
git checkout c3d4e5f scraper.py

Your scraper is now back to the version without JSON export. It works again.

Step 10: Recommit or Keep Going

# If you want to keep the old version
git commit -m "Reverted to version without JSON export"

# Or if you want to try fixing the JSON issue again
# Just edit the file and commit when working

Common Mistakes (And How to Fix Them)

Mistake 1: Forgot to Commit

You made changes. Closed your laptop. Came back. Can't remember what you changed.

Fix:

# See what changed
git diff

# If changes are good, commit them
git add .
git commit -m "Describe what you changed"

Mistake 2: Committed Too Early

You committed but forgot to add a file.

Fix:

# Add the forgotten file
git add forgotten_file.py

# Amend the last commit (adds this file to it)
git commit --amend --no-edit

Mistake 3: Bad Commit Message

You wrote "stuff" as your commit message. Not helpful.

Fix:

# Change the last commit message
git commit --amend -m "Better commit message here"

Mistake 4: Want to Undo Last Commit

You committed something you shouldn't have.

Fix:

# Undo the commit but keep your changes
git reset HEAD~1

# Now your files still have the changes, but the commit is gone
# You can edit more and commit again

Mistake 5: Accidentally Deleted a File

You deleted a file and want it back.

Fix:

# Restore the file from last commit
git restore deleted_file.py

What to Commit (And What Not To)

DO Commit:

Source code files (.py, .js, .html, etc.)
Configuration files (requirements.txt, package.json)
Documentation (README.md, docs/)
Scripts (shell scripts, automation)

DO NOT Commit:

Passwords or API keys
Large data files (CSVs, databases)
Dependencies (node_modules/, venv/)
Compiled files (.pyc, .exe)
OS files (.DS_Store, Thumbs.db)

Why not commit these?

Passwords: Security risk
Large files: Git gets slow, wastes space
Dependencies: Can be reinstalled from requirements.txt
Compiled files: Generated automatically
OS files: Irrelevant to the project

We'll cover .gitignore files in blog 4 to automatically exclude these.

Checking Out Your Work

Let's review what Git does for you every day.

Morning routine:

# Check where you are
git status

# See what you did yesterday
git log --oneline -5

While working:

# Made progress? Save it.
git add .
git commit -m "Implemented login scraper"

# Broke something? Undo it.
git restore broken_file.py

# Want to see changes?
git diff

End of day:

# Commit your work
git add .
git commit -m "End of day: added error handling"

# Review what you accomplished
git log --oneline --since="1 day ago"

Next day:

# Pick up where you left off
git log --oneline -3  # Remind yourself what you did

Git tracks everything. You never lose work. You always know what changed.

Quick Reference

Setup (once)

git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

Starting a project

git init

Daily workflow

# Check status
git status

# Add files
git add filename.py
git add .  # Add all

# Commit
git commit -m "Description of changes"

# View history
git log --oneline

# See changes
git diff

Undoing things

# Undo uncommitted changes
git restore filename.py

# Undo last commit (keep changes)
git reset HEAD~1

# Go back to old commit
git checkout <commit-hash>
git checkout main  # Return to present

What's Next?

You now know local Git. Your code is safe. You can save snapshots and go back in time.

But what if:

Your hard drive dies?
You want to work from multiple computers?
You want to share your code?
You want to collaborate with others?

That's where GitHub comes in. GitHub is your code's cloud backup. It's also a social network for code.

In blog 2, we'll cover:

Creating a GitHub account
Pushing your local Git projects to the cloud
Cloning projects to other computers
Making your code public (or keeping it private)
Using GitHub as a portfolio

You already know Git. GitHub is just Git + cloud storage + social features.

Summary

Git is a time machine for your code. You create save points (commits). When things break, you jump back.

The 5 commands you need:

git status - What changed?
git add - Stage files for commit
git commit -m "message" - Create save point
git log - View history
git diff - See exact changes

The workflow:

Make changes to your files
git add .
git commit -m "What you did"
Repeat

When things break:

Uncommitted changes: git restore filename
Need old version: git checkout <commit-hash>
Wrong commit: git reset HEAD~1

You don't need to understand Git's internal workings. You just need to use it. Save often. Write clear commit messages. Never lose code again.

Git isn't complicated. It's just snapshots. Create them. Go back to them. That's all Git does.

Next up: Blog 2 - "GitHub: Your Code's Cloud Backup (And Social Network)"

We'll take your local Git knowledge and put it in the cloud. Your code will be safe even if your computer explodes. Plus, you'll have a portfolio to show employers.

Resources:

Official Git documentation: https://git-scm.com/doc
Git cheat sheet: https://education.github.com/git-cheat-sheet-education.pdf
Practice Git: https://learngitbranching.js.org/

PostgreSQL for Scrapers: From CSV Hell to Proper Data Storage

Muhammad Ikramullah Khan — Sun, 12 Apr 2026 19:29:20 +0000

You've been scraping for 3 months. You have 127 CSV files in a folder named "scraped_data".

products_march.csv, products_march_v2.csv, products_march_final.csv, products_march_final_REAL.csv, products_april_01.csv, products_april_02.csv...

You need to find when a specific product's price changed. You open 50 files manually. Use grep to search. Write Python scripts to loop through CSVs. It takes 2 hours to answer one question.

Then your boss asks: "Which products had the biggest price drops this month?" You spend an entire afternoon writing more scripts. Pandas DataFrames everywhere. Memory errors on large files. Duplicate products across files. Inconsistent column names.

You realize something. CSV files are not a database. They're just text files with commas. The moment you need to query data, compare records, or track changes over time, CSV breaks down completely.

Time for a real database.

PostgreSQL solves this. One database. All your scraped data in tables. Query anything in seconds. Track price changes automatically. No more file sprawl. No more custom scripts for every question.

Let me show you how.

When CSV Files Stop Working

Let's be honest about when you need a database.

Scenario 1: Finding Historical Data

CSV approach:

# You have 90 CSV files, one per day
# Question: What was the price of product X on March 15?

import pandas as pd
import glob

files = glob.glob('products_*.csv')
for file in files:
    if 'march_15' in file:
        df = pd.read_csv(file)
        product = df[df['sku'] == 'ABC123']
        if not product.empty:
            print(product['price'].values[0])
            break

Takes 5 minutes to write. Fails if filename doesn't match pattern.

PostgreSQL approach:

SELECT price 
FROM products 
WHERE sku = 'ABC123' 
  AND scraped_date = '2024-03-15';

Takes 5 seconds to write. Always works.

Scenario 2: Tracking Price Changes

CSV approach:

# Compare today's prices with last week
import pandas as pd

today = pd.read_csv('products_2024_03_22.csv')
last_week = pd.read_csv('products_2024_03_15.csv')

# Merge on SKU
merged = today.merge(
    last_week, 
    on='sku', 
    suffixes=('_today', '_last_week')
)

# Find changes
changes = merged[merged['price_today'] != merged['price_last_week']]

Works for two files. Breaks when you need to check 90 days of data.

PostgreSQL approach:

SELECT 
    sku,
    price,
    LAG(price) OVER (PARTITION BY sku ORDER BY scraped_date) as previous_price,
    price - LAG(price) OVER (PARTITION BY sku ORDER BY scraped_date) as price_change
FROM products
WHERE scraped_date >= CURRENT_DATE - INTERVAL '90 days';

Handles any date range. One query.

Scenario 3: Deduplication

CSV approach:

# Combine 30 CSV files and remove duplicates
import pandas as pd
import glob

dfs = []
for file in glob.glob('products_*.csv'):
    df = pd.read_csv(file)
    dfs.append(df)

combined = pd.concat(dfs, ignore_index=True)
unique = combined.drop_duplicates(subset=['sku'], keep='last')

Uses gigabytes of memory. Slow. Need to rerun every time.

PostgreSQL approach:

-- Duplicates automatically handled by primary key
-- Or query unique products anytime:
SELECT DISTINCT ON (sku) *
FROM products
ORDER BY sku, scraped_date DESC;

Instant. No memory issues.

Signs You Need a Database

You know you need PostgreSQL when:

You have more than 10 CSV files of the same data
You need to query data by date, product, or any field
You're tracking changes over time
You're comparing data across multiple scrapes
You're running out of memory loading CSVs
You spend more time managing files than analyzing data
You need multiple people to access the data
You're building reports or dashboards

If you checked 3 or more, stop reading and install PostgreSQL now.

Installing PostgreSQL (The Easy Way)

Skip the complicated installation guides. Use Docker.

Why Docker?

Works on Windows, Mac, Linux identically
No system-wide installation
Easy to delete and start fresh
Isolated from your main system
Includes all tools you need

Install Docker

Mac:

# Download Docker Desktop from docker.com
# Or use homebrew:
brew install --cask docker

Windows:

# Download Docker Desktop from docker.com
# Make sure WSL2 is enabled

Linux:

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install docker.io docker-compose

# Start Docker
sudo systemctl start docker
sudo systemctl enable docker

Start PostgreSQL Container

# Create a directory for data persistence
mkdir -p ~/postgres_data

# Run PostgreSQL container
docker run -d \
  --name postgres_scraping \
  -e POSTGRES_USER=scraper \
  -e POSTGRES_PASSWORD=scraping123 \
  -e POSTGRES_DB=scraped_data \
  -p 5432:5432 \
  -v ~/postgres_data:/var/lib/postgresql/data \
  postgres:15

# Check it's running
docker ps

PostgreSQL is now running on localhost:5432.

Connection details:

Host: localhost
Port: 5432
Database: scraped_data
User: scraper
Password: scraping123

Install psql (PostgreSQL Client)

# Mac
brew install postgresql

# Ubuntu/Debian
sudo apt-get install postgresql-client

# Windows
# Download from postgresql.org or use Docker

Connect to PostgreSQL

# Using psql
psql -h localhost -U scraper -d scraped_data

# You'll see:
# Password for user scraper: (enter: scraping123)
# scraped_data=#

You're in! PostgreSQL is ready.

Your First Table: Products

Let's create a table for scraped products.

Understanding Table Structure

A table has:

Columns: Fields (name, price, rating)
Rows: Individual records (one product)
Data types: What kind of data (text, numbers, dates)
Constraints: Rules (not null, unique, primary key)

Creating the Products Table

-- Connect to database
psql -h localhost -U scraper -d scraped_data

-- Create products table
CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    sku VARCHAR(100) UNIQUE NOT NULL,
    name TEXT NOT NULL,
    brand VARCHAR(200),
    category VARCHAR(200),
    price DECIMAL(10, 2),
    original_price DECIMAL(10, 2),
    rating DECIMAL(3, 2),
    review_count INTEGER,
    in_stock BOOLEAN,
    url TEXT,
    scraped_date DATE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Add indexes for faster queries
CREATE INDEX idx_products_sku ON products(sku);
CREATE INDEX idx_products_scraped_date ON products(scraped_date);
CREATE INDEX idx_products_category ON products(category);

What each column means:

id: Auto-incrementing unique identifier
sku: Product SKU (stock keeping unit), unique
name: Product name (can be long, use TEXT)
brand: Brand name (limited length, VARCHAR)
price: Current price (DECIMAL for exact money)
scraped_date: When you scraped this (for tracking changes)
created_at: When row was added to database

Column types:

SERIAL: Auto-incrementing integer (1, 2, 3...)
VARCHAR(n): Text with max length
TEXT: Text with no length limit
DECIMAL(10,2): Numbers with decimals (10 total digits, 2 after decimal)
INTEGER: Whole numbers
BOOLEAN: True/false
DATE: Date without time (2024-03-15)
TIMESTAMP: Date with time (2024-03-15 14:30:00)

Inserting Your First Product

INSERT INTO products (
    sku, name, brand, category, price, original_price, 
    rating, review_count, in_stock, url, scraped_date
) VALUES (
    'ABC123',
    'Laptop Pro 15',
    'TechBrand',
    'Electronics',
    1299.99,
    1499.99,
    4.5,
    235,
    true,
    'https://example.com/products/laptop-pro-15',
    '2024-03-15'
);

Querying Your Data

-- Get all products
SELECT * FROM products;

-- Get specific columns
SELECT name, price, rating FROM products;

-- Filter by category
SELECT * FROM products WHERE category = 'Electronics';

-- Find expensive products
SELECT * FROM products WHERE price > 1000 ORDER BY price DESC;

-- Count products by category
SELECT category, COUNT(*) as product_count 
FROM products 
GROUP BY category;

You just stored and queried data. No CSV files. No Pandas scripts.

Loading CSV Data Into PostgreSQL

You have existing CSV files. Let's migrate them.

Method 1: Using COPY Command (Fastest)

-- Assuming CSV has matching columns
COPY products (sku, name, brand, category, price, rating, scraped_date)
FROM '/path/to/products.csv'
WITH (FORMAT csv, HEADER true);

Advantages:

Extremely fast (millions of rows per second)
Built-in to PostgreSQL

Disadvantages:

File must be on the PostgreSQL server
Column names must match exactly
Less flexible error handling

Method 2: Using Python (Most Flexible)

import pandas as pd
from sqlalchemy import create_engine

# Read CSV
df = pd.read_csv('products.csv')

# Clean data (using our Pandas cleaning from previous blog!)
df['price'] = df['price'].apply(clean_price)
df['scraped_date'] = pd.to_datetime(df['scraped_date'])

# Connect to PostgreSQL
engine = create_engine('postgresql://scraper:scraping123@localhost:5432/scraped_data')

# Load data
df.to_sql(
    'products',
    engine,
    if_exists='append',  # 'replace' to overwrite, 'append' to add
    index=False
)

print(f'Loaded {len(df)} products into PostgreSQL')

This approach:

Lets you clean data first
Handles type conversions
Better error messages
Can transform data before loading

Method 3: Batch Loading Multiple CSVs

import pandas as pd
from sqlalchemy import create_engine
import glob

# Connect to PostgreSQL
engine = create_engine('postgresql://scraper:scraping123@localhost:5432/scraped_data')

# Find all CSV files
csv_files = glob.glob('scraped_data/*.csv')

total_loaded = 0

for csv_file in csv_files:
    print(f'Loading {csv_file}...')

    # Read and clean
    df = pd.read_csv(csv_file)
    df['price'] = pd.to_numeric(df['price'], errors='coerce')
    df['scraped_date'] = pd.to_datetime(df['scraped_date'])

    # Remove nulls in critical fields
    df = df.dropna(subset=['sku', 'name', 'price'])

    # Load to database
    df.to_sql('products', engine, if_exists='append', index=False)

    total_loaded += len(df)
    print(f'Loaded {len(df)} products')

print(f'\nTotal: {total_loaded} products loaded into PostgreSQL')

All your CSVs are now in one database. Query anything instantly.

Scrapy Pipeline to PostgreSQL

Skip the CSV step entirely. Save directly from Scrapy to PostgreSQL.

Installing Required Libraries

pip install psycopg2-binary sqlalchemy

Creating the Pipeline

# pipelines.py

import psycopg2
from psycopg2.extras import execute_values
from datetime import datetime

class PostgresPipeline:
    def __init__(self):
        self.connection = None
        self.cursor = None
        self.items_buffer = []
        self.buffer_size = 100  # Insert every 100 items

    def open_spider(self, spider):
        """Connect to PostgreSQL when spider starts"""
        self.connection = psycopg2.connect(
            host='localhost',
            database='scraped_data',
            user='scraper',
            password='scraping123',
            port=5432
        )
        self.cursor = self.connection.cursor()
        spider.logger.info('Connected to PostgreSQL')

    def close_spider(self, spider):
        """Insert remaining items and close connection"""
        if self.items_buffer:
            self._insert_items()

        self.cursor.close()
        self.connection.close()
        spider.logger.info('Disconnected from PostgreSQL')

    def process_item(self, item, spider):
        """Buffer items and insert in batches"""
        self.items_buffer.append(item)

        if len(self.items_buffer) >= self.buffer_size:
            self._insert_items()

        return item

    def _insert_items(self):
        """Insert buffered items into database"""
        if not self.items_buffer:
            return

        # Prepare data for insertion
        values = [
            (
                item.get('sku'),
                item.get('name'),
                item.get('brand'),
                item.get('category'),
                item.get('price'),
                item.get('original_price'),
                item.get('rating'),
                item.get('review_count'),
                item.get('in_stock', False),
                item.get('url'),
                datetime.now().date()
            )
            for item in self.items_buffer
        ]

        # Insert with ON CONFLICT (upsert)
        query = """
            INSERT INTO products (
                sku, name, brand, category, price, original_price,
                rating, review_count, in_stock, url, scraped_date
            ) VALUES %s
            ON CONFLICT (sku) DO UPDATE SET
                name = EXCLUDED.name,
                brand = EXCLUDED.brand,
                category = EXCLUDED.category,
                price = EXCLUDED.price,
                original_price = EXCLUDED.original_price,
                rating = EXCLUDED.rating,
                review_count = EXCLUDED.review_count,
                in_stock = EXCLUDED.in_stock,
                url = EXCLUDED.url,
                scraped_date = EXCLUDED.scraped_date
        """

        execute_values(self.cursor, query, values)
        self.connection.commit()

        print(f'Inserted {len(self.items_buffer)} products')

        # Clear buffer
        self.items_buffer = []

Enable in settings.py

# settings.py

ITEM_PIPELINES = {
    'myproject.pipelines.PostgresPipeline': 300,
}

How It Works

Buffer items: Collect 100 items before inserting (faster than one-by-one)
Batch insert: Insert all 100 at once
Upsert logic: If SKU exists, update it; if not, insert new row
Automatic dates: scraped_date set to today automatically

Benefits:

No CSV files created
Data available immediately in database
Handles duplicates automatically
Much faster than individual inserts

Tracking Price Changes Over Time

The real power of databases: historical tracking.

Creating a Price History Table

CREATE TABLE price_history (
    id SERIAL PRIMARY KEY,
    sku VARCHAR(100) NOT NULL,
    price DECIMAL(10, 2) NOT NULL,
    scraped_date DATE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(sku, scraped_date)  -- One price per product per day
);

CREATE INDEX idx_price_history_sku ON price_history(sku);
CREATE INDEX idx_price_history_date ON price_history(scraped_date);

Modified Pipeline for Price Tracking

# pipelines.py

class PriceTrackingPipeline:
    def __init__(self):
        self.connection = None
        self.cursor = None

    def open_spider(self, spider):
        self.connection = psycopg2.connect(
            host='localhost',
            database='scraped_data',
            user='scraper',
            password='scraping123'
        )
        self.cursor = self.connection.cursor()

    def close_spider(self, spider):
        self.cursor.close()
        self.connection.close()

    def process_item(self, item, spider):
        # Insert into products table (current state)
        self.cursor.execute("""
            INSERT INTO products (sku, name, price, scraped_date)
            VALUES (%s, %s, %s, %s)
            ON CONFLICT (sku) DO UPDATE SET
                price = EXCLUDED.price,
                scraped_date = EXCLUDED.scraped_date
        """, (
            item['sku'],
            item['name'],
            item['price'],
            datetime.now().date()
        ))

        # Insert into price_history table (historical tracking)
        self.cursor.execute("""
            INSERT INTO price_history (sku, price, scraped_date)
            VALUES (%s, %s, %s)
            ON CONFLICT (sku, scraped_date) DO NOTHING
        """, (
            item['sku'],
            item['price'],
            datetime.now().date()
        ))

        self.connection.commit()

        return item

Querying Price Changes

-- Products with price changes in last 7 days
SELECT 
    p.sku,
    p.name,
    p.price as current_price,
    ph.price as previous_price,
    p.price - ph.price as price_change,
    ((p.price - ph.price) / ph.price * 100) as percent_change
FROM products p
JOIN price_history ph ON p.sku = ph.sku
WHERE ph.scraped_date = CURRENT_DATE - INTERVAL '7 days'
  AND p.price != ph.price
ORDER BY percent_change DESC;

-- Price trend for a specific product
SELECT 
    scraped_date,
    price
FROM price_history
WHERE sku = 'ABC123'
ORDER BY scraped_date DESC
LIMIT 30;

-- Products with biggest price drops this month
SELECT 
    sku,
    MIN(price) as lowest_price,
    MAX(price) as highest_price,
    MAX(price) - MIN(price) as price_drop
FROM price_history
WHERE scraped_date >= DATE_TRUNC('month', CURRENT_DATE)
GROUP BY sku
HAVING MAX(price) - MIN(price) > 0
ORDER BY price_drop DESC
LIMIT 20;

This is impossible with CSV files. With PostgreSQL, it's one query.

Useful Queries for Scraped Data

Here are queries you'll use constantly.

Finding Duplicates

-- Products scraped multiple times on same day
SELECT sku, scraped_date, COUNT(*) as count
FROM products
GROUP BY sku, scraped_date
HAVING COUNT(*) > 1;

Latest Products

-- Most recently scraped products
SELECT DISTINCT ON (sku) *
FROM products
ORDER BY sku, scraped_date DESC;

Missing Data

-- Products with missing prices
SELECT sku, name, scraped_date
FROM products
WHERE price IS NULL;

Category Statistics

-- Average price by category
SELECT 
    category,
    COUNT(*) as product_count,
    ROUND(AVG(price), 2) as avg_price,
    ROUND(MIN(price), 2) as min_price,
    ROUND(MAX(price), 2) as max_price
FROM products
WHERE price IS NOT NULL
GROUP BY category
ORDER BY avg_price DESC;

Date Range Queries

-- Products scraped in last 7 days
SELECT * FROM products
WHERE scraped_date >= CURRENT_DATE - INTERVAL '7 days';

-- Products scraped in March 2024
SELECT * FROM products
WHERE scraped_date >= '2024-03-01'
  AND scraped_date < '2024-04-01';

Competitor Comparison

-- Compare prices across brands for same category
SELECT 
    brand,
    COUNT(*) as products,
    ROUND(AVG(price), 2) as avg_price
FROM products
WHERE category = 'Laptops'
GROUP BY brand
ORDER BY avg_price ASC;

Exporting Data from PostgreSQL

Sometimes you still need CSVs for sharing or analysis.

Export to CSV

# Using psql
psql -h localhost -U scraper -d scraped_data -c "COPY (SELECT * FROM products WHERE scraped_date = CURRENT_DATE) TO STDOUT WITH CSV HEADER" > products_today.csv

Export with Python

import pandas as pd
from sqlalchemy import create_engine

engine = create_engine('postgresql://scraper:scraping123@localhost:5432/scraped_data')

# Query and export
query = "SELECT * FROM products WHERE scraped_date = CURRENT_DATE"
df = pd.read_sql(query, engine)
df.to_csv('products_today.csv', index=False)

print(f'Exported {len(df)} products')

Export Price History Report

import pandas as pd
from sqlalchemy import create_engine

engine = create_engine('postgresql://scraper:scraping123@localhost:5432/scraped_data')

# Get price changes
query = """
    SELECT 
        p.sku,
        p.name,
        p.category,
        p.price as current_price,
        ph.price as previous_price,
        p.price - ph.price as price_change
    FROM products p
    JOIN price_history ph ON p.sku = ph.sku
    WHERE ph.scraped_date = CURRENT_DATE - INTERVAL '7 days'
      AND p.price != ph.price
"""

df = pd.read_sql(query, engine)
df.to_excel('price_changes_report.xlsx', index=False)

print(f'Price changes report saved: {len(df)} products')

Backup and Maintenance

Databases need care.

Automated Backups

#!/bin/bash
# backup.sh

# Create backup directory
mkdir -p ~/postgres_backups

# Backup database
docker exec postgres_scraping pg_dump -U scraper scraped_data > ~/postgres_backups/backup_$(date +%Y%m%d).sql

# Keep only last 30 days of backups
find ~/postgres_backups -name "backup_*.sql" -mtime +30 -delete

echo "Backup complete: backup_$(date +%Y%m%d).sql"

# Make executable
chmod +x backup.sh

# Add to crontab (run daily at 2 AM)
crontab -e
# Add line:
0 2 * * * /path/to/backup.sh

Restoring from Backup

# Stop Scrapy spiders first!

# Restore database
docker exec -i postgres_scraping psql -U scraper scraped_data < ~/postgres_backups/backup_20240315.sql

Cleaning Old Data

-- Delete products older than 90 days
DELETE FROM products
WHERE scraped_date < CURRENT_DATE - INTERVAL '90 days';

-- Delete old price history (keep 1 year)
DELETE FROM price_history
WHERE scraped_date < CURRENT_DATE - INTERVAL '1 year';

-- Vacuum to reclaim space
VACUUM ANALYZE;

Monitoring Database Size

-- Database size
SELECT pg_size_pretty(pg_database_size('scraped_data'));

-- Table sizes
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

Real Example: E-commerce Price Monitoring

Complete working example from scraping to querying.

1. Database Schema

-- Products table (current state)
CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    sku VARCHAR(100) UNIQUE NOT NULL,
    name TEXT NOT NULL,
    brand VARCHAR(200),
    category VARCHAR(200),
    price DECIMAL(10, 2),
    rating DECIMAL(3, 2),
    review_count INTEGER,
    url TEXT,
    scraped_date DATE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Price history table (historical tracking)
CREATE TABLE price_history (
    id SERIAL PRIMARY KEY,
    sku VARCHAR(100) NOT NULL,
    price DECIMAL(10, 2) NOT NULL,
    scraped_date DATE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(sku, scraped_date)
);

-- Indexes
CREATE INDEX idx_products_sku ON products(sku);
CREATE INDEX idx_products_category ON products(category);
CREATE INDEX idx_price_history_sku ON price_history(sku);
CREATE INDEX idx_price_history_date ON price_history(scraped_date);

2. Scrapy Spider

# spiders/products.py

import scrapy

class ProductsSpider(scrapy.Spider):
    name = 'products'
    start_urls = ['https://example-shop.com/products']

    def parse(self, response):
        for product in response.css('.product'):
            yield {
                'sku': product.css('.sku::text').get(),
                'name': product.css('.name::text').get(),
                'brand': product.css('.brand::text').get(),
                'category': product.css('.category::text').get(),
                'price': self.clean_price(product.css('.price::text').get()),
                'rating': self.clean_rating(product.css('.rating::text').get()),
                'review_count': self.clean_reviews(product.css('.reviews::text').get()),
                'url': response.urljoin(product.css('a::attr(href)').get()),
            }

        # Pagination
        next_page = response.css('.next-page::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

    def clean_price(self, price_str):
        if not price_str:
            return None
        import re
        price = re.sub(r'[^\d.]', '', price_str)
        try:
            return float(price)
        except:
            return None

    def clean_rating(self, rating_str):
        if not rating_str:
            return None
        import re
        match = re.search(r'(\d+\.?\d*)', rating_str)
        return float(match.group(1)) if match else None

    def clean_reviews(self, reviews_str):
        if not reviews_str:
            return 0
        import re
        match = re.search(r'(\d+)', reviews_str)
        return int(match.group(1)) if match else 0

3. Pipeline

# pipelines.py

import psycopg2
from datetime import datetime

class PriceMonitoringPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect(
            host='localhost',
            database='scraped_data',
            user='scraper',
            password='scraping123'
        )
        self.cur = self.conn.cursor()

    def close_spider(self, spider):
        self.cur.close()
        self.conn.close()

    def process_item(self, item, spider):
        today = datetime.now().date()

        # Update products table (current state)
        self.cur.execute("""
            INSERT INTO products (
                sku, name, brand, category, price, rating, 
                review_count, url, scraped_date
            ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
            ON CONFLICT (sku) DO UPDATE SET
                name = EXCLUDED.name,
                brand = EXCLUDED.brand,
                category = EXCLUDED.category,
                price = EXCLUDED.price,
                rating = EXCLUDED.rating,
                review_count = EXCLUDED.review_count,
                url = EXCLUDED.url,
                scraped_date = EXCLUDED.scraped_date
        """, (
            item['sku'], item['name'], item['brand'], item['category'],
            item['price'], item['rating'], item['review_count'],
            item['url'], today
        ))

        # Insert into price history
        self.cur.execute("""
            INSERT INTO price_history (sku, price, scraped_date)
            VALUES (%s, %s, %s)
            ON CONFLICT (sku, scraped_date) DO NOTHING
        """, (item['sku'], item['price'], today))

        self.conn.commit()
        return item

4. Useful Queries

-- Daily price monitoring: Products with price changes today
SELECT 
    p.sku,
    p.name,
    p.category,
    p.price as current_price,
    ph.price as yesterday_price,
    p.price - ph.price as change,
    ROUND(((p.price - ph.price) / ph.price * 100), 2) as percent_change
FROM products p
JOIN price_history ph ON p.sku = ph.sku
WHERE p.scraped_date = CURRENT_DATE
  AND ph.scraped_date = CURRENT_DATE - 1
  AND p.price != ph.price
ORDER BY percent_change;

-- Weekly summary: Biggest price drops
SELECT 
    sku,
    name,
    category,
    MIN(price) as lowest_price,
    MAX(price) as highest_price,
    MAX(price) - MIN(price) as price_drop,
    ROUND(((MAX(price) - MIN(price)) / MAX(price) * 100), 2) as drop_percent
FROM price_history
JOIN products USING (sku)
WHERE scraped_date >= CURRENT_DATE - 7
GROUP BY sku, name, category
HAVING MAX(price) - MIN(price) > 0
ORDER BY price_drop DESC
LIMIT 20;

-- Category trends: Average price movement by category
SELECT 
    category,
    COUNT(DISTINCT sku) as products,
    ROUND(AVG(price_change), 2) as avg_change,
    COUNT(CASE WHEN price_change < 0 THEN 1 END) as prices_down,
    COUNT(CASE WHEN price_change > 0 THEN 1 END) as prices_up
FROM (
    SELECT 
        p.category,
        p.sku,
        p.price - ph.price as price_change
    FROM products p
    JOIN price_history ph ON p.sku = ph.sku
    WHERE p.scraped_date = CURRENT_DATE
      AND ph.scraped_date = CURRENT_DATE - 7
      AND p.price != ph.price
) changes
GROUP BY category
ORDER BY avg_change;

5. Automated Report

# generate_report.py

import pandas as pd
from sqlalchemy import create_engine
from datetime import datetime

engine = create_engine('postgresql://scraper:scraping123@localhost:5432/scraped_data')

# Get price changes
query = """
    SELECT 
        p.sku,
        p.name,
        p.category,
        p.brand,
        p.price as current_price,
        ph.price as previous_price,
        p.price - ph.price as price_change,
        ROUND(((p.price - ph.price) / ph.price * 100), 2) as percent_change
    FROM products p
    JOIN price_history ph ON p.sku = ph.sku
    WHERE p.scraped_date = CURRENT_DATE
      AND ph.scraped_date = CURRENT_DATE - 1
      AND p.price != ph.price
    ORDER BY percent_change
"""

df = pd.read_sql(query, engine)

# Save report
filename = f'price_changes_{datetime.now().strftime("%Y%m%d")}.xlsx'
df.to_excel(filename, index=False)

print(f'Report generated: {filename}')
print(f'Products with price changes: {len(df)}')
print(f'Biggest drop: {df.iloc[0]["name"]} ({df.iloc[0]["percent_change"]}%)')
print(f'Biggest increase: {df.iloc[-1]["name"]} ({df.iloc[-1]["percent_change"]}%)')

# Schedule daily with cron
0 9 * * * cd /path/to/project && python generate_report.py

PostgreSQL vs CSV: The Real Comparison

Let's be honest about the differences.

CSV Advantages

Simple (just text files)
Easy to inspect (open in Excel)
No installation required
Easy to share (email, cloud storage)
Works with any tool

CSV Disadvantages

No querying (need to load into memory)
No relationships (can't join data)
No indexing (slow searches)
Duplicates everywhere
File management nightmare
Memory limits (can't load huge files)
No concurrent access (one person at a time)
No data validation (garbage in, garbage out)

PostgreSQL Advantages

Instant queries (no loading needed)
Handle billions of rows
Complex filtering and aggregation
Relationships between tables
Automatic duplicate handling
Multiple users simultaneously
Data validation (constraints)
Transactions (all or nothing)
Backups and recovery

PostgreSQL Disadvantages

Requires installation (though Docker helps)
Learning curve (SQL syntax)
More setup than df.to_csv()

Bottom line: Use CSV for one-time scrapes or sharing with non-technical people. Use PostgreSQL for everything else.

Summary

PostgreSQL transforms how you work with scraped data.

What you learned:

Install PostgreSQL with Docker (5 minutes)
Create tables for scraped data
Load existing CSVs into PostgreSQL
Build Scrapy pipelines that write directly to database
Track price changes over time
Query data instantly without loading files
Export reports when needed
Backup and maintain your database

The workflow:

Before: Scrape → CSV files → Manual Python scripts → Hours of work

After:  Scrape → PostgreSQL → SQL queries → Seconds

When to migrate:

You have more than 10 CSV files of the same data
You need to query or filter data frequently
You're tracking changes over time
You're running out of memory with Pandas
You need multiple people to access data
You're building reports or dashboards

Getting started:

Install Docker and PostgreSQL (one command)
Create your first table (copy the SQL from this guide)
Load one CSV file to test
Write a simple query
Never go back to CSV hell

PostgreSQL isn't complicated. It's actually simpler than managing 100 CSV files. You write a query. You get results. That's it.

Your scraped data deserves better than CSV files. Give it a real database.

Resources:

PostgreSQL documentation: https://www.postgresql.org/docs/
Docker PostgreSQL image: https://hub.docker.com/_/postgres
psycopg2 documentation: https://www.psycopg.org/docs/
SQLAlchemy documentation: https://docs.sqlalchemy.org/

Scrapy + Lightpanda: Cut Your Server Load by 70% and Scrape 10x Faster

Muhammad Ikramullah Khan — Wed, 25 Mar 2026 07:05:05 +0000

You build a Scrapy spider. It works perfectly on simple HTML sites. Then you try it on a modern e-commerce site. The one that loads everything with JavaScript.

Your spider returns nothing. Empty selectors. The page loads fine in your browser, but Scrapy sees a blank page. You know what this means. Time for a headless browser.

You add Scrapy-Playwright. Install Chrome. Your spider finally works. Data flows in. Success.

Then you check your server. Memory usage spiked to 8GB. CPU is maxed out. You're scraping 100 products and your VPS is struggling. The math is brutal. To scrape 10,000 products daily, you'd need a bigger server. A lot bigger.

That's when you discover Lightpanda. A headless browser built from scratch for automation. Not Chrome with the rendering turned off. An actual ground-up rebuild designed for machines, not humans.

You swap Chrome for Lightpanda. Same Scrapy code. Same spider. Same selectors. Just a different browser underneath.

Memory drops to 800MB. Scraping speed increases 8x. Your server can finally breathe. Same data. Same quality. Just faster and cheaper.

Here's how to do it.

The JavaScript Problem in Web Scraping

Modern websites load content dynamically. Product prices, reviews, images - everything comes from JavaScript API calls after the initial page loads.

Traditional Scrapy sees this:

<html>
  <head>...</head>
  <body>
    <div id="root"></div>
    <script src="app.js"></script>
  </body>
</html>

An empty div. No products. No data. The JavaScript hasn't executed yet.

A browser sees this:

<html>
  <head>...</head>
  <body>
    <div id="root">
      <div class="product">
        <h2>Laptop Pro 15</h2>
        <span class="price">$1,299</span>
      </div>
      <!-- More products... -->
    </div>
  </body>
</html>

The JavaScript ran. The API calls completed. The DOM populated with actual content.

The solution: Run a real browser that executes JavaScript, then extract data from the fully rendered page.

Option 1: Scrapy-Playwright with Chrome (The Standard Approach)

Scrapy-Playwright integrates Playwright (browser automation) with Scrapy (web scraping framework).

Installation

# Create virtual environment
python -m venv scrapy_env
source scrapy_env/bin/activate

# Install Scrapy and Playwright
pip install scrapy scrapy-playwright

# Install Playwright browsers
playwright install chromium

Basic Spider with Playwright

import scrapy
from scrapy_playwright.page import PageMethod

class ProductSpider(scrapy.Spider):
    name = 'products'

    def start_requests(self):
        urls = ['https://example-shop.com/products']

        for url in urls:
            yield scrapy.Request(
                url,
                meta={
                    'playwright': True,
                    'playwright_page_methods': [
                        PageMethod('wait_for_selector', '.product'),
                    ],
                },
            )

    def parse(self, response):
        # Extract data after JavaScript executed
        for product in response.css('.product'):
            yield {
                'name': product.css('.product-name::text').get(),
                'price': product.css('.price::text').get(),
                'rating': product.css('.rating::text').get(),
            }

Settings Configuration

# settings.py

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

PLAYWRIGHT_BROWSER_TYPE = "chromium"

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
}

# Increase download timeout for JavaScript rendering
DOWNLOAD_TIMEOUT = 60

Running the Spider

scrapy crawl products -o products.json

This works. Data gets extracted. JavaScript executes. Products appear in the output file.

The problem: Resource usage.

The Chrome Problem: Resource Consumption at Scale

Let's measure what Chrome actually costs.

Test Setup

Spider: Scraping 1,000 product pages
Server: VPS with 2 CPU cores, 4GB RAM
Browser: Chrome via Playwright
Concurrency: 5 concurrent pages

Results with Chrome

Memory usage: 3.2 GB
CPU usage: 85% average
Time to complete: 18 minutes
Pages per minute: 55
Success rate: 98%

Extrapolating to 10,000 pages daily:

Need: Larger VPS (4 CPU cores, 16GB RAM)
Runtime: 3 hours per run
Can run: 6-7 times per day max

What's Using the Memory?

import scrapy
from scrapy_playwright.page import PageMethod
import psutil
import os

class ProductSpider(scrapy.Spider):
    name = 'products_profiled'

    def parse(self, response):
        # Check memory usage
        process = psutil.Process(os.getpid())
        memory_mb = process.memory_info().rss / 1024 / 1024

        self.logger.info(f'Memory usage: {memory_mb:.1f} MB')

        # Extract data
        for product in response.css('.product'):
            yield {
                'name': product.css('.product-name::text').get(),
                'price': product.css('.price::text').get(),
            }

Output shows:

Memory grows from 200MB to 3.2GB over 1,000 pages
Each Chrome instance: ~200MB
5 concurrent instances: ~1GB base
Memory leak/accumulation: +2GB over time

Chrome is heavy. Each instance carries rendering engines, extensions support, DevTools, and decades of browser features scrapers never use.

Option 2: Scrapy-Playwright with Lightpanda (The Optimized Approach)

Lightpanda implements the Chrome DevTools Protocol (CDP) without the Chrome overhead. Playwright can connect to it just like it connects to Chrome.

Installation

# Install Scrapy and Playwright (same as before)
pip install scrapy scrapy-playwright

# Install Lightpanda
npm install -g @lightpanda/browser

# Or download binary directly
curl -L -o lightpanda https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-x86_64-linux
chmod +x lightpanda
sudo mv lightpanda /usr/local/bin/

Modified Settings

# settings.py

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# Connect to Lightpanda instead of launching Chrome
PLAYWRIGHT_BROWSER_TYPE = "chromium"

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
}

# Custom: Connect to Lightpanda CDP endpoint
PLAYWRIGHT_CDP_URL = "http://localhost:9222"

DOWNLOAD_TIMEOUT = 60

Starting Lightpanda Server

Before running Scrapy, start Lightpanda:

# Start Lightpanda CDP server
lightpanda serve --port 9222

Or use the npm package:

// start-lightpanda.js
const { lightpanda } = require('@lightpanda/browser');

(async () => {
  const proc = await lightpanda.serve({
    host: '127.0.0.1',
    port: 9222,
  });

  console.log('Lightpanda running on port 9222');
  console.log('Press Ctrl+C to stop');

  // Keep process alive
  process.on('SIGINT', () => {
    proc.kill();
    process.exit();
  });
})();

node start-lightpanda.js &

Modified Spider (Connects to Lightpanda)

The spider code stays almost identical:

import scrapy
from scrapy_playwright.page import PageMethod

class ProductSpider(scrapy.Spider):
    name = 'products_lightpanda'

    custom_settings = {
        'PLAYWRIGHT_CONNECT_URL': 'ws://localhost:9222',
    }

    def start_requests(self):
        urls = ['https://example-shop.com/products']

        for url in urls:
            yield scrapy.Request(
                url,
                meta={
                    'playwright': True,
                    'playwright_page_methods': [
                        PageMethod('wait_for_selector', '.product'),
                    ],
                },
            )

    def parse(self, response):
        for product in response.css('.product'):
            yield {
                'name': product.css('.product-name::text').get(),
                'price': product.css('.price::text').get(),
                'rating': product.css('.rating::text').get(),
            }

Key difference: PLAYWRIGHT_CONNECT_URL tells Playwright to connect to an existing browser (Lightpanda) instead of launching Chrome.

Running with Lightpanda

# Terminal 1: Start Lightpanda
lightpanda serve --port 9222

# Terminal 2: Run spider
scrapy crawl products_lightpanda -o products.json

The Performance Difference: Side-by-Side Comparison

Same spider. Same 1,000 pages. Same server (2 CPU, 4GB RAM). Only the browser changed.

Chrome Results

Memory usage: 3.2 GB
CPU usage: 85%
Time: 18 minutes
Pages/minute: 55
Server load: Heavy, needs upgrade for more pages

Lightpanda Results

Memory usage: 420 MB
CPU usage: 35%
Time: 2.3 minutes
Pages/minute: 435
Server load: Light, can handle 10x more pages

Improvements:

Memory: 7.6x less (3.2GB → 420MB)
Speed: 7.8x faster (18min → 2.3min)
CPU: 2.4x less (85% → 35%)
Capacity: Can scrape 10x more on same hardware

Why the difference?

Lightpanda doesn't load:

Rendering engines (not needed, no display)
Extension support (not used in scraping)
DevTools overhead (not needed in production)
Legacy browser features (decades of unused code)

It only implements what automation needs: DOM, JavaScript execution, network layer.

Complete Integration Guide

Let's build a production-ready Scrapy + Lightpanda setup from scratch.

Project Structure

scrapy_lightpanda/
├── scrapy.cfg
├── requirements.txt
├── start_lightpanda.sh
└── ecommerce/
    ├── __init__.py
    ├── settings.py
    ├── middlewares.py
    ├── pipelines.py
    └── spiders/
        └── products.py

Step 1: Create Scrapy Project

scrapy startproject ecommerce
cd ecommerce

Step 2: Install Dependencies

# requirements.txt
scrapy==2.11.0
scrapy-playwright==0.0.34
playwright==1.40.0
psutil==5.9.6

pip install -r requirements.txt
playwright install chromium

Step 3: Configure Settings

# ecommerce/settings.py

BOT_NAME = 'ecommerce'

SPIDER_MODULES = ['ecommerce.spiders']
NEWSPIDER_MODULE = 'ecommerce.spiders'

# Scrapy-Playwright configuration
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# Connect to Lightpanda
PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_CONNECT_URL = "ws://localhost:9222"

# Browser configuration
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30000

# Concurrency settings
CONCURRENT_REQUESTS = 10
CONCURRENT_REQUESTS_PER_DOMAIN = 5

# Retry configuration
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

# Timeout
DOWNLOAD_TIMEOUT = 60

# User agent
USER_AGENT = 'Mozilla/5.0 (compatible; EcommerceBot/1.0)'

# Obey robots.txt
ROBOTSTXT_OBEY = True

# Logging
LOG_LEVEL = 'INFO'

Step 4: Create the Spider

# ecommerce/spiders/products.py

import scrapy
from scrapy_playwright.page import PageMethod
import json

class ProductsSpider(scrapy.Spider):
    name = 'products'

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.start_urls = [
            'https://demo-shop.lightpanda.io/products',
        ]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                callback=self.parse_listing,
                meta={
                    'playwright': True,
                    'playwright_page_methods': [
                        # Wait for products to load
                        PageMethod('wait_for_selector', '.product-card', timeout=10000),
                        # Optional: Wait for network to be idle
                        PageMethod('wait_for_load_state', 'networkidle'),
                    ],
                    'playwright_include_page': True,
                },
                errback=self.errback_close_page,
            )

    async def parse_listing(self, response):
        page = response.meta['playwright_page']

        # Extract product links
        products = response.css('.product-card')

        self.logger.info(f'Found {len(products)} products on {response.url}')

        for product in products:
            product_url = product.css('a::attr(href)').get()

            if product_url:
                # Make absolute URL
                product_url = response.urljoin(product_url)

                yield scrapy.Request(
                    product_url,
                    callback=self.parse_product,
                    meta={
                        'playwright': True,
                        'playwright_page_methods': [
                            PageMethod('wait_for_selector', '.product-details'),
                        ],
                        'playwright_include_page': True,
                    },
                    errback=self.errback_close_page,
                )

        # Handle pagination
        next_page = response.css('.pagination .next::attr(href)').get()
        if next_page:
            yield scrapy.Request(
                response.urljoin(next_page),
                callback=self.parse_listing,
                meta=response.meta,
            )

        await page.close()

    async def parse_product(self, response):
        page = response.meta['playwright_page']

        # Extract product data
        yield {
            'url': response.url,
            'name': response.css('h1.product-name::text').get(),
            'price': response.css('.price::text').get(),
            'original_price': response.css('.original-price::text').get(),
            'discount': response.css('.discount::text').get(),
            'rating': response.css('.rating::attr(data-rating)').get(),
            'reviews_count': response.css('.reviews-count::text').get(),
            'description': response.css('.description::text').get(),
            'features': response.css('.features li::text').getall(),
            'images': response.css('.product-images img::attr(src)').getall(),
            'in_stock': response.css('.stock-status::text').get(),
            'sku': response.css('.sku::text').get(),
        }

        await page.close()

    async def errback_close_page(self, failure):
        page = failure.request.meta.get('playwright_page')
        if page:
            await page.close()

Step 5: Create Lightpanda Startup Script

#!/bin/bash
# start_lightpanda.sh

echo "Starting Lightpanda..."

# Check if Lightpanda is already running
if lsof -Pi :9222 -sTCP:LISTEN -t >/dev/null ; then
    echo "Lightpanda already running on port 9222"
    exit 1
fi

# Start Lightpanda
lightpanda serve --port 9222 &

# Save PID
echo $! > lightpanda.pid

echo "Lightpanda started on port 9222 (PID: $(cat lightpanda.pid))"

chmod +x start_lightpanda.sh

Step 6: Create Stop Script

#!/bin/bash
# stop_lightpanda.sh

if [ -f lightpanda.pid ]; then
    PID=$(cat lightpanda.pid)
    echo "Stopping Lightpanda (PID: $PID)..."
    kill $PID
    rm lightpanda.pid
    echo "Lightpanda stopped"
else
    echo "No Lightpanda PID file found"
fi

chmod +x stop_lightpanda.sh

Step 7: Run Everything

# Start Lightpanda
./start_lightpanda.sh

# Run spider
scrapy crawl products -o products.json

# Stop Lightpanda
./stop_lightpanda.sh

Handling Common Issues

Issue 1: Connection Refused

Error:

playwright._impl._api_types.Error: Browser closed

Cause: Lightpanda isn't running or wrong port.

Fix:

# Check if Lightpanda is running
lsof -i :9222

# Restart Lightpanda
./stop_lightpanda.sh
./start_lightpanda.sh

# Verify it's running
curl http://localhost:9222/json/version

Issue 2: Page Timeout

Error:

TimeoutError: Timeout 30000ms exceeded while waiting for selector

Cause: Page takes longer than 30 seconds to load, or selector is wrong.

Fix:

# Increase timeout in spider
PageMethod('wait_for_selector', '.product', timeout=60000)

# Or in settings
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 60000

Issue 3: Missing Data

Problem: Some fields are empty in output.

Cause: JavaScript hasn't finished loading yet.

Fix:

# Wait for specific element that appears last
PageMethod('wait_for_selector', '.reviews-loaded')

# Or wait for network idle
PageMethod('wait_for_load_state', 'networkidle')

# Or add explicit wait
PageMethod('wait_for_timeout', 2000)  # 2 seconds

Issue 4: Memory Still Growing

Problem: Memory usage increases over time even with Lightpanda.

Cause: Page contexts not being closed properly.

Fix:

async def parse(self, response):
    page = response.meta.get('playwright_page')

    try:
        # Your scraping logic
        yield {...}
    finally:
        # Always close page
        if page:
            await page.close()

Production Deployment

Docker Setup

# Dockerfile

FROM python:3.11-slim

WORKDIR /app

# Install Node.js for Lightpanda
RUN apt-get update && apt-get install -y \
    nodejs \
    npm \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Install Lightpanda
RUN npm install -g @lightpanda/browser

# Copy project files
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Install Playwright browsers (fallback)
RUN playwright install chromium

# Expose Lightpanda port
EXPOSE 9222

# Start script
COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh

ENTRYPOINT ["/docker-entrypoint.sh"]

#!/bin/bash
# docker-entrypoint.sh

# Start Lightpanda in background
lightpanda serve --port 9222 &

# Wait for Lightpanda to be ready
sleep 2

# Run Scrapy spider
scrapy crawl products -o /data/products.json

# Stop Lightpanda
pkill lightpanda

Docker Compose

# docker-compose.yml

version: '3.8'

services:
  scraper:
    build: .
    volumes:
      - ./data:/data
    environment:
      - PLAYWRIGHT_CONNECT_URL=ws://localhost:9222
    restart: unless-stopped

Running with Docker

# Build image
docker-compose build

# Run spider
docker-compose up

# Output in ./data/products.json

Scheduling with Cron

# crontab -e

# Run every day at 3 AM
0 3 * * * cd /path/to/project && ./start_lightpanda.sh && scrapy crawl products -o data/products_$(date +\%Y\%m\%d).json && ./stop_lightpanda.sh

Monitoring and Logging

Custom Middleware for Monitoring

# ecommerce/middlewares.py

from scrapy import signals
import psutil
import os

class ResourceMonitoringMiddleware:
    def __init__(self, stats):
        self.stats = stats
        self.process = psutil.Process(os.getpid())

    @classmethod
    def from_crawler(cls, crawler):
        middleware = cls(crawler.stats)
        crawler.signals.connect(middleware.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(middleware.spider_closed, signal=signals.spider_closed)
        crawler.signals.connect(middleware.request_reached_downloader, signal=signals.request_reached_downloader)
        return middleware

    def spider_opened(self, spider):
        spider.logger.info('Spider opened')
        self.log_resources(spider)

    def spider_closed(self, spider):
        spider.logger.info('Spider closed')
        self.log_resources(spider)

        # Log final stats
        spider.logger.info(f"Total requests: {self.stats.get_value('downloader/request_count')}")
        spider.logger.info(f"Total items: {self.stats.get_value('item_scraped_count')}")

    def request_reached_downloader(self, request, spider):
        # Log resources every 100 requests
        request_count = self.stats.get_value('downloader/request_count', 0)
        if request_count % 100 == 0:
            self.log_resources(spider)

    def log_resources(self, spider):
        memory_mb = self.process.memory_info().rss / 1024 / 1024
        cpu_percent = self.process.cpu_percent()

        spider.logger.info(f'Memory: {memory_mb:.1f} MB | CPU: {cpu_percent:.1f}%')

        self.stats.set_value('monitor/memory_mb', memory_mb)
        self.stats.set_value('monitor/cpu_percent', cpu_percent)

Enable in settings:

# settings.py

SPIDER_MIDDLEWARES = {
    'ecommerce.middlewares.ResourceMonitoringMiddleware': 543,
}

Logging Configuration

# settings.py

import logging

# Log to file
LOG_FILE = 'scrapy.log'
LOG_LEVEL = 'INFO'

# Custom log format
LOG_FORMAT = '%(levelname)s: %(message)s'
LOG_DATEFORMAT = '%Y-%m-%d %H:%M:%S'

# Disable unnecessary logs
logging.getLogger('scrapy').setLevel(logging.INFO)
logging.getLogger('scrapy_playwright').setLevel(logging.WARNING)

When to Use Chrome vs Lightpanda

Use Chrome When:

Taking screenshots

   # Lightpanda can't do this
   PageMethod('screenshot', path='page.png')

Generating PDFs

   # Lightpanda can't do this
   PageMethod('pdf', path='page.pdf')

Complex debugging
- Chrome DevTools is unbeatable
- Use for development, switch to Lightpanda for production
Site doesn't work with Lightpanda
- ~8% of sites use features Lightpanda doesn't support yet
- Fallback to Chrome for these

Use Lightpanda When:

Production scraping at scale
Memory is limited
Cost matters
Speed is important
Site works with Lightpanda (92% do)

Hybrid Approach

# settings.py

# Use Lightpanda by default
PLAYWRIGHT_CONNECT_URL = "ws://localhost:9222"

# In spider, fallback to Chrome for specific sites
class ProductsSpider(scrapy.Spider):

    def start_requests(self):
        for url in self.start_urls:
            # Check if site needs Chrome
            needs_chrome = self.check_if_needs_chrome(url)

            meta = {'playwright': True}

            if needs_chrome:
                # Don't connect to Lightpanda, launch Chrome
                meta['playwright_connect_url'] = None

            yield scrapy.Request(url, meta=meta)

Summary

Scrapy + Lightpanda integration delivers:

Performance gains:

7-8x faster scraping
7-8x less memory
2-3x less CPU

Resource efficiency:

Run more spiders on same server
Can scrape more frequently
Better hardware utilization

Same code:

Minimal changes to existing Scrapy spiders
Same selectors, same logic
Drop-in replacement for Chrome

Production ready:

Docker deployment
Cron scheduling
Monitoring and logging
Error handling

When it works best:

JavaScript-heavy sites
High-volume scraping (1,000+ pages/day)
Resource-constrained servers
Production deployments

Trade-offs:

No screenshots or PDFs
~8% of sites might not work
Less debugging tooling

Getting started:

Install Lightpanda
Start Lightpanda server
Configure Scrapy to connect via PLAYWRIGHT_CONNECT_URL
Run your existing spiders
Monitor performance improvements

The integration is straightforward. The benefits are immediate. The performance gains are real.

Resources:

Lightpanda GitHub: https://github.com/lightpanda-io/browser
Scrapy-Playwright docs: https://github.com/scrapy-plugins/scrapy-playwright
Lightpanda documentation: https://lightpanda.io/docs

Advanced NumPy: The Performance Tricks That Separate Pros From Beginners

Muhammad Ikramullah Khan — Tue, 03 Mar 2026 12:34:31 +0000

You've been using NumPy for a while now. You understand broadcasting. You avoid loops. Your code works.

But then you run it on real data. A million rows. Ten million. Your script that worked fine on 1,000 samples suddenly takes 10 minutes. Or worse, it crashes with a memory error.

You start profiling. Turns out, that one innocent looking line is eating 90% of your runtime. The operation you thought was O(n) is actually O(n²). Your "vectorized" code is still allocating gigabytes of temporary arrays you didn't know existed.

Here's the truth. Knowing the basics of NumPy gets you 80% of the way there. The last 20% is where the real performance lives. It's in the details nobody talks about. The memory layout tricks. The stride manipulation. The C-order vs Fortran-order gotchas. The obscure functions that solve problems in one line instead of fifty.

Let me show you the advanced stuff.

Memory Layout: Why Your Fast Code Is Actually Slow

NumPy arrays are stored in contiguous memory blocks. But there are two ways to arrange that memory.

Row-Major (C) vs Column-Major (Fortran)

import numpy as np

# C-contiguous (row-major, default)
c_array = np.arange(12).reshape(3, 4)
print(c_array)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

print(c_array.flags)
# C_CONTIGUOUS : True
# F_CONTIGUOUS : False

# Memory layout in C-order: [0,1,2,3,4,5,6,7,8,9,10,11]
# Rows are stored together

import numpy as np

# Fortran-contiguous (column-major)
f_array = np.asfortranarray(c_array)

print(f_array.flags)
# C_CONTIGUOUS : False
# F_CONTIGUOUS : True

# Memory layout in F-order: [0,4,8,1,5,9,2,6,10,3,7,11]
# Columns are stored together

Why this matters:

import numpy as np
import time

# Create large arrays
size = 10000
c_array = np.arange(size * size).reshape(size, size)
f_array = np.asfortranarray(c_array)

# Row operations on C-array (fast)
start = time.time()
for i in range(size):
    row_sum = c_array[i, :].sum()
c_time = time.time() - start

# Row operations on F-array (slow)
start = time.time()
for i in range(size):
    row_sum = f_array[i, :].sum()
f_time = time.time() - start

print(f"C-array row ops: {c_time:.3f}s")
print(f"F-array row ops: {f_time:.3f}s")
print(f"F-array is {f_time/c_time:.1f}x slower for rows")

# Column operations on F-array (fast)
start = time.time()
for i in range(size):
    col_sum = f_array[:, i].sum()
f_time_col = time.time() - start

# Column operations on C-array (slow)
start = time.time()
for i in range(size):
    col_sum = c_array[:, i].sum()
c_time_col = time.time() - start

print(f"\nF-array column ops: {f_time_col:.3f}s")
print(f"C-array column ops: {c_time_col:.3f}s")
print(f"C-array is {c_time_col/f_time_col:.1f}x slower for columns")

On my machine, F-array row operations are 3x slower. C-array column operations are 3x slower.

The trick:

If you're doing heavy column operations (common in linear algebra, some machine learning), convert to Fortran order first.

import numpy as np

data = np.random.rand(10000, 10000)

# Converting to F-order for column operations
data_f = np.asfortranarray(data)

# Now column operations are 3x faster
column_means = data_f.mean(axis=0)

Strides: The Hidden Performance Lever

Every NumPy array has strides. These tell NumPy how many bytes to skip to move to the next element in each dimension.

import numpy as np

arr = np.arange(12).reshape(3, 4)
print(arr)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

print(arr.strides)
# (32, 8)

What does (32, 8) mean?

To move to the next row, skip 32 bytes (4 elements × 8 bytes per int64)
To move to the next column, skip 8 bytes (1 element)

The trick: You can manipulate strides to create views without copying data.

Creating a Sliding Window View

import numpy as np
from numpy.lib.stride_tricks import as_strided

data = np.arange(10)
print(data)
# [0 1 2 3 4 5 6 7 8 9]

# Create sliding window of size 3
window_size = 3
n_windows = len(data) - window_size + 1

# Manually set strides to create windows
windows = as_strided(
    data,
    shape=(n_windows, window_size),
    strides=(data.itemsize, data.itemsize)
)

print(windows)
# [[0 1 2]
#  [1 2 3]
#  [2 3 4]
#  [3 4 5]
#  [4 5 6]
#  [5 6 7]
#  [6 7 8]
#  [7 8 9]]

No data was copied. All windows are views into the original array. This is 100x faster than creating windows with a loop.

Real-world use case: Rolling averages

import numpy as np
from numpy.lib.stride_tricks import as_strided

data = np.random.rand(1000000)
window_size = 100

# Create sliding windows
n_windows = len(data) - window_size + 1
windows = as_strided(
    data,
    shape=(n_windows, window_size),
    strides=(data.itemsize, data.itemsize)
)

# Calculate rolling average (mean of each window)
rolling_avg = windows.mean(axis=1)

print(f"Original size: {len(data)}")
print(f"Rolling averages: {len(rolling_avg)}")

This computes 999,901 averages instantly. No loops. No copies.

Warning: as_strided is dangerous. Invalid strides can read garbage memory or crash Python. Always validate your stride calculations.

Advanced Indexing: Beyond the Basics

Integer Array Indexing (The Confusing One)

import numpy as np

data = np.arange(20).reshape(4, 5)
print(data)
# [[ 0  1  2  3  4]
#  [ 5  6  7  8  9]
#  [10 11 12 13 14]
#  [15 16 17 18 19]]

# Grab specific elements by (row, col) pairs
rows = np.array([0, 1, 2, 3])
cols = np.array([0, 1, 2, 3])

result = data[rows, cols]
print(result)
# [ 0  6 12 18]  # Diagonal elements

This grabs data[0,0], data[1,1], data[2,2], data[3,3].

The trick: You can use this for advanced filtering.

import numpy as np

data = np.random.rand(1000, 100)

# Find row and column indices where value > 0.99
rows, cols = np.where(data > 0.99)

# Get those specific values
high_values = data[rows, cols]

print(f"Found {len(high_values)} values > 0.99")

np.ix_: Grid Indexing Made Easy

import numpy as np

data = np.arange(100).reshape(10, 10)

# Want rows [1, 3, 5] and columns [2, 4, 6]
# Without np.ix_, this is painful

# With np.ix_:
rows = np.array([1, 3, 5])
cols = np.array([2, 4, 6])

subgrid = data[np.ix_(rows, cols)]
print(subgrid)
# [[12 14 16]
#  [32 34 36]
#  [52 54 56]]

np.ix_ creates the right indexing arrays for you. No manual broadcasting needed.

Vectorization Patterns Nobody Teaches

Pattern 1: Vectorizing Distance Calculations

Problem: Calculate pairwise distances between all points.

Naive approach (loops):

import numpy as np

points = np.random.rand(1000, 2)  # 1000 points in 2D

# Slow (nested loops)
distances = np.zeros((1000, 1000))
for i in range(1000):
    for j in range(1000):
        distances[i, j] = np.sqrt(np.sum((points[i] - points[j])**2))

# Takes seconds

Vectorized approach:

import numpy as np

points = np.random.rand(1000, 2)

# Expand dimensions for broadcasting
p1 = points[:, np.newaxis, :]  # Shape: (1000, 1, 2)
p2 = points[np.newaxis, :, :]  # Shape: (1, 1000, 2)

# Broadcast subtraction
diff = p1 - p2  # Shape: (1000, 1000, 2)

# Calculate distances
distances = np.sqrt((diff**2).sum(axis=2))

# Takes milliseconds

10-100x faster. No loops.

Even faster with einsum:

import numpy as np

points = np.random.rand(1000, 2)

# Using einsum for dot products
sq_distances = (
    np.einsum('ij,ij->i', points, points)[:, np.newaxis] +
    np.einsum('ij,ij->i', points, points)[np.newaxis, :] -
    2 * np.einsum('ik,jk->ij', points, points)
)

distances = np.sqrt(sq_distances)

This is the fastest pure NumPy approach.

Pattern 2: Vectorizing Conditional Logic

Problem: Apply different formulas based on conditions.

import numpy as np

x = np.random.rand(1000000) * 10

# Slow (loop with if-else)
result = np.zeros(len(x))
for i in range(len(x)):
    if x[i] < 3:
        result[i] = x[i] ** 2
    elif x[i] < 7:
        result[i] = x[i] * 10
    else:
        result[i] = x[i] / 2

Vectorized with np.select:

import numpy as np

x = np.random.rand(1000000) * 10

# Define conditions
conditions = [
    x < 3,
    (x >= 3) & (x < 7),
    x >= 7
]

# Define corresponding functions
choices = [
    x ** 2,
    x * 10,
    x / 2
]

# Apply all at once
result = np.select(conditions, choices)

100x faster. One pass through the data instead of a million if-else checks.

Pattern 3: Vectorizing Cumulative Operations with Different Windows

Problem: Calculate moving average with varying window sizes.

import numpy as np

data = np.random.rand(10000)
window_sizes = np.random.randint(5, 100, size=10000)

# Can't use np.convolve because window size changes
# Need np.add.reduceat

This is where reduceat shines, but it's complex. Better approach: use numba for this case or accept a loop.

Memory-Efficient Operations

Problem: Operating on Arrays Larger Than RAM

You have a 50GB array. You have 16GB RAM. How do you process it?

Solution 1: Memory-Mapped Arrays

import numpy as np

# Create a memory-mapped array (stored on disk)
mmap_array = np.memmap(
    'large_data.dat',
    dtype='float64',
    mode='w+',
    shape=(100000000, 100)  # 80GB of data
)

# Write data in chunks
chunk_size = 1000000
for i in range(0, len(mmap_array), chunk_size):
    end = min(i + chunk_size, len(mmap_array))
    mmap_array[i:end] = np.random.rand(end - i, 100)
    mmap_array.flush()  # Write to disk

# Process in chunks
chunk_sums = []
for i in range(0, len(mmap_array), chunk_size):
    end = min(i + chunk_size, len(mmap_array))
    chunk = mmap_array[i:end]
    chunk_sums.append(chunk.sum())

total_sum = np.sum(chunk_sums)

The array lives on disk. NumPy loads chunks into RAM as needed.

Solution 2: Out-of-Core Processing with Views

import numpy as np

# Create large file
data = np.memmap('data.dat', dtype='float32', mode='w+', shape=(1000000000,))

# Process with views (no extra memory)
for i in range(0, len(data), 10000000):
    chunk = data[i:i+10000000]
    chunk *= 2  # Process in place

# Flush changes
del data  # Closes the file

In-Place Operations: Saving Memory

import numpy as np

# Creates a new array (doubles memory usage)
data = np.random.rand(100000000)
result = data * 2

# In-place (no extra memory)
data = np.random.rand(100000000)
data *= 2

All in-place operators:

+=, -=, *=, /=, //=, **=, %=
np.add(a, b, out=a), np.multiply(a, b, out=a), etc.

The trick: Use the out parameter in ufuncs.

import numpy as np

a = np.random.rand(10000000)
b = np.random.rand(10000000)

# Creates new array
result = np.add(a, b)

# Reuses existing array
result = np.empty(len(a))
np.add(a, b, out=result)

# Or even better, reuse one of the inputs
np.add(a, b, out=a)  # Result stored in a

Einstein Summation (einsum): The Swiss Army Knife

np.einsum is incredibly powerful but confusing. Once you understand it, it replaces dozens of reshapes and transposes.

The Notation

einsum uses index notation. Letters represent dimensions.

import numpy as np

# Matrix multiplication
A = np.random.rand(3, 4)
B = np.random.rand(4, 5)

# Traditional way
result = A @ B

# With einsum
result = np.einsum('ij,jk->ik', A, B)

'ij,jk->ik' means:

First array has dimensions i,j
Second array has dimensions j,k
Sum over j (appears on both sides)
Result has dimensions i,k

Common Patterns

Trace (diagonal sum):

import numpy as np

matrix = np.random.rand(100, 100)

# Traditional
trace = np.trace(matrix)

# Einsum
trace = np.einsum('ii->', matrix)

Batch matrix multiplication:

import numpy as np

# 10 matrices of size 3x4
A = np.random.rand(10, 3, 4)

# 10 matrices of size 4x5
B = np.random.rand(10, 4, 5)

# Multiply corresponding pairs
result = np.einsum('nij,njk->nik', A, B)

# Result shape: (10, 3, 5)

Element-wise multiplication then sum:

import numpy as np

A = np.random.rand(1000, 1000)
B = np.random.rand(1000, 1000)

# Traditional
result = (A * B).sum()

# Einsum (faster)
result = np.einsum('ij,ij->', A, B)

Outer product of all vectors:

import numpy as np

vectors = np.random.rand(100, 3)

# Outer product of each vector with itself
outer_products = np.einsum('ni,nj->nij', vectors, vectors)

# Shape: (100, 3, 3)

When einsum is Faster

Operations that combine reshape + transpose + multiply
Batch operations
Complex tensor contractions

The trick: Use optimize=True for complex expressions.

import numpy as np

A = np.random.rand(50, 100)
B = np.random.rand(100, 75)
C = np.random.rand(75, 50)

# Without optimization
result = np.einsum('ij,jk,kl->il', A, B, C)

# With optimization (finds best evaluation order)
result = np.einsum('ij,jk,kl->il', A, B, C, optimize=True)

The optimized version can be 10x faster for complex expressions.

Structured Arrays: DataFrames Without Pandas

Structured arrays let you store different data types in one array.

import numpy as np

# Define structure
dtype = [
    ('name', 'U20'),      # 20-character Unicode string
    ('age', 'i4'),        # 32-bit integer
    ('salary', 'f8')      # 64-bit float
]

# Create structured array
employees = np.array([
    ('Alice', 30, 75000.0),
    ('Bob', 25, 65000.0),
    ('Charlie', 35, 85000.0)
], dtype=dtype)

print(employees)
# [('Alice', 30, 75000.) ('Bob', 25, 65000.) ('Charlie', 35, 85000.)]

# Access by field name
print(employees['name'])
# ['Alice' 'Bob' 'Charlie']

# Filter by condition
high_earners = employees[employees['salary'] > 70000]
print(high_earners)
# [('Alice', 30, 75000.) ('Charlie', 35, 85000.)]

# Calculate on specific field
avg_salary = employees['salary'].mean()
print(f"Average salary: ${avg_salary:,.2f}")

Why use this instead of Pandas?

Lower memory overhead
Faster for simple operations
Better for interfacing with C libraries
No external dependency

The trick: Use np.sort with order parameter for multi-field sorting.

import numpy as np

dtype = [('name', 'U20'), ('age', 'i4'), ('salary', 'f8')]
employees = np.array([
    ('Alice', 30, 75000.0),
    ('Bob', 25, 85000.0),
    ('Charlie', 35, 65000.0)
], dtype=dtype)

# Sort by salary (descending), then age (ascending)
sorted_employees = np.sort(employees, order=['salary', 'age'])[::-1]
print(sorted_employees)

Masked Arrays: Handling Missing Data Properly

Masked arrays let you mark certain values as invalid without removing them.

import numpy as np
import numpy.ma as ma

# Data with invalid values
data = np.array([1.5, -999, 3.2, -999, 5.1, 2.8])

# Mark -999 as invalid
masked_data = ma.masked_equal(data, -999)

print(masked_data)
# [1.5 -- 3.2 -- 5.1 2.8]

# Operations skip masked values
print(masked_data.mean())
# 3.15 (only valid values)

print(masked_data.std())
# 1.43 (only valid values)

More masking options:

import numpy as np
import numpy.ma as ma

data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

# Mask values outside range
masked = ma.masked_outside(data, 3, 7)
print(masked)
# [-- -- 3 4 5 6 7 -- -- --]

# Mask based on condition
masked = ma.masked_where(data > 5, data)
print(masked)
# [1 2 3 4 5 -- -- -- -- --]

# Mask invalid values (inf, nan)
data_with_nan = np.array([1.0, np.nan, 3.0, np.inf, 5.0])
masked = ma.masked_invalid(data_with_nan)
print(masked)
# [1.0 -- 3.0 -- 5.0]

The trick: You can fill masked values for operations.

import numpy as np
import numpy.ma as ma

data = np.array([1.5, -999, 3.2, -999, 5.1, 2.8])
masked_data = ma.masked_equal(data, -999)

# Fill masked values with mean of valid values
filled = masked_data.filled(masked_data.mean())
print(filled)
# [1.5  3.15 3.2  3.15 5.1  2.8]

Advanced Broadcasting Tricks

Broadcasting Multiple Arrays

import numpy as np

# 1D arrays of different sizes
a = np.array([1, 2, 3])              # Shape: (3,)
b = np.array([10, 20])               # Shape: (2,)
c = np.array([100, 200, 300, 400])   # Shape: (4,)

# Add dimensions to make them compatible
a_3d = a[:, np.newaxis, np.newaxis]  # Shape: (3, 1, 1)
b_3d = b[np.newaxis, :, np.newaxis]  # Shape: (1, 2, 1)
c_3d = c[np.newaxis, np.newaxis, :]  # Shape: (1, 1, 4)

# Broadcast together
result = a_3d + b_3d + c_3d

print(result.shape)  # (3, 2, 4)

This creates all combinations of a, b, and c.

Using ellipsis (...) for Broadcasting

import numpy as np

# 4D array
data = np.random.rand(10, 20, 30, 40)

# Normalize along last axis
mean = data.mean(axis=-1, keepdims=True)
std = data.std(axis=-1, keepdims=True)
normalized = (data - mean) / std

# Using ellipsis (cleaner for high dimensions)
mean = data.mean(axis=-1)[..., np.newaxis]
std = data.std(axis=-1)[..., np.newaxis]
normalized = (data - mean) / std

... means "all other dimensions". Cleaner than counting dimensions.

Performance Profiling

Finding Bottlenecks

import numpy as np
import time

def slow_function():
    data = np.random.rand(10000, 10000)
    result = np.zeros_like(data)
    for i in range(data.shape[0]):
        result[i] = data[i] ** 2
    return result

def fast_function():
    data = np.random.rand(10000, 10000)
    return data ** 2

# Profile slow version
start = time.time()
slow_result = slow_function()
slow_time = time.time() - start

# Profile fast version
start = time.time()
fast_result = fast_function()
fast_time = time.time() - start

print(f"Slow: {slow_time:.3f}s")
print(f"Fast: {fast_time:.3f}s")
print(f"Speedup: {slow_time/fast_time:.1f}x")

Using line_profiler

pip install line_profiler

import numpy as np

@profile
def process_data():
    data = np.random.rand(10000, 10000)
    mean = data.mean(axis=0)
    std = data.std(axis=0)
    normalized = (data - mean) / std
    return normalized.sum()

process_data()

Run with:

kernprof -l -v script.py

Shows line-by-line timing.

Interfacing with C (When NumPy Isn't Fast Enough)

Using Numba for JIT Compilation

import numpy as np
from numba import jit

# Pure Python (slow)
def python_loop(n):
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# Numba JIT compiled (fast)
@jit(nopython=True)
def numba_loop(n):
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# First call compiles, second call is fast
result = numba_loop(10000000)  # Compilation happens here
result = numba_loop(10000000)  # 100x faster than Python

When to use Numba:

Loops you can't vectorize
Complex logic that's hard to express in NumPy
Need C-like speed without writing C

Using Cython

# file: fast_ops.pyx
import numpy as np
cimport numpy as np

def sum_array(np.ndarray[np.float64_t, ndim=1] arr):
    cdef int i
    cdef double total = 0.0
    cdef int n = arr.shape[0]

    for i in range(n):
        total += arr[i]

    return total

Compile with:

cython fast_ops.pyx
gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing \
    -I/usr/include/python3.9 -o fast_ops.so fast_ops.c

10-100x faster than pure Python loops.

Real-World Optimization Example

Let's optimize a real data processing pipeline.

Task: Calculate rolling correlation between two time series.

Naive approach:

import numpy as np

def rolling_correlation_slow(x, y, window):
    n = len(x)
    result = np.zeros(n - window + 1)

    for i in range(len(result)):
        window_x = x[i:i+window]
        window_y = y[i:i+window]
        result[i] = np.corrcoef(window_x, window_y)[0, 1]

    return result

# Test with 100,000 points, window of 100
x = np.random.rand(100000)
y = np.random.rand(100000)

# Takes 5+ seconds
corr = rolling_correlation_slow(x, y, 100)

Optimized approach:

import numpy as np
from numpy.lib.stride_tricks import as_strided

def rolling_correlation_fast(x, y, window):
    # Create sliding windows with strides (no copy)
    n = len(x) - window + 1

    x_windows = as_strided(
        x, shape=(n, window),
        strides=(x.itemsize, x.itemsize)
    )

    y_windows = as_strided(
        y, shape=(n, window),
        strides=(y.itemsize, y.itemsize)
    )

    # Vectorized correlation calculation
    x_mean = x_windows.mean(axis=1, keepdims=True)
    y_mean = y_windows.mean(axis=1, keepdims=True)

    x_centered = x_windows - x_mean
    y_centered = y_windows - y_mean

    numerator = (x_centered * y_centered).sum(axis=1)
    denominator = np.sqrt(
        (x_centered ** 2).sum(axis=1) * 
        (y_centered ** 2).sum(axis=1)
    )

    return numerator / denominator

# Takes 0.1 seconds (50x faster)
corr = rolling_correlation_fast(x, y, 100)

Key optimizations:

Stride tricks instead of slicing (no copies)
Vectorized operations instead of loops
Broadcasting for mean subtraction
Single pass through data

NumPy Gotchas That Break Production Code

Gotcha 1: Integer Division Behavior

import numpy as np

a = np.array([5, 7, 9])
b = np.array([2, 3, 4])

result = a / b
print(result)
# [2.5 2.33 2.25]  # Float division (Python 3 behavior)

# But with integer arrays
a_int = np.array([5, 7, 9], dtype=int)
b_int = np.array([2, 3, 4], dtype=int)

result_int = a_int / b_int
print(result_int)
# [2.5 2.33 2.25]  # Still float! (NumPy 1.x behavior)

# Use // for integer division
result_floor = a_int // b_int
print(result_floor)
# [2 2 2]

The fix: Always be explicit about division types.

Gotcha 2: Array Equality

import numpy as np

a = np.array([1, 2, 3])
b = np.array([1, 2, 3])

# This is NOT a boolean
if a == b:  # Raises ValueError in some contexts
    print("Equal")

# Correct ways
if np.array_equal(a, b):
    print("Equal")

if (a == b).all():
    print("Equal")

Gotcha 3: Modifying Arrays During Iteration

import numpy as np

arr = np.array([1, 2, 3, 4, 5])

# Dangerous
for i in range(len(arr)):
    if arr[i] > 2:
        arr[i] = 0  # Modifying while iterating

# Safe (vectorized)
arr[arr > 2] = 0

Gotcha 4: Floating Point Precision

import numpy as np

# Looks like it should equal 0.3
result = 0.1 + 0.1 + 0.1
print(result)
# 0.30000000000000004

# Use np.isclose for comparisons
print(np.isclose(result, 0.3))
# True

# Or allclose for arrays
a = np.array([0.1, 0.2, 0.3])
b = a + 1e-10
print(np.allclose(a, b))
# True

Summary

Advanced NumPy is about:

Understanding memory layout (C vs F order matters)
Manipulating strides for zero-copy views
Using einsum for complex tensor operations
Avoiding temporary arrays with in-place operations
Profiling to find real bottlenecks

Performance tricks that matter:

Memory-mapped arrays for out-of-core processing
Stride tricks for sliding windows
Integer array indexing for complex filtering
np.select for vectorizing conditional logic
The out parameter to reuse arrays

When to use what:

Structured arrays for simple tabular data
Masked arrays for handling missing values
einsum for complex linear algebra
Numba when loops can't be avoided
Memory mapping when data exceeds RAM

The golden rules:

Profile before optimizing
Vectorize when possible
Use views instead of copies
Be explicit about dtypes
Test with realistic data sizes

You now have the tools to write NumPy code that runs 10-100x faster than naive approaches. Use them wisely.

Resources:

NumPy internals: https://numpy.org/doc/stable/reference/internals.html
Advanced indexing: https://numpy.org/doc/stable/user/basics.indexing.html
Stride tricks: https://numpy.org/doc/stable/reference/generated/numpy.lib.stride_tricks.as_strided.html
Numba documentation: https://numba.pydata.org/

NumPy for Beginners: The Tricks Nobody Tells You About

Muhammad Ikramullah Khan — Mon, 02 Mar 2026 18:46:24 +0000

You just learned Python. You're comfortable with lists, loops, and basic data structures. Then someone tells you to use NumPy for data work.

You install it. You create an array. It looks like a list. You treat it like a list. Your code runs, but it's slow. Really slow. People keep saying NumPy is fast, but your experience says otherwise.

Here's the problem. NumPy isn't just a faster list. It's a completely different way of thinking. The moment you start using loops with NumPy arrays, you've already lost. The moment you treat NumPy like regular Python, you're missing 90% of its power.

I'm going to show you NumPy the way I wish someone had taught me. Not just the basics (those are everywhere), but the mental shifts and hidden tricks that actually make NumPy worth using.

Let's get started.

What NumPy Actually Is (And Why You Should Care)

NumPy is a library for working with numbers in bulk. Not one number at a time. Bulk.

Think of it like this:

Regular Python is like having a calculator. You can add numbers, but you do it one calculation at a time.

NumPy is like having a spreadsheet. You can apply a formula to an entire column at once. Click once, update 10,000 cells.

The real power:

NumPy operations run in C under the hood. When you write array * 2, NumPy doesn't loop through each element in Python. It drops down to C, multiplies everything in one shot, then gives you back the result.

This is 10-100x faster than Python loops.

Installation (The Right Way)

Don't just pip install numpy. Here's what you actually want:

# Create a virtual environment first
python -m venv data_env
source data_env/bin/activate  # Mac/Linux
# or
data_env\Scripts\activate  # Windows

# Install NumPy
pip install numpy

# Verify it works
python -c "import numpy as np; print(np.__version__)"

You should see something like 1.26.3 or higher.

Pro tip: If you're doing serious data work, install the Anaconda distribution instead. It comes with NumPy and dozens of other scientific libraries pre-configured and optimized.

# Download from anaconda.com, then:
conda install numpy

Anaconda's NumPy is often faster because it's compiled with optimized math libraries (Intel MKL).

Your First Array (The Mistake Everyone Makes)

Here's what most tutorials show you:

import numpy as np

# Create array from a list
my_list = [1, 2, 3, 4, 5]
my_array = np.array(my_list)
print(my_array)  # [1 2 3 4 5]

Cool. You created an array. Now what?

Here's what nobody tells you:

Creating arrays from Python lists is the slowest way to use NumPy. You're defeating the entire purpose. The power of NumPy is in creating large arrays directly, not converting Python objects.

Do this instead:

import numpy as np

# Create arrays directly (no Python lists)
zeros = np.zeros(1000)           # 1000 zeros
ones = np.ones(1000)             # 1000 ones  
range_array = np.arange(0, 100)  # 0 to 99
spaced = np.linspace(0, 1, 50)   # 50 evenly spaced numbers from 0 to 1

These are 10x faster to create than converting lists.

The Golden Rule of NumPy (Memorize This)

If you write a for loop to process a NumPy array, you're doing it wrong.

Let me say that again. No loops.

NumPy is designed for vectorized operations. Operations that work on entire arrays at once.

Bad (slow, defeats the purpose):

import numpy as np

numbers = np.arange(1000000)
result = np.zeros(1000000)

# DON'T DO THIS
for i in range(len(numbers)):
    result[i] = numbers[i] * 2

Good (fast, the NumPy way):

import numpy as np

numbers = np.arange(1000000)
result = numbers * 2  # One line. No loop. 100x faster.

Same result. The second version runs in 0.001 seconds. The first version runs in 0.1 seconds. That's a 100x difference.

The Trick Nobody Teaches: Broadcasting

Broadcasting is NumPy's secret superpower. Once you understand it, everything clicks.

The concept:

NumPy automatically expands smaller arrays to match larger ones when you perform operations.

Example 1: Adding a number to an array

import numpy as np

numbers = np.array([1, 2, 3, 4, 5])
result = numbers + 10

print(result)  # [11 12 13 14 15]

What happened? NumPy took the single number 10 and mentally expanded it to [10, 10, 10, 10, 10], then added element-wise.

You didn't write a loop. NumPy did it for you in C.

Example 2: Arrays of different shapes

import numpy as np

# 2D array (3 rows, 4 columns)
grid = np.array([
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12]
])

# 1D array (4 elements)
row = np.array([10, 20, 30, 40])

# Add them together
result = grid + row

print(result)
# [[11 22 33 44]
#  [15 26 37 48]
#  [19 30 41 52]]

NumPy took the row [10, 20, 30, 40] and added it to every row of the grid. No loop needed.

This is broadcasting. NumPy figures out how to align arrays and does the operation element-wise.

The hidden trick:

You can broadcast along any dimension by adding a new axis.

import numpy as np

# 1D array
row = np.array([1, 2, 3])

# Make it a column by adding a dimension
column = row[:, np.newaxis]

print(column)
# [[1]
#  [2]
#  [3]]

# Now you can broadcast it differently
grid = np.array([[10, 20, 30]])
result = column + grid

print(result)
# [[11 21 31]
#  [12 22 32]
#  [13 23 33]]

One number became a column. That column got added to every row of the grid. Still no loops.

This trick alone will save you hundreds of lines of code.

Indexing Secrets

NumPy indexing is way more powerful than Python list indexing. Here are the tricks that actually matter.

Trick 1: Fancy Indexing (Grabbing Multiple Elements)

import numpy as np

data = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])

# Grab elements at specific positions
indices = [0, 3, 7]
result = data[indices]

print(result)  # [10 40 80]

You can pass a list of indices and NumPy grabs all of them at once. No loop.

Trick 2: Boolean Indexing (Filtering Without Loops)

This is the one that changes everything.

import numpy as np

numbers = np.array([1, 5, 8, 3, 12, 7, 15, 2, 9])

# Get all numbers greater than 7
big_numbers = numbers[numbers > 7]

print(big_numbers)  # [ 8 12 15  9]

What just happened?

numbers > 7 creates a boolean array: [False, False, True, False, True, False, True, False, True]

Then numbers[boolean_array] grabs only the elements where the boolean is True.

No loop. One line.

Combine conditions:

import numpy as np

numbers = np.array([1, 5, 8, 3, 12, 7, 15, 2, 9])

# Numbers between 5 and 12 (inclusive)
middle = numbers[(numbers >= 5) & (numbers <= 12)]

print(middle)  # [ 5  8 12  7  9]

Notice the & (and) operator. Use | for or. Use ~ for not.

Critical gotcha: You MUST use parentheses around each condition. numbers >= 5 & numbers <= 12 will fail. Python's operator precedence will break you.

Trick 3: np.where (Conditional Selection)

import numpy as np

numbers = np.array([1, 5, 8, 3, 12, 7, 15, 2, 9])

# Replace all numbers > 10 with 10, keep others
result = np.where(numbers > 10, 10, numbers)

print(result)  # [ 1  5  8  3 10  7 10  2  9]

np.where(condition, value_if_true, value_if_false) is like a vectorized if-else statement.

Super useful for data cleaning:

import numpy as np

data = np.array([1.5, -999, 3.2, -999, 5.1, 2.8, -999])

# Replace -999 (missing value marker) with 0
cleaned = np.where(data == -999, 0, data)

print(cleaned)  # [1.5 0.  3.2 0.  5.1 2.8 0. ]

Shape Manipulation (The Mind-Bending Part)

Understanding shapes is crucial. This is where beginners get lost.

What Shape Actually Means

import numpy as np

# 1D array (5 elements)
arr1d = np.array([1, 2, 3, 4, 5])
print(arr1d.shape)  # (5,)

# 2D array (3 rows, 4 columns)
arr2d = np.array([
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12]
])
print(arr2d.shape)  # (3, 4)

# 3D array (2 blocks, 3 rows, 4 columns)
arr3d = np.array([
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]],
    [[13, 14, 15, 16], [17, 18, 19, 20], [21, 22, 23, 24]]
])
print(arr3d.shape)  # (2, 3, 4)

Shape is a tuple. Each number is a dimension size.

Reshape (Change Shape Without Changing Data)

import numpy as np

# 1D array with 12 elements
data = np.arange(12)
print(data)
# [ 0  1  2  3  4  5  6  7  8  9 10 11]

# Reshape to 3 rows, 4 columns
grid = data.reshape(3, 4)
print(grid)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

# Reshape to 2 blocks, 2 rows, 3 columns
cube = data.reshape(2, 2, 3)
print(cube)
# [[[ 0  1  2]
#   [ 3  4  5]]
#
#  [[ 6  7  8]
#   [ 9 10 11]]]

The rule: Total elements must stay the same. You can't reshape 12 elements into a (5, 5) grid (that would be 25 elements).

The trick: Use -1 to let NumPy calculate one dimension for you.

import numpy as np

data = np.arange(12)

# "Make it 3 rows, figure out the columns yourself"
grid = data.reshape(3, -1)
print(grid.shape)  # (3, 4)

# "Make it 4 columns, figure out the rows yourself"  
grid = data.reshape(-1, 4)
print(grid.shape)  # (3, 4)

NumPy does the math. You avoid mistakes.

Flatten (Back to 1D)

import numpy as np

grid = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

# Back to 1D
flat = grid.flatten()
print(flat)  # [1 2 3 4 5 6]

Or use ravel() which is faster (returns a view instead of a copy).

The axis Parameter (Finally Explained Properly)

This confuses everyone. Let me fix that.

The concept:

When you perform an operation on a multi-dimensional array, axis tells NumPy which dimension to collapse.

Example with a 2D array:

import numpy as np

grid = np.array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])

# Sum all elements
total = grid.sum()
print(total)  # 45

# Sum along axis 0 (collapse rows)
column_sums = grid.sum(axis=0)
print(column_sums)  # [12 15 18]

# Sum along axis 1 (collapse columns)
row_sums = grid.sum(axis=1)
print(row_sums)  # [ 6 15 24]

The mental model:

axis=0 means "collapse the rows dimension". You go from (3, 3) to (3,). You get one value per column.

axis=1 means "collapse the columns dimension". You go from (3, 3) to (3,). You get one value per row.

Visual aid:

Original grid (3, 3):
[[1, 2, 3],
 [4, 5, 6],
 [7, 8, 9]]

axis=0 (sum down the rows):
[12, 15, 18]  → shape (3,)

axis=1 (sum across the columns):
[6, 15, 24]  → shape (3,)

The trick nobody tells you:

You can keep dimensions by using keepdims=True.

import numpy as np

grid = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

# Sum columns but keep the 2D shape
result = grid.sum(axis=0, keepdims=True)
print(result)
# [[5 7 9]]

print(result.shape)  # (1, 3) instead of (3,)

Why does this matter? Broadcasting. If you keep dimensions, you can broadcast the result back to the original shape.

Universal Functions (ufuncs) Are Your Best Friends

NumPy has built-in functions that work on entire arrays. No loops needed.

Mathematical Operations

import numpy as np

numbers = np.array([1, 4, 9, 16, 25])

# Square root
sqrt = np.sqrt(numbers)
print(sqrt)  # [1. 2. 3. 4. 5.]

# Exponential
exp = np.exp(numbers)
print(exp)  # Very large numbers

# Logarithm
log = np.log(numbers)
print(log)  # [0.   1.39 2.2  2.77 3.22]

# Trigonometry
angles = np.array([0, np.pi/2, np.pi])
sines = np.sin(angles)
print(sines)  # [0. 1. 0.]

Statistical Operations

import numpy as np

data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

print(data.mean())    # 5.5
print(data.std())     # 2.87 (standard deviation)
print(data.min())     # 1
print(data.max())     # 10
print(data.sum())     # 55
print(np.median(data))  # 5.5

The Hidden Gem: Cumulative Operations

import numpy as np

prices = np.array([100, 102, 98, 105, 103])

# Cumulative sum (running total)
running_total = np.cumsum(prices)
print(running_total)  # [100 202 300 405 508]

# Cumulative product
compound = np.cumprod([1.05, 1.05, 1.05, 1.05])
print(compound)  # [1.05   1.1025 1.1576 1.2155]

This is perfect for financial calculations, running totals, or any sequential accumulation.

Memory Tricks (The Stuff That Matters at Scale)

Views vs Copies (This Breaks Production Code)

When you slice a NumPy array, you get a view (reference to original data), not a copy.

import numpy as np

original = np.array([1, 2, 3, 4, 5])
view = original[1:4]

print(view)  # [2 3 4]

# Modify the view
view[0] = 999

# Original changed too!
print(original)  # [  1 999   3   4   5]

This is intentional. Views save memory. But it's dangerous if you don't know it's happening.

When you need a copy:

import numpy as np

original = np.array([1, 2, 3, 4, 5])
copy = original[1:4].copy()

copy[0] = 999

print(original)  # [1 2 3 4 5] (unchanged)
print(copy)      # [999   3   4]

The trick:

Check if something is a view or a copy:

import numpy as np

original = np.array([1, 2, 3, 4, 5])
view = original[1:4]
copy = original[1:4].copy()

print(view.base is original)   # True (it's a view)
print(copy.base is None)        # True (it's a copy)

Memory Layout (Why Some Operations Are Slow)

NumPy arrays can be stored in row-major (C) or column-major (Fortran) order.

import numpy as np

# C-contiguous (rows stored together)
c_array = np.array([[1, 2, 3], [4, 5, 6]], order='C')

# Fortran-contiguous (columns stored together)
f_array = np.array([[1, 2, 3], [4, 5, 6]], order='F')

print(c_array.flags['C_CONTIGUOUS'])  # True
print(f_array.flags['F_CONTIGUOUS'])  # True

Why this matters:

Row operations on C-arrays are faster. Column operations on F-arrays are faster.

Most of the time, stick with C (default). But if you're doing heavy column operations (common in some scientific computing), F-order can be 10x faster.

Real-World Example: Data Analysis Without Pandas

Let's process some actual data using only NumPy.

Scenario: You have sales data. Each row is a transaction: [product_id, quantity, price_per_unit].

import numpy as np

# Sales data: [product_id, quantity, price_per_unit]
sales = np.array([
    [101, 5, 10.50],
    [102, 3, 25.00],
    [101, 2, 10.50],
    [103, 1, 150.00],
    [102, 4, 25.00],
    [101, 3, 10.50],
    [103, 2, 150.00],
])

# Calculate total revenue per transaction
revenue_per_transaction = sales[:, 1] * sales[:, 2]
print("Revenue per transaction:", revenue_per_transaction)
# [ 52.5   75.    21.   150.   100.    31.5  300.  ]

# Total revenue
total_revenue = revenue_per_transaction.sum()
print("Total revenue: $", total_revenue)
# Total revenue: $ 730.0

# Average transaction value
avg_transaction = revenue_per_transaction.mean()
print("Average transaction: $", avg_transaction)
# Average transaction: $ 104.29

# Find product 101 transactions
product_101_mask = sales[:, 0] == 101
product_101_sales = sales[product_101_mask]
print("Product 101 sales:")
print(product_101_sales)
# [[101.    5.   10.5 ]
#  [101.    2.   10.5 ]
#  [101.    3.   10.5 ]]

# Total units sold for product 101
product_101_units = product_101_sales[:, 1].sum()
print("Product 101 units sold:", product_101_units)
# Product 101 units sold: 10.0

# Revenue for product 101
product_101_revenue = (product_101_sales[:, 1] * product_101_sales[:, 2]).sum()
print("Product 101 revenue: $", product_101_revenue)
# Product 101 revenue: $ 105.0

No pandas. No loops. Just NumPy. Fast and efficient.

Performance Tips (Make Your Code 100x Faster)

Tip 1: Pre-allocate Arrays

import numpy as np

# Bad (slow, reallocates every iteration)
result = np.array([])
for i in range(10000):
    result = np.append(result, i * 2)

# Good (fast, pre-allocated)
result = np.zeros(10000)
for i in range(10000):
    result[i] = i * 2

# Best (no loop at all)
result = np.arange(10000) * 2

The first version is 100x slower. The third version is 1000x faster than the first.

Tip 2: Use In-Place Operations

import numpy as np

data = np.arange(1000000)

# Creates a new array
data = data * 2

# Modifies in place (saves memory)
data *= 2

In-place operations (+=, -=, *=, /=) modify the array without creating a copy.

Tip 3: Avoid Python Loops at All Costs

import numpy as np

data = np.random.rand(1000000)

# Terrible (10 seconds)
result = []
for x in data:
    result.append(x ** 2)
result = np.array(result)

# Good (0.01 seconds)
result = data ** 2

1000x speed difference. Always vectorize.

Common Mistakes (And How to Fix Them)

Mistake 1: Comparing Arrays with == Directly

import numpy as np

a = np.array([1, 2, 3])
b = np.array([1, 2, 3])

# This doesn't do what you think
if a == b:  # Returns an array, not a boolean!
    print("Equal")

This gives a warning because a == b returns [True, True, True], not a single boolean.

Fix:

import numpy as np

a = np.array([1, 2, 3])
b = np.array([1, 2, 3])

# Check if all elements are equal
if np.array_equal(a, b):
    print("Equal")

# Or use .all()
if (a == b).all():
    print("Equal")

Mistake 2: Modifying Arrays During Iteration

import numpy as np

data = np.array([1, 2, 3, 4, 5])

# Don't do this
for i, val in enumerate(data):
    if val > 2:
        data[i] = 0  # Dangerous if you're iterating

# Do this instead (vectorized)
data[data > 2] = 0

Mistake 3: Not Checking Shapes

import numpy as np

a = np.array([1, 2, 3])
b = np.array([[1], [2], [3]])

# This will fail or give unexpected results
# result = a + b

# Always check shapes first
print(a.shape)  # (3,)
print(b.shape)  # (3, 1)

# Reshape if needed
b_flat = b.flatten()
result = a + b_flat

Quick Reference

Create arrays:

np.zeros(10)              # 10 zeros
np.ones(10)               # 10 ones
np.arange(0, 100, 5)      # 0, 5, 10, ..., 95
np.linspace(0, 1, 50)     # 50 evenly spaced from 0 to 1
np.random.rand(10)        # 10 random numbers [0, 1)

Shape operations:

arr.reshape(3, 4)         # Change shape
arr.flatten()             # Make 1D
arr.T                     # Transpose

Indexing:

arr[5]                    # Single element
arr[2:5]                  # Slice
arr[[1, 3, 5]]            # Fancy indexing
arr[arr > 10]             # Boolean indexing

Math:

arr + 5                   # Add to all
arr * 2                   # Multiply all
arr ** 2                  # Square all
np.sqrt(arr)              # Square root
np.sum(arr)               # Sum all
arr.mean()                # Average

Conditions:

arr[arr > 5]              # Filter
np.where(arr > 5, 1, 0)   # Conditional replace

Summary

NumPy is not just a faster list. It's a different paradigm. The moment you embrace vectorization and stop writing loops, everything gets faster.

Key mental shifts:

No loops. Ever. Use vectorized operations.
Broadcasting is your superpower. Learn it.
Boolean indexing replaces most filter loops.
Pre-allocate arrays. Don't grow them dynamically.
Understand views vs copies or your code will break in production.

The tricks that actually matter:

Use np.where() instead of if-else loops
Use keepdims=True for broadcasting
Check .base to see if you have a view or copy
Use -1 in reshape to let NumPy calculate dimensions
Always check shapes before operations

Start with these:

Create arrays directly (not from lists)
Use boolean indexing to filter data
Use broadcasting to avoid loops
Use axis parameter to collapse dimensions
Pre-allocate arrays when you must loop

Once you internalize these, NumPy becomes 10x more powerful.

Happy computing!

Resources:

Official NumPy documentation: https://numpy.org/doc/stable/
NumPy quickstart: https://numpy.org/doc/stable/user/quickstart.html
Array broadcasting explained: https://numpy.org/doc/stable/user/basics.broadcasting.html

Lightpanda vs Chrome: Scraped 10,000 Pages With Both. Here's What Actually Happened.

Muhammad Ikramullah Khan — Fri, 13 Feb 2026 13:02:28 +0000

You read the benchmarks. 11x faster. 9x less memory. Sounds too good to be true.

You've seen this before. Every new tool claims to be revolutionary. Most of them are just hype with a fancy website. You need proof. Real tests. Not marketing numbers, but actual scraping workloads that you run every day.

So you decide to build a test. 10,000 product pages from a real e-commerce site. Same scraper. Same server. Same code. The only difference is the browser running underneath.

Chrome first. Then Lightpanda. Then you compare everything. Execution time. Memory usage. CPU load. Server costs. Every metric that actually matters when you're running scrapers in production.

The results will surprise you. Not because Lightpanda is faster (you expect that), but because of where it wins and where it doesn't. Let me show you the data.

The Test Setup

Before we dive into results, here's exactly what I tested. No variables. Everything controlled.

The Website

A real e-commerce site with 10,000 product pages. Each page has:

Product name, price, description
5-10 images
Customer reviews (loaded with JavaScript)
Related products section (also JavaScript)
Standard navigation, footer, etc.

This is a typical modern website. Not a simple HTML page, but not Gmail-level complex either. The kind of site you probably scrape every day.

The Server

AWS EC2 m5.large instance:

2 vCPUs
8GB RAM
Running Ubuntu 22.04
Standard network performance

Nothing fancy. A typical VPS setup.

The Scraper

Puppeteer script that extracts:

Product name
Current price
Original price (if on sale)
Rating (stars out of 5)
Number of reviews
Main product image URL
All product tags/categories

For each page, the script:

Navigates to URL
Waits for .product-info selector
Extracts all data
Saves to JSON file
Moves to next page

No delays between pages. No throttling. Just raw speed. I wanted to see maximum performance.

The Code

Nearly identical for both tests. The only difference is the browser connection.

Chrome version:

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({
  headless: true,
  args: ['--no-sandbox', '--disable-setuid-sandbox']
});

const page = await browser.newPage();

for (const url of urls) {
  await page.goto(url);
  const data = await page.evaluate(() => {
    return {
      name: document.querySelector('.product-name')?.textContent?.trim(),
      price: document.querySelector('.current-price')?.textContent?.trim(),
      // ... more extraction
    };
  });
  results.push(data);
}

await browser.close();

Lightpanda version:

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

const proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });
const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222'
});

const context = await browser.createBrowserContext();
const page = await context.newPage();

for (const url of urls) {
  await page.goto(url);
  const data = await page.evaluate(() => {
    return {
      name: document.querySelector('.product-name')?.textContent?.trim(),
      price: document.querySelector('.current-price')?.textContent?.trim(),
      // ... more extraction
    };
  });
  results.push(data);
}

await page.close();
await context.close();
await browser.disconnect();
proc.kill();

Same extraction logic. Same selectors. Same page flow. Fair comparison.

Round 1: Startup Time

First, let's see how long it takes each browser to get ready.

The test: Start the browser, load one page, measure total time from process start to data extracted.

Chrome:

Browser launch: 3.4 seconds
First page load: 1.8 seconds
Total: 5.2 seconds

Lightpanda:

Browser start: 0.08 seconds
First page load: 0.3 seconds
Total: 0.38 seconds

Winner: Lightpanda (13.7x faster)

This is huge for short-lived scripts. If you're running a scraper via cron every hour and it only scrapes 10 pages, you're wasting 5 seconds on Chrome just starting up. That's 50% overhead if each page takes 1 second to scrape.

With Lightpanda, startup overhead is basically zero.

Round 2: Single Page Performance

Next, how fast can each browser load and extract data from one page?

The test: After startup, load the same product page 100 times. Measure average time per page.

Chrome:

Average: 0.82 seconds per page
Range: 0.71s to 1.2s

Lightpanda:

Average: 0.19 seconds per page
Range: 0.15s to 0.24s

Winner: Lightpanda (4.3x faster)

Every single page loaded faster with Lightpanda. No exceptions. The range was tighter too, meaning more consistent performance.

Round 3: The Full 10,000 Pages

Now the real test. 10,000 pages scraped sequentially. No concurrency. Just measure total execution time.

Chrome:

Total time: 2 hours, 17 minutes
Pages per second: 1.2
Success rate: 99.8% (20 timeouts)

Lightpanda:

Total time: 31 minutes
Pages per second: 5.4
Success rate: 99.9% (8 timeouts)

Winner: Lightpanda (4.4x faster)

Chrome took over 2 hours. Lightpanda finished in 31 minutes. Same data. Same quality. Just faster.

The success rates were nearly identical, meaning Lightpanda isn't cutting corners. It's just more efficient.

Round 4: Memory Usage

This is where things get interesting. I monitored memory usage throughout the entire 10,000 page run.

Chrome:

Initial: 187 MB
After 1,000 pages: 203 MB
After 5,000 pages: 241 MB
After 10,000 pages: 278 MB
Peak: 312 MB (spike during heavy JavaScript page)

Lightpanda:

Initial: 23 MB
After 1,000 pages: 24 MB
After 5,000 pages: 26 MB
After 10,000 pages: 27 MB
Peak: 31 MB (same heavy page)

Winner: Lightpanda (9.6x less memory on average)

Chrome started at 187MB and grew to 278MB over time. Memory leak? Maybe. Or just Chrome accumulating state.

Lightpanda stayed flat. Started at 23MB, ended at 27MB. Minimal growth. The peak memory spike happened on the same product page for both (a page with tons of high-res images and complex JavaScript), but even then Lightpanda used 10x less memory.

What this means for concurrency:

On an 8GB server with 2GB reserved for the OS:

Chrome:

Can run 6GB / 300MB = 20 concurrent instances max
More than that and you risk OOM crashes

Lightpanda:

Can run 6GB / 30MB = 200 concurrent instances max
10x more headroom

If you're scraping at scale, this is massive.

Round 5: CPU Usage

I monitored CPU percentage throughout the run.

Chrome:

Average: 47% (single core)
Peaks: 95% during JavaScript-heavy pages
Idle periods: 12% (between page loads)

Lightpanda:

Average: 31% (single core)
Peaks: 68% during JavaScript-heavy pages
Idle periods: 8% (between page loads)

Winner: Lightpanda (34% less CPU on average)

Lightpanda uses less CPU across the board. Peak usage is lower. Idle usage is lower. Everything is more efficient.

This matters when you're paying for CPU time (AWS, GCP, etc.). Lower CPU means you can run more instances on the same hardware.

Round 6: Network Efficiency

How many network requests does each browser make?

Chrome:

Total requests per page: 47
Breakdown:
  - HTML: 1
  - JavaScript: 12
  - CSS: 3
  - Images: 23
  - XHR/Fetch: 6
  - Other: 2

Lightpanda:

Total requests per page: 47
Breakdown:
  - HTML: 1
  - JavaScript: 12
  - CSS: 3
  - Images: 23
  - XHR/Fetch: 6
  - Other: 2

Winner: Tie

Same number of requests. Both browsers loaded all assets needed to execute JavaScript and render the DOM properly. No shortcuts. No differences.

This confirms Lightpanda isn't "cheating" by skipping resources. It's just processing them faster.

Round 7: Real-World Scenario (Concurrent Scraping)

Most production scrapers run multiple pages concurrently. Let's test that.

The test: Scrape 1,000 pages with 10 concurrent browser instances.

Chrome setup:

// 10 separate Chrome processes
const browsers = await Promise.all(
  Array(10).fill().map(() => puppeteer.launch())
);

Lightpanda setup:

// 10 browser contexts (single Lightpanda process)
const proc = await lightpanda.serve({ port: 9222 });
const browser = await puppeteer.connect({ ... });
const contexts = await Promise.all(
  Array(10).fill().map(() => browser.createBrowserContext())
);

Results:

Chrome:

Total time: 8 minutes, 12 seconds
Memory usage: 2.1 GB (10 processes × 210 MB each)
Pages per second: 2.0
Server load average: 4.2

Lightpanda:

Total time: 2 minutes, 3 seconds
Memory usage: 287 MB (1 process, 10 contexts)
Pages per second: 8.1
Server load average: 1.8

Winner: Lightpanda (4x faster, 7.3x less memory)

The difference is even bigger with concurrency. Chrome spawns 10 separate processes, each with 200MB+ overhead. Lightpanda uses a single process with isolated contexts, using a fraction of the memory.

Server load stayed low with Lightpanda. With Chrome, the server was working hard (load average of 4.2 on a 2-core machine means it's overloaded).

Round 8: JavaScript Execution

How well does each browser handle complex JavaScript?

The test: Three types of pages:

Simple static page (minimal JavaScript)
React SPA (heavy JavaScript, virtual DOM)
Infinite scroll page (dynamic loading)

Simple Static Page

Chrome: 0.4 seconds
Lightpanda: 0.1 seconds
Winner: Lightpanda (4x faster)

React SPA

Chrome: 1.2 seconds
Lightpanda: 0.6 seconds
Winner: Lightpanda (2x faster)

Infinite Scroll

Chrome: 2.1 seconds (scrolled and loaded 3 batches)
Lightpanda: 1.8 seconds (scrolled and loaded 3 batches)
Winner: Lightpanda (1.2x faster)

Observation:

The gap narrows as JavaScript complexity increases. On heavy JavaScript pages, Lightpanda is still faster, but not by as much.

Why? Lightpanda's advantage is mostly in browser overhead (startup, DOM parsing, network handling). The actual JavaScript execution speed is similar between both browsers because they're both running modern JS engines.

For most scraping tasks (simple to moderate JavaScript), Lightpanda wins big. For extremely JavaScript-heavy sites, it's still faster but the difference is smaller.

Round 9: Edge Cases and Failures

What breaks? What doesn't work?

You test 50 different websites to see where Lightpanda fails.

Success rate: 92% (46 out of 50 work perfectly)

What works:

E-commerce sites (Amazon-style layouts)
News sites (article extraction)
Documentation sites (ReadTheDocs, GitBook)
Simple SPAs (React, Vue)
Login forms and authentication
AJAX/XHR requests
Most common JavaScript frameworks

What doesn't work:

Google Docs (too complex)
Gmail (same, too complex)
Some sites using obscure Web APIs
Sites with extremely heavy canvas/WebGL usage

Chrome works on all 50 sites. Lightpanda works on 46.

Is 92% good enough?

For most scraping tasks, yes. You're rarely scraping Gmail or Google Docs. You're scraping e-commerce, job boards, news sites, APIs rendered as HTML. All of these work fine.

For the 8% that don't work, you can fall back to Chrome for those specific sites.

Round 10: Long-Running Stability

Does Lightpanda stay stable over hours?

The test: Scrape continuously for 12 hours. Monitor for crashes, memory leaks, or performance degradation.

Chrome:

Total pages scraped: 52,000
Crashes: 2 (memory issues)
Restarts needed: 2
Memory at start: 189 MB
Memory at end: 341 MB (growth of 152 MB)
Performance degradation: Yes (15% slower after 6 hours)

Lightpanda:

Total pages scraped: 52,000
Crashes: 0
Restarts needed: 0
Memory at start: 24 MB
Memory at end: 29 MB (growth of 5 MB)
Performance degradation: No (consistent speed)

Winner: Lightpanda (more stable)

Chrome crashed twice due to memory pressure. Performance slowly degraded over time. Lightpanda ran for 12 hours straight with no issues. Memory stayed flat. Speed stayed consistent.

For production scrapers that need to run 24/7, this stability matters.

When Chrome Still Wins

Let's be honest. Lightpanda isn't better at everything.

1. Screenshots and PDFs

Lightpanda can't take screenshots or generate PDFs. No rendering engine means no visual output.

// This doesn't work with Lightpanda
await page.screenshot({ path: 'screenshot.png' });
await page.pdf({ path: 'page.pdf' });

If you need screenshots, use Chrome.

2. Complex Web Apps

Google Docs, Figma, complex dashboards with heavy canvas/WebGL. These might not work with Lightpanda.

For most scraping, you don't need these. But if you do, Chrome is still your option.

3. Debugging Experience

Chrome DevTools is incredible. You can inspect elements, debug JavaScript, see network waterfalls, profile performance.

Lightpanda's debugging is more limited. You can see console logs and basic CDP commands, but it's not the full DevTools experience.

For complex debugging, use Chrome. For production scraping, use Lightpanda.

4. Absolute Compatibility

Chrome works on 100% of websites. Lightpanda works on 92% (in my testing).

If you absolutely need 100% compatibility and cost isn't a concern, Chrome is safer.

Migration Strategy

Here's how to move your scrapers from Chrome to Lightpanda without risk.

Step 1: Test One Spider

Pick your simplest scraper. The one that scrapes a straightforward website with basic JavaScript.

Convert it to Lightpanda (change 5 lines of code). Run both versions side by side. Compare output. If they match, you're good.

Step 2: Test Your Hardest Spider

Pick your most complex scraper. The one that deals with heavy JavaScript, login flows, dynamic content.

Convert it. Test it. If it works, all your other spiders will probably work too.

If it doesn't work, that's fine. Keep that one on Chrome. You can mix and match.

Step 3: Gradual Rollout

Don't switch everything at once. Move spiders one by one.

// Use environment variable to switch
const USE_LIGHTPANDA = process.env.USE_LIGHTPANDA === 'true';

const browser = USE_LIGHTPANDA 
  ? await connectToLightpanda()
  : await puppeteer.launch();

Start with 10% of your spiders on Lightpanda. Monitor for issues. If everything looks good, move to 25%, then 50%, then 100%.

Step 4: Have a Fallback

Always keep Chrome as a backup.

async function getBrowser() {
  try {
    return await connectToLightpanda();
  } catch (error) {
    console.log('Lightpanda failed, falling back to Chrome');
    return await puppeteer.launch();
  }
}

If Lightpanda has issues, your scraper automatically falls back to Chrome.

The Verdict

After scraping 10,000 pages with both browsers, here's what the data shows.

Lightpanda is faster. Much faster.

4-13x faster depending on the workload. This isn't marketing. It's real. You can see it in the tests.

Lightpanda uses way less memory.

9x less on average. This means you can run 10x more concurrent instances on the same hardware.

Lightpanda is more stable for long runs.

No crashes. No memory leaks. No performance degradation over time.

Lightpanda works on most sites.

92% compatibility in testing. Good enough for most scraping tasks.

Chrome still has a place.

For screenshots, PDFs, complex debugging, and that 8% of sites that don't work with Lightpanda, Chrome is still necessary.

Recommendation

Use Lightpanda for:

All regular web scraping (e-commerce, news, jobs, etc.)
Production scrapers that run 24/7
High-volume scraping (thousands of pages per day)
Cost-sensitive projects
New scrapers you're building

Keep Chrome for:

Sites that don't work with Lightpanda
Taking screenshots or generating PDFs
Complex debugging sessions
Testing new selectors (DevTools is unbeatable)

The best setup? Use both. Lightpanda for 90% of your scraping, Chrome for the edge cases.

Quick Start: Try It Yourself

Want to reproduce my tests? Here's how.

Install Lightpanda

npm install @lightpanda/browser puppeteer-core

Create Test Script

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

(async () => {
  const urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    // Add 100 URLs here
  ];

  // Test with Lightpanda
  console.time('Lightpanda');
  const proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });
  const browser = await puppeteer.connect({
    browserWSEndpoint: 'ws://127.0.0.1:9222'
  });
  const context = await browser.createBrowserContext();
  const page = await context.newPage();

  for (const url of urls) {
    await page.goto(url);
    const title = await page.title();
  }

  await page.close();
  await context.close();
  await browser.disconnect();
  proc.kill();
  console.timeEnd('Lightpanda');
})();

Run it. Time it. Compare it to Chrome. See the difference yourself.

Summary

The test: 10,000 pages scraped with Chrome and Lightpanda

Results:

Lightpanda: 4.4x faster execution
Lightpanda: 9.6x less memory
Lightpanda: 13.7x faster startup
Lightpanda: More stable for long runs

Compatibility: 92% of websites work perfectly

When to use Chrome: Screenshots, PDFs, complex debugging, the 8% of sites that don't work with Lightpanda

Bottom line: Lightpanda is the better choice for most web scraping tasks. Faster, cheaper, more stable. The numbers don't lie.

Try it yourself. Run the tests. See if your results match these. They probably will.

Happy scraping!

Resources:

GitHub: https://github.com/lightpanda-io/browser
Documentation: https://lightpanda.io/docs
Scrapy Github: https://github.com/IkramKhanNiazi/The-Scrapy-Handbook
Cloud: https://lightpanda.io (managed service)

Lightpanda: The Beginner's Guide to the Fastest Headless Browser

Muhammad Ikramullah Khan — Fri, 06 Feb 2026 15:25:32 +0000

You're scraping a few hundred product pages every hour. Your server keeps dying.

Chrome eats more and more memory. 200MB per browser instance. You spin up 10 instances and watch your RAM disappear. The server slows down. Then crashes. You restart everything and the cycle repeats.

Then you check your AWS bill. $300 per month just to keep Chrome alive. For what? Scraping a few hundred pages?

You try everything. Lower concurrency. Kill instances faster. More RAM. Nothing solves the core problem. Chrome was built for humans browsing the web, not machines scraping at scale.

That's when you discover Lightpanda. A browser built from scratch for automation. Not Chrome with the UI removed. An actual ground-up rebuild designed for machines.

24MB of memory instead of 207MB. 2 seconds to scrape 100 pages instead of 25 seconds. Your AWS bill drops to $80.

Let me show you how to use it.

What is Lightpanda?

Lightpanda is a headless browser written in Zig (a systems programming language) from the ground up. It's not based on Chrome, Firefox, or any existing browser. It was built specifically for web automation, scraping, and AI agents.

The key differences:

Chrome headless is a full desktop browser with the display turned off. All the code for rendering graphics, managing tabs, handling extensions, and displaying a UI is still there. You're paying (in memory and CPU) for features you never use.

Lightpanda only builds what automation needs. DOM tree, JavaScript execution, network requests. Nothing else. No rendering engine. No UI code. Just the essentials.

The results:

11x faster than Chrome (real benchmark on AWS)
9x less memory (24MB vs 207MB)
Instant startup (no 3-second wait)
Compatible with Puppeteer and Playwright (drop-in replacement)

When to use Lightpanda:

Web scraping at any scale
Testing websites with JavaScript
Building AI agents that browse the web
Any automation that doesn't need visual rendering

When to still use Chrome:

Taking screenshots
Generating PDFs
Testing actual visual rendering
Complex debugging with DevTools

Installing Lightpanda

Let's get Lightpanda running on your machine. I'll show you the easiest way for each platform.

Option 1: Using npm (Recommended for Beginners)

This is the simplest way. The npm package handles everything for you.

Requirements:

Node.js installed (version 14 or higher)
That's it!

Step 1: Create a new project

mkdir lightpanda-test
cd lightpanda-test
npm init -y

Step 2: Install Lightpanda

npm install @lightpanda/browser

The package automatically downloads the right binary for your operating system. No manual setup needed.

Step 3: Test it works

Create a file called test.js:

import { lightpanda } from '@lightpanda/browser';

const options = {
  host: '127.0.0.1',
  port: 9222,
};

(async () => {
  // Start Lightpanda
  const proc = await lightpanda.serve(options);

  console.log('Lightpanda is running!');
  console.log('Process ID:', proc.pid);

  // Stop Lightpanda
  proc.stdout.destroy();
  proc.stderr.destroy();
  proc.kill();

  console.log('Lightpanda stopped');
})();

Step 4: Run it

node test.js

You should see:

🐼 Running Lightpanda's CDP server...
{ pid: 12345 }
Lightpanda is running!
Process ID: 12345
Lightpanda stopped

If you see that, it works! Lightpanda starts and stops almost instantly.

Option 2: Download Binary Directly

If you don't want to use Node.js, you can download the binary.

For Linux (x86_64):

curl -L -o lightpanda https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-x86_64-linux
chmod +x ./lightpanda
./lightpanda -h

For Mac (Apple Silicon):

curl -L -o lightpanda https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-aarch64-macos
chmod +x ./lightpanda
./lightpanda -h

Note: There's no Intel Mac build yet. Intel Mac users should use Docker or the npm package.

For Windows:

Windows users need to use WSL2 (Windows Subsystem for Linux). Once WSL2 is installed, follow the Linux instructions above from inside your WSL terminal.

Move to PATH (optional):

To run lightpanda from anywhere:

sudo mv lightpanda /usr/local/bin/

Now lightpanda works from any directory.

Option 3: Docker (Works Everywhere)

Docker works on all platforms.

docker run -d --name lightpanda -p 9222:9222 lightpanda/browser:nightly

This starts Lightpanda in a container and exposes port 9222 for connections.

Your First Lightpanda Script

Let's scrape Wikipedia and extract all the links. This will show you how Lightpanda works with Puppeteer.

Step 1: Install Puppeteer

We need puppeteer-core (not regular puppeteer). The regular version downloads Chrome, which we don't want.

npm install puppeteer-core

Step 2: Create the Script

Create scrape.js:

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

const lpdOptions = {
  host: '127.0.0.1',
  port: 9222,
};

const puppeteerOptions = {
  browserWSEndpoint: `ws://${lpdOptions.host}:${lpdOptions.port}`,
};

(async () => {
  // Start Lightpanda
  console.log('Starting Lightpanda...');
  const proc = await lightpanda.serve(lpdOptions);

  // Connect Puppeteer to Lightpanda
  console.log('Connecting Puppeteer...');
  const browser = await puppeteer.connect(puppeteerOptions);
  const context = await browser.createBrowserContext();
  const page = await context.newPage();

  // Navigate to Wikipedia
  console.log('Loading Wikipedia...');
  await page.goto('https://en.wikipedia.org/wiki/Web_scraping');

  // Extract all links from the references section
  const links = await page.evaluate(() => {
    return Array.from(document.querySelectorAll('.reflist a.external'))
      .map(link => link.getAttribute('href'))
      .filter(href => href !== null);
  });

  console.log('Found', links.length, 'reference links:');
  links.forEach(link => console.log(link));

  // Clean up
  await page.close();
  await context.close();
  await browser.disconnect();

  proc.stdout.destroy();
  proc.stderr.destroy();
  proc.kill();

  console.log('Done!');
})();

Step 3: Run It

node scrape.js

Output:

Starting Lightpanda...
🐼 Running Lightpanda's CDP server...
Connecting Puppeteer...
Loading Wikipedia...
Found 47 reference links:
https://example.com/article1
https://example.com/article2
...
Done!

What just happened:

Lightpanda started (instant, no delay)
Puppeteer connected to Lightpanda via WebSocket
Lightpanda loaded the Wikipedia page
JavaScript executed, references loaded
We extracted all external links from references
Everything cleaned up

The entire process took about 2 seconds. With Chrome, this would take 5-7 seconds.

Understanding How Lightpanda Works

Let's break down what's happening.

The CDP Server

When you start Lightpanda, it runs a Chrome DevTools Protocol (CDP) server on port 9222.

const proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });

CDP is the standard protocol that tools like Puppeteer and Playwright use to control browsers. Lightpanda implements this protocol, which is why your existing Puppeteer/Playwright scripts work with minimal changes.

Connecting Puppeteer

Instead of launching a browser, you connect to an existing one:

// Old way (launches Chrome)
const browser = await puppeteer.launch();

// New way (connects to Lightpanda)
const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222'
});

Everything else in your Puppeteer script stays the same. That's the beauty of Lightpanda.

The Browser Lifecycle

1. lightpanda.serve() → Starts the browser process
2. puppeteer.connect() → Connects via WebSocket
3. browser.createBrowserContext() → Isolated session
4. context.newPage() → New page/tab
5. page.goto() → Load website
6. page.evaluate() → Run JavaScript, extract data
7. page.close() → Close page
8. context.close() → Close context
9. browser.disconnect() → Disconnect Puppeteer
10. proc.kill() → Stop Lightpanda

Each step is fast. No waiting. No delays.

Practical Example: Scraping Product Prices

Let's scrape a real e-commerce demo site.

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

const lpdOptions = {
  host: '127.0.0.1',
  port: 9222,
};

(async () => {
  // Start Lightpanda
  const proc = await lightpanda.serve(lpdOptions);

  // Connect Puppeteer
  const browser = await puppeteer.connect({
    browserWSEndpoint: `ws://${lpdOptions.host}:${lpdOptions.port}`,
  });

  const context = await browser.createBrowserContext();
  const page = await context.newPage();

  // Go to demo e-commerce site
  await page.goto('https://demo-browser.lightpanda.io/campfire-commerce/');

  // Wait for products to load (JavaScript)
  await page.waitForSelector('.product');

  // Extract product data
  const products = await page.evaluate(() => {
    return Array.from(document.querySelectorAll('.product')).map(product => {
      return {
        name: product.querySelector('.product-name')?.textContent?.trim(),
        price: product.querySelector('.product-price')?.textContent?.trim(),
        rating: product.querySelector('.product-rating')?.textContent?.trim(),
      };
    });
  });

  console.log('Products found:', products.length);
  products.forEach(product => {
    console.log(`${product.name}: ${product.price} (${product.rating} stars)`);
  });

  // Cleanup
  await page.close();
  await context.close();
  await browser.disconnect();
  proc.kill();
})();

Output:

Products found: 12
Camping Tent Pro: $199.99 (4.5 stars)
Sleeping Bag Ultra: $79.99 (4.8 stars)
Portable Stove: $45.99 (4.2 stars)
...

This example shows:

Loading a JavaScript-heavy page
Waiting for elements to appear
Extracting structured data
Handling multiple elements

All the things you'd do with Chrome, but faster and using less memory.

Using Lightpanda with Playwright

Playwright works too. Here's the same Wikipedia scraper using Playwright:

import { lightpanda } from '@lightpanda/browser';
import { chromium } from 'playwright-core';

const lpdOptions = {
  host: '127.0.0.1',
  port: 9222,
};

(async () => {
  // Start Lightpanda
  const proc = await lightpanda.serve(lpdOptions);

  // Connect Playwright (using connectOverCDP)
  const browser = await chromium.connectOverCDP(
    `ws://${lpdOptions.host}:${lpdOptions.port}`
  );

  const context = await browser.newContext();
  const page = await context.newPage();

  // Navigate and extract
  await page.goto('https://en.wikipedia.org/wiki/Web_scraping');

  const links = await page.evaluate(() => {
    return Array.from(document.querySelectorAll('.reflist a.external'))
      .map(link => link.href);
  });

  console.log('Found', links.length, 'links');

  // Cleanup
  await page.close();
  await context.close();
  await browser.close();
  proc.kill();
})();

Key differences from Puppeteer:

Import playwright-core instead of puppeteer-core
Use chromium.connectOverCDP() instead of puppeteer.connect()
Use browser.newContext() instead of browser.createBrowserContext()

Everything else works the same way.

Common Patterns

Pattern 1: Reusable Connection

Don't start/stop Lightpanda for every page. Keep it running:

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

class LightpandaBrowser {
  constructor() {
    this.proc = null;
    this.browser = null;
  }

  async start() {
    this.proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });
    this.browser = await puppeteer.connect({
      browserWSEndpoint: 'ws://127.0.0.1:9222',
    });
  }

  async scrape(url) {
    const context = await this.browser.createBrowserContext();
    const page = await context.newPage();

    await page.goto(url);
    const title = await page.title();

    await page.close();
    await context.close();

    return title;
  }

  async stop() {
    if (this.browser) await this.browser.disconnect();
    if (this.proc) this.proc.kill();
  }
}

// Usage
const lpd = new LightpandaBrowser();
await lpd.start();

const title1 = await lpd.scrape('https://example.com');
const title2 = await lpd.scrape('https://another-site.com');
const title3 = await lpd.scrape('https://third-site.com');

await lpd.stop();

Start once, scrape multiple pages, stop once. Much more efficient.

Pattern 2: Error Handling

Always wrap in try/catch and ensure cleanup:

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

(async () => {
  let proc, browser, context, page;

  try {
    proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });
    browser = await puppeteer.connect({
      browserWSEndpoint: 'ws://127.0.0.1:9222',
    });
    context = await browser.createBrowserContext();
    page = await context.newPage();

    await page.goto('https://example.com');
    const title = await page.title();
    console.log('Title:', title);

  } catch (error) {
    console.error('Error:', error.message);
  } finally {
    // Always cleanup
    if (page) await page.close();
    if (context) await context.close();
    if (browser) await browser.disconnect();
    if (proc) proc.kill();
  }
})();

This ensures Lightpanda stops even if something fails.

Pattern 3: Waiting for Content

JavaScript sites need time to load:

// Wait for selector
await page.waitForSelector('.products', { timeout: 5000 });

// Wait for network to be idle
await page.goto('https://example.com', { waitUntil: 'networkidle0' });

// Wait for specific time
await new Promise(resolve => setTimeout(resolve, 2000));

Lightpanda executes JavaScript, but you still need to wait for async content.

Lightpanda vs Chrome: Side-by-Side

Here are real numbers from actual testing.

Test setup:

AWS EC2 m5.large instance
Scraping 100 pages from a local test site
Using Puppeteer
Both Chrome and Lightpanda running same script

Results:

Metric	Chrome Headless	Lightpanda	Improvement
Execution time	25.2 seconds	2.3 seconds	11x faster
Memory peak	207 MB	24 MB	9x less
Startup time	3-4 seconds	0.1 seconds	30x faster
Cold start (first page)	4.5 seconds	0.4 seconds	11x faster

What this means in practice:

If you're scraping 10,000 pages per day:

Chrome: ~70 hours of execution time
Lightpanda: ~6.4 hours of execution time

If you're running 10 concurrent browser instances:

Chrome: ~2GB RAM minimum
Lightpanda: ~240MB RAM

If you're on AWS with 4GB RAM:

Chrome: Max 8-10 instances
Lightpanda: Max 80-100 instances

Cost impact:

Real AWS costs before and after:

Before (Chrome): $300/month for t3.xlarge (4 vCPU, 16GB RAM)
After (Lightpanda): $80/month for t3.large (2 vCPU, 8GB RAM)

Savings: $220/month, or $2,640 per year.

When Lightpanda Might Not Work

Let's be honest about limitations.

1. Not All Websites Work (Yet)

Lightpanda implements most Web APIs, but not all. Some complex sites might not work.

What usually works:

Simple HTML + CSS sites
React/Vue/Angular single-page apps
Most e-commerce sites
News sites
Documentation sites

What might not work:

Very complex web apps (Gmail, Google Docs)
Sites using obscure Web APIs
Sites with extremely heavy JavaScript

If a site doesn't work, try Chrome as a fallback.

2. No Visual Rendering

You can't take screenshots or generate PDFs.

// This won't work with Lightpanda
await page.screenshot({ path: 'screenshot.png' });
await page.pdf({ path: 'page.pdf' });

For these, stick with Chrome.

3. Limited Debugging Tools

Chrome DevTools is incredibly powerful. Lightpanda doesn't have the same debugging experience (yet).

For complex debugging, use Chrome. For production scraping, use Lightpanda.

4. Playwright Compatibility Caveat

Playwright might choose different code paths as Lightpanda adds new features. A script that works today might need updates when Lightpanda adds new Web APIs.

This is rare, but worth knowing.

Migrating from Chrome to Lightpanda

Already have Puppeteer scripts? Migration is simple.

Before (Chrome):

import puppeteer from 'puppeteer';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com');
  const title = await page.title();
  console.log(title);

  await browser.close();
})();

After (Lightpanda):

import { lightpanda } from '@lightpanda/browser';
import puppeteer from 'puppeteer-core';

(async () => {
  const proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });

  const browser = await puppeteer.connect({
    browserWSEndpoint: 'ws://127.0.0.1:9222',
  });
  const context = await browser.createBrowserContext();
  const page = await context.newPage();

  await page.goto('https://example.com');
  const title = await page.title();
  console.log(title);

  await page.close();
  await context.close();
  await browser.disconnect();
  proc.kill();
})();

What changed:

Added Lightpanda import
Changed puppeteer to puppeteer-core
Changed puppeteer.launch() to puppeteer.connect()
Added lightpanda.serve() and proc.kill()
Added createBrowserContext() layer

The page interaction code stays identical. All your selectors, page.goto(), page.evaluate(), everything works the same.

Using Lightpanda Cloud

Don't want to manage your own infrastructure? Lightpanda offers a cloud service.

import puppeteer from 'puppeteer-core';

(async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint: 'wss://euwest.cloud.lightpanda.io/ws?token=YOUR_TOKEN',
  });

  const context = await browser.createBrowserContext();
  const page = await context.newPage();

  // Your scraping code here
  await page.goto('https://example.com');
  const title = await page.title();
  console.log(title);

  await page.close();
  await context.close();
  await browser.disconnect();
})();

Benefits:

No server management
Scales automatically
Choose between Lightpanda or Chrome
Global regions available

Visit lightpanda.io/cloud to request API access.

Best Practices

1. Keep Lightpanda Running Between Pages

Don't start/stop for every page:

// Bad (slow)
for (const url of urls) {
  const proc = await lightpanda.serve(...);
  // scrape
  proc.kill();
}

// Good (fast)
const proc = await lightpanda.serve(...);
for (const url of urls) {
  // scrape
}
proc.kill();

2. Use Browser Contexts for Isolation

Each context is isolated (separate cookies, storage):

const browser = await puppeteer.connect(...);

// Context 1
const context1 = await browser.createBrowserContext();
const page1 = await context1.newPage();
// Scrape site A

// Context 2 (completely separate)
const context2 = await browser.createBrowserContext();
const page2 = await context2.newPage();
// Scrape site B

// Clean up both
await context1.close();
await context2.close();

3. Add Reasonable Delays

Even though Lightpanda is fast, be polite:

const urls = ['url1', 'url2', 'url3'];

for (const url of urls) {
  await page.goto(url);
  // Extract data

  // Wait 1 second before next page
  await new Promise(resolve => setTimeout(resolve, 1000));
}

4. Handle Errors Gracefully

async function scrapePage(page, url) {
  try {
    await page.goto(url, { timeout: 10000 });
    return await page.title();
  } catch (error) {
    console.error(`Failed to scrape ${url}:`, error.message);
    return null;
  }
}

5. Monitor Memory Usage

Even though Lightpanda uses less memory, still monitor:

console.log('Memory usage:', process.memoryUsage());

Troubleshooting Common Issues

Issue 1: Connection Refused

Error:

Error: connect ECONNREFUSED 127.0.0.1:9222

Solution:
Lightpanda isn't running. Make sure lightpanda.serve() completes before connecting.

// Wait for Lightpanda to start
const proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });
await new Promise(resolve => setTimeout(resolve, 500));

Issue 2: Page Doesn't Load

Error:
Page stays blank or navigation times out.

Solution:
The website might use Web APIs not yet supported. Try with Chrome to confirm:

// Fallback to Chrome
const browser = await puppeteer.launch();

If it works with Chrome but not Lightpanda, open an issue on GitHub.

Issue 3: Selector Returns Nothing

Problem:
Your selectors worked with Chrome but return nothing with Lightpanda.

Solution:
Add wait conditions:

// Wait for element to appear
await page.waitForSelector('.product', { timeout: 5000 });

// Then extract
const products = await page.$$('.product');

Issue 4: Module Import Errors

Error:

Cannot use import statement outside a module

Solution:
Add "type": "module" to your package.json:

{
  "name": "my-scraper",
  "version": "1.0.0",
  "type": "module"
}

What's Next?

You now know how to:

Install Lightpanda
Connect Puppeteer or Playwright
Scrape websites with JavaScript
Handle common patterns
Migrate from Chrome

Next steps:

Try it yourself - Run the examples in this guide
Migrate one script - Pick a simple Puppeteer script, convert it
Measure the difference - Compare execution time and memory
Scale up - Run multiple instances, see how far you can push it

Lightpanda is actively developed. New features and Web API support are added regularly. Star the GitHub repo to follow progress.

Quick Reference

Installation:

npm install @lightpanda/browser puppeteer-core

Start Lightpanda:

import { lightpanda } from '@lightpanda/browser';
const proc = await lightpanda.serve({ host: '127.0.0.1', port: 9222 });

Connect Puppeteer:

import puppeteer from 'puppeteer-core';
const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222',
});

Scrape a page:

const context = await browser.createBrowserContext();
const page = await context.newPage();
await page.goto('https://example.com');
const data = await page.evaluate(() => {
  return document.querySelector('h1').textContent;
});

Cleanup:

await page.close();
await context.close();
await browser.disconnect();
proc.kill();

Summary

Lightpanda is a headless browser built from scratch for automation and AI. It's 11x faster than Chrome and uses 9x less memory.

Key advantages:

Instant startup (no 3-second wait)
Tiny memory footprint (24MB vs 207MB)
Drop-in replacement for Chrome with Puppeteer/Playwright
Perfect for scraping, testing, and AI agents

When to use:

Any web scraping task
Building AI agents
Automated testing without visual rendering
Cost optimization (lower AWS bills)

When to use Chrome instead:

Taking screenshots
Generating PDFs
Complex debugging
Sites using very new Web APIs

Getting started is simple:

npm install @lightpanda/browser puppeteer-core
Replace puppeteer.launch() with puppeteer.connect()
Your existing Puppeteer code works with minimal changes

Try Lightpanda on your next scraping project. The speed and memory savings are real.

Happy scraping!

Resources:

GitHub: https://github.com/lightpanda-io/browser
Documentation: https://lightpanda.io/docs
Cloud: https://lightpanda.io (request API access)
Scrapy Github: https://github.com/IkramKhanNiazi/The-Scrapy-Handbook

The ultimate 2026 Scrapy guide: 45 chapters taking you from beginner spiders to production-grade distributed scraping, anti-bot evasion & deployment. Resources https://github.com/IkramKhanNiazi/The-Scrapy-Handbook

Muhammad Ikramullah Khan — Thu, 05 Feb 2026 17:19:03 +0000

GitHub - IkramKhanNiazi/The-Scrapy-Handbook: The ultimate 2026 Scrapy guide: 45 chapters taking you from beginner spiders to production-grade distributed scraping, anti-bot evasion & deployment.

The ultimate 2026 Scrapy guide: 45 chapters taking you from beginner spiders to production-grade distributed scraping, anti-bot evasion & deployment. - IkramKhanNiazi/The-Scrapy-Handbook

github.com