DEV Community: toothbrush

Linux Performance Troubleshooting: The 60-Second Check and Beyond

toothbrush — Mon, 20 Jul 2026 00:49:47 +0000

When a production server starts misbehaving — latency spikes, dropped requests, OOM kills — you don't have time to read man pages. You need a systematic approach that rules out the most common culprits in under a minute, then drills into the root cause with targeted tools. This is exactly how I debug Linux performance issues on production systems, and it works whether you're on a bare-metal box, a VM, or a container host.

This guide covers the full troubleshooting workflow. Here's what we'll walk through: the 60-second checklist (uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, ss, sar), CPU hot-spot analysis with perf and flamegraphs, memory pressure including OOM and swap, disk I/O latency with iostat and blktrace, network troubleshooting with ss, tc, and tcpdump, and finally a decision tree to choose the right tool for any symptom.

1. The 60-Second Checklist

When you first SSH into a struggling box, run these commands in order. Each takes one look at a different subsystem. By the time you're done, you'll know which resource is the problem.

$ uptime                        # load averages — is the system under pressure?
$ dmesg -T | tail -20           # kernel messages — OOM kills, TCP drops, I/O errors
$ vmstat -SM 1 3                # system-wide CPU, memory, and I/O overview
$ mpstat -P ALL 1 1             # per-CPU balance — one core pegged or evenly spread?
$ pidstat 1 3                   # per-process CPU usage — who's burning cycles?
$ iostat -xz 1 1                # disk I/O — await, svctm, %util
$ free -m                       # memory usage — total, available, cached, swap
$ ss -tupwn                     # socket connections — listening, established, TIME_WAIT
$ sar -n DEV 1 1                # network throughput — packets/sec, errors, drops

I have an alias called perf60 that runs all of these. The point isn't to memorize every flag — it's to have a checklist you always run before diagnosing deeper. Skipping this step is how people spend an hour debugging a memory leak that the OOM killer already logged in dmesg.

Pro tip: Save these commands as a shell script or alias and scp it to every new server you provision. Your future on-call self will thank you.

2. CPU — Finding the Hot Spots

If uptime shows load averages significantly higher than the number of CPUs, and mpstat shows high %user or %system across cores, you have a CPU-bound issue. The next step is figuring out what is burning those cycles.

Per-Process with pidstat

$ pidstat -p ALL -u 1 3
Linux 6.8.0 (web-02)  07/20/2026  _x86_64_  (8 CPU)

02:14:33   UID   PID    %usr  %system  %guest  %CPU   CPU  Command
02:14:34  1001  1234   85.2    2.1     0.0     87.3   3    nginx
02:14:34     0  4567    0.3   12.4     0.0     12.7   7    kworker/u16:1

Here, nginx worker 1234 is using 87% of a single core. That's suspicious — either it's stuck in an infinite loop or it's genuinely serving heavy traffic. The high %system on the kworker suggests something happening at the kernel level — check disk or network interrupts next.

Profiling with perf and Flamegraphs

When you need to know which function is hot, nothing beats perf. It uses hardware performance counters and stack sampling with negligible overhead — safe for production:

# Record CPU samples for 30 seconds
$ perf record -F 99 -a -g -- sleep 30

# Generate the report
$ perf report --stdio --sort comm,dso,symbol

The -F 99 flag samples at 99 Hz (below 100 to avoid aliasing with common timer frequencies). The -g flag captures call graphs. Pipe the output into Brendan Gregg's flamegraph scripts for a visual:

$ perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > cpu-flame.svg

A flamegraph immediately shows you the hot code paths. Wide bars at the top are functions spending the most CPU time. If you see __do_fault or page_fault near the top, your application is page-faulting heavily — not a CPU problem at all, but a memory one.

3. Memory — OOM, Swap, and Page Cache

Memory issues show up in three flavors: OOM kills (the kernel murders a process), swap thrashing (the system is paging like crazy), and page cache pressure (the filesystem cache is starving applications).

Check OOM Kills First

$ dmesg -T | grep -i "oom&#124;killed"
[Jul 20 01:23:45] mysqld invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE)
[Jul 20 01:23:45] mysqld: cpuset=/ mems_allowed=0
[Jul 20 01:23:45] Tasks state (memory reserved: 0K, swap_total: 0K)
[Jul 20 01:23:45] [ 1234] mysqld 1001 12345678 23456789  12345      0             0 mysqld
[Jul 20 01:23:45] Out of memory: Killed process 1234 (mysqld) total-vm:12345678kB

The OOM killer logs every candidate process and the final victim. If you see your application here, the fix is not "add more RAM" (well, sometimes it is) — it's first checking for a memory leak.

Read the Numbers Correctly

The free -m output can be misleading. The available field is what matters:

$ free -m
               total  used  free  shared  buff/cache  available
Mem:           15950  8940  1020     420       5990       6010
Swap:           2048   120  1928

Linux uses free RAM for disk cache aggressively. The available field (6010 MB) estimates how much memory is truly free for new allocations — it includes reclaimable cache. If available is below 10% of total, you're in danger territory.

vmstat -SM 1 shows swap activity. If si (swap in) and so (swap out) are consistently non-zero, the system is paging — and performance will be terrible:

$ vmstat -SM 1
procs  -----------memory----------  ---swap--  -----io----  --system--  ------cpu-----
 r  b   swpd   free   buff   cache   si   so   bi    bo    in    cs  us  sy  id  wa  st
 2  1   120   1020    420   5990   50   45  1200   80   1500  2000  12   8  60  20   0

The si/so of 50/45 MB/s means 95 MB/s of swap traffic. That's disk-bound paging. Every page fault now waits for disk I/O. The wa column (20%) confirms it — the CPU is waiting on I/O.

Finding Memory Leaks

Use pidstat -r 1 to watch per-process RSS growth over time:

$ pidstat -r 1 10
02:20:00   UID   PID  minflt/s  majflt/s  VSZ    RSS    %MEM  Command
02:20:01  1001  1234  12345      0       2.5G   1.8G   11.5  java
02:20:02  1001  1234  12350      0       2.5G   1.8G   11.5  java
02:20:03  1001  1234  14420      0       2.7G   2.0G   12.6  java   ← growing

If RSS keeps climbing without stabilizing, you have a leak. For Java, run jmap -histo:live to dump heap. For other apps, /proc/PID/smaps gives the full picture of memory mappings.

4. Disk I/O — Finding the Slow Storage

High wa in vmstat or iowait in mpstat points to disk contention. The iostat -xz 1 command reveals which device and which operation is slow:

$ iostat -xz 1
Device    r/s    w/s    rkB/s    wkB/s  await  svctm  %util
sda      500     0    81920       0     24.0   0.40   20.0
sdb        0   120        0    30720    180.0  1.20   14.4

Key fields to interpret:

await (ms) — average time for an I/O request to complete. Modern SSDs should be under 2 ms. Above 10 ms means something is wrong.
svctm (ms) — the actual service time for a single I/O. If await >> svctm, the requests are queuing (too much concurrency for the device).
%util — percentage of time the device was busy. Important caveat: modern NVMe SSDs can parallelize I/O deeply, so %util can hit 100% even though the device isn't saturated. Use iostat -x 1 -m and watch aqu-sz (average queue size) instead.

Here sdb has an await of 180 ms — terrible. It's a write-heavy workload on what is likely a slow HDD or a network filesystem. The queue is deep: svctm is 1.2 ms while await is 180 ms, meaning requests are spending 99% of their time waiting in line.

Trace Specific I/O with blktrace

When you need to know which process is thrashing the disk:

# Trace sdb for 5 seconds
$ blktrace -d /dev/sdb -o - | blkparse -i - -o /tmp/blk-trace.txt
$ grep -E " D  | A  " /tmp/blk-trace.txt | awk '{print $12}' | sort | uniq -c | sort -rn

This shows you which process names are doing the most I/O. Combine with iotop -oP for a live view.

5. Network — Connections, Drops, and Latency

Network issues masquerade as application slowness all the time. Before blaming the code, rule out the wire.

Connection State with ss

$ ss -s
Total: 1234 (kernel 1567)
TCP:   567 (estab 234, closed 300, orphaned 3, synrecv 0, timewait 156/0), ports 45

Transport Total     IP        IPv6
*         1567      -         -
RAW       0         0         0
UDP       12        10        2
TCP       567       523       44
INET      579       533       46
FRAG      0         0         0

156 TIME_WAIT connections isn't a problem by itself. But if ss -t state time-wait shows thousands, you're exhausting ephemeral ports — typically caused by HTTP/1.0 short connections or a misconfigured proxy pool.

# Check listen backlog drops
$ ss -ltp | grep -E "^(LISTEN|State)"
$ netstat -s | grep -i "listen.*overflowed&#124;LISTEN.*overflows"
    1234 times the listen queue of a socket overflowed

A listen backlog overflow means the application isn't accepting connections fast enough. Increase net.core.somaxconn and the application's backlog parameter. But this is a band-aid — the real fix is making your app accept connections faster (thread pool sizing, event loop tuning).

Packet Drops and Retransmits

$ netstat -s | egrep "segments retransmited|packets pruned|TCPLoss"
    8923 segments retransmited
    156 packets pruned from receive queue
    78 TCP sockets finished time wait in fast timer

Retransmits above 0.1% of total segments mean packet loss on the network. Use tcpdump -i eth0 host $UPSTREAM_HOST to capture and look for duplicate ACKs. It's often a saturated upstream switch port or a duplex mismatch.

Bandwidth and Queue Drops

$ sar -n DEV 1 1 | grep eth0
02:30:01     eth0    65000.00    8000.00    78901.00     6789.00     0.00     0.00     0.00

$ tc -s qdisc show dev eth0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 123456789 bytes 98765 pkt (dropped 456, overlimits 0)
 backlog 0b 0p requeues 0

The dropped 456 in tc output means the transmit queue is overflowing. Your application is sending data faster than the NIC can transmit it. Either increase the txqueuelen (ip link set eth0 txqueuelen 10000) or reduce the application burst rate.

6. Decision Tree — Which Tool for Which Symptom

Here's a quick reference to jump directly to the right tool based on what you observe:

Symptom	First Command	Deep Dive
Load average high, CPU idle low	mpstat -P ALL 1	perf top / flamegraphs
Load high, CPU idle also high	vmstat 1 (check b column)	iostat -xz 1, blktrace
Application slow, no obvious resource issue	ss -tupn (check TIME_WAIT)	tcpdump, strace -c -p PID
Server unreachable or timeout	dmesg -T	tail
OOM kills in logs	free -m, pidstat -r 1	pmap -x PID, /proc/PID/smaps
Disk latency high (wa > 10%)	iostat -xz 1	iotop -oP, blktrace
Network retransmits > 0.1%	sar -n ETCP 1	tcpdump -w /tmp/dump.pcap

Putting It All Together

Effective Linux performance troubleshooting is a skill you build by doing — but always start with the 60-second checklist. It forces you to check CPU, memory, disk, and network before jumping to conclusions. I've lost count of how many "application bugs" turned out to be a server running out of memory or a saturated network link.

The tools here (perf, vmstat, iostat, ss, blktrace, tcpdump) are available on every Linux distribution and have zero dependencies. Learn to read their output fluently, and you'll be the person in the war room who can say "it's the disk queue depth, not the application code" with confidence.

One final rule: instrument before you need it. Set up sar data collection (sysstat package), configure kernel audit logging for OOM events, and deploy a metrics agent that keeps historical data. When the next incident hits — and it will — you'll have the baseline to know whether things are actually worse than normal.

REST API Design Best Practices: A Practical Guide for 2026

toothbrush — Sun, 19 Jul 2026 02:53:51 +0000

Every team builds APIs. Few build ones that survive their second rewrite. After inheriting three different REST APIs in as many years — one with endpoints named /getAllUsers, another that returned { "status": "ok" } for both success and 500 errors — I started keeping a list of the practices that actually distinguish robust APIs from ones that generate PagerDuty alerts at 3 AM.

This guide distills six rules I've validated across production services handling tens of millions of requests. None are theoretical. All come with working code.

1. Resource-Oriented Naming — Not Action-Oriented

The single biggest smell in a REST API is action verbs in URLs:

# Bad — these are RPC, not REST
GET  /api/getUser?id=42
POST /api/createUser
POST /api/deleteUser/42
POST /api/activateUserSubscription

Resources are nouns, not verbs. The HTTP method is the verb:

# Good
GET    /api/users/42
POST   /api/users
DELETE /api/users/42
POST   /api/users/42/subscriptions   # nested resource
DELETE /api/users/42/subscriptions/active

Key conventions that have held across every production API I've consulted on:

Plural nouns : /users, not /user. Consistency with list endpoints (GET /users = a list) makes singular feel like a bug.
Kebab-case for multi-word resources : /order-items, not /orderItems or /order_items. It's URL-safe and matches what browsers expect.
Nest at most two levels : /users/42/orders/7 is fine. /users/42/orders/7/items/3/addresses/9 is a cry for help. At that point, use a query parameter: /items?order_id=7.
Use query params for filtering, not path segments : /users?status=active&role=admin, not /users/active/admins.

2. Consistent Error Responses — The Contract People Actually Rely On

Most API errors are parsable only by humans staring at a screen. That's a bug. Every error response should follow the same schema so clients can handle them programmatically:

{
  "error": {
    "code": "USER_NOT_FOUND",
    "message": "User with id 42 was not found.",
    "details": {
      "resource": "users",
      "identifier": "42"
    },
    "request_id": "req_a1b2c3d4"
  }
}

The code field is the key — it's a machine-readable string clients can switch on. Never change the meaning of a code after release. The request_id is a trace identifier your logs and your users' bug reports can share, so debugging a production error doesn't start with "which request was that."

Here's a FastAPI middleware that enforces this consistently:

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

class APIException(Exception):
    def __init__(self, code: str, message: str, status: int, details: dict = None):
        self.code = code
        self.message = message
        self.status = status
        self.details = details or {}

@app.exception_handler(APIException)
async def api_error_handler(request: Request, exc: APIException):
    return JSONResponse(
        status_code=exc.status,
        content={
            "error": {
                "code": exc.code,
                "message": exc.message,
                "details": exc.details,
                "request_id": request.headers.get("x-request-id", "unknown"),
            }
        },
    )

With this in place, your HTTP status codes still mean what HTTP says they mean (4xx vs 5xx), but the code field gives clients an actionable discriminator. Stripe pioneered this pattern and it's stood the test of a decade.

3. Pagination That Doesn't Surprise

Cursor-based pagination beats offset-based for production APIs. Here's why: if a new record is inserted before page 2, offset pagination shifts every subsequent page — users see duplicates or miss records. Cursor pagination uses a stable pointer:

GET /users?cursor=eyJpZCI6IDQyfQ==&limit=20

Response:
{
  "data": [ ... ],
  "pagination": {
    "next_cursor": "eyJpZCI6IDg0fQ==",
    "has_more": true,
    "limit": 20
  }
}

The cursor is an opaque base64-encoded token, usually containing the last seen record's ID and a timestamp. Clients don't decode it — just pass it to the next request. Implementation is straightforward with SQL's WHERE id > :cursor ORDER BY id pattern:

SELECT id, name, email FROM users
WHERE id > :cursor
ORDER BY id ASC
LIMIT 21   -- fetch one extra to detect has_more

The extra row tells you has_more = rows_returned == limit + 1. Drop that row from the response and encode its id as the next cursor. This pattern handles millions of rows without the performance cliff that OFFSET hits past page 50.

4. API Versioning — Accept or Header, Not URL Path

The old orthodoxy was /api/v1/users. This commits you to maintaining parallel URL trees forever. A better approach: use the Accept header (content negotiation):

# Request
GET /api/users/42
Accept: application/vnd.devopsdaily.v2+json

# Response
Content-Type: application/vnd.devopsdaily.v2+json

Why this wins: the same URL always works. New clients get v2, old clients get v1, your infrastructure stays simple. When v3 ships, you don't redeploy — you update a default-version config. Stripe and GitHub both moved to this model for good reason.

The practical downside is discoverability — a developer can't glance at the URL and see "this is v2." Mitigate by documenting version defaults in your API reference and returning the version in response headers:

X-API-Version: 2
Deprecated: true
Sunset: Sat, 19 Dec 2026 00:00:00 GMT

If you absolutely must use URL versioning (perhaps your consumers are curl-in-a-shell-script types), stick to the simple /v1/ prefix — but deprecate it in favor of the Accept header on your next major release.

5. Rate Limiting That Tells You When to Retry

Rate limiting without feedback headers is just annoying. Every response — successful or not — should carry:

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 483
X-RateLimit-Reset: 1721414400

And when the client exceeds the limit:

HTTP/1.1 429 Too Many Requests
Retry-After: 42
Content-Type: application/json

{
  "error": {
    "code": "RATE_LIMIT_EXCEEDED",
    "message": "Rate limit exceeded. Retry after 42 seconds.",
    "details": {
      "limit": 1000,
      "reset_at": "2026-07-19T14:00:00Z"
    }
  }
}

The Retry-After header is the most important part — without it, clients have to guess, and they'll either hammer you (making things worse) or back off too conservatively (starving your own metrics). If you're using a token bucket algorithm (and you should be), the reset time is simply the bucket's next refill tick.

Here's a minimal implementation using sliding window counters in Python — no external dependency:

import time
from collections import defaultdict

class SlidingWindowRateLimiter:
    def __init__(self, max_requests: int, window_seconds: int = 60):
        self.max_requests = max_requests
        self.window = window_seconds
        self.requests = defaultdict(list)  # key -> [timestamps]

    def check(self, key: str) -> tuple[bool, int, int]:
        now = time.time()
        cutoff = now - self.window
        self.requests[key] = [t for t in self.requests[key] if t > cutoff]

        remaining = self.max_requests - len(self.requests[key])
        if remaining > 0:
            self.requests[key].append(now)
            return True, remaining, int(cutoff + self.window)
        return False, 0, int(cutoff + self.window)

For production, swap this for Redis-backed counters — but the API contract stays identical.

6. Idempotency and Safe Retries

Network failures happen. When a client sends POST /api/payments and gets a timeout, was the payment charged or not? Without idempotency, the safe answer is "I don't know" — which is the wrong answer.

Use idempotency keys. The client generates a unique key (UUID) and sends it on every mutation request:

POST /api/payments
Idempotency-Key: 7c7a3e81-3a1e-4b1e-8e1e-3e1e7c7a3e81
Content-Type: application/json

{
  "amount": 2999,
  "currency": "usd",
  "source": "tok_visa"
}

The server caches the response keyed by (Idempotency-Key, endpoint). If it sees the same key again, it returns the cached response instead of executing the operation twice:

# Fast enough for most use cases — a simple dict
# For production: Redis with TTL matching your max retry window
idempotency_store: dict[str, dict] = {}

@app.post("/api/payments")
async def create_payment(request: Request):
    idem_key = request.headers.get("Idempotency-Key")
    if not idem_key:
        raise APIException("IDEMPOTENCY_KEY_REQUIRED", ...)

    cached = idempotency_store.get(idem_key)
    if cached:
        return JSONResponse(content=cached, status_code=200)

    result = await process_payment(request.json())
    idempotency_store[idem_key] = result
    return JSONResponse(content=result, status_code=201)

Stripe's idempotency layer handles billions of dollars. The pattern has been battle-tested. Copy it.

Putting It All Together

A well-designed REST API is a contract. The six practices above — resource naming, structured errors, cursor pagination, header-based versioning, informative rate limits, and idempotency keys — form the foundation of APIs that developers enjoy integrating with.

The next time you're building an endpoint, skip the framework boilerplate and start with these. Your consumers (and your on-call rotation) will thank you.

I Built a QuickBooks Automation Tool in a Weekend

toothbrush — Sun, 19 Jul 2026 02:52:48 +0000

Last Thursday I spent 36 minutes typing invoice data into QuickBooks. Supplier name. Date. Amount. Line items. Category. Attach PDF. Next invoice. Repeat. I'm a developer — I automate things for a living — and here I was, doing data entry like it was 1998.

So I built InvoiceFlow — a tool that reads your invoice emails and enters the data into QuickBooks automatically.

The Problem

5 million QuickBooks Online users. Many are small business owners — restaurants, contractors, freelancers. They don't have accounting departments. They ARE the accounting department. Every month they process 20-50 supplier invoices. At 3-5 minutes each, that's 2-4 hours per month of pure data entry. Enterprise tools exist (Bill.com, Stampli) but start at $45+/month and assume you have an AP team. Small businesses get left with Copy → Paste → Cry.

What InvoiceFlow Does

1. Forward the invoice email — you get a personal InvoiceFlow address. Supplier sends invoice → you forward it.

2. AI parses everything — vendor name, date, amount, line items extracted from any PDF format.

3. Appears in QuickBooks — auto-entered as a Bill. Categorized, dated, PDF attached. Confirmation email sent.

Your time: 3 seconds (hit Forward). Processing: ~30 seconds. Done.

Real Pain Points (from Reddit & Forums)

"I spend 3 hours every Sunday just entering invoices. I hate it." — Restaurant owner, r/smallbusiness

"My bookkeeper charges $300/month and 80% of that is literally typing numbers from PDFs into QuickBooks." — Contractor, r/QuickBooks

Tech Stack

Email ingestion via IMAP — watches for forwarded invoices
AI vision models for PDF parsing — handles any invoice layout
QuickBooks OAuth2 API — creates Bills via REST
Error detection — flags mismatched totals, duplicates before posting

No blockchain. No microservices. No Kubernetes. Just a focused tool that solves one boring problem well.

Pricing

$29/month. Unlimited invoices. Cancel anytime. 14-day free trial — no credit card. Compare: part-time bookkeeper = $200-400/month. Enterprise tools = $45-100+/month with limits.

Want early access?

Join the waitlist — 50% off for life when we launch.

Get Early Access →

VS Code Extensions You're Not Using (But Should Be)

toothbrush — Sun, 19 Jul 2026 02:47:20 +0000

Everyone has Prettier and ESLint. But there are extensions hiding in the marketplace that will genuinely change how you code. Here are the ones that deserve a spot in your setup.

Error Lens — Errors Where They Happen

Instead of hovering over squiggly lines to see what's wrong, Error Lens shows errors, warnings, and hints inline — right at the end of the offending line. You'll spot bugs 3x faster because you don't have to move your eyes.

Better Comments — Comments That Work For You

Prefix comments with special characters and they highlight in different colors:

! — Red for critical notes
? — Blue for questions
TODO — Orange for todos
* — Green for highlights

Your brain processes color faster than text. Use it.

GitLens — Git Superpowers

Blame annotations on every line, file history at a click, and side-by-side diff comparisons without leaving your editor. GitLens adds 20+ features to VS Code's built-in Git — the free version alone is a massive upgrade.

Thunder Client — API Testing Without Postman

Lightweight REST client built directly into VS Code's sidebar. No separate app, no account required. Create collections, environment variables, and test APIs without leaving your editor. Faster than Postman for everyday use.

Project Manager — Switch Contexts Instantly

If you juggle 5+ projects, you need this. Save projects with Cmd+Shift+P and switch between them in one click. Each project remembers its own window state, open files, and terminal sessions.

Bookmarks — Never Lose Your Place Again

Mark specific lines and jump between them with keyboard shortcuts. When you're jumping between 3 different sections of a 2000-line file, bookmarks are a lifesaver. Works across files too.

Turbo Console Log — Logging at Key Press

Select a variable, press Ctrl+Alt+L, and a console.log with the variable name and value is inserted on the next line. Tiny feature, massive time saver when debugging.

Bottom Line

Install all 7. Give each one a day of use. By the end of the week, you'll wonder how you ever coded without them.

5 Git Tricks That Make You Look Like a Senior Dev

toothbrush — Sun, 19 Jul 2026 02:46:00 +0000

Most developers use about 20% of Git's power. These 5 commands are the other 80%. Master them and you'll solve problems in seconds that others spend hours on.

1. Interactive Rebase — Clean Commit History

Never push a messy branch again. Interactive rebase lets you squash, reorder, and edit commits before anyone sees them.

git rebase -i HEAD~4

This opens an editor with your last 4 commits. Change pick to squash to combine commits, or reword to fix a message. Your PR reviewer will thank you.

2. Git Bisect — Find the Bug in O(log n)

A bug was introduced sometime in the last 100 commits. Instead of checking them one by one, let Git binary-search for you:

git bisect start
git bisect bad HEAD        # current state is broken
git bisect good v1.0.0     # last known good state

Git checks out a commit in the middle. Test it, mark it git bisect good or git bisect bad. In 7 steps, you'll find the exact commit among 100. Then git bisect reset to return.

3. Worktrees — Work on Two Branches Simultaneously

Need to switch branches but don't want to stash your changes? Worktrees let you check out multiple branches into separate directories — all from one repo.

git worktree add ../feature-branch feature-branch

Now you have two working directories: your current one and ../feature-branch. Make changes in either, they share the same .git folder. Perfect for hotfixes while deep in a feature.

4. Reflog — Your Undo Safety Net

Accidentally deleted a branch? Force-pushed the wrong thing? git reflog shows every action Git has taken in the last 90 days:

git reflog
# abc1234 HEAD@{0}: commit: fix login bug
# def5678 HEAD@{1}: reset: moving to HEAD~3
# ghi9012 HEAD@{2}: commit: add payment feature

Find the commit you lost and do git checkout ghi9012 to recover it. The reflog has saved more careers than any StackOverflow answer.

5. Partial Staging — Commit Specific Lines

You made 5 changes in one file but only want to commit 2 of them. Don't copy-paste into a new file. Use interactive staging:

git add -p

Git shows each chunk and asks what to do. Press y to stage it, n to skip, or s to split a large chunk into smaller ones. Commit exactly what you mean to commit.

Bottom Line

Start with git add -p and git reflog — you'll use those weekly. Then add rebase -i for your next PR. Bisect and worktrees are your secret weapons for the tough debugging sessions.

Senior developers don't memorize more commands. They know fewer commands that solve problems faster.

10 Docker Commands Every Developer Should Know in 2026

toothbrush — Sun, 19 Jul 2026 02:42:19 +0000

Docker has been around for a decade, yet most developers still use the same 3 commands. Here are 10 commands that will make you faster, fix production issues quicker, and actually understand what's happening inside your containers.

1. docker exec -it — Your Debug Entry Point

The classic. But most people don't use it right. Always specify the shell explicitly:

docker exec -it my-container /bin/bash
# Alpine containers don't have bash:
docker exec -it my-container /bin/sh

Bonus: pipe commands directly without entering the container:

docker exec my-container cat /etc/nginx/nginx.conf

2. docker logs — Read the Room

Don't SSH into a container to read logs. Use the tools built for it:

docker logs --tail 100 -f my-container   # last 100 lines, follow
docker logs --since 10m my-container       # last 10 minutes
docker logs --timestamps my-container      # show when each line happened

3. docker system df — Where Did My Disk Go?

This single command tells you exactly how much space Docker is eating:

docker system df
# TYPE           TOTAL   ACTIVE   SIZE    RECLAIMABLE
# Images         12      5        2.3GB   1.1GB (47%)
# Containers     8       3        150MB   90MB (60%)
# Volumes        4       2        500MB   200MB (40%)

Follow up with docker system prune -a when you need to clean house. The -a flag also removes unused images, not just dangling ones.

4. docker inspect — The Truth Is in the JSON

Every Docker object has a complete JSON manifest. inspect reveals it. Use Go templates to extract exactly what you need:

docker inspect -f '{{.NetworkSettings.IPAddress}}' my-container
docker inspect -f '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' my-container
docker inspect -f '{{.State.StartedAt}}' my-container

5. docker stats — Real-Time Resource Monitor

Like top but for containers. Shows CPU, memory, network, and disk I/O — live:

docker stats --no-stream          # snapshot
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

If a container is using 2GB of RAM but your app only needs 200MB, you'll spot it here.

6. docker history — What's Actually In That Image?

Before pulling a 1.5GB image from Docker Hub, check what's inside:

docker history node:20-alpine
# IMAGE          CREATED       SIZE    COMMENT
# abc123         2 days ago    5.6MB   /bin/sh -c apk add --no-cache ...
# def456         2 days ago    112MB   /bin/sh -c mkdir /app

This shows every layer and its size. You'll often find 500MB of build dependencies that should've been cleaned up in the Dockerfile.

7. docker cp — Move Files In and Out

No need to mount volumes for one-off file transfers:

docker cp ./config.json my-container:/app/config.json     # host → container
docker cp my-container:/var/log/app.log ./app.log          # container → host

8. docker commit — Snapshot a Running Container

Made changes inside a container and want to save them as an image? Don't rebuild — snapshot it:

docker commit my-container my-debug-image:v1

Useful for debugging, but don't build production images this way. It captures everything including temp files and secrets.

9. docker network — See Who's Talking to Whom

docker network ls                          # list all networks
docker network inspect bridge              # see connected containers
docker network create --driver bridge my-net  # custom network

Containers on the same custom network can resolve each other by name. No more hardcoding IPs.

10. docker events — Watch Everything Happen

Real-time stream of every Docker event — container starts, stops, health checks, image pulls:

docker events --since 5m
docker events --filter 'event=die' --filter 'container=my-app'

Pipe this into your monitoring stack. When a container unexpectedly dies at 3 AM, you'll know exactly when.

Bottom Line

You don't need to memorize all 10. Start with inspect and system df — you'll use them weekly. Add logs --since for production debugging and events for monitoring. The rest will become muscle memory over time.

The gap between "I use Docker" and "I understand Docker" is exactly these 10 commands.

7 AI Coding Tools That Actually Work in 2026: An Honest Comparison

toothbrush — Sun, 19 Jul 2026 02:40:05 +0000

AI coding assistants went from novelty to necessity in 18 months. Every developer I know has at least one running. The problem: there are now dozens of them, and they all claim to be the best. This is the comparison I wish I'd had before burning through trial periods and switching tools five times.

I tested seven tools over two weeks — real projects, not toy examples. Here is what actually works.

1. GitHub Copilot — The Default Choice That Earned It

Copilot is the incumbent for a reason. It integrates with VS Code, JetBrains, and Neovim. It autocompletes entire functions. It understands your project context — imports, neighboring files, even your test suite. The June 2026 update added multi-file editing and a chat panel that can refactor across modules.

Best for: Developers who want zero configuration and broad IDE support.

Weakness: No free tier (unlike most competitors). $10/month or $100/year.

2. Cursor — The IDE That Replaced VS Code

Cursor is not a plugin — it is a full IDE, forked from VS Code. That means it controls the entire editing experience. The key feature: Cmd+K inline editing. Highlight code, describe the change in plain English, and Cursor rewrites it. It also has a sidebar agent that can read your entire codebase and make changes across files.

Best for: Developers willing to switch editors for a massive productivity jump.

Weakness: No JetBrains support. The agent sometimes over-engineers simple fixes.

3. Codeium / Windsurf — The Free Powerhouse

Codeium's free tier is genuinely unlimited. Autocomplete, chat, context-aware suggestions — all free for individual developers. Windsurf, their standalone IDE, adds agentic features similar to Cursor. For students and open-source maintainers, this is the obvious starting point.

Best for: Cost-conscious developers and open-source contributors.

Weakness: Paid enterprise features lag behind Copilot's offerings.

4. Cody by Sourcegraph — The Codebase Whisperer

Cody's killer feature: it indexes your entire repository and answers questions about your codebase. Ask "where is authentication logic implemented?" and it finds the exact files — not just grep results, but actual understanding. It also generates unit tests, explains complex functions, and can fix bugs with natural language prompts.

Best for: Large codebases and onboarding new team members.

Weakness: Free tier limits context size. Indexing takes minutes on first run.

5. Amazon CodeWhisperer — The AWS-Native Option

If you live in the AWS ecosystem, CodeWhisperer is the obvious pick. It suggests IAM policies, CloudFormation snippets, and Lambda handlers that follow AWS best practices. It also scans for security vulnerabilities — flagging hardcoded credentials and overly permissive policies inline.

Best for: AWS developers who want infrastructure-as-code suggestions.

Weakness: Outside AWS, suggestions are less accurate than Copilot.

6. Tabnine — The Privacy-First Alternative

Tabnine runs models locally — your code never leaves your machine. For companies with strict compliance requirements (finance, healthcare, government), this is non-negotiable. The local models are smaller than cloud-hosted ones, so suggestions are slightly less creative, but they are fast — sub-100ms completion times.

Best for: Enterprise teams with data privacy requirements.

Weakness: Local model quality is noticeably behind cloud-hosted competitors.

7. Continue — The Open-Source Wildcard

Continue is an open-source VS Code/JetBrains extension that lets you plug in any LLM — OpenAI, Anthropic, local Ollama models, or even your own self-hosted API. You can chain models: use a fast local model for autocomplete and a cloud model for chat. The configuration is a single JSON file.

Best for: Developers who want full control over models and pricing.

Weakness: Requires setup. Not "install and forget" like Copilot.

The Bottom Line

If you want zero friction, GitHub Copilot is still the answer — $10/month, works everywhere, 90% accurate suggestions. If you are willing to switch editors, Cursor gives you the biggest productivity boost per dollar. Codeium is the best free option by a wide margin. And if you manage a team shipping to AWS, CodeWhisperer saves you hours of reading documentation.

The one tool I would avoid : any AI assistant that does not integrate with your existing tools. Switching workflows is the hidden cost nobody talks about.

10 Python Libraries That Will Save You Hours in 2026

toothbrush — Sun, 19 Jul 2026 02:36:36 +0000

Every year, the Python ecosystem churns out hundreds of libraries. Most fade away. A few become indispensable. Here are 10 libraries that will genuinely cut your development time in 2026.

1. httpx — HTTP for Humans, Now With Async

Requests is great. httpx is Requests with async support and HTTP/2 built in. If you're still writing synchronous HTTP calls in 2026, you're leaving performance on the table.

import httpx

async with httpx.AsyncClient() as client:
    r = await client.get('https://api.example.com')
    print(r.json())

Drop-in replacement for Requests, but async-first. The migration is trivial.

2. rich — Terminal Output That Doesn't Suck

Progress bars, syntax-highlighted tracebacks, tables, markdown rendering — all in your terminal. rich replaces a dozen small utility libraries.

from rich.console import Console

from rich.table import Table

console = Console()

table = Table(title="Server Status")

table.add_column("Host")

table.add_column("CPU")

table.add_row("prod-1", "42%")

console.print(table)

pydantic v2 — Fast Data Validation

If you work with APIs, configs, or any structured data, Pydantic v2 is ~30x faster than v1 and now written in Rust under the hood. Define a model, get validation, serialization, and JSON Schema for free.

4. typer — CLI Apps in Minutes

Built on top of Click, typer uses type hints to auto-generate CLI interfaces. Write a function, add type annotations, and you have a production-ready CLI with help text and error handling.

5. polars — The Pandas Killer

Pandas is slow with large datasets. polars is written in Rust, uses Apache Arrow, and runs 10-100x faster. If you're processing more than a few million rows, switch now.

6. textual — TUI Apps Like Never Before

Want to build a terminal dashboard? textual gives you a React-like component model for terminals. Build TUIs with CSS-like styling and async event handling.

7. duckdb — SQLite for Analytics

Run SQL directly on CSV, Parquet, and Pandas DataFrames. Zero setup, zero server. duckdb is faster than Spark for datasets under 10GB.

8. structlog — Structured Logging Done Right

Stop grepping through log files. structlog outputs JSON logs that you can pipe into any observability stack. Attach context to every log message automatically.

9. ruff — Linting at Rust Speed

Replaces Flake8, isort, pyupgrade, and a dozen other linting tools — all in one binary, 100x faster. Written in Rust, integrates with pyproject.toml.

10. litestar — FastAPI Without the Bloat

FastAPI is great, but litestar is leaner, more modular, and community-driven. Better dependency injection, Class-Based Views, and first-class async support.

Bottom Line

You don't need to adopt all 10. Pick 3 that solve your current pain points and integrate them into one project. The time savings compound fast.

GitHub Actions CI/CD Pipeline: From Zero to Production

toothbrush — Sun, 19 Jul 2026 02:34:50 +0000

Most teams start with GitHub Actions because it's already there — your code is on GitHub, so the CI runner is a checkbox away. But a real production pipeline is more than running npm test on every push. It needs caching, Docker builds, multi-stage environments, artifact management, and deployment that doesn't wake you up at 3 AM.

I've built and broken enough pipelines to know what works. Here's a production-grade GitHub Actions workflow — explained piece by piece so you understand why each part exists.

1. The Workflow Skeleton

Every GitHub Actions workflow lives in .github/workflows/. We'll build a single file called ci-cd.yml that handles test, build, and deploy for a Node.js + Docker application.

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

The on.push trigger runs on main-branch commits. The pull_request trigger runs on PRs targeting main — so you catch failures before merge. The env block defines shared variables so you don't repeat yourself in every step.

2. Job 1: Lint & Test (Fast Feedback)

Your first job should finish in under 2 minutes. Speed matters because this is what runs on every PR commit — slow tests kill developer productivity.

jobs:
  lint-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'

      - run: npm ci
      - run: npm run lint
      - run: npm test -- --coverage

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: coverage-report
          path: coverage/

Key details that matter:

npm ci — not npm install. ci uses the lockfile exactly, fails if lockfile ≠ package.json, and is 2-3x faster. Never use npm install in CI.
cache: 'npm' — built into setup-node. Caches ~/.npm based on package-lock.json. Cuts install time from 45s to 8s.
upload-artifact with if: always() — coverage reports are useful even when tests fail. You can browse them in the Actions UI.
Separation of concerns — this job has nothing to do with Docker or deployment. It fails fast and tells the developer "your code is broken" before anything expensive runs.

3. Job 2: Docker Build & Scan

The second job builds the Docker image. Crucially, it waits for lint-test to pass. GitHub Actions uses needs for this.

  docker-build:
    needs: lint-test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Three things here that beginners get wrong:

Two tags — latest lets you always pull the newest. The sha tag lets you deploy a specific version and roll back. Without the SHA tag, you can't tell which commit is running in production.
cache-from/cache-to: type=gha — this is GitHub Actions' magic cache for Docker layers. Without it, every build starts from scratch. With it, only changed layers rebuild. A 3-minute build drops to 40 seconds on the second run.
permissions: packages: write — the default GITHUB_TOKEN can't push packages. You must explicitly grant write access to the GitHub Container Registry.

4. Job 3: Security Scan

Don't deploy vulnerable images. Add a scan step between build and deploy:

  security-scan:
    needs: docker-build
    runs-on: ubuntu-latest
    steps:
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: '${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-results.sarif'
          category: 'trivy'

Trivy scans your image against known CVE databases. If someone added a vulnerable dependency, you'll see it in GitHub's Security tab before it ever hits production. I've caught 4 critical CVEs this way — including an undici HTTP request smuggling vulnerability that wasn't flagged by npm audit.

5. Job 4: Deploy

Deployment is the most environment-specific part. I'll show a generic SSH-based deploy to a VPS, which applies to most small-to-mid-size setups:

  deploy:
    needs: [security-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy via SSH
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.DEPLOY_HOST }}
          username: ${{ secrets.DEPLOY_USER }}
          key: ${{ secrets.DEPLOY_SSH_KEY }}
          script: |
            cd /opt/my-app
            docker compose pull
            docker compose up -d --remove-orphans
            docker system prune -af --filter "until=24h"

This runs docker compose pull on your server (it fetches the updated image you just pushed), recreates containers, and cleans up old images that are more than 24 hours old. The --remove-orphans flag removes containers defined in the compose file that are no longer running — prevents stale container drift.

The deploy job only runs on the main branch (if: github.ref == 'refs/heads/main'). PR builds stop at security-scan. This is deliberate — you want to know the image is safe before you deploy, but the actual deploy only happens when code merges.

6. Environment Protection Rules

Don't deploy to production just because someone pushed to main. Set up environment protection in your repo settings:

      - name: Deploy to Production
        uses: appleboy/ssh-action@v1.0.3
        with:
          # ...
        environment: production

Then in GitHub → Settings → Environments → production, require:

Required reviewers — a senior dev must approve the deploy
Wait timer — 5 minutes. Gives you a window to cancel if the pipeline just ran on the wrong branch
Deployment branches — only main can deploy to production

This single setting has saved me from deploying broken code more times than I'd like to admit. The 5-minute wait is surprisingly effective — it forces you to sit and watch the first few seconds of deployment logs.

7. Caching: The Single Biggest Speed Win

A pipeline without caching is a slow pipeline. Beyond Docker layer caching, use GitHub's generic cache action for everything else:

      - name: Cache node_modules
        uses: actions/cache@v4
        with:
          path: |
            ~/.npm
            node_modules
            .next/cache
          key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-npm-

This caches npm dependencies AND the Next.js (or similar) build cache. Combined with Docker layer caching, a full pipeline that took 8 minutes now runs in under 3. Your team will stop complaining about slow CI.

8. Matrix Builds (Bonus)

If your app supports multiple runtimes, use a matrix strategy to test them all in parallel:

    strategy:
      matrix:
        node-version: [18, 20, 22]

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
      - run: npm ci
      - run: npm test

Three parallel runners, each testing a different Node version. They all run simultaneously and the needs condition for downstream jobs only waits for this job to succeed on all matrix entries. If Node 22 has a regression, you'll know before your users do.

Putting It All Together

The final workflow has four sequential stages:

lint-test → docker-build → security-scan → deploy

Each stage gates the next. If linting fails, nothing builds. If the image has critical CVEs, nothing deploys. If the deploy fails, the previous version keeps running because docker compose up only restarts containers when the image actually changes.

The total file is about 90 lines. Copy it into .github/workflows/ci-cd.yml, replace the deploy script with whatever your server needs, and you have a production CI/CD pipeline that handles caching, security scanning, and safe deployment.

One final thing: test your pipeline on a branch before pushing to main. Every time I skip this, I end up fixing YAML indentation via 20 commits to a broken main. Push your workflow to a feature branch first, open a PR, and watch the pipeline run before you merge.