DEV Community: Supreet Sethi

Why Stockfish is So Good (and How You Could Write a Chess Engine)

Supreet Sethi — Thu, 19 Feb 2026 01:18:54 +0000

Stockfish 18 plays chess at 3759 Elo—far beyond the strongest humans ever lived. But it's open-source, written in about 50,000 lines of C++, and runs on any laptop. This article surveys how modern chess engines work, why NNUE neural networks changed everything, and most importantly: the design decisions you'd make if you wanted to build one yourself.

The Tournament Results Tell the Story

Every engine in the top 10 worldwide is now open-source. The proprietary era ended. On the CCRL (Chess Computer Rating List) 40/15 time control, the 2025 hierarchy looks like this:

Rank	Engine	Elo	Language	Evaluation	Open Source
1	Stockfish 18	3650	C++	NNUE	Yes
2	PlentyChess 7.0	3643	C++	NNUE	Yes
3	Obsidian 16.0	3635	C++	NNUE	Yes
4	Alexandria 8	3633	Rust	NNUE	Yes
5	Viridithas 19	3629	Rust	NNUE	Yes
6	Berserk 13	3615	C	NNUE	Yes
10	Leela Chess Zero	~3443	C++	Deep NN	Yes

Leela Chess Zero is roughly 200 Elo weaker than Stockfish despite using a 191-million-parameter transformer neural network, simply because Stockfish can evaluate 1,500× more positions per second. Speed matters more than brilliance in chess.

NNUE Was The Breakthrough Nobody Expected

In August 2020, Stockfish introduced NNUE (Efficiently Updatable Neural Networks), originally invented for computer shogi. The jump was +80 to +100 Elo overnight—equivalent to two years of conventional development. Every top engine today uses NNUE or a variation of it.

The genius of NNUE is that it's specialized for the constraints of chess engines:

Sparse activation: Only ~30 of 40,960 input features are active per position (99.93% dormant). Most neural network activations are wasted computation in the context of alpha-beta search.
Incremental updates: Since only 2-4 features change per move, you don't recompute the entire network—you update an "accumulator" in place. This makes evaluation sub-microsecond.
Integer arithmetic: NNUE uses int8 and int16 quantization, optimized for CPU SIMD (AVX2, AVX-512). No float32 overhead. Roughly 60 million evaluations per second on a modern CPU.

Stockfish's current NNUE uses a 1024×2→8→32→1 architecture (inputs expand to 1024, then compress through small hidden layers). The network is trained on billions of self-play positions scored at modest search depths, using PyTorch with custom CUDA kernels.

The completely hand-crafted evaluation function—Stockfish's secret sauce for 15 years—was deleted entirely in 2023. Now NNUE does everything.

Two Paradigms Compete: Alpha-Beta vs. Monte Carlo Tree Search

Alpha-Beta Search with NNUE (Stockfish's Approach)

Stockfish searches a tree systematically, pruning branches that cannot improve the best move. It uses Principal Variation Search (PVS), a variant of alpha-beta that searches the expected best move with a full window and all others with a zero-width window.

The trick is aggressive pruning without missing critical moves:

Late Move Reductions (LMR): Moves ordered later are searched at reduced depth. Estimated +100 Elo when first introduced.
Null Move Pruning: If passing (making no move) already produces a cutoff, prune the subtree. Dangerous in zugzwang, but nets +100-200 Elo.
Singular Extensions: If one move dramatically outperforms all others, search it deeper.
Futility Pruning: At shallow depths, positions too far below alpha are pruned immediately.
SEE Pruning: Moves with losing material exchanges are pruned based on static exchange evaluation.

These techniques shrink the effective branching factor from ~35 to less than 2, enabling search to depth 30-40 plies where NNUE provides the final judgment.

MCTS with Deep Neural Networks (Leela Chess Zero)

Leela Chess Zero uses Monte Carlo Tree Search combined with a 191-million-parameter transformer. Instead of exhaustively searching and pruning, it builds a search tree by repeatedly:

Selecting a node via the PUCT formula: Q(s,a) + c_puct × P(s,a) × √(N_parent) / (1 + N(s,a))
Evaluating with the neural network (value head), which replaces classical random rollouts
Backing up results to update statistics

The network provides both a policy head (prior probability for each move) and a value head (win probability), replacing the random playout entirely.

The outcome: MCTS's deep NN produces higher-quality per-position judgments, but Stockfish compensates by evaluating vastly more positions per second. On TCEC (the engine championship), Stockfish beats Leela roughly 57-43 and has won every superfinal since 2020. The gap appears to be widening.

Why C++ (And Sometimes Rust)

Programming language choice directly impacts Elo rating at the elite level. Each additional ply of search adds roughly 50-70 Elo. A 2× difference in nodes-per-second (NPS) means one extra ply—so language choice matters.

Language	Top Engine	Elo	NPS Relative to C++	Elo Cost
C++	Stockfish	3759	Baseline	0
C	Berserk	3646	100%	0
Rust	Alexandria	3633	95-100%	0-5
C#	Ceres	~3624	70-80%	30-60
Java	chess22k	~3000	60-80%	30-60
Python	–	~1800	1-2%	400+

C and C++ dominate because they offer:

Direct hardware access (cache control, SIMD intrinsics)
No garbage collection pauses
Compile-time optimizations
Fast bitboard operations (64-bit integers for board representation)

Rust is a surprise win. Alexandria and Viridithas prove that Rust—with its zero-cost abstractions and LLVM compilation—is fully competitive at the elite level. Move generation, board representation, and evaluation all run at near-C++ speeds.

Managed languages like C# and Java incur 30-60 Elo penalties from runtime overhead and garbage collection. Below ~3200 Elo, this doesn't matter—algorithmic skill outweighs implementation. But above 3600, the language tax is disqualifying.

Open-Source Development Beats Proprietary

Twelve of the top 15 engines are open-source. This wasn't ideological—it was mechanistic.

In 2013, Gary Linscott created Fishtest, a distributed testing framework for Stockfish using Sequential Probability Ratio Testing (SPRT). Volunteers donate CPU time. The numbers:

19,700+ CPU-years consumed
9.8+ billion games played
~199 concurrent worker machines
For Stockfish 15: ~13,000 candidate changes tested, ~200 merged

Stockfish gained +120 Elo in the first year of Fishtest—a pace no proprietary team could match. Andrew Grant's OpenBench extended this framework to 17+ other engines (Berserk, Koivisto, Stormphrax, Viridithas, etc.).

The reason is simple: you cannot optimize an engine in isolation. Testing every patch requires millions of games. Distributed testing at cloud scale is the only way to move fast. Open-source enabled this. Proprietary teams struggle to match it.

Other Key Technical Insights

Lazy SMP (Shared Memory Parallelization): Going from 1 to 16 threads adds only ~20-32 Elo, far less than the theoretical benefit. Engines are memory-bound, not CPU-bound, at high core counts.

Syzygy Tablebases: 6-man endgame databases provide ~10 Elo improvement by allowing perfect play in simplified positions. Stockfish ships with 6-man tablebases.

Depth vs. NNUE Quality: Modern NNUE networks are compact (several MB), but the old hand-crafted evaluations were more "reasonable" locally. Deeper search compensates: Stockfish searches to 40+ plies where NNUE makes the final judgment.

Each Additional Ply: Worth approximately +50-70 Elo across all engines. Doubling NPS ≈ one extra ply ≈ 50-70 Elo.

If You Want To Build a Chess Engine: Design Sweet Spots

Most programmers who build chess engines do it for fun or learning. Here's a realistic roadmap with reasonable design trade-offs.

Tier 1: A Respectable Hobbyist Engine (1400-1800 Elo)

Goal: Beat humans, work on a laptop, finish in a weekend.

Design choices:

Language: Python or Rust (you care about learning, not top-10 rankings)
Search: Basic alpha-beta with minimal pruning (null move pruning is the only essential trick)
Evaluation: Hand-crafted. A simple function counting material + positional bonuses for piece centralization, king safety, pawn advancement
Board: 8×8 2D array (don't use bitboards yet; they're an optimization)
Move generation: Brute-force checking all 64 squares

Expected Elo: 1400-1800 (beats most casual players)

Time to implement: ~40-80 lines of core move generation + alpha-beta + evaluation = 200-300 lines total

# Pseudocode for minimum viable chess engine

def evaluate(board):
    material = count_material(board)
    position = count_piece_activity(board)
    return material + position

def search(board, depth, alpha, beta):
    if depth == 0:
        return evaluate(board)

    for move in generate_moves(board):
        board.make_move(move)
        score = -search(board, depth - 1, -beta, -alpha)
        board.unmake_move(move)

        alpha = max(alpha, score)
        if alpha >= beta:
            break  # Beta cutoff

    return alpha

This gives you 1600+ Elo with virtually no optimization.

Tier 2: A Competitive Amateur Engine (2000-2500 Elo)

Goal: Beat most online opponents, viable on older hardware.

Design choices:

Language: C++ or Rust (you need speed now, not just learning)
Search: Alpha-beta with:
- Null move pruning
- Late Move Reductions (LMR) with basic formula: reduction = 1 + ln(depth) × ln(moveNumber)
- Move ordering (best moves first)
- Transposition table (hash table caching positions)
- Quiescence search (don't stop at arbitrary depth; handle captures)
Evaluation: Still hand-crafted, but richer:
- Material (pawns=1, knights=3, bishops=3.25, rooks=5, queens=9)
- Piece-square tables (centralized pieces score higher)
- Pawn structure (doubled, isolated, passed pawns)
- King safety
- Mobility (count available squares for each piece)
Board: Bitboards (64-bit integers for each piece type)
Move generation: Bit-scanning optimized

Expected Elo: 2000-2500 (beats most club players)

Time to implement: 1,000-2,000 lines of well-structured C++

Key optimization: Transposition table. Storing and retrieving previously evaluated positions cuts the search tree in half.

struct TTEntry {
    uint64_t hash;
    int depth, score;
    int flag;  // EXACT, LOWER, UPPER
};

std::unordered_map<uint64_t, TTEntry> transposition_table;

int search(Board &board, int depth, int alpha, int beta) {
    uint64_t hash = board.hash();
    if (transposition_table.count(hash)) {
        auto &entry = transposition_table[hash];
        if (entry.depth >= depth) {
            if (entry.flag == EXACT) return entry.score;
            if (entry.flag == LOWER) alpha = max(alpha, entry.score);
            if (entry.flag == UPPER) beta = min(beta, entry.score);
            if (alpha >= beta) return entry.score;
        }
    }

    // ... search continues ...
}

Tier 3a: A Strong Club Engine (~3000 lines, 2400-2600 Elo)

Goal: Impressive hobbyist engine; beats most players, playable against titled opponents.

What you add to Tier 2:

Iterative deepening: Search depth 1, then 2, then 3, etc. Allows precise time management and search stability
Killer heuristic: Track moves that produced cutoffs in sibling nodes; try them early in similar positions
History heuristic: Moves that produced cutoffs anywhere in the tree are tried earlier at shallower depths
Aspiration windows: Search the full depth with a narrow alpha-beta window around the previous iteration's score (much faster when you're right)
Staged move ordering: Generate checking moves first, then captures, then quiet moves
Better evaluation:
- Piece-square tables for all pieces (not just pawns)
- Pawn structure analysis (passed pawns, weak squares)
- King tropism (pieces attacking near the opponent's king)
- Tempo bonus (slight evaluation advantage for the side to move)

Board representation: Still bitboards, but optimized move generation using bit-scanning

Expected Elo: 2400-2600 (beats most club-level players, competitive against titled opponents)

Time to implement: ~3,000 lines of well-optimized C++

Code example (iterative deepening structure):

int iterative_deepening(Board &board, int time_ms) {
    int best_move = 0;
    int best_score = 0;

    auto start = std::chrono::system_clock::now();

    for (int depth = 1; depth <= 30; depth++) {
        int score = search(board, depth, -INFINITY, INFINITY);

        if (score_is_mate(score)) {
            // If we found mate, no point searching deeper
            return best_move;
        }

        best_move = pv[0];  // Principal variation from this depth
        best_score = score;

        auto elapsed = std::chrono::system_clock::now() - start;
        if (elapsed.count() > time_ms * 0.9) {
            // Stop if we've used 90% of allocated time
            break;
        }
    }

    return best_move;
}

Tier 3b: An Intermediate Engine (~5000 lines, 2800-3200 Elo)

Goal: Seriously competitive; tournaments, online rankings, formidable opponent.

What you add to Tier 3a:

Singular extensions: If one move is dramatically better than siblings, search it one ply deeper
Futility pruning: At shallow depths (1-2 plies from qsearch), prune moves that can't possibly improve alpha
SEE (Static Exchange Evaluation): Use a fast algorithm to estimate if a capture wins or loses material
Countermove heuristic: Track which move was played at the parent node; moves that counter it get priority
Quiescence search improvements:
- Only examine captures that win material or give check
- Stop immediately if a "quiet" move (non-capture) is better than alpha
Transposition table refinements:
- Increased size (16 MB → 256 MB)
- Better replacement strategy (always replace, or replace only weaker entries)
- Age tracking (overwrite older entries)
Advanced evaluation (still hand-crafted):
- Separate evaluation for opening, middlegame, endgame (phase calculation)
- Rook on 7th rank bonus
- Opposite-colored bishop endgames (draw tendency)
- Pawn promotion threats
- Mobility counts refined by piece type

Search enhancements:

Multi-threaded search (Lazy SMP): 2-4 threads with shared transposition table (adds ~20 Elo but improves responsiveness)
Mate distance pruning: If we've found mate in N moves, prune branches that can't improve it
Null move window refinements: Adapt null move reduction based on position criticality (zugzwang detection)

Board representation: Bitboards with magic multiplication for bishop/rook attacks (O(1) attack generation)

Expected Elo: 2800-3200 (serious tournament contender; dominates online)

Time to implement: ~5,000 lines of optimized C++

Critical file structure at this level:

src/
  board.cpp/h        (1,000 lines: bitboards, move gen, fen parsing)
  search.cpp/h       (1,500 lines: alpha-beta, pruning, extensions, TT)
  evaluate.cpp/h     (1,200 lines: handcrafted evaluation)
  movegen.cpp/h      (500 lines: move ordering, sorting)
  uci.cpp/h          (300 lines: UCI protocol interface)
  main.cpp           (100 lines: main loop)

Key decision at 5000 lines: You've optimized hand-crafted evaluation as far as it goes. To go further, you either:

Fork an open-source NNUE (adds ~1000 lines for integration), or
Spend months training your own neural network on billions of positions

Most competitive hobbyist engines stop here or move to Tier 3c.

Tier 3c: An Elite Engine (~10,000+ lines, 3000+ Elo)

Goal: Grandmaster territory; competitive in engine championships; serious technical achievement.

What you add to Tier 3b:

NNUE Integration (the pivotal decision):
- Use an existing trained NNUE (most open-source engines do this)
- OR train your own on self-play data using PyTorch
- Integrate int8 NNUE evaluation (~500 lines)
Advanced search techniques:
- Razoring: Near quiescence, if static eval + margin > alpha, go straight to qsearch (worth ~10-20 Elo)
- Internal iterative reduction: On the PV, if no best move is available, reduce depth slightly
- Probcut: At shallow depths, use a wider evaluation margin to make cutoff predictions
Time management II:
- Chebyshev time allocation (give more time to critical moves)
- Resignation detection (stop searching if score is lost)
- Contempt factor (slight bias toward fighting positions over draws)
Endgame improvements:
- Syzygy tablebases (6-man endgame databases; ~20-30 MB download, adds ~10 Elo)
- Pawn endgame tables (7-piece, perfect play)
- Bitbase knowledge (e.g., K+P vs K endgame rules)
Evaluation refinements (hand-crafted component paired with NNUE):
- Piece-square table adjustments by material count
- King safety asymmetry (attacker bonus)
- Evaluation tempo bonus (now learned by NNUE, but can add explicit bonuses)
Distributed testing infrastructure (if you're serious):
- Hook into OpenBench or Fishtest
- Enable crowd-sourced testing of your patches

Search algorithm: Principal Variation Search (PVS) with all optimizations, or a modern variant like Negamax

Threading: 4-16 threads with lazy SMP (shared TT, split search at root)

Expected Elo: 3000-3500 (elite level; competitive in engine tournaments)

Time to implement: 10,000-15,000 lines of production-quality C++

Representative code structure at this level:

src/
  board.cpp/h        (1,200 lines)
  search.cpp/h       (2,500 lines: alpha-beta, all pruning techniques, TT)
  evaluate.cpp/h     (800 lines: NNUE integration + handcrafted bonuses)
  nnue.cpp/h         (800 lines: int8 quantization, accumulator updates)
  movegen.cpp/h      (600 lines)
  uci.cpp/h          (400 lines: UCI options, time management)
  tt.cpp/h           (300 lines: transposition table)
  bitboards.cpp/h    (200 lines: magic bitboards, attacks)
  endgame.cpp/h      (300 lines: tablebases, special rules)
  main.cpp           (100 lines)
  tests/             (2,000 lines: perft, position tests, regression suite)

NNUE Integration Example:

// Simplified NNUE forward pass
class NNUE {
public:
    int8_t input[2][512];    // Two perspectives
    int32_t hidden[8];       // First hidden layer
    int16_t output;          // Final evaluation

    void update_accumulator(Move m) {
        // Incrementally update instead of full recompute
        // Most of the 1,500× speedup comes from here
    }

    int evaluate() {
        // Update accumulator
        update_accumulator(last_move);

        // Forward pass through tiny network
        for (int i = 0; i < 8; i++) {
            hidden[i] = input[0][i*64] + input[1][i*64];  // Simplified
        }

        // Final output layer
        output = (hidden[0] * w0 + hidden[1] * w1 + ...) >> 7;

        return output;
    }
};

The training pipeline (if you're training your own NNUE):

Generate self-play games using your engine
  ↓
Score positions at depth 8-12
  ↓
Collect 1+ billion positions
  ↓
PyTorch training loop (SGD, L1 loss, quantization-aware training)
  ↓
Export to ONNX or custom int8 format
  ↓
Integrate into engine
  ↓
Test on Fishtest/OpenBench
  ↓
If +25 Elo: merge. If -25 Elo: discard.

Maintenance burden at 10K lines:

A comprehensive test suite (perft, mate-in-N puzzles, position regression tests)
Profiling to identify bottlenecks (search takes ~95% of time; move gen ~2%)
Continuous tuning: 50+ parameters (LMR reductions, null move depth limits, eval weights) can be optimized with texel tuning or local search

Where to start: Clone Berserk (simpler than Stockfish, ~4,500 lines) and add NNUE support. This gives you a 2.5x head start.

Code Complexity vs. Elo Gain (Diminishing Returns)

Here's the brutal truth about chess engine development:

Lines of Code	Elo	Time to Build	Improvement Rate
300	1600	1 weekend	+5.3 Elo/line
1,000	2000	1 week	+0.4 Elo/line
3,000	2500	2-3 weeks	+0.17 Elo/line
5,000	2900	1-2 months	+0.08 Elo/line
10,000	3300	3-6 months	+0.04 Elo/line
50,000	3759	Years	+0.01 Elo/line

The first 1,000 lines are magical. Each line of code adds ~4 Elo. By 5,000 lines, you're down to 0.08 Elo per line. The last 40,000 lines of Stockfish are engineering optimization for marginal gains.

This is why starting with 3,000 lines is the sweet spot for learning. You get a genuinely strong engine (beats humans), understand all the major algorithmic concepts (alpha-beta, pruning, TT), and can see your work play real chess. Anything beyond 5,000 lines is specialist territory.

The Real Lesson

The gap between 1800 Elo and 3600 Elo is about 40 years of accumulated engineering wisdom:

LMR (2004): +100 Elo
Null move pruning (1980s): +150 Elo
Singular extensions (2000s): +30 Elo
Killer heuristic + history (1980s): +50 Elo
NNUE evaluation (2020): +100 Elo
Iterative deepening + time management (1980s): +30 Elo

Each innovation compounds. Modern engines are not singularly brilliant; they're the product of thousands of incremental improvements, tested rigorously on billions of games.

If you're building a hobbyist engine, focus on correctness first, optimization second. Get move generation and search right before worrying about evaluation. A simple alpha-beta + null move pruning engine at 1600 Elo is a better learning experience than a buggy, half-finished attempt at NNUE.

If you're aiming for competitive strength, use distributed testing (OpenBench or Fishtest) early. A single developer testing patches locally will trend toward local optima. The open-source ecosystem wins because every improvement is validated against thousands of positions and millions of games.

And remember: Stockfish is 50,000 lines of C++, but the first 1,000 lines give you 1500 Elo. The remaining 49,000 lines bought maybe 500 more.

Why I Built a NAS in 2026: When Your Vendor Abandons You

Supreet Sethi — Thu, 22 Jan 2026 04:14:31 +0000

My decade old WD MyCloud still works perfectly. The drives are healthy. The network functions. But the mobile app stopped connecting years ago;Western Digital sunset the cloud services without warning. The hardware outlived the business relationship.

Meanwhile, my family defaulted to iCloud at $168/year. Forever.

That's when I understood: In 2026, your hardware doesn't fail. Your vendor relationships do.

The Real Cost of Cloud

When you upload to iCloud or Google Photos, you're not storing photos; you're feeding datasets. Your vacation trains facial recognition. Your kids' birthdays become AI training data. Your entire visual timeline becomes product, not property.

And you pay $120-168/year for the privilege. In perpetuity.

iCloud Family (2TB): $1,678 over 10 years. Own nothing.

The terms change. Features paywall. Companies pivot. You keep paying or lose access.

This is the bargain: convenience now, dependency forever.

Why Build Instead of Buy?

Synology DS423 + drives: $790 AUD

Beautiful interface, one-click apps, "it just works"
But: proprietary OS, curated ecosystem, automatic updates you can't control
When they sunset support in 10 years, you're stuck

My Build: $610 AUD

N100 mini PC ($370) + 4× 1TB NVMe ($240)
Ubuntu, ZFS, Immich, PostgreSQL; all open source
When anything changes, I adapt. No vendor can abandon me.

The difference isn't price. It's sovereignty.

Synology optimizes for simplicity by removing control. I optimized for capability by accepting complexity.

The Hardware Choice: NVMe vs HDD

Why NVMe when HDDs are cheaper?

4TB NVMe: $240 upfront, $70 power over 10 years = $310 total
4TB HDD: $60 upfront, $520 power over 10 years = $580 total
NVMe saves $270 long-term, runs silent, fits in your palm

Performance: 1167 MB/s sequential (5× faster than HDDs), 50k+ random IOPS (vs 150 for HDDs). Matters for thumbnails, galleries, facial recognition.

Trade-off: Can't build massive arrays. But I need 3TB for photos, not 20TB for media servers.

NVMe isn't splurging; it's optimizing for silence, performance, and efficiency over raw capacity.

The Software: Choosing Freedom

Ubuntu 24.04 LTS: 6 years of updates, then I choose when to upgrade
ZFS RAIDZ1: Enterprise filesystem, bit rot protection, snapshots; no subscription
Immich: Self-hosted Google Photos with facial recognition, ML search, automatic iPhone backup
PostgreSQL + Valkey: Same stack powering billion-dollar companies

All free. All maintained by communities answering to users, not shareholders.

No vendor can sunset this. No company can paywall features. No terms can change.

The Economics

10-Year Total Cost:

My NAS: $610 + $220 power = $830
Synology + power: $1,310
iCloud: $1,680

20-Year Total Cost:

My NAS: $1,050 (just adding electricity)
Synology: $2,620 (power + forced hardware replacement)
iCloud: $3,358

After year 4, my cost drops to $22/year electricity. Cloud costs never decrease. Synology faces eventual forced upgrades when support ends.

But the real economics aren't financial; they're about control.

Cloud companies own your data relationship. Vendors own your software stack. I own everything.

What Sovereignty Means

Photos on hardware I can touch
No company can raise prices, change terms, or train AI on my family
When something breaks, I fix it—no support tickets
Software choices are mine forever
Zero monthly fees, zero vendor lock-in

What it costs:

$610 upfront
One evening of setup
Willingness to learn

Who Should Build?

Buy Synology/use iCloud if:

Technology intimidates you
Time is more valuable than money
You trust vendors more than yourself

Build your own if:

Control matters more than convenience
You're comfortable with Linux basics
Vendor lock-in bothers you
You value learning how things work

Neither is wrong. But only one is sovereign.

Why 2026 Is Different

Hardware: $370 quad-core mini PC was impossible five years ago. NVMe prices collapsed.

Software: Immich rivals Google Photos. ZFS is enterprise-grade. Ubuntu is rock-solid. The tools exist and are free.

Cloud costs: Rising steadily while adding AI scanning, government requests, and surveillance.

Vendor support: Companies sunset working products when support becomes unprofitable. My old MyCloud proved this.

You can now build sovereign infrastructure for less than pre-built solutions, with more capability and zero ongoing dependencies.

The Reality Check

This isn't for everyone. If Linux scares you, buy Synology. If you can't spare an evening, pay for iCloud.

But if you care about privacy, hate subscriptions, value understanding your tools, and worry about vendor abandonment;

What I Built

3TB usable storage (ZFS RAIDZ1, survives one drive failure)
1167 MB/s writes (10× faster than network can use)
Silent operation (fanless, 15W total)
Automatic iPhone backup with facial recognition and ML search
Total control over every aspect

Works flawlessly. Family uses it daily. Photos sync automatically. Interface is fast.

And no vendor can ever turn it off.

The Conclusion

In 2026, we've normalized renting access to our own files. Photos don't exist until uploaded to someone's server. Privacy is suspicious. Surveillance is convenient. Working hardware gets abandoned when unprofitable.

Building a NAS is a small rebellion against this.

It cost $610 and an evening. It saves $168/year forever. It protects my family's privacy. It can't be sunset.

My NAS hums at 15 watts, hosting memories that belong to us. Not to Google. Not to Apple. Not to any corporation that might change terms or decide our hardware is too old.

Just us.

That's not revolutionary. It's just ownership.

And in 2026, ownership feels like victory.

Hardware: $610. Monthly fees: $0. Vendor lock-in: none.

Still renting your data, or ready to own it?

Digital Transformation: Why 70% Fail; And How 30% Win

Supreet Sethi — Tue, 09 Dec 2025 05:44:34 +0000

A Data-Backed Guide to Why 70% Fail and How Winners Get 80% Odds

Sources: BCG (860 executives, 30 case studies), McKinsey (1,793 respondents), Academic literature

THE BRUTAL FACTS

I have personal experience being part of digital journey across India, South East Asia and Australia. It is disturbing to see uncoordinated heap of tactical monikered as digital transformation.

Only 30% of digital transformations succeed (meet targets, create sustainable change)
70% fail or underperform (limited value, no durability)
Failed transformations destroy 30–65% of stock value (Target: -38% P/E, Sears: bankruptcy)
Successful transformations create 50–100% stock gains (Walmart: +62% P/E, Netflix: sustained premium)

The difference? Not technology. Not budget. Six critical success factors.

THE SIX SUCCESS FACTORS

Companies that nail ALL SIX achieve 80% success rate (vs. 30% baseline)

#	Factor	Done Right	Done Wrong
1	Clear Strategy	Quantified outcomes tied to competitive advantage (Walmart: "omnichannel + retail media margins")	Vague vision ("go digital"); no competitive advantage articulated (Target)
2	Leadership	CEO + CFO visible; middle management compensated for transformation; CDO elevated	CDO role but weak authority; store managers unaligned; no accountability (Target, Sears)
3	Talent	Hire elite external (30%+); free them up full-time; rotation programs (Amazon from Google; Walmart from Amazon)	Internal reskilling only; underpaid vs. competitors; wrong talent profile (Sears hired store ops people)
4	Governance	Weekly steering; decisions <1 week; teams empowered up to $500K (Amazon: "bar raiser" Friday decisions)	Quarterly reviews; months to resolve blockers; slow approval chains (Target, Sears)
5	Monitoring	Outcome metrics only (growth %, margin bps, deployment speed); real-time dashboards (Walmart: weekly e-comm KPIs)	Activity metrics ("people trained," "% adoption"); quarterly reviews (Target, GE Digital)
6	Architecture	Microservices, API-first, real-time data pipelines; deploy weekly (Netflix, Amazon: 1000+ deploys/day)	Monolithic systems, bolted-on integrations, 6-8 week cycles (Target, Sears mainframe)

WINNERS: WHAT THEY DID RIGHT

Walmart (+62.3% P/E, 2020–2025)

The Transformation:

Discount grocer → omnichannel powerhouse with recurring revenue (retail media)

Execution:

Strategy: "E-commerce to 25%+ revenue; retail media to $5B; margin expansion"
Leadership: CEO Doug McMillon visible; middle managers get 20-30% bonus for e-commerce growth
Talent: Hired CTOs from Amazon/Google; 18-24 month rotation program at 30% premium salary
Governance: Weekly steering committee; blockers resolved in <7 days
Monitoring: Published metrics (26% e-commerce growth, 93% same-day reach, $4.4B retail media)
Architecture: Microservices, API-first, real-time data ingestion

Result: Operating margin +28 bps; retail media 70% margins; P/E 21.69 → 35.20

Amazon (1995–Present, Continuous Pivot)

The Transformation:

Bookstore → e-commerce → cloud infrastructure → AI-first platform

Execution:

Strategy: Customer obsession; tied to measurable metrics (speed, cost, defect rate)
Leadership: "Day 1" culture; removed leaders who couldn't shift to growth mindset
Talent: "Hire the best engineer you know"; 40%+ from external
Governance: Weekly decisions; two-pizza rule (small teams, autonomous decisions)
Monitoring: Real-time customer metrics drive product roadmap
Architecture: Service-oriented from day 1; 1000+ deployments/day

Result: 5000%+ stock return since IPO; innovation velocity 3x industry average

Netflix (2007–2025, Multi-Pivot)

The Transformation:

Physical rental → streaming → original content

Execution:

Strategy: Each pivot with clear metrics (subscribers, churn, content ROI)
Leadership: "Context over control"; failed experiments celebrated
Talent: High-performer culture; recruited from startups and tech giants
Governance: Weekly steering; innovation lab runs failed experiments in weeks
Monitoring: Real-time subscriber/engagement dashboards drive spend allocation
Architecture: Cloud-native (AWS); real-time data pipeline

Result: 300M+ subscribers; P/E premium 60–80x sustained; market dominance

Microsoft (2014–2025, Cloud Pivot Under Satya Nadella)

The Transformation:

Windows/licensing → cloud-first + AI strategy

Execution:

Strategy: "Mobile-first, cloud-first" (contradicted legacy Windows strategy)
Leadership: Nadella removed executives who resisted growth mindset
Talent: Hired Chief Cloud Officer from Amazon, CIO from Google
Governance: Quarterly (faster than pre-Nadella annual cycles)
Monitoring: Azure revenue growth, churn, feature time-to-market tied to exec comp
Architecture: Cloud-native; microservices; API-first

Result: Azure $60B+ business (25–35% YoY growth); stock 20x return; P/E +60%

LOSERS: WHAT THEY GOT WRONG

Target (–38.2% P/E, 2019–2025)

Why It Failed:

Strategy: "Become digital retailer" (vague) with no competitive advantage articulated
Leadership: CDO present but not empowered and store managers unsupported
Talent: <30% external hiring; 8-week training programs resulting in weak execution in comparison to competitors
Governance: Quarterly reviews - 4+ quarter blockers and quality of delivered E-commerce experiential inferior to Walmart
Monitoring: Lack of clear activity metrics ("# trained," "% adoption") and no clear measure of gaps
Architecture: Bolted omnichannel onto monolith with 6–8 week deployment cycles

Result: Billions spent, market share lost to Walmart, valuation compressed 38%

GE Digital ($5B+ sunk cost, then divested)

Why It Failed:

Strategy: Predix platform separate from core business (not integrated into manufacturing divisions)
Leadership: Manufacturing divs had no accountability for adoption
Talent: Reskilled mechanical engineers as software engineers (cultural mismatch)
Governance: Annual strategic reviews (too slow for SaaS)
Monitoring: Vanity metrics (customer logos, not product-market fit)
Architecture: Optimized for GE's use case; not scalable to market

Result: $5B investment → zero market traction → divested

Sears & Blockbuster (→ Bankruptcy, 99%+ value destruction)

Why They Failed:

Strategy: Denial ("physical will always dominate")
Leadership: CEO focused on cost-cutting, not strategic positioning
Talent: No hiring; promoted store ops people to digital leadership
Governance: Quarterly decisions; market moved faster
Monitoring: No competitive positioning metrics
Architecture: Mainframe-era systems; $500M+ rearchitecture needed

Result: Stock $100+ (2000s) → bankruptcy (2010–2018)

THE SCORECARD: 6 FACTORS ACROSS 8 COMPANIES

Company	Strategy	Leadership	Talent	Governance	Monitoring	Architecture	Score	Outcome
Walmart	✓	✓	✓	✓	✓	✓	6/6	+62% P/E
Amazon	✓	✓	✓	✓	✓	✓	6/6	+5000%+ stock
Netflix	✓	✓	✓	✓	✓	✓	6/6	Premium maintained
Microsoft	✓	✓	✓	✓	✓	✓	6/6	+60% P/E, 20x stock
Target	✗	✗	✗	✗	✗	✗	0–2/6	–38% P/E
GE Digital	✗	✗	✗	✗	✗	✗	0–1/6	Divested, $5B+ loss
Sears	✗	✗	✗	✗	✗	✗	0–1/6	Bankruptcy
Blockbuster	✗	✗	✗	✗	✗	✗	0/6	Bankruptcy

THE MATH THAT MATTERS

Success Probability by Score:

6/6 factors: 80% success → 50–100% P/E expansion
4–5/6 factors: 40–50% success → flat to slight premium
2–3/6 factors: 20–30% success → valuation stall
0–1/6 factors: 5–10% success → 30–65% compression or bankruptcy

Market Impact:

Winner (Walmart) P/E: 35.20 → 35% premium over sector median
Loser (Target) P/E: 11.13 → 30% discount to peer average
Gap: 3x multiple difference → same industry, same decade

THE FORMULA (ULTRA-CRISP)

Digital transformation succeeds if and only if ALL SIX are in place:

Clear, quantified strategy (not "go digital"; say "hit 25% e-commerce, add $5B margin-rich revenue")
CEO-led commitment (visible, not delegated; middle management compensated)
Right talent (hire elite external; free them up full-time; pay premium)
Fast governance (weekly decisions; <7 day escalation; teams empowered)
Outcome metrics only (growth %, margin improvement, speed); publish weekly
Modern architecture (microservices, API-first, real-time data; 1–2 week deployments)

Miss one: risk spikes to 50–70%.

Miss three: failure almost certain; valuation collapse imminent.

FINAL TAKEAWAY

There is no luck in digital transformation.

Winners (Walmart, Amazon, Netflix, Microsoft) got all six factors right and compound their advantage yearly. Losers (Target, GE Digital, Sears, Blockbuster) skipped one or more factors and paid the price—sometimes bankruptcy.

Your job: Get all six right. Not approximately. All six.

The market knows the difference. Your stock price will reflect it in 2–3 years.

Don't Touch Your Trade: The Most Underrated Strategy in Investing

Supreet Sethi — Mon, 08 Dec 2025 02:21:48 +0000

You've been staring at your portfolio for 15 minutes. A stock is down 3%. Should you rebalance? Check another screener? Buy the dip? Optimize something?

Stop.

That itch to tinker—to debug, iterate, optimize—is making you poorer.

The Uncomfortable Truth: Activity Is the Enemy

Here's what the data says:

Retail investors underperform their own funds by 1–2% annually. Not because they pick bad stocks. Not because of fees. Because they trade too much.

Day traders? Worse. They underperform by 5–10% annually, even accounting for survivorship bias.

Morningstar's "Mind the Gap" study tracked this for two decades. Dalbar's research confirmed it. Academic studies on retail flows all point to the same culprit:

More trades → Emotional decisions → Worse timing → Compounding in reverse.

The math is brutal. A 15–30% wealth gap over a decade. That's not negligible. That's retirement-level money.

And here's the thing: it's not stupidity. It's busyness.

Why Your Developer Brain Backfires

You're trained to:

Increase iterations
Reduce latency
Debug fast
Ship incremental improvements constantly

This works beautifully in code. It's a disaster in markets.

Markets aren't deterministic. They're stochastic, adversarial, and soaked in noise. The signal-to-noise ratio is terrible. Which means your optimization instinct—your superpower in engineering—becomes a liability.

The rule flips:

Code: "Touch the system frequently"
Markets: "Touch the system rarely"

Every decision point is a chance to mess up. Every trade, a potential emotional mistake. The best traders aren't the busiest—they're the most selective.

The Actual Edge: Do Nothing

If you study every winning strategy that's beaten the S&P 500 over 20+ years, they all collapse to the same principle:

Buy quality. Hold. Interfere minimally.

When you stop trading:

Compounding compounds (not interrupted by poor timing)
Fees evaporate (fewer transactions = lower costs)
Taxes shrink (long-term capital gains > short-term realized losses)
Emotional mistakes disappear (can't make bad trades if you don't trade)
Execution becomes simple (fewer decision points = fewer ways to fail)

Think of it like background processes. Compounding is a thread running 24/7 whether you're awake or asleep. Your job isn't to optimize it. It's to not kill it.

Events vs. Signals: The Mental Model That Changes Everything

Most traders confuse noise for information:

Events (ignore these):

A tweet
A Fed headline
A stock pop or drop
Earnings beat by 2%
An analyst upgrade

Signals (trade on these):

Earnings trends (last 4 quarters improving)
Free cash flow inflection
Long-term growth acceleration
Margin expansion (structural, not cyclical)
Industry disruption with clear winners

Events are fast. Signals are slow. If your portfolio is changing weekly, you're trading on events. And you're losing.

A Rule You Can Adopt Monday Morning

The Weekly Rule:

Review portfolio: once per week
Make decisions: once per month
Trade: only when something fundamental shifts

The Quarterly Rule (for long-term folks):

Thesis broken? Trade
Earnings trajectory changed? Consider it
Price moved? Ignore it completely

That's it. Most investors would triple their returns with this discipline alone.

How to Code This Into Your System

If you were automating "trade less, earn more," it'd look like:

def should_trade(signal_strength, thesis_intact):
    if not thesis_intact:
        return True
    if signal_strength > 0.85:
        return True
    return False  # Default: NO TRADE

See that final return? In your codebase, no-op is the most profitable branch in the long run.

The Payoff: Better Returns, Better Sleep

When you trade less:

You stop chasing momentum (the #1 killer of returns)
FOMO-driven buys vanish
Revenge trades disappear
You sleep eight hours instead of checking futures
Your actual returns start matching market returns

Ironically, trading less is how you stop underperforming and finally start outperforming.

The Idempotence Principle

A robust system doesn't break when you run the same command twice. It doesn't require constant intervention. It's stable and predictable.

Your portfolio should be the same.

If your strategy forces 20 decisions per week, it's not sophisticated—it's brittle. The best portfolios are:

Predictable
Repeatable
Boring
Idempotent

And boring portfolios are profitable portfolios.

The One Edge You Actually Need

You don't need to beat hedge funds. You don't need perfect timing. You don't need machine learning or a proprietary screener.

You need one thing:

The discipline to do nothing most of the time.

That's not laziness. That's leverage.

Trade less. Earn more. Sleep better.

I Hired AWS SAs for 5 Years. Now That I've Left, I Can Tell You the Truth

Supreet Sethi — Thu, 27 Nov 2025 21:41:45 +0000

tl;dr: I spent five years interviewing SA candidates at AWS. Comprehensive AWS training doesn't correlate with hire decisions. Production talk and clear thinking do. Now that I'm external, I can say it plainly: most interview prep is wasted effort.

The Uncomfortable Truth

I was a Principal SA at AWS for two years. I sat on hiring panels as a Manager after that for three years. I made hire/no-hire decisions. I watched hundreds of candidates prepare the wrong way and then blame themselves for not studying hard enough.

Now that I've left, I can say this plainly without corporate-speak: your comprehensive AWS study guide is doing more harm than good.

We don't care if you memorized the Well-Architected Framework. We didn't care if you studied all 200 AWS services. We cared if you could think clearly about tradeoffs, defend a decision under pressure, and admit when you didn't know something.

Yet every study guide, bootcamp, and LinkedIn thread screams the same lie: you need comprehensive AWS training.

You don't. In fact, it often works against you.

What Happens in the Room

Your interviewer—especially if they're a Principal or staff engineer—already knows you haven't deployed infrastructure across 47 different AWS services. They're not testing breadth. They're testing depth of thinking and production judgment.

Here's what I paid attention to when candidates walked in:

1. Production context—where's the real decision?
"I've built with EC2 and RDS" is fine. But I listened for why you made choices. Instance families? Why? Scaling problems? Walk me through one. Connection pooling in production? Perfect—now we're talking about real engineering.

The candidates I voted to hire almost always had one or two services they knew cold because they shipped something real. The ones reading from study guides tried to sound like they knew everything, and it was immediately obvious.

2. Can you think on your feet?
I watched how people handled "how would you architect X?" If they recited a framework answer, I'd already mentally moved on to the next candidate. If they asked me clarifying questions, identified genuine tradeoffs, and said "I'd lean toward Fargate here because—," I'd lean forward.

The best moment in an interview is when a candidate says "I haven't used that, but here's my hypothesis based on..." That's the real move. That's how actual architects work. They don't know everything. They think.

3. Honest gaps are revealing
I built my hiring questions specifically to find the difference between "studied this last week" and "this is my blind spot." When someone said, "I'd need to research that, but here's my starting point," they immediately became more credible than someone bulldozing with buzzwords.

The worst interviews? Candidates stalling on basic questions while clearly trying to remember what slide 47 of their bootcamp said.

The Training Industrial Complex (Or: Why Your $500 Course Hurt Your Chances)

Five years on the hiring side has shown me a clear pattern: candidates who take comprehensive AWS training courses often perform worse than those who don't.

It's counterintuitive, but the data's there:

Candidate studies 200 services -> interviews with surface-level knowledge of many, depth in none
Candidate ships one project with EC2/RDS -> interviews with real architectural thinking
Interview panel immediately spots rehearsed answers and deprioritizes
Candidate blames themselves: "I should have studied the ECS section more"

Wrong diagnosis. Your study guide didn't help you; it convinced you that breadth matters more than depth.

The cruel cycle:

Candidate feels anxious > buys $500 "Complete AWS for SAs" course
Course teaches 200 services at surface level
Candidate memorizes facts they'll forget in 2 weeks
Interview panel spots the rehearsed answers within 10 minutes
Candidate doesn't get the job
They blame themselves for not studying harder

I've sat through hundreds of interviews. The training industry profits off this anxiety loop while genuinely hurting candidate performance.

What You Actually Need

For the phone screen:

Production experience with at least one primary service (compute, database, or whatever)
Ability to discuss a real project: what worked, what didn't, what you'd change
Willingness to think through new problems without pretending you have all the answers
AWS fundamentals (VPC, IAM, Security Groups, GW, basic networking)—these are legitimate

For the technical discussion:

Deep knowledge of services you've used
Framework for thinking about tradeoffs (cost vs. complexity, availability vs. simplicity, etc.)
Ability to pivot and explore adjacent solutions
Comfortable saying "I'd need to research that, but here's my starting hypothesis"

That's genuinely it.

The Real Test: A Rugby Social Network Migration

Let me walk you through an actual scenario AWS uses in interviews—and what separates the good architects from the ones who fail despite being technically sound.

The problem: A rugby-focused social network with 2M users, growing 5x in the next 12 months. Currently on-premises in Dallas. Budget: $25K/month (15K hosting, 10K CDN via Akamai). Users spread: US 30%, Europe 50%, Asia 20%. They need to move to AWS.

Bad SA answer (technically competent, but fails):

"We'd use EC2 for compute, RDS for the database, S3 for video storage, CloudFront for CDN to replace Akamai, ElastiCache for sessions, Route53 for DNS, Auto Scaling groups, VPC with public/private subnets across three AZs, IAM roles for service-to-service auth, CloudWatch for monitoring, potentially Lambda for event processing, maybe Kinesis for real-time notifications, SQS for async jobs..."
Goes deeper: task definitions, RTO/RPO, security groups, NACLs, database backup strategies
Panel internally: "They know AWS components. But I have no idea if they can actually make a decision."
Rejected (despite being technically correct)

Good SA answer (the one I hire):

"Three regions: us-east-1 for US traffic, eu-west-1 for Europe, ap-southeast-1 for Asia. Each region gets an application tier and a database. We'll start with EC2 for the app layer—I'd use t3.large with ASGs to handle the 5x growth. RDS Multi-AZ in each region for data durability. We're replacing Akamai with CloudFront, which saves 10K/month immediately and actually improves latency in Europe and Asia."
Pauses. Asks clarifying questions: "Are writes distributed or mostly from content creators? How real-time are notifications? Is the data read-heavy or write-heavy?"
Listens to answer, then adapts: "If it's 80/20 read/write, we could add ElastiCache in front of RDS to reduce database load during peak traffic. If the user base is mostly mobile, we might want to think about API Gateway + Lambda instead of EC2 to reduce operational overhead—but EC2 gives us more predictable scaling for a 5x ramp."
When asked about databases: "RDS is safe here because we're relational. DynamoDB would require a schema redesign we don't have time for during a 5x growth. We validate that choice once we see real query patterns."
Hired (someone who thinks, not someone who lists)

The difference? The good SA made three architectural decisions (regional distribution, EC2 over serverless, RDS over NoSQL) with clear reasoning based on constraints. The bad SA listed every service and hoped some combination was right.

Then I tweaked the problem: "What if we need to migrate 500TB of video content and we can't afford 6 months of dual-running on-prem + cloud?"

Good SA pivots in real time:

"We'd use AWS DataSync to replicate data continuously for the first month while we're in parallel. Once the on-prem systems reach end of life, we flip the DNS. But more importantly, we don't need S3 for video storage if we're staying with Akamai for CDN—Akamai has edge storage. So we replicate video to Akamai's origin once, we store backups in S3, but the application doesn't serve from S3. That keeps our architecture simple and maintains the budget."
Still hired (adapted correctly)

Bad SA in same scenario:

Panics slightly. Falls back to memorized migration strategies.
Starts listing: "We could use AWS DataSync, AWS Direct Connect, AWS Snowball..."
Doesn't actually choose or reason about cost and timeline.
Still rejected (couldn't think under pressure)

Why "Describe Every Component" Actually Kills Your Interview

When you try to faithfully describe every possible AWS component in your architecture, you're signaling something deadly to the panel: you're not making decisions. You're reciting a design.

Here's what happens in the room:

The candidate who maps components:

Lists RDS, EC2, S3, CloudFront, ElastiCache, Route53, IAM, VPC, subnets, security groups, NACLs, KMS, CloudWatch, CloudTrail...
For each one, provides correct technical details
Panel thinks: "They know AWS. But I don't know if they'd actually make this decision if costs were cut in half. Do they know why each component is there, or are they just pattern-matching to 'enterprise architecture template #42'?"

The candidate who makes three decisions with reasoning:

"Multi-region for latency (us-east, eu-west, ap-southeast). EC2+RDS+CloudFront at each region. No serverless yet—5x growth means we need predictable scaling."
For each decision, brief reasoning: cost, operational burden, time-to-migrate
When challenged ("What about DynamoDB?"), they know why they chose RDS instead
Panel thinks: "If requirements change, this person will adapt the design, not rebuild from a different AWS architecture template."

The second person will design better systems under real pressure because they're reasoning about tradeoffs, not memorizing patterns.

The interview is testing your ability to:

Make decisions under constraint (time, budget, team skills)
Explain decisions in 30 seconds (broad strokes)
Change your mind when new information arrives (during the tweaks)

It's not testing whether you know all 200+ AWS services exist.

The cloud industry's obsession with breadth-first interview preparation isn't about better hiring. It's about justifying expensive training platforms while maintaining the illusion that AWS is harder than it actually is.

AWS is not hard. Thinking clearly about tradeoffs is hard. And no course teaches that. Only production teaches that.

Now that I'm building hiring processes at a new company, I'm explicitly deprioritizing "studied the most material" in favor of "shipped the most thoughtful decisions." We ask fewer "how many EC2 instance types exist?" questions and more "walk me through a bad architectural decision you made and how you'd change it." That's actually predictive. That's the kind of candidate who will design better systems under real pressure.

How to Actually Prepare (The Right Way)

Stop thinking in components. Start thinking in decisions.

Pick a real migration or scaling problem. Use the rugby case as a template:

2M users growing 5x in 12 months
Geographic distribution (not just US)
Budget constraint ($25K/month)
Existing infrastructure you're replacing

For your practice problem, write down:

Three architectural decisions. Not "use EC2, RDS, and CloudFront." Write: "Multi-region distribution because 50% of traffic is Europe and latency matters. EC2 instead of Fargate because we need cost predictability during 5x growth and ops team is Linux-comfortable. RDS instead of DynamoDB because schema is relational and migration timeline is tight."
Why each decision, in one sentence. If you need three sentences to justify it, you don't understand it well enough.
What you'd change if... Cost cut in half? Timeline compressed to 6 weeks? User base only in US? Changing any constraint should trigger a design change. If your design doesn't flex, you don't own it.

In the interview, lead with the broad strokes:

"I'd go multi-region, EC2+RDS in each region, CloudFront for CDN, and here's why..."
Don't list components. List decisions.
When they ask about a component you didn't mention ("What about caching?"), say: "Good point—I'd add ElastiCache if query patterns show we need it, but we validate that in week two post-migration. Right now, I'm optimizing for certainty over premature optimization."

Be ready for the tweaks:

"What if migration takes 2 weeks, not 2 months?" → "We'd use DataSync and parallel run, but that changes our budget. What's the hard constraint?"
"What if the team is Node.js only, not infrastructure-familiar?" → "Serverless becomes more attractive. Lambda+RDS+API Gateway probably safer than managing ASGs."
"What if they won't pay for multi-region?" → "Single region initially, but we architect for multi-region addition—CloudFront origin failover, RDS read replicas we can promote. Route 53 health checks point to healthy region."

This is how real architects work. You start with broad strokes, you defend those strokes, and you pivot when constraints change.

Final Thoughts

If you're preparing for an AWS SA role and thinking "I'm not ready because I haven't studied enough," stop. You're probably overthinking this.

You're ready if you can talk about real production problems and think through new ones. That's the job.

Stop cramming. Start shipping. The interview will feel different when you have stories instead of slides.

Curious if your production experience translates? Talk about a real architectural decision you made, no matter how small. That's your interview prep.

The Melbourne Talent War: A Conversation

Supreet Sethi — Thu, 27 Nov 2025 06:45:50 +0000

Setting: Fitzroy café. Afternoon sun. Coffee that tastes like punishment.

JAMES: Supreet, hiring in Melbourne is a nightmare. Everyone wants “top engineers,” no one can find them. And when they do, the person quits in six months for a 20k bump. I’ve never seen the market this thin.

SUPREET: Yeah, I’m hearing the same from clients. And honestly—this isn’t new to me. I lived a version of this in Indonesia back in 2015.

JAMES: What happened there?

SUPREET: We were trying to build a cloud team for a client. AWS was taking off, but Jakarta didn’t have many cloud engineers. Maybe ten people in the whole market who had touched EC2. Singapore companies were paying double. Local companies couldn’t compete.

JAMES: Sounds familiar already.

SUPREET: Exactly. So instead of chasing unicorns, we changed the game. We stopped trying to hire “cloud engineers.” Because they barely existed. We hired for raw talent. Smart people. Curious people. Didn’t matter if they were from helpdesk, QA, support, or networking.

JAMES: So you made cloud engineers instead of finding them?

SUPREET: Pretty much. We trained them from scratch. Linux basics. Network fundamentals. How cloud pricing worked. How to think about architecture. I mentored them directly, pushed them on real client problems, threw them into high-pressure situations.

JAMES: That sounds… intense.

SUPREET: It had to be. But in a year, they were better than people with “five years cloud” on the résumé. Because they learned deeply, not shallowly.

JAMES: Melbourne would lose its mind over that. Everyone wants “experience with Kubernetes” or “five years React,” whatever. No one wants to train.

SUPREET: Right. And yet everyone complains they can’t find talent. That’s the contradiction. You can’t have both.

JAMES: How did you even find these people in Indonesia? If you weren’t filtering for cloud experience, what were you looking for?

SUPREET: Signals. Stuff no job board filters for:

Could they think clearly?
Did they take initiative in past roles?
Were they hungry to learn?
Did their résumé show actual effort, not just job titles?

Some of the best were people who tinkered with Linux at home or fixed their family’s WiFi for fun.

JAMES: (laughs) Classic.

SUPREET: But that curiosity was gold. Teachable. Reliable. Sustainable.

JAMES: And they stayed?

SUPREET: Most did. Because the mentorship was the real glue. They weren’t just employees—they were being built. And they knew it. They knew they were growing faster than anywhere else.

JAMES: So mentoring was the retention strategy.

SUPREET: It was the whole strategy. People stay where they grow. Not where they get the highest salary.

JAMES: But some still left, right?

SUPREET: Of course. Once they got good, Singapore recruiters came calling. Two left. Hurt a little. But the rest stayed because the growth, culture, and trust mattered more than a few extra thousand dollars.

JAMES: Okay, bring this back to Melbourne. How does this translate?

SUPREET: Melbourne right now is what Jakarta was in 2015.
Cloud, platform, DevOps—demand is massive, supply is tiny.

Companies keep trying to “buy” senior talent. It won’t work.

The only consistent strategy is:

Hire for potential. Mentor hard. Build the team you need.

JAMES: And what does “hire for potential” look like here?

SUPREET: Forget the keyword checklist. Look for:

strong fundamentals (networking, Linux, systems thinking)
curiosity
self-learners
people who’ve built anything on their own
people who write clearly, think clearly

Someone who built a home lab is more valuable than someone who wrote “AWS certified” but never designed anything real.

JAMES: And the mentoring? How do you structure that?

SUPREET: Weekly 1:1s. Real feedback. Pairing on real issues. Give them responsibility early. Don’t treat juniors like juniors. Treat them like apprentices.

JAMES: That’s… not common here.

SUPREET: Exactly why it works.
It’s rare.
It creates loyalty.
And it solves the hiring problem permanently.

JAMES: So your message to Melbourne companies is basically:
“Stop complaining about the senior talent shortage. Build it.”

SUPREET: Yep. Because the truth is, the people who will become your best engineers are currently doing jobs where no one notices them.

JAMES: That’s a great line.

SUPREET: It’s also true.

JAMES: (lifts coffee) So—hire potential, mentor properly, grow your own talent.

SUPREET: (raises cup back) And stop waiting for unicorns that don’t exist.

Forget Wall Street: Here’s How to Create the Best Stock Screener Yourself

Supreet Sethi — Thu, 27 Nov 2025 06:19:03 +0000

Stock Screeners for Math People: Build Your Own

A stock screener is just a function. You don't need Wall Street, a fancy terminal, or YouTube. If you know high school math and Python, you can build your own screener that is transparent, tuned to your portfolio, and changeable as your ideas evolve.

1. What Is a Stock Screener, Really?

At its core, a stock screener is a mathematical filter function:

\text{Screener}(\text{stock}) = \begin{cases} \text{TRUE} & \text{if stock passes your rules} \ \text{FALSE} & \text{otherwise} \end{cases}

You have a universe of N stocks, each with data:

\text{Input} = {S_1, S_2, \ldots, S_N}, \quad S_i = {P_i(t), V_i(t), F_i}

Where:

$P_i(t)$ = price of stock i at time t
$V_i(t)$ = trading volume
$F_i$ = fundamental data (EPS, revenue, etc.)

Your screener applies a function:

f(P_i(t), V_i(t), F_i) \rightarrow {\text{TRUE}, \text{FALSE}}

Apply this to 5,000 stocks, keep those returning TRUE, then rank them. That's the entire idea.

2. Why the "Best Screener" Depends on What You Already Own

The biggest mistake beginners make: treating the "best stock screener" as universal.

In reality, it depends on your portfolio vector:

\vec{P} = [p_1, p_2, \ldots, p_n]

Each $p_i$ is a stock with its own return series. When you add a new stock $X$ , what matters is correlation with your existing portfolio.

Correlation Between Return Series

\text{Corr}(X, Y) = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}

The interpretation:

$Corr≈1\text{Corr} \approx 1$ → they move together (bad for diversification)
$Corr≈0\text{Corr} \approx 0$ → independent movements (good)
$Corr≈−1\text{Corr} \approx -1$ → opposite moves (excellent hedge)

The Real Problem

If you already own 10 tech stocks with mutual correlation ≈ 0.8, adding an 11th tech stock just concentrates risk. It's not diversification—it's doubling down.

Your screener must know what you own and prefer low-correlation stocks.

import numpy as np
import pandas as pd

def portfolio_returns(returns_df, weights):
    """Compute weighted portfolio return series"""
    w = np.array(weights).reshape(-1, 1)
    return (returns_df.values @ w).flatten()

def correlation_with_portfolio(candidate_returns, portfolio_ret_series):
    """Correlation between a candidate and your portfolio"""
    corr_matrix = np.corrcoef(candidate_returns, portfolio_ret_series)
    return corr_matrix[0, 1]

# Decision rule: only buy if correlation < 0.5

3. Core Math Building Blocks

3.1 Returns: Rate of Change

Return over n days is the percentage change:

R_n(t) = \frac{P(t) - P(t-n)}{P(t-n)}

Example:

Price 20 days ago: $100
Price today: $115
20-day return: $115−100100=0.15=15%\frac{115 - 100}{100} = 0.15 = 15\%$

3.2 Moving Averages: Smoothing Noise

Simple Moving Average over n days:

\text{SMA}n(t) = \frac{1}{n} \sum{i=0}^{n-1} P(t-i)

If price is above its recent SMA, it often signals an uptrend.

3.3 Volume Ratio: Conviction Metric

Volume tells you "how many people agree with the move":

\text{Volume Ratio} = \frac{\text{Recent Average Volume}}{\text{Historical Average Volume}}

If this is $ParseError: KaTeX parse error: Expected 'EOF', got '}' at position 1: }̲ > 1.5$ , volume is picking up—a strong signal.

4. Your Data: CSV + SQLite

Use two simple data sources you control locally:

Daily Prices in CSV

One file per ticker: AAPL.csv

Date,Open,High,Low,Close,Volume
2024-01-15,150.00,152.50,149.80,151.20,2500000
2024-01-16,151.30,153.00,151.00,152.80,2300000
2024-01-17,152.00,153.80,151.50,152.95,2400000

Quarterly Fundamentals in SQLite

import sqlite3

conn = sqlite3.connect("stocks.db")
cursor = conn.cursor()

cursor.execute("""
CREATE TABLE IF NOT EXISTS fundamentals (
    ticker TEXT,
    quarter_date TEXT,
    eps REAL,
    estimated_eps REAL,
    revenue REAL,
    net_income REAL,
    market_cap REAL
)
""")

cursor.execute("""
CREATE TABLE IF NOT EXISTS companies (
    ticker TEXT PRIMARY KEY,
    name TEXT,
    sector TEXT
)
""")

conn.commit()

You now have a local data warehouse you control—no API limits, fully reproducible.

5. Screener #1: Momentum (First Derivative of Price)

Momentum is just rate of change:

\text{Momentum}_n(t) = \frac{P(t) - P(t-n)}{P(t-n)} \cdot 100\%

A simple rule: select stocks with 20-day momentum $>15%\gt 15\%$ .

import pandas as pd
import numpy as np
import os

def load_price_data(csv_dir, ticker):
    """Load CSV into DataFrame"""
    df = pd.read_csv(f"{csv_dir}/{ticker}.csv")
    df["Date"] = pd.to_datetime(df["Date"])
    return df.set_index("Date")

def momentum_20d(close_series, period=20):
    """Calculate n-day momentum percentage"""
    if len(close_series) < period + 1:
        return np.nan
    p_today = close_series.iloc[-1]
    p_past = close_series.iloc[-period - 1]
    return (p_today - p_past) / p_past * 100.0

def momentum_screen(csv_dir, min_mom_pct=15):
    """Screen: momentum > 15%"""
    tickers = [f[:-4].upper() for f in os.listdir(csv_dir) 
               if f.endswith(".csv")]
    results = []

    for t in tickers:
        try:
            df = load_price_data(csv_dir, t)
            mom = momentum_20d(df["Close"])
            if mom >= min_mom_pct:
                results.append({
                    "ticker": t,
                    "momentum_20d_pct": round(mom, 2)
                })
        except Exception as e:
            print(f"Error on {t}: {e}")

    return sorted(results, key=lambda x: x["momentum_20d_pct"], reverse=True)

# Run it
momentum_stocks = momentum_screen("./stock_data")
print(f"Found {len(momentum_stocks)} momentum candidates")

When to use: When you see a trend forming. Momentum works because markets overshoot—once they start moving, they keep moving.

6. Screener #2: Volume Accumulation (Smart Money Signal)

We want to know if price moves are backed by serious trading volume.

The money flow line combines two facts:

Where did price close in today's range?
How much volume was there?

\text{MoneyFlow}(t) = \frac{\text{Close}(t) - \text{Low}(t)}{\text{High}(t) - \text{Low}(t)} \cdot \text{Volume}(t)

If this is consistently positive AND volume is rising, you likely have accumulation.

def volume_ratio(df, recent_days=5, hist_days=20):
    """Volume momentum: recent average / historical average"""
    recent = df["Volume"].tail(recent_days).mean()
    hist = df["Volume"].tail(hist_days).mean()
    return recent / hist if hist > 0 else 0

def money_flow_sum(df, period=14):
    """Sum of money flow over period"""
    high = df["High"]
    low = df["Low"]
    close = df["Close"]
    vol = df["Volume"]

    hl_range = high - low
    position = np.where(
        hl_range != 0,
        (close - low) / hl_range,
        0.5  # Avoid division by zero
    )
    flow = position * vol
    return flow.tail(period).sum()

def accumulation_screen(csv_dir, min_vol_ratio=1.5, min_price=5.0):
    """Screen: rising volume + positive money flow"""
    tickers = [f[:-4].upper() for f in os.listdir(csv_dir) 
               if f.endswith(".csv")]
    results = []

    for t in tickers:
        try:
            df = load_price_data(csv_dir, t)
            if df["Close"].iloc[-1] < min_price:
                continue

            vr = volume_ratio(df)
            mf = money_flow_sum(df)

            if vr >= min_vol_ratio and mf > 0:
                results.append({
                    "ticker": t,
                    "volume_ratio": round(vr, 2),
                    "money_flow": int(mf)
                })
        except Exception as e:
            print(f"Error on {t}: {e}")

    return sorted(results, key=lambda x: x["volume_ratio"], reverse=True)

# Run it
accum_stocks = accumulation_screen("./stock_data")
print(f"Found {len(accum_stocks)} accumulation candidates")

When to use: When you want to catch early moves before the broader market notices. Volume leads price.

7. Screener #3: Earnings Surprise (When Reality Beats Expectations)

Markets react strongly when earnings differ from estimates.

\text{Surprise\%} = \frac{\text{EPS}{\text{actual}} - \text{EPS}{\text{estimate}}}{\text{EPS}_{\text{estimate}}} \cdot 100

Positive surprises often lead to repricing.

import sqlite3
from datetime import datetime

def get_latest_earnings(db, ticker):
    """Fetch most recent earnings from database"""
    cur = db.cursor()
    cur.execute("""
        SELECT quarter_date, eps, estimated_eps
        FROM fundamentals
        WHERE ticker = ?
        ORDER BY quarter_date DESC
        LIMIT 1
    """, (ticker,))

    row = cur.fetchone()
    if not row:
        return None

    qdate, eps, est = row
    surprise_pct = (eps - est) / est * 100 if est else 0
    return {
        "date": qdate,
        "eps": eps,
        "est": est,
        "surprise_pct": surprise_pct
    }

def earnings_surprise_screen(csv_dir, db_path, 
                            min_surprise=10, max_days_old=45):
    """Screen: positive recent earnings surprise"""
    db = sqlite3.connect(db_path)
    tickers = [f[:-4].upper() for f in os.listdir(csv_dir) 
               if f.endswith(".csv")]
    results = []

    for t in tickers:
        try:
            e = get_latest_earnings(db, t)
            if not e:
                continue

            days_old = (datetime.now() - 
                       datetime.fromisoformat(e["date"])).days

            if e["surprise_pct"] >= min_surprise and days_old <= max_days_old:
                results.append({
                    "ticker": t,
                    "surprise_pct": round(e["surprise_pct"], 2),
                    "days_since_earnings": days_old
                })
        except Exception as ex:
            print(f"Error on {t}: {ex}")

    return sorted(results, key=lambda x: x["surprise_pct"], reverse=True)

# Run it
earnings_stocks = earnings_surprise_screen("./stock_data", "stocks.db")
print(f"Found {len(earnings_stocks)} earnings surprise candidates")

When to use: Right after earnings season. Markets often misprice surprises—you're looking for the market catching up.

8. Screener #4: Valuation (Price Per Unit of Business)

Simple valuation metrics avoid overpaying.

Price-to-Sales Ratio:

\text{P/S} = \frac{\text{Market Cap}}{\text{Annual Revenue}}

Price-to-Earnings Ratio:

\text{P/E} = \frac{\text{Market Cap}}{\text{Net Income}}

Lower ratios are cheaper. But cheap can mean "dying business", so combine with growth/momentum.

def get_latest_fundamentals(db, ticker):
    """Fetch latest quarter fundamentals"""
    cur = db.cursor()
    cur.execute("""
        SELECT market_cap, revenue, net_income
        FROM fundamentals
        WHERE ticker = ?
        ORDER BY quarter_date DESC
        LIMIT 1
    """, (ticker,))

    row = cur.fetchone()
    if not row:
        return None
    return {
        "market_cap": row[0],
        "revenue": row[1],
        "net_income": row[2]
    }

def valuation_screen(csv_dir, db_path, max_ps=2.0, 
                     max_pe=15.0, min_price=5.0):
    """Screen: low P/S and P/E ratios"""
    db = sqlite3.connect(db_path)
    tickers = [f[:-4].upper() for f in os.listdir(csv_dir) 
               if f.endswith(".csv")]
    results = []

    for t in tickers:
        try:
            f = get_latest_fundamentals(db, t)
            if not f:
                continue

            mc = f["market_cap"]
            rev = f["revenue"]
            ni = f["net_income"]

            ps = mc / rev if rev > 0 else float("inf")
            pe = mc / ni if ni > 0 else float("inf")

            if ps > max_ps or pe > max_pe:
                continue

            df = load_price_data(csv_dir, t)
            price = df["Close"].iloc[-1]
            if price < min_price:
                continue

            results.append({
                "ticker": t,
                "ps": round(ps, 2),
                "pe": round(pe, 2),
                "price": round(price, 2)
            })
        except Exception as ex:
            print(f"Error on {t}: {ex}")

    return sorted(results, key=lambda x: x["ps"])

# Run it
value_stocks = valuation_screen("./stock_data", "stocks.db")
print(f"Found {len(value_stocks)} value candidates")

When to use: When you want a "margin of safety"—cheap stocks provide cushion if things go wrong.

9. Screener #5: Combined Signal (Weighted Score)

Now you have four independent signals. Combine them with a weighted linear combination:

\text{Score} = w_1 s_1 + w_2 s_2 + w_3 s_3 + w_4 s_4

Where:

$s_i$ = normalized score from signal i (scaled to 0–1 range)
$w_i$ = weight for signal i (must sum to 1)

Example: equal weights $w_i = 0.25$ for each signal.

def normalize(value, min_val, max_val, invert=False):
    """Normalize value to [0, 1] range"""
    if max_val == min_val:
        return 0.5
    x = (value - min_val) / (max_val - min_val)
    x = max(0, min(1, x))  # Clamp to [0, 1]
    return 1 - x if invert else x

def combine_signals(momentum_dict, volume_dict, 
                   earnings_dict, valuation_dict):
    """Combine four signals into one composite score"""
    all_tickers = (set(momentum_dict) | set(volume_dict) | 
                   set(earnings_dict) | set(valuation_dict))
    scores = {}

    for t in all_tickers:
        score = 0

        # Momentum: 0 to 50 percent (25% weight)
        if t in momentum_dict:
            m = momentum_dict[t]
            score += normalize(m, 0, 50) * 25

        # Volume ratio: 1 to 3 (25% weight)
        if t in volume_dict:
            v = volume_dict[t]
            score += normalize(v, 1, 3) * 25

        # Earnings surprise: 0 to 50 percent (25% weight)
        if t in earnings_dict:
            e = earnings_dict[t]
            score += normalize(e, 0, 50) * 25

        # Valuation: lower P/S better, inverted (25% weight)
        if t in valuation_dict:
            ps = valuation_dict[t]
            score += normalize(ps, 0, 5, invert=True) * 25

        scores[t] = score

    ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
    return [{"ticker": t, "score": round(s, 1)} for t, s in ranked]

# Run all screeners and combine
momentum_results = momentum_screen("./stock_data")
volume_results = accumulation_screen("./stock_data")
earnings_results = earnings_surprise_screen("./stock_data", "stocks.db")
valuation_results = valuation_screen("./stock_data", "stocks.db")

# Convert to dictionaries keyed by ticker
momentum_dict = {r["ticker"]: r["momentum_20d_pct"] for r in momentum_results}
volume_dict = {r["ticker"]: r["volume_ratio"] for r in volume_results}
earnings_dict = {r["ticker"]: r["surprise_pct"] for r in earnings_results}
valuation_dict = {r["ticker"]: r["ps"] for r in valuation_results}

# Get composite scores
combined = combine_signals(momentum_dict, volume_dict, 
                          earnings_dict, valuation_dict)

print("\n🏆 TOP 10 STOCKS (COMPOSITE SCORE):")
for i, result in enumerate(combined[:10], 1):
    print(f"{i:2}. {result['ticker']:6} — Score: {result['score']:>6.1f}/100")

You now have your own best stock screener that:

Uses clear math
Uses data you control
Can be tuned in a few lines of code

10. Portfolio Context: Which Screener Should You Emphasize?

You don't have to use all signals equally. Your weights depend on what you own.

If you own bonds + gold (defensive)

Focus on:

Momentum (high weight)
Volume accumulation (high weight)

Logic: You want high-growth, high-beta assets to offset your defensive base.

If you already own a lot of tech

Focus on:

Valuation (high weight)
Correlation filter (not a weight, but mandatory)

Logic: Reduce concentration risk rather than stacking more tech. Screen for uncorrelated sectors.

If you're just starting

Use equal weights (25% each).

Then gradually shift weights as you learn what matches your trading style.

11. What Your Broker Won't Tell You

Most broker "stock screener" tools:

❌ Use fixed rules not tuned to your portfolio
❌ Don't show you the actual math
❌ Encourage overtrading with flashy filters
❌ Change rules without telling you

You can do better:

✅ Write the math yourself
✅ Implement it in Python
✅ Run it weekly on your own data
✅ Check correlation before buying
✅ Keep a log of rule changes and results
✅ Know exactly what you own

12. Quick Checklist: Your Own Stock Screener

[ ] You understand inputs: price, volume, fundamentals
[ ] You compute returns with $Rn(t)=P(t)−P(t−n)P(t−n)R_n(t) = \frac{P(t) - P(t-n)}{P(t-n)}$
[ ] You use moving averages to smooth noise
[ ] You confirm moves with volume (not price alone)
[ ] You check earnings surprise (not just headlines)
[ ] You use valuation ratios to avoid overpaying
[ ] You combine signals into a weighted score
[ ] You check correlation with your existing portfolio
[ ] You only change rules slowly and record impact
[ ] You keep your data local (CSV + SQLite)

Next Steps

Set up your data pipeline: Download historical CSV files for 50–100 stocks
Populate SQLite: Add quarterly fundamentals from public sources
Run each screener independently: Build intuition for what each detects
Combine with your weights: Start with equal weights, then tune
Backtest: Log your picks and track win rates
Iterate: Adjust weights monthly based on what works

Good luck.

Can simple 4 Core, 16 GB RAM reach 1000 tps?

Supreet Sethi — Thu, 24 Apr 2025 07:14:23 +0000

Databases are tricky. They hold more state as we move towards stateless architectures. Kubernetes setups often start with stated intention of providing governance over all infrastructure until they encounter Databases. Database also turns out to be the largest cost in many workloads, because of compute requirements, DR, backup and skill.

Database costs are fixed, hard to reduce and form almost 1/3rd of Cloud costs for many customers. As a cofounder of startup or VP of Cloud in a Bank, you start to think of this cost item as the cost of doing business and this has always been. Until it isn't. Database like any other application is highly configurable and we look at major levers to increase performance where a 138$ a month database performs like a $3000 a month. A notional Porsche 917K at price of Renault Clio

We cannot start from managed options here clearly here because it it just a more expensive start. Many of the activities and learnings are deployable on managed databases but we want to start at garage sale price for our shiny high octane database. We start with an ordinary node and in case of E2E Cloud with a 4 core 16 GB c3.16GB-288 instance which costs $66 a month.

We will truly snowflake the database by naming the server david. We will turn these learnings in K8S setup later desnowflaking it but for now we want it - to be one of a kind.

$ sudo hostnamectl set-hostname David
$ sudo sed -i "s/$(hostname)/david/g" /etc/hosts

Now let's setup SysBench as our reference benchmark to test david like a dojo testing a pup pupil until it reaches mental and physical toughness.

Firstly we will setup MySQL and SysBench for testing and run the control test cycle. Control provides us the baseline performance, which will show us how much we were able to push the machine to the frontier of becoming a $3000 dollar DB server.

david$ sudo apt install sysbench mysql-server
I will leave setting credentials on MySQL to the reader. We run the benchmark like this;

david$ sysbench oltp_read_write \
    --tables=10 \
    --table-size=1000000 \
    --threads=4 \
    --time=60 \
    --db-driver=mysql \
    --mysql-db=sbtest \
    --mysql-user=sbtest \
    --mysql-password=password \
    run

We will anchor the readings as a baseline here with key items that matter in read/write mixed performance

Transaction/Sec which baselines at 112.24 (higher is better)
Queries/Sec at 2,244.84 (higher is better)
Avg Latency (ms) 35.63 (lower is better)
95th %ile Latency (ms) 94.10 (lower is better)
Min Latency (ms) 4.31 (lower is better)
Max Latency (ms) 1,065.46 (lower is better)

We will be improving on each of these key performance numbers as we go through retrofitting for Porsche makeover. Before saying that we improve these numbers, we should level-set that these are quite good numbers to start and often meet or exceed customer requirements in vast majority of use-cases.

Step 1: InnoDB Block Sizes and File I/O

Block sizes of choice in Database and Filesystem is single most important change that you could do to improve the database performance. Ostensibly XFS is the best filesystem for database performance however we can't larger block sizes beyond 4k because that requires Linux kernel recompilation. However if we place Innodb block size to be 16k that aligns neatly with 4 blocks for 4k size thus it is still a performance win.

Add to /etc/mysql/mysql.conf.d/mysqld.cnf  
innodb_page_size=16384

With a single line change we have almost doubled our performance from 66$ instance. To reach this performance without config change we would have expended a budget of 3 times current spend. We have thus increased value of our setup by $132.

Step 2: Swapiness and network parameters

Reference Only – Not Applied in Benchmarks

There are several popular Swap and network related sysctl parameters which change kernel behaviour at runtime. Network ones are irrelevant since in my setup networking is loopback which is sufficiently fast in normal coarse. Swap settings are risky, or harmful or both in virtualised environment. We include them for sake of completeness but we did not apply and test in our environment.

# MySQL Swap Settings
vm.swappiness = 10
vm.vfs_cache_pressure = 50

# File System Settings
fs.file-max = 65536
fs.aio-max-nr = 1048576

# Network Settings - General
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 4000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# TCP Settings
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_slow_start_after_idle = 0

# Memory Management for Database Workloads
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.dirty_expire_centisecs = 500

It is imperative that you should be testing your workload with each of these changes before ever bringing them to production.

Step 3: MySQL tune-ups

MySQL has history of being used as database of the Internet. Almost every site large and small hosts on MySQL. With that there a myriad of features, bells and whistles, dials and buttons. We decided to press few of these buttons to extract MySQL performance.

Add to /etc/mysql/mysql.conf.d/mysqld.cnf  

# InnoDB Buffer Pool Settings
innodb_buffer_pool_size = 12G
innodb_buffer_pool_instances = 4
innodb_old_blocks_time = 1000

# Transaction and Log Settings
innodb_log_file_size = 3G
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 1
sync_binlog = 1

# I/O and Threading
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
innodb_read_io_threads = 4
innodb_write_io_threads = 4
innodb_page_cleaners = 4

# File and Storage Settings
innodb_flush_method = O_DIRECT
innodb_file_per_table = ON
innodb_open_files = 400
table_definition_cache = 1400
table_open_cache = 2000

# Query Optimization
join_buffer_size = 2M
sort_buffer_size = 4M
read_rnd_buffer_size = 2M
read_buffer_size = 1M
tmp_table_size = 64M
max_heap_table_size = 64M

# Connection and Thread Settings
max_connections = 150
thread_cache_size = 16
max_prepared_stmt_count = 16384

These numbers have been worked out on specific CPU, RAM and Disk Configuration we used. We assumed that MySQL will be the only tenant to this system and it can fully utilise the resources for maximum performance.

With these valuable config change, we once again doubled the performance of our system which still is being charged at $66 per month. Now notional the value we extract out of it is close to 16 core system which could cost us around $660, unlocking value of $600.

Step 4: Deadlines and IO Schedulers

IO schedulers are important but not that important in SSD and NVMe world today. Both these kind of devices have sophisticated way to schedule the IO workload and placing priority on them. While Ubuntu by default includes two schedulers none and mq-deadline. We stick to default. No changes. No performance gains.

Step 5: More NVMe Controllers, More IOPS, More Throughput

At this time, the simplest thing to do is add more volumes which increase IOPS and throughput. We have two options for doing it, first and simplest is to put single large disk and reap high IOPS,

256 GB -> 5000 IOPS
512 GB -> 10000 IOPS
1TB -> 20000 IOPS

The second and more interesting method is to setup RAID - Redundant Array of Inexpensive Disks or in this Redundant Array of Inexpensive NVMe (RAIN). We have to increase our budget by $66 to do this. We take 4 256 GB Volumes at E2E Cloud priced at 8 dollars per Volume per month. Thus our revised total cost of ownership for this new setup is $138 per month. Once attached these volumes will show up as available disks vd*

david$ lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sr0     11:0    1   366K  0 rom
vda    253:0    0 139.7G  0 disk
├─vda1 253:1    0     1M  0 part
└─vda2 253:2    0 139.7G  0 part /
vdb    253:16   0 232.8G  0 disk
vdc    253:32   0 232.8G  0 disk
vdd    253:48   0 232.8G  0 disk
vde    253:64   0 232.8G  0 disk

Now we have four disks available to be setup as RAID, instead of using mdadm based setup, we use zfs striped pool which is quite easy to setup

david$  sudo apt install zfsutils-linux
david$  sudo zpool create mysqlpool /dev/vdb /dev/vdc /dev/vdd /dev/vde 
We can also place some performance garnishes in ZFS to get slight improvements

# Disable access time updates
david$ sudo zfs set atime=off mysqlpool

# Set optimal record size for MySQL
david$ sudo zfs set recordsize=16K mysqlpool

# Cache only metadata to prevent double-caching
david$ sudo zfs set primarycache=metadata mysqlpool

# Enable LZ4 compression for better performance/storage ratio
david$ sudo zfs set compression=lz4 mysqlpool
Finally we can stop mysql server, move the directory /var/lib/mysql and mount new ZFS pool as new /var/lib/mysql

david$ sudo systemctl stop mysql
david$ sudo mv /var/lib/mysql /var/lib/mysql.old
david$ sudo zfs set mountpoint=/var/lib/mysql mysqlpool
david$ sudo rsync -av /var/lib/mysql.old/ /var/lib/mysql/
david$ sudo chown -R mysql:mysql /var/lib/mysql
david$ sudo systemctl restart mysql

After this massive exercise, looking much like entire cars being reassembled on a race track, we are back to running benchmark again. Anchor this that we have increased IOPS count 4x which is 20K now compared to 5K in default system.

Now we again more than doubled ourselves of all key parameters bringing performance in stratospheric levels. This place is where "enterprise" databases live where core counts float in rivers of RAM. This performance levels are not available anything below $3000 in a cloud provider and the managed version being more expensive and serverless version even further expensive.

Benchmark overview

We have improved our system in every key performance point for this benchmark and here is comparison in a snapshot. While this is great performance wise. We can also forgo some performance for higher resilience setup by using RAIDZ instead of Striped to increase resilience with same setup with marginal 18% loss of performance.

Now that we have these improvements how would we integrate it back to K8S setup.

Step 6: Kubernetes Setup with ZFS CSI Driver Method

This approach uses a Container Storage Interface (CSI) driver to dynamically provision ZFS volumes.

Install ZFS on Worker Nodes

Apply the ZFS installation to each worker node that will host MySQL data:

# Run these commands on each designated worker node
sudo apt update
sudo apt install -y zfsutils-linux

# Create ZFS pool as before
sudo zpool create mysqlpool /dev/vdb /dev/vdc /dev/vdd /dev/vde

# Apply MySQL performance optimizations
sudo zfs set atime=off mysqlpool
sudo zfs set recordsize=16K mysqlpool
sudo zfs set primarycache=metadata mysqlpool
sudo zfs set compression=lz4 mysqlpool

Label the Nodes with ZFS Storage

$ kubectl label nodes <node-name> storage=zfs

Install ZFS CSI Driver

# Clone the ZFS CSI driver repository
git clone https://github.com/democratic-csi/democratic-csi.git
cd democratic-csi

# Install using Helm
helm repo add democratic-csi https://democratic-csi.github.io/charts/
helm repo update

# Install the ZFS CSI driver
helm install zfs-csi democratic-csi/democratic-csi \
  --namespace kube-system \
  --create-namespace \
  -f ./examples/zfs-local-dataset.yaml

Create Storage Class for MySQL

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zfs-mysql-sc
provisioner: org.democratic-csi.zfs
allowVolumeExpansion: true
parameters:
  fsType: zfs
  poolName: mysqlpool
  recordsize: "16k"
  primarycache: "metadata"
  compression: "lz4"
  atime: "off"

Deploy MySQL as a StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: database
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: mysql
  replicas: 1
  template:
    metadata:
      labels:
        app: mysql
    spec:
      nodeSelector:
        storage: zfs
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: zfs-mysql-sc
      resources:
        requests:
          storage: 50Gi

Create MySQL Service

apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: database
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None

Conclusion: From Garage Sale Renault Clio to Enterprise Gold Prosche 917K

Databases may be the last bastion of state in an increasingly stateless world—but that doesn’t mean they must also be the last bastion of uncontrolled cost. What we’ve demonstrated is a principled, low-budget approach to extract maximum performance per dollar by:

Leveraging block-level tuning and file system optimizations.
Exploiting MySQL configuration levers to align with system limits.
Deploying ZFS RAID setups for high IOPS, with compression and smart caching.
Seamlessly porting all these wins into Kubernetes using a CSI-driven ZFS architecture.

At every step, we doubled performance, not by spending more, but by knowing more.

What this shows is that infrastructure as craft—when applied with deliberate design and mechanical sympathy—can give you serverless-grade performance without serverless-grade cost.

This isn’t just a benchmark hack—it’s a mindset shift.

This material is also available at

DEV Community: Supreet Sethi

Why Stockfish is So Good (and How You Could Write a Chess Engine)

The Tournament Results Tell the Story

NNUE Was The Breakthrough Nobody Expected

Two Paradigms Compete: Alpha-Beta vs. Monte Carlo Tree Search

Alpha-Beta Search with NNUE (Stockfish's Approach)

MCTS with Deep Neural Networks (Leela Chess Zero)

Why C++ (And Sometimes Rust)

Open-Source Development Beats Proprietary

Other Key Technical Insights

If You Want To Build a Chess Engine: Design Sweet Spots

Tier 1: A Respectable Hobbyist Engine (1400-1800 Elo)

Tier 2: A Competitive Amateur Engine (2000-2500 Elo)

Tier 3a: A Strong Club Engine (~3000 lines, 2400-2600 Elo)

Tier 3b: An Intermediate Engine (~5000 lines, 2800-3200 Elo)

Tier 3c: An Elite Engine (~10,000+ lines, 3000+ Elo)

Code Complexity vs. Elo Gain (Diminishing Returns)

The Real Lesson

Further Reading

Why I Built a NAS in 2026: When Your Vendor Abandons You

The Real Cost of Cloud

Why Build Instead of Buy?

The Hardware Choice: NVMe vs HDD

The Software: Choosing Freedom

The Economics

What Sovereignty Means

Who Should Build?

Why 2026 Is Different

The Reality Check

What I Built

The Conclusion

Digital Transformation: Why 70% Fail; And How 30% Win

A Data-Backed Guide to Why 70% Fail and How Winners Get 80% Odds

THE BRUTAL FACTS

THE SIX SUCCESS FACTORS

Companies that nail ALL SIX achieve 80% success rate (vs. 30% baseline)

WINNERS: WHAT THEY DID RIGHT

Walmart (+62.3% P/E, 2020–2025)

Amazon (1995–Present, Continuous Pivot)

Netflix (2007–2025, Multi-Pivot)

Microsoft (2014–2025, Cloud Pivot Under Satya Nadella)

LOSERS: WHAT THEY GOT WRONG

Target (–38.2% P/E, 2019–2025)

GE Digital ($5B+ sunk cost, then divested)

Sears & Blockbuster (→ Bankruptcy, 99%+ value destruction)

THE SCORECARD: 6 FACTORS ACROSS 8 COMPANIES

THE MATH THAT MATTERS

THE FORMULA (ULTRA-CRISP)

FINAL TAKEAWAY

Don't Touch Your Trade: The Most Underrated Strategy in Investing

The Uncomfortable Truth: Activity Is the Enemy

Why Your Developer Brain Backfires

The Actual Edge: Do Nothing

Events vs. Signals: The Mental Model That Changes Everything

A Rule You Can Adopt Monday Morning

How to Code This Into Your System

The Payoff: Better Returns, Better Sleep

The Idempotence Principle

The One Edge You Actually Need

I Hired AWS SAs for 5 Years. Now That I've Left, I Can Tell You the Truth

The Uncomfortable Truth

What Happens in the Room

The Training Industrial Complex (Or: Why Your $500 Course Hurt Your Chances)

What You Actually Need

The Real Test: A Rugby Social Network Migration

Why "Describe Every Component" Actually Kills Your Interview

How to Actually Prepare (The Right Way)

Final Thoughts

The Melbourne Talent War: A Conversation

Forget Wall Street: Here’s How to Create the Best Stock Screener Yourself

Stock Screeners for Math People: Build Your Own

1. What Is a Stock Screener, Really?

2. Why the "Best Screener" Depends on What You Already Own

Correlation Between Return Series

The Real Problem

3. Core Math Building Blocks

3.1 Returns: Rate of Change

3.2 Moving Averages: Smoothing Noise

3.3 Volume Ratio: Conviction Metric

4. Your Data: CSV + SQLite