DEV Community: ZyVOP

How to Make Real Music With Suno AI in 2026 — Even If You've Never Written a Song

ZyVOP — Thu, 18 Jun 2026 02:02:16 +0000

Something strange happened to music in the last two years.

The barrier that kept most people out — you need an instrument, years of practice, studio access, a producer who believes in you — quietly disappeared. Not for everyone equally, not without tradeoffs, and not in a way the music industry has fully figured out what to do with. But the barrier is gone.

Suno AI launched publicly in late 2023. By 2026, it has 2 million paid subscribers and generates 7 million tracks every single day. The V5 model, released in September 2025, produces audio with what the company called — with some justification — "the largest quality leap to date." The vocals sound human. The production sounds professional. And the whole thing runs in your browser.

The tool is genuinely accessible. The gap between people who make good things with it and people who make mediocre things with it is almost entirely in how they use it, not in whether they have musical talent. This tutorial closes that gap.

What Suno Actually Does

Suno takes a text description and generates a complete song: lyrics, vocals, melody, instrumentation, arrangement, and production — all in under 60 seconds.

Not a loop. Not a beat. A song with a verse, chorus, bridge, and ending that sounds like something a real band recorded, in whatever genre you describe.

The V5 model supports tracks up to 8 minutes long and handles over 1,200 musical genres. It generates in both directions: you can let Suno write the lyrics from your description, or you can write your own lyrics and have Suno perform and produce them. You can extend existing tracks, remix them in a different style, or generate cover interpretations of conceptual ideas.

It runs at suno.com. Nothing to install.

Plans: What You Get and What You Actually Need

Free plan: 50 credits per day, roughly 10 songs. No commercial use rights. Access to the V4 model. Plenty for learning and personal projects.

Pro ($10/month): 2,500 credits per month (about 500 songs), full commercial rights, V5 model access, and the Song Editor with 12-stem separation. This is the sweet spot for content creators, YouTubers, and anyone producing for clients.

Premier ($30/month): 10,000 credits, MIDI export, and Suno Studio — a full AI-native DAW with timeline editing. For anyone running a music-focused business or producing at agency scale.

Start on the free plan. The moment you want to use tracks commercially — in a YouTube video, a client project, a product — upgrade to Pro. The commercial rights question matters; we'll come back to it.

Your First Song: Simple Mode

Simple Mode is exactly what it sounds like. Open Suno, click Create, and type a description in the text field. Suno writes the lyrics, chooses the structure, and produces the full track.

The mistake most first-timers make is being too vague:

❌ "A happy song about summer"

This produces something generic that could have been generated for anyone. Useful for understanding what the tool can do. Not useful for making something you'd actually listen to twice.

A better first prompt describes something specific — a scene, a feeling, a contradiction, a detail:

✅ "Upbeat indie pop about the moment you realise you're over someone, driving home late at night with the windows down, guitar-driven, warm female vocals, nostalgic but hopeful, 2000s alt-pop production"

Notice what's in there: a specific emotional situation, a physical setting, a clear genre, a vocal type, a production reference, and a tonal direction that's more interesting than "happy." The specificity is what separates generic from good.

Going Deeper: Custom Mode and Structure Tags

Custom Mode is where Suno becomes genuinely creative. Instead of describing a song and letting Suno write everything, you write the lyrics yourself and give Suno specific instructions for how to perform and produce them.

Structure tags tell Suno how to treat different sections. The key ones:

[Intro]
[Verse]
[Pre-Chorus]
[Chorus]
[Bridge]
[Outro]
[Instrumental Break]
[Build]
[Drop]

Place these in your lyrics as labels. Suno reads the structure and composes accordingly.

A working example:

[Style: slow-burn R&B, soulful male vocals, late-night jazz club atmosphere, 
sparse production, piano and brushed drums, intimate recording]

[Verse 1]
City lights through dirty glass
Another glass of something dark
I keep your voicemail just to hear
The version of you in the dark

[Chorus]
But you've already moved on
I'm just the last song on your phone
You've already moved on
And I'm still here, still alone

[Bridge]
Maybe by spring I'll stop counting the days
Maybe by spring I'll forget how you say my name

[Outro]
(fade on piano, no vocals)

The style tag at the top is your production brief. The structure tags tell Suno how to build the song. The last line in the outro is a performance instruction — Suno reads and follows directions embedded in brackets.

The Prompt Formula That Actually Works

After testing and tracking results across hundreds of generations, the community has converged on a prompt structure that consistently outperforms vague descriptions. It's a six-element formula:

[Genre] + [Vocals] + [Lead Instrument] + [Production Style] + [Mood] + [Tempo]

In practice:

Element	Example options
Genre	indie folk, lo-fi hip-hop, cinematic orchestral, dark cabaret, bedroom pop, afrobeat
Vocals	warm female vocals, raspy male vocals, harmonised choir, no vocals instrumental, breathy whisper
Lead Instrument	acoustic guitar, electric piano, synthesiser pads, cello, trumpet
Production Style	1970s studio warmth, modern clean mix, tape-saturated, lo-fi cassette, radio-ready
Mood	melancholic, euphoric, tense, yearning, triumphant, eerie, intimate
Tempo	slow and languid, mid-tempo groove, driving beat, 90 BPM, 140 BPM

A prompt built from this formula:

"Cinematic orchestral, no vocals instrumental, strings and solo piano, Hans Zimmer-inspired production, melancholic and building, slow and expansive, emotional crescendo in the final third"

vs.

"A sad instrumental song"

Same intent. Radically different outputs.

Stop Making Songs. Start Building Artists.

This is the mental shift that separates people who get consistently good results from people who get lucky occasionally.

A real music producer doesn't walk into a studio with no brief. They know the artist — their genre, their vocal style, their sonic signature, their tempo range, their lyrical themes. Every song flows from that established identity.

The same applies in Suno. If you build a consistent artist identity, every generation starts from a known foundation. The outputs become recognisable and coherent rather than random.

How to build your Sound DNA:

Step 1 — Define the artist. Give them a genre, a vocal style, a signature instrument, an era reference, a mood territory, and an energy range. Write this down.

Example:

ARTIST: "The Hollow Hours"
Genre: Atmospheric indie folk
Vocals: Layered female harmonies, no falsetto, intimate and close-mic'd
Signature: Fingerpicked acoustic guitar + ambient synth pads underneath
Era reference: 2010s Bon Iver and Fleet Foxes production, but more minimal
Mood: Melancholic, hopeful undertones, never aggressive
BPM range: 70–100
Lyrical themes: Distance, memory, time, water and seasons as metaphors

Step 2 — Build 3–5 prompt templates from that Sound DNA, one per song "type":

Upbeat: "Atmospheric indie folk, layered female harmonies close-mic'd, fingerpicked 
acoustic guitar with synth pads, Fleet Foxes 2010s production aesthetic, hopeful and 
warm, 95 BPM, gentle build to a full vocal chorus"

Slow: "Atmospheric indie folk, solo female vocal whisper-to-full, fingerpicked acoustic 
guitar only, intimate bedroom recording feel, melancholic and still, 72 BPM, no drums 
until chorus"

Step 3 — Test and refine. Generate five songs from each template. Listen critically. When something's wrong — the vocals are too bright, the production is too busy — adjust one element and regenerate. When something's right, write down what worked.

After 10–15 test songs, your templates are calibrated. Every future song starts from a known-good foundation, and what varies is the lyrics and specific song concept — not the entire sonic identity.

Writing Lyrics That Suno Performs Well

Suno performs some types of lyrics significantly better than others. Understanding what works saves you from the frustration of technically well-structured lyrics that produce flat performances.

What Suno handles brilliantly:

Clear emotional through-lines (songs about a specific feeling tend to perform better than abstract or narrative-heavy lyrics)
Phonetically interesting language (words with open vowel sounds on the high notes, hard consonants for rhythmic emphasis)
Standard song structures (verse/chorus patterns it has seen millions of times)

What still trips it up:

Very long verses without natural rhythm breaks
Complex narrative that requires context to understand emotionally
Specific proper nouns that don't carry phonetic weight ("Bartholomew" is a harder word to sing than "summer")
Lyrics where the syllable count per line varies wildly

A practical tip from the community: write your lyrics, then read them aloud rhythmically before pasting them in. If they feel awkward spoken aloud with a beat, they'll feel awkward sung. Adjust the syllable count and stress patterns first.

Advanced Features Worth Knowing

Song Editor and 12-Stem Separation (Pro)

The Song Editor lets you split a generated track into up to 12 individual stems — vocals, lead guitar, bass, drums, synths, backing vocals, and so on. Each stem can be downloaded separately.

Why does this matter? It means AI-generated music is now a starting point for human production, not a finished product you accept or reject. You can take the backing track from a Suno generation, strip the original AI vocals, and record your own. You can take the drum stem, import it into your DAW, and build a different arrangement around it. The stems are commercially licensed with a Pro subscription.

This is where AI music generation connects to traditional music production rather than replacing it.

Extending Tracks

Any Suno-generated track can be extended beyond its initial length. Click the track, select Extend, and describe what should happen next: "build into a bigger chorus," "break down to just piano and vocals," "add a key change and reprise the opening melody."

The community tip worth following: don't extend more than two or three times. After multiple extensions, tracks tend to develop audio artifacts and lose coherence. Generate a new track with a longer initial description rather than extending endlessly.

Covers and Style Remixes

You can generate a track and then ask Suno to reimagine it in a completely different style. A slow folk ballad can become a drum-and-bass remix. A hip-hop track can be reinterpreted as a jazz standard. This is useful for content creators who need music to fit specific moods across the same project, and for musicians exploring how a composition sounds in different genres before committing to an arrangement.

Suno Studio (Premier)

Suno Studio is a full DAW interface built directly into Suno. Timeline editing, MIDI export, layered arrangement controls — it turns Suno from a generation tool into a production environment. This is the tier for anyone building a music business around AI-generated content.

Who This Is Actually For (And Who It Isn't)

Content creators are the clearest beneficiaries. YouTubers, podcasters, TikTok creators, and social media teams producing video content regularly have long faced the same problem: stock music is expensive, licensing is complex, and the tracks available never quite fit the specific mood. A Pro subscription at $10/month gives you 500 original, commercially licensed tracks per month that match whatever you describe. The economics are dramatically better than any licensing alternative.

Hobbyist musicians and songwriters who have always had song ideas but lacked the production skills to realise them now have a tool that closes that gap. If you can describe what you hear in your head, Suno can produce a version of it. Many musicians use it for ideation — generating multiple rough interpretations of a chord progression concept before committing to one direction.

Small businesses needing background music for retail, videos, apps, or events. Generated once, licensed properly with a Pro plan, used indefinitely.

What it isn't for: professional musicians trying to replace traditional production for high-stakes work. The ceiling of Suno's output quality is impressive, but a session musician who can respond in real time to direction, adjust for emotional nuance mid-performance, and bring genuine artistic interpretation to a piece is still a different thing. For music that needs to carry emotional weight that justifies close scrutiny — a film score, an album meant to stand on its own — AI-generated music is still a starting point or a layer, not the finished article.

Suno vs Udio: The Honest Comparison

Udio is Suno's main competitor and is genuinely worth knowing about. The two tools have distinct personalities.

Suno is faster, more structured, and better at following precise style directions. If you know what you want and can describe it, Suno tends to execute the brief more reliably.

Udio generates music with more variation and occasional surprising creativity, but with less predictability. Its model tends to interpret prompts more loosely, which produces unexpected gems and also unexpected misses.

The community consensus: use Suno for professional and commercial work where consistency matters. Use Udio when you want to be surprised — for inspiration, experimentation, and finding sounds you wouldn't have thought to ask for.

Both are worth having free accounts on. They complement rather than replace each other.

The Copyright Question You Need to Understand

Suno's terms of service are worth reading for one specific reason: commercial rights depend on your subscription level.

Free plan: personal use only. Tracks cannot be monetised or used commercially.
Pro and Premier: upon generation, Suno contractually assigns its rights in the output to you. You own the track commercially and can use it in monetised content, client work, and products.

One nuance: if your lyrics are original (you wrote them in Custom Mode), you hold copyright on the lyrics. Suno holds no claim on original creative input you bring to the generation. For instrumental tracks or tracks where Suno wrote the lyrics, the Pro plan assignment covers full commercial use.

The area that remains legally unsettled: what if a generated track sounds substantially similar to an existing song? This has been the subject of ongoing litigation the music industry has brought against AI music companies. Suno (like most generators) filters for direct replication, but similarity claims remain a live legal area. For most commercial uses, this risk is low and theoretical. For high-stakes projects, run tracks past a legal eye first.

Getting Started This Week

Here's the fastest path from zero to something you'd actually share:

Go to suno.com — create a free account in 30 seconds
Use the six-element prompt formula on your first generation: genre + vocals + instrument + production + mood + tempo
Generate four versions of the same prompt — Suno produces different results each time, and variety lets you choose the best interpretation
Try Custom Mode on your second session — write four lines of a verse, add [Chorus] and four chorus lines, put a style tag at the top
Extend the best track once to see how continuation works
Listen back-to-back with a comparable commercial track in the same genre — this calibrates your ear for what to refine in your prompts

By the end of a two-hour session, you'll have a working understanding of where Suno excels, where it needs guidance, and what your specific sound direction should be. That's worth more than any amount of reading about it.

Resources

Suno — start here, free account, no install
Udio — main alternative, stronger on experimentation
r/SunoAI — the most active community; prompt templates, model comparisons, and honest takes on what's working
Suno Prompts Pack — curated prompt templates by genre and mood
AI Video Bootcamp — Suno Complete Guide — deep dive on the commercial use case for video creators
Ideatomusic.com — Sound DNA Framework — the "stop making songs, start building artists" methodology in full

What genre or sound are you trying to make? Drop it in the comments with any context about the project and I'll help you build a starting prompt template for it.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

The Week AI Went Public: Everything That Just Happened and Why It Actually Matters

ZyVOP — Mon, 15 Jun 2026 05:23:02 +0000

Some weeks in tech move fast. This one was different. Between June 9 and June 13, 2026, enough happened to fill a month's worth of newsletters — and almost all of it is genuinely significant rather than hype dressed up in a press release.

A few things landed at once: Anthropic released the model it had been keeping behind locked doors, OpenAI crossed one billion monthly active users, the largest IPO in stock market history closed its first day up 25%, and China quietly started a production line that builds a humanoid robot every 15 minutes. Meanwhile, a GPT-5.6 checkpoint is circulating in developer channels with nobody from OpenAI officially saying a word.

This piece is an attempt to make sense of all of it — what happened, what it actually means, and what it signals about where the next six months go.

1. Anthropic Released the Model It Was Afraid to Release

On June 9, 2026, Anthropic did something it had been explicitly avoiding for the better part of a year: it handed the public a model from its "Mythos" tier.

Until that morning, Mythos-class models were accessible only through Project Glasswing, a controlled program serving a small number of cybersecurity researchers and government-adjacent organisations. The reason for the restriction was unusual candor from an AI lab: Anthropic genuinely believed Mythos was dangerous enough to require vetting before wider release.

What changed? The classifiers caught up. Anthropic ran an external bug bounty — over 1,000 hours of red-team jailbreak attempts — before the June 9 launch. No universal jailbreaks were found. External red-teaming organisations came up empty too. That cleared the internal bar for what Anthropic called a "public-safe version" of Mythos: Claude Fable 5.

The benchmarks are not modest. Fable 5 posts 80.3% on SWE-Bench Pro — a benchmark measuring real software engineering capability — while the next-best competitor sits 11 points behind. Andrej Karpathy, who rarely reaches for superlatives, called it "a major-version-bump-deserving step change forward." One team used it to complete a migration across a 50-million-line codebase in a single day.

There are two versions. Claude Fable 5 is the widely available one. Claude Mythos 5 is the same underlying model without the safety classifiers — and it's still restricted to Project Glasswing customers. The technical difference is essentially: Fable 5 can decline requests; Mythos 5 cannot.

The Pricing Situation Deserves Attention

Through June 22, Fable 5 is free on Pro, Max, Team, and seat-based Enterprise plans. After June 23, it requires usage credits at $10 per million input tokens and $50 per million output tokens. That's the sticking point. It's cheaper than Mythos Preview's premium pricing, but the shift from flat-rate to usage-based billing is landing badly with some enterprise customers.

The billing split is sharper than it first appears. Chatting on claude.ai stays flat-rate. Claude Code running directly in your terminal stays flat-rate. But run Claude through Zed — an interactive session with a human at the keyboard — and it bills against your credit, because it arrives through the Agent SDK. The meter follows the integration surface, not the human. First-party experiences stay subsidised; everything you build yourself becomes metered.

That's a coherent strategy. It's also the kind of thing that generates resentment when it happens without much warning, which brings us to a separate problem Anthropic is dealing with.

The Partner Trust Problem

The Information published a report this week documenting a pattern of Anthropic launching products that compete with its own partners, often without advance notice. The specific example: weeks before Claude Design launched in April, Anthropic asked Figma and Canva to participate as "launch partners" — even though the product directly competed with tools those companies sell.

This matters more than it might seem. Anthropic is simultaneously running an aggressive partner program, preparing for a Q4 IPO, and trying to convince large enterprises to sign long-term contracts. You cannot do all three while also blindsiding your partners with competitive launches. At minimum, enterprise procurement teams will want guarantees before they deepen their commitments. Those guarantees are harder to give with a pattern like this on the record.

2. The Largest IPO in History Just Happened — and It's Mostly an AI Story

On June 12, one day before today, SpaceX (ticker: SPCX) closed its first day of trading on Nasdaq at $168.70, up roughly 25% from its $135 IPO price. Market cap at close: approximately $1.77 trillion. That makes it the largest IPO in stock market history, full stop.

The first-day performance landed in what analysts had described beforehand as the "orderly debut" scenario — a 25% pop without euphoric excess. Open above $150? Market endorses the 90x EBITDA multiples that AI infrastructure companies are being priced at. SPCX opened at exactly $150 and extended through the session. That is as clean a validation signal as you get.

Why does this belong in an AI piece? Because SpaceX's February 2026 acquisition of xAI — which brought Grok, the X social network, and the Colossus data centers into the public entity — means SPCX is now, in part, a bet on xAI's AI infrastructure. The S-1 disclosed a $6.36 billion 2025 operating loss inside the xAI segment alone. Investors bought that willingly. The market priced a loss-making AI unit as an asset, not a liability.

That decision — made publicly, on the largest stage possible — sets the floor price for what Anthropic and OpenAI will ask for when their own S-1s go live. Both companies filed confidential S-1s in early June, just ahead of SpaceX's listing. Anthropic is targeting a raise above $60 billion at a $965 billion valuation. OpenAI is eyeing Q4. SPCX's clean debut gives both of them permission to move.

The Revenue Recognition Wrinkle

One thing that will matter in the coming months: Anthropic and OpenAI count revenue differently. Anthropic treats itself as the principal in cloud reseller transactions and recognises gross revenue. OpenAI nets out Microsoft's cut. Bank of America estimates the cloud partner payments at up to $6.4 billion for Anthropic in 2026 — money that would be excluded from revenue under OpenAI's approach. If the SEC requires harmonised treatment before either company lists, one of them will see its headline revenue number shrink, visibly, overnight.

Anthropic's annualised revenue is currently reported at figures between $30 billion (Reuters) and $47 billion (Sacra), depending entirely on which methodology you apply. That's a meaningful spread when you're pricing an IPO.

3. Anthropic Overtook OpenAI in Business Adoption — and It's Mostly Thanks to Code

The May 2026 Ramp AI Index — which tracks spending across more than 50,000 US businesses using corporate card data — recorded something that would have seemed impossible two years ago: more American businesses now pay for Claude than for ChatGPT. Anthropic sits at 34.4% business adoption. OpenAI has fallen to 32.3%.

The climb is steep. Anthropic was at 0.03% in June 2023. It was at 7.94% by April 2025. It's at 34.44% now, roughly a 4x jump in a year. The engine behind most of that growth is Claude Code, which Anthropic describes as its fastest-growing product ever. One analysis estimated that 4% of all public GitHub commits worldwide are now being authored by Claude Code — double the share from one month earlier.

A different dataset tells a more cautious story. An IDC survey of over 1,000 organisations from March 2026 found only 19% reported extensive use of Claude, with OpenAI and Google still leading on depth of adoption. Both datasets are correct. They measure different things. Ramp captures whether a business has started paying for Claude at all. IDC captures how deeply it's integrated. Anthropic is winning the new customer race but still building toward depth-of-use parity with companies that adopted OpenAI products earlier.

4. ChatGPT Hit a Billion Users. Think About What That Means.

ChatGPT crossed one billion monthly active users in May 2026, confirmed by CNBC on June 12. It's the fastest any application in history has reached that threshold.

The number is worth sitting with for a moment. Facebook took roughly eight years to reach a billion monthly users. Instagram took about seven. ChatGPT, starting from its November 2022 launch, did it in under three and a half years. OpenAI's annualised revenue crossed $25 billion by May.

What's interesting about the adoption curve is what it suggests about the next phase. More than 60% of consumers now begin daily tasks inside AI platforms. That's not power-user behaviour anymore; it's mass-market habit. The question isn't whether AI assistants will be a mainstream consumer product — that question has been answered. The question is which platform locks in enough of those billion users to survive the inevitable commoditisation of the underlying models.

5. GPT-5.6 Is Leaking From Somewhere

A checkpoint identified as GPT-5.6 "kindle-alpha" surfaced in developer channels in early June, apparently through Codex-related testing paths. OpenAI has said nothing officially. But the signal is consistent across multiple independent sources: stronger reasoning, better coding, and — most notably — a marked improvement in vision, which had been a relative weak spot for GPT-5.x.

Earlier codenames "ember-alpha" and "beacon-alpha" were spotted in Codex rollout logs. Testers are reporting behaviour consistent with a 1.5 million token context window, about 43% above GPT-5.5. Polymarket traders are pricing 80-89% odds of a public release before June 30.

The honest read on this: a leaked checkpoint name is not a launched model. Codenames may refer to release candidates, canary builds, or experiments that never ship under those names. But the consistent signal across independent sources — stronger vision, improved reasoning, expanded context — is worth tracking. If GPT-5.6 ships before the end of June, it would land within days of Fable 5's free-trial window closing, creating a direct head-to-head moment for enterprise teams deciding which model to adopt on a paid basis. That timing, if intentional, is sharp.

Also: Microsoft shipped seven new AI models at Build 2026 this week, with MAI Thinking One drawing the most attention for its reasoning and coding performance. Alibaba launched Qwen 3.7 Plus, combining vision, language, and coding into a single multimodal system. The model release cadence in June 2026 is so compressed that it's genuinely difficult to keep track of what's shipping vs. what's rumoured.

6. China Is Building a Robot Every 15 Minutes

The AI story getting least attention from Western developers is probably the most structurally significant one. EngineAI, a Shenzhen robotics company founded in 2023, opened a 12,000 square metre factory in Shenzhen on June 1 and started shipping T800 humanoid robots at a rate designed to produce 10,000 units. The factory is geared to build one robot every 15 minutes.

Three years from founding to a production-rate humanoid robot factory. That's the timeline.

EngineAI filed confidentially for a Hong Kong IPO this week. It isn't alone. Unitree has filed for a $7 billion IPO. BYD-backed PaXini Tech is evaluating a listing. Linkerbot is targeting a $6 billion valuation. Approximately $22.6 billion has already been raised in Hong Kong IPOs in the relevant period, with AI and robotics names accounting for a disproportionate share.

The context behind this wave: China's central government listed embodied AI as a national priority in 2025 and included it as a top "new industry track" in the 15th Five-Year Plan (2026-2030). Beijing launched the "Robot+" initiative and an "AI + Manufacturing" roadmap targeting double China's manufacturing robot density by 2030. This isn't venture-capital-driven froth. It's state-backed industrial policy executing at a manufacturing scale that Western firms are still describing in pilot-program language.

Goldman Sachs has estimated the humanoid robot market at $154 billion within a decade. The Chinese companies IPO-ing right now are betting they'll build most of it.

The Thing Nobody Is Saying Out Loud

Anthropic published a letter to AI labs in the days before the Fable 5 launch. The argument was that AI systems are advancing so quickly they may soon achieve recursive self-improvement — autonomously making themselves smarter without human intervention — and that a coordinated global "brake pedal" was needed.

Then, four days later, Anthropic launched its most powerful model ever to the general public.

The letter and the launch are not necessarily contradictory. Anthropic's position is that Fable 5 cleared the safety bar. But the optics are uncomfortable: the company most publicly warning about AI's risks is also the one racing to list at a trillion-dollar valuation before OpenAI gets there.

That tension — between safety-first positioning and competitive commercial pressure — is the defining contradiction of the current moment in AI. Every lab feels it. Anthropic just happens to be saying it out loud more than the others, which makes the tension more visible.

What Happens Next

The next 45 days are going to be dense. The GPT-5.6 leak points to an OpenAI release before June 30. Fable 5's free window closes June 22, forcing enterprise teams to make actual cost-vs-value decisions. Anthropic's Q4 IPO window means an S-1 goes public before autumn. MSCI began adding SpaceX to indices today; Nasdaq-100 eligibility arrives in roughly early July, creating another wave of structural buying for SPCX.

And somewhere in a Shenzhen factory, a T800 is rolling off the line. Again.

The pace is not slowing down. The best you can do right now is track what's confirmed versus what's leaked, build toward models that are actually available rather than ones that are rumoured, and — if you're running an enterprise team — make your Fable 5 testing decisions before the free window shuts on June 22.

The rest of it? Watch the IPO filings when they go public. That's when we'll find out what all of this is actually worth.

Further Reading

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Environment and Config Management in Node.js: The System That Scales Past One Server

ZyVOP — Sun, 14 Jun 2026 16:36:08 +0000

Every Node.js project starts with a .env file. That is fine. The problem is when the .env approach never evolves — when you still have a single .env.production manually SCP'd to a server at 2 AM before a launch, when a new developer spends three hours figuring out which environment variables are required, when a secret rotates and you realize you have no idea how many places it is set.

Config management is not glamorous but it is the foundation everything else sits on. This guide covers the full system: validated environment variables, environment-specific config, secrets that never touch your codebase, and the patterns that work from a single VPS to a multi-environment deployment.

The Problem With Unmanaged Config

Most teams have at minimum four environments: local development, CI, staging, and production. Each has different values for dozens of variables. Without a system:

Variables that exist in production but not in .env.example cause 3 AM surprises
A typo in an environment variable name silently falls back to undefined
Secrets rotate but old values persist on servers nobody remembers to update
A new developer clones the repo and cannot start the app without asking four people for values

The solution is three things: a schema that documents and validates every variable, a clear hierarchy for where values come from in each environment, and a secrets strategy that does not involve passing plaintext passwords in Slack.

Step 1 — Validated Config with Zod

Define every environment variable your app needs in one place. Validate at startup so the app crashes immediately with a clear error rather than failing mysteriously at runtime when a missing variable is first accessed.

// src/config/env.ts
import { z } from 'zod';

const EnvSchema = z.object({
  // ── Server ──────────────────────────────────────────────
  NODE_ENV: z.enum(['development', 'test', 'production', 'staging']),
  PORT:     z.coerce.number().int().min(1).max(65535).default(3000),
  LOG_LEVEL: z.enum(['trace', 'debug', 'info', 'warn', 'error']).default('info'),
  APP_VERSION: z.string().default('unknown'),
  APP_URL: z.string().url(),

  // ── Database ─────────────────────────────────────────────
  DATABASE_URL: z
    .string()
    .url('DATABASE_URL must be a valid connection string')
    .refine(
      url => url.startsWith('postgresql://') || url.startsWith('postgres://'),
      'DATABASE_URL must be a PostgreSQL connection string'
    ),
  DATABASE_POOL_SIZE: z.coerce.number().int().min(1).max(100).default(10),

  // ── Redis ─────────────────────────────────────────────────
  REDIS_URL: z.string().url().optional(),
  REDIS_HOST: z.string().default('localhost'),
  REDIS_PORT: z.coerce.number().int().default(6379),
  REDIS_PASSWORD: z.string().optional(),

  // ── Auth ──────────────────────────────────────────────────
  JWT_PRIVATE_KEY: z.string().min(1),
  JWT_PUBLIC_KEY:  z.string().min(1),
  JWT_REFRESH_SECRET: z.string().min(32, 'JWT_REFRESH_SECRET must be at least 32 characters'),

  // ── AWS ───────────────────────────────────────────────────
  AWS_REGION: z.string().default('us-east-1'),
  S3_BUCKET:  z.string().optional(),
  AWS_ACCESS_KEY_ID:     z.string().optional(),
  AWS_SECRET_ACCESS_KEY: z.string().optional(),

  // ── Email ─────────────────────────────────────────────────
  RESEND_API_KEY: z.string().optional(),

  // ── Payments ──────────────────────────────────────────────
  STRIPE_SECRET_KEY:      z.string().optional(),
  STRIPE_WEBHOOK_SECRET:  z.string().optional(),

  // ── Observability ─────────────────────────────────────────
  SENTRY_DSN: z.string().url().optional(),
});

// Validate immediately — crash at startup if config is invalid
const result = EnvSchema.safeParse(process.env);

if (!result.success) {
  console.error('\n❌ Invalid environment configuration:\n');
  const errors = result.error.flatten().fieldErrors;
  for (const [field, messages] of Object.entries(errors)) {
    console.error(`  ${field}: ${messages?.join(', ')}`);
  }
  console.error('\nCheck your .env file against .env.example\n');
  process.exit(1);
}

export const env = result.data;

// Type-safe access everywhere — no more process.env.PORT as string
// env.PORT is number, env.DATABASE_URL is guaranteed non-null

Step 2 — The .env.example File

The .env.example file is your documentation. It is committed to git, contains no real values, and shows every variable the app needs with a comment explaining what it is for.

# .env.example — copy to .env and fill in real values
# DO NOT commit .env — it is in .gitignore

# ── Server ──────────────────────────────────────────────────
NODE_ENV=development
PORT=3000
LOG_LEVEL=info
APP_URL=http://localhost:3000

# ── Database ────────────────────────────────────────────────
# PostgreSQL connection string
# Format: postgresql://USER:PASSWORD@HOST:PORT/DATABASE
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/myapp_dev
DATABASE_POOL_SIZE=10

# ── Redis ───────────────────────────────────────────────────
REDIS_HOST=localhost
REDIS_PORT=6379
# REDIS_PASSWORD=  # Not required in local dev

# ── Auth ────────────────────────────────────────────────────
# Generate with: openssl genrsa -out private.pem 2048 && openssl rsa -in private.pem -pubout -out public.pem
JWT_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n..."
JWT_PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\n..."
# Generate with: openssl rand -hex 64
JWT_REFRESH_SECRET=generate_with_openssl_rand_hex_64

# ── AWS ─────────────────────────────────────────────────────
AWS_REGION=us-east-1
# S3_BUCKET=your-bucket-name        # Only needed for file uploads
# AWS_ACCESS_KEY_ID=                # Use IAM roles in production instead
# AWS_SECRET_ACCESS_KEY=

# ── Email ────────────────────────────────────────────────────
# Get from resend.com
# RESEND_API_KEY=re_...

# ── Payments ─────────────────────────────────────────────────
# Get from Stripe dashboard
# STRIPE_SECRET_KEY=sk_test_...
# STRIPE_WEBHOOK_SECRET=whsec_...    # From: stripe listen --print-secret

Step 3 — Environment-Specific Config Objects

Some config is not secret — it is just different per environment. Build rates, feature defaults, timeouts. Put these in code, not in environment variables:

// src/config/index.ts
import { env } from './env';

const configs = {
  development: {
    rateLimits: {
      loginAttempts: 100,    // Relaxed for development
      apiRequests:   10000,
    },
    email: {
      sendReal: false,       // Log emails instead of sending in dev
    },
    cache: {
      ttl: 10,               // Short TTL for development iteration
    },
  },

  test: {
    rateLimits: {
      loginAttempts: 1000,   // High limit so tests don't hit rate limits
      apiRequests:   100000,
    },
    email: {
      sendReal: false,
    },
    cache: {
      ttl: 1,
    },
  },

  staging: {
    rateLimits: {
      loginAttempts: 10,
      apiRequests:   500,
    },
    email: {
      sendReal: true,        // Send real emails in staging
    },
    cache: {
      ttl: 60,
    },
  },

  production: {
    rateLimits: {
      loginAttempts: 5,
      apiRequests:   200,
    },
    email: {
      sendReal: true,
    },
    cache: {
      ttl: 300,
    },
  },
};

export const config = {
  env:    env.NODE_ENV,
  port:   env.PORT,
  isDev:  env.NODE_ENV === 'development',
  isTest: env.NODE_ENV === 'test',
  isProd: env.NODE_ENV === 'production',
  ...configs[env.NODE_ENV],
};

Step 4 — Secrets That Never Touch Your Codebase

For a VPS deployment, secrets belong on the server — not in CI environment variables, not in your .env.example, not in Slack.

Generating secrets correctly:

# JWT RSA key pair
openssl genrsa -out /opt/app/secrets/jwt_private.pem 2048
openssl rsa -in /opt/app/secrets/jwt_private.pem \
  -pubout -out /opt/app/secrets/jwt_public.pem

# Random secrets
openssl rand -hex 64 > /opt/app/secrets/jwt_refresh_secret
openssl rand -hex 32 > /opt/app/secrets/session_secret

# Set permissions — readable only by the app user
chmod 600 /opt/app/secrets/*
chown deploy:deploy /opt/app/secrets/*

The production .env file — set once via SSH, never synced through CI:

# On the VPS — set manually or via an encrypted secret store
cat > /opt/app/.env.production << 'EOF'
NODE_ENV=production
PORT=3000
APP_URL=https://yourapp.com
DATABASE_URL=postgresql://appuser:$(cat /opt/app/secrets/db_password)@localhost:5432/appdb
JWT_PRIVATE_KEY="$(cat /opt/app/secrets/jwt_private.pem | tr '\n' '\n')"
JWT_PUBLIC_KEY="$(cat /opt/app/secrets/jwt_public.pem | tr '\n' '\n')"
JWT_REFRESH_SECRET=$(cat /opt/app/secrets/jwt_refresh_secret)
RESEND_API_KEY=$(cat /opt/app/secrets/resend_api_key)
STRIPE_SECRET_KEY=$(cat /opt/app/secrets/stripe_secret_key)
EOF

chmod 600 /opt/app/.env.production
chown deploy:deploy /opt/app/.env.production

Step 5 — Config Access Patterns

With the validated env and config objects, accessing configuration is consistent everywhere:

import { env } from '../config/env';
import { config } from '../config';

// Environment variables — all typed and validated
const db = new Pool({
  connectionString: env.DATABASE_URL,
  max:              env.DATABASE_POOL_SIZE,
});

// Derived config — environment-appropriate values
const rateLimiter = new RateLimiter({
  maxRequests: config.rateLimits.apiRequests,
});

// Guard against features that need optional config
if (env.STRIPE_SECRET_KEY) {
  initializeStripe(env.STRIPE_SECRET_KEY);
} else if (config.isProd) {
  throw new Error('STRIPE_SECRET_KEY is required in production');
}

Step 6 — CI Config

In GitHub Actions, secrets go in repository secrets (Settings → Secrets), not in env: blocks directly in the YAML file where they appear in logs.

# .github/workflows/deploy.yml
jobs:
  deploy:
    steps:
      - name: Deploy to VPS
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VPS_HOST }}
          username: ${{ secrets.VPS_USER }}
          key: ${{ secrets.VPS_SSH_KEY }}
          script: |
            # Reload app — env file is already on the server
            cd /opt/app
            docker compose pull
            docker compose up -d

Notice what is not passed via CI: DATABASE_URL, JWT keys, API keys. These live on the server in /opt/app/.env.production. CI only needs SSH access.

The Config Validation Script

Add a script that developers run after cloning the repo to verify their .env is complete:

// scripts/check-env.ts
import { EnvSchema } from '../src/config/env';
import * as dotenv from 'dotenv';

dotenv.config({ path: '.env' });

const result = EnvSchema.safeParse(process.env);

if (result.success) {
  console.log('✅ Environment configuration is valid');
} else {
  console.error('❌ Missing or invalid environment variables:\n');
  const errors = result.error.flatten().fieldErrors;
  for (const [field, messages] of Object.entries(errors)) {
    console.error(`  ${field}: ${messages?.join(', ')}`);
  }
  console.error('\nCopy .env.example to .env and fill in the required values.');
  process.exit(1);
}

// package.json
{
  "scripts": {
    "check:env": "tsx scripts/check-env.ts"
  }
}

Add it to the README setup steps. A new developer runs pnpm check:env and immediately knows exactly which variables they are missing.

The Rules That Prevent Config Incidents

✅ Every variable in the app is in .env.example — no undocumented variables
✅ Zod schema validates all variables at startup — no silent undefined falls
✅ .env is in .gitignore — never committed
✅ Secrets generated with openssl — not typed by hand
✅ Production secrets set on the server via SSH — never through CI env vars
✅ Different config objects per environment — not ternary chains in code
✅ check:env script runs on new developer setup
✅ Secrets rotated by updating the file on the server, not redeploying

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

How to Scrape Google Search Results with Python Without Getting Blocked (2026)

ZyVOP — Sat, 13 Jun 2026 16:14:17 +0000

Why Google SERP Scraping Is the Hardest Job in Scraping

Scraping Google Search is arguably the most requested — and most misunderstood — Python scraping task in 2026.

It powers competitive SEO analysis, rank tracking, SERP feature monitoring, AI training data collection, and market research. If you know what keywords your competitors rank for and what SERP features appear for each one, you have a massive strategic advantage.

But Google's anti-bot defences are, in 2026, smarter than ever. If you try to scrape Google using a simple Python script with requests and BeautifulSoup, you will get blocked within 10 queries — often immediately. Google detects and blocks naive scrapers through TLS fingerprinting, JavaScript challenges, CAPTCHA injection, and IP banning at the data-centre level.

This guide gives you the full truth: what works, what doesn't, the legal context, and four working approaches ordered from DIY to fully managed.

Understanding Google's Anti-Bot Stack

Before writing any code, understand what you're up against:

Request arrives at Google
       ↓
TLS fingerprint check — Does your JA3 hash match a known browser?
       ↓
IP reputation check — Datacenter? Banned range? Previous violations?
       ↓
Header analysis — Correct order? sec-ch-ua present? Accept-Language set?
       ↓
JavaScript challenge — Can you execute JS? (Invisible to non-JS clients)
       ↓
Behavioural analysis — Request frequency, query patterns, session depth
       ↓
CAPTCHA — Last resort for uncertain cases

You need to defeat enough layers to score below Google's bot-confidence threshold. The more layers you address, the longer your scraper survives.

The Legal and Ethical Context

Scraping Google Search has been legally contested. The key points:

Google's Terms of Service prohibit automated scraping of search results without explicit permission
The Computer Fraud and Abuse Act (CFAA) in the US has been cited in cease-and-desist letters
However, Google offers its own Custom Search JSON API (100 free queries/day, paid beyond that) for legitimate programmatic access
For SEO tools, most professional platforms use licensed SERP data from providers like SerpApi, DataForSEO, or Bright Data — not DIY scrapers

Bottom line: Use the DIY approaches below for personal projects, learning, and low-volume research. For commercial or production SEO tools, use a licensed API.

Approach 1: The DIY Scraper (curl_cffi + BeautifulSoup)

This is the most technically educational approach and works for low-volume personal research.

Why `requests` fails immediately

import requests

# This gets you blocked within 1-2 queries in 2026
r = requests.get(
    "https://www.google.com/search?q=python+web+scraping",
    headers={"User-Agent": "Mozilla/5.0"}
)
print(r.status_code)  # 429 or 302 (redirected to CAPTCHA page)

Python's requests has a distinctive TLS fingerprint that Google's systems recognise instantly.

The working approach: curl_cffi with TLS impersonation

pip install curl_cffi beautifulsoup4 pandas

from curl_cffi import requests as cffi_requests
from bs4 import BeautifulSoup
import pandas as pd
import time
import random

# Google-like headers — exact order matters
HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
              "image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Cache-Control": "max-age=0",
    "sec-ch-ua": '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
}

PROXIES_POOL = [
    # Add residential proxies here: "http://user:pass@host:port"
    # Without residential proxies, you'll hit rate limits quickly
]

def get_google_results(
    query: str,
    num_results: int = 10,
    country: str = "us",
    language: str = "en"
) -> list[dict]:
    """
    Scrape Google SERP for organic results.
    Returns a list of {rank, title, url, snippet} dicts.
    """
    params = {
        "q": query,
        "num": num_results,
        "hl": language,
        "gl": country,
        "pws": "0",   # Disable personalised results
    }

    # Build query string manually (preserves param order)
    query_string = "&".join(f"{k}={v}" for k, v in params.items())
    url = f"https://www.google.com/search?{query_string}"

    proxy = random.choice(PROXIES_POOL) if PROXIES_POOL else None

    response = cffi_requests.get(
        url,
        headers=HEADERS,
        impersonate="chrome120",  # Exact TLS fingerprint match
        proxies={"https": proxy} if proxy else None,
        timeout=15,
    )

    if response.status_code != 200:
        raise Exception(f"Got status {response.status_code}")

    return parse_serp(response.text)

def parse_serp(html: str) -> list[dict]:
    """Parse organic results from Google SERP HTML."""
    soup = BeautifulSoup(html, "lxml")
    results = []

    # Google's organic result containers — class names change frequently
    # These were accurate as of June 2026; update via DevTools if needed
    for rank, result_div in enumerate(soup.select("div.g"), start=1):
        title_el  = result_div.select_one("h3")
        url_el    = result_div.select_one("a[href]")
        snippet_el = result_div.select_one(".VwiC3b, .lEBKkf")

        if not title_el or not url_el:
            continue

        href = url_el.get("href", "")
        # Google wraps URLs in /url?q= redirects — extract the real URL
        if href.startswith("/url?q="):
            href = href.split("/url?q=")[1].split("&")[0]

        results.append({
            "rank":    rank,
            "title":   title_el.get_text(strip=True),
            "url":     href,
            "snippet": snippet_el.get_text(strip=True) if snippet_el else "",
        })

    return results

def scrape_keyword_list(keywords: list[str], delay: tuple = (4, 9)) -> pd.DataFrame:
    """
    Scrape Google results for a list of keywords.
    Conservative delay between requests reduces ban risk.
    """
    all_results = []

    for i, keyword in enumerate(keywords):
        print(f"[{i+1}/{len(keywords)}] Scraping: '{keyword}'")
        try:
            results = get_google_results(keyword, num_results=10)
            for r in results:
                r["keyword"] = keyword
            all_results.extend(results)

        except Exception as e:
            print(f"  Error on '{keyword}': {e}")

        # Human-like delay between queries — NEVER remove this
        sleep_time = random.uniform(*delay)
        print(f"  Waiting {sleep_time:.1f}s before next query...")
        time.sleep(sleep_time)

    return pd.DataFrame(all_results)

# Example usage
keywords = [
    "python web scraping tutorial 2026",
    "best python scraping libraries",
    "how to scrape without getting blocked",
]

df = scrape_keyword_list(keywords)
df.to_csv("serp_results.csv", index=False)
print(df[["keyword", "rank", "title", "url"]].head(10))

Extracting SERP Features

Beyond organic results, Google SERPs contain rich features worth extracting:

def extract_serp_features(html: str) -> dict:
    """Extract SERP features: featured snippet, PAA, local pack, etc."""
    soup = BeautifulSoup(html, "lxml")
    features = {}

    # ── Featured Snippet (Position 0) ──────────────────────────
    featured = soup.select_one(".hgKElc, .LGOjhe")
    if featured:
        features["featured_snippet"] = featured.get_text(strip=True)

    # ── People Also Ask (PAA) ───────────────────────────────────
    paa_questions = soup.select(".related-question-pair span.CSkcDe")
    if paa_questions:
        features["people_also_ask"] = [q.get_text(strip=True) for q in paa_questions]

    # ── Related Searches ────────────────────────────────────────
    related = soup.select("a .s75CSd")
    if related:
        features["related_searches"] = [r.get_text(strip=True) for r in related]

    # ── Knowledge Panel ──────────────────────────────────────────
    kp_title = soup.select_one(".qrShPb span")
    if kp_title:
        features["knowledge_panel_entity"] = kp_title.get_text(strip=True)

    # ── Local Pack (Map Results) ─────────────────────────────────
    local_results = soup.select(".rllt__details")
    if local_results:
        features["local_pack"] = [r.get_text(strip=True) for r in local_results[:3]]

    return features

Approach 2: Google Custom Search API (Official, Free Tier)

For clean, reliable data without bot-detection risk, Google's official API is often the best choice for low-volume use:

pip install google-api-python-client

from googleapiclient.discovery import build
import pandas as pd

# Requirements:
# 1. Create project at console.developers.google.com
# 2. Enable "Custom Search API"
# 3. Create API key
# 4. Create Programmable Search Engine at programmablesearchengine.google.com
# 5. Get your Search Engine ID (cx)

API_KEY = "YOUR_GOOGLE_API_KEY"
CSE_ID  = "YOUR_CUSTOM_SEARCH_ENGINE_ID"

def google_api_search(query: str, num: int = 10) -> list[dict]:
    """
    Official Google Custom Search API.
    Free tier: 100 queries/day.
    Paid: $5 per 1,000 queries beyond free tier.
    """
    service = build("customsearch", "v1", developerKey=API_KEY)

    results_list = []
    # API returns max 10 results per call; paginate for more
    for start in range(1, num + 1, 10):
        response = service.cse().list(
            q=query,
            cx=CSE_ID,
            start=start,
            num=min(10, num - start + 1),
        ).execute()

        for i, item in enumerate(response.get("items", []), start=start):
            results_list.append({
                "rank":    i,
                "title":   item.get("title"),
                "url":     item.get("link"),
                "snippet": item.get("snippet"),
                "domain":  item.get("displayLink"),
            })

    return results_list

# Clean, simple, and legal
results = google_api_search("python web scraping 2026", num=10)
df = pd.DataFrame(results)
print(df[["rank", "title", "domain"]])

Limitations: The Custom Search API searches your defined search engine scope — not all of Google. Results differ from google.com organic results, making it less suitable for pure rank-tracking.

Approach 3: SerpApi (Managed, Production-Ready)

For production SEO tools and higher volume, SerpApi is the industry standard. It handles all anti-bot complexity and delivers clean, structured JSON:

pip install google-search-results

from serpapi import GoogleSearch
import pandas as pd

def serpapi_search(
    query: str,
    api_key: str,
    location: str = "India",
    num: int = 10
) -> dict:
    """
    SerpApi Google Search — returns structured JSON with all SERP features.
    Pricing: $50/month for 5,000 searches (2026 rates).
    Free tier: 100 searches/month.
    """
    params = {
        "engine":   "google",
        "q":        query,
        "api_key":  api_key,
        "location": location,
        "hl":       "en",
        "gl":       "in",
        "num":      num,
        "no_cache": False,  # Use cache when available to save credits
    }

    search  = GoogleSearch(params)
    results = search.get_dict()

    output = {
        "query":            query,
        "organic_results":  [],
        "featured_snippet": None,
        "people_also_ask":  [],
        "related_searches": [],
        "total_results":    results.get("search_information", {})
                                   .get("total_results"),
    }

    # Organic results
    for r in results.get("organic_results", []):
        output["organic_results"].append({
            "rank":    r.get("position"),
            "title":   r.get("title"),
            "url":     r.get("link"),
            "snippet": r.get("snippet"),
            "domain":  r.get("displayed_link"),
        })

    # Featured snippet
    if "answer_box" in results:
        output["featured_snippet"] = results["answer_box"].get("answer") or \
                                     results["answer_box"].get("snippet")

    # People Also Ask
    for paa in results.get("related_questions", []):
        output["people_also_ask"].append({
            "question": paa.get("question"),
            "snippet":  paa.get("snippet"),
        })

    # Related searches
    output["related_searches"] = [
        r.get("query") for r in results.get("related_searches", [])
    ]

    return output

# Usage
result = serpapi_search(
    query="python web scraping tutorial",
    api_key="YOUR_SERPAPI_KEY",
    location="Mumbai, India"
)

df = pd.DataFrame(result["organic_results"])
print(df[["rank", "title", "domain"]])
print("\nPeople Also Ask:")
for paa in result["people_also_ask"][:3]:
    print(f"  Q: {paa['question']}")

Approach 4: Building a Rank Tracker (Practical SEO Tool)

Putting it all together into a real-world keyword rank tracker:

import pandas as pd
import json
import time
import random
from datetime import datetime, timezone
from pathlib import Path

class KeywordRankTracker:
    """
    Track keyword rankings over time.
    Stores historical data in JSON for trend analysis.
    """

    def __init__(self, project_name: str, api_key: str):
        self.project_name = project_name
        self.api_key      = api_key
        self.data_file    = Path(f"{project_name}_rankings.json")
        self.history      = self._load_history()

    def _load_history(self) -> list:
        if self.data_file.exists():
            with open(self.data_file) as f:
                return json.load(f)
        return []

    def _save_history(self):
        with open(self.data_file, "w") as f:
            json.dump(self.history, f, indent=2)

    def check_rankings(
        self,
        keywords: list[str],
        target_domain: str,
        location: str = "India"
    ) -> pd.DataFrame:
        """
        Check where target_domain ranks for each keyword.
        """
        timestamp = datetime.now(timezone.utc).isoformat()
        records   = []

        for keyword in keywords:
            print(f"Checking: '{keyword}'")
            result  = serpapi_search(keyword, self.api_key, location)
            ranking = None

            for r in result["organic_results"]:
                if target_domain.lower() in (r.get("domain") or "").lower():
                    ranking = r["rank"]
                    break

            record = {
                "timestamp": timestamp,
                "keyword":   keyword,
                "domain":    target_domain,
                "rank":      ranking,          # None = not in top 10
                "in_top_10": ranking is not None,
            }
            records.append(record)
            self.history.append(record)
            time.sleep(random.uniform(1.5, 3.0))

        self._save_history()
        return pd.DataFrame(records)

    def get_trend_report(self) -> pd.DataFrame:
        """Show ranking changes over time for each keyword."""
        if not self.history:
            return pd.DataFrame()

        df = pd.DataFrame(self.history)
        df["timestamp"] = pd.to_datetime(df["timestamp"])
        df = df.sort_values("timestamp")

        # Pivot to show rank by date
        pivot = df.pivot_table(
            index="keyword",
            columns=df["timestamp"].dt.date,
            values="rank",
            aggfunc="first"
        )
        return pivot

# Usage
tracker = KeywordRankTracker(
    project_name="my_blog",
    api_key="YOUR_SERPAPI_KEY"
)

# Check weekly rankings
rankings = tracker.check_rankings(
    keywords=[
        "python web scraping tutorial",
        "python async scraping",
        "scrapy mongodb tutorial",
    ],
    target_domain="yourblog.com",
    location="India"
)

print("\nCurrent rankings:")
print(rankings[["keyword", "rank", "in_top_10"]].to_string(index=False))

# Show historical trend
trend = tracker.get_trend_report()
if not trend.empty:
    print("\nRanking trend:")
    print(trend)

Competitor SERP Analysis: Who's Outranking You and Why

def analyse_serp_competitors(
    keyword: str,
    your_domain: str,
    api_key: str
) -> dict:
    """
    Analyse who ranks in the top 10 for a keyword,
    what their titles/snippets look like, and where you stand.
    """
    result    = serpapi_search(keyword, api_key, num=10)
    organics  = result["organic_results"]
    your_rank = None

    competitor_analysis = []
    for r in organics:
        domain     = r.get("domain", "")
        is_you     = your_domain.lower() in domain.lower()
        if is_you:
            your_rank = r["rank"]

        # Title length analysis (55-60 chars is Google's sweet spot)
        title_len  = len(r.get("title") or "")
        # Snippet length analysis
        snippet_len = len(r.get("snippet") or "")

        competitor_analysis.append({
            "rank":         r["rank"],
            "domain":       domain,
            "title":        r.get("title"),
            "title_length": title_len,
            "snippet_length": snippet_len,
            "is_you":       is_you,
        })

    return {
        "keyword":    keyword,
        "your_rank":  your_rank or "Not in top 10",
        "gap_to_top": (your_rank - 1) if your_rank else None,
        "competitors": competitor_analysis,
        "featured_snippet_exists": result["featured_snippet"] is not None,
        "paa_count":  len(result["people_also_ask"]),
        "paa_questions": [q["question"] for q in result["people_also_ask"]],
    }

# Run competitor analysis
analysis = analyse_serp_competitors(
    keyword="python web scraping tutorial 2026",
    your_domain="yourblog.com",
    api_key="YOUR_SERPAPI_KEY"
)

print(f"\nKeyword: {analysis['keyword']}")
print(f"Your rank: {analysis['your_rank']}")
print(f"Featured snippet: {'YES' if analysis['featured_snippet_exists'] else 'No'}")
print(f"\nTop 5 competitors:")
for c in analysis["competitors"][:5]:
    marker = " ← YOU" if c["is_you"] else ""
    print(f"  #{c['rank']} {c['domain']}{marker} — title: {c['title_length']} chars")

print(f"\nPeople Also Ask ({analysis['paa_count']} questions):")
for q in analysis["paa_questions"]:
    print(f"  • {q}")

Scaling Up: Async SERP Scraping

For bulk keyword research (hundreds of queries), async execution cuts runtime by 5–10x:

import asyncio
from curl_cffi.requests import AsyncSession
import random

SEMAPHORE = asyncio.Semaphore(5)   # 5 concurrent — conservative for Google

async def async_google_search(
    session: AsyncSession,
    query: str,
    delay_range: tuple = (3, 7)
) -> tuple[str, list[dict]]:
    """Async version of Google scraper."""
    async with SEMAPHORE:
        await asyncio.sleep(random.uniform(*delay_range))

        params = f"q={query.replace(' ', '+')}&num=10&hl=en&gl=in&pws=0"
        url    = f"https://www.google.com/search?{params}"

        try:
            r = await session.get(
                url,
                headers=HEADERS,
                impersonate="chrome120",
                timeout=15,
            )
            results = parse_serp(r.text) if r.status_code == 200 else []
            return query, results
        except Exception as e:
            print(f"Failed '{query}': {e}")
            return query, []

async def bulk_serp_scrape(keywords: list[str]) -> pd.DataFrame:
    """Scrape hundreds of keywords asynchronously."""
    all_records = []

    async with AsyncSession(impersonate="chrome120") as session:
        tasks   = [async_google_search(session, kw) for kw in keywords]
        results = await asyncio.gather(*tasks)

    for keyword, serp_results in results:
        for r in serp_results:
            r["keyword"] = keyword
            all_records.append(r)

    return pd.DataFrame(all_records)

# 100 keywords in ~2 minutes instead of ~15 minutes
keywords = [f"python {topic}" for topic in [
    "scraping tutorial", "async tutorial", "data pipeline",
    "mongodb tutorial", "playwright guide", "scrapy example"
]]

df = asyncio.run(bulk_serp_scrape(keywords))
df.to_csv("bulk_serp.csv", index=False)

Choosing Your Approach: Decision Guide

Situation	Best Approach
Learning / personal project	DIY with curl_cffi (Approach 1)
< 100 queries/day, official use	Google Custom Search API (Approach 2)
Production SEO tool, any volume	SerpApi or DataForSEO (Approach 3)
Bulk research, 1000+ keywords	Async DIY with residential proxies
International SERP analysis	SerpApi (supports location targeting)
SERP feature monitoring (PAA, featured)	SerpApi (returns structured features)

Common Errors and Fixes

Getting a CAPTCHA page immediately Your IP is on a blocklist. Switch to a residential proxy or use SerpApi.

HTML parses but no results found Google changed its CSS class names — they change every few weeks. Open DevTools → Inspect → find the new class for .g results and update parse_serp().

429 Too Many Requests You're hitting Google's rate limit. Increase your delay to 6–10 seconds between queries, or switch to a proxy pool with IP rotation.

Results look different from browser You're likely getting a different locale. Add &hl=en&gl=us to your query params and set Accept-Language: en-US,en;q=0.9 in headers.

FAQ

Q: Is scraping Google legal? Technically it violates Google's Terms of Service. However, scraping publicly available search results for personal research purposes has not typically resulted in legal action against individuals. Commercial use of scraped Google data is a different matter — use SerpApi or DataForSEO for that.

Q: How many queries per day can I make before getting blocked? With rotating residential proxies and 4–9 second delays: 200–500 queries/day per IP. Without proxies: 20–50 before hitting a CAPTCHA.

Q: What's the best free alternative to SerpApi? The Google Custom Search JSON API gives 100 free queries/day officially. For DIY, use the curl_cffi approach above.

Q: Can I track rankings for Google India specifically? Yes — add &gl=in&hl=en to your query params, or set location: "India" in SerpApi params.

Summary

Approach	Cost	Volume	Reliability	Best For
curl_cffi DIY	Free	Low-Medium	Medium	Learning, personal use
Google CSE API	Free/paid	Low	High (official)	Authorised access
SerpApi	$50+/month	Unlimited	Very High	Production tools
DataForSEO	Pay-per-use	Unlimited	Very High	Enterprise SEO

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Redis Caching in Node.js: The Patterns That Actually Hold Up in Production

ZyVOP — Fri, 12 Jun 2026 02:44:02 +0000

The benchmark is always impressive. Before Redis: 840 ms average response time. After Redis: 80 ms on cache hits. A 60% reduction in overall latency across the full traffic mix, and the support queue for "the platform is slow" goes to zero.

What the benchmark does not show is the three days spent debugging stale data, the afternoon untangling a cache stampede that took down the database during a traffic spike, and the cache key naming scheme that had to be refactored mid-sprint because it was not composable enough for prefix-based invalidation.

This guide covers both sides of the ledger. The patterns that get you the gains, and the operational reality that keeps those gains from becoming liabilities.

Why Three Layers, Not One

Caching in a Node.js backend is not a single decision — it is a three-layer architecture, with each layer optimized for a different access pattern:

Layer 1 — In-process LRU: Data lives in Node.js heap memory. No network round-trip. Sub-millisecond access. Per-process and non-shared — each server instance has its own copy. Use for: hot reference data (feature flags, config, lookup tables) that changes rarely and can tolerate brief per-instance staleness.

Layer 2 — Redis: Data lives in a shared, external in-memory store. ~1 ms network round-trip. Shared across all server instances. Use for: session data, user state, shared counters, computed results that must be consistent across your fleet.

Layer 3 — CDN edge cache: Data lives at geographically distributed edge nodes. Sub-10 ms for most users globally. Serves HTTP responses, not application data. Use for: public API responses, static assets, anything that can carry a Cache-Control header.

Most production applications need all three, applied deliberately to different data categories. The mistake is treating Redis as the answer to every caching question — some data is better served by a process-local LRU, and some data should never be cached at all.

Layer 1: In-Process LRU with `lru-cache`

The fastest cache is the one that never leaves your process. For data accessed thousands of times per minute that changes infrequently, an in-process LRU eliminates not just database round-trips but Redis round-trips too.

import { LRUCache } from 'lru-cache';

// Cache for feature flags — changes rarely, read on every request
const featureFlagCache = new LRUCache({
  max: 500,
  ttl: 1000 * 60,          // 60-second TTL
  updateAgeOnGet: false,    // Don't reset TTL on read
  fetchMethod: async (flagKey) => {
    // Called automatically on cache miss — deduplicates concurrent misses
    return await featureFlagService.getFlag(flagKey);
  },
});

// Cache for database query results — medium-frequency reads
const queryCache = new LRUCache({
  max: 500,
  ttl: 1000 * 60 * 5,      // 5-minute TTL
  updateAgeOnGet: false,
  allowStale: false,
});

// Usage
async function getFeatureFlag(key) {
  // fetchMethod handles miss automatically — no manual miss handling needed
  return await featureFlagCache.fetch(key);
}

async function getCachedQueryResult(queryKey, queryFn) {
  const cached = queryCache.get(queryKey);
  if (cached !== undefined) return cached;

  const result = await queryFn();
  queryCache.set(queryKey, result);
  return result;
}

The fetchMethod option on lru-cache is worth highlighting. When multiple concurrent requests hit a cache miss for the same key simultaneously, fetchMethod deduplicates them — only one call to the underlying data source is made, and all waiting promises resolve with the same result. This is request coalescing built into the cache.

What belongs in-process:

Feature flags and A/B test assignments
Application configuration
Static lookup tables (country codes, category lists)
JWT public keys / JWKS
Compiled regular expressions or validation schemas

What does not belong in-process:

Session data (users would lose sessions when a server restarts or when load balancer routes them to a different instance)
Anything that must be consistent across multiple server instances immediately after a write

Layer 2: Redis — Patterns That Work in Production

Pattern 1: Cache-Aside (Lazy Loading)

The most common and most appropriate pattern for application data. The cache is populated on demand: miss, fetch, store.

import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function getUser(userId) {
  const cacheKey = `user:${userId}`;

  // 1. Check cache
  const cached = await redis.get(cacheKey);
  if (cached) {
    return JSON.parse(cached);
  }

  // 2. Cache miss — fetch from database
  const user = await db.query(
    'SELECT id, name, email, role FROM users WHERE id = $1',
    [userId]
  );

  if (!user) return null;

  // 3. Store in cache with TTL
  await redis.setEx(cacheKey, 300, JSON.stringify(user)); // 5-minute TTL

  return user;
}

// Invalidation: call this from any write path that modifies a user
async function invalidateUser(userId) {
  await redis.del(`user:${userId}`);
}

The invalidation discipline is as important as the caching logic. Every write path in your application that modifies a user must call invalidateUser. If you add a new endpoint that updates user email without calling invalidation, users will see stale data until the TTL expires. Build invalidation alongside every write, not as an afterthought.

Pattern 2: Write-Through

Every write updates both the cache and the database. Reads are always served from cache. Suited for data that is written and read at similar frequency, where read-after-write consistency matters.

async function updateUserProfile(userId, updates) {
  // 1. Write to database first
  const updated = await db.query(
    'UPDATE users SET name = $1, bio = $2, updated_at = NOW() WHERE id = $3 RETURNING *',
    [updates.name, updates.bio, userId]
  );

  // 2. Immediately update cache
  const cacheKey = `user:${userId}`;
  await redis.setEx(cacheKey, 300, JSON.stringify(updated));

  return updated;
}

Write-through is more expensive per write but eliminates the window between a write and cache population where a read would generate a cache miss and hit the database. For user profile updates where the user immediately sees their own edits, write-through provides a better experience.

Pattern 3: Cache Keys as a System

Cache key design is the most under-discussed aspect of production caching. A key scheme that is not composable makes bulk invalidation impossible — you end up with either over-invalidation (deleting too much) or stale data.

// Structured key conventions
const CacheKeys = {
  user: (id) => `user:${id}`,
  userOrders: (userId) => `user:${userId}:orders`,
  userOrderPage: (userId, page) => `user:${userId}:orders:page:${page}`,

  product: (id) => `product:${id}`,
  productsByCategory: (categoryId) => `products:category:${categoryId}`,

  // Tag-based invalidation: all keys for a user
  userPattern: (userId) => `user:${userId}:*`,
};

// Bulk invalidation using SCAN (never use KEYS in production — it's O(N) and blocks Redis)
async function invalidateAllUserData(userId) {
  const pattern = CacheKeys.userPattern(userId);
  let cursor = 0;

  do {
    const result = await redis.scan(cursor, {
      MATCH: pattern,
      COUNT: 100,
    });

    cursor = result.cursor;

    if (result.keys.length > 0) {
      await redis.del(result.keys);
    }
  } while (cursor !== 0);
}

The SCAN command with COUNT is the correct way to search for keys by pattern. The KEYS command (redis.keys('user:*')) is O(N) across your entire key space — it blocks the Redis event loop for the entire duration, causing latency spikes across every client connected to that Redis instance during the scan. Never use KEYS in production.

Pattern 4: Preventing Cache Stampedes

The classic production failure: a heavily cached key expires. A thousand concurrent requests all see a miss simultaneously and all query the database at once. The database falls over.

This is the thundering herd problem, and it gets more dangerous as your traffic grows.

Solution A: Probabilistic Early Expiration (XFetch)

Recompute the cache value before it expires, with probability proportional to how close to expiry the value is. By the time the key actually expires, it has already been refreshed.

async function getCachedWithXFetch(key, fetchFn, ttl, beta = 1.0) {
  const raw = await redis.get(key);

  if (raw) {
    const { value, expiry, delta } = JSON.parse(raw);
    const now = Date.now() / 1000;

    // Probabilistically recompute before expiry
    // Higher beta = more aggressive early refresh
    if (now - delta * beta * Math.log(Math.random()) < expiry) {
      return value;
    }
  }

  // Cache miss or early refresh triggered
  const start = Date.now();
  const value = await fetchFn();
  const delta = (Date.now() - start) / 1000; // Time taken to compute, in seconds
  const expiry = Date.now() / 1000 + ttl;

  await redis.setEx(key, ttl, JSON.stringify({ value, expiry, delta }));

  return value;
}

Solution B: Mutex Lock on Cache Miss

When a cache miss occurs, acquire a distributed lock before fetching. Other requests wait for the lock holder to populate the cache rather than all racing to the database.

async function getCachedWithLock(key, fetchFn, ttl) {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const lockKey = `lock:${key}`;
  const lockAcquired = await redis.set(lockKey, '1', {
    NX: true,      // Only set if not exists
    EX: 10,        // Lock expires in 10 seconds
  });

  if (lockAcquired) {
    try {
      // This instance won the lock — fetch and populate
      const value = await fetchFn();
      await redis.setEx(key, ttl, JSON.stringify(value));
      return value;
    } finally {
      await redis.del(lockKey);
    }
  } else {
    // Another instance is populating — wait briefly and retry
    await new Promise(resolve => setTimeout(resolve, 50));
    const populated = await redis.get(key);
    return populated ? JSON.parse(populated) : getCachedWithLock(key, fetchFn, ttl);
  }
}

Use probabilistic early expiration for frequently accessed keys where you can afford the occasional early refresh. Use mutex locks for expensive operations (external API calls, heavy aggregations) where parallel execution would be harmful.

Layer 3: CDN Edge Caching for API Responses

For public API endpoints that return the same response for many users, CDN caching eliminates origin load entirely. A well-configured CDN can absorb 90%+ of read traffic for public content.

// Express middleware that sets aggressive caching headers for public endpoints
function setCacheHeaders(maxAge, staleWhileRevalidate = 60) {
  return (req, res, next) => {
    // Only cache GET requests
    if (req.method !== 'GET') return next();

    res.set({
      'Cache-Control': `public, max-age=${maxAge}, stale-while-revalidate=${staleWhileRevalidate}`,
      'Vary': 'Accept-Encoding',  // Separate cache for gzip vs non-gzip
    });

    next();
  };
}

// Product listing: cache for 5 minutes, serve stale for 60 seconds while revalidating
app.get('/api/products',
  setCacheHeaders(300, 60),
  async (req, res) => {
    const products = await productService.list(req.query);
    res.json(products);
  }
);

// User-specific data: never cache at CDN
app.get('/api/user/profile',
  (req, res, next) => {
    res.set('Cache-Control', 'private, no-cache');
    next();
  },
  authenticate,
  async (req, res) => {
    const profile = await userService.getProfile(req.user.id);
    res.json(profile);
  }
);

stale-while-revalidate is the most useful cache directive for API responses. It tells the CDN to serve a stale response immediately while fetching a fresh one in the background. From the user's perspective, the response is always fast. From your origin's perspective, requests arrive at a controlled, background rate rather than all at once when cache entries expire.

The critical Vary header: If you serve compressed and uncompressed responses from the same endpoint (which all production servers should), Vary: Accept-Encoding ensures the CDN maintains separate cache entries for each encoding. Without it, compressed responses get served to clients that cannot decompress them.

Eviction Policies: What Happens When Redis Gets Full

Redis operates entirely in memory. When maxmemory is reached and new keys need to be written, Redis must evict existing keys. The eviction policy determines which keys get removed.

# redis.conf
maxmemory 4gb
maxmemory-policy allkeys-lru

Policy comparison:

Policy	Behavior	Use When
noeviction	Refuse writes when full; return error	You cannot afford data loss — Redis as primary store
allkeys-lru	Evict least recently used keys from entire keyspace	General-purpose cache — this is the right default
volatile-lru	Evict LRU keys that have a TTL set	Mix of cache and persistent data in same Redis instance
allkeys-lfu	Evict least frequently used keys	Access patterns with highly skewed popularity
volatile-ttl	Evict keys closest to expiry	You want to control what survives memory pressure via TTL

For a pure cache (all data has TTLs and can be regenerated from the database), allkeys-lru is almost always the right choice.

The unbounded key growth trap: Forgetting to set TTLs on cache keys is one of the most common Redis production mistakes. Without TTLs, your key count grows monotonically until Redis hits maxmemory and starts evicting keys according to your policy. With noeviction, writes start failing. The symptom appears suddenly at a threshold that can be hard to predict.

Always set TTLs. Always.

Observability: The Metrics That Catch Problems Early

A cache you cannot observe is a cache you cannot trust. These are the signals worth instrumenting from day one.

Cache hit rate is the single most important metric. Below 50% means your strategy is broken — your TTLs are too short, your key space is too fragmented, or you are caching data that changes too frequently.

// Redis INFO stats
const info = await redis.info('stats');

// Parse keyspace_hits and keyspace_misses
const hitRate = keyspaceHits / (keyspaceHits + keyspaceMisses);

Memory usage and fragmentation:

const memoryInfo = await redis.info('memory');
// Watch: used_memory_rss vs used_memory
// High fragmentation ratio (>1.5) means Redis is holding onto
// more OS memory than it is actually using

Latency percentiles: Instrument your cache client to record operation latency. A Redis GET that takes 50 ms indicates network problems or a Redis instance under heavy load — both situations that degrade your application even when the data is cached.

Instrument hit/miss at the application layer:

class InstrumentedCache {
  constructor(redisClient, metrics) {
    this.redis = redisClient;
    this.metrics = metrics;
  }

  async get(key, namespace = 'default') {
    const value = await this.redis.get(key);

    if (value) {
      this.metrics.increment('cache.hit', { namespace });
    } else {
      this.metrics.increment('cache.miss', { namespace });
    }

    return value ? JSON.parse(value) : null;
  }
}

Track hit rate per cache namespace. A single low-performing key namespace degrades overall hit rate but is invisible in aggregate metrics.

The Redis Scaling Progression

Start simple. Add complexity only when metrics demand it.

Phase 1 — Single Node: Sufficient for most applications up to significant traffic. Add read replicas when reads become the bottleneck.

Phase 2 — Redis Sentinel: Automatic failover for high availability without horizontal sharding. Suitable up to approximately 50 GB of data. One primary, multiple replicas, Sentinel monitors and promotes a replica if the primary fails.

Phase 3 — Redis Cluster: Automatic horizontal sharding across 3–1,000 nodes with built-in replication. Handles petabyte-scale workloads. Adds operational complexity and some API constraints (multi-key operations across slots require care).

Most applications never need Phase 3. Most applications that think they need Phase 3 actually need Phase 2 combined with better application-level caching and query optimization.

The Anti-Patterns Worth Naming

Caching everything once you see the results. After the first successful caching implementation improves response times dramatically, the temptation is to cache every expensive operation. Resist it. Understand the data's update lifecycle — who writes it, how often, what triggers a change — before deciding to cache it. Caching data with complex invalidation requirements without a clear invalidation plan turns a performance win into a correctness bug.

Using KEYS in production. KEYS pattern is O(N) across the entire key space and blocks Redis for its entire duration. At 1 million keys with a 100 ms scan, every client connected to that Redis instance experiences 100 ms of additional latency. Use SCAN with a cursor.

Missing TTLs on any key. Set them. Every one.

A cache hit rate below 50%. This is not a caching problem — it is a sign that your keys are either not matching real request patterns (key construction is wrong), your TTL is shorter than your query interval (keys expire before they get read again), or you are caching data that changes faster than it is read (which means you should not be caching it).

What Good Caching Looks Like

Good caching is designed from the beginning, not bolted on after a performance incident. The key scheme is deliberate and composable. Invalidation is built into every write path at the same time as caching is added to the read path. TTLs are set based on how frequently the underlying data changes, not based on a default that someone copy-pasted. And the hit rate is monitored continuously — not checked once during load testing and then forgotten.

The gains are real. A well-implemented Redis caching layer reduces database load by 60–90%, dramatically improves response latency, and lets your database infrastructure scale much further before requiring sharding or read replicas.

The operational discipline to maintain those gains over time is what separates a caching implementation that helps from one that eventually causes a 3 AM incident.

Don't cache what you don't understand. Caching data with complex invalidation requirements without a clear invalidation strategy introduces subtle bugs that are hard to debug in production.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Python Developer Roadmap 2026 — From Zero to Job-Ready (No Fluff, Real Sequence)

ZyVOP — Thu, 11 Jun 2026 19:44:11 +0000

The problem with most Python roadmaps is that they are either too shallow ("just learn the basics!") or overwhelming lists of 50 tools with no idea of what order to learn them in or how long each stage should take.

This post is different. It is written for someone in India in 2026 who wants to go from either zero coding experience or basic Python knowledge to being genuinely employable as a Python developer. The stages are sequenced based on what actually matters in the job market, not what looks comprehensive on a blog post.

One caveat upfront: there is no single "Python developer" role. The language is used across backend web development, data engineering, machine learning, and automation. This roadmap covers the shared foundation that all of those require, and then branches clearly into specialisations. Decide your direction before you start — it will save you months of misdirected effort.

Phase 1: Core Python fundamentals (Weeks 1–8)

This is the most important phase and the one most people rush through. Every mistake at this stage becomes a painful relearning later. Do not move to Phase 2 until you are comfortable with everything here.

What to learn:

Variables, data types, and operators — Python's built-in types (int, float, str, list, tuple, dict, set) and how they behave. Understand mutability. Know why a = [1, 2, 3]; b = a; b.append(4) changes both a and b.

Control flow — if/elif/else, for loops, while loops, break, continue, pass. Know when to use each and why.

Functions — defining functions, arguments (positional, keyword, *args, **kwargs), return values, scope. This is where most beginners go shallow. Understand closures. Understand what global and nonlocal do.

Object-Oriented Programming — classes, objects, __init__, inheritance, super(), @property, @classmethod, @staticmethod. OOP is tested in virtually every Python interview in India. Do not skip it.

File handling — reading and writing files with open(), using context managers (with statements). This is used in almost every real project.

Error handling — try/except/finally, creating custom exceptions, understanding the exception hierarchy.

List comprehensions and generators — these are core Python idioms. Code that uses them looks professional; code that doesn't often looks junior.

Virtual environments — every professional Python project uses venv or conda to isolate dependencies. Learn this early. It avoids hours of debugging dependency conflicts. The Python docs have a clear guide on this.

Where to practice: LeetCode easy problems in Python. Not for competitive programming — for getting comfortable translating problem statements into working code. Aim for 30–50 easy problems in Phase 1.

What to build: A command-line tool. Something simple but real — a todo list manager, a unit converter, a password generator. The project doesn't matter; writing something that actually runs and handles edge cases does.

Phase 2: Developer tools and professional workflow (Weeks 6–10, overlapping with Phase 1)

Most roadmaps put this too late. You should be learning these tools while you are still learning Python fundamentals, because they change how you code from the start.

Git and GitHub — version control is non-negotiable. Learn git init, add, commit, push, pull, branch, merge, and resolving conflicts. Start putting every project on GitHub from day one. Your GitHub profile is your portfolio and your credibility signal. GitHub's official guides are clear and free.

A proper code editor — VS Code with the Python extension is the standard for most Indian developers. Set up linting (flake8 or ruff) and auto-formatting (black). These tools enforce code quality habits automatically.

Basic command line — you don't need to be a Linux expert, but you need to navigate directories, run scripts, manage files, and understand environment variables from the terminal. If you are on Windows, learn to use WSL (Windows Subsystem for Linux) — most production Python code runs on Linux.

pip and package management — how to install packages, read a requirements.txt, and understand what pip freeze does.

Phase 3: Specialise — pick your lane (Month 3–4)

This is the branching point. Which direction you choose should be based on what kind of work you want to do, not what pays the most right now (though we cover salaries in our Python salary guide).

Lane A: Backend web development (Django / FastAPI)

Who this is for: Developers who want to build web APIs, SaaS products, or work at startups and product companies.

What to learn:

HTTP fundamentals — understand requests, responses, status codes, headers, cookies, and sessions before touching a framework. Django and FastAPI will make much more sense with this foundation. MDN's HTTP overview is the best free resource.

Databases and SQL — learn to write SQL before using an ORM. Understand SELECT, WHERE, JOIN, GROUP BY, indexes, and why they matter for performance. PostgreSQL is the database of choice at most Indian product companies.

Django — start here if you are building full-featured web apps. Learn models, views, templates, the ORM, migrations, admin, and Django REST Framework for building APIs. Django's official tutorial at djangoproject.com is genuinely excellent.

FastAPI — learn this if you are building APIs specifically (not full web apps). FastAPI is faster, fully async, and has automatic API documentation built in. It has grown enormously in adoption at Indian startups in 2025–26. The FastAPI official tutorial is one of the best framework tutorials ever written.

REST API design — know what makes a good API: proper HTTP methods, meaningful status codes, versioning, pagination, authentication (JWT), and error response formatting.

Authentication — implement JWT-based auth at least once from scratch. Understanding what a JWT actually contains and how it is verified is a common interview question.

What to build at this stage: A real REST API. Not a todo app — something that solves an actual problem. A personal finance tracker with user authentication. A book collection API with search. A blogging API (very relevant for Zyvop). Something you can demo and discuss in detail in an interview.

Lane B: Data science and machine learning

Who this is for: Developers interested in working with data, building ML models, or moving into AI engineering roles.

What to learn:

NumPy and pandas — these are the foundations of data work in Python. NumPy for numerical operations, pandas for tabular data manipulation. Learn them deeply, not just surface-level. Real Python's pandas guide is one of the best free resources.

Data visualisation — Matplotlib for basic charts, Seaborn for statistical visualisation. Being able to visualise data is essential for understanding what you are working with.

scikit-learn — the standard machine learning library for classification, regression, clustering, and model evaluation. Learn the core concepts: train/test split, cross-validation, overfitting, regularisation. Do not skip understanding why each algorithm works.

Jupyter Notebooks — the standard environment for data exploration and sharing analysis. Learn to write clean, reproducible notebooks.

Statistics fundamentals — mean, median, variance, standard deviation, probability distributions, hypothesis testing. These are tested in data science interviews at Indian companies. Khan Academy's statistics content is free and surprisingly good for this.

For AI/ML specialisation in 2026: Once you have the above, explore LangChain for building LLM applications, the OpenAI or Anthropic APIs for AI integration, and basic model fine-tuning concepts. This is where salaries jump significantly in the current market.

Lane C: Automation and scripting

Who this is for: Developers who want to automate workflows, build internal tools, do test automation, or work in DevOps-adjacent roles.

What to learn: requests for HTTP automation, BeautifulSoup and Selenium for web scraping, schedule for task scheduling, smtplib for email automation, openpyxl for Excel automation, and Playwright for modern browser automation.

This lane has lower salary ceilings than web dev or ML at mid-senior level, but very high demand and relatively easy entry. Good for developers who want to get hired quickly and then upskill over time.

Phase 4: Testing, deployment, and production skills (Month 5–6)

This is what separates developers who "know Python" from developers who are actually employable at product companies. Most tutorials stop before this phase. That's why most tutorial graduates struggle in interviews.

Testing — learn pytest, the standard Python testing framework. Understand unit tests, integration tests, and the difference between mocking and real test data. Writing tests for your projects before your interview is a clear differentiator. The pytest official docs are comprehensive.

Docker — containerising your applications is now expected, not optional, at product companies. Learn to write a Dockerfile, build an image, run a container, and use Docker Compose for multi-service applications. Docker's official getting started guide is well-written.

Cloud basics (AWS or GCP) — you don't need to be a cloud architect, but you should be able to deploy a Python API to the cloud and connect it to a managed database. AWS's free tier lets you do all of this without paying. The AWS Solutions Architect Associate certification adds ₹2–5 LPA to salary offers in India — it is worth the effort if you have 6 weeks to prepare for it.

Basic CI/CD — understand what a GitHub Actions pipeline does and how to set one up to automatically run tests when you push code. This is standard at every product company.

What to build as a portfolio (and why most portfolios are wrong)

The mistake most learners make is building tutorial projects and calling them portfolio pieces. An interviewer who sees "I built a todo app" has seen that 500 times. What gets attention is a project that:

Solves a real problem (even a small personal one)
Has proper error handling and is actually deployed (not just on GitHub)
Has tests written for it
Has a clean README explaining what it does, why you built it, and how to run it

For Zyvop's tech niche, good portfolio project ideas: a real-time price tracker for Indian stocks using the NSE API, a Python bot that summarises job postings from Naukri and sends a daily digest, an API that analyses and scores resumes, a Django-based blog platform (literally what Zyvop is — great talking point in an interview).

The honest timeline

If you study 2–3 hours daily:

Phase 1: 8 weeks
Phase 2: overlapping, no extra time
Phase 3 (one lane): 8–10 weeks
Phase 4: 4–6 weeks
Total: 5–6 months to being genuinely interview-ready

If you study 1 hour daily or less, double that timeline. There are no shortcuts — only people who consistently practice and those who don't.

FAQ

Q: Should I learn Django or FastAPI first in 2026? If you want to work at a larger company or on full-featured web apps, start with Django — it has more structure and teaches you more about how web applications work end-to-end. If you want to build APIs specifically and are comfortable with asynchronous concepts, FastAPI is the better choice for 2026's job market.

Q: Do I need a computer science degree to become a Python developer in India? No, but you need the equivalent skills. Employers at product companies care about whether you can solve problems and write clean, tested code — not your degree. That said, your degree does affect how your resume gets filtered at large service companies like TCS and Infosys, which have formal eligibility criteria.

Q: What is the minimum Python skill level to start applying for jobs in India? Completing Phase 1 + Phase 2 + the basics of Phase 3 is typically enough to apply for junior/trainee roles at service companies. To have a realistic shot at product company roles, you need to complete all four phases and have at least one deployed project in your portfolio.

Following this roadmap and have questions at a specific stage? Drop them in the comments — the Zyvop author community includes developers at every level, and someone will have been exactly where you are.

Related reads on Zyvop:

Python Developer Salary in India 2026 — The Honest, City-Wise Guide
Best AI Tools for Developers in India 2026
Node.js Interview Questions Asked in India in 2026

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Build a Price Monitoring System with Python: Amazon, Shopify & eBay (2026)

ZyVOP — Thu, 11 Jun 2026 05:30:43 +0000

Why Price Monitoring Is the Most Practical Scraping Project

Amazon remains the largest e-commerce platform in the world, and extracting product data from it — prices, reviews, seller information — is one of the most common web scraping use cases. Whether you're building a deal-finder for personal use, doing competitive analysis for a business, or building a side product that alerts users when prices drop, a price monitoring system is arguably the most immediately useful scraper you can build.

Here's what makes it interesting technically in 2026:

Modern e-commerce sites — Shopify stores, Amazon, direct-to-consumer brands — render prices with JavaScript, hide them behind A/B tests, and protect their pages with sophisticated bot detection. A naive scraper that worked in 2022 breaks instantly today.

Amazon reprices millions of products every day, and according to industry monitoring data, 37 percent of active Amazon monitors check at least hourly and 12 percent run at five-minute intervals. Electronics and trending categories can move dozens of times per day.

This guide builds a complete, production-ready price monitoring system step by step — from scraping the price off a single page all the way to a scheduled multi-platform tracker with email alerts.

What We'll Build

By the end of this guide you'll have:

A multi-platform scraper that monitors prices on Amazon, Shopify stores, and eBay
A SQLite price history database that tracks every price change over time
A price-drop alert system that emails you when a product hits your target price
A daily scheduler that runs automatically without you touching it
A trend report showing price movements over time per product

Understanding the Platforms

Each platform has different scraping characteristics:

Platform	Rendering	Anti-Bot	Price Location	Difficulty
Amazon	Mostly server-side	Very aggressive	.a-price-whole + .a-price-fraction	Hard
Shopify	Server + JS hybrid	Low–Medium	/products/handle.json API	Easy
eBay	Server-side	Moderate	.x-price-primary	Medium
Generic sites	Varies	Low	Custom per site	Varies

Shopify is the easiest — every Shopify store exposes a clean JSON API at /products/[handle].json that returns the full product including all variant prices. Amazon is the hardest, as it reprices constantly and blocks aggressively.

Part 1: The Database Layer

Build the storage foundation first — everything else writes into it.

# db.py
import sqlite3
from datetime import datetime, timezone
from pathlib import Path

DB_PATH = "price_monitor.db"

def get_conn():
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row   # Access columns by name
    return conn

def init_db():
    """Create tables if they don't exist."""
    with get_conn() as conn:
        conn.executescript("""
            CREATE TABLE IF NOT EXISTS products (
                id          INTEGER PRIMARY KEY AUTOINCREMENT,
                name        TEXT NOT NULL,
                url         TEXT NOT NULL UNIQUE,
                platform    TEXT NOT NULL,
                target_price REAL,
                alert_email TEXT,
                active      INTEGER DEFAULT 1,
                added_at    TEXT DEFAULT (datetime('now'))
            );

            CREATE TABLE IF NOT EXISTS price_history (
                id          INTEGER PRIMARY KEY AUTOINCREMENT,
                product_id  INTEGER NOT NULL REFERENCES products(id),
                price       REAL,
                currency    TEXT DEFAULT 'USD',
                in_stock    INTEGER,
                scraped_at  TEXT DEFAULT (datetime('now'))
            );

            CREATE INDEX IF NOT EXISTS idx_history_product
                ON price_history(product_id, scraped_at DESC);
        """)
    print("Database initialised.")

def add_product(name, url, platform, target_price=None, alert_email=None):
    """Register a new product to monitor."""
    with get_conn() as conn:
        try:
            conn.execute("""
                INSERT INTO products (name, url, platform, target_price, alert_email)
                VALUES (?, ?, ?, ?, ?)
            """, (name, url, platform, target_price, alert_email))
            print(f"Added: {name}")
        except sqlite3.IntegrityError:
            print(f"Already exists: {url}")

def save_price(product_id, price, currency="USD", in_stock=True):
    """Record a price observation."""
    with get_conn() as conn:
        conn.execute("""
            INSERT INTO price_history (product_id, price, currency, in_stock)
            VALUES (?, ?, ?, ?)
        """, (product_id, price, currency, int(in_stock)))

def get_active_products():
    """Return all active monitored products."""
    with get_conn() as conn:
        return conn.execute(
            "SELECT * FROM products WHERE active = 1"
        ).fetchall()

def get_price_history(product_id, days=30):
    """Return recent price history for a product."""
    with get_conn() as conn:
        return conn.execute("""
            SELECT price, currency, in_stock, scraped_at
            FROM price_history
            WHERE product_id = ?
              AND scraped_at >= datetime('now', ?)
            ORDER BY scraped_at ASC
        """, (product_id, f"-{days} days")).fetchall()

def get_lowest_price(product_id):
    """Return the all-time lowest recorded price."""
    with get_conn() as conn:
        result = conn.execute(
            "SELECT MIN(price) FROM price_history WHERE product_id = ?",
            (product_id,)
        ).fetchone()
        return result[0]

Part 2: Scraping Amazon

Scraping Amazon prices with Python means pulling five related numbers off a product page: the current Buy Box price, the "was" price (list price), any Subscribe and Save price, per-variant prices, and the price of any other-sellers offers.

# scrapers/amazon.py
import re
import random
import asyncio
from curl_cffi.requests import AsyncSession
from bs4 import BeautifulSoup

HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "sec-ch-ua": '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Upgrade-Insecure-Requests": "1",
}

def parse_amazon_price(html: str) -> dict:
    """
    Extract price data from an Amazon product page.
    Targets the most stable selectors as of June 2026.
    """
    soup = BeautifulSoup(html, "lxml")

    result = {
        "price":      None,
        "was_price":  None,
        "currency":   "USD",
        "in_stock":   False,
        "title":      None,
        "asin":       None,
    }

    # ── Title ─────────────────────────────────────────────────
    title_el = soup.select_one("#productTitle")
    if title_el:
        result["title"] = title_el.get_text(strip=True)

    # ── Current Price (Buy Box) ────────────────────────────────
    # Method 1: Standard Buy Box price — most common location
    whole = soup.select_one(".a-price-whole")
    frac  = soup.select_one(".a-price-fraction")
    if whole:
        price_str = whole.get_text(strip=True).replace(",", "").rstrip(".")
        if frac:
            price_str += "." + frac.get_text(strip=True)
        try:
            result["price"] = float(price_str)
        except ValueError:
            pass

    # Method 2: Fallback — hidden offscreen price (more reliable on some pages)
    if not result["price"]:
        offscreen = soup.select_one(".a-offscreen")
        if offscreen:
            price_text = offscreen.get_text(strip=True)
            numbers = re.findall(r"[\d,]+\.?\d*", price_text.replace(",", ""))
            if numbers:
                try:
                    result["price"] = float(numbers[0])
                except ValueError:
                    pass

    # ── Was / List Price ──────────────────────────────────────
    was_el = soup.select_one(".basisPrice .a-offscreen")
    if was_el:
        was_text = was_el.get_text(strip=True)
        numbers = re.findall(r"[\d,]+\.?\d*", was_text.replace(",", ""))
        if numbers:
            try:
                result["was_price"] = float(numbers[0])
            except ValueError:
                pass

    # ── Stock Status ──────────────────────────────────────────
    avail_el = soup.select_one("#availability span")
    if avail_el:
        avail_text = avail_el.get_text(strip=True).lower()
        result["in_stock"] = "in stock" in avail_text

    # ── Currency (detect from symbol) ────────────────────────
    price_symbol = soup.select_one(".a-price-symbol")
    if price_symbol:
        symbol = price_symbol.get_text(strip=True)
        symbol_map = {"$": "USD", "£": "GBP", "€": "EUR", "₹": "INR"}
        result["currency"] = symbol_map.get(symbol, "USD")

    # ── ASIN from URL or page ─────────────────────────────────
    asin_input = soup.select_one("#ASIN")
    if asin_input:
        result["asin"] = asin_input.get("value")

    return result

async def scrape_amazon(url: str, proxy: str = None) -> dict:
    """Fetch and parse a single Amazon product page."""
    proxies = {"https": proxy} if proxy else None

    async with AsyncSession(impersonate="chrome120") as session:
        # Visit homepage first for a more natural session pattern
        await session.get(
            "https://www.amazon.com",
            headers=HEADERS,
            proxies=proxies,
            timeout=15
        )
        await asyncio.sleep(random.uniform(1.5, 3.0))

        response = await session.get(
            url,
            headers=HEADERS,
            proxies=proxies,
            timeout=20
        )

    if response.status_code != 200:
        raise Exception(f"Amazon returned {response.status_code}")

    # Detect block page
    if "Type the characters you see" in response.text or \
       "Enter the characters you see below" in response.text:
        raise Exception("CAPTCHA encountered — rotate proxy and retry")

    return parse_amazon_price(response.text)

Part 3: Scraping Shopify (The Easy Way)

Every Shopify store has a secret weapon: the /products/[handle].json endpoint returns clean JSON with full product data including all variant prices, no scraping needed.

# scrapers/shopify.py
import httpx
import re
from urllib.parse import urlparse

async def scrape_shopify(url: str) -> dict:
    """
    Extract price from any Shopify store using the built-in JSON API.
    Works on all Shopify stores — no CSS selectors needed.

    Supported URL formats:
      - https://store.com/products/my-product
      - https://store.myshopify.com/products/handle
    """
    parsed = urlparse(url)
    base   = f"{parsed.scheme}://{parsed.netloc}"

    # Extract product handle from URL path
    # URL pattern: /products/product-handle or /collections/x/products/handle
    path_match = re.search(r"/products/([^/?#]+)", parsed.path)
    if not path_match:
        raise ValueError(f"Cannot extract product handle from: {url}")

    handle   = path_match.group(1)
    json_url = f"{base}/products/{handle}.json"

    headers = {
        "User-Agent": "Mozilla/5.0 (compatible; PriceBot/1.0)",
        "Accept": "application/json",
    }

    async with httpx.AsyncClient(headers=headers, follow_redirects=True) as client:
        r = await client.get(json_url, timeout=15)
        r.raise_for_status()
        data = r.json()

    product  = data["product"]
    variants = product.get("variants", [])

    if not variants:
        return {"price": None, "in_stock": False, "title": product.get("title")}

    # Get the first available variant's price
    # (or lowest price if all variants have different prices)
    prices = []
    for v in variants:
        if v.get("available", True):
            try:
                prices.append(float(v["price"]))
            except (ValueError, KeyError):
                pass

    lowest_price = min(prices) if prices else None

    return {
        "title":          product.get("title"),
        "price":          lowest_price,
        "was_price":      float(variants[0].get("compare_at_price") or 0) or None,
        "currency":       "USD",  # Shopify JSON doesn't include currency — check storefront
        "in_stock":       any(v.get("available", False) for v in variants),
        "variants_count": len(variants),
        "vendor":         product.get("vendor"),
        "product_type":   product.get("product_type"),
    }

Part 4: Scraping eBay

# scrapers/ebay.py
import httpx
import re
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,*/*;q=0.8",
}

async def scrape_ebay(url: str) -> dict:
    """Extract price from an eBay listing page."""
    async with httpx.AsyncClient(headers=HEADERS, follow_redirects=True) as client:
        r = await client.get(url, timeout=20)
        r.raise_for_status()

    soup = BeautifulSoup(r.text, "lxml")
    result = {"price": None, "was_price": None, "in_stock": True,
              "currency": "USD", "title": None}

    # Title
    title_el = soup.select_one("h1.x-item-title__mainTitle span")
    if title_el:
        result["title"] = title_el.get_text(strip=True)

    # Current price
    price_el = soup.select_one(".x-price-primary span.ux-textspans")
    if price_el:
        price_text = price_el.get_text(strip=True)
        nums = re.findall(r"[\d,]+\.?\d*", price_text.replace(",", ""))
        if nums:
            try:
                result["price"] = float(nums[0])
            except ValueError:
                pass

    # Was price (strikethrough)
    was_el = soup.select_one(".x-additional-info__original-price span")
    if was_el:
        was_text = was_el.get_text(strip=True)
        nums = re.findall(r"[\d,]+\.?\d*", was_text.replace(",", ""))
        if nums:
            try:
                result["was_price"] = float(nums[0])
            except ValueError:
                pass

    # Stock status
    qty_el = soup.select_one(".d-quantity__availability")
    if qty_el:
        qty_text = qty_el.get_text(strip=True).lower()
        result["in_stock"] = "sold out" not in qty_text and "unavailable" not in qty_text

    return result

Part 5: The Unified Monitor

# monitor.py
import asyncio
import random
from db import get_active_products, save_price, get_lowest_price, init_db
from scrapers.amazon import scrape_amazon
from scrapers.shopify import scrape_shopify
from scrapers.ebay import scrape_ebay

SCRAPER_MAP = {
    "amazon":  scrape_amazon,
    "shopify": scrape_shopify,
    "ebay":    scrape_ebay,
}

async def check_product(product) -> dict | None:
    """Scrape price for one product and save to database."""
    scraper = SCRAPER_MAP.get(product["platform"])
    if not scraper:
        print(f"  No scraper for platform: {product['platform']}")
        return None

    try:
        data = await scraper(product["url"])
        if data.get("price") is None:
            print(f"  [{product['name']}] No price found")
            return None

        save_price(
            product_id=product["id"],
            price=data["price"],
            currency=data.get("currency", "USD"),
            in_stock=data.get("in_stock", True),
        )

        print(f"  [{product['name']}] ${data['price']:.2f}"
              f"{' ⚠ OUT OF STOCK' if not data.get('in_stock') else ''}")
        return data

    except Exception as e:
        print(f"  [{product['name']}] Error: {e}")
        return None

async def run_monitor():
    """Run one monitoring cycle across all active products."""
    products = get_active_products()
    if not products:
        print("No products to monitor. Add some with add_product().")
        return

    print(f"\nMonitoring {len(products)} products...\n")
    results = {}

    for product in products:
        data = await check_product(product)
        if data:
            results[product["id"]] = data
            # Polite delay between requests
            await asyncio.sleep(random.uniform(2.0, 5.0))

    return results

Part 6: Price-Drop Email Alerts

# alerts.py
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from db import get_conn, get_lowest_price

# Configure your email credentials
SMTP_HOST     = "smtp.gmail.com"
SMTP_PORT     = 587
SENDER_EMAIL  = "your_monitor@gmail.com"
SENDER_PASS   = "your_app_password"   # Use Gmail App Password, not your real password

def send_alert(to_email: str, product_name: str, current_price: float,
               target_price: float, product_url: str, currency: str = "USD"):
    """Send a price-drop email alert."""
    symbol = {"USD": "$", "GBP": "£", "EUR": "€", "INR": "₹"}.get(currency, "$")

    subject = f"🔔 Price Drop Alert: {product_name}"
    body = f"""
    Great news! The price for a product you're monitoring has dropped.

    Product : {product_name}
    Current price : {symbol}{current_price:.2f}
    Your target   : {symbol}{target_price:.2f}
    Savings       : {symbol}{target_price - current_price:.2f}

    👉 Buy now: {product_url}

    --
    Python Price Monitor
    """

    msg = MIMEMultipart()
    msg["From"]    = SENDER_EMAIL
    msg["To"]      = to_email
    msg["Subject"] = subject
    msg.attach(MIMEText(body, "plain"))

    try:
        with smtplib.SMTP(SMTP_HOST, SMTP_PORT) as server:
            server.starttls()
            server.login(SENDER_EMAIL, SENDER_PASS)
            server.send_message(msg)
        print(f"  📧 Alert sent to {to_email} for {product_name}")
    except Exception as e:
        print(f"  Alert send failed: {e}")

def check_and_alert(product, current_price: float):
    """Check if current price triggers an alert and send email if so."""
    target = product["target_price"]
    email  = product["alert_email"]

    if not target or not email:
        return  # No alert configured for this product

    if current_price <= target:
        print(f"  🎯 Target hit! {product['name']}: ${current_price:.2f} ≤ ${target:.2f}")
        send_alert(
            to_email=email,
            product_name=product["name"],
            current_price=current_price,
            target_price=target,
            product_url=product["url"],
        )

Part 7: Price Trend Reporting

# report.py
import pandas as pd
from db import get_conn, get_active_products, get_price_history

def generate_price_report(days: int = 30) -> pd.DataFrame:
    """Generate a full price trend report for all monitored products."""
    products = get_active_products()
    rows = []

    for product in products:
        history = get_price_history(product["id"], days=days)
        if not history:
            continue

        prices = [row["price"] for row in history if row["price"]]
        if not prices:
            continue

        latest   = prices[-1]
        lowest   = min(prices)
        highest  = max(prices)
        first    = prices[0]
        change   = ((latest - first) / first * 100) if first else 0

        rows.append({
            "product":        product["name"],
            "platform":       product["platform"],
            "current_price":  round(latest, 2),
            "lowest_price":   round(lowest, 2),
            "highest_price":  round(highest, 2),
            "price_change_%": round(change, 1),
            "observations":   len(prices),
            "target_price":   product["target_price"],
            "below_target":   latest <= product["target_price"] if product["target_price"] else None,
        })

    df = pd.DataFrame(rows)
    return df

def print_report():
    df = generate_price_report(days=30)
    if df.empty:
        print("No price history yet. Run the monitor first.")
        return

    print("\n══ PRICE MONITORING REPORT (Last 30 days) ══\n")
    print(df.to_string(index=False))

    below = df[df["below_target"] == True]
    if not below.empty:
        print(f"\n🎯 {len(below)} product(s) currently at or below target price:")
        for _, row in below.iterrows():
            print(f"   • {row['product']}: ${row['current_price']:.2f} "
                  f"(target: ${row['target_price']:.2f})")

Part 8: Complete Setup and Scheduler

# main.py — run this file to start monitoring
import asyncio
import schedule
import time
from db import init_db, add_product
from monitor import run_monitor
from alerts import check_and_alert
from report import print_report
from db import get_active_products, get_price_history

def setup():
    """Initialise database and add products to monitor."""
    init_db()

    # Amazon products — use full product URLs
    add_product(
        name="Sony WH-1000XM5 Headphones",
        url="https://www.amazon.com/dp/B09XS7JWHH",
        platform="amazon",
        target_price=280.00,
        alert_email="you@example.com"
    )

    # Shopify store products
    add_product(
        name="Example Shopify Product",
        url="https://some-shopify-store.com/products/product-handle",
        platform="shopify",
        target_price=50.00,
        alert_email="you@example.com"
    )

    # eBay listings
    add_product(
        name="Vintage Camera on eBay",
        url="https://www.ebay.com/itm/123456789",
        platform="ebay",
        target_price=120.00,
        alert_email="you@example.com"
    )

async def monitoring_cycle():
    """One full monitoring pass with alert checking."""
    products = get_active_products()
    from monitor import check_product
    from scrapers.amazon import scrape_amazon
    from scrapers.shopify import scrape_shopify
    from scrapers.ebay import scrape_ebay

    for product in products:
        data = await check_product(product)
        if data and data.get("price"):
            check_and_alert(product, data["price"])

    print_report()

def run_scheduled():
    """Run monitoring on a schedule."""
    asyncio.run(monitoring_cycle())
    print("Next run in 6 hours.")

if __name__ == "__main__":
    setup()

    # Run immediately on start
    asyncio.run(monitoring_cycle())

    # Then every 6 hours
    schedule.every(6).hours.do(run_scheduled)

    print("\nScheduler running. Press Ctrl+C to stop.")
    while True:
        schedule.run_pending()
        time.sleep(60)

Pro Tips for Reliable Price Monitoring

Tip 1 — Store raw HTML alongside prices If your parser breaks after a site redesign, you can re-parse from stored HTML without re-scraping:

# Add to save_price()
with open(f"html_cache/{product_id}_{timestamp}.html", "w") as f:
    f.write(raw_html)

Tip 2 — Track percentage drops, not just absolute prices

def is_significant_drop(old_price, new_price, threshold_pct=5.0) -> bool:
    """Return True if price dropped by more than threshold_pct percent."""
    if not old_price or not new_price:
        return False
    drop_pct = (old_price - new_price) / old_price * 100
    return drop_pct >= threshold_pct

Tip 3 — Handle dynamic pricing variations Amazon shows different prices to different users based on their location, login state, browsing history, and time of day. To get consistent readings, always scrape with the same headers, same proxy location, and without cookies from previous sessions.

Tip 4 — Monitor variant-level prices on Shopify Shopify products often have multiple variants (sizes, colours) at different prices. Monitor all variants to catch deals on specific options:

all_variant_prices = {
    v["title"]: float(v["price"])
    for v in product["variants"]
    if v.get("available")
}

FAQ

Q: Is scraping Amazon prices legal? Scraping Amazon is possible and legal as long as you follow the guidelines and stay away from any login-protected personal information. Price data displayed publicly is generally fair game for personal use. Don't republish or resell the scraped data.

Q: How do I avoid getting blocked by Amazon? Use curl_cffi with impersonate="chrome120", rotate residential proxies, add 2–5 second delays between requests, and never scrape more than a few hundred products per day per IP.

Q: What if the price selector breaks? Amazon changes its HTML structure regularly. When that happens: open the product page in Chrome DevTools, find the price element, copy its updated selector, and update parse_amazon_price(). Use data- attributes when available — they're more stable than class names.

Q: Can I monitor prices on Indian e-commerce sites like Flipkart or Meesho? Yes — both are server-rendered and moderately protected. Use httpx with curl_cffi impersonation for Flipkart. Meesho is React-rendered and needs Playwright.

Summary

Component	Tool	What it does
Amazon scraper	curl_cffi + BeautifulSoup	TLS impersonation + price extraction
Shopify scraper	httpx + JSON API	Clean structured data, no CSS selectors
eBay scraper	httpx + BeautifulSoup	Server-rendered price extraction
Storage	SQLite	Price history with full trend data
Alerts	smtplib	Email when target price is hit
Reporting	pandas	Price trend summary across products
Scheduling	schedule	Runs every N hours automatically

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

I Built an MCP Server in 40 Minutes. Here's Exactly What Happened.

ZyVOP — Wed, 10 Jun 2026 04:43:49 +0000

I kept seeing MCP mentioned everywhere — in Claude Code docs, in Cursor settings, in every "AI engineering stack" post on LinkedIn. I kept nodding like I understood it.

I didn't.

So last week I sat down and built one from scratch. A real MCP server, connected to a real Postgres database, that Claude can actually query. No toy examples. No hand-waving over the hard parts.

This is the full account of what happened — including the two hours I lost to a configuration mistake that had nothing to do with code, and the exact moment the whole thing clicked.

First: what MCP actually is, in one paragraph

Before MCP existed, connecting an AI to your database meant writing a custom integration for every AI tool separately. One integration for Claude, another for Cursor, another for GitHub Copilot. Three tools, three integrations, all slightly different.

MCP is a standardised protocol — think of it as USB for AI tools. You write one MCP server that connects to your Postgres database, and every MCP-compatible client (Claude Desktop, Claude Code, Cursor, Windsurf, VS Code Copilot) can use it immediately. The official server registry crossed 2,000 community implementations by Q1 2026, and SDK downloads jumped roughly 970x in 18 months. It became the standard faster than almost any protocol I can remember.

That's the pitch. Here's the build.

What we're building

A Python MCP server with two tools:

list_tables — shows Claude what tables exist in your database
query_database — runs a read-only SQL query and returns results

By the end, you'll open Claude Desktop, type "what tables are in my database?" and get a real answer from your actual Postgres instance.

Prerequisites:

Python 3.10 or higher
A Postgres database (local or cloud — Neon has a free tier that works perfectly)
Claude Desktop installed (download here)
Basic familiarity with Python async/await

Step 1: Set up the project

uv is now the standard for MCP Python projects. It's faster than pip and handles virtual environments automatically. If you don't have it:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create the project
uv init zyvop-mcp-server
cd zyvop-mcp-server

# Install dependencies
uv add "mcp[cli]" fastmcp asyncpg python-dotenv

If you prefer pip:

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install "mcp[cli]" fastmcp asyncpg python-dotenv

Create a .env file for your database credentials:

DATABASE_URL=postgresql://user:password@localhost:5432/yourdb

Never hardcode credentials. Never commit .env. Add it to .gitignore right now before you forget.

Step 2: Write the server

Create server.py. This is the full working server — nothing left out:

import asyncio
import asyncpg
from fastmcp import FastMCP
from dotenv import load_dotenv
import os

load_dotenv()

# FastMCP does the heavy lifting — one line to create the server
mcp = FastMCP(
    name="postgres-assistant",
    instructions="You are a database assistant. Use list_tables first to understand 
    the schema, then query_database to answer questions about the data."
)

DATABASE_URL = os.getenv("DATABASE_URL")

async def get_connection():
    """Create a database connection."""
    return await asyncpg.connect(DATABASE_URL)

@mcp.tool()
async def list_tables() -> str:
    """
    List all tables in the database with their column names and types.
    Always call this before querying to understand the schema.
    """
    conn = await get_connection()
    try:
        rows = await conn.fetch("""
            SELECT 
                table_name,
                column_name,
                data_type
            FROM information_schema.columns
            WHERE table_schema = 'public'
            ORDER BY table_name, ordinal_position
        """)

        if not rows:
            return "No tables found in the public schema."

        # Group by table name for readable output
        tables = {}
        for row in rows:
            table = row['table_name']
            if table not in tables:
                tables[table] = []
            tables[table].append(f"  {row['column_name']} ({row['data_type']})")

        result = []
        for table_name, columns in tables.items():
            result.append(f"Table: {table_name}")
            result.extend(columns)
            result.append("")

        return "\n".join(result)
    finally:
        await conn.close()

@mcp.tool()
async def query_database(sql: str) -> str:
    """
    Run a read-only SQL SELECT query against the database.
    Only SELECT statements are allowed — no INSERT, UPDATE, DELETE, or DROP.
    Returns results as a formatted table.

    Args:
        sql: A valid SQL SELECT statement
    """
    # Safety check — only allow SELECT
    sql_clean = sql.strip().upper()
    if not sql_clean.startswith("SELECT"):
        return "Error: Only SELECT queries are allowed. Received: " + sql[:50]

    # Block dangerous keywords even inside SELECT
    dangerous = ["DROP", "DELETE", "INSERT", "UPDATE", "TRUNCATE", "ALTER", "CREATE"]
    for keyword in dangerous:
        if keyword in sql_clean:
            return f"Error: Query contains forbidden keyword: {keyword}"

    conn = await get_connection()
    try:
        rows = await conn.fetch(sql)

        if not rows:
            return "Query returned no results."

        # Format as readable table
        headers = list(rows[0].keys())
        header_row = " | ".join(headers)
        separator = "-" * len(header_row)

        result_rows = [header_row, separator]
        for row in rows[:50]:  # Limit to 50 rows to avoid token bloat
            result_rows.append(" | ".join(str(val) for val in row.values()))

        if len(rows) > 50:
            result_rows.append(f"\n... and {len(rows) - 50} more rows (query returned {len(rows)} total)")

        return "\n".join(result_rows)

    except asyncpg.PostgresError as e:
        return f"Database error: {str(e)}"
    finally:
        await conn.close()

if __name__ == "__main__":
    mcp.run()

Three things worth noting in this code:

The docstrings are not optional. Without a docstring on your tool function, the AI model has no idea what the tool does and will either refuse to use it or use it wrong. The docstring is literally what Claude reads to decide whether to call your tool.

The 50-row limit on results is deliberate. Token bloat is the number one production pain point — developers report tool schemas and responses consuming 15,000+ tokens before the agent even starts reasoning. Return the minimum useful data. Claude can always ask for more.

The safety checks are not paranoia. A SQL injection via your MCP server would be Claude running DROP TABLE users on your behalf. The keyword check is basic but catches the obvious cases.

Step 3: Test locally with the MCP Inspector

Before connecting to Claude, test with the official inspector:

fastmcp dev server.py

This opens a browser UI at http://localhost:6274 where you can call your tools directly, see the raw JSON-RPC messages, and verify everything works before involving Claude.

Call list_tables first. If it returns your schema, you're in good shape. If it returns an error, it's almost certainly the DATABASE_URL — double-check the connection string format.

Step 4: Connect to Claude Desktop

Find your Claude Desktop config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add your server:

{
  "mcpServers": {
    "postgres-assistant": {
      "command": "uv",
      "args": [
        "run",
        "--project",
        "/absolute/path/to/zyvop-mcp-server",
        "python",
        "server.py"
      ],
      "env": {
        "DATABASE_URL": "postgresql://user:password@localhost:5432/yourdb"
      }
    }
  }
}

Use the absolute path. This is where I lost two hours. A relative path like ./zyvop-mcp-server fails silently on some systems — Claude Desktop starts, shows no error, but the server never loads. Absolute path, always.

Restart Claude Desktop completely. Look for a hammer icon in the bottom-right of the chat input — that's your MCP tools indicator. If it's there, your server connected.

The moment it clicked

I typed: "What tables do I have in my database?"

Claude called list_tables. Returned my schema. Then I asked: "How many users signed up last week?"

It wrote a SQL query, called query_database, got the result, and answered me in plain English.

No code. No terminal. Just a question and an answer, backed by my actual data.

That was the moment I understood why MCP had taken off. It's not impressive as a demo. It's impressive as a workflow — the feeling of having a tool that can navigate your actual systems rather than giving you generic advice about systems it imagines.

What breaks in production (and how to fix it)

Connection pool exhaustion. The server above creates a new connection per query. For low-traffic personal tools that's fine. For anything handling multiple simultaneous requests, create a connection pool at startup:

pool = None

@mcp.lifespan
async def lifespan():
    global pool
    pool = await asyncpg.create_pool(DATABASE_URL, min_size=2, max_size=10)
    yield
    await pool.close()

# Then in your tools:
async with pool.acquire() as conn:
    rows = await conn.fetch(sql)

The "dumps 43 tools into context" problem. One developer reported that GitHub's own MCP server "dumps 43 tools into the context window" before doing anything, destroying agent performance. Keep your tool count small — 5 to 8 focused tools outperforms 40 broad ones. Every tool adds tokens to every conversation whether Claude uses it or not.

Responses that aren't strings. A tool that returned a raw Python dict rendered fine in the MCP Inspector but came back truncated inside Claude Desktop. Wrapping the return value as a typed text string fixed it instantly. Always return strings from your tools, not dicts or lists.

What to build next

Once your Postgres server works, the same pattern extends to anything:

A GitHub MCP server that Claude can use to read your issues and PRs
A filesystem server that lets Claude navigate your project directory
An API wrapper that connects Claude to any internal service your company uses

The official MCP server repository has reference implementations for Slack, Google Drive, GitHub, and more — good reading for patterns to follow.

The honest take

Building this took 40 minutes of actual coding and 2 hours of debugging a config file path.

The coding part is genuinely simple — FastMCP eliminates the boilerplate and provides a decorator-based framework that turns a Python function into a fully typed MCP tool with one line. If you know Python async, you know enough to build this.

The debugging part is a rite of passage. The MCP Inspector is your friend. The Claude Desktop logs (at ~/Library/Logs/Claude/ on macOS) are your other friend. Between those two, every error I hit was eventually traceable.

The thing on the other side is worth it. Give it an afternoon.

Built something with MCP? Drop what you connected it to in the comments — the most interesting integrations I've seen have come from developers solving very specific, unglamorous problems.

External links:

MCP Python SDK — official SDK and changelog
FastMCP docs — the fastest way to build Python MCP servers
MCP Inspector — test your server before connecting to Claude
Neon Postgres — free serverless Postgres, works perfectly for MCP development
Official MCP server examples — reference implementations

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Audit Logging in Node.js: Who Did What, When, and How to Prove It

ZyVOP — Tue, 09 Jun 2026 02:33:44 +0000

Most applications log errors. Fewer log the events that matter to the business: who changed a permission, who exported a CSV of customer data, who deleted a record that cannot be recovered, who approved a payment. These are the events that a regulator, an auditor, a support team, or a forensic investigation needs to reconstruct what happened.

Application logs and audit logs are different things. Application logs are operational — they tell you what your system did. Audit logs are evidentiary — they tell you what your users did, in a form you can trust. GDPR Article 30 requires organizations to maintain a record of processing activities, and audit trails are the technical implementation of that requirement.

This guide covers the full implementation: an immutable audit log table, middleware that captures every state change, querying the audit trail, and the GDPR considerations that determine what you log and how long you keep it.

What Belongs in an Audit Log

Not everything. Logging too much is a problem — collecting too much information in logs can violate GDPR principles. Logs themselves become repositories of personal data and require the same protections as primary datasets. Excessive logging increases the attack surface and complicates compliance efforts.

Log the events that answer: "If something went wrong, could I reconstruct exactly what happened and who was responsible?"

Log these:

Authentication events: login, logout, failed login, password change, MFA changes
Permission changes: role assignments, access grants/revocations
Data exports: any bulk export of user or customer data
Destructive actions: delete, archive, purge
Financial events: payment attempts, refunds, plan changes
Admin actions: any action taken by an admin on behalf of another user
Sensitive data access: viewing PII, medical records, financial data

Do not log these:

Read operations on non-sensitive data (viewing a product listing)
Internal system events (cache misses, background job progress)
Raw personal data in the log payload — use IDs and hashed identifiers

The Audit Log Schema

The audit log table must be append-only. No updates, no deletes — including from your own application.

CREATE TABLE audit_logs (
  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),

  -- Who did it
  actor_id     UUID,           -- NULL for system/anonymous actions
  actor_email  TEXT,           -- Denormalized — survives user deletion
  actor_role   TEXT,
  actor_ip     INET,

  -- What they did
  action       TEXT NOT NULL,  -- 'user.login', 'payment.refunded', 'role.changed'
  resource     TEXT,           -- 'user', 'order', 'subscription'
  resource_id  TEXT,           -- The affected record ID

  -- Tenant context
  tenant_id    UUID,

  -- The change
  old_value    JSONB,          -- State before the action
  new_value    JSONB,          -- State after the action
  metadata     JSONB,          -- Request context, extra fields

  -- When
  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Indexes for common query patterns
CREATE INDEX idx_audit_actor_id      ON audit_logs(actor_id);
CREATE INDEX idx_audit_tenant_id     ON audit_logs(tenant_id);
CREATE INDEX idx_audit_resource      ON audit_logs(resource, resource_id);
CREATE INDEX idx_audit_action        ON audit_logs(action);
CREATE INDEX idx_audit_created_at    ON audit_logs(created_at DESC);

-- Prevent updates and deletes — audit logs are immutable
CREATE RULE audit_logs_no_update AS ON UPDATE TO audit_logs DO INSTEAD NOTHING;
CREATE RULE audit_logs_no_delete AS ON DELETE TO audit_logs DO INSTEAD NOTHING;

The old_value and new_value columns capture the state before and after a change — critical for reconstructing what happened. Denormalizing actor_email means the audit trail survives if the user account is later deleted.

The CREATE RULE statements are database-level enforcement. Even if application code has a bug that tries to update or delete an audit record, the database prevents it.

The Audit Logger

// src/lib/auditLogger.ts
import db from './db';

interface AuditEvent {
  actorId?:    string;
  actorEmail?: string;
  actorRole?:  string;
  actorIp?:    string;
  action:      string;   // 'user.created', 'role.changed', 'payment.refunded'
  resource?:   string;
  resourceId?: string;
  tenantId?:   string;
  oldValue?:   unknown;
  newValue?:   unknown;
  metadata?:   Record<string, unknown>;
}

export async function audit(event: AuditEvent): Promise<void> {
  try {
    await db.query(`
      INSERT INTO audit_logs (
        actor_id, actor_email, actor_role, actor_ip,
        action, resource, resource_id,
        tenant_id, old_value, new_value, metadata
      ) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
    `, [
      event.actorId    || null,
      event.actorEmail || null,
      event.actorRole  || null,
      event.actorIp    || null,
      event.action,
      event.resource   || null,
      event.resourceId || null,
      event.tenantId   || null,
      event.oldValue   ? JSON.stringify(event.oldValue)  : null,
      event.newValue   ? JSON.stringify(event.newValue)  : null,
      event.metadata   ? JSON.stringify(event.metadata)  : null,
    ]);
  } catch (err) {
    // Audit log failures must not break the main operation
    // But they should be visible — log the failure loudly
    logger.error({
      error:  (err as Error).message,
      action: event.action,
    }, 'AUDIT LOG WRITE FAILED');
  }
}

Structured Action Names

Use dot-notation action names that are consistent and queryable:

// src/lib/auditActions.ts
export const AuditActions = {
  // Auth
  AUTH_LOGIN:           'auth.login',
  AUTH_LOGIN_FAILED:    'auth.login.failed',
  AUTH_LOGOUT:          'auth.logout',
  AUTH_PASSWORD_CHANGED:'auth.password.changed',
  AUTH_MFA_ENABLED:     'auth.mfa.enabled',

  // Users
  USER_CREATED:         'user.created',
  USER_UPDATED:         'user.updated',
  USER_DELETED:         'user.deleted',
  USER_ROLE_CHANGED:    'user.role.changed',
  USER_INVITED:         'user.invited',

  // Data
  DATA_EXPORTED:        'data.exported',
  DATA_DELETED:         'data.deleted',

  // Billing
  SUBSCRIPTION_CREATED: 'subscription.created',
  SUBSCRIPTION_CANCELLED:'subscription.cancelled',
  PAYMENT_REFUNDED:     'payment.refunded',

  // Admin
  ADMIN_IMPERSONATED:   'admin.impersonated',
  ADMIN_CONFIG_CHANGED: 'admin.config.changed',
} as const;

export type AuditAction = typeof AuditActions[keyof typeof AuditActions];

Using the Audit Logger in Route Handlers

// src/routes/users.ts
import { audit, AuditActions } from '../lib/auditLogger';

router.patch('/users/:id/role', authenticate, requireRole('admin'), async (req, res) => {
  const { id } = req.params;
  const { role } = req.body;

  const existing = await getUserById(id, req.tenant.id);
  if (!existing) return res.status(404).json({ error: 'User not found' });

  const updated = await updateUserRole(id, role, req.tenant.id);

  // Audit the role change with before/after state
  await audit({
    actorId:    req.user.id,
    actorEmail: req.user.email,
    actorRole:  req.user.role,
    actorIp:    req.ip,
    action:     AuditActions.USER_ROLE_CHANGED,
    resource:   'user',
    resourceId: id,
    tenantId:   req.tenant.id,
    oldValue:   { role: existing.role },
    newValue:   { role },
    metadata: {
      requestId: req.id,
      userAgent: req.headers['user-agent'],
    },
  });

  res.json(updated);
});

// Auth events — login and failed login
router.post('/auth/login', async (req, res) => {
  const { email, password } = req.body;
  const user = await findUserByEmail(email);

  if (!user || !(await verifyPassword(password, user.passwordHash))) {
    // Log failed attempts — useful for detecting brute force
    await audit({
      actorIp: req.ip,
      action:  AuditActions.AUTH_LOGIN_FAILED,
      metadata: {
        email,          // Email attempted — not a real user field
        requestId: req.id,
        userAgent: req.headers['user-agent'],
      },
    });
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  const tokens = generateTokens(user);

  await audit({
    actorId:    user.id,
    actorEmail: user.email,
    actorRole:  user.role,
    actorIp:    req.ip,
    tenantId:   user.tenantId,
    action:     AuditActions.AUTH_LOGIN,
    metadata: {
      requestId: req.id,
      userAgent: req.headers['user-agent'],
    },
  });

  res.json(tokens);
});

Querying the Audit Trail

// src/routes/admin/audit.ts

// Get audit trail for a specific resource
router.get('/admin/audit/:resource/:id', authenticate, requireRole('admin'), async (req, res) => {
  const { resource, id } = req.params;
  const limit  = parseInt(req.query.limit as string) || 50;
  const cursor = req.query.cursor as string | undefined;

  const result = await db.query(`
    SELECT
      id, actor_id, actor_email, actor_role, actor_ip,
      action, resource, resource_id,
      old_value, new_value, metadata,
      created_at
    FROM audit_logs
    WHERE
      resource    = $1
      AND resource_id = $2
      AND tenant_id   = $3
      ${cursor ? 'AND created_at < $4' : ''}
    ORDER BY created_at DESC
    LIMIT ${cursor ? '$5' : '$4'}
  `, cursor
    ? [resource, id, req.tenant.id, cursor, limit + 1]
    : [resource, id, req.tenant.id, limit + 1]
  );

  const rows = result.rows;
  const hasMore = rows.length > limit;
  if (hasMore) rows.pop();

  res.json({
    data:     rows,
    hasMore,
    nextCursor: hasMore ? rows[rows.length - 1].created_at : null,
  });
});

// Activity for a specific user — for "session history" or DSAR requests
router.get('/admin/audit/actor/:userId', authenticate, requireRole('admin'), async (req, res) => {
  const result = await db.query(`
    SELECT action, resource, resource_id, metadata, created_at
    FROM audit_logs
    WHERE actor_id  = $1
      AND tenant_id = $2
    ORDER BY created_at DESC
    LIMIT 100
  `, [req.params.userId, req.tenant.id]);

  res.json(result.rows);
});

GDPR Considerations

Logs must have defined retention periods. Exceeding that timeframe without reason, even accidentally, constitutes a breach of the regulation.

Retention policy:

-- Automated cleanup — run as a scheduled job
-- Retain audit logs for 2 years (adjust to your regulatory requirement)
DELETE FROM audit_logs
WHERE created_at < NOW() - INTERVAL '2 years';

Data minimisation in log payloads:

Avoid logging raw personal data such as full names, addresses, phone numbers, or full data records. Where necessary, replace them with pseudonymous identifiers or hashed values.

// BAD — full PII in audit log
await audit({
  action:   AuditActions.USER_UPDATED,
  newValue: {
    name:    'Jane Smith',
    email:   'jane@example.com',
    address: '123 Main St, London',
    dob:     '1985-03-15',
  },
});

// GOOD — reference IDs, not PII
await audit({
  action:     AuditActions.USER_UPDATED,
  resource:   'user',
  resourceId: user.id,
  oldValue:   { fieldsChanged: ['email', 'address'] },  // What changed
  newValue:   { fieldsChanged: ['email', 'address'] },  // Not the values
});

DSAR (Data Subject Access Request) support:

Under GDPR, users can request all data you hold about them including audit logs that reference them.

// Generate DSAR package for a user — all audit records referencing their ID
async function generateDSARReport(userId: string, tenantId: string) {
  const result = await db.query(`
    SELECT action, resource, resource_id, created_at, actor_ip
    FROM audit_logs
    WHERE (actor_id = $1 OR resource_id = $1)
      AND tenant_id = $2
    ORDER BY created_at DESC
  `, [userId, tenantId]);

  return {
    userId,
    generatedAt: new Date(),
    auditTrail:  result.rows,
  };
}

The Compliance Checklist

✅ Audit table is append-only — DB rules prevent UPDATE and DELETE
✅ Actor email is denormalized — audit trail survives account deletion
✅ Action names are structured and consistent (dot notation)
✅ Old and new values captured for state-change events
✅ Audit failures logged loudly but don't break main operations
✅ Retention policy defined and automated — default 2 years
✅ PII not stored in audit payloads — IDs and field names only
✅ DSAR query ready — can export all records for a given user ID
✅ Failed authentication attempts logged — brute force detection
✅ Admin impersonation logged — who accessed whose account

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Automate Your Code Quality with Git Hooks (And Never Argue in Code Review Again)

ZyVOP — Tue, 09 Jun 2026 02:33:21 +0000

The Problem Git Hooks Solve

Picture a typical pull request review. Someone opens a PR and a reviewer digs in — but before getting to the actual logic, they're leaving comments like "unused import on line 12," "forgot to run prettier," "this console.log shouldn't be here," and "can you write a more descriptive commit message than 'fix'?" The author fixes the comments. Another review pass happens. Eventually the real work gets looked at.

None of that first round should have involved a human. Those are deterministic rules — a computer can check them in milliseconds, consistently, for everyone on the team, before the code ever leaves the author's machine. Git hooks are how you do that.

How Git Hooks Actually Work

When you run git init, Git creates a .git/ folder. Inside it is a hooks/ directory filled with sample scripts — pre-commit.sample, commit-msg.sample, pre-push.sample, and others. Remove the .sample extension, make the file executable, and Git runs it automatically at that moment in the workflow.

The mechanism is simple but powerful: if a hook script exits with a non-zero status code, Git stops what it was doing. A failing pre-commit hook aborts the commit. A failing pre-push hook aborts the push. The developer sees the error output and knows exactly what to fix before proceeding.

Here are the hooks you'll actually use:

Hook	When It Fires	Best Used For
pre-commit	Before the commit object is created	Linting, formatting, fast checks
commit-msg	After the commit message is written	Enforcing message conventions
prepare-commit-msg	Before the commit message editor opens	Pre-filling message templates
pre-push	Before data is sent to the remote	Running the full test suite
post-merge	After a successful merge or pull	Auto-installing new dependencies

The Raw Hook Problem (And Why You Need Tooling)

Here's the catch with raw Git hooks: .git/hooks/ is not tracked by Git. That means every hook you write lives only on your machine. When a teammate clones the repository, they get nothing. You'd have to manually distribute scripts and have every developer install them — which in practice means it never happens.

There's a second problem too. Raw hooks run against your entire working tree. On a large project, linting all 4,000 files every time you commit takes 30–60 seconds. Developers get annoyed and start using git commit --no-verify to skip everything. Once that habit forms, your hooks are useless.

Both problems have clean solutions.

Husky: Share Hooks Through the Repository

Husky solves the sharing problem. Instead of putting hook scripts in .git/hooks/, it stores them in .husky/ — a regular directory that IS tracked by Git. It then tells Git to look for hooks there instead of the default location.

Install and initialize:

npm install --save-dev husky
npx husky init

husky init creates the .husky/ directory with a sample pre-commit hook and adds a "prepare": "husky" script to your package.json. That prepare script is the magic: npm runs it automatically after every npm install. So when a developer clones your repo and installs dependencies, Husky is set up without any extra step.

Your project structure now looks like this:

your-project/
├── .husky/
│   ├── pre-commit       ← tracked by Git, shared with everyone
│   └── commit-msg
├── src/
└── package.json

To add a hook, just create a file with the hook name inside .husky/:

# .husky/pre-commit
npx lint-staged

Every developer gets this hook the moment they run npm install. No manual setup. No documentation to follow.

lint-staged: Only Process What Changed

Running your full linter on every commit is what makes hooks feel slow and painful. lint-staged fixes this by running tools only on the files that are staged for this commit — nothing more.

If you changed two files, only those two files are linted. The other 3,998 files are untouched. Most commits go from 30 seconds to under a second.

Install it:

npm install --save-dev lint-staged

Configure in package.json:

{
  "lint-staged": {
    "*.{js,jsx,ts,tsx}": [
      "eslint --fix",
      "prettier --write"
    ],
    "*.{css,scss,json,md}": [
      "prettier --write"
    ]
  }
}

Run ESLint before Prettier — ESLint may restructure code, and Prettier should have the last word on formatting. The --fix and --write flags auto-correct issues where possible, so the developer often doesn't need to do anything manually.

Point your pre-commit hook to it:

# .husky/pre-commit
npx lint-staged

Now every commit automatically lints and formats only the files being committed. Clean output, no waiting, no excuses.

commitlint: Make Every Commit Message Mean Something

A git log full of "fix," "update," "wip," and "asdfghjkl" is useless. When you need to understand why a change was made six months later, or when you want to auto-generate a changelog, or when you're trying to trace a regression — a well-structured commit history is invaluable.

Conventional Commits is the most widely adopted standard. The format is:

type(optional scope): short description

feat: add OAuth login with Google
fix(auth): handle expired JWT tokens gracefully
docs: update API reference for /users endpoint
chore: upgrade ESLint to v9

commitlint enforces this automatically via the commit-msg hook.

Install it:

npm install --save-dev @commitlint/cli @commitlint/config-conventional

Create the config:

// commitlint.config.js
module.exports = {
  extends: ['@commitlint/config-conventional'],
};

Add the hook:

# .husky/commit-msg
npx --no -- commitlint --edit $1

Git passes the path to the commit message file as $1. commitlint reads it, validates it, and either approves or rejects the commit with a clear explanation:

# ✖ Rejected
git commit -m "fixed the thing"
# ✖   subject may not be empty [subject-empty]
# ✖   type may not be empty [type-empty]

# ✔ Accepted
git commit -m "fix(auth): resolve token expiry not being handled"
# ✔ found 0 problems, 0 warnings

The common commit types are: feat (new feature), fix (bug fix), docs (documentation), refactor (code restructure without behavior change), test (adding or updating tests), chore (maintenance, build changes), and perf (performance improvement).

Once your history is consistently structured, tools like semantic-release can read it to automatically determine version bumps and generate release notes — something that previously required manual effort.

The pre-push Hook: The Last Safety Net

pre-commit is fast and runs on every commit. Your full test suite shouldn't run there — it would make every commit feel heavy. But tests absolutely need to run before code reaches the remote branch, where they become someone else's problem.

The pre-push hook is the right place for this:

# .husky/pre-push
npm test

If tests fail, the push is blocked. The remote branch stays clean. No broken builds in CI, no colleagues pulling a broken main branch.

For TypeScript projects, add a type check too:

# .husky/pre-push
npx tsc --noEmit && npm test

--noEmit runs the TypeScript compiler for type checking only without emitting any files. It catches type errors that might slip past your IDE, and it's much faster than a full build.

The post-merge Hook: Stop Hitting "Cannot Find Module"

Here's a frustration every developer knows: you pull from main, run your app, and immediately hit an error about a missing module. A teammate added a package and you forgot to run npm install after pulling.

The post-merge hook runs after every successful merge and pull. You can use it to check whether package.json changed, and auto-install if it did:

# .husky/post-merge
#!/bin/sh

changed_files="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD)"

if echo "$changed_files" | grep -q "package.json"; then
  echo "📦 package.json changed — running npm install..."
  npm install
fi

This small script has saved countless developers from wasted debugging time. It's the kind of automation that makes your repo feel professional.

When to Bypass Hooks

Sometimes you legitimately need to skip the hooks — saving a work-in-progress state, making an emergency fix, committing generated code. Git makes this easy:

git commit --no-verify -m "wip: checkpoint before major refactor"
git push --no-verify

--no-verify tells Git to skip all hooks for that operation. This is a valid escape hatch — just not a habit. If you notice teammates using it regularly, that's a signal that a hook is too slow or too noisy. Investigate and fix the hook, not the behavior.

Complete Setup Reference

Here's everything you need for a new Node.js project in one place:

# Install all tools
npm install --save-dev husky lint-staged @commitlint/cli @commitlint/config-conventional

# Initialize Husky
npx husky init

# Create hooks
echo "npx lint-staged" > .husky/pre-commit
echo "npx --no -- commitlint --edit \$1" > .husky/commit-msg
echo "npm test" > .husky/pre-push

// package.json — add this block
{
  "lint-staged": {
    "*.{js,jsx,ts,tsx}": ["eslint --fix", "prettier --write"],
    "*.{css,scss,json,md}": ["prettier --write"]
  }
}

// commitlint.config.js
module.exports = { extends: ['@commitlint/config-conventional'] };

That's a complete, automatically shared code quality pipeline. Every developer on your team runs the same checks, every time, from the moment they clone the repo.

Summary

Git hooks shift quality enforcement from "someone reviews for it" to "the system enforces it automatically." Husky makes hooks part of the repository so the whole team uses them without thinking. lint-staged keeps things fast by only touching changed files. commitlint makes your git history readable and automation-friendly.

The best part is that once this is set up, it runs invisibly. Nobody has to remember to lint before committing, nobody argues about formatting in review, and nobody merges code that breaks tests. The developer experience gets better for everyone, and the codebase stays cleaner with no extra effort.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

The Developer's Guide to Environment Variables and Secrets Management

ZyVOP — Mon, 08 Jun 2026 15:21:42 +0000

Why This Deserves More Attention Than It Gets

Credential leaks are one of the most common and preventable security incidents in software. Bots actively scan GitHub for newly pushed API keys, database URLs, and private credentials — and they find them within minutes of a commit going public. Rotating compromised credentials is painful, and in some cases the damage is done before you even realize what happened.

This isn't just an enterprise problem. It happens on solo side projects, open-source repos, and internal tools at startups. And the root cause is almost always the same: someone treated secrets like regular configuration and didn't have a clear strategy for keeping them out of version control, logs, and error messages.

The patterns in this guide aren't bureaucratic overhead. They're the minimum viable approach for any app that talks to a real database or a real API.

What Environment Variables Actually Are

An environment variable is a key-value pair that lives in a process's environment — a set of values the operating system makes available to any running program. Every process inherits the environment of the process that spawned it.

In Node.js, you access them through process.env:

const port = process.env.PORT;
const dbUrl = process.env.DATABASE_URL;

In Python:

import os
port = os.environ.get('PORT')
db_url = os.environ.get('DATABASE_URL')

The core idea — and the reason env vars are the standard approach — is that they decouple what the app does from where it runs. The same application code can point at a local development database or a production cluster. The code doesn't change; only the environment does. This is the heart of the 12-Factor App principle: store config in the environment, not in the code.

The .env File: What It Is and What It Isn't

In local development, you don't set environment variables by hand before every terminal session. Instead, you use a .env file — a plain text file at the root of your project with one key=value pair per line:

DATABASE_URL=postgresql://localhost:5432/myapp_dev
STRIPE_SECRET_KEY=sk_test_abc123xyz
JWT_SECRET=a-long-random-secret-string
PORT=3000
NODE_ENV=development

Libraries like dotenv (Node.js), python-dotenv (Python), and godotenv (Go) read this file at startup and load those values into the process environment. In Node.js:

// Entry point — must be first, before anything imports process.env
import 'dotenv/config';

This is the right tool for local development. Here's the distinction that matters most though: a .env file is a local developer convenience. It is not a secrets distribution system, and it should never be committed to version control. It belongs only on the developer's local machine — never on a server, never in a repo, never in a Docker image.

The .gitignore Rule — No Exceptions

Before writing a single value in your .env file, add it to .gitignore:

# .gitignore
.env
.env.local
.env.*.local
.env.development.local
.env.test.local
.env.production.local

If you have already committed a .env file — even once, even months ago — that secret exists in your Git history permanently. Deleting the file from the current tree doesn't remove it from past commits. Your only options are to rotate every exposed credential and use git filter-repo or BFG Repo Cleaner to scrub the history. It's a painful process. The .gitignore entry costs five seconds and prevents all of it.

The .env.example File: Document Without Exposing

Once .env is gitignored, new developers cloning the repo have no idea what environment variables the app needs. The solution is a .env.example file — committed to the repo, containing all the required keys but no real values:

# .env.example
# Copy this file to .env and fill in your local values.

# ── Database ──────────────────────────────────────────
DATABASE_URL=                    # e.g. postgresql://localhost:5432/myapp_dev
DATABASE_POOL_SIZE=10            # Optional. Defaults to 10.

# ── Authentication ────────────────────────────────────
JWT_SECRET=                      # Min 32 chars. Generate: openssl rand -hex 32
JWT_EXPIRES_IN=7d                # Token lifetime. Format: 1d, 7d, 1h

# ── Stripe ────────────────────────────────────────────
STRIPE_SECRET_KEY=               # Stripe Dashboard → Developers → API keys
STRIPE_WEBHOOK_SECRET=           # Stripe Dashboard → Webhooks → Signing secret

# ── Redis ─────────────────────────────────────────────
REDIS_URL=redis://localhost:6379 # Default local Redis

# ── App ───────────────────────────────────────────────
PORT=3000
NODE_ENV=development

Good comments here are worth more than you'd think. Telling a developer where to get a value (not just that it's needed) eliminates a whole class of "how do I set this up?" questions and dependency on tribal knowledge.

The onboarding flow becomes clean and self-contained:

git clone https://github.com/your-org/your-app
cp .env.example .env
# Fill in your values
npm install && npm run dev

No Slack messages. No waiting for someone to send credentials. No undocumented required services discovered at runtime.

Separating Config by Environment

Most apps run in at least three environments: local development, CI testing, and production. Each has different config values, different credentials, and sometimes different behavior. A layered .env file strategy handles this cleanly:

.env                  # Shared non-secret defaults, committed
.env.local            # Local machine overrides, gitignored
.env.test             # CI test runner config, possibly committed (no secrets)
.env.staging          # Staging environment — NOT committed
.env.production       # Production environment — NOT committed

In Node.js, you load the right file based on NODE_ENV:

import dotenv from 'dotenv';
import path from 'path';

dotenv.config({
  path: path.resolve(process.cwd(), `.env.${process.env.NODE_ENV ?? 'development'}`),
});

Frameworks handle this automatically — Next.js, Vite, and Create React App all have their own .env loading conventions. The principle is the same regardless: one environment, one configuration source, no bleed between them.

One rule worth stating directly: production credentials should not exist on developer laptops. If a developer can connect to the production database from their local machine because they have the URL in their .env, that's a real risk. Each environment should have its own credentials with access scoped appropriately.

Validate Configuration at Startup — Don't Discover Problems at Runtime

One of the most common causes of production incidents is an application that starts successfully but then fails at runtime because an environment variable is missing or has the wrong format. The error surfaces in a user-facing request with a cryptic message. The on-call engineer spends 20 minutes tracing it to a misconfigured env var that the app should have caught at boot.

The fix is to validate your entire configuration when the application starts and refuse to start if anything is wrong.

Using zod in TypeScript:

import { z } from 'zod';

const envSchema = z.object({
  DATABASE_URL: z.string().url('DATABASE_URL must be a valid URL'),
  DATABASE_POOL_SIZE: z.coerce.number().int().min(1).max(100).default(10),
  JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 characters'),
  JWT_EXPIRES_IN: z.string().default('7d'),
  PORT: z.coerce.number().int().min(1).max(65535).default(3000),
  NODE_ENV: z.enum(['development', 'test', 'staging', 'production']),
});

const result = envSchema.safeParse(process.env);

if (!result.success) {
  console.error('❌ Invalid environment configuration:');
  console.error(result.error.format());
  process.exit(1);
}

export const env = result.data;

Now import env from this module everywhere instead of accessing process.env directly. You get full TypeScript types on your config, startup validation, and a single place where all required variables are declared. If anything is missing, the app exits immediately with a clear error:

❌ Invalid environment configuration:
{
  JWT_SECRET: { _errors: ['JWT_SECRET must be at least 32 characters'] },
  DATABASE_URL: { _errors: ['DATABASE_URL must be a valid URL'] }
}

That's infinitely better than a TypeError: Cannot read properties of undefined surfacing mid-request in production.

Secrets in CI/CD Pipelines

Your CI/CD pipeline needs secrets too — deploy keys, cloud provider credentials, API keys for integration tests. These should never appear in pipeline config files.

Every major CI platform has a built-in secret store:

GitHub Actions:

# .github/workflows/deploy.yml
jobs:
  deploy:
    steps:
      - name: Run deploy script
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: ./deploy.sh

Secrets are configured in GitHub → Settings → Secrets and Variables → Actions. They're encrypted at rest, masked in logs, and only available during workflow runs.

GitLab CI:

deploy:
  script: ./deploy.sh
  variables:
    DATABASE_URL: $DATABASE_URL  # Set in GitLab → Settings → CI/CD → Variables

The pattern is identical everywhere: secrets live in the platform's encrypted store, referenced by name in your config, injected as environment variables at runtime. Your pipeline file can be committed publicly with no risk.

Secrets Managers for Production

For production, .env files sitting on a server disk are a problem. They're plain text accessible to anyone with shell access, hard to rotate across multiple instances, and they create configuration drift between servers. The right solution is a dedicated secrets manager.

AWS Secrets Manager is the natural fit for AWS-hosted applications. Secrets are stored and encrypted in AWS, fetched by your app at startup using the AWS SDK, and access is controlled through IAM roles — no static credentials needed:

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

const client = new SecretsManagerClient({ region: 'us-east-1' });

async function loadSecrets() {
  const res = await client.send(new GetSecretValueCommand({ SecretId: 'myapp/production' }));
  const secrets = JSON.parse(res.SecretString);
  // Inject into process.env or use directly
  process.env.DATABASE_URL = secrets.DATABASE_URL;
}

Doppler is popular for teams who want something simpler and platform-agnostic. You manage secrets in Doppler's dashboard and run your app through their CLI:

doppler run -- node server.js

Doppler fetches the right secrets for the environment and injects them. Your app code changes nothing — it still reads from process.env. Doppler handles rotation, audit logging, and access control centrally.

HashiCorp Vault is the most powerful option for teams that want full control — dynamic credentials, detailed audit logs, fine-grained access policies, and support for dozens of secret backends. It's more complex to operate but gives you capabilities the hosted services don't.

The common thread: in production, secrets should be fetched at runtime from an encrypted, auditable store — not read from a file sitting on disk.

The Things You Should Never Do

A few hard rules that are worth being explicit about:

Never commit a .env file. If it happened, rotate every credential in it and treat them as compromised. Git history is permanent.

Never hardcode secrets in source code. There's no such thing as "just temporarily." It lives in the history forever, and eventually someone will search the codebase and find it.

Never log process.env. A single console.log(process.env) at startup sends every secret to your log aggregator, monitoring dashboard, and anywhere else logs flow:

// 🚫 Dumps every secret everywhere logs go
console.log('App starting with config:', process.env);

// ✅ Log only what's safe and useful
console.log(`Starting in ${env.NODE_ENV} mode on port ${env.PORT}`);

Never pass secrets as CLI arguments. Arguments show up in ps aux output and in process manager logs. They're not private.

Never bake secrets into Docker images. A COPY .env . in a Dockerfile embeds the secrets permanently in that image layer. Even if you delete the file later in the build, the earlier layer still contains it:

# 🚫 Secrets are now baked into the image layer — forever
COPY .env .

# ✅ Inject at runtime, never at build time
# docker run --env-file .env.production myapp
# docker run -e DATABASE_URL=$DATABASE_URL myapp

Quick Audit Checklist

Run through this whenever you start a new project or review an existing one:

.env and all local env files are in .gitignore
.env.example is committed with all keys, no real values, with comments
npm install fully sets up the environment — zero extra steps for new devs
Config is validated at startup with a schema; app refuses to start on missing vars
process.env is only accessed through the validated config module
No console.log(process.env) anywhere in the codebase
CI/CD secrets live in the platform's secret store, not in config files
Production uses a secrets manager, not flat files on disk
Each environment has its own credentials — no production credentials on laptops
Docker images don't copy any .env files

Summary

Environment variable management is one of those things you either get right from the start or fix after something goes wrong. The cost of a credential leak ranges from embarrassing to catastrophic. The cost of the setup in this guide is a few hours, once. .env.example keeps onboarding smooth. Startup validation catches misconfigurations before they hit users. Secrets managers in production keep your credentials off of disks and out of the hands of anyone who shouldn't have them.

None of this is complicated — it's mostly just being deliberate about where secrets live and how they flow through your system. Build that habit early and it becomes invisible.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

The Best Open Source LLMs for Coding Right Now (June 2026)

ZyVOP — Mon, 08 Jun 2026 11:58:58 +0000

The leaderboard moved — again. Between April and June 2026, at least five major open-weight coding models shipped, two of them from labs most Western developers haven't heard of. If you read a "best open-source LLMs" guide from three months ago, it's already wrong.

This post is current as of June 8, 2026. Every benchmark number below has a source. Where benchmarks are self-reported by labs (which most are — we'll get into that), we say so.

First: Stop Trusting HumanEval Scores

Everyone above 85% on HumanEval can be ignored for ranking purposes. That includes Qwen, DeepSeek, Codestral, Llama — all of them cross that threshold now. The benchmark is saturated, and there's strong evidence of training data contamination across the board.

The numbers that actually discriminate in 2026:

SWE-bench Verified / SWE-bench Pro — Given a real GitHub repository and a bug report, does the model write a patch that makes the tests pass? SWE-bench Verified (500 tasks) is becoming contaminated itself; SWE-bench Pro (1,865 multi-language tasks) is the harder, cleaner signal right now. OpenAI officially stopped reporting SWE-bench Verified scores in early 2026, citing contamination concerns.

Terminal-Bench 2.1 — Long-horizon CLI tasks: scripting, DevOps automation, multi-step workflows in a terminal. Harder to game than single-function benchmarks.

LiveCodeBench — Competitive programming problems pulled from Codeforces, LeetCode, AtCoder continuously. Contamination-resistant because the dataset updates monthly.

FIM pass@1 — For autocomplete specifically. How often does the model correctly fill in code between a prefix and suffix? This is what your Tab key actually calls.

Agentic Coding (LiveBench) — Multi-step task completion in a coding context. The most useful proxy for "can this model actually work autonomously on a repo?"

With that baseline established — here's the actual landscape.

The June 2026 Leaderboard at a Glance

Model	SWE-bench Pro	Terminal-Bench 2.1	License	Best For
MiniMax M3	59.0%	66.0%	Open-weight (weights pending)	Frontier coding + 1M context
GLM-5.1	58.4%	63.5–66.5%	MIT	Long-horizon agentic engineering
Kimi K2.6	58.6%	—	Modified MIT	Agent swarms, autonomous runs
DeepSeek V4 Pro	~56%	—	MIT	API cost play, 1M context
Qwen3-Coder 480B	~55% (SWE-V: 67–70%)	—	Apache 2.0	Agentic coding via API
Qwen3.6-35B-A3B	73.4% SWE-V	51.5% Terminal	Apache 2.0	Best local model, single GPU
Devstral Small 24B	—	—	Apache 2.0	Local agentic workflows
Codestral 22B	—	FIM: 95.3%	Mistral	IDE autocomplete

Self-reported benchmarks from labs are noted where applicable. SWE-bench Pro scores use standardized scaffolding. Numbers current as of June 2026.

1. MiniMax M3 — The New Entrant You Can't Ignore (Released June 1, 2026)

TL;DR: One week old as of this writing. Claims the top open-weight SWE-bench Pro score. API is live. Weights not yet public. Treat with calibrated skepticism, but the architecture is legitimately interesting.

MiniMax dropped M3 on June 1, 2026 — seven days ago. The API is live through MiniMax's platform and OpenRouter. Open weights and the technical report were promised within ten days of launch, which means they should appear on Hugging Face around June 11.

The numbers MiniMax is claiming: 59.0% on SWE-bench Pro (which they say beats GPT-5.5 and Gemini 3.1 Pro), 66.0% on Terminal-Bench 2.1, 70.06% on OSWorld-Verified for computer use. The model supports a 1M-token context window and is natively multimodal — text, image, and video input from a single architecture.

The architecture story is actually compelling. MiniMax built MSA (MiniMax Sparse Attention), which they say delivers more than 9× faster prefill and more than 15× faster decoding at 1M-token context compared to M2, at 1/20th the per-token compute. Standard full attention has quadratic cost as sequence length grows — MSA partitions the KV cache into blocks and uses a "KV outer gather Q" approach where each KV block is read exactly once, with contiguous memory access. If those speedup numbers hold under independent testing, this matters enormously for long-context coding tasks.

What you need to know right now:

Every benchmark number from M3 is vendor-reported. MiniMax ran their own evaluations using Claude Code as the scaffolding. Independent confirmation hasn't landed yet. TechTimes noted that MiniMax's comparison baseline uses Claude Opus 4.7 rather than the more recently released Opus 4.8 — which places M3 further from the frontier than the launch announcement implies.

Additionally: MiniMax is a Shanghai-based lab operating under Chinese law. If data sovereignty matters to your organization, that's a real consideration before routing production coding traffic through their API.

Promo API pricing at launch: ~$0.30/M input, $1.20/M output (50% promotional discount on the standard $0.60/$2.40 rate).

Bottom line: If you're curious and cost-tolerant about experimental models, M3 via API is worth testing right now. Don't make architectural decisions based on it until the weights ship and independent evals come in.

2. GLM-5.1 — The 8-Hour Model (Released April 2026)

TL;DR: 754B MoE, MIT license, 58.4 on SWE-bench Pro, 200K context. Built specifically for tasks that take hours, not minutes. The single most interesting architectural claim this quarter.

Z.AI (formerly Zhipu AI) released GLM-5.1 in April 2026, and it's been quietly sitting at or near the top of the open-weight coding leaderboard ever since. It achieves 58.4 on SWE-bench Pro, outperforming GPT-5.4 and Claude Opus 4.6 on that specific benchmark (self-reported, but corroborated by LiveBench data).

The capability Z.AI is specifically calling out — and which appears to be real, not just marketing — is long-horizon autonomous execution. The model is designed to work continuously on a single complex task for up to 8 hours. Most LLMs are implicitly optimized for single-turn interactions: give it a clear question, get a clean answer. GLM-5.1 is designed for something harder: multi-stage workflows where the model has to plan, execute dozens of dependent steps, encounter failures, course-correct, and deliver production-grade results.

Z.AI documented the model building a complete Linux desktop system from scratch within 8 hours as a demonstration case. Whether that's reproducible in your specific engineering context is a different question, but the underlying capability — sustained execution without degrading into repetitive loops — is architecturally distinct from what other models do.

The benchmark picture:

SWE-bench Pro: 58.4 (SOTA among open-weight at time of release)
AIME 2026: 95.3
GPQA-Diamond: 86.2
Terminal-Bench 2.0: 63.5 (66.5 with Claude Code scaffolding)
MCP-Atlas (Public Set): 71.8 — directly relevant as MCP becomes standard in production agent systems
CyberGym: 68.7 (up from GLM-5's 48.3 — significant jump)

The 754B MoE architecture is MIT licensed on Hugging Face. For API access, it's available through Z.AI's platform, SiliconFlow, and OpenRouter. Local deployment is supported via SGLang (v0.5.10+), vLLM (v0.19.0+), and KTransformers.

One thing worth noting: The model aligns closely with Claude Opus 4.6 on general intelligence benchmarks. It's not replacing frontier proprietary models — it's matching the previous generation of them for free, which is the actual win.

Bottom line: If you're building agents that need to run long, complex jobs autonomously — not just answer a question and wait for the next prompt — GLM-5.1 is the most purpose-built open-weight model for that use case. The 8-hour claim is unusual enough to investigate.

3. Kimi K2.6 — Best Overall Local Coding Model (If You Have the Hardware)

TL;DR: 1T total / 32B active MoE. 58.6 on SWE-bench Pro. preserve_thinking mode maintains reasoning state across turns. Best-in-class for agent workflows running locally.

Moonshot AI's Kimi K2.6 sits at the top of the May 2026 LiveBench snapshot across both key coding metrics: 78.57 Coding Average and 58.33 Agentic Coding Average. On SWE-bench Pro, it hits 58.6. It's the strongest open-weight model you can actually run locally — with the hardware caveat we'll get to.

The architectural detail that matters for real usage: K2.6 introduces preserve_thinking mode, which maintains full reasoning traces across conversation turns. For complex debugging sessions that span multiple messages, most models effectively forget what they reasoned through three turns ago. K2.6 with preserve_thinking maintains consistent reasoning state — which means less re-explaining context, more coherent multi-step diagnosis. This is a genuine quality-of-life improvement for anyone using the model in an interactive engineering session.

K2.6 also introduces agent swarm orchestration — the ability to coordinate multiple sub-agents in parallel. If you're building a coding pipeline where different agents handle different parts of a codebase simultaneously, this is the model with native architecture support for it.

ollama pull kimi-k2.6

Hardware reality (don't skip this):

K2.6 is a 1-trillion total parameter MoE model. For consumer hardware, you need quantization, and even then you need serious memory. On an M4 Ultra Mac Studio with 128GB unified memory, it runs. On dual RTX 4090s (48GB VRAM), quantized versions work. On a single 24GB card — it doesn't. Don't fight it.

If you have the hardware, K2.6 is the current best local coding model. If you don't, read the next section.

Bottom line: Modified MIT license. HuggingFace weights are available. For teams with multi-GPU setups or M-series Macs with large unified memory, this is the model to run locally. For everyone else, consider it as an API target rather than a local model.

4. Qwen3.6-35B-A3B — The Single-GPU King (Released April 16, 2026)

TL;DR: Apache 2.0. 262K context. 73.4% SWE-bench Verified. 3B active parameters out of 35B total. Runs on a 24GB card or M-series Mac with 32GB. This is the model for most developers.

While the headlines went to the trillion-parameter monsters, Alibaba quietly released Qwen3.6-35B-A3B on April 16, 2026, and it's arguably the most practically useful open coding model for the median developer.

The MoE architecture activates only 3B parameters per token out of 35B total, which means inference cost is close to running a 3B model while drawing on the full 35B parameter space. The result: it fits on consumer hardware, runs at usable speeds, and punches far above its size class on coding benchmarks.

73.4% on SWE-bench Verified — from a model that fits on a single RTX 4090 or M-series Mac with 32GB. That number is competitive with what large cloud models were scoring 12 months ago.

Additional benchmark context:

Terminal-Bench 2.0: 51.5% — strong for its hardware tier
262K native context, extendable to ~1M via Yarn
Thinking preservation (new in this release): retains reasoning context from historical messages, reducing overhead in iterative development sessions
Designed specifically for agentic coding — repository-level reasoning, tool calling, multi-step workflows

# Ollama - Mac and Linux
ollama run qwen3.6:35b-a3b

# Unsloth GGUF - for llama.cpp style inference
# Available on HuggingFace under Qwen/Qwen3.6-35B-A3B-GGUF

The Apache 2.0 license is the cleanest available — no commercial restrictions, no weird carveouts. You can fine-tune it, deploy it in a product, and modify the weights without asking Alibaba for permission.

Who this is for: This is the default recommendation for individual developers and small teams who want a serious local coding model without needing enterprise GPU infrastructure. The hardware bar is a single 24GB card or an M-series Mac. The quality bar is frontier-adjacent.

Bottom line: If you have a single 4090 or a Mac with 32GB+, start here. The tradeoff between performance and hardware requirement is better than any other model on this list.

5. Qwen3-Coder 480B-A35B — The Agentic Powerhouse (Via API)

TL;DR: 480B total / 35B active MoE. Comparable to Claude Sonnet 4 on agentic coding benchmarks. 256K native context, 1M with extrapolation. Free on OpenRouter right now. Run via API, not locally.

Alibaba's Qwen3-Coder, released July 2025, has become the default open-weight model for teams running heavy agentic coding workflows via API. The 480B-A35B-Instruct variant is Alibaba's most powerful agentic coder to date and is specifically designed for the kind of work modern coding agents do: function calling, tool use, and long-context reasoning over entire repositories.

On agentic coding, agentic browser-use, and agentic tool-use benchmarks, Qwen3-Coder sets new state-of-the-art results among open models, described as comparable to Claude Sonnet 4. The 256K native context (extendable to 1M) is appropriate for real-world repositories — not just toy examples.

Alibaba also released Qwen Code alongside the model — a command-line tool for agentic coding forked from Gemini Code, adapted with custom prompts and function calling protocols specifically tuned for Qwen3-Coder.

# Configure in Cline:
# API Provider: OpenAI Compatible
# API Key: your-openrouter-key
# Base URL: https://openrouter.ai/api/v1
# Model: qwen/qwen3-coder-480b-a35b-instruct

At $0.22/M input and $1.00/M output on standard OpenRouter pricing — with a free tier currently available — this is one of the most economical options for high-volume agentic coding pipelines. You're not going to self-host 480B parameters; that requires H100-class multi-GPU infrastructure. The value proposition here is entirely API-based.

Bottom line: For teams using coding agents at scale and optimizing for cost, Qwen3-Coder 480B via API is the obvious choice. The free tier on OpenRouter is worth exploring before committing to paid plans.

6. DeepSeek V4 Flash — Self-Hosting the Frontier (Released April 24, 2026)

TL;DR: MIT license. Two variants: V4-Flash (284B, ~158GB) is self-hostable on 4× A100 or 2× H200. V4-Pro (1.6T) needs a real cluster. Flash delivers 85–95% of Pro quality. Best option if you need frontier-quality code on your own infrastructure.

DeepSeek's V4 released April 24, 2026, with two variants designed for different infrastructure profiles. V4-Pro is the flagship at 1.6T total parameters (49B active); V4-Flash is the practical self-hosting target at 284B total (with FP4+FP8 mixed precision, checkpoint weights ~158GB).

The headline for DeepSeek V4 is the KV cache efficiency: V4 uses only 7% of V3.2's KV cache footprint. That's not a rounding error — it's an architectural improvement that makes serving 1M-context inputs practically feasible without absurd memory requirements.

V4-Flash at Q4 quantization fits on 4× A100 80GB or 2× H200 141GB, plus ~256GB system RAM. The hardware math: weights are ~158GB, full 1M-token KV cache adds ~10GB, runtime overhead a few GB more — total ~170-175GB VRAM. Four A100s give you 320GB, which is headroom, not requirement.

Why self-host V4 at all?

The calculus has shifted in 2026. Regulatory frameworks increasingly demand data residency guarantees. For EU healthcare data, US government contracting, or any organization under strict data sovereignty requirements, "send your code to a third-party API" is not a compliant architecture. V4 Flash under MIT license, running on your own H200 nodes, means no compliance conversation needed.

If you're spending less than ~$15,000/month on inference, the API is almost certainly cheaper than owning the hardware. Above that threshold, self-hosting math starts to work.

# vLLM deployment - requires 4x A100 80GB minimum
vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

Bottom line: For individual developers, use the DeepSeek API — it's still one of the best cost-per-quality deals available. For organizations with data sovereignty requirements and the GPU budget, V4 Flash self-hosted is the only serious open-weight frontier model that's actually deployable.

7. Codestral 22B — Still the Only Answer for IDE Autocomplete

TL;DR: 95.3% FIM pass@1. 22B fits on consumer hardware. Co-founder of Continue.dev says it's their recommended autocomplete model. Nothing else competes on this specific task.

Nothing on this list has changed Codestral's position for IDE tab completion. Mistral built Codestral specifically for fill-in-the-middle (FIM), which is what actually happens when you hit Tab in VS Code or Neovim — the model sees code before your cursor, code after your cursor, and fills the gap.

Codestral's 95.3% FIM pass@1 (Codestral 25.01) outperforms DeepSeek Coder V2 (83.5%) and Llama 3 70B (81.7%) on the same tasks. The gap isn't small. Ty Dunn, co-founder of Continue.dev, publicly called it their recommended autocomplete model because "code completion constitutes a large portion of the work, which requires models that are great at fill-in-the-middle."

The 22B size is intentional — small enough to deliver sub-100ms completions locally, large enough to maintain quality. That latency matters: autocomplete that takes 500ms breaks developer flow. Codestral at Q4_K_M runs ~14GB and fits comfortably on an RTX 3080 Ti or better.

// Continue.dev config
{
  "tabAutocompleteModel": {
    "provider": "ollama",
    "model": "codestral:22b",
    "title": "Codestral"
  }
}

ollama pull codestral:22b

Alongside Codestral for autocomplete, Devstral Small 24B (Apache 2.0) remains the best local option for multi-file agentic workflows — multi-file edits, debugging loops, repository-level reasoning. The two are designed to work together: Codestral handles your Tab key, Devstral handles your agent panel.

Bottom line: If your primary use case is IDE autocomplete — and for most developers it is — Codestral 22B is the answer. It was true in early 2026 and it's still true today. Nothing released in the past two months has changed that.

The Hardware Decision Tree (June 2026 Edition)

Be honest about your hardware before you pick a model. Swapping to SSD isn't a model limitation — it's a sizing problem.

8GB VRAM (RTX 4060, laptop GPUs): Qwen3 8B. Competitive programming assistant and code generation, not agentic repo work.

12–16GB VRAM (RTX 3080, RTX 4070): Qwen2.5-Coder 14B or Devstral Small 24B (quantized). Codestral 22B for autocomplete runs fine here too.

24GB VRAM (RTX 3090, RTX 4090, A5000): Qwen3.6-35B-A3B — this is your sweet spot. The 3B active parameter MoE architecture makes it inference-efficient while the 35B total gives you serious quality. Codestral 22B for autocomplete alongside it.

32GB unified memory (Mac M3/M4 Pro, Mac Mini M4 with 32GB): Qwen3.6-35B-A3B at full quality. GGUF via llama.cpp or Ollama.

48–80GB VRAM (dual 4090, A100 40GB, A6000): Kimi-Dev-72B (the original SWE-bench champion at 60.4%) or start looking at Kimi K2.6 quantized.

128GB+ unified (Mac Studio M4 Ultra, Mac Pro M4 Ultra): Kimi K2.6 quantized, GLM-5.1 quantized via KTransformers.

Enterprise GPU cluster (4× A100, 2× H200, 8× H100): DeepSeek V4 Flash or GLM-5.1 full precision. MiniMax M3 once weights ship.

The Practical Stack for June 2026

Stop looking for one model to rule everything. The models that win on agentic coding are not the same ones that win on real-time autocomplete. Here's what a practical setup looks like:

IDE tab completion: Codestral 22B (Ollama + Continue.dev)

Local chat + code review: Qwen3.6-35B-A3B (single 4090 or M-series Mac)

Local agent for complex multi-file work: Devstral Small 24B (via OpenHands or Cline)

API — high-volume agentic pipelines: Qwen3-Coder 480B via OpenRouter (free tier available)

API — best quality, cost-conscious: DeepSeek V4 Pro API (~competitive pricing, MIT model)

Long-horizon autonomous execution: GLM-5.1 via Z.AI API or SiliconFlow

Experimental — early access: MiniMax M3 API (weights pending, verify benchmarks yourself)

What Benchmarks Still Don't Tell You

Three things that don't appear in any leaderboard:

Hallucinated package APIs. Some models confidently call functions with signatures that changed two years ago, or import packages that don't exist in the version your code targets. GLM-5.1 and Qwen3-Coder are relatively strong here; smaller models are worse. Test your specific stack.

Test-passing vs. code-that-looks-right. Kimi-Dev-72B (the predecessor to K2.6) was trained with RL rewards that fire only when test suites pass — not when code compiles, not when it looks plausible. Most other models are still trained on next-token prediction over code that humans wrote, much of which is subtly wrong. The training objective difference is real and shows up when models handle edge cases.

Context degradation over long sessions. GLM-5.1's "8-hour execution" claim is partly an architecture story about not losing coherence over extended sessions. Other models — even with large context windows — tend to produce lower-quality outputs when the context fills up with long conversation history. If your use case involves long autonomous runs, test that specifically, not just single-turn benchmark scores.

The Honest Bottom Line

Open source coding AI in June 2026 is genuinely frontier-competitive. The specific phrasing that was accurate in 2024 — "open source is good enough for most tasks but you'll need proprietary for the hard stuff" — is no longer accurate. GLM-5.1, MiniMax M3, and Kimi K2.6 are competing directly with the previous generation of proprietary frontier models.

That said, "competitive benchmarks" and "production-ready for your specific codebase" are different things. Run the model on your actual tasks, not just leaderboard tasks. The hallucination patterns, context handling, and latency characteristics vary enormously by use case.

The models that aren't on leaderboards — the hardware requirements, the data sovereignty implications, the licensing edge cases — are as important as the benchmark numbers. This guide tries to be honest about both.

Resources

LiveBench Leaderboard — current open-source model rankings with coding and agentic coding breakdowns
SWE-bench.com Viewer — full history of SWE-bench submissions
SWE-rebench Leaderboard — standardized re-evaluation of SWE-bench across models
Kilo.ai Open Source Model Rankings — live leaderboard, updated as models ship
Qwen3-Coder GitHub — weights, Qwen Code CLI, agentic coding tooling
GLM-5.1 on Hugging Face — MIT-licensed weights and deployment docs
Kimi-Dev GitHub — original RL-trained coding model, still relevant for SWE-bench workflows
MiniMax M3 Blog — official technical post (read the caveats section)
Continue.dev — open-source VS Code + JetBrains extension for local LLMs
OpenHands (All Hands AI) — open-source coding agent compatible with Devstral, Kimi, Qwen
Ollama — local model serving with one-line model pulls
OpenRouter — unified API routing for Qwen3-Coder, MiniMax M3, and others

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

DEV Community: ZyVOP

How to Make Real Music With Suno AI in 2026 — Even If You've Never Written a Song

What Suno Actually Does

Plans: What You Get and What You Actually Need

Your First Song: Simple Mode

Going Deeper: Custom Mode and Structure Tags

The Prompt Formula That Actually Works

Stop Making Songs. Start Building Artists.

Writing Lyrics That Suno Performs Well

Advanced Features Worth Knowing

Song Editor and 12-Stem Separation (Pro)

Extending Tracks

Covers and Style Remixes

Suno Studio (Premier)

Who This Is Actually For (And Who It Isn't)

Suno vs Udio: The Honest Comparison

The Copyright Question You Need to Understand

Getting Started This Week

Resources

The Week AI Went Public: Everything That Just Happened and Why It Actually Matters

1. Anthropic Released the Model It Was Afraid to Release

The Pricing Situation Deserves Attention

The Partner Trust Problem

2. The Largest IPO in History Just Happened — and It's Mostly an AI Story

The Revenue Recognition Wrinkle

3. Anthropic Overtook OpenAI in Business Adoption — and It's Mostly Thanks to Code

4. ChatGPT Hit a Billion Users. Think About What That Means.

5. GPT-5.6 Is Leaking From Somewhere

6. China Is Building a Robot Every 15 Minutes

The Thing Nobody Is Saying Out Loud

What Happens Next

Environment and Config Management in Node.js: The System That Scales Past One Server

The Problem With Unmanaged Config

Step 1 — Validated Config with Zod

Step 2 — The .env.example File

Step 3 — Environment-Specific Config Objects

Step 4 — Secrets That Never Touch Your Codebase

Step 5 — Config Access Patterns

Step 6 — CI Config

The Config Validation Script

The Rules That Prevent Config Incidents

How to Scrape Google Search Results with Python Without Getting Blocked (2026)

Why Google SERP Scraping Is the Hardest Job in Scraping

Understanding Google's Anti-Bot Stack

The Legal and Ethical Context

Approach 1: The DIY Scraper (curl_cffi + BeautifulSoup)

Why requests fails immediately

The working approach: curl_cffi with TLS impersonation

Extracting SERP Features

Approach 2: Google Custom Search API (Official, Free Tier)

Approach 3: SerpApi (Managed, Production-Ready)

Approach 4: Building a Rank Tracker (Practical SEO Tool)

Competitor SERP Analysis: Who's Outranking You and Why

Scaling Up: Async SERP Scraping

Choosing Your Approach: Decision Guide

Common Errors and Fixes

FAQ

Summary

Redis Caching in Node.js: The Patterns That Actually Hold Up in Production

Why Three Layers, Not One

Layer 1: In-Process LRU with lru-cache

Layer 2: Redis — Patterns That Work in Production

Pattern 1: Cache-Aside (Lazy Loading)

Pattern 2: Write-Through

Pattern 3: Cache Keys as a System

Pattern 4: Preventing Cache Stampedes

Layer 3: CDN Edge Caching for API Responses

Eviction Policies: What Happens When Redis Gets Full

Observability: The Metrics That Catch Problems Early

The Redis Scaling Progression

The Anti-Patterns Worth Naming

What Good Caching Looks Like

Python Developer Roadmap 2026 — From Zero to Job-Ready (No Fluff, Real Sequence)

Phase 1: Core Python fundamentals (Weeks 1–8)

Phase 2: Developer tools and professional workflow (Weeks 6–10, overlapping with Phase 1)

Phase 3: Specialise — pick your lane (Month 3–4)

Lane A: Backend web development (Django / FastAPI)

Lane B: Data science and machine learning

Lane C: Automation and scripting

Phase 4: Testing, deployment, and production skills (Month 5–6)

Why `requests` fails immediately

Layer 1: In-Process LRU with `lru-cache`