DEV Community: Pau Dang

OAuth2 Account Takeovers: Building a Bulletproof Social Login Architecture

Pau Dang — Wed, 06 May 2026 15:45:54 +0000

When implementing Social Login (Google, GitHub), many developers assume that the heavy lifting is handled by the provider. The truth is: the integration layer is where your system is most vulnerable.

To tackle these vulnerabilities head-on, we must rethink the integration. Here is how to build a bulletproof, Zero-Trust social login architecture.

The Pitfall 1: The Black Box Dependency

Libraries like Passport.js are incredibly popular, but they wrap the OAuth flow into a "black box." In enterprise environments, you need total auditability. We opted for a custom Axios implementation. This reduces the attack surface and allows precise domain-level error handling.

// https://github.com/paudang/nodejs-social-auth/blob/main/src/infrastructure/auth/socialAuthService.ts
export class GoogleProvider implements ISocialProvider {
  name = 'Google';
  async getProfile(code: string, redirectUri: string): Promise<ISocialProfile> {
    const params = new URLSearchParams();
    params.append('code', code);
    params.append('client_id', process.env.GOOGLE_CLIENT_ID!);
    params.append('client_secret', process.env.GOOGLE_CLIENT_SECRET!);
    params.append('redirect_uri', redirectUri);
    params.append('grant_type', 'authorization_code');

    // Deterministic Token Exchange
    const tokenResponse = await axios.post('https://oauth2.googleapis.com/token', params.toString(), {
      headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
    });

    const { access_token } = tokenResponse.data;
    const profileResponse = await axios.get('https://www.googleapis.com/oauth2/v2/userinfo', {
      headers: { Authorization: `Bearer ${access_token}` },
    });

    return {
      id: profileResponse.data.id,
      email: profileResponse.data.email,
      name: profileResponse.data.name,
    };
  }
}

The Pitfall 2: Blind Account Linking

What happens if an attacker registers on a secondary OAuth provider using your email address, and your system automatically links it? You just gave them an Account Takeover (ATO) vector.

Our system prevents this by intelligently linking social profiles and nullifying passwords for OAuth-created accounts:

// https://github.com/paudang/nodejs-social-auth/blob/main/src/usecases/auth/socialLoginUseCase.ts
let user = await this.userRepository.findByEmail(profile.email);

if (!user) {
  user = new User(
    null,
    profile.name,
    profile.email,
    null, // Disable traditional login
    this.provider.name === 'Google' ? profile.id : null,
    this.provider.name === 'GitHub' ? profile.id : null,
  );
  user = await this.userRepository.save(user);
} else {
  // Controlled linking
  let updated = false;
  if (this.provider.name === 'Google' && !user.googleId) {
    user.googleId = profile.id;
    updated = true;
  }
  // Update user...
}

Bridging the Zero-Trust Gap

Once authenticated via Google, we do not trust their session indefinitely. We immediately bridge the user into our internal JWT system, fortified by a Redis "Nuclear Revoke" mechanism.

Note: The Verify 'state' mechanism (CSRF protection) shown in the diagram represents the ideal architecture target. Cryptographic state validation is actively in development and will be codified in the next version of the generator.

You can generate this exact, secure architecture for your next project using:

npx nodejs-quickstart-structure@latest init -n "my-secure-app" -l "TypeScript" -a "Clean Architecture" -d "PostgreSQL" --db-name "demo" -c "REST APIs" --caching "Redis" --ci-provider "GitHub Actions" --auth JWT --social-auth Google GitHub --no-include-security --advanced-options

Check out the CLI tool at Nodejs Quickstart Generator and the reference implementation code at nodejs-social-auth.

If you missed the first part of this architectural series on JWT Revocation, you can read it here: The Illusion of Stateless Security: Rethinking JWT Revocation at Scale

Slashed My Automation Suite from 9 Hours to 1 Hour with This Simple Caching Trick

Pau Dang — Thu, 30 Apr 2026 01:21:32 +0000

We've all been there: you build an amazing automation suite, hit "Run", and realize it's going to take until next Tuesday to finish.

Last week, I faced a massive bottleneck in my CI/CD pipeline. Here’s how I optimized a 9-hour test suite down to just 1 hour using a "Base Cache" strategy.

The Problem: The `npm install` Nightmare

I'm building a Node.js Quickstart Generator. To ensure quality, I have to validate 720 different combinations of technologies (Clean Architecture, MVC, TypeScript, JavaScript, MySQL, MongoDB, Kafka, etc.).

For every single test run, the script was doing a fresh npm install.

The Result: 9 hours of total execution time.
The Culprit: Redundant network fetches and disk I/O for the same packages over and over again.

The Solution: Smart "Base Caching"

The mindset shift was simple: Stop treating every project as a unique snowflake.

Even with 720 combinations, they share 95% of the same core dependencies. Here is the 3-step workflow I implemented:

1. Identify the "Base"

I grouped projects by their heaviest dependencies: Language + Database. This resulted in only 8 unique Base Caches (e.g., TypeScript_PostgreSQL).

2. Bootstrap via Copy

Instead of running npm install from scratch, the script now uses this optimized logic, code example: https://github.com/paudang/nodejs-quickstart-structure/blob/feature/oauht2-google-github/scripts/lib/validation-core.js#L500

// 1. Define the Base Cache key (Language + DB)
const baseHashKey = `${config.language}_${config.database}`;
const cachePath = path.join(testDir, `node_modules_base_${baseHashKey}`);

// 2. If a base cache exists, copy it in seconds
let usedCache = false;
if (await fs.pathExists(cachePath)) {
    await fs.copy(cachePath, path.join(projectPath, 'node_modules'));
    usedCache = true;
}

// 3. Run npm install (Use --prefer-offline for ultra-fast delta updates)
const npmCmd = usedCache 
    ? 'npm install --prefer-offline --no-audit' 
    : 'npm install --no-audit';

await runCommand(npmCmd, projectPath);

// 4. Save the cache if it's the first run
if (!usedCache) {
    await fs.copy(path.join(projectPath, 'node_modules'), cachePath);
}

Because most of the packages are already in the node_modules folder, npm just performs a quick delta update.

The Results

The optimization was a game-changer:

Execution Time: 9 Hours ➡️ 1 Hour.
Storage Efficiency: 80GB (if full-cached) ➡️ 1.2GB.
Dev Experience: I can now run the full validation suite multiple times a day instead of once a week.

Key Takeaways for Developers

Find the Real Bottleneck: Don't micro-optimize your code if your Network/IO is the one killing your productivity.
Think in "Deltas": If you can't cache everything, cache the 90% and compute the rest.
Automate the Automation: Always monitor your scripts. If it's slow, it’s a bug.

How do you handle heavy node_modules in your CI/CD? I'd love to hear your tricks in the comments! 👇

If you found this helpful, feel free to give it a ❤️ and follow for more DevOps and Performance tips!

When Logout is not enough: Defending against Token Theft with Big Tech-grade Rotation.

Pau Dang — Mon, 20 Apr 2026 05:47:58 +0000

Hi, I’m Pau Dang.

Imagine a silent intruder. They steal a single Refresh Token and maintain persistence in your system for months. Your user changes their password, logs out, and feels safe—but the intruder remains. This is the reality of systems that lack Stateless Invalidation.

I’m tired of boilerplates that ignore this Attack Vector. In this post, I want to discuss the architectural blueprint for Enterprise-grade JWT security.

1. The Revocation Gap: The Silent Crisis

A signed JWT is a rogue agent if you cannot kill it. Pure statelessness is a high-risk gamble. Most developers treat JWT as a "set and forget" solution, but if you cannot revoke a session instantly, your security is an illusion.

This is the Revocation Gap—the space between a user's intent to logout and the token's hardcoded expiration.

2. The Solution: Redis Blacklisting & JTI

To mitigate this without introducing a database bottleneck, we use JTI (JWT ID) and Redis.

JTI: Every token must carry a unique identifier.
Revocation List: On logout, we don't "delete" the token; we add its JTI to a high-speed Redis blacklist.
TTL Precision: By calculating the exact remaining life of the token, we ensure the Redis list remains lean and performant.

3. The Climax: "Nuclear Revoke" (Reuse Detection)

Standard token rotation is not enough. We need proactive defense. In my v2.1.0 implementation, we use Refresh Token Rotation with an integrated tripwire.

If the system receives an old Refresh Token that has already been rotated => This is a definitive sign of an attack (Reuse). Instead of just rejecting the request, the system triggers a "Nuclear Revoke": every active session for that user is instantly purged. It represents the ultimate trade-off: forcing a logout for the real user to ensure the absolute eviction of the intruder.

The Implementation: Knowledge-as-Code

View the full production-ready logic (automatically generated by nodejs-quickstart-structure) on GitHub.

// AuthController.ts - Enterprise Refresh Logic
async refresh(req: Request, res: Response) {
  try {
    const { refreshToken } = req.body;
    const decoded = JwtService.verifyRefreshToken(refreshToken);

    // 1. Integrity Check (Redis Whitelist)
    const cacheKey = `refresh_tokens:${decoded.id}`;
    let activeTokens = await cacheService.get<string[]>(cacheKey) || [];

    // --- THE TRIPWIRE: REUSE DETECTION ---
    if (!activeTokens.includes(decoded.jti)) {
        logger.warn(`Security Breach: Session revoking for user ${decoded.id}`);
        await cacheService.del(cacheKey); // "NUCLEAR REVOKE" - Kill all sessions
        return res.status(401).json({ message: 'Critical: Session compromised' });
    }

    // --- ROTATION ---
    activeTokens = activeTokens.filter(t => t !== decoded.jti);
    const newTokens = JwtService.generateTokens({ id: decoded.id });

    activeTokens.push(newTokens.refreshJti);
    await cacheService.set(cacheKey, activeTokens, 7 * 24 * 60 * 60);

    res.json(newTokens);
  } catch (error) {
    next(error);
  }
}

4. Scaffolding as an Architectural Mitigation

I’ve automated this entire "Engineering Soul" into my open-source project, nodejs-quickstart-structure. It supports 5,280 combinations of architecture and databases, but the core security blueprint remains rock-solid in all of them.

Don't settle for "Hello World" security. Automate the excellence.

Visual Configurator: Nodejs Quickstart Generator
Stats: 4,000+ downloads.
Video Demo: Advanced JWT Security: Refresh Token Rotation & Nuclear Revoke Demo
Author: Pau Dang (Senior SE).
Source: The Illusion of Stateless Security: Rethinking JWT Revocation at Scale.

I Tested 5 CI/CD Providers for 2,640 Node.js Projects. Here’s What I Learned.

Pau Dang — Thu, 09 Apr 2026 08:48:29 +0000

Hi DEV community,

Stop manually configuring your .yaml files. After benchmarking 5 major CI providers across 2,640 unique project scaffolding permutations, I’ve gathered the ultimate list of "Gotchas," fixes, and rankings to help you pick the right one for your enterprise Node.js app.

📊 The Ultimate CI/CD Ranking (2026 Edition)

Based on stability, ease of networking, and resource management across 2,640 repositories.

🥇 #1. GitHub Actions (The "It Just Works" Choice)

If your code is on GitHub, this is the winner.

Developer Experience: 10/10.
Secret Management: Seamless.
Pro-Tip: Use actions/cache aggressively. It cuts E2E startup time for databases and Kafka by nearly 40%.

🥈 #2. GitLab CI (The "Total Control" Engine)

Powerful, but requires more networking knowledge than GitHub.

The "Wait-On" Trick: When running healthchecks in GitLab CI, the hostname is almost always docker. Standardize your testing URLs to http://docker:3001 to save hours of debugging.

🥉 #3. Jenkins (The "Swiss Army Knife" of Enterprises)

The hardest to set up, but the most rewarding for complex, private infra.

The "Network Wall": Jenkins running in a container often can't see the app it just "upped" via Docker Compose.
The Fix: Use host.docker.internal as your WAIT_ON_HOST. It allows the Jenkins container to bridge out to the host-mapped ports.

🥉 #4. CircleCI (The Speed Demon)

Fast builds, but stay alert on memory limits.

OOM Prevention: Don't let Jest eat your RAM. We found that adding --maxWorkers=2 to your test script prevents the dreaded SIGKILL on free tier runners.

🥉 #5. Bitbucket Pipelines (The Team Connector)

Great for Atlassian-heavy workflows.

Legacy Stability: If your builds fail with weird gRPC transport errors, add DOCKER_BUILDKIT: "0" to your environment. It fixes compatibility issues with older pipeline runners.

📊 The Comparison Matrix (Developer Experience focus)

I tracked these metrics across 2,640 builds to see which one makes life easiest for us.

Feature	GitHub Actions	GitLab CI	Jenkins	CircleCI	Bitbucket
Ease of Setup	⚡ Instant	✅ Fast	🏗️ Complex	✅ Fast	✅ Fast
"Cool" Factor	Market Actions	DinD Sidecars	Plugins	Orbs	Pipes
Debug Mode	Good Logs	Great Logs	Hard to Read	SSH Debug	Good Logs
Free Tier Limit	2,000 mins	400 mins	Unlimited	2,500 mins	50 mins
Verdict	Best for OSS	Best for DevOps	Best for Pros	Best for Speed	Best for Jira

🛠️ How we validated this at scale

We built an automation tool that generates entire project structures (Clean Architecture, Typescript, Kafka, MySQL, etc.) and injects the perfect CI configuration for any of these 5 providers.

We didn't just write these configs—we refined them through 2,640 builds. We solved the hard stuff so you don't have to:

🐳 Docker-Out-Of-Docker: Fixed the Jenkins permission and networking walls.
🚀 Wait-On Master: Standardized healthchecks across host and container bridges.
📉 OOM Prevention: Added --maxWorkers=2 to keep CircleCI from killing your build.

💡 The Verdict

Pick the provider that matches your code hosting platform first. But if you have complex, multi-service testing needs, GitHub Actions and GitLab CI are currently pulling ahead in the "Enterprise Free" space.

Which one are you using? Drop a comment below with your biggest CI/CD headache! 👇

🛠️ Try it yourself:

🚀 Live Web UI: Node.js Quickstart Generator
🎬 YouTube Guide: Watch the walkthrough
📂 GitHub: paudang/nodejs-quickstart-structure

Love automation? Check out our Node.js Quickstart generator to get these CI configs out-of-the-box.

Stop Wasting Time on Boilerplate: Real-world Kafka & PostgreSQL Demo in 8 minutes

Pau Dang — Tue, 07 Apr 2026 13:00:00 +0000

After releasing the v2.0.0 Web UI for Node.js Quickstart Generator, the most common question was: "How does it handle real-world complexity?"

So, I decided to record a full, 8-minute implementation demo building a Payment Service from scratch.

📺 Watch: UI to Production Code in 8 Minutes

https://youtu.be/PmmxJLloZ1Q

🛠️ The Tech Stack (Zero-Prompt Setup)

Instead of answering 20 CLI prompts, I used our new Web Configurator to generate this exact stack:

Language: TypeScript
Architecture: Clean Architecture (Domain, UseCase, Infra)
Database: PostgreSQL
Caching: Redis
Messaging: Kafka
Security: Snyk Verified

🏗️ Clean Architecture in Action

The video shows exactly how the folder structure reflects a production-grade system:

src/domain: Pure entities with no dependencies.
src/usecases: Where the transaction logic lives.
src/infrastructure: Concrete implementations for Postgres connections and Kafka producers.

📡 Live Kafka Flow

The "Wow" moment is at 05:00 in the video. We trigger a REST API call that producers a Kafka event, which is then picked up by a separate consumer.

Zero configuration required.
Pre-mapped events.
Ready for microservices.

🛡️ Enterprise-Grade Security

We don't skip security. I ran npm run security:check in the video to show how Snyk is integrated from the start. Clean code is nothing if it isn't secure.

💻 Try it yourself

Want to generate the exact same project as in the video? Run this:

npx nodejs-quickstart-structure@latest init -n "payment-service-nodejs" -l "TypeScript" -a "Clean Architecture" -d "PostgreSQL" --db-name "payment-db" -c "Kafka" --caching "Redis" --ci-provider "GitHub Actions" --include-security

Check out our GitHub Repo and let's end boilerplate fatigue together! 🌟

Introducing Node.js Quickstart Generator v2.0.0: Automated Clean Architecture with a Sleek New UI

Pau Dang — Mon, 06 Apr 2026 12:24:35 +0000

Hi everyone,

How many times have you set up a new Node.js microservice by copying a folder from your last project? We all do it out of necessity. But after releasing v1.1.0, the scale of "boilerplate fatigue" became clear: over 4,000 developers joined the journey in just one month.

That growth sparked a realization: we needed more than just a template. Previously known as nodejs-quickstart-structure, we have officially evolved into a full-scale Generator in v2.0.0.

I built this tool to help ourselves: Why can't our starting points be enterprise-ready by default?

🎥 The Evolution: From CLI (v1.1.0) to Web UI (v2.0.0)

How it started: A powerful CLI that 4,000 of you loved (v1.1.0).

https://www.youtube.com/watch?v=Cxbb54T0uo8
How it’s going: A full-blown Web UI with 1:1 structure parity (v2.0.0).

Introducing: Node.js Quickstart Generator 2.0.0

I built this tool to solve my own frustration: Why can't I just have a professional-grade starting point in one command?

✨ What’s in the Box?

Flexible Architecture: Toggle between MVC and Clean Architecture.
Modern Stack: Pick JavaScript or TypeScript.
Communication Protocols: Built-in support for REST, GraphQL (Apollo), and Kafka (Event-driven).
Caching Layer: Pre-configured Redis or In-memory cache.
Enterprise Standards:
- Multi-stage Dockerfiles.
- Snyk & SonarCloud security hardening.
- Global error handling (ApiError, NotFoundError, etc.).
- Jest & Supertest with 80% unit test coverage out of the box.

🌐 Next-Gen Web UI Configurator (v2.0.0)

Type no more! We have completely evolved the configuration experience in our new Web UI Configurator.

It includes a Real-time Folder Simulation. As you toggle between MVC and Clean Architecture, you see exactly what the project will look like. Once you're happy, just copy the Zero-Prompt CLI command and run it in your terminal. No more prompts!

🦾 AI-Native Foundation

We designed v2.0.0 for the future. If you use Cursor or other AI coding agents, our built-in .cursorrules will make your AI assistant 10x smarter about your specific architecture. No more hallucinations about where controllers or repos belong!

🎯 Get Started NOW

npx nodejs-quickstart-structure@latest init

That’s it. No global install required.

🗺️ Road to 10k Downloads

We just crossed 4,000+ downloads and want to reach 10k by the end of the year. If you love open-source tools that actually save you time, please:

Give us a ⭐ on GitHub.
Share this with your team.
Drop a comment below—what feature should we add next? (gRPC? NestJS-style?)

24 Hours of Chaos: Saving My Open Source Project from a Supply Chain Attack (plain-crypto-js)

Pau Dang — Wed, 01 Apr 2026 01:01:37 +0000

Hello world,

I'm a Senior SE. Today, I want to share a "battle-tested" experience that just happened to my open-source project: nodejs-quickstart-structure.

This isn't just about code; it’s a lesson in Incident Response when facing professional malware designed to hijack npm, GitHub, and sensitive developer credentials.

1. The Threat: Axios & plain-crypto-js

While developing version v2.0.0, I fell victim to a Typosquatting attack. A malicious package or a "shell" dependency injected malware into my local environment.

The Suspect: Linked to the plain-crypto-js incident (a malware variant targeting devs using Axios).
The Behavior: It didn't just break my system; it silently exfiltrated:
- Browser Cookies: Hijacking active sessions for Gmail, GitHub, and LinkedIn.
- SSH Keys: Gaining unauthorized access to push code to repositories.
- npm Tokens: Attempting to publish malicious releases under my name.

2. 0h00: Detection & Containment

Immediately after noticing suspicious logs and file modifications, I followed the "Security Textbook" or you can check at Axios npm Package Compromised: Supply Chain Attack Delivers Cross-Platform RAT:

Deleted Local Repos: Wiped the execution environment of the malware.
Revoked All Sessions: Used a clean device (mobile) to remotely sign out of Google, GitHub, Microsoft, and LinkedIn.
Untrusted Devices: Removed my current machine from the "Trusted Devices" list of all critical accounts.

3. The Battle for npm (The Support Battle)

The worst-case scenario: The attacker hijacked the session and invalidated my 2FA (my stored Recovery Codes returned Invalid).

I immediately contacted npm Support:

Ticket ID: 4223695 was created.

The Strategy: Providing proof of ownership through my GitHub account (which I still control) and the project's long-standing commit history.

4. The Decision: Eradication (Wipe & Rebuild)

As an Architect, I know that if an OS is compromised by a Rootkit/Trojan, no antivirus can guarantee a 100% clean state. The only solution: Wipe & Rebuild.

The Method: Reset PC > Remove everything > Cloud download Windows.
Why Cloud Download? To ensure a fresh installation image directly from Microsoft, avoiding any malware lurking in the local Recovery partition.

5. Lessons Learned for Developers

Dependency Vigilance: Always double-check new packages, especially those with names similar to popular libraries.
2FA is Not Enough: Attackers can bypass 2FA via Session Hijacking. Always be ready to Revoke Sessions remotely.
Offline Recovery Codes: Don't just store them on your computer. Print them or use a decoupled password manager.
Incident Response Mindset: When hacked, stay calm and follow: Containment -> Asset Protection -> Eradication -> Recovery.

Currently, I am in the process of restoring a "sterile" environment to finalize v2.0.0 for nodejs-quickstart-structure. You can check out the v2.0.0 beta details here:

Next gen Web UI - Browser Generator

The project will return with a higher security standard. I hope this story helps fellow developers protect their "digital children"!

How I Built a Node.js Generator with 1,680+ Combinations, Clean Architecture or MVC (Kafka)

Pau Dang — Thu, 26 Mar 2026 10:01:34 +0000

Hi all,

We’ve all been there: starting a new Node.js microservice from scratch. You spend 4 hours setting up Express, another 2 hours arguing over folder structure (MVC or Clean Architecture?), 3 hours configuring Prisma or Mongoose, and half a day trying to get a Kafka KRaft container to talk to your local app.

By the time you're ready to write your first line of business logic, you're already exhausted.

I decided to solve this once and for all. I built nodejs-quickstart-structure—a CLI that doesn't just give you a "hello world," but scaffolds a production-ready engine tailored to your exact needs.

The Problem: The "Boilerplate exhaustion"

Most generators give you a fixed stack. But in the real world, requirements vary:

"This needs to be a simple MVC app."
"This needs to be a Clean Architecture domain-driven service."
"We need GraphQL, not REST."
"We need Kafka for event-driven messaging."

The Solution: 1,680+ Scenarios in One CLI

The core of this tool is its flexibility. By combining different technologies, it supports over 1,680 unique project combinations:

Languages: TypeScript (Recommended) / JavaScript.
Architectures: MVC or Clean Architecture.
Databases: MySQL, PostgreSQL, MongoDB, or None.
Communications: REST APIs, GraphQL, or Kafka (Event-Driven).
Caching: Redis or Memory Cache.
Security: Hardened with Helmet, HPP, and automated Snyk/SonarCloud configs.
CI/CD: Ready-to-use workflows for GitHub Actions, GitLab CI, or Jenkins.

Why "Clean Architecture"?

For many projects, MVC starts fast but becomes a nightmare to test. I’ve implemented a robust Clean Architecture template where:

Domain: Pure business logic (Entities/Use Cases).
Infrastructure: Database, Messaging (Kafka Client), Caching.
Interfaces: Controllers, Express Routes, GraphQL Resolvers.

This separation ensures your business logic doesn't care if you're using MongoDB or PostgreSQL.

AI-Native: Coding in 2026

I realized that if the folder structure is standard, AI tools (like Cursor, ChatGPT, or Gemini) can understand the codebase 10x faster.
Every project generated includes:

.cursorrules: Pre-configured rules so Cursor knows exactly how to add new features following your chosen architecture.
prompts/: A library of "Agent Skills" to help you generate Use Cases or Repositories without breaking patterns.

Try it out

You don't even need to install it globally. Just run:

npx nodejs-quickstart-structure init

The CLI will walk you through the setup. Within seconds, you'll have a project with 80%+ unit test coverage, ESLint/Prettier configured, and a Docker Compose file that actually works.

Checking the official docs:
👉 Official Documentation

Check out the repo and give it a ⭐ if it helps your workflow:
👉 GitHub: paudang/nodejs-quickstart-structure

I'd love to hear your feedback! What's the one thing you always struggle with when starting a new Node project?

Stop Writing Flaky Tests: The Ultimate Node.js Testing Strategy (Unit + E2E)

Pau Dang — Mon, 23 Mar 2026 13:00:00 +0000

Hi DEV community,

If you are a Backend Developer working with Node.js, you have likely experienced the dreaded scenario: "It passes on my machine, but randomly fails on the CI/CD pipeline."

This phenomenon is known as Flaky Tests. It usually stems from writing End-to-End (E2E) tests that share database states across test files, or due to network and infrastructure services (Redis, Kafka) not being fully initialized when the test begins.

Today, I’m going to share the complete testing architecture and lessons I learned while building the automation framework nodejs-quickstart-structure. We will solve the core problem: How to build blazing fast Unit Tests and completely deterministic E2E Tests.

1. The Crisis in 90% of Projects

Many teams implement testing "half-heartedly":

Unit Tests: Dependencies like the Database or Redis aren't mocked, causing the tests to drag on because they wait for network I/O.
E2E Tests: Developers use their local dev database to run E2E suites. Test A creates a User, Test B checks the total number of Users. If they run in a different order, the test suite explodes! Some even use conditionals in tests: if (statusCode == 404) expect(404) else expect(201).

This is a massive Anti-pattern! A test must strictly return exactly one predictable result based on static inputs.

2. The "Big Tech" Strategy: Draw a Hard Line

To fix this, you must strictly delineate your boundaries:

Unit Tests (Fast & Isolated)

Goal: Verify Business Logic (Use cases, Services, Domain).
Rule: MOCK EVERYTHING. No real database connections, no external APIs, no touching Redis or Kafka.
Speed: Thousands of test cases should execute in less than 2 seconds.

E2E Tests (Black-box & Automated Infra)

Goal: Verify the entire request flow (Route -> Controller -> Usecase -> Repo -> DB -> Response).
Rule: Use the REAL Database, Redis, and Kafka (spinned up via isolated Docker Containers or Testcontainers).
Characteristics: Data must be TRUNCATED/Teared down before or after each test suite to guarantee a "clean room" environment. It runs slower, but absolute correctness is guaranteed.

3. The Recipe for Perfect E2E Tests

To prevent E2E tests from interfering with the developer's local development environment, follow these steps and source demo nodejs-service-redis-kafka:

Step 1: Fully isolate `jest.config.js`

Do not share your Unit test configurations with E2E. Create a dedicated jest.e2e.config.js with a higher testTimeout (e.g., 30 seconds to allow databases to boot).

module.exports = {
  ...require('./jest.config'),
  testMatch: ['<rootDir>/tests/e2e/**/*.test.ts'],
  testPathIgnorePatterns: ['/node_modules/'],
  testTimeout: 30000, 
  clearMocks: true
};

Step 2: Use Node.js Scripts to Manage Docker Lifecycle

Instead of forcing developers to manually type docker-compose up before running tests, write an automated orchestration script:

Assign a dedicated port (PORT=3001 instead of 3000) to avoid Dev Server collisions.
execSync('docker-compose up -d db redis kafka').
Use the wait-on npm package to poll the healthcheck until dependencies are fully green.
Run npm run test:e2e:run.
Clean up gracefully: execSync('docker-compose down').

// Wait for dependencies to prevent "Flaky connections"
execute(`npx wait-on http-get://127.0.0.1:${TEST_PORT}/health -t 120000`);
execute('jest --config ./jest.e2e.config.js');

Step 3: Fix Kafka's "read ECONNRESET" Locally

The most common issue when testing Kafka in a local E2E run is that the tests run on the host network while Kafka is stuck in the Docker bridged network.

The Fix: Explicitly map the PLAINTEXT_HOST listener to a dedicated port (e.g., 9093):

# docker-compose.yml
kafka:
  ports:
    - "9093:9093" # Host mapping
  environment:
    - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9093
    - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:9093,CONTROLLER://:9094

Then, in your .env.test, simply point KAFKA_BROKER=localhost:9093.

Step 4: Write "Iron-clad" Assertions

Eliminate loose assertions. Because your database is wiped clean (or relies on random seeds like Date.now()), the output status must be absolutely rigid!

it('should create a user successfully via REST', async () => {
    const uniqueEmail = `test_${Date.now()}@example.com`;
    const response = await request(SERVER_URL)
        .post('/api/users')
        .send({ name: 'Test User', email: uniqueEmail });

    // Strictly expect a 201 Created
    expect(response.statusCode).toBe(201);
});

4. Conclusion

Shifting to an isolated, automated Docker test strategy will cost you 1-2 days of initial setup infrastructure work. But in return, it brings absolute peace of mind to the team as the codebase scales.

If you find this setup process too tedious, you can simply grab the exact folder structures, Docker automation scripts, and Jest configurations that I’ve already pre-configured out of the box in my open-source CLI generator:
👉 nodejs-quickstart-structure on GitHub

Drop a star (⭐) if you find it helpful! Happy coding, and here's to never seeing Test Failed randomly in GitHub Actions again!

Master Caching Patterns: A Clean Architecture Guide with AI-Native Tooling

Pau Dang — Thu, 19 Mar 2026 15:02:06 +0000

In high-performance systems, caching is the ultimate "lifesaver" for reducing database load and optimizing response times. However, choosing the right pattern (Cache-Aside, Write-Through, etc.) depends heavily on your specific use case. In this article, I’ll walk you through a hands-on demo project that showcases 5 essential caching patterns, built with Clean Architecture and the power of an AI-Native tool: nodejs-quickstart-structure.

🚀 Why Caching?

When scaling applications, the database often becomes the bottleneck. Caching strategically places frequently accessed data in memory (like Redis) to bypass slow disk I/O. But not all caching is equal. To understand the nuances, I built a showcase service from scratch.

To skip the boilerplate and focus on the patterns, I used nodejs-quickstart-structure, a CLI tool designed for AI-ready, structured development.

📂 5 Caching Patterns You Must Know

Here is a breakdown of the 5 patterns implemented in this demo:

1. Cache-Aside (Lazy Loading)

Purpose: The application manages the cache. Data is only loaded into the cache when specifically requested.
When to use: Best for systems with Read >> Write ratios (e.g., User Profiles).
Pros: Memory efficiency; cache only contains data that is actually needed.

2. Read-Through

Purpose: Offloads data-fetching logic to the Cache Provider. The app simply calls "Get" from the cache.
When to use: To keep the business logic (Use Case) clean and decouple from the database.
Pros: Extremely clean business code.

3. Write-Through

Purpose: Data is written to both the Cache and the Database simultaneously.
When to use: When Strong Consistency is required.
Pros: Minimal data inconsistency risk.

4. Write-Around

Purpose: Data is written directly to the DB, and the corresponding cache entry is invalidated (deleted).
When to use: When the written data might not be read again immediately.
Pros: Prevents "polluting" the cache with data that won't be reused soon.

5. Write-Back (Write-Behind)

Purpose: Data is written to the Cache first; the DB update happens asynchronously in the background.
When to use: For exceptionally high write performance (e.g., Logging, Real-time Metrics).
Pros: Near-instant write response.

🛠️ Technical Implementation (Step-by-Step)

Follow this roadmap to see how I build a production-ready demo using Clean Architecture:

1. Project Initialization

Bootstrap the project using the AI-Native CLI:

npx nodejs-quickstart-structure@latest init

Selection Guide:

Project name: nodejs-service-caching-pattern
Architecture: Clean Architecture
Database: MySQL
Caching: Redis
Communication: REST APIs

2. Defining Domain Interfaces

Abstract the caching and repository logic to ensure true decoupling.

ICacheService.ts

export interface ICacheService {
  get<T>(key: string): Promise<T | null>;
  set(key: string, value: unknown, ttl?: number): Promise<void>;
  del(key: string): Promise<void>;
  getOrSet<T>(key: string, fetcher: () => Promise<T>, ttl?: number): Promise<T>;
}

IUserRepository.ts

3. Infrastructure Layer

Implementation for Redis and Sequelize (MySQL).

4. Database Seeding: Generate 1,000 dummy users for demo purposes.

V20260319__seed_1000_users.sql

5. Implementing the Use Cases

This is where the pattern logic lives.

Read-Through Example: getUserInfoReadThrough.ts

// The cache provider handles DB fetching upon miss
return await this.cacheService.getOrSet<User | null>(cacheKey, async () => {
    return await this.userRepository.findById(id);
}, 3600);

Write-Back Example: updateUserWriteBack.ts

// Update cache immediately, DB updates in background
await this.cacheService.set(cacheKey, updatedUser, 3600);
this.asyncDatabaseUpdate(updatedUser);

You can check sample code at here:

6. API & Documentation

🔍 How to Test & Verify

Once deployed, you can trigger these endpoints to verify the internal logic:

1. Test Cache-Aside / Read-Through (Read Path)

First Call (Cold Cache): curl -X GET http://localhost:3000/api/demo/cache-aside/1
- Internal: App checks Redis (Miss) -> Queries MySQL -> Updates Redis -> Returns.
Subsequent Call (Warm Cache): Repeat the command.
- Internal: App checks Redis (Hit) -> Returns immediately with zero DB overhead.

2. Test Write-Through (Sync Write)

Trigger: curl -X POST http://localhost:3000/api/demo/write-through -H "Content-Type: application/json" -d '{"name": "Performance", "email": "perf@test.com"}'
- Internal: Service writes to MySQL AND Redis simultaneously.

3. Test Write-Around (Clean Write)

Trigger: curl -X POST http://localhost:3000/api/demo/write-around -H "Content-Type: application/json" -d '{"name": "Guest", "email": "guest@test.com"}'
- Internal: Writes to MySQL, then calls DEL to invalidate the Redis key. The next read will be a Miss to load the fresh data.

4. Test Write-Back (Async Write)

Trigger: curl -X PUT http://localhost:3000/api/demo/write-back/1 -H "Content-Type: application/json" -d '{"name": "Fast Update"}'
- Internal: Updates Redis immediately (super fast response). The DB update happens after a short delay asynchronously.

💡 Final Thoughts on AI-Native Tooling

What makes nodejs-quickstart-structure special is not just the speed of initialization, but its AI-Native nature. With .cursorrules and pre-built prompts, it ensures that your project maintains Clean Architecture and consistent coding standards automatically from development to production.

Full Source Code: GitHub Repository

I hope this guide helps you choose the right caching pattern for your next project

Why Your Docker Container Works on Windows but Fails on Linux: The Case-Sensitive Naming Nightmare

Pau Dang — Thu, 19 Mar 2026 00:00:00 +0000

Hi Everyone,

Have you ever faced that frustrating moment: Your code runs perfectly on your local machine (Windows/macOS), but the moment you containerize it with Docker or deploy it to a Linux server, everything breaks with a cryptic Module not found error?

The culprit usually isn't your logic—it’s a "sweet lie" told by your operating system.

1. The Kernel-Level Reality Check

The trouble starts with how different Operating Systems handle their File Systems:

Windows (NTFS): Microsoft prioritizes user convenience. To Windows, User.service.ts and user.service.ts are semantically the same. Therefore, NTFS is designed to be Case-Insensitive.
Linux (Ext4/XFS): The philosophy of Linux (and Unix) is absolute precision. In ASCII, 'U' (65) and 'u' (117) are two completely different entities. Linux treats User.ts and user.ts as two distinct files that can coexist in the same folder.

2. The Docker "Trap" on Local Machines

Docker on Windows or macOS doesn't actually run directly on those kernels. It runs inside a Lightweight Linux VM.

When you mount (bind mount) your code from a Windows host (Case-Insensitive) into that Linux VM (Case-Sensitive):

The Mistake: You name your file UserMapper.ts but write import { UserMapper } from './usermapper'.
The Local Pass: Windows tells the engine: "Sure, I know what you mean, here is the file."
The Docker Crash: The Linux environment inside Docker yells: "I don't see any file named usermapper (lowercase) here!"

The Result: Your system crashes at runtime or fails the build process immediately.

3. Vaccinating Your Node.js Projects

Instead of relying on human memory, we should use tools to enforce consistency. In my Open Source project, nodejs-quickstart-structure, I’ve implemented a 3-layer defense system:

Layer 1: ESLint Enforcement (Stop it at the source)

We use the eslint-plugin-import plugin to flag errors directly in VS Code if your import path doesn't match the actual filename 100%.

"rules": {
  "import/no-unresolved": "error"
}

Layer 2: CI/CD Pipeline (The Ultimate Gatekeeper)

Our GitHub Actions run on Ubuntu (Linux). If you accidentally push a naming mismatch, the pipeline will fail during the test suite, preventing broken code from ever reaching Production.

Layer 3: Standardized Naming Conventions

Stick to a strict naming convention across the entire project. Whether it's kebab-case or PascalCase, consistency is your best friend.

Final Thoughts

Linux environments don't create bugs; they simply expose the inconsistencies that Windows hides from you. Synchronizing your environment from Local to Production is a hallmark of a Senior Developer.

If you’re looking for a Node.js boilerplate that is pre-configured to be "immune" to these environment issues (along with Kafka, Redis, and Clean Architecture), feel free to check out my project:

👉 GitHub: nodejs-quickstart-structure

👉 NPM: npx nodejs-quickstart-structure init

I hope this saves you a few hours of meaningless debugging! Feel free to drop a ⭐ if you find the project helpful!

15 Minutes to "Ship It": (Clean Architecture + REST API + Kafka + Docker & CI/CD) From Zero to Production with Node.js

Pau Dang — Mon, 16 Mar 2026 02:02:12 +0000

Starting a new Node.js project often involves tedious repetitive tasks: scaffolding directory structures, setting up Express, configuring database connections, managing migrations, and integrating messaging systems like Kafka. This "boilerplate phase" can eat up hours of your initial development time.

Today, I’ll show you how to go from zero to a production-ready environment in minutes. We will build a high-performance Node.js service using Clean Architecture, TypeScript, MySQL, Flyway for database migrations, Kafka for real-time event-driven messaging, Docker Compose for orchestration, and GitHub Actions for CI/CD.

Let’s dive in!

🎯 "Ready-to-Run" Source Code for You:
Instead of manually copying snippets, I’ve packaged the entire source code for this article into a production-grade template on GitHub. This project has already reached 3,000+ downloads and is being used by developers for real-world services.

🔗 Repo: paudang/nodejs-clean-rest-kafka

(Just git clone, run docker-compose up -d, and you're live!)

Step 1: Initialize Project & Install Dependencies

First, let's create the project directory:

mkdir nodejs-clean-rest-kafka
cd nodejs-clean-rest-kafka
npm init -y

Install the essential production libraries:

npm install express cors helmet hpp express-rate-limit dotenv morgan kafkajs sequelize mysql2 winston

Install development dependencies:

npm install -D typescript @types/node @types/express @types/cors @types/morgan ts-node tsconfig-paths tsc-alias jest ts-jest @types/jest

Initialize tsconfig.json with Path Aliases (@/*):

{
  "compilerOptions": {
    "target": "es2020",
    "module": "commonjs",
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"]
    }
  },
  "include": ["src/**/*"]
}

Step 2: Architecting for Scale (Clean Architecture) 🏗️

Using Clean Architecture ensures your codebase remains decoupled and highly testable. We divide the source code into distinct layers:

mkdir -p src/domain src/usecases src/interfaces src/infrastructure src/utils

src/domain: Core business entities and logic (framework-independent).
src/usecases: Application-specific business rules (The "Interactors").
src/interfaces: Adapters such as Controllers and HTTP Routes.
src/infrastructure: Technical details like Database connections, Kafka clients, and Loggers.
src/utils: Shared utility functions.

This separation allows you to swap your database or framework without touching the core business logic.

Step 3: Event-Driven Messaging with Kafka 🚀

In a microservices architecture, Kafka acts as the "heartbeat" for asynchronous communication. We’ll build a KafkaService using the Connection Promise pattern to ensure the Producer is fully connected before sending messages, preventing data loss during startup.

Inside src/infrastructure/messaging/kafkaClient.ts:

import { Kafka, Producer } from 'kafkajs';

export class KafkaService {
    private producer: Producer;
    private isConnected = false;
    private connectionPromise: Promise<void> | null = null;

    constructor() {
        const kafka = new Kafka({ clientId: 'user-service', brokers: ['localhost:9092'] });
        this.producer = kafka.producer();
    }

    // "Connection Promise" ensures we only connect once efficiently
    async connect() {
        if (this.connectionPromise) return this.connectionPromise;

        this.connectionPromise = (async () => {
            await this.producer.connect();
            this.isConnected = true;
            console.log('[Kafka] Producer connected successfully');
        })();
        return this.connectionPromise;
    }

    async sendEvent(topic: string, action: string, payload: any) {
        await this.connect(); // Always wait for readiness
        await this.producer.send({
            topic,
            messages: [{ value: JSON.stringify({ action, payload, ts: new Date() }) }],
        });
        console.log(`[Kafka] Triggered ${action} to ${topic}`);
    }
}

export const kafkaService = new KafkaService();

Step 3.5: Scaling Consumers with Clean Interfaces 🛠️

Managing dozens of event types can quickly become messy. We use Abstract Base Classes and Schema Validation to keep consumers organized:

1. BaseConsumer (The Blueprint)

At src/interfaces/messaging/baseConsumer.ts, we define a template for all consumers. It handles JSON parsing and error logging, so subclasses can focus solely on business logic:

export abstract class BaseConsumer {
  abstract topic: string;
  abstract handle(data: unknown): Promise<void>;

  async onMessage({ message }: EachMessagePayload) {
      const rawValue = message.value?.toString();
      if (!rawValue) return;
      const data = JSON.parse(rawValue);
      await this.handle(data);
  }
}

2. Schema Validation (The Contract)

Using Zod at src/interfaces/messaging/schemas/userEventSchema.ts, we define the contract between Producer and Consumer. This ensures type safety across the wire.

3. WelcomeEmailConsumer (The Implementation)

Logic that triggers an email when a USER_CREATED event is received:

export class WelcomeEmailConsumer extends BaseConsumer {
  topic = 'user-topic';

  async handle(data: unknown) {
    const result = UserEventSchema.safeParse(data);
    if (result.success && result.data.action === 'USER_CREATED') {
      console.log(`[Kafka] 📧 Sending welcome email to ${result.data.payload.email}...`);
    }
  }
}

Step 4: Database Version Control with Flyway

Manual database changes are a nightmare in production. Flyway manages your schema versioning through simple SQL files.

Example V1__Create_Users_Table.sql:

CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL UNIQUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Step 5: Clean UseCases & Controllers

1. UseCase (The Interactor)

Decoupled logic at src/usecases/createUser.ts:

export const createUserUseCase = async (name: string, email: string) => {
    const user = await userRepository.save({ name, email }); // Repository abstraction
    await kafkaService.sendEvent('user-events', 'USER_CREATED', { id: user.id, email: user.email });
    return user;
};

2. Controller (Interface Layer)

The Controller's sole task is to receive requests, call use cases, and return responses. Very clean!

import { Request, Response } from 'express';
import { createUserUseCase } from '@/usecases/createUser';

export const createUser = async (req: Request, res: Response) => {
    try {
        const { name, email } = req.body;
        const user = await createUserUseCase(name, email);
        res.status(201).json(user);
    } catch (err) {
        res.status(500).json({ error: 'Internal Server Error' });
    }
};

Step 6: Docker for Production Excellence

1. `Dockerfile` (Multi-stage Build)

We separate the build environment from the runtime to minimize image size and attack surface:

FROM node:22-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-alpine AS production
WORKDIR /app
ENV NODE_ENV=production
ENV NPM_CONFIG_UPDATE_NOTIFIER=false
COPY package*.json ./
RUN npm ci --only=production
COPY --from=builder /app/dist ./dist
EXPOSE 3000
CMD ["npm", "start"]

2. `docker-compose.yml` (The Full Stack)

One command to rule them all:

services:
  app:
    build: .
    ports: ["3000:3000"]
    depends_on: [db, kafka]
    environment:
      - KAFKA_BROKER=kafka:29092
      - DB_HOST=db
  db:
    image: mysql:8.0
    ports: ["3306:3306"]
    volumes: ["./flyway/sql:/docker-entrypoint-initdb.d"]
  kafka:
    image: confluentinc/cp-kafka:7.4.0
    depends_on: [zookeeper]
    ports: ["9092:9092"]

One Last Surprise... 🤫

Think this took hours to set up?

👉 The Truth Is: The entire project structure—Clean Architecture, Kafka Producers/Consumers, Flyway migrations, Docker configs, and CI/CD pipelines—was generated in under 60 seconds using an automation tool I built.

Time-to-market is everything. Stop reinventing the wheel and start shipping business value.

Want to generate a "perfect" Node.js repo like this yourself? Try my CLI tool:

npx nodejs-quickstart-structure init

Read the full document here: Nodejs Quickstart Structure

Building production-ready software shouldn't be a chore. I hope this helps you ship your next big idea faster than ever. If you found this useful, don't forget to give a Star ⭐ on GitHub! 🔥