DEV Community: Programming with Shahan

How to Build an AI-Ready Web Data Pipeline Using Bright Data and Node.js

Programming with Shahan — Mon, 21 Sep 2026 11:19:22 +0000

A scraper can succeed and still give you bad data.

A website can change its HTML, a selector can stop matching, or a scraper can return a partial record without throwing an obvious error. Your scraper may still report success while your application quietly stores incorrect data.

That’s the problem this guide solves.

🏗️ What we’re building

A Texas real-estate tracker that collects live HAR.com listings, filters them by city and bedrooms, and detects price changes over time.
A reusable pipeline that separates web extraction from our application logic.
A self-healing workflow that lets us repair the scraper without rewriting the Node.js pipeline when the target site’s structure changes.

Concretely, we’ll build a Node.js pipeline that uses Bright Data Scraper Studio for web extraction while our application handles normalization, validation, deduplication, snapshots, and change detection.

🔎 Build your first prompt-based scraper in minutes using Scraper Studio AI Agent for Free.

Then we’ll point our pipeline at a real Texas real-estate domain, track listings across multiple runs, deliberately break the scraper, and repair it without changing the pipeline itself.

🔗 The complete source code is available in my GitHub repository. You can also download The Web Data Pipeline handbook for free:

1. The Data Extraction Pipeline Structure

Our data extraction pipeline divides responsibilities into two layers:

The Infrastructure Layer. It handles proxies, browser fingerprints, CAPTCHAs, and heavy JavaScript rendering to turn unpredictable HTML into stateless structured data.
The Pipeline Layer (Node.js App). It owns the application logic, validation, data state, and historical comparisons.

For this guide specifically, the final outline looks like this:

Therefore, by following this Separation of Concerns (SoC) principle, the same pipeline can be reused for e-commerce, crypto rates, travel listings, job boards, and competitor/financial data monitoring; the list goes on.

2. The Two Non-negotiable Data Pipeline Rules

In a scalable data pipeline, neither layer should depend on the other’s internal implementation details.

We achieve that with two rules.

RULE 1: The stable data contract is the boundary**

Our Node.js application should NEVER depend on the target website's HTML or DOM structure.

Instead, the extraction layer produces data that conforms to a stable contract. The extraction adapter knows about the external source; our application knows about the contract.

For example, our application can work with:

{
  "address": "string",
  "price": 305000,
  "bedrooms": 4
}

It doesn't need to know whether those values came from a CSS selector, an AI-generated scraper, or another extraction system.

This is the practical idea behind the Dependency Inversion Principle (DIP) and the Ports and Adapters (Hexagonal) approach: keep volatile external systems away from the core application logic.

Grab my book Clean Code Zero to One to master those skills.

Later, we'll implement this boundary with an adapter that maps source-specific data into our stable contract.

RULE 2: The pipeline must be payload-agnostic**

A common mistake is coupling the processing pipeline to one specific dataset. For example, hardcoding the entire pipeline around:

{
  "address": "string",
  "price": "number"
}

works for real estate, but becomes a problem when the same system needs to process products, jobs, news, or financial data.

The pipeline should therefore remain independent of specific payload keys. Its job is to handle common operations such as:

ingest → normalize → validate → deduplicate → store → compare

Source-specific rules belong in the adapter and validation level. That way, changing the data source does not require cloning the entire pipeline.

3. Why Traditional Web Scrapers Break

When an AI industry or quantitative hedge fund needs to track market trends or competitor headcount shifts across platforms, they aren’t looking for static web pages; rather, they are hunting for massive, continuous datasets to feed predictive analytics systems, RAG engines, or LLM applications.

However, modern Web Application Firewalls (WAFs) like Cloudflare or Akamai frequently issue “soft bans", returning fake HTTP 200 OKresponses that contain CAPTCHAs instead of data:

Furthermore, the traditional scraper is a victim of three primary forces: fragile selectors, silent failures, and aggressive anti-bot systems. When a dev writes a Playwright or Puppeteer script, they often rely on specific DOM paths. If the target site updates its UI, that selector vanishes.

Worst of all are the silent failures. The scraper doesn’t crash; it simply returns _null_ for the price while successfully extracting the address. Your database is now being poisoned with partial records, a far more dangerous outcome than a hard crash.

3.1. The extraction problem

Anti-bot defenses can create another failure mode. Even when your headers look clean, making too many requests per second or scraping too aggressively from a single IP can trigger a site’s edge defenses. Instead of allowing the request to reach the site’s internal systems, the CDN can reject it with responses such as 503 Slow Down or 503 Service Unavailable:

A managed scraper API, such as the Web Unlocker or the Scraping Browser API, can auto-fix the 503 error. But getting past the security layer is only the first half of the problem.

Even when a handcrafted scraper successfully retrieves the page, the output may still be a messy, unstructured collection of nested strings. That output is difficult to reuse reliably across applications, AI agents, vector databases, and analytics systems.

You do not rise to the level of your application goals; you fall to the level of your data collection infrastructure.

The goal is therefore changing:

We are moving away from treating a scraper as a “static script” that we write and toward treating it as a “managed infrastructure” that we describe.

This change allows us to stop worrying about how the data is pulled and start focusing on what the data represents to our application logic.

3.2. Turning a web page into structured data

To see this transformation in practice, consider tracking the US real estate market using a Zillow Seattle search page:

This raw page contains property cards, filtering dropdowns, and pagination elements. Our backend application cannot consume this visual presentation directly. It requires a structured payload closer to this:

{
  "price": 1450000,
  "bathrooms": 2,
  "square_feet": 2230,
  "address": "2714 10th Avenue W",
  "city": "Seattle",
  "zip_code": "98119",
  "status": "Active"
}

That transformation is the first data engineering problem: turning a human-facing webpage into a reliable data payload.

To solve this, a traditional Playwright scraper might locate an element like this:

const price = await page
.locator('[data-test="property-card-price"]')
.textContent();

Or perhaps the developer finds a class in the current HTML:

const price = await page
.locator('.property-card-price')
.textContent();

Both approaches may work today. But they tightly couple the scraper to the site’s current HTML structure. When that structure changes, the selectors stop matching. Even worse, the scraper might not throw an obvious runtime error; it will simply pass silent null values down your pipeline:

{
  "price": null,
  "bedrooms": 4,
  "bathrooms": 2
}

plaintext

Fortunately, that is manageable for one page. It becomes much harder when a traditional scraper needs to collect thousands of records.

3.3. Scaling beyond a single page

One property page is not the real challenge. A real estate or e-commerce application needs thousands of listings across cities, ZIP codes, and neighborhoods. Now the scraper has to deal with pagination, filtering, changing page structures, and thousands of individual records.

At this point, the HTML document is no longer the product. The data is the product.

And that data still isn’t useful until we decide how the application will consume it. Instead of keeping raw HTML, we may want:

JSON
CSV
database records
Data API responses
Webhook events
object storage

The important question becomes:

How can I reliably turn changing web pages into structured data that our software can use?

This is where the responsibilities of the scraper and the application need to separate:

3.4. From extracted data to application logic

Once we start collecting useful data, a completely different set of engineering questions appears. These questions determine whether our system is simply collecting data or actually producing useful application logic.

For instance:

Which records are new?
Which records changed?
What changed between two runs?
Can we reuse the same pipeline for another target or application?
How do we identify meaningful changes in historical data?

Eventually, a user might ask:

“Find 3 bedroom houses under my budget that recently dropped in price and explain which ones look like the best opportunities.”

A scraper cannot answer that question. It only extracts today’s price and doesn’t know what that price was yesterday or whether the change is significant.

But our application does.

A scraper, even an advanced extraction system such as Scraper Studio, is very good at turning changing web pages into structured data, but they are inherently stateless.

This distinction becomes much clearer with this example. Our pipeline could determine:

_“This product lost 31.27% of its value compared with yesterday and crossed a critical risk threshold.”_

To achieve this, our application needs to remember yesterday’s price, compare the two values, calculate the change, and decide what that change means.

Ultimately, the real business value is rarely the raw extraction itself. It comes from the intelligence built around the data.

3.5. The maintenance loop

At this point, you might wonder why we don’t simply clone this existing repo and move on.

Because the goal isn’t just to make one scraper work. I want you to understand how the mechanism works under the hood and build each part yourself.

If you rely entirely on AI automation loops without understanding what happens underneath, you’ll eventually be lost in the ocean. You will stop working on the actual application and start debugging the extraction infrastructure again.

This creates a maintenance loop: an expensive cycle where developers spend hours debugging proxy routing, browser behavior, selectors, and DOM mutations instead of shipping application features.

4. The Rise of Self-Healing AI-Native Scrapers

To break the “maintenance loop", let’s compare the three most common approaches developers use today:

DIY (Playwright / Scrapy). Take Weeks. You write the crawler, wire up proxies, run headless browsers, handle queues, retries, and anti-bot.
Apify. Take days to weeks. You build your own actor, meaning writing the crawler, integrating a proxy, handling retries, and packaging it as an actor. Using a third-party actor from the Apify Store is faster, but you do not own the code.
Scraper Studio. Take minutes per site. Describe the data you need in plain English, and its AI Agent can generate a scraper and output schema. You can then test and modify the generated JavaScript in the built-in IDE, or work with Bright Data through its CLI and coding-agent (Claude/Cursor) integrations by installing Bright Data skills via npx skills add brightdata/skills.

The important difference is how much of the extraction infrastructure you have to build and maintain yourself.

4.1. The Self-Healing Advantage

Bright data has a unique one-click Self‑Healing feature for updating an existing scraper when its extraction logic becomes outdated. Plus, it’s AI-assisted scraper generation takes few minutes to crawl.

Try generating your first prompt-based scraper using Scraper Studio here.

For example, without writing Playwright project and manually designing selectors, you can describe a task like

“Extract the property address, price, bedrooms, bathrooms, square footage, city, ZIP code, and listing status from this real-estate website.”

The AI agent will generate the scraper and schema, which we can test and refine before running it. Scraper Studio can then be triggered through its interface, API, or scheduled runs.

If you don't like IDE, you can also run Bright Data CLI locally. We’ll use that later when we deliberately break the scraper and repair it.

When a target site is redesigned and a scraper begins returning null or missing fields, you fix it in place using bdata scraper heal or using the built-in IDE's “Self-Healing" feature like this:

bdata scraper heal <COLLECTOR_ID> \
  "The price field is returning null after the page redesign. \
   Extract the current price and currency from the new page structure."

Because this process preserves the stable Collector ID, every API trigger, schedule, and integration keeps working without a single line of code being changed in our application layer.

This converts the “Maintenance Loop” from a hours-long engineering hurdle into a five-minute terminal command.

4.2. Why Scraper Studio Fits Our Architecture

At the beginning of this guide, we established two rules:

The application trusts a stable data contract.
The pipeline remains payload-agnostic.

Scraper Studio fits underneath those rules.

We are not giving up control of our application. Our Node.js pipeline still owns the stable data contract. The extraction layer can change while the application layer remains stable.

That’s exactly the separation we designed in Section 1.

4.3. What About Cost, Scale, and Concurrency

Cost also depends on what the scraper properly loads. Bright Data currently lists 5,000 page loads per month in the free Scraper Studio tier and $1.50 per 1,000 page loads on pay-as-you-go plan with unlimited concurrency.

Keep in mind that a page load is not necessarily the same as one output record. A single page can produce multiple records, so pagination and navigation behavior matter when estimating usage and cost.

We’ll see why this matters when we build the Texas real-estate tracker.

The main thing is, offloading these complexities means we don’t have to build, manage, or scale proxy pools and headless browsers ourselves. That gives us the architecture we really want.

Now let’s build it.

5. Project Setup

Before building the pipeline logic, let’s create a small Node.js application that will eventually consume our scraped data.

We’ll keep the project modular so the same structure can be reused with other data sources later.

Open your terminal and create the project:

mkdir self-healing-ai-data-pipeline
cd self-healing-ai-data-pipeline
git init
npm init -y

Install the dependencies:

npm install express dotenv
npm install -D nodemon

Then open the project in VS Code:

code .

We’ll build the application incrementally instead of creating a large codebase upfront.

5.1 Configure `package.json`

Open package.json and configure the scripts and ECMAScript Module support:

{
  "type": "module",
  "scripts": {
    "start": "node src/server.js",
    "dev": "nodemon src/server.js"
  }
}

The "type": "module" setting allows us to use modern import and export syntax throughout the project.

5.2 Add Environment Variables

Create a .env file in the project root:

PORT=3000
BRIGHTDATA_API_TOKEN=your_api_token
BRIGHTDATA_COLLECTOR_ID=your_collector_id

We’ll use the Bright Data credentials later when we connect the application to Scraper Studio.

Also create a .gitignore file:

node_modules/
.env
.DS_Store
data/*.tmp

javascript

This keeps local dependencies, environment variables, and temporary files out of Git.

5.3 Create the Node.js Server

Now create the following file: src/server.js

Add the initial server:

import express from "express";
import dotenv from "dotenv";
dotenv.config();
const app = express();
const PORT = process.env.PORT || 3000;
app.get("/", (req, res) => {
  res.json({
    message: "Self-healing AI Data Pipeline is Activated!"
  });
});
app.listen(PORT, () => {
  console.log(`Pipeline running on port ${PORT}`);
});

Start the development server:

npm run dev

You should see:

Pipeline running on port 3000

Open http://localhost:3000 in your browser.

You should get:

{
  "message": "Self-healing AI Data Pipeline is Activated!"
}

We are now ready to step away from the text editor and construct our remote infrastructure producer. No! Not yet; we need to build our house first: the pipeline prototype.

6. Building the Pipeline Prototype

Now that the Node.js application is running, let’s build the processing layer that sits behind it.

The goal is to keep this layer independent of any specific website or dataset. Instead of hardcoding fields such as bedrooms, symbol, or salary, we'll translate different source formats into one small internal contract:

id
value
label

The source-specific code will live in an adapter. The pipeline itself will only process this common structure.

6.1 Design the Prototype Structure

Create the required directories and modules:

macOS/Linux:

mkdir -p src/api src/collectors src/pipeline src/storage/snapshots
mv src/server.js src/api/server.js
touch src/pipeline/normalize.js src/pipeline/validate.js src/pipeline/deduplicate.js src/pipeline/index.js src/collectors/mock.js

Windows PowerShell:

New-Item -ItemType Directory -Force src/api, src/collectors, src/pipeline, src/storage/snapshots
Move-Item -Path src/server.js -Destination src/api/server.js -Force
'normalize.js', 'validate.js', 'deduplicate.js', 'index.js' | ForEach-Object {
  New-Item -ItemType File -Force "src/pipeline/$_"
}
New-Item -ItemType File -Force src/collectors/mock.js

Because the server has moved, update the development script in package.json:

"dev": "nodemon src/api/server.js"

Our project now looks like this:

self-healing-ai-data-pipeline/
├── src/
│   ├── api/
│   │   └── server.js
│   ├── collectors/
│   │   └── mock.js
│   ├── pipeline/
│   │   ├── normalize.js
│   │   ├── validate.js
│   │   ├── deduplicate.js
│   │   └── index.js
│   └── storage/
│       └── snapshots/
├── .env
├── .gitignore
└── package.json

6.2 Normalize the Data

The normalizer converts messy source values into the types expected by our internal contract.

For example, a scraper might return $502,000 as a string. Our application should receive 502000 as a number.

Open:

src/pipeline/normalize.js

Add:

/**
 * Strips formatting artifacts and typecasts values to strict numeric floats.
 */
export function toNumber(value) {
  if (value === null || value === undefined) return null;
  const cleaned = String(value).replace(/,/g, "").replace(/[^0-9.]/g, "");
  if (!cleaned) return null;
  const number = Number(cleaned);
  return Number.isFinite(number) ? number : null;
}
/**
 * Normalizes an individual raw payload token into our unified application envelope.
 */
export function normalizeRecord(record, index) {
  return {
    id: record.id ?? `entity-${index + 1}`,
    value: toNumber(record.value),
    label: typeof record.label === "string" ? record.label.replace(/\s+/g, " ").trim() : "Untitled Entry",
    captured_at: new Date().toISOString()
  };
}
export function normalizeCollection(records) {
  if (!Array.isArray(records)) return [];
  return records.map((record, index) => normalizeRecord(record, index));
}

The key part is that the rest of the application no longer needs to understand how the original value was formatted.

6.3 Validate the Contract

Normalization gives us a consistent shape, but consistent shape does not necessarily mean valid data.

The validator acts as the integrity gate before records continue through the pipeline.

Open:

src/pipeline/validate.js

Add:

/**
 * Evaluates core schema boundaries for an individual normalized record.
*/
export function validateRecord(record) {
  const infractions = [];
  if (!record.id || record.id.startsWith("entity-")) {
    infractions.push("Contract Failure: Missing primary unique identification parameter.");
  }
  if (record.value === null || record.value <= 0) {
    infractions.push("Contract Failure: Invalid or zero numeric metric value.");
  }
  return {
    valid: infractions.length === 0,
    errors: infractions
  };
}
export function validateCollection(records) {
  if (!Array.isArray(records)) return [];
  return records.map((record) => ({
    record,
    ...validateRecord(record)
  }));
}

Invalid records are not allowed to silently continue. Instead, the validation result tells us which records passed and which need attention.

6.4 Remove Duplicate Records

Next, we need to prevent the same entity from appearing multiple times in a collection.

Open: src/pipeline/deduplicate.js

Add:

/**
 * Filters out duplicate records from the collection using our unique identifier.
*/
export function deduplicateCollection(records) {
  if (!Array.isArray(records)) return [];
  const uniqueKeys = new Set();
  return records.filter((record) => {
    if (uniqueKeys.has(record.id)) return false;
    uniqueKeys.add(record.id);
    return true;
  });
}

The Set gives us a simple in-memory lookup for identifiers we've already seen.

6.5 Compose the Pipeline

Now combine the individual processing stages into one entry point.

Open: src/pipeline/index.js

Add:

import { normalizeCollection } from "./normalize.js";
import { deduplicateCollection } from "./deduplicate.js";
import { validateCollection } from "./validate.js";
/**
 * Runs raw extraction arrays through our universal processing middle tier.
 */
export function processRawIngestion(rawData) {
  const normalized = normalizeCollection(rawData);
  const unique = deduplicateCollection(normalized);
  const validationReport = validateCollection(unique);
  const cleanRecords = validationReport.filter(item => item.valid).map(item => item.record);
  const quarantineRecords = validationReport.filter(item => !item.valid);
  return {
    pipeline_state: quarantineRecords.length > 0 ? "WARNING" : "HEALTHY",
    telemetry: {
      raw_processed: normalized.length,
      purged_duplicates: normalized.length - unique.length,
      passed_compliance: cleanRecords.length,
      failed_compliance: quarantineRecords.length
    },
    data: cleanRecords,
    quarantine: quarantineRecords
  };
}

Our processing flow is now:

Raw Data
   ↓
Normalize
   ↓
Deduplicate
   ↓
Validate
   ↓
Clean Records + Quarantine

This is deliberately small. Each stage has one responsibility, and the pipeline does not know where the data originally came from.

6.6 Test Multiple Data Sources

Now let’s prove that idea. Create three small mock datasets in: src/collectors/mock.js

Each source uses completely different field names:

// Test Datasets mimicking HAR, Yahoo Finance, and RemoteOK raw fields
const rawHarPayload = [{ "id": "har-992", "price": "$502,000", "address": "2623 Pomeran Dr" }];
const rawYahooPayload = [{ "symbol": "AAPL", "price": "$234.12", "company_name": "Apple Inc." }];
const rawRemoteOkPayload = [{ "job_id": "remote-77", "salary": "$185,000", "position": "Software Engineer" }];
/**
 * Universal Source Adapter Layer translating source-specific 
fields to our pipeline interface
 */
export function getAdaptedMockSource(source) {
  if (source === "har") {
    return rawHarPayload.map(item => ({ id: item.id, value: item.price, label: item.address }));
  }
  if (source === "yahoo") {
    return rawYahooPayload.map(item => ({ id: item.symbol, value: item.price, label: item.company_name }));
  }
  if (source === "remoteok") {
    return rawRemoteOkPayload.map(item => ({ id: item.job_id, value: item.salary, label: item.position }));
  }
  return null;
}

Notice what happened here.

HAR uses price and address.

Yahoo Finance uses symbol and company_name.

RemoteOK uses salary and position.

The pipeline doesn’t care.

The adapter translates each source into:

{
  "id": "...",
  "value": "...",
  "label": "..."
}

That is the Adapter Strategy we designed earlier in action.

6.7 Connect the Pipeline to Express

Now expose the pipeline through a small test endpoint.

Open: src/api/server.js

Replace the initial server with:

import express from "express";
import dotenv from "dotenv";
import { getAdaptedMockSource } from "../collectors/mock.js";
import { processRawIngestion } from "../pipeline/index.js";
dotenv.config();
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
app.get("/api/pipeline/test-mock", (req, res) => {
  const source = req.query.source; // har, yahoo, or remoteok
  const adaptedPayload = getAdaptedMockSource(source);
  if (!adaptedPayload) {
    return res.status(400).json({ error: "Invalid source flag configuration." });
  }
  try {
    const processingResult = processRawIngestion(adaptedPayload);
    res.json({
      status: "success",
      pipeline_telemetry: processingResult.telemetry,
      results: processingResult.data
    });
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});
app.listen(PORT, () => console.log(`Pipeline server online on port: ${PORT}`));

6.8 Test the Prototype

Start the server:

npm run dev

Then test each source:

curl "http://localhost:3000/api/pipeline/test-mock?source=har"

curl "http://localhost:3000/api/pipeline/test-mock?source=yahoo"

curl "http://localhost:3000/api/pipeline/test-mock?source=remoteok"

Each request passes through the same pipeline.

For example, the “Remoteok” request produces a normalized record similar to:

{
  "status": "success",
  "pipeline_telemetry": {
    "raw_processed": 1,
    "purged_duplicates": 0,
    "passed_compliance": 1,
    "failed_compliance": 0
  },
  "results": [    {
      "id": "remote-77",
      "value": 18500,
      "label": "Software engineer"
    }
  ]
}

We’ve now proven the core architectural idea: different source payloads can enter the same processing pipeline without changing the pipeline code.

The adapter handles source-specific structure. The pipeline handles application-level processing.

That separation will become vital when we replace these mock collectors with real web data.

6.9 Commit the Prototype

The prototype is working, so commit the current state:

git add .
git commit -m "Add reusable pipeline prototype with mock sources"

We now have a working processing core.

Next, we’ll replace the mock source with the real web-data extraction layer.

7. Teaching Our Pipeline to Remember

Our business goal is not just to collect the current state of a website. We want to track how that state changes over time.

Look at a simple example:

Yesterday: House A — $500,000
Today: House A — $450,000

How does the scraper know the price dropped?

It doesn’t.

A scraper normally sees the current page. Remembering what happened during previous runs is the responsibility of our application.

7.1. Why a Scraper Run Is Stateless

As we discussed this concept earlier, a scraper sees the current state of a page:

House A → $450,000

The previous crawl is not automatically available to the next run.

This is where our pipeline needs to add state. We’ll store the result of the previous run as a local snapshot. When a new run completes, we’ll compare the new dataset against that snapshot.

The basic flow looks like this:

Current Run
     ↓
Processed Records
     ↓
latest_snapshot.json
     ↑
Previous Run
     ↓
Compare
     ↓
Created / Updated / Removed

Created: A new listing appeared.
Updated: An existing listing changed, such as a price drop.
Removed: A previously tracked listing is no longer present.

For these comparisons, we’ll use JavaScript Map objects later in the pipeline. This gives us linear O(N) comparison work rather than repeatedly scanning the entire dataset for every record.

We don’t need a database server for this prototype. A local JSON snapshot is enough to demonstrate the state-management layer before introducing more infrastructure.

7.2. Create Local Snapshot Storage

The storage directory already exists from our project structure. Now create the snapshot module:

mkdir -p src/storage/snapshots
touch src/pipeline/snapshot.js

Open src/pipeline/snapshot.js.

We’ll use Node.js’s native asynchronous fs/promises API to read and write the snapshot file.

Add:

import fs from "node:fs/promises";
import path from "node:path";
const snapshotDir = path.join(
  process.cwd(),
  "src",
  "storage",
  "snapshots"
);
const snapshotFile = path.join(
  snapshotDir,
  "latest_snapshot.json"
);
/**
 * Saves the processed dataset to local disk.
 */
export async function saveSnapshot(records) {
  try {
    await fs.mkdir(snapshotDir, { recursive: true });
    await fs.writeFile(
      snapshotFile,
      JSON.stringify(records, null, 2),
      "utf8"
    );
    return true;
  } catch (err) {
    console.error(
      `[Snapshot Write Failure] Disk IO stalled: ${err.message}`
    );
    return false;
  }
}
/**
 * Loads the previous snapshot into memory.
 */
export async function loadSnapshot() {
  try {
    const rawBuffer = await fs.readFile(
      snapshotFile,
      "utf8"
    );
    return JSON.parse(rawBuffer);
  } catch (notFoundError) {
    // Return an empty array when no previous snapshot exists.
    return [];
  }
}

There are only two duties here:

saveSnapshot() writes the latest processed records to disk.

loadSnapshot() retrieves the previous snapshot when the next pipeline run starts.

On the first run, there is no previous snapshot, so loadSnapshot() simply returns an empty array.

7.3. Test Snapshot Persistence

Before using snapshots for change detection, let’s verify that our storage layer can actually write and read data.

Open:

src/api/server.js

For this isolated storage test, use:

import express from "express";
import dotenv from "dotenv";
import { saveSnapshot } from "../pipeline/snapshot.js";
dotenv.config();
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
app.get("/health", (req, res) => res.json({ status: "ok" }));
// Snapshot Persistence Verification Route
app.post("/api/save", async (req, res) => {
  const records = req.body.records;
  if (!Array.isArray(records)) {
    return res.status(400).json({
      error: "Invalid Payload: Input must be a 'records' array."
    });
  }
  // Basic inline mapping to standardize test elements instantly inside server context
  const standardized = records.map((item, idx) => ({
    id: item.id ?? `id-${idx + 1}`,
    value: Number(
      String(item.value ?? item.price ?? "")
        .replace(/[^0-9.]/g, "")
    ),
    label: item.label ?? item.address ?? "Untitled Entry"
  }));
  const successfullySaved = await saveSnapshot(standardized);
  if (!successfullySaved) {
    return res.status(500).json({
      error: "Failed to persist snapshot container to disk."
    });
  }
  res.json({
    message: "Snapshot saved successfully.",
    record_count: standardized.length
  });
});
app.listen(PORT, () =>
  console.log(`Pipeline server listening on port: ${PORT}`)
);

Reload the program:

npm run dev

Open a second PowerShell window and create a test payload:

$TestData = @{
  records = @(
    @{
      id = "123"
      price = "$450,000"
      address = "2623 Pomeran Dr"
    }
  )
} | ConvertTo-Json -Depth 10 -Compress

Send it to the local endpoint:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:3000/api/save" `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body $TestData

You should receive a successful response confirming that the snapshot was saved.

Now verify the actual file on disk:

Get-Content .\src\storage\snapshots\latest_snapshot.json

You should see the stored canonical record:

🧠😎 Our pipeline now has memory.

The scraper can continue to focus on extraction. The application owns the historical state.

And that gives us the missing piece for the next step: comparing two snapshots to determine exactly what changed.

8. Detecting Historical Changes

Now that our pipeline can remember the previous run, we can FINALLY answer the question we started with:

What changed since the last crawl?

Consider these two snapshots:

Yesterday:

House A → $500,000
House B → $350,000

Today:

House A → $450,000
House B → $350,000
House C → $600,000

A human can immediately spot three different states:

House A: The price changed from $500,000 to $450,000.
House B: The value stayed unchanged.
House C: A new listing appeared.

Our pipeline needs to detect those states programmatically.

We’ll represent them with three arrays:

created[] — records that exist today but not in the previous snapshot.
updated[] — records that exist in both snapshots but whose tracked value changed.
removed[] — records that existed previously but are missing from today's dataset.

8.1 Building the Dataset Comparison Engine

Create:

src/pipeline/compare.js

We could compare every incoming record against every historical record using nested loops. That approach can reach O(N²) time complexity as the dataset grows.

Instead, we’ll index both datasets by their unique IDs using JavaScript Map.

Building the maps takes O(N), and each lookup is approximately O(1), allowing the overall comparison to remain O(N).

Add the following to src/pipeline/compare.js:

/**
 * Executes a delta analysis between two historical runs.
 *
 * @param {Array<Object>} yesterdayBaseline - Previous snapshot.
 * @param {Array<Object>} todayIncoming - Current dataset.
 * @returns {Object} Created, updated, and removed records.
 */
export function computeHistoricalDelta(yesterdayBaseline, todayIncoming) {
  // Guard against malformed parameters to keep processing loops safe
  const baselineArray = Array.isArray(yesterdayBaseline) ? yesterdayBaseline : [];
  const incomingArray = Array.isArray(todayIncoming) ? todayIncoming : [];
  // Pre-index collections into highly efficient O(1) maps to avoid O(N^2) array scan stalls
  const baselineMap = new Map(baselineArray.map(item => [item.id, item]));
  const incomingMap = new Map(incomingArray.map(item => [item.id, item]));
  const created = [];
  const updated = [];
  const removed = [];
  // Stage 1: Isolate additions and modifications by scanning today's fresh input keys
  for (const [id, incomingItem] of incomingMap.entries()) {
    const historicalItem = baselineMap.get(id);
    // If item doesn't exist in baseline, it's a new lifecycle record
    if (!historicalItem) {
      created.push(incomingItem);
      continue;
    }
    // Identify value metric variance since the previous system sweep
    if (incomingItem.value !== historicalItem.value) {
      const numericDelta = incomingItem.value - historicalItem.value;
      // Senior Practice: Guard against division-by-zero crashes if historical value is 0
      const baseDivisor = historicalItem.value === 0 ? 1 : historicalItem.value;
      const percentageShift = (numericDelta / baseDivisor) * 100;
      updated.push({
        id,
        label: incomingItem.label,
        previous_value: historicalItem.value,
        current_value: incomingItem.value,
        delta: numericDelta,
        // Safely invoke toFixed now that non-finite Infinity properties are blocked
        percent_change: Number(percentageShift.toFixed(2))
      });
    }
  }
  // Stage 2: Isolate deletions by verifying which historical items are missing today
  for (const [id, historicalItem] of baselineMap.entries()) {
    if (!incomingMap.has(id)) {
      removed.push(historicalItem);
    }
  }
  return { created, updated, removed };
}

The algorithm has two passes. The first scans today’s records to find created and updated entries.

The second scans yesterday’s records to find removed entries.

For our real estate tracker, this gives us a clean delta such as:

{
  "created": ["House C"],
  "updated": ["House A"],
  "removed": []
}

The actual updated records also contain the previous value, current value, absolute delta, and percentage change. That information will become useful when we classify significant changes.

8.2 Testing the Comparison Engine

Let’s connect the comparison engine to a temporary Express endpoint and test it against two simulated crawl runs.

Open: src/api/server.js

WRite:

import express from "express";
import dotenv from "dotenv";
import { saveSnapshot, loadSnapshot } from "../pipeline/snapshot.js";
import { computeHistoricalDelta } from "../pipeline/compare.js";
dotenv.config();
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
app.get("/health", (req, res) => res.json({ status: "ok" }));
// Delta Analysis Local Verification Route
app.post("/api/analyze", async (req, res) => {
  const incomingRecords = req.body.records;
  if (!Array.isArray(incomingRecords)) {
    return res.status(400).json({ error: "Invalid Payload: Must pass an array of 'records'." });
  }
  try {
    // 1. Fetch yesterday's snapshot reference state from local disk storage
    const historicalBaseline = await loadSnapshot();
    // 2. Compute mutations across lifecycle boundaries via our comparison engine
    const deltaAnalysis = computeHistoricalDelta(historicalBaseline, incomingRecords);
    // 3. Update local snapshot file to save today's records as tomorrow's baseline
    await saveSnapshot(incomingRecords);
    res.json({
      created: deltaAnalysis.created.length,
      updated: deltaAnalysis.updated.length,
      removed: deltaAnalysis.removed.length,
      raw_deltas: deltaAnalysis
    });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});
app.listen(PORT, () => console.log(`Pipeline running on port: ${PORT}`));

Reload the server:

npm run dev

We’ll use PowerShell to mimic two separate crawl executions.

First, seed the snapshot with yesterday’s data:

$BaselineData = @{
  records = @(
    @{ id = "A"; value = 500000; label = "House A" },
    @{ id = "B"; value = 350000; label = "House B" }
  )
} | ConvertTo-Json -Depth 5 -Compress

Send it to the analysis endpoint:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:3000/api/analyze" `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body $BaselineData

This creates our baseline snapshot.

Now recreate today’s crawl:

$TodayData = @{
  records = @(
    @{ id = "A"; value = 450000; label = "House A" },
    @{ id = "B"; value = 350000; label = "House B" },
    @{ id = "C"; value = 600000; label = "House C" }
  )
} | ConvertTo-Json -Depth 5 -Compress

Send it through the same endpoint:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:3000/api/analyze" `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body $TodayData

Success Result

The backend loads the previous snapshot, compares it with today’s records, and then saves today’s dataset as the new baseline.

You should see counts similar to:

created : 1
updated : 1
removed : 0

plaintext

Note: I run the seeding scripts two times during local testing; that's why the terminal is showing “_created: 2_"

The detailed response should identify House C as created and House A as updated.

There’s an important detail here.

After the request finishes, today’s dataset becomes latest_snapshot.json.

That means the next execution will compare against today, not the original baseline.

This is what turns a simple file into a rolling state mechanism:

Run 1
   ↓
Snapshot A
Run 2
   ↓
Compare against Snapshot A
   ↓
Save Snapshot B
Run 3
   ↓
Compare against Snapshot B
   ↓
Save Snapshot C

Our pipeline can now detect historical changes.

But not every change deserves the same response.

A $5,000 price adjustment and a 90% price collapse are both technically updated events, but they shouldn't necessarily receive the same treatment.

That’s where event classification comes in.

8.3 Classifying System Events

For this prototype, we’ll classify value changes using configurable thresholds.

These numbers are our example thresholds, not universal rules:

Under 15% — **MINOR**: Small change that may not require immediate attention.
15% to under 30% — **MAJOR**: Significant change worth monitoring.
30% or more downward — **CRITICAL**: Large negative change that should be investigated.

A large change could be a genuine business event, but it could also indicate an upstream extraction problem. The classification layer doesn’t decide which one it is. It simply makes the change visible and machine-readable.

Go to: src/pipeline/classify.js

Add:

/**
 * Classifies calculated deltas into structured system events.
 *
 * @param {Object} deltas - Created, updated, and removed records.
 * @returns {Array<Object>} Structured event timeline.
 */
export function classifySystemEvents(deltas) {
  const alertsTimeline = [];
// Process value changes.
  for (const change of deltas.updated) {
    let severity = "MINOR";
    let type = "VALUE_MUTATED";
    const absoluteShift = Math.abs(
      change.percent_change
    );
    if (absoluteShift >= 15 && absoluteShift < 30) {
      severity = "MAJOR";
      type = "SIGNIFICANT_VALUE_SHIFT";
    }
    if (change.percent_change <= -30) {
      severity = "CRITICAL";
      type = "ANOMALOUS_VALUE_COLLAPSE";
    }
    alertsTimeline.push({
      id: change.id,
      type,
      severity,
      message: `${change.label} (ID: ${change.id}) shifted by ${change.percent_change}%.`,
      details: change
    });
  }
  // Process newly created records.
  for (const record of deltas.created) {
    alertsTimeline.push({
      id: record.id,
      type: "RECORD_CREATED",
      severity: "MINOR",
      message: `New entry mapped to tracking logs: ${record.label}`,
      details: record
    });
  }
  // Process removed records.
  for (const record of deltas.removed) {
    alertsTimeline.push({
      id: record.id,
      type: "RECORD_DELETED",
      severity: "MINOR",
      message: `Entry removed from upstream dataset: ${record.label}`,
      details: record
    });
  }
  return alertsTimeline;
}

Now the pipeline doesn’t just say:

House A changed.

It can produce a structured event:

{
  "id": "A",
  "type": "ANOMALOUS_VALUE_COLLAPSE",
  "severity": "CRITICAL"
}

That distinction becomes valuable when another system needs to consume these events.

8.4 Connect Event Classification

Now integrate the classifier into our Express endpoint.

Open: src/api/server.js

Write:

import express from "express";
import dotenv from "dotenv";
import { saveSnapshot, loadSnapshot } from "../pipeline/snapshot.js";
import { computeHistoricalDelta } from "../pipeline/compare.js";
import { classifySystemEvents } from "../pipeline/classify.js";
dotenv.config();
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
app.post("/api/events", async (req, res) => {
  const incomingRecords = req.body.records;
  if (!Array.isArray(incomingRecords)) {
    return res.status(400).json({
      error: "Invalid Payload."
    });
  }
  try {
    const historicalBaseline = await loadSnapshot();
    const deltaAnalysis = computeHistoricalDelta(
      historicalBaseline,
      incomingRecords
    );
    const structuredAlertsTimeline =
      classifySystemEvents(deltaAnalysis);
    await saveSnapshot(incomingRecords);
    const criticalAnomaly =
      structuredAlertsTimeline.find(
        (event) => event.severity === "CRITICAL"
      );
    if (criticalAnomaly) {
      return res.json({
        severity: "CRITICAL",
        message: `${criticalAnomaly.details.label} lost ${Math.abs(criticalAnomaly.details.percent_change)}% of its value.`,
        timeline: structuredAlertsTimeline
      });
    }
    res.json({
      severity: "HEALTHY",
      message:
        "Ingestion finalized with no critical anomalies registered.",
      timeline: structuredAlertsTimeline
    });
  } catch (error) {
    res.status(500).json({
      error: error.message
    });
  }
});
app.listen(PORT, () =>
  console.log(`Event Classification server active on port: ${PORT}`)
);

8.5 Test Event Classification

Let’s deliberately create a large negative change and see whether our classifier catches it.

Open:

src/storage/snapshots/latest_snapshot.json

Replace its contents with:

[  {
    "id": "A",
    "value": 5000000,
    "label": "House A"
  }
]

Now create the test payload in PowerShell:

$AnomalyTestData = @{
  records = @(
    @{ id = "A"; value = 325000; label = "House A" }
  )
} | ConvertTo-Json -Depth 5 -Compress

powershell

Send it to /api/events:

Invoke-RestMethod `
  -Uri "http://127.0.0.1:3000/api/events" `
  -Method POST `
  -Headers @{"Content-Type"="application/json"} `
  -Body $AnomalyTestData

The value changes from $5,000,000 to $325,000, which is a -93.5% change.

Because that is below our -30% threshold, the classifier should return a CRITICAL event. Have a look:

Now run the same request again. This time, the result should no longer be critical.

Why?

Because the previous request already saved $325,000 as the latest snapshot. The second request is therefore comparing $325,000 against $325,000, producing no value change.

That’s the rolling state mechanism working as intended. Have a look:

We now have three important abilities:

Extract
  ↓
Remember
  ↓
Compare
  ↓
Classify
  ↓
Machine-readable Events

The pipeline can now detect when its data changes and distinguish ordinary updates from unusually large shifts.

The next challenge is more interesting: what happens when the scraper itself breaks?

Let’s deliberately break the extraction layer and see whether the rest of our pipeline can detect the failure.

9. Building a Reusable Collector Factory

So far, our pipeline has been working with local datasets.

That was intentional. We first built and tested the processing layer independently from external extraction services. Now we can connect that pipeline to a real web data provider without mixing provider-specific logic into our application.

Our pipeline already knows how to:

Normalize inconsistent values
Validate incoming records
Remove duplicates
Store historical snapshots
Compare datasets over time
Classify notable market changes

The next step is to make the extraction layer replaceable.

If we change scraping providers later, the processing pipeline should not need to change.

9.1 Designing the Collector Abstraction

A common mistake is to put provider-specific code directly inside the main pipeline.

That creates a dependency like this:

Node.js Pipeline
      ↓
Bright Data-specific code
      ↓
Bright Data API

If we later switch providers, we have to modify the pipeline itself.

Instead, we’ll put an abstraction between the pipeline and the extraction provider:

                  ┌─────────────────────┐
                  │   Node.js Pipeline  │
                  └──────────┬──────────┘
                             │
                    Collector Interface
                             │
              ┌──────────────┴──────────────┐
              │                             │
   ScraperStudioCollector          FirecrawlCollector
              │                             │
        Bright Data                    Firecrawl

The pipeline only needs one contract: Give the collector a target and receive structured records.

The implementation behind that contract can change.

Our collector directory will look like this:

src/
└── collectors/
    ├── BaseCollector.js
    ├── ScraperStudioCollector.js
    ├── FirecrawlCollector.js
     ...other collectors
    └── CollectorFactory.js

This is the practical side of the Port and Adapter pattern we introduced earlier: the application depends on an interface, while individual providers implement that interface.

9.2 Building the Base Collector

Create:

src/collectors/BaseCollector.js

This class defines the contract that every collector must follow.

/**
 * Abstract Base Collector Interface.
 */
export class BaseCollector {
  /**
   * Programmatically retrieves raw, unsanitized extraction payload
   * arrays from a source target.
   *
   * @param {string} url - Target public website URL.
   * @returns {Promise<Array<Object>>} - Unnormalized data array.
   */
  async collect(url) {
    throw new Error(
      "Architecture Violation: Method 'collect(url)' must be implemented by subclass."
    );
  }
}

The base class does not know anything about Bright Data, Firecrawl, Playwright, or any other extraction system.

It only defines the expected behavior.

Every concrete collector must implement collect(url).

9.3 Defining the Primary Provider

Our primary provider will be Bright Data Scraper Studio.

Create:

src/collectors/ScraperStudioCollector.js

import { BaseCollector } from "./BaseCollector.js";
/**
 * Our primary provider (infrastructure layer).
 */
export class ScraperStudioCollector extends BaseCollector {
  async collect(url) {
    // Provider-specific implementation will be added here.
    throw new Error(
      "Bright Data Infrastructure Pipeline Placeholder. Activate via configuration parameters."
    );
  }
}

For now, this is intentionally a placeholder.

We are testing the architecture before adding the external API implementation. This makes it easier to verify that the rest of the application does not depend on Bright Data-specific details.

9.4 Adding a Secondary Provider

Now let’s add a second collector to prove that the abstraction is not tied to one provider.

Create:

src/collectors/FirecrawlCollector.js

import { BaseCollector } from "./BaseCollector.js";
/**
 * Alternative web crawler provider used to demonstrate
 * multi-provider collector architecture.
 */
export class FirecrawlCollector extends BaseCollector {
  async collect(url) {
    // Provider-specific implementation will be added here.
    throw new Error(
      "Firecrawl Infrastructure Pipeline Placeholder. Activate via configuration parameters."
    );
  }
}

Again, this is only an architectural placeholder.

The important part is that both providers expose the same collect(url) method.

The pipeline does not need to know which provider is behind it.

9.5 Implementing the Collector Factory

Now we need a single place that decides which provider to instantiate.

Create:

src/collectors/CollectorFactory.js

import { ScraperStudioCollector } from "./ScraperStudioCollector.js";
import { FirecrawlCollector } from "./FirecrawlCollector.js";
export class CollectorFactory {
  /**
   * Instantiates a concrete data provider dynamically
   * based on the configured provider type.
   *
   * @param {string} providerType - Collector identifier.
   */
  static create(providerType) {
    const type = String(providerType).toUpperCase();
    if (type === "SCRAPER_STUDIO") {
      return new ScraperStudioCollector();
    }
    if (type === "FIRECRAWL") {
      return new FirecrawlCollector();
    }
    throw new Error(
      `Factory Exception: Unsupported collector provider type [${providerType}]`
    );
  }
}

Now the application can request a collector without directly importing or constructing a provider.

For example:

const collector = CollectorFactory.create("SCRAPER_STUDIO");

Later, changing the configuration to:

const collector = CollectorFactory.create("FIRECRAWL");

changes the concrete implementation while keeping the pipeline interface the same.

That is the main benefit of the factory: provider selection is centralized instead of scattered throughout the application.

9.6 Verifying the Architecture

Before connecting real external services, let’s verify that the abstraction works at runtime.

Create:

src/testCollector.js

import dotenv from "dotenv";
import { CollectorFactory } from "./collectors/CollectorFactory.js";
dotenv.config();
async function runSanityCheck() {
  console.log("======================================================================");
  console.log("           COLLECTOR ABSTRACTION ARCHITECTURE SANITY CHECK");
  console.log("======================================================================");
  try {
    // Test 1: Verify factory error mechanics on unsupported providers
    try {
      console.log("[Factory Test] Attempting to create an invalid provider...");
      CollectorFactory.create("UNKNOWN_PROVIDER");
    } catch (e) {
      console.log(
        `✅ Success: Factory correctly caught boundary exception -> "${e.message}"`
      );
    }
    // Test 2: Verify dependency inversion polymorphism
    console.log(
      "\n[Factory Test] Creating concrete Scraper Studio class polymorphically..."
    );
    const collector = CollectorFactory.create("SCRAPER_STUDIO");
    console.log(
      `✅ Success: Instance verified. Class Name: ${collector.constructor.name}`
    );
    console.log("----------------------------------------------------------------------");
    console.log("Result Status: ARCHITECTURE CHECK PASSED.");
    console.log("======================================================================");
  } catch (err) {
    console.error(`❌ Unexpected Architecture Error: ${err.message}`);
  }
}
runSanityCheck();

Save all modules. Run the verification script from your terminal:

node src/testCollector.js

Output:

The test verifies two important behaviors.

First, an unsupported provider is rejected at the factory boundary.

Second, the factory can create a concrete collector through the common abstraction.

We haven’t connected a real external provider yet. That’s deliberate.

The architecture is now ready for that integration without forcing provider-specific code into the processing pipeline.

Commit the current changes if you want:

git add .
git commit -m "add polymorphic collector factory layer + change detection"

The next step is where this abstraction becomes useful: we’ll replace the Scraper Studio placeholder with the real Bright Data integration and start feeding live web data into the pipeline.

10. Generate the First Scraper with Scraper Studio AI Agent

Our processing pipeline is ready. We now need to connect it to a real extraction source.

For this project, we’ll use Bright Data Scraper Studio's built-in Web IDE AI agent to generate the scraper and expose the resulting data to our Node.js pipeline.

Our target is a live Texas real-estate listing page on HAR.com:

The goal isn’t to build a HAR.com-specific application.

We’re using real estate as the case study because it gives us a useful combination of structured fields, detail pages, pagination, and changing values such as property prices.

The processing pipeline remains independent of the source.

Step 1: Initialize the Target URL

You’ll be prompted to enter a target URL.

For this project, enter: https://www.har.com/houston/realestate/for_sale

Below the URL field, you’ll find an optional Add additional instructions field.

The AI agent can inspect the target and ask questions during the configuration process, but providing the extraction requirements upfront gives it a clearer starting point.

For our real-estate tracker, use this custom prompt:

"Collect property information from the listings.
Follow links from the search results page into individual property pages.
Extract:
- Price
- Address
- Bedrooms
- Full bathrooms
- Half bathrooms
- Square footage
- Lot size
- Property type
- MLS number
- Listing status
Handle pagination across multiple result pages.
Exclude advertisements, sponsored placements, and sidebar links.
Return a structured JSON array."

After processing the target, Scraper Studio generates a proposed data schema as below:

Review the proposed fields before approving them.

The Generated Schema Is Temporary:

The schema generated by Scraper Studio describes the external source. It is not the same thing as the internal data contract we designed earlier.

Our Node.js pipeline should not care whether the scraper gets a value from a CSS selector, an AI-generated extraction rule, or a completely different provider. The external extraction layer can change. The internal contract should remain stable.This separation is what allows us to replace the extraction provider later without rewriting the processing pipeline.

Step 2: Configure the Extraction Workflow

If you don’t provide enough information in the additional instructions, Scraper Studio can guide you through an interactive configuration process.

Depending on the target, you may be asked whether you want to:

Extract data from individual property pages
Extract data only from the listing page

For our use case, select “Extract data from individual property pages”.

The listing page gives us links to properties, while the individual pages contain richer property information.

Next, Scraper Studio can ask whether pagination should be handled. Select Yes. This tells the scraper that the collection should account for additional result pages rather than stopping at the first page.

After the page analysis finishes, Scraper Studio presents the proposed extraction fields as below:

Review the fields and click Approve.

Now, we have our first functional scraper without writing the extraction logic ourselves. But we’re not going to treat the generated scraper as a black box.

The exciting part starts when we open the generated code.

Step 3: Understanding the Scraper Studio Generated Workflow

After approval, Scraper Studio opens the scraper inside its workspace editor.

The generated JavaScript is divided into two important parts:

Interaction code—controls navigation and crawling behavior
Parser code—extracts structured information from the current page

Understanding this distinction is useful because these two layers solve different problems.

The interaction layer decides where the browser goes. The parser decides what data to extract from the page.

1. Auditing the Interaction Code

The interaction code controls browser navigation, pagination, and the movement between crawling stages. Let's demonstrate key ideas behind this:

FunctionResponsibilitynavigate()Opens the target URL in the browser session.parse()Passes the current page to the parser.rerun_stage()Runs another instance of a stage, useful for distributing pagination work.next_stage()Passes discovered URLs or data into another crawling stage.

The scraper isn’t necessarily limited to one long browser session walking through every page sequentially. Instead, the generated workflow can split work into separate stages.

2. Auditing the Parser Code

When the interaction layer calls parse(), Scraper Studio passes the current page to the parser.

The parser is responsible for turning page content into structured records.

One useful pattern you’ll see in generated parser code is converting matching elements into an array before mapping them into structured results.

For example, using .toArray().map() keeps the transformation in a single expression and produces a normal JavaScript array that can be passed into the next stage.

More importantly, notice what the parser is doing in our workflow. It doesn’t try to extract every property field from the search-result card.

Instead, it first discovers the property URLs. Those URLs can then become inputs to the next crawling stage.

This is precisely the kind of multi-stage workflow we want for our tracker. But our Node.js application doesn’t know how to start this scraper yet.

So the next step is to connect the Scraper Studio collector to our Collector abstraction through its API by replacing the placeholder implementation from the previous section with a real cloud collection request.

11. Connecting Scraper Studio API to Node.js

Unlike traditional scraping frameworks, Scraper Studio wraps the browser automation layer behind two HTTP endpoints:

POST /dca/trigger — Start a new collection job

GET /dca/dataset — Download the completed dataset

11.1 Initiate by API

Open your published scraper in the Scraper Studio workspace and open the Initiate by API tab:

You need two values:

API Token: Your Bright Data API token from Account Settings.
Collector ID: The ID of your published Scraper Studio collector. You get this from your Scraper Studio Dashboard after creating the scraper (it will be an ID starting with c_ inside your scraper settings).

Now open your local .env file and add the values:

BRIGHTDATA_API_TOKEN=""
BRIGHTDATA_COLLECTOR_ID=""

Keep these values private. Never commit your .env file to Git.

11.2 Triggering Scraper Studio API from Node.js

Create an isolated new file for testing scraper studio API; we will later connect with baseCollector:

src/collectors/scraperStudio.js

We will use Node.js’s built-in fetch() to send the request to Scraper Studio.

The function below starts a new collection job and returns the response from Bright Data:

import dotenv from "dotenv";
dotenv.config();
/**
 * Dispatches an asynchronous trigger call to Scraper Studio's cloud queue.
 * @param {string} targetUrl - The target URL to scrape.
 * @returns {Promise<Object>} The asynchronous trigger response.
 */
export async function triggerRemoteScraper(targetUrl) {
  const token = process.env.BRIGHTDATA_API_TOKEN;
  const collectorId = process.env.BRIGHTDATA_COLLECTOR_ID;
  if (!token || !collectorId) {
    throw new Error(
      "Adapter Error: Missing valid Bright Data workspace credentials inside .env file."
    );
  }
  const triggerEndpoint = `https://api.brightdata.com/dca/trigger?collector=${collectorId}&queue_next=1`;
  const parameterPayload = JSON.stringify([    {
      url: targetUrl,
      max_page: 2 // Bound pagination limits during sandbox evaluation passes
    }
  ]);
  console.log(
    `[Cloud Post] Triggering remote automation collection for target: ${targetUrl}`
  );
  const response = await fetch(triggerEndpoint, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${token}`,
      "Content-Type": "application/json"
    },
    body: parameterPayload
  });
  if (!response.ok) {
    throw new Error(
      `Cloud Trigger Rejection: HTTP status code [${response.status}]`
    );
  }
  return await response.json();
}

Notice that we are not launching a browser from Node.js. Our application simply sends the target URL to Scraper Studio. The cloud service takes care of the browser automation and collection job.

The max_page value is also intentionally limited here. During development and testing, this prevents an accidental request from crawling hundreds or thousands of pages.

11.3 Mounting the Express Route

Next, connect the scraper trigger to our Express server.

Open:

src/api/server.js

Replace its contents with:

import express from "express";
import dotenv from "dotenv";
import { triggerRemoteScraper } from "../collectors/scraperStudio.js";
import { processRawIngestion } from "../pipeline/index.js";
dotenv.config();
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
app.get("/health", (req, res) => res.json({ status: "ok" }));
// Asynchronous Trigger Endpoint Route
app.get("/api/collect", async (req, res) => {
  const baselineUrl =
    "https://www.har.com/houston/realestate/for_sale";
  const targetUrl = req.query.url || baselineUrl;
  try {
    const triggerReceipt = await triggerRemoteScraper(targetUrl);
    res.json(triggerReceipt);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});
app.listen(PORT, () =>
  console.log(`Pipeline Server actively running on port: ${PORT}`)
);

The new /api/collect route gives us a simple way to trigger the cloud scraper from our local application. If no URL is provided, it uses our Houston real-estate page as the default.

You could also pass another URL through the query string.

11.4 Verification Pass

Start the development server:

npm run dev

You should see your server’s startup message in the terminal.

Now open a second PowerShell or terminal window and run:

curl "http://127.0.0.1:3000/api/collect"

Your local Express server receives the request and calls the Scraper Studio API. If everything is configured correctly, Bright Data returns a successful trigger response as below:

This confirms that our Node.js application can communicate with Scraper Studio and start a real cloud collection job.

We have now moved from a local mock input to a real remote scraper.

11.5 Open the Runs Tab

After triggering the collector, open your Scraper Studio workspace and select the Runs tab.

You should see the collection job running there:

The Runs page lets you monitor the collection and access the resulting dataset. Scraper Studio provides export formats such as:

JSON
CSV
XLSX
NDJSON

At this point, our Node.js application can successfully start a remote scraping job.

But we still have a problem.

We do not want a human to open Scraper Studio every time, wait for a run to finish, and manually download a file.

The next step is to bring the completed dataset back into our Node.js pipeline programmatically. That will let our application fetch, save, process, and analyze the data dynamically.

12. Building a Texas Real Estate Delta Tracker CLI

Every chapter up to this point built one isolated piece of the system: a stable contract, a pipeline that normalizes and validates anything shaped like { id, value, label }, a snapshot engine that gives the pipeline memory, a comparison engine that turns two snapshots into created / updated / removed, an event classifier that separates market noise from real anomalies, a polymorphic collector factory, and a live Scraper Studio trigger endpoint.

None of that is worth much until it survives contact with a real website.

This chapter is the payoff.

We will build a Texas real estate price tracker for Houston and Dallas cities. It will pull live listings from HAR.com through Scraper Studio and send them through the same pipeline we already built and tested.

The person running the tracker never needs to see a HAR.com URL, paste a property URL, or use the Collection API manually. They simply choose a city from a menu and watch real listings move through the pipeline.

There is no new architecture here. Only the pieces that genuinely need to change for real-world data will change.

Before connecting everything from Node.js, we must customize the scraper’s pagination logic in the real-world extraction workflow. This matters because a real website produces gigantic data than a small mock dataset. You will waste hours/reducing your valuable free credits for testing a single function if you don’t know how to handle pagination.

12.1. How to Handle Pagination when Scraping

Let's look at our generated har.com scraper’s interaction code on scraper studio IDE:

So where’s the actual risk for a repeatable real-world testing workflow? It’s in what the “cap” is actually capping:

const max_page = Math.min(input.max_page || 10, 84); 
// HAR.com has 84 pages max (by default):

At first glance, 84 looks like a safety limit. It is not.

It is simply the current maximum number of pages for this HAR.com search. Math.min(x, 84) prevents a request from asking for page 85, but it still allows a request to crawl all 84 pages.

There is another problem. If max_page is not supplied, the scraper falls back to 10. That sounds small until we look at what happens next.

Every property URL found on a search page becomes its own next_stage() call. In other words, every listing can create another browser job.

If a search page contains roughly 20 listings, the numbers quickly grow:

Unset (Default): 10 pages = 200 results
Set to 50: 50 pages = 1,000 results
Set to 84: 84 pages = 1,680 results

Here is how much time it takes for har.com product page after I limit it to 242 pages. You have to wait 3–8 minutes for testing each request as below:

None of this is a runaway bug. The scraper is doing exactly what we asked it to do. But it is a bad default for a tutorial where we may run the scraper repeatedly while testing snapshots, comparisons, and anomaly detection.

I have also seen much larger crawls in testing. One request produced more than 1,000 pages, while another test with CoinMarketCap reached more than 8,000 pages. That is a lot of browser work for a simple test.

Bright Data’s batch scraper concurrency can reach up to 100 concurrent jobs per scraper. Once that limit is exceeded, the API can return:

Maximum limit of 100 jobs per scraper has been exceeded. Please reduce the number of parallel jobs.

So the issue is not that our scraper is broken. The issue is that the default is too large for a repeatable demo.

Put an explicit cap on the scraper:

To fix that long-waiting scraping job for each test, we want a small and predictable run for local testing.

We will:

Limit listing pagination to 2 pages so it runs in seconds
Limit property-detail crawling to 5 listings per page
Prevent property-detail pages from starting another pagination loop

Therefore, rewrite the “interaction code” in the Scraper Studio IDE as follows:

const url = new URL(input.url);
// Guard: If this is an isolated property detail page, navigate, parse, and exit immediately.
// This prevents detail page crawls from recursively triggering search directory loops.
if (input.is_detail) {
    navigate(url.href);
    collect(parse());
    return;
}
// Otherwise, we are on the main search directory page
navigate(url.href);
const { property_urls } = parse();
// 💡 EXPLICIT PAGINATION CAP: Limit listing discovery to exactly 2 pages max
if (!input.is_rerun) {
    const max_page = 2; // Hard cap listing crawl page depth

    for (let page = 2; page <= max_page; page++) {
        const next_page_url = new URL(input.url);
        next_page_url.searchParams.set('page', page.toString());

        rerun_stage({
            url: next_page_url.href,
            is_rerun: true
        });
    }
}
// 💡 EXPLICIT CRAWL CAP: Limit deep property-detail crawls to exactly 5 listings per page
const target_listings = (property_urls || []).slice(0, 5);
for (let property_url of target_listings) {
    next_stage({
        url: property_url,
        is_detail: true // Flag to activate our detail-page guard in the next stage
    });
}

With that customized interaction code, you’ll see each API request takes only 30 seconds on average in IDE “Recent Runs” tab as below:

You may still see more pages than expected in some runs. In my case, the scraper returned 12 pages even though the interaction code was capped at 2. The parent worker spawned the two intended pages, but some sub-workers also discovered pagination links inside the site’s footer.

This is a useful reminder: a crawl limit in one part of a scraper does not automatically prevent a website’s own links from creating additional work.

After changing the interaction code, save it and click Save to production, then click Continue.

Note: Scraper Studio keeps the draft and production/development versions separate. Our Node.js specifically trigger runs the published production version, not an unsaved draft.

Always sync the production version after manually changing the scraper code. You can verify the production update from the Changelog tab in the top-right corner:

12.2. Delivery Preferences and Calling the Collection API

Open Delivery preferences in the scraper dashboard.

You will see several choices:

For this tracker, we want the complete dataset after the collection finishes.

So we use:

On a job completion (batch)
JSON
API download

That configuration is necessary because it determines how our Node.js collector gets the data.

Our flow is:

POST /dca/trigger
        ↓
{ collection_id }
        ↓
GET /dca/dataset?id=...
        ↓
[ records ]

The dataset becomes available after the batch job finishes. If you choose a webhook instead, the architecture changes. Scraper Studio would push the results to your application instead of your application polling for them.

For this guide, we will keep API download.

12.3. Waiting for the Job Without Timing Out

Our trigger endpoint from Chapter 10 starts a collection and returns its receipt. That proved our connection worked.

But a real tracker needs to wait for the completed dataset.

Bright Data’s Collection API returns a status object while the dataset is still being built and returns the completed records when the job is finished. That means our collector needs to poll the dataset endpoint until the result is ready.

There are also two different timeout problems we need to keep separate:

HTTP request timeout: How long one network request is allowed to hang.
Collection timeout: How long the entire scraping job is allowed to run.

A single network request should fail relatively quickly. A real browser crawl, however, may legitimately take several minutes.

So a slow or broken network request should be retried, while a collection that never finishes should eventually stop.

Open: src/collectors/ScraperStudioCollector.js

and update it with this below:

// src/collectors/ScraperStudioCollector.js
import { BaseCollector } from "./BaseCollector.js";
const TRIGGER_ENDPOINT = "https://api.brightdata.com/dca/trigger";
const DATASET_ENDPOINT = "https://api.brightdata.com/dca/dataset";
const POLL_INTERVAL_MS = 5000;              // how often we check in
const REQUEST_TIMEOUT_MS = 10000;           // how long ONE http call may hang
const MAX_COLLECTION_WAIT_MS = 20 * 60 * 1000; // how long the WHOLE job may take
function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}
export class ScraperStudioCollector extends BaseCollector {
  constructor() {
    super();
    this.token = process.env.BRIGHTDATA_API_TOKEN;
    this.collectorId = process.env.BRIGHTDATA_COLLECTOR_ID;
    if (!this.token || !this.collectorId) {
      throw new Error("Missing BRIGHTDATA_API_TOKEN or BRIGHTDATA_COLLECTOR_ID in your .env file.");
    }
  }
  /**
   * Fires one HTTP request with its own short timeout. This protects us
   * against a dead socket — it has nothing to do with how long the
   * overall scraping job is allowed to run.
   */
  async #request(url, options = {}) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
    try {
      return await fetch(url, { ...options, signal: controller.signal });
    } finally {
      clearTimeout(timer);
    }
  }
  async collect(url) {
    const collectionId = await this.#trigger(url);
    return this.#waitForDataset(collectionId);
  }
  async #trigger(targetUrl) {
    const triggerUrl = `TRIGGERENDPOINT?collector={this.collectorId}&queue_next=1`;
    const response = await this.#request(triggerUrl, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${this.token}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify([{ url: targetUrl, max_page: 2 }])
    });
    if ([401, 403, 407].includes(response.status)) {
      throw new Error(`Scraper Studio rejected the request credentials [HTTP ${response.status}]. Check BRIGHTDATA_API_TOKEN.`);
    }
    if (!response.ok) {
      throw new Error(`Scraper Studio trigger failed [HTTP ${response.status}]`);
    }
    const body = await response.json();
    const collectionId = body.collection_id || body.id;
    if (!collectionId) {
      throw new Error("Trigger response did not include a collection id.");
    }
    console.log(`[Scraper Studio] Job triggered. Collection ID: ${collectionId}`);
    return collectionId;
  }
  async #waitForDataset(collectionId) {
    const datasetUrl = `${DATASET_ENDPOINT}?id=${collectionId}`;
    const startedAt = Date.now();
    let lastStatus = "INITIALIZING";
    while (Date.now() - startedAt < MAX_COLLECTION_WAIT_MS) {
      const elapsedSec = Math.floor((Date.now() - startedAt) / 1000);
      process.stdout.write(`\r[Scraper Studio] ${lastStatus.padEnd(12)} elapsed: ${elapsedSec}s   `);
      let response;
      try {
        response = await this.#request(datasetUrl, {
          headers: { Authorization: `Bearer ${this.token}` }
        });
      } catch (requestError) {
        // A single request timing out or dropping is a network hiccup,
        // not a job failure — try again on the next polling pass.
        await sleep(POLL_INTERVAL_MS);
        continue;
      }
      // Auth problems are a request problem, not a "still running" state.
      // Fail immediately instead of burning the 20-minute budget on a
      // token that will never work.
      if ([401, 403, 407].includes(response.status)) {
        process.stdout.write("\n");
        throw new Error(`Scraper Studio rejected the dataset request [HTTP ${response.status}]. Check BRIGHTDATA_API_TOKEN.`);
      }
      // A brief 404 right after triggering (before the snapshot id has
      // fully registered) is normal and self-resolves within a poll or
      // two. Any other 4xx means something about the request is wrong
      // and won't fix itself by waiting, so we fail fast on those.
      if (response.status === 404 && elapsedSec < 30) {
        lastStatus = "INITIALIZING";
        await sleep(POLL_INTERVAL_MS);
        continue;
      }
      if (response.status >= 400 && response.status < 500) {
        process.stdout.write("\n");
        throw new Error(`Scraper Studio rejected the dataset request [HTTP ${response.status}]`);
      }
      // 5xx and anything else non-2xx is treated as transient — the job
      // itself is still fine, only this one poll failed.
      if (!response.ok) {
        lastStatus = "RETRYING";
        await sleep(POLL_INTERVAL_MS);
        continue;
      }
      const body = await response.json();
      // Bright Data's documented contract: a plain JSON array means the
      // snapshot is finished. Anything else — e.g. { status: "building" }
      // — means the job is still running.
      if (Array.isArray(body)) {
        process.stdout.write("\n");
        console.log(`[Scraper Studio] Collection complete. Records received: ${body.length}`);
        return body;
      }
      lastStatus = body.status ? String(body.status).toUpperCase() : "BUILDING";
      await sleep(POLL_INTERVAL_MS);
    }
    process.stdout.write("\n");
    throw new Error(`Scraper Studio job ${collectionId} did not finish within the 20-minute wait budget.`);
  }
}

Here is what each part protects us from:

Our scraper implements a fail-fast polling architecture designed to handle diverse network conditions. It uses AbortController timeouts and strict HTTP error handling to fail immediately on unrecoverable 4xx client errors, while safely retrying transient 5xx server errors and tolerating brief 404 windows for delayed datasets. Finally, strict ceilings like MAX_COLLECTION_WAIT_MS prevent infinite hangs, while a live status line ensures full operational visibility.

The core distinction is that one request timing out does not fail the collection. A collection that never finishes does.

Bright Data’s quickstart uses a shorter overall wait for simpler jobs. We use 20 minutes because we may scrape thousands of pages using our pipeline in a production app in the future, and the scraper will be doing more work.

12.4. Building the Texas Tracker’s Data Layer

Now we can build the part that is specific to our Texas real estate model.

The collector gives us raw HAR.com records. Our pipeline, however, expects the stable contract:

id
value
label
...metadata

So we need an adapter between the two.

First, let’s look at a live JSON record returned by Scraper Studio:

{
  "price": { "value": 305000, "currency": "USD", "symbol": "$" },
  "address": "5238 Kylie Springs Ln, Houston, TX 77066",
  "bedrooms": 4,
  "bathrooms_full": 2,
  "bathrooms_half": 1,
  "square_feet": 2479,
  "lot_size": 10724,
  "property_type": "Single-Family",
  "mls_number": "53228136",
  "listing_status": "For Sale",
  "product_page_url": "https://www.har.com/homedetail/5238-kylie-springs-ln-houston-tx-77066/3690464",
  "input": { "url": "https://www.har.com/houston/realestate/for_sale", "max_page": 2 }
}

Two fields worth noticing before we map anything:

1. **price** is an object

The price is not simply: 305000

Instead, Scraper Studio returns:

price.value
price.currency
price.symbol

We only need price.value for the main pipeline value.

2. **input** is request metadata

The input object tells us which URL and settings produced the record.

That is useful while debugging. But it is not part of the property itself.

Our adapter will ignore it.

Open:

src/pipeline/adapters.js

and replace its contents:

// src/pipeline/adapters.js
/**
 * Texas city registry. Each entry pairs a simple selection key with the
 * real HAR.com search URL our published Scraper Studio scraper targets.
 * The person running the tracker never sees or types this URL — they
 * just pick a city from a menu.
 */
export const cityRegistry = {
  1: { key: "houston", name: "Houston", url: "https://www.har.com/houston/realestate/for_sale" },
  2: { key: "dallas", name: "Dallas", url: "https://www.har.com/dallas/realestate/for_sale" }
};
/**
 * Universal Dataset Adapter.
 * Converts HAR.com's raw field names into our stable pipeline contract:
 * id, value, label, metadata. This is the ONLY file that knows HAR.com's
 * field names — normalize.js, validate.js, and compare.js never do.
 * Note that `input` (the echoed request) is deliberately never read here.
 */
export class DatasetAdapter {
  /**
   * @param {Array<Object>} rawData - Raw records from the Scraper Studio collector.
   * @param {string} city - The city key ("houston" | "dallas") for tagging.
   */
  static transform(rawData, city) {
    if (!Array.isArray(rawData)) return [];
    return rawData.map((item) => ({
      id: item.mls_number ? String(item.mls_number) : null,
      value: item.price ? item.price.value : null,
      label: item.address ? String(item.address).trim() : "Untitled Listing",
      metadata: {
        city,
        currency: item.price ? item.price.currency : "USD",
        currency_symbol: item.price ? item.price.symbol : "$",
        bedrooms: toInt(item.bedrooms),
        bathrooms_full: toInt(item.bathrooms_full),
        bathrooms_half: toInt(item.bathrooms_half),
        square_feet: toInt(item.square_feet),
        lot_size: toInt(item.lot_size),
        property_type: item.property_type ?? null,
        listing_status: item.listing_status ?? null,
        product_page_url: item.product_page_url ?? null
      }
    }));
  }
}
function toInt(value) {
  const number = parseInt(String(value ?? "").replace(/[^0-9]/g, ""), 10);
  return Number.isFinite(number) ? number : null;
}

The adapter isolates HAR.com from the rest of our system.

The rest of the pipeline does not need to know that HAR.com calls the identifier mls_number.

It only sees:

id
value
label
metadata

Choosing a stable ID:

Our id comes from the MLS number rather than the address. That addresses can change formatting between crawls. An address might appear slightly differently while still referring to the same listing.

The MLS number gives us a much more stable identity for comparison.

If our identity key changes between runs, the comparison engine could treat one listing as two different records: one removed and another created.

Keeping website-specific fields in metadata:

Everything specific to HAR.com goes inside metadata. Bedrooms, bathrooms, square footage, property type, and listing status do not become part of the core pipeline contract.

That means normalize.js, validate.js, and compare.js remain completely independent of HAR.com.

This is the abstraction we designed earlier finally working with real data.

Preserving metadata during normalization

There is one small generic change we need in the pipeline. Our current normalize.js carries the core fields forward, but we also want to preserve the optional metadata bag.

Open:

src/pipeline/normalize.js

and add the metadata field to normalizeRecord:

export function normalizeRecord(record, index) {
  return {
    id: record.id ?? `entity-${index + 1}`,
    value: toNumber(record.value),
    label: typeof record.label === "string" ? record.label.replace(/\s+/g, " ").trim() : "Untitled Entry",
    metadata: record.metadata && typeof record.metadata === "object" ? record.metadata : {},
    captured_at: new Date().toISOString()
  };
}

That’s the only change to the pipeline core in this chapter.

The change is generic. Any future adapter can attach metadata without teaching normalize.js anything about the source.

Filtering by bedrooms:

The bedroom filter belongs to the application, not the pipeline. So it runs after normalization, validation, and deduplication.

Add this helper to src/pipeline/adapters.js:

/**
 * Application-level filter. Runs strictly after the pipeline core has
 * already normalized/validated/deduplicated the records.
 */
export function filterByBedrooms(records, bedrooms) {
  if (!bedrooms || bedrooms === "any") return records;
  const target = Number(bedrooms);
  return records.filter((record) => record.metadata.bedrooms === target);
}

Now the pipeline answers:

Is this data valid?

The application answers:

Which valid records do I want to see?

Those are different responsibilities.

City-specific snapshots

There is one more problem to solve before we build the CLI. Our snapshot system currently stores one file:

latest_snapshot.json

That works for one dataset. It does not work for two cities. If we run Dallas after Houston and both cities use the same snapshot, the Dallas listings would be compared against Houston’s previous data.

Every Dallas listing could appear to be CREATED, while Houston listings could appear to be REMOVED.

We need one snapshot per city.

Open:

src/pipeline/snapshot.js

and replace it with:

// src/pipeline/snapshot.js
import fs from "node:fs/promises";
import path from "node:path";
const snapshotDir = path.join(process.cwd(), "src", "storage", "snapshots");
function snapshotFileFor(city) {
  return path.join(snapshotDir, `snapshot_${city}.json`);
}
export async function saveSnapshot(city, records) {
  try {
    await fs.mkdir(snapshotDir, { recursive: true });
    await fs.writeFile(snapshotFileFor(city), JSON.stringify(records, null, 2), "utf8");
    return true;
  } catch (err) {
    console.error(`[Snapshot Write Failure] ${city}: ${err.message}`);
    return false;
  }
}
export async function loadSnapshot(city) {
  try {
    const rawBuffer = await fs.readFile(snapshotFileFor(city), "utf8");
    return JSON.parse(rawBuffer);
  } catch (notFoundError) {
    // First run for this city — start from an empty baseline.
    return [];
  }
}

The comparison engine still does not know anything about cities. Only the snapshot layer and the caller know which city is being processed.

Now Houston gets:

snapshot_houston.json

and Dallas gets:

snapshot_dallas.json

Each city now has its own memory.

12.5. Assembling an Interactive Tracker

Create:

src/cli/trackTexas.js

This is our terminal client. Notice how little logic it owns. It asks the user for a city and bedroom filter, then connects the pieces we already built:

CLI
 ↓
Collector
 ↓
Adapter
 ↓
Pipeline
 ↓
Filter
 ↓
Snapshot
 ↓
Compare
 ↓
Classify

The person running the tracker never needs to enter a HAR.com URL or know the collector ID. They simply choose a city and a bedroom filter:

// src/cli/trackTexas.js
import readline from "node:readline/promises";
import dotenv from "dotenv";
import { CollectorFactory } from "../collectors/CollectorFactory.js";
import { cityRegistry, DatasetAdapter, filterByBedrooms } from "../pipeline/adapters.js";
import { processRawIngestion } from "../pipeline/index.js";
import { loadSnapshot, saveSnapshot } from "../pipeline/snapshot.js";
import { computeHistoricalDelta } from "../pipeline/compare.js";
import { classifySystemEvents } from "../pipeline/classify.js";
dotenv.config();
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
function formatMoney(amount) {
  return `$${Number(amount).toLocaleString()}`;
}
function movementIcon(percentChange) {
  if (percentChange > 0) return "📈";
  if (percentChange < 0) return "📉";
  return "➖";
}
async function main() {
  console.log("======================================================================");
  console.log("                     TEXAS REAL ESTATE PRICE COMPARISON  TRACKER               ");
  console.log("======================================================================");
  console.log("1) Houston");
  console.log("2) Dallas");
  const cityChoice = await rl.question("Select a city (1-2): ");
  const city = cityRegistry[cityChoice.trim()];
  if (!city) {
    console.error("Invalid selection.");
    rl.close();
    process.exit(1);
  }
  console.log("\nBedroom filter:");
  console.log("1) 2 bedrooms");
  console.log("2) 3 bedrooms");
  console.log("3) Any");
  const bedroomChoice = await rl.question("Select a filter (1-3): ");
  const bedrooms = { 1: 2, 2: 3, 3: "any" }[bedroomChoice.trim()] ?? "any";
  console.log(`\n[Tracker] City: ${city.name}`);
  console.log(`[Tracker] Bedroom filter: ${bedrooms}`);
  try {
    const baseline = await loadSnapshot(city.key);
    console.log(`[Tracker] Loaded ${baseline.length} record(s) from ${city.key}'s previous snapshot.`);
    const provider = process.env.COLLECTOR_PROVIDER || "SCRAPER_STUDIO";
    const collector = CollectorFactory.create(provider);
    const rawData = await collector.collect(city.url);
    console.log(`[Tracker] Raw records received: ${rawData.length}`);
    const standardized = DatasetAdapter.transform(rawData, city.key);
    const processed = processRawIngestion(standardized);
    console.log("[Tracker] Pipeline telemetry:", processed.telemetry);
    const filtered = filterByBedrooms(processed.data, bedrooms);
    console.log(`[Tracker] Records after bedroom filter: ${filtered.length}`);
    if (filtered.length === 0) {
      console.warn("[Tracker] Nothing left after filtering. Nothing to compare or save.");
      return;
    }
    const deltas = computeHistoricalDelta(baseline, filtered);
    const events = classifySystemEvents(deltas);
    console.log("\n----------------------------------------------------------------------");
    console.log(`   ${city.name.toUpperCase()} — FIRST ${Math.min(10, filtered.length)} RECORDS`);
    console.log("----------------------------------------------------------------------");
    for (const record of filtered.slice(0, 10)) {
      const previous = baseline.find((item) => item.id === record.id);
      console.log(`  [${record.id}] ${record.label}`);
      console.log(`    Price: ${formatMoney(record.value)}  |  Beds: ${record.metadata.bedrooms ?? "?"}  |  Status: ${record.metadata.listing_status ?? "unknown"}`);
      if (previous) {
        const change = record.value - previous.value;
        const percent = previous.value ? (change / previous.value) * 100 : 0;
        console.log(`    Previous: ${formatMoney(previous.value)}  ${movementIcon(percent)} change>=0?"+":""{formatMoney(change)} (${percent.toFixed(2)}%)`);
      } else {
        console.log("    Previous: [no baseline yet]");
      }
    }
    console.log("\n----------------------------------------------------------------------");
    console.log(`   CREATED: ${deltas.created.length}  |  UPDATED: ${deltas.updated.length}  |  REMOVED: ${deltas.removed.length}`);
    console.log("----------------------------------------------------------------------");
    for (const event of events) {
      console.log(`  [${event.severity}] ${event.type}: ${event.message}`);
    }
    const merged = new Map(baseline.map((item) => [item.id, item]));
    for (const record of filtered) {
      merged.set(record.id, record);
    }
    await saveSnapshot(city.key, [...merged.values()]);
    console.log(`\n[Tracker] Snapshot saved: snapshot_${city.key}.json`);
  } catch (error) {
    console.error(`\n[Tracker Error] ${error.message}`);
  } finally {
    rl.close();
  }
}
main();

Add the CLI command to package.json:

"scripts": {
  "track": "node src/cli/trackTexas.js"
}

Now run:

npm run track

The terminal itself is not the interesting part. It is only a thin client that connects the components we already built.

The same sequence could later run behind:

An Express API
A scheduled job
A background worker
An AI agent tool call

The underlying pipeline would not need to change.

12.6. Final Testing Workflow

Now we can test the complete system with real data.

Test 1: Real Data Baseline

Run:

npm run track

From the terminal user query box: Select 1) Houston

And then in the next query for bedroom filter, I have chosen 3 (Any) to watch the real flow happen end to end.

For a normal repeatable test, keep the scraper bounded to the two-page configuration from Section 11.1. If you want a larger real-world dataset like m and are willing to wait longer, you can increase the crawl size for that specific run (e.g., max_page = 10).

The first run has no previous snapshot. That means every valid listing should show:

Previous: [no baseline yet]

The validator may also quarantine records that do not contain the required fields, such as a usable price or MLS number.

That is exactly what we want. The same validation rules that worked against our mock data are now working against real listings.

After the run, the tracker writes:

snapshot_houston.json

Scraper Studio will also show the corresponding collection in its Runs tab:

Test 2: Real Historical Price Comparison

Now we can test whether the comparison engine detects a price change.

Open src/storage/snapshots/snapshot_houston.json and locate a live listing to use as our test case. For instance, I have chosen 7942 Hammerly Blvd, Houston, TX 77055, which was recently stored by our pipeline’s latest scraper run:

Live price right now: $439900
Local snapshot (manually edited for testing): $539900

Because real estate prices change slowly over weeks or months, we can’t just wait around for a real-world market drop to test our code. Instead, we will manually edit our local snapshot data to replicate a live price swing:

By artificially inflating our historical data, we force a mismatch. When the scraper runs next, it will perceive the actual market price as a sudden drop, allowing us to instantly verify that our comparison engine catches real-world fluctuations correctly.

Run the tracker again against Houston:

npm run track

The live value is lower than our edited historical value, so the comparison engine detects a price drop.

In my test, the result was -18.52% price drop:

That is below the 30% critical anomaly threshold while still exceeding the 15% major threshold discussed in earlier chapter, so it is classified according to the rules we already defined there.

See? we did not create a second comparison system for real estate. The exact same compare.js and classify.js code is now analyzing a real listing.

Test 3: New and Removed Listings

Edit snapshot_houston.json again.

This time:

Delete a few real listings from the snapshot.
Add a fake listing with an ID that will not appear in the live crawl, such as "81".

Now Run: npm run track

The listing you deleted from the snapshot shows up as CREATED (it’s new relative to your edited baseline), and the fake listing you added shows up as DELETED (it’s absent from the live crawl). This is the exact compare.js logic from Chapter 8, it has no idea it’s looking at real estate, it’s still just matching id keys through two Map objects.

Test 4: Bad Data Protection

Open the snapshot again. Give one listing an obviously misleading value, such as:

"value": 1

This mimics a scraper accidentally extracting a stray number instead of the actual property price. Run the tracker again.

You should see an anomaly similar to:

[CRITICAL] ANOMALOUS_VALUE_COLLAPSE: 5238 Kylie Springs Ln, Houston, TX 77066 (ID: 53228136) shifted by -99.99%.

The value is technically a valid number, so it can pass basic validation. But the comparison engine sees an extreme shift and the classifier flags it as critical. This is the reason why validation and anomaly detection are separate steps.

A number can be valid without being trustworthy.

Test 5: Multiple Cities

Run the tracker again and select: 2) Dallas

Complete the sync. Now check:

src/storage/snapshots/

You should have two separate files:

snapshot_houston.json
snapshot_dallas.json

Open both files. Houston and Dallas should remain separate. Running Dallas should not cause Houston listings to appear as REMOVED. Running Houston should not cause Dallas listings to appear as CREATED.

That is the small change we made to snapshot.js doing its job.

Each city now has its own memory. The terminal was the visible part of these five tests, but it was never the important part. The important part was everything underneath it: the contract, the pipeline, the memory, the comparison engine, and the event classifier. We built those pieces before we had any real estate data.

This chapter proved that the architecture survives contact with a real website. Let’s save this checkpoint:

git add . && git commit -m "feat: Texas real estate valuation delta tracker, bounded pagination, city-scoped snapshots memory"

Next, we will look at what happens when HAR.com changes its layout underneath us.

More importantly, we will see how Scraper Studio’s Self-Healing can repair a broken extraction without us touching normalize.js, compare.js, or the rest of our pipeline core.

13. Scraper Studio Self-Healing in Production

If you ran the Texas real-estate tracker in the previous chapter and all records looked healthy, great. The scraper is working against the live HAR.com site. But websites change.

A developer might rename a CSS class or change the page structure. When that happens, a traditional scraper can fail silently. Instead of throwing an error, it may return incomplete values.

We cannot wait for HAR.com to change its website just to test this. So we will break our scraper ourselves.

In this chapter, we will:

Set up the Bright Data CLI.
Intentionally break a scraper selector in Scraper Studio.
Run our Node.js tracker and let validation catch the bad data.
Use the Bright Data CLI to heal the scraper.
Review the proposed fix before deploying it.
Run the tracker again locally and verify that the data is healthy.

13.1. Installing the Bright Data CLI

The Bright Data CLI is available as the [@brightdata/cli](https://docs.brightdata.com/products/cli/installation) npm package.

You can install it globally:

npm install -g @brightdata/cli

You can also run it without a permanent installation using npx. Plus its fastest way to run the Bright Data CLI, which runs the latest version with no global install:

npx -p @brightdata/cli brightdata --version

For this chapter, I will use the bdata command after installing the CLI globally. Check that it works:

bdata --version

Logging In: Authenticate the CLI with:

bdata login

This opens a browser for authentication. After login, the CLI stores the credentials locally and checks for the required Bright Data zones. If the cli_unlocker and cli_browser zones do not exist, the CLI can create them automatically.

You do not need to copy an API key into your project just to use the CLI.

Once the login is complete, the terminal is ready to manage your Bright Data scrapers.

13.2. Deliberately Breaking the Scraper

Now let’s imitate a website redesign. A live website might change a selector from:

h2.font_size--large_extra_extra.color_carbon

to a completely different one. Our scraper could then keep running while returning missing values.

Step 1: Open the Web IDE

Open your Bright Data dashboard and go to our previous har.com scraper dashboard. In the left sidebar, open Parser Code. Find the line where the scraper extracts the price, address, or bedrooms. Let’s corrupt the price selector:

Step 2: Break the Selector

Change the working selector:

const price = extractPrice(
  'h2.font_size--large_extra_extra.color_carbon'
);

to a selector that does not exist:

const price = extractPrice(
  'div.completely-broken-price-class-that-does-not-exist'
);

Step 3: Publish the Broken Version and Test

Click the dropdown next to Save and choose Save to production. The active scraper now contains our broken selector. Delete all our local snapshot files that we generated earlier.

Run the tracker again: npm run track

Because the website elements no longer match our corrupted selector, the scraper returns null for bedrooms. Watch your terminal:

This is where our pipeline protects us. Our validate.js validation layer sees that required values are missing and prevents those records from entering the local data storage (failed_compliance: 10) and quarantined the dirty data safely.

The pipeline successfully protected our database from corruption.

Now we need to repair the extraction layer.

13.3. Ask Bright Data to Heal the Scraper

You have two choices to heal a broken scraper: you can use the web console “Self-Healing”, or you can use their terminal CLI.

While the web console works well for edits, the CLI is the superior tool for AI coding agents (Claude Code, Cursor, or Codex). I will intentionally use CLI to help you better understand both IDE (UI) and terminal self-healing workflow.

Run the healing command with your Collector ID and a clear description of the problem:

bdata scraper heal c_mssne7uc11iej44z4s \
  "The price selector is broken and returns null. Re-capture the price from the property details section." \
  --url https://www.har.com

The CLI sends the healing request to Bright Data’s scraper infrastructure. The AI agent analyzes the scraper and the target page, then prepares a new version of the scraper.

Review the Proposed Fix:

The healing process stops at an approval gate. You should see a status similar to:

Status: awaiting_approval

Refresh your Scraper Studio web IDE and you’ll see the this popup window:

Check that the price is being extracted from the correct part of the page. Do not approve a fix just because the command succeeded; make sure it returns the right value.

Approve or Reject:

If the proposed change looks correct, approve it using the CLI workflow.

bdata scraper approve c_mssne7uc11iej44z4s

If you decide that the proposed fix is wrong, reject it instead:

bdata scraper approve c_mssne7uc11iej44z4s --reject

Rejecting the fix leaves the existing scraper unchanged. You can then run heal again with clearer instructions.

One useful detail is that the Collector ID stays the same after a successful healing workflow. Your Node.js application therefore does not need a new collector ID just because the scraper implementation changed.

Verifying the Fix:

The scraper is repaired. Now run the tracker again: npm run track

The same Node.js pipeline receives data from the same collector. But this time, the required fields are populated again:

Our ScraperStudioCollector.js calls the same endpoints and Collector ID as before. The scraper now outputs clean, populated data, the compliance failure count drops back to zero, and the new real estate listings flow into your local snapshots seamlessly:

That is the main benefit of the architecture we built throughout this guide. The scraper can be repaired at the infrastructure layer while the Node.js pipeline continues working with the same stable data contract.

14. Deployment

Our pipeline is now a modular Node.js application. Moving it to a cloud server is the next step. The deployment steps depend on your hosting provider, but the process is usually:

Push your project to a GitHub repository.
Connect the repository to a Node.js hosting platform such as Render, Railway, or AWS.
Add your environment variables through the platform’s environment-variable settings.

Once deployed, your Express application can expose the same API routes we tested locally. An AI agent or app can then call the API without knowing implementation details.

Ultimately, Scraper Studio handles the website. Node.js pipeline handles the data. Snapshots store historical state. Delta comparison detects changes. The API makes the result reusable.

That is the final framework we set out to build.

15. Scaling and Production

To be honest, it is incredibly tempting to keep writing to set up managed databases, implement RAG workflows, configure cloud queues, or build a frontend. But doing so would violate the very principles of clean software design we set out to learn. Our two-layer pipline is already complete.

We don’t need to build all of that to prove the design works.

Instead, let’s look at three production pitfalls you should think about when moving this example toward production.

1: The Temporary Cloud Filesystem

Some hosting environments use temporary filesystems. A restart or new deployment can remove files created by the running application.

If the snapshot disappears, the pipeline loses its historical baseline.

For a simple single-instance deployment, your hosting provider may offer persistent disk or volume storage. You can then point the snapshot directory to that location.

For example:

const snapshotDir =
  process.env.SNAPSHOT_DIR ||
  path.join(process.cwd(), "src", "storage", "snapshots");

For a larger application with multiple instances or concurrent writers, a database becomes a better choice. You could replace the file-based implementation behind:

loadSnapshot()
saveSnapshot()

with PostgreSQL, Redis, or another persistent store.

The rest of the pipeline does not need to know where the snapshot is stored. That is one of the benefits of keeping storage behind a small interface.

2: Time-Based Data Decay

Our snapshot keeps the latest version of each listing. But a listing that disappears from the source can remain in the snapshot if we never remove it.

Over many runs, that can leave old records in the dataset. One simple solution is time-based pruning.

Our normalized records already contain captured_at, so we can use that timestamp to remove records that have not been seen recently.

For example:

export function pruneStaleRecords(records, maxAgeDays = 30) {
  const cutoff =
    Date.now() -
    maxAgeDays * 24 * 60 * 60 * 1000;
return records.filter((record) => {
    const seenAt = new Date(record.captured_at).getTime();
    return Number.isFinite(seenAt) && seenAt >= cutoff;
  });
}

The 30-day value is only an example. Your application might use 7 days, 30 days, 90 days, or another rule depending on the data.

There is also an important rule here: Prune the complete validated dataset, not a filtered view.

For instance, if the user runs the tracker with a two-bedroom filter, a three-bedroom listing may not appear in that run. Historical storage should therefore operate on the full dataset before application-level filters are applied.

3: Long-Running HTTP Requests

A large crawl may take longer than the HTTP timeout allowed by your hosting platform. The exact timeout depends on the platform and the type of service you are using.

That means a request such as:

POST /api/tracker/houston/sync

should not always be responsible for waiting until a large collection finishes. A better production design is to separate starting a job from checking its result.

The application can:

Start the collection job.
Receive a job or snapshot identifier.
Return control to the caller.
Check the job status later.
Process the final dataset when the collection is complete.

This is commonly called asynchronous job processing or polling.

It becomes especially useful when your scraper needs to process multiple pages. For the small project in this guide, a synchronous flow is easier to understand. For larger crawls, asynchronous processing is the natural next step.

16. What You Can Build Next

We kept this pipeline small on purpose. Because it is reusable and reliable, you can easily plug it into large-scale production workflows.

Here are two high-demand popular use cases you can build right now:

1. A Real-Estate RAG Pipeline for AI Agents

Static documents make AI real estate analysts obsolete fast. You can use this pipeline to feed an LLM real-time data instead.

How to build it: Set up a daily cron job (like Render Cron) to hit your /api/track/houston endpoint.
The Flow: The pipeline pulls live listings, normalizes the data, and flags major changes (like a 15%+ price drop). The server converts these updates into vector embeddings and saves them to Pinecone or Chroma.
The Result: Your RAG system gets verified, daily market shifts. Your AI analyst can spot live investment opportunities without any manual data entry.

2. Capped-Budget Market Analyzers

Running web crawlers inside autonomous AI loops is financially risky. A single runaway loop on a dynamic billing platform can cost thousands in surprise server fees.

How to build it: Run this pipeline on a flat-rate model (like Scraper Studio’s $1.50 per 1,000 pages). Cap each run at 2 directory pages.
The Math: Every daily sync costs around $0.003.
The Advantage: Strict pagination boundaries give you a hard safety ceiling. You can calculate and lock down your maximum monthly data budget with absolute certainty.

Conclusion

We started with a basic problem: A scraper can succeed and still give you bad data. Therefore, we built a data pipeline around it. The scraper extracts data. The pipeline makes that data useful.

We first created a stable contract. The pipeline core works with that contract instead of knowing the fields of a specific website.

Then we built the core processing stages: we added snapshots so the application could remember previous data. Then we added comparison and classification. Next, we created the collector factory so the application could work with different extraction providers.

Finally, we tested everything against live Texas real-estate listings from HAR.com. We did not put HAR.com-specific field names into the pipeline core. Instead, the adapter translated HAR.com’s response into our stable contract.

For testing our pipeline, we intentionally broke the live scraper. Our validation layer detected the bad data. Bright Data’s self-healing workflow repaired the scraper. We reviewed the proposed change and deployed it without changing the Node.js pipeline.

That is the main idea of this guide. The terminal tracker itself is not the most notable part. It is ONLY one client of the application. The reusable part is the architecture underneath.

FAQ

Why build a custom pipeline instead of parsing whatever the scraper returns?

Because doing that for every source creates a maintenance problem.

Without a stable contract, every new source introduces different field names and different business logic. With the contract, each source needs an adapter.

The rest of the pipeline can remain the same.

Why Node.js instead of Python?

There is no special requirement to use Node.js. We use it here because Scraper Studio’s interaction and parser code also use JavaScript, so the entire example stays in one language.

The core ideas are not tied to Node.js. The normalization, validation, comparison, and classification logic could be implemented in Python, Go, or another language.

Why use JSON files instead of a database?

For an initial MVP (Minimum Viable Product), using a JSON is the smartest, fastest choice. But as soon as the project goes live for hundreds of users or needs long-term history, you must transition that historical snapshot into a real database.

Is web scraping legal?

The examples and workflows in this guide are about publicly accessible web data. Bright Data does not utilize private accounts, bypass authentication mechanisms, or collect sensitive personal information (PII).

It is important to note that public visibility does not automatically grant a blanket right to collect or reuse data. The exact legal requirements heavily depend on the target website, the nature of the data, your specific jurisdiction, and your intended end-use.

Before deploying any scraper to production systems, developers must thoroughly review the target site’s terms of service, applicable local and international laws, strict data-protection regulations (GDPR or CCPA), and their own internal organizational policies.

Can I use this pipeline other than a single data source?

Yes. That is one of the main reasons we created the collector factory and adapter layer.

For a new source, you need:

A scraper or collector for the source.
An adapter that maps its fields to { id, value, label, metadata }.
A registration in the collector factory.

The generic pipeline does not need to know whether the source contains houses, products, jobs, or another type of data.

How do I run the pipeline on a schedule?

The pipeline does not need to know who started it.

Instead of calling it from the interactive CLI, you can trigger the same sequence from a scheduled job:

CollectorFactory.create()
        ↓
DatasetAdapter.transform()
        ↓
processRawIngestion()
        ↓
computeHistoricalDelta()
        ↓
classifySystemEvents()
        ↓
saveSnapshot()

A cron job, scheduled cloud function, webhook, or another job scheduler can become the trigger. The pipeline itself does not need to change.

11 open-source tools every full-stack dev should know

Programming with Shahan — Thu, 16 Jul 2026 10:00:00 +0000

After years of building JavaScript applications, AI agents, browser automation projects, and technical content, these are the open-source tools I keep installing on every new machine.

Recommended Developer Tool: BugHerd

If you build websites for clients or work in a product team, it’s one of those tools that saves countless hours of back-and-forth communication.

One of the biggest bottlenecks in software projects is collecting clear feedback. Instead of receiving messages like:

“The button over there doesn’t work.” or “Something looks broken on mobile.”

I prefer tools that let clients leave feedback directly on the page. That’s exactly what ⁠BugHerd does.

Developers, designers, QA engineers, and clients can simply click anywhere on a website, leave visual feedback, and automatically create actionable tasks with browser details, screenshots, and page metadata already attached.

Recently they also introduced BugHerd MCP, where AI coding tools like ChatGPT, Claude, Cursor, VS Code, and other MCP compatible assistants to work directly with your BugHerd tasks that make AI-assisted debugging and website QA significantly smoother.

Now let’s move into the open-source tools I install on every new machine.

1. Git

Every developer eventually breaks something.

The question isn't if. It's when.

Git is the reason those mistakes rarely become catastrophes.

Instead of thinking about Git as "version control," think about it as an undo button for your entire project.

With Git you can:

experiment without losing work
create separate feature branches
collaborate with teammates
inspect old versions
recover deleted work
understand who changed what

Without Git? Start over.

With Git?

git checkout main

Problem solved.

2. Visual Studio Code

Even with AI editors like Cursor becoming popular, VS Code remains the foundation of most modern development environments.

VS Code is probably the application I spend more time inside than my browser.

Yes it's "just" a code editor.

But the extension ecosystem turns it into an entire development platform.

My favorite extensions include:

ESLint
Prettier
GitLens
Error Lens
Docker
Thunder Client
Playwright
GitHub Copilot

Together they create an environment where formatting, linting, debugging, testing, and Git management happen without leaving the editor.

3. Wave Terminal

Wave Terminal is one of the most exciting open-source terminals I've used recently.

Unlike traditional terminals, it transforms your command line into an interactive workspace where commands, outputs, notes, AI assistance, and multiple sessions live together.

Why I like it? It makes working inside the terminal feel modern instead of intimidating.

4. Docker

Every developer has heard it.

"It works perfectly on my computer."

Usually five minutes before production explodes.

Docker solves this by packaging your application together with everything it needs to run.

Your runtime. Dependencies. Libraries. Configuration. Operating environment.

Instead of shipping code...

...you ship the entire environment.

You can run it on Windows, macOS, Linux, or a cloud server, it behaves exactly the same.

That consistency alone has probably saved software teams millions of debugging hours worldwide.

Typical workflow

If your goal is to build your local files and launch your entire multi-container stack at once, you only need a single command:

docker compose up --build

5. Playwright

Modern software increasingly interacts with browsers.

Testing. Automation. AI agents. Web scraping. End-to-end validation.

Playwright has become my favorite browser automation framework because it feels incredibly reliable.

Unlike older automation libraries, Playwright supports:

Chromium
Firefox
WebKit

from one API.

A few lines of JavaScript can launch an entire browser.

await page.goto(url);

await page.click(selector);

await page.fill(selector, value);

But here's something many tutorials never explain: Launching a browser isn't the hard part anymore.

The hard part is: getting real, usable data from websites to trust that browser.

Today's websites inspect:

browser fingerprints
TLS handshakes
IP reputation
WebGL fingerprints
cookies
behavioral signals

Long before your automation clicks its first button.

That's why browser automation has become less about writing Playwright scripts...

...and more about browser infrastructure.

This is exactly where platforms like Bright Data become valuable.

Instead of spending weeks bypassing anti-bot protections yourself, Bright Data provides production-ready browser infrastructure, including Browser API, Web Unlocker, AI Scraper Studio, and residential proxy networks. So I can focus on building automation instead of constantly fighting anti-bot systems.

If you're building browser AI agents or production scraping pipelines, it's one of the few paid developer tools I genuinely recommend.

While building AI agents, Bright Data delivers ready-to-use browser infrastructure that lets your Playwright automation focus on the task.

For hobby projects, local Playwright is fantastic.

For production automation, browser infrastructure matters just as much as browser code.

6. Scalar

Scalar is one of the cleanest open-source API platforms available today.

It combines beautiful API documentation, request testing, and SDK generation into a single modern interface.

Why I like it? Its strict OpenAPI compliance, robust CLI/API registry, and smooth CI/CD integration with built-in test generation code for a variety of targets. I love its user interface.

Why should you learn it?

Every backend developer works with APIs. Scalar makes analyzing them enjoyable.

7. Supabase

Supabase has become one of my favorite open-source backend platforms.

It combines PostgreSQL, authentication, storage, edge functions, realtime features, and APIs into one developer-friendly stack.

Why I like it? You can build production-ready applications without spending days configuring infrastructure.

It's one of the fastest ways to build modern full-stack applications.

8. TensorFlow

TensorFlow is a tool used for teaching computers to learn, like how your brain learns new things.

It’s mostly used for machine learning, which is how computers can learn patterns and make decisions. For example, writing a block of code to teach a computer how to recognize pictures or understand text.

You can create really smart programs that can learn from data.

Why do you need it? If you want to get into AI and make machines that can think a bit like humans, TensorFlow is your go-to tool.

9. Selenium

Selenium tests websites for you. If you build a website and want to make sure all the buttons work and no errors pop up, you can use Selenium to do the job automatically.

Plus, you don’t have to test everything by yourself; Selenium can do it faster and more accurately. It saves time and helps you find bugs quickly.

10. pnpm

pnpm is the package manager I install before writing my first line of JavaScript.

It installs dependencies faster while using dramatically less disk space than traditional package managers.

Why should you learn it?

Once you switch to pnmp, it's difficult to go back :)

11. DBeaver

DBeaver is my favorite open-source database client.

It supports PostgreSQL, MySQL, SQLite, SQL Server, MongoDB, and dozens of other databases from one interface.

One tool for nearly every database. Why should you learn it?

Sooner or later, every developer needs to inspect data directly.

My Advice to New Developers

One mistake I made early in my career was constantly chasing new frameworks.

Every month there was something "better." Eventually I realized that frameworks change.

Good engineering habits don't.

Learning a new framework might take a weekend. Building a productive development workflow pays dividends for years.

If you're just starting out, I'd spend less time asking:

"What's the hottest framework?"

and more time asking:

"What tools will make me a better engineer every single day?"

That's a much better long-term investment.

Read more: If you're interested in writing cleaner, more maintainable JavaScript, not just shipping code that works, you might also enjoy my book:

📘 Clean Code: Zero to One

Inside, I cover practical clean code principles, design patterns & data structure rules, project organization, reusable patterns, and real-world JavaScript examples that have helped me build production applications over the years.

You can subscribe to my weekly newsletter Horsecoder for latest AI agent building tactics and programming news. You can also read some of my articles from Mindset for Programmers on my LinkedIn profile.

How to Build an Unlocked AI Agent for Browser Automation with Node.js, Bright Data, Gemini, and Playwright

Programming with Shahan — Sun, 05 Jul 2026 06:53:41 +0000

In this full guide, you’ll learn:

📛 Why most AI browser agents fail on modern websites.
🧱 How browser fingerprinting and anti-bot systems work.
⛑️ How to build an AI browser agent using JavaScript (Node.js) that combines Gemini, Playwright, and Bright Data to browse real websites, extract live data, analyze, reason, and generate reports locally without maintaining fragile anti-bot infrastructure ourselves that breaks 5 days later.
🗃️ How to setup Bright Data production-ready browser sessions for AI agent automation without user’s assistance manually.

🪁Introduction

Building unrestricted anonymous browser automation has developed far beyond writing Playwright scripts that click buttons and scrape HTML. Modern websites actively detect automated traffic using browser fingerprints, TLS signatures, IP reputation, and behavioral analysis, making reliable automation significantly more challenging than it was just a few years ago.

Modern AI browser agents don’t usually fail because they’re arbitrary. Their reasoning, prompts, and planning loops are often sophisticated. The execution layer underneath is fragile.

Most tutorials show how to connect an LLM to a browser, execute a few Playwright commands, and declare you’ve built an autonomous agent.

await page.goto(url)
await page.click(selector)
await page.type(selector, text)

In reality, you’ve ONLY automated a browser. Commercial sites don’t gauge how intelligent your agent is. They judge whether they believe your browser is genuine.

Before a page even finishes loading, they inspect what your browser actually is: the TLS handshake, IP reputation, browser fingerprints, canvas and WebGL fingerprints, cookies, device characteristics, and even the rhythm of your connection. Dozens of signals are examined in the time it takes the page to start loading.

If those signals don’t look authentic, your agent rarely reaches the real application. Instead, it encounters CAPTCHA challenges, verification pages, silent redirects, incomplete datasets, or HTTP 200 responses that quietly return entirely different content.

This is one of the biggest challenges in modern browser automation:👇

I’ve always wanted to build an autonomous software system that could function on the web as naturally as a human, not another scraper that extracts a few HTML elements or another automation script that breaks the moment a production website changes.

Like many developers, I started with HTTP requests, graduated to browser automation, and eventually began building AI agents powered by large language models.

Along the way, I realized something surprising: intelligence without reliable web access is meaningless.

Fortunately, solving this problem doesn’t require building your own fingerprinting or anti-detection infrastructure.

Instead, we’ll let Bright Data handle browser identity, network reputation, and anti-bot bypass while our AI agent focuses on reasoning and execution.

Because on today’s web, access is an engineering problem.

🏗️ What We Are Building

By the end of this article, we’ll build a fully functional browser automation AI agent called SIKKI Agent.

Think of it as an unpaid intern who somehow thinks harder at 3 a.m. than I do: it clicks, scrolls, extracts products, grabs ratings, checks prices, and patiently does the repetitive work I’d rather avoid; the most overqualified intern you’ll never have to pay.

The entire project is built with plain JavaScript. No proprietary framework. No billion-dollar AI platform. Just Node.js, Playwright, Bright Data, and Gemini working together.

Here’s a quick overview of what we’re going to build:

🍋‍🟩Note: The complete source code is available on GitHub at the end of this article.

At a high level, the agent follows this workflow:

And this is the engineering architecture we’ll build throughout the tutorial:

🪪 Prerequisite

Basic JavaScript/TypeScript or Python. Even though I used javascript all over this project, you can follow through this article if you know any programming language. But familiarity with asynchronous programming (async/await) is required, as both scraping and AI calls rely heavily on it.
Fundamental Node.js or Python Environment: Ability to install packages via npm or pip and manage local environment variables (.env).
API Keys: Active accounts and API keys for both Google AI Studio (Gemini API) and Bright Data. Both are free to use.
Basic DOM Understanding: A general awareness of HTML structure, though our SIKKI AI agent will minimize the need for complex CSS selectors.

We will use:

Gemini as the reasoning engine
Node.js as the runtime
Playwright as the browser controller
Bright Data Scraping Browser as the cloud browser infrastructure
Memory and retry loops for robustness
Structured outputs and screenshots for observation

Our Sikki agent will:

Accept natural language goals
Plan browser actions dynamically
Connect to a remote cloud browser over CDP (Chrome DevTools Protocol)
Access real websites
Extract structured information
Take screenshots
Recover from failures
Produce clean results by itself

But more importantly

You will understand why browser AI agents fail. Why local Playwright breaks. Why websites detect automation. Why browser identity matters.

And how modern AI agents separate rational thinking from execution to run successfully on the real web.

We are NOT building another toy model. We are building an architecture.

And like all architectures, we begin with the foundations.

🧑‍🍼 Building Sikki Project

In this section, I will show you how to setup Sikki project locally using Gemini, Playwright, and Bright Data and run a quick result in terminal to extract information from CNN website. You can skips this section if you know exactly how to integrate Gemini, Playwright, and Bright Data locally.

Then in the coming section, I will go deeper building the final Sikki agent engine by showing the blueprint and functional implementations step by step.

So lets start with a basic test to see whether our API keys and tech stacks are working properly:

Step 1: ⛏️Create the Project

Open terminal:

mkdir sikki-agent
cd sikki-agent
npm init -y

Enable ES Modules:

npm pkg set type=module

Note: Node.js treats .js files as legacy CommonJS by default. We enable ES Modules to cleanly use modern import statements and native await commands at the root level of our AI agent script

Now install the necessary dependencies:

npm install playwright

npm install @google/genai
npm install dotenv
npm install chalk
npm install ora

Our project’s folder structure should look like this (create these other directories manually):

sikki-agent
 index.js
 .env
 screenshots/
 reports/
 package.json

Step 2: 🗝️ Get Gemini API Key

Go to: https://aistudio.google.com

Create Gemini API Key by creating a new project called “Sikki-Agent”

Inside your .env file, paste your Gemini API key:

GEMINI_API_KEY = your_gemini_api_key

BRIGHT_WS_ENDPOINT = //i will show you later

Step 3: 😶‍🌫️ Setting up Bright Data Cloud Browser Infrastructure

Once you are inside the main console sidebar, do not search for generic proxy settings. Navigate directly to the Web Access API section on the left-hand navigation menu.

Click on the Top right corner “Create API” and choose option labeled Browser API:

then click “continue” and name your API key or leave it default.

Now click “Add API”; you will find your unique WebSocket Connection Endpoint. Copy that first puppeteer/playwright endpoint starting with “wss://brd-customer-hl…” Paste it in your .env file in your local project.

BRIGHT_WS_ENDPOINT = "your_endpoint"

It will look exactly like this perpetual connection string:

This is Because:

We are NOT launching:

chromium.launch()

Instead:

chromium.connectOverCDP()

Meaning: Your browser lives in the cloud.

To help you understand lets write a simple code block to browse CNN’s title from our local terminal using Playwright and Bright Data.

Step 4: 🦧 Connect Playwright to Bright Data

Inside your index.js file write that below to access CNN title from terminal:

import { chromium }
from "playwright";

const BRIGHT_WS_ENDPOINT = "your_endpoint"

const browser =
await chromium.connectOverCDP(BRIGHT_WS_ENDPOINT
);

const page =
await browser.newPage();
await page.goto(
"https://cnn.com"
);

console.log(
await page.title()
);

If you run this on the terminal, you will see a real CNN website title has been shown. We successfully extracted information and displayed the result:

If you now look at your Bright Data event log inside web access panel, you will see successful authentication. That means our BrightData and Playwright are working properly.

So connectOverCDP() does NOT launch a browser locally.

Instead, this is what happened:

This separation is one of the most important notions in today’s browser agents:

Your AI reasons locally. Your browser executes remotely.

Hope that makes sense.

🐦‍⬛ Going Deeper into SIKKI Agent Before Implementation

Wait! in this section I will not show you the actual Sikki codebase. I will explain the arthictecture behind SIKKI and how we will implement it in the next setion. You can skip this section if you want.

Up until now, we have built something that resembles a browser automation script. We opened a cloud browser. We visited a website. We extracted information. And we displayed the results.

That is useful, but it is not intelligence.

The truth is that many developers stop at this stage and call their projects AI agents. They build a browser script, attach an LLM somewhere in the pipeline, and assume they have built something autonomous.

I don’t think that’s enough.🙅‍♂️

An autonomous agent is not defined by the presence of an LLM. It is defined by its ability to observe an environment, reason about the information it receives, decide on actions, execute those actions, and continue this cycle until it reaches its objective.

In other words:

The browser is not the agent. The model is not the agent. The agent is the system that connects both.

This is the architectural philosophy behind SIKKI.

🪛 SIKKI is Not A Scraper

I intentionally avoided calling SIKKI a scraper.

Scrapers collect data. Agents pursue goals.

Suppose I ask SIKKI:

Find the best gaming laptop under $1000.

A traditional scraper might return the following:

ProductPriceAcer Nitro V$899ASUS TUF$949MSI Thin$979

Useful. But not intelligent. An actual agent should ask:

Which products exceed the budget?
Which product has the highest rating?
Which seller has the most reviews?
Which laptop offers the best price-to-performance ratio?
Is the market trending upward or downward?

And finally:

Which one would I personally recommend?

That is no longer data extraction. That is reasoning. And that is the moment a browser becomes an intelligent e-commerce agent.

🛣️ The Architecture Behind SIKKI

Internally, SIKKI is composed of four independent layers. Each layer solves a completely different problem:

SIKKI
Strategic Intelligence Commerce Kernel
┌──────────────────────────────┐
│ Browser Layer                │
│ Playwright + Bright Data     │
└──────────────────────────────┘
                │
                ▼
┌──────────────────────────────┐
│ Extraction Layer             │
│ Product                      │
│ Price                        │
│ Rating                       │
│ Seller                       │
│ Availability                 │
└──────────────────────────────┘
                │
                ▼
┌──────────────────────────────┐
│ AI Layer                     │
│ Gemini                       │
└──────────────────────────────┘
                │
                ▼
┌──────────────────────────────┐
│ Analysis Layer               │
│ Cheapest Products            │
│ Market Average               │
│ Competitor Comparison        │
│ Trend Analysis               │
└──────────────────────────────┘
                │
                ▼
        Markdown / CSV
        Dashboard Export

Because browser automation problems and reasoning problems are fundamentally different. If Gemini cannot understand your prompt, changing Playwright selectors won’t fix it.

If Bright Data cannot open a page, increasing Gemini’s context window won’t help. Each layer has a different responsibility. And each layer can evolve independently.

🏙️ Building The Extraction Layer

Let’s make SIKKI more useful. Instead of extracting a single CNN headline, let’s teach it how to understand products.

In the real world, commerce websites display several critical pieces of information:

Product title
Current price
Rating
Review count
Seller information
Availability
Product link

A human sees these immediately. Our goal is to teach the browser to see them too.

Inside Playwright, we can execute JavaScript directly in the page context using page.evaluate().

For example:

const products = await page.evaluate(() => {const items = [];
    document.querySelectorAll('[data-component-type="s-search-result"]')
      .forEach(card => {
        const title =
          card.querySelector('h2 span')?.innerText;
        const price =
          card.querySelector('.a-price .a-offscreen')
          ?.innerText;
        const rating =
          card.querySelector('.a-icon-alt')
          ?.innerText;
        const link =
          card.querySelector('h2 a')
          ?.href;
        if(title && price){
            items.push({
                title,
                price,
                rating,
                link
            });
        }
    });
    return items;
});
console.log(products);

The output will look like this:

[ {
   title: "Acer Nitro V Gaming Laptop",
   price: "$899",
   rating: "4.5 out of 5 stars",
   link: "https://..."
 },
{
   title: "ASUS TUF Gaming A15",
   price: "$949",
   rating: "4.4 out of 5 stars",
   link: "https://..."
 }
]

At this point, SIKKI has eyes. But it still doesn’t know what it is looking at.

That is where Gemini enters the architecture.

🧠 Giving SIKKI A Brain

I personally think many developers misunderstand LLMs. Gemini is not a magical replacement for code. It is a reasoning engine. This makes it perfect for commerce intelligence.

After extracting products, we simply send structured JSON to Gemini.

const prompt = `You are an ecommerce analyst.
Analyze the following products.
${JSON.stringify(products)}
Return:
1. Cheapest product
2. Average price
3. Best rated product
4. Summary of the market
5. Recommendation
`;

Gemini may answer:

I analyzed 12 products. The cheapest option is Acer Nitro V at $899.

The average market price is $942.

ASUS TUF has slightly better reviews but is more expensive.

Based on value and user ratings, Acer Nitro V provides the best balance between performance and affordability.

Now SIKKI is not only obtaining information but also analyzing the output.

🛸 Building the Final SIKKI Agent

At this point, we already have everything we need.

🌩️ A cloud browser powered by Bright Data
🧐 Playwright connected through CDP
⛑️ Gemini API access
🚚 A local Node.js project

Now, let’s put everything together and build the final version of SIKKI.

The goal is straightforward:

Open an e-commerce website.
Search for a product.
Extract product information.
Ask Gemini to analyze the results.
Export the findings into a Markdown report.

By the end of this section, you will have a fully functional browser-use AI Agent running locally.

Step 1: Create the Agent File

Replace your previous local Sikki project index.js file (if you didn’t follow through with our initial “Building Sikki Agent” section) and import the required modules. Paste your Browser CDP endpoint inside your index.js file if you want instant access or follow best practice by creating an .env file (using the dotenv library) to secure your API keys:

import "dotenv/config"; //to use environment variables from .env file
import fs from "fs"; //to read and write files
import { chromium } from "playwright";
import { GoogleGenAI } from "@google/genai";
//api keys and endpoints from environment variables
const BRIGHT_WS_ENDPOINT = process.env.BRIGHT_WS_ENDPOINT;
const GEMINI_API_KEY = process.env.GEMINI_API_KEY;

Initialize Gemini with a fixed prompt text first, then we will make it dynamic:

//initialize the GoogleGenAI client with the Gemini API key
const ai = new GoogleGenAI({
  apiKey: GEMINI_API_KEY,
});
//the prompt to send to the Gemini API
const prompt = "Write a short introduction to the Gemini API."; 
const response = await ai.models.generateContent({
  model: "gemini-2.5-flash",
  contents: prompt,
});
console.log(response.text);

Here is the results in our local terminal:

Step 2: Setup Playwright to connect to Bright Data’s WebSocket

As I have already shown you how to set up Bright Data with Playwright earlier, I will go straight to searching for e-commerce products.

Note: Unlike a Playwright’s normal local browser chromium.launch() we used chromium.connectOverCDP(). This browser runs inside Bright Data’s infrastructure. IP rotation, browser fingerprinting, and anti-bot protection are handled automatically. Your local machine only sends instructions.

To test our playwright and bright data working properly, let’s search for “gaming laptops” from amazon:

//setup Playwright to connect to Bright Data's WebSocket endpoint and navigate to Amazon search results for "gaming laptop"
async function main(){
  const browser = await chromium.connectOverCDP(BRIGHT_WS_ENDPOINT);
  const page = await browser.newPage();
const query = "gaming laptop";
  const searchUrl = `https://www.amazon.com/s?k=${encodeURIComponent(query)}`;
  console.log("Navigating Bright Data Playwright to:", searchUrl);
  await page.goto(searchUrl, { waitUntil: "domcontentloaded" });
  const title = await page.title();
  console.log("Amazon search page title:", title);
  console.log("Bright Data navigation complete for query:", query);
  await browser.close();
}
main();

Here is our terminal output 👇:

You can replace Amazon with: BestBuy, Walmart, Product Hunt, g2, Ebay etc. The logic remains exactly the same.

Step 3: Extract Product Information

Now let’s collect structured data.

// Playwright to connect to Bright Data's WebSocket endpoint and navigate to Amazon search results for "gaming laptop"
async function main(){
  const browser = await chromium.connectOverCDP(BRIGHT_WS_ENDPOINT);
  const page = await browser.newPage();
const query = "gaming laptop";
  const searchUrl = `https://www.amazon.com/s?k=${encodeURIComponent(query)}`;
  console.log("Navigating Bright Data Playwright to:", searchUrl);
  await page.goto(searchUrl, { waitUntil: "domcontentloaded" });
  const products = await page.evaluate(() => {
    const data = [];
    document
      .querySelectorAll('[data-component-type="s-search-result"]')
      .forEach((card) => {
        const title = card.querySelector("h2 span")?.innerText;
        const price = card.querySelector(".a-price .a-offscreen")?.innerText;
        const rating = card.querySelector(".a-icon-alt")?.innerText;
        if (title && price) {
          data.push({
            title,
            price,
            rating,
          });
        }
      });
    return data;
  });
  console.log("Step 3: Extract Product Information");
  console.log(products);
  const title = await page.title();
  console.log("Amazon search page title:", title);
  console.log("Bright Data navigation complete for query:", query);
  await browser.close();
}
main();

Our Result 👇:

At this point, SIKKI has successfully observed the environment.

Now we give it the ability to reason.

Step 4: Ask Gemini To Analyze The Market

Create a prompt:

const prompt = `You are an ecommerce analyst.
Analyze these products:
${JSON.stringify(products)}
Return:
1. Cheapest Product
2. Average Market Price
3. Best Rated Product
4. Competitor Comparison
5. Final Recommendation
`;

Generate:

const result = await model.generateContent(prompt);
const analysis =
result.response.text();

Terminal output:

Here is the “Best reated product” result and analysis that I pasted from my terminal:

### 3. Best Rated Product
There are three products tied for the highest rating of **5.0 out of 5 stars**:
1.  **HP OMEN 16" Gaming Laptop Computer** (Intel Ultra 9 285H, NVIDIA GeForce RTX 5070 8GB, 16GB DDR5, 1TB PCIe SSD, 144Hz IPS WUXGA Display) at **$1,745.99**
2.  **HP OMEN 16 Slim RTX 5070 Gaming Laptop** (Intel Ultra 9 285H, NVIDIA RTX 5070, 32GB DDR5, 1TB SSD, 16" FHD+ Anti-Glare) at **$1,929.00**
3.  **ASUS ROG Strix G16 Gaming Laptop** (Intel Core i7 14650HX, NVIDIA RTX 5060, 32 GB DDR5 RAM, 1 TB SSD, 16" FHD+ LED Display, & Gaming Accessories) at **$1,999.99**
**Analysis:** It's noteworthy that two HP OMEN laptops achieved perfect ratings, both featuring high-end Ultra 9 CPUs and RTX 5070 GPUs. 
This suggests strong customer satisfaction with HP's premium gaming line. 
The ASUS ROG Strix G16 also achieved a 5.0 rating, 
despite having a slightly lower-tier GPU (RTX 5060) than the 
HP OMENs in this bracket, possibly due to the included 
"Gaming Accessories" and overall perceived value. 
The Alienware product has no rating, so it cannot be considered here.

Notice what just happened? The browser extracted raw information.

Gemini transformed it into business intelligence. That small architectural separation is what turns automation into an AI Agent.

Step 5: Export The Sikki Market Report

Now we save the scraped product data and Gemini analysis into real files.

Instead of dumping raw JSON into the output, the final agent:

extracts clean product titles
normalizes URLs
writes a Markdown report
writes a CSV table
writes a short summary.txt dashboard

Example report generation:

const report = `
# SIKKI Market Report
## Search Query
${query}
## Products
${JSON.stringify(products, null, 2)}
## Gemini Analysis
${analysis}
Generated: ${new Date().toISOString()}
`;
fs.writeFileSync(reportPath, report, "utf8");

You will see sikki-report.md after running this agent in your terminal.

Here is the full code:

import "dotenv/config"; // use environment variables from .env file
import fs from "fs";
import { chromium } from "playwright";
import { GoogleGenAI } from "@google/genai";
const BRIGHT_WS_ENDPOINT = process.env.BRIGHT_WS_ENDPOINT;
const GEMINI_API_KEY = process.env.GEMINI_API_KEY;
if (!BRIGHT_WS_ENDPOINT) {
  throw new Error("Missing BRIGHT_WS_ENDPOINT environment variable.");
}
if (!GEMINI_API_KEY) {
  throw new Error("Missing GEMINI_API_KEY environment variable.");
}
const ai = new GoogleGenAI({
  apiKey: GEMINI_API_KEY,
});
async function retryWithBackoff(fn, maxRetries = 5, initialDelayMs = 2000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (error) {
      const errorMsg = error?.message || error?.status || JSON.stringify(error);
      if (attempt === maxRetries) {
        console.error(`Final attempt failed. Error: ${errorMsg}`);
        throw error;
      }
      const delayMs = initialDelayMs * Math.pow(2, attempt - 1);
      console.log(`Attempt ${attempt} failed: ${errorMsg}`);
      console.log(`Retrying in ${delayMs}ms... (${attempt}/${maxRetries})`);
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
  }
}
async function main() {
  let browser;
  try {
    browser = await chromium.connectOverCDP(BRIGHT_WS_ENDPOINT);
    const page = await browser.newPage();
    const query = "gaming laptop";
    const searchUrl = `https://www.amazon.com/s?k=${encodeURIComponent(query)}`;
    console.log("Connecting Playwright to Bright Data...");
    console.log("Navigating to:", searchUrl);
    await page.goto(searchUrl, { waitUntil: "domcontentloaded" });
    const products = await page.evaluate(() => {
      const data = [];
      document
        .querySelectorAll('[data-component-type="s-search-result"]')
        .forEach((card) => {
          const title = card.querySelector("h2 span")?.innerText;
          const price = card.querySelector(".a-price .a-offscreen")?.innerText;
          const rating = card.querySelector(".a-icon-alt")?.innerText;
          if (title && price) {
            data.push({
              title,
              price,
              rating,
            });
          }
        });
      return data;
    });
    console.log(`Extracted ${products.length} products.`);
    const analysisPrompt = `You are an ecommerce analyst.
Analyze these products:
${JSON.stringify(products)}
Return:
1. Cheapest Product
2. Average Market Price
3. Best Rated Product
4. Competitor Comparison
5. Final Recommendation
`;
    console.log("Sending products to Gemini for analysis (with retry logic)...");
    const analysisResult = await retryWithBackoff(async () => {
      return await ai.models.generateContent({
        model: "gemini-2.5-flash",
        contents: analysisPrompt,
      });
    }, 5, 2000);
    const analysis = analysisResult.text || "No analysis text returned.";
    const report = `# SIKKI Market Report
## Search Query
${query}
## Products
${JSON.stringify(products, null, 2)}
## Gemini Analysis
${analysis}
Generated: ${new Date().toISOString()}
`;
    fs.writeFileSync("sikki-report.md", report, "utf8");
    console.log("Step 5: Export the Results");
    console.log("Report generated: sikki-report.md");
    console.log("Final analysis saved. Open sikki-report.md to review the result.");
    await browser.close();
  } catch (error) {
    console.error("Error:", error.message || error);
    if (browser) {
      await browser.close().catch(() => {});
    }
    process.exit(1);
  }
}
main();

Here is the output of Sikki market report:

Step 6: 📸Teach SIKKI To Capture Evidence

Now that SIKKI can scrape product metadata, the next job is proof.

Capture evidence by taking:

one search-results summary screenshot
a product card screenshot for the top result
direct product page screenshots for the highest-ranked items

Here is how:

const searchScreenshot = path.join(screenshotDir, "search-results.png");
await page.screenshot({ path: searchScreenshot, fullPage: true });
const summaryPath = path.join(screenshotDir, "top-product-summary.png");
await cards[0].screenshot({ path: summaryPath });

Result:

Step 7: Capture Every Product Page

Now we’re able to extract product information using Bright Data (Playwright) without getting blocked by antibot walls, analyze the market using Gemini, and make a report that will be stored inside our /Reports directory. SIKKI should visit each selected product URL and save the page image and it should be graceful without error loops.

Here is the fun part: we don’t want only search result screenshots; we want each product page detail picked by Gemini intelligence. We will generate up to five screenshots; the first screenshot will be a summary. The remaining 4 screenshots should be: top products and individual product details pages. They will be generated based on the user search query and carefully analyzed by Gemini.

Let’s update our code, but this code is too long to paste here (you should check my git repo) so I will explain what I am changing along the way.

Let’s first have a look at what changes our new updated code made. This time I will search for “ryzen processor”:

Key points we are making:

normalize the URL first
visit the direct product page
wait briefly for the page to render
save up to 4 product screenshots

Example:

const normalized = normalizeAmazonUrl(p.url, p.title);
await retryWithBackoff(async () => 
 await page.goto(normalized, { waitUntil: "domcontentloaded", timeout: 120000 }), 
 3, 
 1500
);
await page.waitForTimeout(800);
await page.screenshot({ path: outPath, fullPage: true });

Here is our intelligent agent picking the top product from Bright Data’s extracted data after a successful analysis and saving a screenshot inside the “screenshot” folder:

This turns SIKKI from “just data” into “evidence-backed market intelligence.”

Step 8: Create a CSV Export

A CSV file is essential for downstream analysis and makes the agent practical.

Write a simple export with:

title
price
rating
normalized URL

Example:

const csvLines = ["TITLE,PRICE,RATING,URL"];
enriched.forEach((p) => {
 const safe = (s) => `"${String(s || "").replace(/"/g, '""')}"`;
 csvLines.push([safe(p.title), safe(p.price), safe(p.rating || ""), safe(p.url || "")].join(","));
});
fs.writeFileSync(csvPath, csvLines.join("\n"), "utf8");

The result is a readable products.csv file:

Step 9: Generate A Dashboard Summary

Finally, create a human-readable summary file with the important metrics.

Compute:

total products extracted
cheapest offer
average price
best rated product
top 5 product summary

Example:

const priced = enriched.filter((p) => !isNaN(p.priceValue));
const cheapestObj = priced.length ? priced.reduce((a, b) => (a.priceValue <= b.priceValue ? a : b)) : null;
const avgPriceVal = priced.length ? (priced.reduce((s, p) => s + p.priceValue, 0) / priced.length).toFixed(2) : “N/A”;
const bestRatedObj = enriched.slice().sort((a, b) => {
 const ra = parseFloat((a.rating || “”).split(“ “)[0]) || 0;
 const rb = parseFloat((b.rating || “”).split(“ “)[0]) || 0;
 return rb — ra;
})[0] || null;

Then write:

const dashboard = [];
dashboard.push(“==================================”);
dashboard.push(“SIKKI MARKET DASHBOARD”);
dashboard.push(“==================================”);
dashboard.push(`Query: ${query}`);
dashboard.push(`Products Extracted: ${enriched.length}`);
dashboard.push(`Cheapest Product: ${cheapestObj ? `${cheapestObj.title} | ${cheapestObj.price}` : “N/A”}`);
dashboard.push(`Average Price: ${avgPriceVal === “N/A” ? “N/A” : `$${avgPriceVal}`}`);
dashboard.push(`Best Rated: ${bestRatedObj ? `${bestRatedObj.title} | ${bestRatedObj.rating || “N/A”}` : “N/A”}`);
dashboard.push(“\nTop 5 Products:”);
enriched.slice(0, 5).forEach((p, i) => {
 dashboard.push(`${i + 1}. ${p.title} | ${p.price} | ${p.rating || “N/A”} | ${p.url || “N/A”}`);
});
fs.writeFileSync(dashboardPath, dashboard.join(“\n”), “utf8”);

Lets have a look at the result:

That’s it for today. What we’ve built is a foundation, not a finished product. The interesting part is what you scale it into, and there’s a lot of room.

Here is the github repo of this project (SIKKI agent): https://github.com/codewithshahan/sikki-agent

🏌️Where To Go From Here

This article intentionally focused on the foundations. But SIKKI can become significantly more powerful. You can extend it with:

🤹Multi-site product comparison
🔎Competitor tracking
⏰Scheduled market reports
📊Interactive dashboards
📈Price trend monitoring
🛍️Autonomous shopping assistants
🗃️Agent memory using databases
🍶Multi-agent workflows

You could even create an agent that monitors hundreds of e-commerce websites every day and sends market intelligence directly to your inbox.

The possibilities become surprisingly large once you stop thinking of browser automation as scraping and start thinking about it as an intelligent system.

If you’re serious about mastering clean code and taking your programming career to the next level, then my book is for you: Clean Code Zero to One. I am offering a 50% discount using the code “earlybird” during checkout ;only for the first 50 copies! Plus, enjoy a 30-day money-back guarantee. no risk, all reward.

🏁Conclusion

Many developers think AI Agents begin with LLMs. I think they begin with architecture. An agent is not just a prompt.

It is not just a browser. It is not just an API.

It is the coordination between observation, reasoning, and action.

That is why I built SIKKI: What began as a simple browser session eventually became something much more interesting.

A system capable of exploring the web, interpreting information, and transforming raw data into actionable knowledge.

And I am convinced this is only the beginning.

The future of browser automation isn’t scraping pages anymore. It’s building agents that can understand them.

🧲 Frequently Asked Questions

1. Why use Bright Data instead of running Playwright locally?

Local Playwright is perfect for development and testing, but production websites often evaluate browser fingerprints, IP reputation, TLS signatures, and other anti-bot signals before serving content. Bright Data provides a production-ready browser infrastructure that handles these challenges behind the scenes that allows your AI agent to interact with websites more reliably while you focus on building application logic.

2. Do I still need Playwright if I’m using Bright Data?

Yes. Bright Data and Playwright solve different problems. Playwright controls the browser by clicking, typing, navigating, and extracting data, while Bright Data provides a cloud browser environment designed to access modern websites reliably. Together they create a much more robust browser automation stack than using either tool alone.

3. What kinds of AI agents can I build with Bright Data?

Once you have reliable browser access, the possibilities extend far beyond web scraping. The same architecture can power shopping assistants, market research agents, competitor monitoring systems, travel assistants, browser testing agents, lead generation workflows, news monitoring, and autonomous research assistants. By combining an LLM for reasoning, Playwright for browser control, and Bright Data for browser infrastructure, you can build AI agents capable of operating on real websites with minimal changes to the overall architecture.

How to Bypass Anti-Bot Walls for Production-Ready Apps

Programming with Shahan — Thu, 02 Jul 2026 22:40:15 +0000

In this guide, you’ll learn:

Why traditional scraping approaches fail against modern anti-bot systems, even when using rotating User-Agents or browser automation.
How enterprise websites actually detect and block automated traffic through fingerprinting, reputation scoring, behavioral analysis, and network signals.
How to architect a production-ready scraping pipeline using cloud browsers, proxies, and AI agents instead of crumbling scripts.
How to build a multi-target autonomous agent called Vanguard that can reliably collect data from modern websites at scale.

If you’re anything like me, you’ve probably found that many web scraping tutorials stop being useful the moment you point them at a real production website.

The common advice is surprisingly predictable: use fetch(), automate a browser with Playwright or Puppeteer, rotate a User-Agent, maybe add a proxy, and you’re supposedly ready to scrape almost anything.

That approach might work against a small personal website. It usually falls apart against companies like Amazon, Product Hunt, G2, Walmart, or CNN.

Developers often assume anti-bot protection is just another HTTP check that can be bypassed by changing a few request headers. Modern detection systems work quite differently: they review browser fingerprints, TLS characteristics, IP reputation, behavioral patterns, request timing, JavaScript execution, and dozens of other signals before deciding whether your traffic looks human.

Changing your User-Agent is only one tiny piece of that puzzle.

To understand why so many automation projects eventually fail, we first need to understand how modern Web Application Firewalls (WAFs) actually inspect and classify traffic.

Corporate website’s WAF ecosystem:

If you’ve built automation that kept getting blocked after a few days, you’re not alone. I’ve written far more broken scraping systems than successful ones, and every failure taught me something about how these defenses actually work.

The good news is that the problem usually isn’t your code. It’s the architecture behind it.

Throughout this guide, we’ll look beyond individual requests and focus on designing systems that can survive production environments. Regardless of how you’re building a price-monitoring platform, collecting market intelligence, or fueling AI agents with fresh web data, the underlying principles remain the same.

By the end, you’ll understand the mechanics behind modern bot detection and build a fully operational multi-target agent called Vanguard. More importantly, you’ll understand why its architecture continues working long after traditional scraping scripts have failed.

P*rerequisite:* All examples use plain JavaScript with Node.js so we can focus on the underlying architecture instead of framework-specific details

I — The Illusion of the Clean Browser

When an automation script fails, developers immediately look for a quick proxy patch. They focus on changing their actions to make progress toward the goal (which is low impact) instead of fundamentally changing how their connection behaves so that success naturally follows (which is high impact).

Typically, developers write a script, test it on their local machine, and watch it succeed. They look at a target link (e,g. https://www.amazon.com/s?k=ddr5+ram+128gb), paste it into their personal browser, and see the clean results page layout load instantly without a single CAPTCHA block:

They assume the target is wide open. But they are trying to build a great pipeline on a rotting foundation.

Your personal browser loads that link because it brings a lifetime of clean human history. It presents a validated residential IP address, deep cookie pools, active browser session context, and a highly complex, hardware-accelerated TLS cryptographic fingerprint. Your browser doesn’t have to “grind” or “prove” it’s human; its natural behavior inherently aligns with what the security system requires.

A vanilla automated script has none of this context. To show you exactly how fast a standard production system rejects unmanaged automation, let’s look at the naive code most engineers write when they are being naive.

II — The Anatomy of an Immediate Rejection

If you want to understand how anti-bot engines work, you must look at how they treat raw, unmanaged server requests.

If this doesn’t make sense, let’s run through an example:

When you execute this, your connection is dropped. Amazon’s WAF doesn’t care that your string header says “Mozilla”. It looks at the TCP handshake behavior, notices that your TLS cipher suite order doesn’t match an actual Windows Chrome installation, maps your datacenter or unrotated local IP, and flags the machine instantly.

So… security systems read what your connection does. Your headers are irreverent. Take a look:

Do you think the real human browser has to “grind” to bypass a CAPTCHA? Does their machine have to trick the target to load a product page? To you, it may seem like a basic request on the surface, but the truth is that the security gateway cannot see them as anything other than authentic. The human browser has to try incredibly hard to trigger an anti-bot wall. The bare automated script has to force its way through a firewall, and it gets flagged every single second of it.

To some developers, a professional enterprise proxy mesh seems extreme and over-engineered. To me, it’s natural, and I don’t say that to contrast it with any other kind of architecture. I simply enjoy running production systems that do not break. When my peers tell me that I should just use a basic string rotation utility… I hold my tongue from telling them, “If your scripts are constantly breaking, why are you doing what you’re doing?”

This next sentence may sound simple, but it is baffling how many people don’t get it.

If you want to scrape enterprise websites reliably, the request itself is only part of the equation. Modern Web Application Firewalls (WAFs) inspect your browser fingerprint, network behavior, TLS fingerprint, session history, IP reputation, and many other signals before deciding whether you’re a legitimate user or an automated bot.

To understand why most scraping scripts eventually fail, we first need to look at how modern WAFs make those decisions.

III — The “Fake” 200 Response Trap

Lets look at a real-world scenario where all corporate websites play with agentic AI developers by using “soft block” tricks. I will be using Node.js axios library locally to send asynchronous network requests to REST endpoints, in this case our target URL endpoint is Amazon’s DDR5 ram listing page (you can use any website you like for scraping data):

// The Lazy Scraping Pattern (Guaranteed to fail at scale)

const axios = require('axios');
async function naiveScrape() {
const targetUrl = 'https://www.amazon.com/s?k=ddr5+ram+128gb';
try {
const response = await axios.get(targetUrl, {
headers: { 'User-Agent': 'Mozilla/5.0…' }
});
console.log("Success! Status:", response.status, "Data Length:",
response.data.length);
} catch (error) {
console.error("Blocked!");
}
}
naiveScrape();

If you run this code inside your terminal right now, you might see an output that looks exactly like this:

When scraping a gigantic popular website, a 200 OK response status is often a false victory. Large e-commerce sites use a technique called a Soft Block.

You look at that output. You see nearly a megabyte of data. You see no errors. You see a pristine “200 Success” code. You assume your code cleanly outsmarted the infrastructure. You are wrong.

This is the cleverest trap in modern web architecture. High-tier platforms rarely throw a loud 403 Forbidden or 503 Error code to standard scripts anymore. If they give you an error code, you know instantly you’re blocked, and you immediately start changing your code.

Instead, they route your connection away from the raw data asset entirely. Let’s prove exactly what is inside that 650KB+ length data string. Add these logging lines directly into your try block inside the above codeblock:

console.log("Does it contain Amazon's soft-block tag?", response.data.includes("a-no-js"));

console.log("First 300 characters of the page code:\n", response.data.substring(0, 300));

When you run it, you won’t see the actual marketplace data from Amazon. You will see lines of obfuscated, tracking JavaScript loops meant to run in a real browser:

See? Your terminal exposed the trap. The real clue is the content that was returned. Instead of the product page DOM you expected, Amazon responded with its verification challenge wrapper. In other words, AWS WAF intercepted your request and served a challenge page rather than the actual marketplace content because your script wasn’t operating as a real browser session. Because your script lacked a continuous browsing fingerprint, AWS WAF silently intercepted your connection and handed you a dead-end sandbox loop.

Your script spends precious compute power downloading and trying to parse almost a megabyte of empty authentication wrappers, completely blind to the fact that it didn’t capture a single item. Automated agents cannot run the tracking scripts required to clear that page, putting your pipeline at a complete dead end.

Even though a target website tricks your code with a fake success redirect or drops you completely with a 403 Forbidden block, you are losing the exact same thing: access to the data layer.:

I will now use CNN news website as target URL, instead of falling into Amazon’s tricky “soft block” trap, you will receive an instant block since CNN use Cloudflare; a service that will instantly reject your vanilla Axios request with a hard HTTP 403 Forbidden error code and terminate your script. Have a look:

const axios = require('axios');
async function naiveScrape() {
// Switching to a Cloudflare-protected target for a clean firewall error
const targetUrl = 'https://www.cnn.com/';
try {

console.log("Initiating naive data extraction request…");
const response = await axios.get(targetUrl, {
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36'
}
});
console.log("Success! Data Length:", response.data.length);
} catch (error) {
console.error("\n❌ Request Crushed by Anti-Bot Wall.");
if (error.response) {
// This will capture and print the hard 403 error code
console.error(`Status Code: ${error.response.status}`);
console.error(`Status Text: ${error.response.statusText}`);
console.error("Reason: Cloudflare WAF rejected the vanilla Axios
handshake.");
} else {
console.error("Error Message:", error.message);
}
}
}

naiveScrape();

Here is the result inside the terminal:

A 403 Forbidden error is universally recognized by developers as a hard security rejection. It removes all ambiguity. The errors will be the same for g2.com or any other protected sites, and your automated agents cannot run that script, so your code is completely stuck at a dead end.

IV — Moving to an Automated Gateway

I realized I couldn’t spend my life engineering custom solutions for every single browser fingerprint update, JS challenge modification, or proxy rotation algorithm. It’s an endless game of whack-a-mole. I needed a production-ready infrastructure that handled the network-level deception for me so I could focus purely on writing data logic.

That is why I migrated this workflow to Bright Data’s Web Unlocker. It treats the entire bypass pipeline as a simple managed endpoint. It acts as an abstraction layer that intercepts your raw request, automatically executes the necessary TLS handshakes, decrypts hidden challenges, mimics human behavior, and pipes back the clean HTML data.

Here is exactly how to set this up and test it without turning your local computer into a messy testing ground.

Step 1: Account Activation and Token Generation

First, you need access to the proxy cluster infrastructure.

Signup/login to the dashboard account on Bright Data Web Unlocker.
From the main dashboard sidebar, navigate to Web Access:

3. Select Web Unlocker API and press continue. This is the specialized cluster that houses the machine-learning-driven browser emulation layer:

4. Name your routing zone (default is: web_unlocker1), activate it, and generate an API Token by clicking into the “API key” box.

5. Allow Your Server IP: Before sending requests, there’s one important configuration step that’s easy to miss.

Open your Web Unlocker zone settings, go to the configuration tab, then security settings, and finally check Allowed IPs. Make sure to add your current IPs in the allowlist. Go to “Configuration, then “Security Settings”:

Here is “Add allowed IP” box. Paste your current IP from your machine or check “recent IPs allowlist”:

Without this step, your API key may be valid and your zone may be configured correctly, but your requests can still fail because Bright Data rejects connections from unauthorized IP addresses.

Once your IP has been added and the changes have propagated, you’re ready to send your first request.

Step 2: Testing Inside the Native Bright Data Playground

Before you deploy a single line of application code to your servers, you can visually verify that the unlocking layer works by using their built-in Playground. It allows you to run requests directly from the cloud console to check how targets respond.

Paste your target address directly into the Target URL field. For example: https://www.amazon.com/s?k=ddr5+ram+128gb
Keep the configuration to API mode and device type to desktop.
Hit the blue Run Request button:

The interface will display a green RESPONSE 200 success state. If you look at the bottom visual block, you’ll see a rendering preview of the exact Amazon product directory list. You’ll also find the pure market tables and real product tags by managing to bypass the security wall without any problems.

It is a 3-in-1 tool that manages IP rotation, dynamic browser fingerprinting, and CAPTCHA solving automatically. The API bypasses the anti-bot wall and returns the clean HTML source code to the software in a single response.

Note on verification: If you download that specific raw response file from the playground UI, your computer will automatically open it up inside a local web browser, rendering it visually. If you want to see the actual raw data your backend will process, simply open that downloaded file with a basic text editor like VS Code. You will find thousands of lines of pristine, uninhibited DOM structural code strings.

V — Building LaunchRadar Agent

If you are trying to feed live web data into an LLM using basic Axios configurations or unmanaged connection structures, you are forcing your AI agent to ride a bicycle with flat tires. Modern security layers look for advanced browser behaviors and TLS cipher sequences. If you try to manually concatenate your configuration parameters as raw text blocks, a single trailing comma will break strict backend string parsing, throwing a frustrating 400 Bad Request or an empty payload block.

Therefore, we need to install Bright Data’s official SDK in our project. With this, we completely eliminate Axios-style network proxy bugs and let Bright Data handle them.

Lets go trough the code blocks step by step to build our first AI agent using Bright Data. We are going to assemble a product hunt’s trend intelligence agent called LaunchRadar. Its target is specific: it bypasses the tracking walls of Product Hunt, ingests the real-time tech product feeds, and structures raw document strings into clean text data arrays ready to feed straight into an LLM context window.

Setup LaunchRadar Project from Scratch

To ensure your runtime application environment maps dependencies cleanly, we will completely drop out of folders controlled by cloud backup networks (like Microsoft OneDrive or iCloud) that lock active file permissions and disrupt local script loops.

Open your terminal window (Terminal app on macOS, or your preferred shell) and run these clean setup commands sequentially to construct an isolated project workspace from scratch:

# 1. Move directly to your local User Home root path to avoid file syncing errors
cd ~

# 2. Provision a dedicated application folder
mkdir LaunchRadar

# 3. Enter your newly initialized project workspace
cd LaunchRadar

# 4. Initialize an isolated local package architecture
npm init -y 

# 5. Tell the Node.js runtime to process modern ES Module imports natively
npm pkg set type="module"

Now install the official bright data SDK locally using node.js:

npm install @brightdata/sdk

Now, create a fresh file named index.js inside your application directory workspace and paste this exact production script to scrap raw HTML data from Product Hunt:

/**
 * Project: LaunchRadar
 * Core Stack: Node.js + Official Bright Data JavaScript SDK
 */

import { bdclient } from '@brightdata/sdk';
const BRIGHTDATA_KEY = process.env.BRIGHTDATA_API_KEY || '';
async function deployLaunchRadar() {
    // Targeting Product Hunt's active daily launch asset path
    const targetUrl = 'https://www.producthunt.com/';

    console.log("🤖 [LaunchRadar]: Initializing autonomous target ingestion loop...");
try {
        // Initialize the client gateway via the native constructor
        const client = new bdclient({
            apiKey: BRIGHTDATA_KEY
        });
console.log("🤖 [LaunchRadar]: Tunneling through Web Unlocker cluster...");

        // Execute the single-step browser-emulated extraction natively
        const htmlResponse = await client.scrapeUrl(targetUrl, {
            zone: 'web_unlocker1',
            format: 'raw' // Streams the target webpage as a raw HTML text string
        });
console.log("\n=================== PIPELINE SUCCESS ===================");
        console.log(`🤖 [LaunchRadar]: Ingestion complete. Streamed: ${htmlResponse.length} bytes.`);
        console.log("========================================================\n");

        // This will print true-proving your paid credit bypassed the security loop
        const dataIsValid = htmlResponse.includes("Product Hunt");
        console.log(`🤖 [LaunchRadar]: Target Data Authenticated: ${dataIsValid}`);
// Gracefully disconnect active execution sockets
        await client.close();
} catch (error) {
        console.error("❌ [LaunchRadar Fatal]: Pipeline exception encountered.");
        console.error(`Reason: ${error.message}`);
    }
}
deployLaunchRadar();

If you run that code inside your terminal window, you will see:

Congratulations! You extracted nearly a ~megabyte of raw HTML data. Now you understand how we can collect raw HTML data, it’s time to build our final agent application: Vanguard Agent. It interfaces directly with Bright Data’s official JavaScript SDK, targets the verified backend data pipeline endpoint natively, and bypasses the front-end rendering layer completely to return beautifully structured insights straight to your terminal window.

VI — Building Vanguard Agent

Now that Web Unlocker is handling browser fingerprinting, JavaScript challenges, TLS negotiation, and request routing behind the scenes, we can focus on the part that actually creates value: extracting useful information.

Now let’s build Vanguard Agent. It will work as a lightweight agent that searches live CNN articles, extracts relevant headlines, and automatically generates a structured Markdown report.

Instead of supporting multiple unrelated websites with different search endpoints and DOM structures, we’ll focus on a single production-quality example. The same architecture can later be extended to other targets by swapping the parser while keeping the networking layer exactly the same.

First, install Cheerio:

npm install cheerio

Then, trigger the Bright Data SDK request with the following:

const htmlResponse = await client.scrapeUrl(targetUrl, {
zone: "web_unlocker1",
format: "raw",
country: "us",
dataFormat: "html"
});

dataFormat: “html” is the correct SDK option for returning rendered HTML. Browser emulation is already handled internally by Web Unlocker, so no additional rendering flag is required.

Next, I will point the application directly at CNN’s search endpoint. This produces reproducible search results while keeping the extraction logic straightforward.

After that, the HTML has been retrieved by Bright Data; Cheerio loads the document and extracts article links.

Once the extraction finishes, Vanguard writes the collected results into a Markdown report that can be consumed by downstream AI agents, summarization pipelines, or analytics workflows. I will explain more clearly inside the code blocks one by one.

Here is the full source code of our agent to test on your local machine:

/**
* Project: Vanguard Agent
* Function: Autonomous CNN Trend & Headline Generator
* Core Stack: Node.js (ESM) + Official Bright Data JavaScript SDK + Cheerio
*
* IMPORTANT: CNN's own robots.txt disallows /search (Disallow: /search),
* so Bright Data's Web Unlocker refuses it in immediate-access mode -
* that's not a bug, it's the site opting out of that specific path for
* compliant bots. Instead, this hits CNN's own News Sitemap
* (https://www.cnn.com/sitemap/news.xml), which robots.txt explicitly
* lists as a Sitemap: entry - meaning it's meant to be fetched by bots.
* It contains real headline titles and real publish timestamps for the
* last ~48 hours of coverage across every section, which is arguably a
* better fit for "trend monitoring" than a search page anyway.
*/

import fs from 'fs';
import readline from 'readline';
import * as cheerio from 'cheerio';
import { bdclient } from '@brightdata/sdk';
const REPORT_FILE = 'vanguard_trend_report.md';
const CNN_NEWS_SITEMAP = 'https://www.cnn.com/sitemap/news.xml';

// Key comes ONLY from the environment - no hardcoded fallback, ever.
// If a real key has been pasted into a file, a chat, or a log at any
// point, treat it as compromised and rotate it in your Bright Data
// dashboard immediately, even after removing it from the code.

const BRIGHTDATA_KEY = process.env.BRIGHTDATA_API_KEY || '';
if (!BRIGHTDATA_KEY) {
    console.error(
        "\n❌ [Vanguard Config Error] BRIGHTDATA_API_KEY is not set.\n" +
        "   Run it like this instead:\n\n" +
        "   PowerShell:   $env:BRIGHTDATA_API_KEY=\"your_actual_key\"; node vanguard.mjs\n" +
        "   cmd.exe:      set BRIGHTDATA_API_KEY=your_actual_key && node vanguard.mjs\n" +
        "   macOS/Linux:  BRIGHTDATA_API_KEY=your_actual_key node vanguard.mjs\n"
    );
    process.exit(1);
}

// CNN article URLs follow /YYYY/MM/DD/category/slug - used here purely to
// derive a topic category for the breakdown, not for date (the sitemap
// gives us a real publish timestamp directly).
const CNN_ARTICLE_PATTERN = /^\/(\d{4})\/(\d{2})\/(\d{2})\/([a-z0-9-]+)\//i;
const printDashboardHeader = () => {
    console.clear();
    console.log("\x1b[38;5;39m%s\x1b[0m", " ┌────────────────────────────────────────────────────────┐");
    console.log("\x1b[38;5;39m%s\x1b[0m", " │   VANGUARD INTELLIGENCE SERVICE // CNN TREND ENGINE     │");
    console.log("\x1b[38;5;39m%s\x1b[0m", " └────────────────────────────────────────────────────────┘");
};

const promptUser = (queryMessage) => {
    const interfaceInstance = readline.createInterface({
        input: process.stdin,
        output: process.stdout
    });
    return new Promise((resolve) => {
        interfaceInstance.question(queryMessage, (inputAnswer) => {
            interfaceInstance.close();
            resolve(inputAnswer.trim());
        });
    });
};

function escapeRegExp(string) {
    return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}

function categoryFromUrl(loc) {
    try {
        const path = new URL(loc).pathname;
        const dated = path.match(CNN_ARTICLE_PATTERN);
        if (dated) return dated[4].replace(/-/g, ' ');
        // Some CNN URLs skip the date prefix entirely (e.g. video pages
        // like /politics/video/slug). Fall back to the first path segment
        // instead of dumping everything into "uncategorized".
        const segment = path.split('/').filter(Boolean)[0];
        return segment ? segment.replace(/-/g, ' ') : 'uncategorized';
    } catch {
        return 'uncategorized';
    }
}

async function deployVanguardService() {
    printDashboardHeader();
    console.log("\x1b[38;5;244m%s\x1b[0m", "\n [CONFIGURE INTELLIGENCE FILTER]");
    const searchQuery = await promptUser('🔍 Enter a keyword to filter recent CNN headlines (e.g., startup, election, climate): ');
    if (!searchQuery) {
        console.error("\n❌ [Vanguard Error] Input token context empty. Pipeline aborted.");
        return;
    }
    printDashboardHeader();
    console.log(`\n\x1b[38;5;214m🤖 [Vanguard]: Connecting to proxy fabric for target: ${CNN_NEWS_SITEMAP}\x1b[0m`);
    console.log("🤖 [Vanguard]: Initializing automated browser emulation...");
    const client = new bdclient({ apiKey: BRIGHTDATA_KEY });
    try {
        // dataFormat: 'html' returns the raw response body as-is, which
        // works fine for XML too - we just parse it with cheerio's XML
        // mode below instead of treating it as an HTML page.
        const xmlResponse = await client.scrapeUrl(CNN_NEWS_SITEMAP, {
            zone: 'web_unlocker1',
            dataFormat: 'html',
            country: 'us'
        });
        console.log(`\x1b[38;5;46m🤖 [Vanguard]: Data stream ingestion complete. Length: ${xmlResponse.length} bytes.\x1b[0m`);        if (!xmlResponse || xmlResponse.length < 500) {            const preview = xmlResponse ? xmlResponse.slice(0, 300) : '(empty response)';            throw new Error(                `Target returned only ${xmlResponse ? xmlResponse.length : 0} bytes - this is too small to be ` +                `a real sitemap. Raw response below:\n\n${preview}\n\n` +                `Checklist:\n` +                `  1. Confirm BRIGHTDATA_API_KEY is a valid, active key.\n` +                `  2. Confirm 'web_unlocker1' matches an active Web Unlocker zone name in your ` +                `Bright Data dashboard - zone names are account-specific.\n` +                `  3. If the text above mentions a block/captcha, retry in a moment or try a ` +                `different 'country' value.`            );        }        console.log("🤖 [Vanguard]: Processing resilient extraction analytics...");        // xmlMode: true keeps namespaced tags (e.g. news:title) intact        // instead of cheerio's default HTML-parsing/lowercasing behavior.
        const $ = cheerio.load(xmlResponse, { xmlMode: true });
        const insights = [];
        $('url').each((_, el) => {
            const loc = $(el).find('loc').first().text().trim();
            const title = $(el).find('news\\:title').first().text().trim();
            const pubDateRaw = $(el).find('news\\:publication_date').first().text().trim()
                || $(el).find('lastmod').first().text().trim();
            if (!loc || !title) return; // skip entries without a real headline (e.g. galleries/video-only)
            insights.push({
                title: title.replace(/\s+/g, ' '),
                date: pubDateRaw ? pubDateRaw.slice(0, 10) : 'unknown',
                category: categoryFromUrl(loc),
                postUrl: loc
            });
        });
        const uniqueRawInsights = Array.from(new Map(insights.map(item => [item.postUrl, item])).values());
        // Sort newest first using the sitemap's own publish timestamp.        uniqueRawInsights.sort((a, b) => (a.date < b.date ? 1 : a.date > b.date ? -1 : 0));        // Whole-word match, not substring - plain .includes() would match        // "war" inside "Warner" or "warrant", which are false positives        // for a search on "war". \b enforces a real word boundary.        const keywordPattern = new RegExp(`\\b${escapeRegExp(searchQuery)}\\b`, 'i');        let filteredInsights = uniqueRawInsights.filter(item => keywordPattern.test(item.title));        // Phase 2: fallback to the newest overall headlines if no direct match        let isFallbackMode = false;        if (filteredInsights.length === 0) {            filteredInsights = uniqueRawInsights;            isFallbackMode = true;        }        const uniqueInsights = filteredInsights.slice(0, 10);        if (uniqueInsights.length === 0) {            throw new Error('Zero real headline entries isolated inside the sitemap buffer.');        }        const categoryCounts = uniqueRawInsights.reduce((acc, item) => {            acc[item.category] = (acc[item.category] || 0) + 1;
            return acc;
        }, {});
        const topCategories = Object.entries(categoryCounts)
            .sort((a, b) => b[1] - a[1])
            .slice(0, 5);
        console.log("\n\x1b[38;5;51m┌────────────────────────────────────────────────────────────────────────┐");
        console.log(`│ ► VANGUARD INTELLIGENCE STREAM // CNN // '${searchQuery}'`.padEnd(76, ' ') + '│');
        console.log("└────────────────────────────────────────────────────────────────────────┘\x1b[0m");
        if (isFallbackMode) {
            console.log(`\x1b[33m ⚠️ [Notice] No direct matches for '${searchQuery}'. Showing newest CNN headlines instead.\x1b[0m\n`);
        }
        uniqueInsights.forEach((item, index) => {
            console.log(` \x1b[38;5;46m[${String(index + 1).padStart(2, '0')}]\x1b[0m \x1b[1m${item.title.substring(0, 75)}\x1b[0m`);
            console.log(`     ├─ Category : ${item.category}`);
            console.log(`     ├─ Date     : ${item.date}`);
            console.log(`     └─ Link     : \x1b[38;5;39m\x1b[4m${item.postUrl}\x1b[0m\n`);
        });
        console.log("\x1b[38;5;51m└────────────────────────────────────────────────────────────────────────┘\x1b[0m");
        console.log('\x1b[38;5;244m📊 Top categories in this result set:\x1b[0m');
        topCategories.forEach(([cat, count]) => console.log(`   • ${cat} (${count})`));
        const markdownReport = [
            `# Vanguard CNN Intelligence Digest`,
            `* **Search Keyword:** \`${searchQuery}\``,
            `* **Source:** CNN News Sitemap (robots.txt-approved)`,
            `* **Data Capture Status:** ${isFallbackMode ? 'Newest-headlines fallback' : 'Direct keyword match'}`,
            `* **Timestamp:** ${new Date().toISOString()}`,
            `\n## Top Categories`,
            ...topCategories.map(([cat, count]) => `* **${cat}** - ${count} article(s)`),
            `\n## Headlines`,
            ...uniqueInsights.map((p, idx) =>
                `### ${idx + 1}. ${p.title}\n* **Date:** ${p.date}\n* **Category:** ${p.category}\n* **Link:** [${p.postUrl}](${p.postUrl})`
            )
        ].join('\n');
        fs.writeFileSync(REPORT_FILE, markdownReport, 'utf8');
        console.log(`\x1b[38;5;46m💾 [Vanguard] System Report written successfully -> ${REPORT_FILE}\x1b[0m\n`);
    } catch (pipelineException) {
        console.error("\n❌ [Vanguard Fatal]: Operational routine aborted.");
        console.error(`Reason: ${pipelineException.message}`);
    } finally {
        await client.close();
    }
}
deployVanguardService();

At this point, the application submits a keyword, retrieves the rendered page through Web Unlocker, extracts structured information, and generates a clean report automatically. The anti-bot layer is handled by the gateway, allowing your application to stay focused on parsing and processing data instead of maintaining browser fingerprints or bypass logic.

Here is our agent functionality; you can scale it as you wish:

VII — Moving Past The Anti-Bot Maintenance

As websites continue strengthening their anti-bot systems, maintaining custom bypass logic becomes an increasingly expensive engineering task. Every browser update, fingerprinting change, or new challenge introduces additional maintenance that doesn’t contribute to your product.

Using a managed gateway such as Bright Data Web Unlocker shifts that responsibility away from your application. Instead of maintaining infrastructure that keeps requests alive, you can focus on collecting, processing, and analyzing the data your users actually care about.

Whenever you’re building AI agents, market intelligence platforms, monitoring systems, or large-scale data pipelines, separating anti-bot handling from your business logic results in software that is easier to maintain, easier to scale, and more reliable over time.

If you’re serious about mastering clean code and taking your programming career to the next level, then my book is for you: Clean Code Zero to One. I am offering a 50% discount using the code “earlybird” during checkout, only for the first 50 copies! Plus, enjoy a 30-day money-back guarantee. No risk, all reward.

Frequently Asked Questions

Can Web Unlocker scrape JavaScript-heavy websites?

Yes. Web Unlocker performs browser emulation behind the scenes, allowing it to retrieve rendered HTML from websites that rely heavily on JavaScript, anti-bot challenges, or advanced fingerprinting.

Do I still need to rotate proxies manually?

No. Proxy rotation, browser fingerprint management, retry logic, and challenge handling are managed by the Web Unlocker infrastructure, so your application can focus on extracting data instead of maintaining network logic.

Can I use this architecture for AI agents?

Absolutely. AI agents often require reliable access to fresh web data. By separating data collection from application logic, you can feed structured information into LLM workflows without continually rebuilding scraping infrastructure.

Can I extend Vanguard to other websites?

Yes. The networking layer remains the same. In most cases, you only need to replace the target URL and update the parsing logic for the new site’s HTML structure while continuing to use the same Web Unlocker workflow.

How to Write Better Code for Agile Software Development

Programming with Shahan — Thu, 18 Jun 2026 13:25:43 +0000

Building scalable software applications demands writing clean code that’s so simple that any dev can understand it.

In this article, I’ll explain and demonstrate what clean code is. Then I’ll share my favorite clean code patterns for building modern Agile applications.

I won’t use complex jargon. I’ll provide you with simple, clear JavaScript examples that focus on the core concepts. Straight to the point. But this article will be a bit longer than my usual posts. So take your time wisely.

Let’s get started.

The Cost of Bad Code
Clean Coder vs. Messy Coder
AI Can’t Save You If Your Code is a Mess 🗑️
12 Clean Code Design Patterns for Building Agile Applications ⚖️
- 🌿 Use Names That Mean Something
- 🔨 Keep Functions Laser-Focused (SRP)
- 🚪 Use Comments Thoughtfully
- ⚡ Best Practices for Writing Good Comments
- 🧩 Make Your Code Readable
- 🏌️ Test Everything You Write
- 💉 Use Dependency Injection
- 📂 Clean Project Structures
  - 5.tests
- 🤹‍♂️ Be Consistent with Formatting
  - 1️⃣ Consistent Indentation
  - 2️⃣ Meaningful Whitespace: Giving Code Room to Breathe
  - 3️⃣ Clear Separation of Logic: Modular Thinking
  - 4️⃣ Readable Error Handling
- ✋ Stop Hardcoding Values
- 🤏 Keep Functions Short
- ⛺ Follow the Boy Scout Rule
- 🏟️ Follow the Open/Closed Principle
  - Before: Violating the principle
Modern Best Practices to Help You Write Clean Code: A Summary 🥷
- 🔎 Common Code Smells and How to Fix Them
- 💬 Effective Commenting Practices
- 🧼 Refactoring Techniques for Clean Code
- 🧪 Testing and Clean Code
- 🐛 Error Handling and Clean Code
- 🌱 Code Reviews and Clean Code
Automated Tools for Maintaining Clean Code ⚓
- 1️⃣ Static Analysis
  - How does it work?
  - 🚨 Before static analysis:
  - 🚑 After static analysis (using ESLint):
- 2️⃣ Automated Code Formatting
  - After using Prettier:
- 3️⃣ Continuous Integration (CI) Testing
  - How Does CI Testing Work?
- 4️⃣ CI/CD pipelines
- ⚡ Continuous Integration (CI) Testing with GitHub Actions
  - How to Set Up CI Testing with GitHub Actions
The Role of Documentation in Agile Software Development 🚣
- 1. 🚡 Create Useful Docs (Explain Why, Not Just How)
- 2. ⏳ Keep the Docs Updated (Outdated Docs Are Worse Than No Docs)
- 3. 🚆 Integrate Comments (Good Comments in Code Are Part of Documentation)
Conclusion 🏁
Frequently Asked Questions About Clean Code 🧯

In Agile, where change is the only constant, clean code is your armor. It makes you adaptable, swift, and, most importantly, in control.

Here’s the truth: writing clean code is not optional if you want to survive in the software development industry. Fortunately, we human beings are able to master clean code with some effort and practice.

The Cost of Bad Code

To explain this stack bar graph, in the initial development phase, bad code is slightly more costly to change than clean code.

But as we move into the maintenance and refactoring phases, the gap widens significantly, with bad code costing nearly twice as much as clean code.

By the legacy phase, bad code reaches 100% cost; now it’s extremely expensive to update, while clean code remains more manageable at 45%.

As of now, the most recent analysis on the cost of poor software quality in the U.S. is the 2023 report by the Consortium for Information and Software Quality (CISQ). This report estimates that poor software quality cost the U.S. economy at least $2.41 trillion in 2022, with technical debt accounting for about $1.52 trillion of this amount.

You can read more about that here.

Recent discussions continue to highlight the significant impact of technical debt on software quality and business performance.

For instance, a 2024 survey indicated that for more than 50% of companies, technical debt accounts for greater than a quarter of their total IT budget. And this can really hinder innovation if it’s not addressed.

As you can see, there’s no doubt that bad code is a costly problem in software development.

Clean Coder vs. Messy Coder

Here’s a graph that shows the journey of two types of coders:

⚠️ The Messy Coder (Red line): Starts fast but crashes hard. The more lines they write, the more trouble they make.
⚡ The Clean Coder (Blue line): Starts slow but stays consistent. Growth doesn’t stop; it expands.

🫵 Now, you decide which line you want to follow.

AI Can’t Save You If Your Code is a Mess 🗑️

When you get stuck writing code, you might turn to AI. But let me tell you something: AI can’t save you if your code is a mess.

It’s like building a house on sand. Sure, it stands for a while, but one strong gust of wind or big wave, and it collapses.

Remember: AI is just a tool. If you don’t know how to write clean, scalable applications, you're setting yourself up for failure.

If you can’t maintain the code you write, you’re in trouble.

I’ve seen it over and over again: developers who know five programming languages. They can build apps, websites, software. They know algorithms and data structures like the back of their hand.

But when faced with a large project or someone else’s messy code, they crumble.

They’re like an aerospace engineer who designs and builds their own planes but doesn’t know how to fly them. They crash into their own code.

This was me...once upon a time. I’d write thousands of lines of code, only to realize I couldn’t even understand what I wrote last week. It was chaos for me.

But then it hit me;every developer struggles with this. It wasn't about how much I knew. It was about how I organized and structured what I knew. In other words, it was about knowing the art of programming itself.

I decided to escape this trap. After five months of intense work; four to five hours a day writing, designing, and researching. I created something I wish I had when I started programming. A book that’s a complete beginner’s guide: Clean Code Zero to One.

If you want to learn more about the book, I give you all the details at the end of this tutorial. So read on to learn more about writing clean code.

12 Clean Code Design Patterns for Building Agile Applications ⚖️

If your code doesn’t follow these modern clean code design patterns, you could be creating a ticking time bomb. These patterns are your tools. Master them and enjoy the success of your projects. Let me show you one by one.

🌿 Use Names That Mean Something

Naming your variables or functions b or x is not helpful. Call them what they are so they’re easier to understand. Here’s an example of both a bad and good variable name:

// Weak and vague
let b = 5;

// Strong and clear
let numberOfUsers = 5;

People who write unclear names don’t want to own their mistakes. Don’t be that person.

🔨 Keep Functions Laser-Focused (SRP)

A function should do one thing—and do it perfectly. This is called the Single Responsibility Principle ( SRP).

Good code is like a hammer. It hits one nail, not ten. For example, if you are hiring someone to do everything in your company—finance, sales, marketing, janitorial work, and so on;they'll likely fail miserably because they can’t focus one one thing. The same goes for your classes in code.

🚧 When a class or function does more than one thing, it becomes a tangled mess. Debugging it feels like solving a puzzle upside down. If your class handles both user input and database operations, for example, it’s not multitasking—it's madness. Break it up. One method, one job.

🔥 My Rule: Your code works for you. Keep it sharp, focused, and controllable, or it’s going to control you. Here is how to make that happen:

// Clean: One job, one focus
function calculateTotal(a, b) {
    return a + b;
}

function logTotal(user, total) {
    console.log(`User: ${user}, Total: ${total}`);
}

// Messy: Trying to do EVERYTHING
function calculateAndLogTotal(a, b, user) {
    let total = a + b;
    console.log(`User: ${user}, Total: ${total}`);
}

🪧 When you mix tasks, you mix in confusion. As simple as that.

🚪 Use Comments Thoughtfully

There is a great saying among professional developers:

“ Code speaks for itself. ”

You don’t explain what a door does every time someone walks into a room, do you? Your code should work the same way.

Comments aren’t bad, but if your code can’t stand on its own, then you may have a problem.

🪧 A good comment should tell “why” not “how or what”. If a developer doesn’t understand “how” something works, then they likely aren’t going to understand “why” either.

Here are some short examples of good comments vs bad comments. I’ll also show you a real-world project for writing clean comments.

Example 1: Bad Comment 👎

// Multiply the price by the quantity to calculate the total
const total = price * quantity;

This is a bad comment because it simply repeats what the code already says. The code price * quantity is self-explanatory, so the comment doesn’t add anything useful.

Good Comment: 👍

If the code is clear and simple, you don’t need a comment.

const total = price * quantity;

Example 2: Bad Comment 👎

// Check if the user logged in
function isUserLoggedIn(session) {
    return !!session.user;
}

This comment is bad because it doesn’t explain why the isUserLoggin() exists. It just explains what happens. But we already know that this is an auth function. This comment is a waste of time.

Good Example 👍

// The user is authenticated before accessing protected resources
function isUserLoggedIn(session) {
    return !!session.user;
}

This is a good comment because it explains why the code exists. It tells us that the function checks if the user is authenticated before allowing access to sensitive parts of the app. It focuses on the bigger picture.

⚡ Best Practices for Writing Good Comments

Explain the “Why,” not the “What”:

Write comments to explain the purpose or context of the code, not what the code is doing.
Avoid obvious comments:

Don’t write comments for things the code already makes clear.
Keep them short and precise:

Write concise comments that are easy to read and directly explain the purpose.
Update comments regularly:

Outdated comments can mislead developers, so always update them when the code changes.

Real-World Example (with Good Comments) 🛒

Let’s implement these practices into a real-world project: a large e-commerce application. One function calculates shipping costs based on the order details. Here's the full code, I will explain each comment below:

// Shipping rules:
// - Free shipping for orders over $100
// - Standard shipping ($10) for orders below $100
// - Additional $5 for international orders

function calculateShipping(order) {
    let shippingCost = 0;

    // Check if the order qualifies for free shipping
    if (order.total >= 100) {
        shippingCost = 0; // Free shipping
    } else {
        shippingCost = 10; // Standard shipping cost
    }

    // Add additional cost for international orders
    if (order.isInternational) {
        shippingCost += 5;
    }

    return shippingCost;
}

// Example usage
const order1 = { total: 120, isInternational: false };
const order2 = { total: 80, isInternational: true };

console.log(calculateShipping(order1)); // Output: 0
console.log(calculateShipping(order2)); // Output: 15

At the start of the function, we include a comment explaining the rules for shipping costs. This gives the reader an overview of the logic without needing to read the full code.

// Shipping rules:
// - Free shipping for orders over $100
// - Standard shipping ($10) for orders below $100
// - Additional $5 for international orders

Then, the first condition checks if the order total is greater than or equal to $100. A comment here clarifies why free shipping is applied.

// Check if the order qualifies for free shipping
if (order.total >= 100) {
    shippingCost = 0; // Free shipping
}

The second condition applies an additional charge for international shipping. The comment explains why the extra cost is added.

// Add additional cost for international orders
if (order.isInternational) {
    shippingCost += 5;
}

Why are these comments good?

Imagine you’re working in a team of 20 developers. Someone reads the calculateShipping function six months later. Without these comments, they might waste time guessing why international orders have an extra fee. Good comments clarify the why and save hours of frustration.

🧩 Make Your Code Readable

If someone reading your code feels like they’re solving a riddle, you’ve already become a troublemaker. Here is the proof:

// Clean: Reads like a story
if (isLoggedIn) {
    console.log("Welcome!");
} else {
    console.log("Please log in.");
}

// Messy: Feels like chaos
if(isLoggedIn){console.log("Welcome!");}else{console.log("Please log in.");}

If your code is messy and hard to read, it will confuse others;and even yourself later! Think about coming back to your own code after six months and feeling like you’re reading a foreign language. Readable code saves time, reduces bugs, and makes everyone’s life easier.

🍵 Why is Readability Important?

For yourself: When you revisit your code after weeks or months, clean code helps you pick up where you left off without wasting time figuring out what you did.
For your team: If someone else reads your code, they shouldn’t have to solve a puzzle. Clean code makes teamwork smoother and prevents miscommunication.
Fewer bugs: Clear code is easier to debug because you can quickly spot mistakes.

🧙‍♂️ How to Write Readable Code

Let’s build a simple program to manage books in a library. We’ll make it clean and readable and then I will break down this code below:

// A class to represent a book
class Book {
    constructor(title, author, isAvailable) {
        this.title = title;
        this.author = author;
        this.isAvailable = isAvailable;
    }

    borrow() {
        if (this.isAvailable) {
            this.isAvailable = false;
            console.log(`You borrowed "${this.title}".`);
        } else {
            console.log(`Sorry, "${this.title}" is not available.`);
        }
    }

    returnBook() {
        this.isAvailable = true;
        console.log(`You returned "${this.title}".`);
    }
}

// A function to display available books
function displayAvailableBooks(books) {
    console.log("Available books:");
    books.forEach((book) => {
        if (book.isAvailable) {
            console.log(`- ${book.title} by ${book.author}`);
        }
    });
}

// Example usage
const book1 = new Book("The Clean Coder", "Robert Martin", true);
const book2 = new Book("You Don’t Know JS", "Kyle Simpson", false);
const book3 = new Book("Eloquent JavaScript", "Marijn Haverbeke", true);

const library = [book1, book2, book3];

displayAvailableBooks(library); // Show available books
book1.borrow(); // Borrow a book
displayAvailableBooks(library); // Show available books again
book1.returnBook(); // Return the book
displayAvailableBooks(library); // Final list

We created a Book class to represent each book. It has properties like title, author, and isAvailable to track its status.

The borrow method checks if the book is available. If yes, it marks it as unavailable and prints a message.
The returnBook method makes the book available again.
The displayAvailableBooks function loops through the library and prints only the books that are available.
We create three books (book1, book2, book3) and store them in a library array.
We borrow and return books, showing how the list of available books changes.

As you can see, readable code is not just about style. It saves time, prevents bugs, and preserves your code as useful for years to come.

🏌️ Test Everything You Write

If you don’t take the time to write tests, you shouldn’t be surprised if your code breaks. If you do want to write tests, follow this unit testing strategy to catch problems ahead.

What Is Unit Testing?

Concretely, unit testing checks individual parts of your code (like functions or classes) to verify that they work correctly. Just like checking each brick of your house for soundness before building the walls.

Let me give you an example of how unit testing works:

class Calculator {
    add(a, b) { return a + b; }
    subtract(a, b) { return a - b; }
}

// Test it (Unit Test)
const calculator = new Calculator();
console.assert(calculator.add(2, 3) === 5, "Addition failed");
console.assert(calculator.subtract(5, 3) === 2, "Subtraction failed");

Here’s what’s going on in this code:

First, we create the calculator class:

class Calculator {
    add(a, b) { return a + b; }
    subtract(a, b) { return a - b; }
}

The Calculator class has two methods: add and subtract.

add(a, b) takes two numbers and returns their sum.
subtract(a, b) takes two numbers and returns their difference.

Next, we set up the tests:

const calculator = new Calculator();

Here, we’re creating an instance of the Calculator class to test its methods.

Then we write test cases:

console.assert(calculator.add(2, 3) === 5, "Addition failed");
console.assert(calculator.subtract(5, 3) === 2, "Subtraction failed");

console.assert(condition, message) checks if the condition is true. If it’s false, the message ("Addition failed" or "Subtraction failed") is displayed in the console.

First test: calculator.add(2, 3) === 5
- Calls the add method with 2 and 3.
- Checks if the result is 5.
Second test: calculator.subtract(5, 3) === 2
- Calls the subtract method with 5 and 3.
- Checks if the result is 2.

So what happens if something breaks? It’s pretty simple to solve any issues that arise here. In this case, if the add or subtract method doesn’t work correctly, the test will fail. For example:

console.assert(calculator.add(2, 3) === 6, "Addition failed");

The condition calculator.add(2, 3) === 6 is false.
The console will display: "Addition failed".

Real-World Example: Testing a Login System 👥

Let’s test a simple login system to see how unit testing works in a real-world scenario.

class Auth {
    login(username, password) {
        return username === "admin" && password === "1234";
    }
}

// Test the Auth class
const auth = new Auth();
console.assert(auth.login("admin", "et5t45#@") === true, "Login failed for valid credentials");
console.assert(auth.login("user", "wrongpassword") === false, "Login succeeded for invalid credentials");

First, create the Auth class:

class Auth {
    login(username, password) {
        return username === "admin" && password === "1234";
    }
}

The login method checks if the username is "admin" and the password is "1234". If both match, it returns true – otherwise, false.

Next, set up the tests:

const auth = new Auth();

Create an instance of the Auth class. Then write the test cases:

console.assert(auth.login("admin", "1234") === true, "Login failed for valid credentials");
console.assert(auth.login("user", "wrongpassword") === false, "Login succeeded for invalid credentials");

First test: Checks if valid credentials ("admin", "1234") succeed. If not, "Login failed for valid credentials" is displayed.
Second test: Checks if invalid credentials ("user", "wrongpassword") fail. If not, "Login succeeded for invalid credentials" is displayed.

🌱 Why testing results in clean code:

You naturally write smaller, more focused functions to make your code testable
Tests verify that your code behaves as expected under different conditions.
With tests in place, you can confidently update your code, knowing the tests will catch any mistakes.

💉 Use Dependency Injection

Hardcoding dependencies is like tattooing someone’s name on your forehead — it’s permanent, can be abrasive, and locks you in.

So, what does Dependency Injection do? It lets you manage your code's relationships by passing dependencies as arguments. It’s flexible, adaptable, and maintainable.

To demonstrate how it works, here I’m using the Nodemailer dependency for sending emails to users:

// Dependency: Sending emails with Nodemailer
const nodemailer = require('nodemailer');
function sendEmail(to, subject, message) {
    const transporter = nodemailer.createTransport({ /* config */ });
    return transporter.sendMail({ from: "programmingwithshahan@gmail.com", to, subject, text: message });
}

⚠️ To save yourself from risk, make sure to avoid hardcoding dependencies. Use abstraction or configuration files for secure maintenance.

This is just one example. As a developer, you may use hundreds of libraries or dependencies.

I’m not saying you shouldn’t rely on dependencies/libraries at all, as nowadays it is hard to avoid them. But you should be very careful before installing them in your coding projects.

You should check the security, performance, quality, or functionality of an organization's software systems. Because they sometimes contain risks that can ruin your entire project.

🚧 Always control your tools, don't let them control you.

📂 Clean Project Structures

A well-organized project is the difference between a trash heap and a high-end boutique.

Here is how each folder should be organized:

If your codebase looks like a junk drawer, you’ve already caused trouble for your future self.

Let’s go through the clean project structure you can see above to better understand it:

1.myProjet/src

This is the main container for your entire application. Everything your app needs is stored inside this folder. It has subfolders to keep things tidy and managed in one place.

2.components

This is where you keep all the reusable pieces of your app. You can use these components in multiple places without building them again.

3.services

This is the "brain" of your app. It handles all the work behind the scenes for both the frontend and backend. emailService.js, userService.js and productService.js are some of the example files for your services folder.

4.utils

This contains all the small, handy tools you need to make your application run smoothly and make your life easier. For example, formatedate.js, validateEmail.js and generateId.js are some of the common utils files to make reusable pieces of components for your entire project.

5.`tests`

Conventionally, test files are typically located outside the src folder, at the project root level. This keeps your production code (src) separate from your test code (tests), making it cleaner and easier to manage. Have a look:

myProject/
├── src/              # Production code
│   ├── components/
│   ├── services/
│   └── utils/
├── tests/            # Test files
│   ├── components/
│   ├── services/
│   └── utils/
├── package.json      # Project configuration
└── README.md         # Documentation

Some developers may prefer creating one testing file inside the test folder to test everything in one place. Unfortunately, it may feel clean at first, but as your project grows, you’ll have to find and search for specific code blocks. It’s ugly and can produce unexpected testing results. So breaking them into multiple testing files inside the tests folder is highly recommended.

Real-world example 📧

So let me create a clean, durable project structure for you to apply in any future projects you might work on. Needless to say, clean project structure is the foundation of building a maintainable project.

From our previous email sending application example, we will write a clean project structure for this app. We want to build an application that sends emails to users. Your clean project structure for this app should look like this:

As you can see, I packed every subfolder and file inside the src folder which is the main container of our application. Inside the src folder, we created components, services, utiles. Finally, we have a manageable test folder outside the src folder to test each component independently. This test folder has nothing to do with our production code that is located inside the src folder.

🤹‍♂️ Be Consistent with Formatting

Don’t write code like you’re 10 different people. Be consistent with your formatting.

Use tools like Prettier or ESLint to enforce a consistent style. If every file looks different, you’re creating chaos that no one wants to fix.

I would say that consistency in formatting is one of the most important aspects of writing clean code.

Have a look...

// Always use 2 spaces for indentation
function calculateArea(width, height) {
  if (width <= 0 || height <= 0) {
    throw new Error("Dimensions must be positive numbers.");
  }
  return width * height;
}

// Add meaningful whitespace for readability
const rectangle = {
  width: 10,
  height: 20,
};

// Clear separation of logic
try {
  const area = calculateArea(rectangle.width, rectangle.height);
  console.log(`Area: ${area}`);
} catch (error) {
  console.error(error.message);
}

Let’s examine some of the aspects of this code that make it clean:

1️⃣ Consistent Indentation

Why 2 or 4 spaces? It’s clean, minimal, and universally accepted in many JavaScript style guides. It doesn’t overwhelm the eyes, and the code structure stands out clearly. When you mix inconsistent indentation (2 spaces here, 4 spaces there), you confuse people—and confused people make mistakes.

2️⃣ Meaningful Whitespace: Giving Code Room to Breathe

That extra line break between the rectangle definition and the try block is like a pause in a sentence—it gives the reader time to process.

3️⃣ Clear Separation of Logic: Modular Thinking

try {
  const area = calculateArea(rectangle.width, rectangle.height);
  console.log(`Area: ${area}`);
} catch (error) {
  console.error(error.message);
}

Look at how the logic is divided into clear sections:

First, the calculation (calculateArea function).
Then, the output (console.log).
Finally, error handling (catch block).

Each task has its own space and purpose.

4️⃣ Readable Error Handling

When you throw errors or log messages, you format them cleanly. No vague or cryptic messages here. A developer seeing this will immediately know the problem.

throw new Error("Dimensions must be positive numbers.");

🐦‍⬛ General tips for consistent formatting:

Use 2 or 4 spaces for indentation consistently throughout your codebase. Avoid tabs to maintain uniformity across different editors.
Keep lines to a maximum of 100-120 characters to prevent horizontal scrolling and improve readability.
Group related logic together and separate blocks of code with blank lines to highlight their purpose.
Finally, avoid over-aligning code. Instead, let indentation naturally guide the flow of logic.

✋ Stop Hardcoding Values

Hardcoding values is a lazy way to code. Here is the proof:

// Bad: Hardcoded and rigid
function createUser() {
    const maxUsers = 100;
    if (currentUsers >= maxUsers) throw "Too many users!";
}

// Clean: Dynamic and flexible
const MAX_USERS = 100;
function createUser() {
    if (currentUsers >= MAX_USERS) throw "Too many users!";
}

You see, changing this variable won’t surprise you in the future. You know exactly where to find it to change uncertain values.

Its best to store your fixed values in the global configuration (config) file.

🪧 So, avoid hardcoding values at all costs. Hardcoding is the shortcut that may drive your future self (or others) crazy.

🤏 Keep Functions Short

If your function is longer than 20 lines, it’s probably trying to do too much .

Short functions are sharp functions. They hit their mark every time.

Long, bloated functions are messy and hard to read, but short functions are clear and focused. Here is how your large functions should break down:

function updateCart(cart, item) {
    addItemToCart(cart, item);
    let total = calculateTotal(cart);
    logTransaction(item, total);
    return total;
}

function addItemToCart(cart, item) {
    cart.items.push(item);
}

Let me explain this code so you understand why breaking down large functions is a winning strategy.

The Main Function:updateCart() calls smaller helper functions to handle specific tasks like:

Adds the item to the cart.
Calculates the total price.
Logs the details of the transaction.
Finally, it returns the total price.

Instead of one long block of code that tries to do everything, it delegates tasks to helper functions.

Helper Function:addItemToCart() This function only handles adding the item to the cart. if you need to change how items are added (for example, checking for duplicates). You could just edit this small function instead of digging through a giant block of code in updateCart. That’s how you write clean code functions that’s a joy to read and easy to maintain.

What Happens If Functions Are Too Long? 💤

Let’s say you didn’t break down the updateCart function. Here’s what it might look like:

function updateCart(cart, item) {
    cart.items.push(item);
    let total = 0;
    for (let i = 0; i < cart.items.length; i++) {
        total += cart.items[i].price;
    }
    console.log(`Added ${item.name}. Total is now $${total}.`);
    return total;
}

What are the problems here?

It’s trying to do everything.
It’s hard to read, especially if it grows bigger.
If something breaks, you’ll waste time figuring out which part is the problem.

Now the choice is yours: stick with the messy all-in-one approach or practice the clean one function one job mindset.

⛺ Follow the Boy Scout Rule

Always leave your campsite cleaner than you found it.

Let me break it down for you. You don’t just use something and leave it worse than before. That’s inconsiderate behavior. Real professionals leave things better than they found them.

In coding terms, every time you touch the codebase, make it better. Clean it up, refactor messy parts, and improve readability. If you don’t, you’re just piling on garbage that will eventually collapse on your head.

Here is an example. Instead of improving it, we’re just adding more layers of complexity:

// Original code: Hard to read, poorly named variables
function calc(a, b) {
  let x = a + b;
  let y = x * 0.2;
  return y;
}

// We're adding to it but not cleaning it up
function calcDiscount(a, b, discountRate) {
  let total = calc(a, b);
  let final = total - discountRate;
  return final;
}

After: it gets better every time. Here’s how a disciplined coder works — they improve as they go:

// Improved code: Clear names, refactored for clarity
function calculateSubtotal(price, quantity) {
  return price * quantity;
}

function calculateDiscountedTotal(price, quantity, discountRate) {
  const subtotal = calculateSubtotal(price, quantity);
  const discount = subtotal * discountRate;
  return subtotal - discount;
}

Now, anyone can tell what’s happening at a glance. Because we’ve broken down the code into smaller, more focused functions. Thus, adding new features won’t break existing functionality. 🏕️

🏟️ Follow the Open/Closed Principle

This design principle suggests your code should be designed to allow extensions without changing the existing foundation.

You want to add features — not rip it apart every time you upgrade . Modifying old code to fit new requirements is exactly like trying to rebuild your house every time you buy new furniture. It’s not sustainable.

Let’s see how you can build smarter, scalable code that lets you add features without breaking everything else.

Before: Violating the principle

You’ve got a class to handle payments — simple enough. At first, it just handles credit cards.

But then your boss shows up and says, “Hey, now we need PayPal support.”

And because you didn’t bother learning clean code, your code looks like a spaghetti monster straight out of a legacy enterprise system from 1995. Here’s the masterpiece you’ve crafted:

class PaymentProcessor {
  processPayment(paymentType, amount) {
    if (paymentType === "creditCard") {
      console.log(`Processing credit card payment of $${amount}`);
    } else if (paymentType === "paypal") {
      console.log(`Processing PayPal payment of $${amount}`);
    } else {
      throw new Error("Unsupported payment type");
    }
  }
}

const paymentProcessor = new PaymentProcessor();
paymentProcessor.processPayment("creditCard", 100);
paymentProcessor.processPayment("paypal", 200);

Alas! Every new payment type (like Apple Pay, Google Pay, and so on) requires modifying the processPayment method. Needless to say, you risk breaking existing functionality while adding new features. If you had learned this principle, you might not be in this mess.

Don’t worry: I’ll help you to fix this. First, we need to refactor the code. Instead of modifying the existing class, we’ll extend its functionality using polymorphism:

javascriptCopy code// Base class
class PaymentProcessor {
  processPayment(amount) {
    throw new Error("processPayment() must be implemented");
  }
}

// Credit card payment
class CreditCardPayment extends PaymentProcessor {
  processPayment(amount) {
    console.log(`Processing credit card payment of $${amount}`);
  }
}

// PayPal payment
class PayPalPayment extends PaymentProcessor {
  processPayment(amount) {
    console.log(`Processing PayPal payment of $${amount}`);
  }
}

// Adding a new payment type? Just extend the class!
class ApplePayPayment extends PaymentProcessor {
  processPayment(amount) {
    console.log(`Processing Apple Pay payment of $${amount}`);
  }
}

// Usage
const payments = [\
  new CreditCardPayment(),\
  new PayPalPayment(),\
  new ApplePayPayment(),\
];

payments.forEach((payment) => payment.processPayment(100));

Now, adding new payment methods doesn’t require changing the existing PaymentProcessor class. You just created a new subclass. So the original code remains untouched, meaning there’s no risk of breaking existing features.

Each payment type has its own class, and adding PayPal payment support, for example, doesn’t break the code. Now you can reply to your boss: “Of course, I will add this feature in 5 minutes.” Your promotion is waiting for you to accept it.

I share even more tips in my book Clean Code Zero to One.

Modern Best Practices to Help You Write Clean Code: A Summary 🥷

Now let me show you the best practices and summarise our 12 Clean Code design principles to help you write clean code for agile application development.

🔎 Common Code Smells and How to Fix Them

💊 Duplication: If you're copying code, you’re creating more work for yourself. Extract it into a function, and do it right.
🛤️ Long methods: If your method needs a scroll bar, it's doing too much. Break it down, keep it focused.
👑 King objects: No class should be doing everything. Simplify responsibilities, or your codebase will become messy.

💬 Effective Commenting Practices

💭 When to comment: Only comment if the code isn't clear. If it is, comments are just clutter.
🫗 Clarity: Comments should tell why, not what. If your code needs explaining, it might be too complex.
🌴 Avoid redundancy: Don't comment what's obvious. If your function is addNumbers, don't comment it does that.

🧼 Refactoring Techniques for Clean Code

🏭 Extract methods: Big methods? Break them down. It's not just about cleanliness –– it's about control.
🫕Rename variables: If your variable names don’t shout their purpose, change and improve them. Precision in naming is precision in thought.
🍃 Simplify conditionals: If your conditionals look like algebra, simplify them. If a == true, just write if(a).

🧪 Testing and Clean Code

🧙 Unit tests: Test every piece of code like you're interrogating a suspect. No stone unturned.
🏇TDD (Test Driven Development): Write tests first. It's not just about catching bugs, it's about knowing exactly what your code should do before you write it.
🧽 Clean tests: Your tests should be as clean as your code. If they're messy, they’re not going to be helpful.

🐛 Error Handling and Clean Code

⁉️ Exceptions: Use them. They're not just for errors, they're also for keeping your code clean from error clutter.
🖍️ Fail fast: If something's wrong, stop right there. Don't let errors add up. Deal with them immediately.
🚨 Logging: Log like you're documenting a crime scene. Clear, precise, and only what's necessary.

🌱 Code Reviews and Clean Code

🚢 Process: Have a system. No cowboy coding. Review, critique, improve.
🔪 Tools: Use tools that make reviews easy. They're not just for catching mistakes, they're also for teaching discipline.
🧦 Culture: Cultivate a culture where feedback is gold. Help your team learn how to handle and receive critiques.

Automated Tools for Maintaining Clean Code ⚓

Tools and automation techniques can be really helpful in writing clean code. If you’re not using the right tools and automating things to save yourself time, you’re missing out.

You think you can "eyeball" your way through code quality? Guess again. Without automation, this is what happens:

👎 You miss obvious mistakes because you're "too busy."
🤕 Your code looks different in every file, making collaboration a headache.
🪦 Deployment breaks because you skipped a critical test.

Successful developers use the right tools to automate code and get things done. Here are four strategies for maintaining clean code using modern tools.

1️⃣ Static Analysis

Static analysis is actually a code inspector that reads through your code and points out potential issues early on. The best part? It works before runtime, catching errors that could otherwise lead to crashes, downtime, or embarrassing bugs.

How does it work?

Syntax checking: It looks at your code to analyze everything written in the correct syntax. If you misspell a variable or forget a closing bracket, it’ll call you out instantly.
Code quality rules: Tools like ESLint enforce rules like consistent indentation, avoiding unused variables, and sticking to best practices.
Error prevention: It identifies logic errors, such as using variables that haven’t been defined, or making comparisons that don’t make sense.

Here’s how static analysis works in action:

🚨 Before static analysis:

let sum = (a, b) => { return a + b; }
console.log(sume(2, 3)); // Typo, unnoticed until runtime

Problem: The typo in sume will only cause an error when the code runs, and that could lead to frustrating debugging sessions or worse — breaking the app in production.

🚑 After static analysis (using ESLint):

codeError: 'sume' is not defined.

Solution: ESLint immediately flags the typo before you even run the code. The error is caught early, saving you time and headaches.

2️⃣ Automated Code Formatting

Before Formatting:

function calculate ( x , y ){ return x+ y;}
console.log( calculate (2,3 ) )

Problem: Inconsistent spacing and formatting make the code harder to read.

After using Prettier:

function calculate(x, y) {
  return x + y;
}
console.log(calculate(2, 3));

Solution: Clean, consistent, and professional formatting is applied automatically. No more nitpicking over spaces or alignment.

Pretty basic stuff though. I covered this in case you write code in notepad or something where IDE is not provided (for example, a job interview).

3️⃣ Continuous Integration (CI) Testing

CI testing make sure every new change to your code is verified automatically. It’s like a safety net that catches bugs introduced during development. CI tools run your tests every time you push code, so nothing breaks after deployment.

How Does CI Testing Work?

Triggers on change: Each time code is committed, the CI tool (like GitHub Actions, Jenkins) runs automated tests.
Feedback: It gives you instant feedback if something fails.
Prevents broken code: Commits only clean, and the working code gets merged into the main branch.

4️⃣ CI/CD pipelines

We also use CI/CD pipelines as a continuous process that includes code building, testing, and deployment, while CI testing is a part of that process that focuses on automating the testing of code changes.

Differece between CI/CD pipelines vs CI testing:

CI/CD pipelines: A CI/CD pipeline combines code building, testing, and deployment into a single process. This process make sure that all changes to the main branch code are releasable to production. CI/CD pipelines can reduce deployment time, decrease costs, and improve team collaboration.
CI testing: CI testing is the process of automatically testing code changes that are integrated into a central repository. CI testing focuses on making sure the codebase is stable and that integration issues are resolved. CI testing help developer build software that is stable, bug-free, and meets functional requirements

🚧 This is what CI testing CI/CD pipelines concepts are really about. Not as complex as it seems. So let me elaborate more on CI testing with GitHub Actions, as we usually run tests through automated tools nowadays.

⚡ Continuous Integration (CI) Testing with GitHub Actions

As I said earlier, CI tools run automated tests every time you push code or open a pull request. This guarantees that only working, bug-free code gets merged into the main branch.

How to Set Up CI Testing with GitHub Actions

Step 1: Create Your Repository

Set up a GitHub repository for your project. Then, push your code to GitHub using the following commands:

git init
git add .
git commit -m "Initial commit for CI Testing"
git branch -M main
git remote add origin https://github.com/codewithshahan/codewithshahan.git
git push -u origin main

Or you can create a new repo from your GitHub account without using the command. Just log in to your GItHub account and visit dashboard. Here you will find a “New” button to create a brand new repo:

Step 2: Add a GitHub Actions Workflow

Navigate to your repository’s Actions tab. To do this, first, you have to visit your repo on Github (you will find the link after creating your repo). In this case, I created a new repo called “codewithshahan”. Here, look for the Actions tab on the right side of the navigation bar.

After navigating the Actions tab, scroll down a little and you will find the continuous integration section:

Choose a setup workflow yourself. I will use Node.js for this project.

After clicking the configure button, a node.js.yml file will be created automatically, and you can adjust the code depending on your goals.

I won’t go into detail about how should modify your .yml file. It depends on your project goals and personal preference. Also, it is a whole different broader topic and as this article has already become quite long, so I’ll explain it in a future article. For now, just stick with this foundational knowledge.

This CI Testing workflow is best for modern application development. Your app remains stable while incorporating key features including testing (e,g. Dark Mode), Building and deploying applications directly within your GitHub repository. This way, you can push your code confidently, knowing your code is always clean and ready for production.

The Role of Documentation in Agile Software Development 🚣

If you want your code to be top-notch, you need to understand how to write good documentation. If you think documentation is just about scribbling down how the code works, you’re wrong. It's about explaining why it works, not just how it works. That’s where most people miss the mark.

1. 🚡 Create Useful Docs (Explain Why, Not Just How)

When you write documentation, you're not just throwing down some instructions for how to use the code. You're telling the next person (or even yourself in the future) why this piece of code exists in the first place. That’s the difference between good and bad documentation.

Bad docs leave people scratching their heads. They’re too vague, too simple, and they don’t answer the big questions. If your documentation is unclear, that likely means your thinking is unclear. You’re basically saying, "I don’t care if you understand this, it works, just use it." That’s not helpful.

Great documentation answers the tough questions:

✅ Why did you choose this approach over another?
✅ Why does this function exist? What problem does it solve?
✅ Why did you write this code the way you did?

If your docs are just regurgitating how to use the code, you’re not being as helpful as you can be. Start thinking deeper and explaining the reasoning behind everything.

2. ⏳ Keep the Docs Updated (Outdated Docs Are Worse Than No Docs)

Outdated documentation is the worst. In fact, it can be worse than having no docs at all. When you leave documentation that’s out of sync with the code, you’re doing your future self — or whoever has to deal with it next — a huge disservice.

Every time your code changes, your documentation needs to change too. It has to reflect the current state of things. Don’t mislead future developers (or yourself) by leaving outdated info that will only confuse them and waste their time. If something is no longer relevant, delete it. Outdated documentation is the equivalent of a cluttered mind — it holds you back.

Make it a habit to check and update your documentation regularly. The minute a line of code changes, so should your documentation. Period.

3. 🚆 Integrate Comments (Good Comments in Code Are Part of Documentation)

Here’s the deal — comments in your code should integrate with your documentation. Good comments aren’t just a crutch for developers who can’t explain their code elsewhere. They should improve your docs, not replace them.

Comments are supplements to your documentation. You write clean, understandable code that needs minimal explanation, but when something isn’t crystal clear, throw in a comment. Remember the rule for comments in your code: explain the why, not the how. It’s the same here. Don’t repeat yourself. Let your code do the talking. Comments should complement the bigger picture of your documentation, not act as a band-aid for sloppy code.

🪧 Great code should be self-explanatory. Fix the code, then add comments for clarification if necessary. Keep comments clean, short, and to the point.

If you want to write clean, efficient, and maintainable code, documentation is key. Stop thinking of docs as an afterthought or something you do to fill space. It’s an extension of your code — your way of communicating clearly and effectively. It’s your roadmap for future developers, and it’s a reflection of your thought process.

Conclusion 🏁

Clean code isn't a nice-to-have –– it's a must-have for those who aim to lead. It's about control, efficiency, and improvement over time in the long run. And ultimately, it’ll help you succeed in the game of agile software development.

🪧 If you want to truly master your craftsmanship, write clean code, and let the efficiency speak for itself.

Frequently Asked Questions About Clean Code 🧯

What is clean code? It's code that doesn't make you want to throw your laptop out the window.
Why is clean code important in Agile? Because Agile is about speed and change, and you can't be quick with a mess.
What are code smells? Signs that you're about to lose control of your codebase.
How can I improve commenting? Only comment on what's necessary, and make sure each comment adds value, not noise.

Thank you for being with me.

If you’re serious about learning clean code and taking your programming career to the next level, then this book is for you: Clean Code Zero to One. Grab this and master the art of programming. After reading this book you can pick any programming language you like and build software without facing countless bugs using AI agents:

40% discount using the code "earlybird", only for the first 100 copies. Plus, there’s a 30-day money-back guarantee. no risk, all reward.

📘: https://codewithshahan.gumroad.com/l/cleancode-zero-to-one

Anthropic is paying $570k for engineers..why not just use Claude Code? 🤔

Programming with Shahan — Sun, 26 Apr 2026 11:36:11 +0000

Anthropic is paying $570k for engineers

why not just use Claude Code? 🤔

dev job market is insane in 2026!

Programming with Shahan — Wed, 22 Apr 2026 10:06:35 +0000

Learning to code in 2026 is not what it used to be.

And if you still think it is

You are already behind.

And the way you become a developer?

It's completely different.

You may ask: "should I even learn to code in 2026?"

Yes.

100% yes.

Because some people think AI replaced programmers.

It didn’t.

It removed bad programmers.

And increased demand for REAL BUILDERS.

This is still one of the highest-leverage skills.

Plus the software engineer (SWE) job market is picking up in 2026.

SignalFire reported that firms like Meta, Netflix, Uber, and Google are hiring engineers FASTER than people are leaving:

The US Bureau of Labor Statistics still expects software developer jobs to grow by about 15%, which is way faster than most other careers:

Moreover, 72% of organizations outsource software development to achieve better access to talent [32%], meet increasing customer demands more efficiently [35%], optimize spending [34%], and improve quality or performance [33%]:

Meanwhile, Python is the most sought-after programming language, with 45.7% of recruiters seeking candidates with Python skills, followed by JavaScript [41.5%], Java [39.5%], TypeScript [27.9%], and C# [24.4%].

according to devjobsscanner Python is the most in-demand language as of January 2026 with 43,300+ posted job openings:

AND

Gartner predicts that by 2027, 80% of the engineering workforce will need to upskill to keep pace with generative AI.

so if you can code in 2026 you certainly can

📲 build complex application
😴 automate your entire projects
💼 land your dream job
🦮 and instruct AI instead of being replaced by it

The good news is: learning to code today is much easier than ever before.

So in this article, I will show you:

1. 🍱 what coding looks like now in 2026
2. 🏫 what you MUST learn for the future
3. ⚡ AND how to speed up your learning massively

Let's jump in.

💊TABLE OF CONTENTS:

FIRST: Why "Vibe coding" to "Agentic Engineering" shift happened?
🍼 STEP 1: Coding Fundamentals (non-negotiable)
- Language choice
🍌 STEP 2: Software Architecture
🧪 STEP 3: Testing
🛫 STEP 4: Deployment
🛩️🎮 STEP 5: Version Control
🔐 STEP 6: Security & Privacy
🚢 STEP 7: Microservices & Containers
💪 NOW you are ready for AI coding
- ⚡ The new coding shift: Agentic Engineering
🧠 How to learn code faster
- 🤔 1. Use AI to THINK, not just code
- 🔍 2. Reverse engineer code
- 🏗️ 3. Projects = everything
- 🧠 4. Ask AI for multiple solutions
- 📄 5. Documentation shortcut
📚 Recommendation
🧭 Final thoughts

FIRST: Why "Vibe coding" to "Agentic Engineering" shift happened?

In 2025, Andrej Karpathy introduced something called

👉 “Vibe coding”

// Detect dark theme var iframe = document.getElementById('tweet-1886192184808149383-130'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1886192184808149383&theme=dark" }

It meant

🤖 you talk to AI
🍎 you let it generate code
🥱 you don’t care about details

It worked for small projects.

It was fun.

But it was never serious engineering.

Then things changed in 2026.

By 2026, a new workflow became normal:

👉 Agentic Engineering

This hits different.

Now you don’t just “talk to AI”

You:

🤹 manage AI coding agents
🏗️ structure systems
🦮 guide architecture
🪸 correct mistakes
🛫 control outputs like a system designer

You are no longer a coder.

You are now a Chief of Commander of machine.

But here’s the trap:

You can’t command what you don’t understand.

So fundamentals still rule everything.

🍼 STEP 1: Coding Fundamentals (non-negotiable)

Before AI, coding fundamentals was everything.

Now?

Still everything.

You must learn:

[variables]
[data types]
conditions (if/else)
loops
[functions]
object-oriented programming
APIs
privacy & security
clean code

Because if you cannot read, scale or maintain code

You CAN'T control AI-generated code.

You become dependent.

Your software becomes useless.

Language choice

You should pick only one tech field first and go deep.

To begin with:

learn Python for AI, automation, data systems

learn JavaScript for web/mobile apps, frontend/backend

Pick ONE. Don’t jump.

🍌 STEP 2: Software Architecture

This is where beginners quit.

And professionals are built.

You must understand:

how software is structured
how systems are designed (clean code)
how tech stacks are chosen
how APIs connect systems
how data flows
where data is stored (databases)

Now think: you want to build an app.

You must answer these questions:

Where does data go?
How does it scale?
How do users interact with it?
How fast is it under load?

These are NOT coding questions.

These are engineering decisions

And AI cannot reliably decide them for you.

🧪 STEP 3: Testing

AI can write code.

But AI also makes mistakes silently.

That’s dangerous.

So you must learn:

🍽️ how to write test cases
🍳 how to verify logic
🪠 how to validate features
🫗 how to catch failures early

Concretely, without testing you are guessing.

With testing:

You are controlling your output.

🛫 STEP 4: Deployment

Code that ONLY runs locally is useless.

learn:

🪂 how to host applications
🗼 how servers work
🪟 how environments are structured
🪈 how deployment pipelines function

Job recruiters often ask about it.

You must ship to create history.

You must understand and explain WHY things does/doesn't work.

🛩️🎮 STEP 5: Version Control

Version control (e,g. Git) is not optional. Its the baseline.

You need it even if AI is knocking on your door.

Learn what version control does and master the most useful commands.

You will thank me later.

Now in AI-driven coding

This becomes even more important.

Because AI can break your project while you blink.

Git or version control system will help you tackle it.

🔐 STEP 6: Security & Privacy

This is where AI fails the most.

AI acts as a child to "security and privacy"

And that's where beginners get destroyed without knowing.

So learn

🥷 authentication
👨‍🦯‍➡️ authorization
🛟 data safety
🗝️ secure APIs
🔏 secret management

AI does NOT care about security unless you force it to.

And real engineers know this:

"Paranoia is not fear. it is protection."

Which means build a proactive security-first mindset where you assume threats exist and systems will fail.

🚢 STEP 7: Microservices & Containers

This is advanced.

But important.

You should learn:

-0 how applications are packaged
-01 how dependencies are bundled
-02 how systems run in isolation
-04 how environments stay consistent

Why does this matter now?

Because AI agents can break your projects, break environments and create unpredictable setups

Containers (e.g., Docker) protect you from drowning.

💪 NOW you are ready for AI coding

Only after all this…

You use AI properly.

But as a tool.

Now you can

📢 direct AI agents
🍔 build systems faster
🤜 correct mistakes
🏠 design architecture

⚡ The new coding shift: Agentic Engineering

Now coding looks like this:

You open your machine.

Multiple AI agents working:

One builds backend. One writes tests. One handles docs

You assign tasks, correct mistakes, guide structure, AND enforce logic.

This is not coding anymore.

This is agentic engineering.

Make sure to learn it early.

🧠 How to learn code faster

Most people fail here because they consume too much content.

or learn but never build

Before, it took 2–5 years to become solid developer

Now ~6 months if you are focused.

Because tools will multiply your output.

So here is the solid strategy to learn code faster:

🤔 1. Use AI to THINK, not just code

Don’t ask:

“write code”

Ask:

"explain this"
"break it down"
"give analogies"
"show mistakes"

🔍 2. Reverse engineer code

Take real projects.

Then: break them, modify them and ask AI to explain them

That is actual serious learning.

🏗️ 3. Projects = everything

No projects = no skill.

You must:

☑️ build APIs
☑️ build apps
☑️ build features
☑️ break and fix things

That’s how neural memory forms.

Reading or watching tutorials is another word for procrastination.

🧠 4. Ask AI for multiple solutions

NEVER accept one answer.

Ask:

“give me 3 approaches”

this way you'll learn: tradeoffs. efficiency. structure

📄 5. Documentation shortcut

Nobody likes reading documentation.

Don’t struggle through docs.

Use AI to translate them.

But still understand them.

📚 Recommendation

If you want a clean, structured path:

📘 My Clean Code Zero to One book is for you:

After reading this book you'll think like a system designer. Write clean architecture. Scale and maintain your application and avoid beginner's mistakes.

🧭 Final thoughts

Coding in 2026 is easy.

BUT its not "too easy" as 63% of non-developers so called "vibe coders" think. They all face the reality when seriously get involved in buidling softwares.

either you learn fundamentals

Or you skip fundamentals and get replaced.

There is no middle.

Now open your code editor and build something cool from scratch! 🥂

You can follow me on LinkedIn. Thanks.

learn these 12 git commands and forget

Programming with Shahan — Fri, 17 Apr 2026 22:31:10 +0000

🌱 Introduction

Most devs think they can vibe code with a few Git commands.
They run git add ., "push commit" … and start acting like they built Linux.

I’ve seen the opposite.

they trust AI agents and then the project turns into something nobody wants to touch.

I’ve done that too, early on.

The difference is that you eventually get tired of fixing avoidable problems. That’s when Git stops being a checklist of commands and starts becoming a way of working.

These 12 git commands I actually rely on when things get messy or when I want to keep things clean from the start.

now lets get started.

🎂 1) Rebasing

AI code reviewers and merge queues DO NOT babysit premature branches.

They expect discipline.

most so called git expert stack commits like this:

fix
fix again
testing
oops
final-final-v3

Then they open a PR (pull request) and hope it works.

Unfortunately it doesn’t.

🍄‍🟫 What rebase actually is

Look at the image.

Bottom = old commits
Top = latest main
Dog = your feature branch

Right now your branch is behind.

The dog is sitting somewhere in the middle.

But he wants the top.

So what does he do?

He cLimbs.

Step by step.

And then he sits on top of the latest main.

That’s rebase.

You don’t build sideways.
You don’t stay behind.

You move your work to the top.

Clean. Straight. Done.

🤔 You ask why?

Look, modern teams use AI merge queues, automated reviewers, async PR pipelines.

These systems are NOT your mother.

THey don’t review your failure.

AI pipelines don’t fix your alienated history.

They see this:

random commits
unclear changes
outdated branch

🙅‍♂️ And they reject it.

Not because you’re bad.

Because your git history is turmoil. 🧐🌡️

WitH rebase you fix everything before PR.

Now merge is simple.

🍨 Rebase workflow

This is what you actually do:

git switch feature-login
git fetch origin
git rebase origin/main

git add .
git rebase --continue

git push --force-with-lease

🏗️ Code demonstrations for beginners:

1) 🧗‍♂️ Switch to your feature branch = git switch feature-login

You move into your own work area.

2) ⛅ Get latest updates from remote = git fetch origin

You look at what changed on the server using fetch.

3) 🛟 Rebuild your work on top of latest main = git rebase origin/main

This is the core moment.

Git takes your commits and replays them one by one on top of the newest main.

So instead of:

messy branch history
outdated base

You now get:

clean linear history
same work, new foundation

If conflicts happen, Git stops here.

4) 🚤 Fix conflicts (if any)

git add .
git rebase --continue

This means:

you fix the files
you tell Git: “okay continue”

Git keeps replaying your commits until everything is applied.

5) Push updated history = git push --force-with-lease

Important part:

You are not just pushing code.

You are rewriting history.

So Git says:

“Only push if nobody else changed this branch while you were working.”

That’s what --force-with-lease protects you from.

It checks if someone else updated the branch.

If yes, it stops you.

So you don’t destroy their work.

again, rebase changes history.

That’s why you don’t just --force push blindly.

🪦 2) Reset, Revert, Restore

Mistakes will happen.

That’s normal.

Panicking is optional.

Some developers break things… then freeze.

Others fix it in 10 seconds and move on.

Read Clean Code Zero to One and become like them.

The difference is knowing which command to use.

Git gives you three weapons to reverse damage:

git revert HEAD 
git reset --hard HEAD~1
git restore --staged app.js
git restore app.js

[ code demonstration is below ]

1) 🦺 git revert | safe undo

You don’t delete anything.

git revert HEAD = Makes a new commit that cancels the last one.

History stays clean.

Nothing is hidden.

This is what serious teams want.

2) 🗑️ git reset | delete it completely

This one is different.

It moves your branch back.

The commit is gone.

Like it never happened.

git reset --hard HEAD~1 = Deletes the last commit and your changes.

Fast. Clean. But dangerous.

If someone already pulled that commit… you just created a big mess.

Use it when you’re alone on the branch.

3) 🗃️ git restore | fix files only

This doesn’t touch history.

It just fixes your working files.

git restore --staged app.js = Removes app.js from staging (undo git add). Your edits stay untouched.

git restore app.js = Deletes your changes. File goes back to last commit.

🧠 Concretely:
revert → undo safely
reset → erase history
restore → fix files

Three different jobs.

Where people mess up?

They use reset when they should use revert.

Now history is broken.

Team is confused.

Time is wasted.

🦐 2026 RRR Best Practices

git switch and git restore replaced the overloaded confusing git checkout. This isn’t optional anymore.

Modern teams would love to see you using them.

Old messy git checkout command did everything:
Switch branches. Restore files. Confuse everyone.

Now it’s split into two:

git switch: move between branches
git restore: fix files / undo file changes

ITs much cleaner and accurate.

That’s why you will never see me use checkout in this article.

It’s outdated.

Just like a serious dev who refuses to shallow AI hypes.

I will show you EXACTLY how they work in the next section (#3).

🫰3) Switch & Restore

git switch main
git switch -c feature-payment

git restore README.md
git restore --staged utils.js

👉 `git switch main`

with that, you leave your current branch and return to mission control.

Before:
You’re on: bugfix/login-issue
You’ve got random work-in-progress edits.

After running:

git switch main

You are now on the main branch.

Your files instantly match the main branch’s state.

✍️ NOTE: Uncommitted edits may follow, unless switching would create conflicts, in which case Git stops you to avoid damage. That's protection.

either way, you’ve returned to home base. You’re back to clean ground..

👉 `git switch -c feature-payment`

Result:
A new branch is created which is feature-payment
And you immediately enter that branch.

Now your payment feature lives in isolation.

No risk of breaking main.
No overlap with other features.

🪜 One branch per mission.

👉 `git restore --staged utils.js`

This is your emergency fix when you accidentally run:

git add .

and stage files you didn’t want to include.

Fix it:


git restore --staged utils.js

Result:

utils.js is unstaged which is removed from the commit list.

But your edits remain safely in your working directory.

which means, you fix the mistake without losing a single line.

🐾 4) --amend Commits [Fix Without Noise]

You forgot a file?
You wrote a typo?
You missed a small edit?

You --amend.

🧱 What “--amend” actually DOES?

Amend means:

“Take my LAST commit and update it as if the mistake NEVER existed.”

It does NOT create a new commit. It replaces the previous one.

But most commits like this:

“fix again”
“final fix”
“real final fix”

Now history looks like garbage.

No worries.. you can time travel and --amend.

Real example (this happens all the time)

You accidentally committed .env

Sensitive keys. Big mistake.

You don’t say:

“oops let me fix it in next commit”

No.

You clean it properly:

git rm --cached .env
echo ".env" >> .gitignore
git commit --amend --no-edit

Done.

That commit never had .env.

⏹️ You remove the sensitive file
and rewrite the last commit like it never existed.

Let me explain..

git rm --cached .env 👉 This removes .env from Git tracking (not your computer)

Then:

echo ".env" >> .gitignore 👉 So you don’t repeat the same mistake

Now your changes are staged.

THEN you run:

git commit --amend --no-edit

🫡 This rewrites the last commit
👉 Same message
🦵 But WITHOUT the .env file

Why --no-edit?

Because you don’t want to waste time rewriting the message.

You’re saying:

“Same commit message. Just fix the mistake.”

🤔 What if you WANT to edit?

Then DO NOT use --no-edit.

Just do:

git commit --amend

Now Git opens the editor.

You can fix the message, explain what changed and make it more professional

Now let me illustrate an everyday use case scenario:

git commit --amend
git rebase -i HEAD~5
--amend → fix your last mistake instantly
rebase -i → squash, reorder, clean commits like a surgeon

This is how you turn:

“fixed stuff lol”

into [for example]:

“Refactor auth flow + fix token expiry bug”

Now your history makes sense.

⚠️ WARNING (read twice):

Only --amend local commits.

Never --amend commits already pushed to a team branch.

Why?

Because you rewrite history.

Automated pipelines, merge queues and AI agents HATE rewritten history.

🥍 Push amended commits ONLY if you're 100% sure NO ONE else has pulled them.

🏫 5) Git Log / Learn From History

Git doesn’t lie.

Your code can lie. Your memory can lie. Your teammates can lie.

nowadays devs zombie scroll through Git logs.

They don’t understand the POWER hidden in "history".

Good history = easy debugging
Bad history = endless suffering

⚖️ The 50/72 rule

Follow 50/72 Rule for efficient bug tracing AND debugging.

keep titles under 50 characters and wrap descriptions at 72.

write messages like "Add feature" instead of "Added feature".

and do not mix code refactors with new features in the same commit.

Bad example:

commit 1: "fixed stuff"
commit 2: "final update
commit 3: "changes lol.."

Good example:

commit 1: "fix(auth): resolve token expiry issue"

Now you know what happened without opening code.

Basic survival commands

git log --oneline
git log --oneline --graph --decorate
git log --oneline --author="Shahan"
git log --oneline -- path/to/file

What they really mean:

--oneline = fast history scan + short, readable commits
--graph = see branch chaos visually
--graph --decorate → visualize merges & branches
--author = see who broke what
-- path/to/file = see only a single file’s history

This is how you stop guessing.

When things get serious you type
git log -p
git log -S "functionName"
git blame file.js
git reflog

Now you’re not reading history.

You’re hunting problems.

-p shows exact code changes
-S finds when a bug was introduced
git blame shows who touched each line
git reflog brings back “lost” commits

6) 🪝 Git Hooks to Automate Discipline

Junior devs hope code is fine.

Serious devs don’t hope anything.

They build systems that refuse bad code.

That’s what Git hooks are.

Hooks are simple:

They run automatically when you commit, push, or merge.

Without asking. Without reminders.

If something is wrong they stop you.

The idea is simple:

You try to commit.

Git says:

“Wait. Let me check that first.”

🔧 example of git hooks pre-commit (gatekeeper)

You create this file:

.git/hooks/pre-commit

Now you add rules:

#!/usr/bin/env bash

echo "Running tests..."

npm test || {
  echo "❌ Tests failed. Commit blocked."
  exit 1
}

Then activate it:

chmod +x .git/hooks/pre-commit

What just happened?

You didn’t “add a tool.”

You added a guard at the door.

Now:

tests run automatically
failure = commit rejected
broken code never enters history

Therefore, there's NO debate anymore.
No “I’ll fix it later” nap issue.

🪫2026 Git Hooks Reality

Nowadays most teams don’t hand-write hooks anymore.

They use tools:

they use Husky to manage hooks cleanly

or lint-staged to run checks only on changed files

Because raw hooks don’t scale well.

Tools make it consistent.

Important truth most tutorials skip

Local hooks are not secure.

Anyone can bypass them:

git commit --no-verify

And everything gets ignored.
So if you rely ONLY on local hooks you're mistaken.

So hooks alone are not enough.
Let me explain why..

Real system (what professionals use):

You combine two layers:

1. Local hooks (Fast feedback. Instant rejection.)

2. CI/CD pipelines (GitHub Actions, etc.)

SO
⚡Local = speed
⚖️ Server = enforcement

Local catches mistakes early.
Server enforces rules for everyone.

Together,nothing broken gets through. YOU WIN.

🍋‍🟩🪞 7) Commit References

beginners copy full commit hashes like this:

a3f9c1d8b7e...

Long. Ugly. Useless in daily work.

You don’t need that noise.

🔧 Core Commands

git diff HEAD~2..HEAD
git rev-parse --short HEAD
git tag -a v2.0 -m "Release v2.0"
git push origin v2.0

-- HEAD~2..HEAD = compare recent changes

-- rev-parse --short HEAD = quick reference (human-friendly ID)

-- tag -a = mark a stable checkpoint (this matters)

-- push origin v2.0 = share that checkpoint with the world

🧠 Understand this properly

HEAD = where you are
HEAD~1 = one step back
HEAD~2 = two steps back

You’re not memorizing hashes anymore.

You’re jumping up/down in a timeline. its super productive.

🆔 Why Tags Actually Matter??

Tags are not decoration.

They are anchors.

When you say:

git tag -a v2.0 -m "Release v2.0"

You’re declaring:

“This version is stable. This is deployable.”

Now your CI/CD pipelines + Deployment systems and AI tools

All know exactly what to ship.

NOTE: ⚠️ Fix this weak idea

“Full hashes are sh$t”

No.

Blind usage is actual sh$t.

Full hashes are for finding exact location, scripting and debugging critical issues,

Short refs are for speed.

Know both.

8) 🧳Stash [Pause Work Without Panic]

Interruptions are coming.

Boss calls. Bug report drops. Production is on fire.

And you’re sitting there with half-written code…

Now what?

Commit garbage?
Lose your work?
Panic?

NO.. professionals don’t panic

They stash.

What stash actually is?

Stash is simple:

Save everything I’m doing right now… and bring me back to a clean state.

No commit. No history pollution.

Just pause.

EXAMPLE:

Let say you’re working on a feature.

Suddenly you recieved a message from colleage:

“URGENT: Fix login bug NOW.”

You don’t argue.

You run:

git stash push -m "WIP: profile fix"

Everything disappears.

Not deleted.

Just stored safely.

Your workspace becomes clean again.

Now switch safely

git switch main
git pull

Now you fix the urgent issue without your half-work getting in the way.

See what you saved

git stash list

It’s like a small vault of paused work.

Nothing lost. Just parked.

Bring it back

When you’re ready:

git stash apply

Your work comes back exactly as it was.

Or:

git stash pop

Same thing, but removes it from stash after restoring.

Advanced move

git stash branch hotfix-auth stash@{0}

This does something smarter:

creates a new branch
restores your work there
keeps main branch clean

So you don’t mix emergency fixes with unfinished features.

No conflict with main work.

⚠️🛩️ Reality Check

Stash is NOT permanent storage.

It’s temporary.

Forget it… and it becomes a graveyard.🪦

So use it. Don’t hoard it.

When to use stash

Use it when:

you saves unfinished work safely and then switch tasks instantly

anytime create a branch from stash (stash branch) for experiment

In summary.. need to switch tasks mid-flow? Stash temporarily saves your changes.

9) Bisect 🐞

Manual bug hunting? WEAK MOVE.

Manual bug hunting is slow.

And slow means you suffer.

Bisect = sniper rifle for bug hunting.

Think now: something breaks in production.

No logs. No clear reason.

Only one truth:

It worked yesterday. It’s broken today.

And now you have 50+ commits in between.

Amateurs go to sleep.

Professionals starts hunting.

So what's the solution? you divide the problem.

you use bisect:

git bisect start      # Begin the hunt
git bisect bad        # Current version is broken
git bisect good v2.1  # Last known good version

How the process works

You test the code.

Then you answer only two things:

works = git bisect good
broken = git bisect bad

That’s it.

Git keeps cutting the search space in half.

Again. And again.

Why this is powerful?

50 commits becomes

~6 checks.

its called the power of binary search.

🔥 Now Finish the Bug Hunting mission

When the bug is found:

git bisect reset

Back to normal.

Bug identified.

Target eliminated. ☠️

💡Tips:

You can automate the entire hunt:

git bisect run npm test

Now Git runs your tests automatically on each step.

No manual checking.

No human error.

You just sit back and watch it find the culprit like quillbot ;).

NOTE:

Bisect only works if:

you hAve a known good version
your tests actually work

If your tests are broken, bisect becomes useless.

10) 🧑‍🦯‍➡️ Don’t Merge Blindly

You’ve got a feature-cat branch.

It “works on your machine.”

Now you’re about to merge into main.

And you’re thinking:

“Should be fine.”

That sentence along has destroyed more codebases than bugs ever did.

Rule #1) Check What files are changing

git diff --name-only main..feature-cat

This is your first reality check.

It shows exactly what will be affected.

Rule #2) What actually changed

git diff --color-words main..feature-cat

Now you see the real story.

Not “files changed”.

But what inside those files changed.

This is where bad merges get exposed.

Two ways to compare branches:

git diff main..feature-cat
git diff main...feature-cat

Two-dot vs Three-dot:

-- .. = direct comparison (simple view)
-- ... =true branch difference (REAL merge view)

Most people never learn this difference.

And that’s exactly why they merge wrong.

Before merge, you don’t ask:

“Will this work?”

You ask:

“What exactly am I about to change?”

To sum up, know the battlefield before the fight.

Compare first, merge second.

🫳 11) Cherry-Pick (pick ONLY What You Need)

Sometimes you only need ONE commit from another branch.

You Merging entire branches for one fix?

That’s how you import bugs you NEVER asked for.

lets cherry-pick..

Real Case scenario:

Your feature-login branch has 10 commits:

LoginUI.tsx redesign ❌
authStyles.css change UI ❌
useAuth.ts refactor ❌
✅ Fix: token refresh bug (authService.ts)

Production is breaking.

Users getting logged out every 5 minutes.

You don’t need UI changes.
You don't need refactor.

BUT you need that ONE fix: token refresh bug in authService.ts

💆 Step 1: Find the exact commit

git log --oneline feature-login

You see:

a1b2c3d fix: refresh token expiry bug in authService
f4e5d6c refactor: auth hooks
9a8b7c6 style: login UI update

You take the one that matters: 👉 a1b2c3d

🩺 Step 2: Surgical extraction

git checkout main
git pull
git cherry-pick a1b2c3d

Now only this change is applied:

authService.ts → FIXED
Everything else → untouched

No UI risk. No side effects.

🔨 Step 3: Clean it like a pro

git cherry-pick -e a1b2c3d

Update message:

fix(auth): resolve token refresh logout issue (hotfix)

Now your history makes sense to your team + your CI/CD pipeline

🙎‍♂️What just happened after that 3 steps workflow??

-- Bug reported in production
-- You located the fix commit
-- You shipped ONLY the fix
-- You avoided merging unstable work

This is how senior SWE teams move fast without breaking things.

But hey.. this is also where juniors get destroyed

Cherry-pick can bite you if you have:

-- Same fix exists in two branches later
-- Merge conflicts when syncing branches
-- Duplicate commits + confusing history

You didn’t remove the problem.

You copied the solution.

Big difference.

🧠 So when cherry-pick is the RIGHT move?

Use cherry-pick when:

🚨 Production hotfix
🎯 One isolated commit
⛔ Full merge is risky

Avoid when:

🌆 You need long-term branch sync
♾️ You’re patching randomly without plan

🧱 12) Git Add

If you think git mistakes happen during commits.

You're wrong.

They happen before that.

At staging.

You worked on:

login.js
profile.js
utils.js

Only login.js is ready.

But only amateurs do this:

git add .

And suddenly:

/ half-finished code enters history
/ unrelated changes get bundled
/ reviews become useless
/ AI gets confused
/ debugging becomes harder later

That’s how conflicts starts.

🧑‍⚕️Professional workflow

You don’t git add . everything.

You choose what deserves to exist in history.

git add -p login.js

👉 Stage only specific parts of a file
👉 Build clean, intentional commits

🔍 Preview before committing

git add -n .

👉 Shows what would be staged

⚠️ Advanced control (use carefully)

git add --force .env

👉 Used only when you intentionally need ignored files
👉 Dangerous if you don’t understand why

Staging is not “pre-commit.”

It is decision-making.

You are deciding:

“What deserves to become history?”

✍️ NOTE:

Amateurs commit everything and fix later. Professionals stage with intention and never create havoc in the first place.

Conclusion 🏁

Git is not complicated.

Most of the commands above aren’t even complicated.

They just force you to slow down at the right moments.

It becomes messy when you stop paying attention.

After a while, you stop thinking in commands and start thinking in states:

what changed, what matters, and what should actually be saved.

That’s basically it.

Hope you find this article useful. let me know in the comments section what you have learned so far. Its great to hear. Cheers 🥂

📘 Recommendations: Clean Code Zero to One

Git is a tool. it handles history. Clean Code handles everything inside that history. Developers don’t fail because of Git. They fail because the code itself becomes way too HARD to manage. If you fix that part early, Git becomes your oblige coworker.

use code: POWER40 40% OFF for one week.

👉 Clean Code Zero to One

for those building apps Bright Data takes care of your data: proxies, scraping, data pipelines, all handled. You build. It runs on your behalf.

10 hard truths EVERY DEV must learn in 2025

Programming with Shahan — Wed, 09 Apr 2025 17:26:43 +0000

I've been in the coding game for 5 years.

Written messy code.

Struggled with debugging nightmares.

Made dumb mistakes.

Learned the hard way.

Here are the cold, hard lessons I had to learn the hard way (every developer must learn)...

1. Looks help but they’re not everything.

I remember when I first started using AI code assistants. I thought once I had them, coding would be effortless.

They provided suggestions, completed lines... but my projects still had issues. Because I still lacked the understanding of clean code principles. AI tools got me started, but my lack of foundational knowledge closed doors just as fast.

What I learned is this: AI tools don’t just respond to what you input; they amplify your existing coding habits. A developer who codes with clarity, who structures functions with purpose, who doesn’t rely solely on AI -- that developer will outperform someone who blindly follows AI suggestions every time.

2. If you’re always chasing, you’re not in control.

This one stung. I used to think effort was everything. If I just kept tweaking and adding features, the code would improve. So, I’d refactor endlessly, integrate every new library, follow every trend... and I ended up with bloated, unmanageable codebases.

Because when you’re doing 90% of the work without a clear direction, you’re not developing; you’re floundering. Real progress is intentional. If a feature truly adds value, it will integrate seamlessly. Stop chasing every new tool or framework that barely fits your project. Start valuing simplicity and purpose.

3. Your purpose is more attractive than your tools.

I had phases where I’d focus all my energy on mastering the latest coding tools. Everything revolved around having the newest gadgets in my developer toolkit. I’d spend hours customizing my IDE, setting up complex workflows... and I got nowhere.

But when I locked in on my goals—building meaningful projects, solving real problems—when I was too busy creating to constantly tweak my setup, suddenly my work stood out.

Clients and peers would say, “You’re different,” “You’re focused,” “You’ve got something going.” That’s when I realized: developers who chase purpose attract recognition without even trying. Because that energy is rare, and people feel it.

4. I’ve learned to stop coding so much.

Early on, I thought I had to prove myself. I’d over-engineer. Add unnecessary complexity. Implement features no one asked for. It came off as overcompensation and it killed project momentum.

I started studying minimalism in code, the beauty of simplicity. I realized users notice everything.

The way an application flows. The intuitiveness of the interface. The speed of execution. A developer who can deliver powerful functionality with clean, simple code is a developer who gets attention. Simplicity is power. Let your code speak for itself.

5. Your past code matters.

I worked on projects where I ignored my previous mistakes—spaghetti code, lack of documentation, poor structure — and I told myself, “I’ll do better next time.” I wanted to move forward without addressing the past. I couldn’t. I became trapped in a cycle of repeating errors.

A developer’s past work is a preview of their future. It doesn’t mean you have to be perfect, but you do need accountability. You need to learn and grow. If all your past projects were chaotic, ask why you approached them that way. If you consistently avoided testing and documentation, ask what that means for your development process.

Don’t ignore the signs just because you’re eager to move on.

6. Not every tool is worth your time.

When I was younger, I treated every new framework or library as a must-learn. If it was popular, that was enough. I’d ignore its relevance to my work, its learning curve, its community support—just to have it on my resume. I paid for it every time.

Now? I care more about effectiveness than trendiness.

Does this tool solve a problem I have? Does it integrate well with my existing stack? Does it have long-term viability? If it’s just trendy but brings nothing substantial, I pass. My time is valuable. My focus is sacred. And not every tool deserves attention.

7. The less you rely, the better things go.

This doesn’t mean rejecting help or tools. It means independence. I used to depend heavily on AI code assistants. Let them dictate my coding style. Follow their suggestions blindly. But when I stopped relying so much—when I focused on understanding, improving, and staying grounded—things shifted.

My code became cleaner. My problem-solving skills sharpened. And ironically, that’s what made me a better developer. The one who can code without assistance holds the power. Not out of arrogance but out of competence.

Clean code isn’t optional... it’s your edge. AI can’t fix chaos. If your code is unreadable, your future’s unstable. I learned the hard way. You don’t have to. Grab my book "Clean Code Zero to One" on Gumroad and level up forever:

📘: https://codewithshahan.gumroad.com/l/cleancode-zero-to-one

8. Being a “yes developer” doesn’t work.

I used to think agreeing to every feature request was the way to please clients. I’d say yes to everything, avoid pushing back, always accommodate. It made me overworked.

Even worse—it made me look unprofessional.

Clients don’t want a pushover. They want a professional with a backbone. Someone who sets boundaries. Someone who’s accommodating but not afraid to say no. Someone who makes informed decisions.

Be a good developer but don’t be a submissive one. There’s a difference.

9. Comments are cool, but clarity is better.

“// This function calculates the sum.” Blah, blah, blah. Every developer writes that. But the one who names the function calculateSum()... who structures code so its purpose is self-evident... who doesn’t rely on excessive comments to explain convoluted logic—that’s the developer whose code is respected.

Clarity isn’t verbose. It’s in the structure, the naming, the organization. I stopped trying to explain my code with comments and started writing code that needed no explanation. That’s when my work became exemplary.

10. Your project should add to your life, not be your life.

I’ve taken on projects where I lost myself. I stopped learning new skills. Stopped networking. Stopped pursuing personal growth. And every time, it ended with regret.

That's all for today. Hope you learned something new. Feel free to follow me and subscribe to my weekly horsecoder newsletter for the latest updates.

🏖️ This article is sponsored by FN Tech Solution, where you can find people to do marketing for you: create apps/websites/designs, grow your business, and more.

🌱🗃️ More Learning Resources:
📘 My Book: Clean Code Zero to One
📹 YouTube: Programming with Shahan
🩻 𝕏: shahancd
🌐 Website: codewithshahan.com

How to land your first frontend job (11 Rules!)

Programming with Shahan — Sat, 15 Mar 2025 23:55:40 +0000

You wanna land your first front-end job in 2025? You think slapping together a few React components and calling yourself a "developer" is enough? News flash: it's not.

Most of you are out here writing spaghetti code, hoping some company will bless you with a job. Wrong. Employers want weapons—clean, scalable, maintainable code machines. You follow these 11 rules, and you’ll have hiring managers fighting over you. Ignore them? Enjoy your job at McJavaScript's.

Let’s get into it

📽️ 1. Not building projects

If you really want to know how far you have come, create real-world projects. A simple quiz app can be entertaining to start with.

Make sure to resist the temptation to copy code. Do it from scratch.

// A simple quiz app with array-based questions
const questions = [
  { prompt: "What does HTML stand for?", answer: "HyperText Markup Language" },
  { prompt: "What is the output of 2+2 in JS?", answer: "4" },
];

function startQuiz() {
  let score = 0;
  questions.forEach((q) => {
    const userAnswer = prompt(q.prompt);
    if (userAnswer && userAnswer.toLowerCase() === q.answer.toLowerCase()) {
      score++;
    }
  });
  alert(`Your score is: ${score}/${questions.length}`);
}

startQuiz();

🍽️ 2. Choose Your Learning Resources Wisely

You want to read every tutorial, watch every video, jump on every trendy platform. That’s a crime in software development.

Pick one or two solid resources/courses. Stick to them until you grasp critical concepts 100%.

You are not gonna make it unless you grasp these two principles of learning coding online:

🫗 Jumping around too much = zero progress.
🌴 A single consistent source builds a strong mental framework.

🪛 3. Start with the Basics

Most devs want to build an entire e-commerce platform on Day One.

Don’t. It’s like trying to fight a heavyweight champion before you’ve thrown your first punch.

Build hundreds of small projects with pure HTML, CSS, and plain JavaScript.

Then choose a front-end framework like Vue.js, Angular or React.js and stick with it. Master them, and everything else is easier.

🐣 4. Begin with Easy Projects

You want to jump into a giant game or a social media clone?

Resist. Start small. A to-do list, a simple calculator, or a color picker.

These little wins matter.

👇 Easy Project Example (Color Picker):

const colorInput = document.getElementById("colorInput");
const body = document.body;

colorInput.addEventListener("input", () => {
  body.style.backgroundColor = colorInput.value;
});

One input, one background change. Easy. But it cements your knowledge.

🚤 5. Take Your Time to Learn

Some of you want to “speedrun” coding. Big mistake.

This is not a race.

True mastery comes from slow, deliberate practice.

Don’t sin by rushing. That only leads to half-baked knowledge.

🎐 6. Regular Practice Is Good, But Don’t Overdo It

Coding every day is nice, but if you burn out, you’ll crash in the long run.

Pace yourself. Show up consistently, but take breaks. That’s how you stay hungry without losing your mind.

🥞/🍋 7. Stuck on Something? Take a Break

When your code refuses to work, you might feel the urge to fight it for hours. Sometimes, that’s necessary.

But other times, walk away. Let your mind breathe. Come back fresh.

🔥 Break Trick

Step away from your desk.
Drink water or take a short walk.
Eat delicious food. I personally do push-ups instead.
Fight the bug with a brand new fresh mind.

⛷️ 8. Learn by Doing

Reading docs, watching videos—good. But if you’re not coding, you’re lying to yourself. Real skill is built by typing, failing, debugging, and repeating.

💪 Mini-Challenge

Pick a small feature from any website you love.
Recreate it from scratch in your own code.
Push to github (your future self will be surprised)

📜 9. Plan Your Projects

Moving into code without a plan is a sin.

That’s how you end up with spaghetti code and random errors. Sketch a quick layout. Identify data structures. Then code.

Let me give an example of planning a Note App prior to writing code:

Data: Where to store notes (array, localStorage?).
Functions: addNote(), deleteNote(), displayNotes().
UI: Minimal HTML to show notes in a list.

🏎️ 10. Focus on Quality, Not Speed

Speed is tempting. “Let me build this in one night!” Then you brag about it. But your code is a big mess. Always breaks, and most often, the entire project collapses.

That’s the result of short-term glory.

Real devs write maintainable code. They think about the future. That's why writing clean code matters more than anything in the tech industry.

The reason most software companies fire these types of devs is clear:

Poor code 🆚 Clean Code

// Poor code: Hard-coded, zero flexibility
function calcPrice() {
  let price = 100;
  let tax = price * 0.15;
  console.log(price + tax);
}

// Clean code: Configurable, easy to adapt
function calcPrice(basePrice, taxRate = 0.15) {
  const total = basePrice + basePrice * taxRate;
  console.log(total);
}

📘 Grab my book, Clean Code Zero to One to master clean coding strategies and join a few who truly can write adaptable, maintainable software.

🧘‍♂️ 11. Be Patient with Your Learning

You see others landing big dev jobs. You panic. Stop comparing. Your path is your own. Focus on your front-end path and complete your 6-month or 1-year commitment regardless how you feel.

This is how all devs find their first life-changing job.
Consistency is the key. Be patient.

🏁. Conclusion

Coding is a journey, not a sprint.

Don’t let mistakes or slow progress discourage you.
Fight through the struggle.

🛬 The more you resist the temptation to quit, the closer you are to landing your dream job.

Now Go Build Something and build it right.
That’s the ONLY way you survive in this game.

Happy coding and share your thoughts in the comment section. I would love to hear that.

9 frontend devs mistakes

Programming with Shahan — Mon, 10 Mar 2025 18:43:06 +0000

You’re a newbie? Good.

That means you have nowhere to go BUT UP.

it can feel overwhelming. That’s natural.

Frontend development is all about [how well] you understand.

Not how [fast] you learn.

🪜 You either step up and do it right, or you watch your code breaks.

👉 Let me walk you through the biggest mistakes beginner frontend devs ever make, and how to avoid them.

1. 🤧 NOT Optimizing Images

Some of you test locally on a blazing-fast machine.

You forget real users have slower networks.

Some devs are too lazy to think about their users.

Let me get it straight: A 6MB image shrunk to 200px is a SIN.

Optimize your images. Every. Single. Time.

Some criminal devs try to reduce image file size by using CSS like this:

img {
  width: 200px;
  height: auto; /* keeps the aspect ratio */
}

👉 However, this does not reduce the ACTUAL file size.

If your image is still 6MB, it will still take a LONG TIME to download CSS just displays it smaller.

What a [BIG WASTE] of bandwidth. 🚮

🪧 So, CSS resizing is mostly for layout, not real optimization.

So what is the solution?

Well, the first solution I would suggest is to use online image compressor tools.

I found tinypng is a great tool for optimizing website images without reducing their quality.

You can see my photo file size reduced to -75% (4mb to 890kb); that's crazy.

⚡ Compressed images = faster load times = happy users.

⚠️ Warning: DO NOT compress images for [larger screens]. Use HTML srcset attribute instead:


<!DOCTYPE html>
<body>
  <h1>Responsive Images Example</h1>
  <!-- 
    The browser will pick the most appropriate image based on 
    the user's device width. 
    - "small.jpg" is for screens up to ~400px wide
    - "medium.jpg" is for screens up to ~800px wide
    - "large.jpg" is for anything bigger

    Each file is physically smaller in size, so smaller devices 
    don't waste bandwidth on huge images.
  -->
  <img
    src="images/large.jpg"
    alt="A beautiful scenic view"
    srcset="
      images/small.jpg   400w,
      images/medium.jpg  800w,
      images/large.jpg   1200w
    "
    sizes="
      (max-width: 400px)  400px,
      (max-width: 800px)  800px,
      1200px
    "
  />
</body>
</html>

🪨🔨 How to Optimize Images Using Node.js?

My second solution would be... use a trustworthy JavaScript library.

If you have some experience with Node.js, then here is a popular Node.js library to optimize image without reducing quality: Sharp

👇 Here’s how to use it:

const sharp = require("sharp");

sharp("large-image.jpg")  // The original, large file
  .resize(200, 150)       // Shrink dimensions
  .toFormat("jpeg", { quality: 80 }) // Convert to JPEG with 80% quality
  .toFile("optimized-image.jpg")
  .then(() => {
    console.log("Image optimized successfully!");
  })
  .catch((err) => {
    console.error("Error optimizing image:", err);
  });

2. 🥸 Using Inline Styles

Let's say you're working on a website that has [10 job offers] listed. You want them to be red to catch the user’s eye.

👇 You took the easy path:

<!-- Inline style approach -->
<a href="/job-offer/123" style="color: red;">Junior Developer</a>
<a href="/job-offer/456" style="color: red;">Frontend Engineer</a>
<a href="/job-offer/789" style="color: red;">Backend Specialist</a>
<!-- ...and so on for all your job offer links -->

Sure, it works. But the moment your boss says:

“Hey dude, make all those links blue instead. My wife hates red.🥱”

You’re in TROUBLE.
You’d have to edit [each] inline style to color: blue;.

If you have 100 links scattered across multiple pages,
that’s a MASSIVE chore.

That's one of the [quick-fix traps] most devs fall into.

✅ The CSS-Only Approach

Now, let’s do it the right way. You define a simple class in your CSS:

.link-job-offer {
  color: red;
  text-decoration: none;
  /* maybe other styles, too */
}
.link-job-offer:hover {
  color: darkred;
}

👇 Then, in your HTML:

<!-- Class-based approach -->
<a href="/job-offer/123" class="link-job-offer">Junior Developer</a>
<a href="/job-offer/456" class="link-job-offer">Frontend Engineer</a>
<a href="/job-offer/789" class="link-job-offer">Backend Specialist</a>
<!-- ...repeat for all job offer links -->

👉 Change one line in .link-job-offer for a global update. That’s power.

3. 🎭 Using Inadequate Heading Tags

Don’t use <h1> just because it’s [big and bold]. That’s an SEO sin.

Headings must reflect hierarchy. If it’s purely decorative, use a <div> or a CSS class:

<!-- Bad -->
<h1 style="font-size: 50px;">Promoted</h1>

<!-- Good -->
<div class="promo-text">Promoted</div>

Then style it in your CSS with .promo-text.

🔥🧯 Don’t sabotage your SEO for a cheap hack.

4. ❌📲 Ignoring Mobile Media Queries

You tested on desktop? Great. But [half] your users are on mobile.

If you skip media queries, you CHEAT half your audience.

It doesn't take much time; you just need to add the following CSS codes:

.container {
  width: 60%;
  margin: auto;
}

/* Mobile first approach */
@media (max-width: 600px) {
  .container {
    width: 95%;
  }
}

Resist the lazy mindset. Always check your phone or dev tools. Because if it breaks on mobile, you lose. Your users matter most.

5. 🪟 Not Benefitting from Flexbox or CSS Grid

It’s 2025. Floats are archaic. Resist your old habits. Use flex or grid. It’s faster, cleaner, simpler.

Flexbox Example:

.navbar {
  display: flex;
  justify-content: space-between;
  align-items: center;
}

🙅‍♂️ No more float nightmares.

6. 🗂️ Not Abstracting Elements

If you find yourself reusing .newsletter__title all over, that’s a sign of mistakes. Abstract it. Make a .section-title that you can use anywhere. That’s maintainable.


<!-- ❌ Wrong Approach: Using .newsletter__title everywhere -->
<section class="newsletter">
    <h2 class="newsletter__title">Subscribe to Our Newsletter</h2>
</section>

<section class="blog">
    <h2 class="newsletter__title">Latest Blog Posts</h2> <!-- Wrong! -->
</section>

<section class="events">
    <h2 class="newsletter__title">Upcoming Events</h2> <!-- Wrong! -->
</section>



<!-- ✅ Correct Approach: Abstracting with .section-title -->
<section class="newsletter">
    <h2 class="section-title">Subscribe to Our Newsletter</h2>
</section>

<!-- and so on... -->

7. 🧹 NOT Removing Redundant Styles

Some of you love to add display: block and float: right in the same rule. Or width: 100% on a block element.

That’s dead code. Clean it up.

Wrong:

.foo {
  display: block;
  float: right;
}

Right:

.foo {
  float: right; /* automatically block */
}

🤏 Less is more.

8. 🏯 Uppercasing Things in HTML

Typing <h2>JOB OFFERS</h2> is the easy route.

But that’s a sin against separation of concerns. Use CSS:

<h2 class="section-title">Job Offers</h2>

.section-title {
  text-transform: uppercase;
}

Done. Now, your text is uppercase only in presentation.

9. 💎 NOT Focusing on QUALITY

Speed is tempting. “Let me build this in one night!” Then you brag about it. But your code is a mess. Most often, the entire project collapses.

That’s the sin of short-term glory.

Real devs write maintainable code. They think about the future.

That's why writing clean code matters most in the tech industry. Grab my book Clean Code Zero to One to master this skill and join a few who truly understand everything about clean code.

The reason most software companies fire these types of devs is clear:

Poor code 🆚 Clean Code

// Poor code: Hard-coded, zero flexibility
function calcPrice() {
  let price = 100;
  let tax = price * 0.15;
  console.log(price + tax);
}

// Clean code: Configurable, easy to adapt
function calcPrice(basePrice, taxRate = 0.15) {
  const total = basePrice + basePrice * taxRate;
  console.log(total);
}

🏁. Conclusion

Frontend dev is a journey, not a sprint. Mistakes happen. That doesn't mean you SHOULD NOT look into them and struggle later.

🥊 Fight through the struggle NOW before it puts you in a painful punishment.

Happy coding and share your thoughts in the comment section. I would love to hear that.

🌱🗃️ More Learning Resources:

📘 My Book: Clean Code Zero to One
🌐 Website: codewithshahan.com
🩻 𝕏: shahancd

Which JavaScript framework is best (React or Vue in 2025?)

Programming with Shahan — Thu, 20 Feb 2025 13:34:41 +0000

React vs. Vue.js in 2025, Which One Should You Choose?

React remains one of the most popular JavaScript libraries for building single-page applications in 2025. Meanwhile, Vue.js continues to call itself a “progressive framework” and its popularity skyrocketing day by day. Both share more in common than, say... Angular, which is often considered a “true” framework.

In this article, I will walk you through a detailed comparison between React.js and Vue.js, updated with the latest 2025 trends, so you can decide which library or framework is best for your next project.

⚖️ Framework Overview

React was created by Jordan Walke, a software engineer at Facebook (now Meta). It’s maintained by Meta and a large community of individual developers and companies. In 2025, React’s ecosystem is richer than ever, making it a top choice for building complex user interfaces and handling data processing tasks.

Furthermore, React Native remains a popular extension of React.js, allowing developers to create hybrid mobile applications using the same skill set.

Vue.js, on the other hand, was created by Evan You after he worked on several Google projects using AngularJS. Vue focuses on simplicity and efficiency in UI design and development. By 2025, its intuitive syntax and built-in features (like transitions and official state management via Pinia) have continued to attract developers seeking a straightforward framework for building interfaces.

🍏 Reasons for Using Vue.js

Vue.js has maintained its reputation for simplicity. Developers appreciate its clear and logical design, which speeds up development workflows. Unlike some other frameworks, Vue.js retains a consistent syntax throughout UI functionality. That's why I love it personally.

Efficient Official Plugins: Vue’s core team provides well-structured solutions for common tasks like routing and state management.
No Corporate Ownership: As of 2025, Vue remains community-driven and independent, appealing to developers wary of data exploitation by large tech companies.
Great Ecosystem: Plugins like Vue Router and Pinia cover most app needs out of the box.

⚛️ Reasons for Using React

React.js brings unique features that keep it at the top of many developers’ lists (even though its popularity has been declining over time).

Here is why it is still popular nowadays:

JSX Syntax: JSX (JavaScript XML) makes creating custom components more intuitive. You can write HTML-like code within JavaScript, streamlining component creation.
SEO Support: React.js applications are relatively easy to optimize for search engines, especially when combined with frameworks like Next.js.
React Native: Developers can reuse their React.js knowledge to build native mobile apps for iOS and Android.
React Dev Tools: In 2025, the React Developer Tools are more handy than ever, offering deep insights for debugging and organizing complex apps.

🥇 Comparison of Popularity and Performance

Both React.js and Vue.js remain in high demand in 2025. React has a slight edge in terms of overall popularity, backed by its extensive ecosystem and the community around React Native.

Performance: Both use a virtual DOM for efficient rendering. Vue.js often shines in component creation and updates due to its super fast optimizations.
Usage Stats: According to a 2025 developer survey, over 45% of respondents have used React.js for web development, while around 20% have used Vue.js.

Note: The Statista survey report from previous years still shows React leading in global adoption, though Vue continues to grow steadily.

🧊 Scalability

React.js has the idea of “smart” and “dumb” components to manage complex architectures. Smart components handle data-heavy tasks, while dumb components focus on rendering.

Vue.js offers official libraries like Pinia for state management, enabling developers to handle larger applications. By 2025, Vue 3+ has proven its ability to scale effectively in enterprise projects.

📲 Adjust to Mobile Devices

Both frameworks have a growing demand for cross-platform solutions, although Vue.js is becoming more popular day by day. Let's have a brief overview:

React Native: An established leader for building native iOS and Android apps with React.js.
Weex (Vue): A mobile UI framework by Alibaba Group, allowing Vue.js components to run on mobile. While not as widespread as React Native, Weex provides a solid Vue-based mobile solution.

🚎🔩 Size and Community Support

Bundle Size: Vue.js typically ships with a smaller bundle size, making it ideal for projects prioritizing fast load times.
Community: React benefits from Meta’s backing and a massive open-source ecosystem. Vue.js, though smaller, boasts an active and dedicated community with thorough documentation and ongoing improvements. I think it's becoming more popular than React.js as recent data suggests.

🏒 Use Cases

React.js: Perfect for complex web apps, large-scale projects, and teams needing to pivot between web and mobile with React Native.
Vue.js: Ideal for small to medium-sized apps, real-time platforms, and content delivery systems where simplicity and performance are key.

🏇 Recommendation: Clean Code Book (2025)

If you’re serious about mastering clean code and taking your programming career to the next level, then my book for you: Clean Code Zero to One. I am offering 50% discount using the code "earlybird" during checkout — only for the first 50 copies! Plus, enjoy a 30-day money-back guarantee — no risk, all reward.

👏 Conclusion

I would say React.js is the first choice for developing large and complex apps. Also React Native for mobile apps is a huge plus. But you have to write separate code for web and mobile apps which is a huge downside as well. Therefore, plus-minus = zero.

Vue.js, meanwhile, is loved for its simplicity, fast performance, and smaller default size. It’s a more lightweight framework without sacrificing modern features. The great news is, you don't have to write separate code for web or mobile. Write once and now you can build desktop, mobile, or web apps in one go. So no downside. Only plus plus.

I would choose Vue.js for its simplicity and super fast performance.

Thank you for your time. For more regular posts visit: codewithshahan.com

If you enjoyed this, check out my 𝕏 (Twitter) account for more frontend development insights.

Related Articles:

DEV Community: Programming with Shahan

How to Build an AI-Ready Web Data Pipeline Using Bright Data and Node.js

🏗️ What we’re building

1. The Data Extraction Pipeline Structure

2. The Two Non-negotiable Data Pipeline Rules

RULE 1: The stable data contract is the boundary**

RULE 2: The pipeline must be payload-agnostic**

3. Why Traditional Web Scrapers Break

3.1. The extraction problem

3.2. Turning a web page into structured data

3.3. Scaling beyond a single page

3.4. From extracted data to application logic

3.5. The maintenance loop

4. The Rise of Self-Healing AI-Native Scrapers

4.1. The Self-Healing Advantage

4.2. Why Scraper Studio Fits Our Architecture

4.3. What About Cost, Scale, and Concurrency

5. Project Setup

5.1 Configure package.json

5.2 Add Environment Variables

5.3 Create the Node.js Server

6. Building the Pipeline Prototype

6.1 Design the Prototype Structure

6.2 Normalize the Data

6.3 Validate the Contract

6.4 Remove Duplicate Records

6.5 Compose the Pipeline

6.6 Test Multiple Data Sources

6.7 Connect the Pipeline to Express

6.8 Test the Prototype

6.9 Commit the Prototype

7. Teaching Our Pipeline to Remember

7.1. Why a Scraper Run Is Stateless

7.2. Create Local Snapshot Storage

7.3. Test Snapshot Persistence

8. Detecting Historical Changes

8.1 Building the Dataset Comparison Engine

8.2 Testing the Comparison Engine

Success Result

8.3 Classifying System Events

8.4 Connect Event Classification

8.5 Test Event Classification

9. Building a Reusable Collector Factory

9.1 Designing the Collector Abstraction

9.2 Building the Base Collector

9.3 Defining the Primary Provider

9.4 Adding a Secondary Provider

9.5 Implementing the Collector Factory

9.6 Verifying the Architecture

10. Generate the First Scraper with Scraper Studio AI Agent

Step 1: Initialize the Target URL

Step 2: Configure the Extraction Workflow

Step 3: Understanding the Scraper Studio Generated Workflow

11. Connecting Scraper Studio API to Node.js

11.1 Initiate by API

11.2 Triggering Scraper Studio API from Node.js

11.3 Mounting the Express Route

11.4 Verification Pass

11.5 Open the Runs Tab

12. Building a Texas Real Estate Delta Tracker CLI

12.1. How to Handle Pagination when Scraping

12.2. Delivery Preferences and Calling the Collection API

12.3. Waiting for the Job Without Timing Out

12.4. Building the Texas Tracker’s Data Layer

12.5. Assembling an Interactive Tracker

12.6. Final Testing Workflow

13. Scraper Studio Self-Healing in Production

13.1. Installing the Bright Data CLI

13.2. Deliberately Breaking the Scraper

13.3. Ask Bright Data to Heal the Scraper

14. Deployment

15. Scaling and Production

1: The Temporary Cloud Filesystem

2: Time-Based Data Decay

3: Long-Running HTTP Requests

16. What You Can Build Next

1. A Real-Estate RAG Pipeline for AI Agents

2. Capped-Budget Market Analyzers

Conclusion

FAQ

5.1 Configure `package.json`