DEV Community: zahidkhan-xen

AI-Powered Web Scraper

zahidkhan-xen — Sun, 17 May 2026 10:57:14 +0000

I Built an AI-Powered Web Scraper That Understands Plain English — Here's How

No XPaths. No CSS selectors. No regex nightmares. Just tell it what you want, and it figures out the rest.

1. Introduction — The Problem I Was Solving

If you've done any web scraping, you know the pain:

You write a perfectly working scraper today.
The website changes its layout tomorrow.
Your scraper breaks. You fix it. It breaks again.

The real frustration isn't writing the scraper — it's maintaining it. Every time a site updates its HTML structure, you're back to inspecting elements, rewriting selectors, and re-testing.

I kept thinking: What if the scraper could just understand what I want, the same way I'd explain it to a person?

That's what led me to build this project. Instead of writing brittle rules about where data lives in the HTML, I wanted to just ask a question in plain English and get an answer back.

2. Solution — What I Built

I built a Python AI Web Scraper that combines two powerful tools:

ScrapeGraphAI — a Python library that uses LLMs to intelligently scrape websites
Groq + Llama 3.1 — a fast, free AI model that reads and understands web page content

The result? You give it a URL and a question like:

"What is the purpose of this website?"

And it returns the answer — no HTML parsing, no CSS selectors, no fragile scraping rules.

Tech Stack at a glance:

Python 3.x
scrapegraphai — the AI scraping framework
playwright — for browser-based page rendering
Groq API — free LLM inference (Llama 3.1 8B model)

3. Code Explanation — Step by Step

Here's the complete main.py (it's beautifully short):

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "api_key": "your-groq-api-key-here",
        "model": "groq/llama-3.1-8b-instant",
    },
    "verbose": True,
    "headless": True,
}

smart_scraper_graph = SmartScraperGraph(
    prompt="Extract purpose of this website: https://qaul.ai/",
    source="https://qaul.ai/",
    config=graph_config
)

result = smart_scraper_graph.run()
print(result)

Let's break it down piece by piece.

Step 1 — Import the SmartScraperGraph

from scrapegraphai.graphs import SmartScraperGraph

SmartScraperGraph is the core class from the ScrapeGraphAI library. Think of it as a smart agent that knows how to visit a webpage, read its content, and answer your question using an LLM.

Step 2 — Configure the AI Model

graph_config = {
    "llm": {
        "api_key": "your-groq-api-key-here",
        "model": "groq/llama-3.1-8b-instant",
    },
    "verbose": True,
    "headless": True,
}

This config dictionary has three things:

llm — tells ScrapeGraphAI which AI model to use. Here we're using groq/llama-3.1-8b-instant, which is fast and available for free via the Groq API.
verbose: True — prints what the scraper is doing at each step, great for debugging.
headless: True — runs the browser in the background (no visible window), which is standard for production scrapers.

💡 Why Groq? Groq offers blazing-fast LLM inference with a generous free tier. Llama 3.1 8B is more than capable for most scraping tasks.

Step 3 — Define Your Scraping Task

smart_scraper_graph = SmartScraperGraph(
    prompt="Extract purpose of this website: https://qaul.ai/",
    source="https://qaul.ai/",
    config=graph_config
)

This is the magic part. You provide:

prompt — your question in plain English. This is what you'd normally express as complex CSS selectors or XPath queries. Here, you just ask.
source — the URL of the website to scrape.
config — the configuration we defined above.

Step 4 — Run It and Print the Result

result = smart_scraper_graph.run()
print(result)

Calling .run() triggers the entire pipeline:

Playwright opens the webpage (headless browser)
The page content is extracted
The content is sent to the Llama model via Groq
The model reads it and answers your prompt
The result is returned as a Python dictionary

4. How to Use — Setup & Instructions

Getting this running on your machine takes about 5 minutes.

Prerequisites

Python 3.8 or higher
A free Groq API key (get one at console.groq.com)

Step 1 — Clone the Repository

git clone https://github.com/zahidkhan-xen/ai-powered-web-scraper
cd ai-powered-web-scraper

Step 2 — Create a Virtual Environment

# macOS/Linux
python3 -m venv venv
source venv/bin/activate

# Windows
python -m venv venv
venv\Scripts\activate

Step 3 — Install Dependencies

pip install scrapegraphai
playwright install

⚠️ playwright install downloads the browser binaries needed to render web pages. This may take a minute.

Step 4 — Add Your Groq API Key

Open main.py and replace the placeholder with your actual key:

"api_key": "paste-your-groq-key-here",

Step 5 — Run the Scraper

python main.py

You'll see verbose output as the scraper works, and the final result printed at the end.

Customizing Your Scrape

To scrape a different site or extract different information, just change these two lines in main.py:

smart_scraper_graph = SmartScraperGraph(
    prompt="What products are listed on this page?",   # ← your question
    source="https://your-target-website.com/",          # ← your URL
    config=graph_config
)

More prompt examples you can try:

# Extract pricing information
prompt="List all pricing plans and their features"

# Get contact details
prompt="What is the contact email and phone number on this page?"

# Summarize an article
prompt="Summarize this article in 3 bullet points"

# Extract job listings
prompt="List all job titles and their locations"

5. Conclusion

What I love about this approach is how it flips the traditional scraping paradigm on its head. Instead of you adapting your code to the website's structure, the AI adapts to the website for you.

This is still an early-stage project — a single main.py — but it proves a powerful concept. Some directions you could take it further:

Add a CLI interface so you can pass URLs and prompts as arguments
Save results to JSON or CSV for downstream processing
Loop over multiple URLs to scrape at scale
Add error handling for pages that fail to load
Swap in different models (OpenAI, Anthropic, etc.) depending on task complexity

Traditional scrapers break when websites change. This one just needs a better question. That's a fundamentally more resilient way to build.

GitHub: zahidkhan-xen/ai-powered-web-scraper

If you found this useful, consider giving the repo a ⭐ — it helps others find it too!

From CRA to Lightning Fast Vite with Cesium: My Real-World Migration Experience

zahidkhan-xen — Mon, 21 Jul 2025 10:44:53 +0000

With Create React App (CRA) officially deprecated, it was time for me to step out of my comfort zone and migrate my React project to a modern alternative Vite. In this article, I’ll walk you through my exact steps, learnings, and optimizations while migrating from CRA + CRACO to Vite, including Cesium integration, JSX conversion, and asset handling.

Why Migrate to Vite?

CRA has served well for years, but:

Hot Module Replacement (HMR) in CRA is relatively slow (~1000ms) compared to Vite (~200ms).

Cold Start: CRA took ~15 seconds to start dev server; Vite does it in just ~2 seconds.

Modern Tooling: Vite uses esbuild under the hood — it's 10–100x faster than Webpack.

CRA is officially deprecated — see this article.

https://build5nines.com/create-react-app-is-now-deprecated-time-to-migrate-to-vite-or-next-js/

Initial Cleanup: Say Goodbye to CRA & CRACO

Start by removing all CRA/CRACO-related dependencies:
npm uninstall react-scripts @craco/craco craco-cesium

Update package.json scripts:

"scripts": {
  "dev": "vite",
  "build": "vite build",
  "preview": "vite preview",
  "lint": "eslint src/**/*.{js,jsx,ts,tsx,json}",
  "format": "prettier --write src/**/*.{js,jsx,ts,tsx,css,md,json,scss} --config ./.prettierrc"
}```



## Installing Vite & Required Plugins

Install core dependencies for React + Cesium support:



```jsx
npm install --save-dev vite @vitejs/plugin-react
npm install --save-dev vite-plugin-cesium
npm install cesium

Create vite.config.js

Here’s a basic configuration with Cesium support and API proxy:

import { defineConfig, loadEnv } from 'vite';
import react from '@vitejs/plugin-react';
import cesium from "vite-plugin-cesium";

export default defineConfig(({ command, mode }) => {
  // Load env file based on `mode` in the current working directory
  const env = loadEnv(mode, process.cwd(), '');

  return {
    plugins: [
      react({
        include: ['**/*.jsx', '**/*.js', '**/*.tsx', '**/*.ts'],
        babel: {
          plugins: [
            ['@babel/plugin-transform-react-jsx', { runtime: 'automatic' }]
          ]
        }
      }),
      cesium(),
    ],
    server: {
      proxy: {
        '/api': env.VITE_PORT_API_KEY,
        '/tilesets': env.VITE_PORT_API_KEY
      },
    },
    resolve: {
      alias: {
        '@': '/src',
      },
    },
    optimizeDeps: {
      exclude: ['react-virtualized'],
      include: ['prop-types'],
    },
    build: {
      assetsInlineLimit: 0,
      commonjsOptions: {
        include: [/react-virtualized/, /node_modules/],
      },
      chunkSizeWarningLimit: 1600,
    },
  };
});

Special Handling: Cesium Static Assets in Vite

Cesium relies on static assets (like Assets, Widgets, Workers) that need to be available at runtime.

Here’s how I handled it:

# Remove old public copy if exists
rm -rf public/cesium

# Copy Cesium assets from node_modules
cp -r node_modules/cesium/Build/Cesium public/cesium

Make sure your Vite config includes:

define: {
  CESIUM_BASE_URL: JSON.stringify('/cesium'),
}

Now Cesium will load correctly from /cesium in production.

Project Structure & HTML Changes

Replace %PUBLIC_URL%
Vite doesn’t support %PUBLIC_URL%. Just use root-relative paths:

<!-- index.html -->
<link rel="icon" href="/favicon.ico" />
<script type="module" src="/src/index.jsx"></script>

Convert .js to .jsx
If you’re using JSX inside .js files, rename them to .jsx. Here's a script to automate it:

rename-jsx.sh

#!/bin/bash
echo "🔍 Scanning for JSX in .js files..."

find ./src -name "*.js" | while read file; do
  if grep -qE "</?[A-Za-z].*?>" "$file"; then
    newfile="${file%.js}.jsx"
    mv "$file" "$newfile"
    echo "✅Renamed: $file → $newfile"
  fi
done

echo "🎉 Done! All JSX files renamed to .jsx"

Run it:

chmod +x rename-jsx.sh
./rename-jsx.sh

Handling Environment Variables

Vite uses the VITE_ prefix for env variables. Example:

VITE_FLAG_SMITH_API_KEY=abc123

Use it like this:

const key = import.meta.env.VITE_FLAG_SMITH_API_KEY;

Handling Static Assets

Move assets like images, logos, fonts to the /public folder.

Replace any require(...) statements with static import paths or import.

Old CRA way:

require('../../assets/logo.png')

Vite way:

src="/logo.png" // if inside public folder

Optional: To handle require in legacy code, you can use:

npm install vite-plugin-commonjs --save-dev

Cesium Integration Notes

Vite does not automatically handle Cesium static files like CRA + craco-cesium did.

Manually copy Cesium assets:

rm -rf public/cesium
cp -r node_modules/cesium/Build/Cesium public/cesium

Make sure CESIUM_BASE_URL is defined in vite.config.js as shown earlier.

Clean Up Dev Dependencies

Remove unused CRA-related packages:

npm uninstall html-webpack-plugin

Final Thoughts
Migrating from CRA to Vite brought massive speed improvements:

Cold Start: 15s ⏩ 2s

HMR: 1000ms ⏩ 200ms

Cleaner setup, no more heavy Webpack configs

Easier Cesium integration with vite-plugin-cesium

Whether you’re optimizing for speed or staying up-to-date with modern tooling, Vite is definitely the way forward.

Vite not loading Cesium.3DTileset

zahidkhan-xen — Thu, 17 Jul 2025 06:16:44 +0000

Problem arises when i upload 3D Tileset but its not rendering the map in 3D view , dont know if cesium need specific config with vite while i have shared my configs with you

Context API vs. Redux: When and Why to Use Them in React

zahidkhan-xen — Tue, 02 Jul 2024 16:03:07 +0000

Before starting to explore the differences and use cases of both Context API and Redux, it is essential to understand why these two solutions were developed.

In simple React, we can pass down data from parent components to child components. The parent component is where your main component is rendering. For example:

Let's say we have a component named Greet.js, and this component is rendering in App.js, making App.js its parent component. If we pass a constant firstName with the value "John", it will look like this:

//App.js
export const App = () => {
  const firstName = "John";

  return (
    <Greet name={firstName} />
    <Profile name={firstName} />

  );
}

// Greet.js
export const Welcome = ({ name }) => {
  return <h1>Welcome : {name}</h1>;
}

//Profile.js
export const Profile = ({ name } => {
return <h1> userName : {name} </h1>

This above method also known as Prop Drilling .Passing props can become inconvenient when you need to pass some prop deeply through the tree, or if many components need the same prop the prop drilling becomes complex and then for passing props we use context API.

Context API:

Context is a way to pass data through the component tree without having to pass it down manually at every level.It's beneficial for sharing data that can be considered "global" for a tree of React components, such as the current authenticated user, theme, or preferred language.
There are three step to implement Context API:

Create a Context: Define a context to hold the state.

Provide Context: Wrap the top-level component with the context provider.

Consume Context: Use the context in the child components to access the state.

1-Create a Context

// UserContext.js
import React, { createContext, useState } from 'react';

const UserContext = createContext();

const UserProvider = ({ children }) => {
  const [name, setName] = useState("John");

  return (
    <UserContext.Provider value={{ name }}>
      {children}
    </UserContext.Provider>
  );
};

export { UserContext, UserProvider };

2-Provide Context

// App.js

import React from 'react';
import { UserProvider } from './UserContext';
import Welcome from './Welcome';
import Profile from './Profile';

const App = () => {
  return (
    <UserProvider>
      <div>
        <Welcome />
        <Profile />
      </div>
    </UserProvider>
  );
};

export default App;

3-Consume Context

// Welcome.js
import React, { useContext } from 'react';
import { UserContext } from './UserContext';

const Welcome = () => {
  const { name } = useContext(UserContext);

  return <h1>Welcome: {name}</h1>;
};

export default Welcome;

// Profile.js
import React, { useContext } from 'react';
import { UserContext } from './UserContext';

const Profile = () => {
  const { name } = useContext(UserContext);

  return <h1>userName: {name}</h1>;
};

export default Profile;

Using the Context API allows data to flow anywhere in the component tree, making it a superior method for managing application state compared to prop drilling. However, as applications grow in complexity and certain side effects become challenging to manage within individual components, Redux emerges as a powerful solution.

React Redux

Redux simplifies state management by centralizing it through a global store. It not only eliminates the need for prop drilling but also provides an efficient way to handle side effects using middleware. This approach ensures that side effects, such as asynchronous data fetching or logging, are managed centrally, keeping components focused solely on UI rendering.

Let's understand it with example :
These steps involved in Redux data flow:

Create Action : Define action to be perform

Create Reducer : Define how to perform action and update it

Create Store : Recieved updated state from Reducer and make it available to call in any component.

1.Create Action

// userActions.js
export const setName = (name) => ({
  type: 'SET_NAME',
  payload: name,
});

2.Create Reducer

// userReducer.js
const initialState = {
  name: 'John', // initial state
};

const userReducer = (state = initialState, action) => {
  switch (action.type) {
    case 'SET_NAME':
      return { ...state, name: action.payload };
    default:
      return state;
  }
};

export default userReducer;

*3.Create Store *

// store.js
import { createStore } from 'redux';
import userReducer from './userReducer';

const store = createStore(userReducer);

export default store;

Wrap your application with the Redux Provider component to make the store available to all components.

// App.js
import React from 'react';
import { Provider } from 'react-redux';
import store from './store';
import Welcome from './Welcome';
import Profile from './Profile';

const App = () => {
  return (
    <Provider store={store}>
      <div>
        <Welcome />
        <Profile />
      </div>
    </Provider>
  );
};

export default App;

Now, you can use this globally in any component by using useSelector hook :

// Welcome.js
import React from 'react';
import { useSelector } from 'react-redux';

const Welcome = () => {
  const name = useSelector(state => state.name);

  return <h1>Welcome: {name}</h1>;
};

export default Welcome;

Conclusion:

In the end, whether you choose React's Context API or Redux depends on what your application specifically requires and the particular challenges it faces.

How we can change the color of enableSearch in LazyLog?

zahidkhan-xen — Thu, 10 Aug 2023 12:01:44 +0000