DEV Community: Victor Yakubu

Let's Build a Tool that Can Extract Information from Any Website in Seconds with AI and Proxies

Victor Yakubu — Thu, 19 Dec 2024 09:43:22 +0000

“Useful” data is the backbone of informed decision-making and strategic planning for any successful business. However, the real challenge isn’t just scraping data — it’s extracting valuable information from the data you’ve collected. This step often requires significant time and effort, making it a bottleneck in the process.

In this article, you’ll learn how to build a scraper that not only extracts web data but also processes and analyzes it in seconds. You’ll walk through a step-by-step process, from integrating Bright Data Scraping Browser to bypass common scraping obstacles like Captcha and IP bans to using LangChain and OpenAI for real-time data processing and building a sleek user interface with Next.js to display your results.

At the end of this article, you should have an application that can do this:

Prerequisites:

Familiarity with JavaScript and TypeScript and frameworks like React or Next.js.
Node.js (v16 or higher): Required to run the Next.js application.
npm or yarn: This is used to manage dependencies in your project.

Getting your Credentials

Before starting the project, you’ll need access credentials for the Bright Data Scraping Browser and an OpenAI API key. This section provides a step-by-step guide to obtaining these credentials. Skip this step if you already have them.

1. Bright Data Scraping Browser Access Details

To bypass anti-scraping mechanisms like this, you need the Scraping Browser.

Bright Data’s Scraping Browser is a powerful tool that automatically manages all website unlocking operations under the hood, including CAPTCHA solving, browser fingerprinting, automatic retries, selecting headers, cookies, & Javascript rendering, and more, so you can save time and resources.

Sign up: On the homepage, click “Start Free Trial”. If you already have an account, log in.
Click on “Get Proxy Products”.

Click on the “Add” button and select “Scraping Browser.”

Next, you will be taken to the “Add zone” page, where you will be required to choose a name for your new scraping browser proxy zone. After that, click on “Add”.

After this, your proxy zone credentials will be created. (You will need these details in your code to bypass any anti-scraping mechanisms used on any website.

2. OpenAI API Key

OpenAI’s API will be used to process and analyze the scraped data. Here’s how to get your API key:

Create an OpenAI Account if you haven’t already done so.
Click on “Start Building” and fill in the necessary details
Once logged in, navigate to the API section in your dashboard.
In the API Keys section of the dashboard, click Create New Key.

Copy the generated key and store it in a secure location, as OpenAI will not display it again. If this is your first time, you may need to set up a billing account. Add your payment details to unlock access to GPT-4 and other models.

Building the Scraper

Now that you have your credentials ready, it’s time to dive into building the application. The project will involve creating a Next.js application, setting up components and routes, and implementing the logic for web content extraction and analysis. Here’s a breakdown of the steps:

1. Setting up the Next.js application

Start by creating a new Next.js project. While installing, initialize Typescript and Tailwind CSS, and use the src folder and App router.

npx create-next-app@latest web-content-ai-scraper
cd web-content-ai-scraper

Install the required dependencies:

npm install axios puppeteer-core dotenv cheerio @langchain/community @langchain/openai @langchain/core langchain

2. Project Structure

Organize your project with the following structure:

web-content-ai-scraper/  
├── src/  
│   ├── app/  
│   │   ├── api/  
│   │   │   └── query-web-content/  
│   │   │       └── route.ts  
│   │   ├── components/  
│   │   │   ├── QueryResults.tsx  
│   │   │   └── WebContentQuerierForm.tsx  
│   │   ├── layout.tsx  
│   │   └── page.tsx  
├── utils/  
│   └── WebContentQuerier.ts  
├── .env

3. Setting Up Environment Variables

Create a .env file in the root of your project to store your API keys and credentials securely:

OPENAI_API_KEY=<your_openai_api_key>
BRIGHT_DATA_ENDPOINT=<your_bright_data_endpoint>

Ensure you add the .env file to your .gitignore to prevent accidental exposure of sensitive information.

4. Implementing the Components

To handle the user input and display the results, we will create two React components that will reside in the src/app/components folder.

1. QueryResults Component

The QueryResults.tsx file is used to display the query results and the context sources returned from the backend.

Here’s the complete code:

import React from 'react';

interface QueryResultsProps {
  answer: string;
  // eslint-disable-next-line @typescript-eslint/no-explicit-any
  context: any[];
}

const QueryResults: React.FC<QueryResultsProps> = ({ answer, context }) => {
  return (
    <div className="mt-6 space-y-4">
      <div className="bg-blue-50 border-l-4 border-blue-500 p-4">
        <h2 className="text-lg font-semibold text-gray-900 mb-2">
          Query Result
        </h2>
        <p className="text-gray-700">{answer}</p>
      </div>

      <details className="bg-gray-50 border border-gray-200 rounded-lg">
        <summary className="px-4 py-2 cursor-pointer text-gray-800 font-medium">
          View Context Sources
        </summary>
        <div className="p-4 space-y-2">
          {context.map((doc, index) => (
            <div
              key={index}
              className="bg-white p-3 rounded-md shadow-sm border border-gray-100"
            >
              <p className="text-sm text-gray-600 line-clamp-2">
                {doc.pageContent}
              </p>
            </div>
          ))}
        </div>
      </details>
    </div>
  );
};

export default QueryResults;

Explanation:

The QueryResults Component accepts two props:

answer: The processed output from the AI.
context: An array of context sources related to the query.

The query result is displayed in a highlighted box with a title.
A collapsible details section allows users to view content sources, making it clean and user-friendly.

2. WebContentQuerierForm Component

The WebContentQuerierForm.tsx file creates a user interface for inputting the URL and query.

Here’s the complete code:

'use client';

import React, { useState } from 'react';

interface WebContentQuerierFormProps {
  onSubmit: (url: string, query: string) => void;
  isLoading: boolean;
}

const WebContentQuerierForm: React.FC<WebContentQuerierFormProps> = ({
  onSubmit,
  isLoading,
}) => {
  const [url, setUrl] = useState('');
  const [query, setQuery] = useState('');

  const handleSubmit = (e: React.FormEvent) => {
    e.preventDefault();

    // Basic validation
    if (!url.trim() || !query.trim()) {
      alert('Please enter both a URL and a query');
      return;
    }

    onSubmit(url, query);
  };

  return (
    <form onSubmit={handleSubmit} className="space-y-4">
      <div>
        <label
          htmlFor="url"
          className="block text-sm font-medium text-gray-700"
        >
          Web Page URL
        </label>
        <input
          type="url"
          id="url"
          value={url}
          onChange={(e) => setUrl(e.target.value)}
          placeholder="https://example.com"
          required
          className="mt-1 block w-full rounded-md border-gray-300 shadow-sm focus:border-indigo-300 focus:ring focus:ring-indigo-200 focus:ring-opacity-50"
        />
      </div>

      <div>
        <label
          htmlFor="query"
          className="block text-sm font-medium text-gray-700"
        >
          Your Query
        </label>
        <textarea
          id="query"
          value={query}
          onChange={(e) => setQuery(e.target.value)}
          placeholder="Enter your question about the web page"
          required
          rows={3}
          className="mt-1 block w-full rounded-md border-gray-300 shadow-sm focus:border-indigo-300 focus:ring focus:ring-indigo-200 focus:ring-opacity-50"
        />
      </div>

      <div>
        <button
          type="submit"
          disabled={isLoading}
          className={`w-full flex justify-center py-2 px-4 border border-transparent rounded-md shadow-sm text-sm font-medium text-white 
            ${
              isLoading
                ? 'bg-gray-400 cursor-not-allowed'
                : 'bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500'
            }`}
        >
          {isLoading ? 'Processing...' : 'Query Web Content'}
        </button>
      </div>
    </form>
  );
};

export default WebContentQuerierForm;

Explanation:

The form includes input fields for a URL and a query.
Validation ensures both fields are filled out before submission.
A submit button triggers the onSubmit function, which processes the input.
The button dynamically updates to indicate the loading status (Processing…) when the form is submitted.

The next step will involve setting up the backend API to handle and integrate the query with Bright Data and OpenAI.

5. Creating the API Route

In this section, we will set up an API route to handle the request from the frontend form, process the provided URL and query, and return the result to be displayed.

File: app/api/query-web-content/route.ts

Below is the complete code for the API route:

import { NextRequest, NextResponse } from 'next/server';
import { WebContentQuerier } from '../../../../utils/WebContentQuerier';

export async function POST(request: NextRequest) {
  try {
    const { url, query } = await request.json();

    if (!url || !query) {
      return NextResponse.json(
        { message: 'URL and query are required' }, 
        { status: 400 }
      );
    }

    const querier = new WebContentQuerier();

    // Extract web page content
    await querier.extractWebPageContent(url);

    // Query the content
    const result = await querier.queryContent(query);

    return NextResponse.json(result);
  } catch (error) {
    console.error('Error processing web content query:', error);
    return NextResponse.json(
      { 
        message: 'Error processing web content query', 
        error: error instanceof Error ? error.message : 'Unknown error' 
      }, 
      { status: 500 }
    );
  }
}

How It Works

Request Handling:

This route uses the POST method to process data from the frontend.
The NextRequest object extracts the url and query from the request body.

2*. Input Validation*:

If either url or query is missing, the API responds with a 400 status and an error message.

Web Content Processing:

An instance of the WebContentQuerier the utility is created.
The extractWebPageContent method fetches the content of the web page from the provided url.
The queryContent method processes the extracted content using the provided query to generate a meaningful result.

Error Handling:

Any errors encountered during the process are logged to the console.
The API responds with a 500 status and includes the error message for debugging.

Response:

On successful execution, the API sends back the processed result (e.g., the answer and related context sources) as a JSON response.

The next step is to implement the code for WebContentQuerier utility.

6. Writing the Core Logic

The core logic of our project is encapsulated in the WebContentQuerier class. This utility handles fetching web page content, processing, and indexing it, and performing queries to provide context-based answers. Below is the complete implementation and a breakdown of its functionality.

File: utils/WebContentQuerier.ts

import { ChatOpenAI } from "@langchain/openai";
import { OpenAIEmbeddings } from "@langchain/openai";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { HumanMessage, SystemMessage } from "@langchain/core/messages";
import puppeteer from 'puppeteer-core';
import * as cheerio from 'cheerio';
import dotenv from 'dotenv';


// Load environment variables
dotenv.config();

export class WebContentQuerier {
  private model: ChatOpenAI;
  private embeddings: OpenAIEmbeddings;
  private vectorStore: MemoryVectorStore | null;

  constructor() {
    // Initialize OpenAI model and embeddings
    this.model = new ChatOpenAI({
      openAIApiKey: process.env.OPENAI_API_KEY,
      temperature: 0.7,
      model: 'gpt-3.5-turbo',
    });

    this.embeddings = new OpenAIEmbeddings({
      openAIApiKey: process.env.OPENAI_API_KEY,
    });

    this.vectorStore = null;
  }

  async extractWebPageContent(url: string) {
    const BROWSER_WS = process.env.BROWSER_WS; // Scraping browser WebSocket URL

    if (!BROWSER_WS) {
      throw new Error("Scraping browser WebSocket URL is not defined in .env file.");
    }

    try {
      console.log('Connecting to Scraping Browser...');

      // Connect to the Scraping Browser
      const browser = await puppeteer.connect({
        browserWSEndpoint: BROWSER_WS,
      });

      const page = await browser.newPage();
      page.setDefaultNavigationTimeout(60000);
      console.log(`Navigating to URL: ${url}`);
      await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 });

      // Extract the content
      const html = await page.content();

      // Use Cheerio to parse the HTML
      const $ = cheerio.load(html);
      const pageText = $('body').text()
        .replace(/\s+/g, ' ')
        .trim();

      // Create a document-like object
      const docs = [{
        pageContent: pageText,
        metadata: { source: url }
      }];

      // Split the document into smaller chunks
      const textSplitter = new RecursiveCharacterTextSplitter({
        chunkSize: 1000,
        chunkOverlap: 200,
      });

      const splitDocs = await textSplitter.splitDocuments(docs);

      // Create vector store in memory
      this.vectorStore = await MemoryVectorStore.fromDocuments(
        splitDocs,
        this.embeddings
      );

      console.log(`Successfully extracted and indexed content from ${url}`);
      await browser.close();
      return splitDocs.length;
    } catch (error) {
      console.error('Error extracting web page content:', error);
      throw error;
    }
  }

  async queryContent(query: string, maxResults: number = 4) {
    if (!this.vectorStore) {
      throw new Error(
        'No content has been loaded. Call extractWebPageContent first.'
      );
    }

    try {
      // Perform similarity search
      const relevantDocs = await this.vectorStore.similaritySearch(
        query,
        maxResults
      );

      // Use the relevant documents as context for the query
      const context = relevantDocs.map((doc) => doc.pageContent).join('\n\n');

      // Create chat messages with system and human messages
      const messages = [
        new SystemMessage(
          'You are a helpful assistant that answers questions based strictly on the given context.'
        ),
        new HumanMessage(
          `Context:\n${context}\n\nQuery: ${query}\n\nProvide a concise and accurate answer based strictly on the context.`
        ),
      ];

      // Generate an answer using the chat messages
      const response = await this.model.invoke(messages);

      return {
        answer: response.content,
        context: relevantDocs,
      };
    } catch (error) {
      console.error('Error querying content:', error);
      throw error;
    }
  }

  // Optional method to clear the vector store
  clearContent() {
    this.vectorStore = null;
    console.log('Vector store cleared');
  }
}

export default WebContentQuerier;

Explanation

`extractWebPageContent(url: string)`

This method extracts and indexes the content from a web page.

Scraping Content:
- Connects to a Puppeteer browser using a WebSocket endpoint (BROWSER_WS).
- Navigate to the URL and retrieve the HTML content of the page.
Content Parsing:
- Uses Cheerio to extract and clean the textual content of the page ($(‘body’).text()).
Document Splitting:
- Splits the text into smaller chunks using RecursiveCharacterTextSplitter to ensure better embeddings and search performance.
Vector Store Creation:
- Converts the split documents into embeddings using OpenAIEmbeddings and stores them in a memory-based vector store.
Return Value:
- Returns the number of chunks created.

`queryContent(query: string, maxResults: number = 4)`

This method processes user queries based on the indexed content.

Similarity Search:
- Uses the vector store to find the most relevant text chunks for the query.
Context Generation:
- Combines the content of the relevant documents into a single string, providing the context for the query.
Chat Completion:
- Sends the context and query as system and human messages to the OpenAI ChatOpenAI model.
- The system message instructs the AI to respond only based on the context. You can modify the prompts for more accurate information
Return Value:
- Returns an object containing the AI’s answer and the relevant context.

7. Updating Layout and Page Files

The layout.tsx file defines the global layout for your application. And

Here’s the code for the layout.tsx file:

import type { Metadata } from 'next';
import { Inter } from 'next/font/google';
import './globals.css';

const inter = Inter({ subsets: ['latin'] });

export const metadata: Metadata = {
  title: 'Web Content Querier',
  description: 'Query web content using AI',
};

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body className={`${inter.className} bg-gray-100`}>{children}</body>
    </html>
  );
}

The page.tsx file defines the content and functionality for the /api/query-web-content route in the app.

'use client';

import { useState } from 'react';
import WebContentQuerierForm from './components/WebContentQuerierForm';
import QueryResults from './components/QueryResults';

export default function Home() {
  const [queryResult, setQueryResult] = useState<{
    answer: string;
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    context: any[];
  } | null>(null);
  const [isLoading, setIsLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);

  const handleQuery = async (url: string, query: string) => {
    setIsLoading(true);
    setError(null);
    setQueryResult(null);

    try {
      const response = await fetch('/api/query-web-content', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ url, query }),
      });

      if (!response.ok) {
        const errorData = await response.json();
        throw new Error(errorData.message || 'Failed to fetch query results');
      }

      const data = await response.json();
      setQueryResult(data);
    } catch (err) {
      setError(
        err instanceof Error ? err.message : 'An unknown error occurred'
      );
    } finally {
      setIsLoading(false);
    }
  };

  return (
    <div className="min-h-screen py-6 flex flex-col justify-center sm:py-12">
      <div className="relative py-3 sm:max-w-xl sm:mx-auto">
        <div className="absolute inset-0 bg-gradient-to-r from-cyan-400 to-light-blue-500 shadow-lg transform -skew-y-6 sm:skew-y-0 sm:-rotate-6 sm:rounded-3xl"></div>
        <div className="relative px-4 py-10 bg-white shadow-lg sm:rounded-3xl sm:p-20">
          <div className="max-w-md mx-auto">
            <div className="divide-y divide-gray-200">
              <div className="py-8 text-base leading-6 space-y-4 text-gray-700 sm:text-lg sm:leading-7">
                <h1 className="text-3xl font-extrabold text-center text-gray-900 mb-6">
                  Web Content Querier
                </h1>

                <WebContentQuerierForm
                  onSubmit={handleQuery}
                  isLoading={isLoading}
                />

                {error && (
                  <div
                    className="bg-red-100 border border-red-400 text-red-700 px-4 py-3 rounded relative"
                    role="alert"
                  >
                    {error}
                  </div>
                )}

                {isLoading && (
                  <div className="flex justify-center items-center">
                    <div className="animate-spin rounded-full h-8 w-8 border-b-2 border-gray-900"></div>
                  </div>
                )}

                {queryResult && (
                  <QueryResults
                    answer={queryResult.answer}
                    context={queryResult.context}
                  />
                )}
              </div>
            </div>
          </div>
        </div>
      </div>
    </div>
  );
}

Wrapping Thoughts

By combining AI and web scraping, you have created a powerful and efficient system for extracting and querying web content. This approach leverages the precision of AI models like OpenAI’s GPT-3.5-turbo alongside robust scraping tools like the Bright Data Scraping Browser to transform raw web data into meaningful insights. This can be useful in research and content analysis, customer support automation, and a lot more.

The code for this tutorial is available on GitHub; tweak it and improve it for your use.

How to Build a Custom MedusaJS Admin Dashboard with NextJS, Supabase, and Tailwind CSS

Victor Yakubu — Mon, 16 Dec 2024 01:30:01 +0000

When it comes to managing an e-commerce business, no two business needs are the same. For example, one e-commerce store might need a dashboard focused on tracking real-time inventory, while another prioritizes visualizing sales trends and customer engagement. That’s why off-the-shelf admin dashboards or pre-built templates frequently fall short: they’re not designed to accommodate the specific needs of individual businesses.

However, a customizable solution like Medusa solves this by providing the building blocks and REST API endpoints you need to build custom e-commerce solutions that can adapt to the business’s current needs. This ensures you retain control over every aspect of your e-commerce ecosystem.

In this article, you’ll learn how to:

Set up an e-commerce backend using MedusaJS and Supabase.
Build a custom sales dashboard using Next.js and Tailwind CSS.
Extend MedusaJS’s admin functionality to meet specific business requirements.

By the end of this guide, you’ll have an admin dashboard that looks like this:

Prerequisites

To follow along with this tutorial, you need the following:

Basic knowledge of Next.js and PostgreSQL
Node.js (v16 or later) — Install it from nodejs.org.

Setting up your Medusa store locally

Follow these steps to set up your Medusa store:

Step 1: Create a PostgreSQL database using Supabase

Visit Supabase and sign up.
Click “Start your project” and create a new account.
From your dashboard, click “New Project”
Fill in the details for your database, i.e., Project Name and Database password. You can either use the default region or select one closer to you.
After that, click “ Create new project”. It will create a new Supabase project with an entire Postgres database.

Step 2: Get your database connection string

Navigate to Project Settings > Connect.

You will find the connection string under Connection String > URL. Save it — you’ll need it for the Medusa setup.

Step 3: Install and set up Medusa locally

Run the following command to install and set up your Medusa project, connecting it to your Supabase database:

npx create-medusa-app@latest - seed - db-url postgresql://postgres:<password>@<host>.supabase.co:5432/postgres

Replace <password> in the command with the database password you created in Step 1.

Command flags explained:

— seed: Seeds the database with demo data.
— db-url: Specifies your database connection URL.

You can see the other CLI options that create-medusa-app accepts in the documentation.

Once the installation has been completed, your project will be served in the following ports:

Medusa backend: http://localhost:9000
Medusa admin dashboard: http://localhost:7001

To access the admin dashboard, create an admin account or use the default credentials ( Email: admin@medusa-test.com Password: supersecret)

Add a custom Sales Overview page.

Aside from the default features, MedusaJS allows you to add custom routes to your admin dashboard, enabling you to track specific business metrics like sales performance. Here’s how you can add a custom Sales Overview page:

1. Creating a custom Admin UI route

MedusaJS admin routes are React components in the src/admin/routes directory. To create a custom route for sales:

Navigate to src/admin/routes in your project directory.

Create a folder structure: sales/page.tsx.

└── my-store/
└── src/
    └── admin/
        └── routes/
            └── sales/
                └── page.tsx

Inside page.tsx, export a React component for the sales page:

const Sales = () => {
  return (
    <div>
      See the number of products sold and the number remaining in stock
    </div>
  )
}

export default Sales

Visit http://localhost:7001/a/sales to see your custom page. However, it is not accessible from the sidebar; you will fix that in the next section.

2. Adding the route to the Sidebar

To make the route accessible from the sidebar:

Import RouteConfig from @medusajs/admin and define the route configuration:

import { RouteConfig } from "@medusajs/admin"
import { CircleStack } from '@medusajs/icons';

const Sales = () => {
  return (
    <div>
      See the number of product sold and the number left in stock
    </div>
  )
}

export const config: RouteConfig = {
  link: {
    label: "Sales",
    icon: CircleStack,
  },
}

export default Sales

The Sales route will now appear on the admin dashboard sidebar.

Extending your Product Entity to add custom fields

To make the Sales Overview page functional, you need to ensure your database supports all the required data. Specifically, you’ll want to display each product’s current stock, price, and units sold. While the first two are available in the default schema, the units_sold data isn’t. So, how do you add custom fields?

Step 1: Reviewing the current schema

Start by exploring the existing data structure. You can make a GET request to the /admin/products endpoint to see the available fields for each product. This endpoint is accessible at:

http://localhost:9000/admin/products

You’ll notice there isn’t a column for units_sold. This is where customization becomes necessary.

Step 2: Modifying your database schema to the units_sold Column

Since Medusa uses a PostgreSQL database with TypeORM as the ORM, you’ll need to update both the database schema and the MedusaJS product entity model to add new columns.

To start, modify your database schema to include a units_sold column. For better granularity, you can add quarterly columns (units_sold_q1, units_sold_q2, etc.). Run the following SQL commands directly on your database:

ALTER TABLE product ADD COLUMN units_sold_q1 INT DEFAULT 0;
ALTER TABLE product ADD COLUMN units_sold_q2 INT DEFAULT 0;
ALTER TABLE product ADD COLUMN units_sold_q3 INT DEFAULT 0;
ALTER TABLE product ADD COLUMN units_sold_q4 INT DEFAULT 0;

💡 After running these commands, the new columns will exist in your database but won’t yet appear in API responses or the admin dashboard. To fix this, you’ll need to update the MedusaJS entity model.

Step 3: Extending the Product Entity Model

Medusa uses TypeORM to define its models. To add the new columns to the product entity, create a file named product.ts in ./src/models/:

    import { Column, Entity } from "typeorm";
    import { Product as MedusaProduct } from "@medusajs/medusa";
    @Entity()
    export class Product extends MedusaProduct {
    @Column({ default: 0 })
    units_sold_q1: number;
    @Column({ default: 0 })
    units_sold_q2: number;
    @Column({ default: 0 })
    units_sold_q3: number;
    @Column({ default: 0 })
    units_sold_q4: number;
    }

The @Column decorator ensures that TypeORM maps the fields to your database.

Step 4: Updating type definitions

If you’re using TypeScript, add the new columns to your IDE’s autocomplete by extending Medusa’s core Product interface. Create a file at src/index.d.ts:

    export declare module "@medusajs/medusa/dist/models/product" {
    declare interface Product {
    units_sold_q1: number;
    units_sold_q2: number;
    units_sold_q3: number;
    units_sold_q4: number;
     }
    }

Step 5: Exposing new fields in the API

Medusa’s API won’t return your custom columns by default. To include these fields, you need to modify the products endpoint configuration. Create a file at src/loaders/extend-product-fields.ts:

    export default async function () {
    const imports = await import(
    "@medusajs/medusa/dist/api/routes/admin/products/index"
    ) as any;
    imports.defaultAdminProductFields = [
    …imports.defaultAdminProductFields,
    "units_sold_q1",
    "units_sold_q2",
    "units_sold_q3",
    "units_sold_q4",
     ];
    };

Then, register this loader in your Medusa project by adding it to the loaders array in medusa-config.js.

Step 6: Creating a TypeORM data source

To synchronize your new fields with the database, create a TypeORM data source file at the root of your Medusa project. Save it as datasource.js:

    const { DataSource } = require("typeorm");
    const AppDataSource = new DataSource({
    type: "postgres",
    port: 5432,
    username: "<YOUR_DB_USERNAME>",
    password: "<YOUR_DB_PASSWORD>",
    database: "<YOUR_DB_NAME>",
    entities: ["dist/models/*.js"],
    migrations: ["dist/migrations/*.js"],
    });
    module.exports = {
    datasource: AppDataSource,
    };

Step 7: Writing and running migrations

To update the database schema programmatically, create and execute a migration:

Generate a migration file:

npm run build
npx typeorm migration:create src/migrations/AddUnitsSoldColumnToProduct

Edit the generated migration file:

import { MigrationInterface, QueryRunner } from "typeorm";

export class AddUnitsSoldColumnToProduct implements MigrationInterface {
  public async up(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(`
      ALTER TABLE product ADD COLUMN units_sold_q1 INT DEFAULT 0;
      ALTER TABLE product ADD COLUMN units_sold_q2 INT DEFAULT 0;
      ALTER TABLE product ADD COLUMN units_sold_q3 INT DEFAULT 0;
      ALTER TABLE product ADD COLUMN units_sold_q4 INT DEFAULT 0;
    `);
  }

  public async down(queryRunner: QueryRunner): Promise<void> {
    await queryRunner.query(`
      ALTER TABLE product DROP COLUMN units_sold_q1;
      ALTER TABLE product DROP COLUMN units_sold_q2;
      ALTER TABLE product DROP COLUMN units_sold_q3;
      ALTER TABLE product DROP COLUMN units_sold_q4;
    `);
  }
}

Run the migration:

npm run build
npx medusa migrations run

Step 8: Verifying your changes

After running the migration, your database schema will include the new columns, and they’ll be visible in your /admin/products API responses. Test these changes by sending a GET request to the endpoint.

Now, your MedusaJS project is equipped to track and display quarterly sales data, enabling you to create a fully functional Sales Overview page tailored to your needs.

Building the Sales Page UI

Now that the data flows seamlessly between your admin dashboard and the Medusa backend, it’s time to implement the code for your Admin UI. This section will guide you through building the user interface for a Sales Overview Page.

MedusaJS comes bundled with Medusa UI, a React-based design system that includes components, hooks, utility functions, icons, and pre-made Tailwind CSS classes. These tools ensure a consistent and professional design across your admin dashboard.

Your Sales Overview Page will feature three main components:

Header: Displays key metrics like revenue and customer counts.
Bar Chart: Visualizes sales data by quarter.
Recent Orders Section: Lists recent customer orders.

Creating the Sales Page layout

In the page.tsx file, structure the layout of your Sales Page:

    import { RouteConfig } from '@medusajs/admin';
    import { CircleStack } from '@medusajs/icons';
    import TopCards from './components/TopCards';
    import BarChart from './components/BarChart';
    import RecentOrders from './components/RecentOrders';
    const Sales = () => {
    return (
    <div>
    <TopCards />
    <div className="p-4 grid grid-cols-2 gap-4">
    <BarChart />
    <RecentOrders />
    </div>
    </div>
    );
    };
    // Adding route into the admin dashboard sidebar
    export const config: RouteConfig = {
    link: {
    label: 'Sales',
    icon: CircleStack,
    },
    };
    <style></style>;
    export default Sales;

Next, create a component folder under src/admin/routes/sales; inside the component folder, create the following files: TopCards.tsx, BarChart.tsx and RecentOrders.tsx.

Header section

The Header Section displays sales statistics such as quarterly revenue and total customers. Create this component in TopCards.tsx:

src/admin/routes/sales/components/TopCards.tsx

import { Text } from '@medusajs/ui';
const TopCards = () => {
  return (
    <div className="gap-y-large flex flex-col">
      <div className=" 'p-4 gap-y-2xsmall flex flex-col">
        <h2 className="inter-xlarge-semibold">Sales</h2>
        <Text className="inter-base-regular text-grey-50">
          See the number of product sold and the number remaining in stock
        </Text>
      </div>

      <div className=" flex sm:flex-none gap-4 p-4">
        <div className="lg:col-span-2 col-span-1 bg-white flex justify-between w-full border p-4 rounded-lg cursor-pointer hover:shadow-lg transform hover:scale-[103%] transition duration-300 ease-out border-l-[4px] border-[#1F2937]">
          <div className="flex flex-col w-full pb-4">
            <p className="text-2xl font-bold">$7,845</p>
            <p className="text-gray-600">Quarterly Revenue</p>
          </div>
          <p className="bg-green-200 flex justify-center items-center p-2 rounded-lg">
            <span className="text-green-700 text-lg">+18%</span>
          </p>
        </div>
        <div className="lg:col-span-2 col-span-1 bg-white flex justify-between w-full border p-4 rounded-lg cursor-pointer hover:shadow-lg transform hover:scale-[103%] transition duration-300 ease-out border-l-[4px] border-[#1F2937]">
          <div className="flex flex-col w-full pb-4">
            <p className="text-2xl font-bold">$1,14,783</p>
            <p className="text-gray-600">YTD Revenue</p>
          </div>
          <p className="bg-green-200 flex justify-center items-center p-2 rounded-lg">
            <span className="text-green-700 text-lg">+11%</span>
          </p>
        </div>
        <div className="bg-white flex justify-between w-full border p-4 rounded-lg cursor-pointer hover:shadow-lg transform hover:scale-[103%] transition duration-300 ease-out border-l-[4px] border-[#1F2937]">
          <div className="flex flex-col w-full pb-4">
            <p className="text-2xl font-bold">10,845</p>
            <p className="text-gray-600">Customers</p>
          </div>
          <p className="bg-green-200 flex justify-center items-center p-2 rounded-lg">
            <span className="text-green-700 text-lg">+17%</span>
          </p>
        </div>
      </div>
    </div>
  );
};

export default TopCards;

Bar chart section

The Bar chart visualizes sales revenue by quarter. Create this component in BarChart.tsx:

src/admin/routes/sales/components/BarChart.tsx

import React, { useState, useEffect } from 'react';
import { Bar } from 'react-chartjs-2';
import {
  Chart as ChartJS,
  CategoryScale,
  LinearScale,
  BarElement,
  Title,
  Tooltip,
  Legend,
} from 'chart.js';
ChartJS.register(
  CategoryScale,
  LinearScale,
  BarElement,
  Title,
  Tooltip,
  Legend
);

import {useAdminProducts} from 'medusa-react'

const BarChart = () => {

    const { products} = useAdminProducts();
    const [totalRevenueQ1, setTotalRevenueQ1] = useState(null);
    const [totalRevenueQ2, setTotalRevenueQ2] = useState(null);
    const [totalRevenueQ3, setTotalRevenueQ3] = useState(null);
    const [totalRevenueQ4, setTotalRevenueQ4] = useState(null);

    useEffect(() => {
      async function calculateTotalRevenue() {
        // Check if products is available
        if (!products) {
          console.error('Products data is not available.');
          return;
        }

        // Step 1: Map through Products and calculate revenue for each product, each quarter
        const revenuePerProductQ1 = products.map((product) => {
          const amount = product.variants[0].prices[0].amount;
          const sold_q1 = product.units_sold_q1;
          console.log('amount:', amount, 'sold_q1:', sold_q1);
          return amount * sold_q1;
        });
        const revenuePerProductQ2 = products.map((product) => {
          const amount = product.variants[0].prices[0].amount;
          const sold_q2 = product.units_sold_q2;
          return amount * sold_q2;
        });
        const revenuePerProductQ3 = products.map((product) => {
          const amount = product.variants[0].prices[0].amount;
          const sold_q3 = product.units_sold_q3;
          return amount * sold_q3;
        });
        const revenuePerProductQ4 = products.map((product) => {
          const amount = product.variants[0].prices[0].amount;
          const sold_q4 = product.units_sold_q4;
          return amount * sold_q4;
        });

        // Step 2: Sum the individual multiplication results to get the total revenue for each quarter
        const totalRevenueQ1 = revenuePerProductQ1.reduce(
          (total, revenue) => total + revenue,
          0
        );
        const totalRevenueQ2 = revenuePerProductQ2.reduce(
          (total, revenue) => total + revenue,
          0
        );
        const totalRevenueQ3 = revenuePerProductQ3.reduce(
          (total, revenue) => total + revenue,
          0
        );
        const totalRevenueQ4 = revenuePerProductQ4.reduce(
          (total, revenue) => total + revenue,
          0
        );

        // Set quarterly totalRevenue states with the calculated value
        setTotalRevenueQ1(totalRevenueQ1);
        setTotalRevenueQ2(totalRevenueQ2);
        setTotalRevenueQ3(totalRevenueQ3);
        setTotalRevenueQ4(totalRevenueQ4);
      }
      // calculate total revenue when products data is available
      if (products) {
        calculateTotalRevenue();
      }

    }, [products]);

   const data = {
     labels: ['Q1', 'Q2', 'Q3', 'Q4'],
     datasets: [
       {
         label: 'Total Revenue',
         data: [totalRevenueQ1, totalRevenueQ2, totalRevenueQ3, totalRevenueQ4],
         backgroundColor: '#32de84',
         borderColor: 'rgb(0,128,0)',
       },
     ],
   };

  return (
    <>
      <div className="w-full md:col-span-2 relative lg:h-[70vh] h-[50vh] m-auto p-4 border rounded-lg bg-white">
        <div className="w-100% h-[70vh]">
          <Bar data={data} />
        </div>
      </div>
    </>
  );
};

export default BarChart;

The useAdminProducts hook gives you access to the list of available products in the database and also automatically handles auth, passing along credentials in its requests to the Medusa backend as long as you’re signed in to the Admin dashboard. You can learn more about the Admin APIs for managing products here.

Recent Orders section

The Recent Orders Section lists customer orders. Create this component in RecentOrders.tsx:

src/admin/routes/sales/components/RecentOrders.tsx

import { ShoppingBag } from '@medusajs/icons';
import { useAdminOrders } from 'medusa-react';

const RecentOrders = () => {
  const { orders, isLoading } = useAdminOrders();

  return (
    <div className="w-full col-span-1 relative lg:h-[70vh] h-[50vh] m-auto p-4 rounded-lg bg-white overflow-scroll">
      <h1 className="text-center inter-large-semibold text-[#1F2937]">
        Recent Orders
      </h1>
      {isLoading && <span>Loading...</span>}
      {orders && !orders.length && <span>No Orders</span>}
      {orders && orders.length > 0 && (
        <ul>
          {orders.map((order, id) => (
            <li
              key={order.id}
              className="bg-gray-50 hover:bg-gray-100 rounded-lg my-3 p-2 flex items-center cursor-pointer"
            >
              <div className="bg-green-200 rounded-lg p-3">
                <ShoppingBag />
              </div>
              <div className="pl-4">
                <p className="text-gray-800 font-bold">
                  €{order.payments[0].amount}
                </p>
                <p className="text-gray-400 text-sm">
                  {order.customer.first_name}
                </p>
              </div>
              <p className="lg:flex md:hidden absolute right-6 text-sm">
                {order.items[0].title}
              </p>
            </li>
          ))}
        </ul>
      )}
    </div>
  );
};

export default RecentOrders;

The useAdminOrders hook gives you access to the list of orders made by clients, and only authorised credentials have access to this.
The useAdminOrders hook also gives you access to all of its properties like first_name, payments_amount, title, etc.

Your Sales Overview Page is now functional and visually appealing with these components. You can modify and add as many functionalities as you want to suit your needs.

Conclusion

This tutorial demonstrated how to build a custom MedusaJS admin dashboard using Next.js, Supabase, and Tailwind CSS. By following these steps, you learned how to set up a Medusa backend, create a dynamic user interface, and extend the Medusa functionalities to build tailored components like the Sales Overview Page.

Check out the Medusa documentation to learn more about advanced features and other customization options you can add to your e-commerce experience.

How to Scrape and Analyse Data for Free using AI: From Collection to Insight

Victor Yakubu — Sun, 15 Dec 2024 22:31:33 +0000

While some websites are straightforward to scrape by using just Selenium, Puppeteer, and the like, other websites that implement advanced security measures such as CAPTCHAs and IP bans may prove difficult. To overcome these challenges and ensure you can scrape 99% of websites for free using the Scraper, you will be building this in this article; you will be integrating a proxy tool in your code that will help in bypassing these security measures.

However, collecting the data is just one step; what you do with that data is equally, if not more, important. Often, this requires painstakingly sifting through large volumes of information manually. But what if you could automate this process? By leveraging a language model (LLM), you can not only collect data but also query it to extract meaningful insights — saving time and effort.

In this guide, you’ll learn how to combine web scraping with AI to build a powerful tool for collecting and analysing data at scale for free. Let’s dive in!

Prerequisites

Before you begin, ensure you have the following:

Basic Python knowledge, as this project involves writing and understanding Python code.
Install Python (3.7 or later) on your system. You can download it from python.org.

Installation and Setup

To continue with this tutorial, complete the following steps:

Follow these steps to set up your environment and prepare for building the AI-powered scraper.

1. Create a Virtual Environment

First, set up a virtual environment to manage your project’s dependencies. This will ensure you have an isolated space for all the required packages.

Create a New Project Directory: Open your terminal (or Command Prompt/PowerShell on Windows) and create a new directory for your project:

mkdir ai-website-scraper
cd ai-website-scraper

2. Create the Virtual Environment:

Run the following command to create the virtual environment:

On Windows:

python -m venv venv

On macOS/Linux:

python3 -m venv venv

This creates a venv folder that will store the virtual environment.

2. Activate the Virtual Environment

Activate the virtual environment to begin working within it:

On Windows:

.\venv\Scripts\activate

On macOS/Linux:

source venv/bin/activate

Your terminal prompt will change to show (venv), confirming you’re now inside the virtual environment.

3. Install Required Dependencies

Now, install the libraries your project needs. Create a requirements.txt file in your project directory and add the following dependencies:

streamlit
selenium
Beautifulsoup4
langchain
langchain-ollama
lxml
html5lib

These packages are essential for scraping, data processing, and building the UI:

streamlit: This is used to create the interactive user interface.
Selenium: For scraping website content.
beautifulsoup4: For parsing and cleaning the HTML.
langchain and langchain-ollama: This is for integrating with the Ollama LLM and processing text.
lxml and html5lib: For advanced HTML parsing.

Install the dependencies by running the following command:

(Ensure that you are in the folder where the file is located before running the command)

pip install -r requirements.txt

Building the UI with Streamlit

Streamlit makes it easy to create an interactive user interface (UI) for Python applications. In this section, you will build a simple, user-friendly interface where users can input a URL and display the scraped data.

1. Set Up the Streamlit Script

Create a file named ui.py in your project directory. This script will define the UI for your scraper. Use the code below to structure your application:

import streamlit as st
import pathlib
from main import scrape_website

# function to load css from the assets folder
def load_css(file_path):
with open(file_path) as f:
st.html(f"<style>{f.read()}</style>")
# Load the external CSS
css_path = pathlib.Path("assets/style.css")
if css_path.exists():
load_css(css_path)
st.title("AI Scraper")
st.markdown(
"Enter a website URL to scrape, clean the text content, and display the result in smaller chunks."
)
url = st.text_input(label= "", placeholder="Enter the URL of the website you want to scrape")
if st.button("Scrape", key="scrape_button"):
st.write("scraping the website…")
result = scrape_website(url)
st.write("Scraping complete.")
st.write(result)

The st.title and st.markdown functions set up the application title and provide instructions for users.
The st.text_input component lets users input the URL of the website they want to scrape.
Clicking the “Scrape” button triggers the scraping logic, displaying progress messages using st.info.

You can learn more about streamlit components from their documentation.

2. Add Custom Styles

To style your application, create an assets folder in your project directory and add a style.css file. Customize the Streamlit interface with CSS:

.stAppViewContainer {
background-image: url("https://images.unsplash.com/photo-1732979887702-40baea1c1ff6?q=80&w=2832&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D");
background-size: cover;
color: black;
}
.stAppHeader {
background-color: rgba(0, 0, 0, 0);
}
.st-ae {
background-color: rgba(233, 235, 234, 0.895);
}
.st-emotion-cache-ysk9xe {
color: black;
}
.st.info, .stAlert {
background-color: black;
}
.st-key-scrape_button button {
display: inline-block;
padding: 10px 20px;
font-size: 16px;
color: #fff;
background-color: #007bff;
border: none;
border-radius: 5px;
cursor: pointer;
animation: pulse 2s infinite;
}
.st-key-scrape_button button:hover {
background-color: #0056b3;
color: #fff;
}

3. Run the Streamlit app

In your project directory, run the following command:

streamlit run ui.py

This will launch a local server, and you should see a URL in the terminal, usually http://localhost:8501. Open this URL in your browser to interact with the web application.

Scraping website with Selenium

Next, write the code to extract the HTML content of any webpage using Selenium. However, for the code to work, you need a Chrome WebDriver.

Install ChromeDriver for Selenium

Selenium requires a WebDriver to interact with web pages. Here’s how to set it up:

Download ChromeDriver:
Visit the ChromeDriver website and download the version matching your Google Chrome browser.
Add ChromeDriver to PATH

After downloading ChromeDriver, extract the file and copy the application file name “chromedriver” and paste it in your project folder.

When this is done, create a new file called main.py and implement the code below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Function to scrape HTML from a website
def scrape_website(website_url):
# Path to WebDriver
webdriver_path = "./chromedriver" # Replace with your WebDriver path
service = Service(webdriver_path)
driver = webdriver.Chrome(service=service)
try:
# Open the website
driver.get(website_url)
# Wait for the page to fully load
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME, "body")))
# Extract the HTML source
html_content = driver.page_source
return html_content
finally:
# Ensure the browser is closed after scraping
driver.quit()

Save and run the code; you should get all the HTML of the page you scraped displayed in your streamlit application like this:

Using a Proxy Provider to Bypass Website with Captcha and IP Bans

While you can now retrieve the HTML of a website, the above code may not work for sites with advanced anti-scraping mechanisms such as CAPTCHA challenges or IP bans. For example, scraping a site like Indeed or Amazon using Selenium may result in a CAPTCHA page blocking access. This happens because the website detects that a bot is trying to access its content. If this behaviour persists, the site may eventually ban your IP address, preventing further access.

To fix this, integrate Bright Data’s Scraping Browser into your script. The scraping browser is a robust tool that leverages multiple proxy networks, including residential IPs, to bypass anti-scraping defenses. It handles unblocking pages by managing custom headers, browser fingerprinting, CAPTCHA solving, and more. This ensures that your scraping efforts remain undetected while accessing content seamlessly.

Setting up Bright Data’s Scraping Browser for free

Signing up — go to Bright Data’s homepage and click on “Start Free Trial”. If you already have an account with Bright Data, you can just log in.
After logging in, click on “Get Proxy Products”.

Click on the “Add” button and select “Scraping Browser.”

Next, you will be taken to the “Add zone” page, where you will be required to choose a name for your new scraping browser proxy zone. After that, click on “Add”.

After this, your proxy zone credentials will be created. You will need these details in your script to bypass any anti-scraping mechanisms used on any website.

You can also check out Bright Data’s developer documentation for more details about the scraping browser.

In your main.py file, modify the code to include the scraping browser. You will notice that this code is much cleaner and shorter than the previous code.

from selenium.webdriver import Remote, ChromeOptions
from selenium.webdriver.chromium.remote_connection import ChromiumRemoteConnection
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
AUTH = '<username>:<passord>'
SBR_WEBDRIVER = f'@brd.superproxy.io:9515'">https://{AUTH}@brd.superproxy.io:9515'
# Function to scrape HTML from a website
def scrape_website(website_url):
print("Connecting to Scraping Browser…")
sbr_connection = ChromiumRemoteConnection(SBR_WEBDRIVER, "goog", "chrome")
with Remote(sbr_connection, options=ChromeOptions()) as driver:
driver.get(website_url)
print("Waiting captcha to solve…")
solve_res = driver.execute(
"executeCdpCommand",
{
"cmd": "Captcha.waitForSolve",
"params": {"detectTimeout": 10000},
},
)
print("Captcha solve status:", solve_res["value"]["status"])
print("Navigated! Scraping page content…")
html = driver.page_source
return html

Replace <username> and <password> with your scraping browser username and password.

Cleaning the Dom content

After scraping the HTML content of a website, it’s often filled with unnecessary elements such as JavaScript, CSS styles, or unwanted tags that do not contribute to the core information you’re extracting. To make the data more structured and useful for further processing, you need to clean the DOM content by removing irrelevant elements and organizing the text.

This section explains how to clean the HTML content, extract meaningful text, and split it into smaller chunks for downstream processing. The cleaning process is essential for preparing data for tasks like natural language processing or content analysis.

Code Walkthrough for Cleaning DOM Content

Here’s the code that will be added to main.py to handle cleaning the DOM content:

from bs4 import BeautifulSoup

# Extract the body content from the HTML
def extract_body_content(html_content):
soup = BeautifulSoup(html_content, "html.parser")
body_content = soup.body
if body_content:
return str(body_content)
return ""
# Clean the body content by removing scripts, styles, and other unwanted elements
def clean_body_content(body_content):
soup = BeautifulSoup(body_content, "html.parser")
# Remove <script> and <style> tags
for script_or_style in soup(["script", "style"]):
script_or_style.extract()
# Extract cleaned text with each line separated by a newline
cleaned_content = soup.get_text(separator="\n")
cleaned_content = "\n".join(
line.strip() for line in cleaned_content.splitlines() if line.strip()
)
return cleaned_content
# Split the cleaned content into smaller chunks for processing
def split_dom_content(dom_content, max_length=5000):
return [
dom_content[i : i + max_length] for i in range(0, len(dom_content), max_length)
]

What the Code Does

Extracting the Body Content:

The extract_body_content function uses BeautifulSoup to parse the HTML and extract the
tag’s content.
If a <body> tag exists, the function returns it as a string. Otherwise, it returns an empty string.

Cleaning the Content:

The clean_body_content function processes the extracted content to remove unnecessary elements:
1. <script> and <style> tags are removed to eliminate JavaScript and CSS.
2. The function retrieves the plain text from the cleaned content.
3. It formats the text by stripping empty lines and extraneous spaces.

3. Splitting the Content:

The split_dom_content function takes the cleaned content and splits it into smaller chunks with a default maximum length of 5,000 characters.
This is useful for processing large amounts of text in manageable pieces, especially when passing data to models with token or input size limits.

Save your changes and test your application. You should get an output like this after scraping a website.

Parsing the Dom content to Ollama

Once the DOM content is cleaned and prepared, the next step is parsing the information to extract specific details using Ollama, a large language model (LLM) integrated with LangChain. Ollama is a CLI tool used to download and run LLMs locally. However, before using Ollama, you have to do the following installations:

If you haven’t, download and install Ollama from the official website. You can install it on Mac using the command Homebrew.

brew install ollama

Next, install any model from this list; there are models like Phi3, Mistral, Gemma 2, etc.; each has its own system requirements. This code uses the phi3 mainly because it's lightweight.

ollama pull phi3

After installation, you can call on that model from your script using LangChain to provide meaningful insights from the data that will be sent to it.

Here’s how to set up the functionality to parse DOM content into phi3 model

Code walkthrough for llm.py

The following code implements the logic to parse DOM chunks with Ollama and extract relevant details:

from langchain_ollama import OllamaLLM
from langchain_core.prompts import ChatPromptTemplate
# Template to instruct Ollama for parsing
template = (
"You are tasked with extracting specific information from the following text content: {dom_content}. "
"Please follow these instructions carefully: \n\n"
"1. **Extract Information:** Only extract the information that directly matches the provided description: {parse_description}. "
"2. **No Extra Content:** Do not include any additional text, comments, or explanations in your response. "
"3. **Empty Response:** If no information matches the description, return an empty string ('')."
"4. **Direct Data Only:** Your output should contain only the data that is explicitly requested, with no other text."
)
# Initialize the Ollama model
model = OllamaLLM(model="phi3")
# Function to parse DOM chunks with Ollama
def parse_with_ollama(dom_chunks, parse_description):
prompt = ChatPromptTemplate.from_template(template)
chain = prompt | model
parsed_results = []
for i, chunk in enumerate(dom_chunks, start=1):
if not chunk.strip(): # Skip empty chunks
print(f"Skipping empty chunk at batch {i}")
continue
try:
print(f"Processing chunk {i}: {chunk[:100]}…") # Print a preview
print(f"Parse description: {parse_description}")
response = chain.invoke(
{
"dom_content": chunk,
"parse_description": parse_description,
}
)
print(f"Response for batch {i}: {response}")
parsed_results.append(response)
except Exception as e:
print(f"Error parsing chunk {i}: {repr(e)}")
parsed_results.append(f"Error: {repr(e)}")
return "\n".join(parsed_results)

What the code does.

Instruction Template:

Provides precise guidance for Ollama on what information to extract.
Ensures the output is clean, concise, and relevant to the parsing description.

2. Chunk Processing:

The parse_with_ollama function iterates through the DOM chunks, processing each with the LLM.
Skips empty chunks to optimize performance.

3. Error Handling:

Handles errors gracefully, logs them, and continues processing remaining chunks.

Updating the file ui.py file

Add the following code to the ui.py file to allow users to input parsing instructions to the LLM and view results:

from main import scrape_website, extract_body_content, clean_body_content, split_dom_content

from llm import parse_with_ollama

rest of the code....

if "dom_content" in st.session_state:

parse_description = st.text_area("Enter a description to extract specific insights from your scraped data:")

if st.button("Parse Content", key="parse_button"):

if parse_description.strip() and st.session_state.get("dom_content"):

st.info("Parsing the content…")

dom_chunks = split_dom_content(st.session_state.dom_content)

parsed_result = parse_with_ollama(dom_chunks, parse_description)

st.text_area("Parsed Results", parsed_result, height=300)

else:

st.error("Please provide valid DOM content and a description to parse.")

How It Works in the UI

User Input:

The user provides a natural language description of the data to extract in a text area.

2. Parsing Trigger:

When the Parse Content button is clicked, the cleaned DOM content is split into manageable chunks and passed to parse_with_ollama.

3. Results Display:

The parsed results are displayed in a text area, allowing users to review the extracted information.

With this done, the scraper can now provide responses to your prompts based on the data scraped.

What’s next?

The combination of web scraping and AI opens up exciting possibilities for data-driven insights. Beyond collecting and saving data, you can now leverage AI to optimize the process of gaining insight from the data scraped. This is useful for marketing and sales teams, data analysis, business owners, and a lot more.

You can find the complete code for the AI scraper here. Feel free to experiment with it and adapt it to your unique needs. Contributions are also welcome — if you have ideas for improvements, consider creating a pull request!

You can also take this further. Here are some ideas:

Experiment with Prompts: Tailor your prompts to extract specific insights or address unique project requirements.
User Interface
Integrate other LLM Models: Explore other language models like OpenAI, Gemini, etc to further optimize your data analysis.

The Top 10 AI Tools Every SaaS Startup Needs in 2025

Victor Yakubu — Mon, 25 Nov 2024 13:14:39 +0000

HubSpot AI CRM, Fiverr Logo Maker, Jasper, Zapier, ClickUp, Notion AI, Synthesia, and more. Discover the essential AI tools your SaaS startups need for success.

SaaS AI tools allow companies to streamline workflows, automate repetitive tasks, and uncover insights that drive smarter business decisions. By boosting productivity and enhancing customer interactions, AI has become a powerful resource for startups looking to scale without stretching resources.

But with the array of AI tools available, which ones are right for your startup? This article will guide you through the top ten AI tools every SaaS startup should consider in 2025, covering solutions that balance innovation and affordability to help you stay competitive without breaking the bank.

What Should You Look for in SaaS AI Tools?

Selecting the right AI tools can accelerate growth and efficiency for startups. Here’s what to prioritize:

Customizability: Look for tools that allow adjustments to fit your business needs, as flexible AI solutions adapt better to unique startup goals.
Predictive Analytics: Choose tools with predictive capabilities to forecast trends, helping you make data-driven decisions and stay ahead in a competitive market.
Text Analysis: Text analysis tools can interpret customer feedback, enabling more personalized services and enhancing customer engagement.
Workflow Automation: Automate repetitive tasks with tools that streamline operations, allowing teams to focus on strategic goals rather than routine processes.
Easy to Use: Prioritize user-friendly tools that require minimal training, ensuring quick adoption across your team, even if you lack in-house AI expertise.
Data Security: Select tools with strong data security features, safeguarding sensitive information and maintaining compliance with privacy regulations.
Integration Powerhouse: Opt for AI tools that easily integrate with existing platforms, allowing seamless data exchange across various functions like CRM, marketing, and analytics.
Scalability: Startups should invest in scalable tools that grow with their business so they won’t need to switch platforms as their requirements expand.

If you want a customized AI functionality integrated into your products, you can find vetted freelance AI specialists on Fiverr Pro who can help you configure and integrate AI into your products on a budget, ensuring you make the most of your investment without the need for a dedicated AI team.

Top 10 Essential AI Tools for SaaS Startups in 2025

HubSpot AI CRM — Customer Relationship Management

HubSpot’s AI-powered CRM, enhanced with the advanced “Breeze” suite, offers SaaS startups over 80 AI-driven tools designed to streamline workflows across sales, marketing, and customer support. Announced at INBOUND 2024, the CRM integrates automation tools like Breeze Copilot, which analyzes customer interactions and creates competitor research reports, helping teams save time and make data-driven decisions.

For marketing, Breeze’s Content Agent generates on-brand content, while the Social Media Agent schedules and adapts posts to audience preferences. In sales, the Prospecting Agent personalizes outreach messages and optimizes timing for engagement. Lastly, the Customer Support Agent efficiently handles common inquiries, escalating only when needed, making HubSpot’s CRM an ideal choice for startups looking to scale customer management.

Key Features

Copilot and Breeze Agents: Copilot assists users with various tasks, and there are specialized agents for content creation, social media, prospecting, and customer support. Meanwhile, the Breeze agents leverage AI to simplify workflows, improving productivity for marketing and sales teams.
Breeze Intelligence: This feature offers data enrichment using a vast database to provide insights into buyer profiles and intent, helping teams focus on high-fit leads.
Content Remix and Marketing Analytics Suite: In the Marketing Hub, Content Remix repurposes videos into other formats for extended campaigns, while the Analytics Suite centralizes performance metrics, helping users optimize campaigns more effectively.
Agent.ai: An innovative addition, agent.ai allows users to build custom agents within HubSpot’s ecosystem, which can be hired to perform complex marketing and sales tasks, promoting an extensive network of customizable AI-driven agents.

Pricing

HubSpot offers a tiered model with Starter plans around $15/month, Professional around $800/month, and Enterprise plans around $3,600/month. Specific Breeze features, especially within the new Breeze Intelligence and agent.ai, may have additional costs or be available at higher plan levels.

Fiverr Logo Maker: Streamline Your Branding with AI

**Fiverr Logo Maker** is a powerful AI-driven tool tailored for startups and businesses looking for professional branding solutions. Designed for simplicity and effectiveness, it enables users to create stunning logos without needing advanced design skills. Its standout features make it one of the most versatile tools for branding in 2025:

Key Features

AI-Powered Customization: Fiverr’s Logo Maker uses advanced algorithms to generate logo options based on user preferences, including industry, style, and colour palette.
All-Inclusive Branding Packages: It provides high-resolution logos, transparent background files, and additional assets like social media kits and brand style guidelines, ensuring a cohesive brand presence.
Unlimited Edits and Revisions: Users can refine their logos as needed, creating seasonal or campaign-specific variations for consistent adaptability.
Commercial Usage Rights: All logos come with full commercial rights, allowing businesses to use them on merchandise, advertisements, and more.

Why Choose Fiverr Logo Maker?

Conversation Analytics: Analyzes customer calls, meetings, and emails, highlighting trends and areas for improvement in sales pitches and customer engagement.
Deal Intelligence: Tracks deal progress and provides real-time insights into deal risks, helping teams take proactive action on high-priority deals.
Coaching Insights: Identifies coaching opportunities based on team performance, enabling sales managers to train and develop their teams more effectively.
Forecasting and Pipeline Management: Provides accurate forecasting tools, helping startups maintain better control over their sales pipeline.

Pricing

Gong operates on a custom pricing model based on team size and specific needs. This pricing aligns with Gong’s value for revenue-focused teams who rely on high-quality analytics to drive sales performance.

Conclusion

In this age and time, AI tools are not just an option but a necessity for SaaS startups aiming to streamline operations, enhance productivity, and remain competitive in a fast-paced digital landscape. From HubSpot AI CRM’s smart customer relationship management to Synthesia’s advanced video creation capabilities, these tools empower businesses to automate workflows, improve customer engagement, and make data-driven decisions.

Factors such as customizability, scalability, and data security are crucial when selecting AI tools, especially for startups that might lack in-house AI expertise. Fiverr Pro offers vetted AI specialists who can bridge this gap and ensure the successful integration of these technologies.