DEV Community

The AI Entrepreneur
The AI Entrepreneur

Posted on

One API Call to Analyze Any Website: Tech Stack, SEO, Contacts and More

#javascript #python #api #webdev

I got tired of calling 7 different APIs just to understand a single website. So I built one that does everything in a single call.

Historically, getting a complete picture of a website required a "Frankenstein" stack of integrations. You would need one API for tech stack detection (like Wappalyzer), another for SEO metrics, a third for email/contact scraping, a fourth for WHOIS data, and yet another to clean the HTML into Markdown for your LLMs.

Managing seven different API keys, seven different billing cycles, and seven different JSON formats is a developer's nightmare. It introduces latency, increases costs, and creates multiple points of failure.

Enter the WebSight API. Hosted on the Apify platform, this tool collapses those seven layers into a single, high-performance API call. In this tutorial, we will explore how to use the WebSight API to extract tech stacks, SEO audits, contacts, and AI-ready content in one go.

The Problem: The Seven-API Burden

Before we dive into the solution, let's look at what developers typically have to juggle to get comprehensive website intelligence:

  1. Tech Stack Detection: Identifying if a site uses React, Shopify, or WordPress.
  2. SEO Audit: Checking meta tags, H1 headers, and internal link structures.
  3. Contact Extraction: Finding emails, LinkedIn profiles, and Twitter handles.
  4. Structured Data: Parsing JSON-LD or Microdata for products or events.
  5. AI Content Scoring: Determining the readability and "human-ness" of the content.
  6. Domain Intel: Checking SSL status, WHOIS records, and server location.
  7. Clean Content Extraction: Converting messy HTML into clean Markdown for GPT-4 or Claude.

Using separate services for these tasks means your application has to wait for seven different round-trips to various servers. WebSight API solves this by performing all these checks server-side in a single execution.

Introducing WebSight API

The WebSight API is a comprehensive website analysis api designed for speed and depth. By calling a single endpoint, you receive a structured JSON object containing everything from the site's underlying framework to the social media profiles of its founders.

Endpoint: https://george-the-developer--websight-api.apify.actor/analyze?url=URL

The 7 Modules of WebSight

  1. Tech Stack Detection API: Identifies over 5,000 technologies, including CMS, UI frameworks, analytics, and payment processors.
  2. SEO Audit API: Provides a real-time health check on title tags, descriptions, canonicals, and image alt texts.
  3. Contact & Social Intel: Scrapes the DOM and headers for emails, phone numbers, and links to platforms like X, LinkedIn, and Instagram.
  4. Structured Data Parser: Extracts and flattens Schema.org data, making it easy to read product prices or review scores.
  5. AI Content Metrics: Scores the content for reading ease and provides a summary of the page's intent.
  6. Infrastructure & Security: Returns SSL details, IP addresses, and hosting providers.
  7. Markdown Engine: Delivers the main content of the page in clean Markdown, stripped of ads and navigation menus—perfect for RAG (Retrieval-Augmented Generation) applications.

Implementation: JavaScript and Python

Integrating WebSight into your workflow is straightforward. Below are examples of how to call the API using the two most popular languages for web automation.

JavaScript (Fetch API) 

const analyzeWebsite = async (targetUrl) => {
    const apiEndpoint = \`https://george-the-developer--websight-api.apify.actor/analyze?url=\${encodeURIComponent(targetUrl)}\`;

    try {
        const response = await fetch(apiEndpoint);
        if (!response.ok) throw new Error('Network response was not ok');

        const data = await response.json();
        console.log("Tech Stack:", data.tech_stack);
        console.log("SEO Health:", data.seo.score);
        console.log("Main Content (Markdown):", data.content.markdown.substring(0, 100) + "...");
    } catch (error) {
        console.error("Error analyzing website:", error);
    }
};

analyzeWebsite('https://stripe.com');
Enter fullscreen mode Exit fullscreen mode

Python (Requests) 

import requests
import json

def get_website_intel(target_url):
    api_url = f"https://george-the-developer--websight-api.apify.actor/analyze?url={target_url}"

    response = requests.get(api_url)

    if response.status_code == 200:
        data = response.json()

        # Accessing specific modules
        contacts = data.get('contacts', {})
        tech = data.get('tech_stack', [])

        print(f"Emails Found: {contacts.get('emails')}")
        print(f"Technologies: {', '.join(tech[:5])}")

        return data
    else:
        print(f"Failed to retrieve data: {response.status_code}")

intel = get_website_intel('https://stripe.com')
Enter fullscreen mode Exit fullscreen mode

Realistic Output: Analyzing Stripe.com

When you point the WebSight API at a sophisticated site like Stripe.com, you get a glimpse of its true power. Instead of a wall of HTML, you get a clean, actionable website intelligence api response.

JSON Output Fragment (Stripe.com) 

{
  "url": "https://stripe.com",
  "status": 200,
  "metadata": {
    "title": "Stripe | Financial Infrastructure for the Internet",
    "description": "Stripe is a suite of APIs powering online payment processing and commerce solutions for internet businesses of all sizes."
  },
  "tech_stack": [
    "React",
    "Next.js",
    "Stripe Payments",
    "Amazon Web Services",
    "Google Analytics",
    "Marketo",
    "HSTS"
  ],
  "seo": {
    "score": 94,
    "h1_count": 1,
    "missing_alt_tags": 0,
    "is_mobile_friendly": true,
    "sitemap_found": true
  },
  "contacts": {
    "emails": ["support@stripe.com", "sales@stripe.com"],
    "social": {
      "twitter": "https://twitter.com/stripe",
      "linkedin": "https://linkedin.com/company/stripe",
      "github": "https://github.com/stripe"
    }
  },
  "infrastructure": {
    "ip": "3.233.126.24",
    "provider": "Amazon Data Services",
    "ssl_expiry": "2025-12-01T00:00:00Z"
  },
  "content": {
    "word_count": 1240,
    "readability_score": "College Level",
    "markdown": "# Financial infrastructure for the internet..."
  }
}
Enter fullscreen mode Exit fullscreen mode

Key Use Cases

1. Competitive Intelligence

Monitor your competitors' tech stacks. Did they just switch from Shopify to Next.js? Are they using a new A/B testing tool? WebSight API allows you to track these changes at scale without manual inspection.

2. Hyper-Targeted Lead Generation

For B2B companies, the "Tech Stack Detection API" is a goldmine. If you sell a React performance monitoring tool, you can use WebSight to find every company in a specific niche that is using React and has a low SEO score, then reach out to them via the extracted emails.

3. SEO Auditing at Scale

Instead of running a manual audit tool for every new client, agencies can programmatically generate a health report in seconds. By checking meta tags, H1s, and site speed indicators in one call, you can provide instant value during discovery calls.

4. Giving AI Agents "Eyes"

If you are building an AI agent that needs to "browse" the web, providing it with raw HTML is expensive and noisy. WebSight API provides the AI with clean Markdown and structured metadata. This reduces token usage and improves the accuracy of the AI's summaries and actions.

Conclusion

The era of fragmented web scraping is over. By consolidating tech detection, SEO auditing, contact extraction, and content cleaning into a single website analysis api, the WebSight API allows developers to focus on building features rather than managing infrastructure.

Whether you are building the next big SaaS or a simple automation script, the ability to get 360-degree website intelligence with one API call is a game-changer for your development velocity.

Ready to try it? Head over to the WebSight API on Apify and start analyzing.

Top comments (0)