DEV Community

jsmanifest
jsmanifest

Posted on

How to Monitor Any URL for Dead Links (Including Tricky Soft 404s)

#javascript #webdev #api #programming

The Problem: Not All Dead Links Return 404

When a webpage goes down, most developers expect a 404 Not Found response. But many services don't play by those rules.

Soft 404s return 200 OK with error content:

Request: GET https://example.com/deleted-page
Response: 200 OK
Body: <html>...Page not found...</html>
Enter fullscreen mode Exit fullscreen mode

Standard uptime monitors see 200 OK and report everything as healthy. The link is actually dead.

This happens everywhere:

  • Any website: Custom error pages, removed content, expired pages
  • Documentation sites: Outdated links, reorganized docs, deprecated APIs
  • E-commerce: Out-of-stock products, discontinued items
  • File hosts: K2S, Nitroflare, Rapidgator, MEGA, MediaFire
  • Video platforms: YouTube ("Video unavailable"), Vimeo, Dailymotion
  • Social media: Deleted posts, suspended accounts
  • Cloud storage: Expired Google Drive shares, revoked permissions
  • APIs & webhooks: Deprecated endpoints, changed URLs

Anyone managing external links—whether it's a documentation site, resource page, or link directory—has likely experienced users reporting dead links that monitoring tools missed.

The Solution: Universal Link Monitoring

DeadLinkRadar monitors any URL with smart dead link detection. It works in two ways:

  1. Universal monitoring: Any URL gets checked with smart soft-404 detection
  2. Deep integrations: 30+ platforms (YouTube, file hosts, social media) get specialized detection

This tutorial covers:

  1. Building a JavaScript API client
  2. Bulk importing links (any URL type)
  3. Setting up Discord and Telegram alerts
  4. Automating daily link audits

Building the API Client

First, create an API key from the DeadLinkRadar dashboard. Then build a client class:

// deadlinkradar-client.js
const API_BASE = 'https://deadlinkradar.com/api/v1'

class DeadLinkRadarClient {
  constructor(apiKey) {
    this.apiKey = apiKey
  }

  async request(endpoint, options = {}) {
    const response = await fetch(`${API_BASE}${endpoint}`, {
      ...options,
      headers: {
        'X-API-Key': this.apiKey,
        'Content-Type': 'application/json',
        ...options.headers,
      },
    })

    if (!response.ok) {
      const error = await response.json()
      throw new Error(error.error?.message || 'API request failed')
    }

    return response.json()
  }

  // Account & Usage
  getAccount = () => this.request('/account')
  getUsage = () => this.request('/usage')

  // Links
  getLinks = (params = {}) => {
    const query = new URLSearchParams(params).toString()
    return this.request(`/links${query ? `?${query}` : ''}`)
  }

  createLink = (url, groupId = null) =>
    this.request('/links', {
      method: 'POST',
      body: JSON.stringify({ url, group_id: groupId }),
    })

  deleteLink = (id) => this.request(`/links/${id}`, { method: 'DELETE' })

  checkLinkNow = (id) => this.request(`/links/${id}/check`, { method: 'POST' })

  getLinkHistory = (id) => this.request(`/links/${id}/history`)

  // Batch operations
  batchCreateLinks = (links) =>
    this.request('/links/batch', {
      method: 'POST',
      body: JSON.stringify({ action: 'create', links }),
    })

  // Groups
  getGroups = () => this.request('/groups')

  createGroup = (name, description = '') =>
    this.request('/groups', {
      method: 'POST',
      body: JSON.stringify({ name, description }),
    })

  // Webhooks
  getWebhooks = () => this.request('/webhooks')

  createWebhook = (url, events = ['link.dead']) =>
    this.request('/webhooks', {
      method: 'POST',
      body: JSON.stringify({ url, events }),
    })

  testWebhook = (id) => this.request(`/webhooks/${id}/test`, { method: 'POST' })
}

export default DeadLinkRadarClient
Enter fullscreen mode Exit fullscreen mode

Importing Links

With the client ready, bulk import links from any source—websites, APIs, file hosts, anything:

import DeadLinkRadarClient from './deadlinkradar-client.js'

const client = new DeadLinkRadarClient(process.env.DEADLINKRADAR_API_KEY)

// Mix of link types - all supported
const linksToMonitor = [
  // Regular websites
  'https://docs.example.com/api/v2/authentication',
  'https://blog.company.com/2024/product-launch',
  'https://partner-site.com/integration-guide',

  // APIs and webhooks
  'https://api.stripe.com/v1/status',
  'https://hooks.slack.com/services/T00/B00/XXX',

  // File hosting (deep integration)
  'https://k2s.cc/file/abc123/archive.zip',
  'https://mega.nz/file/ABC123',
  'https://drive.google.com/file/d/1234/view',

  // Video platforms (deep integration)
  'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
  'https://vimeo.com/123456789',

  // Social media (deep integration)
  'https://bsky.app/profile/user.bsky.social/post/abc123',
  'https://github.com/owner/repo',
]

async function importLinks() {
  // Create groups to organize different link types
  const { data: docsGroup } = await client.createGroup(
    'Documentation',
    'External documentation and API references',
  )

  const { data: downloadsGroup } = await client.createGroup(
    'Downloads',
    'File hosting and download links',
  )

  console.log(`Created groups: ${docsGroup.name}, ${downloadsGroup.name}`)

  // Batch import (up to 1000 URLs per request)
  // Format: array of objects with url (required), group_id (optional), check_frequency (optional)
  const linksPayload = linksToMonitor.map((url) => ({ url }))
  const { data: result } = await client.batchCreateLinks(linksPayload)

  console.log(`Imported: ${result.success}`)
  console.log(`Failed: ${result.failed}`)

  if (result.errors?.length) {
    console.log('Errors:', result.errors)
  }

  if (result.created?.length) {
    console.log(
      `Created links:`,
      result.created.map((l) => l.url),
    )
  }
}

importLinks()
Enter fullscreen mode Exit fullscreen mode

How Detection Works

DeadLinkRadar automatically determines the best detection strategy for each URL:

URL Type Detection Method
Known platforms (YouTube, K2S, etc.) Specialized platform detection
Any other URL Smart soft-404 detection

This catches most dead links even on sites without proper 404 responses.

Setting Up Discord Alerts

Discord webhooks provide instant notifications when any link dies. Create a webhook in Discord (Server Settings → Integrations → Webhooks), then register it:

import DeadLinkRadarClient from './deadlinkradar-client.js'

const client = new DeadLinkRadarClient(process.env.DEADLINKRADAR_API_KEY)

async function setupDiscordAlerts() {
  const discordWebhookUrl = process.env.DISCORD_WEBHOOK_URL

  // Register webhook for dead link and recovery events
  const { data: webhook } = await client.createWebhook(discordWebhookUrl, [
    'link.dead',
    'link.alive',
  ])

  console.log('Discord webhook registered')
  console.log(`Webhook ID: ${webhook.id}`)
  console.log(`Signing secret: ${webhook.signing_secret}`) // Store securely

  // Send a test notification
  await client.testWebhook(webhook.id)
  console.log('Test notification sent')
}

setupDiscordAlerts()
Enter fullscreen mode Exit fullscreen mode

When a link dies, Discord receives the webhook payload:

{
  "event": "link.dead",
  "timestamp": "2024-01-15T10:30:00Z",
  "webhook_id": "550e8400-e29b-41d4-a716-446655440000",
  "data": {
    "linkId": "550e8400-e29b-41d4-a716-446655440001",
    "url": "https://docs.example.com/api/v2/authentication",
    "previousStatus": "active",
    "currentStatus": "dead",
    "checkedAt": "2024-01-15T10:30:00Z"
  }
}
Enter fullscreen mode Exit fullscreen mode

Setting Up Telegram Alerts

For Telegram notifications, create a webhook receiver that forwards to the Telegram Bot API. This can run on Vercel, Cloudflare Workers, or any serverless platform:

// api/telegram-forwarder.js
const TELEGRAM_BOT_TOKEN = process.env.TELEGRAM_BOT_TOKEN
const TELEGRAM_CHAT_ID = process.env.TELEGRAM_CHAT_ID

export default async function handler(req) {
  if (req.method !== 'POST') {
    return new Response('Method not allowed', { status: 405 })
  }

  const payload = await req.json()

  // Format the message
  const emoji = payload.event === 'link.dead' ? '🔴' : '🟢'
  const status = payload.event === 'link.dead' ? 'Dead Link' : 'Link Alive'

  const message = `
${emoji} *${status}*

*URL:* \`${payload.data.url}\`
*Previous:* ${payload.data.previousStatus}
*Checked:* ${new Date(payload.data.checkedAt).toLocaleString()}
`.trim()

  // Forward to Telegram
  await fetch(`https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      chat_id: TELEGRAM_CHAT_ID,
      text: message,
      parse_mode: 'Markdown',
    }),
  })

  return new Response('OK', { status: 200 })
}
Enter fullscreen mode Exit fullscreen mode

Register the forwarder endpoint as a webhook:

await client.createWebhook('https://your-domain.com/api/telegram-forwarder', [
  'link.dead',
  'link.alive',
])
Enter fullscreen mode Exit fullscreen mode

To get the Telegram bot token and chat ID:

  1. Create a bot with @BotFather
  2. Start a chat with the bot
  3. Get the chat ID from https://api.telegram.org/bot<TOKEN>/getUpdates

Querying Link Status

Check the current state of monitored links:

import DeadLinkRadarClient from './deadlinkradar-client.js'

const client = new DeadLinkRadarClient(process.env.DEADLINKRADAR_API_KEY)

async function getDeadLinks() {
  // Fetch all dead links with pagination
  const { data: links, pagination } = await client.getLinks({
    status: 'dead',
    per_page: 100,
  })

  console.log(`Found ${pagination.total} dead links:\n`)

  for (const link of links) {
    console.log(`URL: ${link.url}`)
    console.log(`Service: ${link.service}`) // "generic" for regular URLs
    console.log(`Last checked: ${link.last_checked_at}`)
    console.log('---')
  }
}

getDeadLinks()
Enter fullscreen mode Exit fullscreen mode

Filter by service, group, or search:

// Get all dead links from regular websites (generic detection)
const genericLinks = await client.getLinks({
  status: 'dead',
  service: 'generic',
})

// Get all dead YouTube links (specialized detection)
const youtubeLinks = await client.getLinks({
  status: 'dead',
  service: 'youtube',
})

// Get links in a specific group
const groupLinks = await client.getLinks({
  group_id: '550e8400-e29b-41d4-a716-446655440000',
})

// Search by URL substring
const searchResults = await client.getLinks({
  search: 'docs.example.com',
})
Enter fullscreen mode Exit fullscreen mode

Investigating Link History

When a link dies, check its history to understand when and why:

async function investigateLink(linkId) {
  const { data: history } = await client.getLinkHistory(linkId)

  console.log('Check history:\n')

  for (const check of history) {
    const emoji = check.status === 'active' ? '🟢' : '🔴'
    console.log(`${emoji} ${check.checked_at}`)
    console.log(`   Status: ${check.status}`)
    console.log(`   Response time: ${check.response_time_ms}ms`)

    if (check.error_message) {
      console.log(`   Error: ${check.error_message}`)
    }
    console.log()
  }
}

investigateLink('550e8400-e29b-41d4-a716-446655440000')
Enter fullscreen mode Exit fullscreen mode

Automated Daily Audit

Combine everything into a daily audit script that posts a summary to Discord:

import DeadLinkRadarClient from './deadlinkradar-client.js'

const client = new DeadLinkRadarClient(process.env.DEADLINKRADAR_API_KEY)
const DISCORD_WEBHOOK = process.env.DISCORD_SUMMARY_WEBHOOK

async function dailyAudit() {
  // Fetch usage and dead links
  const { data: usage } = await client.getUsage()
  const { data: deadLinks, pagination } = await client.getLinks({
    status: 'dead',
    per_page: 100,
  })

  // Group dead links by service
  const byService = deadLinks.reduce((acc, link) => {
    acc[link.service] = acc[link.service] || []
    acc[link.service].push(link)
    return acc
  }, {})

  const summary = {
    monitored: usage.links.used,
    dead: pagination.total,
    byService: Object.entries(byService).map(([service, links]) => ({
      service,
      count: links.length,
    })),
  }

  console.log('Daily Audit Summary')
  console.log(`Monitored: ${summary.monitored}`)
  console.log(`Dead: ${summary.dead}`)

  // Post to Discord
  if (DISCORD_WEBHOOK) {
    const embed = {
      title: '📊 Daily Link Audit',
      color: summary.dead > 0 ? 0xff0000 : 0x00ff00,
      fields: [
        { name: 'Monitored', value: summary.monitored.toString(), inline: true },
        { name: 'Dead', value: summary.dead.toString(), inline: true },
      ],
      timestamp: new Date().toISOString(),
    }

    if (summary.dead > 0) {
      embed.fields.push({
        name: 'By Type',
        value: summary.byService.map((s) => `**${s.service}**: ${s.count}`).join('\n'),
      })
    }

    await fetch(DISCORD_WEBHOOK, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ embeds: [embed] }),
    })

    console.log('Summary posted to Discord')
  }

  return summary
}

dailyAudit()
Enter fullscreen mode Exit fullscreen mode

Run this with a cron job or scheduled task:

# crontab - run daily at 9am
0 9 * * * node /path/to/daily-audit.js
Enter fullscreen mode Exit fullscreen mode

Webhook Signature Verification

For production use, verify webhook signatures to ensure requests come from DeadLinkRadar:

import crypto from 'crypto'

function verifyWebhookSignature(payload, signature, secret) {
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(JSON.stringify(payload))
    .digest('hex')

  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expectedSignature))
}

// In the webhook handler
export default async function handler(req) {
  const signatureHeader = req.headers.get('x-deadlinkradar-signature')
  // Format: "sha256=<hex>"
  const signature = signatureHeader?.replace('sha256=', '')
  const payload = await req.json()

  if (!signature || !verifyWebhookSignature(payload, signature, process.env.WEBHOOK_SECRET)) {
    return new Response('Invalid signature', { status: 401 })
  }

  // Process the webhook...
}
Enter fullscreen mode Exit fullscreen mode

Real-World Example: Docusaurus Documentation Monitor

Documentation sites often link to external APIs, libraries, and resources that can go offline without notice. Here's a complete example for monitoring all external links in a Docusaurus documentation site.

Step 1: Extract Links from Markdown Files

First, create a utility to recursively scan documentation files and extract external URLs:

// extract-docs-links.js
import fs from 'fs'
import path from 'path'

/**
 * Recursively find all markdown files in a directory
 */
function findMarkdownFiles(dir, files = []) {
  const entries = fs.readdirSync(dir, { withFileTypes: true })

  for (const entry of entries) {
    const fullPath = path.join(dir, entry.name)
    if (entry.isDirectory()) {
      findMarkdownFiles(fullPath, files)
    } else if (entry.name.match(/\.(md|mdx)$/)) {
      files.push(fullPath)
    }
  }

  return files
}

/**
 * Extract external links from markdown content
 */
function extractLinks(content, filePath) {
  const links = []

  // Match markdown links: [text](url)
  const linkRegex = /\[([^\]]*)\]\(([^)]+)\)/g
  let match

  while ((match = linkRegex.exec(content)) !== null) {
    const url = match[2]

    // Only include external https:// links
    if (url.startsWith('https://') || url.startsWith('http://')) {
      links.push({
        url,
        text: match[1],
        file: filePath,
        section: path.dirname(filePath).split(path.sep).pop() || 'root',
      })
    }
  }

  return links
}

/**
 * Extract all external links from Docusaurus docs directory
 */
export function extractLinksFromDocs(docsDir = './docs') {
  const markdownFiles = findMarkdownFiles(docsDir)
  const allLinks = []

  for (const file of markdownFiles) {
    const content = fs.readFileSync(file, 'utf-8')
    const links = extractLinks(content, file)
    allLinks.push(...links)
  }

  // Deduplicate by URL
  const uniqueUrls = new Map()
  for (const link of allLinks) {
    if (!uniqueUrls.has(link.url)) {
      uniqueUrls.set(link.url, link)
    }
  }

  return Array.from(uniqueUrls.values())
}
Enter fullscreen mode Exit fullscreen mode

Step 2: Setup Monitoring with Groups

Now import the links into DeadLinkRadar, organized by documentation section:

// setup-docs-monitoring.js
import DeadLinkRadarClient from './deadlinkradar-client.js'
import { extractLinksFromDocs } from './extract-docs-links.js'

const client = new DeadLinkRadarClient(process.env.DEADLINKRADAR_API_KEY)

async function setupDocsMonitoring() {
  console.log('Extracting links from documentation...')
  const links = extractLinksFromDocs('./docs')
  console.log(`Found ${links.length} unique external links`)

  // Get unique sections
  const sections = [...new Set(links.map((l) => l.section))]

  // Create a group for each documentation section
  const groupIds = {}
  for (const section of sections) {
    const { data: group } = await client.createGroup(
      `docs-${section}`,
      `External links from ${section} documentation`,
    )
    groupIds[section] = group.id
    console.log(`Created group: ${group.name}`)
  }

  // Prepare links with group assignments
  const linksPayload = links.map((link) => ({
    url: link.url,
    group_id: groupIds[link.section] || null,
  }))

  // Batch import (API supports up to 1000 per request)
  const { data: result } = await client.batchCreateLinks(linksPayload)

  console.log(`\nImport complete:`)
  console.log(`  Imported: ${result.success}`)
  console.log(`  Failed: ${result.failed}`)

  if (result.errors?.length) {
    console.log(`  Errors:`, result.errors.slice(0, 5))
  }

  // Set up webhook for dead link alerts
  const webhookUrl = process.env.DOCS_ALERT_WEBHOOK_URL
  if (webhookUrl) {
    const { data: webhook } = await client.createWebhook(webhookUrl, ['link.dead', 'link.alive'])
    console.log(`\nWebhook configured: ${webhook.id}`)
    console.log(`Signing secret: ${webhook.signing_secret}`)
  }

  return result
}

setupDocsMonitoring().catch(console.error)
Enter fullscreen mode Exit fullscreen mode

Running the Monitor 

# Set environment variables
export DEADLINKRADAR_API_KEY="your-api-key"
export DOCS_ALERT_WEBHOOK_URL="https://your-server.com/api/docs-webhook"

# Run the setup script
node setup-docs-monitoring.js
Enter fullscreen mode Exit fullscreen mode

Example output:

Extracting links from documentation...
Found 47 unique external links
Created group: docs-getting-started
Created group: docs-api-reference
Created group: docs-guides

Import complete:
  Imported: 47
  Failed: 0

Webhook configured: 550e8400-e29b-41d4-a716-446655440000
Signing secret: whsec_abc123...
Enter fullscreen mode Exit fullscreen mode

Once configured, DeadLinkRadar checks each link daily. When a documentation link dies, the webhook fires with the full context—allowing automated Slack alerts, GitHub issues, or custom notification flows.

The free tier includes 25 links with daily checks—enough to validate the solution before scaling up.

Summary

Traditional uptime monitors fail because they only check HTTP status codes. DeadLinkRadar provides:

  • Universal monitoring for any URL with smart soft-404 detection
  • Deep integrations for 30+ platforms (YouTube, file hosts, social media) with specialized detection
  • REST API for programmatic link management
  • Webhooks for Discord, Slack, Telegram, and custom integrations
  • Batch operations for bulk imports
  • Link groups for organization
  • Check history for debugging

Whether monitoring documentation links, API endpoints, download pages, or social media posts—the JavaScript client above covers the full API surface. Adapt it to any workflow.

Resources:

Top comments (0)