DEV Community

eatyou eatyou
eatyou eatyou

Posted on

URL Unfurling: How Slack, Discord and Twitter Generate Link Previews

#tutorial #javascript #html #webdev

You paste a URL in Slack. A second later, a rich card appears — title, description, image, favicon. Magic.

But it's not magic. It's a well-defined process called URL unfurling. Every chat app, social platform, and messaging service does it. Understanding how it works makes you a better web developer — and helps you debug why your links sometimes look broken.

Let's break it down.

What Happens When You Paste a URL

Here's the sequence, step by step:

Step 1: Detection

The app detects a URL pattern in your message. Most apps use a regex like:

https?://[^\s]+
Enter fullscreen mode Exit fullscreen mode

Some apps unfurl all URLs. Others only unfurl the first one. Slack unfurls up to 5 per message.

Step 2: Server-Side Fetch

The app's server (not your browser) makes an HTTP GET request to the URL. This is important — it means:

  • The request comes from the app's IP, not yours
  • JavaScript doesn't execute (no browser engine)
  • The server follows redirects (usually up to 5-10 hops)
  • There's a timeout (typically 3-10 seconds) 
GET https://example.com/blog/my-post HTTP/1.1
User-Agent: Slackbot 1.0 (+https://api.slack.com/robots)
Accept: text/html
Enter fullscreen mode Exit fullscreen mode

Notice the User-Agent. You can detect unfurling bots by checking for strings like Slackbot, Twitterbot, Discordbot, LinkedInBot, WhatsApp, facebookexternalhit.

Step 3: HTML Parsing

The server parses the HTML response and extracts metadata. The priority order is usually:

  1. Open Graph tags (og:title, og:description, og:image)
  2. Twitter Card tags (twitter:title, twitter:description, twitter:image)
  3. Standard HTML (<title>, <meta name="description">, <link rel="icon">)
  4. Fallbacks (first <h1>, first paragraph, first image)

Here's simplified pseudocode of what the parser does:

function extractMetadata(html) {
  const doc = parseHTML(html);

  return {
    title: 
      doc.querySelector('meta[property="og:title"]')?.content ||
      doc.querySelector('meta[name="twitter:title"]')?.content ||
      doc.querySelector('title')?.textContent ||
      doc.querySelector('h1')?.textContent,

    description:
      doc.querySelector('meta[property="og:description"]')?.content ||
      doc.querySelector('meta[name="twitter:description"]')?.content ||
      doc.querySelector('meta[name="description"]')?.content,

    image:
      doc.querySelector('meta[property="og:image"]')?.content ||
      doc.querySelector('meta[name="twitter:image"]')?.content,

    favicon:
      doc.querySelector('link[rel="icon"]')?.href ||
      doc.querySelector('link[rel="shortcut icon"]')?.href ||
      new URL('/favicon.ico', url).href,
  };
}
Enter fullscreen mode Exit fullscreen mode

Step 4: Image Validation

The app fetches the og:image URL to verify it exists, check its dimensions, and sometimes generate a thumbnail. This is why:

  • Broken image URLs show no preview image
  • Very large images take longer to unfurl
  • Some apps reject images under a minimum size

Step 5: Caching

The result is cached. Cache duration varies by platform:

Platform Cache Duration How to Clear
Slack Per-workspace Edit message, re-paste URL
Facebook ~24 hours Use Sharing Debugger
Twitter ~7 days Use Card Validator
LinkedIn ~7 days Use Post Inspector
Discord Minutes Automatic

This is why fixing your OG tags doesn't immediately update old previews.

Step 6: Rendering

Finally, the app renders a preview card with the extracted metadata. The layout depends on the platform and whether an image was found.

Why Your Link Previews Break

Now that you know the process, here are the most common failure points:

1. Client-Side Rendered Content

If your page is a React/Vue/Angular SPA that renders OG tags with JavaScript, the unfurling bot won't see them. Bots don't execute JS.

Fix: Use SSR (Next.js, Nuxt, Remix) or serve OG tags from the server.

2. Redirect Chains

URL shorteners and marketing tools add redirects. Each redirect adds latency. After 5-10 hops or 3-10 seconds, the bot gives up.

Fix: Minimize redirects. Test with curl -L -v your-url to see the chain.

3. Blocking Bots

Your firewall, CDN, or rate limiter might block the unfurling bot's IP or reject its User-Agent.

Fix: Whitelist known bot User-Agents:

# Nginx example
if ($http_user_agent ~* "Slackbot|Twitterbot|Discordbot|LinkedInBot|facebookexternalhit|WhatsApp") {
  # Allow through rate limiting
}
Enter fullscreen mode Exit fullscreen mode

4. Relative Image URLs 

<!-- Bot can't resolve this -->
<meta property="og:image" content="/images/og.png" />

<!-- This works -->
<meta property="og:image" content="https://example.com/images/og.png" />
Enter fullscreen mode Exit fullscreen mode

5. Slow Response Times

If your page takes more than 3-5 seconds to respond, the bot times out and shows no preview.

Fix: Serve a lightweight HTML response with OG tags. You don't need to server-render the full page — just the <head> section matters for unfurling.

Building Your Own Unfurler

If you're building a chat app or any app that needs link previews, here's the architecture:

[User pastes URL]
       |
       v
[URL Detection (regex)]
       |
       v
[Server-side HTTP fetch]
  - Follow redirects (max 5)
  - Timeout: 5 seconds  
  - User-Agent: "MyApp/1.0"
       |
       v
[Parse HTML for meta tags]
  - OG tags > Twitter tags > HTML tags > fallbacks
       |
       v
[Validate & fetch image]
  - Check dimensions
  - Generate thumbnail
       |
       v
[Cache result]
  - Key: URL hash
  - TTL: 1-24 hours
       |
       v
[Render preview card]
Enter fullscreen mode Exit fullscreen mode

Key Design Decisions

Timeouts: Be aggressive. 5 seconds max for the HTML fetch, 5 seconds for the image. Users are waiting.

Security: Never fetch private IPs (10.x, 192.168.x, 127.x). This prevents SSRF attacks. Also block file:// and other non-HTTP schemes.

function isSafeUrl(url) {
  const parsed = new URL(url);
  if (!['http:', 'https:'].includes(parsed.protocol)) return false;
  // Block private IPs
  const ip = await resolve(parsed.hostname);
  if (isPrivateIP(ip)) return false;
  return true;
}
Enter fullscreen mode Exit fullscreen mode

Caching: Cache aggressively. Most pages don't change their OG tags often. 1-hour TTL is a good default.

Concurrency: If 100 users paste the same URL at once, don't fetch it 100 times. Use request coalescing — one fetch, broadcast the result.

Build vs. Buy

Building a robust unfurler is more work than it looks:

  • Handling character encodings (Shift-JIS, EUC-KR, etc.)
  • Dealing with malformed HTML
  • Following redirect chains correctly
  • Image validation and thumbnailing
  • SSRF protection
  • Rate limiting outbound requests
  • Caching layer

If link previews aren't your core product, using an API is often the pragmatic choice. Options include self-hosted libraries (open-graph-scraper on npm) or hosted APIs that handle all the edge cases for you.

Testing Your Unfurling

Before sharing your URL anywhere:

  1. Check your OG tags exist — view source, search for og:
  2. Use platform debuggers — Facebook Sharing Debugger, Twitter Card Validator
  3. Test from a fresh context — incognito browser, different Slack workspace
  4. Check mobile — some platforms render previews differently on mobile
  5. Test after deploy — cached previews from staging URLs can be misleading

Recap

URL unfurling is: detect URL → server-side fetch → parse HTML meta tags → validate image → cache → render card.

The most common problems are: missing OG tags, client-side rendering, blocking bots, and slow response times.

Now you know exactly what happens when someone pastes a link — and how to make sure yours looks good.

Building something that needs link previews? I'd be curious to hear about the edge cases you've hit.

Top comments (0)