Netlify Dynamic Site Challenge: Web Voyager, a website screenshot service using Netlify Blob storage.

Shikhar — Sun, 12 May 2024 18:18:38 +0000

This is a submission for the Netlify Dynamic Site Challenge: Build with Blobs.

https://github.com/shikhar13012001/netlify-dyn-site-challenge

What I Built

Introducing a cutting-edge service that revolutionizes web monitoring by allowing users to capture high-quality screenshots directly from input URLs or CSV files containing multiple URLs. Each URL generates a screenshot, and CSV files have persistent URLs for easy sharing and future reference. This service leverages Netlify's blob storage, utilizing a deployment blob store with a key: blob|JSON|.. storage model, ensuring efficient, reliable, and scalable screenshot management. Ideal for developers and marketers, this tool simplifies web monitoring and enhances collaboration. Thank you, Netlify, for powering this innovative solution!

Demo

The deployment is in a digital ocean droplet http://64.227.150.39/ because of the timeouts in Netlify lambda functions.

Single URL webshot

The single web shot URL is persistent at /screenshot/image/<key_id>

As you can observe, the screenshot has both an instant download feature and a shareable link.

Bulk screenshot feature

Note: I have made a URL csv (urls.csv) file for convenience.
For simplicity I have taken 4 URLs in the urls.csv

I just wanted to show the cool loading screen while it works on it.

Bulk screenshot network tab view
The persistent URL of the bulk shot exists at /screenshot/bucket/<generated_bucket_key>

Platform Primitives

Leveraging Netlify Blobs for Screenshot Service

In developing the website screenshot service, Netlify Blobs was crucial in managing and storing screenshots efficiently. Here’s a detailed breakdown of how we harnessed this platform primitive to enhance our service:

Blob Storage Integration

Initialization and Configuration: The integration begins with setting up the Netlify Blobs storage, a straightforward process thanks to the @netlify/blobs package. We initialize the blob store with our site's unique identifier and an API token for authentication. This setup ensures that our screenshots are securely stored and easily accessible.

   const { getStore } = require("@netlify/blobs");
   const NETLIFY_API_TOKEN = "your_netlify_api_token";
   const NETLIFY_SITE_ID = "your_site_id";
   const store = getStore({
     name: "screenshots",
     siteID: NETLIFY_SITE_ID,
     token: NETLIFY_API_TOKEN,
   });

Screenshot Capture: Using Puppeteer, we capture screenshots of the specified URLs. Puppeteer is configured to run in a headless mode with several options to optimize performance and compatibility with different web pages.

   const browserOptions = {
     headless: true,
     args: [
       "--disable-setuid-sandbox",
       "--no-sandbox",
       "--single-process",
       "--no-zygote",
       "--autoplay-policy=user-gesture-required",
       "--disable-background-networking",
     ],
   };

   const browser = await puppeteer.launch({ ...browserOptions });
   const page = await browser.newPage();
   await page.setViewport({ width: 1920, height: 1080 });
   await page.goto(url, { waitUntil: "domcontentloaded" });
   const screenshot = await page.screenshot({ fullPage: true });
   await browser.close();

Storing Screenshots: After capturing the screenshot, we use the Netlify Blob store to save the image. Each screenshot is stored with a unique key generated by uuidv4(), ensuring easy retrieval and avoiding filename conflicts.

   const key = uuidv4();
   await store.set(key, screenshot, {
     metadata: {
       url,
       type: "image/png",
       size: { width: 1920, height: 1080 },
     },
   });

Persistent URLs: Netlify Blobs provide persistent URLs for each stored screenshot, making it easy to share and reference them. This feature is particularly useful for bulk screenshot functionality, where multiple URLs are processed, and each screenshot is stored and accessed via its unique URL.

   return res.json({ key });

Bulk screenshot storage: For bulk screenshot persistence, I have used a function called saveKeys, a member function of Screenshot.api.ts' Webshot class. The code below gets all keys from the screenshots, which runs in parallel to lower the compute time to the max of all screenshot capture times( around a minute right now!!!).

 const keys = await Promise.all(
        urls.map((url) => this.takeScreenshot(url, OPTIONS))
      );

Another route, /api/bucketList, has a conditional statement for saving or retrieving keys using the setJson function available in Netlify blob storage.

let bucketKeys;

    // Handle operations
    if (opCode === "save") {
      bucketKeys = await store.setJSON(bucketKey, { keys });
    } else if (opCode === "retrieve") {
      bucketKeys = await store.get(bucketKey, { type: "json" });
    }

    return NextResponse.json(
      { bucketKeys },
      {
        status: 200,
        headers: {
          "Netlify-CDN-Cache-Control":
            "public, s-maxage=31536000, stale-while-revalidate=59",
        },
      }
    );

Network call for bucketList call

By integrating Netlify Blobs, we have built a robust screenshot service that caters to single and bulk URL processing needs. This service not only simplifies web monitoring for developers and marketers but also leverages the powerful features of Netlify Blobs to provide a scalable, reliable, and secure solution.