DEV Community

Cover image for Netlify Dynamic Site Challenge: Web Voyager, a website screenshot service using Netlify Blob storage.
Shikhar
Shikhar

Posted on

Netlify Dynamic Site Challenge: Web Voyager, a website screenshot service using Netlify Blob storage.

This is a submission for the Netlify Dynamic Site Challenge: Build with Blobs.

https://github.com/shikhar13012001/netlify-dyn-site-challenge

What I Built

Introducing a cutting-edge service that revolutionizes web monitoring by allowing users to capture high-quality screenshots directly from input URLs or CSV files containing multiple URLs. Each URL generates a screenshot, and CSV files have persistent URLs for easy sharing and future reference. This service leverages Netlify's blob storage, utilizing a deployment blob store with a key: blob|JSON|.. storage model, ensuring efficient, reliable, and scalable screenshot management. Ideal for developers and marketers, this tool simplifies web monitoring and enhances collaboration. Thank you, Netlify, for powering this innovative solution!

Demo

The deployment is in a digital ocean droplet http://64.227.150.39/ because of the timeouts in Netlify lambda functions.

Single URL webshot

The single web shot URL is persistent at /screenshot/image/<key_id>
Single Url web shot of netlify.com from my service

Obtained screenshot
As you can observe, the screenshot has both an instant download feature and a shareable link.

Bulk screenshot feature

Note: I have made a URL csv (urls.csv) file for convenience.
For simplicity I have taken 4 URLs in the urls.csv

urls.csv content

urls.csv is attached to the service from the local file system
I just wanted to show the cool loading screen while it works on it.

Magic UI loading screen(animated beams)

Bulk screenshot network tab view
The persistent URL of the bulk shot exists at /screenshot/bucket/<generated_bucket_key>

Bulk screenshot network tab view

Platform Primitives

Leveraging Netlify Blobs for Screenshot Service

In developing the website screenshot service, Netlify Blobs was crucial in managing and storing screenshots efficiently. Here’s a detailed breakdown of how we harnessed this platform primitive to enhance our service:

Blob Storage Integration

  • Initialization and Configuration: The integration begins with setting up the Netlify Blobs storage, a straightforward process thanks to the @netlify/blobs package. We initialize the blob store with our site's unique identifier and an API token for authentication. This setup ensures that our screenshots are securely stored and easily accessible.
   const { getStore } = require("@netlify/blobs");
   const NETLIFY_API_TOKEN = "your_netlify_api_token";
   const NETLIFY_SITE_ID = "your_site_id";
   const store = getStore({
     name: "screenshots",
     siteID: NETLIFY_SITE_ID,
     token: NETLIFY_API_TOKEN,
   });
Enter fullscreen mode Exit fullscreen mode
  • Screenshot Capture: Using Puppeteer, we capture screenshots of the specified URLs. Puppeteer is configured to run in a headless mode with several options to optimize performance and compatibility with different web pages.
   const browserOptions = {
     headless: true,
     args: [
       "--disable-setuid-sandbox",
       "--no-sandbox",
       "--single-process",
       "--no-zygote",
       "--autoplay-policy=user-gesture-required",
       "--disable-background-networking",
     ],
   };

   const browser = await puppeteer.launch({ ...browserOptions });
   const page = await browser.newPage();
   await page.setViewport({ width: 1920, height: 1080 });
   await page.goto(url, { waitUntil: "domcontentloaded" });
   const screenshot = await page.screenshot({ fullPage: true });
   await browser.close();
Enter fullscreen mode Exit fullscreen mode
  • Storing Screenshots: After capturing the screenshot, we use the Netlify Blob store to save the image. Each screenshot is stored with a unique key generated by uuidv4(), ensuring easy retrieval and avoiding filename conflicts.
   const key = uuidv4();
   await store.set(key, screenshot, {
     metadata: {
       url,
       type: "image/png",
       size: { width: 1920, height: 1080 },
     },
   });
Enter fullscreen mode Exit fullscreen mode
  • Persistent URLs: Netlify Blobs provide persistent URLs for each stored screenshot, making it easy to share and reference them. This feature is particularly useful for bulk screenshot functionality, where multiple URLs are processed, and each screenshot is stored and accessed via its unique URL.
   return res.json({ key });
Enter fullscreen mode Exit fullscreen mode
  • Bulk screenshot storage: For bulk screenshot persistence, I have used a function called saveKeys, a member function of Screenshot.api.ts' Webshot class. The code below gets all keys from the screenshots, which runs in parallel to lower the compute time to the max of all screenshot capture times( around a minute right now!!!).
 const keys = await Promise.all(
        urls.map((url) => this.takeScreenshot(url, OPTIONS))
      );
Enter fullscreen mode Exit fullscreen mode

Another route, /api/bucketList, has a conditional statement for saving or retrieving keys using the setJson function available in Netlify blob storage.

let bucketKeys;

    // Handle operations
    if (opCode === "save") {
      bucketKeys = await store.setJSON(bucketKey, { keys });
    } else if (opCode === "retrieve") {
      bucketKeys = await store.get(bucketKey, { type: "json" });
    } 
Enter fullscreen mode Exit fullscreen mode
    return NextResponse.json(
      { bucketKeys },
      {
        status: 200,
        headers: {
          "Netlify-CDN-Cache-Control":
            "public, s-maxage=31536000, stale-while-revalidate=59",
        },
      }
    );
Enter fullscreen mode Exit fullscreen mode

Network call for bucketList call
Network call for bucketList call

By integrating Netlify Blobs, we have built a robust screenshot service that caters to single and bulk URL processing needs. This service not only simplifies web monitoring for developers and marketers but also leverages the powerful features of Netlify Blobs to provide a scalable, reliable, and secure solution.

Top comments (3)

Collapse
 
shreshthgoyal profile image
Shreshth Goyal

Great work @shikhar13012001 !!

Collapse
 
akshat_srivastava_3700d4a profile image
Akshat Srivastava

Very well put man!!

Collapse
 
mattycoderx profile image
Mattycoderx

Great informations