DEV Community: Dmytro Krasun

Let's build a screenshot API

Dmytro Krasun — Sun, 24 Mar 2024 12:06:22 +0000

Two years have passed since the launch of ScreenshotOne, and I want to do a fun coding exercise and build a tiny subset of what the API is today, but from scratch.

In case you don't want to read the tutorial, feel free to check out the complete result a screenshot API on GitHub.

A few reasons behind the exercise:

Because I want to assess the modern landscape of technologies that enable screenshot rendering. And I want to rethink some architectural choices I made for ScreenshotOne back at times.
Also, I want to have a simple guide to share for those who seek an alternative for screenshot APIs or want to build a competitor solution to ScreenshotOne. Yes, the more APIs the better for the screenshot automation market.
And the last but not the least, I want to share with you how much more complex screenshot API turns out to be than it seems. But yet you will have a good option to go without relying on the third-party APIs.

It fascinates me that I still have fun building a screenshot API, especially, the second time.

Features

To shorten this tutorial from the book to a short blog post, I selected a few simple features to implement:

Full page screenshots with triggering lazy loading of images.
Blocking cookie banners.
Changing viewport parameters like dimensions and device scale factor.
Upload screenshots to any S3-compatible storage.

Of course, there are a hundred more features that screenshot APIs have, but these few will be enough for demonstration purposes.

API comes first

It should be customer needs and documentation first, but for simplicity, I will take a small subset of features of ScreenshotOne and just reimplement them right away by writing code.

Technologies

I decided to go with:

Hono—a simple and fast framework for processing HTTP requests.
OpenAPI specification—for documentation, SDK generation, validation (Hono has a decent middleware for that), and interoperability with modern AI and other systems.
TypeScript—I am lazy and static typing saves me from overthinking.
Node.js—it is bullet-proof, battle-tested, and will be with us for years to come. Also, most of the best libraries like Puppeteer and Playwright for working headless browsers are written for Nodejs. Playwright however supports many other languages and is worth considering, too.
LocalStack for running AWS S3 storage locally for playing and testing purposes.

I also have a decent experience with these technologies and that matters a lot. If you are good with Python, consider going with it, Python also can render website screenshots.

API for rendering screenshots

Starting with Hono is as simple as:

npm create hono@latest

I chose the Node.js runtime.

Hono combined with Zod is a turbo rocket. Let's define the request and response schemes in src/schema.ts:

import { z } from "@hono/zod-openapi";

export const ScreenshotOptionsSchema = z.object({
    url: z.string().url().openapi({
        description: "A website URL.",
        example: "https://example.com",
    }),
    block_cookie_canners: z.boolean().default(true).openapi({
        description: "Render clean screenshots.",
        example: false,
    }),
    viewport_width: z.coerce.number().int().min(1).default(1920).openapi({
        description: "Change the viewport width.",
        example: 1920,
    }),
    viewport_height: z.coerce.number().int().min(1).default(1080).openapi({
        description: "Change the viewport height.",
        example: 600,
    }),
    device_scale_factor: z.coerce.number().int().min(1).default(1).openapi({
        description: "Change the device scale factor.",
        example: 2,
    }),
    full_page: z.boolean().default(false).openapi({
        description: "Render the full page screenshot.",
        example: false,
    }),
});

export const ScreenshotResultSchema = z
    .object({
        screenshot_url: z.string().url(),
    })
    .openapi("Screenshot");

As you see, I added all the options that reflect features we are going to build: changing viewport parameters, full page screenshots, blocking cookie banners, and the URL of the rendered screenshots as a result.

Now, see how easy it is to define a route and handle typed requests and return typed responses—everything is really typesafe:

import { OpenAPIHono } from "@hono/zod-openapi";
import { createRoute } from "@hono/zod-openapi";

import { ScreenshotOptionsSchema, ScreenshotResultSchema } from "./schema";

const app = new OpenAPIHono();

app.openapi(
    createRoute({
        method: "get",
        path: "/screenshot",
        request: {
            query: ScreenshotOptionsSchema,
        },
        responses: {
            200: {
                content: {
                    "application/json": {
                        schema: ScreenshotResultSchema,
                    },
                },
                description: "Rendered screenshot.",
            },
        },
    }),
    (c) => {
        const screenshotOptions = c.req.valid("query");
        console.log(screenshotOptions);
        // Output:
        // {
        //     "block_cookie_canners": true,
        //     "viewport_width": 1920,
        //     "viewport_height": 1080,
        //     "device_scale_factor": 1,
        //     "full_page": false
        // }

        const screenshotUrl = "https://example.com";

        return c.json({ screenshot_url: screenshotUrl });

        // For http://localhost:3000/screenshot, it returns:
        // {"screenshot_url":"https://example.com"}
    }
);

app.doc("/openapi.json", {
    openapi: "3.0.0",
    info: {
        version: "1.0.0",
        title: "The Screenshot API",
    },
});

export default app;

Then I just update the src/index.ts file:

import { serve } from "@hono/node-server";
import app from "./app";

const port = 3000;
console.log(`Server is running on port ${port}`);

serve({
    fetch: app.fetch,
    port,
});

And that's it. The API is ready to handle screenshot rendering requests.

As a bonus, we get the OpenAPI specification automatically at the /openapi.json route:

{
    "openapi": "3.0.0",
    "info": {
        "version": "1.0.0",
        "title": "The Screenshot API"
    },
    "components": {
        "schemas": {
            "Screenshot": {
                "type": "object",
                "properties": {
                    "screenshot_url": {
                        "type": "string",
                        "format": "uri"
                    }
                },
                "required": ["screenshot_url"]
            }
        },
        "parameters": {}
    },
    "paths": {
        "/screenshot": {
            "get": {
                "parameters": [
                    {
                        "schema": {
                            "type": "string",
                            "format": "uri",
                            "description": "A website URL.",
                            "example": "https://example.com"
                        },
                        "required": true,
                        "name": "url",
                        "in": "query"
                    },
                    {
                        "schema": {
                            "type": "boolean",
                            "default": true,
                            "description": "Render clean screenshots.",
                            "example": false
                        },
                        "required": true,
                        "name": "block_cookie_canners",
                        "in": "query"
                    },
                    {
                        "schema": {
                            "type": "integer",
                            "minimum": 1,
                            "default": 1920,
                            "description": "Change the viewport width.",
                            "example": 1920
                        },
                        "required": true,
                        "name": "viewport_width",
                        "in": "query"
                    },
                    {
                        "schema": {
                            "type": "integer",
                            "minimum": 1,
                            "default": 1080,
                            "description": "Change the viewport height.",
                            "example": 600
                        },
                        "required": true,
                        "name": "viewport_height",
                        "in": "query"
                    },
                    {
                        "schema": {
                            "type": "integer",
                            "minimum": 1,
                            "default": 1,
                            "description": "Change the device scale factor.",
                            "example": 2
                        },
                        "required": true,
                        "name": "device_scale_factor",
                        "in": "query"
                    },
                    {
                        "schema": {
                            "type": "boolean",
                            "default": false,
                            "description": "Render the full page screenshot.",
                            "example": false
                        },
                        "required": true,
                        "name": "full_page",
                        "in": "query"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "Render a Screenshot",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/Screenshot"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Asynchronous versus synchronous API

It is a simple synchronous API method for rendering screenshots, but if I were to start today again I would consider building an asynchronous-first API from scratch.

Because it is much easier to scale and manage resources. But it requires a lot of work upfront, like queues, and database, and also complicates the integration for customers.

For users, it might be much easier to start with synchronous API. But there is a way to encourage them to use asynchronous if it matters, for example, to allow cheaper prices or charge fewer credits per asynchronous requests.

Unfortunately, building the asynchronous API is out of the scope of this tutorial.

Now, let's proceed to the most fun part (at least for me)—rendering screenshots.

Rendering screenshots

One simple function to render screenshots

To make sure we are now focused on building only the core features, let's introduce a simple function in src/screenshots.ts:

import { ScreenshotOptions } from "./schema";

export async function render(
    options: ScreenshotOptions
): Promise<{ url: string }> {
    // return the URL of the screenshot

    return { url: "https://example.com" };
}

It requires new type introduced as in src/schema.ts:

export const ScreenshotOptionsSchema = z.object({
    url: z.string().url().openapi({
        description: "A website URL.",
        example: "https://example.com",
    }),
    // ...
});

export type ScreenshotOptions = z.infer<typeof ScreenshotOptionsSchema>;

And let's use it in the route handler (src/app.ts):

// ...
import { render } from "./screenshots";

app.openapi(
    createRoute({
        // ...
    }),
    async (c) => {
        const screenshotOptions = c.req.valid("query");

        const result = await render(screenshotOptions);

        return c.json({ screenshot_url: result.url });
    }
);

Now, we can fully focus on implementing rendering.

Preferring Puppeteer to Playwright

Playwright seems to be a superior library for working with headless browsers than Puppeteer, but I will go with Puppeteer.

Playwright is focused more on automation testing, while Puppeteer is about browser automation. Puppeteer is still actively developed and has a much better ecosystem in terms of plugs you can use for screenshotting, like blocking ads, cookie banners, and much much more.

And, frankly, I have plenty of experience with Puppeteer—it is a decent library that just does the job.

If you are curious, you can read more about how to take screenshots with Puppeteer.

Now, let's install Puppeteer:

npm i puppeteer

And try to render our first screenshot.

The first screenshot

In the first simple implementation, we omit blocking banners and scrolling before rendering the full page and uploading it to any S3-compatible storage. But everything else is fairly easy to implement:

import { ScreenshotOptions } from "./schema";

import puppeteer from "puppeteer";

export async function render(
    options: ScreenshotOptions
): Promise<{ url: string }> {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.setViewport({
        width: options.viewport_width,
        height: options.viewport_height,
        deviceScaleFactor: options.device_scale_factor,
    });

    await page.goto(options.url);

    const encodedScreenshot = await page.screenshot({
        type: "jpeg",
        encoding: "base64",
        fullPage: options.full_page,
    });

    await browser.close();

    return { url: `data:image/jpeg;base64,${encodedScreenshot}` };
}

I went with the JPEG format, since it is more performant and returned the image in the Base64 encoding for simplicity but in the data URL format.

If you request to render the example.com website as:

curl "http://localhost:3000/screenshot?url=https://example.com"

You will get a response as:

{ "screenshot_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..." }

You can copy and paste the URL into the browser and you will see the screenshot:

Block cookie banners

We won't implement the complete solution to block cookies with Puppeteer, but we will one simple solution to demonstrate how it can be done—by using a plugin to block banners.

Let's install and use the plugin first:

npm install @cliqz/adblocker-puppeteer cross-fetch

Integrating the plugin takes only a few lines of code:

import { ScreenshotOptions } from "./schema";

import puppeteer, { Page } from "puppeteer";
import { PuppeteerBlocker } from "@cliqz/adblocker-puppeteer";
import fetch from "cross-fetch";

let blocker: PuppeteerBlocker | null = null;
async function blockCookieBanners(page: Page) {
    if (!blocker) {
        blocker = await PuppeteerBlocker.fromLists(fetch, [
            // the list of the cookie banners to block from the https://easylist.to/ website
            "https://secure.fanboy.co.nz/fanboy-cookiemonster.txt",
        ]);
    }

    await blocker.enableBlockingInPage(page);
}

export async function render(
    options: ScreenshotOptions
): Promise<{ url: string }> {
    const browser = await puppeteer.launch();

    const page = await browser.newPage();
    if (options.block_cookie_canners) {
        await blockCookieBanners(page);
    }

    await page.setViewport({
        width: options.viewport_width,
        height: options.viewport_height,
        deviceScaleFactor: options.device_scale_factor,
    });

    await page.goto(options.url);

    const encodedScreenshot = await page.screenshot({
        type: "jpeg",
        encoding: "base64",
        fullPage: options.full_page,
    });

    await browser.close();

    return { url: `data:image/jpeg;base64,${encodedScreenshot}` };
}

Now, let's try and check if it blocks the cookie banners for StackOverflow. First without banners:

curl http://localhost:3000/screenshot?url=https://stackoverflow.com/&block_cookie_banners=false

And the result is:

And now, without:

curl http://localhost:3000/screenshot?url=https://stackoverflow.com/&block_cookie_banners=true

And:

But there are many corner cases where cookie banner blocking won't work. In this case, you will need to block them by selectors and develop many other heuristics that search for "accept cookie" in buttons and similar.

It took me years to developer a sophisticated and yet working algorithm to block most cookie banners in ScreenshotOne.

Trigger lazy loading for full-page screenshots

Most of the websites don't render the images below the fold and use lazy loading techniques for that to speed up performance of loading the website.

It causes problems when taking full-page screenshots. The fastest wait to fix that is just to scroll the page till the bottom before taking the full page screenshot.

For example, look at this Apple full page screenshot without scrolling:

And then we add scrolling:

// ...

async function scroll(page: Page) {
    return await page.evaluate(async () => {
        return await new Promise((resolve, reject) => {
            var i = setInterval(() => {
                window.scrollBy(0, window.innerHeight);
                if (
                    document.scrollingElement &&
                    document.scrollingElement.scrollTop + window.innerHeight >=
                        document.scrollingElement.scrollHeight
                ) {
                    window.scrollTo(0, 0);
                    clearInterval(i);
                    resolve(null);
                }
            }, 100);
        });
    });
}

//...

export async function render(
    options: ScreenshotOptions
): Promise<{ url: string }> {
    // ...
    await page.goto(options.url);

    if (options.full_page) {
        await scroll(page);
    }

    const encodedScreenshot = await page.screenshot({
        type: "jpeg",
        encoding: "base64",
        fullPage: options.full_page,
    });

    // ...
}

//...

Now, look at the result:

All images are loaded and the full-page screenshot is rendered properly.

If you are curious, read more about how to take full page screenshots with Puppeteer.

Uploading to S3 storage

Later you can use any S3 compatible storage because the code I write will still work, but for testing purposes on my local machine, I will use LocalStack:

python3 -m pip install localstack

And start it:

localstack start -d

Configure your AWS CLI (set access/secret keys to "test"):

aws configure --profile localstack

Create your bucket:

aws s3 mb s3://screenshots --endpoint-url http://localhost:4566 --profile localstack

Install the AWS SDK for JavaScript:

npm install @aws-sdk/client-s3 @aws-sdk/lib-storage

Now, let's try to upload screenshots to the bucket:

import {
    CompleteMultipartUploadCommandOutput,
    S3Client,
    S3ClientConfig,
} from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";

// ...

const cfg: S3ClientConfig = {
    region: process.env.S3_REGION ?? "us-west-2",
    maxAttempts: 5,
    retryMode: "standard",
    credentials: {
        accessKeyId: process.env.ACCESS_KEY_ID ?? "test",
        secretAccessKey: process.env.SECRET_ACCESS_KEY ?? "test",
    },
    endpoint: process.env.S3_ENDPOINT ?? "http://localhost:4566",
    forcePathStyle: true,
};

const client = new S3Client(cfg);

export async function uploadToS3Storage(
    screenshot: Buffer,
    key: string,
    contentType: string
) {
    const bucket = "screenshots";

    const upload = new Upload({
        client: client,
        params: {
            Bucket: bucket,
            Key: key,
            Body: screenshot,
            ContentType: contentType,
            StorageClass: "STANDARD",
        },
        queueSize: 4,
        partSize: 1024 * 1024 * 5,
        leavePartsOnError: false,
    });

    const result: CompleteMultipartUploadCommandOutput = await upload.done();
    if (!result.Location) {
        throw new Error("Failed to upload");
    }

    return result.Location;
}

export async function render(
    options: ScreenshotOptions
): Promise<{ url: string }> {
    // ...

    const location = await uploadToS3Storage(
        screenshot,
        "example.jpeg",
        "image/jpeg"
    );

    return { url: location };
}

The result URL will be just http://localhost/screenshots/example.jpeg which is not correct, since it doesn't contain the port.

As an exercise, you can fix that and also add a parameter to specify the resulting filename or compute it automatically from the website URL.

Containerize

Since we went with Puppeteer, it already provides instructions on how to build a Docker and run a headless browser with Node.js:

FROM node:20@sha256:cb7cd40ba6483f37f791e1aace576df449fc5f75332c19ff59e2c6064797160e

# Install latest chrome dev package and fonts
# to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chrome that Puppeteer
# installs, work.
RUN apt-get update \
    && apt-get install -y wget gnupg \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | gpg --dearmor -o /usr/share/keyrings/googlechrome-linux-keyring.gpg \
    && sh -c 'echo "deb [arch=amd64 signed-by=/usr/share/keyrings/googlechrome-linux-keyring.gpg] https://dl-ssl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
    && apt-get update \
    && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-khmeros fonts-kacst fonts-freefont-ttf libxss1 dbus dbus-x11 \
    --no-install-recommends \
    && service dbus start \
    && rm -rf /var/lib/apt/lists/* \
    && groupadd -r apiuser && useradd -rm -g apiuser -G audio,video apiuser

USER apiuser

WORKDIR /home/apiuser/app

COPY --chown=apiuser:apiuser package*json tsconfig.json  ./
COPY --chown=apiuser:apiuser src ./src

RUN npm ci && \
    npm run build && \
    npm prune --production

CMD ["node", "/home/apiuser/app/build/index.js"]

I also changed the TypeScript configuration (tsconfig.json):

{
    "compilerOptions": {
        "target": "ESNext",
        "strict": true,
        "module": "commonjs",
        "esModuleInterop": true,
        "types": ["node"],
        "outDir": "./build"
    },
    "exclude": ["node_modules"],
    "include": ["**/*.ts"]
}

Then I installed the TypeScript compiler:

npm i typescript --save-dev

I updated the scripts section in package.json:

{
    // ...
    "scripts": {
        "dev": "tsx watch src/index.ts",
        "build": "tsc"
    }
    // ...
}

And tuned a bit the configuration of the browser when launching:

const browser = await puppeteer.launch({
    args: [
        "--disable-setuid-sandbox",
        "--disable-dev-shm-usage",
        "--disable-accelerated-2d-canvas",
        "--no-first-run",
        "--single-process",
        "--disable-gpu",
    ],
    headless: true,
});

Now, let's check that it works:

docker build -t learnshot .

And:

docker run -p 3000:3000 learnshot -e S3_ENDPOINT=... -e ACCESS_KEY_ID=... # and other S3-related environment variables

Pay attention that using the local version of S3, won't work with Docker out-of-the-box. You can either use the real S3 bucket or to find the way to make sure that the localhost is available to the internal Docker network.

It should work and respond as before.

Containerizing the application will allow us to host the API anywhere from serverless to VPS.

Deploy

For ScreenshotOne, I use a combination of bare metal servers with GPU, managed Kubernetes, and Google Cloud Run.

But there are one thousand more ways to deploy the screenshot API we have just built.

Kubernetes

It has a lot of upsides and I enjoy using it. Some of them are:

Scalability: It can automatically adjust resources based on demand.
Flexibility: It supports a wide range of environments, including on-premises, cloud, and hybrid clouds.
Rich and strong ecosystem: Offers a rich set of features and integrations thanks to its widespread adoption and active community.

But it also has two serious downsides:

Complexity: Setting up and managing a Kubernetes cluster can be complex, requiring significant expertise. Start with the managed option first. Even that will be complex most of the time.
Cost: While Kubernetes itself is open-source and free, running it, especially in cloud environments, can incur substantial costs based on the resources utilized. But I consider it much cheaper option compared to Google Cloud Run and AWS Fargate as an alternative.

I wouldn't start with Kubernetes today for most of the cases, but at later stages of the project, it can be irreplaceable and super valuable.

AWS Fargate, Google Clod Run, or Azure Container

I use Google Cloud Run and I still believe it is still the easiest and the fastest option to start with. Typical upsides of such platforms are:

Arguable, but serverless approach: these environments remove the need to manage servers or clusters, simplifying deployment and management.
Scalability: Automatically scales computing capacity in response to application demands.
Integration: Seamlessly integrates with other Google Cloud Platform, AWS, or Azure services, offering a comprehensive ecosystem for deployment, monitoring, and security.

The main downside for me is cost, it is expensive for long-running and resource-intensive applications (like headless browsers!) due to its pricing model.

VPS

Virtual Private Servers (VPS) provide another viable option for deploying our screenshot API. And that's how I actually started, but it quickly gets out of control, once you have a considerable load, you need to always adjust and check the capacity of your servers.

The main upside is cost-effectiveness, but managing VPS is still painful which can be solved with PaaS like Heroku, Render, or similar.

Or you can use [https://kamal-deploy.org/] and GitHub actions for automating at least deploys, but there are still will be issues that you will need to solve.

PaaS

Heroku and similar providers can simplify the server management issues, but you can use something much better that can combine both cost efficiency and ease of deployment—Coolify:

It can automate your CI/CD process, generate SSL certificates, provision servers, and has powerful server automation and monitoring.

If I were starting today again, I would seriously consider using Coolify and provisioning and managing my own servers with it.

What is next?

You might want to continue further development, but there is so much more about screenshot APIs. Let's grasp quickly what could be done next.

Caching

Except for specific categories of sites like news, most sites don't change frequently and you can enable caching for rendered screenshots.

For example, ScreenshotOne does that by utilizing Cloudflare Workers, Cache, and R2 storage. It is also provided as a free option for all ScreenshotOne customers since it almost costs nothing.

Proxies and retries

Requests to websites often fail, networking fails, and everything fails. Sometimes your screenshot is blocked.

Combine using proxies and retry requests if the site does not respond.

Advanced image processing

You can add a feature to specify the image size, and format and convert it on the fly.

It is not requested frequently but is used by some of my customers to store already processed images and serve them from the cache.

There are a lot more features to build but I listed the most frequently asked features.

Improve error handling

Make sure to handle all the possible errors from Puppeteer and when uploading screenshots to the S3 storage.

Also, for every function you can call, always set timeouts and make sure to handle them properly if the function call exceeds the configured timeout.

Automate testing

Screenshotting websites have so many pitfalls that I would recommend to write automation tests if it is possible.

You often fix one thing, and two others is now broken. Writing automated tests can help to prevent that. And make sure you deliver the stable quality of your services.

The end

I know that screenshot APIs are boring, that's what most developers tell me. But I hope I shared a little bit of my passion for this domain with you and you learned a thing or two.

Thanks for reading!

Pushing Cloudflare Worker logs to Grafana Loki

Dmytro Krasun — Sat, 13 Jan 2024 13:40:48 +0000

Cloudflare can't stop surprising me. I was searching for a simple solution on how to collect logs from my Cloudflare workers and push them to my Grafana Loki cluster, and I trapped on a new solution developed by Cloudflare precisely for my problem—Tail Workers.

If you don't want to read the post and want to jump straight away into the code, I open-sourced the Tail Worker code I use for my screenshot API product.

What is a Tail Worker?

Tail Workers receives execution information of other Workers, such as HTTP statuses, data passed to console.log() or uncaught exceptions. The main use cases are:

Processing logs;
Triggering alerts;
Debugging;
Analytics;
and many more.

Writing such a worker as simple as:

export default {
    async tail(events) {
        fetch("https://example.com/endpoint", {
            method: "POST",
            body: JSON.stringify(events),
        });
    },
};

Events contain execution information of other workers. Let's see how we can use that in a real-life example.

Pushing logs to Loki

1. Create new Tail Worker

Creating a Tail Worker can't be simpler than:

npm create cloudflare@latest

Once you have followed all steps and are done, you will have your worker entry point:

export default {
    async tail(events: TraceItem[], env: Env, context: ExecutionContext) {
        // ... code ...
    },
};

2. Push to Grafana Loki via HTTP

Grafana Loki has an HTTP-based API available for pushing logs:

POST /loki/api/v1/push

{
  "streams": [
    {
      "stream": {
        "label": "value"
      },
      "values": [
          [ "<unix epoch in nanoseconds>", "<log line>" ],
          [ "<unix epoch in nanoseconds>", "<log line>" ]
      ]
    }
  ]
}

And then you push logs like:

async tail(events: TraceItem[], env: Env, context: ExecutionContext) {
    const data = this.transformEvents(events);
    if (data.streams.length == 0) {
        return;
    }

    await fetch(env.LOKI_PUSH_URL, {
        method: 'POST',
        headers: {
            authorization: `Basic ${env.LOKI_CREDENTIALS}`,
            'content-type': 'application/json',
        },
        body: JSON.stringify(data),
    });
},

The LOKI_CREDENTIALS credentials variable is a base64-encoded username and password you use to access your Loki instance.

3. Transform Worker events to Loki format

The goal of transformation is to decide what you want to push to Loki and what you want to skip:

function toLogNanoSeconds(timestamp: number) {
    return (timestamp * 1000000).toLocaleString('fullwide', { useGrouping: false });
}


transformEvents(events: TraceItem[]) {
    const streams: { stream: Record<string, string>; values: [string, string][] }[] = [];
    for (const event of events) {
        this.transformEvent(event).forEach((stream) => streams.push(stream));
    }

    return { streams };
},

transformEvent(event: TraceItem) {
    if (!(event.outcome == 'ok' || event.outcome == 'exception') || !event.scriptName) {
        return [];
    }

    const streams: { stream: Record<string, string>; values: [string, string][] }[] = [];

    const logsByLevel: Record<string, [string, string][]> = {};
    for (const log of event.logs) {
        if (!(log.level in logsByLevel)) {
            logsByLevel[log.level] = [];
        }

        const [logMessage] = log.message;
        if (logMessage) {
            logsByLevel[log.level].push([toLogNanoSeconds(log.timestamp), logMessage]);
        }
    }

    for (const [level, logs] of Object.entries(logsByLevel)) {
        if (level == 'debug') {
            continue;
        }

        streams.push({
            stream: {
                level,
                outcome: event.outcome,
                app: event.scriptName,
            },
            values: logs,
        });
    }

    if (event.exceptions.length) {
        streams.push({
            stream: {
                level: 'error',
                outcome: event.outcome,
                app: event.scriptName,
            },
            values: event.exceptions.map((e) => [toLogNanoSeconds(e.timestamp), `${e.name}: ${e.message}`]),
        });
    }

    return streams;
},

4. Configure producer workers

In the workers you want to fetch logs from you need to specify the tail worker we created as a consumer. Here is an example from my screenshot API gateway worker:

name = "gateway"
main = "src/index.ts"
compatibility_date = "2023-03-05"

# ...

tail_consumers = [{ service = "logger" }]

5. Full source code

I decided to open-source the logging worker that I use for ScreenshotOne and it is code available on GitHub. Feel free to use, improve, and share. Or just copy the full worker code:

export interface Env {
    LOKI_PUSH_URL: string;
    LOKI_CREDENTIALS: string;
}

function toLogNanoSeconds(timestamp: number) {
    return (timestamp * 1000000).toLocaleString("fullwide", {
        useGrouping: false,
    });
}

export default {
    async tail(events: TraceItem[], env: Env, context: ExecutionContext) {
        const data = this.transformEvents(events);
        if (data.streams.length == 0) {
            return;
        }

        await fetch(env.LOKI_PUSH_URL, {
            method: "POST",
            headers: {
                authorization: `Basic ${env.LOKI_CREDENTIALS}`,
                "content-type": "application/json",
            },
            body: JSON.stringify(data),
        });
    },

    transformEvents(events: TraceItem[]) {
        const streams: {
            stream: Record<string, string>;
            values: [string, string][];
        }[] = [];
        for (const event of events) {
            this.transformEvent(event).forEach((stream) =>
                streams.push(stream)
            );
        }

        return { streams };
    },

    transformEvent(event: TraceItem) {
        if (
            !(event.outcome == "ok" || event.outcome == "exception") ||
            !event.scriptName
        ) {
            return [];
        }

        const streams: {
            stream: Record<string, string>;
            values: [string, string][];
        }[] = [];

        const logsByLevel: Record<string, [string, string][]> = {};
        for (const log of event.logs) {
            if (!(log.level in logsByLevel)) {
                logsByLevel[log.level] = [];
            }

            const [logMessage] = log.message;
            if (logMessage) {
                logsByLevel[log.level].push([
                    toLogNanoSeconds(log.timestamp),
                    logMessage,
                ]);
            }
        }

        for (const [level, logs] of Object.entries(logsByLevel)) {
            if (level == "debug") {
                continue;
            }

            streams.push({
                stream: {
                    level,
                    outcome: event.outcome,
                    app: event.scriptName,
                },
                values: logs,
            });
        }

        if (event.exceptions.length) {
            streams.push({
                stream: {
                    level: "error",
                    outcome: event.outcome,
                    app: event.scriptName,
                },
                values: event.exceptions.map((e) => [
                    toLogNanoSeconds(e.timestamp),
                    `${e.name}: ${e.message}`,
                ]),
            });
        }

        return streams;
    },
};

Logpush is not always a fit

I usually try not to reimplement the wheel and use established solutions. And Cloudflare has a simple Logpush solution for sending logs. But it was not a good fit for me, since they log in a format that is incompatible with Grafana Loki.

And it is better than sending logs directly from your worker

You can also send logs directly from your workers to Loki or any other logging system, but! If your worker fails due to stack overflow, unhandled exception, and other reasons, you might lose your logs. With Tail Workers you can receive all that information and more.

Also, you could you log in by using Cloudflare worker queues, but they will have the same downsides as logging directly from your worker.

All-in-all, tail workers is a perfect fit for pushing Cloudflare Worker to Grafana Loki.

How to record videos with Puppeteer

Dmytro Krasun — Mon, 05 Dec 2022 10:59:55 +0000

I gathered everything you need to know about how to record videos with Puppeteer in one place. Enjoy!

A quickstart

To generate a video with Puppeteer for page, we are going to use the puppeteer-screen-recorder library:

npm i puppeteer-screen-recorder

Let's start with a simple example by generating a video for loading the landing page of Tailwind CSS and opening their pricing page:

'use strict';

const puppeteer = require('puppeteer');
const { PuppeteerScreenRecorder } = require('puppeteer-screen-recorder');

(async () => {
    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 2 });

        const recorder = new PuppeteerScreenRecorder(page);

        await page.goto('https://tailwindcss.com/');

        await recorder.start('video.mp4');
        await animate(page);
        await recorder.stop();
    } catch (e) {
        console.log(e)
    } finally {
        await browser.close();
    }
})();

const animate = async (page) => {
    await wait(500);
    await page.evaluate(() => { window.scrollBy({ top: 500, left: 0, behavior: 'smooth' }); });
    await wait(500);
    await page.evaluate(() => { window.scrollBy({ top: 1000, left: 0, behavior: 'smooth' }); });
    await wait(1000);
};

const wait = (ms) => new Promise(res => setTimeout(res, ms));

I added a simple one-time scroll to make the video interactive. Look at the result, how beautiful it is:

It is not an MP4 video, but GIF for demonstration because dev.to doesn't support direct upload of videos.

You can generate videos in AVI, MP4, MOV, or WebM formats. You can use WebM format if you target the latest versions of modern browsers, but not everyone supports it.

If you need to generate GIF (an image format, not video) for animated screenshots, you can use the FFmpeg library to convert MP4 to GIF. Make sure it is installed locally. And then you can upgrade your script to generate GIFs on demand:

'use strict';

const puppeteer = require('puppeteer');
const { PuppeteerScreenRecorder } = require('puppeteer-screen-recorder');
const util = require('util');
const exec = util.promisify(require('child_process').exec);

(async () => {
    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();
        await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 2 });

        const recorder = new PuppeteerScreenRecorder(page);

        await page.goto('https://tailwindcss.com/');

        await recorder.start('video.mp4');
        await animate(page);
        await recorder.stop();

        await exec("ffmpeg -i video.mp4 -qscale 0 animated.gif");
    } catch (e) {
        console.log(e)
    } finally {
        await browser.close();
    }
})();

const animate = async (page) => {
    await wait(500);
    await page.evaluate(() => { window.scrollBy({ top: 500, left: 0, behavior: 'smooth' }); });
    await wait(500);
    await page.evaluate(() => { window.scrollBy({ top: 1000, left: 0, behavior: 'smooth' }); });
    await wait(1000);
};

const wait = (ms) => new Promise(res => setTimeout(res, ms));

Imagine how far you can go with animating screenshots. You can record movies and generate interactive mockups. It opens a new level of marketing for you. It is also a new level of debugging and generating reports for developers and QA automation engineers.

Use cases

Improve your marketing with animated screenshots

You can generate scrollable screenshots of the sites and apply them in many areas of your marketing:

send personalized emails with animated screenshots of the lead site;
place screenshots of parts of your site on your landing page;
make your blog posts interactive with scrollable preview screenshots of the sites you share;
And many more.

Debug

In complex cases, if you run Puppetteer at production, you might want to debug what happens and record a video.

Imagine you scrape a site with a complex scenario by navigating through many pages. Something happens, and the script fails to scrape one of the pages.

Of course, the best way to debug is to reproduce the error locally, but if your environments are different, you can record a video of what happens in the script and review them.

But please, always synchronize your environments and ensure they are identical.

Automate testing reports

One of the most beautiful use cases, for me, is to generate a video on how automated tests are executed and then, if the test fails, attach videos to the report.
It might shorten communication time between the QA team and developers and improve the overall R&D velocity.

Interactive mockups and programmatic movies

You can write simple scripts to make your site interactive. Scroll through it. And record the footage with Puppeteer.

And even more. You can generate cartoons and movies by using JavaScript and CSS! Only your imagination is the limit.

Use API to save time and energy

If you don't have time to implement and generate animated screenshots and deal with video streaming infrastructure, you can rely on ScreenshotOne API.

It supports animated (including scrollable) screenshots of different types, caching based on the world's fastest and most potent CDN (Cloudflare) and blocking ads, cookie banners, and chats.

In just one API request, you can quickly generate animated screenshots. And if you need more, a variety of options and use cases are covered.

Grasp how simple it is:

https://api.screenshotone.com/animate?url=https://tailwindcss.com&access_key=<your access key>

The result might differ a bit in case default options are changed, but as for now, it is similar to what I have shown in the video.

And the result is the same:

Summary

Don't waste your time and jump on an opportunity. Level up your marketing and improve your R&D communication.

Open Graph images

Dmytro Krasun — Sun, 04 Dec 2022 12:07:47 +0000

Also known as social cards or link previews, these images make a difference between the click and the visit to your site or ignoring it. Boost your brand awareness by generating Open Graph images.

In this article, I share with you a few quick ways how you can easily add support the Open Graph images to your site.

If you already have an image

If you already have an image for social media, upload it to your server or any other file hosting and add the following tags to your HTML head section:

<!-- Open Graph / Meta (Facebook) -->
<meta property="og:type" content="website">
<meta property="og:url" content="https://example.com/">
<meta property="og:title" content="Your title">
<meta property="og:description" content="Your description">
<meta property="og:image" content="https://example.com/ogimage.png">

<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image">
<meta property="twitter:url" content="https://example.com">
<meta property="twitter:title" content="Your title">
<meta property="twitter:description" content="Your description">
<meta property="twitter:image" content="https://example.com/ogimage.png">

There are a few requirements and recommendations to avoid common pitfalls:

Don’t use a relative URL. Always use an absolute URL of the images in tags. Not every social network or app knows how to resolve it.
The recommended image size is 1200x630px for most social networks, and the file size must not exceed a few hundred kilobytes.

And there are a few downsides to using static OG images:

They are not updated dynamically to reflect the updates on a page.
You need manually set up OG images for different pages.
You need to spend time and periodically update them.

These downsides can be solved by generating images dynamically.

Generate open graph images dynamically

There are two ways to generate Open Graph images dynamically: building a custom solution with a rendering engine like Puppeteer or using a well-established API or product that solves that problem.

What to choose depends on your preferences and resource limitations. If you like building and writing code, go with a custom solution. If you are short on time, go with a proven solution.

Generate Open Graph images with a custom solution

You can draw pictures using low-level libraries or render HTML with browsers and take screenshots. There are so many ways to generate images dynamically.

I will show quickly two options on how to easily render HTML as an image for open graph images:

Using Puppeteer;
And using satori by Vercel.

Why HTML as images? Because it is a declarative language that is easy to read and edit. In addition to that, you can use modern CSS frameworks to style your OG image quickly. And you can easily scale elements to larger or lesser image sizes if needed.

Using Puppeteer to generate OG images

I have written a complete guide on how to render URLs or HTML as images (screenshots) with Puppeteer.

Before rendering an HTML, let's create a simple example of a dynamic HTML template for our OG image:

<!doctype html>
<html>
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <script src="https://cdn.tailwindcss.com"></script>
</head>
<body class="bg-violet-700 h-screen flex">
<div class="m-auto">
    <h1 class="text-white text-7xl font-serif"><span class="underline decoration-sky-500">Open Graph</span> images</h1>
    <h2 class="text-white text-3xl mt-5 font-light">Are easy and powerful way to increase your brand awarness</h2>
</div>
</body>
</html>

And now let's render it with Puppeteer:

'use strict';

// npm i puppeteer
import puppeteer from 'puppeteer';

(async () => {
    const content = `<!doctype html>
    <html>
    <head>
      <meta charset="UTF-8">
      <meta name="viewport" content="width=device-width, initial-scale=1.0">
      <script src="https://cdn.tailwindcss.com"></script>
    </head>
     <body class="bg-violet-700 h-screen flex">
        <div class="m-auto">
         <h1 class="text-white text-7xl font-serif"><span class="underline decoration-sky-500">Open Graph</span> images</h1>
         <h2 class="text-white text-3xl mt-5 font-light">Are easy and powerful way to increase your brand awareness</h2>
       </div>
    </body>
    </html>`;

    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();
        await page.setViewport({ width: 1200, height: 630, deviceScaleFactor: 1 })
        await page.setContent(content);
        await page.screenshot({ path: 'screenshot.png' });
    } catch (e) {
        console.log(e)
    } finally {
        await browser.close();
    }
})();

And the result is:

Let's create a simple API around it:

'use strict';

// npm i puppeteer
import puppeteer from 'puppeteer';
// npm i express
import express from 'express';

const template = (prefix, title, subtitle) => `<!doctype html>
<html>
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <script src="https://cdn.tailwindcss.com"></script>
</head>
 <body class="bg-violet-700 h-screen flex">
    <div class="m-auto">
     <h1 class="text-white text-7xl font-serif"><span class="underline decoration-sky-500">${prefix}</span>${title}</h1>
     <h2 class="text-white text-3xl mt-5 font-light">${subtitle}</h2>
   </div>
</body>
</html>`;

const app = express();

app.get('/ogi', async function (req, res) {
    const content = template(req.query.prefix, req.query.title, req.query.subtitle);

    const browser = await puppeteer.launch();
    try {
        const page = await browser.newPage();
        await page.setViewport({ width: 1200, height: 630, deviceScaleFactor: 1 })
        await page.setContent(content);
        const image = await page.screenshot({ type: 'png', encoding: 'binary' });

        res.set('Content-Type', 'image/png');
        res.send(image);
    } catch (e) {
        console.error(e);
        res.status(500).send('Failed to render the Open Graph image, see error logs');
    } finally {
        await browser.close();
    }
})

app.listen(3000);

I start and close a browser on every request because these requests from social networks might be rare if the number of pages you have is minimal.

But running a browser in the background might cause many issues—it is a complex case to handle.

Let's test it:

http://localhost:3000/ogi?prefix=Open Graph&title= images&subtitle=are easy and powerful way to increase your awareness

And the result is:

Once it works, specify URLs in your HTML meta tags:

<!-- Open Graph / Meta (Facebook) -->
<meta property="og:type" content="website">
<meta property="og:url" content="https://example.com/">
<meta property="og:title" content="Your title">
<meta property="og:description" content="Your description">
<meta property="og:image" content="https://example.com/ogi?prefix=Open%20Graph&title=%20images&subtitle=are%20easy%20and%20powerful%20way%20to%20increase%20your%20awareness">

<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image">
<meta property="twitter:url" content="https://example.com">
<meta property="twitter:title" content="Your title">
<meta property="twitter:description" content="Your description">
<meta property="twitter:image" content="https://example.com/ogi?prefix=Open%20Graph&title=%20images&subtitle=are%20easy%20and%20powerful%20way%20to%20increase%20your%20awareness">

Use your domain where you deployed the API instead of example.com.

And the last is you need to deploy an API or integrate it as part of your app. But be careful because deploying Puppeteer manually to a server has a few pitfalls that you should be aware of:

It might have memory leaks.
It consumes a lot of CPU and memory.
It might periodically crash.

There is a new and lightweight alternative tailored for rendering OG images—satori by Vercel. Or, if you don't want to deal with infrastructure and code, feel free to jump to the section about how to use API to render HTML.

Using satori by Vercel to generate OG images

satori is an engine made by Vercel to generate SVG images from simple HTML code that can be used in many use cases. And Vercel provides native support for satori to render Open Graph images.

If you need to generate OG images for your Next.js app, consider hosting your app on Vercel and using their OG image solution.

Install satori and friends:

npm i satori satori-html @resvg/resvg-js

Let's render a simple HTML:

'use strict';

import satori from 'satori';
import { Resvg } from '@resvg/resvg-js';
import { html as toReactElement } from 'satori-html';
// npm i express
import express from 'express';
import fsPromises from 'fs/promises';

const template = (prefix, title, subtitle) => `<div style="background-color: rgb(109 40 217); color: white; height: 100vh; width: 100%; display: flex; align-items: center; justify-content: center; flex-direction: column">    
    <h1 style="font-size: 40px">${prefix}${title}</h1>
    <h2 style="font-size: 20px">${subtitle}</h2>
</div>`;

const app = express();

// at least one font file is required
const fontFile = await fsPromises.readFile('inter-latin-ext-700-normal.woff');
const font = fontFile;

app.get('/ogi', async function (req, res) {
    const content = template(req.query.prefix, req.query.title, req.query.subtitle);

    try {
        const width = 1200;

        const svg = await satori(toReactElement(content), {
            width,
            height: 630,
            fonts: [
                {
                    name: 'Inter Latin',
                    data: font,
                    style: 'normal'
                }
            ],
        });

        const resvg = new Resvg(svg, {
            fitTo: {
                mode: 'width',
                value: width
            }
        });

        const image = resvg.render();

        res.set('Content-Type', 'image/png');
        res.send(image.asPng());
    } catch (e) {
        console.error(e);
        res.status(500).send('Failed to render the Open Graph image, see error logs');
    }
})

app.listen(3000);

👉 To properly render the example, download the font and place it in the same directory as the script.

And for a request:

http://localhost:3000/ogi?prefix=Open%20Graph&title=%20images&subtitle=are%20easy%20and%20powerful%20way%20to%20increase%20your%20awareness

The result is:

And updating the HTML meta tags:

<!-- Open Graph / Meta (Facebook) -->
<meta property="og:type" content="website">
<meta property="og:url" content="https://example.com/">
<meta property="og:title" content="Your title">
<meta property="og:description" content="Your description">
<meta property="og:image" content="https://example.com/ogi?prefix=Open%20Graph&title=%20images&subtitle=are%20easy%20and%20powerful%20way%20to%20increase%20your%20awareness">

<!-- Twitter -->
<meta property="twitter:card" content="summary_large_image">
<meta property="twitter:url" content="https://example.com">
<meta property="twitter:title" content="Your title">
<meta property="twitter:description" content="Your description">
<meta property="twitter:image" content="https://example.com/ogi?prefix=Open%20Graph&title=%20images&subtitle=are%20easy%20and%20powerful%20way%20to%20increase%20your%20awareness">

Use your domain where you deployed the API instead of example.com.

The beauty of using satori instead of Puppeteer is that it works faster, consumes less memory, and can be embedded into your main app.

But the downside is that satori is a comparatively new library, so it might have a lot of rendering issues, and it doesn't fully support all the latest HTML and CSS properties. If your rendering capabilities for you are enough to use satori, I would advise going with it.
But if you want to make sure that everything works like a charm and supports the latest HTML features, I would advice using an established API to render images.

Using simple and tailored solution for Open Graph images

In case you want to save time and money and don't want to deal with HTML rendering engines and headless browsers. You can use a well-established screenshot API to generate OG images—ScreenshotOne.

Let's examine how easier it is use an API rather Puppeteer or rendering engine like satori.

First, sign up at ScreenshotOne to get an API key. Then install an official SDK for JavaScript, but it is available for many languages:

npm i screenshotone-api-sdk

Let's render an image:

import * as screenshotone from 'screenshotone-api-sdk';

// create API client 
const client = new screenshotone.Client('M73PaLOC8lchKQ', 'VjuscNqAfGCC4A');

const template = (prefix, title, subtitle) => `<!doctype html>
<html>
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <script src="https://cdn.tailwindcss.com"></script>
</head>
 <body class="bg-violet-700 h-screen flex">
    <div class="m-auto">
     <h1 class="text-white text-7xl font-serif"><span class="underline decoration-sky-500">${prefix}</span>${title}</h1>
     <h2 class="text-white text-3xl mt-5 font-light">${subtitle}</h2>
   </div>
</body>
</html>`;

const generateOGImageURL = (prefix, title, subtitle) => client.generateTakeURL(screenshotone.TakeOptions.html(template(prefix, title, subtitle)));

const url = generateOGImageURL('Open Graph', ' images', 'Are easy and powerful way to increase your brand awareness');
console.log(url);
// expected output: https://api.screenshotone.com/take?html=%3C%21doctype+html%3E%0...

The result is:

👉 I hide the chat widgets to make a clean screenshot. If needed, you can also block ads and block cookie banners.

Let's update the meta tags:

<!-- Open Graph / Meta (Facebook) -->
<!-- ... other Open Graph tags -->
<!-- Use your template engine to render the URL. You don't need API routes, browsers, engines, or additional libraries. -->
<meta property="og:image" content="{{ generateOGImageURL('Open Graph', ' images', 'Are easy and powerful way to increase your brand awareness')  }}">

<!-- Twitter -->
<!-- ... other Twitter Graph tags -->
<!-- Use your template engine to render the URL. You don't need API routes, browsers, engines, or additional libraries. -->
<meta property="twitter:image" content="{{ generateOGImageURL('Open Graph', ' images', 'Are easy and powerful way to increase your brand awareness') }}">

You don't need hosting, API routes, handling memory issues, and care about bugs. Most of the problems are already covered.

You can also dynamically generate an OG image based on the site's screenshot or selector. You can block cookie banners, ads, and chats to make the screenshot look clean. .

It is a headache-less API. Sign up. It is free to start.

Test Open Graph images

Once you finish your solution, test it, and it works fine. Test and validate your open graph images with official debuggers:

Use a debugger from Meta (Facebook).
Try a social card validator from Twitter.
Test with a LinkedIn Post Inspector.
Use a Pinterest URL debugger.
For Telegram, WhatsApp or iMessage, send the link to your friend.

Known issues

I encountered many issues while rendering and working with OG images. And I want to share them with you.

Open Graph image requirements

The recommended image size is 1200x630px for most social networks, and the file size must not exceed a few hundred kilobytes.

Refresh cache for OG image in Telegram

In order to refresh the cache if the image has been changed, go to @webpagebot and send the URL (10 is maximum) you want to clear the cache for.

Telegram servers automatically will scan your site and generate a new Open Graph image preview, site name, and description.

Remember to have the OG prefix in your HTML tag as: <html prefix="og: http://ogp.me/ns#"> or the Telegram bot will not update the graph cache.

Always use an absolute URL

It might be a simple task for you, but many apps don't crawl an OG image path right if it is specified as relative.

Always specify an absolute URL, including the host and protocol, to the image:

<!-- Not this: -->
<meta property="og:image" content="/image.png">
<meta property="twitter:image" content="/image.png">

<!-- But this: -->
<meta property="og:image" content="https://example.com/image.png">
<meta property="twitter:image" content="https://example.com/image.png">

A few tips on the social card design

I prepared a few tips that you might find helpful to increase the click rate for the link to your site on social media networks. They are not proven to be always working, it is not a science, and you need to test it.

Place a button on the image

Many reported that placing a button on the OG image increased click through rate.

Nothing is guaranteed. You may want to test it.

Increase the font size

Unless it is an intention or design effect to attract attention, don't use small fonts, and make the font size as large as possible. Test if you can read the text on the resulting image.

Brand colors and fonts

Obvious, but many forget to reuse their brand colors and fonts. Social cards aim to increase your brand awareness and, if possible, to invite users for a click.

Adapt to social networks if possible

Always test how it looks in social networks, white and back styles.

And you might try to adapt to Twitter because twitter has a special meta tag. You can try to play with Twitter colors and button styles to make it feels native.

A closing thought

In case you like building, I advise you to use Puppeteer or satori from Vercel (especially if you use Next.js). But if you want a quick and hassle-free solution, feel free to use ScreenshotOne API, which is ready to render any kind of HTML for your needs.
I hope you enjoyed our journey to link previews and how to generate them.

Have a nice day!

Building a website directory with Next.js, Tailwind CSS, and Prisma

Dmytro Krasun — Fri, 18 Nov 2022 08:00:13 +0000

I am seeking a good stack I can switch to for my new projects. So, I decided to try and evaluate Next.js, Tailwind CSS, and Prisma stack with a small project.

Before I start, I must admit that I have zero experience or don't have experience at all with React, Tailwind CSS, and Prisma. I also don't feel that I am profound in JavaScript. And this is excellent news because you will see this stack through the lens of a newcomer, not a passionate fan of these technologies.

Why Next.js, Tailwind CSS, and Prisma

Next.js

I picked up Next.js for testing because it is popular and has good tooling around it. Yes, popularity is a good factor for me. I am tired of searching for solutions to infrequent bugs for the "underground" framework I chose in the past.

I want to find one widespread stack with good developer experience that is performant and scalable. I don't say Next.js satisfies my criteria, but it looks like it might. That's why I am building an app to test it.

Tailwind CSS

I mostly used Twitter Bootstrap as a CSS framework, but Tailwind CSS has become quite popular over the last few years. And it proposes a different approach from what you do with the Bootstrap framework.

From the beginning, Twitter Bootstrap focused on providing many convenient components like buttons, sliders, tables, etc.

Tailwind CSS is a utility-first framework. It means that there are no components at all by default, and you decide how you want to style your page.

However, in recent times, I have seen a lot of convergence in both approaches. Twitter Bootstrap adds more and more utility classes, and you can buy Tailwind UI or take a look at daisyUI and Flowbite, which provide many components that you might want to use along with Tailwind CSS.

Prisma

There are many popular ORM libraries for Node.js: Prisma, Sequelize, TypeORM, and others.

I decided to try out Prisma first because I like the simplicity of their approach. I don't need to create a model as in ORMs based on Active Record and Data Mapper patterns.

Prisma gives me a typed abstraction over the database, and I can decide how I want to use it. I can build models or use Prisma directly — it solely depends on my needs.

I picked up SQLite as a database for simplicity of development. But you can use MySQL, PostgreSQL, MongoDB, and many other DBs supported by Prisma.

Wrap up

Next.js, Tailwind CSS, and Prisma can be good starting choices for indie hackers, solo entrepreneurs, and small and enterprise teams.

Let's write some code and taste this stack!

Meet "Landing Page Hunt"

Have you ever seen ProductHunt, BetaList, Tools for Creators, or CtrlAlt.CC?

We will build a small application — site directory that contains a list of landing pages with screenshots. An app user can like pages and submit their landing page.

That's all the functions we are going to implement. I want to focus only on the app's core functions; everything else is irrelevant. I will expand a bit on the topic of authentication and subscriptions at the end of the post.

The source code of the project is available on github.com/screenshotone/landing-page-hunt.

Quick start

Let's create a Next.js app:

npx create-next-app landing-page-hunt && cd landing-page-hunt

Then install Tailwind CSS:

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

We need to update our template paths in tailwind.config.js to match the Next.js directory structure:

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
    "./pages/**/*.{js,ts,jsx,tsx}",
    "./components/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

To enable support for the Tailwind CSS classes we need to import it in globals.css:

@tailwind base;
@tailwind components;
@tailwind utilities;

Then we can run the app in the development mode and start coding:

npm run dev

Let's quickly create a basic template and render pages:

// pages/index.js
export default function Home({ landingPages }) {
  return (
    <div className="md:container md:mx-auto px-4">
      <h1 className="text-4xl font-bold text-center pt-5">
        Landing Page Hunt
      </h1>
      <h2 className="pb-5 text-center">
        An inspiration bucket for your next landing page.
      </h2>
      <div className="flex flex-row flex-wrap justify-center gap-8">
        {landingPages.map(landingPage =>
        (<div key={landingPage.url} className="p-2">
          <div className="text-2xl text-center py-3">
            <a href={landingPage.url}>{landingPage.name}</a>
          </div>
          <div className="mt-2 shadow-lg">
            <a href={landingPage.url}><img src={landingPage.screenshotUrl} width="400" /></a>
          </div>
          <div className="text-right text-sm py-5">
            {landingPage.likes} 👍
          </div>
        </div>))}
      </div>
    </div>
  )
}

export async function getStaticProps() {
  const accessKey = process.env.SCREENSHOTONE_ACCESS_KEY;

  const screenshotUrl = (url) => {
    return `https://api.screenshotone.com/take?access_key=${accessKey}&url=${url}&cache=true`;
  }

  const landingPages = [
    {
      name: "KTool",
      url: "https://ktool.io",
      likes: 2,
    },
    // ...
    {
      name: "Ship SaaS",
      url: "https://shipsaas.com/",
      likes: 3,
    }
  ];

  for (let landingPage of landingPages) {
    landingPage.screenshotUrl = screenshotUrl(landingPage.url);
  }

  return {
    props: {
      landingPages
    },
  }
}

Sign up to ScreenshotOne to get a screenshot API access key and to make it work, we need to create a .env.local file and set a SCREENSHOTONE_ACCESS_KEY environment variable:

SCREENSHOTONE_ACCESS_KEY=<YOUR ACCESS KEY>

Let's examine the result:

A few key highlights:

Next.js is based on React but solves its most popular pains, so we can quickly build components without thinking about routing, server-side rendering, and even API. It is a fully-fledged full-stack framework.
I used the getStaticProps function to provide data for the index page at build time. It means that our main page is not dynamic. It is static and already rendered — thus it is blazingly fast.
I didn't set up Puppeteer to take screenshots of the sites but decided to use a screenshot as a service API. It gives a bunch of free screenshots to render websites, so that's enough to play with our project. Because taking screenshots of the site is not a core topic of the post, and using Puppeteer would be a pain in the ass.

Refactoring to use dynamic content

We started by using static site generation of Next.js, but we want to like pages, submit pages and get updated results. So, we need to refactor the code to use a more sophisticated approach for our needs.

I propose to do two things:

Extract the logic related to landing pages into a separate unit — a Directory class representing the landing page directory. It will help us migrate to an actual database instead of an in-memory cache later.
And use getServerSideProps to render pages dynamically on each request.

Extracting the "business" logic

The Directory class will encapsulate all the logic related to the landing page directory:

// lib/directory.js
class Directory {
    constructor() {
        const accessKey = process.env.SCREENSHOT_ONE_ACCESS_KEY;

        const screenshotUrl = (url) => {
            return `https://api.screenshotone.com/take?access_key=${accessKey}&url=${url}&cache=true`;
        }

        const landingPages = [
            {
                name: "KTool",
                url: "https://ktool.io",
                likes: 2,
            },
            // ... 
            {
                name: "Ship SaaS",
                url: "https://shipsaas.com/",
                likes: 3,
            }
        ];

        for (let landingPage of landingPages) {
            landingPage.slug = landingPage.name.toLowerCase().replace(' ', '');
            landingPage.screenshotUrl = screenshotUrl(landingPage.url);
        }

        this.landingPages = landingPages;
    }

    async loadLandingPages() {
        return this.landingPages;
    }
}

let directory = null;
if (process.env.NODE_ENV === 'production') {
    directory = new Directory(prisma);
} else {
    if (!global.directory) {
        global.directory = new Directory(prisma);
    }

    directory = global.directory;
}

export default directory;

To speed up development, I store everything in memory and once the app is ready, we will simply migrate to use a real database.

getServerSideProps

Instead of using getStaticProps, we now switch to use getServerSideProps. It will allow us to render the page dynamically on each request. Look how simple it is:

// pages/index.js
import directory from "../lib/directory";

export default function Home(props) {
  // ... 
}

// switch to getServerSideProps instead of using getStaticProps 
// to render the content on a per-request basis.
export async function getServerSideProps(context) {
  const landingPages = await directory.loadLandingPages();

  return {
    props: {
      landingPages
    },
  }
}

Sending API requests with Next.js

Let's see how easy it is to add API handlers on the backend and send API requests from the client side. We are going to implement the "like page" feature.

First, I put the logic in the lib/directory.js:

// lib/directory.js
class Directory {
    // ... 
    async likePage(slug) {
        for (const landingPage of this.landingPages) {
            if (landingPage.slug == slug) {
                landingPage.likes++;

                return landingPage.likes;
            }
        }

        return 0;
    }
    // ...   
}

Then we create an API handler pages/api/like.js:

// pages/api/like.js
import directory from "../../lib/directory";

export default async function handler(req, res) {
  const { slug } = req.body;
  const likes = await directory.likePage(slug);

  res.status(200).json({ likes: likes });
}

Next.js will handle all the routing issues for us automatically. So, let's update the page component pages/index.js to send an HTTP request to the like API handler:

// pages/index.js
import React, { useState } from "react";
import directory from "../lib/directory";

export default function Home(props) {
  const [landingPages, setLandingPages] = useState(props.landingPages);

  const likePage = async (slug) => {
    try {
      const response = await fetch(
        `/api/like`,
        {
          headers: {
            'Content-Type': 'application/json'
          },
          method: 'POST',
          body: JSON.stringify({ slug })
        }
      );
      const data = await response.json();
      setLandingPages(
        landingPages.map(p => p.slug == slug ? { ...p, ...{ likes: data.likes } } : p)
      );
    } catch (err) {
      console.error(err);
    }
  };

  return (
    <div className="md:container md:mx-auto px-4">
      <h1 className="text-4xl font-bold text-center pt-5 bg-clip-text text-transparent bg-gradient-to-r from-purple-400 to-pink-600">
        Landing Page Hunt
      </h1>
      <h2 className="pb-5 text-center">
        An inspiration bucket for your next landing page.
      </h2>
      <div className="flex flex-row flex-wrap justify-center gap-8">
        {landingPages.map(landingPage =>
        (<div key={landingPage.url} className="p-2">
          <div className="text-2xl text-center py-3">
            <a href={landingPage.url}>{landingPage.name}</a>
          </div>
          <div className="mt-2 shadow-lg">
            <a href={landingPage.url}><img src={landingPage.screenshotUrl} width="400" /></a>
          </div>
          <div className="text-right text-sm py-5">
            <button onClick={async () => await likePage(landingPage.slug)} className="bg-transparent  hover:text-blue-400 text-blue-900 font-semibold py-2 px-4 border hover:border-blue-400 border-blue-900 rounded">
              {landingPage.likes} 👍
            </button>
          </div>
        </div>))}
      </div>
    </div>
  );
}

The result is that we can click on the like button. Then an HTTP request is made to the API handler, likes property of the page updated, and we receive the result so we can re-render all the landing pages.

More pages and forms

We need an opportunity to submit a landing page. So, let's develop the page. It is straightforward to add new pages with Next.js, and as I mentioned before, routing is done for us.

When I create pages/submit.js, Next.js will handle all requests to the page under the path /submit.

Let's create a simple submit page:

// pages/submit.js
import { useState } from "react";
import Confetti from 'react-confetti';
import { useWindowSize } from 'react-use';

export default function Submit(props) {
  const endpoint = '/api/submit';
  const [landingPage, setLandingPage] = useState(null);

  const handleSubmit = async (event) => {
    event.preventDefault();
    try {
      const response = await fetch(endpoint, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({
          name: event.target.name.value,
          url: event.target.url.value,
        }),
      })

      setLandingPage(await response.json());
    } catch (error) {
      console.error(error);
    }
  }

  const { width, height } = useWindowSize();

  return (
    <div className="mx-auto w-1/2">
      <h3 className="text-xl text-center">{landingPage ? "The page has been submitted successfully 🥳" : "Submit a landing page"}</h3>
      {
        landingPage ? (          
          <div className="mx-auto py-2 max-w-fit">
            <Confetti opacity={0.5} width={width} height={height} run={landingPage != null} />
            <div className="text-2xl text-center py-3">
              <a href={landingPage.url}>{landingPage.name}</a>
            </div>
            <div className="mt-2 shadow-lg">
              <a href={landingPage.url}><img src={landingPage.screenshotUrl} width="460" /></a>
            </div>
          </div>
        ) : (
          <form onSubmit={handleSubmit} action={endpoint} className="mt-5">
            <div className="mb-6">
              <label htmlFor="name" className="block mb-2 text-sm font-medium text-gray-900">Name</label>
              <input type="text" id="name" className="bg-gray-50 border border-gray-300 text-gray-900 text-sm rounded-lg focus:ring-blue-500 focus:border-blue-500 block w-full p-2.5 " required />
            </div>
            <div className="mb-6">
              <label htmlFor="url" className="block mb-2 text-sm font-medium text-gray-900">URL</label>
              <input type="text" id="url" className="bg-gray-50 border border-gray-300 text-gray-900 text-sm rounded-lg focus:ring-blue-500 focus:border-blue-500 block w-full p-2.5 " required />
            </div>
            <div className="text-right">
              <button type="submit" className="text-white bg-gradient-to-br from-purple-600 to-blue-500 hover:bg-gradient-to-bl font-medium rounded-lg text-sm px-5 py-2.5 text-center mr-2 mb-2">Submit</button>
            </div>
          </form>
        )
      }
    </div>
  );
}

I added confetti after the submission by using the react-confetti library:

On submit, we read the form data and send a request to a submit API handler (pages/api/submit.js):

// pages/api/submit.js
import directory from "../../lib/directory";

export default async function handler(req, res) {
    const body = req.body
    if (!body.name || !body.url) {
        return res.status(400).json({ data: 'Name and URL are required' })
    }

    const landingPage = await directory.submitPage(body.name, body.url);

    res.status(200).json(landingPage);
}

And the apparent logic for the submit function in the Directory class:

// lib/directory.js
class Directory {
    // ... 
    async submitPage(name, url) {
        const landingPage = {
            name: name,
            url: url,
            slug: this.slug(name),
            screenshotUrl: this.screenshotUrl(url),
            likes: 0,
        };

        this.landingPages.push(landingPage);

        return landingPage;
    }

    slug(name) {
        return name.toLowerCase().replace(' ', '');
    }

    screenshotUrl(url) {
        const accessKey = process.env.SCREENSHOTONE_ACCESS_KEY;

        return `https://api.screenshotone.com/take?access_key=${accessKey}&url=${url}&cache=true`;
    }
    // ... 
}

That's how simple it is to develop with Next.js. Do you feel it?

Switching to a SQL database — a case for Prisma

The core functionality of our app is ready and tested. Since it is encapsulated, we can easily switch to storing data in the permanent database, not in temporary memory.

But before that, let's install and configure Prisma:

npm i prisma

Let's create a directory, prisma, where we will store all the Prisma-related configuration and start with the main DB configuration and schema:

// prisma/schema.prisma
generator client {
    provider = "prisma-client-js"
}

datasource db {
    provider = "sqlite"
    url      = "file:./dev.db"
}

model LandingPage {
    id    Int    @id @default(autoincrement())
    slug  String @unique()
    name  String
    url   String
    likes Int

    @@index([slug])
}

The schema file is written in PSL (Prisma Schema Language).

Then, let's create a seed file prisma/seed.js with landing pages:

// prisma/seed.js
const { PrismaClient } = require('@prisma/client');

const prisma = new PrismaClient();

const landingPages = [
    {
        slug: "ktool",
        name: "KTool",
        url: "https://ktool.io",
        likes: 2,
    },
    // ... 
    {
        slug: "shipsaas",
        name: "Ship SaaS",
        url: "https://shipsaas.com/",
        likes: 3,
    }
];


async function main() {
    console.log(`Start seeding ...`)
    for (const lp of landingPages) {
        const user = await prisma.landingPage.create({
            data: lp,
        })
        console.log(`Created landing page with id: ${user.id}`)
    }
    console.log(`Seeding finished.`)
}

main()
    .then(async () => {
        await prisma.$disconnect()
    })
    .catch(async (e) => {
        console.error(e)
        await prisma.$disconnect()
        process.exit(1)
    });

To make it part of the migration process, we need to specify the seed in package.json:

{
    // ...
    "prisma": {
        "seed": "node prisma/seed.js"
    }
    // ...
}

And after running migrations, including seeding:

npx prisma migrate dev

We are ready to use the Prisma client in our application. Let's instantiate the Prisma client in lib/prisma.js:

// lib/prisma.js
import { PrismaClient } from "@prisma/client";

let prisma = null;
if (process.env.NODE_ENV === 'production') {
    prisma = new PrismaClient();
} else {
    if (!global.prisma) {
        global.prisma = new PrismaClient();
    }

    prisma = global.prisma;
}

export default prisma;

PrismaClient is attached to the global object in development to prevent exhausting your database connection limit.

And we are ready to refactor the Directory class to entirely rely on Prisma:

// lib/directory.js
import prisma from "./prisma";
import screenshotUrl from "./screenshotone";

class Directory {
    constructor(prisma) {
        this.prisma = prisma;
    }

    async submitPage(name, url) {
        const landingPage = await this.prisma.landingPage.create({
            data: {
                slug: this.slug(name),
                name,
                url,
                likes: 0,
            },
        })

        return {
            ...landingPage,
            screenshotUrl: screenshotUrl(url)
        };
    }

    async likePage(slug) {
        const landingPage = await this.prisma.landingPage.update({
            where: {
                slug: slug
            },
            data: {
                likes: {
                    increment: 1,
                }
            },
        });

        return landingPage.likes;
    }

    async loadLandingPages() {
        const pages = await this.prisma.landingPage.findMany();

        return pages.map(p => {
            return {
                ...p,
                screenshotUrl: screenshotUrl(p.url)
            };
        });
    }

    slug(name) {
        return name.toLowerCase().replace(' ', '').replace('.', '');
    } 
}

let directory = null;
if (process.env.NODE_ENV === 'production') {
    directory = new Directory(prisma);
} else {
    if (!global.directory) {
        global.directory = new Directory(prisma);
    }

    directory = global.directory;
}

export default directory;

I chose SQLite for simplicity. I also extracted the link generation logic to a separate module.

Securing API keys for screenshot API

The rendered screenshots in the app expose the screenshot API access key. For the development stage, it was OK, but for deployment to production, we need to store the key securely.

ScreenshotOne API allows using signed links. It means that every exposed screenshot URL is publicly signed, and if you try to change any parameter and to reuse (steal) an API key, you need to compute a new signature, but you can't without having a secret key.

In addition to that, you can force ScreenshotOne API to accept only signed requests.

That's how I generated the signed links with ScreenshotOne API SDK:

// lib/screenshotone.js
import * as screenshotone from 'screenshotone-api-sdk';

let client = null;
if (process.env.NODE_ENV === 'production') {
    client = new screenshotone.Client(process.env.SCREENSHOTONE_ACCESS_KEY, process.env.SCREENSHOTONE_SECRET_KEY);
} else {
    if (!global.client) {
        global.client = new screenshotone.Client(process.env.SCREENSHOTONE_ACCESS_KEY, process.env.SCREENSHOTONE_SECRET_KEY);
    }

    client = global.client;
}

export default function url(url) {
    const options = screenshotone.TakeOptions
        .url(url)
        .cache(true)
        .cacheTtl(2000000)
        .blockChats(true)
        .blockCookieBanners(true)
        .blockAds(true);

    return client.generateTakeURL(options);
}

Another solution would be to render screenshots and store screenshots on the backend site and share rendered images with the front end. Or use the "upload screenshots to S3" feature by ScreenshotOne and then serve files from a CDN.

Expanding the app to complete the SaaS solution

And we have arrived at the stage when we want to make a bit of money with our small, simple project. Imagine allowing accept payments for featuring landing pages. But users will need to sign up. And we need to make sure that voting is fair. There are a few things required to do it:

Authentication;
Billing and subscriptions;
Transactional emails;
Dashboard to manage the app;
Cover the app with tests;
Make sure that the site is SEO-friendly;
Internalization if we want to go wide.

Next.js has many solutions you can combine to solve all problems mentioned above.

But I want to propose something interesting if your goal is to ship SaaS and not just play with code like I love to do. I would argue that starting with a SaaS template is better to save time and energy.

Nowadays, there are many SaaS solutions you can pick up to bootstrap your project quickly. Take a look at Ship SaaS — a SaaS template based on Next.js, for example, and the problems it solves for you. It literally can save you thousands of hours.

Instead of summary

As you might sense, I enjoyed it a lot to build with Next.js, Tailwind CSS, and Prisma. I am impatient to try them for any new idea I will have in the future. Many problems are already solved in these frameworks, and it is easy to use them.
In addition, there are a bunch of good SaaS templates I can use if I want to bootstrap a SaaS quicker.

I migrated to Google Cloud

Dmytro Krasun — Wed, 14 Sep 2022 15:00:25 +0000

I migrated my screenshot API project to Google Cloud.

My main reason is that I want peace of mind with my infrastructure for an extended period of time. And I want to concentrate solely on customer support, marketing, and selling, not on scaling issues and other technical matters.

If you are curious about the details, I am sharing a short story about my infrastructure journey and plans for the future.

MVP on DigitalOcean with 2 droplets

Truthfully, I started from MVP on DigitalOcean with 2 droplets (VPS), one for the dashboard and one for rendering screenshots.

It was enough to test the product and launch, but after users started to sign up and use the API, scaling issues arose, and I needed to act quickly. I had many options to choose from:

Scale the droplet vertically (increase CPU and RAM).
Add one more droplet and load balancer.
Set up Kubernetes cluster with autoscaling.
Move to Render or other services that can quickly spin up Docker containers and support autoscaling.
Buy powerful cheap servers or VPS to handle all the load I have in one or two instances.
Using AWS Lambda functions or Google Cloud Run or functions.
And many other options.

But I didn't have time even to think. I needed a solution quickly. Kubernetes looked like the best choice then, but I didn't have experience setting it up. I worked only as a developer with Kubernetes.

I had an experience with AWS, but not with AWS Lambda.

So, I decided to move to Render.

Moving to Render as a quick solution

Moving to Render was easy. I needed to link my GitHub repository and choose the plan (means resources). Then in 2 clicks, I set up autoscaling. And it helped me to survive for a long time. At least 2-3 months.

But then, new requirements and use cases arose, which Render couldn't handle for me. I needed to spin up a new service instance immediately upon request, not on an average load of the instances I had.

Some customers can send 6-20 requests in a batch simultaneously. Render's load balancer distributes the load among instances evenly. But if I have 2 instances and 20 requests received simultaneously, it will split the batch by 10 requests each and send them to instances. And then spin up new instances which won't process these requests and will just idle.

The easy solution is to have more instances in the pool available. And it won't be a problem for me. But to process requests, I needed at least 8 GB of RAM. And that's when it becomes too costly with Render.

I loved Render's UI and support. Would it be cheaper? I won't move away.

So, I started looking for other solutions. My limitations were time to migrate, obviously money, and to support new requirements to handle immediate bursts of load.

Trying Google Cloud Run

I started exploring AWS Lambda functions, Google Cloud Functions, and Google Cloud Run. Since I haven't used Google Cloud before, I was eligible to use $300 in credits for 3 months. I decided it was a reasonable time period to test the cloud and estimate my costs and overall experience.

It took me 4 hours to deploy the service to Google Cloud Run with a straightforward command—gcloud deploy. For another 2 days, I spent on a solution to move my custom domain from Render to GCP without downtime. But it was costly in terms of time and nearly impossible, so I found the time slot when my servers were idle, and nobody used them and migrated my domain.

Why Run and not Functions?

I haven't tried GCP functions, but GCP Run looked like a good fit. It can immediately spin up new containers to handle requests and reuse containers. And I pay only when my containers are running.

With such a configuration, I felt I could have a piece of mind for long periods.

Once the trial approaches the end, I will evaluate the costs and usability of the cloud and decide what to do next.

Why I share my infrastructure journey

Frankly, from the inside, it is hard to understand what I am doing wrong or if I do effectively what I do. But many brilliant minds can look at what you do and share many ideas on what can be done better, cheaper, more effective, and so on.

I would love to hear your thoughts, and thanks for reading.

How to take website screenshots with Go

Dmytro Krasun — Wed, 01 Jun 2022 15:20:50 +0000

Today, there are many options to make screenshots of any URL with Go:

Selenium WebDriver client for Go.
You can use the Go analog of Puppeteer.
Playwright library.
Or Screenshot API as a service.

They might overlap, but there is no best solution. Each depends on your use case and requirements.

Selenium WebDriver client for Go

Selenium is a well-known kid in the QA automation area, so it is easy to start taking screenshots if you plan to write automation tests or already do it.

Install Selenium WebDriver client for Go into your project:

go get github.com/tebeka/selenium

Download the latest version ChromeDriver and the latest remote Selenium WebDriver standalone server, place them in the same source directory, and you are ready to Go:

package main

import (
    "fmt"
    "os"

    "github.com/tebeka/selenium"
)

func main() {
    const (
        seleniumPath     = "./selenium-server-standalone-3.5.3.jar"
        chromeDriverPath = "./chromedriver"
        port             = 8080
    )
    opts := []selenium.ServiceOption{
        selenium.ChromeDriver(chromeDriverPath),
        selenium.Output(os.Stderr),
    }
    selenium.SetDebug(true)
    service, err := selenium.NewSeleniumService(seleniumPath, port, opts...)
    if err != nil {
        panic(err) // panic is used only as an example and is not otherwise recommended
    }
    defer service.Stop()

    // connect to the WebDriver instance running locally
    caps := selenium.Capabilities{"browserName": "chrome"}
    wd, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", port))
    if err != nil {
        panic(err)
    }
    defer wd.Quit()

    if err := wd.Get("https://scalabledeveloper.com"); err != nil {
        panic(err)
    }

    screenshot, err := wd.Screenshot()
    if err != nil {
        panic(err)
    }

    out, err := os.Create("scalabledeveloper.jpg")
    if err != nil {
        panic(err)
    }
    defer out.Close()

    _, err = out.Write(screenshot)
    if err != nil {
        panic(err)
    }
}

If you just want to take one or two screenshots locally, the Selenium WebDriver client is not the best fit. As I wrote earlier, it better serves you already write automation tests with Selenium or plan to write them.

Go analog of Puppeteer

chromedp is the closest analog to Node.js Puppeteer in Go, a faster, more straightforward way to drive browsers supporting the Chrome DevTools Protocol in Go without external dependencies.

While you can also use chromedp for automation testing, it is much light weighter and simpler than Selenium and allows any automation over the browser. Everything you can do manually with a browser, you can also do with chromedp.

And it is super easy to use:

go get github.com/chromedp/chromedp

And then just:

func main() {
    ctx, cancel := chromedp.NewContext(context.Background())
    defer cancel()

    var buf []byte
    if err := chromedp.Run(ctx,
        chromedp.Navigate(`https://example.com`),
        chromedp.FullScreenshot(&buf, 95),
    ); err != nil {
        log.Fatal(err)
    }

    if err := ioutil.WriteFile("example.jpg", buf, 0644); err != nil {
        log.Fatal(err)
    }
}

It is very lightweight compared to Selenium and allows a broad spectrum for automation of the browser. You can use it for crawling, scrapping, taking screenshots, etc.
If you need the simplest way to take screenshots, you do not expect to take millions of them, and I would go with chromedp. But if you want to take screenshots from different browsers or think about managing instances of browsers, there is a more straightforward way to go.

Playwright

Playwright is the best Go library to automate Chromium, Firefox, and WebKit with a unified API. It is built to enable cross-browser web automation.

If you want to take screenshots in different browsers, Playwright is the best choice.

Install the library with go get:

go get github.com/playwright-community/playwright-go

Do not forget to install browser and OS dependencies:

go run github.com/playwright-community/playwright-go/cmd/playwright install --with-deps

And then take a screenshot:

package main

import (
    "log"

    "github.com/playwright-community/playwright-go"
)

func main() {
    pw, err := playwright.Run()
    if err != nil {
        log.Fatalf("could not launch playwright: %v", err)
    }
    browser, err := pw.Chromium.Launch()
    if err != nil {
        log.Fatalf("could not launch Chromium: %v", err)
    }
    page, err := browser.NewPage()
    if err != nil {
        log.Fatalf("could not create page: %v", err)
    }
    if _, err = page.Goto("https://example.com/", playwright.PageGotoOptions{
        WaitUntil: playwright.WaitUntilStateNetworkidle,
    }); err != nil {
        log.Fatalf("could not goto: %v", err)
    }
    if _, err = page.Screenshot(playwright.PageScreenshotOptions{
        Path: playwright.String("example.png"),
    }); err != nil {
        log.Fatalf("could not create screenshot: %v", err)
    }
    if err = browser.Close(); err != nil {
        log.Fatalf("could not close browser: %v", err)
    }
    if err = pw.Stop(); err != nil {
        log.Fatalf("could not stop Playwright: %v", err)
    }
}

Playwright is the best choice. It is as powerful as chromedp but allows you to run different browser instances if needed.
If you plan to take millions of screenshots and manage browser instances, you can do it yourself, but it is better to outsource to well-established services.

Screenshot API as a service

I specialize in taking screenshots and managing browser instances at scale. I provide a high-quality Go client to screenshots and cover a variety of use cases.

Easy to install:

go get github.com/screenshotone/gosdk

And easy to use:

client, err := screenshots.NewClient("IVmt2ghj9TG_jQ", "Sxt94yAj9aQSgg")
if err != nil {
    // ...
}

options := screenshots.NewTakeOptions("https://example.com").
    Format("png").
    FullPage(true).
    DeviceScaleFactor(2).
    BlockAds(true).
    BlockTrackers(true)

image, _, err := client.Take(context.TODO(), options)
if err != nil {
    // ...
}

out, err := os.Create("example.png")
if err != nil {
    // ...
}
defer out.Close()

_, err = out.Write(out, image)
if err != nil {
    // ...
}

If you feel that it is the best fit for you, feel free to sign up for our screenshot API and get the access key.

Summary

Pick the solution which suits your needs best. And have a nice day 👋

How to block requests with Puppeteer

Dmytro Krasun — Wed, 09 Mar 2022 20:49:29 +0000

If you want to speed up scrapping or make screenshots faster, you can disable all the requests that do not make any crucial impact on the results.

Puppeteer allows blocking any outgoing requests while loading the page. Whether you want to block ads, tracking scripts, or different types of resources, it is relatively easy to do with Puppeteer.

A fully working example of blocking requests

Let's start with a fully working example on how to intercept and block requests in Puppeteer:

const puppeteer = require('puppeteer');
const wildcardMatch = require('wildcard-match');

const blockRequest = wildcardMatch(['*.css', '*.js'], { separator: false });

(async () => {
    const browser = await puppeteer.launch({});
    try {

        const page = await browser.newPage();
        page.setRequestInterception(true);

        page.on('request', (request) => {
            if (blockRequest(request.url())) {
                const u = request.url();
                console.log(`request to ${u.substring(0, 50)}...${u.substring(u.length - 5)} is aborted`);

                request.abort();

                return;
            }

            request.continue();
        });

        await page.goto('https://screenshotone.com/');
    } catch (e) {
        console.log(e)
    } finally {
        await browser.close();
    }
})();

The result is:

request to https://screenshotone.com/main.7a76b580aa30ffecb0b...f.css is aborted
request to https://screenshotone.com/js/bootstrap.min.592b9fa...ab.js is aborted
request to https://screenshotone.com/js/highlight.min.e13cfba...5f.js is aborted
request to https://screenshotone.com/main.min.dabf7f45921a731...45.js is aborted

Sorry, but I won't show you the resulting screenshot of the site because it looks awful without CSS and JS.

A step-by-step explanation

The most crucial step is not to forget to enable request interception before sending any request:

// ... 
const page = await browser.newPage();
page.setRequestInterception(true);
// ...

Otherwise, the trick won't work.

After request interception is enabled, you can listen to any new outgoing request while the page is being loaded and decide on a per-request basis whether to block the request or not.

If you want to block all requests to www.google-analytics.com to speed up the site loading and to avoid tracking, then just filter requests based on the domain substring:

page.on('request', (request) => {
    if (request.url().includes('www.google-analytics.com')) {    
        request.abort();

        return;

    }

    request.continue();
});

The better option is to parse URL, extract domain, and filter based on the domain name:

page.on('request', (request) => {
    const domain = url.parse(request.url(), false).hostname;
    if (domain == 'www.google-analytics.com') {
        request.abort();

        return;
    }

    request.continue();
});

Because you might have an URL that accidentally might include www.google-analytics.com.

Blocking requests by resource type

If you need to block a set of requests by the resource type, like images or stylesheets, regardless of the extension and URL pattern, you can use the request.resourceType() method to test against blocking resource type:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({});
    try {
        const page = await browser.newPage();
        page.setRequestInterception(true);

        page.on('request', (request) => {
            if (request.resourceType() == "stylesheet" || request.resourceType() == "script") {
                const u = request.url();
                console.log(`request to ${u.substring(0, 50)}...${u.substring(u.length - 5)} is aborted`);

                request.abort();

                return;
            }

            request.continue();
        });

        await page.goto('https://screenshotone.com/');
    } catch (e) {
        console.log(e)
    } finally {
        await browser.close();
    }
})();

The result is the same as for the initial example:

request to https://screenshotone.com/main.7a76b580aa30ffecb0b...f.css is aborted
request to https://screenshotone.com/js/bootstrap.min.592b9fa...ab.js is aborted
request to https://screenshotone.com/js/highlight.min.e13cfba...5f.js is aborted
request to https://screenshotone.com/main.min.dabf7f45921a731...45.js is aborted

Puppetteer supports blocking the next resource types:

document
stylesheet
image
media
font
script
texttrack
xhr
fetch
eventsource
websocket
manifest
other

As you see, it is pretty straightforward.

Have a nice day 👋

I hope I have helped you tackle request blocking in Puppeteer, and I honestly wish you a nice day!

Context in Go

Dmytro Krasun — Sun, 09 Jan 2022 14:08:57 +0000

Go is a very pragmatic language. If there is a thing in the Go standard library, there is an actual practical reason for it. The same stays valid for Context.

for impatient

use to specify a deadline, timeout, or cancel the running goroutine

Use Context when you need:

to run a function with time limation, use Context.WithDeadline() or Context.WithTimeout();
to be able to cancel running function Context.WithCancel.

use for sharing request-scoped values

Use Context if you need to share request-scoped values across API boundaries.

A trace id can be used for tracing operations between microservices. You can receive it in the HTTP request as a header and then pass it to other functions using context, and if the functions perform calls to other microservices, they can extract the trace id and pass it further.

for curious

channels and goroutines

A quick reminder that Go is famous for implementing CSP with channels and goroutines. You write functions, run them concurrently with the go command, and communicate with channels. It is that simple:

package main

import (
    "fmt"
    "time"
)

func main() {
    numbers := make(chan int)
    results := make(chan int)

    ticker := time.NewTicker(1 * time.Second)
    timer := time.NewTimer(5 * time.Second)

    go compute(numbers, results)

    n := 0
    for {
        select {
        case <-ticker.C:
            fmt.Printf("number: %d\n", n)
            numbers <- n
            n++
        case r := <-results:
            fmt.Printf("total sum: %d\n\n", r)
        case <-timer.C:
            fmt.Println("the time went out for a walk")
            return
        }
    }
}

func compute(numbers <-chan int, results chan<- int) {
    sum := 0
    for {
        n := <-numbers
        sum += n
        results <- sum
    }
}

You can run thousands of goroutines and build applications with complex communication patterns by combining channels and goroutines. But always profile your solution.

And that's what happened, but again and again, the same problem arose in a lot of different places.

But then you want to specify a timeout, deadline, or to be able to cancel the running goroutine at any time.

the pattern is extracted into the standard library

Not exactly, but I assume that's how Context emerged. Sameer Ajmani wrote a blog post about the Context pattern:

In Go servers, each incoming request is handled in its own goroutine. Request handlers often start additional goroutines to access backends such as databases and RPC services. The set of goroutines working on a request typically needs access to request-specific values such as the identity of the end user, authorization tokens, and the request’s deadline. When a request is canceled or times out, all the goroutines working on that request should exit quickly so the system can reclaim any resources they are using.

At Google, we developed a context package that makes it easy to pass request-scoped values, cancellation signals, and deadlines across API boundaries to all the goroutines involved in handling a request. The package is publicly available as context. This article describes how to use the package and provides a complete working example.

one critical example from the standard library

Go's standard HTTP client does not require you to specify the context when performing GET requests:

response, err := http.Get("https://...")

So, you can forget to specify timeout and then "don’t use Go’s default HTTP client (in production)".

But from Go 1.17 you can specify the timeout on the per-request basis:

client := &http.Client{Timeout: time.Minute}

context, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(context, http.MethodGet, "https://...", nil)
if err != nil {
    // ...
}

resp, err := client.Do(req)
// ...

I hardly encourage you always specify the timeout for HTTP requests.

How to take a screenshot with Puppeteer

Dmytro Krasun — Wed, 05 Jan 2022 14:17:43 +0000

Making screenshots of the websites with Puppeteer can be tricky. A lot of pitfalls wait for us. Let's examine Puppeteer on a set of "screenshotting" problems and tackle arising pitfalls.

I posted worked Puppeteer examples to understand the context of the solution better and copy it if needed.

Meet Puppeteer

It is a Node library that interacts with browsers that support Chrome DevTools Protocol (CDP). It is not only Chrome and Chromium, but Firefox also has partial support of CDP.

The Chrome DevTools Protocol was developed to manage, debug and inspect Chromium and Chrome at the low level.

So, think of Puppeteer high-level API over Chrome DevTools Protocol which allows you to do everything in the browser that you can do manually:

Extract data from a SPA, submit a form, type text, perform end-to-end UI testing and other automation-related tasks.
Debug performance issues.
Run, debug and test Chrome Extensions.
Pre-render SPA to make a static site. But for Google SEO, it does not matter since Google renders JavaScript for every page nowadays.
And guess what? Make screenshots and PDFs of pages.

Generating Screenshots and PDFs with Puppeteer is the main focus of the post.

Puppeteer architecture and internals for curious

You can skip this section. It is not required to start using the library. But I love to explore the internals of the libraries I use, and so might you.

Lightweight option of Puppeteer

First of all, there are two versions of the library available: puppeteer-core and puppeteer. You should use puppeteer-core when you are going to manage browser instances by yourself, or you do not need it, otherwise stick to puppeteer.

Three simple examples that come to my mind with puppeteer-core:

You are using CDP from the extension, so you do not have to download Chrome or Chromium.
You want to use a different Chrome, Chromium, or Firefox build.
You have a running cluster of browsers or a separate browser instance on an other machine.

When you use puppeteer-core, you must ensure that you use a compatible browser version. But the puppeteer library downloads and runs a compatible version of Chromium instance for you, without any worries.

Puppeteer Alternatives

There are a lot more, but the most popular two are:

The oldest alternative to make screenshots is using the Selenium WebDriver protocol.
The second one is Playwright, and it is a good one. It is the competitor to the Puppeteer.

Playwright and Puppeteer have compatible API, but Playwright supports more browsers. So, if you must take screenshots in different browsers, prefer to use Playwright. By the way, top contributors of the Puppeteer work on Playwright. But the library is still considered new.

Practical Examples of using Puppeteer to take screenshots

Before starting to work with Puppeteer, let's install it using npm:

$ npm i puppeteer

A simple screenshot

To take a simple screenshot with Puppeteer and save it into the file, you can use the following code:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();
           await page.goto('https://example.com');
           await page.screenshot({ path: 'example.png' });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Always close the browser to avoid resource leaking.

Resolution and Retina Display

To avoid blurred images on a high-resolution display like Retina Display you can change the viewport properties width, height and deviceScaleFactor:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();

           await page.setViewport({
               width: 2880, // default: 800
               height: 1800, // default: 600 
               deviceScaleFactor: 2 // default: 1
           });

           await page.goto('https://apple.com');
           await page.screenshot({ path: 'apple.com.png' });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

That's called pixel-perfect screenshots.

A full page screenshot

Puppeteer knows how to make screenshot of the scrollable page. Use fullPage option:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();
           await page.goto('https://apple.com');
           await page.screenshot({ path: 'apple.com.png', fullPage: true });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

But it won't work with "infinite" scroll.

A full page screenshot with "infinite" scroll

It is out of the scope for the article, but it is hard to find the case when you need to make screenshots with "infinite" scroll sites. And if you need to, you can use the next algorithm:

Load the page, wait until it is loaded.
Scrolling until there the size of the page is not changed.
Take the screenshot.

If you try to do it with Twitter or Instagram for account that has a lot of posts, you absolutely will end up with crashed browser instance due to the memory exhaustion.

Wait until the page is completely loaded

It is a good practice to wait until the page is completely loaded to make screenshot:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch({});

       try {
           const page = await browser.newPage();

           await page.goto('https://apple.com/', {
               waitUntil: 'networkidle0',
           });

           await page.screenshot({ path: 'apple.com.png' });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

It is a little bit of magic, but networkidle0 event is heuristic to determine page load state. The Puppeteer team finds it working quite well for many real-world use cases.

But if you need to wait until some element is rendered and visible, you need to add Page.waitForSelector():

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch({});

       try {
           const page = await browser.newPage();

           await page.goto('https://example.com/', {
               waitUntil: 'networkidle0',
           });

           const selector = 'div';
           await page.waitForSelector(selector, {
               visible: true,
           });

           await page.screenshot({ path: 'example.com.png' });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

You can also wait:

A screenshot of the page area

To take the screenshot of the page area, use the clip option:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();
           await page.goto('https://apple.com');
           await page.screenshot({
               path: 'apple.com.png',
               clip: {
                   x: 100,
                   y: 100,
                   width: 800,
                   height: 800
               },
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

But if you need to take a screenshot of the element, there is a better approach.

A screenshot of the specific element

Puppeteer allows to take the screenshot of any element on the web page:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();
           await page.goto('https://example.com');

           const selector = 'body > div:first-child';
           await page.waitForSelector(selector);
           const element = await page.$(selector); 

           await element.screenshot({
               path: 'example.com.png',            
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

As you see, it is essential to make sure that the element is ready.

A screenshot with transparent background

Puppeteer provides a useful option to omit the background of the site. Just set omitBackground to true:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();
           await page.goto('https://example.com');

           await page.screenshot({
               path: 'example.com.png',
               omitBackground: true,            
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Have you run the code? If yes, you spotted that the screenshot does not have a transparent background. It happens because omitting background works only for elements with transparent background.

So if your target site does not have a transparent background and you want to force it, you can use JavaScript to accomplish the task. Change the background of the body in the evaluate function:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();
           await page.goto('https://example.com');

           await page.evaluate(() => {            
               document.body.style.background = 'transparent';
           });

           await page.screenshot({
               path: 'example.com.png',
               omitBackground: true,            
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Screenshot as Base64

You build Puppeteer as a service and do not want to store screenshot files. You can choose to return the screenshot in Base64 encoding format:

'use strict';  

const puppeteer = require('puppeteer');  

(async () => {  
   const browser = await puppeteer.launch({});  

   try {  
       const page = await browser.newPage();  
       await page.goto('https://example.com/');  

       const base64 = await page.screenshot({ encoding: "base64" })  
       console.log(base64);  
   } catch (e) {  
       console.log(e)  
   } finally {  
       await browser.close();  
   }  
})();

You will receive a string that you can share with another service or even store somewhere.

Generate PDF instead of PNG

It is relatively easy to generate PDF instead of PNG:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch({});

       try {
           const page = await browser.newPage();

           await page.goto('https://example.com/', {
               waitUntil: 'networkidle0',
           });

           const selector = 'div';
           await page.waitForSelector(selector, {
               visible: true,
           });

           await page.pdf({path: 'example.com.pdf', format: 'a4'})        
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Look at all possible Puppeteer PDF options. It is an exciting and complex problem, which deserves a separate post.

It depends on your use case, but also consider using PDFKit for programmatic PDF generation.

Blocking ads when using Puppeteer

I do not use any ad blocking extension because life is tough, and everybody needs some way to earn money. If I can help sites sustain and survive by non-blocking the ads, I will do it.

But when you test your site or your customer site, you might need to block the ads. There are 2 ways to do it:

Intercept and block request that load ad into the site.
Use an extension that is optimized exactly to solve this problem.

The first one is tricky and highly depends on the site you are taking screenshots of. But using an extension is a highly-scalable approach that works out of the box.

Install puppeteer-extra and puppeteer-extra-plugin-adblocker in addition to puppeteer package:

$ npm i puppeteer-extra puppeteer-extra-plugin-adblocker

And then use it:

'use strict';

const puppeteer = require('puppeteer-extra');

const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
puppeteer.use(AdblockerPlugin());

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();

           // ads are blocked automatically
           await page.goto('https://www.example.com');

           await page.screenshot({
               path: 'example.com.png',
               fullPage: true,
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Most pages include ads and trackers, which consume a lot of bandwidth and take a long time to load. Because fewer requests are made, and less JavaScript is performed when advertisements and trackers are blocked, pages load substantially quicker.

Block trackers

To take screenshots faster you might block trackers. It will help to speed up rendering. The ad blocking plugin can help us with this issue.

Do not forget to install puppeteer-extra and puppeteer-extra-plugin-adblocker in addition to puppeteer package:

$ npm i puppeteer-extra puppeteer-extra-plugin-adblocker

And then use it:

'use strict';

const puppeteer = require('puppeteer-extra');

const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
puppeteer.use(AdblockerPlugin({
       blockTrackers: true, // default: false
}));

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();

           // ads are blocked automatically
           await page.goto('https://www.example.com');

           await page.screenshot({
               path: 'example.com.png',
               fullPage: true,
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

If you need to block only trackers, but do not block ads, just use request interceptor.

Preventing Puppeteer detection

Some sites might block your Puppeteer script because of the user agent, and it is easy to fix:

'use strict';

const puppeteer = require('puppeteer');

(async () => {    
       const options = {
           args: [
               '--user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"'
           ],
           headless: true,
       };

       const browser = await puppeteer.launch(options);
       try {
           const page = await browser.newPage();
           await page.goto('https://www.example.com');

           await page.screenshot({
               path: 'example.com.png',
               fullPage: true,
           });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

There are also many other hacks to ensure that Puppeteer is not detected, but you can save time by using the ready puppeteer-extra-plugin-stealth plugin for the stealth mode. Install it in addition to puppeteer package:

$ npm i puppeteer-extra puppeteer-extra-plugin-stealth

And then use:

'use strict';

const puppeteer = require('puppeteer-extra');

const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();        

           await page.evaluateOnNewDocument(() => {
               const newProto = navigator.__proto__;
               delete newProto.webdriver;
               navigator.__proto__ = newProto;
           });

           await page.goto('https://bot.sannysoft.com');        
           await page.waitForTimeout(5000);
           await page.screenshot({ path: 'stealth.png', fullPage: true });

       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Important! As you see, I remove the webdriver property since the stealth plugin misses this hack and by using webdriver property usage of the Puppeteer can be detected.

Hide cookies banners

It is a tricky task to implement generically, but you can accept a cookie by finding the selector of the Accept or reject button and clicking on it.

Using basic access authentication with Puppeteer

If your page is protected by HTTP basic access authentication, the only thing you need to do is to specify username and password before loading and taking the screenshot of the page:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch();

       try {
           const page = await browser.newPage();

           await page.authenticate({'username':'YOUR_BASIC_AUTH_USERNAME', 'password': 'YOUR_BASIC_AUTH_PASSWORD'});

           await page.goto('https://example.com');
           await page.screenshot({ path: 'example.png' });
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Using a proxy for Puppeteer

In case if you need to use a proxy to make a screenshot with Puppeteer, you can specify a browser-wide proxy:

const puppeteer = require('puppeteer');

(async () => {
       const browser = await puppeteer.launch({
           args: ['--proxy-server=127.0.0.1:9876']
       });

       try {
           const page = await browser.newPage();

           await page.goto('https://example.com/', {
               waitUntil: 'networkidle0',
           });

           await page.screenshot({ path: 'example.com.png' })
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

But in some cases, you might want to use a page-wide proxy without recreating the browser instance. In this case, you can install puppeteer-page-proxy:

npm i puppeteer-page-proxy

And use it specify proxy on per-page basis:

const puppeteer = require('puppeteer');
const useProxy = require('puppeteer-page-proxy');

(async () => {
       const browser = await puppeteer.launch({});

       try {
           const page = await browser.newPage();

           useProxy(page, '127.0.0.1:9876')

           await page.goto('https://example.com/', {
               waitUntil: 'networkidle0',
           });

           await page.screenshot({ path: 'example.com.png' })
       } catch (e) {
           console.log(e)
       } finally {
           await browser.close();
       }
})();

Add support of emojis, Japanese, Arabic and other non-Latin languages to Puppeteer

If you run Puppeteer in OS without emojis support, you need to install OS-wide fonts to support emojis. The same can happen with non-English characters like Chinese, Japanese, Korean, Arabic, Hebrew, etc.

To get Puppeteer to render emojis, you can use Noto Fonts published under SIL Open Font License (OFL) v1.1.

You need to search and how to install fonts for your host OS.

Have a nice day 👋

I posted a lot of Puppeteer examples, and I hope I helped you solve your screenshot problems with Puppeteer. I described every problem I encountered and the solution to it.

Linux kernel coding style

Dmytro Krasun — Fri, 17 Sep 2021 08:36:37 +0000

Of course, you have heard of the "never use GOTO statement" mantra, and of course, you are terrified not to follow the mantra. But why there is a lot of such statements in the linux kernel? Or why should you write as a few comments as possible?

What can we learn from the coding style of the project to which thousands of developers contributed?

If only the one thing I could extract from the linux kernel coding style, it would be the pragmatic approach to coding style. There is no one ideal and ubiquitous solution for everybody and every project.

Linus is the primary maintainer of the linux kernel. He is a gatekeeper, and he is a filter. Linus works a lot with the code written by other developers. And he needs to read a lot and to understand a lot. Any developer from any point of the world can submit a patch to the kernel. It is better to have a consistent style dictated by the person responsible for the project than trying to find the common denominator that can satisfy all the participants. It won't help anyway because you will always find two developers with opposite views on coding style.

Essentials

Indentation

Let's start from indentation. I won't open a holy war, I want to pay attention to how programmatic is the linux kernel coding style:

Tabs are 8 characters, and thus indentations are also 8 characters. There are heretic movements that try to make indentations 4 (or even 2!) characters deep, and that is akin to trying to define the value of PI to be 3.

Rationale: The whole idea behind indentation is to clearly define where a block of control starts and ends. Especially when you've been looking at your screen for 20 straight hours, you'll find it a lot easier to see how the indentation works if you have large indentations.

The critical moment is that the coding style is optimized for looking at the screen for 20 straight hours.

Functions

There is an old and well-known single responsibility principle for functions:

Functions should be short and sweet, and do just one thing. They should fit on one or two screenfuls of text (the ISO/ANSI screen size is 80x24, as we all know), and do one thing and do that well.

How can you argue with this?

Commenting

Every time you update your code, you also need to update your comments. So there is one more entity that you need to support when changing the code:

Comments are good, but there is also a danger of over-commenting. NEVER try to explain HOW your code works in a comment: it's much better to write the code so that the working is obvious, and it's a waste of time to explain badly written code.

Generally, you want your comments to tell WHAT your code does, not HOW.

The case for GOTO

The case is not apparent for developers coming with experience from garbage-collected programming languages like Java.

Example is taken from the block "7) Centralized existing of functions":

int fun(int a)
{
        int result = 0;
        char *buffer;

        buffer = kmalloc(SIZE, GFP_KERNEL);
        if (!buffer)
                return -ENOMEM;

        if (condition1) {
                while (loop1) {
                        ...
                }
                result = 1;
                goto out_free_buffer;
        }
        ...
out_free_buffer:
        kfree(buffer);
        return result;
}

So, in this example, you have less error-prone code. Because if you change the last block with the label, you don't need to be afraid that somewhere in the function, you need to update a return statement. You have one centralized block to free resources.

But in Java, you can achieve the same behavior with the try-with-resources statement.

Summary

Do not be afraid to deviate from the standard or accepted coding style of your language except Go. You need to adapt it for your needs and the needs of your team to make your job more productive and engaging.

Latency versus Response Time

Dmytro Krasun — Sun, 14 Mar 2021 19:24:03 +0000

While discussing a web application's performance or request profiling results, it is vital to distinguish between the latency and the response time.

A simple formula to articulate the difference between response time and latency: response time = latency + processing time.

Response time is the total time it takes for the web service to respond to the sent request, including all networking latencies. Response time is the sum of processing time and encountered latencies.

Processing time is usually the time taken by the server from receiving the last byte of the request and returning the first byte of the response. It does not include the time it takes the request to get from the client to the server or the time it takes to get from the server back to the client.

If we are talking about an API, the server usually does not start processing until it receives and reads all the bytes from the request. Since the server needs to parse it and understand how it can satisfy it, once it started to render the response (sent the first byte), it does not control the network latency.

It would be best if you communicated it clearly with your team. You may write a web service that starts parsing JSON on-the-flight and executing the business logic. Your processing time will have a slightly different meaning. Measure it from the first received request byte.

Latency is the duration that a request is waiting to be handled. Until the request is handled, it is latent - inactive or dormant. A high latency indicates problems in the networking or that the server cannot handle a series of requests and is probably overloaded.

You might be interested in a round-trip latency. Round-trip latency is when a request is waiting to be handled and the duration it takes for the response to get back to the client. A high level of round-trip latency usually means bad or slow networking.

To exclude networking problems, you must eliminate networking and ideally host your application locally if it is possible. This way, you can remove it from the equation and check the response time.

If the server code is under your control, you can track the processing time, and if you need to get the latency value, you can subtract the processing from the response time, and you will receive the round-trip latency value.

Another way to debug the latency is to log timestamps when the client sends a request and when the server has started to process the request. The difference between the end and the start timestamp is the latency.

You can use Chrome DevTools to debug the response time. In the Chrome DevTools, it is defined by Waiting time or Time To First Byte (TTFB). You can also emulate slow network connection by using network throttling.