DEV Community: Peter Perlepes

Row-level access for your Airtable-powered application with Clerk

Peter Perlepes — Tue, 31 Aug 2021 06:27:01 +0000

Haven't you heard about Airtable ? 🤔

Airtable is an online platform to access, manage, and collaborate on relational or spreadsheet-like information. The folks at Airtable have done an amazing job on both the user experience and the technical aspects of the product. As a no-code tool – for most use cases – it can help with your whole team’s efficiency around data management.

Airtable is a great choice as a database for any kind of resource you want to serve in an application, but lacks the granular access management capabilities that most web applications need.

Apartment Hunting Application 🏘

As an example, consider an apartment hunting application where realtors need to add and manage the most attractive apartments for each of their clients. Each apartment will be listed in a single table, and you need to make sure that clients can only access the apartments selected for them. To achieve that, we can leverage some Clerk magic to provide authenticated user access to only certain rows in your Airtable apartment hunt database.

The full code repository can be found in the clerk-airtable-apartment-hunt.

Setup for the Apartment Hunt 🛠

To kickstart the apartment hunt project, you can start by creating your Airtable account and then use the Apartment Hunting Template from the template gallery.

In the created dataset you will need to add a column that represents the email that the Apartment has been assigned to. Go ahead and create the Email column of type “Email”.

Setting the column's type to Email adds more capabilities to the field, including validation that the email address is valid.

For the sake of our example, you can go ahead and fill the Email column with the email address you will use to access your Apartment Hunting application. For me, it's peter@clerk.dev.

Create a Clerk application 🥊

If you are new to Clerk you will need to create an account on our platform, then follow the steps to create a new application.

After you create an account and a new application for this example, you can move on to the repository setup.

Show me the code

To run the full example locally, you will need to follow a few small steps. First, go ahead and clone the example application.

git clone https://github.com/clerkinc/clerk-airtable-apartment-hunt.git

Go inside your project folder and copy the .env.example file to a .env.local file.

cp .env.example .env.local

Clerk Environment Variables

You will need the Frontend API value which can be found on the dashboard on your development instance's home page. Set this value as the NEXT_PUBLIC_CLERK_FRONTEND_API.

Next you will need the Clerk API key which can also be found on your dashboard under Settings ➜ API keys. Add that as CLERK_API_KEY in your .env.local file.

Finally your .env.local file should look something like:

AIRTABLE_API_KEY=keyojbaeZ5KBe9JMR
AIRTABLE_BASE_ID=appBMXDYAGWAgvH8S
NEXT_PUBLIC_CLERK_FRONTEND_API=clerk.2ct1o.leet.lcl.dev
CLERK_API_KEY=test_avDIYjpk0SqaTGF1Wx8MdrEHZIkg2zSObU

Now you just need to install the project dependencies with yarn install inside the project folder, then yarn dev to start the application locally.

How Clerk provides authenticated access to your data 🔐

To authorize Airtable data access with Clerk, we introduce a thin and customizable access management layer over the Airtable API in our backend.*

*The Airtable Rest API does not restrict us from calling it directly from the browser, but it is not recommended since we would need expose sensitive information. For more information, please see this community forum answer.

In the Apartment Hunting application, @clerk/nextjs takes care of the frontend of user authentication. For apartment data access, we use Next.js API routes to interact with the Airtable API in a secure manner. These routes use @clerk/nextjs/api to determine the signed in user.

Only showing apartments assigned to the current user 🙋

To make sure users only have access to the properties assigned to them, we create a /api/apartments endpoint to fetch this information. The code for this endpoint can be seen below:

async function handler(
  req: WithSessionProp<NextApiRequest>,
  res: NextApiResponse
) {
  switch (req.method) {
    case "GET":

      /** 
       * Get the user email from the userId attached on the request.
       */
      const userId = req.session?.userId as string;
      const user = await ClerkInstance.users.getUser(userId);
      const primaryEmailAddress =
        user.emailAddresses.find(
          (emailAddress) => emailAddress.id === user.primaryEmailAddressId
        )?.emailAddress || "";

      /** Use the email to retrieve the assigned apartments. */
      const apartments = await getApartmentsByEmail(primaryEmailAddress);
      res.status(200).json(apartments);
      break;
    default:
      res.status(405).end();
  }
}

/** 
 * Only allow authenticated access or respond with status code 403 Forbidden.
 * Add the req.session attribute on the NextApiRequest object
 */
export default requireSession(handler);

The requireSession helper guarantees that an authenticated user is accessing the endpoint, and also populates req.session attribute on the request object coming from Next.js.

In this endpoint, we retrieve the primary email address of the authenticated user and use it to fetch only apartments assigned to this email. Here, we only check for the primary email address of the user, but since Clerk also supports multiple email addresses per account, you could adjust the logic accordingly.

Only allow assigned users to modify the apartment status 🙅

In a similar manner, we want to restrict editing the apartment status to only the assigned user. The logic for restricting that access can be seen below:

async function handler(
  req: WithSessionProp<NextApiRequest>,
  res: NextApiResponse
) {
  switch (req.method) {
    case "PUT":
      const apartment = req.body;
      const userId = req.session?.userId as string;

      /** We make sure prevent a user with different account to update the visitation status. */
      const user = await ClerkInstance.users.getUser(userId);
      const primaryEmailAddress = user.emailAddresses.find(
        (emailAddress) => emailAddress.id === user.primaryEmailAddressId
      )?.emailAddress;

      /** We check if the persisted apartment email matches the requesters. */
      const persistedApartment = await getApartmentById(apartment.id);


      /** If the emails do not match, return 401 Unauthorized */
      if (primaryEmailAddress !== persistedApartment.fields.Email) {
        res.status(401).end();
        break;
      }

      const results = await updateApartment(apartment);
      res.status(200).json(results);
      break;
    default:
      res.status(405).end();
      break;
  }
}

export default requireSession(handler);

In the same manner as the apartment fetch, we only allow authenticated access by using the requireSession middleware. We perform an extra check with the signed in user's email address to ensure they are assigned to the apartment.

Recap 🏖

This was just a simple example of how Clerk can be used to add row-level access an to application that uses Airtable as it's database. While we built this example, we were really impressed with how powerful Airtable can be at managing project data, with little to no code involved.

In the same manner, Clerk abstracts away the intricacies of authentication and user management, allowing a robust solution to be deployed with little code, and users to managed with no code through our dashboard.

If you have any feedback, are running into trouble, or just want to share what you've built - we'd love to hear from you! Reach out to us on Twitter @ClerkDev, on our community Discord server, or through any of our support channels.

Build a premium recipes app with Clerk and Firebase🔥

Peter Perlepes — Mon, 23 Aug 2021 16:34:56 +0000

Firebase is among the top Platform-as-a-Service (PaaS) providers for web and mobile applications. It packs tons of powerful and well designed features for developers to spin up a fully fledged application with minimal effort, like storage, analytics, and authentication.

Clerk integrates directly with Firebase, so developers can easily add our beautiful Sign Up, Sign In, and User Profile UIs to their Firebase application. The integration allows developers to use all the full feature set of Firebase without compromising on user management.

In this post, we are going to show you a full example integrating Clerk with Firebase, to make a "premium" recipe showcase application. The recipe data is going to be stored in Firebase Firestore and will only be available to authenticated users.

The full code for this guide is available in the clerk-firebase-starter repository, and includes instructions for how to set up Firebase and connect it to Clerk. The application demo is live at https://fir-clerk.web.app/.

If you would like to read the documentation before getting started, please refer to our Firebase integration documentation.

Setting up the Firebase project 🏕

To start off, we need a Firebase Web project. Go to the Firebase Console and create a new project:

After giving it a valid name and confirming, you will find yourself in the Firebase dashboard.

Enabling Firestore and adding recipes 🍳

From the Firebase dashboard, you can go ahead and create a new Firestore Database for our example project. The Firestore instance will serve as our database where we will store and retrieve our recipes. If you want to learn more about Firestore, you can take a look at the starter documentation.

During the database instance creation, you should choose the storage location somewhere close to your users. Also keep in mind that by selecting the production mode ruleset, by default you have disabled any reads/writes to your database from outside the platform. We are gonna change that right after!

If you are not familiar with Firebase Security Rules, they are basically a Domain-specific language to limit the access to important data in Firebase storage solutions. As we mentioned previously, since these are premium recipes, only authenticated users will be allowed to view them.

To allow authenticated users to read any database but not write, you can use the security rule shown below:

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read: if request.auth != null;
    }
  }
}

Adding recipes 👨‍🍳

To add a few recipes, go ahead and create a recipes collection with recipes of your liking, but please conform to the same attribute schema shown below:

If it's helpful, here is the TypeScript type for each recipe document:

type Recipe = {
  /** The recipe description */
  description: string;

  /** The amount of guilty you should feel */
  calories: number;

  /** Cooking time in minutes */
  cookingTimeMin: number;

  /** Publicly accessible image full URL */
  imageUrl: string;

  /** Number of ingredients needed */
  ingredientsNum: number;

  /** The title of the recipe */
  title: string;
};

After adding a few recipes, you are all set from the data side.

Enabling Firebase Authentication 🔒

Since this is a new project, you will need to enable the Authentication feature. No further action is needed, since Clerk will handle the rest.

Enabling the Firebase integration on Clerk 👇

If you are new to Clerk you will need to create an account on our platform, then follow the steps to create a new application.

After you create an account and a new application for this example, you can follow these instructions to enable the Firebase integration on Clerk for your application instance.

With that done, you are now able to authenticate Clerk users in your Firebase application.

Show me the code 👩‍💻

To run the full example locally, you will need to follow a few small steps. First, go ahead and clone our Firebase starter application.

git clone git@github.com:clerkinc/clerk-firebase-starter.git

Go inside your project folder and copy the .env.example file into a .env.local file.

cp .env.example .env.local

Take the Frontend API value which can be found on your application dashboard and add it as the NEXT_PUBLIC_CLERK_FRONTEND_API value. Your .env.local file should look something like:

NEXT_PUBLIC_CLERK_FRONTEND_API=clerk.sample.api.lcl.dev

The final configuration step is to replace the firebase.web.ts config file with one from your own Firebase project. You can find a specification for the config object in Firebase's documentation.

After you create new Firebase Web project, you will be able to find the required values under Project settings ➜ General.

Now you are ready to install the dependencies and run your project in development mode. Go to the root directory of the project and run:

yarn install

and after completion

yarn dev

Your application is now running in your local environment and you can experience the same functionality as the live demo.

Where the magic happens 💫

If you take away the application setup, the integration is seamless and works out of the box with just a few copy & paste steps across Clerk and Firebase. Here's how it works:

Let us go over the way the integration works in your web application code and what are the actions you need to authenticate a Firebase user with Clerk.

The firebase application object houses the .auth() namespace which includes methods to authenticate a user. One of those methods is signInWithCustomToken, which allows third-party providers like Clerk to pass authenticated user data to Firebase.

Where does this "custom token" come from ?

After setting up Firebase integration on Clerk, you can retrieve the necessary "custom token" by calling the getToken method on the Clerk User object.

Combined, it's just two lines of code:

const firebaseToken = await clerkUser.getToken("firebase");
await firebase.auth().signInWithCustomToken(firebaseToken);

From that point on, your user is authenticated and can complete all the actions that require privileges of an identified Firebase user.

You can see this in action in our useRecipes hook implementation. If you remove these two lines, the request will fail since the Clerk user will not also be authenticated in Firebase. (Remember, we set a Security Rule - allow read: if request.auth != null; - which restricts access to authenticated users.)

Moving forward ⚡

This end to end example showed how you can use Clerk and Firebase together for a new web project. Firebase is an exceptional development platform and we are really excited to see what you build with this integration.

If you have any feedback, and running into trouble, or just want to share what you've built - we'd love to hear from you! Reach out to us on Twitter @ClerkDev, on our community Discord server, or through any of our support channels.

How HttpOnly cookies help mitigate XSS attacks 🍪

Peter Perlepes — Fri, 20 Aug 2021 08:14:08 +0000

TLDR: HttpOnly cookies do not prevent cross-site scripting (XSS) attacks, but they do lessen the impact and prevent the need to sign out users after the XSS is patched. HttpOnly cookies are not a substitute for XSS prevention measures.

Our very first architecture decision at Clerk was to use HttpOnly cookies for session management. It has long been understood that HttpOnly cookies help mitigate cross-site scripting (XSS) attacks, and we felt it was important to include this best-practice directly in our product.

But while there's strong consensus that using HttpOnly cookies is a best practice, we've found many developers are unsure of how they help with XSS. We think this stems from the guidance, which often just says what to do rather than explaining why:

A cookie with the HttpOnly attribute is inaccessible to the JavaScript Document.cookie API; it is sent only to the server. For example, cookies that persist server-side sessions don't need to be available to JavaScript, and should have the HttpOnly attribute. This precaution helps mitigate cross-site scripting (XSS) attacks.
Mozilla MDN Web Docs

Cookies [...] SHOULD be tagged to be inaccessible via JavaScript (HttpOnly).
NIST 800-63B

The attack vector ⚔

After reading this guidance, you might be surprised to learn that HttpOnly cookies do not prevent XSS attacks.

Instead, HttpOnly cookies are helpful when you assume an XSS attack will happen and want to lessen the impact. Ultimately, they mitigate XSS attacks by making it easier for organizations to respond.

The specific threat HttpOnly cookies protect against is called session token exfiltration, which is a fancy way of saying that the attacker is able to steal a user's session token.

When a session token is stored in a cookie without the HttpOnly flag, the token can be stolen during an XSS attack with document.cookie. This is problematic because session tokens are the primary mechanism used by backends to authenticate a user.

Once an attacker has access to a session token, they can often act on behalf of a user until that token expires or is revoked. Actions can be taken remotely - even if the user is no longer visiting the page with the XSS vulnerability - which can serve to dramatically increase the surface area of the attack.

Conversely, when a session token is stored in a cookie with the HttpOnly flag, that token cannot be directly exfiltrated during an XSS attack. This minimizes the surface area of the XSS attack and makes it easier for organizations to respond.

Responding to XSS attacks - without HttpOnly cookies

When an organization is responding to an XSS attack, the first step is always patching the XSS vulnerability.

If HttpOnly cookies were not used, organizations should then assume that session tokens were exfiltrated. This means that - even with the XSS vulnerability patched - the attacker may still have the ability to act on behalf of users.

The next step is revoking the session of any user who may have been subjected to the XSS vulnerability, since those are the users who may have had their session tokens exfiltrated. These users will need to sign in again next time they visit the website.

Lastly, the organization will need to reverse any actions the attacker took on behalf of their users, from the time the vulnerability began to the time session tokens were revoked.

Responding to XSS attacks - with HttpOnly cookies 🍪

With HttpOnly cookies, organizations still need to patch the XSS vulnerability and reverse any actions taken on behalf of their users, but they do not need to revoke sessions and ask users to sign in again.

What about localStorage and sessionStorage? 🤔

Although window.localStorage and window.sessionStorage are newer client-side storage APIs, they function like cookies without the HttpOnly flag. HttpOnly cookies are still the only standard mechanism for persisting session tokens that cannot be exfiltrated during an XSS attack.

Server responses so slow the user abandoned! Trace using NEL and an example in Node.js

Peter Perlepes — Mon, 26 Oct 2020 08:55:34 +0000

How you can use another relatively new browser capability to retrieve reports on your own endpoints when user abandoned your service due to slow server response 🤦‍♂️

Network Error Logging ? 🤔

If you have never heard about Network Error Logging (NEL) before and you are working on the web, you might get really excited, as I did when I read the specification about the capabilities it provides.

We will not go into an introduction of this specification (in this article) but just a few words to whet your appetite.

If you are interested in the topic and want to follow along, take a look at the specification or introductory articles first.

NEL gives web service operators visibility into problems related to network reliability that the users of those services are experiencing.

The most exciting part of NEL in my eyes is the “users“ part. The place where the reports are stored and transmitted over to your NEL collector service is client browser.

In a more systemic language, the NEL agent is the user’s browser. That makes NEL reports, the ground truth about whether problems impact your users at any point in the network interactions of an HTTP request.

An amazing capability without requiring bespoke instrumentation, specialized infrastructure or new tools to get started.

The power to monitor “abandonment”

Except for all the TCP, DNS, TLS and HTTP specific errors being reported by NEL, for more than a year now the “abandonment” error type has been available for Chromium based browsers.

"abandonment", as not yet well described in the official documentation, covers a significant blind spot which, if I can talk in marketing terms, is as low in the funnel as possible regarding a user reaching our product.

The “abandonment” error type is generated when the user closed the page and the requirements below are met:

Valid, not erroneous(400–500) or redirect, headers are received in the server response.
The browser has not managed to fully read the response body from the server.

In plain English:

For a resource request, let’s take the main document as an example which is probably the most important, this error report covers cases where all the network infrastructure has done its job but now due to the server or the CDN serving the request being slow the user has left. 🙅‍♀️

This slowness can most of the times be attributed to:

Slow Time To First Byte (ttfb)
Time to generate and transmit the full body response. Server-side rendering, slow database queries are a couple of things that come to mind.

The Importance 🏆

This spot is of extreme value to web engineers, user behaviour analysts and web performance engineers to say a few.

Client side analytics have no power there as the whole response body is not yet fully transmitted, let alone analytics scripts being run.
Server logs depending on your infrastructure at the worst case scenario have logged a 200 status code or nothing at all*.* When a CDN is serving your pages, you usually don’t even have access to their logs.

Common ways of monitoring are leaving this range of behaviours not easily traceable but nevertheless crucial to profits.

Utilizing the NEL abandoned error type you can now become aware of this issue and act upon it depending on your application.

Example Node.js implementation 👨‍💻

To showcase how you could achieve the NEL reporting functionality and understand which failures are treated as *abandoned, w*e are gonna go about implementing an example Node.js web server using a couple of helper libraries and tools.

In this example we are going to be using:

Express.js for the web server framework.
nodemon to avoid restarting the local server all the time.
ngrok for easy access to public secure URL. (NEL does not work for insecure localhost endpoints)

A Simple Web Server

Let’s start by creating our simple web server project:

$ mkdir nel-abandoned && cd nel-abandoned
$ npm init -y
$ npm install express
$ touch app.js

Open up *app.js *with your favorite text editor and add the required setup to start an express web server:

 const express = require("express");
 const app = express();
 const PORT = 3000;

 /*
  Allow express to parse the special content type 
  of the NEL report.
 */
 app.use(express.json({ type: "application/reports+json" }));

 /* Home route just sending nothing back*/
 app.get("/", async (req, res) => {
   res.end();
 });

 /* 
  NEL collector endpoint. 
  In a real environment, the reporting endpoint would be 
  in a completely different server IP, domain and CDN.
 */
 app.post("/report", (req, res) => {
   // Log the reports received on the terminal 
   console.log(JSON.stringify(req.body));
 });

 app.listen(PORT, () => {
   console.log(`Listening on ${PORT}`);
 });

Now run your server and ngrok on different terminals.

 $ npx nodemon app.js
 // On different terminal now
 $ ngrok http 3000

To let the clients know that you want to use NEL and report back to a specific endpoint, you want to send to the client the Reporting API/NEL required headers.

To do that we will create a NelMiddleware that will send to every request the NEL headers we defined back to the client.

 const NelMiddleware = function (req, res, next) {

  res.setHeader(
     "Report-To",
     JSON.stringify({
       group: "network-errors",
       // Expire in day
       max_age: 86400,
       // Here use the secure URL you are gonna get from ngrok
       endpoints: [{ url: "NGROK_URL/report" }],
     })
   );

   res.setHeader(
     "NEL",
     JSON.stringify({
       report_to: "network-errors",
       // Cache the policy for a day
       max_age: 86400,
     })
   );

   next();
 };

 /* Use the middleware before registering the routes */
 app.use(NelMiddleware);

If everything went well, trying the home route of your application and inspecting the DevTools network panel you would be able to see the NEL headers included in the document request.

Simulating & monitoring an “abandonment” error

To help us in the analysis and simulation of our experiment, we can use the reporting dashboard of ngrok by opening http://localhost:4040 where we get a free request traffic inspector. We would be able to inspect the reports posted to our service by the NEL agent later on.

To simulate an abandonment case as we described earlier you can try simply adding an artificial delay to the home route. This would be the case of a slow Time to First Byte.

 app.get("/", async (req, res) => {
   setTimeout(() => res.end(), 10000);
 });

Open the secure ngrok URL on the home route and before the 10 second timeout runs, close the current tab. You can do this a few times to populate more reports.

In a few minutes you will be seeing either in the ngrok inspector or the console logger of the /report *endpoint, some reports coming from the browser with the error type *abandoned.

Case of not fully delivered response body

The second common case that would trigger the abandonment would be to have a part of the response body being triggered slowly and the user leaving the page before the full completion.

 /* Helper for artificial delay */
 const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

 app.get("/", async (req, res) => {
     res.setHeader("Content-Type", "text/html");
     await res.write(`
       <!DOCTYPE html>
       <html lang="en">
         <head>
           <meta charset="UTF-8">
           <title>Document</title>
         </head>
       <body>
    `);

     await delay(5000);
     res.write(`<p>Hello World</p></body>`);

     await delay(5000);
     res.write(`</html>`);
     res.end();
 });

What this function does is it delivers parts of an HTML response on the response stream but without fully completing the body after the two delay calls of 5 seconds each.

Trying out the same trick of opening the page for a couple of seconds and then closing it down would trigger more abandonment type reports.

That’s about it 🎉

Closing notes

Thanks for reaching this far into the article. At this point I want to again stress how important this capability of the NEL proposal is in my eyes. Congratulations on the team at Google that introduced initially this idea and also the folks on the Web Performance Working Group for taking care of this specification.

Just as with all things in life, there are some caveats for now that you might need to be aware of. Currently NEL is supported only in Chromium based browsers like Chrome, Edge & Opera but hopefully this is gonna increase as time passes.

Hope you found something interesting and useful with this relatively new abandonment reporting capability, and if you liked the article it would be nice to try and spread the word!

Feel free to reach out to me on any of my social media for any questions 😊

Picture by Song Kaiyue at Pexels

Google, You Might Be Wrong Now and the Testing Pavlova 🍰

Peter Perlepes — Sat, 04 Jul 2020 08:13:42 +0000

The Statement 📜

One post on the Google Testing Blog in 2015 about End-to-End testing has influenced the way we approach software testing even today.

In the article, there are points that I wholeheartedly agree and points in which I disagree. This fact does not mean nothing more than that professionals with different experiences and context can provide their unique take on things. That is how we grow as a community and individuals.

Now that this is out of the way, let's go over some article points that in my opinion we need to re-evaluate based on the new normal.

The Google Testing Blog

For people that have not come across it yet, Google Testing Blog is a hub with amazing resources for any kind of topic around software and testing. Written by industry professionals and guest experts, the content there has benefitted our craft and made people rethink how to approach software testing in more than one aspects.

The Incident 👀

"Just Say No to More End-to-End Tests"
was the title of an article posted on the blog and really dealt a powerful blow to End-to-End testing enthusiasts back then.

People from then on had an excuse to ditch honest efforts towards automation and discourage initiatives that were only aiming towards validating if the product works as a whole for the end user.

While this might not have been the original intent of the author, that is expected to happen when a so powerful title is posted on a blog that "comes from Google".

I seriously hope I am the only one that came across senior engineers or tech/team leads that stated a fact which for them seemed an undeniable truth and a correct approach:

"Why do we need to do those 'End-to-End' tests when we don't have good unit test coverage first ? " 😶

Certainly this can be the result of tons other factors, many of them reasonable, but a clear point is that we as a community of developers and testers can be considered poor in education on the topic of modern web automation testing.

For a recent data source, take a look at the open source survey done by Tricentis. We are missing education and training, that is why people cannot stand firm against absurd statements...

What Was Then, Is Not What Stands Now 👨‍🚀

I will now be walking down into memory lane for most people. Right below we can see the staple "Testing Pyramid" as can also be observed in the said article and a presentation that went along with it:

This was absolutely the truth and the way to go, but you don't see people interested in Backbone.js either now, do you?

Slow to Orchestrate 🐢

Spinning up a new environment was indeed painstaking and the capabilities of modern DevOps were a pipe dream. Now we have on-demand environments in amazing speeds, Serverless, Review Apps, Deployment Previews and tons more great tools.

Slow to Run 😴

Spawning a full browser has its difficulties as it requires specialized preparation to provide GUI capabilities. Now with tools like Puppeteer, Playwright and Cypress just to name a few, you can run End-to-End tests headless in any minimally equipped environment. Even Serverless!

Flaky 🎰

Selenium architecture was not capable at the time to easily evaluate things like element ready state or network state. That ultimately resulted in explicit wait() calls which even a millisecond delay would trigger an error for the whole test.

This also made scaling and parallelization not as effective as it could be. The tools mentioned above are closer to the browser platform and can automatically wait for the correct moment to trigger actions without hacks or explicit wait.

Hard to Maintain* 🐛

Without wanting to dive deep into this point, I will just add a statement that goes on the grapevine. At the time the people hired for the role were not as technically strong or had the training required to excel at the expectations for the position.
*This was and is true up to this day for situations that an application is in its infant stages and is constantly experiment with new experiences.

Let us jump to the present reality...🚀

Thinking Big is Required

Quoting the article:

Think Smaller, Not Larger

The thought of unit level tests providing actual confidence for a frontend application is moving away from the truth as time passes. Actual confidence is the thing that we are paid to create and that is
"Working software for the end user".

Modern frontend applications have indisputably moved to being:

Heavy in business logic and state 🏋

Your typical product these days has shifted business logic on the frontend side which is in one way or the other intermingled with routing tied in-memory browser state.To translate the above statement to layman's terms, when you want to test your e-commerce cart in React, you have to mock/stub the state manager and the router just to get things going.

At least in my eyes, this is magnitudes faster than an end-to-end test for your cart but by stubbing a shared state manager and a global router, how confident are you that this unit test proves the cart is working ?

Multi-dependent on distributed services 🤹

In one way or the other, the shift towards distributed computing, in all its shapes, is bound to permeate the ways we approach software. Same thing is starting to happen on the frontend applications.

Their functionality, their validity and their usefulness depends on external services. Either that be another team's microservice, an authentication API or Google Places Autocomplete. That is in many ways reflected on the interfaces of your application.

Think larger and more distributed. 🌐

What Happens Now for Frontend Engineers ? 🤔

The above points, when attempted to be translated into unit or integration tests, can quickly become tightly coupled to multiple mocking techniques and external APIs that are out of your control.

To not get confused here, this is no pushing against them, but comparing them to a Puppeteer End-to-End test, they are not conceptually so far away in terms of setup and mocking required. There are even techniques to provide whole environment mocks in minutes!

The suggestion here is to consider adding a couple of End-to-End tests to validate that a user flow works before throwing it out to QA. Just add Puppeteer or Playwright, throw in some lines of JavaScript that as a frontend engineer you are so familiar with, and you are good to go.

As an ecosystem, we have reached the maturity levels that this is easily possible, so please do 🙏

Furthermore something that I want to get out of my chest... Unlike backend tests, frontend tests if not run against different environments, are not guaranteed to be working, even if they give you the green heavy checkmark.

Never let someone with no frontend expertise dictate a testing strategy for the frontend. It is like getting a swimming coach to prep you for a boxing match.

The New Normal 🏄

Great individuals as Kent C. Dodds and Guillermo Rauch have been introducing unconventional thoughts about the "Testing Pyramid".

Let me now give you my take on the new normal. Enjoy!

As you can observe in what I like to call "Testing Pavlova", the layers and the ratio of expected tests are kept the same as with the "Testing pyramid", only a bit more stylish.

Adding up the vertical axis that developers tend to confuse in many discussions.

Now for the revolutionary part, something that I have not seen being illustrated yet. You are receiving a shaped container, but you are not explicitly shown how to fill it.

We will show how this Pavlova should be filled, parallelizing that with the timeline of your project.

No Top Down but Side to Side

Some Useful Gists

> Consider the product "ok" to be launched when it also contains as few End-to-End tests as to cover your main revenue flows 💸

> Code coverage should be viewed as an organic side-effect of product maturity, not an explicit target 🌲

> The diminishing return occurs when you just add Unit level tests that will never confirm a product function, but probably just if a library works 🤦

🔥A Hot Take🔥

The final points I wish to tackle in the article is one that I cannot find myself familiarizing with, no matter how many years I have been in the industry.

If the loop (test feedback loop) is fast enough, developers may even run tests before checking in a change.

For the first point, I am actually confident that we as software engineering community, have invested in the tools and culture to have scheduled and automated test runs in many parts of our coding efforts assembly line. That is in our CI environment, pre-commit hooks and other examples depending on the project. If at your project/team, the mentioned techniques are a "backlog" item, please push it up, it is essential.

and now...

Developers like it (End-to-End testing) because it offloads most, if not all, of the testing to others.

For the final point, we will be including only Frontend engineers into the mix because this shift in mindset is another part of the new normal.

Taking into account the change in how applications are architected, API mocking capabilities and technotropies like Jamstack, it is to our best interest to invest in including End-to-End testing as a "frontend-related" topic.

Frontend engineers 👩‍💻

Are responsible for adding the new UI/UX interactions to be tested
Are already working with JavaScript, so Node.js automation tools are easy to onboard
Know how the browser/network operates and its quirks
Are the ones who will fix most frontend issues in the first place

With all the great tools, services and community resources we have at our disposal it is not making much sense for frontend engineers to not take part in creating and maintaining End-to-End tests for their interfaces. Even the bare minimum.

Closing 📔

Thank your for reaching the end of the article and I seriously hope you took your notes. Not to bash me online, but to just reconsider some of your past or present discussions about web automation testing, specifically the End-to-End topic.

A Good Idea That Often Fails In Practice

That was the characterization of End-to-End testing in the said blog post and I hope we will practically and conceptually get over this, with all the amazing people and tools around our ecosystem.

Onwards 🚀

Cross-posted from The Home of Web Automation

Automation for Web Performance - All Flavours Lighthouse

Peter Perlepes — Fri, 26 Jun 2020 18:30:29 +0000

Google Lighthouse is an amazing tool that has changed the way we view web performance and now directs our efforts to improve it. Let us show how easy it is to automate the audit and collect the data in many formats!

A Glimpse Into Web Performance 🚄

During the last four to five years, web performance has grown to be a huge topic of discussion and innovation across the internet-connected industry. It would be really convenient for anyone to be able to define briefly the components that gather under the umbrella of this term, but that is not the case.

At a really high level we can list a few that are already specific towards the web:

Browser internal workings as a host platform
Data payloads
User Interface/User Experience

Just the three items that popped up pretty easily in my head, are topics that deserve tomes and tomes of information, details and technotropies. The good thing though is that we do not need to digest all the information to be able to improve them on the context of our own projects.

Lighthouse

Just a pinch of understanding would be enough, and here is where Lighthouse comes in the picture.

Lighthouse is tool that helps technologists audit, monitor and improve the overall quality of web pages, open-sourced under the Google Chrome umbrella. If you do not know it yet, it is already included in your Chrome and Edge browsers. Some details on how to run it if you haven't yet.
Lighthouse tab in Chrome, Lighthouse tab in Edge.

"Overall" can sound cheeky and soft-handed to people that are software specialists. To their surprise Lighthouse is, at least in my opinion, one of the projects that has, is and will be improving the quality of the web as we experience it in many aspects. By default it includes suggestions for:

Performance
Progressive Web App
Best practices
Accessibility
SEO

Great engineering effort from many slices of people across the tech community, has made this tool a powerhouse and indisputably the "go-to" for performance monitoring.

If I were you, I would explore the categories and capabilities right away, the ROI is huge!

After you have gotten over the stroke that you got from the awesomeness of the tool...🤯

Let me make you tremble once again by telling you that the whole report that Lighthouse has generated, together with granular performance metrics and more data, can be retrieved in an automated way using a single command or in more complicated scenarios just a few lines of JavaScript code.

In action now!

Lighthouse One-Line 🛤

If the specific page or set of pages that you wish to run Lighthouse against, are valid for all the criteria below:
- Publicly accessible
- Non differentiating between known and anonymous users
- Not requiring previous application state

Then you can just use the lighthouse-cli or go with Google Pagespeed Insights (that uses Lighthouse internally) and you do not need anything additional.

You can follow along with a simple setup, that will provide a separate place to run and store your Lighthouse reports from using the command line interface, starting with a bare Node.js project.

In your command line, let's build a new project for our task at hand:

mkdir auto-lighthouse
cd auto-lighthouse
npm init -y

Having scaffolded a project, installing the required library comes next:

npm install lighthouse

Only with the Lighthouse npm module installed, you can now run:

npx lighthouse https://dev.to

What happens now is that you will see a Google Chrome instance launching in your machine, automatically navigating to the supplied URL, doing some magics 🧙‍♂️ and finally producing a single file inside the project folder. The file generated is an HTML file with a name structured like:
{SUPPLIED_URL}{DATE}

Go and open this file in your browser of choice. There you have your amazing report to distribute in your team to reveal their incompetencies 🤣

Special Hint: If you want to save the file but also open the report automatically in your browser you can run the cli command with the --view option.

** If you wondering why your report is showing the device field as "Emulated Moto G4", you are not the only one confused. Monitor the rabbit holes here 🐰

One Step Further Tailored for You 👔

Since you made it to this step you are either interested in the shiny content we have or the situation you want to automate your Lighthouse reporting is a bit more intricate for example...

The user needs to be authenticated to see the target page
There must be some browser/application state initialized

Hmm...
We cannot pre-bake state in the browser using the lighthouse cli...
We cannot authenticate the user safely without going through a login action at least...

How would we go about those scenarios then ? 🤔

Of course we can use our favorite Node.js browser automation libraries, Puppeteer or Playwright. Using these libraries, we would be able to orchestrate the steps needed to arrive at the correct application state, and then run Lighthouse.

Now The Code 👨‍💻

*I am sure you are eager to jump into the code and this is what we are gonna do. But as we progress through it, I will try my best to explain what seemed unclear (at least to me) when trying to get this process down.

First up, you should go ahead and install the additional npm package we are gonna need to make this work. For demonstration purposes we will install Puppeteer.

npm install puppeteer

Next create a new file called index.js with the following context:

const puppeteer = require("puppeteer");
const lighthouse = require("lighthouse");

(async function(){
  const url = "https://dev.to"; /* Random */
  const PORT = 9222; /* Not arbitrary, the default Lighthouse would look for if not specified in options */

  const browser = await puppeteer.launch({
    args: [`--remote-debugging-port=${PORT}`], /* Expose this so we can use it below */
  });

  const lighthouseOpts = {
    port: PORT,
    disableStorageReset: true /* For the custom steps we will show later */
    logLevel: 'info', /* To observe the good stuff */
  };

  /* Run Lighthouse, using the options specified */
  const lighthouseResult = await lighthouse(url, lighthouseOpts);
  console.log(lighthouseResult.lhr) /* Inspect the "lhr" (lighthouse report) property in the console */

  /* Kill the browser 🔪 */
  await browser.close()
})();

Probably the first question that comes into your mind is: Why are we using this remote-debugging-port thing ?

One of Lighthouse programmatic capabilities is that it can connect to an existing browser instance by using the provided port the browser is using.

If you run the script now, it will have the exact same behavior with the CLI version but it will not result in a new HTML report in your directory, instead it will just log the lhr property on the console.

Adding The Custom Steps 🔧

As you might have suspected already, there is a place in which your additional logic can go in, in order to drive the app to the state you require.

const browser = await puppeteer.launch({/*...*/});

/*
 * Additional Web Automation logic
 * await authenticate(browser, ...args);
 * await fillEphemeralStorage(browser, ...args);
 * ...
 */

const lighthouseOpts = { /*...*/ };

An example of a function that you would use:

async function authenticate(browser, signinUrl){
  const page = await browser.newPage();
  await page.goto(signinUrl);

  const emailInput = await page.$("input[type=email]");
  await emailInput.type("admin@admin.com");
  await page.click("button[id=go]");
  await page.waitForNavigation();

  await page.close();
}

Functions like these will result in a state specific to your application needs but remember that it will probably have to do with something along the lines of:

- Browser stored credentials (cookies) 🍪
- Local browser state (IndexedDB, LocalStorage, etc) 🏦
- Ephemeral app specific conditions 👻

After the setup functions are completed, Lighthouse can be called to run at the target URL.

An Important Lighthouse Parameter 🚦

Because of the Lighthouse inner workings, the metrics to be collected and heuristics of the collection, every time an audit is requested, a new page will open but with the browser storage cleaned up, unless stated otherwise. That is understandable in order to start from a clean" slate. You can see the code for this process here.

To make sure our setup is not reset by Lighthouse, we pass the parameter

disableStorageReset: true

Now our setup is safe!

Handling The Report 📈

Currently the only processing we are doing in the report is logging it out in the console. Not so handy!

To our benefit, the Lighthouse API exposes a method in which we can generate a full report by providing the collected data and the format we want to report to be produced in.

/* ... */
const { generateReport } = require("lighthouse/lighthouse-core/report/report-generator");
const fs = require("fs");
/* ... */
const { lhr } = await lighthouse(url, lighthouseOpts);
fs.writeFileSync("./report.json", generateReport(lhr, "json") );

What we have done here is demonstrate how we can use the generateReport function to create a new report in JSON format and put it in a new file called report.json in the current directory.

*The generateReport function can be used to output JSON, CSV and HTML formats for now.

If you go and open this file, you can see the whole slew of web quality metrics. There is a really high chance that you were not aware how many things are collected and reported by Lighthouse 🎉

Schedule the script to run for all the websites you manage and you are good to go!

Closing 🎗

Thank you for going through yet another recipe. I hope that you have a bit more ammo to convince you project leads that an automated Lighthouse audit can give you many bangs over zero buck. My suggestion is to take a day off from work and just see and study all the metrics reported. Time well spent!

Spread the report and specialized insights inside your team/organization in order to drive change for the better that will benefit the end user. In the near future we will see some more stuff around web automation and performance monitoring. See you then!

Cross-posted from The Home of Web Automation

Full Guide: Introducing The Web Automation Initiative to Your Organization and Team

Peter Perlepes — Wed, 24 Jun 2020 07:51:04 +0000

This is not a post to discuss and compare the low-level capabilities of each web automation library or framework. Here we plan to lay out some foundational concerns that you should keep in mind when choosing the right tool for the job.

Enjoy!

Stepping with the "right" foot in the door for Web Automation

Proposing a Web Automation Initiative 👔

There comes the time that that you want to start your web automation initiative. That requires a quite smart deal of preparation.

First you have got to convince your team that this is worthwhile.

In the same manner you have to make the decision of which web automation strategy and which library or framework you should consider based on your application use cases.

Let us go over some foundational concerns to help guide that decision and effort.

Getting In The Mood 🤖

The first best thing is that you arrived at the point of discussing something related to web automation 👏

In most discussions around web automation, especially when planning to introduce a new initiative or re-evaluate your technology stack, the topics that can quickly drive the discussions are cross-browser compatibility, programming language API and community backing that available tools offer.

To not get confused here, by browser compatibility we mean the range of available browsers a web automation task can run seamlessly on without any change required.

Aim For Efficiency 🚀

For modern software products and businesses, efficiency is a vital factor.

That makes it all the more important for us to be pragmatic. That means evaluate solutions that

Minimize wasted effort
Do just enough to complete the task at hand safely and with the expected quality levels

For our context we need to pick the right web automation tool serving our own needs.

-How many people in the team have experience with web automation testing ?
-Do we have to run the suite in different browsers ?
-Do we need to support Internet Explorer ?

These are just some standard and common questions, but we will go over the process on how you can evaluate your requirements and tradeoffs even in more complex situations.
Let's go! 🏄

Evaluating Your Reality

The first step for exploring the best possible solution is to rule out the cases you do not even need to consider such decision.

They are probably self explanatory but compiling down a table with rough estimations to get you out of this situation pretty quickly:

Type of web automation task	What path to take
Task that aims to automate otherwise manual labour.	Pick the tool that you feel most comfortable with except if execution speed is important. If it is, probably Puppeteer or Playwright would be just fine.
Proof Of Concept for initiating an End To End testing initiative.	Go with the tool that requires less setup and needs the minimum amount of code to achieve the required use cases. You could go with Puppeteer or Playwright if you are into vanilla but Cypress would be my go-to here.
Designing an End To End testing platform in an enterprise setting.	Selenium and Selenium-based frameworks would probably fit better as portability around teams and cross-browser compatibility is an absolute must for those businesses.
Generate PDF from a remote context on a user action	Speed is of the essence and you have all the control. In that case you can look into a solution like serverless puppeteer/playwright.

These situations as coarse grained as they might seem, can at least point you to a first solid direction. Efficiency metrics and team feedback will guide you for the rest of the decisions you have to make down the line depending on the path you chose initially.

The Messy Middle of Web Automation 🌀

Since you reached this part of the article, means we can borrow this amazing phrase Scott Belsky to describe a common but certainly messy situation you need to wrestle with. The Messy Middle.

This phrase will be used to describe the state that at some point in time your web automation endeavors are bound to reach.

One of the characteristics of this state is choice overload 🤯

As browser technology is progressing rapidly, the ecosystem and available tools for web automation are growing along with it, especially in the End to End testing league. The absolutely wrong path to take is start exploring the vast ecosystem without any guidance from you actual goals, data and expectations. These main drivers as you can understand are highly dependent on the scenario that brought you to this stage from the start.

This Messy Middle most commonly can occur in one of the following scenarios:

Trying to introduce the concept of End to End testing to a team (or in a whole organization).
Re-evaluating the state of a legacy web automation codebase that bears the weight of time. Flakiness, Configuration & Setup mayhem, takes hours to run etc...

Above we list these two specific scenarios as they share the common requirements:

Low resource cost
High impact
A twist of wow factor

In both cases you absolutely need to convince and inspire your team, that is why we need to invest up front on the things that matter.

Look Further Into Requirements Not Options

As technologists, and software enthusiasts at that, we love diving in and analyzing a plethora of options that we have to solve a particular problem. It feels like a playground with thousands of toys to pick up and play.

We want to try the latest, touch the bleeding edge and feel that we are joining the innovation march, leaving expectations as an afterthought that we will tackle after we make our choices.

-Should we assemble our own minimal framework or start with something like Gauge or WebdriverIO ?
-Should we integrate tests using Cucumber or try it together with TestNG ?
-Might it be a good case to try the Screenplay pattern ?

At that time, which is bound to happen, take a step back and bring again into your mind the word "Pragmatic". We are operating under the umbrella of a business and we have so many edges that our choices will have an effect on.

From our peers and seniors but finally to the customer, we ought as professionals to make our decisions as informed as possible. Not that the technology questions are unimportant, on the contrary, since they are important they should not be taken lightly.

Gather Real User Data 👨‍👩‍👧‍👦

There is no better indicator about your real requirements than from the people that actually use your product.

For most use cases, the main difference you first need to consider is how cross-browser compatible your solution needs to be.

You are trying to get a foot in the door but if the proposed path does not cover 30% of user technology specifications, it is not the "right" foot you are getting into with. It is totally reasonable to look for the option that would allow broader amount of coverage, but you have to act smart here, you need to base "broader" on something, which is not 100% of the browser market.

This step is skipped for most developers as they become allergic to step out of the comfort zone of development tools 😥

Data analytics tools such as Google Analytics, can provide you with fine-grained information about which browser/browser version/device combination your actual users are preferring to navigate your product with.

Now you can start working your case with reality as your guide.
Special Hint: Look also for the conversions/combination metric! If Internet Explorer users are just 5% but bring 15% of the conversions, it matters a lot!

It will be clear as day that if your userbase is composed by a considerable amount of traffic from Internet Explorer, you should go for a solution that supports Selenium. On the other hand, if your application is mostly used on Safari, you should give Playwright its fair chance against Puppeteer.

Thankfully though, there are free and open source options that want to lift the burden of this choice from our shoulders as much as possible. Frameworks like WebdriverIO and CodeceptJS provide the capabilities to switch between Selenium, Puppeteer or even Playwright(only for CodeceptJS) as your automation core library.

Evaluate Team Experience & Preferences 🤝

Presenting your case in a team that already has experience with web automation is always a good thing. In the best case scenario you will be having more eyes on the target along with more technical knowledge from your peers.

On the other side of the spectrum, a team that had a bad experience with similar efforts in the past, can be seen as a source of feedback for what went wrong and ways to avoid the same mistakes.

⭐ Do not forget that this situation can be fertile ground to change perspectives for the better (that is why you should continue reading)

Bring The Excitement 🏂

Talking about perspectives, it might be by far one of the strongest factors on the success of the initiative, you should strive to excite all the more people from your team.

Try to create a group of "followers", people that will support your claim that this change will have positive impact for the team as a whole. In the same note, having your peers excited will have a high chance of them being willing to collaborate with you and contribute in the effort. This can only lead to much better results.

Keep in mind that more experienced managerial stuff, will not naturally prefer a "one-man" initiative as they will need to account for the bus factor of a new system.

On the other hand, as any solid initiative, it needs to pull its own weight, that is why you must strive to require as few slices of people as possible in order to make this a reality.

As an example, if you get the feeling that the CI pipeline is convoluted or it requires significant involvement of Operations people, try to choose a solution that makes the process as lightweight and minimal as possible.

👉 Minimizing the effort from other individuals and the changes required in other parts of the system, is sure to increase the chance the initiative is accepted and not thrown in a "low priority" bucket.

Maximum Output - Low Effort

Independently of the results of your preparation, since hopefully you did not skip any corners, it is time to get the wheels spinning. Together with your team you have made the choices, keeping in mind all the needs of the application, and in order to actually start the initiative you need to present to management the incremental steps towards your desired end goal.

Special Hint: Do not come empty handed on this aspect, expecting the question to be asked by someone else, be prepared with the correct amount of effort that can fit into the work schedule that your team is following.

"Incremental" can mean different things for different people, but we need to have this down for our initiative so that we can produce the maximum amount of output in low effort iterations. What better way to illustrate such a case than with an example.

Strap in!

Example: Monolith with Multiple Frontend Applications 🤹

The Goal

Let's suppose your case is a monolithic application (Django, Rails etc.) with multiple frontend apps for distinct parts of your product (Data dashboard, Signup flow, Landing pages).

The application does not have any End to End tests as of yet, but has a simple test pipeline that spins up a local instance of the monolith, runs the backend tests and the different tests for the frontend that happen to be run using jest for all applications.

In this scenario your end goal is to create a containerized suite of End to End tests with Puppeteer that run in every PR merge of the application and tests all the parts at once.

Some additional features of the suite that everybody agreed on include:

Screenshots on failure cases
Statistics of past and current test runs
Slack notifications when the suite fails to build a PR

First Iteration Proposal

For the first iteration, you should aim for the least amount of setup required and happy path coverage of couple of the main flows that are most valuable to your business.

Our first targets can look like:

The landing page is loaded successfully and the CTA is visible within the viewport.
The dashboard application is loaded successfully and the main 3 routes can be clicked through and render the main graphs of the user expects to see.

Common Mistake: Setup For the First Tests ❌

One of the main mistakes that individuals make when introducing a web automation initiative is to try to setup the "final" view of the pipeline integration right at the start of the initiative.

This is where many trip up and the whole initiative gets discarded when you depend on other people to kickstart it. Some examples where you will need to take time from your DevOps team...

Nobody in the team knows how to setup Jenkins for the Selenium Server to run in parallel
Your Docker setup is failing for "reasons" 🐳
You are not feeling comfortable modifying the Jenkinsfile on your own

Without going deeper here into the reasons and remediation of this, you have to go with the most unobtrusive setup as possible.

The Unobtrusive Setup 👻

In our case this means integrating the first scenarios inside the existing frontend unit testing suite. Just the plain npm test that your frontend apps are already using in one of the build hooks (hopefully 😅) is more than enough.

The application is already running the tests in the pipeline, so you can easily go with the solution of adding jest-puppeteer to your project and add your two scenarios then and there on their respective frontend repositories.

No further containerization is required
No additional commands on the build pipeline
No further effort, just your two test scenarios

First Iteration Complete 🎖

With the scenario described above that would take:
- One additional NPM package
- 100 or so lines of code
- Some seconds added on the pipeline total build time

With that we have completed the first iteration of our initiative. Every time our application goes through the simple test pipeline, we make sure that two of the main parts of our product are not broken. This is no small guarantee for the business and you manage to deliver it in a really short timespan.

Hopefully you have surprised a couple of people that did not know the power you can have with End to End tests giving a little effort.

Full steam ahead for the next iteration! 🚄

Next Iteration!

Actually, for the next iteration, you are on your own.

You have already something going on and that is enough to take some time from each iteration cycle and enhance it. Keep showing the value and advocate the confidence that it gives while shipping code into production.

Do not forget that value is expressed differently in the ears of different people.

Remind developers that they can be confident that they did not break something important while trying new features refactorings 🛠
Show the product owners that time-to-release will keep decreasing as QA has less manual work to do for each feature ⏲
Indicate to managers and stakeholders that the main revenue flows are safe while the product is becoming better and more feature-rich 💰

You got this! 🔥

Closing

Thank you for taking the time over yet another not so short writeup, which I hope you enjoyed and found useful ideas about how to tackle a web automation initiative, both in the technical and in the non-technical aspects 🙇‍♂️

From personal experience, committing and going through the process of gathering data, evaluate current requirements and analyze past experiences from your team is probably the most important decision you could have made.

Not because it will make the effort more likely to succeed, which can never be guaranteed, but for the sole reason that it will arm you with the feeling of innate confidence and the joy of solving a real problem for your business, not just a tech test-drive that might have a chance to fit the requirements.

Cross-posted from The Home of Web Automation

Resources to explore:
- Getting started with Puppeteer
- Getting started with Playwright
- Google Analytics technology report

Image by Mateusz Dach

Automate SSL Certificate renewal and issue monitoring using Puppeteer 📜

Peter Perlepes — Sat, 20 Jun 2020 18:19:46 +0000

Your SSL Backstory 🕵

To end up in this post, you probably are in some way involved with an entity that has a web presence, either that be a website, a service or a product. This entity of yours needs to be discovered by your clients/users through the World Wide Web and most of the times using a web browser directly (that is also true for mobile devices). To make this happen you took your path together with your team and designed and developed a kind of web program that aims to make the world a better place.

At some point depending on your role, you came across the SSL or HTTPS thing that hopefully people said is a must for all websites independently of purpose and content. They are right in what they said and you should believe them.

What happened after that is, either with an automatic commercial setup or a hosting service cross-sell, you purchased the magic certificate that guarantees safety, freedom from Google's restless eye and a, not so stylish now, lock badge on the visitor's address bar 🔒

What Can Go Wrong ? 🤷‍♂️

When you bought your certificate and the process has gone smoothly, your server gained the capability to encrypt between your site and your visitors. This certificate is also used to authenticate that you are capable and authorized to claim this security standard.

This certification is not printed once and valid forever though. The authority organization behind these certificates have decided an expiration date of
"2 years maximum"

As browser vendors really care about serving secure information around the web in the case that your certificate is expired, or has any kind of "issue", your web visitors will be prompted by the dreaded screen of a faulty SSL certificate.

Yes, this is the screen layout that the browser will prompt to your users 🙊

In some cases they will need to explicitly state they want to enter this web space or in some cases they are not allowed at all.

What could that translate to ?

Tremendous reduction in authority and credibility by your users 🙅‍♂️
Lost revenue & leads 💸
SEO penalties 👮‍♀️

By now you are probably are shaken enough about not forgetting to renew your SSL certificate but also to detect issues with it as fast as possible.

One thing you can do now is send a message to your webmaster to check on the certificate expiration date. The next best thing to do is to schedule a regular check about when is the required time to renew the certificate and also monitor incidents similar in nature.

Can we automate that ?
We sure can!

Here Comes Web Automation 🤖

You decided to automate this process of checking your certificate. Good for you and web automation tools have you covered. To showcase how we can do that we will be using Puppeteer as it has the utility of providing that kind information out of the box, but probably any CDP capable solution can do.

If you are not sure how to start of with Puppeteer, you can have a look at Recipe #1 and come back.

Looking through the documentation, we can see that Puppeteer's API exposes directly information about the security details of a specific network response through the SecurityDetails interface.

This interface exposes really neat information, like certificate issuer and the ending date of the certificate validity, that we will be using for our demonstration.

Let's jump in!

Show Me the Code 👨‍💻

After you have done your casual Puppeteer setup, the first thing to do is enable request interception for the newly created Page object.

await page.setRequestInterception(true);

With that, you have now gained access to some page events that will allow you to listen to the request/response cycle of every request on the pages you navigate.

// We do not need to do anything on the Request event, just let it move forward
page.on("request", (request) => request.continue());

page.on("response", (response) => {
  // Here the magic will occur
});

await page.goto("https://www.thehomeofwebautomation.com/")

The request that we need to cater for in our case is the initial DOC type request that is done to fetch the first HTML content of our page. That request carries all the information about the certificate we own.

/* Inside the response handler */
const contentType = response.headers()["content-type"]; // Get the content-type of the response

if (contentType.match("text/html")) { // Check for an HTML specific response
  /* Retrieve the security details */
  const securityDetails = response.securityDetails();

  /* Authority that issued the certificate */
  const certificateIssuer = securityDetails.issuer();

  /* SecurityDetails.validTo() returns a Unix Timestamp so we need to convert it */
  const validToDate = new Date(securityDetails.validTo() * 1000);

  /* ... */
}

Run this code and log some things on the console to get a fill for the data you can retrieve! For the next part we would aim to notify the responsible individuals on an upcoming certificate expiration date, so that they can take action accordingly.

/* Continue inside the conditional */
const diffInDays = computeDateDiffInDays(new Date(), validToDate); // Calculate the difference in days

if(diffInDays < 90){ // If the expiry is in less than 90 days
  notify(diffInDays, certificateIssuer); // Send a notification
}

/* The notify function can be something like */
function notify(daysRemaining, message, whoToContact = "some channel or some email address"){
  // sendSlackNotification(...) https://blog.nodeswat.com/simple-node-js-and-slack-webhook-integration-d87c95aa9600
  // sendMailToServiceManagement(...) https://blog.mailtrap.io/sending-emails-with-nodemailer/
}

With that, we are mostly done!

What I would advise you to do is schedule this as a job to run every day and leave it running for one or for all the domains you own and take care of.

Some Additional Checks

👉 More than one text/html responses

In many scenarios, there more than one requests that responds with HTML content (fitting the content-type conditional), but for our needs, the first one will suffice. What you can do is introduce a simple boolean flag.

/* Higher scope from the response handler */
let initialHtmlFound = false;

/* Inside the response handler */
if (contentType.match("text/html") && !initialHtmlFound) {
  // ...
  initialHtmlFound = true;
}

👉 Redirects

There is probably no web engineer that has not been bitten by a case of redirects. What you can do here is check for the response status and continue to the next one.

if (contentType.match("text/html") && !initialHtmlFound) {
 /* You should also check for possible redirects using response.status() >= 300 && response.status() < 400 */
}

👉 Different kinds of SSL errors

There are different kinds of SSL errors that might come up and they would throw an error right at the navigation step. To be on the safe side you can also take care of those.

// Add this simple check for the rest of the certificate error cases
await page.goto("https://expired.badssl.com/").catch(err => {
  if(err.message.match("net::ERR_CERT_")){
    notify(0, "Now we need to worry")
  }
  // Do whatever you want here
});

👉 SecurityDetails is `null`

The final special case that I would mention here is the possibility of the SecurityDetails being null. When does this happen ? Most often when we intercept a response that does not have any security details to expose. That can happen if you decide to navigate to a page that has no SSL certificate at all e.g. http://example.com

Now you are really ready!

Side Note 🖋

If For Some Reason you have lost the plot at some point and a charlatan convinced you that you do not need an SSL certificate for your so special case, please let the next thing you do today is go to any vendor (Let's Encrypt is free) and just set up the goddamn thing. If you do not believe us here, I will leave the experts do the talking and Mr. Troy Hunt is the real deal.

Closing

Thanks for reading this recipe and I hope your learned one or two useful things for your web automation efforts. Stay safe on the web and do not allow your users to see one of these embarrassing screens again.

Cross posted from The Home of Web Automation

Picture from user Danny Meneses at Pexels

Getting started with Puppeteer 🤖

Peter Perlepes — Sat, 20 Jun 2020 08:19:14 +0000

Puppeteer may currently be the most known headless browser automation library out there. It provides a high-level Node.js API which allows you to spin up and send commands to a Chromium or Chrome browser instance. It has proven itself to be easy to install, simple to use and performant by nature.

Some Backstory 📖

The way that Puppeteer works is that it provides a thin layer above the DevTools Protocol.

The DevTools Protocol is what gives you the power to do all the cool stuff in the actual "Inspect Element" toolbar in your browser. Actually this protocol is the same that powers up most Blink-based browsers (Chrome, Chromium etc.) providing the tools for DOM inspection, network profiling, debugging and all the other cool capabilities we have access to.
In Puppeteer you can do almost anything you can do in the actual browser without hacks included.

Puppeteer belongs under the Google Chrome umbrella and specifically is maintained by the Chrome DevTools team. That fact alone should give you some confidence about the long-term sustainability of the project. Additionally it is guaranteed to be up to date with the latest features that are shipped in the Chromium/Chrome browsers. You will not usually have to wait about a feature being ported to the library.

So let's get to it!👷

Get The Library

Initially make sure you are in a machine with Node.js >=v10.18.1 installed so we can go with the latest Puppeteer version.

Make a new project folder called puppeteer-example so we can start going through the process.

  mkdir puppeteer-example
  cd puppeteer-example

Now we can go ahead and bootstrap the required Node.js setup.

  npm init -y

With this you are ready to install your favorite libraries like left-pad or browser-redirect but you can skip it for now 😂. Back to our target:

  npm install puppeteer@4

While installing the library, you probably came across a message on your console stating Downloading Chromium xxx. That message is there to let you know that with the Puppeteer library, a specific version of Chromium for your operating system is also downloaded (inside node_modules) to be used by your installation of Puppeteer. The reason for that is every Puppeteer version is only guaranteed to work with a specific Chromium version it comes bundled with.
Special Hint: If you are a bit disk-space constrained, delete your node_modules directory from your test or unnused Puppeteer projects after you are done.

First Encounter🤞

We got through the installation and now we can start writting some code. You will probably be surprised with how much you can do with a few lines of code.

For our first task, we will try to explore the official Puppeteer website https://pptr.dev/.
Create a test file index.js with the following contents:

const puppeteer = require("puppeteer");

(async function () {
  const browser = await puppeteer.launch({ headless: false }); // We use this option to go into non-headless mode
  const page = await browser.newPage(); // Create a new page instance
  await page.goto("https://pptr.dev"); // Navigate to the pptr.dev website

  await page.waitFor(5000); // Wait for 5 seconds to see the beautiful site
  await browser.close(); // Close the browser
})();

Now by running this code using node test.js you will witness a Chromium instance launching and navigating to the pptr.dev website for 5 seconds before closing down.

I am sure that this now feels a comfortable place for web automation enthusiasts. The only component missing is the scenarios you need to run and getting the feel for the intuitive and simple API that Puppeteer advertises.

Why not take a look ?

Exploring a Simple Scenario 🕵

Skipping the pleasantries, our aim will be to explore the autocomplete search functionality that pptr.dev website has for our convenience.

Thinking Out Loud

So let us go about describing what does an actual user needs to do to get this autocomplete feature to achieve its purpose.

We expect the user to:
1. Open the page
2. Try to find the autocomplete search
3. Type his query for the API method he is looking for
4. Click the most relevant result on the list
5. Expect to see the section with the item he selected

To test out if the Puppeteer API is as intuitive as it claims to be, we can go ahead and translate this thinking to Puppeteer commands.

/* Somewhere else... */
const Homepage = {
  autocompleteSearchInput: "input[type='search']",
};
const apiSearchTerm = "metrics"; // The API method we are looking for
/* ... */

await page.goto("https://pptr.dev");
await page.waitForSelector(Homepage.autocompleteSearchInput);
await page.type(Homepage.autocompleteSearchInput, apiSearchTerm);
await page.click("search-item");

// Find the API name using XPath
const $apiMethod = await page.$x(
  "//api-method-name[text()='" + apiSearchTerm + "']"
)[0];

// Check if this method name section is actually visible on the viewport
const isApiMethodVisible = await $apiMethod.isIntersectingViewport();

assert.equal(isApiMethodVisible, true);

Well that was it! 🎉
The code above, containing also some housekeeping, in my eyes seems pretty straightforward based on the thinking process we laid out, I do not think I even need to explain what most of the commands contribute to. The API successfully translates to clear language without relying on other external abstractions.

A point that we can stand on a bit is the combination of commands that are used to detect if the API method that we were looking for is actually inside the browser viewport. People with experience in the field know that to assert this fact you would either create your own custom command (doing viewport dimension calculations) or rely on a framework command that has already been implemented for us.

The differentiating factor here is that the command we get directly from Puppeteer could be considered the most reliable, just from the fact that it is provided by the platform itself.

One or Two Things Missing 🙈

After we all agree that the API is rather intuitive and simple to use, we can go over and mention a couple of things that might seem to be "missing" in making our development experience a tad much better.

1) Filling your code with the `async` keyword

As you have definitely observed, there is this async keyword you have to sprinkle all around your code, and it feels a bit noisy for me at least. This keyword is required because of the event-driven nature of the browser APIs. The way to code around asynchronous and event-driven platforms in JavaScript is by using Promises to model your operations, and Puppeteer has done just that.

To make handling of those asynchronous operations a bit less painful, JavaScript has added some new keywords to the language syntax. These keywords are the async & await that you see on our code. Because Puppeteer's API needs to use Promises, the best way we can write our code is to use this async/await syntax for most commands.

2) No chaining available yet

Due to some design decisions and the nature of the library, as we have mentioned in the point above, there is currently no support for what we can call method chainning. With this capability our code could become so much more fluent to read and follow through. Picture something like:

await page.$("input[type='search']").click().type("metrics").submit();

I cannot vouch for but I think there are some third-party library solutions you can try. If you want to go a bit over the state and the possible external solutions, you start by taking a look at one relevant GitHub issue.

Closing

You just got through a super fast introduction on how to setup Puppeteer and code a simple scenario for an autocomplete search. From here on out you are on your own, except for all the recipes that will come on The Home of Web Automation.

My suggestion would be to start experimenting on your own use case and as a bedtime story, go over the detailed API documentation on GitHub. It is almost certain you will find a couple of surprising things you did not expect to do using the native commands.

Cross posted from The Home of Web Automation

Photo from Kevin Ku at Pexels

Introducing Playwright 🎭

Peter Perlepes — Thu, 18 Jun 2020 08:52:14 +0000

Introducing Playwright, one of the newest and most rapidly growing headless automation libraries out there. Released in January 2020 by Microsoft, Playwright is a Node.js library that advertises performant, reliable and hustle-free browser automation.

The Charming Browser Qualities 🐈

One of the main advantages that you will find on Playwright versus other similar solutions is the range of browsers it can orchestrate. It supports Chromium, Firefox and WebKit based browsers on Linux, Windows and Mac operating systems. Yes you heard that right, you can run a "Safari-like" browser on Linux or Windows; nothing new, just WebKit. An amazing advantage with the speed and reliability of protocol-driven browser libraries paired with a really broad range of browser coverage.
How does Playwright achieve that out of the box ?

As mentioned above for Chromium based browsers, in a similar way as Puppeteer does, downloads a version of Chromium/Chrome and uses the Chrome DevTools Protocol to orchestrate the browser instance. For Firefox and WebKit engines, what Playwright does is again downloading the actual browser but extends their debugging protocol capabilities to provide a unified API and features. There is no modification of the actual browsers, so that it is expected to work exactly the same in the testing and the real user's browser.
To get a fill for the "patches" you can probably take a look at the repository under the browser_patches folder.

Moving on From Puppeteer 💼

If you have used Puppeteer in the past and you were excited about its good parts, Playwright promises that with even more power. Just by taking a glimpse of the API on the Playwright official website you will quickly notice that it kinda looks the same with the Puppeteer API. You are certainly right and it is not by accident.
The truth is that the same team that built Puppeteer, has now moved on to Microsoft and continued Playwright from a Puppeteer fork 👀

Without feeling the need to get into company politics or open source dynamics, the Playwright team promises an even better and more testing-friendly API along with significant improvements that target multi-page scenario performance, cloud-native operations and other goodies. All that while keeping the migration scenario from a Puppeteer codebase, an almost "mechanical" and straightforward task.

Let's jump in then!

The Installation Step

Initially make sure you are in a machine with Node.js >=v10.15.0 installed so we can go with the current Playwright version.

Make a new project folder called playwright-example so we can start cooking 🍳

  mkdir playwright-example
  cd playwright-example

Now for the setup of our Node.js project.

  npm init -y

The pre-setup (funny that we have these stuff 😅) is ready, now for the actual setup:

  npm install playwright

Installing Playwright as you can see in your console pulls specific versions of Chromium, Firefox and WebKit. With the additional ~250mb of downloads in a special place in your machine cache, you get the browser support the library is rightfully advertising.

Small Detour 🏝

To ease up the tension that you might have been building with this thought running through your mind, we will take a small detour:

What is this WebKit thing and how can I be confident it behaves like Safari on Linux/Windows ?

First off some foundational knowledge. The 'commercial' browsers as you know them like Google Chrome, Mozilla Firefox, Apple Safari and others, are built on top of rendering/browser engines and each vendor adds on top of it a few goodies for its userbase. The most well known engines are Blink, Gecko and WebKit that are used by Chrome/Chromium/Microsoft Edge/Opera, Firefox and Safari respectively. In other words, it is the base for a browser's main functionalities.

You can run WebKit with Playwright on Windows/Linux and expect similar results with the real Safari browser, as the layouting on the page and the JavaScript execution (handled by JavaScriptCore) are mostly the same. There might be differences in more specialized fields as how rendering works, performance, audio, video and images, but probably will fit your use case.

If you want to know more or keep up with the latest news on Playwright, go ahead and follow Arjun Attam, you won't be dissappointed.

The Launchpad 🚀

Let's create our launchpad with the bare essential commands to get up and running with Playwright.

Touch an index.js file with the following contents:

const playwright = require("playwright");

(async function(){
  const browser = await playwright.webkit.launch({ headless: false }); // Non-headless mode to feel comfy
  const context = await browser.newContext(); // So much to say, but another time
  const page = await context.newPage(); // Create a new Page instance which handles most of your needs

  await page.goto("https://playwright.dev"); // Navigate to the Playwright webpage
  await page.waitForTimeout(5000); // Rest your eyes for five seconds
  await browser.close(); // Close the browser
})();

Starting with Something Simple

To get our feet wet, we aim to test the autocomplete search functionality on the official Playwright website. The search component is there for users to search through the topics, documentation and API methods or Playwright. Our goal is to simulate the scenario where a user browses the page and searches for a specific method using this component.

Well structured, dynamically updated component inside a Single Page App type website, it seems like a really good deal for a test drive. Our first objective is to build out the steps that a user needs to take to reach the goal of finding the API method she is looking for.

User Expectations 🥂

1. Open the page
2. Try to find the autocomplete search
3. Type his query for the API method he is looking for
4. Click the most relevant result on the list
5. Expect to see the section with the item he selected

Let's now see how the steps, that we expect the user to take, can be translated to Playwright commands.

/* Somewhere else... */
const Homepage = {
  autocompleteSearchInput: "search-view input",
};
const apiSearchTerm = "context"; // The API method we are looking for
/* ... */

await page.goto("https://playwright.dev", { waitUntil: "networkidle" });
await page.type(Homepage.autocompleteSearchInput, apiSearchTerm);

// Check for 'methods' that have the specific search term 
await page.click(`//search-suggestions/a[contains(@href, 'api.md')]//mark[.='${apiSearchTerm}']`);

// Find the method name title using XPath
const $apiMethod = await page.$("xpath=//header-with-link//h4[contains(.,'context')]");

// Check if this method name section is actually visible on the viewport
const isApiMethodVisible = await $apiMethod.boundingBox();
assert.notEqual(isApiMethodVisible, null);

As you can see right above, the API that is expressing the user interactions down into code is at least in my view pretty intuitive. Similar to Puppeteer, you can expect that most actions the user can take are translated into direct Page instance methods (type, click, dblclick, etc...).

The differentiating factor here is that the command we get directly from Playwright could be considered the most reliable, just from the fact that it is provided by the platform itself.

One or Two Things Missing 🙈

1) Filling your code with the `async` keyword

To make handling of those asynchronous operations a bit less painful, JavaScript has added some new keywords to the language syntax. These keywords are the async & await that you see on our code. Because Playwright's API needs to use Promises, the best way we can write our code is to use this async/await syntax for most commands.

2) No chaining available yet

await page.$("search-view input").click().type("context").submit();

But at some point, we might get there!

Closing 🧘‍♂️

So this was a glimpse to get you started with your first Playwright script to assert an actual user scenario. Loads of things to mention for each command and the capabilities but we are gonna see them closer in the recipes that will come on The Home of Web Automation.

Especially the Playwright's BrowserContext as advertised is an abstraction that can unlock much more power and performance using parallelization locally or from even the cloud. Pretty excited to try that!

Playwright might seem new to the scene but on the contrary it has some long history as we mentioned earlier. If you want to pit it against another tool or introduce it as 'X killer', sorry but we are not doing that here. The least I can say is that if your application has a considerable slice of WebKit-based browser users, then give Playwright a try, your users will thank you for that 💪

As with every tool though, start with anything that catches your attention, feels comfortable and addresses your actual needs in a simpler way.

Cross posted from The Home of Web Automation

Image by Devanath from Pixabay

How Selenium Works - High Level Architecture

Peter Perlepes — Tue, 16 Jun 2020 06:44:05 +0000

Selenium WebDriver undoubtedly remains the king/queen of web automation and testing. Tooling, frameworks, community, job postings, millions of people interacting with it and the rest of the ecosystem is resting on the shoulders of a project that started back at 2004. That proves that powered by the effort of all those people maintaining it and contributing to it, it is that good on what aims to do.

Even with how lean the introduction is, independently if you are working currently with Selenium or not, it stands to reason that taking a peek on what the architecture of this tool formed out to be will only be to your benefit. Enjoy the view.

The High Level

First off, most people that are using Selenium directly, by directly meaning a client library, are writting scripts, using their favorite language, that describe specific steps and interactions they want to simulate on a real browser instance. That could be from navigating to pages, clicking elements to submitting forms and scrapping content without permission.

WebDriver driver = new ChromeDriver();                  

driver.get("https://www.selenium.dev/projects/"); // Open this URL in the browser instance      

WebElement searchElement = driver.findElement(By.name("search")); // Find the hipster search of this website            
searchElement.sendKeys("firefox"); // Fill the input with the search term

This above is just a simple example of code in Java that QA engineers are used to writting for their day to day responsibilities. You run this script (with the rest of the boilerplate) and you get a Chrome window on the Selenium website, with the "firefox" term on the search field, straightforward. The exact same API we used is exposed in different language bindings like JavaScript, Python and Ruby.

For some individuals though, the process and the parts that these commands have to go through to arrive at the browser and how they operate together is kind of a blurry picture.

We can start clearing the blur with the illustration below:

Some of the components might seem familiar, others not so much. We are gonna break them down one by one so as to connect the dots and fill any potential bumps in your understanding of Selenium WebDriver.

The Components

Client Libraries

Client libraries or otherwise known as language bindings, are libraries that allow developers to use the Selenium high level API in their language of choice. Without knowing all the details, usually what "client libraries" provide, is wrappers over some transport mechanism to communicate with a fixed set of endpoints. Think of them as just a thin layer that most of the times translates _HTTP handling from one language to another._

This seems to be the case at least with the JavaScript wrapper of WebDriver.

JsonWire Protocol

As mentioned in (1) client libraries use a specific HTTP API, kinda rest-ish as David Burns say, to communicate their desired commands to the browser driver that we will speak about in a moment.

In order to prevent chaos, pains and expected deviances from client library authors and maintainers, a system needs to define a standarized way to communicate with the outside world. The decision for WebDriver, to communicate as uniformly as possible with browser drivers, was the implementation of the JsonWire protocol and as you can imagine by now, it is JSON over HTTP. The specification for this API endpoints can be found here.

To see how straightforward it can be, here is a small example:

/* 
 * Send the command to the session with id=sessionId 
 * to initiate a navigation to the "url" body parameter 
 */

POST /session/:sessionId/url 
{
  "url": "https://example.com"}
}

Going to the JsonWire specification page from GitHub, you must have noticed the OBSOLETE warning, but do not falter. It does not mean that it is an abandoned project or that the information there is irrelevant, on the contrary it has taken a big step forward. As widely used as this protocol came to be, with all the huge ecosystem of tech and people behind it, managed to become an actual Working Draft of the W3C standard.

That fact makes it an official protocol that user agents aim to implement and conform, in order for programs to remotely instruct the behaviour of web browsers. A huge success and recognition for the WebDriver project in my opinion.

Browser Driver

The browser driver can be considered the "backend" of the WebDriver architecture. It acts as the end host of the JsonWire protocol accepting commands in the form we shown earlier and responding in kind with values, errors, stats or anything that the protocol defines. On one side it actually works exactly like a regular standalone HTTP server for the client libraries. On the other end, it is using its own HTTP API to communicate, send commands and receive responses from the actual browser instance as it instruments its behaviour.

As you might expect or have noticed while trying to use Selenium, there are specific drivers for different browsers that implement the meaty part of actually sending the commands that retrieved from the client libraries to the browser instance. For example there is the GeckoDriver that supports Firefox, the ChromeDriver that supports Chrome/Chromium, the OperaDriver for Opera and more.

Each of these browser drivers uses its own way and implementation to communicate with the actual browser instance. Some examples:

ChromeDriver uses the DevTools Protocol.
The GeckoDriver uses the Marionette remote protocol.
The OperaDriver (for Opera>=26) is based on ChromeDriver with some adaptations.

As you can understand some of the drivers may not be open sourced by the maintaining companies but I am confident you can find more information if you look through the pages and documentation for each one.

Browser

The actual browser instance that will receive the commands for its respective browser driver, and finally simulate the web automation task we planned all along.

Closing

That was all about the close look we attempted at the high level architectural components that make up the 'Selenium Architecture' as I see it. Hope you enjoyed it as much as I did while I was looking into how all the actors come together and how much work and dilligence is put in the development and maintenance of each project.

Cross posted from The Home of Web Automation

Cover image by Peter H from Pixabay

The Home of Web Automation - A different take on a software blog

Peter Perlepes — Mon, 15 Jun 2020 10:43:23 +0000

About a month ago, I decided to take my passion and knowledge on web automation and testing and turn them into a blog. That came to be The Home of Web Automation

The Actual Need

In the last few years the scene of Web Automation has received a surge of attention from the software industry, and that is not without reason.

Reliability is a must for both new-found companies and large enterprises alike.
Applications are more interactive and a fair amount of times frontend heavy, meaning that they are responsible for orchestrating lots of different actors.
Manual repetitive tasks and testing is still proving to be a real pain when trying to move fast.

All of the above (and tons more) appear to be fair reasons to invest in the skills, technologies and methodologies that revolve around web automation.

Additionally there seems to be a considerable gap in the technical skills that individuals require to succeed in these efforts, I aim to help as much as possible with that. For people that want more information on the topic, there is a 2020 survey around open-source web automation testing tools that is quite revealing.

Being in the shoes that most people reading this post have been, I can sympathize with the fact that this can be just another topic to learn. Understanding the information fatigue that the software industry has been bearing on its denizens, I will try my best to make the experience as joyful as possible.

The Different Take

From my experiences in the world of software and how rapidly it is progressing, all the frameworks, the libraries, the techniques, it can make one feel really anxious and fatigued about being required to consume and understand all of these intricacies. More articles to read, more podcasts to listen to, more conference talks that you would put in your "todo" list.

Is it important ? Heck yes it is. Your career, financials and dreams might depend on it. But in this race, can there be a relaxing spot where you can keep getting the relevant technical stuff ?

Why not give it a try? That was what I felt like at least while building this website.

During the process of putting together the visual aspect, pallete, fonts, image sources and the like, I decided to take a different route than what would be expected on a software related site.

A warm kind of pallete

Playful, almost meaningless drawings

I cannot stress enough how much I love Excalidraw💙 and how extensively I will be using it to create some meaningless drawings (But also for the diagrams required) for each post. As it sparks some joy for me, I hope it does the same to you!

Weird cursive fonts

Using Caveat from Google Fonts seemed to fit my target amazingly well.

A try on software jokes

If at some point I prompt you to install left-pad or throw a joke at some website's UX decision, please understand it is benevolent 😁

The interface is composed of elements that their purpose is to make you more energetic and spark the feeling of happiness when you open a page, even while not being the most readable. Actually this became one of the top goals to be reflected on the user interface.

Closing

If you are into web automation, testing and all the ecosystem around those, you can follow for new content @HomeOfWebAuto.
I would be writting about how I built the blog in a later post

Any feedback is appreciated, if you have any questions/request reach out to me and I hope you enjoy it!