DEV Community: Michael Wanyoike

Open Source Node.js Ecommerce Platform for Remix

Michael Wanyoike — Thu, 07 Apr 2022 11:25:32 +0000

Introduction

In this tutorial, you'll learn how to build a front-end UI for Medusa using Remix.

Medusa is an open source ecommerce platform in Node.js that provides you with many ecommerce features such as order management, RMA flows, customer management, and more. Medusa also focuses on providing a good developer experience allowing you to get started within minutes and with a strong support community and documentation to back it up.

Remix is a full-stack web framework that allows you to build slick user interfaces with a resilient user experience. It renders pages on the server, unlike most React frameworks.

For this tutorial, we'll only focus on the basics which include:

Setting up the storefront layout
Listing products
Displaying a single product page with options

Below is a snapshot of what we'll be building:

You can find the complete project in this GitHub repository.

Prerequisites

This article is for intermediate to advanced React developers. You need to be familiar with the following:

Why Remix

Remix is a newer React framework that has been quickly gaining popularity in the last couple of years. It's been built by the authors of the popular React Router library.

For ecommerce, server-based frameworks are recommended in order to provide better search engine optimization capabilities, enhanced API security and faster dynamic pages for end users. Remix has a lot of key advantages including:

It renders dynamic content really fast as content processing and third-party API calls are done on the server and not on the client
It performs really well on slow networks such as 2G and 3G
Remix websites work even if JavaScript is disabled in the browser
Build times and performance are not affected by the size of the data

Why Medusa

Medusa’s headless architecture makes it easier to create a storefront with the language or framework of choice. Whether you choose Remix, Gatsby, Next.js, or any other framework, you can utilize Medusa’s APIs to create a storefront that has all main ecommerce capabilities.

You can also connect your Medusa server to the Medusa admin take full power over your ecommerce store. The Medusa admin allows store operators to manage products, orders, customers, discounts, and much more.

Medusa Server Setup

The first step in this tutorial is to set up a Medusa server where data storage and backend processing are handled. First, install the software locally as follows:

# Install Medusa CLI
npm install -g @medusajs/medusa-cli

# Create a new Medusa project
medusa new my-medusa-store --seed

The --seed option adds dummy products to your store as well as some other configuration.

Set up Medusa Admin

As mentioned before, Medusa provides a strong admin interface that you can use to manage your store, products, orders, and much more! The admin is easy to install and use, however it is completely optional. So, if you’re not interested in an admin you can skip to the next section.

In a separate directory, run the following command to install the admin:

git clone https://github.com/medusajs/admin medusa-admin

This will create a new directory medusa-admin. Change to that directory and install the dependencies:

npm install

Now, run the Medusa server from the my-medusa-store directory:

npm start

Then, run your Medusa admin from the medusa-admin directory:

npm run develop

If you open localhost:7000 in your browser, you’ll see a login screen. The --seed option you used earlier when you created your Medusa store adds an admin user with the email “admin@medusa-test.com” and password “supersecret”.

Then, from the sidebar, choose Products. You’ll see the list of products available in your store and you can add a new product using the New Product button at the top right.

If you click New Product or you click on an existing product to edit it, you’ll be able to enter a lot of information about the product. You’ll also be able to add variants, upload images, and much more.

Remix + Tailwind CSS Setup

In this section, you’ll quickly create a Remix project and set up Tailwind CSS for rapid UI styling. The UI won’t be fully responsive for the sake of the simplicity of the tutorial.

You’ll also use JavaScript to write the code, however, I highly recommend using TypeScript and a Test-Driven Development framework for real-world production frontends.

We can quickly create our Remix project as follows:

npx create-remix@latest remix-medusa-storefront

? What type of app do you want to create? Just the basics
? Where do you want to deploy? Remix App Server
? TypeScript or JavaScript? JavaScript
? Do you want me to run `npm install`? (Y/n) Y

After installation, navigate to the project folder via terminal and confirm that everything is working by executing the command npm run dev. Check to see that localhost:3000 is loading properly. If everything loads fine, kill the dev server before proceeding to the next step.

Next, with the help of the official Tailwind CSS Remix integration guide, set up Tailwind CSS in your remix-medusa-storefront project as follows:

Step 1: Install package dependencies

# Install Dev packages
npm install -D tailwindcss postcss autoprefixer concurrently

# Generate `tailwind.config.js` file
npx tailwindcss init -p

Step 2: Update the content field in tailwind.config.js to configure the files used for Tailwind CSS purging process.

module.exports = {
  content: ["./app/**/*.{js,jsx}"],
  theme: {
    extend: {},
  },
  plugins: [],
};

Step 3: Change the dev and build scripts in package.json to add the CSS compilation steps:

"scripts": {
        ...,
    "build": "npm run build:css && remix build",
    "build:css": "tailwindcss -m -i ./styles/app.css -o app/styles/app.css",
    "dev": "concurrently \"npm run dev:css\" \"remix dev\"",
    "dev:css": "tailwindcss -w -i ./styles/app.css -o app/styles/app.css"
  },

Step 4: Create the file ./styles/app.css at the root of the project with the following content:

@tailwind base;
@tailwind components;
@tailwind utilities;

Step 5: Add this code to app/root.jsx to allow loading of Tailwind's compiled CSS onto the pages:

import styles from "~/styles/app.css";

export function links() {
  return [{ rel: "stylesheet", href: styles }];
}

Step 6: Test your Tailwind CSS setup by replacing the code in app/routes/index.jsx with the following code:

export default function Index() {
  return (
   <div className="container mx-auto mt-8">
    <h1 className="text-3xl font-bold text-gray-700 underline">
      Hello world!
    </h1>
   </div>
   );
 }

Execute npm run dev and confirm that Tailwind CSS styles are being loaded at the index page on localhost:3000/.

Do note that when you launch your project, whether in dev or build mode, the file /app/styles/app.css gets generated for you based on the source input of ./styles/app.css. Hence, you shouldn't touch the generated file when making CSS customizations.

Adding /app/styles/app.css to .gitignore is a good idea since the file will be generated during the deployment stage.

Site Layout

Now that you have successfully integrated Tailwind CSS in a working Remix project, you can proceed with setting up a basic layout for your Storefront. Create the app/layouts folder and create the following files inside this folder:

footer.jsx
navbar.jsx
index.jsx

In app/layouts/footer.jsx add the following code:

export default function Footer() {
  const currentYear = new Date().getFullYear();

  return (
   <div className="py-4 text-sm text-center text-gray-200 bg-gray-800">
    &copy; Copyright {currentYear} [Brand name]. All Rights Reserved
   </div>
   );
 }

This snippet simply displays copyright information in the current year.

For the Navbar, you need to display:

Logo
Navigation links
Shopping cart icon

For the logo, you can include your own logo or you can copy this placeholder logo for now. Rename the filename to logo.svg and place it in the /public directory.

For navigation links, you’ll use [<NavLink>](https://remix.run/docs/en/v1/api/remix#navlink) which is a special kind <Link> that knows whether or not the page is the currently loaded page. You’ll need to provide styling for .active CSS class to provide a visual indication.

For the shopping cart icon, you’ll simply import from the React Icons package. Install as follows:

npm install react-icons

With the required resources now set up, you can now proceed with inserting the following code in app/layouts/navbar.jsx

import { Link, NavLink } from "@remix-run/react";
import { BiShoppingBag } from "react-icons/bi";

export default function Navbar() {
 const links = [
   {
       label: "Home",
       url: "/",
   },
   {
       label: "Products",
       url: "/products",
   },
   {
       label: "About",
       url: "/about",
   },
  ];

 return (
  <nav className="flex items-center justify-between px-8 pt-2">
    {/* Site Logo */}
   <div className="font-mono text-3xl font-extrabold uppercase">
    <Link to="/">
     <img className="w-28" src="/logo.svg" alt="Medusa" />
    </Link>
   </div>

    {/* Navigation Links */}
   <div className="space-x-4">
     {links.map((link, index) => (
     <NavLink key={index} to={link.url} className="navlink">
       {link.label}
     </NavLink>
     ))}
   </div>

    {/* Shopping Cart Indicator/Checkout Link */}
   <div className="font-semibold text-gray-600 hover:text-emerald-500">
    <NavLink
     to="/checkout"
     className="inline-flex items-center space-x-1 transition-colors duration-300"
    >
     <BiShoppingBag className="text-xl" /> <span>0</span>
    </NavLink>
   </div>
  </nav>
  );
}

Next, insert the following code in app/layouts/index.jsx which will be your main site layout:

import Footer from "./footer";
import Navbar from "./navbar";

export default function Layout({ children }) {
 return (
  <>
   <header className="border-b">
    <Navbar />
   </header>
   <main className="container flex justify-center flex-grow mx-auto">
     {children}
   </main>
   <Footer />
  </>
  );
}

Add this code to ./styles/app.css after Tailwind’s base styles to include your custom layout and navigation styles:

/*
Layout styling
*/
html {
  @apply antialiased font-sans text-gray-800 bg-gray-200;
 }

 body {
  @apply flex flex-col min-h-screen overflow-x-hidden;
 }

 /*
 Typography styling
 */

 h1 {
  @apply text-3xl font-bold;
 }

 h2 {
  @apply text-xl;
 }

 p {
  @apply text-gray-700;
 }

 /*
 Navigation menu styling
 */

 .navlink {
  @apply inline-block w-20 py-2 font-semibold text-center text-gray-500 hover:text-emerald-500;
 }

 .navlink:after {
  @apply block pb-2 border-b-2 border-emerald-400 transition ease-in-out duration-300 origin-[0%_50%] content-[""] scale-x-0;
 }

 .navlink:hover:after {
  @apply scale-x-100;
 }

 a.active {
  @apply font-bold text-gray-700;
 }

Finally, replace the entire code in app/root.jsx which includes your new site layout:

import {
  Links,
  LiveReload,
  Meta,
  Outlet,
  Scripts,
  ScrollRestoration,
} from "@remix-run/react";

import Layout from "./layouts";
import styles from "~/styles/app.css";

export function links() {
  return [{ rel: "stylesheet", href: styles }];
 }

 export function meta() {
  return {
   charset: "utf-8",
   title: "Medusa Remix StoreFront",
   viewport: "width=device-width,initial-scale=1",
   };
 }

 export default function App() {
  return (
   <Document>
    <Layout>
     <Outlet />
     <ScrollRestoration />
     <Scripts />
     <LiveReload />
    </Layout>
   </Document>
   );
 }

 function Document({ children }) {
  return (
   <html lang="en">
    <head>
     <Meta />
     <Links />
    </head>
    <body>{children}</body>
   </html>
   );
 }

 export function ErrorBoundary({ error }) {
  return (
   <Document>
    <Layout>
     <div className="text-red-500">
      <h1>Error</h1>
      <p>{error.message}</p>
     </div>
    </Layout>
   </Document>
   );
 }

You can now start the dev server again by running the command npm run dev. Your localhost:3000 index page should now appear similar to the screenshot below:

Page Routes

You’ll now add the products, about, and checkout pages. Create the following files in app/routes folder:

products/index.jsx
about.jsx
checkout.jsx

You won't implement any logic for this section. You’ll simply place some placeholder code starting with app/routes/products/index.jsx:

export default function ProductsIndexRoute() {
  return (
   <div className="w-full mt-8">
    <h1>Products Page</h1>
    <p>List of products</p>
   </div>
   );
 }

Copy the following final code for app/routes/about.jsx:

export default function AboutRoute() {
  return (
   <div className="w-full mt-8">
    <h1>About</h1>
    <p className="mt-4 text-justify">
      Lorem ipsum dolor sit amet, consectetur adipisicing elit. Labore aperiam
      maxime assumenda dolore excepturi ipsam accusantium repudiandae ducimus
      eum, voluptatibus, adipisci nam temporibus vel ex! Non iure dolore at
      mollitia.
    </p>
   </div>
   );
 }

Copy the following placeholder code for app/routes/checkout.jsx:

export default function CheckoutRoute() {
  return (
   <div className="w-full mt-8">
    <h1>Checkout Page</h1>
   </div>
   );
 }

Finalize the home page by implementing a simple Hero banner inspired by TailwindUI. Replace the entire code in app/routes/index.jsx with the following:

import { Link } from "@remix-run/react";

export default function IndexRoute() {
 return (
  <div>
    {/* Hero Banner */}
   <div className="px-12 py-32 text-center text-gray-200 bg-gray-800">
    <h1 className="text-5xl text-gray-100">New arrivals are here</h1>
    <p className="px-8 mt-2 font-semibold text-gray-300">
      The new arrivals have, well, newly arrived. Check out the latest
      options from our summer small-batch release while they're still in
      stock.
    </p>
    <Link
     to="/products"
     className="inline-block px-6 py-2 mt-8 text-sm font-semibold text-gray-700 transition duration-300 bg-gray-100 rounded-md hover:bg-white hover:text-gray-900 hover:scale-110 color"
    >
      Shop New Arrivals
    </Link>
   </div>
  </div>
  );
}

Your home page should appear similar to the screenshot below:

Navigate and check all pages to ensure that the placeholder code is working properly. In the next section, you’ll start implementing the logic for the /products route.

Products Page

In this section, you’ll implement the Products page by pulling data from the Medusa server and in a CSS grid.

First, ensure your Medusa Store server is running at localhost:9000. If it's not, you can navigate to Medusa's project folder in the terminal and execute the command npm start. Once it's running, you can proceed to the next step.

Coming back to your remix-medusa-storefront project, install the Medusa JS Client package to provide you with easy access to the Medusa API:

npm install @medusajs/medusa-js

Next, you need to create a utility that will help you create and access an instance of the medusa-js client. Create the file app/utils/client.js with the following code:

import Medusa from "@medusajs/medusa-js";

const BACKEND_URL = process.env.PUBLIC_MEDUSA_URL || "http://localhost:9000";

export const createClient = () => new Medusa({ baseUrl: BACKEND_URL });

Next, open apps/routes/products/index.js and replace with the following:

import { json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import { createClient } from "~/utils/client";

export const loader = async () => {
 const client = createClient();
 const { products } = await client.products.list();
 return json(products);
};

export default function ProductsIndexRoute() {
 const products = useLoaderData();

 return (
  <div className="w-full mt-8">
   <h1>Latest Arrivals</h1>
   <ul>
     {products.map((product) => (
     <li key={product.id}>{product.title}</li>
     ))}
   </ul>
  </div>
  );
}

In the above code, you are using Remix's data loading feature to query data from the Medusa server. This data is passed on to the rendering function via the useLoaderData hook. Check out Medusa's Product API and examine what the JSON structure looks like. You should expect the following output on the /products page:

Now that you have the data, you can start building the UI using CSS grids and Product Cards. But first, we need to create a little helper that will display the product's price.

In Medusa, a product contains multiple variants, and each variant has different prices for multiple currencies.

The data you seeded earlier when you created the Medusa server contains pricing for USD and EUR for each product variant. Since this is an introductory tutorial that aims to be simple, you’ll not be able to fully implement all the necessary logic for a production app which includes:

Auto-detecting users’ region and currency using geolocation
Giving users the options for choosing region and currency
Displaying product prices based on the selected region/currency

Create the file app/utils/prices.js and copy the following simplified code:

// TODO: Detect user language
const locale = "en-US";

// TODO: Detect user currency/Allow currency selection (usd | eur)
const regionCurrency = "usd";

export function formatPrice(variant) {
  const price = variant.prices.find(
    (price) => price.currency_code == regionCurrency
  );
  return new Intl.NumberFormat(locale, {
    style: "currency",
    currency: regionCurrency,
  }).format(price.amount / 100);
}

The code above uses hard-coded constants in place of configurable variables. The function formatPrice accepts a Product Variant as input and returns the price as a formatted String currency.

Next, you’ll need to create the component ProductCard which will display:

Thumbnail
Title
Price (for the 1st variant)

Create the file app/components/product-card.jsx and copy the following code:

import { Link } from "@remix-run/react";
import { formatPrice } from "~/utils/prices";

export default function ProductCard({ product }) {
 const variant = product.variants[0];

 return (
  <section className="overflow-hidden bg-white rounded-lg shadow:md hover:shadow-lg w-80">
   <Link to={`/products/${product.id}`}>
    <img className="w-80" src={product.thumbnail} alt={product.title} />
    <div className="p-4">
     <h3 className="text-lg font-bold text-gray-700 hover:underline">
       {product.title}
     </h3>
     <p className="font-semibold text-teal-600">{formatPrice(variant)}</p>
    </div>
   </Link>
  </section>
  );
}

Finally update the code in apps/routes/products/index.js as follows:

import { json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import ProductCard from "~/components/product-card";
import { createClient } from "~/utils/client";

export const loader = async () => {
 const client = createClient();
 const { products } = await client.products.list();
 return json(products);
};

export default function ProductsIndexRoute() {
 const products = useLoaderData();

 return (
  <div className="w-full p-4 my-8">
   <h1 className="text-center">Latest Arrivals</h1>
   <div className="grid grid-cols-1 gap-6 px-4 mt-8 md:px-12 lg:px-6 xl:px-4 xl:gap-6 2xl:px-24 2xl:gap-6 justify-items-center md:grid-cols-2 lg:grid-cols-3 2xl:grid-cols-4">
     {products.map((product) => (
     <ProductCard key={product.id} product={product} />
     ))}
   </div>
  </div>
  );
}

Implementing these updates should yield the following result:

Single Product Page

To create a single product page, you need to use the slug file naming convention. Create the file apps/routes/product/$productId.jsx with the following content:

import { useState } from "react";
import { json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";
import { BiShoppingBag } from "react-icons/bi";

import { createClient } from "~/utils/client";
import { formatPrice } from "~/utils/prices";

export const loader = async ({ params }) => {
  const client = createClient();
  const { product } = await client.products.retrieve(params.productId);
  return json(product);
};

export default function ProductRoute() {
  const product = useLoaderData();
  const [variant, setVariant] = useState(product.variants[0]);
  const [image, setImage] = useState(product.images[0]);
  const [quantity, setQuantity] = useState(1);

  const handleVariantChange = (index) => {
    setVariant(product.variants[index]);
    setQuantity(1);
  };

  const handleQuantityChange = (action) => {
    switch (action) {
      case "inc":
        if (quantity < variant.inventory_quantity) 
          setQuantity(quantity + 1);
        break;

      case "dec":
        if (quantity > 1) setQuantity(quantity - 1);
        break;

      default:
        break;
    }
  };

  const handleImageChange = (id) => {
    setImage(product.images.find((img) => img.id === id));
  };

  return (
    <div className="w-full">
      <div className="grid items-center md:grid-cols-2">
        <div>
          <img
            className="w-full rounded-lg"
            src={image.url}
            alt={product.title}
          />
          <div className="flex justify-center p-4 space-x-2">
            {product.images.map((imageItem) => (
              <img
                className={`w-16 border-2 rounded-lg ${
                  imageItem.id === image.id ? "border-teal-400" :      null
                }`}
                key={imageItem.id}
                src={imageItem.url}
                alt={product.title}
                onClick={() => handleImageChange(imageItem.id)}
              />
            ))}
          </div>
        </div>
        <div className="flex flex-col px-16 py-4 space-y-8">
          <h1>{product.title} </h1>
          <p className="font-semibold text-teal-600">{formatPrice(variant)}</p>
          <div>
            <p className="font-semibold">Select Size</p>
            <div className="grid grid-cols-3 gap-2 mt-2 md:grid-cols-2 xl:grid-cols-4">
              {product.variants.map((variantItem, index) => (
                <button
                  key={variantItem.id}
                  className={`px-2 py-1 mr-2 text-sm hover:brightness-90 ${
                    variantItem.id === variant.id
                      ? "bg-gray-700 text-gray-100"
                      : "bg-gray-300 text-gray-700"
                  }`}
                  onClick={() => handleVariantChange(index)}
                >
                  {variantItem.title}
                </button>
              ))}
            </div>
          </div>
          <div>
            <p className="font-semibold">Select Quantity</p>
            <div className="flex items-center px-4 mt-2 space-x-4">
              <button
                className="px-4 py-2 hover:shadow-sm hover:text-teal-500 hover:font-bold"
                onClick={() => handleQuantityChange("dec")}
              >
                -
              </button>
              <span>{quantity}</span>
              <button
                className="px-4 py-2 hover:shadow-sm hover:text-teal-500 hover:font-bold"
                onClick={() => handleQuantityChange("inc")}
              >
                +
              </button>
            </div>
          </div>
          <div>
            <button className="inline-flex items-center px-4 py-2 font-semibold text-gray-200 bg-gray-700 rounded hover:text-white hover:bg-gray-900">
              <BiShoppingBag className="mr-2 text-lg" />{" "}
              <span>Add to Cart</span>
            </button>
          </div>
          <div>
            <p className="font-semibold">Product Description</p>
            <hr className="w-2/3 mt-2 border-t-2 border-gray-300" />
            <p className="mt-4 text-gray-700">{product.description}</p>
          </div>
        </div>
      </div>
    </div>
  );
}

Let’s break down the logic into multiple stages. First, you load the single product using the productId route parameter.

import { json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import { createClient } from "~/utils/client";
import { formatPrice } from "~/utils/prices";

export const loader = async ({ params }) => {
 const client = createClient();
 const { product } = await client.products.retrieve(params.productId);
 return json(product);
};

export default function ProductRoute() {
  const product = useLoaderData();

  return (
      <div className="w-full mt-8">
       <h1>{product.title}</h1>
       <p>{formatPrice(variant)}</p>
       <p>{product.description}</p>
      </div>
  );
}

You should be familiar with this code as it's quite similar to app/components/product-card.jsx. The main difference is that you are using Medusa's client.products.retrieve(id) to retrieve data for a single product.

Second, you need to give customers the ability to select product variants, which in this case are sizes. You can implement this feature with the help of the useState React hook:

import { useState } from "react";

export default function ProductRoute() {
    const product = useLoaderData();
    const [variant, setVariant] = useState(product.variants[0]);

      const handleVariantChange = (index) => {
        setVariant(product.variants[index]);
        setQuantity(1);
      };

    return (
        <div>
           ...
           <div>
                {product.variants.map((variantItem, index) => (
                <button
                 key={variantItem.id}
                 onClick={() => handleVariantChange(index)}
                >
                  {variantItem.title}
                </button>
                ))}
              </div>
        </div>
    )
}

The code above will generate a series of clickable buttons corresponding to each variant that the product has.

Third, you need to give end-users the ability to view different images of the product. Here’s the logic for this feature:

import { useState } from "react";

export default function ProductRoute() {
    ...
  const [image, setImage] = useState(product.images[0]);

  const handleImageChange = (id) => {
    setImage(product.images.find((img) => img.id === id));
  };

  return (
    <div>
        ...
        <div>
          <img src={image.url} alt={product.title}
          />
          <div>
            {product.images.map((imageItem) => (
              <img
                className={`w-16 border-2 rounded-lg ${
                  imageItem.id === image.id ? "border-teal-400" : null
                }`}
                key={imageItem.id}
                src={imageItem.url}
                alt={product.title}
                onClick={() => handleImageChange(imageItem.id)}
              />
            ))}
          </div>
        </div>
    </div>
  )
}

The snippet above will display a list of clickable thumbnails below the main product image that users can click to switch and view different images of the product.

Fourth, you need to provide end-users with a quantity input. You need to validate this input to ensure that:

Quantity is not less than 0
Quantity is not more than the variant's inventory

Here's the logic for the quantity input:

import { useState } from "react";

export default function ProductRoute() {
    ...
  const [quantity, setQuantity] = useState(1);

  const handleQuantityChange = (action) => {
    switch (action) {
      case "inc":
        if (quantity < variant.inventory_quantity) setQuantity(quantity + 1);
        break;

      case "dec":
        if (quantity > 1) setQuantity(quantity - 1);
        break;

      default:
        break;
    }
  };

  return (
    <div>
        ...
        <div>
          <p>Select Quantity</p>
        <div>
          <button onClick={() => handleQuantityChange("dec")}>
            -
          </button>
          <span>{quantity}</span>
          <button onClick={() => handleQuantityChange("inc")}>
            +
          </button>
        </div>
      </div>
    </div>
  )
}

The + and - buttons will allow users to increment or decrement the desired quantities of a particular variant. The function handleQuantityChange performs the validation for this input.

Now that you have understood the different logic sections of a Single Product page, let’s see what the completed page looks like in the browser:

You should be able to select variants (size), thumbnails and set the quantity. Make sure every product page is loading without errors.

What’s Next

There are more important ecommerce functionalities that you still need to implement. Those include checkout, payment, shipping, and customer accounts, among other features.

You can check out the Medusa documentation for more details on how you can go forward including:

How to Add Plugins. You can also check the list of plugins available in Medusa:
Add payment methods like Stripe.
Add custom shipping methods.
Add product search using Algolia.

If you’re interested in a storefront with all the ecommerce features ready to use, Medusa has Next.js and Gatsby storefronts that you can use. These storefronts include functionalities like customer accounts, product listings, cart management, and full checkout workflow.

Should you have any issues or questions related to Medusa, then feel free to reach out to the Medusa team via Discord. You can also reach out to the Remix team for support via Discord.

How to Debug Cloudflare Workers with AppSignal

Michael Wanyoike — Wed, 22 Sep 2021 14:26:19 +0000

In this article, you'll learn how to capture error logs in your Cloudflare Workers application using AppSignal. We'll build a simple workers project and integrate AppSignal's code to collect the necessary metrics. We'll also learn how to utilize AppSignal's dashboard to analyze and track errors.

Let's get stuck in!

What Are Worker Functions in Cloudflare Workers?

Cloudflare Workers is a serverless platform that allows developers to build and quickly deploy backend logic using JavaScript. Unlike most serverless platforms, worker functions can respond to a request in less than 500 milliseconds in cold start conditions. Unfortunately, building a worker application does get frustrating at times due to the difficulty of debugging the code.

Here's a snapshot of the metrics collected from a deployed workers application:

In the example above, the majority of the requests have passed, while a few have failed. Unfortunately, there's no way to dig into the dashboard for more information about errors.

The recommended solution is to use a remote logging service to capture error data. This service should have an intuitive dashboard that allows you to inspect the error messages captured.

Prerequisites

This article assumes you have at least a beginner's experience in building a Cloudflare Workers application. That means you should have:

A Cloudflare Workers account
The latest version of Wrangler CLI tool installed and authenticated to connect to your account

For ease in testing, we'll use the Visual Studio Code editor and REST Client extension.

Alternatively, you can use another code editor, but you'll have to use a tool like Postman to make GET and POST requests.

Project Setup

You can find the source code for the project we'll use in this article on GitHub. If you would like to follow along, set up a new workers application from scratch using the following commands:

# scaffold project
wrangler generate appsignal-debug-worker

# navigate inside the project
cd appsignal-debug-worker

# install dependencies
npm install

We'll build a simple POST request handler that calculates a person's age based on their date of birth. This POST handler will accept date information via JSON. We'll use the date-fns package to handle date calculations.

Install the package:

npm install date-fns

Update index.js as follows:

import { parseISO, differenceInYears } from 'date-fns'

addEventListener('fetch', (event) => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  if (request.method === 'POST') {
    return handlePostRequest(request)
  } else {
    return handleGetRequest(request)
  }
}

/**
 * Respond with hello worker text
 * @param {Request} request
 */
async function handleGetRequest(request) {
  return new Response('Hello worker!', {
    headers: { 'content-type': 'text/plain' },
  })
}

/**
 * Respond with 'Hello, {name}. You are ${age} years old!' in json
 * @param {Request} request
 */
async function handlePostRequest(request) {
  const body = await request.json()
  const { name, dob } = body
  // Calculate age
  const dobDate = parseISO(dob)
  const today = new Date()
  const age = differenceInYears(today, dobDate)
  const data = JSON.stringify({
    message: `Hello, ${name}. You are ${age} years old!`,
  })
  return new Response(data, {
    headers: { 'content-type': 'application/json' },
  })
}

Next, update the wrangler.toml file and ensure type has been set to webpack:

name = "appsignal-debug-worker"
type = "webpack"
usage_model = "bundled"

account_id = "<insert account id here>"
workers_dev = true

Create a new rest.http file and copy the following HTTP request commands:

# Test GET request
GET http://localhost:8787/ HTTP/1.1

###

# Test POST - invalid request
POST http://localhost:8787/ HTTP/1.1

###

# Test POST - valid request
POST http://localhost:8787/ HTTP/1.1
Content-Type: application/json

{
  "name": "John",
  "dob": "1990-03-15"
}

You can now use the command wrangler dev to launch the dev server. As the REST client is installed in VS Code, you can easily hover over each request and click the Send Request link. Execute all three request commands and take note of the response.

Below is the output of the invalid request:

[2021-09-08 13:15:07] POST appsignal-debug-worker.brandiqa.workers.dev/ HTTP/1.1 500 Internal Server Error
SyntaxError: Unexpected end of JSON input at line 0, col 0
{
  "exceptionDetails": {
    "columnNumber": 0,
    "exception": {
      "className": "SyntaxError",
      "description": "SyntaxError: Unexpected end of JSON input",
      "preview": null,
      "subtype": "error",
      "type": "object",
      "value": null
    },
    "lineNumber": 0,
    "text": "Uncaught",
    "url": "undefined"
  },
  "timestamp": 1631096101393
}

As we are running on the dev server, we can easily take note of the error. However, once you publish your application, you won't be able to access this essential error information.

In the next section, we'll look at how you can integrate AppSignal to capture and send errors when your application runs in deployment.

Javascript Integration with AppSignal

Let's start by signing up for a new 30-day trial account (no credit card required). If you sign up with your email address, you'll need to go through the usual verification process. Afterward, the process is as follows:

Create a new organization (or choose an existing one).
Pick a language — choose JavaScript.
Create an app — use appsignal-debug-worker for name and staging for environment.
Install the AppSignal JavaScript package. You'll be presented with your API key.
Next, you'll be given a snippet of code to confirm whether you've successfully integrated AppSignal.

Steps 1 to 3 are pretty straightforward.

Let's go through steps 4 and 5. In your terminal, install the AppSignal package and upload the API key:

# Install package
npm install @appsignal/javascript

# Upload api key
wrangler secret put APPSIGNAL_API # supply your api key in the next prompt

To complete step 5, update index.js as follows:

import Appsignal from '@appsignal/javascript'

const appsignal = new Appsignal({ key: APPSIGNAL_API })

async function handleGetRequest(request) {
  appsignal.demo()
  return new Response('Hello worker!', {
    headers: { 'content-type': 'text/plain' },
  })
}

Run the wrangler dev command and execute the first GET request specified in your rest.http file. Your code should run as normal.

Go back to your AppSignal dashboard and navigate to the 'Errors' page. You should have received an incident report that looks similar to the following screenshot:

That's just a simple test to confirm that we've successfully integrated AppSignal into our project. In the next step, we'll look at how we can actually implement error reporting in existing code.

Remove the appsignal.demo() line before you proceed.

How to Send Errors to Your AppSignal Dashboard

There are a couple of ways we can catch and send errors to our AppSignal dashboard using the JavaScript API. We can use the appsignal.sendError(error) error:

try {
  // write main logic here
} catch(error) {
  appsignal.sendError(error)
}

A second option is to use the appsignal.wrap() function as follows:

try {
  await appsignal.wrap(() => {
    // write main logic here
  })
} catch (e) {
  // return error response, the error
  // has already been sent to AppSignal
}

Here's a clearer example, involving asynchronous code:

async function handlePostRequest(request) {
  try {
    const data = await appsignal.wrap(async () => {
      // write main logic here
    })
    // return success response
    return new Response(data, {
      headers: { 'content-type': 'application/json' },
    })
  } catch (error) {
    // return error response
    return new Response(error, {
      headers: { 'content-type': 'text/plain' },
      statusText: error,
      status: 500,
    })
  }
}

When you use the wrap method, any errors within the block are captured and automatically sent to your AppSignal dashboard.

Next up, we'll look at how we can send more error information.

Using Breadcrumbs to Help Find Errors

Breadcrumbs can provide additional information about an error. The information is sent to your dashboard, which should help you with re-tracing the cause of the problem in your code.

The syntax is as follows:

appsignal.addBreadcrumb({
  category: "", // e.g. "UI", "Network", "Navigation", "Console"
  action: "", // e.g "The user clicked a button", "HTTP 500 from http://blablabla.com"
  metadata: {} // key/value metadata in <string, string> format
})

Here's a more complete example that you can use to update index.js:

/**
 * Respond with 'Hello, {name}. You are ${age} years old!' in json
 * @param {Request} request
 */
async function handlePostRequest(request) {
  try {
    const body = await request.json()
    const { name, dob } = body
    // Calculate age
    const dobDate = parseISO(dob)
    const today = new Date()
    const age = differenceInYears(today, dobDate)
    const data = JSON.stringify({
      message: `Hello, ${name}. You are ${age} years old!`,
    })
    return new Response(data, {
      headers: { 'content-type': 'application/json' },
    })
  } catch (error) {
    return handleError(error, {
      category: 'Backend',
      action: 'handlePostRequest',
      metadata: {
        fileName: 'index.js',
        message: error.message,
        foo: 'bar',
      },
    })
  }
}

function handleError(error, breadcrumb) {
  appsignal.addBreadcrumb(breadcrumb)
  appsignal.sendError(error)
  return new Response(error, {
    headers: { 'content-type': 'text/plain' },
    statusText: error,
    status: 500,
  })
}

After updating the file, launch the dev server and test all the HTTP REST URLs. The valid POST request should return the following similar response:

HTTP/1.1 200 OK
date: Wed, 08 Sep 2021 10:16:57 GMT
content-type: application/json
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=qS42jXbBLK13PlG6m1o4sBYLw%2B%2Bbp79hiVQ%2BSR8hRI85U548xXWp3NCG7T6vIDMUQcMMDkG%2FZPOM0elIzB3vs1UXWNUiLQkErgzi%2FgzpnBSubg%2FxLjOSj6lO4osFYCxe9UL1x691hEBEQAHz1ku9hgc17r7jjMA3WrhndnaT"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
vary: Accept-Encoding
cf-ray: 68b76c9d5dadd73d-DAR
content-encoding: gzip
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
server: cloudflare
transfer-encoding: chunked
{
  "message": "Hello, John. You are 31 years old!"
}

The invalid POST request will output the same as before.

Let's now take a look at our AppSignal dashboard and what information got captured. The above type of error is placed under the SyntaxError entry. See the screenshot below for comparison:

As you can see, all the data we included in the addBreadcrumbs function has been captured. Implementing this strategy in your code will help you debug your code in production better.

Troubleshoot Your Errors Easily with AppSignal

To recap, we've learned how to:

Integrate AppSignal into our workers code using the @appsignal/javascript npm package
Use the appSignal.sendError() function to send error data
Use the appSignal.wrap() function to automatically capture and send error data
Use the breadcrumbs function to send additional details about an error

While AppSignal provides a myriad of features, including performance and anomaly tracking, the @appsignal/javascript package only supports the error reporting feature.

Even so, error reporting allows you to build and deploy your worker applications with confidence. You'll be able to quickly troubleshoot problems before end-users submit support tickets.

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Michael Wanyoike writes clean, readable, and modular code. He loves learning new technologies that bring efficiencies and increased productivity to his workflow.

How to Perform Load Testing with k6 using AWS CodeBuild Platform

Michael Wanyoike — Mon, 13 Sep 2021 12:50:07 +0000

You can find a collection of k6 scripts and build specification for CodeBuild in this tutorial here.

📖What you will learn

How to integrate load testing with k6 into AWS CodeBuild Platform.

Different implementation paths, and when to use each.

Introduction

In this tutorial, we will look into how to integrate performance testing in your development process with AWS CodeBuild and k6.

k6 is an open-source load testing tool for testing the performance of APIs, microservices, and websites. Developers use k6 to test a system's performance under a particular load to catch performance regressions or errors.

AWS CodeBuild is a fully managed continuous integration service hat compiles source code, runs tests, and produces software packages that are ready to deploy. It uses a pay-as-you-go pricing model with 100 free build minutes per month on the AWS Free Tier plan.

Documentation for AWS CodeBuild is tightly bound with other AWS services which makes it difficult to understand AWS CodeBuild as a standalone product. To put it simply, AWS has multiple products in it's CI/CD offering which includes:

AWS CodeBuild : continuos integration service
AWS CodeDeploy : managed deployment service
AWS CodePipeline : continuos deployment service
AWS CodeCommit : managed source control service

You can read this blog post to understand how all these services are integrated to form a complete CI/CD pipeline. Fortunately for us, we only need to concern ourselves with AWS CodeBuild and Amazon EventBridge in order to automate k6 load testing. We recommend having a look at the following links:

Writing your performance test

In order to learn how to write a performance test, we'll work in small steps and improving the existing script as our knowledge develops. The test will be made up of the following:

An HTTP request against our system under test.
A load configuration controlling the test duration and amount of virtual users.
A performance goal, or service level objective, expressed as a threshold.

Creating the test script

The test script we will write first will simulate a virtual user accessing a specified website. To ensure we don't overload the system under test, we'll put the virtual user to sleep for one second before continuing.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

Configuring the load

In this step, we'll look at how we can define parameters. For this example, we'll have 50 virtual users accessing the specified website for the duration of one minute. Due to the sleep we specified earlier, the script will generate 50 interactions per second, resulting in a total of about 2900 iterations.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

If you have installed k6 in your local machine, you can run your test locally in your terminal using the command: k6 run test.js.

Configuring our thresholds

Next, we need to define our Service Level Objectives, also known as SLOs. This is basically a measurement that specifies the desired performance levels at a given load. With k6, this is done using Thresholds where you can define SLOs as Pass/Fail criteria.

If any of the thresholds you define fail your test, k6 will halt and return a non-zero exit code, communicating to the CI tool that the step has failed. Let's see how we can add a Threshold to our previous script. We'll ensure that the 95th percentile response time remains below 500ms.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,  
    thresholds: {    
        http_req_duration: ['p(95)<500'],  
    },
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

Thresholds are powerful features that allow us to define different types of Pass/Fail criteria in the same test run. For example:

The 99th percentile response time must be below 700 ms.
The 95th percentile response time must be below 400 ms.
No more than 1% failed requests.
The content of a response must be correct more than 95% of the time.

Take a look at our Thresholds documentation for additional details on the API and its usage.

Setting up the CodeBuild workflow

We'll use Docker to run load testing in AWS CodeBuild's environment:

docker run -i loadimpact/k6 run - <scripts/test.js

In order to automate the build process, we'll need to create a buildspec.yml file with the following configuration:

version: 0.2

phases:
  pre_build:
    commands:
      - echo Pull loadimpact/k6 image..
      - docker pull loadimpact/k6
  build:
    commands:
      - echo Run performance test
      - docker run -i loadimpact/k6 run - <scripts/test.js
  post_build:
    commands:
      - echo Performance test complete

At the time of writing AWS CodeBuild supports repositories hosted on GitHub, BitBucket, Amazon S3 and AWS CodeCommit. For reference, we'll use GitHub for this tutorial. Once you've prepared a project containing the above files, you'll need to commit and push the changes to your remote repository.

Once your remote repository is ready, you can start working on your AWS Console. Using the Services menu, locate AWS CodeBuild and open the page:

Click the Create project button and populate the form as follows:

Project Configuration
- Project name: Load-Test-Example
- Description: Load testing example with k6
Source
- Source provider: Github
- Repository: 'Public' or 'Repository in my GitHub account' for private
- Repository URL: https://github.com/brandiqa/k6-example-aws-codebuild
- Service Role Permissions: Set true for 'Allow AWS CodeBuild to modify this service role so it can be used with this build project'
Environment
- Operating system: Ubuntu
- Runtime: Standard
- Privileged: true. If you don't enable privilege, the docker command won't have the system level permissions to run.
- Leave all other settings in their default state
Buildspec
- Select 'Use a buildspec file'

Leave all other settings in the default state. Ensure CloudWatch logs is enabled. Click the Create build project button.

The next step is to run the project which you can do using the Start build project. Once it's complete, you should expect the following results:

Running k6 cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

Locally on the CI server.
In k6 cloud, from one or multiple geographic locations.

You might want to use cloud tests in these common cases:

If you're going to run a test from multiple geographic locations (load zones).
If you're going to run a high-load test, that will need more compute resources than available in the runner.

If any of those reasons fit your needs, then running k6 cloud tests is the way to go for you.

⚠️ Try it locally first

Before we start with the configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the cloud execution guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to trigger cloud tests using AWS CodeBuild. If you do not have an account with k6 Cloud already, you should go here and start your free trial. After that, get your account token from the cloud app and save it somewhere temporarily.

Next, let's create a modified version of the testing script and call it cloud-test.js:

import http from 'k6/http';
import { sleep } from 'k6';


export const options = {
    ext: {
        loadimpact: {
            name: 'cloud-test',
        },
    },
    duration: '1m',
    vus: 50,
    thresholds: {
        http_req_duration: ['p(95)<500'],
    },
};

export default function () {
    const res = http.get('https://test.k6.io');
    sleep(1);
}

The ext property enables passing of the script filename which is required for cloud execution. The next step is to update buildspec.yml as follows:

version: 0.2

env:
  exported-variables:
    - K6_CLOUD_TOKEN

phases:
  pre_build:
    commands:
      - echo Pull loadimpact/k6 image..
      - docker pull loadimpact/k6
  build:
    commands:
      - echo Run performance test
      - docker run -i -e K6_CLOUD_TOKEN=${K6_CLOUD_TOKEN} loadimpact/k6 cloud - <scripts/cloud-test.js
  post_build:
    commands:
      - echo Performance test complete

You'll need to commit and push the new changes to your remote repository. Next, we need to modify the current build and specify the K6_CLOUD_TOKEN as an environment variable:

After making these changes, run the build. In about a minute or so, you should expect the following output:

And if we copy the highlighted URL and navigate to it in a new tab

Scheduling Builds

It is highly recommended to perform load testing at a time when most users aren't accessing your system. This is because performance testing often places significant stresses on your infrastructure which may cause user experience issues.

To configure scheduled nightly build that runs at a given time of a given day or night, head over to Amazon EventBridge.

Head over to the Amazon EventBridge console and click the Create rule button. Fill in the form as follows:

Name: Schedule-Load-Test-Example
Description: Nightly build
Define pattern: Schedule
Cron Expression: 0 12 * * ? *
Target: CodeBuild project
Project ARN: arn:aws:codebuild:region-ID:account-ID:project/project-name

For the Project ARN, simply replace the values in italics with values from your account i.e.

region-ID: found at the top navigation bar, click the drop-down location menu to see your region id in the format 'us-east-1' or 'us-west-1'
account-ID: follow this guide
project-name: Load-Test-Example

Your fully constructed project ARN should look similar to this:

arn:aws:codebuild:us-east-2:123456789123:project/Load-Test-Example

AWS doesn't fully recognize the standard CRON format. Hence you'll need to use the format specified in their cron guide.

After filling the form, click the 'Create' button to save the changes. The Load-Test-Example CodeBuild project will run at the specified time.

Summary

Building your CI/CD pipeline on AWS platform can be overwhelming at first. Hopefully, this guide should help alleviate the pain by showing you exactly what you need you to configure in order to setup an automated load testing pipeline on AWS.

As demonstrated in this article, you can execute an AWS build that runs k6 either locally or on the cloud. Automating performance testing in your CI/CD pipeline will help you identify issues early before they affect your customers.

How to Perform Load Testing with k6 using Google Cloud Build

Michael Wanyoike — Wed, 01 Sep 2021 10:50:01 +0000

You can find a collection of k6 scripts and Google Cloud Build workflows referenced in this tutorial here.

📖 What you will learn

How to integrate load testing with k6 into Google Cloud Build CI/CD Platform.

Different implementation paths, and when to use each.

Introduction

In this tutorial, we will look into how to integrate performance testing in your development process with Google Cloud Build and k6.

Google Cloud Build is a serverless CI/CD platform that allows developers to create custom workflows for their software development lifecycle within the Google Cloud Platform(GCP) ecosystem. It uses a pay-as-you-go pricing model with 120 free minutes given everyday.

If you've not used Google Cloud Build before, we recommend looking at the following links to get a hold of how it works:

Writing your performance test

We'll start small by writing a simple test that measure the performance of a single endpoint. As with most, if not all, development efforts, performance testing yields the best results if we work in small increments, iterating and expanding as our knowledge increases.

Our test will consist of three parts:

An HTTP request against our system under test.
A load configuration controlling the test duration and amount of virtual users.
A performance goal, or service level objective, expressed as a threshold.

Creating the test script

When we execute our test script, each virtual user will execute the default function as many times as possible until the duration is up. To make sure we don't flood our system under test, we'll make the virtual user sleep for a second before it continues.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
    const res = http.get('https://test.k6.io');
    sleep(1);
}

Configuring the load

We'll configure our test to run 50 virtual users continuously for one minute. Because of the sleep we added earlier, this will result in just below 50 iterations per second, giving us a total of about 2900 iterations.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

If you have installed k6 in your local machine, you can run your test locally in your terminal using the command: k6 run test.js.

Configuring our thresholds

The next step is to define your service level objectives, or SLOs around your application performance. SLOs are a vital aspect of ensuring the reliability of your systems and applications. If you do not currently have any defined SLAs or SLOs, now is an excellent time to consider your requirements.

You can define SLOs as Pass/Fail criteria with Thresholds in your k6 script. k6 evaluates them during the test execution and informs about the Threshold results. If any of the thresholds in our test fails, k6 will return with a non-zero exit code, communicating to the CI tool that the step has failed.

Now, we will add one Threshold to our previous script to validate than the 95th percentile response time must be below 500ms.

// ./test.js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {  
    duration: '1m',  
    vus: 50,  
    thresholds: {    
        http_req_duration: ['p(95)<500'],  
    },
};

export default function () {  
    const res = http.get('https://test.k6.io');  
    sleep(1);
}

Thresholds are a powerful feature providing a flexible API to define various types of Pass/Fail criteria in the same test run. For example:

The 99th percentile response time must be below 700 ms.
The 95th percentile response time must be below 400 ms.
No more than 1% failed requests.
The content of a response must be correct more than 95% of the time.

Check out the Thresholds documentation for additional details on the API and its usage.

Setting up the Cloud Build workflow

Ideally, the easiest way to setup load testing in a CI/CD environment is to use docker:

docker run -i loadimpact/k6 run - <scripts/test.js

Unfortunately, the above command fails to run within the Cloud Build environment. The solution is to mount the source code files and execute them within a docker container environment. To ensure a smooth CI/CD workflow in Google Build's environment, we'll run our load performance test using docker-compose. First, we need to define our docker-compose.yml file:

version: '3.4'
services:
  k6:
    image: loadimpact/k6:latest
    command: run /scripts/test.js
    volumes:
      - ./scripts:/scripts

Basically, we have defined a single service called k6 that performs the following:

Pulls the k6 docker image from Docker Hub registry
Executes a command that runs the local k6 test
Mounts the scripts folder in the /script path

Next we need to define the CI/CD steps that will be executed on the Cloud Build platform. We'll do this using cloudbuild.yaml file:

steps:
# Load Test
- name: 'gcr.io/cloud-builders/docker'
  args: ['run', '-i', 'loadimpact/k6', 'run', '-', '<scripts/test.js']

Normally, we would have to define steps for install, test, build and deploy. Since we are testing with an app that has already been deployed, test.k6.io, we'll only define a single step, Load Test.

At the time of writing, only GitHub, BitBucket and Google Cloud Source repositories are supported by Google Cloud Platform. For reference, we'll use GitHub for this tutorial. Once you've prepared a project containing the above files, you'll need to commit and push the changes to your remote repository.

Once your remote repository is ready, you can start working on your Google Cloud Console. If you have never used Cloud Build before, you'll need to enable the API first. The setting below is located under API & Services > Library > Developer tools:

By enabling the API, the appropriate IAM service account will be created automatically that will allow you to run the build steps successfully. With Cloud Build, Google provides us with a selection of cloud builders a.k.a execution environments for running our CI/CD steps. These builders include yarn, npm, docker, mvn, go and a few more. Unfortunately, majority of the building tools we need for our projects are not available out of the box.

To add the tools we need, we have to visit the community contributed builders repository where all the tools needed in a software development lifecycle can be found. In our case, we simply need the docker-compose builder. The steps to install this builder in our own Google Cloud account is as follows:

Install and configure gcloud CLI tool to connect with your GCP account from your machine
Run the update command: gcloud components update
Install docker-compose builder on your GCP account using the following commands:

   # Clone the entire cloud builders community repo onto your hard drive
   git clone https://github.com/GoogleCloudPlatform/cloud-builders-community

   # Navigate into the docker-compose folder
   cd cloud-builders-community/docker-compose/

   # Install docker-compose builder into your account
   gcloud builds submit --config=cloudbuild.yaml .

Confirm the docker-compose builder has been installed. See Cloud Build > Container Registry > Images

Once you have completed this step, you can now proceed with creating a Trigger which is the basis of building a CI/CD pipeline in Google Cloud Build.

Head over to Cloud Build > Triggers page and click the Create Trigger button. Populate the form as follows:

Name - Run-Performance-Test
Description - Run k6 load test
Event - Push to a branch
Source - link to your GitHub repository
Branch - main
Configuration - Cloud Build configuration file (yaml or json)
Location - Repository

After hitting save, click the Run button. You should expect the following results:

Running k6 Cloud tests

There are two common execution modes to run k6 tests as part of the CI process.

Locally on the CI server.
In k6 Cloud, from one or multiple geographic locations.

You might want to use cloud tests in these common cases:

If you're going to run a test from multiple geographic locations (load zones).
If you're going to run a high-load test, that will need more compute resources than available in the runner.

If any of those reasons fit your needs, then running k6 Cloud tests is the way to go for you.

⚠️ Try it locally first

Before we start with the configuration, it is good to familiarize ourselves with how cloud execution works, and we recommend you to test how to trigger a cloud test from your machine.

Check out the cloud execution guide to learn how to distribute the test load across multiple geographic locations and more information about the cloud execution.

Now, we will show how to run cloud tests on the Google Cloud Build platform. If you do not have an account with k6 Cloud already, you should go here and start your free trial. After that, get your account token from the cloud app and save it somewhere temporarily.

Next, we need to update our docker-compose.yml file as follows:

version: '3.4'
services:
  k6:
    image: loadimpact/k6:latest
    command: cloud /scripts/test.js
    volumes:
      - ./scripts:/scripts
    environment:
      - K6_CLOUD_TOKEN=${K6_CLOUD_TOKEN}

There are only 2 changes we have made here:

In the command section, we've changed from run to cloud
We've added an environment section. The ${K6_CLOUD_TOKEN} environment variable value needs to be defined outside the docker container, in this case, we'll do it in the cloudbuild.yaml config file where execution call is made.

With most platforms, there is a system for securely storing environment variables. Unfortunately for Google Cloud Build, there isn't. Instead, what we'll paste the token value in plain format within the Trigger configuration. We'll use the inline YAML editor. Simply edit the current Trigger and update the following settings:

Configuration Location: Inline
Click Open Editor button and copy the following:

  steps:
    - name: gcr.io/$PROJECT_ID/docker-compose
      env:
        - 'K6_CLOUD_TOKEN=<paste your token here>'
      args:
        - run
        - k6

Paste in your token key as indicated. Click Done and then Save.

Once we commit and push the changes we made earlier, k6 will run our test using the k6 Cloud, and output the URL to our test results as part of the workflow logs:

And if we copy the highlighted URL and navigate to it in a new tab

Nightly Builds

Triggering a subset of performance tests at a specific time is a best-practice for automating your performance testing.

It's common to run some performance tests during the night when users do not access the system under test. For example, to isolate more extensive tests from other types of testing or to generate a performance report periodically.

To configure scheduled nightly build that runs at a given time of a given day or night, head over to your Trigger configuration and edit as follows:

Event: Manual invocation

Go back to the main Triggers page and click on the menu (vertical ellipses) located at the right end of the row containing your trigger. You should see a new option called Run on schedule. Click on it to start setting up the schedule. You'll need to:

Enable the Schedule API
Select a service account (accept the default which should start with cloud-build-trigger-scheduler@..)
Provide name, description and frequency of the schedule in POSIX cron syntax. Here is an interactive tool for creating crontab scheduling expressions. You can also select your desired time zone.

You can use the following values for reference:

Once you save your schedule settings, Google Cloud Build will take care of running the workflow at the time intervals you specified.

You can also visit the Cloud Scheduler page and trigger the scheduled job manually. In case this fails, edit the Job Schedule and check the Configure the job's target > Body field. By default the branch name is set to master. If you are using a different branch such as main, you need to update as follows:

{"branchName":"main"}

After saving, you should be able to successfully run the job manually.

Summary

While there is a learning curve with setting up a full CI/CD pipeline for your project with Google Cloud, integrating k6 performance tests into a new or existing Google Cloud Build pipeline is quick and easy as only a couple of configuration files are involved.

The configurations mentioned throughout the article, provide the same flexibility and capabilities as you're used to from running k6 locally. By integrating performance tests into your CI/CD pipelines, you'll be able to identify and correct performance regressions as they occur.

How to Autoscale Kubernetes Pods with Keda - Testing with k6

Michael Wanyoike — Tue, 26 May 2020 18:22:23 +0000

Overview

This article demonstrates how to Load Test an application deployed in a Kubernetes cluster, verify that the autoscaling is working, and identify potential performance bottlenecks.

When you deploy an application to production on Kubernetes, you'll need to monitor traffic and scale up the application to meet heavy workloads during various times of the day. You'll also need to scale down your application to avoid incurring unnecessary costs. Doing this manually is impractical since heavy traffic can come at any time including late at night when your entire team is asleep. And even if you are awake, a spike might come and go before you had a chance to address it.

The best approach is to automate scaling. This can be triggered by monitoring custom metrics such cpu usage, network bandwidth or http requests per second. Scaling an application running on a Kubernetes platform can be done in the following ways:

Horizontal : Adjust the number of replicas(pods)
Vertical : Adjust resource requests and limits imposed on a container

In this article, we'll focus on horizontal scaling based on a custom metric. Which metric you should choose to trigger scaling is outside the scope of this article. We will use the http request rate metric in this example, however other metric types can be used. We'll cover how to obtain these metrics and visualize them for analysis. We'll use the following to perform this task:

k6 OSS: an open-source load testing tool. We'll use it to simulate heavy traffic(load) for our Kubernetes application. There's also Grafana Cloud k6 which you can use to scale load testing beyond your local computing and networking infrastructure limitations.
Prometheus: an open-source monitoring platform. We'll use it to scrape metrics from our application and the Kubernetes API in real-time as while the load testing tool is running.
Grafana: an open-source analytics platform. We'll use it to visualize real-time metrics being collected by Prometheus so that we can see the performance of our application across time. Below is a screenshot of how metrics are visualized on a Grafana dashboard.

The outline of this tutorial is as follows:

Application Deployment
Prometheus, Kube State Metrics and KEDA Deployment
InfluxDB and Grafana Setup
Configuring Horizontal Pod Autoscaling with Keda

You can follow along this entire tutorial on a single machine. Just note that results will be skewed since your CPU will handle both running the application and load testing it. For accurate results, these tasks need to be performed on separate machines. For learning and simplicity purposes, one machine will be sufficient.

The source code of the application we'll be load testing is provided in our GitHub repository. The load testing script is included there as well.

Below is a complete illustration of the entire setup:

Prerequisites

This article assumes you at least have some basic knowledge about running and configuring a Kubernetes cluster. We'll be using Minikube for this guide. However, feel free to use any other k8s implementation that can run on your computer.

Before we proceed, you'll need to have the following installed on your machine. For windows users, you can use Chocolatey package manager to install most of these requirements. For macOS, use brew. For Linux, use the instructions provided in the following links:

k6
Node.js
Docker
GNU Make (pre-installed for Linux, install xcode for macOS users)
kubectl
Minikube

Application Setup & Deployment

If you haven't, download the example project to your workspace now:



# Download project
git clone git@github.com:k6io/example-kubernetes-autoscaling-nodejs-api.git
cd example-kubernetes-autoscaling-nodejs-api

# Install dependencies
npm install

# Run development server
npm run dev

Open the URL http://localhost:4000/ to confirm the application is running. If you click on the crocodiles API link or visit the url http://localhost:4000/crocodiles, you should see the following:

Let's quickly inspect the code. Open the file db.json in your favorite code editor. This is where data is stored. You can add more records if you want. Next open server.js. This is the complete project code where the server logic lies. You'll see it's a simple project that uses json-server package to provide CRUD API services.

If you observe this section of code, you'll notice that an artificial delay has been implemented. Because the API application is so simple, we need to slow it down a bit just to help us simulate the results of a real-world application.



const minDelay = 30;
const maxDelay = 250;

// Add a delay to /crocodiles requests only
app.use('/crocodiles', function (req, res, next) {
  let delay = Math.floor(Math.random() * (maxDelay - minDelay)) + minDelay;
  setTimeout(next, delay);
});

The next section of the code I want to show you is how to export the metrics from our application to Prometheus. Later on, we will query these metrics to monitor the status of the application.



const prometheusExporter = require('@tailorbrands/node-exporter-prometheus');
const options = {
  appName: 'crocodile-api',
  collectDefaultMetrics: true,
  ignoredRoutes: ['/metrics', '/favicon.ico', '/__rules'],
};
const promExporter = prometheusExporter(options);
app.use(promExporter.middleware);
app.get('/metrics', promExporter.metrics);

If you would like, you can define additional metrics for your application. The above code will output several default metrics. In your browser, open the URL http://localhost:4000/metrics to view these metrics:

For this application, we are using the npm package @tailorbrands/node-exporter-prometheus to help us export metrics from our application in a format compatible with Prometheus scraping requirements with very little effort on our part. You can find other Prometheus client library for Node.js applications in the npmjs registry repository. If you are working with a different programming language, you can find official and non-official Prometheus client libraries on Prometheus website.

The metrics displayed in the above screenshot will update when you interact with application. Simply refreshing the URL http://localhost:4000/crocodiles in your browser or executing a curl command on the site will cause the values of the metrics to update. For example, the metric node_http_requests_total keeps track of the number of times a HTTP request is performed on the application. Do note the metrics page doesn't use AJAX. So you'll have to refresh from time to time to see the results.

When you setup Prometheus, it will fetch the values of this metrics every 5 - 15 seconds and store it in a time-series database. This is what is referred to as scraping. With this information, Prometheus can plot for you a graph so that you can see how the metric values change over time.

If you inspect the metrics provided by our application, you'll notice we have different metric types. For this guide, we'll focus on the metric node_http_requests_total which is of type counter. A counter is a cumulative metric that always keeps going up. It can be reset to 0 if the application is restarted.

Now, we run the k6 load testing tool to generate some traffic, and we will visualize how this counter metric changes over time. At the root of the application project, locate the script performance-script.js which contains instructions on how to perform the load test.

Below are 2 examples of the k6 load test configuration. The first option is a quick 3 minute load test you can use to quickly confirm metrics are being captured. The second option allows us to scale the number of virtual users over a duration of 12 minutes. This will give us enough data to analyze performance and behavior of our autoscaling configuration.



// First Load Testing Option : Quick Test with fixed no. of virtual users
export let options = {
  duration: '3m',
  vus: 200,
  thresholds: {
    http_req_duration: ['p(95)<700'],
  },
};

// Second Load Testing Option : Long Test with varying no. of virtual users
export let options = {
  stages: [
    { duration: '1m', target: 50 },
    { duration: '1m', target: 150 },
    { duration: '1m', target: 300 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 800 },
    { duration: '3m', target: 1200 },
    { duration: '3m', target: 50 },
  ],
};

Before we can run the load testing script, we need to deploy Prometheus to scrape the application's metric. Our application needs to be deployed as well in order for Prometheus to discover our application. To deploy our project to our Kubernetes(minikube) node, simply execute the following commands:



# Build the application's docker image inside the minikube's environment
make image
# Deploy the application
make apply

If you have trouble executing the above commands, just visit the file Makefile and execute the commands under image and apply sections. Below should be output of the last command:

Take note of the IP address and the port. In the above case, we can access our application through a web browser using the following address: http://10.98.55.109:4000. Unfortunately, the page will likely refuse to load since we haven't fully configured how our load balancer will expose our service. A quick work around is to create a tunnel by running the following command in a separate terminal:



minikube tunnel

Once the tunnel is up and running, you should be able to access the application in your browser. Proceed to the next section and deploy Prometheus.

Deploy Prometheus, Kube State Metrics and KEDA

To deploy Prometheus to our minikube node, follow this guide. You'll also need to deploy Kube State metrics. This is a service that accesses the Kubernetes API and provides Prometheus with metrics related to API objects such as deployments and pods. We need this service to track the number of running pods.

Since we are deploying, let's also deploy KEDA, a Kubernetes Event-driven Autoscaling service. It works alongside Horizontal Pod Autoscaler to scale pods up and down based on a threshold that we'll need to specify. Instructions for deploying using YAML files can be found on this page. For convenience, these are the commands you need to execute to deploy KEDA:



git clone https://github.com/kedacore/keda && cd keda

kubectl apply -f deploy/crds/keda.k8s.io_scaledobjects_crd.yaml
kubectl apply -f deploy/crds/keda.k8s.io_triggerauthentications_crd.yaml
kubectl apply -f deploy/

Use the command kubectl get po -A to ensure all the pods we have deployed are running. Use the command kubectl get service -n monitoring to find the ip address and port where Prometheus dashboard can be accessed. Construct the URL and launch the Prometheus dashboard in your browser. First let's confirm that Prometheus has discovered metrics from our application. From the top menu, go to Status > Targets page and scroll down to the bottom and look for the label crocodile-service:

If you see a similar status like the above, you are good to go on the next step. You should also confirm that the service kube-state-metrics has been discovered as well. On the top menu, click the Graph link to go to the graph page. This is where we enter query expressions to access the vast information that Prometheus is currently scraping.

Before we execute a query, fire up the k6 load testing tool to get some data to work with. You'll need to execute the load test script like this:



ENDPOINT=10.98.55.109:4000/crocodiles k6 run performance-test.js # replace ip address with yours

On the Graph page, enter this expression: node_http_requests_total. It should autofill for you as you type. Click on the Graph tab and you should see the following output.

This is how the results are visualized after the test has completed. Notice how it flattens at the top. This is because the metric stopped incrementing when the test ended. If you click on the Table tab, you should see two fields that look like this:

The node_http_requests_total metric keeps track of the total number of requests per HTTP status code. If you were to run the test, the line graph will start shooting up from where it left of. This metric doesn't seem useful in it's current form.

Fortunately, we can apply a function to make it useful. We can use the rate() function to calculate the number of requests per second over a specified duration. Update the expression as follows:

rate(node_http_requests_total[2m])

This function will give us the number of requests per second within a 2 minute window. Basically, it calculates how fast are the increments increasing per second. When the increments stop, the rate() function will give us 0. Below are the results of the load test with the rate function applied in the expression:

As you can see, our loading test script peaked at approximately 65 requests per second. When the load test script completed, the graph line goes back to 0. This is now a useful metric we can use to determine if scaling our pods is needed.

In the next section, we'll setup a more advance visualization dashboard that can display multiple metrics at once with automatic refresh rate.

Install InfluxDB and Grafana

In this section, we'll install InfluxDB which is an open-source time series database. InfluxDB is widely supported by many tools and applications. This provides new opportunities for collecting additional metric data from other sources such as Telegraf.

Grafana is an open-source analytic and monitoring platform that allows engineers to build dashboards for visually monitoring metrics from multiple sources in real-time. It supports the use of expressions to interpolate raw metric data into information that is easy to consume.

Both InfluxDB and Grafana can be deployed on the Kubernetes node. However, its easier and faster installing them locally. Do not use the Docker option as it's not easy for containers to communicating with Kubernetes services. Below are the download links:

Download InfluxDB : Get 1.8 OSS version
Download Grafana : Get latest version

Once you have installed both applications, make sure the services are running. For Ubuntu:



sudo systemctl start influxdb grafana-server

You can interact with influxDB database server via the influx command line interface. You can also interact with influxDB via http://localhost:8086 by passing query parameters.

Grafana is accessed by visiting http://localhost:3000. The default username and password should be admin admin. If this doesn't work, just edit the file /etc/grafana/grafana.ini and ensure the following lines are enabled:



[security]
# default admin user, created on startup
admin_user = admin

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = admin

You'll need to restart grafana server for the the change to take effect. Once you log in, you'll need to go to Configuration > Data Sources and add the following sources:

InfluxDB K6 Database
Prometheus data source

Feel free to use other sources. In the screenshot provided, I've additionally installed Telegraf which is a service that collects real-time metrics from other database systems, IoT sensors and system performance metrics for CPU, memory, disk and network components. We won't use it for this article though.

Below are the settings that I've used for InfluxDB data source. The rest of the fields not mentioned here are left blank or in their default setting:

Name: InfluxDB-K6
URL: http://localhost:8086
Access: Server
Database: k6
HTTP Method: GET
Min time interval: 5s

Click the Save & Test button. If it says 'Data source is not working', it's because the database has not been created. When you run the following k6 command, the database will be created automatically:



# replace IP address
ENDPOINT=10.98.55.109:4000/crocodiles k6 run -o influxdb=http://localhost:8086/k6 performance-test.js

Below are the settings that I've used for Prometheus data source.

Name: InfluxDB-K6
URL: http://<prometheus ip address>:8080/
Access: Server
Scrape interval: 5s
Query timeout: 30s
HTTP Method: GET

Click Save & Test and make sure you get the message "Data source is working". Otherwise, you won't be able to proceed to the next step.

The next step is to create a new dashboard by importing the Crocodile Metrics Dashboard from this link. Copy and paste the JSON code and hit save. This custom dashboard will allow you to visually track:

HTTP Request Rate (sourced from both k6 and application via Prometheus)
Number of virtual users
Number of active application pods and their status
99th percentile response time (measured in milliseconds)
Memory usage per pod (measured in megabytes)

All the panels are easy to configure. You can change widget type, adjust the query and display more values. You can also re-organize the layout and add new panels. Below is how the Crocodile Metrics Dashboard should look like when installed:

In the next section, we'll configure KEDA to monitor and scale up our application.

Configuring Horizontal Pod Autoscaling with Keda

Before we configure auto scaling, let's run a quick load test. Open performance-test.js and ensure the following code is active:



export let options = {
  stages: [
    { duration: '1m', target: 50 },
    { duration: '1m', target: 150 },
    { duration: '1m', target: 300 },
    { duration: '2m', target: 500 },
    { duration: '2m', target: 800 },
    { duration: '3m', target: 1200 },
    { duration: '3m', target: 50 },
  ],
};

Run the k6 script using the following command ensuring InfluxDB is collecting K6 metrics.:



# replace IP address
ENDPOINT=10.98.55.109:4000/crocodiles k6 run -o influxdb=http://localhost:8086/k6 performance-test.js

The Grafana dashboard should start populating with data. As the number of http requests per second increases, the number of pods stays constant. You can wait for the load test to complete or you can cancel mid-way.

Let's now configure KEDA to monitor and scale our application. Open the YAML config file keda/keda-prometheus-scaledobject located in the project and analyze it:



apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: default
  labels:
    deploymentName: crocodile-api
spec:
  scaleTargetRef:
    deploymentName: crocodile-api
  pollingInterval: 10 # Optional. Default: 30 seconds
  cooldownPeriod: 15 # Optional. Default: 300 seconds
  minReplicaCount: 1 # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  triggers:
    - type: prometheus
      metadata:
        # Required
        serverAddress: http://10.103.240.12:8080
        metricName: access_frequency
        threshold: '50'
        query: sum(rate(node_http_requests_total[2m]))

Take note of the query we provided:

sum(rate(node_http_requests_total[2m]))

KEDA will run this query on Prometheus every 10 seconds. We've added the sum function to the expression in order to include data from all running pods. It will check the value against the threshold we provided, 50. If the value exceeds this amount, KEDA will increase the number of running pods up-to a maximum of 10. If the value is less, KEDA will scale our application pods back to 1. To deploy this configuration, execute the following command:



kubectl apply -f keda/keda-prometheus-scaledobject

Once again, run the k6 script test just like before and observe how the number of pods increases as the number of requests per second increase as well. Below is the final result after the test has completed.

Take note of the Running Pods chart. The number of application pods increased as the load increased. When the load throttled down, the number of pods decreased as well. And that's the beauty of automation.

Conclusion

How to Autoscale Kubernetes Pods with Keda - Testing with k6

How to autoscale pods using KEDA and Prometheus
How to analyse and visualize metrics using Grafana
How to extract metrics from applications and services

With the information you have at hand, you can benchmark any application and monitor the performance with accuracy. In addition to application metrics, you can also use:

CPU usage
Memory usage
Network usage
Disk usage
Database performance
Temperature

However, I would like to point out some important facts we haven't covered here that you need to learn:

For a real world application, you would have a separate pod containing the database. The data for this type of pod needs to synced when scaled out.
You will need to determine the request rate threshold that would make sense to scale

The more metrics you can monitor, the easier it is to identify bottlenecks affecting the performance of your application. For example, if your application has high CPU usage, optimizing the code can greatly boost the performance. If disk usage is high, using memory caching solutions can help a big deal.

The point is, scaling the number of application pods shouldn't be the only solution when it comes to handling heavy traffic. Scaling vertically or increasing the number of nodes should be considered and could greatly improve performance during heavy loads. By using k6 load testing tool, and using Grafana to analyze results, you can discover where bottlenecks for your application are located.