DEV Community: Žiga Patačko Koderman

Lighthouse vs. PageSpeed Insights: The Lack of Correlation

Žiga Patačko Koderman — Mon, 05 Aug 2024 12:23:00 +0000

What's the problem?

Have you ever run PageSpeed Insights (PSI) on your website only to receive a wildly different performance score than when running Lighthouse via Chrome Developer Tools? Let me show you what I mean:

The overall scores are relatively similar, but there is almost no correlation between the metrics that make them up. This is a problem because you cannot run PageSpeed Insights on your local build of the website, forcing you to publish any changes and test them online, which is slow, inconvenient, and a bad practice. Let's take a look at where the difference between Lighthouse and PageSpeed comes from.

The reason for the discrepancy

PageSpeed uses Lighthouse under the hood, so what is the deal here? Googling this tells you that PageSpeed also includes some real-world data and so on, but this does not address the problem at stake. After some lengthy deep dives into the issue, we found the reason for it here:

PageSpeed Insights run Lighthouse using a relative 4x CPU slowdown.

The fact that it is relative results in a different score based on the device running the benchmark.

You might be tempted to say "just multiply the times here by X to compensate for having a faster computer." However, this won't work. We won't go into the depths of why this is the case beyond saying that those metrics are complex and don't scale in that way. You can find out more here (TBT is perhaps the simplest example of that).

Solution

Let us start by installing the tools we'll need further down the line.

1. Install the Lighthouse CLI

You cannot tweak the CPU slowdown of a Lighthouse simulation through the Chrome Dev Tools. Hence the need for Lighthouse CLI. You can install it using



npm install -g lighthouse

2. Estimate the slowdown factor (optional)

Here we will show how to extract the benchmarkIndex for your device and then calculate an approximate slowdown factor that you need to use. This is the procedure found in the Chrome documentation. However, you will probably need to readjust the slowdown even after that. Therefore, we prefer to skip this step and find a suitable constant using trial and error.

2.1 Extract `benchmarkIndex` from the report

Here is a command that extracts the benchmarkIndex from the report. It presumes that you have jq installed on your system.



lighthouse YOUR_WEBSITE_URL --output=json --chrome-flags="--headless" --form-factor=mobile | jq '.environment.benchmarkIndex'

On a baseline 2021 M1 MacBook Pro, this number is around 2900.

2.2 Convert `benchmarkIndex` into a slowdown estimate

You can use the CPU Throttling Calculator to estimate the slowdown factor you need. For benchmarkIndex 2900, this yields 9.9. However, it later turns out that this is not enough by almost a factor of two.

Let's take a look at why this might be the case. By reverse engineering the website code, we found that this calculator uses a linear estimation based on a few calibrated points. Let's take a look at a chart of their estimation function:

The last calibrated point is around benchmark index 1300, more than two times lower than what a modern laptop is capable of. This explains why the estimated CPU slowdown of around 10 does not produce satisfactory results. In other words, it is not (yet) calibrated in the range that matters to us.

For a baseline 2021 M1 MacBook Pro, the slowdown factor that produces reasonable results sits around 19.

3. Run Lighthouse

You can run Lighthouse with a desired CPU slowdown like this:



lighthouse YOUR_WEBSITE_URL --throttling.cpuSlowdownMultiplier YOUT_CPU_SLOWDOWN --output=html --chrome-flags="--headless" --form-factor=mobile --output-path report.html

and then open up the generated HTML report.

This produces somewhat more relevant results:

The values are not the same, but we should not expect them to be — this is a complex benchmark running on a simulated device, after all. Go ahead and tweak your slowdown factor until it resembles what online PageSpeed Index test is telling you.

Here is a one-liner using a JSON report that also extracts the metrics in question:



lighthouse YOUR_WEBSITE_URL --throttling.cpuSlowdownMultiplier YOUT_CPU_SLOWDOWN --output=json --chrome-flags="--headless" --form-factor=mobile | jq -r '{LCP: .audits["largest-contentful-paint"].numericValue, "LCP Score": .audits["largest-contentful-paint"].score, FCP: .audits["first-contentful-paint"].numericValue, "FCP Score": .audits["first-contentful-paint"].score, SI: .audits["speed-index"].numericValue, "SI Score": .audits["speed-index"].score, TBT: .audits["total-blocking-time"].numericValue, "TBT Score": .audits["total-blocking-time"].score, CLS: .audits["cumulative-layout-shift"].numericValue, "CLS Score": .audits["cumulative-layout-shift"].score, "TOTAL": .categories.performance.score} | to_entries[] | "\(.key): \(.value)"'

It is not the prettiest, but it can be useful.

4. Repeat multiple times

The last thing we wanted to touch on is repeatability. Lighthouse will produce vastly different performance results each run. Therefore, we usually like to run it at least 5 times and compute the averages. This allows us to have a better sense of whether we are moving in the right direction.

The gist here contains a script that does just that for you.

5. Dive deeper

The Lighthouse JSON report contains much more information that the HTML one, so it is worth examining a bit. It contains all the reasons why the webpage might not be performing well. One of the more useful things we found is the largest-contentful-paint-element, which tells you which element you should optimize in order to achieve a better LCP score.

Let's get back to work

Well, now the hard part begins - improving the webpage performance. However, do not forget that Lighthouse is a (very relevant) made-up metric - in reality, you need to optimize your webpage for real-world users. And web crawlers, but we will get into the SEO debate some other time.

This blog post was written by the awesome team at zerodays.dev.

How Our Infrastructure Supports Last-Minute Studying

Žiga Patačko Koderman — Thu, 04 Jul 2024 08:15:21 +0000

The past few weeks, one of our clients, Astra AI - an AI powered math tutor, has been recording a steep increase in traffic. This makes sense, as in June, students were frantically studying to improve their final grades right before the school year ended and preparing for the Slovenian national high school final exam - Matura.

Although it’s been years since our teachers warned us not to study in the last days and avoid cramming, the data shows that this is still the case. Well, little has changed since our study days (not that we listened to teachers back then either, of course).

But this time we are experiencing this phenomenon from a completely new perspective - intensely studying in the last few days before the exam means a sudden increase in traffic. The chart below represents the number of OpenAI tokens used by Astra per day:

We won’t be sharing the absolute numbers of course, but suffice to say that we believe this to be one of the biggest usages in our region.

This was Astra’s first time through such (admittedly expected) load increase. However, we were fairly confident in our infrastructure and did nothing special in advance to handle this. And in fact no technical issues arose.

Since the peak, we’ve analyzed the logs, reviewed the data, and attributed this to a few key factors:

Some clever load balancing between multiple API keys in order to climb OpenAI's tier ladder.
Great choice of hosting providers allowing for easy elastic scaling, namely:
- Vercel for the Next.js frontend and part of the backend. The killer features for us include autoscaling, CDN and instant rollbacks out of the box 📦.
- Railway.app for the background machinery handling the more complicated requests.
Great choice of monitoring and analytics tools (Sentry, Axiom, Posthog and Uptime Kuma) coupled with amazing Slack integrations that allowed us to iron out any issues way before the traffic spike while the troubling features were still fresh from the oven.

As we could talk about our infrastructure choices for days, we decided to keep this post short and simple but are planning on doing more of a deep dive in one of the future posts, so keep an eye out for that.

This blog post was written by the zerodays.dev team.

The Hunt for a Perfect Headless CMS

Žiga Patačko Koderman — Wed, 03 Jul 2024 06:57:50 +0000

We recently decided to leave Webflow (the reasons why will be further explained in our next blog) and migrate our company website to a headless content management system. For that reason we decided to thoroughly research the options available and select the best solution for us.

🗒️ Requirements (and wishes ✨)

Our main goal was to select a suitable headless CMS for managing the content of a NextJS-based landing page, allowing for full customization and seamless integration. Based on that we developed a list of requirements for the system we wanted, not just for our website, but also for the majority of other landing pages we develop for our clients.

Must have:

Localization support (RTL support in editor is a bonus)
Good integration with NextJS:
Good React integration.
Possible server-side (pre)rendering.
Supports custom sections or components to emulate a purposely primitive website builder.
Rich text editor support.
User-friendly admin interface.
SEO support.
Multi-user support.
Media hosting.
Easy backup and restore.
Good reputation.

Nice to have:

Open source.
Ability to self-host.
Support for drafts and versioning
Live preview inside the content editor.
Admin panel customization options.
A/B testing support.

Other considerations:

Migration challenges from the existing website.
Multi-region hosting and caching (can be done on NextJS side).

Based on those requirements we tested and considered lots of CMS options. Let’s jump straight into why the Payload ended up our first choice.

🥇 The winner - Payload CMS

Reasons for Selection:

Excellent integration with NextJS. Latest version (3.0) is now a part of a NextJS application.
Native localization support where localization can be done per field, instead of only per-block.
Open source and widely used.
Ability to be self-hosted.
Admin dashboard is easy to extend with custom React components.
Easy to extend with custom functionalities since CMS code is co-located with the frontend.
Native TypeScript support.
Live editing preview option.

Cons:

The version 3 that includes all the sweet features we desire is still in beta - this is why we are using it only for our website at the moment and not experimenting on our clients.

🥈 Other CMS systems considered

Below is the list of all the other options we’ve tested. This is not to say that none of those could work for us - Strapi was extremely close too, and others might be best suited for your specific use case.

Strapi

Pros:

Open source.
(Probably) the most popular headless CMS option.
Self-hosted easily.
Official plugin to generate Swagger documentation so NextJS integration is trivial (although generated schema doesn’t include localization parameters).
Native localization support
Customizable admin dashboard.

Cons:

Locales are still separate, when translating content all fields need to be set in all locales instead of preserving some fields (images, slugs, etc.). This might cause some friction for editors.
Poor TypeScript support.

WordPress

Pros: Very popular option, familiarity.

Cons: Requires additional plugins and configuration to achieve full headless functionality. Worse NextJS integration and development experience. Forces you to use different technologies on frontend and backend.

Sanity

Pros: NextJS support, UI editor integration, flexible localization methods.

Cons: No self hosting, high price (per user, per request and bandwidth).

Prismic

Pros: NextJS support, live preview editor.

Cons: No self-hosting, clunky localization (all locales are totally separate), restrictive pricing plans (max 8 locales in platinum plan).

Tina.io

Pros: Simple, developer friendly, has live editing capabilities.

Cons: Poor i18n support, needs to be handled by developers.

Directus

Cons: More a dataset editor than a CMS.

Contentful

Cons: Limited language support without enterprise tier.

KeystoneJS

Pros: Seems inactive, development progress on the official website was not updated since 2022.

Contentstack

Cons: CMS is a minor part of the product. Company’s focus is elsewhere.

Storyblok

Cons: Expensive. Subpar developer experience.

🏁 Conclusion

In the end, Payload took the crown for us. What ultimately sold us on it, was its superb integration with Next.js - our technology of choice for web development.

This blog and its underlying research was made by the awesome team at zerodays.dev.

All Paths Lead To Jira

Žiga Patačko Koderman — Wed, 22 May 2024 07:03:24 +0000

zerodays’ journey through the world of project management tools

Right after we started our company in late 2021, we knew a good project management tool will be absolutely essential for our work, but we had some specific requirements. We do a lot of agency work, developing for multiple different clients at the same time so the tool had to be amazing for cross project planning, as well as enable some type of transparent sharing of the project progress with the client.

Now, almost two and a half years later and with four tools tested in detail, few others considered and most of them tossed, we feel like we finally landed on a good option for us - Jira. But, as this was definitely not a smooth path, we wanted to do an overlook of everything we have tried, why it worked and why not, in hopes it eases the process for anyone else trying to navigate the world of project management tools.

What methodology do we follow?

The methodology we follow of course defines the tools needed for our project management, so let’s start with that. We follow agile principles, but no specific methodology strictly, however, we use Kanban boards. So, we were looking for a tool that would offer us flexible and user-friendly Kanban boards and would allow us to plan as well as track the time spent on various tasks/tickets.

Trying out the Tools

GitHub Issues and Projects

Our first project management tool we took on was GitHub Projects. We knew and loved GitHub Issues from the open source projects and believed that Projects would be the tool that would fit us. However, unfortunately, it left quite a lot to be desired.

What we disliked:

All the project management is connected only to the repository. So if your project exists in two separate repositories (frontend and backend for example), you have to make two separate tickets instead of one and cross linking through the repositories is not a practical solution.
It isn’t really a fit for our office team, as the non-development tasks don’t fit in, and it’s expensive as a project management tool.
Every ticket you create is a draft by default (this isn’t really a deal-breaker on its own, but it definitely makes the user experience a lot less enjoyable).
There is no hierarchy when it comes to tickets, which makes having an overview of the issue timeline, the main issue, the sub issues etc. difficult.
Poor time tracking.

So after a while, we made the decision to move on and try out Clickup 2.0.

ClickUp

ClickUp is advertised as an “all in one productivity platform” which drew us to it. It definitely solved most of the issues we mentioned with GitHub, such as cross project tracking and ticket hierarchy, but unfortunately we still weren’t completely satisfied.

What we disliked:

Firstly, the app had a lot of bugs and issues, which made it difficult to use. For example, often when creating a ticket, it didn’t show up until a couple of minutes later, when two same tickets popped up. Slow loading times were also a common issue.
The layout and organization of the platform was in our opinion a bit cluttered, so the project tracking and overlook of the projects is messy.

We have to mention that the bugs mentioned above are supposedly resolved in the 3.0 version, however as the app didn’t convince us, we decided not to stay aboard, and jumped ship to Notion.

Notion

Notion is an everyday organization tool, combining project management tools with wikis, docs, planning tools, etc. Its customizability and the lure of having one place for the documentation and project management are what convinced us to try it. We used it for most of our development and non-development documentation and tried out two Kanban boards - a free Kanban template offered by the platform, and our own customizable Kanban board. What we liked about the platform was the easy addition of external guests and sharing the projects with people outside of the company, and good tools for non-development related team collaboration.

What we disliked:

Although Notion is quite known to be very customizable for everyday use, it was too limited for our development team. We specifically had issues with the Kanban boards - they were too rigid. Both the template and our own version lacked the ability to move, rename columns, add filters, automatically hide done tickets, etc.
Poor GitHub integration.
Sometimes it doesn't work - some tickets are not saved on the first try and other bugs.
Similar to GitHub - there was no ticket hierarchy, which made the project overlook difficult.
Columns don’t have an option to hide tickets, so it gets cluttered.
Project planning and filter options were quite limited. There was also no option to integrate it with our time-tracking tool.

With enough experience to finally know what we like and what not, we started considering our final move to either Linear or Jira.

Jira

In the end we settled on Jira, because of the configurability of the tool and integrations it offers, specifically with Slack, Figma and GitHub, the last one being absolutely amazing. So far, it solves the majority of the issues we had with other tools, most notably, it is automated where it can be, which makes a more hassle-free user experience for the development team.

What do we still dislike:

Although we love the configurability it offers, it is quite difficult to set up everything - it is confusing, cluttered in different places, and the learning curve is very steep. On the good side, it is only a pain for the admins, not the developers.
It doesn’t really offer good template options. The way around we found is that we have a project that we named template and we then initiate new projects based on it.. The downside is we still have to manually set up filters and settings, so it takes a while. Again, thankfully it is the issue only for the admins.
We had to make our own integration for the time tracking software we use - Clockify. There are some apps that offer some integration, but we wanted time tracking to be synced with Jira’s own time tracking. However, with Jira, we at least had an option to do this ourselves.

Wrapping it up

We also considered Linear and although it seemed to cover most of our requirements, we decided to go for Jira because of the integrations it offers. In all honesty, we have to admit that we might have not considered it enough and will maybe return back to it at some point, but a decision had to be made.

So, here we are at last - Jira. Is this the best solution for eternity? Probably not. But, we are happy at the moment, as we configured it to our specific needs and it doesn’t restrict our developers, while being fairly automated, so the hassle of moving tickets etc. is minimized.

zerodays is a custom software development team based in Ljubljana, Slovenia. We focus on AI development and consulting, web and mobile app development, business digitalization, and process automation solutions. If you’re interested in what we do and want to keep up with us, visit our website or follow us on LinkedIn.

Generative UI in React Native

Žiga Patačko Koderman — Thu, 11 Apr 2024 11:09:47 +0000

Let's build a generative UI Weather Chatbot!

This article is inspired by Vercel's Generative UI. It expands on OpenAI's functions/tools by allowing the model to visualize data to the user.

Let's get straight into it.

Create an Expo Typescript app 📱

npx create-expo-app -t expo-template-blank-typescript demo
cd demo

Install 📦

yarn add react-native-gen-ui zod

This adds the react-native-gen-ui package that offers:

Exposes useChat hook for easy access to OpenAI's completions api.
Supports streaming.
Enables Generative UI via tools.
Is fully type-safe.

The package was initially developed by us at zerodays.dev and was open-sourced for everyone to use.

We also install zod here for validation of tool parameters.

Import dependencies

At the top of the App.tsx add

import { z } from "zod";
import { OpenAI, isReactElement, useChat } from 'react-native-gen-ui';

Initialize OpenAI

Below imports, write:

const openAi = new OpenAI({
  apiKey: process.env.EXPO_PUBLIC_OPENAI_API_KEY!,
  model: 'gpt-4',
});

Then make sure to add EXPO_PUBLIC_OPENAI_API_KEY to your environment or in .env file. You can obtain an API key in the OpenAI Platform.

Utilize `useChat` hook.

At the top of the App function, let's use the useChat hook.

export default function App() {
  const { input, onInputChange, messages, isLoading, handleSubmit } = useChat({
    openAi,
    initialMessages: [
      { content: "You are a nice little weather chatbot.", role: "system" },
    ],
  });

  return <View/>;
}

Here we are passing initial messages into the useChat hook, then allow it to manage the state of messages, stream the content from OpenAI and update the UI.

But we are still missing a way to render this - make the function return the snippet below:

return (
  <View
    style={{
      flex: 1,
      backgroundColor: "#fff",
      alignItems: "center",
      justifyContent: "center",
    }}
  >
    {/* Iterate over all messages and render them */}
    {messages.map((msg, index) => {
      // Message can be either a React component or a string
      if (isReactElement(msg)) {
        return msg;
      }
      switch (msg.role) {
        // Render user messages in blue
        case "user":
          return (
            <Text style={{ color: "blue" }} key={index}>
              {msg.content?.toString()}
            </Text>
          );
        case "assistant":
          // Render assistant messages
          return <Text key={index}>{msg.content?.toString()}</Text>;
        default:
          // This includes tool calls, tool results and system messages
          // Those are visible to the model, but here we hide them to the user
          return null;
      }
    })}
    {/* Text input for chatting with the model */}
    <TextInput
      style={{
        borderColor: "gray",
        borderWidth: 1,
        textAlign: "center",
        width: "100%",
      }}
      autoFocus={true}
      value={input}
      onChangeText={onInputChange}
    />
    <Button
      onPress={() => handleSubmit(input)}
      title="Send"
      disabled={isLoading}
    />
  </View>
);

Then run the app using npx expo start. More details about running an Expo app can be found here.

Voilà - we can have a basic chat with 🤖 now!

Add a tool

Now the real magic begins 🪄. A tool is defined by:

its name,
description (a simple string for model to understand what this tool does),
parameters (a type-safe zod schema of what this tool accepts as arguments) and
a render function for handling what both the user and model see when this tool is called.

Let's add a “get weather tool“ to tell us what the weather is like at a certain location:

const { ... } = useChat({
  ...
  tools: {
    getWeather: {
      description: "Get weather for a location",
      // Parameters for the tool
      parameters: z.object({
        location: z.string(),
      }),
      // Render component for weather - can yield loading state
      render: async function* (args) {
        // Fetch the weather data
        const data = await getWeatherData(args.location);

        // Return the final result
        return {
          // The data will be seen by the model
          data,
          // The component will be rendered to the user
          component: (
            <View
              key={args.location}
              style={{
                padding: 20,
                borderRadius: 40,
                backgroundColor: "rgba(20, 20, 20, 0.05)",
              }}
            >
              <Text style={{ fontSize: 20 }}>
                {data.weather === "sunny" ? "☀️" : "🌧️"}
              </Text>
            </View>
          ),
        };
      },
    },
  },
});

This allows the model to call the getWeather function. The model receives the data and can comment upon it, while the user is presented with an emoji representation of the weather (either ☀️ or 🌧️).

The getWeatherData function is still missing, let's add it:

async function getWeatherData(location: string) {
  // Wait 3 seconds to simulate fetching the data from an API
  await new Promise((resolve) => setTimeout(resolve, 3000));

  // Randomly return either `sunny` or `rainy`
  return {
    weather: Math.random() > 0.5 ? "sunny" : "rainy",
  };
}

The result is a chat interface that goes beyond just words and can display custom components (even though, this is just emojis in our example).

Show loading states

The react-native-gen-ui allows us to yield zero or more components before returning the actual data and the final component. This can be used to display progress to the user until the data is fetched.

Add the following code at the beginning of render:

render: async function* (args) {
  // Yield a loading indicator
  yield <ActivityIndicator key="activity-indicator" />;

  // Fetch the weather data
  ...

The render function is in fact a generator - this is denoted by the * after the function keyword. The interface will always display the last component that was either yielded or returned, allowing us to swap what the user sees over time.

This can be especially useful if the the render function takes a lot of time and has multiple steps. The user can stay informed of whatever is happening in the background. For example, one could pre-render partial information before the entire data pipeline completes.

References

📦 react-native-gen-ui
A minimal example like the one in this post.
A complete weather bot example from the GIF at the top.

This blog post and the react-native-gen-ui package were made by the awesome team at zerodays.dev.

DEV Community: Žiga Patačko Koderman

Lighthouse vs. PageSpeed Insights: The Lack of Correlation

What's the problem?

The reason for the discrepancy

Solution

1. Install the Lighthouse CLI

2. Estimate the slowdown factor (optional)

2.1 Extract benchmarkIndex from the report

2.2 Convert benchmarkIndex into a slowdown estimate

3. Run Lighthouse

4. Repeat multiple times

5. Dive deeper

Let's get back to work

How Our Infrastructure Supports Last-Minute Studying

The Hunt for a Perfect Headless CMS

🗒️ Requirements (and wishes ✨)

🥇 The winner - Payload CMS

🥈 Other CMS systems considered

🏁 Conclusion

All Paths Lead To Jira

zerodays’ journey through the world of project management tools

What methodology do we follow?

Trying out the Tools

GitHub Issues and Projects

ClickUp

Notion

Jira

Wrapping it up

Generative UI in React Native

Create an Expo Typescript app 📱

Install 📦

Import dependencies

Initialize OpenAI

Utilize useChat hook.

Add a tool

Show loading states

References

2.1 Extract `benchmarkIndex` from the report

2.2 Convert `benchmarkIndex` into a slowdown estimate

Utilize `useChat` hook.