DEV Community: Paige Niedringhaus

Reviewing the BenQ RD Series Monitor for Developers

Paige Niedringhaus — Mon, 08 Sep 2025 12:46:00 +0000

I didn’t think developers needed a special monitor, until I tried the BenQ RD280U.

Like many developers, I began working from home full time when the pandemic hit and the office I commuted to every day was forced to close. My initial WFH setup was at the dining room table with my 16" MacBook Pro, a leftover 18" Dell monitor as my second screen, and an old Microsoft wireless keyboard.

v0 of the work from setup. It came from humble beginnings.

As the weeks turned into months and it became apparent no one would be going back to an office anytime soon, I began to slowly but steadily build a solid work from home station: laptop stand to raise my laptop to eye level, adjustable desk, extra office chair in the garage.

Then mid-pandemic, I joined the fully remote tech startup I work for now, Blues, and it was game on: upgraded Logitech wireless keyboard, dual 27" HP monitors, upgraded office chair (with the rollerblade wheels, I might add), Thunderbolt docking station, external microphone, key light, and Elgato prompter (for the weekly web dev podcast I started with my friends), and for quite a while I've felt very dialed in at this command station each day. After 5 years of incremental upgrades, you'd hope that would be the case.

The v1 wfh command station I've been perfecting for years now.

About two months ago, out of the blue, the folks at tech product company BenQ reached out to me asking if I'd be interested in trying out one of its RD series programming monitors: a product line designed with developers' needs in mind. Having heard good things about BenQ, but never actually trying their products, I said "Sure!", and received a 28", 4K RD280U monitor shipped to my front door about a week later.

I was a little dubious some of the monitor's claims like coding-optimized color modes and auto dimming features for low-light environments were just marketing fluff, but once I tried it out, boy was I wrong.

Today, I'm going to review the BenQ RD280U 28" 3:2 Monitor for Programming, and tell you how it's improved my day-to-day coding workflow in ways I never anticipated.

Initial assembly and setup of the RD280U was a breeze

The RD280U ships in a what seems like large box, even for a 28" monitor, but all the pieces are well organized and protected, and my assembly of the monitor was done within minutes following the detailed illustrations in the Quick Start guide included in the box.

The box is quite large even though it's not actually that heavy. Based on its bulk, I would recommend you find a second person to help you get it inside and near to where you plan to assemble and use it, if possible.

The screen itself can tilt between -5˚ - 20˚ up and down, swivel right and left 15˚ , and adjust the height on its stand by 4.3" (110mm).

It comes packaged with HDMI, USB-C, and USB-B cords for connecting to your laptop, and also has a display port as well, if that's your preferred connection method. One pleasant surprise for me was the lack of an external power pack that I'd have to secure somewhere behind my desk. Instead there's simply a power cord that runs from the monitor to an outlet.

Additionally, there's a flexible arm version of the monitor (the RD280UA) you can choose instead of the stand the RD280U comes with if you want to clamp it to your desk instead.

Noteworthy RD280U features for devs

Ok, let's talk about the things that developers care most about: the monitor's features! The RD280U has a lot of them.

"Coding Mode"

The BenQ RD series monitors have a feature called "Coding Mode", which is a series of specialized display settings aimed at providing the best possible text clarity for programmers. Coding Mode adjusts settings to sharpen text and reduce eye strain, and has both Dark and Light themes depending on devs' user preferences.

I'm Team Dark Mode all day, by the way.

Below are more specifics on how Coding Mode enhances a typical coding session.

Higher text clarity

This RD monitor has a max resolution of 4K+ UHD, which means four times more pixels and detail than a 1080p display. What this translates to in practice, is less eye strain when looking at code for hours on end. Coding Mode takes this a step further by fine-tuning the contrast, brightness, sharpness, saturation, and gamma settings levels for text, enhancing the overall sharpness and readability of code, even during long working sessions.

Personally, I can see a difference when I compare two IDE windows side-by-side on my existing HP monitor vs the new BenQ monitor: the BenQ monitor does make the code sharper and easier to read.

Anti-reflection screen properties

Until recently (when I was traveling for work and working from coffee shops and hotel cafes), I didn't think too much about screen glare and light reflection. It must just be where my office is set up at home, but it wasn't really an issue for me.

Working remotely, however, screen glare was an issue I had to contend with and the BenQ Nano Matte Panel technology that actively minimizes contrast glare on the screen would have been most welcome.

If you have lights set up behind you that cause reflections on your monitor or even just windows that let the sunlight in during a typical day, you'll probably really appreciate this feature as well.

The Function Bar

An addition the RD monitors include that is very geared towards developers is the addition of a Function Bar at the bottom center of the monitor.

It has a "Coding HotKey" on the front of the bar that looks like a closing HTML tag </>, and when touched, it provides quick access to features like coding modes, screen brightness and contrast, and a customizable "Function Key" button on the bottom righthand side of the bar that can be set to different options you might want quick access to like Color Mode or input source.

Here's a shot of the settings the Coding HotKey displays when tapped on the front of the Function Bar.

In addition to the Function Key on the far right of the Function Bar, there's also a power button for the monitor and a center button that displays all the RD monitor's settings (there's a lot of them). The settings are easy to navigate through as the buttons allow for left/right toggling through menus and the naming is also pretty intuitive.

The full settings panel is accessed by pressing the center button on the bottom of the Function Bar.

The bar also has some LED indicators on the front next to the Coding HotKey to show which settings are currently on.

It has:

Night Hours Protection indicator (an owl head), which adjusts the screen to minimum brightness for coding in low light or at night,
Brightness Intelligence Gen2 (a polygon-esque lightbulb), that auto detects ambient light and adjusts the screen's brightness accordingly,
And Low Blue Light Plus (a lightbulb), which reduces blue light while preserving color quality.

For me, after toggling through my screen's settings on initial setup and ensuring my Color Mode theme was set to "Coding - Dark Theme", I haven't needed to use the Function Bar or Coding HotKey much since.

I don't have multiple different computers I want to use this monitor with and switch between (although it supports that!), nor do I feel the need to change my Color Mode theme frequently, but for some devs this may be of greater value.

MoonHalo backlight

Another interesting feature I hadn't consciously considered but really like, is the backlight "MoonHalo" built into the RD280U.

It's essentially a ring light built into the back of the monitor that can be set to turn on and provide ambient light while working. It can also auto dim to ultra low brightness with color-balance based on how dark your work surroundings are, which is influenced by the sensor in the Function Bar mentioned above that automatically detects and adjusts the display brightness based on the ambient brightness throughout the day.

The auto dimming feature, I appreciate a lot. It makes me feel less like I'm working in a cave at my desk after dark.

Here's an angled shot of what the MoonHalo looks like on the back of the RD280U monitor. At this time of day, it's at a low brightness level.

While the MoonHalo certainly doesn't take the place of a key light if you do any video recording or streaming, it is nice for providing some extra ambient light and offering adjustable color temperature without needing a separate lamp, and doesn't do the "blinding white screen at night scenario" when the monitor screen is set for daytime brightness conditions by default.

Additional monitor features

Those are some of the technologies and thoughtful additions that the RD series monitors offer specifically for programmers via Coding Mode, but there's some additional things worth pointing out in this review as well, that may just sell you on the BenQ RD monitor otherwise.

Display Pilot 2 software

The BenQ line of monitors comes with its own software control app called Display Pilot 2, to let users efficiently control their display settings on their computer.

The app offers a software interface for adjusting many of the settings I outlined above: audio levels, color mode, screen brightness, MoonHalo, switching screen orientation, and so on.

Honestly, until I was writing this review, I hadn't even bothered to download the Display Pilot 2 software to my Mac, and while it provides a nice looking interface to control all the bells and whistles from my computer, I don't see myself using it often after I've dialed in my initial settings.

I should also note that while I was messing with the settings via Display Pilot 2, the software crashed a couple of times and caused my BenQ RD monitor to blink on and off more than once, so your mileage may vary when using it.

Multi-source support for seamless source switching

A lot of developers have different rigs for different purposes: a gaming PC, a MacBook work, a personal laptop, or some other combination of devices.

In the past, it's always been kind of a a pain to switch one machine for another to use the same peripherals (keyboard, external monitor, mouse, headphones, etc.).

Well, the BenQ RD monitor can support more than one machine at a time. It has an HDMI port, display port, two USB-C ports, 1 USB-B port, 1 USB-A port, and a headphone jack. It also has a built-in KVM (keyboard, video, mouse) switch to make shifting from one machine to another easy.

Simply connect Computer 1 and Computer 2 to the RD monitor, and both will be identified. Once both are recognized, you can switch the input video source from Computer 1 to Computer 2, and the monitor will also switch the USB signal to computer 2 as well. No muss, no fuss.

I have not tested this feature out myself, as I do not use multiple laptops with my current setup, but I trust the documentation the BenQ website provides about the KVM switch.

My favorite monitor feature (I didn't see this one coming tbh)

By far my favorite feature of this new RD monitor is (and I'm as shocked about this as anyone else considering all the cool tech I've covered up to now): its 3:2 aspect ratio design!

Aspect ratio? Really?? That's what you're most jazzed about???

The 3:2 screen aspect ratio

My friends, it is.

I (apparently) am so used to the more standard 16:9 aspect ratio most monitors come with, that after a few days of adjusting to this new, bigger screen where I can see more lines of code or more info in my browser's dev tools without having to scroll, I'm surprised no one really talks about how much of a difference it can make when evaluating external monitors.

According to the BenQ website, this RD280U screen shows 7 more lines of code, but I tell you that for me it is quite a bit more than 7 lines, and I absolutely love it. And all this without me even flipping the monitor into a vertical orientation!

I am very seriously considering getting another RD280U just so I can have this much extra screen real estate for both my external displays at home.

Just look at the amount of code visible on my old HP monitor (right side) vs the amount of code visible on the new RD monitor (left side). It's like night and day.

Conclusion

As humans, we adapt quickly to our current conditions until we get so used to it that they become "normal" to us. Another term for this is habituation.

Over the years, I've developed what I consider a very effective and comfortable work from home setup: laptop stand, external monitors, keyboard and mouse, ergonomic desk chair, and even a small teleprompter screen, and I was quite content with it.

Then, BenQ reached out to me with the chance to try one of its RD programming series monitors, and my eyes were opened to features I hadn't even considered I needed as a developer, but made the daily process of writing and reviewing code for hours so much nicer.

A dedicated "Coding Mode" (with both dark and light modes) that enhances text clarity and contrast, anti-reflective screen, a backlight built in to the monitor itself, auto dimming software measuring ambient light, and a 3:2 aspect ratio that dramatically increased my screen real estate.

None of these features were things even on my radar before the BenQ RD280U, but now, I have a hard time imagining working without them.

If you're considering an external monitor to enhance or upgrade your own coding practice, I would strongly encourage you to consider the BenQ RD280U or its other RD series monitors built specifically with developers and programmers in mind. I was skeptical at first that a monitor could really improve my day-to-day workflow, but I was wrong.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

If you'd like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading, I hope you’ll check out the BenQ RD line of monitors the next time you need a new monitor. BenQ took the time to understand what improves the developer quality of life, and built a monitor with that really raises the bar with a ton of useful features.

References & Further Resources

Build a Custom Time Slider Component with Ant Design and Next.js

Paige Niedringhaus — Fri, 24 Jan 2025 00:00:00 +0000

Introduction

Part of my job as a software engineer working for a startup involves building proof of concept (POC) web apps quickly, and one of my preferred tech stacks for such a project is a Next.js application in conjunction with the Ant Design component library.

I'm a fan of Ant Design because it's a library I'm familiar with, and it offers a lot of more complicated components (think filterable tables, nested trees, complex forms, date pickers, and so on) that are highly customizable and just work right out of the box.

When I was tasked recently with building a web app that contained a time slider akin to the adjustable sliders you see in interactive weather maps or other time series data applications that would allow a user to adjust the visible time range within a predetermined set of dates, because I couldn't find a suitable, React-based component or package already available, I ended up modifying a basic Ant Design slider component to suit my needs.

My time slider component modifies its tick mark date formatting appropriately based on how big or small the original time span is, it allows users to drag the slider from either end to adjust the currently selected date range, it allows for date ranges set in the future, and it even delineates on the slider between past dates and future dates with different colors and styling.

Today, I'll show you how to build your own custom time slider inside of a Next.js application using the Ant Design component and the Day.js date formatting library.

Here's what the final adjustable time slider component looks like and a link to the live demo site in CodeSandbox.

Install Ant Design and Day.js

The two libraries you'll need to add to a React-based application to build this custom time range slider are the Ant Design library and the Day.js library.

Run this command in the terminal in the root of your project to add them both.

npm install antd dayjs

As I said before, Ant Design is a great, full-featured, very well documented component library I've worked with many times in the past and it makes building complex web apps that contain elements like drawers, cards, cascading selects, and a host of other components much quicker and easier to get going.

Ant Design uses Day.js by default, and Day.js is a lightweight, JavaScript library that parses, validates, manipulates, and displays dates and times with a largely Moment.js-compatible API. If you've ever used Moment to handle dates in a JavaScript application, you'll feel right at home with Day.js, with the added bonus that it's a fraction of the size of Moment.

Ant Design Slider

One particularly useful component that Ant Design offers is a <Slider /> component with a lot of customizable functionality not many other sliders contain.

For my app, I needed a slider that could:

Select a range within the slider's bound (instead of being anchored to one end of the slider or the other),
Allow that range track to be dragged within the slider.
Offer customizable tick marks, tooltips, and range track styles.

Day.js

Working with dates in JavaScript applications is still quite the pain, even today without a supporting date library. The Day.js library is a faster, smaller alternative to Moment.js, an extremely popular library for handling dates and times, with a very similar API and great internationalization support.

To make dates work for my app with the Ant Design slider, I needed a library that could:

Transform human readable dates like "2024-12-01" into Unix millisecond timestamps, because the slider expects numbers (not date formatted objects) to be passed to it in order to work correctly
Format those same dates appropriately for tick marks of varying date spans
Format said date for hoverable tooltips as well.

The Day.js library is capable of all this and more with simple, easy to understand functions and great documentation, and since it's the default library for Ant Design anyway, it make sense to just fall in line with what Ant D knows best.

Create a TimeSlider component

Once those two libraries are installed in the project, it's time to move on to actually creating the <TimeSlider /> component. Below is the starting code for the component. I'll go through all the pieces that comprise it afterwards.

import { Col, Row, Slider } from "antd";
import dayjs, { Dayjs } from "dayjs";

type TimeSliderProps = {
  outerDateRange: { from: Dayjs; upto: Dayjs };
  innerDateRange: { from: Dayjs; upto: Dayjs };
  onChange: (innerDateRange: { from: Dayjs; upto: Dayjs }) => void;
};

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
  const min = outerDateRange.from.valueOf();
  const max = outerDateRange.upto.valueOf();
  const values: [number, number] = [
    innerDateRange.from.valueOf(),
    innerDateRange.upto.valueOf(),
  ];

  return (
    <div>
      <Row>
        <Col span={24}>
          <Slider
            range={{ draggableTrack: true }}
            min={min}
            max={max}
            value={values}
            onChange={(value: number[]) => {
              onChange({
                from: dayjs(value[0]),
                upto: dayjs(value[1]),
              });
            }}
          />
        </Col>
      </Row>
    </div>
  );
};

export default TimeSlider;

I created a new file called TimeSlider.tsx where all this code is going to live, and made a basic shell of the component that accepts the props outerDateRange, innerDateRange, and onChange() from the parent component. Later on in this article I'll show you how to set up the required state and functions in the parent component to pass down to this component.

As this is actually a TypeScript file, this component also gets its own TimeSliderProps types defined for each of the props being passed to it. This is where the Day.js library initially comes into play as well, as the type for outerDateRange, innerDateRange, and even the onChange() function expect Day.js dates to be passed to them.

NOTE: All of my JavaScript-related code is actually written in TypeScript, so if you prefer to use JavaScript, just know that it can be fairly easily adapted to plain JS by removing things like types from the components and variable declarations.

Inside of this component we're going to set a bunch of variables required by the Ant <Slider /> component to render properly.

A min and max variable to tell the slider what it's beginning and ending values are.
A values object that is an array of the innerDateRange's from and upto values.

NOTE: For the range slider to work as you'd expect with date objects, the dates must be converted to Unix Epoch time, that is why you'll notice both the outerDateRange and innerDateRange dates are being called with the Day.js library's valueOf() function. This takes the date and turns it into a Unix Timestamp in milliseconds, which can then be passed to the <Slider/> component.

This is the bare minimum of data the Ant Design component will need to actually render onscreen. Then, it's time to create the component with the help of the Ant Design <Slider/> element and pass the necessary details to it.

For a bit of nicer formatting, I also employed Ant Design's <Col/> and <Row/> elements, but those are not required for this component to work. They help the component to lay out in a horizontal fashion and take up as much space as the screen width will allow.

The <Slider/> component itself requires the min and max variables, the values array, the onChange() function, and in order to make it a range slider whose track can be dragged, the range={{draggableTrack: true}} value must be added to the slider's props.

If this component were to receive valid dates right now, it should render, albeit without much to see. Let's keep going.

Calculate and style the tick marks to handle different date ranges appropriately

The next step is to calculate the tick marks that will appear along the slider's track showing the range of the slider.

Here's the code that will get added to the TimeSlider.tsx component to make that happen.

import { Col, Row, Slider } from "antd";
import { SliderRangeProps } from "antd/lib/slider";
import dayjs, { Dayjs } from "dayjs";

type TimeSliderProps = {
  outerDateRange: { from: Dayjs; upto: Dayjs };
  innerDateRange: { from: Dayjs; upto: Dayjs };
  onChange: (innerDateRange: { from: Dayjs; upto: Dayjs }) => void;
};

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
  const min = outerDateRange.from.valueOf();
  const max = outerDateRange.upto.valueOf();
  const tickCount = 15;
  const tickInterval = Math.floor((max - min) / tickCount);
  const values: [number, number] = [
    innerDateRange.from.valueOf(),
    innerDateRange.upto.valueOf(),
  ];

   const fmt = (date: number) => {
    const d = dayjs(date);
    const { from, upto } = outerDateRange;
    const range = dayjs(upto).diff(dayjs(from)); // Calculate time difference in milliseconds

    // Pick an appropriate level of granularity for the date range we're viewing
    if (range < 1000 * 60 * 60 * 24) return d.format("hh:mm A"); // range is less than a day, show hours and minutes

    if (range < 1000 * 60 * 60 * 24 * 7) return d.format("M/D, HH:mm"); // range is less than a week, show day of the week

    if (range < 1000 * 60 * 60 * 24 * 30) return d.format("M/D"); // range is less than a month, show month / day

    if (range < 1000 * 60 * 60 * 24 * 365) return d.format("MMM D"); // range is less than a year, show month and day

    if (range < 1000 * 60 * 60 * 24 * 365 * 10) return d.format("MMM YYYY"); // range is more than a year, show year and month

    return d.format("YYYY"); // range is more than 10 years, show year
  };

  const marks: Record<number, { label: string; style?: React.CSSProperties }> =
    {
      [min]: { label: fmt(min) },
      [max]: { label: fmt(max) },
    };

  // add ticks to marks
  for (let i = min + tickInterval; i <= max; i += tickInterval) {
    marks[i] = {
      label: fmt(i),
    };
  }

   return (
    <div>
      <Row>
        <Col span={24}>
          <Slider
            range={{ draggableTrack: true }}
            min={min}
            max={max}
            marks={marks}
            value={values}
            onChange={(value: number[]) => {
              onChange({
                from: dayjs(value[0]),
                upto: dayjs(value[1]),
              });
            }}
          />
        </Col>
      </Row>
    </div>
  );
};

export default TimeSlider;

First a couple of new variables are introduced at the top of the component: tickCount and tickInterval.

tickCount is the number of ticks that should be displayed on the slider.
tickInterval is a value calculated by determining how far away from each other the min and max dates are in time and then dividing that difference by the tickCount number to figure out how to evenly space the ticks along the slider, regardless of what the actual date values are.

Then, I created a function called fmt() which handles the heavy lifting of formatting the dates associated with the tick marks appropriately based on how far apart in time the dates actually are.

   const fmt = (date: number) => {
    const d = dayjs(date);
    const { from, upto } = outerDateRange;
    const range = dayjs(upto).diff(dayjs(from)); // Calculate time difference in milliseconds

    // Pick an appropriate level of granularity for the date range we're viewing
    if (range < 1000 * 60 * 60 * 24) return d.format("hh:mm A"); // range is less than a day, show hours and minutes

    if (range < 1000 * 60 * 60 * 24 * 7) return d.format("M/D, HH:mm"); // range is less than a week, show day of the week

    if (range < 1000 * 60 * 60 * 24 * 30) return d.format("M/D"); // range is less than a month, show month / day

    if (range < 1000 * 60 * 60 * 24 * 365) return d.format("MMM D"); // range is less than a year, show month and day

    if (range < 1000 * 60 * 60 * 24 * 365 * 10) return d.format("MMM YYYY"); // range is more than a year, show year and month

    return d.format("YYYY"); // range is more than 10 years, show year
  };

The fmt() function takes in a date (which will eventually be a tick mark), translates it to a Day.js date (d) that can be easily formatted as needed, figures out the range between between the min and max dates passed into the component from its parent, and then formats the d date according to how wide the total timeline range is.

If range is less than a day, the tick marks will be styled to show hours and minutes, if the range is less than a week, it will show day of the week plus hours and minutes, and so on as the range increases. It takes a little bit of effort to decide when it makes sense to switch from one tick display format to the next, but it ends up working out pretty well in practice. Feel free to adjust your formatting of different date ranges to suit your needs.

Next I created the marks that will get passed to the Ant Design slider component.

These marks are created in two parts.

  // marks for each end of the time slider component
  const marks: Record<number, { label: string; style?: React.CSSProperties }> =
    {
      [min]: { label: fmt(min) },
      [max]: { label: fmt(max) },
    };

  // add ticks to marks
  for (let i = min + tickInterval; i <= max; i += tickInterval) {
    marks[i] = {
      label: fmt(i),
    };
  }

The first half of the code, where const marks is declared, is where the marks for each end of the slider are created. Each end's date is passed to the fmt() function to have it correctly rendered based on the length of the time span.

{
  [min]: { label: fmt(min) },
  [max]: { label: fmt(max) },
};

Then, each tick between the two ends is run through the formatting function as well via this for loop.

for (let i = min + tickInterval; i <= max; i += tickInterval) {
  marks[i] = {
    label: fmt(i),
  };
}

Finally, the resulting marks array is passed to the <Slider/> component, along with all the other values covered earlier.

<Slider
range={{ draggableTrack: true }}
min={min}
max={max}
marks={marks}
value={values}
onChange={(value: number[]) => {
    onChange({
    from: dayjs(value[0]),
    upto: dayjs(value[1]),
    });
}}
/>

Format the tooltip

After marks for the timeline, it's time to format the tooltip when a user hovers over either end of the range slider. This is actually one of the easier bits of code within this component as well.

Inside of the component under where all the variables are declared at the top add the following function.

import { SliderRangeProps } from "antd/lib/slider";

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
// variables declared up here

 const formatter: NonNullable<SliderRangeProps["tooltip"]>["formatter"] = (
    value: string | number | undefined
  ) => `${value ? dayjs(value).format("MM/DD/YYYY HH:mm A") : ""}`;

This formatter() function takes in the current value of the end of the range slider a user is hovering over, and reformats it to a nicely readable date.

Then call this formatter() function within the <Slider/> component at the bottom of the component file.

<Slider
range={{ draggableTrack: true }}
min={min}
max={max}
marks={marks}
value={values}
onChange={(value: number[]) => {
    onChange({
    from: dayjs(value[0]),
    upto: dayjs(value[1]),
    });
}}
tooltip={{ formatter }}
/>

Voila! Nicely formatted tooltips for the time range slider on hover.

Style the track and marks to handle current and future dates differently

Now for the extra credit additions to this time slider component: styling future dates differently within the same component. Sounds impossible? I thought so too, at first, but now I can assure you it's not.

There's two changes I wanted to implement to help users know when they were looking at dates in the past/present versus dates in the future with the time slider:

I wanted to change the color of the track from Ant Design's light blue default to a different color, and
I wanted to change change the color of tick marks in the future, as well.

Lucky for me, Ant Design allows for immense customization of its components, even down to things as specific as track and tick mark styling.

The track styling was more complicated, so let's tackle that one first.

import dayjs, { Dayjs } from "dayjs";

const TimeSlider = ({
  outerDateRange,
  innerDateRange,
  onChange,
}: TimeSliderProps) => {
  // other variables declared at top of component here
  const current = dayjs().valueOf();

  // Calculate track styles based on the current date
  const getTrackStyle = () => {
    if (values[1] <= current) {
      // All within past/present
      return {
        backgroundColor: "#91caff",
      };
    } else if (values[0] >= current) {
      // All in future
      return {
        backgroundColor: "rgb(165, 222, 129)",
      };
    } else {
      // Split between past/present and future
      const pastWidthPercentage =
        ((current - values[0]) / (values[1] - values[0])) * 100;
      return {
        background: `linear-gradient(
            to right,
            #91caff 0%,
            #91caff ${pastWidthPercentage}%,
            rgb(165, 222, 129) ${pastWidthPercentage}%,
            rgb(165, 222, 129) 100%
          )`,
      };
    }
  };

return (
    <div>
      <Row>
        <Col span={24}>
          <Slider
            range={{ draggableTrack: true }}
            min={min}
            max={max}
            marks={marks}
            value={values}
            styles={{
              track: getTrackStyle(),
            }}
            onChange={(value: number[]) => {
              onChange({
                from: dayjs(value[0]),
                upto: dayjs(value[1]),
              });
            }}
            tooltip={{ formatter }}
          />
        </Col>
      </Row>
    </div>
  );
};

export default TimeSlider;

In order to get the track to change color midway through (for when it straddled date ranges both in the past and the future), I needed to declare a new variable called current to keep track of what the actual current date is.

At the top of the component where the other variables are declared, current is added to the mix, and it's simply calling the Day.js library's valueOf() function) to get the current date as a Unix timestamp.

Once the current date is identified, I made a function called getTrackStyle() that compares the values captured by either end of the draggable date range slider and sets the color of the track accordingly.

When both slider values are less than or equal to the current time, the entire track is styled light blue, when both slider values are greater than current time (i.e. the future), the entire track is styled light green. When the range spans both past and future dates, the function creates a gradient effect relying on the pastWidthPercentage() function to determine where to transition the colors and the CSS linear-gradient to create a split color effect.

I'll be honest with you: ChatGPT helped me with this track styling magic. It's been extremely helpful for many complex things I've needed to do with Ant Design components, and I'd recommend working with it if you get stuck.

The getTrackStyle() function was passed to the <Slider/> component inside its styles property as styles={{ track: getTrackStyle() }}.

Then, I styled the tick marks based in the future in a different color as well.

  for (let i = min + tickInterval; i <= max; i += tickInterval) {
    const futureDate = i > current;
    marks[i] = {
      label: fmt(i),
      style: futureDate
        ? {
            color: "rgb(73, 163, 15)",
          }
        : {},
    };
  }

To accomplish this, I modified the for loop that went through all the predetermined ticks and compared each one to the current variable declared above. If any of the marks were in the future, those marks got a little extra styling applied to them in the form of a green font color. If the marks were not in the future, they retained their original font color.

Bonus section: I also wanted a "Now" tick mark to clearly delineate where the time slider track went from blue dates in the past/present to green dates in the future.

  const marks: Record<number, { label: string; style?: React.CSSProperties }> =
    {
      [min]: { label: fmt(min) },
      [current]: {
        label: "Now",
        style: {
          color: "rgb(89, 147, 251)",
          transform: "translate(-50%, -36px)",
          fontWeight: "bold",
        },
      },
      [max]: { label: fmt(max) },
    };

To add an extra tick mark entitled "Now" along my time slider track, I modified the marks variable, adding in the current variable as one more pre-defined tick mark, and giving it a label of "Now" and a little styling to stand apart from the rest of the tick marks along the track.

The "Now" tick mark only shows up when the dates passed to the time slider actually cross the current date, but I think it aids in understanding of what's going on onscreen.

With all of that done, the time slider component itself is complete, and it's time to add it to a parent component that will pass it the dates and functions it requires.

Import the TimeSlider component into the parent page

Ok, now it's time to give the custom <TimeSlider/> component the props it needs.

In my case, I had other components that also needed to be aware of any changes to the date range for display purposes (I had chart and map child components that needed to be updated as well when selected dates changed), so I had a page within my Next.js app holding the date state variables and onChange() function that were passed to the <TimeSlider/> component.

Inside of the index.tsx file within the pages/ folder, I added the following code:

import { useState } from "react";
import dayjs, { Dayjs } from "dayjs";
import TimeSlider from "@/components/TimeSlider";
import styles from "@/styles/MainContent.module.css";

type HomeProps = {
  startDate: string;
  endDate: string;
};

export default function Home({ startDate, endDate }: HomeProps) {
  const [initialDate, setInitialDateRange] = useState({
    startDate: dayjs(startDate),
    endDate: dayjs(endDate),
  });

  const [selectedDate, setSelectedDateRange] = useState({
    startDate: dayjs(startDate),
    endDate: dayjs(endDate),
  });

  const handleDateSliderChange = (innerDateSpan: { from: Dayjs; upto: Dayjs }) => {
    setSelectedDateRange({
      startDate: innerDateSpan.from,
      endDate: innerDateSpan.upto,
    });
  };

  return (
    <>
      <main className={styles.mainContent}>
        <div className={styles.page}>
          <h2 style={{ textAlign: "center", margin: "20px auto 0 auto" }}>
            Custom Ant Design Next.js Time Slider
          </h2>
          <div className={styles.timeSlider}>
            <TimeSlider
              outerDateRange={{
                from: initialDate.startDate,
                upto: initialDate.endDate,
              }}
              innerDateRange={{
                from: selectedDate.startDate,
                upto: selectedDate.endDate,
              }}
              onChange={handleDateSliderChange}
            />
          </div>
      </main>
    </>
  );
}

export async function getStaticProps() {
  const startDate = dayjs("2024-12-01").utc().format("YYYY-MM-DDTHH:mm:ss[Z]");
  const endDate = dayjs("2025-02-28").utc().format("YYYY-MM-DDTHH:mm:ss[Z]");

  return {
    props: {
      startDate,
      endDate,
    },
  };
}

In the index.tsx page, I created three variables related to the <TimeSlider/> component.

The initialDate and selectedDate React states, which are both objects containing a startDate and endDate property.
The handleDateSliderChange() function, which accepts new startDate and endDate values, and updates the selectedDate state accordingly.

For convenience, I chose to set my start and end date values inside the Next.js getStaticProps() function call which runs before the page renders in the browser, but it's not required, and I set both date state objects equal to the defined dates.

The initialDate values (which end up determining the min and max ends of the time slider won't change again until they are modified in getStaticProps() and the page is refreshed), but the selectedDate values are free to change within those bounds thanks to the handleDateSliderChange() function which will update the state of the selectedDate values whenever the user interacts with the <TimeSlider/> component.

The <TimeSlider/> component was imported into the index.tsx file and the initialDate object was passed as its outerDateRange prop, the selectedDate object was passed as its innerDateRange prop, and the handleDateSliderChange() function was passed as its onChange() prop.

At this point, the component should have everything it needs to work.

Test out the Time Slider component

If you'd like to see a working demo of this <TimeSlider /> component in action, I've got a CodeSandbox demo for you to try out.

Open a terminal and type npm run dev to get it started.

In addition to the time slider component, I also included a few nice extras: some Ant Design date picker components and buttons with values like "Last month", "Last week", and "Last day" to show other ways of manipulating the selectedDate state controlled by the parent component.

If you'd like to see how the time slider adjusts to larger or smaller date ranges or play around with past and future dates, just adjust the dates inside the getStaticProps() function in the index.tsx file and refresh the browser.

Conclusion

As I was building a React-based proof of concept web app that needed a time slider that users could drag and adjust to zoom in or out on particular pieces of data over time, I couldn't find a ready-made solution for such a need. So I built one myself.

Using the Ant Design component library's highly customizable <Slider/> component as the base and manipulating dates with the help of the Day.js library, I was able to build a flexible, full-featured, and decent looking time slider without a whole lot of extra code on my end.

Then I took it a few steps further making the slider handle future dates and style them differently and even display a little "Now" tick mark when the slider crossed over dates in the past to dates in the future dates. It's a pretty specific use case, I know, but I think there's more possible applications for interesting sliders like this once you start to think beyond the typical date time options.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope if you need a way to control time series data views within an app and don't want to rely on just date pickers or inputs, you'll consider giving this solution a try. Enjoy!

References & Further Resources

Automatically Publish a Repo as a PyPI Library with GitHub Actions

Paige Niedringhaus — Mon, 21 Oct 2024 00:00:00 +0000

Introduction

Last year I had the opportunity to turn my company's API into a JavaScript SDK by relying on the project's openapi.yaml file and the OpenAPI Generator CLI.

Notehub JS

For the full details of how I did created Notehub JS, I encourage you to read the original blog post I published about it here.

I did this as much for myself as for other developers, because I and a group of my coworkers were building a lot of JavaScript-based web apps to display and interact with the IoT data our company Blues specializes in transporting from a device in the real world to the cloud via cellular. To make it easier to update the JavaScript library as the API it's based on continues to grow and evolve, I set up a bunch of GitHub Actions workflows to do most of the tedious, repetitive tasks for me, and I learned a bunch of useful new things along the way.

This year, one of my other coworkers who works frequently in Python, asked if I could create a Python SDK for the API, and I agreed, feeling pretty confident about most of the steps involved. One new thing I did have to learn was how to build and publish a Python package to the Python Package Index, PyPI.

In this blog, I'll show you how to set up a GitHub Actions workflow to automatically publish a new version of a GitHub project to PyPI when a new release is made - no muss, no fuss, very little manual input required.

Notehub Py

As with the original Notehub JS project, the Notehub Py project's structure is a bit different from your typical repo because it is automatically generated from the Notehub API's openapi.yaml file. The openapi.yaml file follows the OpenAPI specification standards, and can be used with the OpenAPI Generator CLI to build a Python-based SDK to interact with the Notehub API in just a few commands from a terminal.

Here's a simplified view of the Notehub Py repo's folder structure:

.
├── .github/
│ └── workflows/
│ └── GH Action files
├── lib_template/
│ └── python library template files
├── src/
│ ├── notehub_py/
│ │ └── Python-based API and model files
│ ├── docs/
│ │ └── MD documentation
│ ├── test/
│ │ └── unit tests
│ ├── dist/
│ │ └── bundled .tar and .whl binaries for PyPi
│ ├── pyproject.toml
│ ├── requirements.txt
│ └── setup.py
├── openapi.yaml
├── config.json
├── README.md
└── scripts.py

The openapi.yaml file lives at the root level of the project along with its config.json file, a scripts.py file, and a few other other bits and pieces (license, contribution guidelines, code of conduct, etc.), but the real meat of the library lives inside of the src/ subfolder.

What's unique about this subfolder is that it is regenerated each time the openapi.yaml file is updated, and it has all the API endpoints, models, docs, and the dist/ folder for the notehub-py SDK that actually gets published on PyPI. Publishing just a subfolder of a project inside a GitHub repo is a little unusual, but not to worry, it can be done in an automated fashion.

Create a `scripts.py` and `config.json` file to automate generating the library and packaging it for distro to PyPI

To automate the build and publish steps to deploy Notehub Py to the PyPI registry, we need a couple of things: a config.json file and a scripts.py file.

The config.json file is a configuration file of additional properties used by the OpenAPI Generator and its Python library template to define certain variables like package name, package version, GitHub repo URL, etc. Here is what the Notehub Py's config.json file looks like.

config.json

{
  "packageName": "notehub_py",
  "packageUrl": "https://github.com/blues/notehub-py",
  "projectName": "notehub-py",
  "packageVersion": "1.0.2"
}

Every time a new version of the Notehub Py library needs to be published to PyPI, the packageVersion for the project will be updated in this file, then the commands to regenerate the library and its distribution packages (which I'll cover next) gets run.

The scripts.py file is a set of reusable commands to automate the steps of updating this repo based on the latest version of the openapi.yaml, and packaging it up for publishing to PyPI.

scripts.py

import sys
import subprocess
import shutil
import os

def generate_package():
    try:
        subprocess.run([
            "openapi-generator-cli",
            "generate",
            "-g",
            "python",
            "--library",
            "urllib3",
            "-t",
            "lib_template",
            "-o",
            "src",
            "-i",
            "openapi.yaml",
            "-c",
            "config.json"
        ])
    except Exception as e:
        print("Exception when generating package: %s\n" % e)

def build_distro_package():
    try:
        os.chdir("src/")
        # Check if the 'dist/' folder exists
        if os.path.exists("dist"):
            # If it exists, delete it and its contents
            shutil.rmtree("dist")

        # Upgrade the 'build' module
        subprocess.run([
            "python3",
            "-m",
            "pip",
            "install",
            "--upgrade",
            "build"
        ])

        # Generate a new 'dist/' folder  
        subprocess.run([
            "python3",
            "-m",
            "build"
        ])  
    except Exception as e:
        print("Exception when building distro package: %s\n" % e)       

if __name__ == " __main__":
    if len(sys.argv) != 2:
        print("Usage: python3 scripts.py [generate_package | build_distro_package]")
        sys.exit(1)

    script_to_run = sys.argv[1]
    if script_to_run == "generate_package":
        generate_package()
    elif script_to_run == "build_distro_package":
        build_distro_package()
    else:
        print("Invalid script name. Use one of: generate_package, build_distro_package")
        sys.exit(1)

The first function, generate_package(), uses the subprocess module to run the openapi-generator-cli tool to generate a new version of the Notehub Py library. Let's break down each of the arguments in the command:

"generate": Specifies the action to generate code.
"-g", "python": Indicates that the target language for the generated code is Python.
"--library", "urllib3": Specifies that the generated code should use the urllib3 library for HTTP requests.
"-t", "lib_template": Points to a template directory named lib_template that contains custom templates for the code generation.
"-o", "src": Sets the output directory to src, where the generated code will be placed.
"-i", "openapi.yaml": Specifies the input OpenAPI specification file.
"-c", "config.json": Uses a configuration file named config.json to customize the generation process.

The second function, build_distro_package(), builds the distribution package for the the Notehub Py project that will be deployed to PyPI.

First, the function checks if a dist folder exists inside the src folder where all the Notehub Py API code lives, and deletes the older version of the folder if it does exist to ensure any previous build artifacts are removed before adding a new distro package.

os.chdir("src/")
# Check if the 'dist/' folder exists
if os.path.exists("dist"):
    # If it exists, delete it and its contents
    shutil.rmtree("dist")

Next, the function upgrades the build module to ensure the latest version is installed. This module is essential for generating the distribution packages.

subprocess.run([
    "python3",
    "-m",
    "pip",
    "install",
    "--upgrade",
    "build"
])

Finally, the function generates a new dist directory by invoking the build module to create the distribution package, which includes both source distributions and wheel distributions.

subprocess.run([
    "python3",
    "-m",
    "build"
])

Both of these functions can be invoked from the command line by running python scripts.py generate_package or python scripts.py build_distro_package.

Set up a PyPI account and library to publish to

Now that the Notehub Py config file and scripts to automate the generation and building of the SDK are done, it's time to prepare PyPI to receive the library.

Follow these steps:

Create a PyPI user account if you haven’t done so already.
After logging into PyPI, go to the “Account Settings” page and select the “Publishing” tab.

Get to this screen by clicking the “Account Settings” tab in the user’s dropdown menu, then select “Publishing” under “Your account”.

Scroll down to the “Add a new pending publisher” section of the page, and input the details of your project you want to publish to PyPI: publisher platform (GitHub), project name to display on PyPI, project owner, repo name, publishing workflow name (something like publish-pypi.yaml), and click the "Add" button when you're done.
1. Using PyPI’s trusted publishing option utilizes OpenID Connect (OIDC) technology to provide credential-free publishing authority to trusted third party services like GitHub Actions. This allows us to automate the release process without needing to use API tokens or passwords.

Now, we're ready for the final step in the process: a GitHub Actions workflow to automate publishing the newly generated distribution packages to PyPI.

Write a GitHub Actions workflow to publish to PyPI

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

For me, it made the most sense to trigger a GitHub Actions workflow by creating a new release in GitHub to publish the Notehub Py src/ subfolder to PyPI.

GitHub specifically defines a release as a deployable software iteration that you can package and make available for a wider audience to download and use, which is exactly what I want.

Once I'd chosen this as the trigger for my workflow, it was a pretty straightforward set of steps to publish to PyPI.

Here's what the finished publish-pypi.yml file looks like inside of the ./github/workflows/ folder - I'll break it all down below.

publish-pypi.yml

name: Upload Python Package

on:
  release:
    types: [created]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: 
      name: pypi
      url: https://pypi.org/p/notehub-py
    permissions:
      id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
    defaults:
      run:
        working-directory: ./src
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.x"

      - name: Install dependencies
        run: |
          python3 -m pip install --upgrade pip
          pip install build

      - name: Build package
        run: |
          python3 -m pip install --upgrade build
          python3 -m build

      - name: Publish package to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          packages-dir: ./src/dist/

Each GH Actions workflow file needs a name , so I chose: Upload Python Package. It tells users exactly what this script's purpose is.

As I said earlier, this workflow is triggered whenever a new release is created in GitHub, which is where the following lines take effect.

on:
  release: 
    types: [created]

on is how a workflow is triggered.
release is the event that triggers the workflow.
types: [created] is the activity type for a release event that triggers the workflow. This gives us more fine-grained control of when the workflow should run.

Then the jobs section runs inside of the workflow. This particular script only has one job, deploy, but if there's multiple jobs, they'll run sequentially unless otherwise specified.

The deploy job defines that it runs on the latest version of Ubuntu in runs-on.

The job is configured to operate within an environment named pypi, with a URL that points to the PyPI project page for notehub-py, and the permissions section grants the job the necessary id-token: write permission, which is required for trusted publishing to PyPI. All of this will correspond to the details filled out in PyPI in the previous section setting up the trusted publishing option via OpenID Connect.

 environment: 
      name: pypi
      url: https://pypi.org/p/notehub-py
    permissions:
      id-token: write # this permission is mandatory for trusted publishing

The defaults.run section sets the working directory to ./src for all run steps, ensuring the commands are executed within the source subdirectory of the project.

Finally, we get to the steps.

The steps are as follows:

Check out the code so the workflow can clone the repository's code into the runner with actions/checkout@v4.
Set up a Python environment with version 3.x using actions/setup-python@v5.
Upgrade pip and install the build modules necessary for building the distribution package.
Build the distribution package using python3 -m build.
Publish the built packages to PyPI using the pypa/gh-action-pypi-publish@release/v1 action. The packages-dir parameter specifies the directory containing the distribution files (./src/dist/).

And there you have it: each time a new release is created in the Notehub Py repo, this GitHub Actions workflow will run and deploy the updated code to PyPI.

Conclusion

After publishing my first JavaScript SDK of my company's API on npm last year, I was asked to build a similar SDK in Python and distribute it to PyPI.

I was able to repurpose a lot of the same steps and workflows I used for generating the JavaScript library to generate the Python library, but configuring the deployment to PyPI was a bit different as that package platform recommends using trusted publishing to deploy new package versions.

With just a few Python scripts, a config file, and a little bit of set up on the PyPI site, I was able to deploy the auto-generated subfolder inside of the Notehub Py project to PyPI, helping developers more easily interact with the Notehub API. GitHub Actions allowed me to build the code, package it up for distribution, and publish it to PyPI in no time.

Check back in a few weeks — I’ll be writing more about the useful things I learned while building this project in addition to other topics on JavaScript, React, IoT, or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning how to deploy a subfolder of a project to PyPI through GitHub Actions workflows comes in handy for you in the future. Enjoy!

References & Further Resources

Notehub Py GitHub repo
notehub-py SDK on PyPI
OpenAPI Generator CLI

Animate an Auto-Scrolling Carousel with Only HTML and CSS

Paige Niedringhaus — Fri, 23 Aug 2024 00:00:00 +0000

Introduction

As web developers working with powerful computers we don't often stop to consider that while things like memory and stable Internet connections aren't a problem for us, they may be where our apps end up running.

Case in point: building a web app to run on a Raspberry Pi - which could be quite possible for something like a kiosk in a mall, a doctor's office or even an amusement park.

Last year, I was tasked with building a small demo for work to demonstrate how to package up a simple web application and download it to a Raspberry Pi using a cellular-based Blues Notecard.

If you're interested in how to do this, you can read the full details here. But that is outside the scope of this post.

I decided a good demo would be a weather app where a list of cities would be provided, and the app would fetch the weather forecast for each city and display it in an endlessly looping carousel.

In order to keep the app size small (and quicker to download via cellular data to a Raspi), I built it with pure HTML, CSS, and a little vanilla-JavaScript.

For the auto-scrolling feature of each city's forecast I'd normally look for a carousel library to help me, but since I needed the app to stay lightweight I did some research and found that CSS alone could do what I needed.

Today, I'll show you how to use CSS keyframes animations along with scroll snap module and a few well-placed CSS classes to make auto-scrolling, CSS-only carousels possible.

Here's what the final auto-scrolling carousel animation looks like.

CSS keyframes and scroll snap

Before we get to the implementation of this auto-scrolling carousel, let's talk a bit about the CSS features that make it possible: CSS keyframes and the CSS scroll snap module.

CSS keyframes

The @keyframes rule controls the intermediate steps in a CSS animation sequence by defining styles for keyframes (or waypoints) along an animation sequence. This gives more control over the intermediate steps of the animation sequence than CSS transitions.

Not only do @keyframes allow for complex animations without the need for JavaScript, but they also offer a straightforward syntax, and reusability across multiple elements (as you'll soon see in my own code examples below).

CSS scroll snap

The CSS scroll snap module provides properties that let you control the panning and scrolling behavior by defining snap positions. Content can be snapped into position as the user scrolls overflowing content within a scroll container, providing paging and scroll positioning.

It has two properties in particular we'll take advantage of today for the carousel.

CSS scroll-snap-type

The CSS scroll-snap-type property sets how strictly snap points are enforced on the scroll container.

It accepts a variety of different values, but the most common ones are:

mandatory - The scroll container must snap to the nearest snap point after scrolling.
proximity - The scroll container will only snap to a point if the scrolling comes to a natural rest near a snap point.
none - No snapping behavior is enforced.

In practice, you might use mandatory when you need precise control over the final position of the scroll (like with carousels or paginated content where each item should be fully visible after scrolling), and you might use proximity when you want to give users more control over the scroll stopping point, which can feel smoother and less jarring.

CSS scroll-snap-align

The other part of this scroll snap equation, scroll-snap-align, specifies a box's snap position within its snap container when using the CSS scroll snap module.

A child element of the scroll snap container can have the scroll-snap-align property set to control how it snaps into the viewport when scrolling stops. Its values include:

start - Aligns the start edge of the item with the start edge of the scroll container.
end - Aligns the end edge of the item with the end edge of the scroll container.
center - Centers the item in the scroll container.
none - No snap alignment will occur.

With that bit of background out of the way, let's get on to the code.

Set up the HTML and CSS

First, the proper HTML must be set up for the CSS auto-scrolling to work correctly.

1. Wrap the HTML elements to be part of the carousel in a parent div.

To begin, wrap all the slides to be scrolled through inside a parent container. This is where the CSS scroll-snap-type property will be applied so that the child elements will be able to interpret the scroll-snap-align property applied to them.

NOTE: In my full code sample, I defined an HTML template to hold all my weather data for each city, and the template.content.cloneNode() function to clone the template for each city in my list and add it inside the weather-forecast-container element

That's outside the scope of what's needed for this article, but if you'd like to see the full code, it's available here.

Here is my HTML file where the parent container and its children are defined.

index.htm

  <div class="weather-forecast-container">
    <div class="weather-template">
      <div class="current-header-wrapper">
        <div class="current-date"></div>
        <h2 class="location">Loading current weather...</h2>
      </div>
      <div class="current-conditions">
        <div>
          <span class="conditions">
            <img class="condition-icon" src="" />
            <span class="condition-text" />
          </span>
        </div>
        <div class="current-temp"></div>
        <div class="details">
          <div class="wind-speed"></div>
          <div class="rainfall"></div>
          <div class="pressure"></div>
        </div>
      </div>
      <div class="forecast">
        <table>
          <tbody class="future-forecast"></tbody>
        </table>
      </div>
      <div class="weather-carousel-snapper"></div>
    </div>
  </div>

As you can see from the HTML above, the weather-forecast-container class wraps the div with the class of weather-template which holds all the weather info for a given city, and contains an empty div at the bottom with a class of weather-carousel-snapper.

2. Position the empty div at the bottom of the weather-template div to be the carousel's snapper element.

Make sure that this element is inside of the weather-template div and not a sibling to it. It will continue to be empty, but it needs to be present.

<div class="weather-carousel-snapper"></div>

3. Add base styling to the parent container and position the weather template and scroll-snap div inside it.

This is where the CSS scroll snap module comes into play.

styles.css

.weather-forecast-container {
  position: absolute;
  top: 0;
  right: 0;
  bottom: 0;
  left: 0;
  display: flex;
  height: clamp(290px, 88vh, 430px);
  max-width: 780px;
  margin: 8px auto;
  scroll-snap-type: x mandatory;
  scroll-behavior: smooth;
  overflow-x: scroll;
}

Inside the weather-forecast-container class, we're going to go beyond the basic styling of the container like positioning and defining its height and width (both set to the width of a small screen attached to the Raspi), and add the first CSS scroll snap properties here.

scroll-snap-type: x mandatory; - this defines a mandatory snapping in the horizontal axis.
scroll-behavior: smooth; - makes the carousel appear to scroll to the next page, instead of just jumping there as it would by default.
overflow-x: scroll; - overflow content is clipped if necessary to fit horizontally inside the element's padding box.

Next up, is the styling for the weather-template div that contains the weather info and the empty weather-carousel-snapper div.

styles.css

.weather-template {
  position: relative;
  flex: 0 0 100%;
  background-size: 780px 100%;
  background-repeat: no-repeat;
  border: 1px solid rgba(0, 0, 0, 0.25);
  border-radius: 4px;
  padding: 0 10px;
  display: flex;
  flex-direction: column;
  justify-content: space-evenly;
}

I've included the stying for the weather-template div as it's the true parent of the weather-carousel-snapper element, but this is just standard CSS styling.

styles.css

.weather-carousel-snapper {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  scroll-snap-align: center;
}

Last but not least, we add the last bit of scroll snap CSS to the empty div with the class of weather-carousel-snapper.

scroll-snap-align: center; - this will align each item to the center of the container.

4. Create the animation keyframes to auto-scroll the forecast slides

After the CSS styling is done, and scroll snap is enabled, it's time to make the @keyframes animations that will actually make the different cities appear to slide through the viewport from one city to the next.

styles.css

@keyframes tonext {
  75% {
    left: 0;
  }
  95% {
    left: 100%;
  }
  98% {
    left: 100%;
  }
  99% {
    left: 0;
  }
}

@keyframes tostart {
  75% {
    left: 0;
  }
  95% {
    left: -300%;
  }
  98% {
    left: -300%;
  }
  99% {
    left: 0;
  }
}

@keyframes snap {
  96% {
    scroll-snap-align: center;
  }
  97% {
    scroll-snap-align: none;
  }
  99% {
    scroll-snap-align: none;
  }
  100% {
    scroll-snap-align: center;
  }
}

In the code above, there are three separate animations defined.

tonext and tostart - manage the horizontal position of each city's weather forecast at different stages of the carousel, creating a sliding effect either to the right or far left, followed by a reset to the starting position (for getting the animation to endlessly loop back to the first city in the list again).
snap - controls the scroll-snap-align property, adjusting the snapping behavior at specific points in the animation timeline. This creates the dynamic snapping behavior that centers up the next city forecast in the carousel.

By combining these three keyframes, we'll have the slick-looking animation in the demo video where the carousel appears to smoothly slide from one forecast to the next.

5. Apply the keyframe animations to the appropriate HTML elements inside the weather-forecast-container parent div.

Now we're going to apply the @keyframes animations defined in the previous step.

styles.css

@media (hover: hover) {
  .weather-carousel-snapper {
    animation-name: tonext, snap;
    animation-timing-function: ease;
    animation-duration: 4s;
    animation-iteration-count: infinite;
  }

  .weather-template:last-child .weather-carousel-snapper {
    animation-name: tostart, snap;
  }
}

@media (hover: hover) - in an effort to ensure that the carousel animations and scroll snap are not applied to devices where hovering isn't possible, such as with most smartphones and tablets, I wrapped these animations inside of the hover media query.

The .weather-carousel-snapper class is given the tonext and snap animations to run simultaneously with animation-name: tonext, snap;, and it will ease over 4 seconds and loop infinitely.

A rule targeting the last child of the .weather-template element with the class of weather-carousel-snapper, applies a different animation combination (tostart and snap), to reset the carousel after the last city's weather forecast has been displayed onscreen to get the carousel back to it's starting position (and first city in the list).

6. And there you have it: a CSS only, auto-scrolling carousel with just HTML and CSS.

The finished product demoed once more in this video.

Conclusion

In certain situations, it's really crucial that web applications be as small as possible. For instance, when that app may be running on a low power or underpowered system (like a Raspberry Pi).

When I needed to build a small demo app that would download over cellular to a Raspi, I decided to do it with plain HTML, CSS, and a bit of vanilla-JS to keep the zip file size as small as possible - no JS frameworks like React or Svelte to help me out.

I wanted to build a weather app that would display the forecast for a list of cities and auto-rotate through the list before returning to the beginning of the list to start over. At first, I thought I might need a carousel library to make this happen, but it turns out CSS scroll snap and keyframe animations can do exactly the same thing, no JS needed. That's pretty darn sweet.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope if you need a carousel in the future (auto rotating or not) that you'll try implementing it yourself with CSS - I think you'll be pleasantly surprised how little code is needed to make it happen. Enjoy!

References & Further Resources

CSS-Tricks, CSS-Only Carousel article
MDN docs, CSS @keyframes
MDN docs, CSS scroll snap module
MDN docs, CSS scroll-snap-type
MDN docs, CSS scroll-snap-align
GitHub repo, Cellular-Connected Electronic Kiosk

Lean on CSS Clip Path to Make Cool Shapes in the DOM without Images

Paige Niedringhaus — Wed, 19 Jun 2024 00:00:00 +0000

Introduction

Up until a few years ago if you wanted background shapes or sections of a website that were anything besides rectangles you most likely needed a designer to provide you with a static PNG or JPEG image that would be added as required, but CSS has come a long way since then, my friends.

When I was working on a website update that broke up the contents on the page into different colored background sections, alternating between pure white and soft gray colors, the design mock up I had included one section whose bottom edge tilted up and to the right instead of going across the page at a perfect 90 degree angle, as a typical block element does.

Now I could have asked the designer to make a background image to do this for me, but instead I wanted to see if I could do it on my own with the power of CSS. And lo and behold I could, with CSS clip-path.

Interesting shapes and visuals in the DOM are no longer purely the domain of designers, with tools like CSS clip-path, devs have the power to reshape elements and I'll show you how.

CSS clip-path

If you're less familiar with the CSS clip-path property, like me, it creates a clipping region that sets which parts of an element should be shown. Parts that are inside the region are shown, while those outside are hidden.

A demo from the MDN clip-path docs. Different clip-path options provide different views of the hot air balloon and text.

The clip-path property can accept a large variety of values:

<clip-source>, which accepts values like url for an SVG element with clipping path defined.
<geometry-box>, which accepts values like margin-box and border-box.
<basic-shape>, which accepts values like circle() and rect().
global-values, which accepts values like inherit and revert.

The <geometry-box> and <basic-shape> values can even be combined together in one clip-path.

/* this CSS combines two different clip path properties */
clip-path: padding-box circle(50px at 0 100px);

This post doesn't go into great detail about all of the properties clip-path can accept and how they can be combined to create quite complex shapes. If you want more information and examples of clip=path in action, I recommend starting with the Mozilla documentation.

One of the <basic-shape> properties clip-path accepts is polygon(), and this ended up being the solution I needed for my tilted background section.

The polygon I needed to recreate with CSS

The gray polygon background I needed to create with CSS.

The image above is a screenshot of the gray background section I needed to recreate with CSS clip-path's polygon() property. And the first thing I needed to do was create some HTML elements to apply the CSS to.

polygon() clip-path vs rect() clip-path

You might be wondering why I chose to use the polygon() property instead of the rect() property with clip-path. While the two are similar, polygon() can create more complex polygonal shapes and offers greater versatility for advanced designs by accepting pairs of coordinates to define each vertex of the polygon, whereas rect() can only handle rectangular shapes.

Set up the HTML and CSS

The site I was working on relied on the static site generator Hugo, a Go-based framework. Hugo uses templates to render the site's HTML, so the example code below should look relatively familiar to you if you know HTML.

A note on templates:

If you've ever used JSX components, Node.js with Pug or Handlebars, or Jekyll - Hugo's templates are similar: HTML elements with Go variables and functions sprinkled in with {{ }} to render the correct information wherever the templates are injected.

Here's the code for what I'd nicknamed the "puzzle section" of the page due to the puzzle piece in the foreground of this section. For the purposes and clarity of this article, I've replaced the Go variables injected into the template with the generated HTML.

single.html

  <div class="about-body">
    <!-- more HTML elements up here -->

    <section class="puzzle-section section">
      <div class="container">
        <div class="row">
          <div class="col-12 col-md-6 col-lg-6">
              <h4 class="mb-3">
                Lorem ipsum dolor
              </h4>
              <p class="mb-5">
                Sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Ipsum dolor sit amet consectetur adipiscing elit pellentesque.
              </p>
              <h4 class="mb-3">
                Duis aute irure dolor in reprehenderit
              </h4>
              <p class="mb-5">
                in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Consectetur adipiscing elit pellentesque habitant morbi tristique senectus et.
              </p>
          </div>
          <div
            class="col-sm-8 offset-sm-2 col-md-6 offset-md-0 col-lg-6 offset-lg-0"
          >
            <img
              class="img-fluid"
              src="/images/about/puzzle-pieces.png"
              alt="Puzzle pieces"
            />
          </div>
        </div>
      </div>
    </section>

     <!-- more HTML elements below -->
  </div>

This section of code is relatively compact, but it deserves discussion. In addition to the HTML elements, there are quite a few CSS classes which come from the Bootstrap library, one of the original open source CSS frameworks for responsive web designs.

Among the custom classes like about-body, which I used for adding custom styling, there are classes like container, row, col-12 or col-md-6, mb-5, and mb-3.

All of the latter classes are Bootstrap classes, which serve to make the text and image elements onscreen share the width of the page when the viewport is over a certain width (col-md-6), or apply a margin-bottom of a certain amount to the <p> tags (mb-3 or mb-5).

The Bootstrap classes are beside the point for this post though, the class to focus on is the puzzle-section which wraps all the text and puzzle piece image.

This puzzle-section class is where we're going to add the clip-path property to display the light grey background behind the text and image with the slightly tilted, up-and-to-the-right design.

Add the CSS clip-path to shape puzzle-section

As I wasn't quite sure how to style a normal, rectangular <div> into an uneven shape, I started looking for a solution online and found this helpful, interactive clip-path-focused site, CSS clip-path maker.

This CSS clip-path maker website is fantastic because it has a whole slew of preset shapes, adjustable image sizes and backgrounds, and the currently displayed image's vertices can be dragged into any arrangement you want. The line at the bottom of the screen shows the exact clip-path CSS values that you can copy/paste into your own project's CSS.

I chose the parallelogram preset shape as my starting point, and then dragged the corners to match the angle of the background section I was trying to recreate from scratch. Once I was satisfied it looked accurate, I copied the CSS line at the bottom of the page to my clipboard.

In my project's SCSS file, I added the copied clip-path CSS in addition to the light grey background-color property and some padding to give the text and puzzle piece images some breathing room on the page.

NOTE: Even though this file shown in the example code is SCSS instead of pure CSS, for this post it shouldn't make a difference here. It should be a direct 1:1 comparison.

about.scss

.about-body {
  // this white sets the white background color for the whole webpage
  background-color: white; 

  .puzzle-section {
    // clip-path code copied from the clip-path maker website
    clip-path: polygon(0 0, 100% 0%, 100% 75%, 0% 100%);
    background-color: light-grey;
    padding: 2rem 0 10rem 0;
  }
}

That little bit of CSS for clip-path was all that was needed to take my perfectly rectangular DOM element and turn it into an imperfect polygon instead. Not too shabby!

Conclusion

CSS is pushing the boundaries of what web developers can do without resorting to images, videos, and custom designed elements all the time. And the satisfaction of figuring out how to do a cool little bit of design on all on your own feels pretty empowering.

A recent example of this was using the CSS clip-path property to create a background box for some text and images that had an uneven bottom edge. With the help of an interactive website dedicated decoding to clip-paths of all shapes and sizes, I was able to make quick work of this slightly skewed polygon.

And let me take a moment to shout out how much I appreciate the folks putting out those little sites or code snippets that solve a very specific problem for another developer - you folks continue to make the Internet a better place.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning to reshape how elements look in the DOM with just the power of CSS helps you as much as it's helped me.

Further References & Resources

MDN docs, CSS clip-path
CSS clip-path generator website

Create a Custom Formatted CSV from Python Data

Paige Niedringhaus — Fri, 19 Apr 2024 00:00:00 +0000

Introduction

In 2023, a friend of mine asked me to write a program to collect the data from an NFT collection on the NFTrade website. He was interested in the following information: all the NFTs currently for sale in the collection, the current price of BNB (the cryptocurrency the NFTs are listed for sale in), the price of each NFT converted to US dollars based on the current price of BNB, and put all that data into a spreadsheet that's easy to manipulate, sort, and filter.

I couldn't use my go-to JavaScript skills to fetch data via HTTP calls from the NFTrade site because there is no public-facing API, so instead, I taught myself to build a small web scraping script to visit the site and "scrape" that data from it.

Python seems to be a very popular language for such projects, so I went with it, and as I worked on it, the requirements got a bit more complex, and I learned a lot of useful things about Python as a result, which I've shared in a series of blog posts. This post will be the last in my series covering this web scraper.

In previous articles I:

And now, with all the data in one place, I needed to make it into a spreadsheet my friend could work with.

The Python csv.DictWriter module makes turning a list of Python objects into a standard CSV file a straightforward task, which I'll demonstrate in this article.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Sample Python data

I need to set the stage and show you the shape of the my Python data before I show you the solution I ended up using. As I mentioned in the introduction, the data I wanted for each NFT included the following things:

NFT ID
NFT list price (in BNB)
NFT rarity score (a number randomly assigned to each NFT in the collection)
Current price of BNB in US dollars
Cost of NFT in US dollars
Cost of each rarity point assigned to the NFT in US dollars

What I ended up with is a list of objects (or "dictionaries" in Python parlance) because each object in the list is made up of key value pairs. Here's a sample of what the list of data looks like.

Sample NFT data to be added to CSV file

[
    {'bnb': 352.44, 'cost_per_rs': 28.2, 'id': 4, 'nft_price': '0.8', 'price_usd': 281.95, 'rs': 10},
    {'bnb': 352.44, 'cost_per_rs': 77.54, 'id': 42, 'nft_price': '1.1', 'price_usd': 387.68, 'rs': 5},
    {'bnb': 352.44, 'cost_per_rs': 98.68, 'id': 174, 'nft_price': '1.4', 'price_usd': 493.42, 'rs': 5}, 
    {'bnb': 352.44, 'cost_per_rs': 29.68, 'id': 184, 'nft_price': '1.6', 'price_usd': 563.9, 'rs': 19},
    {'bnb': 352.44, 'cost_per_rs': 46.99, 'id': 256, 'nft_price': '2', 'price_usd': 704.88, 'rs': 15},
    # more NFT data
]

NOTE: If you're interested in seeing how I got this list of data, check out my previous blog post for a more in-depth explanation.

Ok, now that I've established what the data looks like we can move on to how I turned this list into a CSV, complete with a header row of each key in the dictionary object.

The csv.DictWriter module

Unlike core JavaScript, which tends to be pretty bare-bones and makes users install packages for most everything they want to do, the Python standard library is quite robust.

It comes with a csv module that implements classes to read and write tabular data in CSV format out of the box, as the CSV (Comma Separated Values) format is the most common import and export formats for spreadsheets and databases.

The csv module had two packages of particular interest to me:

writer - a writer object responsible for converting the user's data into delimited strings on a given file-like object.
DictWriter - an object like a writer but it maps dictionaries to output rows. This object requires a fieldnames parameter that is a sequence of keys which identifies the order in which values in the dictionary are passed to the writerow() method.

Since my Python data is formatted as a list of dictionaries, as shown in the previous section, it made the most sense for me to use the csv.DictWriter module.

For me, the fieldnames parameter would be composed of the keys of each dictionary: id, bnb, nft_price, etc. and that would guarantee that each dictionary of NFT info passed through the DictWriter object would match up the proper value with its corresponding key. And DictWriter also has two handy public methods I wanted:

writeheader() - a method to write a row with the field names to the writer's file object (i.e. the top row in a CSV that typically has the column names listed for each column of data).
writerows() - the method that writes all the elements in rows to the writer's file object (i.e. the list of NFT objects I wanted added to the CSV file).

DictWriter was the right choice for my situation, so let's get to how I put it into practice to make my file next.

Format the Python data into a CSV and save it locally

To transform the list of Python dictionaries into a CSV using the csv.DictWriter module, I created a new function called download_csv(), and here is what the code looked like in my for_sale_scraper.py file.

def download_csv(self, card_list):
      """Turn card list into CSV file."""
      date_time = datetime.now().strftime("%Y_%m_%d-%I_%M_%p_")
      # add date time to the front of the file name 
      with open(date_time + 'NFTs_For_Sale.csv', 'w', encoding='utf8', newline="") as output_file:
          dict_writer = csv.DictWriter(output_file, fieldnames=['id', 'bnb', 'nft_price', 'price_usd', 'rs', 'cost_per_rs'])
          dict_writer.writeheader()
          dict_writer.writerows(card_list)
      pprint("CSV NFTs For Sale file generated!")

It will probably be easiest to go through this function line by line, so let's start from the top.

The function takes in a card_list argument which is what I want written to each row in the CSV, and the first thing it does is create a variable named date_time.

date_time = datetime.now().strftime("%Y_%m_%d-%I_%M_%p_")

date_time is a formatted date and time string representing the current date in the format of "Year_Month_Day-Hour_Minute_AM/PM". When called, it will generate a string like "2024_04_11-03_30_PM_", which will be tacked on to the beginning of the CSV's file name so it's easy to sort and identify the files after they're generated.

On to the next line of the function:

with open(date_time + 'NFTs_For_Sale.csv', 'w', encoding='utf8', newline="") as output_file:

with open opens a file for writing with the specified filename, which is the newly created date_time variable concatenated with the "NFTs_For_Sale.csv" string. The 'w' mode indicates the file will be opened for writing, the encoding='utf8' parameter specifies the character encoding to be used, and the newline="" ensures the proper newline character is used for writing rows to the CSV file. The whole thing is named output_file for reference later in the function.

Then we come to the DictWriter.

    dict_writer = csv.DictWriter(output_file, fieldnames=['id', 'bnb', 'nft_price', 'price_usd', 'rs', 'cost_per_rs'])
    dict_writer.writeheader()
    dict_writer.writerows(card_list)

It is DictWriter's time to shine, as a new instance of it is created and the output_file is passed to it, along with the fieldnames list which includes all the dictionary keys as the headers for the CSV file. In the following two lines of code, DictWriter writes the header row to the CSV file with the names specified in fieldnames, and writes the list of dictionaries (the card_list passed to this function) to the CSV file. Each dictionary in the list represents a row in the CSV file.

Job complete!

pprint("CSV NFTs For Sale file generated!")

Finally, a success message is printed out at the end of the function indicating the CSV file has been successfully generated, and the file should show up in the repo alongside all the other previously project files.

Note all the generated CSV files, neatly ordered in descending order by their filename dates.

And the download_csv() function is called from the main Python function like in the code snippet below.

I've included the other functions called before download_csv so you can see how I divvied up the work this file was doing, but I have not included the details of those functions because it's outside the scope of this post.

for_sale_scraper.py

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # add rarity scores to all cards in the list by merging them with the id_rs_list
   cards_with_rs = scraper.get_cards_rarity_score(cards_for_sale)

     # add the current bnb price, current usd price of cards and current usd price of each rs point
   cards_with_bnb_rs = scraper.add_pricing_to_cards(cards_with_rs)

   # generate csv file 
   scraper.download_csv(cards_with_bnb_rs)

Now, my friend can generate CSVs of the NFT collection whenever he wants to, or he could even set up a cron job on his machine to run this script on a regular basis and then review the generated files at his convenience.

Conclusion

Building a web scraper to collect data from the NFTrade site was a good exercise for me to learn more about Python, and it was a great opportunity for me to share my learnings in a series of blog posts for anyone else looking to do something similar.

I scraped the data with Selenium Python, narrowed it down to the pieces I wanted, added some extra info like the cost of each NFT in US dollars, and then bundled all data up into a downloadable CSV for easy scanning and sorting.

The extra nice thing about making it into a CSV is that the CSV reading and writing functions are part of the core Python library so I didn't even have to install any third party packages to make it work - it was actually quite straightforward in the end.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning to use Python's csv.DictWriter module to make your own CSVs from list of data comes in handy in your own apps and projects.

References & Further Resources

NFTrade website
Python DictWriter documentation
First blog post about scraping data from a lazy-loading website using Selenium Python
Second blog post about limiting data searches to a particular element on a page instead of the whole page when using XPath
Third blog post about filtering, merging, and updating lists of objects in Python

Filter, Merge, and Update Python Lists Based on Object Attributes

Paige Niedringhaus — Fri, 23 Feb 2024 00:00:00 +0000

Introduction

Last year, I wrote a web scraping program to collect data from one of the NFT collections on the NFTrade site. My friend wanted the following data included in a CSV: all the NFTs currently for sale in the collection, the total price of each NFT in US dollars based on the current market price of the BNB cryptocurrency that the NFT is for sale in, and the price in USD per rarity point (a value randomly assigned to each NFT in the collection).

The NFTrade website does not have a public API so instead of writing a Node.js script to fetch the data via HTTP calls, I built a small site scraping script to go to the website and actually "scrape" the data from it.

Having not written a web scraper before, I chose to write the program in Python, and as I built the scraper, the project requirements got a bit more complex, and I learned a bunch of useful techniques when coding in Python, which I'm sharing in a series of posts.

After choosing the Selenium Python package to use Selenium WebDriver to scrape the data from NFTrade and extract the details from each NFT that I wanted (the NFT's ID and price in BNB), I needed to update my new list of NFT data in several ways:

I needed to filter out any NFTs that weren't currently for sale (some that were scraped off the site weren't actually for sale),
I needed to match all the NFTs for sale with their "rarity scores" (as defined in a separate JSON list) and include those scores along with the rest of the NFT data,
I needed to compute the total cost and cost per rarity point for each NFT in USD based on the current market price of BNB and add those prices to each NFT in the list as well.

I know this sounds quite complicated, but I broke each of these requirements down into separate methods inside my Python script and learned a lot about working with lists in Python along the way.

Today, I'll show you how to filter lists by whether an attribute exists in an object, how to merge two lists of items together based on matching attributes, and even how to add new object properties to the objects within a list in Python.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Sample Python Data

Before I dive into the specifics of my list manipulations in Python, let me give you a little background on what the data looks like that I was working with. Here's a small sample of what the list of NFT data looked like before I started mutating it.

Sample NFT data scraped from the NFTrade site

[
    {'id': 6774, 'nft_price': '0.22'},
    {'id': 5710, 'nft_price': '0.16'},
    {'id': 3187, 'nft_price': '0.8'},
    {'id': 6482, 'nft_price': '1.1'},
    {'id': 7689, 'nft_price': '0.5'},
    {'id': 335, 'nft_price': '4'},
    {'id': 7025, 'nft_price': '1.057'},
    {'id': 597, 'nft_price': '5'},
    {'id': 3936, 'nft_price': '3.1'},
    {'id': 2834, 'nft_price': '0.649'},
    {'id': 763, 'nft_price': '1.65'},
    {'id': 7683, 'nft_price': None},
    {'id': 7914, 'nft_price': None}
 ]

As you can see from the output above, the original data I started with was pretty sparse: the ID number for each NFT and the price in BNB (if it existed) were the only pieces of data present in each object from the info scraped off the NFTrade site. I had my work cut out for me to clean this list up and add more useful data to it, so let's move on to how I did so in the next section.

NOTE: If you'd like to see more about how to scrape the browser data and gather just the necessary bits, read my first couple of blog posts here and here.

Filter objects in a list on whether an attribute exists or not

As I mentioned in the introduction, the first thing I needed to do was clean this list up by removing any NFTs that didn't have a price.

Due to how I had to lazily load and scrape the data from the NFTrade website initially, there was a good chance there were a handful of NFTs I gathered up that weren't for sale, and therefore didn't have prices, so I needed to weed them out first.

Technically every NFT in my list had an nft_price attribute, but if there was no price listed in the card's scraped data, the nft_price attribute was assigned None, which proved very useful.

Inside of the __main__ method in my Python script, I'd already scraped the data from the webpage with the get_cards() method, then looped through the NFT data to grab just the bits of relevant data with the get_nft_data() method. Now I wanted to filter down the cards to only include ones listed for sale.

Here's the __main__ method code first:

for_sale_scraper.py

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

And here's the method I came up with to filter down to just the NFTs for sale: filter_priced_cards().

def filter_priced_cards(self, card_list):
    """Filter card list to only cards with NFT cost."""

    # filter out any cards in the list that don't have an NFT price equal to None
    cards_for_sale = list(filter(lambda card: card['nft_price'] != None, card_list))
    return cards_for_sale

Let's break down what's happening in the second line of the filter_priced_cards() function.

I used Python's built-in filter() function to iterate over the card_list passed to the function to create a new list named cards_for_sale. The anonymous lambda function inside of filter() takes each card in the card_list and returns True if the nft_price attribute of the card is not None, and False if it is - this is how it filters out all the cards that don't have a price.

The list() function that wraps the filter() converts the result back to a list, because filter() returns a filter object which is an iterator, not a list.

And finally, the new cards_for_sale list is returned.

[Merge two lists together by matching object keys

Once the NFTs not for sale have been filtered out, the next step is to add the rarity score to each NFT based on its ID.

For this particular set of NFTs, each NFT had a "rarity score" that had been randomly assigned to it. The rarity scores for each NFT were listed in a separate JSON file in the project and they look like this.

id_rs_score.json

[
  {"id": 1, "rs": 18},
  {"id": 2, "rs": 13},
  {"id": 3, "rs": 14},
  {"id": 4, "rs": 10},
  {"id": 5, "rs": 22},
  {"id": 6, "rs": 13},
  {"id": 7, "rs": 10},
  {"id": 8, "rs": 13},
  {"id": 9, "rs": 13},
  {"id": 10, "rs": 9},
  // more ids and rarity scores ("rs") below
]

I needed to combine my list of cards_for_sale with the rarity scores in the JSON file by matching up the id attribute in each list of objects. For this task, I came up with the following function: get_cards_rarity_score().

def get_cards_rarity_score(self, card_list):
    """Combine rarity scores with card list by ID."""

    # get rs data for each card from json file 
    with open("id_rs_list.json") as file:
        id_rs_list = json.load(file)

    # merge together cards with id_rs_list by their matching ID numbers 
    match_cards_with_rs_list = groupby(sorted(card_list + id_rs_list, key=itemgetter("id")), itemgetter("id"))
    combined_cards = [dict(ChainMap(*g)) for k, g in match_cards_with_rs_list]

    # filter out all the items in the merged list without a "for sale" value 
    filtered_combined_cards = []
    for card in combined_cards:
        if 'nft_price' in card:
            filtered_combined_cards.append(card)

    return filtered_combined_cards

To combine the rarity score with any of the NFT objects contained in the card_list list, the first thing that had to happen was to read the data from the id_rs_list.json file and assign it to a variable.

    # get rs data for each card from json file 
    with open("id_rs_list.json") as file:
        id_rs_list = json.load(file)

Once the JSON list was extracted from the file, the card_list and id_rs_list needed to be merged together based on their matching IDs.

The groupby() function groups elements with the same ID, and then ChainMap() merged the grouped items into Python dictionaries (objects). The result was a list of dictionaries (combined_cards) where each dictionary represented a card with combined information from both lists.

    # merge together cards with id_rs_list by their matching ID numbers 
    match_cards_with_rs_list = groupby(sorted(card_list + id_rs_list, key=itemgetter("id")), itemgetter("id"))
    combined_cards = [dict(ChainMap(*g)) for k, g in match_cards_with_rs_list]

One thing to note: the combined_cards list has every NFT listed from the id_rs_list, not just the ones whose IDs match the IDs in the card_list. So the combined_cards list looks like the data below - but for every item in id_rs_list.

[ 
 {'id': 1, 'rs': 4},
 {'id': 2, 'nft_price': '3', 'rs': 6},
 {'id': 3, 'rs': 22},
 {'id': 4, 'rs': 4},
 {'id': 5, 'rs': 10},
 {'id': 6, 'nft_price': '5', 'rs': 1},
 {'id': 7, 'rs': 1},
 {'id': 8, 'nft_price': '0.1', 'rs': 14},
 {'id': 9, 'nft_price': '1.5', 'rs': 5},
 {'id': 10, 'rs': 1},
 # more IDs and NFT data 
]

Since the combined_cards list had every single NFT in it (not just ones for sale), once more I had to filter the list down so that every item without an "nft_price" was omitted.

    # filter out all the items in the merged list without a "for sale" value 
    filtered_combined_cards = []
    for card in combined_cards:
        if 'nft_price' in card:
            filtered_combined_cards.append(card)

In this case, since there's a (very likely) chance the NFT data in the combined_cards list did not have the "nft_price" attribute, I checked if each card had the key "nft_price" and if so, the card was added to the new filtered_combined_cards list.

The filtered_combined_cards list ended up looking like the code snippet below.

[
 {'id': 4, 'nft_price': '0.8', 'rs': 10},
 {'id': 42, 'nft_price': '1.1', 'rs': 5},
 {'id': 174, 'nft_price': '1.4', 'rs': 5},
 {'id': 184, 'nft_price': '1.6' 'rs': 19},
 {'id': 256, 'nft_price': '2', 'rs': 15},
 {'id': 335, 'nft_price': '4', 'rs': 2},
 {'id': 562, 'nft_price': '1.2', 'rs': 2},
 {'id': 584, 'nft_price': '5', 'rs': 14},
 {'id': 597, 'nft_price': '5', 'rs': 17},
 # more NFT data here
]

Once all this data manipulation and list combining was done, the function returned the final list of cards (filtered_combined_cards) that had both rarity score information and an "nft_price" attribute included.

return filtered_combined_cards

For reference, here's the __main__ function in the Python script, which called the get_cards_rarity_score().

for_sale_scraper.py

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

Add new object properties to each object in a list

All right, here's the last Python list manipulation tip I'll be sharing in this post: how to add new properties to each object in a list.

After filtering the NFTs to just the ones for sale, and adding the rarity scores from the id_rs_list JSON file, I needed to fetch the current price of 1 BNB compared to US dollars, calculate the current price of each NFT in USD, and calculate the cost per rarity point for each NFT.

Fortunately the cryptocurrency data aggregation site CoinGecko, has a REST API that I could use to get the current market price of BNB cryptocurrency in US dollars, and then calculate the rest of the required data based on the info in my NFT card list.

Here is the add_pricing_to_cards() function I came up with to calculate the prices.

def add_pricing_to_cards(self, card_list):
    """Get current price of BNB and compute cost per rarity point"""

    URL="https://api.coingecko.com/api/v3/simple/price?ids=binancecoin&vs_currencies=USD"
    response = requests.get(URL).json()
    bnb = response['binancecoin']['usd']

    # add the current value of bnb to the card_list 
    cards_bnb_price = [dict(card, bnb=bnb) for card in card_list]

    # compute the current price of usd for each card based on its bnb price       
    cards_with_usd_price= [dict(card, price_usd=round(float(card['nft_price'])*card['bnb'], 2)) for card in cards_bnb_price]

    # compute the current cost usd of each rarity score point         
    cards_with_rs_prices = [dict(card, cost_per_rs=round(card['price_usd']/card['rs'], 2)) for card in cards_with_usd_price]

    return cards_with_rs_prices

In the function, the first thing I did was call the CoinGecko price API to get the current price of BNB in USD.

    URL="https://api.coingecko.com/api/v3/simple/price?ids=binancecoin&vs_currencies=USD"
    response = requests.get(URL).json()
    bnb = response['binancecoin']['usd']

Next, I added the bnb to each object in the input card_list and created a new list named cards_bnb_price.

    # add the current value of bnb to the card_list 
    cards_bnb_price = [dict(card, bnb=bnb) for card in card_list]

After including the current BNB price in USD, I was able to compute the total price in USD for each NFT in the list by multiplying the card's original price in BNB by the current price of BNB in USD.

    # compute the current price of usd for each card based on its bnb price       
    cards_with_usd_price= [dict(card, price_usd=round(float(card['nft_price'])*card['bnb'], 2)) for card in cards_bnb_price]

And I also calculated the price in USD per rarity score point as well, simply by dividing the card's total price in USD by the rarity score number (rs).

    # compute the current cost usd of each rarity score point         
    cards_with_rs_prices = [dict(card, cost_per_rs=round(card['price_usd']/card['rs'], 2)) for card in cards_with_usd_price]

The function then returned the list of cards with the added pricing info, including BNB price, USD price, and USD cost per rarity score point. The final list data looked like this.

[
    {'bnb': 352.44, 'cost_per_rs': 28.2, 'id': 4, 'nft_price': '0.8', 'price_usd': 281.95, 'rs': 10},
    {'bnb': 352.44, 'cost_per_rs': 77.54, 'id': 42, 'nft_price': '1.1', 'price_usd': 387.68, 'rs': 5},
    {'bnb': 352.44, 'cost_per_rs': 98.68, 'id': 174, 'nft_price': '1.4', 'price_usd': 493.42, 'rs': 5}, 
    {'bnb': 352.44, 'cost_per_rs': 29.68, 'id': 184, 'nft_price': '1.6', 'price_usd': 563.9, 'rs': 19},
    {'bnb': 352.44, 'cost_per_rs': 46.99, 'id': 256, 'nft_price': '2', 'price_usd': 704.88, 'rs': 15},
    # more NFT data
]

The add_pricing_to_cards() function is called from the main Python function like so:

for_sale_scraper.py

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # filter out any extra cards that aren't for sale
   cards_for_sale = scraper.filter_priced_cards(card_data)

   # add rarity scores to all cards in the list by merging them with the id_rs_list
   cards_with_rs = scraper.get_cards_rarity_score(cards_for_sale)

And now that I had all the data that my friend requested for each NFT in the collection for sale on NFTrade, all that was left to do was turn the whole list into a downloadable CSV that would be easy to sort and manipulate. I'll save that for a future post.

Conclusion

When I had to use Python to build a website scraper to get NFT data off of the NFTrade site, I learned a lot of useful new coding tricks along the way.

After I'd managed to scrape the data with the help of Selenium Python, and extract the initial data I needed from each NFT by using WebDriver's XPath, my job was far from complete.

I needed to take the little data I had and combine those NFTs with "rarity scores" in a JSON file, fetch the current market price for BNB cryptocurrency in US dollars, and then compute the total cost of each NFT and cost per rarity point, and as I completed these tasks I learned a heck of a lot about how to work with lists of complex objects in various new ways. And I feel confident these new techniques will help me out in any future Python endeavors I might undertake.

Check back in a few weeks — I’ll be writing more blogs about the problems I had to solve while building this Python website scraper in addition to other topics on JavaScript or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning to filter, merge, and alter objects within lists in Python proves helpful for you in your own projects.

Further References & Resources

NFTrade website
Python documentation
CoinGecko cryptocurrency tracker and analytics site
First blog post about scraping data from a lazy-loading website using Selenium Python
Follow up blog post about limiting data searches to a particular element on a page instead of the whole page when using XPath

Use Selenium with Python to Target the XPath of a Particular Object

Paige Niedringhaus — Mon, 18 Dec 2023 00:00:00 +0000

Introduction

Earlier this year, I needed to write a program for my friend to collect the data of one of the NFT collections on the NFTrade site. He wanted the following data included: all the NFTs currently for sale in the collection and the price of each NFT in US dollars based on the current market price of the BNB cryptocurrency that the NFT is for sale in, and he wanted it listed out line by line in a CSV file that he could sort and manipulate.

Unfortunately, the NFTrade website does not have a public API so instead of writing a Node.js script to fetch the data via HTTP calls, I built a small script to navigate to the website page and actually "scrape" the data off of it.

Having not written a web scraper before, I chose to write the program in Python as it seems to be a very popular programming language choice for a task such as this. While I built this scraper, the project requirements evolved and got more complex, and I learned a bunch of useful new techniques when using Python, which I'll be sharing in a series of posts over the coming months.

After I'd settled on using the Selenium Python package to start a Selenium WebDriver instance and scrape the data from NFTrade, I hit a snag: I was collecting all the individual NFT data to loop through and pull the details from, but each time the loop ran it only collected the data from the first NFT in the list.

I was stumped and turned to the Internet for help and (once again) Stack Overflow came through for me.

When extracting scraped web data using Selenium WebDriver's XPath to target the data, in order to search inside a particular element instead of the whole document, a period (.) must be added in front of the XPath. I'll show you how to do it in this post.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Selenium Python package

For my project, I ended up using the Selenium Python package because it can scrape websites with dynamically loaded data, which is how NFTrade works. A user lands on a collection of NFTs and as they scroll down the page, periodically more NFTs are loaded into the browser window via JavaScript.

Selenium Python operates the Selenium WebDriver software and drives a browser just like a user would, and although its original use was for automated end-to-end testing of software, it can also be used to scrape data off of live web pages.

If you'd like to read more about how to use Selenium Python for this, I encourage you to visit my first blog post on this subject where I go in depth on it.

What is XPath?

Among the many useful methods included with the Selenium Python package are the methods find_element(), find_elements(), and By.XPATH.

The find_element methods do what their name implies: find an element (or elements) given a By strategy and locator.

By accepts element IDs, names, attributes, XPaths, class names, etc. And XPath is a syntax used to navigate through elements and attributes in a standard XML document (webpage).

Due to the NFTrade site's structure, I used XPath expressions to identify all the individual NFTs on the page and scrape the data from each NFT to include later in my CSV file.

The Problem: XPath was targeting the whole document

After I'd written the code to initially fire up my Selenium WebDriver instance, navigate to the NFTrade site, and load the NFTs into the browser that I wanted to scrape the data from, I had a list of NFT info I needed to slim down to just the data points I wanted to include in the CSV.

NOTE: If you'd like to see how to scrape the browser data in-depth, read my first blog post here.

For me, this was things like: the NFT ID number (part of the NFT card's name) and the the NFT's sale price in BNB.

Inside of the __main__ method in my Python script, I'd scraped the data from the webpage with the get_cards() method, and then I wanted to loop through the NFT data I'd collected and extract the data points from each NFT card with my get_nft_data() method.

Here's the __main__ method code for reference:

for_sale_scraper.py

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = []
   for card in cards:
    info = (scraper.get_nft_data(card)) 
    card_data.append(info)

    # pprint the card data to ensure we're getting the correct data out of each card
    pprint(card_data)

And here is the code for my get_nft_data() method.

def get_nft_data(self, nft_data):
        """Extracts and prints out NFT specific data."""
        nft_name_element = self.driver.find_element(By.XPATH, '//div[contains(@class, "Item_itemName__ckoHR")]')
        nft_name = nft_name_element.get_attribute("innerHTML")
        nft_id = nft_name.partition('#')[-1]

        nft_price_element = nft_data.find_element(By.XPATH, '//div[contains(@class, "Item_itemPriceValueTxt__lblqJ")]')
        nft_price = nft_price_element.get_attribute("innerHTML")
        return {
            'id': int(nft_id), 'price': nft_price
            }

What my get_nft_data() method is supposed to do is:

Use the XPath within each card to get the NFT's name via get_attribute("innerHTML") (innerHTML targets the text content of the element which includes its ID number at the end of the name), partition() the string into a tuple based on the # included in the name, and take the last element in the tuple (which is the ID number).
It should also get the NFT's price (also using get_attribute("innerHTML")) via a second XPath within the NFT card.
Finally return those two elements together as a new object with the keys of id and price.

In theory, I wanted that to happen for every NFT I'd collected into my card_data list. In practice, I got 200 elements that all contained the ID and price data from the very first card in my card_data list.

Not quite what I was expecting.

The Solution: How to restrict XPath inside a particular element

After several failed variations of the code above and searching through many Stack Overflow posts without success, I finally wrote my own SO post explaining my situation and asking for help from the greater web development community.

If you'd like to see my original Stack Overflow post and the helpful responses provided, here is a link.

Just over 30 minutes after posting my question, a kind soul answered it and got me moving forward again. That's the power of the web dev community at its finest.

Below is the corrected get_nft_data() code that actually gets the data from each individual NFT as the data is looped over. I also added some comments between lines of code to explain what's happening at each step.

def get_nft_data(self, nft_data):
      """Extracts and prints out card specific data."""
      # get full card name "NFT_CARD #1234" by XPATH
      nft_name_element = nft_data.find_element(By.XPATH, './/div[contains(@class, "Item_itemName__ckoHR")]')
      nft_name = nft_name_element.text
      # parse out just ID number from name
      nft_id = nft_name.partition('#')[-1]

      # get nft recently sold value by XPATH
      nft_bnb_sale_price = nft_data.find_elements(By.XPATH, './/div[contains(@class, "Item_itemPriceValueTxt__lblqJ")]')

      # if there is a for sale price, take it 
      if nft_bnb_sale_price:
          nft_price = nft_bnb_sale_price[0].text
      else: 
      # if there's no value, just put None in place of a value    
          nft_price = None   

      return {
          'id': int(nft_id), 
          'nft_price': nft_price
          }

In this version of the Python code, there's three main differences.

The first is that instead of using self.driver.find_element, this code is using nft_data.find_element. By substituting nft_data instead of self.driver, it allows the XPath search to be restricted to a particular element.
Second, inside of each find_element method referencing By.XPATH, the XPath being passed has a . in front of it. So '//div[contains(@class, "Item_itemName_ckoHR")]' becomes './/div[contains(@class, "Item_itemName__ckoHR")]'.
Last, the user suggested I could use .text to get the NFT name and BNB price instead of having to write out the lengthier .get_attribute("innerHTML") each time to reach the text in the NFT, which was a nice improvement in code readability.

The solution also mentioned there was a chance that some of the NFTs collected from the page may not have a price listed (NFTrade displays all NFTs in a collection, not just the ones for sale), and recommended wrapping the code that gets the nft_price in an if / else block so if the price is present, it will be collected and returned, and if it's not, it'll return None in place of the value and not throw an error in the code.

Hence this code for checking sale price:

 # if there is a for sale price, take it 
      if nft_bnb_sale_price:
          nft_price = nft_bnb_sale_price[0].text
      else: 
      # if there's no value, just put None in place of a value    
          nft_price = None

Test the refactored code

With my newly refactored get_nft_data() method at the ready, it was time to test it out in my Python script.

As a reminder, my __main__ method looked like this:

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();
   cards = scraper.get_cards(max_card_count=200)
   card_data = [] 
   for card in cards:
    info = (scraper.get_nft_data(card))
    card_data.append(info)

    # pprint the card data to ensure we're getting the correct data out of each card
    pprint(card_data)

This time, when I ran the script from the command line with python for_sale_scraper.py, here's a screenshot of the output I received.

If you look closely you'll see that each object in this area has a different ID and price in it.

As you can see from the image, I got an array of items and each item in the array had a different id and nft_price from the others. Now the get_nft_data() method was working correctly, targeting the next NFT in the card_data array with each successive iteration in the loop and pulling out the data specific to that card.

Success!

With that hurdle overcome, I was ready to move on to the next steps of this project: converting the NFT's BNB prices into the current USD prices and assembling them into a CSV spreadsheet. Those tasks will be covered in detail in upcoming blog posts.

Conclusion

When I was asked to put together a spreadsheet of all the NFTs for sale in a particular collection on NFTrade, I ended up using Python to build a website scraper to accomplish it and learned a lot of new problem-solving techniques in the process.

I managed to load and collect all the NFT data from a web page with the assistance of the Selenium Python package, but I got throughly stuck when trying to iterate through my data to extract the ID and price for each NFT from it.

Luckily, the Stack Overflow community came through for me and taught me about the finer points of using Selenium WebDriver's XPath to target specific elements within a page instead of the whole page when looking for particular pieces of data, which got me unstuck and on my way to building my spreadsheet of data.

Thank goodness for the knowledge sharing of the online web development community - I am truly grateful to be able to turn to them when I've exhausted my own ideas to solve a problem.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning how to restrict XPath searches to a particular element on a page instead of the entire page proves helpful for you in the future like was for me.

Further References & Resources

Selenium Python docs
Selenium WebDriver docs
NFTrade website
XPath documentation
Original Stack Overflow post
Previous blog post about scraping data from a lazy-loading website using Selenium Python

Scrape Data from a Lazy Loading Website with Selenium Python

Paige Niedringhaus — Sun, 22 Oct 2023 00:00:00 +0000

Introduction

A few months ago, my friend wanted me to write a program to collect the data of one of the NFT collections on the NFTrade site, compute the current price of each NFT in US dollars based on the current market price of the BNB cryptocurrency it was listed for sale in, and compile all of the NFTs for sale into a CSV file that he could sort and manipulate.

Unfortunately, the NFTrade website does not have a public API so writing a Node.js script to fetch the data from the API and format it as required was not an option. Instead, I needed to make a site scraper to actually go to the website page and "scrape" the data off of it.

Having not written a web scraper before (and also wanting to make the script easier for my friend to update and run on his own machine), I decided to write the program in Python (it seems to be a very popular programming language choice for a task such as this). Along the way, my little web scraper's requirements evolved and got more complex, and I learned a bunch of useful new techniques about using Python for my project, which I intend to share in a series of posts over the coming months.

My first attempt to scrape the data from NFTrade was unsuccessful beyond locating the first 75 NFTs on the page. I figured out this was because NFTrade (as many other websites do) lazy loads NFTs onto the page 75 at a time: once the user's scrolled down far enough to reach the end of the currently visible items, the site loads the next batch of elements onto the page (essentially a fancier version of pagination). So I needed a way to have my web scraper program collect whatever data was available on the page then scroll down far enough to trigger more data to load and collect that, and rinse and repeat.

After some trial and error, I finally found a working solution with the help of a Python package named Selenium Python, and I'll share with you today how to write your own Python script to scrape data from a lazy loading website with Selenium WebDriver.

NOTE: I am not normally a Python developer so my code examples may not be the most efficient or elegant Python code ever written, but they get the job done.

Selenium with Python package

There are a few different popular Python packages available for web scraping which I tried before reaching for Selenium, but I had an issue with them in that they only worked for static websites that were generated at build time, not for sites that are generated on the client-side via JavaScript, like NFTrade is.

To that end, I had to do a little digging to find a package that could work with scraping sites with dynamically loaded data, and I ran across the Selenium Python package during my investigation.

Selenium Python is a Python-based API that allows users to write scripts or automated tests using Selenium WebDriver in an intuitive, Python-flavored way. And Selenium WebDriver is a software that can drive a browser natively, as a user would, either locally or on a remote machine. Originally created back in 2004, some version of Selenium has been around for years and is considered one of the earliest versions of automated testing that emulates user actions on a web page (commonly known today as end-to-end testing).

The cool thing about WebDriver though, is that its uses span beyond automation testing, as scripts can actually be written to scrape data off of live web pages, and that's just what I ended up doing in my Python script, so let's get started.

Install Selenium Python in the Python project

As with most projects, the first thing to do is add the Selenium Python package to the Python project. The easiest way is to use pip to install the Selenium package.

Assuming you have pip on your machine, at the root of your Python project folder, run the following command from a terminal.

pip install selenium

Then, add the selenium package to your requirements.txt file so anyone downloading the repo in the future can install all the necessary project dependencies.

requirements.txt

selenium

And that's all it takes to be ready to use WebDriver in your Python script. Simple enough.

Import Selenium WebDriver into Python script

After adding the Selenium Python bindings to the project, it's time to import Selenium's WebDriver and some of its helpful configuration options to the actual Python script that does the website scraping. I named my file for_sale_scraper.py since I was specifically looking for NFTs that are for sale (not all of the NFTs listed on NFTrade are - some are just visible but not actually available to purchase), but you can choose any sort of file name that makes sense for you.

Below are the imports I added to my file. I'll break down what each one is doing below.

for_sale_scraper.py

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By

The very first import line brings in the selenium.webdriver module and provides all the WebDriver implementations.

from selenium import webdriver

Next, as I chose to use Chrome as the browser I wanted WebDriver to interact with (Selenium supports Firefox, Chrome, Edge, and Safari browsers), I imported the Options class from the selenium.webdriver.chrome.options module. This allowed me to add specific config details about how I want the Chrome browser to be set up when the Python script runs against it: things like headless mode or disable extensions, etc.

from selenium.webdriver.chrome.options import Options

I'll cover the arguments I passed here in detail in the next section.

WebDriverWait, added in the third line of imports, is part of the special sauce that makes WebDriver a good solution for sites like NFTrade that dynamically fetch data on the client side: it allows for implicit and explicit wait times before trying to locate an element on the page, which gives the browser time for data to come back from the server and populate in the DOM.

from selenium.webdriver.support.wait import WebDriverWait

This type of wait is an "explicit wait", meaning I manually set a period during which the code will wait before continuing to try and execute.

And finally, there is the import for By. By is what allows me to locate elements on the page - it is immeasurably useful and powerful.

The By class accepts element IDs, names, attributes, XPaths, link text, tag names, class names, and CSS selectors just to name a few, and once again, it is a key player when it comes to scraping data off of the web page, as I'll demonstrate soon.

from selenium.webdriver.common.by import By

Right, all the Selenium WebDriver imports are now present in the Python file, time to initialize them and get to work.

Add methods to scrape data and lazy load more data

Before WebDriver can begin scraping the data from NFTrade, an instance of the browser that WebDriver will interact with must be instantiated and the proper options supplied to it.

1. Initialize the Selenium WebDriver instance

In my attempt to try to follow good Python coding practices (again, disclaimer: I don't write Python as my primary coding language), I created a class for the the file named class ForSaleNFTScraper, and created an __init__ () method immediately inside of it where I created the Chrome WebDriver instance that the whole script will be able reference in the remainder of its methods.

class ForSaleNFTScraper:
    def __init__ (self):
        options = Options()
        options.add_argument('--headless')
        options.add_argument('--start-maximized')
        self.driver = webdriver.Chrome(options=options)
        self.wait = WebDriverWait(self.driver, 5)

# more code here

The first thing I did inside of the __init__ () method was to add a couple of Chrome browser configs via the Options import from the last section by declaring a new options variable.

    options = Options():

Since I wanted this script to run without actually opening a browser window on the user's local machine, I added the config argument of --headless and the argument of --start-maximized, so the (unseen) window would take up as much screen size as was available (and hopefully load as many NFTs as quickly as possible by doing so).

    options.add_argument('--headless')
    options.add_argument('--start-maximized')

Then I passed the new options object to the instance of webdriver.Chrome, which was set to the variable of self.driver (self is a variable accessible throughout the rest of the methods within this ForSaleNFTScraper class), and instructed the new WebDriver to wait for 5 seconds after startup (which would presumably give it time to go to the specified NFTrade web URL and load the data onto the page before attempting to scrape it).

    self.driver = webdriver.Chrome(options=options)
    self.wait = WebDriverWait(self.driver, 5)

There's plenty happening in that first method, but it's all pretty straightforward once you go through the code line by line and understand what the arguments mean to the Chrome WebDriver instance, and why it's doing what it's doing. Now that the WebDriver instance was configured and ready to go, I could write the code fetching the NFT card data, and lazy loading more data once the end of the currently visible info was reached.

2. Write the get_cards() and get_current_card_count() methods

This is where the code really starts to get interesting in my opinion, because it's where I learned to collect whatever data was currently visible in a (headless) browser and then load more data to add to the list. Pay close attention, because this is where the lazy loading code resides that gets more and more data onto the page.

def get_current_card_count(self):
    """Get the count of cards loaded into list of cards."""
    return len(self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]'))    

def get_cards(self, max_card_count = 500):
    """Extract and returns card ID and price."""
    URL="https://nftrade.com/collection/[NFT_COLLECTION_NAME]"
    self.driver.get(URL)
    last_card_count = 0
    # loops through lazy loading cards on site until max_card_count number reached 
    while last_card_count < max_card_count:
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)
        last_card_count = self.get_current_card_count()
    # grab all the cards now loaded into the browser by XPATH
    cards = self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]')
    return cards

# more code here

Ok, here we go.

For starters, there are two methods that I'm displaying in the code snippet here. The first method, get_current_card_count() is how I keep track of how many NFT cards in a collection are currently visible on the screen.

As I've said, NFTrade lazy loads its NFT collections onto a site to make initial page load quicker, and when a user scrolls down to the end of the currently loaded batch of elements, the NFTrade page then triggers to load more cards into the DOM at that point in time.

The second method is get_cards(), which handles going to the NFTrade collection URL and scraping all the available card data. It relies on get_current_card_count() to help it know to load more NFT cards until the desired number of cards has been loaded in the browser to scrape data from.

get_cards() method

I'll talk about get_cards() first as it's the more complicated of the two methods. The first thing the method does is declare a new variable named URL - this variable is set to the URL of the NFTrade collection page I want WebDriver to navigate to and scrape the data from. I used the Selenium WebDriver driver.get() method to navigate to the page given by the URL.

   URL="https://nftrade.com/collection/[NFT_COLLECTION_NAME]"
   self.driver.get(URL)

After navigating to the proper URL, I created a variable called last_card_count and set it equal to 0: this variable will be used to track how many NFTs are currently visible on the page and compare it to the max_card_count variable passed to the get_cards() method (if a number isn't passed for max_card_count it defaults to 500).

Below is the key code to lazy loading more and more data in the browser

Inside of get_cards(), there's a while loop set up to compare the last_card_count and max_card_count variables. As long as last_card_count is less then max_card_count, the loop will run, and each time it executes WebDriver uses the driver.execute_script() method to scroll down the page, wait for 3 seconds (allowing more cards to load onscreen), and then updating the last_card_count variable equal to the new amount of cards on the page using the get_current_card_count() method.

NOTE: The window.scrollTo() method is critical

driver.execute_script() allows for the synchronous execution of JavaScript in the current window, so when you see the code self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);"), what's happening is that WebDriver is using the JavaScript window.scrollTo() method to scroll the browser all the way to the bottom of the page (that's why document.body.scrollHeight is present - it's a measurement of the height of the whole document.body page element), which triggers the page to load more NFT cards into view.

    last_card_count = 0
    # loops through lazy loading cards on site until max_card_count number reached 
    while last_card_count < max_card_count:
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)
        last_card_count = self.get_current_card_count()

And this is a perfect time to segue into discussing the get_current_card_count() method, which is short and sweet.

get_current_card_count() method

This method exists simply to find the count of the current elements loaded in the browser, and it does so by combining the WebDriver find_elements() method with the By.XPATH element locator method.

Due to how the NFTrade site is built, there are no easily identifiable classes, IDs, or other consistent ways to identify all the cards on the page, so I had to resort to XPath expressions to identify each element and include it in my count to update the last_card_count variable. I cobbled together the XPath below by using my Chrome DevTools to inspect the elements on the page and construct the XPath from there through trial and error.

NOTE: What is XPath?

If you're unfamiliar like I was, XPath is a syntax that can be used to navigate through elements and attributes in a standard XML document (or webpage). The link I provided to W3Schools has some good examples of what typical XPath expressions look like and how to interpret them.

So the code inside of the get_current_card_count() method is just the one line of code:

return len(self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]'))

In the code snippet, I'm getting the count (using the build-in Python method len()) of all the elements on the page that match the XPath of a <div> containing the class of "Item_itemContent__1XIcH", because each NFT on the page is wrapped by that <div> with that class. It's not the prettiest thing to read and understand, but it gets the job done.

And finally, jumping back to the get_cards() method again, once the last_card_count variable has been updated and surpassed the max_card_count variable (i.e. enough NFT cards are loaded into the browser), the while loop ends, and all the cards on the screen are targeted (using the very same XPath used in the get_current_card_count() method, I might add) and set equal to the cards variable defined at the top of the get_cards() method. That variable then gets returned to the __main__ method running the whole script, which I'll cover next.

    # grab all the cards now loaded into the browser by XPATH
    cards = self.driver.find_elements(By.XPATH, '//div[contains(@class, "Item_itemContent__1XIcH")]')
    return cards

There's quite a bit going on here, but hopefully it makes more sense now what these methods are doing. Time to test out this lazy loading script functionality and see how WebDriver does.

3. Run the Python script

All right, now that all the code and logic to load multiple sets of NFTrade cards into the browser and collect the data has been written, it's time to run the code.

To do that, I declared a __main__ method at the bottom of the file which can be started from the terminal with the following command.

python for_sale_scraper.py

Here is what the __main__ method includes.

if __name__ == ' __main__':
   scraper = ForSaleNFTScraper();

  # get all the cards from nftrade site
   cards = scraper.get_cards(max_card_count=200)

   # pprint the card data to ensure we're getting data
   pprint(cards)

   print("Total cards collected:", len(cards))
   # more code here

The first thing the method does is create a new instantiation variable named scraper by calling the ForSaleNFTScraper() class. It then proceeds to fetch all the card data and set it equal to a variable named cards by calling the method scraper.get_cards(max_card_count=200) and supplying a max_card_count variable of 200.

After this step, as a sanity check, I used the Python pprint() and print() methods to print out all the card data and a count of the total cards fetched by the get_cards() method, and ensure all the info I needed to include in the CSV (NFT price, NFT ID, etc.) was available to me. Here's a screenshot of some of the data printed out in my console helping me know my code is doing what I expect.

Here is what the raw NFT card data gathered from the get_cards() method looks like printed in the terminal.

Since I set my max_card_count to 200, but NFTrade loads NFTs in batches of 75 at a time, it makes sense that the total count of NFTs scraped off the page equals 225.

And after verifying the right data's there (and the right amount of data as well), I continued on extracting the data, calculating the current price in USD for each NFT, and assembling a CSV of all the data. But I'll save those steps for future blog posts.

Conclusion

Building a Python-based website scraper to create a CSV of NFTs available for sale on NFTrade was a unique challenge I learned a lot of new things from.

After my first attempt failed due to NFTrade dynamically lazy loading NFTs in batches of 75 onto the page as a user scrolled further down, I had to come up with a more creative solution that would allow me to trigger the site to load more cards on the page first, then grab the data on the cards for sale.

I found the solution I was looking for with the help of a Python package called Selenium Python. Selenium Python is a powerful Python-based API that allows users to write scripts or automated tests leveraging Selenium WebDriver. And it was up to the task at hand: with just a few methods I was able to specify as many NFTs as I wanted loaded on the page before scraping and collecting all their data all at once.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope seeing how to make a Python Selenium WebDriver load data onto a dynamic webpage before scraping it comes in handy for you in the future.

Further References & Resources

Copy Files from One Repo to Another Automatically with GitHub Actions

Paige Niedringhaus — Sun, 13 Aug 2023 00:00:00 +0000

Introduction

Welcome to the third (and final) installment in my series about the many things I learned while building my first open source API library for Blues, the Internet of Things startup I work for.

It was a good challenge and a great learning opportunity for me.

I automated as many pieces of the library's maintenance and deployment to as I could with the help of GitHub Actions workflows, one of them being: copy the openapi.yaml file from the Blues cloud repo, Notehub, whenever that file was updated, and push it into a feature branch in the Notehub JS repo (the name of the API library).

Interacting with two different repos in one workflow might sound tricky, and at first it was, but it is possible and not too bad once you see how it can be done.

In this tutorial, I'll demonstrate how to create a GitHub Actions workflow that automatically copies a file from one repo to another and pushes the changes to another repo inside a feature branch.

I know the use case sounds a bit specific, but it came in really handy for me, and you never know when it might for you too.

Notehub JS

Before we get to the actual GitHub Actions workflows, I'll give you a little background on the Notehub JS project.

This section outlines the folder structure for the repo in case you want to explore it in GitHub, if you just want to get to the solutions, feel free to jump down to the next section.

Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it is actually generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

The OpenAPI Generator CLI is a tool that uses the openapi.yaml file to create an entire API library complete with documentation, models, endpoints, and scripts to package it up for publishing as an npm module. The end library that is published to npm is a subfolder inside of the main Notehub JS repo. At the root of the project are the openapi.yaml file, the GitHub Actions workflows, and a few other config files.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── create-pr.yml 
| ├── PULL_REQUEST_TEMPLATE.md
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

Based on the diagram above, the openapi.yaml file that the library is generated from lives at the root of the repo, and the src/ folder is what holds all the Notehub API JavaScript code that powers the Notehub JS library.

This openapi.yaml file gets copied from the Notehub repo when changes are made to it. And changes are made to that file whenever the Notehub API is updated with new features and functionality, so making sure that the Notehub JS library keeps up with the new additions to the original API it's based on is important.

Now that I've explained more about the Notehub JS repo and why keeping it in sync with the Notehub API is important (and depends on the openapi.yaml file), we can get down to the business of ensuring the file gets copied to this repo whenever it's updated.

Looking for more details about Notehub JS?

Check out my first blog post about how to use GitHub Actions to automatically publish new releases to npm - I do a fairly deep dive on Notehub JS there.

Create a GitHub Actions workflow to run when the `openapi.yaml` file is updated

First things first: creating a new GitHub Actions workflow that's triggered when the openapi.yaml file changes in the Notehub repo.

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

Below is the final version of the copy-openapi-file.yml file, which lives inside of the ./github/workflows/ folder. I'll walk through each step of the script below.

.github/workflows/copy-openapi-file.yml

name: Copy updated OpenAPI file

# only run this workflow when the openapi file has changed on the master branch
on: 
  push:
    branches: 
      - master
    paths:
      - 'notehub/api/openapi.yaml'

jobs:
  copy_openapi_file_to_notehub_js: 
    runs-on: ubuntu-latest
    steps:
    # check out notehub project
    - name: Check out Notehub project
      uses: actions/checkout@v3
    # check out notehub-js project using token generated in that project to successfully access it from inside the Notehub GitHub Action workflow
    - name: Check out Notehub JS project
      uses: actions/checkout@v3
      with:
        repository: blues/notehub-js
        path: ./notehub-js
        token: ${{ secrets.NOTEHUB_JS_TOKEN }}
    # make a copy the openapi file from notehub project
    - name: Copy OpenAPI file
      run: bash ./.github/scripts/copy-openapi-file.sh
      env:
        DESTINATION_PATH: ./notehub-js/
    # make a branch in notehub-js repo and push the copy of the openapi file there
    - name: Push to notehub-js repo
      run: bash ./.github/scripts/push-openapi-to-notehub-js.sh
      env:
        BRANCH: feat-openapi-update

Each GitHub Actions workflow needs a name , and I try to choose descriptive ones like this one: Copy updated OpenAPI file.

Then, I want this workflow to only run when the openapi.yaml file in the master branch of the project is changed. master is the branch that gets deployed to production, and in my case it's a safe assumption that when there's a change to this file in master, those changes are going to prod, which is when I want to copy the file over to the Notehub JS repo so it can reflect those same changes.

The code snippet below does just that.

on: 
  push:
    branches: 
      - master
    paths:
      - 'notehub/api/openapi.yaml'

on is how a workflow is triggered.
push is the event that triggers the workflow.
branches defines which branch names this workflow should run for.
paths is another filter to even more finely determine when the workflow runs.

What it boils down to is: when there's a change to the master branch in the repo for the openapi.yaml file at this particular file path, run the GitHub Actions workflow.

From there, the copy_openapi_file_to_notehub_js job runs. There's only one job in the jobs section of this script, but if there were multiple jobs, they'll run sequentially.

The create_pr_repo_sync job defines that it runs on the latest version of Ubuntu in runs-on.

And finally, I get to the steps section. Here's how it goes:

Check out the Notehub code so the workflow can access it using the GitHub Action actions/checkout@v3.
Check out the Notehub JS code so the workflow can access it as well using the GitHub Action actions/checkout@v3.
- NOTE : Notice that in this second checkout action I had to include a with parameter that allows me to specify another repo to check out by providing a repository, path, and token. The token is a GitHub Personal Access Token (PAT) required when checking out other libraries beyond the current one the workflow is operating within.
Run a shell script named copy-openapi-file.sh with the GitHub Actions environment variable DESTINATION_PATH passed to the script.
Run another shell script named push-openapi-to-notehub-js.sh with the GitHub Actions environment variable BRANCH passed to the script.

In the following sections I'll go through what those two shell scripts do in detail.

Ok, so the workflow file is done. Time to get into the details of the two scripts doing the heavy lifting.

Write one shell script to copy the `openapi.yaml` file to the Notehub JS repo

Premade GitHub Actions can do many things, but something as specific as copying a file from one repo to another is a bit beyond them.

Luckily, GitHub Action workflows can run shell scripts, so if you can use a scripting language like PowerShell or Bash, you're in luck.

Here is the Bash script I wrote to copy the file from the Notehub repo to the Notehub JS repo.

copy_openapi_file.sh

#!/usr/bin/env bash
# bash boilerplate
set -euo pipefail # strict mode
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
function l { # Log a message to the terminal.
    echo
    echo -e "[$SCRIPT_NAME] ${1:-}"
}

# File to copy from Notehub
OPENAPI_FILE=./notehub/api/openapi.yaml

# if the file exists in Notehub, copy it to Notehub-JS repo
if [-f "$OPENAPI_FILE"]; then
    echo "Copying $OPENAPI_FILE"
    cp -R ./notehub/api/openapi.yaml $DESTINATION_PATH
fi

echo "OpenAPI file copied to $DESTINATION_PATH"

The 9 lines at the top of the file are Bash script set up, so don't worry too much about what's happening there.

The first line to pay attention to is where the OPENAPI_FILE variable is defined with a path to the openapi.yaml file in the Notehub repo.

After identifying the file to copy from the Notehub repo, a standard Bash if statement checks if the file exists in the Notehub repo (just to be safe), and if it does, it copies the file recursively using cp -R to the $DESTINATION_PATH variable that was provided by the GitHub Actions environment variable.

In this case, the $DESTINATION_PATH is pointing to the root folder of the Notehub JS project where the file should be copied to, but you could specify it copy the file anywhere within the receiving repo that makes sense.

During the copying process and when the script finishes copying the file I added a few echo commands to print out useful messages in the logs to let me know things are going to plan.

And that's the first shell script that actually copies the file that will be taken from one repo to the other. I hope it's not too complicated once the actions in the script are broken down.

Now, it's time to add that file to the Notehub JS repo.

Use a second shell script to push the copied file to a feature branch in the other repo

When I checked the GitHub Marketplace for a ready-made action to create a new feature branch in the repo receiving the copied file, there were a few to choose from, but I wanted more control over what was happening than they offered.

I wanted an action that would:

Create a new branch in the Notehub JS receiving repo if it didn't exist, add the copied file, and then push that branch to GitHub.
Check out the already existing branch in the Notehub JS repo, overwrite any previous file changes in the branch, and then push the new file changes in that branch back to GitHub.

Here's the bash script that I came up with to handle those two scenarios.

push-openapi-to-notehub-js.sh

#!/usr/bin/env bash
# bash boilerplate
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
function l { # Log a message to the terminal.
    echo
    echo -e "[$SCRIPT_NAME] ${1:-}"
}

# move to the root the notehub-js repo
cd "./notehub-js"
echo "Open root of Notehub JS repo"

# check if there's already a currently existing feature branch in notehub-js for this branch
# i.e. the altered openapi.yaml file's already been copied there at least once before
echo "Check if feature branch $BRANCH already exists in Notehub JS"
git ls-remote --exit-code --heads origin $BRANCH >/dev/null 2>&1
EXIT_CODE=$?
echo "EXIT CODE $EXIT_CODE"

if [[$EXIT_CODE == "0"]]; then
  echo "Git branch '$BRANCH' exists in the remote repository"
  # fetch branches from notehub-js
  git fetch
  # stash currently copied openapi.yaml
  git stash
  # check out existing branch from notehub-js
  git checkout $BRANCH 
  # overwrite any previous openapi.yaml changes with current ones
  git checkout stash -- .
else
  echo "Git branch '$BRANCH' does not exist in the remote repository"
  # create a new branch in notehub-js 
  git checkout -b $BRANCH
fi

git add -A .
git config user.name github-actions
git config user.email github-actions@github.com
git commit -am "feat: Update OpenAPI file replicated from Notehub"
git push --set-upstream origin $BRANCH

echo "Updated OpenAPI file successfully pushed to notehub-js repo"

Once more, the first 9 lines of code are boilerplate for Bash; no comments necessary.

The first thing this script does is situate itself in the root of the Notehub JS folder just as you would when navigating around a computer via the command line: cd "./notehub-js".

Once there, it checks if the feature branch environment variable passed in by the GitHub Action $BRANCH already exists in the Notehub JS repo with this one-liner.

git ls-remote --exit-code --heads origin $BRANCH >/dev/null 2>&1

Essentially, this code is:

Getting a list of references in the remote repository with git ls-remote,
Checking if env var $BRANCH (passed from the GitHub Action) exists only in the refs/heads of the origin remote with --heads
- (i.e. does the branch name already exist in the GitHub repo where it keeps track of the names of all of its branches),
And returning an --exit-code with status "2" when no matching refs are found in the remote repo and a status of "0" if a matching ref is found.

When the exit code is returned, it's set equal to the variable $EXIT_CODE.

Then, an if/else statement checks what $EXIT_CODE equals.

EXIT_CODE = "0"

If the exit code is 0, that means the branch already exists in the repository; this could happen if multiple changes are made to the openapi.yaml file before the feature branch in Notehub JS gets merged in and deleted.

When that happens the following steps take place:

The branches for the Notehub JS repo are fetched via git fetch,
The current changes (the freshly copied openapi.yaml file) are stashed with git stash,
The existing branch of the same name as the $BRANCH env var is checked out locally,
The stashed changes are applied to that branch overwriting whatever was there before with git checkout stash -- ..

EXIT_CODE = "2"

If $EXIT_CODE is equal to 2, that means the branch doesn't already exist remotely in GitHub and the script can simply checkout a new branch with the name in $BRANCH via git checkout -b $BRANCH.

Finally, after all this is completed, a standard set of git commands follows:

Use git-add -A . to stage all changes in the working directory,
Configure a username and email address for who's doing the commit with git config user.name and git config user.email,
Add a commit message with git commit -am "Standard commit message here",
Lastly, push it to GitHub with git push --set-upstream origin $BRANCH.

At this point, you should be good to go. The file should be successfully copied over to the other repository.

NOTE: Inspiration for this second Bash script was initially based on this blog post by Remarkable Mark.

Check that the workflow runs

The simplest way to test this new functionality is just to keep an eye out for changes in the main branch of the repo that would trigger the workflow to run and see if it works.

When you name the workflow a descriptive name, it's easy to find it amongst the various GitHub Action workflows the repo may have.

Here's a screenshot of the workflow runs of the Copy updated OpenAPI file job inside the GitHub repo's Actions page.

And if you want to dig in further to any of the runs to check the logs and job steps (especially handy for debugging purposes), just click on a workflow run, then the name of the job, and inside the job you can expand out each step.

Here's another screenshot of the Push to notehub-js repo step expanded to see exactly what the Bash script in that step is logging to the console via echo.

And you're done! The workflow's running, the steps are successfully completing, and, most importantly, the file from the Notehub repo is successfully copied to the Notehub JS repo. Mission accomplished.

Conclusion

My first foray into building an open source software library on behalf of one of my company's APIs was a unique challenge that taught me a lot. Especially as I got to leverage GitHub Action workflows to automate as many parts of the process as I could to make ongoing maintenance and upkeep easier.

One such task that I automated involved copying a file from one repo into another whenever a change was made to that particular file. And while I was able to leverage a few pre-existing GitHub Actions, I also had to write a couple custom Bash scripts for some of the trickier, more-unique-to-my-use-case steps.

Fortunately, GitHub Action workflows allows for both types of steps to be combined in a workflow, making for some very powerful, turnkey solutions that did just what I needed.

Check back in a few weeks — I’ll be writing more about JavaScript, React, IoT, or something else related to web development.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning to copy a file from one repo to another automatically with GitHub Actions workflows is as helpful for you as it has been for me. Enjoy!

References & Further Resources

Previous OSS article using GH Actions to publish a package to npm
Previous OSS article using GH Actions to create a new PR branch
Notehub JS GitHub repo

Automatically Create a Pull Request Against a Feature Branch with GitHub Actions

Paige Niedringhaus — Sun, 11 Jun 2023 00:00:00 +0000

Introduction

This blog post is the second in a short series I'm writing about the many things I learned in the course of building my first open source API library for the IoT startup I work for, Blues.

It was a great learning experience for me and a new, unique challenge because I needed to do the following things:

Make a copy of the openapi.yaml file from the Blues cloud's repository, Notehub, whenever the file was updated.
Open a new pull request against the Notehub JS repo with the copy of the openapi.yaml file.
Generate a fresh version of the Notehub JS API library based on the that file via the OpenAPI Generator CLI.
And publish the updated Notehub JS library to npm.

And I wanted to automate as many of these steps as possible through the use of GitHub Actions workflows.

Since I didn't know how often the Notehub's openapi.yaml file would be updated, I needed a way to notify myself when a new version of the Notehub's API file needed review in the Notehub JS repository. The best solution I could think of was to open a new pull request in the Notehub JS repo after copying the updated openapi.yaml file into a feature branch and tagging myself to review it.

Today, I'll walk through how to use a GitHub Actions workflow to create (or update) a pull request whenever a new feature branch is made in that repository.

Notehub JS

Before we get to the actual GitHub Actions workflow, let me give you just a little background on the Notehub JS project because it's a bit different than most.

This section helps explain the folder structure for the repo in case you want to explore it in GitHub, if you just want the solutions, feel free to jump down to the next section.

Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it's generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

The OpenAPI Generator CLI is a tool that can use the openapi.yaml file to create an entire library complete with documentation, models, endpoints, and scripts to package it up for publishing as an npm module. The end library that I care about publishing to npm is a subfolder inside of the main Notehub JS repo. At the root of the project are the openapi.yaml file, the GitHub Actions workflows, and a few other config files.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── create-pr.yml 
| ├── PULL_REQUEST_TEMPLATE.md
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

As you can see in the diagram above, the openapi.yaml file that this library is generated from lives at the root of the repo, and the src/ folder is what actually holds all the Notehub API JavaScript code that powers the Notehub JS library.

This openapi.yaml file is what gets copied from the Notehub repo when changes are made to it. And changes are made to the file whenever the Notehub API is updated with new features and functionality, so making sure that the Notehub JS library keeps up with the new additions to the API it's designed to interact with is important.

Now that I've explained a bit more about the Notehub JS repo and why keeping it in sync with the Notehub API is important, we can get down to the business of automating PRs for this repo.

Looking for more details about Notehub JS?

Check out my previous blog post about how to use GitHub Actions to automatically publish new releases to npm - I do a fairly deep dive on Notehub JS there.

Create a standard PR template for the repo

When I was just a few years into my own web development career, I was introduced to GitHub PR templates, an easier way to keep pull requests uniform for a repository worked on by multiple developers. It made the task of creating decent PRs so much better.

There is not a team or repo I join now without adding a ./github/ directory folder and PULL_REQUEST_TEMPLATE.md file if it doesn't already exist.

For the Notehub JS repo, as it's mostly autogenerated, I included a couple of sections to fill in: Problem Context: a brief description of the updates in the PR, and Changes: what code changes were actually made in the PR. Simple enough for any dev to fill out.

PULL_REQUEST_TEMPLATE.md

# Problem Context

## Changes

That's all I wanted to add to the repo before focusing on the GitHub Actions workflow itself. We'll get to it next.

Set up a GitHub Actions workflow to automatically create pull requests

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

For this particular workflow, I was able to string together a few premade GitHub Actions to do just what I needed:

Create a new pull request whenever a new feature branch was pushed to the GitHub repository.
Use the PULL_REQUEST_TEMPLATE.md file to format the PR.
Notify me when a PR is there for review.

Here's what the final create-pr.yml file looks like inside of the ./github/workflows/ directory. I'll dissect it below.

create-pr.yml

name: Automatically create / update pull request

# run this workflow only on new feature branches, not when they're merged to main
on:
  push:
    branches-ignore:
      - "main"

jobs:
  create_pr_repo_sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Create pull request
        id: open-pr
        uses: repo-sync/pull-request@v2
        with:
          destination_branch: "main"
          pr_title: "feat: PLACEHOLDER TITLE"
          pr_template: ".github/PULL_REQUEST_TEMPLATE.md"
          pr_reviewer: "paigen11"
          pr_draft: true

Each workflow file needs a name , so I chose to name this one: Automatically create / update pull request.

This workflow should be triggered whenever a new branch is pushed to the repo but it should ignore the "main" branch. "main" is the main branch of this repo that gets published to npm and doesn't need a PR to be created for it.

on:
  push:
    branches-ignore:
      - "main"

on is how a workflow is triggered.
push is the event that triggers the workflow.
branches-ignore is how to exclude the workflow from running on certain branch patterns. This gives us more fine-grained control of when the workflow should run.

Then the jobs section runs inside of the workflow. This particular script only has one job, create_pr_repo_sync, but if there's multiple jobs, they'll run sequentially unless otherwise specified.

The create_pr_repo_sync job defines that it runs on the latest version of Ubuntu in runs-on.

Finally, I get to the steps.

The steps are as follows:

Checkout the code so the workflow can access it using the GitHub Action actions/checkout@v3.
Create a pull request using the GitHub Action repo-sync/pull-request@v2
Pass the PR action a custom pr_title, pr_template, and pr_reviewer. Set it to be created as a pr_draft (I like PRs to be drafts until I've looked them over and know they're ready for review), and open the PR branch against the destination_branch: "main".

In the pr_template section, I pass the path to the PR template file I made in the previous section, and for the pr_reviewer I add my own GitHub username so that I get an email notification when the new PR is created in GitHub.

Test out the workflow

If you want to test this functionality out, create a new local branch of the repo, make a change in the branch, and push it to GitHub. When that new branch registers, the GitHub Actions workflow will be triggered to run.

If you visit the Actions page in the GitHub repo, you should see the "Automatically create / update pull request" job running.

And after the action finishes running, there should be an email sent to the folks tagged as the PR reviewers in the GitHub Action (me, in this case).

And with that, the workflow working has been confirmed, and the reviewers know it's time to take action in that repo.

Conclusion

When I built my first open source software library, I learned a lot. My software is a JavaScript library based on my company's API, Notehub, so every time the Notehub API gets updated, the JS library based off of it, needs to be updated too.

I needed a programmatic way to let myself know whenever new changes were made to the JS repo that needed to be reviewed.

The easiest way I could think to accomplish this was by automating the pull request creation so that when a new feature branch was pushed to the repo, it opened a PR and tagged myself as the reviewer.

Luckily, a GitHub Action existed for just such a scenario, allowing for designating a PR template, a PR reviewer, giving the PR a placeholder title, and a whole host of other configurations. The workflow made it quite straightforward, and gave me the peace of mind that whenever changes are made that I need to review, I get an email to look at what's changed.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning to use GitHub Actions workflows to automate a new PR when a feature branch is pushed to a repository proves useful. Enjoy!

References & Further Resources

Previous OSS article using GH Actions to publish a package to npm
Notehub JS GitHub repo
GitHub PR templates
Pull request GitHub Action

Use GitHub Actions to Automatically Publish a Repo Subfolder as an npm Library

Paige Niedringhaus — Wed, 12 Apr 2023 00:00:00 +0000

Introduction

I've said it before, and I'll say it again, working as a software engineer for a startup is fun and challenging because of the sheer number of different things I get to do on a regular basis.

I work for an Internet of Things startup called Blues which specializes in getting IoT data from devices in the real world into the cloud via cellular. To help show our customers all the possibilities of what they can do with our hardware (Notecards) and our cloud (Notehub), a group of my coworkers and I have been building lots of JavaScript-based web apps to monitor and display IoT data for a variety of different use cases.

A little more about Notecards and Notehub.io

Notecards are low-power, 30mm x 35mm, prepaid cellular-connected devices designed to integrate with any IoT device, and Notecards know, out of the box, how to connect with the Notehub cloud to send as well as receive data from it.

Notehub.io is a cloud-based service for securely connecting to Notecard devices and storing data from them before sending it on to your cloud application of choice.

It quickly became apparent that rewriting the same HTTP calls in multiple projects to fetch data from the native Notehub API endpoints wasn’t easy to reuse or keep up to date as the API continues to grow and evolve.

So we created our own open source, JavaScript-based library for the Notehub API and published it on npm for anyone to use for free. Notehub JS is designed to get you connected to the Notehub API quickly, and allow you to access all of the API routes relevant to interacting with Notehub in a JavaScript-friendly way.

Figuring out how to build an open source library like this and publish it to npm was a first for me, and I learned a lot of interesting things along the way that I'd like to share with you all over the course of a few blog posts in the hope that it might help you if you decide to undertake a similar effort.

In this blog, I'll show you how to set up a GitHub Actions workflow to automatically publish a new release of a GitHub repo to npm - no muss, no fuss, no manual input required.

Notehub JS

Ok, before we get to the GitHub Actions workflow, I want to give you a little background on the Notehub JS project because it's a bit unusual.

As I mentioned above, Notehub JS is a JavaScript-based library for interacting with the native Notehub API, and it's actually generated from the Notehub project's own openapi.yaml file, which follows the OpenAPI specification standards.

OpenAPI Specification

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to HTTP APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with a minimal amount of implementation logic.

An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

OpenAPI Generator CLI

Because the Notehub API has an openapi.yaml file that is updated regularly, I was able to use the OpenAPI Generator CLI to automatically generate a JavaScript-based library to interact with the Notehub API in just a few commands from a terminal.

The catch is, when the new folder is generated which has all the API endpoints and models, the documents, and the dist/ folder that packages everything up for consumption as an npm module, it's stored as a subfolder inside of the main Notehub JS repo.

The openapi.yaml file lives at the root level of the project along with its config.json file, a package.json file, and a few other other bits and bobs (license, contribution guidelines, code of conduct, etc.), but the real meat of the library lives inside of the src/ subfolder.

Here's a simplified view of the Notehub JS repo's folder structure:

root/
├── .github/ 
| ├── workflows/
| | ├── publish-npm.yml 
├── src/ <- this is the folder generated by the OpenAPI Generator CLI
| ├── dist/
| ├── docs/
| ├── src/ 
| | ├── api/ 
| | ├── model/
| | ├── index.js 
| openapi.yaml
| config.json
| package.json

From the diagram above you can see the openapi.yaml file that this library is generated from lives at the root of the repo, and the src/ folder is what actually holds all the Notehub API JavaScript code that powers the Notehub JS library.

This src/ folder is what needs to be published to npm - not the root of the repo as is the case with most projects in GitHub. But not to worry, publishing just this subfolder can be done, and better yet, it can be automated.

Configure the root `package.json` to point to the subfolder where the Notehub JS library code lives

So now that I've covered why I only need to publish the subfolder of the Notehub JS repo to npm, let's get on to the how.

First thing: add a few extra configuration details to the package.json at the root of the project.

Update the main field to point to the subfolder's index.js file

According to the npm docs, the main field points to the primary entry point of an app, and when it's not specifically defined it defaults to index.js at the root of the project.

Since the src/dist/ folder is where the index.js file lives for Notehub JS library code that should be uploaded to npm, main needs to be updated to point to it instead.

Now the package.json's main field should read:

"main": "./src/dist/index.js"

Update repository to include a directory field

Next, the repository field in a package.json specifies where the code lives, and it's generally helpful for people who want to contribute to the project.

It has another benefit though: if the package.json for the npm package is not in the root directory (like for us, where the auto-generated src/ folder has its own package.json), you can specify the directory in which that file lives.

Here's what the repository field should look like:

"repository": {
  "type": "git",
  "url": "https://github.com/blues/notehub-js.git",
  "directory": "src"
}

What this translates to on the npm package page is a handy link that points back to root of the GitHub folder (see image below).

Add the publishConfig field to specify directory

Typically, the publishConfig field is used to set config values that will be used at publish time like tags, registries, or access.

Another field that can be added here is directory, which customizes the published subdirectory relative to the current package.json. This ensures npm knows the package.json it needs to read from (and display) in the npm package page is inside of the auto-generated src/ folder, not at the root of the project.

Add the following publishConfig code to the package.json like so:

"publishConfig": {
  "registry": "https://registry.npmjs.org",
  "directory": "./src"
}

In the end, the whole package.json at the root of the Notehub JS repo looks like the code snippet below.

Note: I've omitted parts of the package.json for clarity, but if you'd like to see the whole file, you can click the file name below. It's linked to the actual file in the repo in GitHub.

package.json

{
  "name": "notehub-js",
  "description": "JavaScript library for accessing the Blues Wireless Notehub API endpoints",
  "main": "./src/dist/index.js",
  "repository": {
    "type": "git",
    "url": "https://github.com/blues/notehub-js.git",
    "directory": "src"
  },
  "publishConfig": {
    "registry": "https://registry.npmjs.org",
    "directory": "./src"
  }
}

Great, the changes to the package.json to publish just the src/ folder to npm are complete, let's move on to the GitHub Actions workflow to make publishing a new version of the Notehub JS library to npm as automated as possible.

Create a GitHub Actions workflow to publish to npm

Need a refresher on GitHub Actions?

If you want a quick primer on what GitHub Actions are, I recommend you check out a previous article I wrote about them here.

I decided that for my use case, creating a new release made the most sense to trigger a GitHub Actions workflow to publish the Notehub JS src/ subfolder to npm.

GitHub specifically defines a release as a deployable software iteration that you can package and make available for a wider audience to download and use, which is exactly what I want.

Once I had settled on this as the trigger for my GitHub Actions workflow, it was a pretty straightforward set of steps to publish to npm.

Here's what the finished publish-npm.yml file will look like inside of the ./github/workflows/ folder - I'll break it all down afterwards.

publish-npm.yml

name: Publish notehub-js to npm

on:
  release:
    types: [created]

jobs:
  npm:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ./src
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup .npmrc file to publish to npm
        uses: actions/setup-node@v3
        with:
          node-version: "16.x"
          registry-url: "https://registry.npmjs.org"

      - name: Install dependencies
        run: npm ci

      - name: Publish to npm
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Each workflow file needs a name, so I chose a descriptive one: Publish notehub-js to npm.

Next, as I already explained above, this workflow is triggered whenever a new release is created in GitHub, which is where

on:
  release: 
    types: [created]

comes into play.

on is how a workflow is triggered.
release is the event that triggers the workflow.
types: [created] is the activity type for a release event that triggers the workflow. This gives us more fine-grained control of when the workflow should run.

Then the jobs section runs inside of the workflow. This particular script only has one job, npm, but if there's multiple jobs, they'll run sequentially unless otherwise specified.

The npm job defines that it runs on the latest version of Ubuntu in runs-on.

defaults.run is an important part of the script because it's where I specify that the working directory for all the following steps is the src/ folder (because I don't need any of the files from the root of the repo, just what's inside the src/ folder, I can define this at the job level).

Finally, I get to the steps.

The steps are as follows:

Checkout the code so the workflow can access it using the GitHub Action actions/checkout@v3.
Download a Node.js version to build the package before publishing it to npm using the GitHub Action actions/setup-node@v3 (specify the node-version and registry-url for npm here).
Install the project's dependencies stored in the src/ folder's package.json file using npm ci.
Publish to npm by running the command npm publish and supplying it with an NPM_TOKEN (generated in npm) to verify this workflow has permissions to publish to the specified npm package registry.

And there you have it: each time a new release is created in the Notehub JS repo, this GitHub Actions workflow will run and deploy the updated code to npm.

Conclusion

Working as an engineer for a startup company is cool because no two days are alike for me. One day I'm building web apps and updating documentation on our developer experience site, the next I'm reading up on best practices for npm packages and how to automate tasks with GitHub Actions.

Venturing into the realm of open source software and publishing my first JavaScript package to npm was quite a learning experience for me, but I'm so grateful I had the chance to push myself into this space and create something useful for my team (and hopefully our users as well!).

One thing I knew I wanted, right from the start, was to automate the tasks I knew I'd be repeating fairly regularly over time: namely, regenerating the JavaScript library as the Notehub API continued to evolve, and redeploying a new version of the library to npm with the latest updates.

With some extra configurations in the Notehub JS repo's package.json, I was able to focus in on the auto-generated subfolder that contains the useful JavaScript-based code for easily interacting with the Notehub API, and with a few GitHub Actions inside of a workflow, I was able to build the code for the package and publish it to npm with the click of a button. Not too bad for an end result.

If you’d like to make sure you never miss an article I write, sign up for my newsletter here: https://paigeniedringhaus.substack.com

Thanks for reading. I hope learning how to target a specific folder inside of a project and automating deployments to npm through GitHub Actions workflows come in handy for you in the future. Enjoy!

References & Further Resources

Notehub JS GitHub repo
notehub-js npm library
OpenAPI Generator CLI

DEV Community: Paige Niedringhaus

Reviewing the BenQ RD Series Monitor for Developers

I didn’t think developers needed a special monitor, until I tried the BenQ RD280U.

Initial assembly and setup of the RD280U was a breeze

Noteworthy RD280U features for devs

"Coding Mode"

Additional monitor features

My favorite monitor feature (I didn't see this one coming tbh)

Conclusion

References & Further Resources

Build a Custom Time Slider Component with Ant Design and Next.js

Introduction

Install Ant Design and Day.js

Ant Design Slider

Day.js

Create a TimeSlider component

Calculate and style the tick marks to handle different date ranges appropriately

Format the tooltip

Style the track and marks to handle current and future dates differently

Import the TimeSlider component into the parent page

Test out the Time Slider component

Conclusion

References & Further Resources

Automatically Publish a Repo as a PyPI Library with GitHub Actions

Introduction

Notehub Py

Create a scripts.py and config.json file to automate generating the library and packaging it for distro to PyPI

Set up a PyPI account and library to publish to

Write a GitHub Actions workflow to publish to PyPI

Conclusion

References & Further Resources

Animate an Auto-Scrolling Carousel with Only HTML and CSS

Introduction

CSS keyframes and scroll snap

CSS keyframes

CSS scroll snap

Set up the HTML and CSS

Conclusion

References & Further Resources

Lean on CSS Clip Path to Make Cool Shapes in the DOM without Images

Introduction

CSS clip-path

The polygon I needed to recreate with CSS

Set up the HTML and CSS

Add the CSS clip-path to shape puzzle-section

Conclusion

Further References & Resources

Create a Custom Formatted CSV from Python Data

Introduction

Sample Python data

The csv.DictWriter module

Format the Python data into a CSV and save it locally

Conclusion

References & Further Resources

Filter, Merge, and Update Python Lists Based on Object Attributes

Introduction

Sample Python Data

Filter objects in a list on whether an attribute exists or not

[Merge two lists together by matching object keys

Add new object properties to each object in a list

Conclusion

Further References & Resources

Use Selenium with Python to Target the XPath of a Particular Object

Introduction

Selenium Python package

What is XPath?

The Problem: XPath was targeting the whole document

The Solution: How to restrict XPath inside a particular element

Test the refactored code

Conclusion

Further References & Resources

Scrape Data from a Lazy Loading Website with Selenium Python

Introduction

Selenium with Python package

Install Selenium Python in the Python project

Import Selenium WebDriver into Python script

Add methods to scrape data and lazy load more data

Conclusion

Further References & Resources

Copy Files from One Repo to Another Automatically with GitHub Actions

Create a `scripts.py` and `config.json` file to automate generating the library and packaging it for distro to PyPI

Create a GitHub Actions workflow to run when the `openapi.yaml` file is updated

Write one shell script to copy the `openapi.yaml` file to the Notehub JS repo

Configure the root `package.json` to point to the subfolder where the Notehub JS library code lives