DEV Community: Fraser

Pre-generating filters on static sites

Fraser — Sat, 05 Mar 2022 11:49:27 +0000

Working with data on a static site can be tricky. In a scenario where data doesn't change too regularly, it would be ideal to be able to build the data into the static site. But how does that change if users are allowed to filter the dataset?

I recently started rebuilding LottoStats, a project that uses scheduled functions to scrape various UK lottery results and display them on the LottoStats website. The site uses Next.JS and Supabase and my goal was to pre-generate all pages at build time. I also wanted users to be able to filter the lottery games displayed on the results table.

Keeping everything static was easy when displaying all the data and not permitting any user interaction, however when it came to adding filters, I immediately started reaching for a client side fetch to get the filtered results. This went against my goal of all pages being pre-generated, so I looked for an alternative.

I started by considering how a server rendered site handles filtering. Filters are likely encoded in the page URL, extracted by the server, and used in a database call.

My initial mental model of a page with filters applied was that it was a subset of an "all results" page. However, by re-considering I realised that a page with filters applied could be thought of as a page in its own right. By thinking of each filtered page as a separate page I would be able to generate a static page for every combination of filters and then use routing to fake filters being toggled.

For LottoStats there are four filters each of which can either be on or off, leading to 16 possible combinations.

Next.JS helps with the implementation by giving us getStaticPaths to create pages based on URL paths. For only 16 combinations, the paths can be hardcoded using the filter query string as the param. For sites with many more combinations a catch-all getStaticPaths is likely to work better.

Next.JS passes the query string used for getStaticPaths over to getStaticProps via context automatically. getStaticProps can then use the query string to fetch the required data from the database during build.

The result of this is a set of pages with each page representing a specific combination of filters.

Now we have a set of pre-generated filtered result pages, attention can turn to getting the correct page in front of the user during runtime. Filters are typically presented to the user as a collection of buttons, so we can deal with routing in the onClick handler.

When the user clicks a filter toggle, the name of the toggle they clicked is passed into the onClick handler. We can find the current path using the Next.JS useRouter hook. By comparing the toggle that was clicked with the current path we can determine what the user is trying to achieve.

There are a few possibilities,

there are currently no filters applied and the user wants to add one
there are filters applied and the user wants to add another
there are filters applied and the user wants to remove one
all the filters are applied and the user wants to remove one

Note that for this method, I've made the filters a toggle with it only being possible to add or remove one filter at a time.

We can use a conditional to create a suitable filter string for each of these possibilities. The structure of the filter string should take the same format as the pathnames used to generate pages in getStaticPaths.

This filter string can then be passed into the Next.JS useRouter push method to direct the user to the page that matches the filters that the user is looking for.

This method of pre-generating filter pages excels where site speed is a concern. By removing a database call, filters appear to be applied very quickly. It also has the side-effect of lodging the filter query into the browser history ensuring that filters will remain in effect should the user navigate back to the filter page, somewhere SPA sites often fall short.

I had no hard requirement to keep the whole LottoStats site static, but it was fun working out a way to implement it. Check out the LottoStats source on Github.

Using Cloudinary to improve image loading

Fraser — Mon, 04 Oct 2021 14:39:28 +0000

After switching the SommCellar eCommerce site from Shopify to our own platform, we became responsible for hosting everything. This included a whole stack of product images.

As a quick fix I added all the images to the Git repo for the site and served them as static assets. It wasn't an elegant solution, whenever new products were added I had to manually add the photo to the Git repo and then redeploy, but it worked and at the very least allowed us to keep the site operating while still building.

The issue we ran into was that the images weren't optimised in any way. This had an impact on customer experience as images were taking around 4-5 seconds to download and customers just weren't hanging around to wait for them.

As an eCommerce site, we use the same pictures in multiple views, some as lo-res thumbnails and others as hi-res feature images. If we were to have created the various options manually it would been challenging to keep organised as well as a huge waste of time.

We needed something that would take a single original image and create a set of "production" images that would suit their particular use case.

Incorporating Cloudinary

Cloudinary is a media management tool that does exactly what we needed, accepts a source image and then allows you to apply transformations using URL query parameters.

To get started we uploaded all our original images up to Cloudinary from which we got a set of URLs back. These URLs pointed to the original images with no alterations made. At this stage we were in exactly the same situation as when we had the images in our Git repo - a URL pointing to an unoptimised original image.

But it's at the next stage that things get clever. By editing the URL Cloudinary had given us we can apply transformations to adjust the resolution, shape, and near enough anything else you could imagine doing to an image. The transformations we were particularly interested in were resolution, quality, and format.

Our original images were around 2800x3600 px in size, over 2MB in file size and stored as jpeg. For our thumbnail images we wanted the following spec:

around 300px wide
as small of a file size as possible
delivered in as modern of a format as the requester's browser could handle

These three requirements could be applied by adding the following to our image source URL:

c_scale,f_auto,q_auto,w_300

Making it reusable

The issue we now faced was that we wanted to use the same image in multiple places on the site but our hardcoded source URL was forcing the image to be 300px wide. While this was fine for a thumbnail, for the feature image of a product was too small.

We needed to be able to store the URL for the original image in the database and then apply different transforms depending on the location the image was going to be displayed.

We found the cloudinary-build-url package to help us.

This package takes in the source URL as well as an object of transformations, meaning we were able to swap out the transforms that would be applied depending on the location of the image component.

The impact on the site

We've still got some tweaking to do while chasing the best performance but thanks to Cloudinary we are well on our way.

Images now load much quicker and users don't experience gaps in content while waiting for images to download.

The bandwidth we're using to deliver images has also reduced which translates to reduced Cloudinary credit consumption.

Finally the biggest impact we've found is for the product team. We always wanted images to be optimised and without Cloudinary that would have meant manually adjusting images to suit multiple formats. Being able to upload a single image and leave all the adjustments to a third party is a considerable improvement in workflow and helps take the load off the product team.

Switching from Shopify to Stripe

Fraser — Fri, 24 Sep 2021 13:55:53 +0000

At SommCellar we recently switched from using Shopify to handle e-commerce over to a custom system centred around Stripe. The change was based on Shopify limitations as well as the reduced transaction fees on offer from Stripe.

Why did we start with Shopify?

When we first launched the business our number one concern was getting up and running quickly without an extended development period.

Hearing the positive experiences of others and knowing that we could get a high-quality site launched in a matter of minutes we chose Shopify to power our online shop.

Shopify offers a great all-in-one platform to manage the online sale of physical goods. The speed at which we could get the site up and running was amazing and the initial costs of using the platform were reasonable.

Our launch site may not have been exactly what we envisioned but it allowed us to begin making sales and generate revenue.

What issues did we experience?

While at the start Shopify was great, when we looked to expand the site by adding new features we quickly began hitting barriers.

It turned out that adding near anything beyond the requirements of a basic site requires plugins.

Perhaps calling the requirement for plugins an "issue" is wrong — Shopify themselves market plugins as a feature — but by the time we had priced a few plugins for the site our monthly hosting cost was going to triple.

What we were looking for wasn't lavish, just some tools to help with delivery pricing, bookkeeping feeds, and the like but the total cost quickly crept up.

We were also concerned that the all-in-one platform from a single provider that we had originally signed up for, had fast turned into a melody of service providers that we had to rely on to run our business and provide our customers with great service. Each service provider would become a link in a growing chain of dependencies, and we would be at their mercy if they wanted to make changes to their link because Shopify hadn't built the functionality into their own system themselves.

The final kick in the teeth was a notice from Shopify that app developers would receive their first million dollars of revenue fee-free, while those of us using them to run our shops had to pay fees from the first sale. As a shop owner, it felt like they were rewarding the developers who were helping them build their platform, while ignoring their customers.

What are we using now?

We didn't take the decision to move away from Shopify lightly and spent a long time researching other options. There are a number of e-commerce platforms out there, but we didn't find any that weren't either missing features or exorbitantly expensive.

In a rare moment of clarity, we realised that (ignoring card processing) for small businesses, e-commerce reduces to a glorified CRUD app. At its core, we are only concerned with products, customers, and orders. If we could handle emails, card payments, and wrap everything in a fancy admin UI there was nothing stopping us from running our own platform and customising it exactly to our needs.

Saying we "built our own e-commerce platform" is very grandiose. In reality we wrote down our customer journey and built functionality into our system as the next step on the journey arrived.

By focusing on our own needs only we were able to eliminate irrelevant features and postpone others to a later date. For example, at the moment our products are managed by running SQL queries directly against the database instead of via the admin UI. Adding the ability to manage products via the admin UI will certainly be added in the near future but for now it has saved us the time of building out a management interface, allowing us to concentrate on other areas.

An area that received increased attention was payments. As the centre of the business, it was important we got it right. All our payments are handled by Stripe. Users add items to their shopping bag and enter their delivery information on the SommCellar website. When they hit "Continue to payment" they are redirected to Stripe Checkout, which takes care of charging the customer's card. Users are then redirected back to the success page on our website.

We decided to handle products solely on our own system instead of making use of Stripe's products API. When creating the payment intent we pass Stripe a list of products, including the title, price, and quantity, and Stripe simply calculates the total and charges the user. This saves us from having to maintain two collections of products across our own system and Stripe's.

Using webhooks, Stripe lets us know when payment have been charged successfully and we update the order to reflect the new payment status, after which we begin fulfilment.

For the time being our system is incredibly disjointed but for our small team, that all works in the same space, any issues can be sorted by simply turning to the person sitting next to them.

Our final tech stack is Postgres, Express, Stripe and Next.JS.

What issues do we now face?

While we've managed to move over to a system that does exactly what we want and will help us to develop into the future, we aren't left with nothing to do.

When using someone else's system, they are responsible for performance, maintenance, and uptime, activities that now fall to us. We always knew this would be a trade-off and decided that the benefits of creating our own system outweighed the task of running it.

In the effort to get our own system launched, running activities have taken a back seat and will be added to the tech debt pile. While the business is small, these issues are relatively low impact. We will incorporate performance tasks into the general development of the system and tackle issues as they become apparent.

For us, our main concern is to be aware of any problems as quickly as possible, so we can communicate any issues with our customers. Heavy use of monitoring and automation will be useful to make sure we can stay on top of issues as they come up.

Would I recommend Shopify?

After writing a whole post outlining the reasons we decided to move away from Shopify it may be expected that I would recommend everyone did the same. But that's not the case.

I'm happy we gave Shopify a go, and despite finding their platform didn't suit us, I think they have a good offering perfectly suited to many uses.

If you are running an online shop and you're not looking to deviate from a fairly standard setup then Shopify will work well and will take care of a lot of the challenges of running your shop.

I do think the pricing of plugins can be excessive and quickly lead to a high monthly hosting cost. I also feel for non-technical shop owners who have no choice but to accept the costs as the cost of rolling their own is prohibitive.

Overall, I am happy we went down the route of an in-house e-commerce system. The control and flexibility in developing the system into the future will allow us to maintain a sharp focus on delivering the best possible service to our customers.