DEV Community: Maciej Krawczyk

I Went to an Archaeological Conference. What Can Software Engineers Learn From Humanities?

Maciej Krawczyk — Thu, 27 Apr 2023 16:16:19 +0000

Last year, one of my colleagues asked if we wanted to present our application at a conference. Hesitantly, I agreed.

CAA conference official banner, source: https://2023.caaconference.org/

I have to start with a short disclaimer. Even though I work as a software engineer, and that’s the career path I settled for, I am an archaeologist.

I have a master’s degree in Archaeology and started my Ph.D. in that field but dropped out after a year or so. After some additional career changes, I found my path in software engineering and programming, but I still have a passion for archaeology left in me. Also, my wife is an archaeologist (a real one).

Based on that, I often get involved in side projects and co-operations with archaeologists. Mostly, I work on data-collecting applications for field work and further analysis for projects in archaeology and broader historical sciences. Back then, when I started with those kinds of applications, there were hardly any good open source solutions for such tasks — or so I thought. Also, it was at the beginning of my software engineering path. So, I settled for custom solutions each time. Sometimes, it was in requirements; sometimes, it was due to a lack of better existing solutions. But in most cases, it was my decision. And it was probably the wrong one.

About the Conference

Last year, one of my colleagues asked if we wanted to present our application at a conference. Hesitantly, I agreed. I was not sure if we would have something worth presenting and also if I would find other papers interesting. But after the conference, I can say it was worth every minute spent there. And that’s how I found myself in Amsterdam, at CAA 2023, at the beginning of April.

But what does CAA stand for? According to their website:

Computer Applications and Quantitative Methods in Archaeology (CAA) is an international organisation bringing together archaeologists, mathematicians, and computer scientists. Its mission is to encourage and facilitate dialogue between these disciplines, to provide an overview of the present state of the discipline, and to stimulate discussion to progress the field. […]

CAA organises an annual international scientific conference covering a wide range of topics, including data acquisition and recording, conceptual modelling, semantic technologies, data analysis, data management, digital 3D object reconstruction, image visualisation in archaeology, geophysics, and GIS. The conference format includes thematic paper and poster sessions as well as round tables and workshops.

And this was the 50th annual conference. Yup, the first one took place in 1972 (there was no conference in 2020 due to obvious reasons). Two years after Edgar F. Codd published his paper on relational data model, and two years before D.D. Chamberlin and R.F. Boyce published their paper on SQL (called SEQUEL back then). I was pretty surprised to find out that in that analog world, there were people with such vision to start going in a digital direction despite all of that seeming temporary. They were true early adopters.

Photo by author

The conference took place in the lovely city of Amsterdam. It was fully packed with presentations about deep learning, 3D data modeling, web and mobile data collection solutions, GIS and mobile GIS, and gamification. The first day contained various workshops, and the other three contained presentations.

Why

Most sessions I attended were focused on collecting data (also in the field), creating tools for further analysis, and presenting the research results. To be honest, I wasn’t prepared for what I saw. The amount of clever and well-thought-out tools and solutions I saw a bit were overwhelming and exceeded my wildest expectations. Also, to be perfectly honest, I must mention that many of them caused a little sting of jealousy. The good kind that boosts your creativity, the one saying, “Whoa, that’s awesome. Why I haven’t thought about that?”

But also, many more topics were discussed at the conference, like deep learning, 3D modeling, or gamification. But they were not my primary interests, so I focused on the abovementioned ones.

What I learned

For me, there were three main outcomes of attending such a conference. First, and most importantly, I learned (or rather was reminded) that I don’t always have to go the custom path. Sometimes, it’s better to use existing solutions and even sacrifice a few planned features to use a more standardized approach. There was a terrific quote I saw in one of the presentations:

“Standardized is better than perfect.”

Sadly, I don’t remember whose presentation it was and where it was taken. And I’m sorry I can’t give proper credit to the person who said that. But it reminds us that creating software is a game of compromises between what we want, what clients want, what is even possible, and time and money constraints.

The second great outcome for me was a huge boost in creativity and motivation from all the examples of adapting tools and empowering work empowered by tools instead of being defined by them. For example, there was almost a whole session discussing the usage of LiDAR (Light Detection and Ranging) sensors in the latest versions of the iPhone to document the archaeological features in a three-dimensional way. Back in my day (that sounds awful), when I was still working in archaeology nine or ten years ago, tools used for that were extremely expensive and required trained operators. Now everyone can do this. That’s mind-blowing.

The last outcome for me was a great opportunity to confront my software development approach with effective, radically different, and no less creative approaches. The possibility to see less conservative, free-from-bias points of view helps us better understand our clients and their needs. Even though they rarely have a scientific background. Also, looking for a little science in our work helps us understand what we are doing and aiming for.

What you can learn

Based on what I saw at this conference and what I see in my day-to-day job, many development teams and software developers could benefit from a little change in approach. Being a bit scientific in our work can give us wider perspectives to solve problems. Also, it allows us to be more resourceful. Don’t get me wrong, I don’t want to criticize any development methodologies here. I don’t have anything against Agile and Scrum. Quite the opposite; I love them.

On the other hand, this kind of lean approach, working with what one has and using more complex solutions only when strictly necessary, can benefit everyone. Also, most importantly, it limits bloat in a project, which is great for development teams. To be honest, academias are one of the most bureaucratically-bloated organisations I’ve ever seen (so far). But when we give that to teams working on single projects, they can oppose that trend and work efficiently toward their goals.

Also, I believe this kind of conference is the best place to learn new, creative ideas, as it gives you access to the most creative usage of tools. Sometimes, we are so used to our expertise that we are blind to other solutions, and the only thing needed to realize that is a new perspective. That’s the easiest way to make us leave our cozy programming comfort zone — which is where the growth is.

Why IT and science communities should meet more often

Because it’s the best and easiest way to get a wider perspective to see your software differently. Also, this meeting can easily fuel your curiosity and creativity. Limiting ourselves to tech conferences can cripple our view of the tech landscape.

Why humanities and not, for example, math conferences? If you are not strictly math-oriented in your work, the entry threshold may be too high. Don’t get me wrong, I’m not saying that humanities are easier. You may not be able to get all the intricacies of problems discussed, but when you see the pyramid, you know what it is. Also, I believe that between us, there are a lot of people who don’t have a background in CS. And going back to your roots and previous passions can be most refreshing and beneficial.

If I can convince anyone to try that and be more scientific in their work, read something about that, or maybe attend conferences that bridge one’s interests and software, I would be the happiest person in the world.

Summary

I was amazed to learn that scientific communities are the best place to look if you are looking for true early adopters for your solutions. They may not have the largest budgets, but they have a huge curiosity.

Am I going to next year’s CAA conference? I’d love to, but due to logistic problems, it may not be possible. It will take place in Auckland in New Zealand, directly from my place to the other side of the globe.

Creating Your Own E-Commerce Keystone.js-Based System — Build a Cart

Maciej Krawczyk — Wed, 27 Apr 2022 13:09:46 +0000

Creating Your Own E-Commerce Keystone.js-Based System — Build a Cart

Next steps in our journey to create our own e-commerce system

Photo by Kenny Eliason on Unsplash

Intro

Some time ago I had a wild idea to use Kesytone.js to build an e-commerce system. This journey started a couple of weeks ago, and until now we’ve talked about system requirements, environment setup and base models, and access control. In this article, let’s focus on main cart functionalities. Also, the finished code for this article is available on my GitHub.

Cart Flow

In previous parts of this series, when we set up the basic schemas, we decided that each user would have only one cart, and it would contain all added products until the user created an order from that cart. Based on that, users can perform three kinds of operations on their cart. First, there’s the possibility to add an item to the cart, then remove it and change its quantity.

Also, there’s one major issue to consider, and it’s more of a business than a technical problem. Should we allow users to add more products to the cart than is currently available? I mean, in the case where the product has stock, for example, four items are available, but the user tries to add to cart five, what should happen in that case?

Of course, that issue should be secured in UI, but in some cases, it can happen anyway. Using SSR and Next.js has some drawbacks, and in the most basic case, the quantity of available products is only checked on page rendering. This may lead to cases when product availability may change in the time between render and the moment of adding a product to the cart.

There are two main solutions: first, block adding to the cart in that case or move this validation step forward and block creating orders with products out of stock. Despite our decision, this step is necessary from a security point of view.

I believe there’s a third solution to this problem — something in between the previously mentioned two. Stock schema includes information about the next delivery, so if there aren’t enough items in stock, but together it’s enough then the user can add it to the cart. But the order will be delayed because of that.

On the other hand, if there’s no next delivery information, it will be blocked. This solution should ensure better user retention and what’s more important is more interesting to implement.

With that out of the way, we can focus on each of these three operations. First, add a product to the cart. There are basically two steps in that. Validate stock and update products in the cart. The same applies when updating the quantity. Removing products is just an update to the cart model, right? Not exactly. Let’s take a look on our Cart schema:

There is related to Product list, but there’s no way to store information about the quantity of added products. So, we have to create an intermediate list (called a pivot table in SQL nomenclature) to handle the many-to-many relationship and store the quantity information.

Cart Items List

The main purpose of this list is to store relationships between Cart and Product entities and quantity information. Basically, it should be only requested as a relation from Cart. There’s no point in requesting it directly. Let’s create cart-product.schema.ts:

But wait, there’s only two fields? It’s simple, this list doesn’t need to know anything about Cart or what these products belong to. But on the other hand, Cart model needs this information, so we have to update this list and change the relation from Product to CartProduct. Additionally, there's no longer a need to hide the possibility of creating this entity from Admin UI.

products: relationship({
  ref: 'CartProduct',
  many: true,
}),

OK, now we can update our flows:

Adding to cart:
1. Validate stock
2. Create CartProduct entities
3. Update Cart model
Remove product from cart:
1. Remove CartProduct entity
2. Update Cart
Change quantity in cart:
1. Validate stock
2. Update CartProduct entity
3. Update Cart

A Word or Two on Framework Abstractions and Their Limitations

But why all that trouble? Let’s take a look at the current ER diagram of our database:

We have our Cart, CartProduct, and Product tables, but also there’s _Cart_products table. We didn’t create the last one, right? Undelaying Prisma did that for us. That’s why it’s good to have a basic understanding of the tools we use.

Prisma has two ways of creating many-to-many relations (more information’s available in the docs ), explicit or implicit. In the first one, we are responsible for creating pivot tables and relationships on other tables in our schema.prisma file. In the second one, we skip the pivot table and ORM creates it for us.

But in our case, we don’t have direct control over the schema.prisma file; Keystone takes care of that and uses the implicit method. In most cases, it’s perfectly fine, but sometimes it may have some drawbacks, like this unnecessary table here.

Frameworks usually hide many implementation details under the thick abstraction layer, which is a good thing in most cases. It allows developers to focus on business logic and work faster and more efficiently. But in some cases, we have to accept some issues.

Hooks and Validation Flow

To perform all the steps involved in each cart operation we need a tool that allows us to perform some side effects, including additional validation while updating Cart schema. Fortunately, Keystone has the perfect tool for that.

There’s Hooks API, which does exactly that on the whole schema or particular fields in it. There are five of them:

resolveInput allows us to modify input data before validation on create or update operation.
validateInput and validateDelete gives us the possibility to return additional validation errors in the create/update and delete operations, respectively.
beforeOperation handles side effects before database operation
afterOperation does the same but after operation.

Summary

Now, we’ve finished the cart part of our e-commerce system. To be honest, this part of the application was harder to develop than I’d anticipated at first. But also, the implementation was not so difficult. Most of the work was thinking about the best way to solve that problem, not the problem itself.

For various reasons, it took me longer than I’d planned, and I hope you liked it. If you have any questions or comments, feel free to ask them.

A side project has one nasty characteristic: At first, they are exciting and interesting, but after some work, we don’t feel that way any longer. And I believe that’s why writing this part took me so long.

Don’t get me wrong, I’m still planning to finish this series and build this system, but in order to not lose the fun in it — and to prevent it from becoming a chore in the next article — I’ll take a break and write about something else.

See you there!

Building Your Own E-Commerce Keystone.js-Based System — Access Control

Maciej Krawczyk — Fri, 01 Apr 2022 00:59:15 +0000

Building Your Own E-Commerce Keystone.js-Based System — Access Control

Improving upon our e-commerce based system

Photo by Nathália Rosa on Unsplash

Introduction

It’s time to continue our series on creating a Keystone.js-based e-commerce system. Previously, we focused on creating all basic models and environment setups. This time we will work on access control and users privileges. Let’s be honest, not all users should be allowed to access all data and have the ability to modify it. It is a serious security risk, so this time we are going to fix it. Also, the finished code for this article is available on my GitHub.

Requirements

So, what are our requirements here? First, we need to restrict users' access based on their role in the system. Basically, customers can only modify data associated with them, but admins have much more privileges. All of this should affect the system on three levels: first, restrict access to Admin UI, then filter access based on operation type to each schema, and lastly on the level of some fields in schemas.

Access Control Implementation

First, let’s update user roles in our system. In the previous article, we started with two of them — admin and customer. The basic assumption was that we need two different levels of security with user privileges, but after some consideration, I decided we need another one. So it’s time to update our roles.enum.ts and roles.const.ts and add the employee role. For now, privileges for admin and employee will stay the same, but with further system development it may change, as you can see below:

With that out of the way, we can focus on Admin UI access. For sure, all admins and employees should be able to use it; it’s necessary. But customers should not be able to access it. Of course, they need to log into the system and create sessions, but the Admin UI contains sensitive information that mustn’t be widely accessible.

Out of the box, Keystone offers the perfect solution for this kind of problem. To restrict certain users from accessing Admin UI, we just have to update our main config file, keystone.ts and ui.isAccessAllowed option. Its function with the context parameter will return a boolean value. Previously, we were only checking if users had valid session data. Now we have to check if the user has a valid session and if their role is not equal to customer.

And that’s all; only users with the right role will be able to access Admin UI. Now, it’s time to focus on access control at the level of each schema. Basically, there are two types of restrictions we are going to use: first, everyone can query but only internal users (admin and employee) can modify it. The second case covers a scenario when internal users have full access, but customer user can only query and update dates that belong to that user.

Let’s walk through all the schemas and update the access property on each of them. First, the Address list: it contains sensitive data so only admins and owners of specific records should be able to access it.

In Keystone.js’s access API, there are three levels of control (read more), operation, filter, and item. First one, operation allows restricting what kind of operation can be done on the list (query, create, update, and delete). Second, filter allows adding specific GraphQL filters to every operation performed on this list with distinction to its type (same as before, but without create). Last one, item works more on the data level and allows us to inspect data passed to each operation, but it works only on mutations, so has no effect on queries.

All properties in this section of the list configuration are functions. The first and last should return boolean; the second one should return GraphQL filter.

In case of Address list on the operation level, we only need to check if the user is authenticated, but on the filter level, we have to check if the user is admin or employee. Then there’s no necessity for an additional filter.

But when the user’s role is customer, we have to create and add filters limiting results only to this user. These functions will be rather similar for each schema, so I’ve decided to move them to another file to prevent unnecessary repetition. I’ve created a shared folder and index.ts inside. Also, I’ve created a file for our new method called filter-customer-access.ts and added it into exports in index.ts.

Basically, it creates and returns new GraphQL filter checking if the user ID in the record we are trying to access is equal to the ID stored in the current session. Additionally, I’ve added an interface to describe the session parameter, mostly because, in core Keystone files, a session has a type any that is not so convenient.

We are going to use this method to add this filter to query, update, and delete operations, but not create. First, it’s not possible, and it needs a different approach. We are going to uralitize item level restrictions in this case. The function used here will also be reused, so I’ve created another file in the shared folder called filter-customer-access-create.ts.

It looks pretty similar to the previous one, but there are two main differences: first, it returns a boolean value, not filter, and secondly, it checks if the data passed to create mutation has information about the user and if it’s the same user to whom current session belongs. Also, I’ve added export for this method in index.ts. Now it’s time to add it to the Address schema config file and its access section, as you can see below:

Order, Payment, Shipment has the exact same restrictions, so I’ve updated their schemas too. Additionally, in the Order schema, I’ve updated user and employee fields. Previously, they had an option to create a new entity from inside the list which is not really desired here. So I’ve added to this fields’ config:

ui: {
 hideCreate: true,
},
``

Cart schema restrictions are pretty similar with some exceptions. First, there’s no check on the item level. Instead of that, I’ve added another config option — graphql. It contains options directly applied to GraphQL schema and allows disabling certain operations, create in our case.

Because of that, there’s no way to create new Cart entity via GraphQL API, and it’s something that we want. Cart for user can be created only once, on sign up process. More about that later. I’ve also updated some fields to block the possibility to create a new user and new products from inside this list. The last change was to add the default value to sum field. Here’s the whole updated schema:

The next group of lists has much simpler restrictions: all users can query them (not authenticated too), but only internal users can create/modify them. This group contains Catgegory, ProductImage, Product, and Stock. In order to apply these restrictions, we have to add access rules on the operation level:

In the Stock schema, I’ve also added an additional access rule on field level, amountInNextDelivery to be exact. Information about stock delivery may not be strictly confidential or sensitive, but competition is watching, so better keep it for authenticated users.

amountInNextDelivery: integer({
  access: { read: ({ session }) => !!session },
}),

The last updated schema is User; most changes are pretty similar to the Address list, but there are some exceptions. First, on the item level, we are not restricting the create operation. If we do, no one will be able to register.

Instead, I’ve added the same restriction to the update operation. Additionally, I’ve utilized hooks here, precisely the afterOperation one. We will talk more about hooks in further parts of this series, but for now, all we have to know is that they allow you to perform side effects in reference to operations performed on the current list.

Our hook first checks the type of current operation; if it was not the create operation, it returns undefined. But in the case of create, we want to proceed and create a corresponding Cart entity for that new user. But there’s a catch, previously when updating Cart schema I’ve disabled GraphQL API responsible for new cart creation, so we have to use prisma API (more about that). It allows us to skip this restriction without the risk of allowing users to create carts independently. Here’s the updated User schema:

There’s one thing left to do for this part. We need some test data — users in this case. Prisma.js included in Keystone has a nice way to import initial data (more), so let’s use this feature. Start with installing the ts-node package as devDependency:

yarn add -D ts-node

In the meantime, create the folder, seed-data, and two files inside: data.json and index.ts. The first one will be our source of test data; the second will import that data. But additionally, we have to make small changes to our tsconfig.josn in order to be able to directly import data from the JSON file. So, under compiler options, add:

“resolveJsonModule”: true

Now, let’s add our test data:

The password used here is 1234ABcd@@ for each user. Next, we have to prepare import logic, so let's update index.ts:

There are two main parts to this importer: first, we are creating a main method responsible for importing data into the database, and the second part that calls this function in a safe way with proper error catching and the ability to disconnect the database after the job is done.

The basic importer logic is pretty simple; it loads user data from imported JSON then loops over it. For each iteration of this loop, it checks if that user exists in the database; if not it creates it. With that done, the last thing we need here is a way to start this import. So, let’s update our package.json and add:

"prisma": {
  "seed": "ts-node seed-data"
},

Additionally, we need to create a script to start all that using yarn, so under the scripts add:

"seed": "keystone prisma db seed"

With that done, we are almost at the finish line of this part. We just need to create migrations for the updates we’ve done and import test data. So, let’s first start our database with the following command:

docker-compose -f docker-compose.dev.yml up database

Then, start our backend app outside the container. So in the backend folder run:

yarn dev

The script will detect changes in the database schema and create a migration for us. We only have to specify a name for it. For example, new_user_role_and_cart_default_values. After that we can stop this script and seed our test data:

yarn seed

With that done, we can restart the whole system and test it out with this command:

docker-compose -f docker-compose.dev.yml up

Summary

That’s all for this part. With that change, data security in our system improved much, and now we can safely proceed to the next tasks and work on the core elements of every e-commerce system — cart business logic.

I hope you liked it. If you have any questions or comments, feel free to ask them.

Have a nice day!

Build Your Own E-Commerce Keystone.js-Based System — Environment Setup and Base Models

Maciej Krawczyk — Fri, 18 Mar 2022 16:24:55 +0000

Build Your Own E-Commerce Keystone.js-Based System — Environment Setup and Base Models

This week, we are going to get our hands dirty and start coding

Photo by Ludovic Charlet on Unsplash

Introduction

Last week's article was the first of the series about building an e-commerce system using Keystone.js, and it was mostly focused on system requirements and its basic architecture. This week we are going to get our hands dirty a little and start coding. First, we will talk about the development environment and its setup. Then about Keystone and Next setup, and lastly, about basic models setup. Also, the finished code for this article is available on my GitHub.

Dev Environment Setup

Let’s start and create a project folder for our system, keystone-e-commerce, in my case, and create some necessary files. Run the following command in the console:

mkdir keystone-e-commerce && cd keystone-e-commerce
touch README.md docker-compose.dev.yml .gitignore .env.example .env

According to the previous article, the data layer of this system contains two elements: database and search engine. The easiest way to use them locally in our development environment is to use Docker for that. So, it’s time to update our docker-compose.dev.yaml file. Just add the following:

Also, I’ve added here pgAdmin. This is a nice to have and very useful tool, especially in the dev environment. Next, the necessary thing is to set up environment variables for the system. The easiest way to do it is to create one .env file and use it across all containers. OK, let’s add all the necessary variables as shown below:

I’ve already added variables required by Cloudinary integration; we will get

back to them later. Next update the .gitgnore file. For now, this is enough. The following code will help:

**/node_modules
.env

Now, with this basic setup, we can start our containers with this command:

docker-compose -f docker-compose.dev.yml up

Kesytone.js and Next.js Setup

With this setup done, we can focus on other parts of the system. First, let’s create our Next.js client app. To be honest, here we are only going to create it and add it to our Docker setup. More work with it will be done in upcoming articles in this series. Let’s run the appropriate command:

yarn create next-app --typescript

The script will ask us for the name of the app. I’ve called mine client. After installation, we have to create Dockerfile.dev for this app to use with other containers. It’s a rather simple one, as you can see:

FROM node:16

WORKDIR /usr/app

CMD yarn dev -p 3000

Additionally, update the docker-compose.dev.yml file under services section with this code:

A solution like this allows one main .env file, which is nice and gives centralized control of all system secrets. Also, it encapsulates the whole system in one closed docker network. And for now, that’s all about this part of the app. So, let’s switch to the backend part and set up Kesytone.js.

First, run the script to create the app. I’ve chosen backend as the folder name for it.

yarn create keystone-app

After that, it’s basically ready to use, but first, we have to update the database and other configurations. In keystone.ts, add credentials and update import for lists:

Then, create additional folders in order to have a nice and easy to understand structure, with this code:

mkdir consts enums schema

Also, importing in the main config file requires us to add index.ts in schema folder to work properly. The last setup element left to do is to create Dockerfile.dev file and update docker-compose.dev.yml. It will be pretty similar to the previous one, as shown below:

FROM node:16

WORKDIR /usr/app

CMD yarn dev

This will also allow us to start the entire system with one command.

Models Setup

With setup work done, we can start and create all necessary Keystone lists (and data models in our case). Let’s start with User model. Basically, it’s going to hold all user data including roles and privileges inside the system. Create user.schema.ts in schema folder. For now, we are only concerned by the fields property of the list. We have to add all necessary fields there, as you can see below:

Fields name, email, and password are rather obvious. Its purpose is to identify user, and it is necessary for the authorization and authentication process. All three of them are required. Additionally, email has to be unique, and in Admin UI it can be used to filter all the users. More interesting is the role field. Its type is select and holds information about user privileges in the system. It will create enum column in the database to keep it nice and clean. I’ve moved the options values to a separate file in the consts folder.

Here’s the code:

Also, I’ve moved all the values into the file in enums folder:

For now, these two roles are sufficient. Later, we are going to need more of them. The last field holds a reference to Address list. Precisely, it is a one-to-many relationship. The next list contains all addresses associated with users; each one can have more than one. So, create address.schema.ts as shown below:

This list contains all of the necessary address information that will be needed in the shipment of user orders. Most fields are required in order to provide sufficient delivery information. Also, this list holds timestamps of the creation of the record and its last modification. The last field is a reference to user, owner of this address, and in this case, it is a many-to-one relationship.

Next, tightly associated with the user list is Cart model. It contains all the information about the products added to the cart by the user, their sum, and the date of the last modification. Each user has one Cart, so it is a one-to-one relationship. Here’s the code:

In each Cart there can be many products added, and the same product can be added to multiple carts, so it creates a many-to-many relationship.

With that out of the way, we can focus on other models. Next, related to User and loosely coupled to Cart is Order list. It contains all the information about orders currently in processing and already processed. The first field is a reference to user, owner of this order, a one-to-may relationship. Each user can have multiple orders, but each order has only one owner.

The next field contains information about products in this order serialized into JSON. This way we can hold information about all the products in order, not only ones currently in stock, but also removed from offer too.

Next, two fields hold a relationship to Payment and Shipment lists, both one-to-one relationships. The last three fields contain information about the creation date, last modification, and order status. The last one is in select type, I moved all options and values to separate files like with user roles before.

The next two lists complement Order model and hold information about Payment and Shipment associated with it. Both of them hold basic information about timestamps and statuses of both business processes (created the same way as order status) and their relationship to Order.

The first of them additionally holds information about the sum of orders, its currency, and transaction ID from a third-party provider. I haven’t thought yet about specific integration in this matter, but probably it will be Stripe because I’m most familiar with it.

On the other hand, the Shipment model holds information about the employees responsible for processing this shipment, packing, and sending it. Similarly, like in the previous one, there’s also information about an external ID from a third-party system responsible for processing the delivery.

For now, all status fields contain only one option — Created. In the next parts of this series more focused on these sections of the system, we are going to add other necessary options.

The last group of models is focused on products. The main Product list contains basic product information like name, description, SEO description, price, rating (stars), and timestamps. The rest of the fields establish various relationships to more specific product information like product images, categories, and stocks.

The first one creates a one-to-many relationship; the second one a many-to-many, and the last one is a one-to-one relationship. Basically, a product can have multiple images, but the image belongs only to one product. Each may have categories and category has many products, and lastly, each has only one stock information (as I mentioned in the previous article I’ve decided to support only one warehouse setup).

The next list holds products images. There are two important fields here: alt and image. The first one contains the information necessary to fill in the HTML alt attribute for each image. The second allows uploading images directly into Cloudinary CDN. All that is supplemented with timestamp fields.

The next list, Stock, contains information about the amount of available-to-order products. Additionally, there’s information about the next expected delivery and the number of products in it. It’s necessary for cases when a user tries to order more products than are available.

The last model, Category, holds information about products categories and their relationships to one another. Fields here include category name, related products, and parent category (if there’s one) and usual timestamps. This internal relationship allows creating categories tree easily which will be useful when creating our system’s frontend.

The last thing to do here is to import all models into index.ts, which is imported into keystone.ts main config file shown below:

A quick note about timestamps, each one of them has a set of default values and should not be edited by users. But more on that topic in the next article.

ER diagram

OK, that concludes all models we have to create. Now it’s time to create necessary migrations and corresponding tables in the database. Start with spinning up the database container.

docker-compose -f docker-compose.dev.yml up database

After that, start in the separate terminal of our backend:

cd backend && yarn dev

The script will ask for the name of the new migration, type initial_models and hit enter. It will handle the table's creation and generate migration files. Now we can stop both processes and start the whole system.

docker-compose -f docker-compose.dev.yml up database

Summary

Now we have a working dev environment and basic models setup, so it gives us a solid foundation for our system. Next time, we will start building up additional features needed in the e-commerce system. First, we will focus on access control and user privileges to access different parts of the system.

I hope you liked it. If you have any questions or comments, feel free to ask.

Have a nice day!

Build Your Own E-Commerce Keystone.js-Based System — Requirements and Architecture

Maciej Krawczyk — Tue, 08 Mar 2022 17:46:12 +0000

Build Your Own E-Commerce Keystone.js-Based System — Requirements and Architecture

A basic overview of our e-commerce system

Photo by Markus Winkler on Unsplash

Introduction

Not so long ago I was working on a system based on Keystone.js CMS. But there it was used much more sophisticated way than just as basic headless CMS. I was easily able to extend it with search engine (Rust-based Meilisearch) and connect to external APIs.

After that, I had a couple of ideas for projects that could be built with Keystone.js and one of them was an e-commerce system. The idea was so appealing that I’ve decided to give it a try and actually build it. This article is the first of the series about this topic.

After brief research, I found one similar project, but it’s using the previous version of Keystone.js (details). The project is great, but there are some things that I believe I will do differently.

First, it uses MongoDB, and I am not pretty sure this is the best solution for e-commerce (and also I am an SQL man). Additionally, from my point of view, some features are missing, like delivery handling and so on.

Another similar project comes from another Node.js headless CMS — Strapi. But it’s a rather simplified, tutorial type (details) and what’s more, uses Nuxtjs. It’s not my cup of tea; I am a React type of guy.

In my professional career, I worked on one rather large e-commerce system for a couple of years, so I believe I have some ideas how it should look like (and how it should not). Let’s start with our goals here.

The Goal

OK then, but what should this e-commerce system look like? First, scalability is not my main concern here, so I won’t pay much attention to it. For sure, I will leave some doors open to allow some basic scaling, but that’s all.

With that out of the way, let’s assume our system will be constructed with the three-layered approach. The first one is the user-facing, presentational layer, the second is the business/logic layer, and the last one is the data layer.

Presentational layer

Contains the frontend part of application, which should allow users to browse a catalog of products. They can also filter them by category, search them, add them to their cart, and finalize their order with payment and delivery handling.

Additionally, it should allow the user to view the status and history of orders.

Logic layer

The next layer contains all the backend processes of the system: handling products, their stocks, orders processing, payments, and delivery integrations.

Also, users authorization and authentication with access control are crucial here. Basically, its goal is to transform and process data from presentational or data layer and pass further in both directions.

Data layer

The last layer is responsible for data storage and its persistence. Basically, it’s the database of our system. Its goal is to maintain data availability and consistency and ensure that we follow ACID principles.

So basically, I’m going to create an e-commerce system that contains all that layers. To sum up, this system should contain:

frontend layer allowing users to browse products and process orders
logic layer responsible for processing data and allowing to manage orders, products, and so on
data layer holding whole necessary data in an ACID way

I believe there’s a need for a bit of clarification. This three-layered model does not relate to software architecture. We can still create microservices architecture using it, and we can stick to old but gold monolithic model.

My main goal here is to exercise skills in the design and implementation of a working system (and write about it), not to get involved in the theoretical discussion of the superiority of one approach over another.

System Elements

Now it’s time to talk about all the elements building each separate layer but also about connections between them. Although I’m going to give a brief analysis and description of each layer in this article, I’ll mainly focus on the logic layer.

Presentational layer elements

The frontend layer will be built from three main elements: main app, Next.js application in this case, and CDN service and Nginx web server.

It’s a rather straightforward solution: the main app built in React with the help of the server site rendering from Next.js creates a kind of sweet spot with single page application interactivity — and has all of the SEO advantages and faster initial loads from server-side rendering.

The role of the CDN here is to help with serving static files, mostly images. Nginx works like a reversed proxy and load balancer. Nothing too complicated. All the communication with the logic layer happens via GraphQL.

Logic layer elements

This main layer basically contains only two elements: The Keystone.js app working as main processing, and the command unit and GraphQL resolver, which act as an extension to the built-in one and work as a wrapper for text search capabilities for the underlying search engine.

But this statement is a huge simplification, so let’s get to the details.

Keystone manages three models fundamental for the whole system: User, Product, and Order.

The rest of the models can be directly or indirectly derived from them. First, User holds the basic information of customers and staff of our e-commerce system, mainly authorization info and assigned roles.

Also User has one Cart and can have multiple Order models and multiple Address models. Each Order model has one connected Payment and Shipment and holds multiple Product models.

Product has one Stock model (for the sake of simplicity, we assume we won’t use a multi-warehouse setup) and multiple ProductImage models.

Lastly, it has a connection to multiple Category models, and each one of them can be related to the parent Category forming tree.

It looks complicated, but it’s not (yet).

Basic models structure

Also, the role of Keystone is to manage the content of basic pages for the main frontend app. But more important, its role is also to manage side effects and external integrations.

An example of this side effect may be e-mail notifications associated with order processing steps and informing clients about it or indexing products data in search engines after changes.

On the other hand, examples of external integrations contain the usage of Stripe in order to process payments or connect to external API handling delivery (e.g., DHL API).

These functionalities will mainly harness the power of hooks in Keystone schemas. But not only that, we will need to create additional rest endpoints as webhooks in order to handle async responses from these integrations.

Lastl, Keystone admin UI works here as a kind of dashboard for staff to manage orders and process them.

Data layer elements

The last layer of the system's main responsibility is to collect data and ensure its persistence. Like in the previous one, there will be two elements here: PostgreSQL database and Meilisearch search engine.

For the moment, I’m writing this with no choice of the database. Keystone.js supports only PostgreSQL or SQLite, and I believe the second one is a bit too small for e-commerce system.

For the search engine, we need custom integration, so the choice here is much bigger, but I’m most familiar with Elasticsearch and Meilisearch.

I decided to choose the second one because I feel its JavaScript integration is a little better, and querying using facet filters is easier. Additionally, its footprint is a couple of times smaller than Elastic.

That concludes all the elements of our yet to be built e-commerce system:

System overview

Summary

Ok, let’s sum up. Now we have the basic overview of what we are going to build in the next part of this series.

Building an e-commerce system may look hard and daunting, but when we split this task into small, well-planned chunks, it’s not so terrifying.

Next time, we will write a little code and start with the project’s setup.

I hope you liked it. If you have any questions or comments, feel free to ask them.

Have a nice day!

Keystone.js custom fields: menu component

Maciej Krawczyk — Sun, 27 Feb 2022 21:16:17 +0000

Photo by Bicanski on Pixnio

Introduction

Last week my series about Keystone.js custom components had a little break, instead of building next component I’ve focused on more basic and low level topics. But this week, I’m back on track and main focus of this article is menu component. This time, we will build more content than data collection component.

Basic field types in Keystone.js works fine for crating pages and posts, but there’s no easy way to build page menu. Built-in JSON field has all necessary features to hold data describing page menu, but its visualization is pretty basic, contains only simple text area. From developer point of view it maybe sufficient, but for the end user of CMS system may be hard to grasp. And additionally there’s no guarantee that data from user input follows schema for our menu object. And from security point of view allowing user to input raw JSON data into system is at best bad idea. So we need to create custom view allowing user to create menu configuration object without possibility to add raw JSON data.

Requirements

Ok, let’s recap our requirements then. First, our component have to output menu configuration as JSON and store it inside database. This data has to follow known upfront schema and do not allow user to input raw data. Basically what we need is UI to hide JSON creation in user-friendly way.

Component creation

Like in the previous parts of we are going to use build in field type, and only creates custom visualization, but here also the goal is to abstract from user the duty of following the menu schema. But to begin with we have to create this schema and decides how this configuration should look like.

Let’s assume that our system requires menus which contains items with name and URL and optionally array of sub-items, but they can’t have additional children. So interfaces for our menu schema should look like this:

Based on that, we know that our example configuration should look something like this:

Like in previous component I’ve created separate subfolder menu for this field in views folder. Basic component structure will be pretty similar to previous components, so we have to import controller, Cell and CardValue from built-in version of JSON component and reexport them. Also, I’ve setup basic JSX using built-in FieldContainer and FieldLabel components:

I’ve already imported styles and two components InputField and IconButton, which were moved to other files in order to keep structure somewhat clean. First we have to crate state to hold our new menu and effect to react to this state changes and call props.onChange method:

Nothing fancy here, if value exists (case of editing menu) we are parsing JSON into object and set it as initial state, or if it doesn’t exist we set it as empty array. Effect at the other hand stringify current value of state and sets it as value. Next we have to create button allowing to create new menu items:

The only purpose of this button is to call method onAddItem on click event, create blank menu item and inset it into state:

Also, we have to render this newly added item with all necessary buttons including remove, move up or down and add children and inputs for setting name and path of this menu item.

Ok, here starts to happen much more. First we render a bunch of input and buttons components. Each input holds string information about item name or its path. Here it’s abstracted to other component:

Each field has own event handler, but logic here is pretty similar, so I’ve moved it into separate method. But two handlers stayed in order to provide clear structure:

Basically it takes field name, event and field index, creates new state object (in order to prevent mutating state), changes its value based on event value, but only when indexes are equal. And then sets it back to state. Next elements of item are functional buttons, their props contains also icon image path and additional CSS classes:

More important here are callback methods, e.g., onRemoveItem. It takes item index as prop and then removes this item from state, like before it creates new state and then sets it:

Next two buttons allows changing order of items, to move it up or down. Basically it rearranges the array of items and sets it back into state:

Purpose of last button is slightly different, it adds new children item to this menu element (creates drop-down), takes only index as parameter:

Ok, now we have the possibility to add children to our items, so we have to render them, they are pretty similar to main items. The main difference is in missing add children button.

Main difference here is in event handlers mostly due to need of finding of right children of right item to update/set. Most of them has additional parameter which is parent index. onRemoveChild method maps through all items and removes child only when both parent and child indexes matches:

Similarly, childChangeOrder maps children order only when both indexes match:

Input onChange handlers are also slightly different, due to nested nature of children elements:

To finish our component we have to add a bit more styling than usually, mostly due to the fact that we’ve created more custom elements than before, here are all the styles:

Here we have finished component inside admin UI:

And here is code for finished component:

Summary

Truth to be told it’s the biggest Keystone.js custom field view component I’ve ever build. Even though I had to build special visualization for this component I still was able to utilize built-in fields underneath. I believe it’s something what makes Keystone.js pretty useful and versatile tool. Everything depends on our knowledge of its possibilities and creative way of finding them usage.

This was the fourth article in my series about Keystone.js. I hope You liked it, if You have any questions or comments feel free to ask them. I’m not pretty sure what next article will be about, maybe I will write something more about usage of Keystone in real life projects, or maybe I’ll venture into other topics. Not sure. See you next time.

Keystone.js 101

Maciej Krawczyk — Sun, 20 Feb 2022 20:17:30 +0000

Photo by Bicanski on Pixnio

Introduction

Some time ago I was working on small project consisting website and app to collect research data. It’s a kind of project I am typically working at for my clients. In most cases my stack for that kind of project consists of WordPress as headless CMS, Next.js front, and custom-built Node.js app for data collecting. But often I feel this is kind of overkill. Especially in that last case. So I decided to look for better solution, best in the way to solve CMS and data collecting app in one. After brief research I’ve decided to stick to Keystone.js, it has all I need. But also it required a bit of customization (more about in previous articles, here and here). These customizations were so interesting I’ve decided to write about them, but when I was writing last article I realized that they were pretty niche and specialized. And there’s hardly any introduction to Keystone.js in its newest version, released at the end of last year. And that’s why I’ve decided to take brake from writing about custom fields and components and concentrate on simple, beginner-friendly introduction to Keystone.

New version of Keystone.js was released at the end of 2021. Basically it’s CMS system build with Express.js backend and admin UI in Next.js. But also it’s features packed including types safe Prisma ORM and TypeScript to elevate developer experience as stated in release blog post:

We extended Keystone’s foundations in Node, Express, and JavaScript, by adding Prisma to take care of the database layer and NextJS to power Keystone’s Admin UI. We also introduced a strongly-typed developer experience to make Keystone **the most advanced TypeScript Headless CMS**.

Thanks to usage of Prisma Keystone can generate migrations every time there’s update in data schema. It definitely allows co concentrate on building application business logic without concern about low level implementation details. In some cases it may not be what we want but in general it’s nice to have it working out of the box. There may be some performance issues due to ORM’s issue with 1+N queries, but it can be mitigated easily. Additionally, we have here built-in support for GraphQL what is always pleasant.

Ok, all that features are rally impressive, but what use case is prefect for Keystone? First it’s CMS, so managing content for websites, landing pages or blogs with custom frontends are usages where it shines. But not only, due to its wide possibilities of creating data structure, it can be used in many more cases. E.g., e-commerce — the challenge is only in designing right models for cart, and products, additional functionalities like payment processing can be added using built in extensibility of underlying Express.js.

Installation

I think It’s the high time to get our hands dirty and start our own Keystone instance.

Dependencies

Basically only thing we require to have is Node.js installed. Out of the box Keystone comes with SQLite, so there’s no need for additional database. But always we can switch to PostgreSQL, for now I am gonna stick to first option in sake of simplicity.

Let’s start

To begin with we only require one command. Open your terminal and type:

$ npx create-keystone-app

First, CLI will ask for name of our app, let’s try simple-keystone-blog:

Next, as prompted cd into that folder (assuming You are using Linux or Mac, if You need, use Windows equivalent commands), and start project with yarn dev and open http://localhost:3000 in browser.

Here You can set initial user account and login into admin dashboard. Out of the box there are two lists already created. User and Post, so we have all the basics to start writing a blog. Also, we can access all the data via GraphQL API or use sandbox accessible under http://localhost:3000/api/graphql.

Files structure

Ok, let’s have a look on the files structure in our project:

First two folders contains auto-generated views for admin UI, third dependencies to NPM packages. keystone.db, schema.graphql and schema.prisma are database itself, GraphQL and ORM configuration, each one is also auto-generated, based on our changes. Last three important files are keystone.ts, auth.ts and schema.ts. First contains whole configuration of our project — database configuration and imported from other two auth (session) settings and lists definitions.

Auth settings work really nice out of the box, and I won’t focus on them If You are curious of them check the docs. Much more interesting are lists settings, let’s have a look on them.

Schemas

This file exports object of type of Lists, which contains configurations for all the lists visible (and invisible) in admin UI. There are three defined here: User, Post and Tag. Each one calls function list with additional configuration object. This object can have up to six properties, but only one is mandatory — fields. It holds list of all the fields in that list (with their additional config). Amount of field types here is not huge, but it’s enough for all basic needs (more), it contains fields for all data types, files and images, relations to other lists and rich text editor. Rest of the props holds more advance configuration options, like display options in ui prop, side effects associated with this list in hooks prop, additional database or GraphQL config in db and graphql. Additionally, some of this config properties can be used directly on the level of each field to extend their functionality.

All this config options gives us wide possibility to build complex schemas for each list including side effects allowing to integrate external tools and extend it further. In topic of extensibility there are two additional things worth mention. On top of all the fields in main config object (keystone.ts) we can modify setting of underlying Express.js server. Options here are pretty basic, except one — extendExpressApp. Basically this option allows us to add more REST endpoints or middleware to our backend. Second important option is extendGraphqlSchema, generally it does the same job as previous one, but allows adding custom GraphQL resolvers.

Summary

I know I’ve barely scratched the surface of what really Keystone.js can do, but on the other hand I hope I’ve showed the real potential in it. At first glance it looks like pretty basic CMS with limited options. But when we dig deeper there are tons of possibilities and after careful planing and clear requirements it may turn out that we don’t need to build another custom system. Instead, use something nice as a base and extend it.

That’s the third article in series about Keystone.js. In this one I’ve decided to take step back, and get to basics. I hope it’s enough to convince You to give it a try. Next week, I’m getting back to series and custom components, this time we are going to extend JSON field and use it as data for building pages navigation menu.

Keystone.js custom fields: map component

Maciej Krawczyk — Sun, 13 Feb 2022 20:30:46 +0000

Photo by Bicanski on Pixnio

Background

This article is the second one in the series about Keystone.js custom fields. Here you can find the first one.

Projects I am usually working one are rather small but has its own specificity due to their background (more about that in first article). In most cases they contain full-fledged CMS system and custom-built data collection tool. But last one left me wondering is there any way to kill two birds with one stone and simplify whole system. Solution here was to use Keystone.js possibility to create models for content data but also for research qualitative data. The only challenge here was that built-in set of fields is nice but in some cases too basic. Here we are going to focus on map field allowing user to store localization of points in database (coordinates to be exact). Like in the previous one, built-in text field was sufficient to store data, but I had to create visual part of this input almost from scratch.

Requirements

Main goal here was to create field showing interactive map to the user allowing to zoom and pan view and also click to add point. Then as a result save coordinates of this point into database. Also, we have to store this data into text field. Alternatively it can be stored in two separate columns in database, one for latitude and one for longitude. But I believe it’s more complicated solution, it requires custom field controller and also changing backend part of the field (see details). In that case solution with one text field seams like much better. To sum up, we need to:

Display map,
Add controls (pan and zoom),
Add possibility to add point to map,
Save point coordinates to database.

Component creation

Fortunately we don’t have to build everything from scratch. Most of the heavy lifting will be handled by OpenLayers library. There are many NPM packages handling maps, but the most important advantage of this one is great and complete documentation (most parts). So first we have to add it to our Keystone project:

$ yarn add ol

I am using Yarn here, but also you can install it using NPM:

$ npm i ol

Additionally, due to some dependencies mismatch I had to install separately geotiff.js, depending on actual version at the moment you read this it may not be necessary.

$ yarn add geotiff

Like in previous component I’ve created separate subfolder coordinates for this field in views folder. Basic component structure is the same as in previous component, so we have to import controller, Cell and CardValue from built-in version of text component and reexport them. Also, I’ve setup basic JSX using built-in FieldContainer and FieldLabel components.

The base of our map component here is this div tag. And basically that’s all JSX needed. Whole logic and map rendering is going to happen inside this useEffect hook. Additionally, I had to add useRef hook to keep reference to that mentioned before div.

First, we need to import needed elements from ol library:

Basically map created with OpenLayers is only a container, we have to add layers in order to present our desired map. First, I created base map layer source using TileImage class and map tiles from Digital Atlas of the Roman Empire (more info):

Then, I had to create map instance:

Here as you can see Map requires a couple of configuration properties. First, we have to set reference to DOM element which will contain our map, mapRef.current in that case. Next property is an array of initially created layers. Here I’ve created TileLayer based of source created before. Last property here is view, it sets map initial center (coordinates, here in EPSG:3857 coordinate system) and zoom. Actually this is the only one obligatory property when creating map (docs). After this steps, we have ready map visualization which can be controlled by user. Next, we have to add another layer to hold point created by user. In this case it’s VectorLayer with corresponding VectorSource and set of styles for points. And then we have to add it into our existing map.

Additionally, here I’ve created styling for the point added by the user. In order to do that I have to instantiate Style object with configuration with property image. There are other ways of doing it, but I prefer this one (check docs). Value of this property is instance of Circle class (in my case aliased as CircleStyles), with configuration object containing radius, fill and stroke props. Also last two are instances of corresponding classes. Basically it sets point visualization to circle with radius of 5 pixels, red, slightly transparent fill and opaque red border. Now map is ready to add our custom handler for singleclick event to allow user to add point. But first we need a way to store our point coordinates:

Also, here in case of situation when field already have value (e.g., when we are editing the record) we are setting coordinates variables to this value. This little complicated way of reading value is mostly caused by the way that Keystone internally handles data for text field. Ok, next we have to create handler for the event I’ve mentioned before:

To create this handler we have to call on method on our map object. It takes two parameters, string with event type and callback function which has one parameter, evt being type of MapBrowserEvent. Basically there are two purposes of this callback, to set new value of field (also lat and lon variables) and call addPointAndFocus method. Here it is:

This method does three things, if variables lat and lon exists and has values then it removes all previous points. In our case there can be only one to remove, but method getFeatures returns array. Important thing to note here is that we are operating on vectorSource not vectorLayer. Next, new point feature is created with current coordinates. Lastly map view is set to have center on newly created point and increased zoom (in case if it’s smaller than 8). And now our component is almost ready, lastly we have to add a bit of styles to it. Mostly because map container has to have height set to value in pixels:

One last thing to add was all styles from OpenLayers library. Here I’ve used React-Helmet package to help with that task:

Ok, here is finished component, and how it looks like inside the app:

Component in action.

Summary

Creating new fields in Keystone maybe easier than it looks, I hope I was able to show that. At first It may look daunting, but it’s no different from creating other ordinary React components. Everything depends on our requirements, and how complicated they are. Also, libraries like OpenLayers may be a little scary at first glance, additionally quick start tutorial in documentation is focused mainly on usage in static sites (by static I mean static like ten or more years ago) what can cause some problems to users used to current approach with single page applications and gigabytes of NPM packages. But when we dig a little deeper API documentation is really great and informative.

This was second article in series about custom Keystone field components and I planed to finish it with the next one about slightly more complicated menu component, utilizing JSON field underneath. But when I was writing this one I realized that this topic is pretty niche and there’s a need for simpler, more introductory overview of Keystone.js as a CMS. So, the next one will be kind of Keystone.js v 6 101, and then we will get back to menu custom field. See you in the next one!

Keystone.js custom fields: text with autocomplete

Maciej Krawczyk — Sun, 06 Feb 2022 20:25:55 +0000

Photo by Bicanski on Pixnio

Problem background

Some websites and apps I’m working for my clients are research projects in humanities especially in archeology. And because of that their needs are pretty specific. On the one hand they require simple webpage to present their research results and fulfil administrative obligations, but on the other hand, they also require custom tailored tools to catalog research data. Usually these are qualitative rather than quantitative data.

Typically, my stack to fulfil this needs is pretty extensive. It starts with WordPress as CMS, working in headless mode with Next.js frontend. And this covers website part. Data collection tool was in most cases custom-made app in Node.js and Express or PHP app with Laravel framework. It was completed by Elasticsearch and another large tools. But most of the time I have gut feeling that’s a bit of overcomplicated solution and for sure bloated one. When I was starting current project, I made decision that I have to find better and lighter way to do that, especially because in this case data model was not so complicated. My research started with looking for other CMS, best in Node.js and headless out of the box. I knew about two of this kind of systems, Strapi and Keystone.js. First one is a little to rigid for my taste, but second one looks like perfect solution for my needs. Possibility to define schemas in code and easy configuration was really nice. But after that, I had great aha moment. What if I can also use it to define models for data catalog? Is there possibility to kill two birds with one stone? I believe there is. There was one issue thou, but about this later.

About Keystone.js

On Keystone website we can read that it is superpowered CMS for developers, with built-in GraphQL support and Management UI in place. Also, there’s new, just released version 6. It has really impressive set of features, authentication and authorization, UI, it uses TypeScript and much more useful and necessary features. The only thing we have to worry is our entities structure. But there are a couple of problems. Abstraction layer is really thick in here. In general, it’s not an issue but in some places may become a concern.

Built-in set of fields which can be used in lists and schemas here is more than enough in most basic cases. But unfortunately my case turned out to be not so basic. Maybe it’s caused by specificity of my project, but I needed some extra fields to work with. The issue was not in the field types provided by Keystone, text or JSON fields are versatile enough to serve many purposes. But their visualization in admin panel can be insufficient sometimes.

During development of this project I had to create three custom fields, but only their UI aspects had to change. Underneath they still are text and JSON fields. First of this three is text field with additional autocomplete and I gonna focus on that in this article.

Requirements

Text field with autocomplete? It sounds like relationship field from Keystone basic set. Yes, but not really. Relationship means in that case that there are at least two entities connected to each other. In this case I just need to get suggestions from table column corresponding to that field. So, what this field has to have? First, it has to have a possibility to get all suggestions from database (or search engine in my case) while typing, then present them to user and let her/him select one or finish writing own version.

Documentation info about custom field

Mostly because it’s new, just released version of Keystone.js documentation is not so clear in some places. But part about using custom fields is really nice. Even though for me the most useful part of it was the link to source code of built-in fields (for the moment when I am writing this the link is broken, use this instead).

In our case interesting is text field (code). We can use this file as a template for what we want to achieve. In documentation, it’s stated that custom field view have to exports four things: field controller and three React components — Field, Cell and CardValue. Each one responsible for different visualizations. But here we are in comfortable position, to fulfill our requirements we have to change only Field visualization, others are fine, so we can reuse built-in ones. Also, there are no changes in controller, so original one can be reused too.

Steps to resolve

First what I did was to create separate folder views to hold all custom fields visualizations and subfolder autocomplete for our current visualization. Main file there is view.tsx containing our new field. Because we won’t be modifying field controller and Cell and CardValue components, so we can import them and re-export right away.

I believe it’s not necessary, but in sake of completion and code readability is reasonable to do it. Also, here we can add rest of necessary imports:

Here we are importing basic React hooks which will be necessary later, some types from Keystone core library, and built-in field components to blend in our custom components into other fields in Admin UI. Now is time to create our Field component end exports it. Also, we have to declare some basic state of the component. We are going to need it later.

According to documentation props in this case have several properties: field which is object containing all methods and properties form field controller, value holding field value, in this case as an object:

Mostly it depends on kind of operation we are currently performing. It looks different when user creates entity and differently when it’s updated. Also, there are two boolean values autoFocus and forceValidation, both rather self-explanatory. Last one there is method onChange which is value setter here. Additionally, I am not pretty sure why, but it is optional, and we have to keep that in mind. Ok, now it’s time to construct our component JSX:

There are to main parts of this component. First one creates text input using built-in components (with some extra events handlers) and simple suggestions list created using ul and li tags. Also, here we can see usage of visible component state. It’s responsible for showing list of suggestions on focus event (and hiding on blur).

Skeleton of the component is in place, now we have to add some muscles to it and add missing methods onChange (our one, not the one coming from props), onSuggestionClick and setOnChange helper.

First of these methods, setOnChange is a kind of helper function responsible for calling props.onChange method and setting new value of field. As I mentioned before, it’s optional, so we have to close it inside if statement and then call it with new value (keeping in mind, how this value should look like). Next one, onChange fires on each change event. It calls SetOnChange first and then starts to look for suggestions, if the input is more than 3 characters long it calls async method getSuggestions with current input value and then calls set state method setSuggestions. Otherwise, if input is too short it sets suggestions to empty array. I believe, in some cases it maybe necessary to debounce this method, but I’ve decided not to. Last one onSuggestionClick sets field value to the value of clicked suggestion.

Next we need to create previously mentioned method getSuggestions:

Its purpose is simple, it fetches data from server in JSON format and returns suggestions. In this case it is a custom endpoint created according to docs (more):

It’s simple Express.js route calling autocomplete method and returning results as JSON. It takes three parameters: query to search, index to search in and field to return. In my case under the hood it uses Meilisearch search engine and its JS integration — Meilisearch JavaScript. It’s really nice, reliable and what’s the most important lightweight search engine, useful alternative to huge Elasticsearch. But I am using it because it’s also needed in other part of the system, in other cases using built-in QueryAPI should be enough.

Lastly we have to add a bit of reactiveness to our component, in order to do that we have to add two useEffect hooks and one useCallback:

First, we create memoized callback to load initial suggestions on component loads and value is already set (in case of editing record). I used this hook to make sure that this method is only declared once, on component render. And then we call it inside first useEffect hook, the one running only on component render. Whole idea here is to already have the suggestions when user starts editing the corresponding field. Last hook changes visibility of suggestions based on length of suggestions state array.

Now our component is working as planed, but there’s still one issue. It’s not pretty at all, so we have to add some styling to blend it better with the rest of UI. I moved entire styles to separate file just to not make it messy, but it’s strictly personal.

Here I’ve decided to use Emotion library just because I know it enough, and it does its job perfectly. Styles here mostly mimic built-in ones to help our custom component to blend into UI and look like it was always there.

So, here’s complete component:

Component in action

Summary

To sum up: using custom fields in Keystone.js is not that hard (at least custom visualizations for fields). It requires a bit of research and clear idea what we want to achieve, but it can easily resolve minor problems. And most important can save use from writing full-fledged custom system where it’s not needed. When I was starting my first job as a junior web developer, someone told me that real dev knows when to use library/framework and only adopt it and when is better to make custom solution. I believe even though couple of years passed I am still learning this.

To be honest it’s the first article I’ve ever written. I hope it’s clear and informative, and I really appreciate your feedback. I am also planing to write about another two custom fields I’ve needed to create. And I am challenging myself to publish one each week. I hope it will work out. See you in the next one!