DEV Community: Varun A P ⚡️

Fast Search for Static Sites (JAMStack)

Varun A P ⚡️ — Sun, 07 Jun 2020 00:00:00 +0000

The JAMStack revolution is already here!

If you are into frontend, you might have already heard about this, The JAMStack. From the time Netlify popularized the concept and with the push from the Gatsby community, people have started using this extensively. This is nice considering the benefits that come along.

✔️ Faster Page loading
✔️ No more server-side rendering (no need to maintain FE setup)
✔️ Developer Experience

Overall, it is an advancement to the entire web, reaching far more countries and devices.

Ok. So?

I've built a bunch of JAMStack sites (mostly for fun, some for work) using various tools and frameworks like Gatsby, Sapper, Hugo, VuePress and more. Among those sites, only a few actually conceptualize JAMStack's potential. I mean, anyone can build a 5 page website with just HTML, CSS and maybe a bit of JS. We did that for years without any of the tools that we have at our disposal.

In this article, let's uncover more about a crucial part of any large website ▶️ SEARCH ◀

I explored most of the things stated below while I was building Confs.space over a few weekends.

How is this site different? Let's take a look at the stats:

📺 Hosts 120+ Conferences
🎤 3000+ talks
🖨 Generates 4000+ Pages

The above metric is something to keep in mind while we go over the topic.

Adding Search

When you have a site that has so much content, it is important to provide the ability to find what the user is interested in. And search is a quick and accustomed way of doing so.

With this thought, I started looking at how I can do it effectively as there is so much data.

Tech stack and the considerations before implementing search:

→ Hugo static site generator (template based)
→ JavaScript, for few operations like persisting darkmode, toggling menu, etc.
→ Hosted on Vercel's now (Brings in the possibility of having a serverless function)
→ Content is stored as markdown files
→ Performance should not degrade (this is personal 😛)

Available Options

With all this in mind, the most obvious thought was that it might need some form of backend setup to have an effective search. I've also heard about Algolia, which offers site-wide discoverability services.

At the same time, I explored options for some form of client side search as well.

Breaking it down

Let's breakdown the things that would be required for each of the options.

Option 1: Custom Backend

a) Require to setup the entire flow, reading data from repository to searching it effectively

b) Need separate hosting (Serverless might not make the cut when there is a cold start)

c) Full customizability - total control over the implementation

d) Maintenance effort

Option 2: Third-party services

a) Setup some CI like Github Actions to update committed data to third-party's index for the site

b) Searching would involve API calls to the third-party service

c) Add custom javascript code to the site (usually adds between 25kb and 75kb of additional JS on load)

d) No need to worry about scaling (this is probably the reason third-party search providers exists)

e) Cost

Option 3: Client-side search

I came across two contenders for client-side search: Lunr and Fuse.js.

a) Requires a search index to be generated - either during runtime or build time (pre-computed index)

b) Effectiveness depends on the library - mostly does fuzzy search

c) No additional API calls (for any number of searches), can make the search feel quick (UX)

d) Adds ~10kb additional load to the page

Implementation

Finally, I decided to go with a client side implementation as it's of least effort. Also, with hugo, generating additional file(s) for the index is a breeze (just adds a few milliseconds to the build).

In the first version of search for confs.space, I decided to go with Lunrjs. It was quite straight forward to implement as the index only includes a bunch of JSON objects with fields required for the search.

Added the following lines for generating the index:

{{- $.Scratch.Add "index" slice -}}
{{- range where .Site.Pages ".Params.template" "==" "talk" -}}
{{- $.Scratch.Add "index" (dict "title" .Title "ref" .Params.url "tags" .Params.tags) -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

For confs.space, in order to allow for searching videos, I needed to include all talks to the search index. The above code adds the title and tags of the videos for providing appropriate suggestions.

Scratch in Hugo is much like a state. The content is added to the state as dict, then JSON stringified and added to the page.

But..

Although I set it up quickly, there were issues with fuzzy search. Even after exhausting combinations, I was not able to get accurate results. It seemed to have an issue with field weights.

So, I decided to try Fuse.js. It didn't require any change for the index generation. Only the JS code had to be changed. As I hoped, the search was perfect.

PS: Fusejs is lighter than lunr, just ~4Kb.

Optimization

Now that the functionality is as expected, I wanted to improve the user experience.

You might wonder what more can be improved. Surprisingly, there are some crucial things to consider.

1 - Size of the index

Since there are a huge number of videos, the size of the generated index is quite large. Loading this index for all the users would result in an overhead.

Fix: Since the index is required only for the users who perform the search, the index is only loaded when the search field is in focus. The index is then used to initialize Fuse.

Improvement: It is possible to split the index into multiple smaller indices based on categories. Doing so will enable users to search with a specific category while loading smaller data.

2 - Search page

By moving the search to client side, there is no way of indexing search pages (since the results are dynamic). Because of this, the search results need to be loaded without any delay in UI. [Easy if the page was pre-rendered]

Partial Fix: For now, I haven't added any pagination for searches and limited the search results to be a maximum of 30 videos in length. And this result is then dynamically added/modified in the DOM (not quite ideal in the virtual DOM world but gets the job done).

Since the index is already loaded, the search experience feels quite snappy (try at Confs.space).

To Conclude

Client-side search is actually considerable for JAMStack sites based on the scale

For Confs.space, I might have to look at a different solution as site's content increases, next step being splitting of index into smaller chunks.

I'll soon be making the source of confs.space open and so I'd be able to use Algolia under its open source initiative.

If you have any questions or suggestions, I'd love to hear them.

Disclaimer: The article in no way is opposing the use of Lunr. It's just that I had troubles with getting it to work.

Let's talk about CSS Subgrids

Varun A P ⚡️ — Sun, 09 Jun 2019 00:00:00 +0000

CSS Subgrid is a part of CSS Grid Level 2 Spec by CSS working group. It solves a common use case which wasn’t so obvious when grids were launched. Let’s dig deep into what it is!

CSS grids are primary used for building complex layouts for the web. And when you start building such layouts, there would be use-cases where you would want to use grids for child components as well. This is great! But there is one non-obvious thing that happens when you do so.

Each time you write display: grid, a new grid context is created which is totally unaware of the parent grid.

Now, due to this obscure spec, you would end up with layouts that are not aligned to your main grid(parent’s grid) of the webpage. And it’s because of this problem, Subgrid are here.

Subgrid allows a grid-item with its own grid to align in one or both dimensions of the parent’s grid

Here is an example of a two-column grid container with two child elements.

In this case, the two cards are a part of grid and are aligned properly. But in you see the content of the grid, they are aligned differently for each other although the card itself is a grid.

/* Styles for grid without subgrid */
.container {
  display: grid;
  grid-template-columns: repeat(2, minmax(50px, 1fr));
  grid-gap: 10px;
  .card {
    display: grid;
  }
}

With Subgrid

When you specify the child card as a subgrid, then we can specify the child to use parent’s grid (with auto rows in this case).

Here, the grid line No. 2 is the same for both the cards, thus making the alignment same.

/* Styles for grid with subgrid */
.container {
  display: grid;
  grid-template-columns: repeat(2, minmax(50px, 1fr));
  grid-gap: 10px;
  .card {
    display: grid;
    grid-row: span 2;
    grid-template-rows: subgrid;
  }
}

PS: Properties like grid-gap and names grid lines are also inherited by the subgrid from parent grid.

Checkout the above example for yourself in Codepen.

Currently the Subgrid spec is only available in Mozilla Firefox Nightly release, but support for other browsers are also currently being worked upon. caniuse.com

Bonus: Here is a Scrimba Screencast that explains Subgrid with code.

https://scrimba.com/c/cgq2knS6

Chrome DevTools: Content Overrides

Varun A P ⚡️ — Fri, 31 May 2019 17:23:18 +0000

Have you debugged a bug in production? Or made multiple deployments to verify one specific issue?

Web Devs are likely to encounter or have encountered such situations numerous times. So, let's look at a way how we can ease such situations with Chrome DevTools.

In order to figure out issues, one might log values just before the code breaks or go a bit further to add a breakpoint before a few lines, update the breaking code & continue execution. (especially for code that runs on page load)

Yes, this is a pain!

But, there is another simpler way of doing this.

Chrome supports a feature called Overrides in its sources panel. The feature allows to override any files that is received from network with your own copy.

In order to use this,

you'll have to select a folder on local machine for chrome to save the modified version of files
Once chosen, edit any file and hit CMD + S or Ctrl = S to save the file to overrides folder

Once the file is saved, reload the page so that chrome can use the overridden file on page load. this helps you to modify and test code right in production before pushing out a fix. 🥳

Preview (overriding file content):

Try it yourself at Lessons: Content Overrides (Demo Page)

You can also clone the repo below and run the same in your local machine.

apvarun / learndevtools

Debug Better With DevTools

Learn Dev Tools

Collection of walk-throughs for learning to use developer tools in the browser. Read more here

Getting Started

Visit https://learndevtools.now.sh/ to learn and try out various features of devtools to help you be more productive at work.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

View on GitHub

Previous Post: Logging Without console.log

Chrome DevTools: Logging Without console.log

Varun A P ⚡️ — Thu, 30 May 2019 17:51:37 +0000

If you have ever written JavaScript code, you are highly likely to have come across console.log statements. They are life savers to identify logic-based issues in code.

But there is a limitation here. You'll need to modify code to add console statements in order to verify values (another option being adding breakpoints).

But, that's no more true.

Chrome now allows you to add logging statements without actually adding it in your code. And they call it logpoints.

In order for you to add a logpoint, open up the code where you need to log and right-click on line number. Select "Add logpoint" option and enter the list of variables/values to be logged (separated by comma).

Once set, Chrome will log the values to console each time the line is executed. 🥳

Preview (logging without console logs):

Try it yourself at Lessons: Conditional Breakpoints (Demo Page)

You can also clone the repo below and run the same in your local machine.

apvarun / learndevtools

Debug Better With DevTools

Learn Dev Tools

Collection of walk-throughs for learning to use developer tools in the browser. Read more here

Getting Started

Visit https://learndevtools.now.sh/ to learn and try out various features of devtools to help you be more productive at work.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

View on GitHub

Previous Post: Conditional Breakpoints

Chrome DevTools: Conditional Breakpoints

Varun A P ⚡️ — Wed, 29 May 2019 17:24:49 +0000

Have you ever tried to debug a loop or code that executes at intervals? I know it's hard to navigate through the loop to understand the bug.

Conditional breakpoints might be just the one for you.

Conditional breakpoints

You would be familiar with adding breakpoints in your source code in order to debug code execution. This is a pretty common practice of finding bugs or understand the execution of code.

But Chrome also has a feature called conditional breakpoints, which allows you to break at a particular line only if a specific condition is evaluated to be true.

The condition can include any variable in code that is available at that particular line. Even the variables inside the closures can be accessed in the condition evaluation.

Preview (debugging with conditional breakpoints):

apvarun / learndevtools

Debug Better With DevTools

Learn Dev Tools

Collection of walk-throughs for learning to use developer tools in the browser. Read more here

Getting Started

Visit https://learndevtools.now.sh/ to learn and try out various features of devtools to help you be more productive at work.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

View on GitHub

Direct Link: Lessons: Conditional Breakpoints

BONUS: The new version of Chrome's Canary release (at the time of writing this post) has landed with inline breakpoints which allow you to break at specific points in a line itself. This is helpful in cases where you do multiple function calls or computations in a single line.

Previous Post: Detecting Element Changes

Chrome DevTools: Detecting Element Changes

Varun A P ⚡️ — Tue, 28 May 2019 18:31:26 +0000

Google chrome comes with a lot of handy tools to assist web developers in building a better experience for users/customers. And it has added numerous features over the last few years, forcing other browsers to follow the same path.

This article will talk about how one can detect changes on elements in the DOM during runtime, primarily for debugging. This is the first in a series of posts covering useful tools of Chrome Dev tools.

Elements Panel

Elements panel, as you know, is the place where you can see the elements that are currently available in the DOM (Document Object Modal).

Chrome allows to listen to three specific events that occur to an element in the DOM tree.

Add/Update of an attribute value
Change in the element's children (subtree)
Removal of node (child elements)

How to add listeners?

Open the Dev Tools and select the Elements tab. In the list of DOM nodes, select the node on which you need to add listener.

Once the node is selected, a three-dot menu will appear on the left of the node. Clicking on the dots will open a context menu with a bunch of options.

Now, for adding the listener, Select the "Break on" option and then the type of event. Once selected, chrome will listen to changes to that particular element and stop JS execution at the line of code which modifies the selected node.

Once the breakpoint is hit, you will get to see all the contextual information and memory snapshot, which can be used in identifying the issue source.

Preview (breaking on subtree modification):

If you want to try it out yourself, head over to the repo below. It has the file that is shown in the Video. Clone the repo and play around :)

apvarun / learndevtools

Debug Better With DevTools

Learn Dev Tools

Collection of walk-throughs for learning to use developer tools in the browser. Read more here

Getting Started

Visit https://learndevtools.now.sh/ to learn and try out various features of devtools to help you be more productive at work.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

View on GitHub

Direct Link: Lessons: Detecting Element Updates

Auth for Hasura GraphQL engine with Firebase - My thoughts

Varun A P ⚡️ — Thu, 25 Apr 2019 00:00:00 +0000

Hasura’s GraphQL engine is itself a drop-in backend for most of the use cases. It provides you the flexibility to setup a fully functional backend with GraphQL as interface for the front-end. It acts as a layer between the Postgres database and your frontend applications(web or mobile).

Authentication is entirely a different class of problem which is also common for any kind of application. Hasura already has documentation as to how their platform can be made to work with Auth0 or Firebase authentication service. 🔥

Everything seemed perfect when I started. And I chose firebase as their auth service doesn’t have a user count based pricing (at least at the time of writing this). This would definitely save a couple of $$ as the application scales.

And so I went ahead with the implementation.

The Problems

But I encountered a problem. As per the documentation, I had to create accounts using the firebase client-side SDK. This was a bummer for me as I had to create bulk accounts for my use case.

Another problem was that if I went ahead and created accounts using client-side API, I would also have to relay the same details(except password) to my datastore(GraphQL engine) as well. This is really unintuitive and doesn’t ensure that the information is present in both locations.

How did I address it?

T.L.D.R : Event Triggers + Cloud functions

GraphQL engine also has support for event triggers which can be setup to call web hooks based on DB operations. Any insert, update or delete operation on tables via the GraphQL endpoint will trigger the events associated with it.

I was able to set up cloud functions that can be called on insert events, cloud function being hosted on firebase functions. The end point of the cloud function is given to event triggers, which then makes a POST call when the event is fired.

Sample cloud function to create user

const functions = require("firebase-functions"),
  admin = require("firebase-admin"),
  express = require("express"),
  cors = require("cors"),
  bodyParser = require("body-parser"),
  generator = require("generate-password"),
  env = require("./envfile");

const app = express();
app.use(cors({ origin: true }));
app.use(bodyParser.urlencoded({ extended: false }));
admin.initializeApp(functions.config().firebase);

app.post("/", (req, res) => {

  // Check for POST request
  if (req.method !== "POST") {
    res.status(400).send("Request not supported");
    return;
  }

  // Destructure POST params
  let { email, accountrole, id } = req.body.event.data.new,
    headers = req.headers;

  // Custom header for API call authorization
  if (headers.authorization !== env.authKey) {
    res.status(404).send("Unkonwn error");
    return;
  }

  // Generate account password
  const password = generator.generate({
    length: 12,
    numbers: true
  });

  admin
    .auth()
    .createUser({
      email: email,
      emailVerified: false,
      password,
      uid: String(id)
    })
    .then(userRecord => {
      // See the UserRecord reference doc for the contents of userRecord
      console.log("Successfully created new user foe email ", email);

      // Send them an email with initial password or a password reset link
      // SendEmail(...)

      // Custom claims for validation by the GraphQL engine
      const customClaims = {
        "https://hasura.io/jwt/claims": {
          "x-hasura-default-role": accountrole,
          "x-hasura-allowed-roles": ["admin", "user"],
          "x-hasura-user-id": id
        },
        authorized: false // custom parameter that can be updated later
      };

      // Assign custom claims
      return admin
        .auth()
        .setCustomUserClaims(userRecord.uid, customClaims)
        .then(() => {
          console.log("Custom claims set for " + email);
        })
        .catch(console.log);
    })
    .catch(error => {
      console.log("Error creating new user:", error);
    });
});

exports.user = functions.https.onRequest(app);

As the signup form is filled by the user, a mutation is fired to insert the data into the user table. This will then post the same information to the cloud function using the event trigger.

The trigger runs only on new insertions to the user table.

Note that since we have a check for headers in the cloud function, you will also need to pass the secret token in header as below.

So, once the signup is done, an user will be created in the firebase auth (with email and generated password).

And in order to login, we can use the firebase auth script and directly authenticate from the UI.

What’s achieved

Signup process can inlcude multiple meta information along with username/email. (password being autogenerated by this process).

If we were to signup user directly to firebase using their SDK, we would have to send two parallel API calls at the least to have user information in our DB as well. This is simplified and made one way so that there is no inconsistency in storage of userdata.

You will now be able to create accounts in bulk as the event triggers are fired for each insert to the table. The process becomes to fully event driven.

Hope this was helpful for someone out there trying to tackle the same problems. I’d love to hear from you on twitter in case of any queries or suggestions.

Beyond this year: 2019 goals

Varun A P ⚡️ — Sun, 30 Dec 2018 00:00:00 +0000

As the year comes to an end, it’s time to look back at the events and improve upon it. 2018 has been a roller-coaster with a lot of new experiences. I have learnt so many things, development to design and more.

The biggest one was moving from React ecosystem to Angular for my fulltime job. And yes, I switched my job this year and I couldn’t be any happier. ( Off-the-record : I work at Springboard - an Edtech startup.)

Starting Angular? Create a basic Angular CLI project and build something. You will learn more from it than from the docs.

I also had a lot of learning going with designing of UI elements. As it’s a continuous process, it will be continued in the next year too. Another great experience that I had was working from China for a month. It was a completely different world, and I also did manage to make some good friends there.

Apart from these, I switched to Mac from Windows, tried my hands on starting a podcast and a video series, launched a couple of personal projects, etc.

Setbacks? Yes, there were, and some learning from them too. I got selected as a speaker at React Rally, but was not able to make it due to visa issues.

So as we end 2018, I would like to thank everyone who supported and inspired me to achieve all the above things.

What does this mean for 2019?

The bars are definitely high! But that shouldn’t stop anyone from dreaming big 😉. I have so much stuff in my todos that needs my attention and so I am looking at planning it accordingly.

Major Goals

🚀 Increase open source contribution
👨🏻‍💻 Learn and use GraphQL on the server
✍🏻 Write at least one blog post a month
📹 Create more videos with live coding screen shares
📚 Learn Python (Django)
🛫 Plan at least one trip with family and one with friends
🗣 Speak at a conference
💪🏻 Workout regularly

These are not a new year resolution but goals that I’m committing publicly. I’m also looking at being active on Twitter to learn and support the awesome web dev community there.

Wrapping up

Kudos to you for reaching the end. Let’s make 2019 more fruitful to oneself and to the web community 🥳. Keep learning, respect & help others and stay motivated!

Happy 🎊 2019 🎊!