DEV Community: Abdus Azad

Build a Search API from Scratch

Abdus Azad — Wed, 23 Feb 2022 07:23:08 +0000

Search functionality is one of the most common features you see in any digital product. I would hesitate to use a product that does not contain a search bar (given that the search bar is necessary). However, creating a search engine as big as Google would take lots of time and energy, and may not be possible for a lone developer. So, here I will demonstrate a simple way to build a search engine for small to medium-sized products.

The Stack

Before getting into the actual coding, let me introduce you to the tech stack. I will be using JavaScript for both front-end and back-end, and LunrJS to index and search through the text content.

In case you have not heard of LunrJS, it is a full-text search library that is a bit like Solr, but much smaller and not as bright. A library that is written in JavaScript for both client-side and server-side. LunrJS indexes text-based content into a JSON document. The production bundle of LunrJS 8.2 KB in size, which makes it a good fit on the front-end too.

Some of the Lunr alternatives are: js-search, flexsearch, fuse, wade.

Flow

To integrate search functionality into a website, we need some data. We will be searching for specific information from this data lake (well, quite a small lake for now). To store data, we can use any of the available databases depending on the project's needs. For this demonstration, I am using MongoDB (via Mongoose ORM).

Here's how to initialize a database connection using Mongoose in a serveless environment:

import mongoose from "mongoose";

let mongoDBConn: mongoose.Connection | null = null;
const connectionStr = process.env.DATABASE_URI;

if (typeof connectionStr !== `string`) {
  throw new Error(`database uri: not a string`);
  process.exit(1);
}

if (!mongoDBConn) {
  mongoose
    .connect(connectionStr)
    .then((m) => (mongoDBConn = m.connection))
    .catch(console.error);
}

You may notice an unusual way of initializing the database connection object. I am caching it inside a variable. This way, the subsequent serverless invocation will be able to reuse it.

function getBlogSchema() {
  const BlogCollection = new mongoose.Schema({
    title: { type: String, required: true, unique: true },
    // rest of the document fields
  });

  BlogCollection.index({ url: 1, title: 1, description: 1 });

  const model = mongoose.model(`Blog`, BlogCollection);
  model.syncIndexes();
  return model;
}

export const blogModel = mongoose.models.Blog
  ? mongoose.models.Blog
  : getBlogSchema();

Again, another non-conventional way of creating a database model, all thanks to serverless. Since we cached the database into a variable, we should check if the model exists in the cache. We can't recreate a model in Mongoose. Trying to do so will throw an error.

Moving on, we have to install the package lunr by running yarn add lunr. Once done, it is time to setup lunr. Let's start with the imports.

import fs from "fs";
import lunr from "lunr";
import { blogModal } from "./path/to/blogModel";

Then, I am going to write a few helper functions. These functions will help us execute the search systematically.

buildSearchIndex: As the name suggests, this function will build a search index. A search index is a collection of data stored in a disk or inside the memory. This collection is parsed and stored in a way so that querying it becomes easier and more efficient.

  export async function buildSearchIndex(): Promise<lunr.Index> {
    try {
      const docs = await blogModel?.find();
      const index = lunr((builder) => {
        builder.ref(`_id`);

        builder.field(`title`);
        // additional fields if any

        for (let i = 0; i < docs.length; i++) {
          const d = docs[i];
          builder.add(d);
        }
      });

      return index;
    } catch (err) {
      console.log(err);
      throw err;
    }
  }

Let's understand what is going on. First, we are calling the lunr function, which accepts a callback. The first parameter of the callback is an object called builder (passed automatically by lunr).

builder.ref method is used to reference the original document. Assign a unique field to it. _id, for example.

builder.field method tells the builder what field to index. Add all the fields you want to search in.

Finally, we are adding documents to the index by calling builder.add method inside a for loop.

saveSearchIndex: This function saves a given search index to the file system. When the data lake size increase, it is no longer efficient to create an index on every API call. In such cases, lunr could load a pre-created index from the disk.

  export function saveSearchIndex(index: lunr.Index) {
    try {
      fs.writeFileSync(INDEX_PATH, JSON.stringify(index, null, 2), {
        encoding: "utf-8",
      });
    } catch (err) {
      console.log(err);
    }
  }

loadSearchIndex: This function loads an index to the memory so that lunr could perform operatons on it.

  export function loadSearchIndex(): lunr.Index {
    try {
      const content = fs.readFileSync(INDEX_PATH, {
        encoding: `utf-8`,
      });

      return lunr.Index.load(JSON.parse(content));
    } catch (err) {
      console.log(err);
      throw err;
    }
  }

deleteSearchIndex and hasSearchIndex: These functions are used to delete an existing search index from disk, and to check whether a given index exists.

  export function deleteSearchIndexFile() {
    return fs.unlinkSync(INDEX_PATH);
  }

  export function hasSearchIndex() {
    return fs.existsSync(INDEX_PATH);
  }

Now that we have all the helper functions ready, we could start implementing the feature. Inside the API endpoint file, we are going to initialize the lunr index.

A point worth noting, we have to update the index after a certain period. Otherwise, the index will not have all the data from the database.

let searchIndex: lunr.Index;
let indexBuiltAt: Date;
const TEN_MIN_IN_MILI = 600000;

In the above code snippet, I declared a few variables. Variable indexBuiltAt stores the most recent build timestamp. Based on this timestamp, I will be updating the index.

function createSearchIndex() {
  buildSearchIndex()
    .then((index) => {
      searchIndex = index;
      saveSearchIndex(index);
      indexBuiltAt = new Date();
    })
    .catch(console.log);
}

The above function creates a search index and stores them in the variables declared earlier.

Finally, it's time to glue everything together and make it a working solution.

Following code-block is pretty much explains itself. I used setImmediate so that it does not block the main event loop.

setImmediate(() => {
  if (hasSearchIndex()) {
    searchIndex = loadSearchIndex();
  } else {
    createSearchIndex();
  }

  setInterval(() => {
    // reload search index at every 10 mins
    if (
      indexBuiltAt &&
      indexBuiltAt?.getTime() + TEN_MIN_IN_MILI < new Date().getTime()
    ) {
      if (hasSearchIndex()) {
        searchIndex = loadSearchIndex();
      } else {
        createSearchIndex();
      }
    }
  }, 30 * 1000);
});

At this point, everything is done. And we are ready to run queries on this index. To run a query using lunr, we have to call the search method.

const ids = [];
const result = searchIndex.search(`*${search.split(` `).join(`*`)}*`);

for (let i = 0; i < result.length; i++) {
  const doc = result[i];
  mongoose.isValidObjectId(doc.ref) && ids.push(doc.ref);
}

I am collecting all the matching ids into an Array. Using these ids, I will retrieve the actual documents, and send them as the API response.

Conclusion

This set-up is ideal if your product is relatively small (and do not have a huge amount of data to run the operations upon). I have used the same setup in one of the projects I built.This can be improved a lot. For instance, you could build the search index every time there is a new entry in the database.

For more information on lunr, please check the official website. It has many other useful things built-in.

Why TypeScript

Abdus Azad — Mon, 26 Jul 2021 15:09:12 +0000

For many of the developers, TypeScript seems like a completely new language. But to me, it is just JavaScript with added type system. In this post, I am going to discuss why we should use TypeScript over plain old JavaScript.

Type System

Let me lead the post with TypeScript’s awesome typing-system. In plain JavaScript, there are no types. Values are assigned to variables dynamically. When I say a language is dynamic, I essentially mean that we can re-assign variables with different data types. For example, if we a declare variable let whatever = 10, this variable will be of type Integer. Now in a strongly-typed static language, we cannot reassign this variable with a different data type, say a String.

If a programmer has not used any statically-typed languages in the past, this additional type-system on top of JavaScript can be a bit hectic and frustrating while working on a project. But if we look at the bigger picture, we can prevent so many bugs right at the compile time by passing values of the correct type.

For example, if a function parameter type is set as String, and we pass a value of type Number while invoking it, TypeScript compiler will not compile the code. On the other hand, JavaScript will report no error at all.

Self Documenting

Since each variable in TypeScript has a type associated with it, it’s very easy to read and understand that codebase. Check out the below code for example:

interface DbOpts {
  host: string;
  port: number;
  username: string;
  password: string;
  database: string;
}
function connectDB(opts: DbOpts) {
  return db.connect(opts);
}

In the above code snippet, if we call connectDB() function without the correct parameter (or even a single property, as all of the properties, arerequired), the TypeScript compiler will throw an error saying that there’s a type mismatch. Isn’t that useful?

Of course, we can use JSDoc, but to be honest, JSDoc is not reliable when it comes to type-checking. The reason is, JSDoc is a documentation tool, not a compiler or type checker.

Object-Oriented Programming

TypeScript fully supports object oriented programming. Class field modifiers are a recent addition to JavaScript. Thanks to V8 engine, we can use it in Node without any polyfill. TypeScript had these features for a long time. It supports class field modifiers such as private, public and protected.

Autocompletion

Almost all of the popular IDE and text editors have great support for TypeScript. With TypeScript, you get better support for auto-completion of code. And most of the time, editors would highlight any type of errors before you even run the program.

Again, JSDoc can be a replacement for this. And it is what we used when TypeScript was not so popular. But both of these tools have their own tasks to accomplish.

Transpile Code

TypeScript can compile to many versions of ECMAScript. You can use all the latest features of JavaScript happily and still be stress-free about supporting the older environments.

Point to be noted here: TypeScript does not polyfill anything. It converts code from one version of JavaScript to another. For example, if your codebase has some code snippets that the target version does not support, TypeScript will convert the code so that it is compitable with the older version of ECMAScript.

const isTrue: Boolean | undefined = undefined;
const hello = isTrue ?? "not set to a boolean value";

If we choose ES3 as target, above snippet will be compiled to:

"use strict";
var isTrue = undefined;
var hello =
isTrue !== null && isTrue !== void 0 ? isTrue : "not set to a boolean value";

Conclusions

There are certainly some drawbacks of using TypeScript over JavaScript. But those drawbacks are so negligible that they don’t even count.

TypeScript produces a maintainable codebase. Maybe it won’t matter when the codebase is small. But in the long run, we get a codebase that could be maintained by People without losing their sanity. If someone in the team has never used a statically-typed language before, they will need some training.

Session Management in Vim

Abdus Azad — Mon, 26 Oct 2020 10:52:18 +0000

If you ask me about the most underrated feature in Vim, my answer would be
'Session Management' straightaway! Just like any other "modern" editors, Vim
does support sessions.

What is a Session

Before I go in-depth, let me explain what exactly is a Session in Vim. Simply
put(from :help session): A Session keeps the Views for all windows, plus the global settings.

An example of a session could be: If I have opened a project and worked for an
hour, and then closed it. My Vim session would be one hour long.

Sessions in Vim are not persistent by default. One has to make it persistent.
Most people do not know about it because they probably have never felt the
need for it.

I was one of those people who didn't feel the need for storing sessions, until
recently. And I am not ashamed of it. It just happens... mostly because, we
don't need it yet.

Importance of Session

To answer the question 'How a persistent Session is Useful', let's find out what
we lack without it.

Time. Yes! Imagine, you are working on a project, and for some reason, you closed Vim. So when you open it again, you gotta recreate those perfectly sized windows, open files or config options etc.
Context. It's not easy to remember which file you were editing, the last time you have worked on a project.

Now, what if I say that you could boot into Vim without losing any of the
previous windows, tabs etc? not even the cursor location. That would be
great, is not it?

That's where persistent session comes into action. It enables Vim to load
the last session, much like what we see in editors like Visual Studio Code.

Storing Sessions

There are multiple techniques one may use to handle Sessions. I will, however,
talk about the one I use.

First, I let Vim look for any existing session files. If one exists for the
current session, Vim would update it automatically when I close Vim. The Reason
I don't let Vim save Sessions for which no session file exists, because that
makes me counter-productive. For example, I needed to edit a config file, say,
.vimrc from my $HOME directory. Had I enabled auto-create session file, a
new session file for $HOME directory will be created, which I obviously don't
want. That session does not make any sense.

So, if a session file is not created automatically, how do I create it then?
Well, it's really simple. I defined a command for creating new Sessions, called
SessCreate using Vim Language.
Now, I can simply call this command from inside Vim whenever I need it.

Let me show you how I did it.

Snippet

fu! SessionCreate()
  if !isdirectory(expand("~/.vim/sessions"))
    execute "call mkdir(expand('~/.vim/sessions', 'p'))"
  endif
  execute 'mksession ~/.vim/sessions/' . split(getcwd(), '/')[-1] . '.vim'
endfunction

command SessCreate call SessionCreate()

Explanation:

if !isdirectory(expand("~/.vim/sessions"))
  execute "call mkdir(expand('~/.vim/sessions', 'p'))"
endif

Check if a directory $HOME/.vim.sessions exist. If not, create it with
execute "call mkdir(expand('~/.vim/sessions', 'p'))". This is the directory
where sessions are stored. An alternate location for storing session is the
project directory itself. But that is not something I want to use, as not
everyone uses Vim. So, this would be an added bloat for them.

call in Vim Script is used to call a function. mkdir() is used to create a
new directory. p is a flag that can be passed to mkdir() so that it creates
parent directories as and when necessary. Lastly, expand() is used to expand
any Shell expandables. Here, it converts ~ to /home/username.

execute 'mksession ~/.vim/sessions/' . split(getcwd(), '/')[-1] . '.vim'

mksession is a built-in command for creating a session from the current Vim state.
It saves the created session in ~/.vim/sessions/DIR_NAME.vim. Here, the
DIR_NAME is the working directory where Vim is opened. So, session for
~/Projects/github/awesome-vim would be saved into
~/.vim/sessions/awesome-vim.vim.

Saving a Session in Existing Session File

Now, it's time for some automation. Check out the following code snippet:

fu! SessionSave()
  if filewritable(expand('~/.vim/sessions/' . split(getcwd(), '/')[-1] . '.vim'))
    execute 'mksession! ~/.vim/sessions/' . split(getcwd(), '/')[-1] . '.vim'
  endif
endfunction

This function checks whether the existing session file is writable. And save
the latest session if true. That's it. The exclamation mark(!) is for overwriting
the previous session.

Restoring a Session

Finally, when I open Vim again, I want saved buffer(if any) to be restored.
Following function is responsible for this:

fu! SessionRestore()
  let l:session_file = '~/.vim/sessions/' . split(getcwd(), '/')[-1] . '.vim'
  if filereadable(expand(session_file))
    echo session_file
    execute 'source ~/.vim/sessions/' .  split(getcwd(), '/')[-1] . '.vim'

    if bufexists(1)
      for l in range(1, bufnr('$'))
        if bufwinnr(l) == -1
          exec 'sbuffer ' . l
        endif
      endfor
    endif
  endif
endfunction

Let me explain what it does exactly.

let l:session_file = '~/.vim/sessions/' . split(getcwd(), '/')[-1] . '.vim'
if filereadable(expand(session_file))
  echo session_file
  execute 'source ~/.vim/sessions/' .  split(getcwd(), '/')[-1] . '.vim'
endif

First, check if the file is readable. There's no point in sourcing it if Vim
can't access it. Then simply source it to Vim. That's it.

if bufexists(1)
  for l in range(1, bufnr('$'))
    if bufwinnr(l) == -1
      exec 'sbuffer ' . l
    endif
  endfor
endif

bufexists() function checks if any
buffer exists. bufnr() returns all
available buffers. ...and so on. Read more about them using :help FUNC_NAME
in Vim.

So, that's it. Persistent Sessions makes our life dramatically easier when using
Vim (or NeoVim, GVim whatever flavour you use).

P.S.: I am a new Vim user. I am open to suggestions. Feel free to suggest
anything that would make use Vim in a more productive way!!

Fiber: a Super Minimal Hugo Blog Theme

Abdus Azad — Sun, 30 Aug 2020 02:03:21 +0000

Hello DEV community! I am here to brag about my recently launched Hugo theme!

In short, Fiber is a Hugo theme for the Minimalists. It features a Clean and Simple UI™. It also has a 'Notes' section so that you won't have to ditch nicely written notes from your personal website. It's your choice anyway.

Features

Simple and Clean UI™
Shortcodes - Currently, it has two custom shortcodes. Both embed ASCIINema Casts and CanIUse data respectively into the page. (Of course, I plan to add more)
Code Highlighting
Click-to-copy Codes (maintains indentations)
Notes - Uses rough-notations to spice things up
Most importantly; this theme just works, without throwing any errors!!!

Screenshots

Check out some screenshots! Or you can check this theme in action here

(ignore those red lines at the bottom. they are coming from window manager)

How to Use

Since you made it up to here, I would assume you are interested in trying this theme out! Installation and Usage are documented in Readme.

It would be great if you Star this project on Github. This step is completely optional though ;)

Have a good day!

Using Github Profile Readme as 'Twitter-like' Feed

Abdus Azad — Thu, 20 Aug 2020 01:16:55 +0000

I had this weird idea of using GitHub profile README as Twitter-like(well,
Mastodon TBH) feed! There was no specific reason for doing so. I roughly
thought it would boost my Mastodon profile(which is quite new at the moment)
reach.

Another reason was to see how far I could go with limited tools (as Github
ReadMe does not support JavaScript yet).

Generating those Cards

There was only one thing to figure out. How to create beautiful status cards
with limited CSS! I knew I could use SVG and make whatever I want. But not
all browser supports every feature of SVG yet. So, generating and
storing SVG codes as images may not always work.

Since GitHub has introduced workflows(better be late than never), I chose to
render SVG on a webpage and convert them to PNG, all in Server-side.

This way, I am assured that SVG would be rendered in the latest browser and
converted PNG. But how do I do it?

There is a package called Puppeteer in the NPM registry. This library gives
the user a way to control Chromium/Firefox browser programmatically. This,
by default, runs browser in headless mode. So, the user would be able to run it
without a Desktop Environment.

Getting 'Toots' from Mastodon

If you don't know about Mastodon, you can think of it as Open-Source Twitter!
Read more about Mastodon.

Mastodon have an API to fetch public toots of any profile without using an
API key. The response is in JSON, so parsing it was quite simple.

Biggest surprise: If I add a Middot (·) somewhere inside SVG, loading image
won't work! It throws an encoding error.

Installing Font

After pushing the initial version to GitHub, I noticed that the font I was
using locally was not available in Github Workflows environment. So, I had to
install them manually. Although I could simply add a font CDN using style
tag within SVG, why not do it the Linux' way? :P

Installing font through command-line as just three simple steps.

download font as a zip archive
extract font files to ~/.fonts using unzip
run fc-cache -f -v to manually update the font cache

Writing Workflow Steps

I set-up the workflow to run in two events. First is, when I push something to
the repository. And second is, on every hour (using a Cronjob). This way, I won't
have to worry about updating ReadMe manually when I post a new toot.

Setting up Node and NPM was done by an action called setup-node.
Installing required font was done by a custom bash script I wrote.
Finally, the image creation script was run using Node, commit the changes, and push them back to GitHub.

And the work is done here!

Conslusion

Sure, this thing was just for fun. But I learnt quite a few new things.

I can't use Middot(or similar characters) within an SVG tag. Now I know what to look for when there's an encoding issue.
I understand how action/checkout works. This will help me write better workflows in future.
I learnt about some SVG tags. For example, switch and foreignObject.

If you liked this, I would be happy if you star 🌟 this repository.
You can follow me on Mastodon. My handle is @abdus@linuxrocks.online.

Terminating DOM Operations at will: 'AbortController' in JavaScript

Abdus Azad — Sun, 26 Jul 2020 16:21:49 +0000

AbortController is an interface which provides a way for terminating one or
more web request as and when desired.

This generally means that a request can be terminated by a user whenever needed,
irrespective of whether the operation is finished or not.
AbortSignal can be implemented in any web platform API which uses
Promise.

The API

AbortController provides a few things for users to implement it effectively in
code. At the time of writing this, the constructor would return an instance which
contains a method AbortController.abort() and a property
AbortController.signal(read-only).

AbortController.signal - Returns a AbortSignal instance
AbortController.abort() - Aborts a DOM Request

AbortSignal

AbortController.signal returns an instance of type AbortSignal which
represents the current state of the AbortController instance.
It has a read-only property AbortSignal.aborted. Type of property aborted
is Boolean.

AbortSignal is also responsible for communicating with DOM as an Event Listener
can be attached to it.

Using AbortController

To use AbortController in a Promise, one must adhere to a few rules.
Source

Function should accept AbortSignal through a signal property. For example:

  function abortThis(arg1, { signal }) {
    return new Promise((resolve, reject) => {
      // function body
    })
  }

When method aborted() is called, reject the Promise with a DOMException AbortError

  throw new DOMException('aborted by user', 'ABORT_ERR');

Abort immediately if the aborted flag on AbortSignal is set to true.

Browser Support

Despite being a relatively new API, browser support for AbortController is
quite awesome! All major browser fully supports it. Following is a chart
from Can I Use.

<div
class="ciu_embed"
data-feature="abortcontroller"
data-periods="future_1,current,past_1,past_2"
data-accessible-colours="false"

Data on support for the abortcontroller feature across the major
browsers

Examples

Following are a few examples of AbortController API. First one is using fetch
API. On the second example, I will try to implement it in a Promise.

Using Fetch API

const controller = new AbortController();
const signal = controller.signal;

// fetch a URL
fetch('https://jsonplaceholder.typicode.com/todos/1', { signal })
  .then(raw => raw.ok && raw.json())
  .then(console.log)
  .catch(e => console.log(e.message));

// driver code
setTimeout(() => {
  // abort network request if it takes
  // more than a second
  controller.abort();
}, 500);

The above code allows a network request to run for 500ms. If data has been
fetched within that period, it would return JSON representation of the response.
Otherwise, it would abort the request.

Implementing using Promise

In this section, I will try to implement AbortController for a function which
returns Promise. Should be quite easy and straightforward.

/* this function would return a Promise
 * which would resolve with value `hello`
 * after 4s
 */
function abortTask({ signal }) {
  return new Promise((resolve, reject) => {
    // 1. check if it's already aborted
    if (signal && signal.aborted) {
      return reject(new DOMException('`signal` is in `aborted` state', 'ABORT_ERR'));
    }

    // wait for 4s
    setTimeout(() => resolve("hello"), 4000);

    // 2. add a listener to `signal` to check its state
    signal && signal.addEventListener('abort', () => {
      // reject promise when signal.aborted changes to `true`
      return reject(new DOMException('aborted by user', 'ABORT_ERR'));
    })
  })
}



/* DRIVER CODE */
let signal;   // just so that I could re-use it


// when `signal.aborted` is `false`
const abortController_1 = new AbortController();
signal = abortController_1.signal;

abortTask({ signal })
  .then(t => console.log(t)) // hello
  .catch(e => console.error(e.message))


// when `signal.aborted` is `true`
const abortController_2 = new AbortController();
signal = abortController_2.signal;

abortController_2.abort();

abortTask({ signal })
  .then(t => console.log(t))
  .catch(e => console.error(e.message)) // err


// when user calls AbortController.abort()
const abortController_3 = new AbortController();
signal = abortController_3.signal;

// abort task in 2s
setTimeout(() => abortController_3.abort(), 2000);

abortTask({ signal })
  .then(t => console.log(t))
  .catch(e => console.error(e.message)) // err

Get Your Own Heroku - a Guide on How to Run a Custom 'Platform as a Service' for deploying Apps

Abdus Azad — Sun, 26 Apr 2020 07:02:06 +0000

Heroku is a fantastic platform for Node.js developers. It made deployment of Node.js apps super easier. But obviously. it have so many limitations on free-tier, which makes it less ideal for deploying certain kind of projects, like a telegram bot etc.

The Idea

I was having a pretty similar issue. I needed to host a bot. Initially, it was in Heroku, but due to the fact that every apps on Heroku free tier goes for sleep after inactivity of around 30 minutes. This, generally, is not a problem for websites, but bots won't work. There is always a delay on bot response.

First, I thought I would build my own Heroku-like platform. But it's not really easy. So, I went off with an open-source 'Platform as a Service' software.

Likely, there are many such open-sourced products available. One of them is CapRover.

Features

Docker-based deployment. Each of the deployed app would have their own isolated space.
Support for Dockerfile. A Dockerfile would maximize the possibilities of what you could do with the deployments.

Prerequisites

Nothing much required, really! All you need is:

A VPS to host CapRover. Preferably DigitalOcean It should also have a Linux OS installed. Ubuntu is preferred.
Basic Linux Shell Skills for installing required applications.
A Domain Name

Initial Server Setup

If you are using Digital Ocean, first you need to create a droplet. Digital Ocean have a nice guide on how-to setup a droplet.

Once you are done creating a Droplet, it's time to update the system. To do that, you need to access your droplet using SSH. This tutorial would help you do that.

After connecting to droplet via SSH, update and upgrade the system using sudo apt update && sudo apt upgrade. Note that, these commands are for Ubuntu.

We also need a couple of other software in order to get our CapRover server up and running. We will be installing them one by one.

Docker

CapRover is built on top of Docker. So, it's an essential essential software. To install docker, checkout the official Docker documentation for Ubuntu.

You may need to add the current user to docker group. To do so, run sudo usermod -aG docker <user_name>. Otherwise, you will get a docker error.

Node.js and NPM

Yeah, these Node thing is everywhere these days :/ But anyways, it should be relatively easier to install. Note that, by default Ubuntu repository have node v8.x.x. We need at least v10.x.x. So to install node v13.x.x via NodeSource:

curl -sL https://deb.nodesource.com/setup_13.x | sudo -E bash -

Then run apt-get install command.

sudo apt-get install -y nodejs

To confirm that you have Node installed, run node -v from terminal.

NPM

It's possible that NPM was installed along with Node.js. You can verify that using npm -v. In case it is not installed, install it from Ubuntu repository using sudo apt install npm.

By default, NPM needs sudo to install global packages, which is not recommended. To change this behavior, change prefix in NPM config. Run npm config set prefix dir_name(replace dir_name with a directory name where you want to have npm packages). You also, need to add dir_name/bin to your PATH variable.

Domain Setup

By now, you should have a static IP address assigned to your Digital Ocean droplet. If you have a domain name purchased, you need to point it to your server IP address using A record. For example: you want captain.yourdomain.com as your app host. You need to setup two different a records.

cap              A         3.23.76.12
*.captain        A         3.23.76.12

At this point, your server is ready to host CapRover. You may additionally need to allow some ports in order to make CapRover work. You can allow them using ufw command in Ubuntu.

ufw allow 80,443,3000,996,7946,4789,2377/tcp; ufw allow 7946,4789,2377/udp;

Note that, this command must run in root shell.

Install CapRover and CapRover CLI

Thanks to the devloper of CapRover, it's super easy to install. All we need is to run one command, sit back and watch it installing.

docker run -p 80:80 -p 443:443 -p 3000:3000 -v /var/run/docker.sock:/var/run/docker.sock -v /captain:/captain caprover/caprover

Once the CapRover installation is done, visit cap.domainname.com:3000 to verify if CapRover is working. We also need to install CapRover CLI in order to manage deployment, sertup server etc. To install CapRover from NPM, run npm i -g caprover. Once installed, run caprover serversetup, and follow the interactive command prompt.

At this point, you have a CapRover instance running successfully. Check this guide for various deployment methods.

Before deploying a project, make sure to create an app using the Web GUI. Otherwise, you won't be able to deploy.

For more information about CapRover and it's configurations, check out its official docs.

Conclusions

Having a self-managed PaaS could be very handy. Following are a few reasons I choose to go with a self-managed PaaS for hosting apps.

I would be saving money. Generally, if I go for Heroku non-free tier, I would be paying a lot of money compared to what I pay now.
Shared Resources. Server resources won't get wasted because each and every app would be able to access resources from the same resource pool. So, let's say, I have 2 GB of RAM. And I need only 1 GB to host XYZ webapp. Now, I can use the remaining amount of RAM to spin-up a new instance ABC webapp.
Support for [insert techstack name] Since, CapRover uses docker to deploy a new instance, I can basically run apps built using any back-end technologies. And write how to deploy it using a Dockerfile. I have so much flexibility now.

The only downside I could think of is, I have to look after the server. This is won't be a big deal, thanks to AWS ❤️

Securely sending Emails from static sites with a 3rd party API

Abdus Azad — Thu, 07 Nov 2019 18:30:00 +0000

This are the exact same steps I am following to make this form work. If you find any flaws, I should be the first one to know :)

I have been looking for a way to send emails and/or other data from a static site using a third party mailing API. There are a few awesome services for sending mails but none of them are ‘secure’. By secure, I mean to say that whatever a person is sending to me through a mailing API is delivered as plain text. That means, anyone having access to the API server could see any mail sent to me via their API. And I don’t want to setup my own back-end just to send emails, either.

The solution to this problem is pretty simple! Encrypt all data before you are sending using a public key which belongs to you. And whenever it is delivered to you, decrypt with your public key.

Explanation

Okay. Here you may have a question, what the hell is a key? To answer that, a key is an unique id using which a person could encrypt a message and decrypt it on demand later on.

So, what is the benefit? Well, a lot of benefits! First, no other person than you could ever look into it(unless you share your private key). It won’t let anyone modify the message which is being sent to you in midway.

To encrypt/cipher the plain text message, we will be using Public Key Encryption. So, before moving forward, let’s learn about a few related terms related to cryptography which I will be using throughout the post.

Private Key : It is a cryptographic key used for decryption. By nature, it is a secret key and should only be with you. Once you think you lost the key or it get compromised, time to get a new one. And invalidate previous public and private key.
Public Key : It comes with your private key, as a pair. This key is used to do encryption so that the private key could decrypt. This key can freely be shared with anyone. A message which is encrypted with a public key must be decrypted using the private key of that pair. I have my public key here. Send encrypted message haha..
Fingerprint : A fingerprint is a short version(consists bytes) of a longer public key. It is used to simplify key management in certain apps. It’s generated by applying a cryptographic hash function in public key. My fingerprint is FA20 8736 8166 715F E3C1 60C6 F2AE 28CC C8CA FB9A

Implementation

Steps to implement:

We will generate a key pair for us. So that, we can encrypt/decrypt a message.
Encrypt message using our own public key.
Send message from our static site using a mail delivery API.
Once the API delivers the message to us, decrypt using private key.
That should be enough! Let’s see how it goes.

Generate a Key Pair

To handle encryption, we would be using OpenPGP. OpenPGP is an encryption standard derived from PGP. GnuPG(gpg) is a software which implements OpenPGP. Generally GnuPG comes pre-installed. If you don’t have it installed, download it from here.

Pretty Good Privacy (PGP) is an encryption program that provides cryptographic privacy and authentication for data communication. PGP is used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications. Phil Zimmermann developed PGP in 1991. - wikipedia

Generate a new Key from command line: Type gpg --full-gen-key in your terminal, hit enter and follow the on-screen instruction.
To find out fingerprint of your key, run gpg --fingerprint.
To export public key(you have to), run gpg --armor --export FINGERPRINT > public-key.txt. This will generate an ASCII armored public key and save it in public-key.txt.

Encrypt Data

Now that we have our key, we can go ahead and geneate encrypted text using the public key. To encryt/decrypt, we would be using a library called openPGP.js(developed by Signal). You can simply add this library to a project using CDN.

Let’s start with writing some codes.

Create a form in HTML

Copy following HTML inside index.html(create file in project root)

<html>
  <body>
    <form action="">
      <input type="email" placeholder="mail@abdus.xyz" /> <br />
      <input type="name" placeholder="Abdus" /> <br />
      <textarea
        name="message"
        cols="30"
        rows="10"
        placeholder="Your awesome message"
      ></textarea
      ><br />
      <input type="submit" value="Encrypt and Send" />
    </form>
    <script
      src="https://cdnjs.cloudflare.com/ajax/libs/openpgp/4.6.2/openpgp.min.js"
      integrity="sha256-txcrHjb7Yt1zrebRnGkk5OwDkoBrvJTjvJx5b7qBtWc="
      crossorigin="anonymous"
    ></script>
    <script src="assets/js/main.js"></script>
  </body>
</html>

Also copy following JavaScript inside assets/js/main.js(create in project root)

const PUBLIC_KEY = `PUT YOUR PUBLIC KEY HERE(copy it from public-key.txt we generated earlier)`;

async function encrypt(message) {
  const enc_message = await openpgp.encrypt({
    message: openpgp.message.fromText(message),
    armor: true, // export as ascii armored text
    publicKeys: (await openpgp.key.readArmored(PUBLIC_KEY)).keys,
  });

  sendMail(enc_message.data); // send the mail
}

document.querySelector('form').addEventListener('submit', e => {
  // dom manipulation
  const sender = {
    name: document.querySelector('#name').value,
    email: document.querySelector('#email').value,
    message: document.querySelector('#message').value,
  };

  // prevent page from reload on form submit
  e.preventDefault();

  // finally call our encrypt function
  encrypt(`${sender.name} <${sender.email}> \n\n\n ${sender.message}`);
});

function sendMail(encrypted_message) {
  // here, we will make a POST request to formspree
  // so that formspree could deliver our data
  fetch('https://formspree.io/YOUR_FORMSPREE_ID', {
    method: 'POST',
    mode: 'no-cors',
    body: JSON.stringify({
      message: encrypted_message,
    }),
  })
    .then(console.log)
    .catch(console.log);
}

Above JavaScript is pretty much self-explanatory. You shouldn’t have much problem with it. If you face any issue, let me know using this form.

Also, we need a Formspree account. This is an API to deliver our form data/email. It’s easy to create an account in Formspree. You can do that.

Decrypt a message

Once you recieve the message through Formspree, it’s time to decrypt and see what is inside it. To do that, we would be using command line gpg client. Before decrypting, save your encrypted text in a file and name it something.pgp. Then finally, run gpg --decrypt something.pgp(assuming something.pgp is in current working directory). The output with be piped to standard output, i.e. (mostly) terminal/cmd window. To save it in a file, add this flag to the command: --output something.txt.

If you prefer a GUI application, you’ll find many open-source applications for free. Just make sure it implements the OpenPGP standards.

That’s it! By this time, you should have an encrypted email which is delivered securely with the help of a third-party website from your static site.

Footnote

This encrypting process does not sign the document.
It’s always better to use your own server.
Never disclose your private key. Never.

How to Load a Website Faster

Abdus Azad — Mon, 28 Jan 2019 18:30:00 +0000

Page Load-time is an important factor for any kind of websites. The lower the load-time, higher the site traffic and low bounce rate.

If you don’t know what a page load time is, it’s the time taken by a browser to render a webpage. This process includes: making a request to the server, downloading the resources, processing DOM elements and rendering.

To decrease a website’s page load time, you have to follow certain things while building/creating a website.

Place scripts just before the closing body tag

No matter what a script does, it should always be placed at end of <body>. i.e. the closing </body> tag. Scripts are generally huge in size and costs browser some extra time to load it. Moreover, you don’t want to execute any scripts before the DOM even loads, right?

Minify

Yes, minify stylesheets and scripts. minify is a process of removing whitespace and unwanted characters from your code. It will help you reduce the size of files so that browser could download it faster.

Don’t use too many redirects

Whenever we redirect a link, it takes an extra request to the server. For example, if you are loading some resources from example.com which redirects to example-2.com. You are forcing the client(browser) to make an extra request. So, minimize the use of redirected links.

Use CDN

CDN or Content Delivery Network help you reduce the server load and serve pages based on user’s location. For example, if you are somewhere in the US and using a CDN, the website would be served from the nearest server available and not from the original server. This method helps you cut-down the distance travelled by resources.

Code Splitting

This might look something new to you. This helps you load only the code you require to render the very first instance of any website. So, if you have a big CSS file which contains many selectors which don’t need during initial render, eliminate them or create a new CSS file and load them on a later stage.

async and defer for scripts

if you have some scripts which you don’t need immediately, just defer them. async and defer tells the browser to load these scripts on a later stage. Example,

<script async src="example.com"></script>
<script defer src="example.com"></script>

A light colored Dev badge

Abdus Azad — Thu, 19 Apr 2018 12:28:01 +0000

I need a light colored badge to render on my website because my website's having a dark background. So, the logo doesn't look as expexted on my website. And I think everyone with a dark themed website facing this issue.

So, It would be great for us if Dev.to provide a light colored badge to showcase on our website.