DEV Community: Div Rhino

Create a passkey-authenticated app with Expressjs, Sequelize and Docker

Div Rhino — Fri, 05 Jan 2024 14:51:01 +0000

In this tutorial, we will learn more about the Web Authentication API (a.k.a. Webauthn) by building a simple app that uses Passkeys for authentication. Webauthn allows servers to register and authenticate users through the use of public key cryptography instead of a password. We will be using Express.js for this demo.

Caveat

At the time this article was first published, Webauthn is still an emerging technology. So while it is already supported in most major browsers, it only has partial support in Firefox because it does not yet cater to TouchID.

Scope

To keep the scope of the content manageable, we will only be focusing on the happy path of the implementation.

We will not be covering:

security
error handling
form data validation, or
any other authentication edge cases

Prerequisites

We will be using Docker in this tutorial, so to follow along, you will need to have Docker installed and running. You can head to their download page to find the version that suits your environment.

Setting up a new project

In the terminal, we can change into the directory where our projects are stored. In my case this would be the Sites folder, it may be different for you. We will create a new directory for our new project, and immediately change into it

mkdir divrhino-passkeys-express
cd divrhino-passkeys-express

We will be using Docker for local development, so let’s create our necessary config files

touch Dockerfile
touch docker-compose.yml
touch .dockerignore

If you’re new to using Docker and are wondering why we need both a Dockerfile and a docker-compose.yml file, you can read more about the differences between these two files in this article: dockerfile-vs-docker-compose"

Now let’s open up our Dockerfile and add Node.js to our instructions. We will be installing Node.js in a container, so we do not need to have it install on our machine. On the next line, we will specify the path to our working directory. That’s enough to get us started for now.

FROM node:19
WORKDIR /usr/src/app

Docker containers run on Linux, so here we’re saying that we want our app to live in the /usr/src directory of the Linux filesystem in a project folder we want to be named app"

Opening up our docker-compose.yml file, we can add config for our first service called web, which will run on port 3000.

version: "3.8"

services:
  web:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app

We do not have a node_modules directory just yet, but let’s tell our .dockerignore file to ignore it along with any npm debugging logs

node_modules
npm-debug.log

And with that, we can head into our terminal and spin things up with following command. At this point, it will only install Node.js in our Docker container.

docker compose up

Once the container is created and Node.js is installed, successfully, we can access the shell inside our Docker container by running the following command

docker compose run --service-ports web bash

Just as a sanity check, let’s see if the correct version of Node.js has been installed in our container. If you get an output similar to v19.9.0, then all is right with the world.

node -v

While still in the container, let’s initialise this project to use npm to handle our dependencies. We use the --yes flag to skip the setup wizard and give us a basic package.json file

npm init --yes

We will be using Express.js to give us web server functionality. So we can install that as a dependency

npm install express --save

Then we can create an index.js file to act as an entry point to our application

touch index.js

We can head over to the Express.js documentation and copy over their hello world example into our index.js file. Then we will make some minor changes to it

const express = require("express");
const app = express();
const port = process.env.PORT || 3000;
const host = "0.0.0.0";

app.get("/", (req, res) => {
  res.send("Hello World!");
});

app.listen(port, host, () => {
  console.log(`Example app listening on http://${host}:${port}`);
});

While still in the container, we can now spin up our barebones app by running the following command

node index.js

And if we visit http://localhost:3000/, we will see the Hello World! message appear in our browser

Our app is running, but if we make any changes to our index.js file and say “Hello” to the Universe instead our app does not, automatically, reflect these changes in the browser. We will have to kill our server and restart it to see the changes.

Restarting the server, manually, every time we make a change is not ideal. We can work around this by installing a package called nodemon as a dev dependency. The nodemon package will help us rebuild our app whenever a change is detected.

While our app is running, we can open up a new terminal tab and execute the following command

docker compose exec web npm install nodemon --save-dev

And now when we want to restart our server, we can use nodemon to do so. Because we installed nodemon, locally, in our application, it will not be available in our system path, so we will need to prefix the command with npx

npx nodemon index.js

npx allows developers to execute any packages available in the NPM registry without needing to install them."

Now what if someone clones our repository without the node_modules folder. Will our app still work as expected? Let’s delete our node_modules folder and find out.

If we enter our container’s shell

docker compose run --service-ports web bash

And try to run our app

npx nodemon index.js

We will see a message that says nodemon is not installed in our project.

Need to install the following packages:
  nodemon@3.0.1
Ok to proceed? (y)

We could very well proceed to install it. But maybe it would make more sense if it was installed whenever our Docker image was built. We can make changes to our Docker config to introduce this behaviour.

In our Dockerfile, we can add a line that will copy the package.json and package-lock.json files from our host machine into the container. Then we can also add instructions to run the npm install command

FROM node:19
WORKDIR /usr/src/app

COPY package*.json /usr/src/app/
RUN npm install

Then in our docker-compose.yml file, we can add a volume to persist our node_modules data and also map a start-up command to docker compose up

version: "3.8"

services:
  web:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
      - node_modules:/usr/src/app/node_modules
    command: npx nodemon index.js
volumes:
  node_modules:

Now we can exit our container to rebuild our image and fire up our app by running

docker compose build
docker compose up

If we refresh our browser, our app should work the same.

Now let’s add the final touches to finalise the Dockerfile. We want to add instructions that will:

copy all the contents into out working directory,
expose port 3000, and
add a CMD to run our app

We’ve added a few more steps and now our Dockerfile should look like this

FROM node:19
WORKDIR /usr/src/app

EXPOSE 3000

CMD [ "npx", "nodemon", "index.js" ]

COPY package*.json /usr/src/app/
RUN npm install

COPY . /usr/src/app/

These changes will not change anything visually in the browser, but it will make it easier for our teammates to spin up the app in one command.

Initialise Sequelize

Now that we’re done with the Docker configuration, let’s keep going. Since we are implementing a login feature, we need Users and some way to store any data related to them.

In this section of the tutorial, we will learn how we can use Sequelize to represent a User model. We will also set up a Postgres database to persist this data. Sequelize is a widely-used ORM for the JavaScript/Nodejs ecosystem. We will use it to interact with our database instead of writing any raw SQL.

With our app running in another tab, we can install the sequelize package as a dependency

docker compose exec web npm install sequelize --save

We will also install the sequelize-cli as a dev dependency to help us manage migrations and project bootstrapping

docker compose exec web npm install sequelize-cli --save-dev

Now we can use the Sequelize CLI to initialise some necessary config and directories

docker compose exec web npx sequelize-cli init

This command will create the following files and folders in our project root

config/config.json
/usr/src/app/models
/usr/src/app/migrations
/usr/src/app/seeders

It might be a good time for us to start thinking about organising the structure of our project, so let’s shuffle some of these files around.

First let’s make a couple of new directories to hold our app code and our db code. To do this, we can run the following command in our terminal

mkdir {app,db}

Then we need to move some of the Sequelize files into these newly created directories. Let’s move the /models folder into the app directory

mv models ./app/models

Then let’s move the /migrations and /seeders folders into the /db directory

mv migrations ./db/migrations
mv seeders ./db/seeders

We will also rename our config and turn it into a JavaScript file, instead of a JSON file. We do this so that we can use environment variables within in

mv config/config.json config/database.js

We should also fix the line in our app/models/index.js file where we import our config. Change it from this

const config = require(__dirname + "/../config/config.json")[env];

To this

const config = require("../../config/database")[env];

We’ve made quite a few changes, but Sequelize expects the folders it generated to be found in specific locations. So we need a way to override the default paths that Sequelize expects. Let’s create a .sequelizerc config file to help us

touch .sequelizerc

Then open it up and add the following config to tell Sequelize the new locations of its required files and folders

const path = require('path');

module.exports = {
  'config': path.resolve('config', 'database.js'),
  'models-path': path.resolve('app', 'models'),
  'seeders-path': path.resolve('db', 'seeders'),
  'migrations-path': path.resolve('db', 'migrations')
};

Now we need to open up the config/database.js file and replace the old JSON content with the following

const dbSettings = {
  development: {
    username: process.env.PGUSER,
    password: process.env.PGPASSWORD,
    database: process.env.PGDATABASE,
    host: process.env.PGHOST,
    dialect: "postgresql",
  },
  test: {
    username: process.env.PGUSER,
    password: process.env.PGPASSWORD,
    database: process.env.PGDATABASE,
    host: process.env.PGHOST,
    dialect: "postgresql",
  },
  production: {
    username: process.env.PGUSER,
    password: process.env.PGPASSWORD,
    database: process.env.PGDATABASE,
    host: process.env.PGHOST,
    dialect: "postgresql",
  },
};

module.exports = dbSettings;

We’re pulling our settings in from values stored in environment variables and we’ve set our dialect to postgresql instead of mysql.

Sequelize CLI assumes mysql by default. If we want to use a different database, we will need to update the dialect key with our alternative database."

Environment variables

We used some environment variables within our config/database.js file, but they do not exist yet. Let’s create a .env file to store them

touch .env

Within the .env file, we can add the following variables. We’ve formatted them in this way because this is how Railway names their Postgres related variables. We can leave the PGHOST blank for now, but we will come back to it

PGHOST=db
PGPORT=5432
PGUSER=divrhinopasskeys
PGPASSWORD=divrhinopasskeys
PGDATABASE=divrhinopasskeys_dev

Now we can go ahead and update the web service config in our docker-compose.yml file so that it pulls variables from our new .env file

version: "3.8"

services:
  web:
    build: .
    env_file:
      - .env
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
      - node_modules:/usr/src/app/node_modules
    command: npx nodemon index.js

volumes:
  node_modules:

We do not need to use a package such as dotenv to load environment variables from a .env file into the process.env object. We can set docker-compose up to do that for us."

Postgres service in docker-compose

In the previous section, we initialised the ORM, Sequelize, with the intention of using it with a Postgres database. We don’t have a database just yet. So let’s quickly add a new service for it in our docker-compose.yml file

version: "3.8"

services:
  web:
    build: .
    env_file:
      - .env
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
      - node_modules:/usr/src/app/node_modules
    depends_on:
      - db
    command: npx nodemon index.js
  db:
    image: postgres:alpine
    environment:
      - POSTGRES_USER=${PGUSER}
      - POSTGRES_PASSWORD=${PGPASSWORD}
      - POSTGRES_DB=${PGDATABASE}
      - POSTGRES_HOST=${PGHOST}
    ports:
      - "5432:5432"
    volumes:
      - postgres-db:/var/lib/postgresql/data

volumes:
  node_modules:
  postgres-db:

The docker-compose.yml file will use these values in our .env file to spin up our Postgres database.

We can kill the server and rebuild our app and check that our Postgres service is working by starting up our app

docker compose build
docker compose up

Then entering the db service’s shell in another terminal tab

docker compose exec db bash

and using psql to try and connect to our database

psql -h 0.0.0.0 -U divrhinopasskeys -d divrhinopasskeys_dev

We have to explicitly pass in the database name because it is different from our username. If we did not pass it in, Postgres would assume that our username and database share the same name."

If all is successful, we should see the following prompt in our terminal

psql (16.0)
Type "help" for help.

divrhinopasskeys_dev=#

We can also use a GUI database client to connect to the database. This is similar to using psql in the terminal, but with the benefit of not having to look up the various commands

Wiring up Postgres with Sequelize

We’re added both Sequelize and Postgres to our project with the intention of having them work together. However, they are not connect to one another yet, so we need to start wiring up them up.

Let’s install the Postgres npm package to make it easier to interface with Postgres from our JavaScript code

docker compose exec web npm install pg --save

Then we can create a little helper function so we can easily pass our database around to different parts of our app. Let’s first create a db/helpers directory to hold our new module

mkdir db/helpers

then we can create a new file within it to initialise our database

touch db/helpers/init.js

In the db/helpers/init.js file we can create a new Db class that uses the environment variables to connect to the database. Then we initialise and export a database instance to be used in other parts of our app

class Db {
  constructor() {
    this.Sequelize = require("sequelize");
  }

  init() {
    const dbName = process.env.PGDATABASE;
    const dbUsername = process.env.PGUSER;
    const dbPassword = process.env.PGPASSWORD;
    const dbHost = process.env.PGHOST;
    const dbPort = process.env.PGPORT;
    return new this.Sequelize(dbName, dbUsername, dbPassword, {
      host: dbHost,
      port: dbPort,
      dialect: "postgres",
    });
  }
}

module.exports = new Db().init();

First database model

Now everything is set up to allow us to create our first database model.

While our app is still running, we can open up a new terminal tab and run the following command to generate a new Sequelize model called User. This will also create a corresponding migration file. For now we will only set the email attribute

docker compose exec web npx sequelize-cli model:generate --name User --attributes email:string

Once the User model file has been generated, we can manually add the remaining attributes to our model and migration. First, in our model, we can update the init function to include all the other attributes we need. We will also specify the modelName and tableName

"use strict";

const { Model } = require("sequelize");

module.exports = (sequelize, DataTypes) => {
  class User extends Model {
    /**
     * Helper method for defining associations.
     * This method is not a part of Sequelize lifecycle.
     * The `models/index` file will call this method automatically.
     */
    static associate(models) {
      // define association here
    }
  }
  User.init(
    {
      email: {
        type: DataTypes.STRING,
        unique: true,
      },
      handle: {
        type: DataTypes.BLOB,
        unique: true,
      },
      createdAt: {
        allowNull: false,
        type: DataTypes.DATE,
      },
      updatedAt: {
        allowNull: false,
        type: DataTypes.DATE,
      },
    },
    {
      sequelize,
      modelName: "User",
      tableName: "users",
    }
  );
  return User;
};

And over in our migration file, we can add the same attributes. Making sure to use Sequelize instead of DataTypes. The migration file has an up method that executes when the database is migrated, and a down method that is executed then the database migration is rolled back

"use strict";
/** @type {import('sequelize-cli').Migration} */
module.exports = {
  async up(queryInterface, Sequelize) {
    await queryInterface.createTable("users", {
      id: {
        allowNull: false,
        autoIncrement: true,
        primaryKey: true,
        type: Sequelize.INTEGER,
      },
      email: {
        type: Sequelize.STRING,
        unique: true,
      },
      handle: {
        type: Sequelize.BLOB,
      },
      createdAt: {
        allowNull: false,
        type: Sequelize.DATE,
      },
      updatedAt: {
        allowNull: false,
        type: Sequelize.DATE,
      },
    });
  },
  async down(queryInterface, Sequelize) {
    await queryInterface.dropTable("users");
  },
};

With those edits, our first model and migration are ready. Let’s migrate our database by executing the following command

docker compose exec web npx sequelize-cli db:migrate

If we head into our database client of choice, we will notice that a users table was created, containing all the columns that we had specified, above. This can give us confidence that our migration was successful

Routes and controllers

So far, we’ve only been working with the default / route. But in this section, we will be creating a few new views and adding corresponding routes for each of them. We will create a new routes.js file in the /config directory to organise things

touch config/routes.js

And within this new file, we will require the express package. We will use it to create a new router and export it

const express = require("express");
const router = express.Router();

module.exports = router;

Now we will move our one and only route from index.js into the config/routes.js file

const express = require("express");
const router = express.Router();

router.get("/", (req, res) => {
  res.send("Hello Universe!");
});

module.exports = router;

Then we can import out routes file into index.js. The file should now look like this

const express = require("express");
const app = express();
const port = process.env.PORT || 3000;
const host = "0.0.0.0";

// Routes
app.use("/", require("./config/routes"));

app.listen(port, host, () => {
  console.log(`Example app listening on http://${host}:${port}`);
});

We can further clean up our routes.js file by introducing controllers. We will create a new /controllers directory to organise them

mkdir app/controllers

Our first controller will be the pages controller

touch app/controllers/pages.js

Within this file, we will create a PagesController call and export it

class PagesController {}

module.exports = PagesController;

Now we will move the callback function out of our route and put it in the PagesController. We can refer to this function as a controller action. We will also modify it a little so that it only renders when the user is not logged in. The controllers/pages.js file should now look like this

class PagesController {
  welcome(req, res, next) {
    if (!req.user) return res.send("Hello Universe!");
    next();
  }
}

module.exports = PagesController;

Now we can import and instantiate our PagesController into the routes file. The routes.js file should now look like this:

const express = require("express");
const router = express.Router();

// Controllers
const pages = new (require("../app/controllers/pages"))();

router.get("/", pages.welcome);

module.exports = router;

That gives us our first route and corresponding controller action. We can head into our browser to see the results.

Add support for frontend views

Now we’re in a good place to start creating some views. We will use the ejs package for templating and the express-ejs-layouts package to help us create generic layouts that all our views can share. Let’s install them both using the following command

docker compose exec web npm install ejs express-ejs-layouts --save

Once they are installed, we can head into our index.js file and configure our app to use these two packages. We will import and use layouts, then we will tell express where to join all our view files by setting the views property. We will also tell express which layout to use by setting the layout property and then set our view engine to use ejs so it understands that we are using ejs

const express = require("express");
const app = express();
const port = process.env.PORT || 3000;
const host = "0.0.0.0";

const path = require("path");
const layouts = require("express-ejs-layouts");

// Templates
app.use(layouts);
app.set("views", path.join(__dirname, "app/views"));
app.set("layout", "layouts/application");
app.set("view engine", "ejs");

// Routes
app.use("/", require("./config/routes"));

app.listen(port, host, () => {
  console.log(`Example app listening on http://${host}:${port}`);
});

Now we can start creating some view files. We will need a views directory to organise them

mkdir app/views

The first view file we create will be the application layout file that will be shared between all our views

mkdir app/views/layouts
touch app/views/layouts/application.ejs

In application.js, we can add our boilerplate HTML structure

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Divrhino Passkeys</title>
  </head>
  <body>
    <div class="container"><%- body %></div>
  </body>
</html>

Now let’s create out first view file. The directory will match the name of the controller

mkdir app/views/pages
touch app/views/pages/welcome.ejs

And in the welcome.ejs file, we will add some markup

<div class="container">
  <main class="centralising-wrapper">
    <a href="/">
      <img
        src="/images/divrhino-logo.png"
        alt="DivRhino logo"
        class="site-logo"
      />
    </a>
    <h1>Div Rhino Passkeys</h1>

    <section>
      <p class="site-intro">
        This app is built with Express.js It demonstrates how to implement a
        password-less login flow with passkeys (webauthn).
      </p>

      <div class="auth-options">
        <a href="/login" class="button-link">Login</a>
        <a href="/register" class="button-link">Register</a>
      </div>
    </section>
  </main>
</div>

Now in our PagesController, we can update the welcome function to render a view file instead of just sending through some text

class PagesController {
  welcome(req, res, next) {
    if (!req.user) return res.render("pages/welcome");
    next();
  }
}

module.exports = PagesController;

If we refresh our browser now, we should see the new markup for our welcome page.

Static assets

Now that we have our initial views, we can start considering how to handle the different types of static assets we can use within our views. In this section, we will work on adding stylesheets, script files and images.

By default, Expressjs expects static assets to live in the /public folder. This folder does not exist yet, so let’s create it now

mkdir public

Then within in, we can make a subfolder to organise our stylesheets

mkdir public/styles

Since our app is pretty small, we will have one stylesheet called main.css

touch public/styles/main.css

And in main.css, we will add some arbitrary style so that we can test it out in the browser to see if it is imported correctly

body {
  background-color: lightpink;
}

We can import the main.css stylesheet into our application layout file so that all our pages have access to it

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />

    <link href="/styles/main.css" rel="stylesheet" type="text/css" />
    <title>Divrhino Passkeys</title>
  </head>
  <body>
    <%- body %>
  </body>
</html>

If we head over to our browser and refresh, nothing happens. This is because our app doesn’t know anything about the static files in the public folder yet. Let’s change that. In our index.js file, let’s add this config under our templates config

// Templates
...

// Static files
app.use(express.static(__dirname + '/public'))

// Routes

Now when we refresh our browser, we can see the pink background colour has been applied to the body.

Now that we know our stylesheet is loading, correctly, we can copy over some ready-made styles from the project repo on Github. CSS is out of scope for this current tutorial, so we won’t be explaining any of it. But please feel free to dive into it on your own if you’re curious.

Frontend views

We will not be taking a deep look at HTML and EJS in this tutorial, as the focus is to gain a better understanding of passkey authentication. So we will be including the finished markup without any elaboration.

We will add

auth/register
auth/login
admin/dashboard

auth/register

Let’s create a new file to house our registration form. This will sit in a new views folder called auth

mkdir app/views/auth
touch app/views/auth/register.ejs

We have our register view, but if we navigate to http://0.0.0.0:3000/register in the browser, we see the following error:

Cannot GET /register

This let’s us know that the /register route does not currently exist, so we will have to create it. In our routes.js file, we can add a new register route. And we can also import the auth controller that does not exist yet

...

// Controllers
...
const auth  = new (require('../app/controllers/auth'))

// Routes
...
router.get('/register', auth.register)

module.exports = router

We will create a new file for our auth controller. This is where we will add all our auth-related actions

touch app/controllers/auth.js

And within the auth controller, we can add the register action that we referenced in our /register route

class AuthController {
  register(req, res) {
    res.render("auth/register");
  }
}

module.exports = AuthController;

Then we can add the related markup to the views/auth/register page. We will not be explaining this

<div class="container">
  <main class="centralising-wrapper">
    <a href="/">
      <img
        src="/images/divrhino-logo.png"
        alt="DivRhino logo"
        class="site-logo"
      />
    </a>
    <h1>Register</h1>

    <section class="auth-form">
      <form id="registration-form">
        <label for="email">
          <span class="input-label">email</span>
          <span class="input-field">
            <input
              id="email"
              type="email"
              name="email"
              autocomplete="email"
              placeholder="Enter your email"
              required
              autofocus
            />
          </span>
        </label>

        <input type="submit" value="Register" />
      </form>

      <p class="help">Already have an account? <a href="/login">Sign in</a></p>
    </section>
  </main>
</div>

<script type="text/javascript" src="/scripts/base64url.js"></script>
<script type="text/javascript" src="/scripts/attestation-register.js"></script>

If we visit the /register page in the browser now, we will notice that there is a broken image tag. Our app is already configured to handle static assets, so we can just created an images folder in the public directory and it will work without any additional wiring

mkdir public/images

All the images we need are available in the project repository on Github. So let’s download them and place them in the public/images directory. And now when we refresh our browser, our logo image should show up. We should also be able to see our custom cursors.

auth/login

While we are still on our /register page, let’s click the link to Sign In. We would normally expect to be redirected to the login page. However, we see a familiar error, instead

Cannot GET /login

This indicates that the /login route does not exist just yet, so let’s create a new file for it in the auth views directory

touch app/views/auth/login.ejs

Over in our routes.js file, we can add a new login route, which references a login action that does not exist yet

...

// Routes
...

router.get('/login', auth.login)

module.exports = router

So we can head into the auth controller to add a new login method which will render a login view

class AuthController {

    ...

    login(req, res) {
        res.render('auth/login')
    }
}

module.exports = AuthController

We can paste the following markup into our login EJS view

<div class="container">
  <main class="centralising-wrapper">
    <a href="/">
      <img
        src="/images/divrhino-logo.png"
        alt="DivRhino logo"
        class="site-logo"
      />
    </a>
    <h1>Login</h1>

    <section class="auth-form">
      <form>
        <label for="email">
          <span class="input-label">email</span>
          <span class="input-field">
            <input
              id="email"
              name="email"
              type="email"
              autocomplete="email webauthn"
              placeholder="Enter your email"
              required
              autofocus
            />
          </span>
        </label>
        <input type="submit" value="Login" />
      </form>

      <p class="help">
        Don't have an account yet? <a href="/register">Register</a>
      </p>
    </section>
  </main>
</div>

<script type="text/javascript" src="/scripts/base64url.js"></script>
<script type="text/javascript" src="/scripts/assertion-login.js"></script>

Now if we refresh our browser, the /login page should be rendering correctly.

admin/dashboard

The final page we need to create is the dashboard. This page should only be visible when a user has logged in.

First, let’s create a new admin folder in our app/views directory. Then within this directory, we can create our dashboard.ejs file

mkdir app/views/admin
touch app/views/admin/dashboard.ejs

In our routes file, we can import the AdminController and let’s update the root route (/) to take another action for our admin.dashboard

...

// Controllers
...
const admin = new (require('../app/controllers/admin'))

// Routes
router.get('/', pages.welcome, admin.dashboard)

...

module.exports = router

If you notice, the admin.dashboard action does not exist yet. In fact, we’re even importing the admin controller when it doesn’t even exist yet either. So let’s create it

touch app/controllers/admin.js

Now we can create a new dashboard action in this new AdminController

class AdminController {
  dashboard(req, res) {
    res.render("admin/dashboard", { user: req.user });
  }
}

module.exports = AdminController;

We can paste the following markup into our dashboard EJS view

<div class="container-wide">
  <nav class="site-nav">
    <h1>Div Rhino Dashboard</h1>
    <div class="logout-form">
      <form action="/logout" method="post">
        <input type="submit" value="Logout" />
      </form>
    </div>
  </nav>
  <main class="admin-wrapper">
    <section class="empty-dashboard">
      <img
        src="/images/divrhino-logo-greyscale.png"
        alt=""
        class="empty-logo"
      />
      <p>Your dashboard is currently empty.</p>
    </section>
  </main>
</div>

We won’t look at our dashboard until after we sign in. So for now, let’s just trust that it works as expected.

And those are the all the view we will need to start implementing our authentication workflow.

The PublicKeyCredentials table

Next up, we need to create a new database table. We already have a users table in our database. Now we need to a table to store our users’ public keys. We will call it public_key_credentials. For now, we will only include the public_key attribute in our command

docker compose exec web npx sequelize-cli model:generate --name PublicKeyCredentials --attributes public_key:string

If the command executes successfully, we should have a new model and migration file. Now we can manually add the remaining attributes to each file. Let’s edit the model file first. We can add more columns for our table. A user can have many public_key_credentials, so we will need to configure a user_id as a foreign key under associations. Then we will have two string columns for an external_id and a public_key. We will also set the modelName and tableName

"use strict";
const { Model } = require("sequelize");
module.exports = (sequelize, DataTypes) => {
  class PublicKeyCredentials extends Model {
    /**
     * Helper method for defining associations.
     * This method is not a part of Sequelize lifecycle.
     * The `models/index` file will call this method automatically.
     */
    static associate(models) {
      // define association here
      this.belongsTo(models.User, { foreignKey: "user_id" });
    }
  }
  PublicKeyCredentials.init(
    {
      user_id: DataTypes.INTEGER,
      external_id: DataTypes.STRING,
      public_key: DataTypes.STRING,
    },
    {
      sequelize,
      modelName: "PublicKeyCredentials",
      tableName: "public_key_credentials",
    }
  );
  return PublicKeyCredentials;
};

Then over in our migration file, we can add the same attributes. Making sure to use Sequelize instead of DataTypes.

"use strict";
/** @type {import('sequelize-cli').Migration} */
module.exports = {
  async up(queryInterface, Sequelize) {
    await queryInterface.createTable("public_key_credentials", {
      id: {
        allowNull: false,
        autoIncrement: true,
        primaryKey: true,
        type: Sequelize.INTEGER,
      },
      user_id: {
        type: Sequelize.INTEGER,
        allowNull: false,
      },
      external_id: {
        type: Sequelize.STRING,
        unique: true,
      },
      public_key: {
        type: Sequelize.STRING,
      },
      createdAt: {
        allowNull: false,
        type: Sequelize.DATE,
      },
      updatedAt: {
        allowNull: false,
        type: Sequelize.DATE,
      },
    });
  },
  async down(queryInterface, Sequelize) {
    await queryInterface.dropTable("public_key_credentials");
  },
};

We should also update our user model to have a hasMany association for public_key_credentials, remembering to also specify the foreignKey

class User extends Model {
  /**
   * Helper method for defining associations.
   * This method is not a part of Sequelize lifecycle.
   * The `models/index` file will call this method automatically.
   */
  static associate(models) {
    // define association here
    this.hasMany(models.PublicKeyCredentials, { foreignKey: "user_id" });
  }
}

Those are the only edits we need for this resource, so we can migrate our database by executing the following command

docker compose exec web npx sequelize-cli db:migrate

Now if we look in our database client, we should see a new public_key_credentials table with all the columns we just added.

Configure Passportjs

We now have all the database tables we need, so we can start configuring Passport.js.

Passport.js is one of the most widely-used auth solutions in the JavaScript ecosystem. It comes with a variety of “strategies” that you can explore in their docs. We will be using the WebAuthn strategy in this tutorial.

First we need to install the base passport package as a dependency

docker compose exec web npm install passport --save

Then we will install the WebAuthn strategy to give us the functionality we need to use passkeys. This strategy will allow our server to generate a challenge that can be used during the attestation (register) and assertion (login) phases

docker compose exec web npm install passport-fido2-webauthn --save

We’re going to create a little Passport service to act as a wrapper around our Passport-related code. Using a service will allow us to keep this contained in its own module so our index.js file is cleaner. It will also make it easier to swap out the code if we decide to go with another strategy in future. Let’s create a new app/services directory to organise our services

mkdir app/services

Then we can make a new file for our passport-service

touch app/services/passport-service.js

We will add a basic skeleton for the module

class PassportService {
  init(store) {
    // 1. configure passport to use WebAuthn Strategy
    // 2. passport serialise user
    // 3. passport deserialise user
  }
}

module.exports = PassportService;

And import our new service into our routes file so that is has access to the challenge store. We will pass this same challenge store to a couple of future routes too.

...

// Passport
const PassportService = require('../app/services/passport-service')

const SessionChallengeStore =
    require('passport-fido2-webauthn').SessionChallengeStore

const passportService = new PassportService()
const store = new SessionChallengeStore()

passportService.init(store)

...

Opening up our PassportService file, we can start putting our service together. We will essentially need to set up 3 main functions

A function that sets passport up to use the WebAuthnStrategy
A callback that passport can use to serializeUer
A callback that passport can use to deserializeUser

The first function will be called useWebauthnStrategy(), because that’s what we’re hoping to use it for. It will take in an instance of SessionChallengeStore as it’s only argument and it will return a new instance of WebAuthnStrategy.

Within the function body, we will return a new WebAuthnStrategy instance. We will need to pass it three bits of information:

a store object
a verify() callback, and
a register() callback.

The store object is pretty straightforward. We will create a new Object literal with a store key set its value to the instance of SessionChallengeStore that we has passed it. This is what we will use to generate a challenge.

The verify() callback will be used when logging the user in, and the register() callback will be used when registering a new user.

const passport = require("passport");
const WebAuthnStrategy = require("passport-fido2-webauthn");

class PassportService {
  init(store) {
    // 1. configure passport to use WebAuthn Strategy
    passport.use(this.useWebauthnStrategy(store));
    // 2. passport serialise user
    // 3. passport deserialise user
  }

  useWebauthnStrategy(store) {
    return new WebAuthnStrategy(
      { store: store },
      this.verify, // needs to be fleshed out
      this.register // needs to be fleshed out
    );
  }

  // Verify callback
  async verify(id, userHandle, done) {}

  // Register callback
  async register(user, id, publicKey, done) {}
}

module.exports = PassportService;

Let’s start fleshing out the verify() method first. The end goal of this function is to look up a specified user in the database and get their public key. Again, as we mentioned above, this will allow us to log them in.

We will pass in the id, userHandle and a done callback as the 3 arguments. And we will wrap all the database querying actions within a database transaction. Using a transaction means that we can rollback, without committing any changes to the database, if something goes wrong. We will also use a try...catch statement to facilitate a rollback path, just in case we need it

// Verify callback
async verify(id, userHandle, done) {
    const transaction = await db.transaction()
    try {
        // Add database actions here

        await transaction.commit()

                // execute done callback function here
    } catch (error) {
        await transaction.rollback()
        throw error
    }
}

We don’t need to add anything more to our catch block, so we can just focus on the try block. First, we will query the database to findOne result for a PublicKeyCredentials record that is associated with the current user’s external_id. If we are unable to find one, we will call the done() callback with an error message

// Verify callback
async verify(id, userHandle, done) {
    const transaction = await db.transaction()
    try {
                // Find PublicKeyCredentials for current user
        const currentCredentials =
            await models.PublicKeyCredentials.findOne(
                {
                    where: { external_id: id },
                },
                { transaction }
            )

        if (currentCredentials === null) {
            return done(null, false, { message: 'Invalid key. ' })
        }

                // Find associated user

        await transaction.commit()

                // execute done callback function here
        return done(null, currentCredentials, currentCredentials.public_key)
    } catch (error) {
        await transaction.rollback()
        throw error
    }
}

If we are able to get a PublicKeyCredentials record for the current user, we will use its user_id value to find the actual user record too. And if the user record does not exist, we will execute the done() callback and send back an error message. However, if we are able to find a relevant user record, we will compare it’s handle column against the userHandle we pass into the function. Then we will commit the transaction. At this point, the verify() code would have successfully determined who current user, so it invokes the done callback with the user record and a public key

// Verify callback
async verify(id, userHandle, done) {
    const transaction = await db.transaction()
    try {
                // Find PublicKeyCredentials for current user

                ...

                // Find associated user
        const currentUser = await models.User.findOne(
            {
                where: { id: currentCredentials.user_id },
            },
            { transaction }
        )

        if (currentUser === null) {
            return done(null, false, { message: 'No such user. ' })
        }

                // Compare user record's handle to the handle we pass in
        if (Buffer.compare(currentUser.handle, userHandle) != 0) {
            return done(null, false, { message: 'Handles do not match. ' })
        }

                // Commit the transaction
        await transaction.commit()

                // When current user is determined, invoke done() callback
                // with user record and its public key
        return done(null, currentCredentials, currentCredentials.public_key)
    } catch (error) {
        await transaction.rollback()
        throw error
    }
}

In this phase, the public key is used to cryptographically verify the WebAuthn assertion and authentication the user. The verify() function is used when a user logs in to the system."

The final verify() method should look like this

// Verify callback - used when a user wants to sign in
    async verify(id, userHandle, done) {
        const transaction = await db.transaction()
        try {
            // Find public key by id
            const currentCredentials =
                await models.PublicKeyCredentials.findOne(
                    {
                        where: { external_id: id },
                    },
                    { transaction }
                )

            if (currentCredentials === null) {
                return done(null, false, { message: 'Invalid key. ' })
            }

            // Find user by publicKey's user_id
            const currentUser = await models.User.findOne(
                {
                    where: { id: currentCredentials.user_id },
                },
                { transaction }
            )

            if (currentUser === null) {
                return done(null, false, { message: 'No such user. ' })
            }

            // Compare user.handle from db with userHandle from request
            if (Buffer.compare(currentUser.handle, userHandle) != 0) {
                return done(null, false, { message: 'Handles do not match. ' })
            }

            // Commit transaction
            await transaction.commit()

            // When current user is determined, invoke done() callback
            // with user record and its public key
            return done(null, currentCredentials, currentCredentials.public_key)
        } catch (error) {
            await transaction.rollback()
            throw error
        }
    }

Because we were only reading from the database, we didn’t technically need to use a transaction. However, demonstrating it here will help us get used to the idea of it a little more, as we will see it again very soon.

Before we move on, we should also remember to import the db and the Sequelize models at the top of our passport-service.js file because we are using them within the code we just wrote

const db = require("../../db/helpers/init");
const models = require("../models");

Now we can start working on the register() callback. The end goal of this function is to create a new user and a new associated PublicKeyCredentials record in the database.

We will pass in the user, id, publicKey, and a done callback as the 4 arguments to the function. We will also wrap everything in a try...catch statement and execute the database actions within a database transaction. As we mentioned previously, using a transaction will give us the ability to rollback the changes if anything goes wrong.

The user refers to the user data we pass in from the request. The id will be used as the external_id of the PublicKeyCredentials record. We will use this to “label” or “identify” public key records. The publicKey refers to the encoded publicKey and challenge. Once we successfully persist the necessary database records, we invoke the done() callback with the newly-created user record

// Register callback
async register(user, id, publicKey, done) {
    const transaction = await db.transaction()
    try {
                // Create new user using email and handle

                // Create new public key credentials from user.id, the passed-in id,
                // and the passed-in publicKey


                // If all goes well, we commit the changes from this transaction
        await transaction.commit()

                // Once complete, invoke done() callback with new user record
        return done(null, newUser)
    } catch (error) {
        await transaction.rollback()
        throw error
    }
}

We will use the handy create() method we get from Sequelize to create a new user record using the email and handle. Since this is happening in a database transaction, we also pass in a options object that contains the transaction. And if, for whatever reason, we are unable to create a new user record, we will invoke our done callback with an error message

// Register callback
async register(user, id, publicKey, done) {
    const transaction = await db.transaction()
    try {
                // Create new user using email and handle
        const newUser = await models.User.create(
            {
                email: user.name,
                handle: user.id,
            },
            { transaction }
        )

        if (newUser === null) {
            return done(null, false, { message: 'Could not create user. ' })
        }

                // Create new public key credentials from user.id, the passed-in id,
                // and the passed-in publicKey

                // If all goes well, we commit the changes from this transaction
        await transaction.commit()

                // Once complete, invoke done() callback with new user record
        return done(null, newUser)
    } catch (error) {
        await transaction.rollback()
        throw error
    }
}

After we’ve successfully created a new user record, we will move on to creating its associated public key record. Since this action is also being performed in the same database transaction as our previous action, we will pass in the transaction in the options object. And if, for some reason, we are unable to persist a new PublicKeyCredentials record, we will invoke the done callback with an error message

// Register callback
async register(user, id, publicKey, done) {
    const transaction = await db.transaction()
    try {
                // Create new user using email and handle

        ...

                // Create new public key credentials from user.id, the passed-in id,
                // and the passed-in publicKey
        const newCredentials = await models.PublicKeyCredentials.create(
            {
                user_id: newUser.id,
                external_id: id,
                public_key: publicKey,
            },
            { transaction }
        )

        if (newCredentials === null) {
            return done(null, false, { message: 'Could not create public key. ' })
        }

                // If all goes well, we commit the changes from this transaction
        await transaction.commit()

                // Once complete, invoke done() callback with new user record
        return done(null, newUser)
    } catch (error) {
        await transaction.rollback()
        throw error
    }
}

The final register() method should look like this

// Register callback - new user wants to sign up
    async register(user, id, publicKey, done) {
        const transaction = await db.transaction()
        try {
            // Create new user using email and handle
            const newUser = await models.User.create(
                {
                    email: user.name,
                    handle: user.id,
                },
                { transaction }
            )

            if (newUser === null) {
                return done(null, false, { message: 'Could not create user. ' })
            }

            // Create new public key credentials from user.id, the passed-in id,
            // and the passed-in publicKey
            const newCredentials = await models.PublicKeyCredentials.create(
                {
                    user_id: newUser.id,
                    external_id: id,
                    public_key: publicKey,
                },
                { transaction }
            )

            if (newCredentials === null) {
                return done(null, false, { message: 'Could not create public key. ' })
            }

            // If all goes well, we commit the changes from this transaction
            await transaction.commit()

            // Once complete, invoke done() callback with new user record
            return done(null, newUser)
        } catch (error) {
            await transaction.rollback()
            throw error
        }
    }

The verify() and register() methods are the most involved parts of the PassportService. But we still have a little more to go before our service is ready. We need to use passport to serialise and deserialise the user.

Here we are using Node’s process.nextTick(). Node has an event loop and every time the event loop takes a full trip, we call it a tick. So when we pass a function to process.nextTick(), we are telling the engine to invoke this function at the end of the current operation, before the next event loop tick start."

Let’s start with creating a token via the passport.serializeUser() function. Passport will call the serializeUser function to serialise a user to the session. So it will take the regular JavaScript object containing the user details and convert it into an encrypted string (token). When a user is serialised, we call pass in a custom callback function (i.e. this.serialiseUserFn) that invokes done() callback with a regular JavaScript object that contains the user’s id and email

const passport = require('passport')
const WebAuthnStrategy = require('passport-fido2-webauthn')

const db = require('../../db/helpers/init')
const models = require('../models')

class PassportService {
    init(store) {
        // 1. configure passport to use WebAuthn Strategy
        passport.use(this.useWebauthnStrategy(store))
        // 2. passport serialise user
        passport.serializeUser(this.serialiseUserFn)
        // 3. passport deserialise user
    }

    useWebauthnStrategy(store) { ... }

        // Serialise user to token
    serialiseUserFn(user, done) {
        process.nextTick(() => {
            done(null, { id: user.id, email: user.email })
        })
    }

    // Verify callback
    async verify(id, userHandle, done) { ... }

    // Register callback
    async register(user, id, publicKey, done) { ... }

}

module.exports = PassportService

To read user information from the token, we can use the passport.deserializeUser() function. Passport will call the deserializeUser to read and deserialise user information from the token. And when we are deserialising, we pass in a custom callback function (i.e. this.deserialiseUserFn) that will invoke the done() callback with the user

const passport = require('passport')
const WebAuthnStrategy = require('passport-fido2-webauthn')

const db = require('../../db/helpers/init')
const models = require('../models')

class PassportService {
    init(store) {
        // 1. configure passport to use WebAuthn Strategy
        passport.use(this.useWebauthnStrategy(store))
        // 2. passport serialise user
        passport.serializeUser(this.serialiseUserFn)
        // 3. passport deserialise user
        passport.deserializeUser(this.deserialiseUserFn)
    }

    useWebauthnStrategy(store) { ... }

        // Serialise user to token
    serialiseUserFn(user, done) { ... }

        // Deserialise user from token
    deserialiseUserFn(user, done) {
        process.nextTick(() => {
            return done(null, user)
        })
    }

    // Verify callback
    async verify(id, userHandle, done) { ... }

    // Register callback
    async register(user, id, publicKey, done) { ... }

}

module.exports = PassportService

And with that, we can wrap up our PassportService. You can refer to the finished service in the Github repo to compare your code if you get stuck. Let’s keep moving.

Sessions

The WebAuthn API requires session support. So we will need to set that up. We will utilise the express-session middleware to help us.

So let’s install that first and we will configure it later

docker compose exec web npm install express-session --save

We will also need to install connect-session-sequelize so that we can use Sequelize to store our session data in the database

docker compose exec web npm install connect-session-sequelize --save

Now that those two packages are installed, we can configure them in the index.js file. We will create a new sessionStore and add some settings like the maxAge of the cookie. Then we will call sessionStore.sync()

...

const passport = require('passport')
const session = require('express-session')
const SequelizeStore = require('connect-session-sequelize')(session.Store)

...

const db = require('./db/helpers/init')

// Session store
const sessionStore = new SequelizeStore({
  db: db,
})

...

// Sessions
app.use(
    session({
        secret: process.env.SESSION_SECRET,
        resave: false,
        saveUninitialized: false,
        store: sessionStore,
        cookie: {
            maxAge: 1000 * 60 * 60 * 24 * 7, // 1 week
        },
    })
)

sessionStore.sync()
app.use(passport.authenticate('session'))

...

app.listen(port, host, () => {
    console.log(`Example app listening on http://${host}:${port}`)
})

If you noticed, we have used an environment variable process.env.SESSION_SECRET in our session settings. This doesn’t exist yet, so we can add it to our .env file

PGHOST = db;
PGPORT = 5432;
PGUSER = divrhinopasskeys;
PGPASSWORD = divrhinopasskeys;
PGDATABASE = divrhinopasskeys_dev;

SESSION_SECRET = randomstringabcde123;

Now if we refresh our browser, we can look in our database to see that a new Sessions table has been automatically created for us.

Configuration for form submission

Because we need to be able to submit form data within our application, we need to configure our app to support this.

First let’s ensure our app knows how to work with JSON. We can use the express.json() function to add this functionality. This is a built-in middleware that parses incoming requests that contain JSON payloads

// Parse JSON requests
app.use(express.json());

And since we’re also working with multipart form data, we should also install a middleware called multer

docker compose exec web npm install multer --save

Then we can configure it in index.js

const multer = require('multer')
...

app.use(multer().none())

And since our sessions use cookies, we will also add the cookie-parser package

docker compose exec web npm install cookie-parser --save

Then configure it

const cookieParser = require('cookie-parser')
...

app.use(cookieParser())

While we are here, we should also tell our app to handle urls that contain query param data. When the extended option is set to false, express will use the querystring library to parse the url-encoded data

app.use(express.urlencoded({ extended: false }));

Passkeys

We have finally reached the main attraction: Passkeys!

Before we dive in and start implementing our authentication with passkeys, let’s take a little time to go over a high-level view of the whole passkey authentication process.

Passkeys are used in 2 different phases

the attestation phase, and
the assertion phase

The attestation phase can be thought of as the user registration part of the process, while the assertion can be seen as the user login part. And during these phases, there are 3 main entities working together

the client
the authenticator, and
the relying party

We will implement the two different phases in the next few sections.

Phase 1: Attestation (creating a passkey)

The first phase we will go over is the attestation phase. In the attestation phase, a new public key credential and associated user is registered. In a regular web application, a user would create an account with their email and password. When using WebAuthn, a user is created with an associated passkey to be used in place of a password.

To create a passkey, a few things need to happen between the client, the authenticator and the relying party. Below is a high level summary of the 3 main steps in the process

The user submits the registration form, prompting the client to request a challenge from the relying party

The client calls the navigator.credentials.create(challenge) method, passing in the challenge that was received. The navigator.credentials.create() method is made available to us from the Web Authentication API. This will prompt the authenticator to use the challenge it receives to create private+public key credential pair. The user is verified via thumbprint. The private key is stored on the authenticator and is used to sign the challenge. Then the public key, the signed challenge and credential id are sent back to client

The client sends the signed challenge, credential id, and public key onwards to the relying party (server). The server verifies the signed challenge with the public key and session information. If the verification process is successful, the server will store the public key credentials and user in the database

Now that we have an idea of the flow we want to achieve, let’s start working on the code. We will try to mirror the process we just went through in the overview, above.

Starting with the registration form. If we take a look at the register view file, we can see that we have already imported the attestation-register.js script file into the register view. This file will contain code that will allow us to submit the registration form. This script file does not exist yet, so we will create it soon. You may notice that we are also importing the base64url package. We will copy the contents for this from our project repo

<!-- register view -->

<!-- Import base64url above our attestation-register script -->
<script type="text/javascript" src="/scripts/base64url.js"></script>
<script type="text/javascript" src="/scripts/attestation-register.js"></script>

Let’s create a scripts folder in the public folder to organise our frontend JavaScript files

mkdir public/scripts

Then within it, we can create our attestaion-register.js file

touch public/scripts/attestation-register.js

And also our base64url.js file

touch public/scripts/base64url.js

We can head to our project repo and copy over the contents for this base64url.js file.

Now inside the attestation-register.js file, we can start creating our Register module. It will just have an init method for now. The init method will take the event as its only argument. Then we will add some code that will listen for the window object to load. Once the window has loaded, we will add an eventListener to the registration form, so whenever it is submitted, we will create a new instance of our Register class and call init(). We will also prevent the default behaviour that the browser gives to the form element

class Register {
  async init(event) {
    // 1. Get Challenge from server (Relying Party)
    // 2. Use challenge to create public key credential pair
    // 3. Send publicKey+challenge to server to create new user
    // 4. Redirect to user's dashboard
  }
}

window.addEventListener("load", async () => {
  document
    .querySelector("#registration-form")
    .addEventListener("submit", async (event) => {
      event.preventDefault();

      const register = new Register();
      await register.init(event);
    });
});

We’ve listed the four functions we will need in order to complete our Register module. The first 3 correspond to the steps we discussed in the overview for phase 1. Step 4 will just redirect the user to the appropriate route.

The first function we need will get fetch a random challenge from the server by using the fetch API to make a request to the /register/public-key/challenge endpoint. We will store this challenge in a variable so we can pass it onto the next function

class Register {
    async init(event) {
        // 1. Get Challenge from server (Relying Party)
        const challenge = await this.getChallenge(event)
        // 2. Use challenge to create public key credential pair
        // 3. Send publicKey+challenge to server to create new user
        // 4. Redirect to user's dashboard
    }

    async getChallenge(event) {
        const response = await fetch('/register/public-key/challenge', {
            method: 'POST',
            headers: {
                Accept: 'application/json',
            },
            body: new FormData(event.target),
        })

        return response.json()
    }

}

...

The /register/public-key/challenge endpoint does not exist yet so lets switch over to our routes file on the server side and create it. It will take in the challenge store as its only argument

router.post("/register/public-key/challenge", auth.createChallengeFrom(store));

Our new route calls an AuthController action named createChallengeFrom(), which does not exist yet. So let’s head into the AuthController file so we can create it. Within the auth controller, we can add the new createChallengeFrom method. As mentioned above, it takes in the challenge store as its only argument and returns a middleware function.

Within this middleware function body, we will set up a user object. It will have an id and a name. The id will be a unique uuid that we generate with the help of the uuid package we import at the top of the file. The name will be set to the email value the user provided in the registration form. The user object and the challenge will be returned in the JSON response

const base64url = require('base64url')
const uuid = require('uuid').v4

class AuthController {
    ...

    createChallengeFrom(store) {
        return (req, res, next) => {
            const user = {
                id: uuid({}, Buffer.alloc(16)),
                name: req.body.email,
            }

            store.challenge(req, {user: user}, (err, challenge) => {
                if (err) return next(err)

                user.id = base64url.encode(user.id)

                res.json({
                    user: user,
                    challenge: base64url.encode(challenge),
                })
            })
        }
    }

    ...
}

module.exports = AuthController

We are using a package called base64url in this controller code. This package will ensure that binary data and urls are properly encoded to plain text to avoid ambiguity. We imported it, but haven’t added it to our codebase yet, so let’s do that now

docker compose exec web npm install base64url --save

We also imported the uuid package without installing it, so let’s install that too

docker compose exec web npm install uuid --save

And with that, our createChallengeFrom(store) method is done. We can head back to our attestation-register.js file and continue our main quest.

The second function will use the navigator.credentials.create() method to prompt the authenticator to use the challenge it receives to create a private+public key credential pair. The authenticator will also ask the user for verification in the form of a fingerprint or similar. The private key is stored on the authenticator and used to sign the challenge.

Within the options, we have indicated that would like to create new credentials that have the type publicKey. Within the publicKey option, we send along:

the name of the relying party
user details
the signed challenge
information about the authenticator, and
an array of supported pubKeyCredParams

class Register {
    async init(event) {
        // 1. Get Challenge from server (Relying Party)
        const challenge = await this.getChallenge(event)
        // 2. Use challenge to create public key credential pair
        const credentials = await this.createPublicKeyPairWith(challenge)
        // 3. Send publicKey+challenge to server to create new user
        // 4. Redirect to user's dashboard
    }

    ...

    async createPublicKeyPairWith(challengeResponse) {
        const options = {
            publicKey: {
                rp: { name: 'Divrhinopasskeys' },
                user: {
                    id: base64url.decode(challengeResponse.user.id),
                    name: challengeResponse.user.name,
                    displayName: challengeResponse.user.name,
                },
                challenge: base64url.decode(challengeResponse.challenge),
                pubKeyCredParams: [
                    {
                        type: 'public-key',
                        alg: -7, // ES256
                    },
                    {
                        type: 'public-key',
                        alg: -257, // RS256
                    },
                    {
                        type: 'public-key',
                        alg: -8, // Ed25519
                    },
                ],
                authenticatorSelection: {
                    userVerification: 'preferred',
                },
            },
        }

        const newCredentials = await navigator.credentials.create(options)
        return newCredentials
    }

}

...

The pubKeyCredParams indicates the type of encrypted credentials the relying party will accept. The numbers assigned to the alg keys refer to COSE algorithm identifiers and they represents the cryptographic algorithm used for the encrypting the credentials. Based on the order of the array objects, the Relying Party will accept either an ES256, RS256, or Ed25519 credentials, but prefers ES256"

The third function will create a new user in the database and allow this new user to loginWith(credentials). It will use the fetch API to make a POST request to the /login/public-key endpoint and pass along options that include clientDataJSON and an attestationObject. Then it will return the logged in user object, which we will store in a variable called currentUser

class Register {
    async init(event) {
        // 1. Get Challenge from server (Relying Party)
        const challenge = await this.getChallenge(event)
        // 2. Use challenge to create public key credential pair
        const credentials = await this.createPublicKeyPairWith(challenge)
        // 3. Send publicKey+challenge to server to create new user
        const currentUser = await this.loginWith(credentials)
        // 4. Redirect to user's dashboard
    }

    buildLoginOptionsWith(userCredentials) {
        const body = {
            response: {
                clientDataJSON: base64url.encode(
                    userCredentials.response.clientDataJSON
                ),
                attestationObject: base64url.encode(
                    userCredentials.response.attestationObject
                ),
            },
        }

        if (userCredentials.response.getTransports) {
            body.response.transports =
            userCredentials.response.getTransports()
        }

        return body
    }

    async loginWith(userCredentials) {
        const options = this.buildLoginOptionsWith(userCredentials)

        const response = await fetch('/login/public-key', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify(options)
        })

        return response.json()
    }

}

...

The /login/public-key endpoint does not exist yet, so let’s switch gears a little bit and head into the backend to work on it. We’ll start by adding a route to our routes file. We have 3 actions listed. First we will use passport.js, under the hood, to see if the user is authenticated. Then we will call the next handler in the list. If the user is authenticated, we will admit them to the dashboard. However, if they are not authenticated, we will fall through to the last action and deny them entry

router.post(
  "/login/public-key",
  auth.passportCheck(),
  auth.admitUser,
  auth.denyUser
);

We haven’t created any of these three AuthController actions just yet, so let’s head into the AuthController file and create them now.

For the passportCheck(), we will use the passport.authenticate() method with the webauthn option and we will add a couple of settings for the error messages

passportCheck() {
    return passport.authenticate('webauthn', {
        failureMessage: true,
        failWithError: true,
    })
}

And since we’re using the passport package, we should also ensure it is imported at the top of the AuthController file

const passport = require("passport");

Next we can add the admitUser middleware function. This will return an object with an ok field set to true and destination value set to the root route i.e. /

admitUser(req, res, next) {
    res.json({ ok: true, destination: '/' })
}

Lastly, the denyUser action which will check if the error is a 4xx type error and return an object where with the ok field set to false and the destination set to the login page

denyUser(err, req, res, next) {
    const cxx = Math.floor(err.status / 100)

    if (cxx != 4) return next(err)

    res.json({ ok: false, destination: '/login' })
}

Passport.js will do its thing, and will be able to appropriately redirect a user when they successfully login. This concludes this little sidequest into the backend, we can go back to the frontend and continue working on our Register module.

The fourth function and final function will redirect the user to the appropriate page. We will use the destination that is returned from the passportCheck() step. If the verification is successful, this destination would be the dashboard route

class Register {
    async init(event) {
        // 1. Get Challenge from server (Relying Party)
        const challenge = await this.getChallenge(event)
        // 2. Use challenge to create public key credential pair
        const credentials = await this.createPublicKeyPairWith(challenge)
        // 3. Send publicKey+challenge to server to create new user
        const currentUser = await this.loginWith(credentials)
        // 4. Redirect to user's dashboard
        this.redirect(currentUser)
    }

    ...

    redirect(currentUser) {
        window.location.href = currentUser.destination
    }

}

...

We’ve added quite a lot of new config to our app, so we should restart our server. Now if we test this out in the browser now, we should see a popup dialogue that will allow us to use an authenticator to create a passkey for our new user.

Logout

The logout process is pretty straightforward and is mostly identical to web applications that use regular password authentication.

We can start by adding a new route

router.post("/logout", auth.logout);

Our new logout route is calling an auth controller action that does not exist yet. So let’s open up the AuthController and add the auth.logout action. If there is an error, we will just use the next() middleware to call the next middleware function in the stack. But if there are no errors, we will redirect the user to the root route

logout(req, res, next) {
    req.logout((err) => {
        if (err) return next(err)
        res.redirect('/')
    })
}

Now is we head into the browser and hit the logout button, we should be taken back to the root route.

Phase 2: Assertion (using the passkey)

Now that we can register a new user and also log them out, let’s work on the remaining phase of the process, which is to authenticate the user with the passkey we created. As a reminder, this phase is called the assertion phase, which can be thought of as the user login. Instead of a user keying in their password, they will use their passkey for authentication.

To use a passkey, a few things need to happen between the client, the authenticator and the relying party. The following is a brief summary of the 3 main steps in the process

The user submits the sign in form, prompting the client to request a new random challenge from the relying party. This first step is more or less the same as the first step of the attestation phase

The client calls navigator.credentials.get(challenge) method, passing in the challenge that was received. The navigator.credentials.get() is another method made available to us by the Web Authentication API. This will prompt the authenticator to request verification from the user via thumbprint. The private key that is stored on the authenticator is used to sign the challenge. The signed challenge, the credential id, and username are sent back to client

The client then sends the signed challenge, credential id, and username onwards to the relying party (server). The server verifies the signed challenge with the public key from the database. If the verification is successful, the server will find the correct user in the database and log them into our application

Now that we have gone over the flow we want to achieve, let’s start working on the code. We will try to mirror the process we just went through in the overview, above.

Starting with the login form. If we take a look at the register view file, we can see that we have already imported the the assertion-login.js script file into the login view. This file will contain code that will allow us to submit the login form. This file does not exist yet, so we will create it soon. We are also importing the base64url package too. We previously created this file in phase 1

<!-- login view -->

<script type="text/javascript" src="/scripts/base64url.js"></script>
<script type="text/javascript" src="/scripts/assertion-login.js"></script>

Now we can create the script file in the /public folder to organise our assertion/login code

touch public/scripts/assertion-login.js

Inside the assertion-login.js file, we can start creating our Login module. It will just have an init method for now. Then we will add some code that will listen for the window object to load before it creates a new instance of our Login class. We will only init the login instance if the user agent supports the PublicKeyCredential interface

class Login {
  async init() {
    // 1. Check that conditional mediation is supported
    // 2. Get challenge from server (Relying Party)
    // 3. Use existing public key credential to authenticate user
    // 4. Use public key credential to login user
    // 5. Redirect to user's dashboard
  }
}

window.addEventListener("load", async () => {
  const login = new Login();

  if (window.PublicKeyCredential) {
    await login.init();
  }
});

We’ve listed out 5 functions we will need in order to complete our Login module. The middle 3 correspond to the steps we discussed in the overview for phase 2. But steps 1 and 5 are additional steps for a nicer user experience.

The first function we need to implement will check if the User Agent supports conditional mediation. If conditional mediation is available, the browser can show any available passkeys to the user in a dialog box. If conditional medication is not supported, we will return early and the rest of the login code will not be run

class Login {
    async init() {
        // 1. Check that conditional mediation is supported
        await this.checkConditionalMediationSupport()
        // 2. Get challenge from server (Relying Party)
        // 3. Use existing public key credential to authenticate user
        // 4. Use public key credential to login user
        // 5. Redirect to user's dashboard
    }

    async checkConditionalMediationSupport() {
        const isCMA =
            await window.PublicKeyCredential.isConditionalMediationAvailable()
        if (!isCMA) return
    }
}

...

This is a sneak peek of what the dialog box would look like in Chrome once it is implemented.

The second function will get fetch a random challenge from the server by using the fetch API to make a request to the /login/public-key/challenge endpoint. We will store this challenge in a variable and pass it onto the next function

class Login {
    async init() {
        // 1. Check that conditional mediation is supported
        await this.checkConditionalMediationSupport()
        // 2. Get challenge from server (Relying Party)
        const challenge = await this.getChallenge()
        // 3. Use existing public key credential to authenticate user
        // 4. Use public key credential to login user
        // 5. Redirect to user's dashboard
    }

    ...

    async getChallenge() {
        const response = await fetch('/login/public-key/challenge', {
            method: 'POST',
            headers: {
                Accept: 'application/json',
            },
        })

        return response.json()
    }
}

...

The /login/public-key/challenge route does not exist yet so lets switch over to our routes file on the server side and create it now. This route will work exactly like the route we created in the attestation phase. But for the purposes of simplicity, we will duplicate the functionality again for our login phase

router.post("/login/public-key/challenge", auth.getChallengeFrom(store));

Our new route calls an auth controller action named getChallengeFrom(), which does not exist yet. So let’s head into the AuthController file so we can create it. We will use the store to generate a new challenge. The store is the instance of SessionChallengeStore that we pass through from our routes file. If there is an error, we will just use the next() middleware to call the next middleware function in the stack. But if there are no errors, will send back a JSON response object that contains a randomly generated challenge.

getChallengeFrom(store) {
    return (req, res, next) => {
        store.challenge(req, (err, challenge) => {
            if (err) return next(err)
            res.json({ challenge: base64url.encode(challenge) })
        })
    }
}

And that’s it for the backend parts of the login phase. We can return to the assertion-login.js file and forge ahead.

The third function will use the navigator.credentials.get() method to prompt the authenticator to seek verification from the user in the form of a thumbprint or something similar. The authenticator will also use the stored private key to sign the challenge we pass to it. Within the options, we have indicated that we will use conditional mediation and our credentials are of the type publicKey. The publicKey option tells the user agent to use an existing set of public key credentials to authenticate to a relying party

class Login {
    async init() {
        // 1. Check that conditional mediation is supported
        await this.checkConditionalMediationSupport()
        // 2. Get challenge from server (Relying Party)
        const challenge = await this.getChallenge()
        // 3. Use existing public key credential to authenticate user
        const credentials = await this.authenticateUserWith(challenge)
        // 4. Use public key credential to login user
        // 5. Redirect to user's dashboard
    }

    ...

    async authenticateUserWith(challengeResponse) {
        const options = {
            mediation: 'conditional',
            publicKey: {
                challenge: base64url.decode(challengeResponse.challenge),
            },
        }
        const credentials = await navigator.credentials.get(options)
        return credentials
    }

}

...

The fourth function will allow the user to loginWith(credentials). This looks a little long, but all we are doing is putting together some login options that the relying party needs to verify the user. In the overview, we mentioned that during this step, we will send over the credential id, signed challenge and username. Along with those bits of information, we will also send some information about the authenticator and the client. These options are sent via the fetch API to the /login/public-key endpoint

class Login {
    async init() {
        // 1. Check that conditional mediation is supported
        await this.checkConditionalMediationSupport()
        // 2. Get challenge from server (Relying Party)
        const challenge = await this.getChallenge()
        // 3. Use existing public key credential to authenticate user
        const credentials = await this.authenticateUserWith(challenge)
        // 4. Use public key credential to login user
        const currentUser = await this.loginWith(credentials)
        // 5. Redirect to user's dashboard
    }

    ...

    buildLoginOptionsWith(userCredentials) {
        const body = {
            id: userCredentials.id,
            response: {
                clientDataJSON: base64url.encode(
                    userCredentials.response.clientDataJSON
                ),
                authenticatorData: base64url.encode(
                    userCredentials.response.authenticatorData
                ),
                signature: base64url.encode(
                    userCredentials.response.signature
                ),
                userHandle: userCredentials.response.userHandle
                    ? base64url.encode(userCredentials.response.userHandle)
                    : null,
            },
        }

        if (userCredentials.authenticatorAttachment) {
            body.authenticatorAttachment =
            userCredentials.authenticatorAttachment
        }

        return body
    }

    async loginWith(userCredentials) {
        const options = this.buildLoginOptionsWith(userCredentials)

        const response = await fetch('/login/public-key', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify(options)
        })

        return response.json()
    }

}

...

For our fifth and final function, we will use the destination from our passportCheck() step and redirect the user to the appropriate page. If the verification is successful, this page would be the dashboard. Again this function is more or less identical to the redirect function in phase 1, but we have duplicated it here for simplicity

class Login {
    async init() {
        // 1. Check that conditional mediation is supported
        await this.checkConditionalMediationSupport()
        // 2. Get challenge from server (Relying Party)
        const challenge = await this.getChallenge()
        // 3. Use existing public key credential to authenticate user
        const credentials = await this.authenticateUserWith(challenge)
        // 4. Use public key credential to login user
        const currentUser = await this.loginWith(credentials)
        // 5. Redirect to user's dashboard
        this.redirect(currentUser)
    }

    ...

    redirect(currentUser) {
        window.location.href = currentUser.destination
    }

}

With those five functions, we have successfully implemented the Login module. We can now restart our server and log in with our passkey.

Conclusion

And there you have it. In this tutorial we built an Expressjs app using passkeys to learn more about the Web Authentication API (a.k.a. Webauthn). We also used docker during local development so we could easily set up a Postgres service for our app. The finished code for this project can be found in the Github repository.

Going further

At the beginning of the article, we went over the scope of the tutorial and listed out the areas we would not be covering. If you’d like to challenge yourself to take this project further, you can consider adding

better session security
a deployment step
CI/CD pipeline
error handling, and
form data validation

Congratulations, you did great! Keep learning and keep coding. Bye for now, <3

divrhino / divrhino-passkeys-express

Create a passkey-authenticated Expressjs app with Passport.js. Video tutorial available on the Div Rhino YouTube channel.

Divrhino Passkeys Express

Text tutorial: https://divrhino.com/articles/passkeys-express
YouTube tutorial: https://youtu.be/1R68BRM5dyA

View on GitHub

Create a CRUD app with Go Fiber, docker, and Postgres

Div Rhino — Wed, 15 Feb 2023 09:02:17 +0000

Introduction

CRUD refers to the four basic operations an app should be able to perform. The acronym represents the Create, Read, Update, and Delete functions.

In the first and second parts of this tutorial series, we learnt how to Create and Read from a database. In this third instalment, we will learn how to Update and Delete data from a database.

Prerequisites

To follow along, you will need to have Docker installed and running.

You will also need the source code from the second part of the tutorial series. If you did not happen to follow that tutorial, you can download the finished code and use it as your starting point.

Remember to also set up your environment variables to avoid errors.

Revisiting fact creation

Before we start adding new functionality, let’s first improve some existing code.

In the previous tutorial, we had set up a confirmation page which we redirected to after we successfully created a new Fact. We did this to get some practice creating new handlers, now we can update this behaviour so that we are redirecting to the index page, instead.

In the handlers/facts.go file, let’s update the CreateFact handler. If we get errors when trying to parse the body of the request, we will render the new template. And if we get an error when trying to save the Fact to the database, we will also just render the new template. But if can successfully save the new Fact, then we will redirect to the index page to see our newly created Fact being listed:

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    // Parse request body
    if err := c.BodyParser(fact); err != nil {
        return NewFactView(c)
    }

    // Create fact in database
    result := database.DB.Db.Create(&fact)
    if result.Error != nil {
        return NewFactView(c)
    }

    return ListFacts(c)
}

Now that our CreateFact handler is no longer using the ConfirmationView handler, we can delete it.

Read

In the first part of this tutorial series, we already learnt how to read all our Facts from the database. Then in the second part, we built a UI to display them all. To get more practice with reading from the database, we will now learn how to Show a single fact on a page of its own.

In our browser window, we can navigate to localhost:3000/fact/1 because this will be the url format we will use to show a fact, given its id. Since we newly created a fact, we expect its id to be 1. However, this currently results in an error because we have nothing set up to support a show action.

Route

Let’s head into the routes file to add a new show route so that our app recognises this new url format. Within the body of the setupRoutes() method, we can add a new route to show a single Fact. This route will use the GET action. It will also have a :id path parameter.

package main

import (
    "github.com/divrhino/divrhino-trivia/handlers"
    "github.com/gofiber/fiber/v2"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.ListFacts)

    app.Get("/fact", handlers.NewFactView)
    app.Post("/fact", handlers.CreateFact)

    // Add new route to show single Fact, given `:id`
    app.Get("/fact/:id", handlers.ShowFact)
}

We are referencing the ShowFact handler that does not exist yet. We can create that next.

Handler

Within the handlers package, we will create the ShowFact handler that will retrieve and render a single fact.

As with all handlers in Go Fiber, we pass in the context and return an error. Then within the body of the ShowFact handler, we create a new variable called fact to store a Fact model instance to hold the fact we will retrieve.

To be able to retrieve the Fact from the database, we will need to know its id. We can use the c.Params() method to pull this information from the URL.

A typical URL will look something like this:

We use Go Fiber’s c.Params() method when we’re specifically trying to get values from the path parameters. So c.Params("id") would get us the id value from the above example URL.

Once we have the id value, we can use it to query the database and get the First record where the id matches.

Then we can use the c.Render() method to render a frontend view.

func ShowFact(c *fiber.Ctx) error {
    fact := models.Fact{}
    id := c.Params("id")

    database.DB.Db.Where("id = ?", id).First(&fact)

    return c.Render("show", fiber.Map{
        "Title": "Single Fact",
        "Fact":  fact,
    })
}

Let’s use the frontend views we already have to add a new Fact. If we try to view this newly-created Fact in our browser, we get the following error:

We can fix this by creating a new template called show:

touch views/show.html

Then within this view/show.html file, we can add some markup. The classes used in this markup correspond with styles that have been pre-prepared in the CSS. We will not go into depth about them, as CSS is out of scope for this tutorial.

<div class="container flex">
    {{if .Fact}}

    <div class="content">
        <p>{{ .Fact.Question }}</p>
        <p>{{ .Fact.Answer }}</p>
    </div>

    <div class="actions">
        <a href="" class="btn-secondary">Edit</a>
        <button class="btn-danger">Delete</button>
    </div>
    {{end}}
</div>

We use an {{if}} block to check for the presence of a Fact and conditionally display content. If a Fact exists, we will render it along with an Edit link and a Delete button. If we head into the browser now, we should be able to see our single Fact being displayed by the show template.

However, you may notice that none of our CSS styles are being applied to this page. We can quickly fix this by heading into our view/layouts/main.html file and update the link to our stylesheet so that is has a preceding / in the path. So it should look like this:

<link rel="stylesheet" href="/style/main.css?v=1">

And if your logo image is also missing, we can fix this by heading into the views/partials/header.html file and update the link to our image so that is has a preceding / in the path. So it will look like this:

<img src="/image/divrhino-logo.png" alt="">

Now when we refresh the browser, we should see our styled show page

404 page

So far things are looking promising, but what happens if we try to request a fact that does not exist? We can find out by changing the id in our path parameter to 99.

We’re getting an empty result but our show template is still rendering. This behaviour would be confusing to a user because they have no way of knowing what is going on. We can improve the experience by introducing a 404 page. If a fact does not exist, we should render this 404 page instead.

In the cmd/routes.go file, we update the ShowFact handler to return a NotFound when we are unable to retrieve the record. This NotFound handler does not exist yet, so we will need to create it.

func ShowFact(c *fiber.Ctx) error {
    fact := models.Fact{}
    id := c.Params("id")

    result := database.DB.Db.Where("id = ?", id).First(&fact)
    if result.Error != nil {
        return NotFound(c)
    }

    return c.Status(fiber.StatusOK).Render("show", fiber.Map{
        "Title": "Single Fact",
        "Fact":  fact,
    })
}

func NotFound(c *fiber.Ctx) error {
    return c.Status(fiber.StatusNotFound).SendFile("./public/404.html")
}

Then we can go and update our app config in cmd/main.go to also use the NotFound handler. Remember to place this after the other setup, but right before the call to app.Listen

package main

import (
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/handlers"
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/template/html"
)

func main() {
    database.ConnectDb()

    engine := html.New("./views", ".html")

    app := fiber.New(fiber.Config{
        Views:       engine,
        ViewsLayout: "layouts/main",
    })

    setupRoutes(app)

    app.Static("/", "./public")

    // Set up 404 page
    app.Use(handlers.NotFound)

    app.Listen(":3000")
}

If we try to test this in the browser now, we will get a blank page. We need to create a 404.html file in our public folder:

touch public/404.html

Then inside the public/404.html file, we can paste some ready-made content from the project repository.

All the markup and styles for this page will be inline. It is a good idea to keep error pages self-contained so they don’t throw additional errors if they are unable to location any external dependencies.

Now when we try to navigate to a route that does not exist, we see our new 404 page:

Our show page is done! Let’s keep going.

Update

The next CRUD action we will tackle is the Update operation. Again let’s start in our cmd/routes.go file.

If you find yourself not knowing where to start when implementing any action, try adding a new route and let the errors guide you.

In our cmd/routes.go file, we need to add 2 new routes. We need a GET route that will take us to the edit form and another PATCH route that will actually update the record we are editing. GET and PATCH are HTTP verbs (methods) and aren’t specific to Go Fiber.

package main

import (
    "github.com/divrhino/divrhino-trivia/handlers"
    "github.com/gofiber/fiber/v2"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.ListFacts)
    app.Get("/fact", handlers.NewFactView)
    app.Post("/fact", handlers.CreateFact)
    app.Get("/fact/:id", handlers.ShowFact)

    // Display `Edit` form
    app.Get("/fact/:id/edit", handlers.EditFact)
    // Update fact
    app.Patch("/fact/:id", handlers.UpdateFact)
}

Neither of these handlers exist yet, so we can head into our handlers package to create them.

Within the body of the EditFact handler, we create a new variable called fact to store the Fact we will be editing. Then we use the c.Params("id") method call to get the id value from the URL.

We query the database and retrieve the first record where the id matches. If we can’t find it, we render the NotFound handler. Otherwise, we will Render the view template and send the fact variable through to the frontend.

You may notice that this handler is very similar to the ShowFact handler, the main difference being we’re rendering different frontend views for each. If you’re feeling adventurous, you could consider DRYing out the code that is common between the two handlers.

func EditFact(c *fiber.Ctx) error {
    fact := models.Fact{}
    id := c.Params("id")

    result := database.DB.Db.Where("id = ?", id).First(&fact)
    if result.Error != nil {
        return NotFound(c)
    }

    return c.Render("edit", fiber.Map{
        "Title":    "Edit Fact",
        "Subtitle": "Edit your interesting fact",
        "Fact":     fact,
    })
}

The next handler we’ll need to create is the UpdateFact handler which will persist the changes to the database. If we run into an error during the parsing step, we will send back a Service Unavailable status. If we get an error when trying to write these changes to the database, we will redirect to the edit template

func UpdateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    id := c.Params("id")

    // Parsing the request body
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusServiceUnavailable).SendString(err.Error())
    }

    // Write updated values to the database
    result := database.DB.Db.Model(&fact).Where("id = ?", id).Updates(fact)
    if result.Error != nil {
        return EditFact(c)
    }

    return ShowFact(c)
}

Now we need a frontend view so users can edit facts in the browser.

touch views/edit.html

Inside the edit.html file, we will create a simple form with the id attribute of form-update-fact. We will also store the current fact’s id column value inside a data attribute called data-factid. These id values might be confusing, but just be mindful that the first id is an HTML attribute that can be targeted by JavaScript and the fact’s id refers to the database column.

Then within the form body, we will add an input field for question, an input field for answer, and a submit button.

<div class="container">
    <form id="form-update-fact" data-factid="{{ .Fact.ID }}">
        <label for="question">
            <span>Question</span>
            <input type="text" name="question">
        </label>

        <label for="answer">
            <span>Answer</span>
            <input type="text" name="answer">
        </label>

        <input type="submit" value="Update">
    </form>
</div>

We will also head into the show.html page and update the edit link so that its href value is pointing to our new edit route. We use {{ .Fact.ID }} to populate the id of the current fact we want to edit.

<div class="container flex">
    {{if .Fact}}

    <div class="content">
        <p>{{ .Fact.Question }}</p>
        <p>{{ .Fact.Answer }}</p>
    </div>

    <div class="actions">
        <a href="/fact/{{ .Fact.ID }}/edit" class="btn-secondary">Edit</a>
        <button class="btn-danger">Delete</button>
    </div>
    {{end}}
</div>

If we take a look at your Fact in the browser, it should be able to click on the Edit button. This will take you to the edit form

Currently, our form fields are empty. This isn’t ideal, because we should be able to see the values we want to change. We can remedy this by adding a value attribute to each of the fields. So the markup for our form should now look like this:

<div class="container">
    <form id="form-update-fact" data-factid="{{ .Fact.ID }}">
        <label for="question">
            <span>Question</span>
            <!-- Add value for Question field -->
            <input type="text" name="question" value="{{ .Fact.Question }}">
        </label>

        <label for="answer">
            <span>Answer</span>
            <!-- Add value for Answer field -->
            <input type="text" name="answer" value="{{ .Fact.Answer }}">
        </label>

        <input type="submit" value="Update">
    </form>
</div>

If we refresh our browser, the fields of our edit form should now have values

But if we try to submit the form now, it does not work as expected. Why is this?

The HTML form element only supports the GET and POST methods, and the default method for a form is GET. However, our update route is using the PATCH method to update our facts. We will need a way to send a PATCH request from our edit form. We can achieve this with a little bit of JavaScript.

Send PATCH request with JavaScript

In the previous part of this tutorial series, we added some JavaScript to our views/index.html page because this was the only place in our app that was using JavaScript. Now, we’ve reached a point where we need JavaScript on other pages too, so we will need to import our script in a place that is accessible throughout the app. Let’s move our script tag from views/index.html into our main layout. So our views/layout/main.html file should now look like this:

<!DOCTYPE html>
<head>
    <title>Div Rhino Trivia</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link rel="stylesheet" href="/style/main.css">
</head>
<body>
    ...
</body>
</html>

<!-- import javascript file -->
<script src="/javascript/app.js"></script>

By default, all our static assets are cached so that the app is faster. This makes sense when our app is in production, because we do not want to keep fetching styles and scripts that have not changed. However, this can be annoying in development, because we need to make frequent changes to our CSS and JS while we’re building things.

We can work around the caching by appending a timestamp to the end of the asset urls. This hack should only be used, locally, though. We do not want to have this behaviour in production.

First, we can add an id attribute to our <link> and <script> tags so we can target them with JavaScript. Then we can add a little snippet that will append a timestamp to the end of each path. Notice that we also removed the ?v=1 query param from the end of the stylesheet path.

<!DOCTYPE html>
<head>
    <title>Div Rhino Trivia</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">

    <!-- Add id to stylesheet link tag -->
    <link id="styles" rel="stylesheet" href="/style/main.css">
</head>
<body>
    ...
</body>
</html>

<!-- Add id to script tag -->
<script id="main" src="/javascript/app.js"></script>

<!-- For local development only: Snippet to append timestamps -->
<script>
    const timeStamp = Date.now()
    const stylesheetLink = document.querySelector("link#styles")
    const appScriptLink = document.querySelector("script#main")
    stylesheetLink.href += `?t=${timeStamp}`
    appScriptLink.src += `?t=${timeStamp}`
</script>

We have now set up everything we need for the PATCH request. Inside the public/javascript/app.js file, we can paste in a snippet. This snippet uses the JavaScript fetch API to make a PATCH request to edit the current fact. JavaScript is outside the scope of this tutorial, so we will not step through every line of this snippet, but this is what is should look like:

// Get the editForm element and the current fact ID to edit
const editForm = document.querySelector('#form-update-fact')
const factToEdit = editForm && editForm.dataset.factid

// Add an event listener to listen for the form submit
editForm && editForm.addEventListener('submit', (event) => {
    // Prevent the default behaviour of the form element
    event.preventDefault()

    // Convert form data into a JavaScript object
    const formData = Object.fromEntries(new FormData(editForm));

    return fetch(`/fact/${factToEdit}`, {
        // Use the PATCH method
        method: 'PATCH',
        headers: {
            'Content-Type': 'application/json'
        },
        // Convert the form's Object data into JSON
        body: JSON.stringify(formData),
    })
    .then(() => document.location.href=`/fact/${factToEdit}`)// Redirect to show
})

Now we can try this out in the browser. If we visit http://localhost:3000/fact/1/edit and change the values, we can submit the edit form to update the current Fact.

Delete

The last CRUD action we will implement is the delete action. First, we will head into our routes.go file to add a new delete route.

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.ListFacts)

    app.Get("/fact", handlers.NewFactView)
    app.Post("/fact", handlers.CreateFact)

    app.Get("/fact/:id", handlers.ShowFact)

    app.Get("/fact/:id/edit", handlers.EditFact)
    app.Patch("/fact/:id", handlers.UpdateFact)


    // Delete fact
    app.Delete("/fact/:id", handlers.DeleteFact)
}

We are passing in a handler called DeleteFact, but it does not exist yet, so let’s head into our handlers package to create it. First we will find the current Fact using the provided id and then we will delete the fact using the Delete() method that GORM gives us. Once we have successfully deleted the fact, we will redirect to the index page

func DeleteFact(c *fiber.Ctx) error {
    fact := models.Fact{}
    id := c.Params("id")

    result := database.DB.Db.Where("id = ?", id).Delete(&fact)
    if result.Error != nil {
        return NotFound(c)
    }

    return ListFacts(c)
}

Now that we can delete the app in the API, let’s wire up the delete button that appears on our show page. We will give it an id of delete-button so we can target it with JavaScript. Then we will also add a data-factid attribute so that we can pass the current fact’s id column value to JavaScript

<div class="container flex">
    {{if .Fact}}

    <div class="content">
        <p>{{ .Fact.Question }}</p>
        <p>{{ .Fact.Answer }}</p>
    </div>

    <div class="actions">
        <a href="/fact/{{ .Fact.ID }}/edit" class="btn-secondary">Edit</a>

        <!-- Add `id` and `data-factid` attributes to delete button -->
        <button id="delete-button" data-factid="{{ .Fact.ID }}" class="btn-danger">Delete</button>
    </div>
    {{end}}
</div>

Then in our app.js file, we can add a little snippet that sends a DELETE request. We won’t go over this snippet in too much detail, but this is the basic idea:

When the deleteButton is clicked, we ask the user to confirm their decision to delete
Then we use the fetch API to send a DELETE request to our endpoint
After the fact is successfully deleted, we redirect to the index page

// Get deleteButton element and the current fact ID to delete
const deleteButton = document.querySelector('#delete-button')
const factToDelete = deleteButton && deleteButton.dataset.factid

// Add event listener to listen for button click
deleteButton && deleteButton.addEventListener('click', () => {
    // We ask the user if they are sure they want to delete the fact
    const result = confirm("Are you sure you want to delete this fact?")

    // If the user cancels the prompt, we exit here
    if (!result) return

    // If the user confirms that they want to delete, we send a DELETE request
    // URL uses the current fact's ID
    // Lastly, we redirect to index
    return fetch(`/fact/${factToDelete}`, { method: 'DELETE' })
            .then(() => document.location.href="/")
})

If your JavaScript is not updating as expected, you may need to clear your browser’s cache or open the app in a private window.

Improving the index page

So far, we have been manually typing in the URL every time we want to see a single Fact. It would be a nicer experience if we could click on the facts from our List without having to remember the IDs for each one.

Back in browser, let’s add a few facts to work with:

Since we have a show page now, we no longer need the toggle answer button. So we can update our index.html page to remove it. The updated markup looks like this:

<div class="container">
    {{if .Facts}}
        {{ range .Facts }}
        <a href="/fact/{{ .ID }}">
            <div class="fact-item">
                <div class="question-wrapper">
                    <p class="question">{{ .Question }}</p>
                    <div class="actions">
                        {{template "icon-arrow-right" .}}
                    </div>
                </div>
            </div>
        </a>

        {{ end }}
    {{else}}
        <div class="no-facts">
            <p>No facts yet.</p>
            <p><a href="/fact">Add now</a></p>
        </div>
    {{end}}
</div>

We are using a new partial called icon-arrow-right, but it does not exist yet. So let’s create that now.

First let’s create a new folder to hold all our icon partials

mkdir views/icons

Then inside this new folder, let’s create our icon-arrow-right partial

touch views/icons/icon-arrow-right.html

Then within icon-arrow-right.html, we can paste some SVG code:

{{define "icon-arrow-right"}}
<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="#4b5366" width="24" height="24">
    <path stroke-linecap="round" stroke-linejoin="round" d="M12.75 15l3-3m0 0l-3-3m3 3h-7.5M21 12a9 9 0 11-18 0 9 9 0 0118 0z" />
</svg>
{{end}}

If we refresh our app in the browser now, this is what it should look like

Now we can click on each fact and be taken to its individual show page where we can edit and delete it, easily.

And with that, we’ve come to the end.

Conclusion

In this tutorial we implemented all the CRUD actions - Create, Read, Update and Delete. We used JavaScript’s fetch API to make PATCH and DELETE requests. We also added a little hack to bust our browser’s cache during development.

Congratulations, you did great! Keep learning and keep coding. Bye for now, <3.

divrhino / divrhino-trivia-crud

Build a CRUD app with Go Fiber, docker and Postgres.

Build a CRUD app with Go Fiber, Docker, and Postgres

Text tutorial: https://divrhino.com/articles/crud-go-fiber-docker-postgres/
Video tutorial: https://youtu.be/6dZiD5cC69Q

View on GitHub

Build a fullstack app with Go Fiber, Docker, and Postgres

Div Rhino — Sat, 19 Nov 2022 06:30:55 +0000

Originally posted on divrhino.com

In the first part of this tutorial series, we built a REST API with Go Fiber, Docker, and Postgres. We added endpoints that would allow us to create facts and list all our facts. In this second part, we’re going to convert the API into a fullstack Go Fiber app by adding some frontend views.

Prerequisites

To follow along, you will need to have Docker installed and running.

If you did not follow the first tutorial, but would like to follow along with this instalment, you can use the finished code from the first part as your starting point. You can find that in the repo for the first tutorial.

Preparing the app for templates

Out of the box, Go Fiber already has mechanisms that will allow us to add some frontend views to our app. We will need to use a template engine. Go Fiber supports several template engines. The full list of supported template engines can be found on their website. For this tutorial, we’re going to use the html template engine.

To install the necessary package, we can enter our web service container. Again, we’re assuming you have the code from the first part of this tutorial series.

docker compose run --service-ports web bash

Note:
⚠️ If you get an error about environment variables not being set, you may need to create an .env file in your repository following the instructions from the first part of this tutorial series.

Then, within the container, we can use the go get command to install the Go Fiber html template engine

go get github.com/gofiber/template/html

Next we will need to configure our Go Fiber app to handle templates correctly. We can head into the cmd/main.go file to do the necessary set up. First, we will import the html template package we just installed, and then we will use html.New() to initialise a new engine instance.

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/template/html" // 1. import
    "github.com/divrhino/divrhino-trivia/database"
)

func main() {
    database.ConnectDb()

    engine := html.New("./views", ".html") // 2. new engine

    app := fiber.New()

    setUpRoutes(app)

    app.Listen(":3000")
}

We will also have to update the app initialisation code and pass in some configuration options. In the options, we can set the Views property to the engine instance we created above.

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/template/html"
    "github.com/divrhino/divrhino-trivia/database"
)

func main() {
    database.ConnectDb()

    engine := html.New("./views", ".html")

    app := fiber.New(fiber.Config{
        Views: engine, // new config
    })

    setUpRoutes(app)

    app.Listen(":3000")
}

Now that our app is configured for templates, we can add our first frontend view.

Index view

The first view we will create will be the index view. This is where we will display the list of all our Facts.

To organise our html template files, we will need to first create a views directory

mkdir views

Then inside this directory, we can create a new index.html file

touch views/index.html

Our index view is currently empty, but we will come back to it shortly.

Now we will head into our handlers/facts.go file because we will need to update our ListFacts handler so it returns an html template. Go Fiber gives us a Render() method on the fiber.Ctx, which will allow us to render a view.

We can pass Go data to this Render() method and it returns a text/html response. For now, will pass through some string data for a Title and Subtitle field.

package handlers

import (
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/models"
    "github.com/gofiber/fiber/v2"
)

func ListFacts(c *fiber.Ctx) error {
    facts := []models.Fact{}
    database.DB.Db.Find(&facts)

    return c.Render("index", fiber.Map{
        "Title": "Div Rhino Trivia Time",
        "Subtitle": "Facts for funtimes with friends!",
    })
}

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": err.Error(),
        })
    }

    database.DB.Db.Create(&fact)

    return c.Status(200).JSON(fact)
}

These fields can be displayed in our html file using the Go template syntax i.e. double curly braces ({{ }}).

Now we can return to our views/index.html file and add the following content. Here we are using dot notation to display the Title and Subtitle fields of the current page.

<p>{{ .Title }}</p>
<p>{{ .Subtitle }}</p>

We can pass in any kind of Go data structure when rendering templates. For our Title and Subtitle, we rendered string data. Later on in the tutorial, we will see how we can loop over and render nested data.

We can already test this out in the browser. We can exit from the web service container and run our app from the host terminal using the command:

docker compose up

Our .Title and .Subtitle should be present on the page at http://localhost:3000/

Global layout

HTML pages share a lot of common elements like the <head> tag and navigation menus. Instead of rendering these elements on every page, individually, we can create a global layout. We can then use the global layout as the base for all our pages. In this section, we will update our app config settings to accomodate this.

Heading back to cmd/main.go, we need to revisit our app’s config object. We will add a ViewsLayout property and set it to "layouts/main". This will apply the main layout to all our pages, by default.

package main

import (
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/template/html"
)

func main() {
    database.ConnectDb()

    engine := html.New("./views", ".html")

    app := fiber.New(fiber.Config{
        Views: engine,
        ViewsLayout: "layouts/main", // add this to config
    })

    setUpRoutes(app)

    app.Listen(":3000")
}

This main layout file does not exist yet. So when we try to test this in the browser, we get this error:

We can resolve the error by creating the main layout. Let’s first create a new folder for our layouts.

mkdir views/layouts

Then we can create the main.html layout file here

touch views/layouts/main.html

Opening up views/layouts/main.html, we’ll add the following content.

<!DOCTYPE html>
<head>
    <title>Div Rhino Trivia</title>
</head>
<body>
    <div class="wrapper">
        <h1>{{.Title}}</h1>
        <h2>
            {{.Subtitle}}
        </h2>
        {{embed}}
    </div>
</body>
</html>

It’s just a basic HTML layout with a couple of headings. We’re rendering our .Title data within <h1> tags and our .Subtitle data within <h2> tags. Below these, we have an {{embed}}. The {{embed}} is where our individual page content will be rendered.

We have arbitrarily decided that all of our pages will have a {{.Title}} and {{.Subtitle}} field, so we can render them in the main layout, so that all our pages get the same layout. Of course, the contents of the fields will be different on each page because each page has its own context.

Remember that we already have a {{.Title}} and {{.Subtitle}} rendered in our index.html template. So if we check our browser now, we see double

To fix this, we need to remove the fields from our index.html file and replace them with some temporary text:

<!-- index.html -->

This content is unique to the index page

If we head into our browser now, we should see our main layout content has rendered along with the unique text we put in our index.html file.

Displaying all facts on the index page

In the previous step, we rendered a couple of string values in our template. It was a good demonstration of how static content can be displayed in a template. In the following section, we will build on this knowledge to be able to render dynamic content.

Let’s update the ListFacts handler so that it sends facts from the database to our frontend template. As mentioned previously, we can send any type of Go data to our templates. The facts variable holds a slice of Fact, the underlying type of a Fact is struct.

package handlers

import (
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/models"
    "github.com/gofiber/fiber/v2"
)

func ListFacts(c *fiber.Ctx) error {
    facts := []models.Fact{}
    database.DB.Db.Find(&facts)

    return c.Render("index", fiber.Map{
        "Title": "Div Rhino Trivia Time",
        "Subtitle": "Facts for funtimes with friends!",
        "Facts":    facts, // send the facts to the view
    })
}

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": err.Error(),
        })
    }

    database.DB.Db.Create(&fact)

    return c.Status(200).JSON(fact)
}

Inside the index.html view, we can range over the Facts and display each of them as follows.

<div>
    {{ range .Facts }}
    <div>
        <p>{{ .Question }}</p>
        <p>{{ .Answer }}</p>
    </div>
    {{ end }}
</div>

We should also handle scenarios where no Facts are present. We can use if / else blocks to do some conditional rendering. If there are Facts present, we will range over and render each one. If there are no Facts, we will display a message that says there are no facts with a link to /fact where we can create new facts.

<div>
    {{ if .Facts }}
        {{ range .Facts }}
        <div>
            <p>{{ .Question }}</p>
            <p>{{ .Answer }}</p>
        </div>
        {{ end }}
    {{ else }}
        <div>
            <p>No facts yet.</p>
            <p><a href="/fact">Add new</a></p>
        </div>
    {{ end }}
</div>

We’ve cleared the database, so there are currently no Facts present. Our index view currently looks like this in the browser:

The Add new link goes to a page that does not exist yet, so we will create it in the next section.

Create new fact view

The next view we will create will be the new.html view. This template will contain a form that can be used to submit new Facts.

In the views folder, we will create a new template file called new.html:

touch views/new.html

In the first part of this tutorial series, we added a route to POST new facts to the API. We now need to add a route that will GET the “new fact” template.

package main

import (
    "github.com/divrhino/divrhino-trivia/handlers"
    "github.com/gofiber/fiber/v2"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.ListFacts)

    app.Get("/fact", handlers.NewFactView) // Add new route for new view
    app.Post("/fact", handlers.CreateFact)
}

The handlers.NewFactView function doesn’t exist yet, so let’s head into the handlers/facts.go file to create it. For now, we will just add a Title and a Subtitle, we will build on it later.

package handlers

import (
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/models"
    "github.com/gofiber/fiber/v2"
)

func ListFacts(c *fiber.Ctx) error {
    facts := []models.Fact{}
    database.DB.Db.Find(&facts)

    return c.Render("index", fiber.Map{
        "Title": "Div Rhino Trivia Time",
        "Subtitle": "Facts for funtimes with friends!",
        "Facts":    facts,
    })
}

// Create new Fact View handler
func NewFactView(c *fiber.Ctx) error {
    return c.Render("new", fiber.Map{
        "Title":    "New Fact",
        "Subtitle": "Add a cool fact!",
    })
}

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": err.Error(),
        })
    }

    database.DB.Db.Create(&fact)

    return c.Status(200).JSON(fact)
}

We can test out this new template by visiting http://localhost:3000/fact in the browser.

At the moment, we only have the default content from the main layout rendering on the page because our new.html file is empty. We will work on creating our form in the next step.

Use a form to create new facts

Back in our new.html view, we will write some markup for our new form:

<form id="new-form" method="POST" action="/fact">
    <label for="question">
        <span>Question</span>
        <input id="question" type="text" name="question">
    </label>

    <label for="answer">
        <span>Answer</span>
        <input id="answer" type="text" name="answer">
    </label>

    <input type="submit" value="Submit">
</form>

Without adding anything more, we can already submit this form data. However, when we do submit a new fact, the UX of it is a little awkward, as it redirects to the JSON response (in Firefox, anyway). We can fix that by redirecting to a success or confirmation page.

Success page

Our success or confirmation view will show us a message to confirm that our form data has been submitted successfully.

Let’s create a new template called confirmation.html:

touch views/confirmation.html

This confirmation.html file will contain a link that takes users back to the fact creation page

<div class="add-more">
    <p><a href="/fact">Add more</a></p>
</div>

Then we can head into the handlers/facts.go file to add a handler for our confirmation view. We won’t add a route for this handler because we only need to use it within our CreateFact() handler.

We will also update the CreateFact() handler to return our new ConfirmationView:

package handlers

import (
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/models"
    "github.com/gofiber/fiber/v2"
)

func ListFacts(c *fiber.Ctx) error {
    facts := []models.Fact{}
    database.DB.Db.Find(&facts)

    return c.Render("index", fiber.Map{
        "Title": "Div Rhino Trivia Time",
        "Subtitle": "Facts for funtimes with friends!",
        "Facts":    facts,
    })
}

func NewFactView(c *fiber.Ctx) error {
    return c.Render("new", fiber.Map{
        "Title":    "New Fact",
        "Subtitle": "Add a cool fact!",
    })
}

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": err.Error(),
        })
    }

    database.DB.Db.Create(&fact)

    return ConfirmationView(c) // 2. Return confirmation view
}

// 1. New Confirmation view
func ConfirmationView(c *fiber.Ctx) error {
    return c.Render("confirmation", fiber.Map{
        "Title":    "Fact added successfully",
        "Subtitle": "Add more wonderful facts to the list!",
    })
}

We can head into the browser to test out the confirmation page. Once we’ve submitted a form, we will be redirected to our confirmation page rather than the JSON response

Partials

There are several ways we can make our markup more reusable. Previously, we saw how to create a layout to share common structural page elements between our views. Another technique would be to make use of partials.

We need a header, but we don’t want to copy/paste the header markup for every one of our views. So let’s put the header in a partial. We can start by creating a views/partials folder

mkdir views/partials

Then add a new header.html file to it:

touch views/partials/header.html

Inside header.html, we can define our partial

{{define "header"}}
<header>
    <div>
        <a href="/">
            <img src="image/divrhino-logo.png">
        </a>
    </div>
    <div class="">
        <a href="/fact">
            <span>+ New Fact</span>
        </a>
    </div>
</header>
{{end}}

We can now use our header partial in views/layouts/main.html. The . means we’re passing through the current context

{{define "main"}}
<!DOCTYPE html>
<head>
    <title>Div Rhino Trivia</title>
</head>
<body>
    <div class="wrapper">
        {{template "header" .}} <!-- import partial -->

        <h1>{{.Title}}</h1>
        <h2>
            {{.Subtitle}}
        </h2>
        {{embed}}
    </div>
</body>
</html>

If we look at our header partial, we will notice that we are importing a logo image that doesn’t exist yet. And while the image is not present, our app doesn’t know how to handle images yet, either. In the next section, we’ll configure our app to handle static assets.

Serving static assets

Although our app is functional, it isn’t much to look at. We can improve that with some static assets such as images, CSS or a bit of JavaScript.

Go Fiber has methods to help us achieve this fairly easily. Let’s head back into our main.go file and use the app.Static method to tell our app where to locate our static assets. In our case, we will be putting our images, CSS and JS in a folder called public. We have chosen to call it public because its somewhat of a convention, but you can call this folder anything you like.

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/template/html"
    "github.com/divrhino/divrhino-trivia/database"
)

func main() {
    database.ConnectDb()

    engine := html.New("./views", ".html")

    app := fiber.New(fiber.Config{
        Views:       engine,
        ViewsLayout: "layouts/main",
    })

    setUpRoutes(app)

    app.Static("/", "./public")

    app.Listen(":3000")
}

The public folder does not exist yet, so let’s create it in the root of our project

mkdir public

Now we can move on and adding our logo image.

Images

To keep our images organised, we will create a folder in our public directory called image:

mkdir public/image

We can put our logo image file into this folder.

Our logo image is now present in the header on our index page. If we visit other pages, the header partial is also present.

Linking stylesheets

Apart from images, we can also add some CSS stylesheets.

To keep our stylesheets organised, let’s create a folder in our public directory called style:

mkdir public/style

Then we can create a stylesheet within it:

touch public/style/main.css

We don’t have anything in our main.css file yet, but let’s link it in our head tag first and add some styles later. In the main layout, we can add this link tag to the <head>. We’ve added a query param of v=1 after the href value as a rudimentary way to bust the cache while we’re developing, locally. I’m sure there are other ways to do this.

<link rel="stylesheet" href="style/main.css?v=1">

Let’s also add the viewport meta tag to the <head> so our app can be responsive and display correctly on mobile devices:

<meta name="viewport" content="width=device-width,initial-scale=1">

The index.html template should look like this:

<!DOCTYPE html>
<head>
    <title>Div Rhino Trivia</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link rel="stylesheet" href="style/main.css?=1">
</head>
<body>
    <div class="wrapper">
        {{template "header" .}}

        <h1>{{.Title}}</h1>
        <h2>
            {{.Subtitle}}
        </h2>
        {{embed}}
    </div>
</body>
</html>

Inside our main.css file, we can change the body element’s background colour

body {
    background-color: aquamarine;
}

We will have to update our markup and paste in some pre-prepared styles. We will not go over the specifics here, as this is not a CSS tutorial. But the completed markup and stylesheet can be found in the Github repo.

JavaScript

We’ve added a bit more markup to make the layout a little more interesting. We’ve added a little toggle button next to each of our facts.

Right now, they are not functional. Let’s add some javascript to make these buttons interactive.

To keep our JavaScript organised, let’s create a folder in our public directory called javascript:

mkdir public/javascript

Then we can create a JavaScript file within it:

touch public/javascript/app.js

We don’t have anything in our app.js file yet, but let’s link it in our head tag first and add our JS later. In the index.html view, we can import this <script> tag at the bottom of the file.

<div class="container">
    {{if .Facts}}
        {{ range .Facts }}
            <div class="fact-item">
                <div class="question-wrapper">
                    <p class="question">{{ .Question }}</p>
                    <button id="answer-toggle" class="answer-toggle">Toggle answer</button>
                </div>
                <div class="answer-wrapper">
                    <p class="answer">{{.Answer}}</p>
                </div>
            </div>
        {{ end }}
    {{else}}
        <div class="no-facts">
            <p>No facts yet.</p>
            <p><a href="/fact">Add new</a></p>
        </div>
    {{end}}
</div>

<script src="/javascript/app.js"></script>

We will also paste in some pre-prepared JS. Again, the final javascript file can be found in the Github repo.

And with that, we’ve come to the end.

Conclusion

In this tutorial we learnt how to add a form and other frontend views to a Go Fiber app that was build from scratch using Go and Docker. We started with a simple API and extended it to include templates and static assets.

Congratulations, you did great! Keep learning and keep coding. Bye for now, <3.

divrhino / divrhino-trivia-fullstack

Convert a REST API into a fullstack Go Fiber app by adding some frontend views.

Build a fullstack app with Go Fiber, Docker, and Postgres

Text tutorial: https://divrhino.com/articles/full-stack-go-fiber-with-docker-postgres/
Video tutorial: https://youtu.be/B7hSjNbcVYM

View on GitHub

Build a REST API from scratch with Go, Docker & Postgres

Div Rhino — Mon, 17 Oct 2022 06:46:12 +0000

Originally posted on divrhino.com

In this tutorial, we will learn how to create a simple trivia REST API from scratch, using Go and Docker. We will start with an empty folder and build on it as we go. We won’t need to have Go installed on our machine beforehand, which is arguably the biggest benefit of this approach. At the end of the tutorial, we will have a little Go Fiber app connected to a Postgres database.

Prerequisites

To follow along, you will need to have Docker installed. You can head to their download page to find the version that suits your environment.

Creating a new project

In the terminal, we can change into the directory where our projects are stored. In my case this would be the Sites folder, it may be different for you. Then create a new directory for our REST API project. Then we can immediately change into this new directory



mkdir divrhino-trivia
cd divrhino-trivia

Getting started with Docker

Our project folder is currently empty. Since we’re using Docker to create an app from scratch, the first file we will add is the Dockerfile. Then we will also add a docker-compose.yml file because we will need to manage multiple containers.



touch Dockerfile
touch docker-compose.yml

Dockerfile

First we will look at the Dockerfile.

To begin, we want to start building our own container FROM the official golang image. And we want to use a specific version of the image. In this tutorial, we will use version 1.19.0. Specifying the version will ensure all our dev environments are the same. Think of this step like we’re installing Go onto our machine.

The only other thing we will do in this step is to specify our working directory. Docker containers run on Linux, so here we’re saying that we want our app to live in the /usr/src directory of the Linux filesystem in a project folder we want to be named app.



FROM golang:1.19.0

WORKDIR /usr/src/app

We will continue building on our Dockerfile, but this is good enough to get us started. Let’s move on to our initial docker-compose.yml configuration.

docker-compose.yml

If our app required only one service, we would be able to make do with having just the Dockerfile above. However, we’re eventually going to add a Postgres service as well. So it would be handy to have a docker-compose.yml file to help us manage our container collection.

We’re starting off with a web service. We’ve called it web because this is the container where we will have our Go Fiber web app. Below is a brief explanation of each of the fields:

Field name	Description
`build`	This is the path to the Dockerfile for the service. We use . because the Dockerfile for our web service can be found in the same folder as our docker-compose.yml file.
`ports`	This is were we map the port from the container with our host computer. Our web service will be running on port 3000.
`volumes`	Volumes are used to persist data generated by the service. We want to persist our web service data inside our project directory, i.e. /usr/src/app.



version: '3.8'

services:
  web:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app

And with that, we can head into our terminal and run our app using the following command:



docker compose up

Because it’s our first time running our service, this command will pull the images we need. You will notice that the logs in our console correspond with the commands in our Dockerfile. Let’s kill our container now and go have a little look inside.

We can enter our container by running the following command. Here we’re saying we want to open the bash within our web service:



docker compose run --service-ports web bash

Now inside our container, we can run commands like any other terminal. Let’s check our Go version:



go version

We’ve successfully put together the most basic Docker setup needed to start developing with Go. In the next part of the tutorial, we will install the Go Fiber framework.

Dockerfile vs. docker-compose

Before we go any further, let’s take a little detour to briefly discuss why we have both a Dockerfile and a docker-compose.yml file.

A Dockerfile is a set of commands that will be run to set up a container. Sometimes, it can help to think of it as the list of commands you’d need to run when setting up a brand new computer for development in a particular technology. In the case of this particular tutorial, we can think of it as the list of commands we’d need to run in order to set up a brand new computer for Go development. A project can have one or more Dockerfiles.

The docker-compose.yml file is a configuration file that will allow us to manage all our different containers. As we mentioned above, a project can have one or more Dockerfiles, which means it can be made up of one or more containers. The docker-compose.yml file can be thought of as a single project manager for all these containers.

Installing Go Fiber

Now that we have a functioning containerised Go environment, we can start installing the necessary packages inside it. We’ve decided that Go Fiber will be our framework of choice, however, you can install other packages in much same way.

Using the following command, we can enter the container for our web service:



docker compose run --service-ports web bash

Before we start installing all the packages, we’ll need to initialise Go Modules to manage our dependencies.

It is usually a good idea to name your project using the URL where it can be downloaded. I will use my Github repo URL as the name of my project. But please feel free to substitute the following command with your own Github or website



go mod init github.com/divrhino/divrhino-trivia

With Go Modules set up, we can now install the Go Fiber framework:



go get github.com/gofiber/fiber/v2

Hello World

Go Fiber provides a Hello World example in their docs. We can use that as a starting point for our own app. While we’re still in the container, let’s create our cmd folder and our main.go file:



mkdir cmd
touch cmd/main.go

Then we can add the example code to our cmd/main.go file. We’ve modified the string, but the rest of the code is the same as the example from the Go Fiber docs



package main

import "github.com/gofiber/fiber/v2"

func main() {
    app := fiber.New()

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Hello, Div Rhino!")
    })

    app.Listen(":3000")
}

Still in the container, we can start our Go Fiber web server using the go run command and binding to localhost



go run cmd/main.go -b 0.0.0.0

We can visit http://localhost:3000/ in the browser to see our app running and our printed-out string.

Starting your app from host machine

At the moment, every time we want to start our gofiber app, we need to enter our web service container to run the command. It would be nice if we could just run our dockerised app from our host machine.

We will need to make a few changes to our Dockerfile and docker-compose.yml files to achieve this.

Updating Dockerfile

In our Dockerfile, we will add 2 new lines. First, we will use the COPY instruction to copy all the files into the container’s working directory. Then we will run the command go mod tidy to install and clean up our dependencies



FROM golang:1.19.0

WORKDIR /usr/src/app

COPY . .
RUN go mod tidy

docker-compose.yml

In our docker-compose.yml configuration, we just need to add the command we want to map to docker compose up. You may notice that it is the same command we use inside the web service container



version: '3.8'

services:
  web:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
    command: go run cmd/main.go -b 0.0.0.0

If our container is still running, we can shut it down. Then in our terminal, we can now run our app using the following command:



docker compose up

We can head to our browser to see that our app is running. Now let’s update our cmd/main.go file. When we refresh the browser, our changes aren’t reflected. This is because we’ll need to rebuild our app first before we can see the changes.

Hot reloading

It would be great if we had a mechanism to rebuild our app whenever we made changes to the code. We can use a package called air to help us achieve this.

We’ll want to install it as part of our docker setup, so let’s open up our Dockerfile again and add a RUN instruction to install the air package.

Your Dockerfile should now look like this:



FROM golang:1.19.0

WORKDIR /usr/src/app

RUN go install github.com/cosmtrek/air@latest

COPY . .
RUN go mod tidy

Add .air.toml

We also need to add a configuration file for the air package. First we can create a new dotfile called .air.toml



touch .air.toml

Then we can head to the air package’s Github repo to copy the sample config file from there. We will only need to change the command under [build] so that it is pointing to our cmd directory. This is where our main.go file lives.



[build]
# Just plain old shell command. You could use `make` as well.
cmd = "go build -o ./tmp/main ./cmd"

Update command in docker-compose.yml

Then we have to change the command in our docker-compose.yml file so that it uses the air to run our app.

Our web service in the docker-compose.yml file should currently look like this:



version: '3.8'

services:
  web:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
    command: air ./cmd/main.go -b 0.0.0.0

Now rebuild our container



docker compose build

And run the app



docker compose up

Now if we make changes to the cmd/main.go file, we can refresh our browser and see our changes.

Environment variables

It is good practice to keep all our sensitive keys in a .env file rather than checking them in to version control. We can use docker-compose.yml file to read in our environment variables without installing any additional packages. Under the web service config, we can add the env_file key and point it to our .env file



version: '3.8'

services:
  web:
    build: .
    env_file:
      - .env
    ports:
      - "3000:3000"
    volumes:
            - .:/usr/src/app
    command: air ./cmd/main.go -b 0.0.0.0

You may have noticed that this file does not exist yet, so let’s create it in your project’s root directory:



touch .env

And that’s all we need in order to use environment variables.

Adding Postgres

In the next few sections of the tutorial, we will be setting up Postgres as our database of choice. We will have to set up a second service for it, so let’s head straight back into our docker-compose.yml file and add a new db: key under services.



version: '3.8'

services:
  web:
    build: .
    env_file:
      - .env
    ports:
      - "3000:3000"
        volumes:
              - .:/usr/src/app
    command: air ./cmd/main.go -b 0.0.0.0
  db:

Then under this db: key, we’ll have to add a few more fields

Field name	Description
`image`	We will use the postgres:alpine image that is available directly from docker hub. Since we won’t be adding any additional instructions, we don’t need an accompanying standalone Dockerfile for the db service.
`ports`	This is where we map the port from the container with our host computer. Our db service will be running on port 5432, which is the usual port for Postrgres.
`volumes`	Volumes are used to persist data generated by the service. We want to persist our db service in postgres-db:/var/lib/postgresql/data.



version: '3.8'

services:
  web:
    build: .
    env_file:
      - .env
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
    command: air ./cmd/main.go -b 0.0.0.0
  db:
    image: postgres:alpine
    ports:
      - "5432:5432"
    volumes:
      - postgres-db:/var/lib/postgresql/data

volumes:
  postgres-db:

Named volumes

You may have noticed that we have added another volumes key, with an empty postgres-db field, on a line of its own. We then used postgres-db in our db service config. This type of volumes config is know as a named volume.

Named volumes persist data even after a container is restarted or removed. The data here will also be accessible to other containers. The path to the actual volume is handled by docker internals. Volumes that are defined in this way would need to be removed manually.

This makes sense for a database because we’d, ideally, like to persist our data even after we’ve shut down everything and gone to bed.

Database credentials

In the next step of our Postgres-related work, we need to store our database credentials (i.e. DB_USER, DB_PASSWORD and DB_NAME somewhere safe. We don’t want to be pushing these values into our version control, so let’s keep them in the .env file



DB_USER=divrhinotrivia
DB_PASSWORD=divrhinotrivia
DB_NAME=divrhinotrivia

And now we can access them in our docker-compose.yml file:



version: '3.8'

services:
  web:
    build: .
    env_file:
      - .env
    ports:
      - "3000:3000"
    volumes:
      - .:/usr/src/app
    command: air ./cmd/main.go -b 0.0.0.0
  db:
    image: postgres:alpine
    environment:
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=${DB_NAME}
    ports:
      - "5432:5432"
    volumes:
      - postgres-db:/var/lib/postgresql/data

volumes:
  postgres-db:

Now that we’ve configured our db service to use Postgres, we need to start communicating with it.

Communicating with the database using GORM

For a small app, we could get away with using vanilla SQL queries to communicate with our database. However, for the purposes of learning more about using “models” to represent database entities, let’s use an ORM library called GORM.

ORM or Object Relational Mapping is a technique that will allow us to query and manipulate data from a database in an object-oriented way. In our case, Go structs will be the “objects” that represent our database entities.

To install GORM, we will enter our web service container



docker compose run --service-ports web bash

And run the following command



go get gorm.io/gorm

While we’re here, we should also install the postgres driver for GORM



go get gorm.io/driver/postgres

Now we’re ready to start setting up a GORM model.

GORM model

As we mentioned in the introduction to this tutorial, we are building a trivia app. So we’ll have to stores Facts in our database within a table of the same name (i.e. a facts table). We will create a Go struct to represent our Facts and then we’ll use GORM to translate our struct into a database entity.

First, let’s create a models folder in our project root directory. This is where we will store our GORM models



mkdir models

And let’s add a models.go file to the models folder



touch models/models.go

Inside models/models.go, let’s add our Fact model. A Fact will have a Question, which will be of type string and an Answer, which will also be of type string. At the very top of the struct body, we’ll indicate that it is a gorm.Model. So we should also make sure we’re importing the gorm package



package models

import "gorm.io/gorm"

type Fact struct {
    gorm.Model
    Question string
    Answer   string
}

Struct tags

Struct tags are small pieces of metadata attached to fields of a struct. They are used to provide instructions to other Go code, communicating how to work with the struct fields.

In the following snippet, we’re using the json keyword in our struct tags to describe the corresponding JSON keys we’d like to associate to each field of the struct.

Any of our Go code that works with JSON will see these struct tags and understand that:

the Question field is represented by the question key in JSON, and
the Answer field is represented by the answer key in JSON

JSON keys follow some naming conventions. For instance, JSON keys are usually lowercase and may also be snake_case in some cases. This is why we will often see this sort of “mapping” in codebases that use a lot of JSON, e.g. APIs and web apps.



package models

import "gorm.io/gorm"

type Fact struct {
    gorm.Model
    Question string `json:"question"`
    Answer   string `json:"answer"`
}

Now let’s also add some struct tags for GORM to use. We will use the gorm keyword to specify some initial database rules for each field.

We’re telling GORM the following:

In the database, both the question and answer columns will be of type TEXT
In the database, neither column should be allowed to have NULL values
We also set the default value for each column to NULL so that we can return an error if the user does not provide their own values when they create a new Fact



package models

import "gorm.io/gorm"

type Fact struct {
    gorm.Model
    Question string `json:"question" gorm:"text;not null;default:null`
    Answer   string `json:"answer" gorm:"text;not null;default:null`
}

Pay attention to the syntax of the struct tag. We can use more than one keyword type within the same struct tag, but everything is encapsulated between backticks.

Now we’re ready to move on and make a database connection.

Create a database connection

We need to open a connection to the database so that we can read and write new Facts. Let’s start by creating a new database directory



mkdir database

And within the database directory, we’ll create a new file



touch database/database.go

Inside database/database.go file, we indicate that this file belongs to the database package. Then we import the GORM package and set up a custom struct type called Dbinstance to represent our database instance.

We will also create a new package-level variable to hold our global database. This variable will have the name of DB with a type of Dbinstance. We put this on the package level because we need to access it everywhere in our app.



package database

import "gorm.io/gorm"

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

Now let’s create a function called ConnectDb(). As the name implies, this is the function we will use to connect our app to the database.



package database

import "gorm.io/gorm"

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {}

Inside the body of ConnectDb(), we will use a GORM method called Open(). gorm.Open() takes 2 arguments. The first argument is of type gorm.Dialector and the second argument is of type gorm.Options



package database

import "gorm.io/gorm"

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    gorm.Open(gorm_dialector, gorm_options)
}

First argument: gorm_dialector

Let’s start putting together what we need for the first argument, which is the dialector. Since we’re using postgres, we need to import the driver package.

Then we will call the postgres.Open() method which takes one argument, which is a DSN (data source name) string.

To construct the DSN string, we need to import the fmt and os packages. We will use the os.Getenv method to access the environment variables we had set up in our docker-compose.yml file earlier. And we will use the fmt.Sprintf() method to interpolate the string with the relevant variables.



package database

import (
    "fmt"
    "os"

    "gorm.io/driver/postgres"
    "gorm.io/gorm"
)

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    dsn := fmt.Sprintf(
        "host=db user=%s password=%s dbname=%s port=5432 sslmode=disable TimeZone=Asia/Shanghai",
        os.Getenv("DB_USER"),
        os.Getenv("DB_PASSWORD"),
        os.Getenv("DB_NAME"),
    )

    gorm.Open(postgres.Open(dsn), gorm_options)
}

Second argument: gorm_options

The second argument to gorm.Open is a GORM config object. In our config, we set the kind of logger we want to use. We should also remember to import the gorm/logger package



package database

import (
    "fmt"
    "os"

    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "gorm.io/gorm/logger"
)

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    dsn := fmt.Sprintf(
        "host=db user=%s password=%s dbname=%s port=5432 sslmode=disable TimeZone=Asia/Shanghai",
        os.Getenv("DB_USER"),
        os.Getenv("DB_PASSWORD"),
        os.Getenv("DB_NAME"),
    )

    gorm.Open(postgres.Open(dsn), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Info),
    })
}

The gorm.Open() method returns a database and an error. So let’s do some quick error handling before we move on.

If we get an error here, we want to log a fatal error and exit. If the database can’t connect, it’s a bit of a non-starter, so I think it’s okay to use log.Fatal() here. We also exit with the exit code of 2, because our operation did not complete successfully.



package database

import (
    "fmt"
    "log"
    "os"

    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "gorm.io/gorm/logger"
)

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    dsn := fmt.Sprintf(
        "host=db user=%s password=%s dbname=%s port=5432 sslmode=disable TimeZone=Asia/Shanghai",
        os.Getenv("DB_USER"),
        os.Getenv("DB_PASSWORD"),
        os.Getenv("DB_NAME"),
    )

    db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Info),
    })

    if err != nil {
        log.Fatal("Failed to connect to database. \n", err)
        os.Exit(2)
    }
}

But if there are no errors, we log a message that says we are connected and we set the Logger value for our db



package database

import (
    "fmt"
    "log"
    "os"

    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "gorm.io/gorm/logger"
)

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    dsn := fmt.Sprintf(
        "host=db user=%s password=%s dbname=%s port=5432 sslmode=disable TimeZone=Asia/Shanghai",
        os.Getenv("DB_USER"),
        os.Getenv("DB_PASSWORD"),
        os.Getenv("DB_NAME"),
    )

    db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Info),
    })

    if err != nil {
        log.Fatal("Failed to connect to database. \n", err)
        os.Exit(1)
    }

    log.Println("connected")
    db.Logger = logger.Default.LogMode(logger.Info)
}

Next we want to use AutoMigrate to create the tables that we need. We pass all our GORM models to AutoMigrate. In this tutorial, we only have one GORM model, which is the Facts model.



package database

import (
    "fmt"
    "log"
    "os"

    "github.com/divrhino/divrhino-trivia/models"
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "gorm.io/gorm/logger"
)

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    dsn := fmt.Sprintf(
        "host=db user=%s password=%s dbname=%s port=5432 sslmode=disable TimeZone=Asia/Shanghai",
        os.Getenv("DB_USER"),
        os.Getenv("DB_PASSWORD"),
        os.Getenv("DB_NAME"),
    )

    db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Info),
    })

    if err != nil {
        log.Fatal("Failed to connect to database. \n", err)
        os.Exit(1)
    }

    log.Println("connected")
    db.Logger = logger.Default.LogMode(logger.Info)

    log.Println("running migrations")
    db.AutoMigrate(&models.Fact{})
}

Lastly, we set the value of our global DB variable to the database we just set up.



package database

import (
    "fmt"
    "log"
    "os"

    "github.com/divrhino/divrhino-trivia/models"
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
    "gorm.io/gorm/logger"
)

type Dbinstance struct {
    Db *gorm.DB
}

var DB Dbinstance

func ConnectDb() {
    dsn := fmt.Sprintf(
        "host=db user=%s password=%s dbname=%s port=5432 sslmode=disable TimeZone=Asia/Shanghai",
        os.Getenv("DB_USER"),
        os.Getenv("DB_PASSWORD"),
        os.Getenv("DB_NAME"),
    )

    db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
        Logger: logger.Default.LogMode(logger.Info),
    })

    if err != nil {
        log.Fatal("Failed to connect to database. \n", err)
        os.Exit(2)
    }

    log.Println("connected")
    db.Logger = logger.Default.LogMode(logger.Info)

    log.Println("running migrations")
    db.AutoMigrate(&models.Fact{})

    DB = Dbinstance{
        Db: db,
    }
}

We will open the database connection in func main() so that the database is accessible throughout the app:



// cmd/main.go

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/database"
)

func main() {
    database.ConnectDb()

    app := fiber.New()

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Div Rhino Trivia App!")
    })

    app.Listen(":3000")
}

Routes & Endpoints

Our app is now able to connect to our database. Now we can move on and set up some endpoints.

To set up our first endpoint, we can head into out cmd/main.go file. Here, we can call the setupRoutes() function right after where our new gofiber app is created.



package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/database"
)

func main() {
    database.ConnectDb()
    app := fiber.New()

    setupRoutes(app)

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Div Rhino Trivia App!")
    })

    app.Listen(":3000")
}

This setupRoutes() function doesn’t exist yet, so let’s create it now. We can put this function in its own file, but it will belong to the main package. Let’s create a new file to hold all our routes



touch cmd/routes.go

We can move our existing routes into cmd/routes.go, and clean up the cmd/main.go file



// cmd/routes.go

package main

import (
    "github.com/gofiber/fiber/v2"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Div Rhino Trivia App!")
    })

}



// cmd/main.go

package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/database"
)

func main() {
    database.ConnectDb()
    app := fiber.New()

    setupRoutes(app)

    app.Listen(":3000")
}

We can further clean this up by moving the handler code into a separate package.

Handler: home

To keep things organised, we can make a new handlers folder to hold the handler code



mkdir handlers

Then create a new file for all the handlers that are related to facts



touch mkdir handlers/facts.go

We can move our existing handler into handlers/facts.go



package handlers

import "github.com/gofiber/fiber/v2"

func Home(c *fiber.Ctx) error {
    return c.SendString("Div Rhino Trivia App!")
}

Our routes.go file should look like this:



package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/handlers"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.Home)
}

Now let’s open our API client to test this home route. We are using Insomnia in this tutorial, but you can use your client of choice. We should see our string in the response

Creating Facts

Now that we understand how to set up endpoints, let’s go ahead and add an endpoint that we can use to create new facts. It will make a POST request



package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/handlers"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.Home)

    app.Post("/fact", handlers.CreateFact)
}

The handlers.CreateFact handler doesn’t exist yet, so let’s create that now:



package handlers

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/models"
)

func Home(c *fiber.Ctx) error {
    return c.SendString("Div Rhino Trivia App!")
}

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": err.Error(),
        })
    }

    database.DB.Db.Create(&fact)

    return c.Status(200).JSON(fact)
}

Use Insomnia to test this endpoint

Listing Facts

Now that we can create new facts, we should also have a way to list all our facts. Let’s update the home route and adapt it to ListFacts



package main

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/handlers"
)

func setupRoutes(app *fiber.App) {
    app.Get("/", handlers.ListFacts)

    app.Post("/fact", handlers.CreateFact)
}

Then let’s also change the home handler to be ListFacts



package handlers

import (
    "github.com/gofiber/fiber/v2"
    "github.com/divrhino/divrhino-trivia/database"
    "github.com/divrhino/divrhino-trivia/models"
)

func ListFacts(c *fiber.Ctx) error {
    facts := []models.Fact{}
    database.DB.Db.Find(&facts)

    return c.Status(200).JSON(facts)
}

func CreateFact(c *fiber.Ctx) error {
    fact := new(models.Fact)
    if err := c.BodyParser(fact); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": err.Error(),
        })
    }

    database.DB.Db.Create(&fact)

    return c.Status(200).JSON(fact)
}

We can use Insomnia get a list of all our facts. Currently we only have one. Let’s add a few more and get the list with all the new facts.

Conclusion

And there you have it. In this tutorial we learnt how to create a simple trivia app from scratch using Go and Docker. We started with a blank folder and worked our way up to creating a multi-container app with a Postgres database.

If you enjoyed this article and you'd like more, consider subscribing to Div Rhino on YouTube.

Congratulations, you did great! Keep learning and keep coding. Bye for now, <3

divrhino / divrhino-trivia

Create a simple trivia REST API from scratch, using Go, Docker & Postgres. Video tutorial available on the Div Rhino YouTube channel.

Build a REST API from scratch with Go and Docker

Text tutorial: https://divrhino.com/articles/rest-api-docker-go-fiber-from-scratch
Video tutorial: https://youtu.be/p08c0-99SyU

View on GitHub

A gentle introduction to interactive git rebase

Div Rhino — Thu, 07 Apr 2022 08:33:57 +0000

Introduction

It may be fair to assume that almost every development team uses git for version control and to collaborate on code. Incorporating git into your workflow for the first time can be quite daunting. Another related challenge is trying to get your head around rebasing. In this tutorial, we will learn a practical beginner’s workflow that utilises interactive rebasing.

Prerequisites

To follow along, you will need to have git installed. You can head to their website and download the version that suits your environment.

If you’re unfamiliar with vim commands, you can check out the little table of commands at the end of this article to get a small crash course.

Thoughts about pull requests

Before we dive into the meat of this walkthrough, let’s take a moment to talk about pull requests. I find it helpful to think about a pull request as if I were documenting a potion recipe that I’d like to eventually pass on to my future self.

Having this mental model helps me organise my commits to tell a coherent story. It helps me decide the sequence and composition of my commits.

Creating a new branch

Beginning a new task means we’ll need to create a new branch. At this point, we can pick our base branch. By default, our base branch would either be the main branch or the develop branch. Occasionally, we may choose another feature branch as our base. Regardless of the base branch we choose, we’ll want to make sure it’s up-to-date before creating our new branch.

For this example, we have decided to base our work off of develop. We can checkout develop and once we’re on develop, we can pull the latest changes:

git checkout develop
git pull --rebase --autostash

The --rebase and --autostash flags are a nice way to avoid conflicts when pulling changes. If you have uncommitted local changes, they are stashed before the rebase.

💡 I personally use the git pull --rebase --autostash command a lot, so I have saved it as an alias called git up. If you’d also like to do this, you can run this in your terminal:
git config --global alias.up 'pull --rebase --autostash'

Once our chosen base branch has the latest changes, we can create our new branch:

git checkout -b feature/potion-for-posterity

Writing commit messages

As we happily code away, we should remember to periodically commit our changes to avoid unexpectedly losing our work. It is a good idea to make small, but frequent, commits. It is also a good idea to group relevant changes under a commit message that is descriptive and applicable. Remember, these are messages we are writing for our future selves, so they should act as documentation.

Calling back to our potion analogy, our commit messages should read like simple instructions for a recipe or a to-do list. Some examples of commit messages could be:

Add Posterity model
Add Posterity controller
Add migration for Posterity
Add tests for Posterity

We can add and commit our changes by running the following commands and writing our message in the resulting editor:

git add .
git commit

Git allows us to write our commit messages inline by using the -m flag. However, writing our commit message in the editor lets us see when our message has gotten too long. In the screenshot below, we can see that the commit message text has changed colour to let us know we’re exceeded the recommended character limit. There is no rule that requires us to adhere to this character limit, it’s just a personal preference.

💡 Having this length constraint means we won’t have to expand it to see the full message on Github. It’s not a big deal, though, just a matter of preference.

Handling changes coming into the base branch

It is normal to have teammates merging their work into the base branch throughout the course of a work day, especially if the base branch is develop. To avoid potential conflicts later on, we’ll want to make sure we’re keeping up-to-date with the latest changes when we can. It is a good idea to periodically reset our base branch.

To do this, we can commit or stash the changes on our current branch. We can push these changes up to Github just to be extra safe. Once the changes are safe and out of the way, we can checkout our base branch and pull the latest changes. As a reminder, our base branch is develop for this example, but it can be any other arbitrary branch as well.

git checkout develop
git up # alias for `git pull --rebase --autostash`

Once our base branch has all the latest changes, we can switch back to our working branch and rebase against our base branch

git checkout feature/potion-for-posterity
git rebase origin/develop

Interactive rebase

When we use rebase, we can think of it as resetting our base branch. Rebasing helps us keep our commit history clean. As we talked about in the beginning, a PR is a recipe for a potion we’d like to pass along to our future selves or to other developer friends. So it should read like documentation.

A rebase can be done manually or interactively. I personally find that having a visual guide is helpful, so I usually opt to use interactive rebase. In this section, we’ll walk though some scenarios where we may want to “rewrite history”. We won’t go over every single action, just the ones we’ll find ourselves using most frequently.

Squash

There will be cases where we may want to squash several commits together. For instance, we may find that we have a handful of small commits that don’t really tell a full story on their own and could work better as part of a single, larger, commit.

In the animated GIF example, below, we have 3 commits we would like to act on. So we can kick off an interactive rebase by running:

git rebase --interactive HEAD~3

Executing this will take us into the editor, where we can choose which commits we’d like to squash. We can mark a commit for squashing by changing the pick into a squash.

When we save (using :wq if your editor is vim), we are taken into another editor interface where we can choose the message we’d like to use for our newly combined commit. In the example, we’ve chosen to use the second commit message, so we delete all the other ones. Once we’re happy with the commit message, we can save and exit out of the editor.

Fixup

There are some cases where we’d like to combine commits and keep the message of the original commit. A fixup is similar to a squash, but we just use the commit message of the original commit and discard the messages from any commits marked for fixup.

In the animated GIF example, below, we want to combine the last two commits with the first one. This first commit can be thought of as the original commit we want to fuse the other commits into. We also want to keep the first commit’s message and discard the other messages. We’ve decided to consolidate these commits because all these commits are related to the model, and tell a richer story when they are combined.

We have 7 commits in total and we want to see our full list of commits. We can kick off an interactive rebase by running:

git rebase --interactive HEAD~7

Executing this command will take us to the editor. Here, we rearrange the commit history by moving the last two commits up to the second and third positions, then we can mark them for fixup by changing the pick into a fixup.

We remember to save and exit out of the editor (:wq). Finally we force push our changes to Github by running:

git push --force-with-lease origin feature/potion-for-posterity

💡 I prefer to use the --force-with-lease flag when force pushing my changes because it is safer. It does not overwrite any work on the remote branch if more commits were added to the remote branch by another team member.

Unlike a squash, we did not encounter the additional step to choose which commit message to use. We just used the message of the first (original) commit.

Reword

Sometimes we can make typos in our commit messages. Other times we can realise, later on, that our commit message does not accurately describe the changes we are adding. In such cases, we can reword our commit message.

In the animated example GIF, below, we are rewording the last commit message. Because we only need 1 commit (the last commit), we can start an interactive rebase with the following command:

git rebase --interactive HEAD~1

Executing this command will bring us to the editor where we can mark the commit for rewording. It should be noted that all these actions have a shorter version. In this case, we can replace pick with r. This works the same as if we were to use reword, r is the short form of reword.

When we save (using :wq if your editor is vim), we are taken into another editor interface where we can change the message of our commit. Then we can save, exit and force push our changes to Github.

ggpush --force-with-lease

💡 I personally use ggpush as an alias for git push --force-with-lease origin feature-branch.

If I am not mistaken, you can now get this out-of-the-box with the latest versions of git.

Edit

An edit will allow us to edit both the commit message and the contents of a commit. In the unfortunate situation where we have accidentally committed sensitive information like a password or a secret, we can use rebase to edit the commit.

In the animated GIF example, we realise that we’ve accidentally save secrets and password in our README. We’d like to pull up all our commits just as a sanity check. We can start an interactive rebase using this command:

git rebase --interactive HEAD~6

We are brought into the editor where we can see all our commits. We identify the commit that contains the secret password and mark it with edit. Once we save, the rebase process will be paused to give us a chance to go into our code and make the necessary changes.

We can remove the sensitive information, then add our changes and amend the previous commit.

git add .
git commit --amend

We can then ask the rebase process to continue by running:

git rebase --continue

Lastly, we remember to force push our changes to Github. Our secret password has been successfully removed from our commit history

ggpush --force-with-lease

Drop

Sometimes we realise that some commits just aren’t necessary. Perhaps we cherry-picked commits from another branch that have since become out-dated. Or we realise that we’ve added code that we don’t really need anymore. We can use a rebase to drop these commits.

In the animated GIF, below, we want to drop an unnecessary commit. Not only did we make a typo in the commit message, we realised that we no longer need these changes, altogether. We only want to act on the last commit, so we can start an interactive rebase using this command:

git rebase --interactive HEAD~1

We are transported into the editor where we can see the commit in question. We can mark it with drop. Once we save and exit the editor, that unnecessary commit no longer exists. We can force push our changes to Github.

ggpush --force-with-lease

Additional information

A small tip about discarding changes

Sometimes we’re just experimenting and trying stuff out, so we may not necessarily want to keep all the changes we made. In situations like that, we can discard them all by running:

git checkout .
git clean -df

Some vim commands used in this guide

When rebasing, we spend a lot of time in the editor. If your default editor is vim, here are some of the commands we use frequently throughout this guide:

Command	Description
i	To enter INSERT mode. This is the vim mode that allows us to type or insert text.
esc or control + c	Exits from INSERT mode and takes us back to NORMAL mode. NORMAL mode is where we can run commands or navigate around the editor with h, j, k, l and other navigational commands.
:wq	This command will save our work (w) and exit vim (q). Commands are executed in NORMAL mode and start with a colon (:)
:q!	Hard quit without saving anything. The exclamation point (!) is like a force flag.

Going further

If you'd like to continue learning about git and rebase, you can consider doing the following

read up on the differences between merging and rebasing
spin up a little repository and play around with the actions we covered, or
spin up a demo repository and try all the other actions we did not cover

Conclusion

In this tutorial we learnt how to use git rebase --interactive to squash, fixup, reword, edit and drop commits. We saw how these actions can improve our git workflow by helping keep our commit history clean and allowing us to tell a coherent story in our pull request.

Congratulations, you did great! Keep learning and keep coding. Bye for now, <3

Resources

How to build an interactive CLI app with Go, Cobra & promptui

Div Rhino — Mon, 05 Jul 2021 11:56:22 +0000

Originally posted on divrhino.com

This tutorial will guide us through building an interactive CLI app with Go, Cobra and promptui. We will learn how to prompt the user for input data and persist this data to an SQLite database. Unlike the previous app we built, this experience will feel more like a two way conversation.

Prerequisites

To follow along with this tutorial, you will need to have Go and the Cobra generator installed.

Installation guides:

Demo of what we're building

We will build a note-taking app called Studybuddy. The app will ask us a series of questions and save our answers as a note. We will then be able to see all the notes we've saved. The following video demonstrates some interactive behaviour we will build:

Commands we will implement

This article will focus on implementing interactivity with promptui. We will create several commands to achieve our goal.

studybuddy init - creates a database
studybuddy note - displays information about commands related to notes
studybuddy note new - opens a prompt to collect data from the user
studybuddy note list - displays all the notes we've created

Setting up a cobra app

We will be using the cobra package to give us the functionality we need to build our CLI app. We will only need the ability to create a handful of commands and subcommands, so our cobra app will be very simple.

Starting in our Sites folder, or wherever you keep your projects, we will create a new project folder called studybuddy. Then we will change into it



mkdir -p studybuddy && cd studybuddy

Using the cobra generator, we can initialise a new cobra app. It is a good idea to name your project using the URL where it can be downloaded. I will use my Github repo URL as the name of my package. Please feel free to substitute the following command with your own Github account or personal website



cobra init --pkg-name github.com/divrhino/studybuddy

We will be using go modules to manage our project dependencies, so we can set it up using the same package name we used in the above cobra init command



go mod init github.com/divrhino/studybuddy

To add module requirements and sums, let's run



go mod tidy

Then we should also update descriptions in cmd/root.go file



Use:   "studybuddy",
Short: "Use studybuddy to learn and retain vocabulary",
Long:  `Learn a new language with the studybuddy CLI app by your side`,

And that's our basic cobra app. Let's build it and try it out. Run the following command to build the current project



go build .

An executable binary will be created in your project folder. We have not installed it in our GOPATH so we can't execute it simply by running studybuddy in the terminal. Instead we can run it relative to the current project directory



./studybuddy

If all has gone according to plan, we should get a print out in the terminal showing a bunch of information about our new CLI app.

Opening the database connection

In order to read and write to our database, we will need to open the database connection pool. To do this, we must first create a new SQLite database. As the name suggests, SQLite is a "lite" database, so it is less complicated than something like Postgres. This makes it a great choice for a situation like ours where the primary focus is not the database.

All our database-related functionality should be kept in an appropriate place. Let's create a data folder and also create a file called data.go within it



mkdir data
touch data/data.go

Then let's install the go-sqlite3 package so we can use it in our project



go get github.com/mattn/go-sqlite3

Opening up data/data.go, we can start by importing the database/sql and go-sqlite3 packages. These are the 2 packages we need to work with SQLite in our app



package data

import (
    "database/sql"
    _ "github.com/mattn/go-sqlite3"
)

We can create our function that will open up the database connection pool. We can start writing our code after the imports. In the following code:

We set up a package-level variable called db to hold our database connection pool. This variable will be used in several functions.
We set up OpenDatabase() function.
Then inside the body of OpenDatabase(), we declare a variable for err that we will use later in the sql.Open() function
Next we use sql.Open() to open up a connection pool, passing in our driver name (sqlite3) and the path to our database. Notice that we are not re-declaring the db or the err variables. We want to assign the return value from sql.Open() to the package-level db variable we declared previously
We do some quick error handling before we move on
Then we just return a db.Ping() to verify that the connection is alive



var db *sql.DB

func OpenDatabase() error {
    var err error

    db, err = sql.Open("sqlite3", "./sqlite-database.db")
    if err != nil {
        return err
    }

    return db.Ping()
}

Now that our OpenDatabase() function is ready, will call it from func main(). We do this so the entire app can have access to the connection pool. Our entire main.go file should look something like this:



package main

import (
    "github.com/divrhino/studybuddy/cmd"
    "github.com/divrhino/studybuddy/data"
)

func main() {
    data.OpenDatabase()
    cmd.Execute()
}

Creating the database table

Next, we will need to set up a table into which we can insert our study notes.

We can create a function to create a new table called studybuddy. Inside data/data.go, and in the following code:

we set up CreateTable() function
Inside the body of the CreateTable() function, we set up a variable called createTableSQL to hold the SQL statement we need to create a table with the necessary columns. i.e. idNote, word, definition, category
We prepare the SQL statement using the db.Prepare() method on the package-level db variable we created earlier
If db.Prepare() returns an error, we do some quick error handling
Then we execute the statement using Exec()
Here as well, just so we can see what's happening, we log a message using log.Println



func CreateTable() {
    createTableSQL := `CREATE TABLE IF NOT EXISTS studybuddy (
        "idNote" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
        "word" TEXT,
        "definition" TEXT,
        "category" TEXT
      );`

    statement, err := db.Prepare(createTableSQL)
    if err != nil {
        log.Fatal(err.Error())
    }

    statement.Exec()
    log.Println("Studybuddy table created")
}

We will be using this CreateTable() function in our init command, which we will create next.

The `init` command

Our first custom command will be the init command. We want to run this command to create a new database table. The database connection pool will also open because once we start using the app, func main() will call the OpenDatabase() function.

Making sure we're in our project directory, let's create our init command



cobra add init

Then let's open the newly created cmd/init.go file to quickly update the short and long descriptions. Our short and long descriptions do not have to be the same. But since we don't want to spend too much time worrying about the content here, we will just have them be the same.



Short: "Initialise a new studybuddy database and table",
Long:  `Initialise a new studybuddy database and table.`,

Still in the cmd/init.go file, we will:

import the data package we just created
Inside the Run function of the initCmd, we will call the function we just created above. i.e. CreateTable()



package cmd

import (
    "github.com/divrhino/studybuddy/data"
    "github.com/spf13/cobra"
)

var initCmd = &cobra.Command{
    Use:   "init",
    Short: "Initialise a new studybuddy database and table",
    Long:  `Initialise a new studybuddy database and table.`,
    Run: func(cmd *cobra.Command, args []string) {
        data.CreateTable()
    },
}

func init() {
    rootCmd.AddCommand(initCmd)
}

Now we can build our app



go build .

In the terminal, we can now run the following command. We will know that it's working when a file called sqlite-database.db appears in our project root.



./studybuddy init

The base `note` command

With the initial database setup sorted out, we can move on to adding commands that are directly related to note-taking. The next command we will implement is the base note command. This command won't actually do a whole lot. It will just allow us to get more information about its subcommands

Using the cobra generator, we can go ahead and add the note command



cobra add note

Now we just have to do two things:

First, we have to update the command descriptions
Second, we have to remove the Run field. Removing Run makes this command behave more like the help command

Our whole noteCmd variable should look like this:



var noteCmd = &cobra.Command{
    Use:   "note",
    Short: "A note can be anything you'd like to study and review.",
    Long:  `A note can be anything you'd like to study and review.`,
}

And that's it for the base note command. Let's build the app



go build .

Then run the binary from within the project directory



./studybuddy note

If everything was built correctly, we should get a print out in the terminal with some information about our new note command.

The `note new` command

This is the most involved section of the tutorial because this is where the bulk of the interactivity occurs. Here we will utilise promptui to help us accept user input data. Then we will persist it to our database.

First we need to create a subcommand under the previously-created note command. When used, the subcommand will take this form: studybuddy note new. In the terminal, execute the following line:



cobra add new -p 'noteCmd'

For this command as well, we will need to update the descriptions



Use:   "new",
Short: "Creates a new studybuddy note",
Long:  `Creates a new studybuddy note`,

We will need a way to get user input. We can use the promptui package to help us. We will have to install it first by running the following command in the terminal:



go get github.com/manifoldco/promptui

Every prompt has a similar shape, so we can go ahead and create a custom promptContent struct type



type promptContent struct {
    errorMsg string
    label    string
}

We should probably also import the promptui package along with the other packages we need before we continue



package cmd

import (
    "errors"
    "fmt"
    "os"

    "github.com/manifoldco/promptui"
    "github.com/spf13/cobra"
)

A prompt from the promptui package has a few key concepts:

validate - a function to validate the user's input
templates - used to style text for different state messages
prompt - the actual prompt struct. Running the prompt will allow us to collect data from the user. This feedback will be stored as result



func promptGetInput(pc promptContent) string {
    validate := func(input string) error {
        if len(input) <= 0 {
            return errors.New(pc.errorMsg)
        }
        return nil
    }

    templates := &promptui.PromptTemplates{
        Prompt:  "{{ . }} ",
        Valid:   "{{ . | green }} ",
        Invalid: "{{ . | red }} ",
        Success: "{{ . | bold }} ",
    }

    prompt := promptui.Prompt{
        Label:     pc.label,
        Templates: templates,
        Validate:  validate,
    }

    result, err := prompt.Run()
    if err != nil {
        fmt.Printf("Prompt failed %v\n", err)
        os.Exit(1)
    }

    fmt.Printf("Input: %s\n", result)

    return result
}

After prompting the user for data, we want to use this data to create a new note. Using the custom promptContent struct we created, above, we will set up a prompt to capture a word and a definition for the word.



func createNewNote() {
    wordPromptContent := promptContent{
        "Please provide a word.",
        "What word would you like to make a note of?",
    }
    word := promptGetInput(wordPromptContent)

    definitionPromptContent := promptContent{
        "Please provide a definition.",
        fmt.Sprintf("What is the definition of %s?", word),
    }
    definition := promptGetInput(definitionPromptContent)
}

While we can prompt the user for text input, we can also give the user some options to choose from. We achieve this by using SelectWithAdd from promptui. In the following code:

We start with some pre-determined items , which will be of type []string
We give the user the option of adding their own category by adding the label Other
We set the initial index to -1 because this index will never actually be present inside items
As long as the index is -1, the prompt is kept open and more categories can be appended to the items slice
Once the user chooses an item, the prompt will close and the result will be returned



func promptGetSelect(pc promptContent) string {
    items := []string{"animal", "food", "person", "object"}
    index := -1
    var result string
    var err error

    for index < 0 {
        prompt := promptui.SelectWithAdd{
            Label:    pc.label,
            Items:    items,
            AddLabel: "Other",
        }

        index, result, err = prompt.Run()

        if index == -1 {
            items = append(items, result)
        }
    }

    if err != nil {
        fmt.Printf("Prompt failed %v\n", err)
        os.Exit(1)
    }

    fmt.Printf("Input: %s\n", result)

    return result
}

Once we're done with our promptGetSelect function, we can return to our createNewNote() function to capture the category



func createNewNote() {
    wordPromptContent := promptContent{
        "Please provide a word.",
        "What word would you like to make a note of?",
    }
    word := promptGetInput(wordPromptContent)

    definitionPromptContent := promptContent{
        "Please provide a definition.",
        fmt.Sprintf("What is the definition of the %s?", word),
    }
    definition := promptGetInput(definitionPromptContent)

    categoryPromptContent := promptContent{
        "Please provide a category.",
        fmt.Sprintf("What category does %s belong to?", word),
    }
    category := promptGetSelect(categoryPromptContent)
}

We are able to capture input data from the user about notes, but we haven't created a function to insert the notes into our database, so let's do that now. Back in data/data.go, we need to create another method that interacts with our database. This time we want to INSERT data:



func InsertNote(word string, definition string, category string) {
    insertNoteSQL := `INSERT INTO studybuddy(word, definition, category) VALUES (?, ?, ?)`
    statement, err := db.Prepare(insertNoteSQL)
    if err != nil {
        log.Fatalln(err)
    }
    _, err = statement.Exec(word, definition, category)
    if err != nil {
        log.Fatalln(err)
    }

    log.Println("Inserted study note successfully")
}

Now we're ready to insert notes into the database. Jumping back into our new command in the file new.go, we'll have to import the data package into cmd/new.go



import (
    ...

    "github.com/divrhino/studybuddy/data"

    ...

)

At the bottom of our createNewNote() function, call data.InsertNote() and pass in the word, definition and category data we collected from the user:



func createNewNote() {
    wordPromptContent := promptContent{
        "Please provide a word.",
        "What word would you like to make a note of?",
    }
    word := promptGetInput(wordPromptContent)

    definitionPromptContent := promptContent{
        "Please provide a definition.",
        fmt.Sprintf("What is the definition of the %s?", word),
    }
    definition := promptGetInput(definitionPromptContent)

    categoryPromptContent := promptContent{
        "Please provide a category.",
        fmt.Sprintf("What category does %s belong to?", word),
    }
    category := promptGetSelect(categoryPromptContent)

    data.InsertNote(word, definition, category)
}

Then to tie everything up, we have to call our completed createNewNote() function inside Run:



var newCmd = &cobra.Command{
    Use:   "new",
    Short: "Create a new note to study",
    Long:  `Create a new note to study.`,
    Run: func(cmd *cobra.Command, args []string) {
        createNewNote()
    },
}

Let's build everything so we can test our changes



go build .

Then run the note new command to trigger out little "interview":



./studybuddy note new

Add a few notes so we have something to see in the next section.

The `note list` command

This final command will give us the ability to display all our notes. We will need a way to retrieve all our records from the database.

Back in the data/data.go files, we can create a function to do that. In the following code:

we set up a function called DisplayAllNotes()
We then set up an SQL query to select everything from the studybuddy table and order the results by word
We do some quick error handling
Then we refer the closing of the row
We loop through the rows and print out the columns in a loosely formatted string



func DisplayAllNotes() {
    row, err := db.Query("SELECT * FROM studybuddy ORDER BY word")
    if err != nil {
        log.Fatal(err)
    }

    defer row.Close()

    for row.Next() {
        var idNote int
        var word string
        var definition string
        var category string
        row.Scan(&idNote, &word, &definition, &category)
        log.Println("[", category, "] ", word, "—", definition)
    }
}

We should also remember to create our list subcommand. When used, the subcommand will take this form: studybuddy note list. In the terminal, execute the following line:



cobra add list -p 'noteCmd'

Now we can open up cmd/list.go and update it. In the following code:

We import the data package.
We update the Short and Long descriptions
We call data.DisplayAllNotes() inside Run



package cmd

import (
    "github.com/divrhino/studybuddy/data"
    "github.com/spf13/cobra"
)

// listCmd represents the list command
var listCmd = &cobra.Command{
    Use:   "list",
    Short: "See a list of all notes you've added",
    Long:  `See a list of all notes you've added.`,
    Run: func(cmd *cobra.Command, args []string) {
        data.DisplayAllNotes()
    },
}

func init() {
    noteCmd.AddCommand(listCmd)
}

Build the app



go build .

And run the following command to see a list of all our notes:



./studybuddy note list

Going further

Interactive CLI apps are a simple, yet powerful tool to have at your disposal. You can combine them with webscrapers or other techniques to build custom productivity apps and beef up your personal workflow.

If you’d like to extend this project further, you can consider doing the following:

Add a feature that lets you test yourself. Display each word in a question and execute a prompt to collect the answer to test questions.
Add a feature that allows you to see which words you have trouble remembering.

Conclusion

In this tutorial we learnt how to build an interactive CLI app that we can use to save fun and interesting new words.

If you enjoyed this article and you'd like more, consider subscribing to Div Rhino on YouTube.

Congratulations, you did great. Keep learning and keep coding. Bye for now, <3.

divrhino / studybuddy

Build an interactive CLI application with Go, Cobra and promptui. Video tutorial available on the Div Rhino YouTube channel.

Build an interactive CLI app with Go, Cobra and promptui

Text tutorial: https://divrhino.com/articles/build-interactive-cli-app-with-go-cobra-promptui
Video tutorial: https://www.youtube.com/watch?v=so3VZwdWcBg

View on GitHub

How to create PDFs with Go, Maroto & GoFakeIt

Div Rhino — Sun, 02 May 2021 04:27:37 +0000

Originally posted on divrhino.com

In this tutorial, we will learn how to use the Maroto package to create a PDF. From invoices to certificates, being able to generate a PDF from app data is a pretty handy feature. In this tutorial, we will use the Maroto package to build and save a product list. We will also learn how we can use the GoFakeIt package to generate some random dummy data that we can use in our PDF.

Prerequisites

To follow along, you will need to have Go installed. We are using version 1.16.2 for this tutorial.

The packages

Maroto

The Maroto package is described as being bootstrap-inspired, but you don't need to know Bootstrap to use it. However if you have used Bootstrap before, the approach Maroto takes will feel familiar to you. A document is built using rows, columns and components. The package gives us the ability to include features such as:

images
signatures
Barcodes, and
QR Codes

GoFakeIt

Using mock data is a great way to speed up the prototyping process. We will use the GoFakeIt package to create a little dummy data generator to insert into our PDF.

We will see how to install these in the Getting Started section.

Wireframe of what we're building

We will be creating an product list for an imaginary fruit shop called "Div Rhino Fruit". The document will have a header and a table of products.

Getting started

Now that we've covered the basic background information, we can start setting up our project.

Inside our Sites directory, we can make a new folder which we will call fruitful-pdf and we will change into it

cd Sites
mkdir fruitful-pdf
cd fruitful-pdf

We will be using go modules to manage our dependencies.

It is a good idea to name your project using the URL where it can be downloaded, so I'm going to use my Github repo URL as the name of my package. Please feel free to substitute the following command with your own Github account or personal website

go mod init github.com/divrhino/fruitful-pdf

After the command runs successfully, you should see a go.mod file in your project directory.

Next we will install the Maroto package. This tutorial will be using version v0.31.0. If you're using an older version of the Maroto package, certain properties such as text colours may not be available.

go get -u github.com/johnfercher/maroto
go get github.com/johnfercher/maroto/internal@v0.31.0

We will also need to install the GoFakeIt package for our mock data generator. We will be using version v6.3.0 for this tutorial. You can install it with the following command:

go get github.com/brianvoe/gofakeit/v6

After installing these packages, you should see a go.sum file has been created in your project folder. This can be thought of as a lock file. It is used to verify that the checksum of dependencies have not changed.

Those are the only third-party packages we need. We are now ready to start creating our PDF structure.

Skeleton PDF

Like with most Go projects, we can go ahead and create a main.go file at the root of our project directory.

touch main.go

Inside our main.go file, let's import all our necessary Maroto sub-packages. Each package provides us with useful functions that will allow us to use things like colours, images, fonts and components.

package main

func main() {}

Now, inside the body of our func main(), we can create a new maroto using the pdf sub-package. The NewMaroto() method creates a new maroto instance and returns a pointer to pdf.Maroto. It also expects two arguments: (i) orientation, and (ii) paper size. We can get these values from the consts sub-package:

package main

import (
    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
)

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
}

We also want to give our PDF document some margins so the content isn't falling off the sides. We can do this by using the SetPageMargins method, which takes 3 values: (i) a left, (ii) a top, and (iii) a right margin value.

package main

import (
    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
)

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
    m.SetPageMargins(20, 10, 20)
}

Our content isn't ready yet, but we can go ahead and save an empty file for now using the OutputFileAndClose() method. Let's tell it that we want to save the output as div_rhino_fruit.pdf, in a folder called pdfs.

package main

import (
    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
)

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
    m.SetPageMargins(20, 10, 20)

    m.OutputFileAndClose("pdfs/div_rhino_fruit.pdf")
}

We'll have to remember to create the pdfs folder too.

mkdir pdfs

The OutputFileAndClose() method returns an error, so let's do some quick error handling before we move on. If, for some reason, we're unable to output a PDF file, the program immediately aborts, because its only purpose is to create this PDF file.

package main

import (
    "fmt"
    "os"

    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
)

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
    m.SetPageMargins(20, 10, 20)

    err := m.OutputFileAndClose("pdfs/div_rhino_fruit.pdf")
    if err != nil {
        fmt.Println("⚠️  Could not save PDF:", err)
        os.Exit(1)
    }
}

And just so we know something is actually happening, let's print out a little message every time we run our code:

package main

import (
    "fmt"
    "os"

    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
)

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
    m.SetPageMargins(20, 10, 20)

    err := m.OutputFileAndClose("pdfs/div_rhino_fruit.pdf")
    if err != nil {
        fmt.Println("⚠️  Could not save PDF:", err)
        os.Exit(1)
    }

    fmt.Println("PDF saved successfully")
}

If we run our program now, we'll get an empty PDF file saved to our pdfs folder

go run main.go

Building the header

An empty PDF file isn't all that interesting, so let's add some content, starting with a header. Our header will hold an image of the "Div Rhino Fruit" logo.

We don't want to put everything in our func main(), so let's create a new function to build our header. We will pass in our previously-created instance of pdf.Maroto as an argument. This function will perform some transformations on the pdf.Maroto instance (i.e. m), and we won't be returning any values.

func buildHeading(m pdf.Maroto) {}

The Maroto package gives us a method that lets us register a header "component" that will appear on every page of our PDF document. This RegisterHeader() method accepts an anonymous callback function as the only argument. This anonymous function can be thought of as a header container.

func buildHeading(m pdf.Maroto) {
    m.RegisterHeader(func() {})
}

Within the body of the "header container", we can set up a row that contains a column. Much like the Bootstrap grid system, rows are wrappers for columns. Here we've given our row a height of 50 and we've indicated that we want a full-width column that takes up 12 spaces. The number 12 is significant because most grid systems use 12 columns.

func buildHeading(m pdf.Maroto) {
    m.RegisterHeader(func() {
        m.Row(50, func() {
            m.Col(12, func() {

            })
        })
    })
}

We have a logo that was created beforehand and we've named it logo_div_rhino.jpg. To keep things organised, let's make a new images directory to hold images we use in this project. Feel free to use your own logo image instead.

mkdir images

We will also want to import the props subpackage from maroto

import (
    "fmt"
    "os"

    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
    "github.com/johnfercher/maroto/pkg/props"
)

Inside our full-width column, we can set up an image component to display our "Div Rhino Fruit" logo. We will centralise it and tell it to take up 75% of the height of the cell.

func buildHeading(m pdf.Maroto) {
    m.RegisterHeader(func() {
        m.Row(50, func() {
            m.Col(12, func() {
                m.FileImage("images/logo_div_rhino.jpg", props.Rect{
                    Center:  true,
                    Percent: 75,
                })
            })
        })
    })
}

The FileImage() method returns an error, so let's do some quick error handling before we move on. If the image cannot be loaded, we print a message in the console to let the user know.

func buildHeading(m pdf.Maroto) {
    m.RegisterHeader(func() {
        m.Row(50, func() {
            m.Col(12, func() {
                err := m.FileImage("images/logo_div_rhino.jpg", props.Rect{
                    Center:  true,
                    Percent: 75,
                })

                if err != nil {
                    fmt.Println("Image file was not loaded 😱 - ", err)
                }
            })
        })
    })
}

Next we want to create another row and full-width column to add some descriptive Text — "Prepared for you by the Div Rhino Fruit Company".

func buildHeading(m pdf.Maroto) {
    m.RegisterHeader(func() {
        m.Row(50, func() {
            m.Col(12, func() {
                err := m.FileImage("images/logo_div_rhino.jpg", props.Rect{
                    Center:  true,
                    Percent: 75,
                })

                if err != nil {
                    fmt.Println("Image file was not loaded 😱 - ", err)
                }

            })
        })
    })

    m.Row(10, func() {
        m.Col(12, func() {
            m.Text("Prepared for you by the Div Rhino Fruit Company", props.Text{
                Top:   3,
                Style: consts.Bold,
                Align: consts.Center,
                Color: getDarkPurpleColor(),
            })
        })
    })
}

func getDarkPurpleColor() color.Color {
    return color.Color{
        Red:   88,
        Green: 80,
        Blue:  99,
    }
}

We want to use a custom colour here, so we will also want to make a new function func getDarkPurpleColor() and use the color sub-package that Maroto provides to create a dark purple colour

import (
    "fmt"
    "os"

    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
    "github.com/johnfercher/maroto/pkg/props"
    "github.com/johnfercher/maroto/pkg/color"
)

And that's our header. Let's call it inside out func main()

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
    m.SetPageMargins(20, 10, 20)

    buildHeading(m)

    err := m.OutputFileAndClose("pdfs/div_rhino_fruit.pdf")
    if err != nil {
        fmt.Println("⚠️  Could not save PDF:", err)
        os.Exit(1)
    }

    fmt.Println("PDF saved successfully")
}

We can run our code to generate a PDF file to see what this looks like:

go run main.go

Lay out a table of products

We can build our table of fruit next. We first want to give the whole table a heading. We can create a new teal colour and set the background colour of the cell to this teal colour. Then, like we've done previously, we can add a row that contains a full-width column. We are giving the Text component properties such as top position, size, color, etc.

func buildFruitList(m pdf.Maroto) {

    m.SetBackgroundColor(getTealColor())

    m.Row(10, func() {
        m.Col(12, func() {
            m.Text("Products", props.Text{
                Top:    2,
                Size:   13,
                Color:  color.NewWhite(),
                Family: consts.Courier,
                Style:  consts.Bold,
                Align:  consts.Center,
            })
        })
    })
}

func getTealColor() color.Color {
    return color.Color{
        Red:   3,
        Green: 166,
        Blue:  166,
    }
}

At this point, if you'd like to see what this looks like, you can just generate a PDF and preview it:

go run main.go

Along with content, our table should have column headings as well. Similar to the HTML table structure, we first want to create something that resembles a thead and tbody. We also want to set the cell colour of this section to white and set the background colour of alternate rows to a light purple. We will need to create a new light purple colour function.

We can use the TableList() component that Maroto provides to get a table. We want both the HeaderProps and the ContentProps to have 3 columns. We can do this by giving the GridSizes property a value of []uint{3, 7, 2} (a slice of unsigned integers with the values of 3, 7 and 2). These grid sizes add up to 12.

func buildFruitList(m pdf.Maroto) {
    tableHeadings := []string{"Fruit", "Description", "Price"}
    contents := [][]string{{"Apple", "Red and juicy", "2.00"}, {"Orange", "Orange and juicy", "3.00"}}
    lightPurpleColor := getLightPurpleColor()

    m.SetBackgroundColor(getTealColor())
    m.Row(10, func() {
        m.Col(12, func() {
            m.Text("Products", props.Text{
                Top:    2,
                Size:   13,
                Color:  color.NewWhite(),
                Family: consts.Courier,
                Style:  consts.Bold,
                Align:  consts.Center,
            })
        })
    })

    m.SetBackgroundColor(color.NewWhite())

    m.TableList(tableHeadings, contents, props.TableList{
        HeaderProp: props.TableListContent{
            Size:      9,
            GridSizes: []uint{3, 7, 2},
        },
        ContentProp: props.TableListContent{
            Size:      8,
            GridSizes: []uint{3, 7, 2},
        },
        Align:                consts.Left,
        AlternatedBackground: &lightPurpleColor,
        HeaderContentSpace:   1,
        Line:                 false,
    })

}

...

func getLightPurpleColor() color.Color {
    return color.Color{
        Red:   210,
        Green: 200,
        Blue:  230,
    }
}

Just some notes on the code, above:

the tableHeadings if of type slice of string
the contents are a slice of slice of string. Another way of saying slice of slice of string is to say two-dimensional slice. This type will be important in the next section when we work on our mock data generator.

Call the buildFruitList() in func main()

func main() {
    m := pdf.NewMaroto(consts.Portrait, consts.A4)
    m.SetPageMargins(20, 10, 20)

    buildHeading(m)
    buildFruitList(m)


    err := m.OutputFileAndClose("pdfs/div_rhino_fruit.pdf")
    if err != nil {
        fmt.Println("⚠️  Could not save PDF:", err)
        os.Exit(1)
    }

    fmt.Println("PDF saved successfully")
}

Let's do a quick preview of our PDF again to see what we've just added so far

go run main.go

Mock data generator

Okay, now let's take a small detour out of our main.go file. So far we've used hardcoded content to inject into our table. We're going to try something different and use some randomly-generated data instead. To achieve this, we will create our own custom data package and make use of the GoFakeIt package we installed, earlier.

In our project root, let's make a new folder called data

mkdir data

And within this new data folder, let's create a new file called products.go

touch data/products.go

Let's head into our newly-created products.go file. This is going to be a new package on its own, so we can start by indicating that it's part of package data instead of package main. Then we can import GoFakeIt.

package data

import "github.com/brianvoe/gofakeit/v6"

The GoFakeIt packages gives us all sorts of functions for concepts such as a File, a Person, a Number, among other things. We will be using Fruit in this tutorial.

We can represent the structure of each fruit item using a Fruit struct type. Each Fruit will have a Name, a Description and a Price. Each of these values will be randomly generated using GoFakeIt.

package data

import "github.com/brianvoe/gofakeit/v6"

type Fruit struct {
    Name        string  `fake:"{fruit}"`
    Description string  `fake:"{loremipsumsentence:10}"`
    Price       float64 `fake:"{price:1,10}"`
}

Now that we have our Fruit struct type, we can create a function to make use of it. Every time the generateFruit() function is called, we get a new random fruit.

package data

import (
    "fmt"

    "github.com/brianvoe/gofakeit/v6"
)

type Fruit struct {
    Name        string  `fake:"{fruit}"`
    Description string  `fake:"{loremipsumsentence:10}"`
    Price       float64 `fake:"{price:1,10}"`
}

func generateFruit() []string {
    var f Fruit
    gofakeit.Struct(&f)

    froot := []string{}
    froot = append(froot, f.Name)
    froot = append(froot, f.Description)
    froot = append(froot, fmt.Sprintf("%.2f", f.Price))

    return froot
}

Lastly, we want to create a function we can access outside this data package. We need this in order to generate this random fruit data inside out PDF table that lives in our main.go file.

package data

import (
    "fmt"

    "github.com/brianvoe/gofakeit/v6"
)

type Fruit struct {
    Name        string  `fake:"{fruit}"`
    Description string  `fake:"{loremipsumsentence:10}"`
    Price       float64 `fake:"{price:1,10}"`
}

func generateFruit() []string {
    var f Fruit
    gofakeit.Struct(&f)

    froot := []string{}
    froot = append(froot, f.Name)
    froot = append(froot, f.Description)
    froot = append(froot, fmt.Sprintf("%.2f", f.Price))

    return froot
}

func FruitList(length int) [][]string {
    var fruits [][]string

    for i := 0; i < length; i++ {
        onefruit := generateFruit()
        fruits = append(fruits, onefruit)
    }

    return fruits
}

Just some notes about the code, above:

FruitList() is a public function that we can access from outside the data package, which is why it starts with a capital letter.
As we saw earlier, our TableList component needs the data to come in as a two-dimensional slice of string.
FruitList() takes in one parameter of length so we can dynamically determine how many items of fruit we want to generate.
We then pass this length value into a little for loop that calls our generateFruit() function however many times length determines.
Then we return a two-dimensional slice of string.

Hooking up the dynamic content

Back in our main.go file, we can import our data package and replace our contents variable. Let's use the FruitList() function to generate 20 random fruit.

package main

import (
    "fmt"
    "os"

    "github.com/johnfercher/maroto/pkg/color"
    "github.com/johnfercher/maroto/pkg/consts"
    "github.com/johnfercher/maroto/pkg/pdf"
    "github.com/johnfercher/maroto/pkg/props"

    "github.com/divrhino/fruitful-pdf/data"
)

...

func buildFruitList(m pdf.Maroto) {
    headings := []string{"Fruit", "Description", "Price"}
    contents := data.FruitList(20)

...
}

And now if we run our code, we have a table of 20 randomly-generated fruit.

go run main.go

If we decide to generate many, many more random Fruit, say 100, our PDF automatically adds pages to accommodate this.

Going further

If you'd like to extend this project further, you can consider doing the following

try replacing the mock data with actual data from an API
use the RegisterFooter() method to add a footer to every page of the PDF document
add a signature, barcode, page numbers and a QR code

This repository includes an extended version of this tutorial code in the examples folder. You can also look at the Maroto package Github page for even more examples.

Conclusion

In this tutorial we learnt how to generate some dummy data that we saved in a PDF file. We used the maroto and gofakeit packages to help us achieve this.

If you enjoyed this article and you'd like more, consider following Div Rhino on YouTube.

Congratulations, you did great! Keep learning and keep coding. Bye for now, <3

divrhino / fruitful-pdf

Create a PDF document with Go, Maroto and GoFakeIt. Video tutorial available on the Div Rhino YouTube channel.

Create PDFs with Go, Maroto and GoFakeIt

Text tutorial: https://divrhino.com/articles/create-pdf-document-with-go-maroto-gofakeit
Video tutorial: https://youtu.be/jwOy4JgleTU

View on GitHub

How to add flags to a CLI tool built with Go and Cobra

Div Rhino — Tue, 13 Apr 2021 14:50:59 +0000

Originally posted on divrhino.com

Prerequisites

In the first part of this tutorial series, we created a dad joke command line application that gives us a random dad joke right in our terminal. In this second part, we will be learning how we can use a flag to retrieve dad jokes that contain a specific term.

If you did not follow the first tutorial, but would like to follow along with this tutorial, you can use the finished code from the first part as your starting point. You can find that in the repo for the first tutorial.

Adding a flag

If you're familiar with command-line tools, you will recognise the use of flags. A flag provides a way for the user to modify the behaviour of a command. In this tutorial, we will be modifying the random command to allow us to search for a random joke that includes a specific term.

Cobra has two types of flags:

Persistent flags - available to the command it is assigned to, as well as all its sub-commands
Local flags - only assigned to a specific command

Our example is small enough that this distinction does not have a real impact. We will, however, choose to apply a persistent flag because in an imaginary future, we'd like all sub-commands of random to be able to take in a term.

In the init function of our cmd/random.go file, we can add a persistent flag. We have named it term and given it a description:



func init() {
    rootCmd.AddCommand(randomCmd)

    randomCmd.PersistentFlags().String("term", "", "A search term for a dad joke.")
}

That's all it takes to add a flag to a command. There are several ways we can make use of this flag. Here's how we will approach it:

First we check for the presence of a term
If the random command was run with the term flag, we will run a function that knows how to handle search terms
But if the random command is run without any flags, we will run another function that merely returns a random joke, without knowing anything about search terms. This function is called getRandomJoke() and was created in the previous tutorial

Let's put together the initial skeleton of the function we need to handle search terms. In cmd/random.go, we will add this new function. For now, all it does is print out the search term to the terminal. We will build up the function as we proceed:



func getRandomJokeWithTerm(jokeTerm string) {
    log.Printf("You searched for a joke with the term: %v",  jokeTerm)
}

Now we can move on to check whether the user has used the term flag. In the Run function of our randomCmd, we can add the following check:



var randomCmd = &cobra.Command{
    Use:   "random",
    Short: "Get a random dad joke",
    Long:  `This command fetches a random dadjoke from the icanhazdadjoke api`,
    Run: func(cmd *cobra.Command, args []string) {
        jokeTerm, _ := cmd.Flags().GetString("term")

        if jokeTerm != "" {
            getRandomJokeWithTerm(jokeTerm)
        } else {
            getRandomJoke()
        }
    },
}

What's essentially happening here is:

We are getting the value from our term flag and storing it in the variable jokeTerm
If the jokeTerm value is not empty, we will run our getRandomJokeWithTerm method and pass the jokeTerm value in to it
Else if jokeTerm is empty, we'll just get a random joke by running getRandomJoke. Again, this was a function we created in the previous article.

Now we can go into our terminal and run the random command with and without the term flag:



# random command without the flag
go run main.go random

# random command with the term flag
go run main.go random --term=hipster

Understanding the response

As a reminder, we are using the icanhazdadjoke API for all our jokes data. If we take a look at the API documentation, we can see that we can pass a search term to our request

We can run the example curl command in our terminal:



curl -H "Accept: application/json" "https://icanhazdadjoke.com/search?term=hipster"

We can represent the JSON response as a struct in our code:



type SearchResult struct {
    Results    json.RawMessage `json:"results"`
    SearchTerm string          `json:"search_term"`
    Status     int             `json:"status"`
    TotalJokes int             `json:"total_jokes"`
}

Get data with search term

Now that we have a better understanding of the data that we'll be working with, let's move on and create a method get the data.

First we will need to create a skeleton method. To begin, we just want to be able to pass a search term to which we can pass in to the API call.



func getJokeDataWithTerm(jokeTerm string) {

}

In the previous article, we had already created a method that helped us to get joke data. We even creatively called it getJokeData(). This method takes in an API url as it's only argument. Within the body of our newly created getJokeDataWithTerm function, we can add the following lines:



func getJokeDataWithTerm(jokeTerm string) {
    url := fmt.Sprintf("https://icanhazdadjoke.com/search?term=%s", jokeTerm)
    responseBytes := getJokeData(url)
}

Then we want to unmarshal the returned responseBytes, following the shape of the SearchResult{} struct



func getJokeDataWithTerm(jokeTerm string) {
    url := fmt.Sprintf("https://icanhazdadjoke.com/search?term=%s", jokeTerm)
    responseBytes := getJokeData(url)

    jokeListRaw := SearchResult{}

    if err := json.Unmarshal(responseBytes, &jokeListRaw); err != nil {
        log.Printf("Could not unmarshal reponseBytes. %v", err)
    }
}

Unmarshalling the responseBytes gives us the Results as json.RawMessage. We will have to unmarshal this raw JSON, following the shape of Joke{}, which is a struct we had created in the first article.

Our Results may often contain more than one joke. We can store all the jokes in a []Joke{} (slice of Joke{}).



func getJokeDataWithTerm(jokeTerm string) {
    url := fmt.Sprintf("https://icanhazdadjoke.com/search?term=%s", jokeTerm)
    responseBytes := getJokeData(url)

    jokeListRaw := SearchResult{}

    if err := json.Unmarshal(responseBytes, &jokeListRaw); err != nil {
        log.Printf("Could not unmarshal reponseBytes. %v", err)
    }

    jokes := []Joke{}
    if err := json.Unmarshal(jokeListRaw.Results, &jokes); err != nil {
        log.Printf("Could not unmarshal reponseBytes. %v", err)
    }
}

Next, we'll want to return all the jokes we process using this method. We also want to know how many jokes we got back as well. We can update the function to be able to return these two values:



func getJokeDataWithTerm(jokeTerm string) (totalJokes int, jokeList []Joke) {
    url := fmt.Sprintf("https://icanhazdadjoke.com/search?term=%s", jokeTerm)
    responseBytes := getJokeData(url)

    jokeListRaw := SearchResult{}

    if err := json.Unmarshal(responseBytes, &jokeListRaw); err != nil {
        log.Printf("Could not unmarshal reponseBytes. %v", err)
    }

    jokes := []Joke{}
    if err := json.Unmarshal(jokeListRaw.Results, &jokes); err != nil {
        log.Printf("Could not unmarshal reponseBytes. %v", err)
    }

    return jokeListRaw.TotalJokes, jokes
}

Now that our getJokeDataWithTerm function is fleshed out, we can use it within our getRandomJokeWithTerm function. Replace the contents as follows:



func getRandomJokeWithTerm(jokeTerm string) {
    _, results := getJokeDataWithTerm(jokeTerm)
    fmt.Println(results)
}

For the time being, we throw away the totalJoke value by using an underscore (_). We are only doing this for demonstration purposes, so fret not, we will use it in the next section.

Randomising the search results

If we head into the terminal now and test our command, we just keep getting all the search results.



go run main.go random --term=hipster

This isn't really what we're going for. We want to be able to get one random joke that contains a specified search term each time we run the command with the flag. We can achieve this by introducing a function to randomise our results.

First, though, let's import a couple of packages



import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "math/rand"
    "net/http"
    "time"

    "github.com/spf13/cobra"
)

Then we can write a skeleton randomiser function:



func randomiseJokeList(length int, jokeList []Joke) {

}

Our randomiseJokeList function takes in 2 arguments:

length - to be used as the ceiling for our random range
jokeList - the data we want to randomise

We can update our randomiseJokeList method with the following code:



func randomiseJokeList(length int, jokeList []Joke) {
    rand.Seed(time.Now().Unix())

    min := 0
    max := length - 1

    if length <= 0 {
        err := fmt.Errorf("No jokes found with this term")
        fmt.Println(err.Error())
    } else {
        randomNum := min + rand.Intn(max-min)
        fmt.Println(jokeList[randomNum].Joke)
    }
}

Here's what's going on in the above code snippet:

We are getting a random value within a range
If the number of jokes is less than or equal to zero, we let the user know that we weren't able to find any jokes with that term
But if there are jokes present, then we print out a random one

With our newly created randomiseJokeList function all complete, we can return to our getRandomJokeWithTerm function and update it:



func getRandomJokeWithTerm(jokeTerm string) {
    total, results := getJokeListWithTerm(jokeTerm)
    randomiseJokeList(total, results)
}

If we go into our terminal and test our flag now, we will be able to get a random dad joke that contains a specified term:



# get one random joke that contains the term "hipster"
go run main.go random --term=hipster

# get one random joke that contains the term "apple"
go run main.go random --term=apple

Distributing your CLI tool

Everybody likes a good dad joke from time to time, so I'm sure your friends will want to use your tool. We will be able to distribute this dadjoke CLI tool as a Go package. In order to do this:

Upload your code to a public repo
Install it by running go get <link-to-your-public-repo e.g. go get github.com/example/dadjoke

Then you will be able to run your tool like this:



dadjoke random

# random joke with term
dadjoke random --term=hipster

Conclusion

In this tutorial we learnt how to extend our dadjoke CLI tool so we could implement a flag for our random command.

If you enjoyed this article and you'd like more, consider following Div Rhino on YouTube.

Congratulations, you did great. Keep learning and keep coding. Bye for now.

divrhino / dadjoke2

Add flags to a CLI tool built with Go and Cobra. Video tutorial available on the Div Rhino YouTube channel.

Dadjoke CLI Tool - Part 2: Adding flags

Text tutorial: https://divrhino.com/articles/add-flags-to-command-line-tool-go-cobra/
Video tutorial: https://www.youtube.com/watch?v=kT7Z02bR1IY

View on GitHub

How to build a CLI tool with Go and Cobra

Div Rhino — Tue, 13 Apr 2021 14:26:14 +0000

Originally posted on divrhino.com

Go is very useful for building services and tools for productivity. CLI tools are extremely powerful. They are a great way to automate all sorts of different everyday tasks. And who doesn't need a Dadjoke at least once a day, right? We are going to learn how to build a little CLI tool that will use the icanhazdadjoke api to give us a random dad joke.

Prerequisites

To follow along with this tutorial, you will need to have Go and Cobra installed.

Installation guides:

Initialising the project

In the terminal, we can first create a new directory for our project. We can then immediately change into the new directory and generate a new app, giving it a package name. Usually, a package would be a url you own.

In this case, we've named it as a github repo. You can change the example to your own Github user name.

cd Sites
mkdir dadjoke
cd dadjoke
cobra init --pkg-name github.com/example/dadjoke

If we run the ls command in the terminal, we can see the files that the cobra init command created for us.

ls

We now have a license, a cmd folder and a main.go file

LICENSE
a cmd folder
a main.go file

Cobra just uses the main.go file as an entry point. We won't be putting any of our CLI application code here. Instead, most of our code will be put in the cmd folder.

We will also want to use Go modules in our project, to handle our dependencies. We will run the go mod init command, in the terminal, to initialise Go modules. Here we are using the same package name we had used earlier when generating our cobra app.

go mod init github.com/example/dadjoke

This creates a go.mod file, which will help us manage our dependencies.

Creating commands

If we run go run main.go in our terminal for the first time, all our dependencies will be installed and a go.sum file will also be created. This can be thought of as a lock file. It is used to verify that the checksum of dependencies have not changed.

We will also see a print out about our CLI, including the description, usage and available commands. Right now, we only have the help command.

go run main.go

Cobra gives us some boilerplate content, including a description of what our app does. We should probably go and update this to use a description that better describes the dadjoke app we're building

Let's open up the cmd/root.go file and and update the description of our newly-created root command. Replace the default content with your own Short and Long descriptions:

var rootCmd = &cobra.Command{
    Use:   "dadjoke",
    Short: "Get random dad jokes in your terminal",
    Long:  `Dadjoke CLI is a tool that gives you a random dad joke`,
}

If we run our app now, go run main.go, we will see the description we just wrote. Currently, our app does not have any available commands to list.

So let's now create the random command. Cobra gives us the add command that allows us to do this, easily. In the terminal, make sure you're in your project root and run the following command:

cobra add random

The add command generates a new cmd/random.go file for us.

So if we run go run main.go, we will see that random is now one of our available commands. How cool is that?

go run main.go

If we run our random command right now, we'll see that it has some boilerplate description, just like the root command we saw previously. We will want to update this description too. Go into your cmd/random.go file and add a Short and Long description:

var randomCmd = &cobra.Command{
    Use:   "random",
    Short: "Get a random dad joke",
    Long:  `This command fetches a random dad joke from the icanhazdadjoke api`,
    Run: func(cmd *cobra.Command, args []string) {
        ...
    },
}

The dadjoke API - curl

Let's take a look at the documentation for the API we will be consuming. We will be using the free icanhazdadjoke API. This API doesn't require authentication. The creators are nice enough to let us use it for free. The only thing they're asking is that we add a custom User-Agent header. We can definitely do that.

If we scroll down to the endpoints, we can see the cURL command. Let's run it in our terminal and see what we get.

curl -H "Accept: application/json" https://icanhazdadjoke.com/

Here we see that it returns an ID, a joke and a status. Let's quickly represent this in our code before we move on. Inside cmd/random.go, create a new type Joke struct:

package cmd

import (
    "fmt"

    "github.com/spf13/cobra"
)

var randomCmd = &cobra.Command{
    Use:   "random",
    Short: "Get a random dad joke",
    Long:  `This command fetches a random dad joke from the icanhazdadjoke api`,
    Run: func(cmd *cobra.Command, args []string) {
        ...
    },
}

func init() {
    rootCmd.AddCommand(randomCmd)
}

type Joke struct {
    ID     string `json:"id"`
    Joke   string `json:"joke"`
    Status int    `json:"status"`
}

Get request in Go

Now let's try to make that API call in Go.

We will be doing most of our work in the random.go file. Right now, our Run function merely prints out a message. Let's create a function called getRandomJoke. We will call this function inside the Run method. And let's just print a message for now, just to see if it works.

In our random.go file, add a new getRandomJoke() method and call it from inside Run:

package cmd

import (
    "fmt"

    "github.com/spf13/cobra"
)

var randomCmd = &cobra.Command{
    Use:   "random",
    Short: "Get a random dad joke",
    Long:  `This command fetches a random dad joke from the icanhazdadjoke api`,
    Run: func(cmd *cobra.Command, args []string) {
        getRandomJoke()
    },
}

func init() {
    rootCmd.AddCommand(randomCmd)
}

type Joke struct {
    ID     string `json:"id"`
    Joke   string `json:"joke"`
    Status int    `json:"status"`
}

func getRandomJoke() {
    fmt.Println("Get random dad joke :P")
}

If we run our random command in the terminal now, we will see our message from the Println on line 25

go run main.go random

Looking at the http package

Next, let's create a function that will make a GET request to the API endpoint. We're going to use that to get our random joke data. We can use the net/http package to achieve this.

First things first, let's visit the net/http documentation to get a better idea of how we can use it. We can visit https://golang.org/pkg/net/http/ and search for func Get. Since we know we want to make a GET request. Here, we see this line that says

To make a request with custom headers, use NewRequest and DefaultClient.Do.

If you remember, the API maintainers would like us to add a custom header to our app, so this is what we're looking for.

The getJokeData() method

We will create a function that we can use to make GET requests to the icanhazdadjoke API endpoint

package cmd

import (
    "fmt"

    "github.com/spf13/cobra"
)

var randomCmd = &cobra.Command{
    Use:   "random",
    Short: "Get a random dad joke",
    Long:  `This command fetches a random dad joke from the icanhazdadjoke api`,
    Run: func(cmd *cobra.Command, args []string) {
        getRandomJoke()
    },
}

func init() {
    rootCmd.AddCommand(randomCmd)
}

type Joke struct {
    ID     string `json:"id"`
    Joke   string `json:"joke"`
    Status int    `json:"status"`
}

func getRandomJoke() {
    fmt.Println("Get random dad joke :P")
}

func getJokeData(baseAPI string) []byte {}

Inside the body of the getJokeData() function, we will create a new request using the NewRequest() method from the net/http package

package cmd

import (
    "fmt"
    "net/http"
    "io/ioutil"

    "github.com/spf13/cobra"
)

var randomCmd = &cobra.Command{
    Use:   "random",
    Short: "Get a random dad joke",
    Long:  `This command fetches a random dad joke from the icanhazdadjoke api`,
    Run: func(cmd *cobra.Command, args []string) {
        getRandomJoke()
    },
}

func init() {
    rootCmd.AddCommand(randomCmd)
}

type Joke struct {
    ID     string `json:"id"`
    Joke   string `json:"joke"`
    Status int    `json:"status"`
}

func getRandomJoke() {
    fmt.Println("Get random dad joke :P")
}

func getJokeData(baseAPI string) []byte {
    request, err := http.NewRequest(
        http.MethodGet, //method
        baseAPI,        //url
        nil,            //body
    )

    if err != nil {
        log.Printf("Could not request a dadjoke. %v", err)
    }

    request.Header.Add("Accept", "application/json")
    request.Header.Add("User-Agent", "Dadjoke CLI (https://github.com/example/dadjoke)")
}

New code explanations:

Line 5
- Import net/http package
Line 6
- Import io/ioutil package
Line 35
- Use the http.NewRequest() method to create a new request
Line 36
- First argument is an HTTP method
Line 37
- Second argument is a url
Line 38
- Third argument is a request body. Remember the comma at the end.
Lines 41-43
- Handle the error that is returned from http.NewRequest()
Line 45
- Add a header to tell the API we want our data returned as JSON
Line 46
- Add a custom User-Agent header to tell the API maintainers how we're using their API

The completed getJokeData() method:

func getJokeData(baseAPI string) []byte {
    request, err := http.NewRequest(
        http.MethodGet, //method
        baseAPI,        //url
        nil,            //body
    )

    if err != nil {
        log.Printf("Could not request a dadjoke. %v", err)
    }

    request.Header.Add("Accept", "application/json")
    request.Header.Add("User-Agent", "Dadjoke CLI (https://github.com/example/dadjoke)")

    response, err := http.DefaultClient.Do(request)
    if err != nil {
        log.Printf("Could not make a request. %v", err)
    }

    responseBytes, err := ioutil.ReadAll(response.Body)
    if err != nil {
        log.Printf("Could not read response body. %v", err)
    }

    return responseBytes
}

New code explanations:

Line 48
- Pass the request to the http.DefaultClient.Do() method to get a response
Lines 49-51
- Handle error that is returned from http.DefaultClient.Do() method
Line 53
- Pass the resonseBody to the ioutil.ReadAll() to read it into bytes
Lines 54-56
- Handle error that is returned from ioutil.ReadAll() method
Line 58
- Return response as bytes

Finishing the getRandomJoke() method

Let's re-visit our getRandomJoke method so we can use our getJokeData method.

func getRandomJoke() {
    url := "https://icanhazdadjoke.com/"
    responseBytes := getJokeData(url)
    joke := Joke{}

    if err := json.Unmarshal(responseBytes, &joke); err != nil {
        fmt.Printf("Could not unmarshal reponseBytes. %v", err)
    }

    fmt.Println(string(joke.Joke))
}

New code explanations:

Line 2
- Store the API url in the url variable
Line 3
- Pass url into the getJokeData() method and store the returned reponse bytes in a variable
Line 4
- Create a new Joke struct. We will save data into this when we unmarshal the reponse
Lines 6-8
- Unmarshal the response, passing in responseBytes and url to http.Unmarshal as arguments
- Also handle the error that is returned
Line 10
- Convert joke.Joke to a string and print it to the terminal

Let's go back to our terminal and run the command to get a random joke:

go run main.go random

Conclusion

In this tutorial we learnt how to create a command-line application with Go and Cobra. In part 2, we will learn how to implement a flag for our random command.

If you enjoyed this article and you'd like more, consider following Div Rhino on YouTube.

Congratulations, you did great. Keep learning and keep coding. Bye for now.

divrhino / dadjoke

Build a basic CLI tool with Go and Cobra. Video tutorial available on the Div Rhino YouTube channel.

Dadjoke CLI Tool

Text tutorial: https://divrhino.com/articles/build-command-line-tool-go-cobra/
Video tutorial: https://www.youtube.com/watch?v=-tO7zSv80UY

View on GitHub

How to build a web scraper with Go and Colly

Div Rhino — Sun, 31 Jan 2021 15:10:22 +0000

Originally posted on divrhino.com

Sometimes some things just don't have an API. In those kinds of cases, you can always just write a little web scraper to help you get the data you need. In this tutorial, we're going to learn how to build a web scraper. We will also learn how to save our scraped data into a JSON file. We're going to be working with Go and the Colly package. The Colly package will allow us to crawl, scrape and traverse the DOM.

Prerequisites

To follow along, you will need to have Go installed.

Setting up project directory

Let's get started. First, change into the directory where our projects are stored. In my case this would be the "Sites" folder, it may be different for you. Here we will create our project folder called rhino-scraper

cd Sites
mkdir rhino-scraper
cd rhino-scraper

In our rhino-scraper project folder, we'll create our main.go file. This will be the entry point of our app.

touch main.go

Initialising go modules

We will be using go modulesto handle dependencies in our project.

Running the following command will create a go.mod file.

go mod init example.com/rhino-scraper

We're going to be using the colly package to build our webscraper, so let's install that now by running:

go get github.com/gocolly/colly

You will notice that running the above command created a go.sum file. This file holds a list of the checksum and versions for our direct and indirect dependencies. It is used to validate the checksum of each dependency to confirm that none of them have been modified.

In the main.go file we created earlier, let's set up a basic package main and func main().

package main

func main() {}

Analysing the target page structure

For this tutorial we will be scraping some rhino facts from FactRetriever.com.

Below is a screenshot taken from the target page. We can see that each fact has a simple structure consisting of an id and a description.

Creating the fact struct

In our main.go file, we can write a Fact struct type to represent the structure of a rhino fact. A fact will have:

an ID that will be of type int, and
a description that will be of type string.

The Fact struct type, the ID field and the Description field are all capitalised because we want them to be available outside of package main.

package main

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {}

Inside of func main, we will create an empty slice to hold our facts. We will initialise it with length zero and append to it as we go. This slice will only be able to hold Facts.

package main

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {
    allFacts := make([]Fact, 0)
}

Using the Colly package

We will be importing a package called colly to provide us with the methods and functionality we'll need to build our web scraper.

package main

import "github.com/gocolly/colly"

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {
    allFacts := make([]Fact, 0)
}

Using the colly package, let's create a new collector and set it's allowed domains to be factretriever.com

package main

import "github.com/gocolly/colly"

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {
    allFacts := make([]Fact, 0)

    collector := colly.NewCollector(
        colly.AllowedDomains("factretriever.com", "www.factretriever.com"),
    )
}

HTML structure of a list of facts

If we inspect the HTML structure, we will see that the facts are list items inside an unordered list that has the class of factsList. Each fact list item has been assigned an id. We will use this id later.

Now that we know what the HTML structure is like, we can write some code to traverse the DOM. The colly package makes use of a library called goQuery to interact with the DOM. goQuery is like jQuery, but for Golang.

Below is the code so far. We will go over the new lines, step-by-step

package main

import (
    "fmt"
    "log"
    "strconv"

    "github.com/gocolly/colly"
)

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {
    allFacts := make([]Fact, 0)

    collector := colly.NewCollector(
        colly.AllowedDomains("factretriever.com", "www.factretriever.com"),
    )

    collector.OnHTML(".factsList li", func(element *colly.HTMLElement) {
        factId, err := strconv.Atoi(element.Attr("id"))
        if err != nil {
            log.Println("Could not get id")
        }

        factDesc := element.Text

        fact := Fact{
            ID:          factId,
            Description: factDesc,
        }

        allFacts = append(allFacts, fact)
    })
}

So, here's what's happening:

We import the fmt, log and strconv packages
We are using the OnHTML method. It takes two arguments. The first argument is a target selector and the second argument is a callback function that is called everytime a target selector is encountered
In the body of the OnHTML, we create a variable to store the ID of each element that is iterated over
The ID is currently of type string, so we use strconv.Atoi to convert it to type int
The strconv.Atoi method returns an error as it's second return value, so do some basic error handling
We create a variable called factDesc to store the description text of each fact. Based on the Fact struct type we established earlier, we are expecting the fact description to be of type string.
Here, we create a new Fact struct for every list item we iterate over
Then we append the Fact struct to the allFacts slice

Begin crawling and scraping

We want to have some visual feedback to let us know that our scraper is actually visiting the page. Let's do that now.

package main

import (
    "fmt"
    "log"
    "strconv"

    "github.com/gocolly/colly"
)

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {
    allFacts := make([]Fact, 0)

    collector := colly.NewCollector(
        colly.AllowedDomains("factretriever.com", "www.factretriever.com"),
    )

    collector.OnHTML(".factsList li", func(element *colly.HTMLElement) {
        factId, err := strconv.Atoi(element.Attr("id"))
        if err != nil {
            log.Println("Could not get id")
        }

        factDesc := element.Text

        fact := Fact{
            ID:          factId,
            Description: factDesc,
        }

        allFacts = append(allFacts, fact)
    })

    collector.OnRequest(func(request *colly.Request) {
        fmt.Println("Visiting", request.URL.String())
    })

    collector.Visit("https://www.factretriever.com/rhino-facts")
}

Here's what's happening:

We use fmt.Println to output a Visting message whenever we request a URL
We use the Visit() method to give our programme a starting point

If we run our program in the terminal now, by using the command

go run main.go

It will tell us that our collector visited the rhino facts page on Fact retriever.com

Saving our data to JSON

We may want to use our scraped data in another place. So let's save it to a JSON file.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "os"
    "strconv"

    "github.com/gocolly/colly"
)

type Fact struct {
    ID          int    `json:"id"`
    Description string `json:"description"`
}

func main() {
    allFacts := make([]Fact, 0)

    collector := colly.NewCollector(
        colly.AllowedDomains("factretriever.com", "www.factretriever.com"),
    )

    collector.OnHTML(".factsList li", func(element *colly.HTMLElement) {
        factId, err := strconv.Atoi(element.Attr("id"))
        if err != nil {
            log.Println("Could not get id")
        }
        factDesc := element.Text

        fact := Fact{
            ID:          factId,
            Description: factDesc,
        }

        allFacts = append(allFacts, fact)
    })

    collector.OnRequest(func(request *colly.Request) {
        fmt.Println("Visiting", request.URL.String())
    })

    collector.Visit("https://www.factretriever.com/rhino-facts")

    writeJSON(allFacts)
}

func writeJSON(data []Fact) {
    file, err := json.MarshalIndent(data, "", " ")
    if err != nil {
        log.Println("Unable to create json file")
        return
    }

    _ = ioutil.WriteFile("rhinofacts.json", file, 0644)
}

Here's what's happening in the code above:

We import the ioutil package so we can to write to a file
We import the os package
The OS package provides an interface to operating system functionality
Let's create a function called writeJSON that takes in one parameter of the type slice of fact
Inside the function body, let's use MarshalIndent to marshal the data we pass in
The MarshalIndent method returns the JSON encoding of data and also returns an error
Some error handling. If we get an error here, we will just print a log message saying we were unable to create a JSON file
We can then use the WriteFile method it provides us to write our JSON-encoded data to a file called "rhinofacts.json"
This file does not exist yet, so the WriteFile method will create it with the permissions code of 0644.

Our WriteJSON function is ready to use. We can call it and pass allFacts to it.

Now if we go back to the terminal and run the command go run main.go, all our scraped rhino facts will be saved in a JSON file called "rhinofacts.json".

Conclusion

In this tutorial, you learnt how to build a web scraper with Go and the Colly package. If you enjoyed this article and you'd like more, consider following Div Rhino on YouTube.

Congratulations, you did great. Keep learning and keep coding!

divrhino / rhino-scraper

Learn how to build a web scraper with Go and colly. Video tutorial available on the Div Rhino YouTube channel.

Rhino Scraper

Text tutorial: https://divrhino.com/articles/build-webscraper-with-go-and-colly/
Video tutorial: https://www.youtube.com/watch?v=4VSno5bK9Uk

View on GitHub

How to add TailwindCSS to your Hugo site

Div Rhino — Sun, 03 Jan 2021 06:57:40 +0000

Originally posted on divrhino.com

Tailwind is a utility CSS library that has been gaining a lot of popularity in the past few years. Utility libraries are great because you won't end up fighting against a framework when it comes to building your own custom designs. And with Tailwind, you can build your sites faster because you won't be jumping between your HTML and CSS all the time. In this tutorial, we're going to learn how to add TailwindCSS to a Hugo project.

Prerequisites

To follow along with this tutorial, you will need to have Hugo and NodeJS installed

Installation guides:

Hugo: https://gohugo.io/getting-started/installing/
NodeJS: https://nodejs.org/en/download/

Creating Hugo project and theme

Let's start by creating a new project. Navigation into the directory where you keep all your projects, and run the command to create a new hugo site. We're going to name our site "hugotails".

cd Sites
hugo new site hugotails

Then change directories into our new "hugotails" site and create a new theme. We will call the theme "windytheme".

cd hugotails

hugo new theme windytheme

Initialising NPM and installing dependencies

Now that we've got our hugo site all ready to go, let's install tailwind. To follow along with this section of the tutorial, you will need to have Nodejs installed. So if you haven't installed it already, you might want to do so before continuing.

Make sure you're in the "hugotails" project root because we will be initialising NPM here. We can run the npm init command with the --yes flag so we don't have to go through all the setup questions.

After this command is successfully run, we will have a newly-created package.json in our project root.

npm init --yes

Now that our project has been initialised with NPM, we can go ahead and install the necessary packages. Run:

npm install --save-dev autoprefixer postcss postcss-cli postcss-import tailwindcss

We are passing in the -save-dev flag because we want to save these packages as dev dependencies.

If you noticed, we aren't just installing the tailwind package, we have a few other accompanying packages. Below is a brief description of what each of them are for:

Postcss - a tool for transforming CSS with JavaScript.
postcss-cli - the command-line tool we can use to execute Postcss commands in the terminal
postcss-import - used to resolve the path of an @import rule
autoprefixer - helps us add vendor prefixes to our CSS
tailwindcss - a library of utility classes to help us build unique custom layouts without doing too much

Setting up config files

Next up, we will tell our site to use our new windytheme theme, by adding the following line to our hugotails/config.toml file:

...
theme = "windytheme"

Now we can move onto adding tailwind as a PostCSS plugin. Since we're working with a brand new theme, we don't have a css directory yet. Let's create one. Running mkdir with the -p flag will create nested directories if they don't already exist.

mkdir -p themes/windytheme/assets/css/

Next we need to create configuration files for both PostCSS and tailwind. We will also need a main styles.scss file:

touch themes/windytheme/assets/css/postcss.config.js
touch themes/windytheme/assets/css/tailwind.config.js
touch themes/windytheme/assets/css/styles.scss

Let's tackle the postcss config first. Open up the postcss.config.js file

const themeDir = __dirname + '/../../';

module.exports = {    
    plugins: [   
        require('postcss-import')({
            path: [themeDir]
            }), 
        require('tailwindcss')(themeDir + 'assets/css/tailwind.config.js'),
        require('autoprefixer')({
            path: [themeDir]
        }),
    ]
}

Phew, that was a lot. But we're not done yet, let's head over to our tailwind.config.js and add a basic configuration file for TailwindCSS:

const themeDir = __dirname + '/../../';

module.exports = {
  content: [
    `${themeDir}/layouts/**/*.html`, 
    `${themeDir}/content/**/*.md`
  ],
  theme: {
    extend: {}
  },
  variants: {},
  plugins: []
}

Updating styles.scss

Now we have to include the necessary tailwind imports in our stylesheet. Open up our stylesheet, styles.scss, and add the following imports:

@import "node_modules/tailwindcss/base";
@import "node_modules/tailwindcss/components";
@import "node_modules/tailwindcss/utilities";

Importing CSS in head tag

When we created our new theme, Hugo helped us out by creating some starting templates files. One of these starter files is the head.html partial.

{{ $time := now }}
{{ $styles := resources.Get "css/styles.scss"
    | toCSS
    | postCSS (dict "config" "./assets/css/postcss.config.js")
    | resources.ExecuteAsTemplate (printf "styles.%s.css" $time) $time
}}

{{ if .Site.IsServer }}
    <link rel="stylesheet" href="{{ $styles.RelPermalink }}">
{{ else }}
    {{ $styles := $styles | minify | fingerprint | resources.PostCSS }}
    <link rel="stylesheet" href="{{ $styles.RelPermalink }}">
{{ end }}

So what's going on in our code above? Let's go over it line-by-line:

{{ $time := now }} - We’re getting the current time so we can use to bust our cache in local development
$styles := resources.Get "css/styles.scss" - We got our stylesheet as a resource and stored it in a variable called styles
| toCSS - We used Hugo pipes to first convert the SCSS file to CSS, so the browser can understand it
| postCSS (dict "config" "./assets/css/postcss.config.js") - We point to where our PostCSS config file lives
| resources.ExecuteAsTemplate (printf "styles.%s.css" $time) $time - We execute our stylesheet as a template so we can append a timestamp to the end of the filename
We're going to check if we're in our local dev environment by using the IsServer variable, if we are, we will link to the stylesheets relative URL
However, if we're in a production environment, we're going to use Hugo Pipes to minify our css file
We also want to pipe it through fingerprint so that we know the asset has not been tampered with. This also has a nice side-effect of cache-busting, so we also know we're serving the latest version

Testing some TailwindCSS classes

We've set up all the configuration we need. Now let's set up our index page too. This file can be found at themes/windytheme/layouts/index.html.

{{ define "main" }}
  {{ .Content }}
{{ end }}

Now we can imported our styles and transformed them in a few useful ways, let's see if everything actually works.

Let's create a basic header in the themes/windytheme/layouts/partials/header.html file:

<header>
  <h1>Welcome to HugoTails!</h1>
</header>

While still in themes/windytheme/layouts/partials/header.html, let's add some tailwind utility classes:

<header class="w-full bg-red-300">
  <h1 class="text-center">Welcome to HugoTails!</h1>
</header>

We should also start our Hugo server:

hugo server

Open your browser and visit http://localhost:1313, you should be able to see your changes there.

Conclusion

In this tutorial, you learnt how to add tailwindcss to your hugo site using Hugo Pipes. If you enjoyed this article and you'd like more, consider following Div Rhino on YouTube.

Congratulations, you did great. Keep learning and keep coding!

divrhino / hugotails

Add TailwindCSS to Your Hugo Site. Video tutorial available on the Div Rhino YouTube channel.

🚀 Gettings up and running

Run npm install to get all your node packages installed
Run hugo to build
Run hugo serve to get the server up
Visit http://localhost:1313/ to see site

Versions

Hugo v0.79.0
Node v14.15.3

Caveats

This is only tested for MacOS, so I'm not sure how to troubleshoot for Linux or Windows 😅

Companion resources

Text Tutorial: https://divrhino.com/articles/add-tailwindcss-to-hugo/
Video Tutorial: https://www.youtube.com/watch?v=8Hr19zVHCbo

View on GitHub