DEV Community: Kane Ong

A recommended way to do functional testing with K.O architecture

Kane Ong — Fri, 08 Nov 2019 03:17:46 +0000

In my last article, I introduced a new architecture pattern to improve the productivity of express.js API development. Functional testing with the new architecture requires another article to discuss, so here it is.

How to read this article

It is recommended to read my previous article before continuing.

A Minimalist Architecture Pattern for Express.js API Applications

Kane Ong ・ Oct 1 '19

#productivity #javascript #express #webdev

Demo project

Let's reuse the last project that we created.

dividedbynil / ko-architecture

A Minimalist Architecture Pattern for ExpressJS API Applications

K.O Architecture Demo

Framework: ExpressJS
Database: MongoDB
Authentication: JSON Web Token

Experiment data

origin: restaurants.json

APIs document

Postman APIs collection and environment can be imported from ./postman/

Pre-running

Update the ./config.js file

module.exports = {
  saltRounds: 10,
  jwtSecretSalt: '87908798',
  devMongoUrl: 'mongodb://localhost/kane',
  prodMongoUrl: 'mongodb://localhost/kane',
  testMongoUrl: 'mongodb://localhost/test',
}

Import experiment data

Open a terminal and run:

mongod

Open another terminal in this directory:

bash ./data/import.sh

Start the server with

npm start

Start development with

npm run dev

View on GitHub

Prerequisite

Import testing libraries

npm i -D mocha chai

Here we choose mocha as the testing framework, chai as the assertion library.

Let's add a test script in package.json.

"scripts": {
  ...
  "test": "./node_modules/mocha/bin/mocha"
},

Unit testing

`./common/response.js`

function errorRes (res, err, errMsg="failed operation", statusCode=500) {
  console.error("ERROR:", err)
  return res.status(statusCode).json({ success: false, error: errMsg })
}

function successRes (res, data, statusCode=200) {
  return res.status(statusCode).json({ success: true, data })
}

...

The res arguments in the functions above are slightly tricky for unit testing. However, it is not difficult to mock res.

For example (the code below is using jest testing framework):

const { errData, errorRes, successRes } = require('./response')

// mocking res
const res = () => ({
  err: '',
  errMsg: '',
  statusCode: 0,
  result: {},
  status (code) {
    this.statusCode = code
    return this
  },
  json (obj) {
    this.result = obj
    return this
  }
})

describe('errorRes', () => {
  it('should return default error status code', () => {
    const obj = errorRes(res(), '')
    expect(obj.statusCode).toBe(500)
  })

  it('should return custom error status code', () => {
    const obj = errorRes(res(), '', '', 401)
    expect(obj.statusCode).toBe(401)
  })

  it('should return default errMsg', () => {
    const obj = errorRes(res(), '')
    expect(obj.result.error).toBe('failed operation')
  })

  it('should return custom errMsg', () => {
    const errMsg = 'invalid operation'
    const obj = errorRes(res(), '', errMsg)
    expect(obj.result.error).toBe(errMsg)
  })
})
...

`common/crud.js`

const { errData, errorRes, successRes } = require('../common/response')
const mongoose = require('mongoose')


function create (model, populate=[]) {
  return (req, res) => {
    const newData = new model({
      _id: new mongoose.Types.ObjectId(),
      ...req.body
    })
    return newData.save()
      .then(t => t.populate(...populate, errData(res)))
      .catch(err => errorRes(res, err))
  }
}

function read (model, populate=[]) {
  return (req, res) => (
    model.find(...req.body, errData(res)).populate(...populate)
  )
}
...

The model arguments in the functions above are not that simple to do mocking. Feel free to try it out and leave a code snippet in the comment if you made it.

A recommended way

A recommended way to test K.O architecture is to use an HTTP assertion library to do bottom-up integration testing. By doing so we can avoid creating complex mock functions.

Import http assertion library - supertest

npm i -D supertest

Let's create a minimal express app to test the functions in common/response.js.

test/response.test.js

const { errData, errorRes, successRes } = require('../common/response')
const request = require('supertest');
const bodyParser = require('body-parser');
const express = require('express');
const app = express();

app
.use(bodyParser.json())
.post('/errorRes', (req, res) => errorRes(res, ...req.body))
.post('/successRes', (req, res) => successRes(res, ...req.body))
.post('/errData', (req, res) => {
  const { errMsg, err } = req.body
  errData(res, errMsg) (err, { custom: true })
})

Using the post HTTP method is more convenient as we utilize req.body as the function argument carrier.

Let's test the express app.

test/response.test.js

function test (funcStr, args) {
  return request(app)
          .post('/'+funcStr)
          .send(args)
}

describe ('response', function () {
  describe('errorRes', function () {
    it('should return default status code & message', function (done) {
      test('errorRes', ['error'])
        .expect(500, {
          success: false,
          error: 'failed operation'
        }, done)
    })

    it('should return custom errMsg', function (done) {
      test('errorRes', ['error', 'test'])
        .expect(500, {
          success: false,
          error: 'test'
        }, done)
    })

    it('should return custom errMsg & status code', function (done) {
      test('errorRes', ['error', 'test', 401])
        .expect(401, {
          success: false,
          error: 'test'
        }, done)
    })
  })

  describe('successRes', function () {
    it('should return default status code & data', function (done) {
      test('successRes', [])
        .expect(200, {
          success: true,
          data: {}
        }, done)
    })

    it('should return custom data', function (done) {
      test('successRes', [{ custom: true }])
        .expect(200, {
          success: true,
          data: { custom: true }
        }, done)
    })

    it('should return custom data & status code', function (done) {
      test('successRes', [{ custom: true }, 201])
        .expect(201, {
          success: true,
          data: { custom: true }
        }, done)
    })
  })

  describe('errData', function () {
    it('should return default status code & error', function (done) {
      test('errData', { err: true })
        .expect(500, {
          success: false,
          error: 'failed operation'
        }, done)
    })

    it('should return default status code & data', function (done) {
      test('errData', { err: false })
        .expect(200, {
          success: true,
          data: { custom: true }
        }, done)
    })

    it('should return custom error message', function (done) {
      test('errData', { err: true, errMsg: 'custom' })
        .expect(500, {
          success: false,
          error: 'custom'
        }, done)
    })
  })
})

Let's create another minimal express app to test functions in common/crud.js.

test/curd.test.js

const { create, read, update, remove } = require('../common/crud')
const Restaurant = require('../models/Restaurant')
const User = require('../models/User')
const request = require('supertest')
const expect = require('chai').expect
const bodyParser = require('body-parser')
const express = require('express')
const mongoose = require('mongoose')
// check the updated github readme for example
const { testMongoUrl } = require('../config')
const app = express()

mongoose.connect(testMongoUrl, {
  useNewUrlParser: true,
  autoIndex: false,
  useFindAndModify: false,
  useUnifiedTopology: true,
})

app
.use(bodyParser.json())
.post('/create', create(Restaurant, ['owner']))
.post('/read', read(Restaurant, ['owner']))
.post('/update/:_id', update(Restaurant, ['owner']))
.post('/remove/:_id', remove(User))

Using a new mongo database (testMongoUrl) to do testing is more convenient to remove testing data after each test run and not interfere with existing data.

function test (funcStr, args) {
  return request(app)
          .post('/'+funcStr)
          .send(args)
}

describe('crud', function () {
  const user = {
    "name": "Test",
    "email": "test@test.com",
    "type": "admin",
    "password": "12345"
  }

  const restaurant = {
    "name": "Brand New Restaurant",
    "location": {
      "type": "Point",
      "coordinates": [-73.9983, 40.715051]
    },
    "available": true
  }

  const testUser = new User({
    _id: new mongoose.Types.ObjectId(),
    ...user
  })

  before(function (done) {
    testUser.save(done)
  })

  after(function (done) {
    mongoose.connection.db.dropDatabase(function(){
      mongoose.connection.close(done)
    })
  })

  describe('create', function () {
    it('should not create new restaurant without owner', function (done) {
      const data = restaurant
      test('create', data)
        .expect(500, done)
    })
    it('should return new restaurant and populate', function (done) {
      const data = { owner: testUser._id, ...restaurant }
      test('create', data)
        .expect(200)
        .expect(function (res) {
          restaurant._id = res.body.data._id
          expect(res.body.data).to.be.a('object')
          expect(res.body.data).to.deep.include(restaurant)
          expect(res.body.data.owner.name).to.equal('Test')
          expect(res.body.data.owner.password).to.equal(undefined)
        })
        .end(done)
    })
  })

  describe('read', function () {
    it('should return restaurant and populate', function (done) {
      const data = [{ "available": true, _id: restaurant._id }]
      test('read', data)
        .expect(200)
        .expect(function (res) {
          expect(res.body.data).to.be.a('array')
          expect(res.body.data[0]).to.deep.include(restaurant)
          expect(res.body.data[0].owner.name).to.equal('Test')
          expect(res.body.data[0].owner.password).to.equal(undefined)
        })
        .end(done)
    })
    it('should return no restaurant', function (done) {
      const data = [{ "available": false }]
      test('read', data)
        .expect(200)
        .expect(function (res) {
          expect(res.body.data.length).to.equal(0)
        })
        .end(done)
    })
  })

  describe('update', function () {
    it('should return updated restaurant and populate', function (done) {
      const { _id } = restaurant
      const data = { name: 'New name' }
      test('update/'+_id, data)
        .expect(200)
        .expect(function (res) {
          expect(res.body.data).to.be.a('object')
          expect(res.body.data).to.deep.include({...restaurant, ...data})
          expect(res.body.data.name).to.equal('New name')
          expect(res.body.data.owner.name).to.equal('Test')
          expect(res.body.data.owner.password).to.equal(undefined)
        })
        .end(done)
    })
  })

  describe('remove', function () {
    it('should remove user', function (done) {
      const { _id } = testUser
      test('remove/'+_id)
        .expect(200)
        .expect(function (res) {
          expect(res.body.data).to.be.a('object')
          expect(res.body.data).to.have.property('ok')
          expect(res.body.data.ok).to.equal(1)
        })
        .end(done)
    })
  })

})

Let's create another minimal express app to test functions in common/middleware.js.

test/middleware.test.js

const { notFound, onlyAdmin, notOnlyMember } = require('../common/middleware')
const request = require('supertest');
const bodyParser = require('body-parser');
const express = require('express');
const app = express();

app
.use(bodyParser.json())
.post('/notFound', notFound)
.post('/onlyAdmin', addUserType, onlyAdmin, notFound)
.post('/notOnlyMember', addUserType, notOnlyMember, notFound)

function addUserType (req, res, next) {
  req.user = { type: req.body.type }
  next()
}

function test (funcStr, args) {
  return request(app)
          .post('/'+funcStr)
          .send({ type: args })
}

describe('middleware', function () {
  describe('notFound', function () {
    it('should return not found', function (done) {
      test('notFound')
        .expect(404, done)
    })
  })

  describe('onlyAdmin', function () {
    it('should access', function (done) {
      test('onlyAdmin', 'admin')
        .expect(404, done)
    })
    it('should not access', function (done) {
      test('onlyAdmin', 'member')
        .expect(401, done)
    })
  })

  describe('notOnlyMember', function () {
    it('should access', function (done) {
      test('notOnlyMember', 'admin')
        .expect(404, done)
    })
    it('should access', function (done) {
      test('notOnlyMember', 'owner')
        .expect(404, done)
    })
    it('should not access', function (done) {
      test('notOnlyMember', 'member')
        .expect(401, done)
    })
  })
})

End-to-end testing

After testing all functions in the common folder, it's up to you to continue end-to-end testing in the mocha framework or use other tools.

Feel free to leave a comment if you have any questions.

A Minimalist Architecture Pattern for Express.js API Applications

Kane Ong — Tue, 01 Oct 2019 16:40:20 +0000

Express.js is a minimal web application framework that improves the productivity of web developers. It is very flexible and does not enforce any architecture pattern. This article demonstrates a new architecture pattern which I designed that will further improve your productivity.

How to read this article

This article introduces a pattern that is different from the popular MVC or MSC (Model-Service-Controller) pattern. You can read this article before knowing any of those.

Demo project GitHub

Let's create a restaurant app RESTful API.

Access rules

Public users:
- create an account
- log in
Members:
- read all nearby available restaurants
Owners:
- CRUD all nearby restaurants
Admins:
- CRUD all nearby restaurants
- CRUD all users

Requirements

Each restaurant object must have a name, geolocation coordinates, available status, and owner id.
Each user object must have a name, email, user type (member/owner/admin), and password.

Tech stack in this demo

Database: MongoDB
ORM: Mongoose

JSON Response convention

When we send a JSON data back to the client, we may have conventions that identify a successful or failed operation, for example



{
  success: false,
  error: ...
}



{
  success: true,
  data: ...
}

Let's create functions for the JSON responses above.

./common/response.js



function errorRes (res, err, errMsg="failed operation", statusCode=500) {
  console.error("ERROR:", err)
  return res.status(statusCode).json({ success: false, error: errMsg })
}

function successRes (res, data, statusCode=200) {
  return res.status(statusCode).json({ success: true, data })
}

Here we use default arguments for both functions, the benefit here is we can use the function as:



errorRes(res, err)
successRes(res, data)

and we don't have to check if the optional arguments are null.



// Example when default arguments not in use.
function errorRes (res, err, errMsg, statusCode) {
  if (errMsg) {
    if (statusCode) {
      ...
    }
    ...
  }
}

// or using ternary operator
function successRes (res, data, statusCode) {
  const resStatusCode = statusCode ? statusCode : 200
  ...
}

Feel free to replace console.error with logging function (from other library) you prefer.

Database async callback convention

For create, read, update and delete operations, most database ORMs/drivers have a callback convention as:



(err, data) => ...

knowing this, let's add another function in ./common/response.js

./common/response.js



function errData (res, errMsg="failed operation") {
  return (err, data) => {
    if (err) return errorRes(res, err, errMsg)
    return successRes(res, data)
  }
}

Export all functions in ./common/response.js



module.exports = { errorRes, successRes, errData }

Database operations (CRUD) conventions

Let's define the database operations functions for all models. The conventions here are using req.body as the data source and req.params._id as collections' object id. Most of the functions will take a model and a list of populating fields as arguments, except delete operation (it is unnecessary to populate a deleted record). Since delete is a reserved keyword in JavaScript (for removing a property from an object), we use remove as the delete operation function name to avoid confliction.

./common/crud.js



const { errData, errorRes, successRes } = require('../common/response')
const mongoose = require('mongoose')


function create (model, populate=[]) {
  return (req, res) => {
    const newData = new model({
      _id: new mongoose.Types.ObjectId(),
      ...req.body
    })
    return newData.save()
      .then(t => t.populate(...populate, errData(res)))
      .catch(err => errorRes(res, err))
  }
}

function read (model, populate=[]) {
  return (req, res) => (
    model.find(...req.body, errData(res)).populate(...populate)
  )
}

function update (model, populate=[]) {
  return (req, res) => {
    req.body.updated_at = new Date()
    return model.findByIdAndUpdate(
            req.params._id,
            req.body,
            { new: true },
            errData(res)
          ).populate(...populate)
  }
}

function remove (model) {
  return (req, res) => (
    model.deleteOne({ _id: req.params._id }, errData(res))
  )
}

module.exports = { read, create, update, remove }

The database CRUD function above used the functions from ./common/response.

Ready for development

With all the functions above defined, we are ready for application development. We now only require to define data models and routers.
Let's define the data models in ./models

./models/Restaurant.js



const mongoose = require('mongoose')
const Schema = mongoose.Schema
const ObjectId = Schema.Types.ObjectId
const validator = require('validator')


const restaurantSchema = new Schema({
  _id: ObjectId,
  name: { type: String, required: true },
  location: {
    type: {
      type: String,
      enum: [ 'Point' ],
      required: true
    },
    coordinates: {
      type: [ Number ],
      required: true
    }
  },
  owner: { type: ObjectId, ref: 'User', required: true },
  available: {
    type: Boolean,
    required: true,
  },

  updated_at: Date,
});

module.exports = mongoose.model('Restaurant', restaurantSchema, 'restaurants');

./models/User.js



const mongoose = require('mongoose')
const Schema = mongoose.Schema
const ObjectId = Schema.ObjectId
const validator = require('validator')


const userSchema = new Schema({
  _id: ObjectId,
  name: { type: String, required: true },
  email: {
    type: String,
    required: true,
    unique: true,
    validate: [ validator.isEmail, 'invalid email' ]
  },
  type: {
    type: String,
    enum: ['member', 'owner', 'admin'],
    required: true
  },
  password: { type: String, required: true, select: false },

  updated_at: Date,
});

module.exports = mongoose.model('User', userSchema, 'users');

The models above are very common, nothing new or fancy within.

Routing and handlers

From the database convention above, you may think that using req.body as the data source is very limited if one needs a backend-process JSON field. Here we can use middleware to resolve the limitation.

./api/user.js



router
.use(onlyAdmin)
.post('/', create(User))
.get('/all/:page', usersAtPage, read(User))
.put('/:_id', handlePassword, update(User))
.delete('/:_id', remove(User))

./api/restaurant.js



const express = require('express')
const router = express.Router()
const { create, read, update, remove } = require('../common/crud')
const Restaurant = require('../models/Restaurant')

router
.get('/all/:lng/:lat/:page', nearBy(), read(Restaurant, ['owner']))
.get('/available/:lng/:lat/:page',
  nearBy({ available: true }),
  read(Restaurant, ['owner'])
)

function nearBy (query={}) {
  return (req, res, next) => {
    const { lng, lat, page } = req.params
    req.body = geoQuery(lng, lat, query, page)
    next()
  }
}

./api/auth.js



router
.post('/signup', isValidPassword, hashPassword, signUp)
.post('/login', isValidPassword, findByEmail, verifyPassword, login)

// middlewares below are used for processing `password` field in `req.body`
function isValidPassword (req, res, next) {
  const { password } = req.body
  if (!password || password.length < 6) {
    const err = `invalid password: ${password}`
    const errMsg = 'password is too short'
    return errorRes(res, err, errMsg)
  }
  return next()
}

function hashPassword (req, res, next) {
  const { password } = req.body
  bcrypt.hash(password, saltRounds, (err, hashed) => {
    if (err)
      return errorRes(res, err, 'unable to sign up, try again')
    req.body.password = hashed
    return next()
  })
}

function signUp (req, res) {
...
}

function findByEmail (req, res, next) {
....
}

function verifyPassword (req, res, next) {
  ...
}

function login (req, res) {
  ...
}

module.exports = router;

How to extend

Extending the application only requires to add new models and define new routers for endpoints.

Differences from MSC

The Model-Service-Controller pattern requires every database model to have a set of service functions for data operations. And those service functions are only specifically defined for a particular model. With the new architecture above, we skip the definition of service functions for each model by reusing the common database operations functions, hence improving our productivity.

Summary

This architecture provides great flexibility for customization, for example, it does not enforce a folder structure other than having a common folder, you are free from putting all middleware functions in router files or separating them by your rules. By using and extending the functions in the common folder, you can either start a project from scratch or refactor/continue a large project productively. So far I have been using this architecture for any size of ExpressJS projects.