This article is part of the Aspecto Hello World series, where we tackle microservices-related topics for you. Our team searches the web for common issues, then we solve them ourselves and bring you complete how-to guides. Aspecto is an OpenTelemetry-based distributed tracing platform for developers and teams of distributed applications.
Introduction
Integration tests are never fun to write.
Setting up an environment, finding a way to reproduce the needed state of it, populating it with relevant data, etc – It’s all work no play.
In this blog post, I will show you how to write an integration test and talk about why you want to do it.
The motivation for writing an integration test
Let’s begin by talking about why we even want to write one.
So you finished writing all of the code of your task and everything is working.
How do you make sure it stays this way and does not break when another developer changes just one tiny thing that happens to break your feature? (Not on purpose of course, or so we hope).
The solution: Instead of moving onto the next feature right away, you can write an integration test to make sure that your team gets notified if something breaks (by running it in a CI/CD pipeline that alerts you).
The role of the integration test is to verify that not only does each function you wrote works as a standalone (that’s unit tests), but also to verify that all of the functions play out well together.
Spoiler alert: in this guide, I will also show you how to make sure that a database received the relevant params.
What we will build:
I like to keep everything as simple as possible, giving you only what you need to get on with your day.
Therefore, I will build a simple to-do app: a nodejs express service that authenticates the user, with endpoints for creating & viewing his tasks.
The integration test will verify the following:
- Login
- Create a new TODO item
- Verify the todo item was saved with the relevant email & todo text was saved to the database
- Fetch all todo items
- Verify that the relevant email was used to query the database
The tools I will be using:
- NodeJS + Express
- Jest – a framework for authoring and executing nodejs tests
- Malabi – a Trace-Based Testing library that lets me make assertions on real data passed to moving parts like mongodb / elasticsearch and more (I will further elaborate on this below).
Part 1 – Setting up the tested microservice
Note: If you already have a service you wish to test, feel free to skip to part 2 where I show how to write the actual integration test.
Step 1 – Set up the project
Let’s use express-generator to generate the initial code for our express app, with the pug view engine.
Also, let’s install the packages we will be using:
npx express-generator -v pug
npm i --save jsonwebtoken mongoose passport passport-jwt axios
We will be storing the data in a MongoDB database, which you can run locally using docker like this:
docker pull mongo
docker run -d -p 27017:27017 mongo
Step 2 – create the views
We’ll add a very simple UI for our imaginary users to add TODO items.
login.pug
extends layout
block content
body
form(action='/auth/login?redirectToTodos=true', method='POST')
p
| username:
input(type='text', name='username', value='tom@a.com')
| <br/>password:
input(type='password', name='password', value='password')
input(type='submit', value='Submit')
app.pug
extends layout
block content
p Welcome to the app screen <b>#{user}</b>
form(action='/todos/', method='POST')
p
| new todo text:
input(type='text', name='username', value='hi')
input(type='submit', value='Submit')
ul
each val in todos
Step 3 – Adding routes
In the routes folder, let’s delete the users’ file, we won’t be using it.
Now place the following code in the index.js file.
Index.js
const var express = require('express');
const router = express.Router();
const jwt = require('jsonwebtoken');
const TOKEN_SECRET = 'SECRET';
router.get('/login', function(req, res, next) {
res.render('login', { title: 'Express' });
});
router.post('/auth/login', function(req, res, next) {
const email = req.body.username;
const token = jwt.sign({ email }, TOKEN_SECRET, {
expiresIn: 60 * 60,
});
res.cookie('auth', token, { httpOnly: true });
req.query.redirectToTodos ? res.redirect('/todos') : res.json({ success: true });
});
module.exports = router;
Here we have two endpoints – one for rendering the login view for our users, and the other one for performing the login with a username & password. We’re not verifying passwords here, as authentication is not the focus of the blog.
The way it works in general: login, sign a JWT token with email, send to client with the response.
I added a small modification here – redirectToTodos param, to avoid writing client-side code.
In real life, I would not add it and have the client handle the JSON response. For the purpose of this post, it’s good enough.
Let’s add another routes file called todos.js:
const express = require('express');
const router = express.Router();
const passport = require('passport');
const mongoose = require('mongoose');
async function connectToMongoose() {
await mongoose.connect('mongodb://localhost:27017/test');
}
connectToMongoose().catch(err => console.log(err));
const todoSchema = new mongoose.Schema({
text: String,
email: String
});
const Todo = mongoose.model('Todo', todoSchema);
router.get('/',
(req, res, next) => {
passport.authenticate('jwt', { session: false }, async (err, user, info) => {
if (err) {
console.log('error is', err);
res.status(500).send('An error has occurred, we cannot greet you at the moment.');
}
else {
console.log('user is', user);
console.log('info is', info);
const { email } = user;
const todos = await Todo.find({ email });
res.render('app',{ success: true, user: email, todos });
}
})(req, res, next);
});
router.post('/',
(req, res, next) => {
passport.authenticate('jwt', { session: false }, async (err, user, info) => {
if (err) {
console.log('error is', err);
res.status(500).send('An error has occurred, we cannot greet you at the moment.');
}
else {
console.log('user is', user);
console.log('info is', info);
const todo = new Todo({ text: 'Hey that is my new todo', email: user.email });
await todo.save();
res.json({ success: true });
}
})(req, res, next);
});
module.exports = router;
In this file I define the mongoose Todo model and add two endpoints: GET for retrieving all todos for the current user, and POST for creating one.
I use the passport-jwt strategy to decode the user’s email (which is sent with the request as a cookie)
Step 4 – Adding app.js
The app.js file contains different setups for authentication & routing. I won’t dive into this deeply. If you’re curious feel free to check out my guide to microservices authentication strategies.
var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');
var indexRouter = require('./routes/index');
var todoRouter = require('./routes/todos');
var app = express();
// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');
app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
const passport = require('passport');
const JwtStrategy = require('passport-jwt').Strategy,
ExtractJwt = require('passport-jwt').ExtractJwt;
app.use(passport.initialize());
app.use(passport.session());
app.use('/', indexRouter);
app.use('/todos', todoRouter);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
next(createError(404));
});
// error handler
app.use(function(err, req, res, next) {
// set locals, only providing error in development
res.locals.message = err.message;
res.locals.error = req.app.get('env') === 'development' ? err : {};
// render the error page
res.status(err.status || 500);
res.render('error');
});
const cookieExtractor = function(req) {
let token = null;
if (req && req.cookies)
{
token = req.cookies['auth'];
}
return token;
};
const TOKEN_SECRET = 'SECRET';
const opts = {
jwtFromRequest: ExtractJwt.fromExtractors([cookieExtractor]),
secretOrKey: TOKEN_SECRET,
};
passport.use(
'jwt',
new JwtStrategy(opts, (jwt_payload, done) => {
try {
console.log('jwt_payload', jwt_payload);
done(null, jwt_payload);
} catch (err) {
done(err);
}
}),
);
module.exports = app;
Step 5 – Running the app
Now you can simply run the app using and go to localhost:3000/login, login and submit the form to create a todo item and see that it is working.
npm start
Part 2: Writing an integration test
In this part, we will be using jest & Malabi to write our integration test.
Jest is a library that lets you author and execute tests for nodejs.
As for Malabi – an explanation follows.
Malabi is an open-source Javascript framework based on OpenTelemetry that allows you to leverage trace data and improve assertion capabilities. This library introduces a new way of testing services: Trace-based testing (TBT).
Here’s a quick tutorial on Malabi and how it works:
Trace based testing
So what is Trace Based Testing anyway? Good thing you ask.
It is a new approach that utilizes OpenTelemetry to our advantage in testing.
Opentelemetry gives us SDKs that let us instrument our application, meaning to create data of what calls are being made to endpoints, databases and essentially is aimed at giving us visibility on our microservices.
You can read more about Malabi and Trace-based testing in this blog post.
By instrumenting our tested microservice, we’re able to make assertions on what API calls were made, with what parameters.
We can also assert how a database was queried – in our case, which exact parameter did Mongo receive when it was asked for all todos by the current user. We want to assert the current user and know we don’t have a bug causing us to log in with one user but query another user’s data.
Malabi wraps the opentelemetry SDK, creates spans (actions performed in the service), and exposes an HTTP endpoint for you to query & assert on.
Enough theory for now – Let’s begin.
Step 1 – perform installs
npm install --save-dev jest malabi
Add the following lines at the top of our app.js file
const malabi = require('malabi');
malabi.instrument();
malabi.serveMalabiFromHttpApp(18393);
This tells Malabi to create spans for us (actions we can assert on, for example – a MongoDB query and an HTTP request are 2 examples of possible spans).
It also tells malabi to expose an API endpoint at port 18393 that lets the test runner get these spans and then assert on them.
Step 2- add a new folder & test file: tests/integ.spec.js
const axios = require('axios').default;
const { fetchRemoteTelemetry, clearRemoteTelemetry } = require('malabi');
const getMalabiTelemetryRepository = async () => await fetchRemoteTelemetry({ portOrBaseUrl: 18393 });
describe('testing service-under-test remotely', () => {
beforeEach(async () => {
// We must reset all collected spans between tests to make sure span aren't leaking between tests.
await clearRemoteTelemetry({ portOrBaseUrl: 18393 });
});
it('successful /todo request', async () => {
// simulate login
const loginRes = await axios.post(`http://localhost:3000/auth/login`, { username: 'tom@a.com', password: 'password' });
const authCookie = loginRes.headers['set-cookie'];
// Create a new todo item
const newTodoRes = await axios.post(`http://localhost:3000/todos`, {}, { headers: {
Cookie: authCookie
} });
// console.log('newTodoRes', newTodoRes);
// call to the service under test - internally it will call another API to fetch the todo items.
await axios(`http://localhost:3000/todos/`, { headers: {
Cookie: authCookie
} });
// Get instrumented spans
const repo = await getMalabiTelemetryRepository({ portOrBaseUrl: 13893 });
// This is the span that holds data from the creation of the new todo item
const postTodoSpan = repo.spans.mongo().first;
const emailUsedForCreatingNewTodo = JSON.parse(postTodoSpan.dbStatement).document.email;
console.log('emailUsedForCreatingNewTodo', emailUsedForCreatingNewTodo);
// Assert the email was saved equals the email we logged in with
expect(emailUsedForCreatingNewTodo).toEqual("tom@a.com")
// This is the span that holds data from the fetching of todos from mongo for current user
const getTodosSpan = repo.spans.mongo().second;
const emailUsedInMongoQuery = JSON.parse(getTodosSpan.dbStatement).condition.email;
// Assert the email was used for querying equals the email we logged in with
expect(emailUsedInMongoQuery).toEqual("tom@a.com")
});
});
A few things to note about the above file:
getMalabiTelemetryRepository – fetches spans created in the current run, from the malabi endpoint.
We can then use the received object to query for specific span types, like all Mongo spans, all http requests, etc.
We also have clearRemoteTelemetry that cleans up the in-memory cache of spans between test runs. This helps maintain our test clean & not polluted between test runs.
Notice we have made 2 assertions: one that the post request saved the todo item with the relevant email (the one we logged in with), and the second one verifies that we fetched only todos that belong to that user.
You can now run the test using the “jest” command, and see that everything is working as expected.
As you can see, this is a very simple and clean way we can make sure the connections between our different functions and moving parts are working as expected.
Of course, in production, you would most likely set this up as part of a CI/CD workflow, but this is not the focus of this guide.
That’s basically it, I hope this guide was useful for you, feel free to reach out to me @magnificoder for any questions you may have!
Feel free to check out some of my other articles, like this one: How to Deploy Jaeger on AWS: a Comprehensive Step-by-Step Guide.
Top comments (0)