DEV Community: Himanshu Mishra

Building a Scalable Middleware for Face Recognition Attendance Devices Using Node.js and MQTT

Himanshu Mishra — Sat, 03 May 2025 16:14:27 +0000

Face recognition attendance systems are becoming more common in large enterprises. But integrating with them—especially when you're dealing with thousands of employees and dozens of devices—is far from straightforward. In this post, I’ll walk through how I built a scalable middleware using Node.js and MQTT to interface with smart attendance devices, handling real-time syncing, deletions, and multi-device management.

🧩 The Problem
We were working with face recognition devices that communicate via MQTT. These devices have their own set of protocols for:

Real-time recognition data uploads

Batch user syncing

User deletion

Intruder detection

Device metadata management

Our goal was to:

Sync thousands of employee records across devices.

Handle new employee additions, deletions, and updates efficiently.

Receive and process real-time attendance events.

Scale with increasing devices and users.

🔧 The Tech Stack
Node.js for building the middleware service.

MQTT.js to manage communication with devices.

MySQL for persistent storage of employee and device metadata.

Redis for job queues and temporary state tracking.

Docker + CI/CD for smooth deployments.

🏗 Architecture Overview
plaintext
Copy
Edit
Employee DB/API ↔ Middleware ↔ MQTT Broker ↔ Devices
↕
MySQL, Redis
Key Components:
mqtt.js: Handles connection, reconnection logic, message parsing, and publishing.

syncManager.js: Manages batch sync of employee data to devices.

deletionManager.js: Handles scheduled and manual deletion jobs.

eventHandler.js: Parses and logs attendance and intruder data sent from devices.

🧠 Challenges Faced

Device Acknowledgments Each device expects an acknowledgment after every message. Missing an ACK could cause it to retry or drop the operation.

Solution: Created a robust messageDispatcher that tracks correlation IDs and ensures timely ACKs with retry logic.

Bulk Employee Syncing Some devices had memory limits, so syncing 10k users at once caused crashes.

Solution: Introduced batching and throttling using a queue-based worker system:

js
Copy
Edit
const batchSize = 100;
const delay = 200; // ms
Each batch was sent with a delay, and progress was tracked via Redis.

MQTT Reconnection Logic MQTT connections sometimes dropped silently.

Solution: Added:

Heartbeat topics

Reconnection backoff

On-demand manual reconnect trigger via API

Multi-device Management Some employees needed to be synced only to specific devices.

Solution: Maintained a many-to-many relationship between employees and devices in MySQL, and dynamically selected the devices for each operation.

🛠 Sample Code Snippet
js
Copy
Edit
mqttClient.on('message', async (topic, message) => {
const parsed = JSON.parse(message.toString());

if (topic.includes('attendance')) {
await handleAttendanceLog(parsed);
} else if (topic.includes('ack')) {
ackTracker.resolve(parsed.requestId, parsed.status);
}
});
📈 Performance
After optimizing:

50+ devices could be managed concurrently.

20,000+ employee records synced without timeouts or failures.

Real-time event processing latency: < 500ms

🧪 Testing Tips
Use an MQTT simulator to mimic device behavior.

Validate ACK timeouts and retry logic with offline devices.

Monitor logs with correlation IDs for each operation.

🚀 Final Thoughts
This project taught me a lot about building real-time, fault-tolerant systems in Node.js. MQTT is powerful but tricky—it requires strict protocol adherence, retry handling, and thoughtful scaling strategies.

Have you worked with IoT devices or MQTT in production? I’d love to hear your experiences!

💬 Let’s Connect:
You can find me on GitHub [@himanshumishir] or check out my portfolio [https://himanshumishir.in].

How to send mail using Nodemailer?

Himanshu Mishra — Sun, 20 Feb 2022 18:34:30 +0000

What is nodemailer?

Nodemailer is a module for Node.js applications to allow easy as cake email sending. The project got started back in 2010 when there was no sane option to send email messages, today it is the solution most Node.js users turn to by default.

Why nodemailer?

Nodemailer features
A single module with zero dependencies – code is easily auditable, as there are no dark corners
Heavy focus on security, no-one likes RCE vulnerabilities
Unicode support to use any characters, including emoji 💪
Windows support – you can install it with npm on Windows just like any other module, there are no compiled dependencies. Use it hassle free from Azure or from your Windows box
Use HTML content, as well as plain text alternative
Add Attachments to messages
Embedded image attachments for HTML content – your design does not get blocked
Secure email delivery using TLS/STARTTLS
Different transport methods in addition to the built-in SMTP support
Sign messages with DKIM
Custom Plugin support for manipulating messages
Sane OAuth2 authentication
Proxies for SMTP connections
ES6 code – no more unintentional memory leaks, due to hoisted var’s
Autogenerated email test accounts from Ethereal.email`

Step by Step guide on how to send mail

Open terminal. mkdir node-mail cd node-mail
Create server.js file. touch server.js
Create a node app. npm init -y
Install express and nodemailer. npm install nodemailer express
Create transportOptions.js and message.js. touch message.js transportOptions.js
Open message.js and export an object. module.exports = (email)=>{ return { from: "test@mail.com", to: email, subject:"Sample mail ", text: "Hello", html: "<h1 style="color:#183b56;font-size: 28px;text-align: center;">Hello User</h1> <p style="font-size: 18px;color: #1f2933;text-align: center;">Hello this is a test mail</p>", } };
Open transportOptions.js and export an object here also. module.exports = transportOptions = { host: "smtp.office365.com", port: "587", auth: { user: "test@mail.com", pass: "PASSWORD" }, secureConnection: true, tls: { ciphers: "SSLv3" }, };
Open server.js and create an express server. const express = require('express'); const transportOptions = require('./transportOptions'); const message = require('./message'); const app = express(); app.get('/send-mail', (req, res) => { const {email} = req.body; (async () => { try { const mailTransport = nodemailer.createTransport(transportOptions); await mailTransport.sendMail(message(email)); return res.status(200).json({ message: "Successfully sent mail!", }); } catch (err) { return res.status(400).json({ message: "Sorry No such Email Exist", }); } })(); }); app.listen(3000, () => console.log('Example app is listening on port 3000.'));
Save all files and test.
Please comment for any suggestion or feedback.
You can contact me on HimanshuMishra@duck.com