loading...

Asynchronous processing of data in Expressjs

email2vimalraj profile image Vimalraj Selvam ・1 min read

I've an Express route which receives some data and process it, then insert into mongo (using mongoose).

This is working well if I return a response after the following steps are done:

  • Receive request
  • Process the request data
  • Insert the processed data into Mongo
  • Return 204 response

But client will be calling this API concurrently for millions of records. Hence the requirement is not to block the client for processing the data. Hence I made a small change in the code:

  • Receive request
  • Return response immediately with 204
  • Process the requested data
  • Insert the processed data into Mongo

The above is working fine for the first few requests (say 1000s), after that client is getting socket exception: connection reset peer error. I guess it is because server is blocking the connection as the port is not free and at some point of time, I notice my nodejs process is throwing out Out of memory error.

Sample code is as follows:

async function enqueue(data) {
    // 1. Process the data
    // 2. Insert the data in mongo
}

async function expressController(request, response) {
    logger.info('received request')
    response.status(204).send()

    try {
        await enqueue(request.body)
    } catch (err) {
        throw new Error(err)
    }
}

Am I doing something wrong here?

Posted on by:

email2vimalraj profile

Vimalraj Selvam

@email2vimalraj

Full stack developer and have a history of QA role

Discussion

markdown guide
 

For this use case I suggest using a message broker like RabbitMQ or Google PubSub etc. In the controller you only have to insert the request body into a queue and do the processing and inserting to mongo in completely separate and scalable a worker process by subscribing to that topic.

Also if you need something really fast, check out Fastify and it's benchmarks.

 

Yeah I thought through having a queue which will take care of this process. I'll see if I can put it in a message queue.

Regarding fastify, i'll definitely give a try.