Save Transcripts to MongoDB with a Node.js Webhook

By Vikram Vaswani, Developer Advocate

This tutorial was originally published at https://docs.rev.ai/resources/tutorials/save-transcripts-mongodb-webhooks/ on Jul 12, 2022.

Introduction

Webhooks provide a simple and efficient way to handle events that occur asynchronously outside an application's scope. They are typically set up as URL endpoints that are invoked by upstream applications on specific events. They accept incoming HTTP requests and in turn trigger application-specific processes, such as generating a notification or updating a status record.

Rev AI's Asynchronous Speech-to-Text API uses webhooks to notify applications when a transcription job is complete. On receipt of this notification, the application usually needs to retrieve the transcript and "does something" with it. This "something" could involve saving the transcript to a database, providing the user with the option to edit and correct it, or processing it in a different way.

This tutorial demonstrates how to implement a webhook in a Node.js application for one such use case. It gives you the information (and code) you need to create a webhook that, on job completion, is able to request a complete transcript from Rev AI and save it to an external MongoDB database for further processing.

Assumptions

This tutorial assumes that:

• You have a Rev AI account and access token. If not, sign up for a free account and generate an access token.
• You have a properly-configured Node.js development environment with Node.js v16.x or v17.x. If not, download and install Node.js for your operating system.
• You have some familiarity with the Express framework. If not, familiarize yourself with the basics using this example application.
• You have write privileges to a MongoDB database. If not, download and install MongoDB for your operating system and configure access control, or deploy MongoDB in the cloud and configure a database, user and role with write privileges.
• Your webhook will be available at a public URL. If not, or if you prefer to develop and test locally, download and install ngrok to generate a temporary public URL for your webhook.
• You have an audio file to transcribe. If not, use this example audio file from Rev AI.

Step 1: Install required packages

This tutorial will use:

• The Rev AI Node SDK, to submit transcription requests to the Rev AI Asynchronous Speech-to-Text API;
• The MongoDB Node.js Driver, to save data to MongoDB;
• The Express Web framework and body-parser middleware, to receive and parse webhook requests.

Begin by installing the required packages:

npm i revai-node-sdk mongodb express body-parser

Step 2: Create the webhook handler

The webhook URL is specified as part of the job parameters submitted to the Rev AI API. On job completion, the Rev AI API will send an HTTP POST request containing JSON-encoded details of the completed job to the webhook URL. Here is an example of one such POST request:

{
  job: {
    id: 'Qctu6UgJvTjZ',
    created_on: '2022-04-08T15:01:35.999Z',
    completed_on: '2022-04-08T15:02:05.403Z',
    name: 'FTC_Sample_1.mp3',
    callback_url: 'https://a177-49-36-111-113.ngrok.io/hook',
    media_url: 'https://www.rev.ai/FTC_Sample_1.mp3',
    status: 'transcribed',
    duration_seconds: 107,
    type: 'async',
    language: 'en'
  }
}

The following example demonstrates how to implement a webhook handler that receives and parses the HTTP POST message from the Rev AI API and then makes a subsequent request to the API to retrieve the complete transcript. The handler then saves the received data to a MongoDB database collection as a JSON document.

To use this example, you must replace the <MONGODB_CONNECTION_URI> with the connection URI to your MongoDB database and the <REVAI_ACCESS_TOKEN> placeholder with your Rev AI account's access token.

const { RevAiApiClient } = require('revai-node-sdk');
const { MongoClient } = require('mongodb');
const bodyParser = require('body-parser');
const express = require('express');

// MongoDB connection string
const mongodbUri = '<MONGODB_CONNECTION_URI>';

// Rev AI access token
const revAiToken = '<REVAI_ACCESS_TOKEN>';

// create Express application
const app = express();
app.use(bodyParser.json());

// create Mongo client
const mongo = new MongoClient(mongodbUri);
mongo.connect();
const db = mongo.db('mydb');
const transcripts = db.collection('transcripts')

// create Rev AI API client
const revAiClient = new RevAiApiClient(revAiToken);

// handle requests to webhook endpoint
app.post('/hook', async req => {
  const job = req.body.job;
  console.log(`Received status for job id ${job.id}: ${job.status}`);

  if (job.status === 'transcribed') {
    // request transcript
    const transcript = await revAiClient.getTranscriptObject(job.id);
    console.log(`Received transcript for job id ${job.id}`);

    // create MongoDB document
    const doc = {
      job_id: job.id,
      created_on: job.created_on,
      language: job.language,
      status: job.status,
      transcript
    }

    // save document to MongoDB
    try {
      const result = await collection.insertOne(doc);
      console.log(`Saved transcript with document id: ${result.insertedId}`);
    } catch (e) {
      console.error(e);
    }
  }
});

//  start application on port 3000
app.listen(3000, () => {
  console.log('Webhook listening');
})

Save this code listing as index.js and take a closer look at it:

• This code listing begins by importing the required packages and credentials and creating both a MongoDB client and a Rev AI API client.
• It attempts to connect to the MongoDB database service using the MongoDB client's connect() method and selects a database and collection for use.
• It starts an Express application on port 3000 and waits for incoming POST requests to the /hook URL route.
• When the application receives a POST request at /hook, it parses the incoming JSON message body and checks the job id and status.
• If the status is transcribed, it uses the API client's getTranscriptObject() method to retrieve the complete transcript as a JSON document.
• It prepares a MongoDB document to hold the transcript. This document also includes other parameters of the job, such as the date and time, language and status.
• It then uses the insertOne() method to insert the prepared document into the collection, and prints the unique MongoDB document identifier to the console.
• Errors, if any, in the MongoDB connection and data insertion process are sent to the console.

Step 3: Test the webhook

To see the webhook in action, first ensure that you have replaced the placeholders as described in the previous step and then start the application using the command below.

node index.js

Next, submit an audio file for transcription to Rev AI and include the callback_url parameter in your request. This parameter specifies the webhook URL that the Rev AI API should invoke on job completion.

Here is an example of submitting an audio file with a webhook using curl.

curl -X POST "https://api.rev.ai/speechtotext/v1/jobs" \
     -H "Authorization: Bearer <REVAI_ACCESS_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"media_url":"<URL>","callback_url":"http://<WEBHOOK-HOST>/hook"}'

If you prefer to submit the audio file using the Rev AI Node SDK, use this script instead:

const { RevAiApiClient } = require('revai-node-sdk');

const revAiToken = '<REVAI_ACCESS_TOKEN>';
const webhookUrl = 'http://<WEBHOOK-HOST>/hook';
const fileUrl = '<URL>';

// define job options
const jobOptions = {
  callback_url: webhookUrl,
};

// create Rev AI API client
const revAiClient = new RevAiApiClient(revAiToken);

// submit job
job = await revAiClient.submitJobUrl(fileUrl, jobOptions);

In both cases, replace the <REVAI_ACCESS_TOKEN> placeholder with your Rev AI access token and the <URL> placeholder with the direct URL to your audio file. Additionally, replace the <WEBHOOK-HOST> placeholder as follows:

• If you are developing and testing in the public cloud, your Express application will typically be available at a public domain or IP address. In this case, replace the <WEBHOOK-HOST> placeholder with the correct domain name or IP address, including the port number 3000 if required.
• If you are developing and testing locally, your Express application will not be available publicly and you must therefore configure a public forwarding URL using a tool like ngrok. Obtain this URL using the command ngrok http 3000 and replace the <WEBHOOK-HOST> placeholder with the temporary forwarding URL generated by ngrok.

Once the job is processed, the Rev AI API will send a POST request to the webhook URL. The webhook will then request the complete transcript and save it to the MongoDB database as a JSON document, as explained in the previous section.

Here is an example of the output generated by the webhook handler:

Webhook listening
Received status for job id Qctu6UgJvTjZ: transcribed
Received transcript for job id Qctu6UgJvTjZ
Saved transcript with document id: 62504e6fcc32c9387208c875

You can retrieve the transcript for review or further processing by connecting to the MongoDB database and obtaining the document using its unique identifier. An example of doing this with mongosh is shown below:

test> use mydb
mydb> db.transcripts.find( { '_id':ObjectId('62504e6fcc32c9387208c875') } )

[
  {
    _id: ObjectId("62504e6fcc32c9387208c875"),
    job_id: 'Qctu6UgJvTjZ',
    created_on: '2022-04-08T15:01:35.999Z',
    language: 'en',
    status: 'transcribed',
    transcript: {
      monologues: [
        {
          speaker: 0,
          elements: [
            {
              type: 'text',
              value: 'Hi',
              ts: 0.26,
              end_ts: 0.48,
              confidence: 0.99
            },
            { type: 'punct', value: ',' },
            {
              ...
            }
          ]
        }
      ]
    }
  }
]

If a webhook doesn't work as expected, you can test and inspect the webhook response.

Next steps

Learn more about using webhooks for asynchronous processing by visiting the following links:

• Documentation: Asynchronous Speech-To-Text API job submission and webhooks
• Tutorial: Get Started with Rev AI API Webhooks
• Tutorial: Use Webhooks to Trigger Job Email Notifications with Node.js, SendGrid and Express
• Documentation: Using ngrok
• Documentation: MongoDB
• Documentation: MongoDB Node.js Driver