DEV Community

Cover image for Creating Minute-Based HLOC Bars from WebSockets Data with NodeJS
Shridhar G Vatharkar
Shridhar G Vatharkar

Posted on

Creating Minute-Based HLOC Bars from WebSockets Data with NodeJS

Learn how to convert WebSocket data into minute-based HLOC bars using Node.js. Step-by-step guide for efficient financial data processing.

As WebSockets continues to gain traction in today's tech world, curious developers are eager to explore this exciting technology. Setting up a server-client setup and experimenting with WebSocket data can be pretty thrilling, but the real magic happens when you access WebSockets, which delivers real-time, actionable data.

This tutorial will focus on something super helpful: turning raw WebSocket data into minute-based High, Low, Open, and Close (HLOC) bars. This transformation helps developers and analysts track market trends in one-minute intervals.

The best part? You don't need to be a Node.js pro to follow along! While some essential programming experience (especially with JavaScript) might help, it's unnecessary. Are you not a JavaScript fan? No worries—you can also explore WebSocket implementations in Python or Golang.

Before starting the setup and coding, let's clarify the plan. We'll fetch real-time Forex data via a WebSocket connection and convert it into minute-based HLOC bars using the TraderMade WebSocket API. The documentation provides more details.

Here's our game plan, broken into four simple steps:

  1. Install NodeJS and set up your environment.
  2. Sign up for a free trial and grab your API key.
  3. Build a Node.js WebSocket client to receive data.
  4. Process WebSocket data into minute-based HLOC bars with Node.js.

Want a quick walkthrough? Check out this YouTube video.

Step 1: Install NodeJS and Set Up Your Environment

For Windows and Mac users: Download and install NodeJS.
For Linux fans: Use apt-get to install NodeJS.

Once NodeJS is installed, open your command prompt (Windows) or terminal (Mac/Linux). Now, let's get the necessary dependencies in place. For our client, we're using the "ws" library. You can install it with this command:
(And then continue with the step-by-step instructions as needed.)
Let's dive in and get started!

npm install ws 
Enter fullscreen mode Exit fullscreen mode

Get your free trial and API key

Head over to the tutorial that walks you through starting a free 14-day trial.
Once you have your API key, store it somewhere safe—you'll need it soon!

Setting Up a Node.js WebSocket Client to Receive Data

Find your NodeJS installation directory and create a new file called forexWsClient.js. Open it with your favorite code editor—Atom, VS Code, Notepad (on Windows), or any text editor on Linux.
Ready for some code magic? Let's dive in!

const WebSocket = require ('ws');
const ws = new WebSocket ('wss://marketdata.tradermade.com/feedadv');
Enter fullscreen mode Exit fullscreen mode

First, we'll bring in the WebSocket object and set up a connection to TraderMade using their secure WSS URL. Once the connection is live, we'll send over our API key (the one you grabbed during the free trial signup) and the symbols we want to track. For example, let's focus on EURUSD.

ws.on('open', function open() {
     ws.send("{"userKey":"streaming_api_key", "symbol":"EURUSD"}");
  });

  ws.on('message', function incoming(data) {
    if(data != "Connected"){
            data = JSON.parse(data)
            console.log(data)
    }
  });
Enter fullscreen mode Exit fullscreen mode

Running Your WebSocket Client

  1. Save your forexWsClient.js file.
  2. Open your terminal or command prompt and navigate to the file's location.
  3. Run this command to fire up your client:
node forexWsClient.js 
Enter fullscreen mode Exit fullscreen mode

And That's a Wrap!

Congrats—you've successfully set up a Node.js WebSocket client to receive real-time Forex data! Now, sit back and watch the financial markets come to life with this smooth integration.

Transforming WebSocket Data into Minute-Based HLOC Bars with Node.js

Modules You'll Need

To make this script work, we'll rely on two key Node.js modules:

  1. Fs: This is the File System module, which lets us read from and write to files.
  2. Luxon: A handy JavaScript library designed for working with dates and times.
// Required Modules
const fs = require('fs'); // File System module
const { DateTime } = require('luxon'); // Luxon library for date-time formatting
Enter fullscreen mode Exit fullscreen mode

File Paths

  1. inputFilePath: This refers to the location of the input JSON file that holds the WebSocket data.
  2. outputFilePath: This is where the processed HLOC data will be saved.
// File Paths
const inputFilePath = 'path/to/your/input/file.json'; // Update with your input file path
const outputFilePath = 'path/to/your/output/file.json'; // Update with your output file path
Enter fullscreen mode Exit fullscreen mode

Reading Input Data

The script begins by asynchronously reading the input file using fs.readFile. It splits the file content into individual lines, each treated as a JSON object. This setup assumes that each line represents a separate WebSocket message in JSON format.

fs.readFile(inputFilePath, 'utf8', (err, data) => {
  if (err) {
    console.error('Error reading input file:', err);
    return;
  }

  let jsonData;
  try {
    jsonData = data.split('
').filter(line => line.trim() !== '').map(line => {
      try {
        return JSON.parse(line);
      } catch (error) {
        console.error('Error parsing JSON line:', error);
        return null;
      }
    }).filter(Boolean);
  } catch (error) {
    console.error('Error processing JSON data:', error);
    return;
  }

  if (!jsonData || jsonData.length === 0) {
    console.error('No valid JSON data found in the input file.');
    return;
  }
Enter fullscreen mode Exit fullscreen mode

Grouping Data by Time Intervals

The groupDataByInterval function is key to organizing the data. It groups the WebSocket messages based on a set time interval—1 minute, to be exact. Here's how it works:

  1. It starts by converting each data point's timestamp (ts) into a numerical value.
  2. This timestamp is then divided by the interval duration (60,000 milliseconds), and the result is rounded down to determine which group (or interval) each data point belongs to.
  3. All the data points that fall within the same interval are grouped.
const groupedData = groupDataByInterval(jsonData, 60000);
  const hlocData = calculateHLOC(groupedData);

  fs.writeFile(outputFilePath, JSON.stringify(hlocData, null, 2), (err) => {
    if (err) {
      console.error('Error writing output file:', err);

    } else {
      console.log('HLOC data written to', outputFilePath);
    }
  });
});

function groupDataByInterval(data, interval) {
  const groupedData = {};
  data.forEach(item => {
    const intervalKey = Math.floor(item.ts / interval) * interval;
    if (!groupedData[intervalKey]) {
      groupedData[intervalKey] = [];

    }
    groupedData[intervalKey].push(item);
  });
  return groupedData;
}
Enter fullscreen mode Exit fullscreen mode

Calculating HLOC Data

Once intervals group the data, the calculateHLOC function takes over to compute the key HLOC values for each interval:

  1. Open: The mid-value of the first data point in the interval.
  2. High: The highest mid-value observed within the interval.
  3. Low:The lowest mid-value seen in the interval.
  4. Close: The mid-value of the last data point in the interval.
function calculateHLOC(groupedData) {
  const hlocData = [];
  for (const intervalKey in groupedData) {
    const intervalData = groupedData[intervalKey];
    const open = roundToFiveDecimalPlaces(intervalData[0].mid);
    const close = roundToFiveDecimalPlaces(intervalData[intervalData.length - 1].mid);
    const high = roundToFiveDecimalPlaces(Math.max(...intervalData.map(item => item.mid)));
    const low = roundToFiveDecimalPlaces(Math.min(...intervalData.map(item => item.mid)));

    hlocData.push({
      interval: Number(intervalKey),
      intervalDateTime: formatDateTime(Number(intervalKey)),
      open,
      high,
      low,
      close
    });
  }
  return hlocData;
}
Enter fullscreen mode Exit fullscreen mode

Formatting and Writing Output

Once the HLOC values have been calculated for each interval, the data is organized into a well-structured JSON format, which includes:

  1. Interval: A timestamp marking the start of each minute interval.
  2. IntervalDateTime: The date and time corresponding to the interval timestamp, formatted for clarity.
  3. Open, High, Low, Close: The calculated HLOC values. Finally, the processed HLOC data is saved to an output JSON file using fs.writeFile.
function formatDateTime(timestamp) {
  return DateTime.fromMillis(timestamp).toUTC().toFormat('yyyy-MM-dd HH:mm:ss');
}

function roundToFiveDecimalPlaces(value) {
  return Math.round(value * 1e5) / 1e5;
}
  } else {
    console.log('HLOC data written to', outputFilePath);
  }
});
Enter fullscreen mode Exit fullscreen mode

Our output is all set!

{
    "interval": 1704453720000,
    "intervalDateTime": "2024-01-05 11:22:00",
    "open": 1.09152,
    "high": 1.09152,
    "low": 1.09132,
    "close": 1.09132
  },

  {
    "interval": 1704453780000,
    "intervalDateTime": "2024-01-05 11:23:00",
    "open": 1.09133,
    "high": 1.09153,
    "low": 1.09132,
    "close": 1.09152
  },

  {
    "interval": 1704453840000,
    "intervalDateTime": "2024-01-05 11:24:00",
    "open": 1.09153,
    "high": 1.09153,
    "low": 1.09141,
    "close": 1.09142
  }
Enter fullscreen mode Exit fullscreen mode

Wrapping Up

This tutorial provides a simple and effective way to turn WebSocket data into valuable insights. By converting raw data into minute-based HLOC bars, developers and analysts can track market trends, spot patterns, and make well-informed decisions. The script's modular structure, combined with the Luxon library for handling dates and times, ensures both efficiency and accuracy in data processing and analysis.

Image of AssemblyAI tool

Transforming Interviews into Publishable Stories with AssemblyAI

Insightview is a modern web application that streamlines the interview workflow for journalists. By leveraging AssemblyAI's LeMUR and Universal-2 technology, it transforms raw interview recordings into structured, actionable content, dramatically reducing the time from recording to publication.

Key Features:
🎥 Audio/video file upload with real-time preview
🗣️ Advanced transcription with speaker identification
⭐ Automatic highlight extraction of key moments
✍️ AI-powered article draft generation
📤 Export interview's subtitles in VTT format

Read full post

Top comments (0)

AWS Security LIVE!

Tune in for AWS Security LIVE!

Join AWS Security LIVE! for expert insights and actionable tips to protect your organization and keep security teams prepared.

Learn More

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay