Learn how to convert WebSocket data into minute-based HLOC bars using Node.js. Step-by-step guide for efficient financial data processing.
As WebSockets continues to gain traction in today's tech world, curious developers are eager to explore this exciting technology. Setting up a server-client setup and experimenting with WebSocket data can be pretty thrilling, but the real magic happens when you access WebSockets, which delivers real-time, actionable data.
This tutorial will focus on something super helpful: turning raw WebSocket data into minute-based High, Low, Open, and Close (HLOC) bars. This transformation helps developers and analysts track market trends in one-minute intervals.
The best part? You don't need to be a Node.js pro to follow along! While some essential programming experience (especially with JavaScript) might help, it's unnecessary. Are you not a JavaScript fan? No worries—you can also explore WebSocket implementations in Python or Golang.
Before starting the setup and coding, let's clarify the plan. We'll fetch real-time Forex data via a WebSocket connection and convert it into minute-based HLOC bars using the TraderMade WebSocket API. The documentation provides more details.
Here's our game plan, broken into four simple steps:
- Install NodeJS and set up your environment.
- Sign up for a free trial and grab your API key.
- Build a Node.js WebSocket client to receive data.
- Process WebSocket data into minute-based HLOC bars with Node.js.
Want a quick walkthrough? Check out this YouTube video.
Step 1: Install NodeJS and Set Up Your Environment
For Windows and Mac users: Download and install NodeJS.
For Linux fans: Use apt-get to install NodeJS.
Once NodeJS is installed, open your command prompt (Windows) or terminal (Mac/Linux). Now, let's get the necessary dependencies in place. For our client, we're using the "ws" library. You can install it with this command:
(And then continue with the step-by-step instructions as needed.)
Let's dive in and get started!
npm install ws
Get your free trial and API key
Head over to the tutorial that walks you through starting a free 14-day trial.
Once you have your API key, store it somewhere safe—you'll need it soon!
Setting Up a Node.js WebSocket Client to Receive Data
Find your NodeJS installation directory and create a new file called forexWsClient.js. Open it with your favorite code editor—Atom, VS Code, Notepad (on Windows), or any text editor on Linux.
Ready for some code magic? Let's dive in!
const WebSocket = require ('ws');
const ws = new WebSocket ('wss://marketdata.tradermade.com/feedadv');
First, we'll bring in the WebSocket object and set up a connection to TraderMade using their secure WSS URL. Once the connection is live, we'll send over our API key (the one you grabbed during the free trial signup) and the symbols we want to track. For example, let's focus on EURUSD.
ws.on('open', function open() {
ws.send("{"userKey":"streaming_api_key", "symbol":"EURUSD"}");
});
ws.on('message', function incoming(data) {
if(data != "Connected"){
data = JSON.parse(data)
console.log(data)
}
});
Running Your WebSocket Client
- Save your forexWsClient.js file.
- Open your terminal or command prompt and navigate to the file's location.
- Run this command to fire up your client:
node forexWsClient.js
And That's a Wrap!
Congrats—you've successfully set up a Node.js WebSocket client to receive real-time Forex data! Now, sit back and watch the financial markets come to life with this smooth integration.
Transforming WebSocket Data into Minute-Based HLOC Bars with Node.js
Modules You'll Need
To make this script work, we'll rely on two key Node.js modules:
- Fs: This is the File System module, which lets us read from and write to files.
- Luxon: A handy JavaScript library designed for working with dates and times.
// Required Modules
const fs = require('fs'); // File System module
const { DateTime } = require('luxon'); // Luxon library for date-time formatting
File Paths
- inputFilePath: This refers to the location of the input JSON file that holds the WebSocket data.
- outputFilePath: This is where the processed HLOC data will be saved.
// File Paths
const inputFilePath = 'path/to/your/input/file.json'; // Update with your input file path
const outputFilePath = 'path/to/your/output/file.json'; // Update with your output file path
Reading Input Data
The script begins by asynchronously reading the input file using fs.readFile. It splits the file content into individual lines, each treated as a JSON object. This setup assumes that each line represents a separate WebSocket message in JSON format.
fs.readFile(inputFilePath, 'utf8', (err, data) => {
if (err) {
console.error('Error reading input file:', err);
return;
}
let jsonData;
try {
jsonData = data.split('
').filter(line => line.trim() !== '').map(line => {
try {
return JSON.parse(line);
} catch (error) {
console.error('Error parsing JSON line:', error);
return null;
}
}).filter(Boolean);
} catch (error) {
console.error('Error processing JSON data:', error);
return;
}
if (!jsonData || jsonData.length === 0) {
console.error('No valid JSON data found in the input file.');
return;
}
Grouping Data by Time Intervals
The groupDataByInterval function is key to organizing the data. It groups the WebSocket messages based on a set time interval—1 minute, to be exact. Here's how it works:
- It starts by converting each data point's timestamp (ts) into a numerical value.
- This timestamp is then divided by the interval duration (60,000 milliseconds), and the result is rounded down to determine which group (or interval) each data point belongs to.
- All the data points that fall within the same interval are grouped.
const groupedData = groupDataByInterval(jsonData, 60000);
const hlocData = calculateHLOC(groupedData);
fs.writeFile(outputFilePath, JSON.stringify(hlocData, null, 2), (err) => {
if (err) {
console.error('Error writing output file:', err);
} else {
console.log('HLOC data written to', outputFilePath);
}
});
});
function groupDataByInterval(data, interval) {
const groupedData = {};
data.forEach(item => {
const intervalKey = Math.floor(item.ts / interval) * interval;
if (!groupedData[intervalKey]) {
groupedData[intervalKey] = [];
}
groupedData[intervalKey].push(item);
});
return groupedData;
}
Calculating HLOC Data
Once intervals group the data, the calculateHLOC function takes over to compute the key HLOC values for each interval:
- Open: The mid-value of the first data point in the interval.
- High: The highest mid-value observed within the interval.
- Low:The lowest mid-value seen in the interval.
- Close: The mid-value of the last data point in the interval.
function calculateHLOC(groupedData) {
const hlocData = [];
for (const intervalKey in groupedData) {
const intervalData = groupedData[intervalKey];
const open = roundToFiveDecimalPlaces(intervalData[0].mid);
const close = roundToFiveDecimalPlaces(intervalData[intervalData.length - 1].mid);
const high = roundToFiveDecimalPlaces(Math.max(...intervalData.map(item => item.mid)));
const low = roundToFiveDecimalPlaces(Math.min(...intervalData.map(item => item.mid)));
hlocData.push({
interval: Number(intervalKey),
intervalDateTime: formatDateTime(Number(intervalKey)),
open,
high,
low,
close
});
}
return hlocData;
}
Formatting and Writing Output
Once the HLOC values have been calculated for each interval, the data is organized into a well-structured JSON format, which includes:
- Interval: A timestamp marking the start of each minute interval.
- IntervalDateTime: The date and time corresponding to the interval timestamp, formatted for clarity.
- Open, High, Low, Close: The calculated HLOC values. Finally, the processed HLOC data is saved to an output JSON file using fs.writeFile.
function formatDateTime(timestamp) {
return DateTime.fromMillis(timestamp).toUTC().toFormat('yyyy-MM-dd HH:mm:ss');
}
function roundToFiveDecimalPlaces(value) {
return Math.round(value * 1e5) / 1e5;
}
} else {
console.log('HLOC data written to', outputFilePath);
}
});
Our output is all set!
{
"interval": 1704453720000,
"intervalDateTime": "2024-01-05 11:22:00",
"open": 1.09152,
"high": 1.09152,
"low": 1.09132,
"close": 1.09132
},
{
"interval": 1704453780000,
"intervalDateTime": "2024-01-05 11:23:00",
"open": 1.09133,
"high": 1.09153,
"low": 1.09132,
"close": 1.09152
},
{
"interval": 1704453840000,
"intervalDateTime": "2024-01-05 11:24:00",
"open": 1.09153,
"high": 1.09153,
"low": 1.09141,
"close": 1.09142
}
Wrapping Up
This tutorial provides a simple and effective way to turn WebSocket data into valuable insights. By converting raw data into minute-based HLOC bars, developers and analysts can track market trends, spot patterns, and make well-informed decisions. The script's modular structure, combined with the Luxon library for handling dates and times, ensures both efficiency and accuracy in data processing and analysis.
Top comments (0)