Have you been a Node developer for years, but have never come across Node Buffers? Maybe, you have come across the term a few times, but always shied away from learning what buffers are? You may have never actually needed to use buffers before. Node.js is not a language that forces programmers to deal directly with how programs manage memory. But if you are committed to going the extra mile to become an expert Node developer, you’ll have to grapple with concepts like buffers to understand how Node works under the hood.
At first glance, you’ll feel like Node Buffers are such a complicated topic to understand. But in reality, that’s not the case. The problem starts when all the online tutorials you look at start with creating Node Buffers and manipulating them without first explaining what actually buffers are. To not make the same mistake here, I’ll have to explain what buffers are first. But before understanding what buffers are, we have to tackle a few other concepts that come up when trying to understand buffers.
To understand buffers properly, we should understand about binary data, character encoding, and streams. If you don’t see how these things matter to buffers yet, don’t worry, they all come together, in the end, to explain what buffers are.
What is Binary Data?
If you already know what binary data are, you can skip to the next topic directly. Others should just hold on to see what binary data are.
The binary number system is another number system like the decimal number system we use. Unlike the decimal number system, which uses digits from 0-9 to represent numbers, the binary number system uses only the digits 0 and 1 to represent numbers. Here are some examples of binary numbers.
0, 1, 10, 101, 1011011, 1000101
Each digit in a binary number is considered as a Bit in Computer Science. A combination of 8 bits is called a Byte. But what does Computer Science has to do with binary numbers? Computers use binary numbers to store and represent data. So, every type of data stored in computers is ultimately stored as a set of binary numbers. We call these data Binary Data.
To store every type of data as binary data, computers should know how to convert them to binary data. Computers have different mechanisms to achieve this. Let’s see what they are.
How Do Computers Convert Numbers to Binary Data?
Converting numbers to binary data is just math. You can refer to the math involved with binary number operations and number system conversions to know more about this. However, using this math, decimal number 9 can be represented as 101 in binary, and so are other integers. Computers are equipped to make this conversion on their own.
How Do Computers Convert Characters to Binary Data?
The simple explanation to this question is “each character has a unique binary number associated with it”. This unique number is called the character’s code point or character code. You can use charCodeAt function in Javascript to find each character’s character code.
'a'.charCodeAt() //outputs 97
'A'.charCodeAt() //outputs 65
There are two main standards that are used to assign character codes to each character: ASCII and Unicode. Character codes given to characters are the same regardless of the programming language. ASCII uses up to 7 bits to represent a character and Unicode uses up to 16 bits. This difference gives Unicode a wider range than ASCII to represent more characters and makes Unicode the most popular standard from the two.
Is finding the character point of each character the only thing computers have to do to convert characters to binary data? The answer is: No. There’s one more step to follow to convert characters to binary data. That is character encoding.
What is Character Encoding?
I mentioned before ASCII can use up to 7 bits and Unicode can use up to 16 bits to represent a character. But the computers don’t have to always use all of Unicode’s 16 bits to represent a character. For example, the character ‘A’ can be represented using a minimum of 7 bits. If the computer uses 16 bits to store ‘A’ by filling the binary number with leading 0s, it becomes a waste of system resources.
This is where character encoding steps in. Character encoding standards decide how many bits computers should use to represent a character. UTF-8, UTF-16, and UTF-32 are some examples of character encoding standards.
UTF-8 uses blocks of 8 bits—bytes—to represent characters. It can encode all Unicode characters using 1-4 bytes. Now, if the computer encodes ‘A’ using the UTF-8 standard, the stored binary value is 01000001 with an extra leading 0.
This completes the process of converting characters to binary data. Converting strings to binary data is nothing other than converting each character to binary data. Computers use more standards and methods to convert images, audio, and video data to binary data.
Now comes the concept of Streams. Let’s see what they are.
What are Streams?
Streams are collections of data that are being moved from one place to another. In our case, we are talking about binary data streams, which are collections of binary data being moved from one place to another.
A stream consists of a large amount of data. But the computers don’t have to wait for all the data in the stream to be available to start processing. When sending out the streams to a certain destination, data in the stream is not sent at once because of its large size. Instead, the stream is divided into smaller data chunks. The destination collects receiving chunks and starts processing them once enough chunks are available.
The destination receiving a stream intends to process data in some way—read, manipulate, or write to data. But the capacity of the data processor at the destination has a limit of maximum and minimum amount of data it can process at once. So, what happens when the destination receives data chunks that don't fit into this limit? The destination can't discard them. The destination can, however, use a mechanism to store received chunks until they are accepted by the processor. This is where buffers step in. But first, we should know what exactly buffers are to understand how they help with storing data chunks.
What are Buffers and What do They Do?
A buffer is a small storage space in the memory of a computer, typically in the RAM. Until the destination processor is ready to accept the received data chunks from a stream, buffers act as a waiting area for them.
If the destination receives data from the stream faster than the speed it can process data, these surplus data "waits" in a buffer until the processor frees up to accept more data. If the destination receives data from the stream slower than the speed it can process data—in other words, if the amount of chunks available at the moment is below the minimum amount of data the processor can accept—these data chunks "wait" in a buffer until a sufficient amount of data is available.
So that's what buffers are: a waiting area for the streaming data to wait until the data processor is ready to accept them. Wherever streaming is present, you can expect to see buffers working under the hood to store data chunks that are yet to be processed.
You may have heard of the concept called buffering. When you are watching a YouTube video, sometimes the video keeps loading without resuming the video for some time. This is because your browser is waiting for more data chunks of the video stream to arrive. Until the browser receives enough data chunks, they are stored in these buffers and wait for the processor to accept them. Hence, the name “buffering”. This is exactly what happens with binary streams in Node.js.
The same thing happens when we try to read a large file in a Node program. The buffer used here stores the data chunks sent through the file stream until enough data is available before passing it to the program. This process is, again, called buffering.
But How Does Node.js Use Buffers?
Now, you understand the underlying concept of buffers and why they are needed. But you may be still wondering why Node needs buffers.
Well, the answer is simple. When you send an HTTP request to a web server, the request is sent as a TCP stream through the network, which is a binary data stream. So, all the Node servers you build have to deal with streams and buffers.
When you read a file using the fs.readFile() method, it returns a buffer object through the callback or the promise.
In short, some of the most important modules in Node.js constantly deal with buffers and buffer manipulation. You may have even worked with buffers already, albeit unknowingly. What more reasons do you need to dive deep into learning Node Buffers as a Node developer?
Buffer Creation and Manipulation in Node.js
Node.js offers a Buffer class that lets you easily create buffers and manipulate them. Let’s see what we can do with it.
//create a buffer
let buffer1 = Buffer.alloc(100)
This creates a buffer of size 100, which means the buffer stores 100 bytes of zeros.
You can also create a buffer from strings and integer arrays.
let buffer2 = Buffer.from("I'm learning Node Buffer")
//This encodes the given string and stores it as binary data
let buffer3 = Buffer.from([1, 2, 90, 55])
//stores each integer in binary
You can access each byte in the buffer using the index.
buffer2[0] //returns 73 for ‘I’
buffer2.toString() //returns ‘I'm learning Node Buffer’
Now let’s see how to write to a buffer.
buffer2.write("Hi")
buffer2.toString() //returns ‘Hi’
The write method overwrites the existing content in a buffer and changes it to the value you provide.
//change the stored value at a given index
buffer2[0] = 80
buffer2.toString() //returns ‘Pi’
//get the buffer length
buffer1.length //returns 100
buffer3.length //returns 4
You can check Node.js documentation to see what else you can do with buffers.
Conclusion
As you saw in this article, buffers are fundamental to how Node.js works under the hood. Understanding these concepts is crucial to becoming a better Node developer. This knowledge helps you to write optimized Node programs and realizing the language’s limitations and how to work around them. So, next time you come across an intimidating term related to Node.js, don’t hesitate, meet it head-on as we did with Buffers.
If you like the story, please don't forget to subscribe to our newsletter so we can stay connected: https://livecodestream.dev/subscribe
Top comments (1)
For anyone interested in buffers vs streams performance when it comes to parsing csv files: github.com/pavelloz/streams-vs-buf... - couple benchmarks