Have you ever asked yourself what bitwise operators are for? Why would such a high-level language such as JavaScript ever need such a low-level operator? For one, it actually has its use cases in JavaScript. Most are just not as obvious as others. Actually, most are not even obvious at all unless you really try and squint your eyes at the computer screen. Trust me, I have tried. I'm not even joking. Throughout my relatively short experience with JavaScript (3 years as of writing this article), it has been so rare to find instances of bitwise operators appear in average situations. I might not be looking deep enough, but it does seem pretty clear to me why this is so. By the end of this article, you will see why this is the case.
Bitwise Operators
NOTE: I am not requiring an extensive knowledge on the subject, but I will assume that you are already at least somehow familiar with binary number systems and bitwise operators. If not, I highly recommend that you read up a bit (See what I did there?) before continuing with the rest of this article.
Bitwise operators allow us to manipulate the individual bits that make up a number in binary. For a quick review, here is a "table" of what the common bitwise operators do.
// I will use the binary notation prefix ("0b") a lot in this article.
const num1 = 0b1010; // 10
const num2 = 0b1111; // 15
// NOT num1
~num1; // 0b0101 (complement) === -11
// num1 AND num2
num1 & num2; // 0b1010 === 10
// num1 OR num2
num1 | num2; // 0b1111 === 15
// num1 XOR num2
num1 ^ num2; // 0b0101 === 5
// Bit-shift to the left by 1
num1 << 1; // 0b10100 === 20
// Bit-shift to the right by 1
num >> 1; // 0b0101 === 5
I mean this is great and all for the sake of learning something new everyday, but when would you ever use this knowledge? Is there a practical application for bitwise operators? Short answer, no. Although it can be useful in code minification, memory optimization, and some other use cases, by using bitwise operators, you are opting into less readable code. It is just more cryptic to read because you have to set your "Decimal Mode" brain into "Binary Mode". Nonetheless, that doesn't stop us, right? We're all here to learn. So without further ado, I present bitmasks.
Over-engineering a simple problem
Honestly, I don't have a simple definition for what a "bitmask" is. It's quite a strange monster if you'd ask me. To me, a bitmask can be thought of as a query. Using a bitmask means to query the bits found in some binary number. If you're confused by that definition, I don't blame you. I have to admit that it isn't the best definition. If you can think of a better one, please do leave a comment below. I'd gladly update this article to specifically include your definition.
Anyway, a definition is worthless without its complementary example. Let's say we have an object that stores booleans corresponding to the configurations found in an app.
// Mock app settings
const config = {
isOnline: true,
isFullscreen: false,
hasAudio: true,
hasPremiumAccount: false,
canSendTelemetry: true
};
Our job is done at this point. We can store that as is in a JSON file. That is the straightforward implementation. However, we can use bitmasks to "over-engineer" this problem. In JavaScript, number types can be explicitly converted (or coerced) into booleans by passing it into the Boolean function. Take note that in this case, Boolean is not used as a constructor. It is simply a means to convert the number type (or any type actually) into its equivalent boolean "truthiness". For example:
Boolean(-2); // true
Boolean(-1); // true
Boolean(0); // false
Boolean(1); // true
Boolean(2); // true
Boolean(Math.PI); // true
Boolean(Number.MAX_SAFE_INTEGER); // true
Since 0 is not exactly a "truthy" value per se, it evaluates to false. That relationship gives us an idea on how to convert a bunch of booleans into a single number. Instead of storing the app settings as an object, we can store it as a single number. Yup, you heard, or rather read, that right. First, we think of the booleans as 1s and 0s, where 1 is true and 0 is false. These 1s and 0s correspond to each property in the config object from left to right.
// For reference only
const config = {
isOnline: true,
isFullscreen: false,
hasAudio: true,
hasPremiumAccount: false,
canSendTelemetry: true
};
// isOnline: 1
// isFullScreen: 0
// hasAudio: 1
// hasPremiumAccount: 0
// canSendTelemetry: 1
// Thus, we have the binary number 0b10101.
let configNumber = 0b10101; // 21
Bitmasks
NOTE: Here comes the weird part of the article. This is where I pull out the black magic. I hope you have stretched those brain muscles enough because you'd be doing a strenuous workout with it from this point on. Feel free to read some parts over and over again. This is a pretty difficult topic to say the least.
Now that we have reduced an entire object into a single number, we can use bitwise operators on it. But why, do you ask? Well, this is the essence of bitmasking.
A bitmask is a way to "select" the bits you're interested in. When selecting a single particular bit, it is always a power of 2 because any power of 2 corresponds to that particular bit that is "turned on". Since bit-shifting to the left is essentially multiplying by 2 (analogous to raising 2 by a power), you can think of bit-shifting to the left as a way to "select" the bit you are interested in.
// Selecting the 1st bit from the right
// 2 ** 0
// 1 << 0
0b00001 === 1;
// Selecting the 2nd bit from the right
// 2 ** 1
// 1 << 1
0b00010 === 2;
// Selecting the 3rd bit from the right
// 2 ** 2
// 1 << 2
0b00100 === 4;
// Selecting the 4th bit from the right
// 2 ** 3
// 1 << 3
0b01000 === 8;
// Selecting the 5th bit from the right
// 2 ** 4
// 1 << 4
0b10000 === 16;
If we want to select more than one bit, we can do that, too.
// Selecting the 1st and 5th bit from the right
0b10001 === 17;
// Selecting the 3rd and 4th bit from the right
0b01100 === 12;
// Selecting the 2nd, 4th, and 5th bit from the right
0b11010 === 26;
// Selecting the 1st, 2nd, and 4th bit from the right
0b01011 === 11;
// Selecting ALL the bits
0b11111 === 31;
Getting Values
Bitmasking allows us to extract the value of a single bit in the configNumber number. How do we do this? Let's say we wanted to get the value of hasAudio. We know that the hasAudio property is located at the third bit from the right of the configNumber.
let configNumber = 0b10101; // 21
// Shifting 0b1 to the left 2 times gives the 3rd bit from the right
const bitMask = 0b1 << 2; // 4
// Since we know that the 3rd bit from the right corresponds to the hasAudio property...
const query = configNumber & bitMask; // 4
// ...we can test its "truthiness" by using the AND operator.
const truthiness = Boolean(query); // true
// The truthiness IS the value we want to extract.
truthiness === config.hasAudio; // true
At this point, you may be asking...
"So what if it returns
4?
"So what if it's coerced totrue?"
"So what if it's 'truthy'?"
If you are asking that, then you just answered your own question. 4 has been coerced to true. That is the exact value of the hasAudio property in the original config object. We have successfully extracted the value of the hasAudio property through bitmasking.
Well, what happens if we try to query a "falsy" property such as isFullscreen? Would bitmasking reflect the same value in the original config object? As a matter of fact, it does. We know that the isFullScreen property is located at the fourth bit from the right in the configNumber.
let configNumber = 0b10101; // 21
// Shifting 0b1 to the left 3 times gives the 4th bit from the right
const bitMask = 0b1 << 3; // 8
// Since we know that the 4th bit from the right corresponds to the isFullscreen property...
const query = configNumber & bitMask; // 0
// ...we can test its "truthiness" by using the AND operator.
const truthiness = Boolean(query); // false
// The truthiness IS the value we want to extract.
truthiness === config.isFullscreen; // true
We can go even crazier by selecting multiple bits in our bitMask, but I'll leave that as an exercise for you to ponder on.
You may be noticing a pattern here. The result of the AND bitwise operator determines the truthiness of a query. The truthiness is essentially the actual value of the property we are trying to get in the first place. Yes, I know; it's black magic. I had the same reaction. It was too clever for me to fully comprehend at the time.
So now that we know how to extract a boolean out of a specific bit, how do we manipulate a bit?
Toggling Values
The same logic follows when we want to toggle bits. We still use bitmasks to select the bits we're interested in, but we use the XOR bitwise operator (^) instead of the AND bitwise operator (&) for our query.
Let's say we wanted to toggle the canSendTelemetry property. We know that it is located in the first bit from the right.
let configNumber = 0b10101; // 21
// Shifting 0b1 to the left 0 times gives the 1st bit from the right,
// which corresponds to the canSendTelemetry property
const bitMask = 0b1 << 0; // 1
// Toggling the 1st bit from the right
const query = configNumber ^ bitMask; // 20
// Setting the query as the new configNumber
configNumber = query;
Now if we tried to extract the canSendTelemetry property from the new configNumber, we will find that it is no longer set to true. We have successfully toggled the bit from true to false (or rather from 1 to 0).
All Together Now
This is definitely tedious to do over and over again. Since we all want to save a few keystrokes, let's create some utility functions that does all this for us. First, we will write two utility functions that extracts the "truthiness" of a bit: one extracts the "truthiness" if it's given a bitmask, while the other extracts the "truthiness" if it's given the zero-indexed position (from the right) of the bit being extracted.
/**
* Extracts the "truthiness" of a bit given a mask
* @param {number} binaryNum - The number to query from
* @param {number} mask - This is the bitmask that selects the bit
* @returns {boolean} - "Truthiness" of the bit we're interested in
*/
function getBits(binaryNum, mask) {
const query = binaryNum & mask;
return Boolean(query);
}
/**
* Extracts the "truthiness" of a bit given a position
* @param {number} binaryNum - The number to query from
* @param {number} position - This is the zero-indexed position of the bit from the right
* @returns {boolean} - "Truthiness" of the bit we're interested in
*/
function getBitsFrom(binaryNum, position) {
// Bit-shifts according to zero-indexed position
const mask = 1 << position;
const query = binaryNum & mask;
return Boolean(query);
}
Finally, let's write a utility function for toggling one or multiple bits. The function returns the new binaryNum that comes as a result of toggling the selected bits.
/**
* Returns the new number as a result of toggling the selected bits
* @param {number} binaryNum - The number to query from
* @param {number} mask - This is the bitmask that selects the bits to be toggled
* @returns {number} - New number as a result of toggling the selected bits
*/
function toggleBits(binaryNum, mask) {
return binaryNum ^ mask;
}
We can now use these utility functions with the previous examples.
const config = {
isOnline: true,
isFullscreen: false,
hasAudio: true,
hasPremiumAccount: false,
canSendTelemetry: true
};
let configNumber = 0b10101;
// Extracts hasPremiumAccount
getBits(configNumber, 1 << 1); // false
getBitsFrom(configNumber, 1); // false
// Toggles isOnline and isFullscreen
toggleBits(configNumber, (1 << 4) + (1 << 3)); // 0b01101 === 13
Conclusion: Why should I even bother with bitmasking?
That's a very good question. Frankly, I wouldn't recommend using this regularly, if at all. As clever as it is, it is just too esoteric for common use. It is impractical and unreadable most of the time. Constant documentation and awareness are required to make sure that the correct bits are being selected and manipulated. Overall, there aren't many applications for this, especially in a high-level language like JavaScript. However, that shouldn't discourage you from using it if the need arises. It is our job as programmers to determine which algorithms are the best for both the user (for usability) and the developer (for maintainability).
If that is so, then what is the point of me writing an entire article about this?
- This is for the hardcore computer scientists out there. They are the ones who will benefit the most from this article, especially those who are just beginning to dive deeper into the weird world of computer science. To put it more generally, one does not need to be a computer scientist to benefit from this article. Whoever is interested in such topics will see the value in all of this bitmasking chaos.
- For those who are not into computer science, you now have more tools under your belt. You can use bitmasks in the future if the time calls for it. I hope this article encourages you to think creatively. Over-engineering is a curse we all suffer eventually. It isn't entirely a bad thing, though. Over-engineering is just a negative connotation for thinking (too) creatively. Our brains tend to explore ideas even if it's impractical. Of course, we have to avoid it for productivity, but a little exploration now and then is always healthy. Get that brain working, and that brain would work for you.
- As for me, I wrote this article to test myself. I wanted to know how much I've learned so far. Besides that, I find pleasure in teaching others. One can learn so much by teaching others. This is the primary reason why I write articles for this site. It just has its rewards, you know? If you aren't already, go ahead and try to teach someone something new. It might surprise you to see how much it can help you as well.
Bitmask responsibly.
Oldest comments (29)
Nice discussion!
As I see it, bitmasks are a very important tool for use cases where either performance or memory constraints are important. Performance, because bitwise operations can sometimes achieve the same as more complex arithmetic, but at a fraction of the cost. And memory constraints because storing a single conceptual boolean value in a single bit is about as efficient as you can get, memory-wise.
Although it surely isn't the only one, C++ is truly the language for highly optimized computing. I like the fact that one can have many options for a number data type. For example, we can choose to be as memory-efficient as we want by using a
shortover a normalinttype; unlike in JavaScript where all numbers are 64-bit IEEE 754-compliant. This is good for beginners because they don't have to worry about the intricacies of the language, but when it comes to servers and other large-scale operations, it might be a huge help to the workload if numbers weren't 64 bits long.Great article! If you allow, I will add my example of bitmasking and a very cool one imho
When it comes to data compression algorithms many times you need to write or read just one bit. In nowadays computer systems that kind of granularity is impossible because the minimum amount of data is a byte. So if it weren't for bitmasking and binary operators in language compression algorithms would be very useless.
Sometimes, I feel like bit manipulation is an underappreciated art. Your example proves it. We use compression algorithms everyday, especially in the Web where all network requests must be minimized as much as possible, yet there is little attention and appreciation given to it.
Permissions are the only real way most people will see bitmasks these days unless you're doing something particularly low-level.
Like file permissions:
reads literally as
Not realising this is exactly why everyone abuses
chmod -R 777so wildly.It's also good1 for wrapping up complex permissions in a program:
vs.
Meh, "good". ↩
I was supposed to include an example on complex conditions in the article, but then I thought that it was probably getting too long already. Thanks for bringing this up!
Also, may I ask what the
xs stand for inrwxr-xr-x?They're the execute bits - if it's a file, you can run it as a command, if it's a directory, you can
cdinto it.While JS is a high-level language and it initially certainly wasn't meant to be used for much more than some simple script blocks, the engineering and runtimes that we use this language in have lowered things considerably.
Consider that JS runtimes have multiple tiers of JS code processing, the latter ones compiling it into direct assembly and throwing optimiser passes over it. JS engines detect and optimise shapes of objects to eliminate all the handling around string lookups for the keys of your objects.
So, while you can still use JS for everyday high-level tasks, you can now successfully use it for very high data throughput operations as well, both in the browser and in server contexts. And memory usage and data layout impacts throughput significantly. Consider this made up but actually kind of realistic metadata object:
To create this object, a JS engine has to allocate a standard JS object, allocate 3 strings for the keys and store the 3 values. Now if there are many objects being handled like this then JS engines will get smarter about them, but they still have to create new dynamically allocated objects for each one.
BTW, with "many" I'm thinking of the tens or hundreds of thousands or larger magnitudes.
Given that I know as the developer of this app that the
sortKeyfield is never larger than, say, 1 million and thegroupis at most 50 and thefirstis a boolean, which is 1 or 0.I can use this info to put all this info into a single 32-bit integer.
sortKeycan fit inside 20 bits (2^20 > 1M),groupcan fit into 6 bits (2^6 = 64) andfirstcan fit into a single bit, total 27 bits. I can store these as such:Where F = first, G = group and S = sortKey
The above object can then be represented as:
To create this value, the JS engine has to … allocate nothing (in optimised passes), it's just an integer that fits comfortably in a register inside the CPU, we even have 5 bits to spare! ;) I can now use the bit masking and insertion techniques you mentioned to read/write data in these values (did I mention that bit ops are single instructions on the CPU?)
Now, if this was indeed only for a single variable, we wouldn't have gained much, but we're talking about large amounts of data here. Say, I will process 1 million of these blocks at a time. To do that I can:
And done, all memory accounted for, and with simple indexing I can access each element.
Doing the same for the JS object variant would mean:
The first 2 can be moved to init time with object reuse, but the 3rd one is maybe even more important. Modern CPUs really don't like cache misses and fragmented memory reads, it can have a significant impact on the runtime of your code.
Either way, the end result is that by representing the data more efficiently the code will run (much) faster and take less memory, which means being potentially better than competition, having lower server costs (fewer servers needed with less memory in each), not having to wait for results, being able to handle bigger workloads, etc.
So again, for "normal" usage indeed, don't bother, but for high-volume data processing it really pays to look at data layout optimisation, and bit fields are one of the tools you can use for that!
I really love the example you gave. It's very interesting. I applaud the creativity and cleverness displayed in those 27 bits. I just learned something new and cool today. Thanks!
Stop confusing people with weird words like "CPU" and "register", what's wrong with you? Did you even read the article and especially the context in which it was written?
I really hope you're being satire.
This optimization is fantastic. And horrific at the same time. It can provide performance improvement seemingly out of thin air. And it can be a nightmare to support for anyone other than the author. What if the number of groups changes dramatically? Like, grows to 100, or 200? Or is removed entirely? What if first is replaced by 6 booleans? What if... the metadata changes completely?
My point: bitwise operations are a great tool, but it has extremely specific usage cases. I'd advice average software developers to be extremely skeptical about it (don't use it unless you know what you want to achieve and there's no better way, or it requires significant sacrifices elsewhere).
I think you wrote an explanation of bitfields that is both accessible and in depth, well done.
I think that the best use case for bitfields/bitmasks are found where the various "flags" you toggle on and off are not independent of each other, but have some meaningful relationship, e.g. the values in the field are "additive" in a sense.
For example, log levels are considered additive by some applications: if you log at "info" level, then you also log "error" and "warn" level messages, this relationship between the different possible values is well captured by bitfields by representing the "lowest" value as the smallest value in the field:
Then whatever is logging can decide whether to log a message or not based on whether the configured log level "has" the specified flag by using bitwise and (&) as you mentioned, or making a numeric comparison:
Another example is configuring media "directions" (you can run into this with webrtc):
In these cases, the values have some meaningful relationship with each other and using bitfields feels quite natural to me to represent these relationships.
I never knew that error logs had levels. No wonder they look like hexadecimal gibberish to naive users like me.
WebRTC has a pretty cool API for using bits rather than strings, such as
'forwards','backwards', or'both', to describe media direction. I definitely want to see more of that in modern APIs.This makes me wonder what the rubrics are for determining the "best and optimal move" in Chess.
A while ago I found out that making chess engines is holy hell incredibly intrincate
I found this engine a guy made in Scala, exploiting it's concurrency and DSL capabilities: github.com/marianogappa/ostinato
It was one of those moments when I say, "damn, I do not know crap about programming and I suck"
But back to the post, great job! I'm myself delving a bit more into bit ops, and your post is super well explained.
Definitely not an expert but my take away from my limited experience with bit masks are like this:
For example you can have presets like
admin = 24; user = 17; poweruser = 31;. And just setconfig = adminand you are good to go rather than assigning many variables.Another example, bit masks are used in Unity 3D engine to set the layers a Camera renders in runtime. Unity has two parts, an editor where you set things using GUI, and your code. Layer count and names are customized in the Unity editor. And in your camera code, instead of using a whole list of exposed layer objects, you can provide a bit mask to modify which layers are rendered with that particular camera.
Yeah, I also have very limited experience with bitmasks myself. Seems like bitmasking isn't as impractical as I initially thought. Thanks for sharing!
I feel like this article and others here on DEV regarding bitwise operations are spreading fear about something quite simple. I think that the undertones of the article may make readers more likely to panic if they run into bitwise operations in the wild. I use them quite often for work, and its really not bad. Its just a thing. I do appreciate the depth of the article though, good post!
I don't do much programming in Java script and I don't know why one would use bit masks in a language that is mostly for browser side high level coding.
But calling them esoteric and impractical would lead me to believe you don't deal with embedded coding and good old standard c. Often in embedded systems we can't afford to be wasteful with resources, since c has a much smaller compiled foot print than C++ and higher languages it is used.
Also when working with creating a client server protocol that sends and receives data packets (like a multi player game), you often need to send flags that say things like jump, attack, crouch, move, etc..
You could create a struct to send as a packet using a bunch of C++ BOOL's. But here's the rub, if you're using a bool per flag, a bool requires minimum 1 Byte or 8 bits; however, if you use an unsigned int as a bitfield, and bit masks, you can effectively have 8 bools per byte for a char, or uint8_t.
You get 8 times the bool flags, with 1/8th the data packet size. Reducing the data packet size reduces the latency. Also bitwise operators are very fast for the cpu to process.
For Java Script, I'd probably not bother with bitmasks and bitwise operators, but if you are using c or c++ and trying to minimize the amount of memory you're using, bit masks will always beat bool.
Yup, you are definitely correct there. Perhaps I have been too harsh on bitmasks. I never really considered embedded systems when I wrote this article. I simply had a "JavaScript mindset", so to speak.
I absolutely love working with bitmasks! They pair incredibly well with MongoDB: there are query expressions for $bitsAllSet/$bitsAnySet + $bitsAllClear/$bitsAnyClear to make for some very succinct and expressive queries, and then there also $bit update operators for setting/unsetting bits. Field selection is a lot more straightforward; it indexes well, especially w/ compound indexes; and it just feels clean. I've found that a lot of queries are just much easier to express like this, for instance:
Also, there are a lot of times in which logical expressions are much cleaner with bitmasks. For instance, say you wanted to update a doc and wanted to trigger some behavior based off of the changes; the optimal way to do this IMO is to use a findOneAndUpdate with bitwise updates ($bit) and then to return the old doc / sync the bitwise updates and to compute the XOR. Once you have the XOR of the update, it's really easy to derive events: IE
status & XOR & VERIFIED-> they are verified & they weren't verified before this update. The most powerful use case I've found is for shared documents, where you have a subdocument array for user data w/ an ObjectID ref + a bitmask, and then you can use $elemMatch updates to set each users respective flags, and then you can do awesome things like taking the AND of each users flags + the XOR of an update as to derive events based of when flags are mutually set by both users and became such as a result of this update. This very simple design pattern works for a staggering number of use cases.Ayyyyy. I can't disagree with your level of enthusiasm! Bitmasks are definitely great for these kinds of use cases. As long as the code for these design patterns are relatively straightforward and easy to fathom, bitmasks aren't really as "bad" and "esoteric" as I have made them to seem like in the article.
Hey, your article was very useful. Thank you for that. I think it's useful for use-cases when you have lists of booleans and you want to compare against. Instead of diffing in loops "foo === arr[i] && true === arr[i]" having into numbers is simpler.
I've expanded and made a class for making it simpler to use. I've called it tuples-boolean-bitmasker, the source is in a project on a GitLab monorepo I own, here are the unit tests I've written while building BitMasker
Very nice explanation. In JS React actually uses Bitmasks for the modes it provides since you can wrap many modes into each other it gives you a simple way to pass all the available mode states as a single variable.
In Unix file permissions are handled this way: like 6 = 110 (read & write) or 7 = 111 (rwx)
I worked with bitmasks(a while back) to write a decoder for GRIB, a binary format for weather data. The utility of a bitmask in this context seems much more obvious. It allows you to extract individual bits from a given byte of data. There is NO other way you can do this because processors do not handle anything under 1 byte.
I am new to the software development and we use bitwise masking. It's a real nightmare for newbies as code is totally unreadable. At least I might understand some basics now.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.