Why doesn't throwing more CPU and RAM at your server solve your latency issues under heavy load?
In this article, we will see why buying a more powerful server will not increase the performance of your blocking I/O app. Instead of just throwing hardware at the problem, we are going to get our hands dirty with systems architecture and build a custom, multi-threaded Non-Blocking I/O (NIO) HTTP server in Java from scratch. And yes, we will talk about the painful concurrency bugs you run into when you try to build the parts a framework normally hides from you.
This is Part 2 of a series where I document building an HTTP server in Java from raw sockets. Part 1 covers a buffering bug in blocking I/O — this one covers what happened under concurrent load and the NIO rewrite that followed.
Chapter 1: The Trap of Blocking I/O (BIO)
Imagine you are at a restaurant where one waiter is assigned to one table. When the waiter takes your order, they walk to the kitchen and stand there staring at the wall while the chef cooks your meal. While that waiter is waiting, no other customer in the restaurant can be served. If the restaurant gets busy, the owner's only solution is to hire more waiters. Eventually, the kitchen gets overcrowded, salaries skyrocket, and the restaurant runs out of physical space.
This is exactly how Blocking I/O (BIO) works.
In a classic blocking Java server (using ServerSocket), every incoming TCP connection gets its own dedicated OS thread. When a thread tries to read from a socket, it blocks:
// The thread is now frozen here until the client decides to send something
int bytesRead = inputStream.read(buffer);
If the client has slow internet, or if your database query takes a few seconds, that thread is just sitting there wasting resources.
If you think, "I'll just buy a bigger server with more RAM and spin up more threads," here is the catch:
- Thread Overhead: In Java, each thread takes up about 1MB of memory for its stack. If you have 5,000 active connections, that is roughly 5GB of RAM gone just to keep idle threads sleeping.
- Context Switching: The CPU spends more time swapping between thousands of threads than actually running your code.
- Bottlenecks: Your database connection pool or OS files will max out long before your fancy CPU even hits 50%.
The Non-Blocking (NIO) Solution
Instead of one waiter per table, imagine a single waiter with a pager system. The waiter takes orders from multiple tables, sends them to the kitchen, and moves on to serve other people. When a dish is ready, the kitchen buzzes the pager. The waiter only works when there is an actual event to handle.
Chapter 2: Migrating from BIO to Java NIO
When we migrate from Blocking I/O to Non-Blocking I/O in Java, we swap out our core networking objects:
-
ServerSocketbecomesServerSocketChannel: We can configure it to be non-blocking by callingssc.configureBlocking(false). This lets us listen for connection events without freezing the thread. -
SocketbecomesSocketChannel: Reads and writes become non-blocking. Callingread()returns immediately with whatever data is currently ready.
To understand how this works, we need to talk about Channels, Buffers, and Selectors.
1. Channels (SocketChannel)
Think of a channel as a two-way pipe. Unlike standard blocking streams, you can tell it to be non-blocking. If you ask it to read, it returns immediately: it gives you the bytes if they are there, a 0 if the network is still loading, or -1 if the client closed the connection.
2. Buffers (ByteBuffer)
You cannot read directly from a channel. You must read from the channel into a buffer, and write from a buffer into the channel.
A ByteBuffer is not like a normal array or list. It has three pointers that you have to manage manually:
-
capacity: The size of the buffer. -
position: Where you are currently reading or writing. -
limit: The boundary of how far you can go.
Because there is only one pointer (position) for both reading and writing, you have to tell the buffer when you are switching roles by calling flip() and clear().
To show you what is actually happening under the hood, here is a simple conceptual class:
public class MockByteBuffer {
private byte[] array;
private int position = 0;
private int limit;
private int capacity;
public MockByteBuffer(int capacity) {
this.capacity = capacity;
this.limit = capacity;
this.array = new byte[capacity];
}
// 1. Write mode: putting data into the buffer
public void put(byte b) {
if (position < limit) {
array[position++] = b;
}
}
// 2. Flip: Switch from WRITE mode to READ mode
public void flip() {
limit = position; // Limit becomes the end of written data
position = 0; // Reset pointer to start reading from the beginning
}
// 3. Clear: Switch from READ mode back to WRITE mode
public void clear() {
position = 0; // Reset pointer to the start for writing
limit = capacity; // Restore limit to full capacity
}
}
- When you write bytes into the buffer (like receiving data from a socket), the position moves forward.
- Before you read those bytes, you call
flip(). This resets the position to0and sets the limit to where you stopped writing, so you don't read empty space. - Once you are done reading, you call
clear()to reset the pointer so you can write into it again.
3. Selectors
The Selector is the brain of NIO. It is the multiplexer that monitors multiple channels.
- You register your channel to a selector and tell it what you are interested in (like
SelectionKey.OP_READfor incoming data orSelectionKey.OP_ACCEPTfor new connections). - You call
selector.select(), which blocks your thread only until one of those events happens. - Then you get the
selectedKeys()and process them one by one.
Chapter 3: Architecting the Event Loop (Boss & Workers)
For our server, we are going to implement a simple Boss-Worker model:
-
The Boss Thread (NioServer.java): Its only job is to run a selector that listens for new client connections (
OP_ACCEPT). When a client connects, it accepts the channel, sets it to non-blocking, and hands it over to the worker pool. -
The Worker Pool (SelectorsPool.java): A pool of threads, each running its own selector. They handle the reading (
OP_READ) and processing for the clients assigned to them.
Here is how the Boss Thread sets up the ServerSocketChannel and binds it:
public void start() {
try (ServerSocketChannel ssc = ServerSocketChannel.open()) {
ssc.bind(new InetSocketAddress(host, port));
ssc.configureBlocking(false); // Make it non-blocking
Selector selector = Selector.open();
ssc.register(selector, SelectionKey.OP_ACCEPT); // Listen only for accepts
while (true) {
selector.select(); // Block until a client connects
Set<SelectionKey> keys = selector.selectedKeys();
Iterator<SelectionKey> iter = keys.iterator();
while (iter.hasNext()) {
SelectionKey key = iter.next();
if (key.isAcceptable()) {
SocketChannel sc = ssc.accept();
sc.configureBlocking(false);
// Pass the channel to our worker pool!
pool.execute(sc);
}
iter.remove();
}
}
} catch (Exception e) {
System.out.println("Error in the nio server: " + e);
}
}
Chapter 4: Processing the Data (NioWorker.java)
Once the client is registered to a worker's selector, the worker thread reads the bytes and routes the request:
public class NioWorker implements Runnable {
private SocketChannel sc;
private Dispatcher dispatcher;
public NioWorker(SocketChannel client, Dispatcher dispatcher) {
this.sc = client;
this.dispatcher = dispatcher;
}
@Override
public void run() {
ByteBuffer buffer = ByteBuffer.allocate(1024);
StringBuilder sb = new StringBuilder();
try {
int r;
// Read bytes from channel into buffer
while ((r = sc.read(buffer)) > 0) {
buffer.flip(); // Switch to read mode
sb.append(new String(buffer.array(), 0, buffer.limit(), StandardCharsets.UTF_8));
buffer.clear(); // Clear buffer for next write
}
String message = sb.toString();
// Route the request and write the response back
String res = dispatcher.rout(message.split("\n")[0]);
ByteBuffer resBuffer = ByteBuffer.wrap(res.getBytes());
sc.write(resBuffer);
sc.close();
} catch (Exception e) {
System.out.println("Error in the worker: " + e);
}
}
}
Known simplification: this version reads once per worker invocation and assumes the full request arrived in that pass — it doesn't yet register back with a selector to wait for more OP_READ events if a client's request is split across multiple TCP packets. That's a real gap between this code and the "worker has its own selector" model described above, and it's on my list to fix in a later revision rather than something I'm claiming is solved here.
Chapter 5: The Hidden Trap of Custom NIO
Building your own web server is great for learning, but it exposes you to subtle bugs that modern frameworks hide. Let's look at one of the biggest headaches you will encounter:
The Selector Registration Deadlock
Look at how the Boss hands a socket to a worker's selector:
// Boss Thread:
sc.register(workerSelector, SelectionKey.OP_READ);
And look at how the Worker is waiting:
// Worker Thread:
workerSelector.select(); // Blocks here waiting for network activity
In Java, selector.select() locks the selector's key set. At the same time, calling socketChannel.register(selector, ...) from the Boss thread tries to get that exact same lock.
If the worker thread is blocked inside select(), the Boss thread will hang forever on .register(), and your entire server will freeze.
The Fix
You have to wake up the selector before registering the channel:
workerSelector.wakeup();
sc.register(workerSelector, SelectionKey.OP_READ);
Note: Because wakeup() and register() are not atomic, production frameworks like Netty use a concurrent queue to enqueue registration tasks and run them directly inside the worker thread loop, avoiding this lock contention entirely.
Benchmarks: Proving the Architecture
Same test every time — an artificial 5-second delay in the request handler to simulate a slow database call, hit with ab -n 10 -c 10 http://localhost:8080/:
| Server | Total time (10 requests, concurrency 10) | Requests/sec |
|---|---|---|
| 1 thread, blocking (no pool) | 50.015s | 0.20 |
| 1 NIO worker + boss | 50.015s | 0.20 |
| 10 NIO workers + boss | 10.009s | 1.00 |
The 1-worker NIO server and the 1-thread blocking server land on almost identical numbers — which makes sense, since one worker means every request still queues behind the last one, regardless of blocking or non-blocking I/O underneath. The moment you add nine more workers, ten requests that would've taken 50 seconds serially finish in about 10 — a 5x improvement from spreading load across workers, not from any change in hardware.
(I also ran this with a 10-thread blocking pool instead of one thread, which is the more common real-world setup — I don't have that exact run saved, so I'm leaving it out rather than guessing at a number. Worth noting though: since 10 threads can serve 10 concurrent requests at once, that version should land close to the single request's 5-second delay rather than 50 seconds — the same idea as the NIO version, just paying for it in threads instead of workers.)
Conclusion & Takeaways
Writing a bare-metal web server teaches you three main things:
- Architecture > Hardware: Buying a bigger server is useless if your system is bottlenecked by blocking threads.
- NIO is Hard: Managing buffers, selector states, and thread deadlocks manually is highly complex.
- Appreciate the Ecosystem: This project makes you appreciate why frameworks like Netty, Tomcat, and Spring Boot exist. They handle the low-level heavy lifting so you can focus on writing actual business features.
Further Reading
- "Non-blocking I/O and how Node uses it, in friendly terms" by Studying With Alex — a good visual walkthrough of the blocking vs. non-blocking distinction.
- Baeldung's guide to Java NIO Selectors — a deeper dive into managing selector keys.
Code for this article is tagged at https://github.com/AnasMAC/java-baremetal-server/ — that snapshot matches exactly what's described above. The main branch has since grown to include a database connection pool, dependency injection, and server-side rendering, which are their own upcoming articles in this series.
Top comments (0)