I/O Multiplexing

Blocking I/O

Blocking I/O is a method where the program stops and waits until an I/O operation completes.

char buffer[1024];
int n = read(socket_fd, buffer, 1024);  // Blocks here
printf("Data received: %s\n", buffer);   // Won't execute until data arrives

When the read function is called, the program waits until data arrives. This means if no data comes, it just keeps waiting indefinitely. In a single-threaded environment, if one socket is blocked, other sockets cannot be processed.

---

Problems with Thread per Connection

The approach of allocating one thread per client is intuitive and simple to implement, but it has several critical issues in production environments.

void handleClient(int client_fd) {
    while (true) {
        char buffer[1024];
        int n = read(client_fd, buffer, 1024);  // Blocking
        if (n > 0) {
            process(buffer);
            write(client_fd, response, size);
        }
    }
}

while (true) {
    int client_fd = accept(server_fd, ...);
    std::thread(handleClient, client_fd).detach();
}

Issues

• Thread Resource Waste

  • Each thread spends most of its time in an I/O waiting state (Blocking).
  • Actual data processing time is often less than 1% of the entire lifecycle.

• Context Switching Overhead

  • As the number of threads increases, context switching frequency grows exponentially.
  • Costs incurred during context switching:
  • Saving/restoring register state
  • Cache invalidation (increased cache misses)
  • TLB (Translation Lookaside Buffer) flush
  • A significant portion of CPU time is consumed by switching rather than actual work.

• Memory Exhaustion

  • Each thread requires independent stack memory.
  • In large-scale services, memory shortage can lead to system instability.

• Thread Creation/Deletion Cost

  • OS-level overhead during thread creation:
  • Kernel resource allocation
  • Stack memory allocation and initialization
  • TLS (Thread Local Storage) setup
  • Scheduler registration
  • Accumulated costs become significant in environments with frequent connections.

• Scalability Limits

  • C10K Problem: Difficulty handling more than 10,000 simultaneous connections

---

I/O Multiplexing

I/O Multiplexing is a technique where a single process manages multiple file descriptors simultaneously. The program monitors file descriptors to determine what type of I/O events (read, write, exceptions, etc.) have occurred and whether each file descriptor is in a ready state. Methods for implementing I/O Multiplexing include select, poll, and epoll.

---

How I/O Multiplexing Works

1. A single thread registers multiple file descriptors.
2. The OS kernel monitors the state of registered file descriptors.
3. When an I/O operation becomes possible, the application is notified.
4. The application performs I/O operations in a non-blocking manner only on ready file descriptors.

---

I/O Multiplexing Implementation Methods

1. select

• Used to handle multiple files in a single thread.
• Uses an array that stores up to 1024 file descriptors, and searches for target file descriptors using sequential search, causing performance degradation as the number of file descriptors increases.
• Compatible with older systems, but inefficient for modern requirements.

2. poll

• Unlike select, which was limited to a maximum of 1024 FDs, poll can inspect an unlimited number of file descriptors.
• Still uses sequential search, so performance degrades as the number of file descriptors increases, similar to select.

3. epoll

• Supported only on Linux
• Supports kernel-level multiplexing to overcome select's limitations.
• Manages file descriptor state in the kernel and directly notifies state changes.
  • Returns the list of changed file descriptors itself rather than just the count, eliminating the need to loop through files like select and poll, making it efficient.
• Calling epoll_wait eliminates the need to pass monitoring target information every time.
• Two modes:
  • Level-Triggered: Continuously generates events while data remains in the input buffer. Keeps notifying as long as data exists.
  • Edge-Triggered: Generates an event only at the moment data enters the input buffer.

4. IOCP (I/O Completion Port)

• An API that efficiently processes large volumes of non-blocking sockets in Windows environments.
• Features:
  • Suitable for building high-performance servers.
  • Efficiently manages multiple worker threads.
  • Processes through notifications when I/O operations complete.

5. kqueue

• An event notification mechanism used in BSD-family operating systems like FreeBSD and macOS.
• Features:
  • Can monitor various events including sockets, files, and timers.
  • Efficiently manages asynchronous I/O events.

---

epoll vs IOCP vs kqueue

Item	epoll	IOCP	kqueue
Supported OS	Linux	Windows	FreeBSD, macOS, etc.
Purpose	Large-scale network I/O processing	Asynchronous I/O processing	Event-based I/O processing
Operation Method	Event-based asynchronous I/O processing	Notification queue for completed tasks	Event registration followed by notification
Event Management	epoll_wait()	GetQueuedCompletionStatus()	kevent()
Handling Unit	File Descriptor	I/O Handle	File Descriptor, socket, etc.
Performance	Optimized for large-scale client connections	CPU core-based optimization	Efficient with diverse event management
Features	Edge-triggered, Level-triggered support Simple API Scalable	Notification from work queue after I/O completion Suitable for high-performance servers Thread pool based	File I/O, timer, and other event support Multiplexing capable Flexible structure
Advantages	High scalability Can handle many connections	Optimized CPU utilization through thread pool management Suitable for large-scale task processing	Integrated management of multiple event sources Flexible and powerful functionality