How Go's net/http Handles TCP Connections

Preface

Before we begin, let's briefly explain two concepts, Socket and File Descriptor, to facilitate later understanding.

What is a Socket?

A Socket is a fundamental abstraction for network communication. It provides a standard interface for applications to access the network protocol stack. Simply put, a socket is an endpoint for network communication, allowing programs on different computers to exchange data over the network.

Main features of Socket:

It acts as the interface between the application layer and the transport layer.
It can be regarded as a special kind of file, supporting read and write operations.
There are different types: TCP Socket (connection-oriented), UDP Socket (connectionless), etc.

What is a File Descriptor?

A File Descriptor is an integer value used by the operating system to identify and manage open files. In Unix/Linux systems, everything is a file, including regular files, directories, devices, and even network connections.

Key points about file descriptors:

It is a non-negative integer, usually starting from 0 (0 is standard input, 1 is standard output, 2 is standard error).
In the OS kernel, the file descriptor is an index pointing to a file table entry.
Each process has its own file descriptor table.

Relationship Between Socket and File Descriptor

In Unix/Linux systems, sockets are also considered a special kind of file, and therefore also have corresponding file descriptors. When you create a socket:

The operating system allocates a file descriptor.
This file descriptor can be used for subsequent network operations (read, write, close, etc.).
Applications interact with the socket through this file descriptor.

TCP Connection Establishment Process

Socket Creation

// Internal implementation of the net package
fd, err := socket(family, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)

This step creates a socket file descriptor through a system call.

Server Bind and Listen

// Simplified server code flow
bind(fd, addr)
listen(fd, backlog)

The server binds the socket to a specific address and port, and then starts listening for connection requests.

Accepting Connections

// Simplified version from net/http/server.go
func (srv *Server) Serve(l net.Listener) error {
    for {
        rw, err := l.Accept()  // Accept a new connection
        if err != nil {
            // Handle error
            continue
        }
        go srv.newConn(rw).serve(ctx)  // Create a new goroutine for each connection
    }
}

Client Connect

// Simplified version from net/http/transport.go
func (t *Transport) dialConn(ctx context.Context, addr string) (*conn, error) {
    // Create TCP connection
    netConn, err := t.dial(ctx, "tcp", addr)
    if err != nil {
        return nil, err
    }
    // Wrap as HTTP connection
    return &conn{
        conn: netConn,
        // ... other fields
    }, nil
}

Data Transmission

// Reading data
n, err := syscall.Read(fd, buf)
// Writing data
n, err := syscall.Write(fd, data)

Closing the Connection

err := syscall.Close(fd)

Key Implementation Details

Multiplexing

HTTP/1.1 uses the Keep-Alive mechanism to reuse TCP connections.
HTTP/2 implements multiplexing through streams, allowing multiple HTTP requests to share a single TCP connection.

Connection Pool Management

// net/http/transport.go
type Transport struct {
    // Idle connection pool
    idleConn     map[connectMethodKey][]*persistConn
    // Maximum number of idle connections
    maxIdleConns int
    // ... other fields
}

Timeout Control

// Set connection timeout
conn.SetDeadline(time.Now().Add(timeout))

Error Handling and Retry Mechanism

// Simplified retry logic
for retry := 0; retry < maxRetries; retry++ {
    conn, err := dial()
    if err == nil {
        return conn
    }
    // Wait before retrying
    time.Sleep(backoff)
}

Workflow

When the Client Initiates an HTTP Request:

First, check if there is an available connection in the connection pool.
If not, create a new TCP connection.
Send the HTTP request data.
Wait for and read the response.

When the Server Handles a Request:

The Accept loop accepts new connections.
A goroutine is created for each connection.
The HTTP request is parsed.
The request is processed and a response is returned.

This is the core mechanism by which the net/http package implements HTTP connections on top of the TCP protocol. Through abstraction and encapsulation, developers do not need to handle low-level TCP connection details directly, while also benefiting from efficient connection management and reuse mechanisms.