DEV Community: Joshua Varghese

HAProxy's Zero-Downtime Reload

Joshua Varghese — Sat, 09 Nov 2024 12:01:54 +0000

HAProxy's Reload Architecture

HAProxy uses a sophisticated socket transfer mechanism between old and new processes. This design choice leads to some interesting trade-offs and benefits.

The Master CLI Process

HAProxy's architecture includes a master CLI process that orchestrates the reload:

Example of HAProxy's master-worker socket handling */
static int proc_self_pipe[2];

void master_register_worker(struct worker *w) {
    struct listener *listener;

    list_for_each_entry(listener, &w->listeners, list) {
        /* Transfer listener sockets to the new worker */
        if (listener->state == LI_READY) {
            listener_transfer_fd(listener, w);
        }
    }
}

Socket Transfer Magic

One of the most interesting aspects of HAProxy's implementation is how it handles socket transfers. The process involves several key steps:

Socket Preparation

static int listener_transfer_fd(struct listener *l, struct worker *w) {
    struct cmsg_fd_list fdlist;
    struct msghdr msg;
    struct iovec iov[1];
    int ret;

    /* Prepare socket data structure */
    memset(&msg, 0, sizeof(msg));
    msg.msg_iov = iov;
    msg.msg_iovlen = 1;

    /* Set up control message for FD passing */
    msg.msg_control = &fdlist;
    msg.msg_controllen = sizeof(fdlist);
}

Graceful Connection Handover HAProxy ensures existing connections aren't disrupted during the reload:

void perform_soft_reload(void) {
    /* 1. Keep accepting new connections in old process */
    while (nb_running_tasks() > 0) {
        /* 2. Process existing connections */
        process_runnable_tasks();

        /* 3. Check if we can stop */
        if (should_exit()) {
            break;
        }
    }
}

State Management During Reload

HAProxy's state management during reload is particularly clever. It handles several key aspects:

Connection State Preservation

struct connection {
    unsigned int flags;     /* Status flags */
    enum obj_type *target;  /* What this connection is about */
    void *ctx;             /* Application specific context */
    /* ... other fields ... */
};

/* During transfer */
static void transfer_connection_state(struct connection *conn) {
    /* Save essential connection data */
    struct conn_state state = {
        .flags = conn->flags,
        .protocol_state = conn->ctx,
        .ssl_state = conn->ssl_ctx
    };

    /* Transfer to new process */
    send_state_to_new_process(&state);
}

SSL Session Handling

A particularly tricky part is managing SSL sessions during reload:

static int ssl_session_transfer(SSL *ssl, struct worker *new_worker) {
    unsigned char *session_data;
    unsigned int session_len;

    /* Serialize SSL session */
    session_data = ssl_serialize_session(ssl, &session_len);
    if (!session_data)
        return -1;

    /* Transfer to new process */
    return send_session_to_worker(new_worker, session_data, session_len);
}

Configuration Validation

Before any reload happens, HAProxy performs extensive configuration validation:

int check_config_validity(char *file) {
    struct proxy *curproxy = NULL;
    int cfgerr = 0;

    /* Parse and validate configuration */
    for (curproxy = proxy_list; curproxy; curproxy = curproxy->next) {
        /* Check proxy settings */
        cfgerr += proxy_cfg_check(curproxy);

        /* Validate SSL configurations */
        cfgerr += ssl_cfg_check(curproxy);

        /* Check backend server configurations */
        cfgerr += check_backend_cfg(curproxy);
    }

    return cfgerr;
}

Health Check Continuity

One often-overlooked aspect is maintaining health check states during reload:

struct check {
    unsigned int status;   /* health check status */
    unsigned int result;   /* test result */
    int code;             /* status code */
    int duration;         /* time it took to get the result */
};

void transfer_health_checks(void) {
    struct server *srv;
    struct check *check;

    /* Iterate through all servers */
    for (srv = servers; srv; srv = srv->next) {
        check = &srv->check;

        /* Save health check state */
        if (check->status & CHK_ST_ENABLED) {
            save_check_state(check);
        }
    }
}

Performance Considerations

HAProxy's reload mechanism is designed with performance in mind. Here are some key optimizations:

Minimal Memory Overhead: Only essential state is transferred
Efficient FD Passing: Uses kernel mechanisms for file descriptor transfer
Progressive Transfer: Connections are handled gradually to avoid spikes

Understanding Envoy Proxy's Hot Restart Implementation

Joshua Varghese — Sat, 09 Nov 2024 11:45:16 +0000

As modern distributed systems grow in complexity, the ability to update proxy configurations without dropping active connections has become crucial. In this post, I'll break down how Envoy Proxy implements its hot restart mechanism, a feature that allows seamless configuration updates and binary upgrades without disrupting existing connections.

What is Hot Restart?

Hot restart (or hot reload) is a mechanism that allows a proxy server to reload its configuration or upgrade its binary while maintaining existing client connections. This is achieved by having the new process take over the listening sockets and existing connections from the old process, ensuring zero connection drops during the transition.

Envoy's Approach

Envoy implements hot restart through a parent-child process model, where the parent process manages the handover of socket descriptors to the new child process. Here's how it works:

Shared Memory Architecture Envoy uses shared memory to facilitate communication between the old and new processes. This is implemented in the HotRestartImpl class:

class HotRestartImpl {
private:
  static constexpr uint64_t MAX_STAT_SEGMENTS = 256;
  SharedMemory* shmem_;
  Stats::StatDataAllocator* stats_allocator_;
};

Socket Passing Process The hot restart process follows these key steps:

Initialize Shared Memory: The parent process creates a shared memory segment that both processes can access.
Socket Duplication: The parent process duplicates its listening sockets.
Graceful Handover: Traffic is gradually transferred to the new process.

Here's a simplified version of how Envoy handles socket passing:

class HotRestartingChild {
public:
  void initialize(int argc, char** argv) {
    // Request parent's listen sockets
    std::vector<int> fds = parent_.retrieveListenSockets();

    // Initialize new server with inherited sockets
    for (int fd : fds) {
      Server::createListenerFromSocket(fd);
    }

    // Signal ready to parent
    parent_.sendReady();
  }
};

State Transfer One of the most critical aspects is transferring the state of existing connections:

void HotRestartImpl::drainListeners() {
  // 1. Stop accepting new connections
  for (auto& listener : listeners_) {
    listener->stopAcceptingConnections();
  }


  // 2. Wait for existing connections to complete
  while (hasActiveConnections()) {
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
  }

  // 3. Signal completion to new process
  notifyNewProcess();
}

Key Implementation Challenges

File Descriptor Handling Envoy needs to carefully manage file descriptors to ensure they're properly transferred and not leaked:

Uses SCM_RIGHTS to pass file descriptors between processes
Maintains a registry of active file descriptors
Implements careful cleanup mechanisms

Connection State Management The proxy must maintain connection state during the transition:

TCP connection parameters
TLS session information
Protocol-specific state (HTTP/2 streams, etc.)

Configuration Compatibility Envoy ensures that configuration changes are compatible with existing connections:

bool HotRestartImpl::validateConfig(
  const envoy::config::bootstrap::v3::Bootstrap& new_config) {
  // Verify that critical fields haven't changed
  // Check listener compatibility
  // Validate cluster configurations
  return isCompatible;
}

Conclusion

Diving into Envoy's hot restart implementation has been quite the journey! It's fascinating to see how they've tackled the challenge of swapping out a running proxy without dropping connections. The elegant dance between parent and child processes, the careful handling of file descriptors, and the intricate state management all come together to make this possible.

What really stands out is how much thought went into making the system robust. It's not just about passing sockets around – it's about handling edge cases, ensuring configuration compatibility, and providing fallback options when things don't go as planned.

Note: This is a high-level overview based on Envoy's open-source implementation. For the most up-to-date and detailed information, please refer to the official Envoy documentation and source.[https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/arch_overview]