DEV Community

Ethan
Ethan

Posted on

Encrypting my HTTP server with OpenSSL in C 🔒

#programming #security #c #webdev

Implementing TLS in My C HTTP Server

Quick Summary

After building my HTTP server in C, I was eager to start securing the site's traffic. I had heard a lot about SSL/TLS from studying for the CompTIA Security+ certification, but implementing it gave me a unique and more developed understanding of how it's actually integrated with network protocols.

How TLS Works

Before I dive into the code, it's important that we first understand the fundamentals.

TLS Handshake Diagram

The TLS handshake comes into play after the TCP three-way handshake and before any HTTP pages are served.

Here's a step-by-step overview of how the TLS 1.3 handshake unfolds:

The Client Hello Message. The client initiates the handshake by sending a "hello" message to the server. In this message the server will find:

  • The TLS version the client supports
  • The cipher suites supported
  • A string of random bytes known as "client random"
  • Parameters for calculating the premaster secret

Server Generates Master Secret. Server now has the following to create the master secret:

  • Client random
  • Client's parameters
  • Cipher suites

The Server Hello/Finished Message. The server's hello and finished messages include the following:

  • Server's certificate
  • Digital signature
  • Server random
  • Chosen cipher suite

Client Finished. The client verifies the validity of the signature and certificate, then generates the master secret, and sends a "Finished" message.

Last thing to note before we jump into the code implementation...

The TLS handshake utilizes asymmetric encryption. Asymmetric encryption is defined as a process of encryption that uses a pair of mathematically linked keys: a public key and a private key.

After the TLS handshake is completed, future traffic is encrypted using symmetric encryption. Symmetric encryption uses the same secret key to encrypt and decrypt data.

Code Implementation

OpenSSL is an open-source toolkit for cryptographic and secure communication. With OpenSSL we get access to two important libraries: libcrypto and libssl.

Cryptographic library (libcrypto): Contains a variety of cryptographic functions for symmetric encryption (like AES), asymmetric encryption (like RSA), as well as cryptographic hashing (like SHA256).

TLS/SSL library (libssl): Gives us access to the SSL and TLS protocols, which are used for secure connections.

#include <openssl/ssl.h>    // libssl
#include <openssl/err.h>    // openssl errors
#include <openssl/crypto.h> // libcrypto
Enter fullscreen mode Exit fullscreen mode

OpenSSL in Action

Before, my HTTP server used POSIX functions to establish a basic connection. In order to create a secure connection, we'll need some additional functionality working around those POSIX functions.

setup_tls() 

SSL_CTX *setup_tls() {
  SSL_library_init();
  OpenSSL_add_all_algorithms();
  SSL_load_error_strings();

  SSL_CTX *ctx = SSL_CTX_new(TLS_server_method());
  if (!ctx) {
    perror("Unable to create SSL context");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }

  if (SSL_CTX_set_cipher_list(ctx, "DEFAULT:!aNULL:!eNULL") == -1) {
    fprintf(stderr, "failed to set cipher list\n");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }

  if (SSL_CTX_set_ciphersuites(ctx, "TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256") == -1) {
    fprintf(stderr, "Failed to set TLS 1.3 cipher suites\n");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }

  if (SSL_CTX_use_certificate_file(ctx, "certs/cert.pem", SSL_FILETYPE_PEM) == -1) {
    perror("Unable to use cert file");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }

  if (SSL_CTX_use_PrivateKey_file(ctx, "certs/key.pem", SSL_FILETYPE_PEM) == -1) {
    perror("Unable to use private key file");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }

  if (SSL_CTX_check_private_key(ctx) == -1) {
    fprintf(stderr, "Private Key does not match certificate\n");
    exit(EXIT_FAILURE);
  }

  printf("✅ TLS Configured Successfully\n");

  return ctx;
}
Enter fullscreen mode Exit fullscreen mode

There's a lot to unpack here. Let me break down the key components:

Initialization Functions 

SSL_library_init();
OpenSSL_add_all_algorithms();
SSL_load_error_strings();
Enter fullscreen mode Exit fullscreen mode

These three function calls initialize the OpenSSL library, register all available encryption algorithms, and load human-readable error messages that we can use for debugging.

Creating the SSL Context

The SSL_CTX structure holds all the configuration for our TLS implementation. We create it using TLS_server_method(), which automatically selects the highest TLS version supported by both client and server.

Configuring Cipher Suites 

if (SSL_CTX_set_cipher_list(ctx, "DEFAULT:!aNULL:!eNULL") == -1) {
    fprintf(stderr, "failed to set cipher list\n");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }

  if (SSL_CTX_set_ciphersuites(ctx, "TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256")  == -1) {
    fprintf(stderr, "Failed to set TLS 1.3 cipher suites");
    ERR_print_errors_fp(stderr);
    exit(EXIT_FAILURE);
  }
Enter fullscreen mode Exit fullscreen mode

The first function configures cipher suites for TLS 1.2 and earlier, excluding any that don't provide authentication or encryption. The second function specifically sets cipher suites for TLS 1.3, prioritizing AES-256 for maximum security.

Loading Certificates and Private Keys

The certificate and private key are essential for the TLS handshake. The certificate contains the server's public key and is sent to clients during the handshake. The private key remains on the server and is used to decrypt data encrypted with the public key. The SSL_CTX_check_private_key() function verifies that the private key corresponds to the certificate.

Integrating TLS into the Server Loop

With our TLS context set up, we need to modify how we handle client connections:

// After accepting a connection
    client_len = sizeof(client_addr);
    new_fd = accept(sockfd, (struct sockaddr *)&client_addr, &client_len);
    if (new_fd == -1) {
      perror("accept");
      continue;
    }

    // TLS handshake per connection
    SSL *ssl = SSL_new(ssl_ctx);
    if (!ssl) {
      perror("TLS handshake failed");
      continue;
    }

    if (SSL_set_fd(ssl, new_fd) == -1) {
      perror("Failed to set fd with SSL");
      continue;
    }

    printf("Performing TLS handshake...\n");

    if (SSL_accept(ssl) == -1) {
      fprintf(stderr, "SSL_accept failed\n");
      ERR_print_errors_fp(stderr);
      SSL_free(ssl);
      close(new_fd);
      continue;
    }


// Now use SSL_read() and SSL_write() instead of read() and write()
char buffer[4096];
bytes_received = SSL_read(ssl, buffer, sizeof(buffer) - 1);


// ... process request ...

SSL_write(ssl, file_content, size);

// Clean up
SSL_shutdown(ssl);
SSL_free(ssl);
close(new_fd);
Enter fullscreen mode Exit fullscreen mode

The key difference is that we wrap the standard socket file descriptor with an SSL structure using SSL_new() and SSL_set_fd(). The SSL_accept() function performs the entire TLS handshake we discussed earlier. After that, we use SSL_read() and SSL_write() instead of the standard POSIX equivalents.

Key Takeaways

Implementing TLS from scratch gave me several important insights:

The abstraction is powerful but complex. OpenSSL handles an enormous amount of complexity behind relatively simple function calls. Understanding what happens during SSL_accept() helps appreciate the engineering that goes into secure communications.

Certificate management matters. In production, you'll need properly signed certificates from a Certificate Authority. For development, self-signed certificates work fine, but browsers will show security warnings.

Performance considerations are real. The TLS handshake adds latency to each new connection. This is why HTTP/2 and HTTP/3 emphasize connection reuse and why session resumption exists in TLS.

Error handling is critical. OpenSSL provides detailed error information through ERR_print_errors_fp(), which was invaluable during debugging.

What's Next

Now that the server supports TLS, I'm planning to take security further with:

Security Scanner: Build a tool to test my server against common vulnerabilities like weak cipher suites, certificate validation issues, and protocol downgrade attacks.

HTTP Fuzzer: Create a fuzzer to send malformed requests and edge cases to ensure the server handles unexpected input gracefully without crashes or security issues.

Vulnerability Analysis: Implement security headers (HSTS, CSP, X-Frame-Options) and test against tools like Nikto and OWASP ZAP to identify any remaining vulnerabilities.

Try it Yourself

If you want to experiment with TLS implementation, checkout out my project on Github!

Building a TLS-enabled server from scratch has been one of the most educational projects I've undertaken. It transformed my theoretical knowledge of cryptography and network security into practical, hands-on experience. I highly recommend trying it yourself if you want to deepen your understanding of how secure communications work at a fundamental level.

Have questions or suggestions? Feel free to reach out or leave a comment below!

Top comments (0)