DEV Community: Aaron Peschel

Compiling OpenVPN in an Ubuntu 14.04 Chroot

Aaron Peschel — Tue, 07 Nov 2023 22:02:10 +0000

Originally published by apeschel on July 11, 2016, 6:35 p.m.

Set up chroot build environment

Digital Ocean has a good guide for setting up a build chroot available here:

https://www.digitalocean.com/community/tutorials/how-to-configure-chroot-environments-for-testing-on-an-ubuntu-12-04-vps

The only thing of note is that they recommend putting your chroot in your root / directory on your base host. I suggest using /srv/schroot/ instead.

The instructions provided here are for Ubuntu trusty, which is what we will be using to build OpenVPN.

sudo apt-get update
sudo apt-get install dchroot debootstrap

sudo mkdir -p /srv/schroot/ubuntu14.04

sudo debootstrap --variant=buildd --arch amd64 trusty /srv/schroot/ubuntu14.04/ http://us.archive.ubuntu.com/ubuntu/

proc                        /srv/schroot/ubuntu14.04/proc proc    defaults        0       0
sysfs                       /srv/schroot/ubuntu14.04/sys  sysfs   defaults        0       0

sudo mount proc /srv/schroot/ubuntu14.04/proc -t proc
sudo mount sysfs /srv/schroot/ubuntu14.04/sys -t sysfs

sudo cp /etc/hosts /srv/schroot/ubuntu14.04/etc/hosts

sudo chroot /srv/schroot/ubuntu14.04/ /bin/bash

Set Up Build Environment

https://community.openvpn.net/openvpn/wiki/TesterDocumentation http://packaging.ubuntu.com/html/

Having a text editor in our chroot environment will reduce the number of times we have to leave and enter the chroot, thus it is installed for convenience.

apt-get install vim

Our chroot environment will have a very bare-bones sources.list. We will need to add some entries to it.

vim /etc/apt/sources.list

Add the following text.

deb-src http://us.archive.ubuntu.com/ubuntu trusty main

deb http://us.archive.ubuntu.com/ubuntu trusty universe
deb-src http://us.archive.ubuntu.com/ubuntu trusty universe

We need to update Ubuntu's policy-rc.d to let it know that we are in a chroot environment.

vim /usr/sbin/policy-rc.d

Add the following to the file

#!/bin/sh
exit 101

Now we will install any additional build tools that will be required. We will use bzr (bazzar) to build the OpenVPN package.

cd $HOME
locale-gen en_US.UTF-8
apt-get update
apt-get install net-tools autoconf libtool dialog
apt-get install bzr bzr-builddeb devscripts

Build OpenVPN

apt-get build-dep openvpn

Using Canonical Bazaar

Here we download the 14.04 OpenVPN source in a new package branch.

cd $HOME
bzr branch lp:ubuntu/trusty/openvpn

The TLS keys that are used for the OpenVPN tests in the 14.04 source have expired. This will cause the tests to fail. We can use the keys from the latest OpenVPN release instead to fix this.

apt-get install git
git clone git://git.code.sf.net/p/openvpn/openvpn-testing openvpn-openvpn-testing
cp openvpn-openvpn-testing/sample/sample-keys/* openvpn/sample/sample-keys/

Now to finally build the OpenVPN package.

cd $HOME
cd openvpn
bzr bd -- -b -us -uc

Build Raw binaries using make

This will skip the testing phase, which takes a long time with the OpenVPN package. However, it will not create a package and will require installation via make install

cd $HOME
apt-get source openvpn
cd openvpn-*
autoreconf -vi
./configure
make all
make check

Build openvpn-auth-ldap

This follows the same core process as building the OpenVPN package.

apt-get build-dep openvpn-auth-ldap

Using Canonical Bazaar

cd $HOME
bzr branch lp:ubuntu/trusty/openvpn-auth-ldap
cd openvpn-auth-ldap
bzr bd -- -b -us -uc

Using Make

cd $HOME
apt-get source openvpn-auth-ldap
cd openvpn-auth-ldap-*
./configure
make

Using Native Python Libraries in Lambda

Aaron Peschel — Tue, 07 Nov 2023 20:06:54 +0000

It is possible to use native libraries in Amazon Lambda. In this example, we will be building a native bcrypt library against libcrypt, using virtualenv to manage our libraries.

Create an Amazon EC2 Instance to build the native library

Amazon Lambda runs on the Amazon Linux distribution, so we will need to target our compiled libraries against this distribution. Unfortunately, there doesn't appear to be any sort of debootstrap equivalent for Amazon Linux, so an instance using Amazon Linux will be needed for the build process.

Create an instance using Amazon Linux
Connect to the server

Set up the build environment on the server

We will need to install the build tools for the library we are building, as well as any build dependencies that are required.

$ sudo yum update
$ sudo yum install -y gcc44 gcc-c++ libgcc44 cmake
$ sudo yum install -y python27-devel python27-pip gcc libjpeg-devel zlib-devel gcc-c++
$ sudo yum install libffi-devel

Update/upgrade pip and setuptools

Make sure you are using the latest version of pip to prevent any issues.

$ pip update
$ sudo pip install --upgrade pip setuptools

Set up Virtualenv for your project

We will need to package any libraries that the lambda requires along with the lambda. Virtualenv provides us with a convenient method to bundle the libraries along with our lambda function.

Create a directory for your project, and enter it

$ mkdir $WORK
$ cd $WORK
$ virtualenv venv

Activate virtualenv and install libraries

Here we will build and install the libraries into the virtualenv directory, which will make them available for packaging later.

$ source venv/bin/activate
$ pip install --upgrade pip
$ pip install bcrypt

Place your lambda function into the working directory.

$ cp ~/lambda_function.py $WORK

Verify the script is working as expected.

Run your lambda and/or tests locally to verify all the libraries are working as expected.

Package the project, according to the lambda requirements.

Here's the tricky part -- Amazon Lambda requires all the libraries to be at the top level in the zip file that you upload. The following will create a zip file with the lambda function and all required libraries at the top level of the zip file. Since virtualenv contains all the libraries that we need for the project, we will copy all the libraries from the virtualenv environment to the zip file.

$ zip -9 bcrypt_lambda.zip lambda_function.py
$ cd $VIRTUAL_ENV/lib/python2.7/site-packages
$ zip -r9 $WORK/bcrypt_lambda.zip *
$ cd $VIRTUAL_ENV/lib64/python2.7/site-packages
$ zip -r9 $WORK/bcrypt_lambda.zip *

Upload the zip to Amazon Lambda.

Now that you have the zip file, upload it to Amazon Lambda as you normally would. Remember to set up the test data for the lambda to verify that the function is working as expected.

Impact of Hosting Large Numbers of Nginx SSL VHosts

Aaron Peschel — Tue, 07 Nov 2023 18:22:41 +0000

Originally posted by apeschel on Aug. 19, 2015, 6:08 p.m.

Abstract

A question was raised recently about what the impact of hosting extremely large amounts of SSL certificates via Nginx would be. This document provides an explanation of how Nginx manages SSL, provides data about the impact of hosting large numbers of certificates, and provides an analysis of what the data means.

How Nginx Manages SSL Certificates

Logically, each SSL certificate will be associated with a separate host. In Nginx, separate hosts are managed via VHosts. A VHost is a collection of the IP, Port, Hostname, and other relevant data about a host. Nginx collects the information relevant to a VHosts into a ngx_http_core_srv_conf_t struct. These structs are contained within the server_names_hash, which uses the hostname string as a key.

Nginx defers SSL management to OpenSSL. It stores certificates in OpenSSL's SSL_CTX global context structure, and uses the OpenSSL API for any SSL-related interaction between the client and server.

When a connection is made to an nginx server, nginx looks up the given hostname in its server names hash, to find the relevant structure containing the information for that VHost. Once the struct is found, it can use the OpenSSL API to establish the SSL connection with the correct certificate for the requested domain.

There is an upper limit on the number of VHosts that Nginx can fit into its server hash. This upper limit is configurable via two settings:

server_names_hash_bucket_size
server_names_hash_max_size

For optimal performance, server_names_hash_bucket_size should be set to the same size as your processor's cache line. In Linux, you can find this value at /sys/devices/system/cpu/cpu0/cache/index*/coherency_line_size. This leaves you with server_names_hash_max_size as the value that should be modified to increase the VHosts maximum.

For more information about how Nginx manages this stuff, refer to the following Nginx docs:

Testing and Analyzing Nginx Performance

Setup

A simple Nginx host was set up in Digital Ocean for testing. The tests were run on a 2CPU / 2GB instance.

For each iteration, all VHosts were running SSL on the same IP:Port pair, utilizing SNI to do so. Each VHost was configured with an SSL certificate, a corresponding private key, and a unique static file to host. All SSL certificates were generated using a 4096-bit RSA key.

siege was used for stress testing. All the hostnames were put into urls.txt, which is what siege uses to determine what hosts to send traffic to. Siege was configured to use 100 concurrently running clients, with each client attempting to retrieve 50 pages from hosts listed in urls.txt. This totals to 5000 total pages requested from the Nginx host, spread across all the VHosts running on it.

It was noted to me after initially writing this article that siege did not support SNI. It seemed unlikely that SNI support on the client side would have any impact on the results. In the interest of accuracy, I worked with the siege maintainer to add support for SNI and reran the tests. As expected, there was no impact on the results.

The goal of this testing was to determine the memory and performance overhead of hosting a large number of SSL certificates. This means the servers do not very closely reflect what an actual production host would look like. This is a deliberate attempt to isolate only the dependent variables for testing.

Summarized Results

|Num Certs|Memory (KiB)|Time (seconds)|
---------------------------------------
|        1|        3000|            41|
|        2|        3200|            42|
|      100|        9600|            41|
|     1000|       40000|            44|
|    10000|      370000|            43|

Analysis

Nginx took approximately the same amount of time, regardless of the number of VHosts. This is what would be expected from Nginx's use of a hash for managing the VHosts.

Performing a linear regression against the memory data provides us with the following relationship:

m = 3900 + 37 * n

Where

m = Expected Memory Usage (in KiB)
n = Num of VHosts using SSL

This demonstrates that each additional VHost consumes approximately 37 KiB of memory.

Raw Data

The following values are used unless otherwise noted;

server_names_hash_bucket_size: 64
server_names_hash_max_size: 512

No Certificates

root      1740  0.0  0.1  85880  1344 ?        Ss   18:36   0:00 nginx: master process /usr/sbin/nginx
www-data  1741  0.0  0.2  86224  2268 ?        S    18:36   0:00  \_ nginx: worker process
www-data  1742  0.0  0.2  86224  2276 ?        S    18:36   0:00  \_ nginx: worker process
www-data  1743  0.0  0.2  86224  2408 ?        S    18:36   0:00  \_ nginx: worker process
www-data  1744  0.0  0.1  86224  2028 ?        S    18:36   0:00  \_ nginx: worker process

siege -q -b -c 100 -r 50 http://1.apestest/

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  17.87 secs
Data transferred:               1.83 MB
Response time:                  0.30 secs
Transaction rate:             279.80 trans/sec
Throughput:                     0.10 MB/sec
Concurrency:                   83.88
Successful transactions:        5000
Failed transactions:               0
Longest transaction:            4.11
Shortest transaction:           0.05

One Certificate

root      1740  0.0  0.2  85992  3012 ?        Ss   18:36   0:00 nginx: master process /usr/sbin/nginx
www-data  2377  0.0  0.1  86276  1904 ?        S    19:10   0:00  \_ nginx: worker process
www-data  2378  0.0  0.1  86276  1904 ?        S    19:10   0:00  \_ nginx: worker process
www-data  2379  0.0  0.1  86276  1904 ?        S    19:10   0:00  \_ nginx: worker process
www-data  2380  0.0  0.3  86276  3116 ?        S    19:10   0:00  \_ nginx: worker process

siege -q -b -c 100 -r 50 https://1.apestest/

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  40.97 secs
Data transferred:               1.83 MB
Response time:                  0.73 secs
Transaction rate:             122.04 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   89.43
Successful transactions:        5000
Failed transactions:               0
Longest transaction:            5.80
Shortest transaction:           0.12

Two Certificates

root      1740  0.0  0.3  86680  3216 ?        Ss   Aug10   0:00 nginx: master process /usr/sbin/nginx
www-data 22502  0.0  0.1  86680  1992 ?        S    21:32   0:00  \_ nginx: worker process
www-data 22503  0.3  0.3  86680  3440 ?        S    21:32   0:00  \_ nginx: worker process
www-data 22504  0.0  0.1  86680  1992 ?        S    21:32   0:00  \_ nginx: worker process
www-data 22505  0.0  0.1  86680  1992 ?        S    21:32   0:00  \_ nginx: worker process

siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  41.92 secs
Data transferred:               1.83 MB
Response time:                  0.75 secs
Transaction rate:             119.27 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   89.68
Successful transactions:        5000
Failed transactions:               0
Longest transaction:            7.86
Shortest transaction:           0.12

One Hundred Certificates

root      1740  0.0  0.9  92476  9560 ?        Ss   Aug10   0:00 nginx: master process /usr/sbin/nginx
www-data 30236  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
www-data 30237  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
www-data 30238  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
www-data 30239  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process

siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  40.50 secs
Data transferred:               0.67 MB
Response time:                  0.73 secs
Transaction rate:             123.46 trans/sec
Throughput:                     0.02 MB/sec
Concurrency:                   90.17
Successful transactions:           0
Failed transactions:               0
Longest transaction:            6.91
Shortest transaction:           0.00

One Thousand Certificates

server_names_hash_max_size: 1024

root       912  0.0  1.9 124704 40272 ?        Ss   Aug17   0:00 nginx: master process /usr/sbin/nginx
www-data  6543 18.2  1.9 124704 40288 ?        S    02:54   1:01  \_ nginx: worker process
www-data  6544 17.3  1.9 124704 40288 ?        S    02:54   0:58  \_ nginx: worker process
www-data  6545 18.2  1.9 124704 40288 ?        S    02:54   1:01  \_ nginx: worker process
www-data  6546 16.3  1.9 124704 40288 ?        S    02:54   0:55  \_ nginx: worker process

10079> siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  43.83 secs
Data transferred:               1.83 MB
Response time:                  0.77 secs
Transaction rate:             114.08 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   87.71
Successful transactions:        5000
Failed transactions:               0
Longest transaction:           17.73
Shortest transaction:           0.13

Ten Thousand Certificates

server_names_hash_max_size: 16384

root       912  0.0 17.8 449596 365304 ?       Ss   Aug17   0:04 nginx: master process /usr/sbin/nginx
www-data 15625  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
www-data 15626  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
www-data 15627  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
www-data 15628  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process

siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  42.71 secs
Data transferred:               1.83 MB
Response time:                  0.74 secs
Transaction rate:             117.07 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   87.15
Successful transactions:        5000
Failed transactions:               0
Longest transaction:           14.30
Shortest transaction:           0.12

Bringing Up a Bonded Interface in Linux Without a Reboot

Aaron Peschel — Tue, 07 Nov 2023 11:26:11 +0000

Originally by apeschel on Jan. 28, 2015, 7:18 p.m.

There was an issue with some servers I was working on recently where two interfaces were meant to be set up with bonding. Unfortunately, due to a misunderstanding, the servers were brought up and put into production without the bonding being set up. When the problem was noticed later, the initial assumption was that fixing the bonding would require a reboot, which meant we would need to schedule downtime for the servers. We had a week scheduled to cycle through the servers, but I was determined to find a way to cut down the time by figuring out how to fix the bonding without a reboot.

After digging around, and asking around on FreeNode, I was not able to find anyone who had come up with a solution to this problem. So I did some experimentation, and came up with a process that only resulted in a couple seconds of downtime for each server while the IP switched from eth0 to bond0.

The initial state of the servers looked like the following.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 1.1.1.2/24 brd 1.1.1.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether aa:bb:cc:dd:ee:f1 brd ff:ff:ff:ff:ff:ff

Note that the IP address is associated with the eth0 interface, and neither interface is associated with the bond.

The first step is to get the corrected configs for the interfaces in place. These servers were running Ubuntu, which uses /etc/network/interfaces for managing interfaces. After reading the associated kernel documentation, I came up with the following config. Read through the kernel docs and update the config for your distribution according to what you need.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
  bond-master bond0
  bond-primary eth0 eth1

auto eth1
iface eth1 inet manual
  bond-master bond0
  bond-primary eth0 eth1

auto bond0
iface bond0 inet static
  bond-slaves none
  bond-mode 802.3ad
  bond-miimon 100
  address 1.1.1.2
  netmask 255.255.255.0
  gateway 1.1.1.2
  bond-xmit_hash_policy layer3+4

The final step is to enact the changes. There is risk involved in doing this via a network connection. I was able to do this via SSH by taking down the interfaces and bringing them back up all in one command. Having a local console open on the machine is a better method, in case something goes wrong.

sudo ifdown eth0 && sudo ifdown eth1 && sudo ifup eth0 && sudo ifup eth1

At this point, the IP address will be associated with both the eth0 and bond0 interfaces, and needs to be cleared from eth0.

sudo ip addr del 1.1.1.2/24 dev eth0

Now everything should be set up correctly.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 1.1.1.2/24 brd 1.1.1.255 scope global bond0