Adding Brotli to an Already-built NGINX Instance

#brotli #nginx #bash #server

Do you have a running instance of NGINX and wanted to add Brotli compression? What if you don’t want to recompile a custom web server from source or replace your auto-updating repo-installed version? You can create dynamic Brotli modules which upgrade whenever NGINX does.

The bash script below explains the installation steps. Feel free to study and execute the codes step-by-step. That may help you troubleshoot any peculiarities in your system. Otherwise, simply run the script.

#!/bin/bash

# https://www.majlovesreg.one/tag/code/
# https://www.majlovesreg.one/adding-brotli-to-a-built-nginx-instance
# https://github.com/majal/maj-server/tree/master/nginx-modules-brotli

# Install needed development packages if not yet installed in the system
# sudo apt -y install git libpcre3 libpcre3-dev zlib1g zlib1g-dev openssl libssl-dev

# For predefined NGINX version, use:
# ngver=1.17.1

# For passing the version via the command line (i.e.: user@server:~$ ./mkbrotli 1.17.1), use:
ngver=$1

# For automated detection of currently installed NGINX version (not to be used for auto-updating, see hooks in post), use:
# ngver=$(nginx -v 2>&1 | grep -o '[0-9\.]*')

# Get configure parameters of installed NGINX. Not needed if NGINX was configured '--with-compat'.
# Uncomment one of the lines below if the script sucessfully builds modules but NGINX throws a "not binary compatible" error.
# confparams=$(nginx -V 2>&1 | grep -o -- '--prefix='.*)
# confparams=$(nginx -V 2>&1 | grep -o -- '--[^with]'.*)
# confparams=$(nginx -V 2>&1 | grep -- '--' | sed "s/.*' //")

# To manually set NGINX modules directory:
# moddir=/path/to/modules/directory

# To automatically select NGINX modules directory:
[ -d /usr/share/nginx/modules ] && moddir=/usr/share/nginx/modules
[ -d $(nginx -V 2>&1 | grep -o 'prefix=[^ ]*' | sed 's/prefix=//')/modules ] && moddir="$(nginx -V 2>&1 | grep -o 'prefix=[^ ]*' | sed 's/prefix=//')/modules"
[ -d $(nginx -V 2>&1 | grep -o 'modules-path=[^ ]*' | sed 's/modules-path=//') ] && moddir="$(nginx -V 2>&1 | grep -o 'modules-path=[^ ]*' | sed 's/modules-path=//')"
[ "${moddir}" ] || { echo '!! missing modules directory, exiting...'; exit 1; }

# Set temporary directory and build on it
builddir="$(mktemp -d)"
cd "${builddir}"

echo
echo '################################################################################'
echo
echo "Building Brotli for NGINX ${ngver}"
echo "Temporary build directory: ${builddir}"
echo "Modules directory: ${moddir}"
echo

# Download and unpack NGINX
wget https://nginx.org/download/nginx-${ngver}.tar.gz && { tar zxf nginx-${ngver}.tar.gz && rm nginx-${ngver}.tar.gz; } || { echo '!! download failed, exiting...'; exit 2; }

# Download, initialize, and make Brotli dynamic modules
git clone https://github.com/google/ngx_brotli.git
cd ngx_brotli && git submodule update --init && cd ../nginx-${ngver}
[ "${confparams}" ] && nice -n 19 ionice -c 3 ./configure --add-dynamic-module=../ngx_brotli "${confparams}" || nice -n 19 ionice -c 3 ./configure --with-compat --add-dynamic-module=../ngx_brotli
nice -n 19 ionice -c 3 make modules || { echo '!! configure or make failed, exiting...'; exit 4; }

# Replace Brotli in modules directory
[ -f "${moddir}/ngx_http_brotli_filter_module.so" ] && sudo mv "${moddir}/ngx_http_brotli_filter_module.so" "${moddir}/ngx_http_brotli_filter_module.so.old"
[ -f "${moddir}/ngx_http_brotli_static_module.so" ] && sudo mv "${moddir}/ngx_http_brotli_static_module.so" "${moddir}/ngx_http_brotli_static_module.so.old"
sudo cp objs/*.so "${moddir}/"
sudo chmod 644 "${moddir}/ngx_http_brotli_filter_module.so" || { echo '!! module permissions failed, exiting...'; exit 5; }
sudo chmod 644 "${moddir}/ngx_http_brotli_static_module.so" || { echo '!! module permissions failed, exiting...'; exit 6; }

# Clean up build files
cd "${builddir}/.."
sudo rm -r "${builddir}/ngx_brotli"
rm -r "${builddir}"

echo
echo "Sucessfully built and installed latest Brotli for NGINX ${ngver}"
echo "Modules can be found in ${moddir}"
echo "Next step: Configure dynamic modules and reload/restart NGINX."
echo
echo '################################################################################'
echo

# Start/restart NGINX.
# Not recommended if script is hooked since NGINX is automatically restarted by the package manager (e.g. apt) after an upgrade.
# Restarting NGINX before the upgrade could cause a module version mismatch.
# sudo nginx -t && { systemctl is-active nginx && sudo systemctl restart nginx || sudo systemctl start nginx; } || true
# echo
# systemctl --no-pager status nginx
# echo

You may choose to automate the installation by simply running the script above. In its default settings, it will process the version number you will pass to it. To get your NGINX’s version number run this command:

$ nginx -v
  nginx version: nginx/1.17.1

Since the version number in this example is 1.17.1, after you have downloaded the bash script above, go to the directory where the script is and run these commands:

$ chmod +x mkbrotli
$ ./mkbrotli 1.17.1

You may now chill and relax as the script automagically downloads, builds, and installs the dynamic modules for you. If the script runs through errors please let me know in the comments so I may make adjustments to it.

Activating the modules

Now that we have the modules, there are two configuration steps needed to activate Brotli compression. First, load the modules in NGINX’s main configuration file (mine was at /etc/nginx/nginx.conf). You can get your config file path by running this command:

$ nginx -V 2>&1 | grep -o 'conf-path=[^ ]*' | sed 's/conf-path=//'

Now edit NGINX’s conf file. Outside any block { ... }, maybe on the top section of the file, add these lines:

# nginx/nginx.conf

load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;

# http {
#   ...
# }

Second, activate Brotli (and set its configuration directives, aka settings) either inside the http block of NGINX’s conf file, or the server block of your domain configs (e.g. files in nginx/sites-enabled/). As for me, I placed these directives in the nginx/snippets/ folder and called them via include in my domain configs.

# nginx/nginx.conf

# http {
#   ...

  brotli on;
  brotli_static on;
  brotli_types
    text/plain
    text/css
    text/xml
    text/javascript
    text/x-component
    application/xml
    application/xml+rss
    application/javascript
    application/json
    application/atom+xml
    application/vnd.ms-fontobject
    application/x-font-ttf
    application/x-font-opentype
    application/x-font-truetype
    application/x-web-app-manifest+json
    application/xhtml+xml
    application/octet-stream
    font/opentype
    font/truetype
    font/eot
    font/otf
    image/svg+xml
    image/x-icon
    image/vnd.microsoft.icon
    image/bmp;

#   ...
# }

Auto-update with NGINX

One potential problem with dynamic modules is when NGINX automatically upgrades via a package manager (e.g. apt, yum, or pacman). If you’re like me, all upgrades in my server are automated by scripts. You see, dynamic modules are version-specific. NGINX is set not to load modules that were built for a different version. So how will Brotli modules automatically upgrade together with NGINX?

The answer is to hook on the package manager during upgrades. In apt this may be a two- or three-step process:

Hook to apt and call a worker script. (Alternatively, the hook could directly call mkbrotli and thus skip the need for a worker script.)
The worker script calls one or more build scripts to update the dynamic modules.
Build scripts will build and install the new modules right before NGINX upgrades.

Here is my hook file in /etc/apt/apt.conf.d/:

# /etc/apt/apt.conf.d/05nginxmodules

// Hook to build and install dynamic modules before NGINX upgrades
// Script calls individual build scripts and passes back error codes
// Place this file in /etc/apt/apt.conf.d/

DPkg::Pre-Install-Pkgs {"/usr/local/sbin/nginx-mod-preinstall";};

The worker script I use:

# /usr/local/sbin/nginx-mod-preinstall

#!/bin/bash
# Call NGINX module build scripts and pass error codes to apt hook

# Get NGINX version to upgrade to
read ngfile < <(grep '/nginx_') || exit 0
ngver=$(echo $ngfile | sed 's/-.*//' | sed 's/.*_//')

# List of build scripts to run:
/usr/local/sbin/mkbrotli $ngver || exit $?
# /usr/local/sbin/mkmodsec $ngver || exit $?
# /usr/local/sbin/mkpagespeed $ngver || exit $?

And finally the build script for this one is… (drum roll please)… of course mkbrotli. 😉 I hope that Brotli could be natively supported by NGINX in the future. It’s awesome. For now, we have this workaround. Enjoy the compression!

Google PageSpeed

Another project that aims to speed things up on the web is Google’s best-practices optimization called PageSpeed. I also made a Bash script to automate installation of PageSpeed on a running NGINX server. If you’re interested, check out this post: Adding PageSpeed to an Already-running NGINX Instance.

Adding Brotli to an Already-built NGINX Instance appeared first on Maj Loves Reg and was published by Majal Mirasol.