If you need to adapt content, authorize, filter, change the behavior of Nginx, so njs
can be the solution. Nginx proposes a JavaScript backend to manipulate requests and responses and even streams. This is njs
and it is very powerful.
Nginx is a very good HTTP server and reverse proxy. It is simple to configure, easy to start, and it "does the job". We often use it as a proxy to backends. But, sometimes, you can feel some limitations. For example if you need to authorize users to several backends with a specific identity provider with a weird API. Or, sometimes, this is the backend which is problematic.
For example, I serve a Rancher backend, and Rancher provides its kubeconfig
YAML file (dynamically generated for each user) with a certificate inside that I need to remove.
In short, I need to "filter" the content, because I cannot change the behavior of the backend.
And this is exactly when
njs
can be used!
What I could do is simply to detect that the user claims the KubeConfig file, remove the certificate entry in the YAML, and serve the response file. This is one of the vast variety of manipulation that you can do with njs
.
What it njs
?
njs
is a JavaScript engine integrated to Nginx. njs
provides a different approach that starts a JS VM on each needed process. It actually can use common JavaScript modules (like fs
for example), and it's ES6 compliant. That means that you will code the scripts without the need of changing your habits.
What you can do
There are plenty of things that you can do with njs
:
- Make authorization (with or without backend)
- Manipulate the output content, headers, status...
- Interact with streams
- Use cryptography
- And so on...
You must keep in mind that njs
is not intend to create an application. Nginx is not an application server, it's a web server and a reverse proxy. So, it will not substitute a "real" web framework. But, it will help to fix some things that are hard to do.
Read this before!
Go to the documentation page here to check the global variables which are already defined by Nginx/njs, you'll see that there are many methods to trace, crypt, decode, or manipulate the requests.
Do not spend too much of time to read the page, but just take a look to be aware of the possibilities.
It's not activated by default
njs
is not activated by default. It's a module that you need to load at startup.
The legacy method to activate it is to add load_module modules/ngx_http_js_module.so;
on top of nginx.conf
file.
For docker, you can change the "command" to force the module to be loaded:
docker run --rm -it nginx:alpine \
nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"
Then in your http
services, you can now load a script and call functions for different situation.
Prepare the tests
To follow this article, and be able to test, we will start a Nginx service and a "Ghost" blog engine.
This will help to startup:
mkdir -p ~/Projects/nginx-tests
cd ~/Projects/nginx-tests
mkdir -p nginx/conf.d
touch nginx/conf.d/default.conf
cat << EOF | tee docker-compose.yaml
version: "3"
services:
# our nginx reverse proxy
# with njs activated
http:
image: nginx:alpine
command: nginx -g "daemon off; load_module modules/ngx_http_js_module.so;"
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d:z
ports:
- 80:80
# a backend
ghost:
image: ghost
depends_on:
- http
environment:
url: http://example.localhost
EOF
Debug
Sometimes, your script fails to return somthing, you've got an error 500 and nothing is clearly displayd in the nginx logs.
What you can do is to add this in the "location":
error_log /var/log/nginx/error.log debug;
For docker/podman users, use:
# see the logs in real time
docker-compose logs -f http
And you can use in your script :
r.log("something here")
r.log(njs.dump(variable))
When you finally found the problems, you can then remove the "debug" suffix of the "error_log
" directive.
There are many ways, many options, many situations
Nginx manages 2 types of configuration: http and stream. By chance, njs
can be used for both.
In this article, we will only speak about the "http" mode.
When we work with "http" mode, we can tell to Nginx to use JavaScript to manipulate:
- The content (that means that we will generate the entire request before the call)
- The headers only (after request)
- The body only (after request)
- Set variables
- ...
The pipeline is not hard: import a javascript file, then use a Nginx directive to use a function for a situation. Nothing more.
First example, create a content
A first, let's create a file named example.js
. To make it easy to test, put this in nginx/conf.d/example.js
β in production environment, it's better to change the location of your scripts.
OK, so, this is the content:
function information(r) {
r.return(200, JSON.stringify({
"hello": "world",
}));
}
// make the function(s) available
export default {information}
The r
variable (argument of the function) is a "request" object provided by Nginx. There are many methods and properties that we can use, here we only use r.return()
.
It's now time to make the JavaScript function to be called as the content maker. In nginx/conf.d/default.conf
, append this:
js_import /etc/nginx/conf.d/example.js;
server {
listen 80;
server_name example.localhost;
location / {
js_content example.information;
}
}
Yes, that's all. We import the script, and we use the function as js_content
. That means that Nginx will release the request to the script, and the script can yield the content.
Start (or restart) the container and hit the "example.localhost" domain:
$ docker-compose up -d
$ curl example.localhost
{"hello": "world"}
That's the first example. Here, we only generate a JSON output.
We can now do some nice things, like replacing the content.
Replacing the content
njs
proposes several fetch
APIs to get, sub-request or internally redirect the request.
In this example, we will replace the "/about/" page content by inserting a message inside.
The following configuration is not perfect (actually we could match location /about
and call our JavaScript, but I want to show you the internal redirection) β but it will show you some cool stuffs that njs
allows.
Change the nginx/conf.d/default.conf
file with this:
js_import /etc/nginx/conf.d/example.js;
upstream ghost_backend {
server ghost:2368;
}
server {
listen 80;
server_name example.localhost;
gunzip on;
# call our js for /exemple url
location = /exemple {
js_content example.information;
}
# match everything
location / {
js_content example.replaceAboutPage;
}
# call the "ghost blog" backend
# note the "arobase" that creates a named location
location @ghost_backend {
# important !
subrequest_output_buffer_size 16k;
proxy_pass http://ghost_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
}
In this file, you have to pay attention on this:
We force "gunzip" to uncompress proxied responses. This is important if the backend forces
gzip
responses, and you want to make replacements.
We also make the "subrequest output buffer" size to 16k, you can set it to 128k if needed (for large CSS files if you subrequest them for example).This is important because we will make subrequests later, and the buffer will be filled too fast
Then, in the example.js
file, add the replaceAboutPage
function and export it :
function information(r) {
r.return(
200,
JSON.stringify({
hello: "world",
})
);
}
async function replaceAboutPage(r) {
if (r.uri == "/about/") {
r.headersOut["content-type"] = "text/html";
r.return(200, "Changed");
} else {
r.internalRedirect("@ghost_backend");
}
}
export default { information, replaceAboutPage };
Take a minute to read the replaceAboutPage
function. In this example, we only:
- return a page with "Changed" inside if the URI is "/about"
- either we use
@ghost_backend
location to proxy the blog
Restart the nginx container:
docker-compose restart http
And visit http://example.locahost. Then go to the "/about" page using the "About" link on top.
Nice, so now, we can do a better replacement.
We will need to "subrequest" the page. But a subrequest needs a "real URI path". So, let's add a location in default.conf
first.
# a reversed uri.
# We remove the prefix and use the @ghost_backend
location /__reversed {
internal;
rewrite ^/__reversed/(.*)$ /$1 break;
try_files $uri @ghost_backend;
}
Then, let's go to example.js
and replace the replaceAboutPage
function to this one:
//...
async function replaceAboutPage(r) {
if (r.uri == "/about/") {
r.subrequest(`/__reversed${r.uri}`) // call the reversed url
.then((res) => {
// copy the response headersOut
Object.keys(res.headersOut).forEach((key) => {
r.headersOut[key] = res.headersOut[key];
});
// replace the end of "header" tag to append a title
const responseBuffer = res.responseBuffer
.toString()
.replace("</header>", "</header><h1>Reversed</h1>");
r.return(res.status, responseBuffer);
})
.catch((err) => {
r.log(err);
r.return(500, "Internal Server Error");
});
} else {
// in any other case
r.internalRedirect("@ghost_backend");
}
}
//...
One more time, restart http
container:
docker-compose restart http
Then visit http://example.locahost/about/ β you should see:
Second method, use js_body_filter
A probably better solution is to use js_body_filter
instead of js_content
. This leaves Nginx makes the proxy pass, then we can manipulate the body.
So, let's change the default.conf
file to this:
js_import /etc/nginx/conf.d/example.js;
upstream ghost_backend {
server ghost:2368;
}
server {
listen 80;
server_name example.localhost;
gunzip on;
# call our js for /exemple url
location = /exemple {
js_content example.information;
}
# call the "ghost blog" backend
location / {
js_body_filter example.replaceAboutPage;
subrequest_output_buffer_size 16k;
proxy_pass http://ghost_backend;
proxy_set_header Host $host;
proxy_set_header Accept-Encoding "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
}
Very important, here we force the Accept-Encoding
to be empty, because Ghost will return a gzipped content that is impossible (at this time) to decompress from javascript.
Then change the JavaScript replaceAboutPage
function to this:
async function replaceAboutPage(r, data, flags) {
if (r.uri == "/about/") {
r.sendBuffer(data.toString().replace("</header>", "</header>Reversed"), flags);
} else {
r.sendBuffer(data, flags);
}
}
-
r
is the request -
data
is the data to send, here it's the content taken from the Ghost backend -
flags
is an object with "last" flags set totrue
orfalse
The function needs to use sendBuffer()
to send the data to the client.
Of course there are many others things to do, like changing the "Content-Length" header, but this works.
It's very efficient and that's a good method to make some replacement, content checks or fixes to a response without the need to make a subrequest
.
Make a fetch
to outside
njs
provides a global ngx
variable. This object proposes the fetch
API which is more or less the same as you can find in modern browsers.
Let's add a method to call the Dev.to API.
Please, do not abuse the API, it's a simple example, and you are pleased to not overload the servers
The following function will get the list of my articles.
async function getMetal3D(r) {
ngx
.fetch("https://dev.to/api/articles?username=metal3d", {
headers: {
"User-Agent": "Nginx",
},
})
.then((res) => res.json())
.then((response) => {
const titles = response.map((article) => article.title);
r.headersOut["Content-Type"] = "application/json";
r.return(200, JSON.stringify(titles));
})
.catch((error) => {
r.log(error);
});
}
// don't forget to export functions
export default { information, replaceAboutPage, getMetal3D };
OK, now you think that you'll only need to add a js_content
call inside the default.conf
file. But... there will be some problems.
-
ngx
object need a "resolver" to know how to resolve the domain name - also, you need to define where it can find certificate authorities
But, that's not so hard to do:
location /metal3d {
resolver 9.9.9.9;
js_fetch_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
js_content example.getMetal3D;
}
So, one more time, restart the http
container and call the /metal3d
URI:
docker-compose restart http
curl http://example.localhost/metal3d
# response:
[
"Python, the usefulness of \"dataclass\"",
"Fixing a Promise return object to be typed in Typescript",
"Flask / Quart - manage module loading and splitting",
"Change local container http(s) reverse proxy to Pathwae"
]
Conclusion
Nginx proposes a very useful JavaScript extension system. It's easy to realize the large possibilities that can be made with it.
It's possible to create a custom cache system, to manipulate headers, reading token, validate authorizations, change the content, create a proxy on API, and many others things.
We only checked how to manipulate content, but it's also possible to manipulate "streams" with events.
Go to the documentation pages:
- NJS doc: https://nginx.org/en/docs/njs/
- JS object that you can use: https://nginx.org/en/docs/njs/reference.html
- Module directive:
- Examples on GitHub: https://github.com/nginx/njs-examples/
Top comments (0)