API gateway request transformation policies are incredibly powerful. There are many situations when an API developer can take advantage of request transformations to adjust the shape and values of a request to cleanly fit their API.
Let’s say you’re deprecating a certain endpoint for your API, but you still need to support the old specification for a transition period. Request transformations let you accept requests according to the old specification, transform them to fit the new specification and then forward them.
Or maybe you discovered that incorrect documentation on an endpoint’s query parameter specification has been in the wild for far too long, which is why so many users have been encountering errors. Rather than changing your API to fit bad documentation, you can use request transformations to modify requests as they come in, shaping them according to how your API is supposed to work.
Or perhaps it’s simply time to implement basic security best practices, like stripping or obfuscating sensitive data in the headers before letting them continue upstream.
For these and many other reasons, request transformations can be your go-to solution.
This article will look at how we might use Kong Gateway, coupled with the Request Transformer plugin, to configure, intercept and transform requests as they make their way to an upstream route. Setup and configuration are incredibly simple. We’ll cover several common use cases:
- Copying an API Key from the query string to the header
- Removing a query string value
- Moving an API key from a query string to the header
- Adding a version number to a query string
- Modifying the header token to Bearer Auth
- Moving JWT from header to body
- Sanitizing the body
- Changing the HTTP method
This should give you enough foundation to craft your own transformations, custom-tailored for your application situation.
Overview of Core Concepts
Before we dive in, let’s quickly cover the technologies we’ll be using for our walkthrough.
Kong Gateway
Kong Gateway is a thin-layer, front-line “gateway” sitting in front of your system’s upstream services. Whether those upstream services are API servers, web servers or any other cloud microservices, Kong Gateway handles traffic control, authentication, load balancing and more.
Request Transformer Plugin
The Request Transformer plugin for Kong Gateway comes built in. As requests come through Kong, you can configure the plugin to transform those requests—mutating headers, query string parameters, the request body and so on—before forwarding those requests to their final destination. Transformations are highly configurable and extremely powerful, giving you a lot of control over the precise shape of requests before they hit your endpoint.
Mockbin
Mockbin is an online custom endpoint generator used for testing HTTP requests and tracking responses. Mockbin is exactly what we need for our walkthrough—as we set up transformations, send requests and then inspect the result.
Our Walkthrough Approach
To start, we’ll set up our Mockbin endpoint as our “upstream service.” We only need a single endpoint, and we’ll hit it with GET and POST requests. Then, we’ll make sure Kong Gateway is installed and configured properly.
We’ll look at configuring the Request Transformer plugin for each transformation that we want to demonstrate. Then, we’ll send our request (using curl). Finally, we’ll inspect the request at Mockbin. We want to see what the request looked like when it arrived at Mockbin after passing through Kong and the Request Transformer plugin.
Are you ready? Roll out!
Want to set up a request transformation for your API gateway with clicks instead of code? Try Konnect for free >>
Set Up Mockbin Endpoint
Mockbin is a simple and easy-to-use test endpoint generator. On the website, click on the Create Bin link to get started.
And, just like that, your endpoint has been created! From here, you will need your endpoint URL. This is just https://mockbin.org/bin/{BIN-IDENTIFIER}. But, for good measure, you can right-click on Visit in Browser and copy the link address.
To test it out, you can open a new tab in your browser and visit the link you just copied. Doing so will send a GET request to your Mockbin endpoint. To see the details for this request, go back to the browser tab with your Mockbin details and click on View History. You’ll see a running log of requests to this endpoint. You should see the GET request that you just sent:
Our Mockbin endpoint is up and running and ready to go.
Set Up Kong Gateway
Spinning up Kong Gateway is quite straightforward. First, you’ll need to install Kong on your local machine. There are many different installation options, so you can choose whichever best fits your environment.
After installing Kong, create a project folder on your local machine. For simplicity, I’m creating a sub-folder called “project” in my home folder. Then, in that folder, run the command to generate a declarative configuration file.
~$ mkdir project
~$ cd project
~/project$ kong config init
~/project$ tree
.
└── kong.yml
0 directories, 1 file
For our use of the Request Transformer plugin, we can put all of our service and route setup and plugin configurations in a single file. Then, start Kong. Kong does not need to write to a database, and we don’t need to do any further on-the-fly configuration as we go along. This is Kong’s DB-less and Declarative Configuration, and it’s sufficient for what we need.
The kong.yml file is a starter template for a declarative configuration file. Let’s open it and modify it to look like the following:
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: untouched
service: mockbin
paths:
- /untouched
Let’s briefly go over what we’ve done here. The
_format_version
Next, we create a route, which we will arbitrarily name “untouched.” This route listens on the Kong Gateway for requests that go to the path /untouched, and then it forwards those requests to our upstream service called “mockbin.” In this first example, we are not adding a Request Transformer plugin. Requests will proceed through Kong Gateway untouched, continuing to Mockbin.
We’re almost ready! The last thing we need to do is tell Kong where to look for our declarative configuration file when it starts up. To do this, we need a kong.conf file, copied from the default template provided to us upon installation:
~/project$ cd /etc/kong
/etc/kong$ cp kong.conf.default kong.conf
~/project$ sudo vi kong.conf
In the kong.conf file, there are two lines that we need to edit. At around line 922, we need to tell Kong that we won’t be using a database. And at around line 1106, we need to provide the absolute path to our declarative configuration file. That’s the kong.conf file in your project folder.
# PATH: /etc/kong/kong.conf
...
# AROUND LINE 922
# inherited from the corresponding main connection config described above but
# may be optionally overwritten explicitly using the `pg_ro_*` config below.
database = off # Determines which of PostgreSQL or Cassandra
# this node will use as its datastore.
# AROUND LINE 1106
# This value is only used during
# migrations.
declarative_config = /PATH/TO/YOUR/PROJECT/FOLDER/kong.yml
# The path to the declarative configuration
# file which holds the specification of all
# entities (Routes, Services, Consumers, etc.)
With the file saved, we’re ready to start up Kong.
~/project$ sudo kong start
Now that Kong is running, let’s send a request to localhost:8000, which is the port where Kong is listening. We’ll send a GET request to the /untouched endpoint, and we’ll tack on a query string parameter:
~/project$ curl -X GET 'http://localhost:8000/untouched?hello=world'
In your browser, refresh the page with your Mockbin endpoint history log. You should see a new entry for a GET request.
As you view the Request Details, you can see that the queryString array has a pair that matches the parameters we sent to Kong. It looks like Kong successfully forwarded our (untouched) request to Mockbin!
Just for good measure, let’s also send a POST request to the same endpoint at Kong. We’ll set our “Content-Type” header and tack on a request body like so:
~/project$ curl -X POST \
-H 'Content-Type: application/json' \
-d '{"hello":"world"}' \
'http://localhost:8000/untouched'
Again, we refresh the Mockbin log, and we see our POST request. Here, we see our Request Body, which matches what we sent to Kong.
If we poke around at the Request Details, we also see the “Content-Type” value that we set in our header, along with our POST body data.
It looks like we have finished our setup. We’ve connected all the pieces. It’s time to start playing around with some common request transformations.
Common Request Transformations
1. Copy API key from query string to header
Let’s start with a common scenario. Your API users have been adding their API key as a query string parameter, but you’d like to transition them toward attaching it as a custom header value instead. Until you’re certain that all of your API users have gotten the memo that the key should be in the header, you want to accommodate the stragglers. What you need is a request transformation that copies the value from the query string parameter to a new header value.
Here is how our kong.yml file would look:
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: copy-query-to-header
service: mockbin
paths:
- /copy-q2h
plugins:
- name: request-transformer
route: copy-query-to-header
config:
add:
headers:
- api-key:$(query_params.api_key)
Similar to our first example, we’ve created a service, along with a route that forwards to this service. For this route, Kong will listen on the path /copy-q2h. Next, we add a plugin. The name of this plugin ( not arbitrary) is request-transformer. The plugin is for our specific route.
What kinds of request transformations should this plugin do? It will add a header. The new key for this header will be api-key. The value will be copied from the query string parameter called api_key. This part uses the plugin’s templating feature to grab a value that came in the query string
Don’t forget: Whenever we modify the kong.yml file, we need to restart Kong. Let’s restart Kong and send our request.
~/project$ sudo kong restart
~/project$ curl -X GET 'http://localhost:8000/copy-q2h?api_key=THISISMYAPIKEY'
We’ve sent a GET request to our /copy-q2h endpoint. We tacked on a query string parameter api_key=THISISMYAPIKEY. Let’s see how this request was transformed by refreshing the Mockbin log.
As we scroll down the Request Details, we see in the headers that there is an api-key header with the value THISISMYAPIKEY. It looks like our copy transformation worked.
You will notice that the api_key=THISISMYAPIKEY query string parameter is still in the request, though. That’s okay for now. We’ll deal with that in a little bit.
2. Remove query string value
Another common scenario involves the need to remove certain query string parameters completely. Perhaps for privacy reasons or security reasons, sanitizing query parameters is a common use case for request transformations.
In the following kong.yml example, we want to remove the key-value pairs for the api_key and password query string parameters. Similar to our previous example, we create a route (listening on /strip-q), and we create a plugin on that route that removes our undesirable parameters:
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: strip-query
service: mockbin
paths:
- /strip-q
plugins:
- name: request-transformer
route: strip-query
config:
remove:
querystring:
- api_key
- password
Let’s send a request to this path, adding on a few query parameters. We’ll include the ones we want to strip away, along with one that we would like to keep. Don’t forget to restart Kong!
~/project$ sudo kong restart
~/project$ curl -X GET \
'http://localhost:8000/strip-q?api_key=THISISMYAPIKEY&username=johndoe&password=THISISMYPASSWORD'
Taking a glance at the Mockbin log, we can see that this most recent GET request received the query string parameter we wanted to keep (username=johndoe) but didn’t receive the undesirable ones. That’s a good sign.
A closer look at the query string array in Request Details shows that the plugin worked as expected, removing the api_key and password parameters before forwarding the request to Mockbin.
Just a note: You could write kong.yml with multiple routes and multiple plugins. In this article, our file always shows just a single route and a single plugin. That’s just to keep it simple, but you’re not restricted to do it this way.
3. Move API key from query string to header
Previously, we copied an API key from the query parameter to a header value, but we left the query parameter in the request. In the following example, let’s do something similar, but we’ll clean up after ourselves by removing the query parameter too. Our configuration for the plugin this time around: 1) adds a header based on the query parameter value, and then 2) removes the query parameter.
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: move-query-to-header
service: mockbin
paths:
- /move-q2h
plugins:
- name: request-transformer
route: move-query-to-header
config:
add:
headers:
- api-key:$(query_params.api_key)
remove:
querystring:
- api_key
At this point, the keen observer might be asking: “Wait, if you remove the query parameter, how would you copy its value over to the header? Does the configuration order matter, where you have to copy it before you remove it?” The answer is… no! Kong’s documentation on template values explains it this way:
Note: The plugin creates a non-mutable table of request headers, query strings, and captured URIs before the transformation. Therefore, any update or removal of params used in a template does not affect the rendered value of a template.
Let’s restart Kong and send our request.
~/project$ sudo kong restart
~/project$ curl -X GET 'http://localhost:8000/move-q2h?api_key=THISISMYAPIKEY'
An inspection of the Request Details at the Mockbin log shows the value in our headers:
4. Add version number to query string
Another common case is simply adding a query string parameter to a request. In this example, we’ll add a fixed API version number to all requests that come through our /add-q route:
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: add-to-query
service: mockbin
paths:
- /add-q
plugins:
- name: request-transformer
route: add-to-query
config:
add:
querystring:
- api_version:2.0
We will add the parameter called api_version, and its value will be 2.0. Let’s restart Kong and send our request.
~/project$ sudo kong restart
~/project$ curl -X GET 'http://localhost:8000/add-q?username=johndoe'
While we only set one query parameter (username=johndoe), we inspect our transformed request at the Mockbin log. There we see in the Request Details that Mockbin received two parameters with the request:
It looks like our query parameter add was successful.
5. Modify header token to Bearer Auth
Now, let’s experiment with our headers a bit. Imagine the scenario where your API users are attaching their authorization token as a header called token, but you wrote your API to use the Bearer Auth scheme. The token value should be in a header called Authorization, following the word “Bearer!” Collin, the junior dev who wrote that erroneous API documentation, will get a poor performance review. In the meantime, you’ll use request transformation to fix Collin’s mess.
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: modify-header
service: mockbin
paths:
- /token-to-auth
plugins:
- name: request-transformer
route: modify-header
config:
add:
headers:
- Authorization:Bearer $(headers["token"])
remove:
headers:
- token
Our configuration again takes advantage of template values, copying the token value from the token header and structuring a proper Bearer Auth header value. And, of course, we remove the undesirable token header after we get what we need.
We restart Kong and send our request.
~/project$ sudo kong restart
~/project$ curl -X GET \
-H 'token:thisisanopaquestringthatrepresentsajwt' \
'http://localhost:8000/token-to-auth'
For this request, we sent our token value in the header called
token. Let’s inspect the Request Details at our Mockbin log:
We see the Bearer Auth scheme in our header now. A further inspection of the details shows that the offending token header is not present. On a side note, you might notice above that we had configured our plugin to add a header with the name “Authorization” (capitalized), but what shows up in Mockbin is “authorized” (lowercase). It looks like this might be an issue with Mockbin—which lowercases all of its header names—and not an issue with the Request Transformer plugin (which addressed this specific issue in a pull request).
6. Move JWT from header (Bearer Auth) to body
Let’s do some request transformations that modify the request body. In this example, we’ll take the token in Bearer Auth format, and we’ll write it to our request JSON body as a value. While some upstream services look in the header for an authorization token, others might look to the request body. Request transformations provide flexibility, especially when modifying upstream services isn’t an option.
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: jwt-to-body
service: mockbin
paths:
- /jwt2b
plugins:
- name: request-transformer
route: jwt-to-body
config:
add:
body:
- jwt:$(headers["Authorization"]:gsub("^Bearer ",""))
The syntax for the above plugin configuration is a bit more complicated, but you can probably discern what’s happening. We want to add a key-value pair to our request body. The key is jwt, and the value will be taken with the Authorization header but with the initial “Bearer” (plus space) removed.
The plugin documentation notes that the placeholder is evaluated as Lua expression. So, if you know Lua, you can probably do some pretty powerful transformations.
Let’s restart Kong and send our POST request, along with headers and body data:
~/project$ sudo kong restart
~/project$ curl -X POST \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer thisisanopaquestringthatrepresentsajwt' \
-d '{"username":"johndoe" }' \
'http://localhost:8000/jwt2b'
When we look at our Mockbin log for this request, we immediately see the Request Body. It contains the original body we sent ("username":"johndoe"), but it also contains
jwt, with the token value that was in our Bearer Auth header.
You might notice, from our plugin configuration, that we decided to leave our Bearer Auth header in the request:
7. Sanitize body
Perhaps for sensitive data that might come through the request body, you will need a request transformation that sanitizes it but doesn’t remove it. This helps your upstream service know that the data did come through, but it’s just no longer available. For this type of transformation, your configuration might look like this:
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: sanitize-body
service: mockbin
paths:
- /san-b
plugins:
- name: request-transformer
route: sanitize-body
config:
replace:
body:
- last4: ****
- api_key: ********
- password: ********
In this request transformation, we’re looking in our JSON body for last4, api_key or password. For any of those keys, we want to replace the provided value with a redacted one. After restarting Kong, we send our request:
~/project$ sudo kong restart
~/project$ curl -X POST \
-H 'Content-Type: application/json' \
-d '{"api_key":"THISISMYAPIKEY", "username":"johndoe", "password":"THISISMYPASSWORD", "last4":9876}' \
'http://localhost:8000/san-b'
A quick glance at the Mockbin log shows our transformed Request Body. Recall that Kong transformed this body before it was sent to the upstream service, so we can be sure that none of this sensitive data got past our gateway.
8. Change HTTP method
For our final example, let’s demonstrate a request method transformation. Let’s assume that PUT requests to a given route need to be converted to POST requests, and instead, we’d add an "action":"PUT" key-value pair into the JSON request body.
In this configuration, we configure our route only to listen for PUT requests. Then, we configure the plugin to transform the http_method to POST. Lastly, we add a key-value pair to our body.
# ~/project/kong.yml
_format_version: "2.1"
services:
- name: mockbin
url: https://mockbin.org/bin/REPLACE-WITH-YOUR-BIN-IDENTIFIER
routes:
- name: put-to-post
service: mockbin
methods:
- PUT
paths:
- /put-to-post
plugins:
- name: request-transformer
route: put-to-post
config:
http_method: POST
add:
body:
- action:PUT
Restart Kong and send the PUT request.
~/project$ sudo kong restart
~/project$ curl -X PUT \
-H 'Content-Type: application/json' \
-d '{"username":"johndoe" }' \
'http://localhost:8000/put-to-post'
We see the successful result in our Mockbin log. The most recent request was a POST (not PUT) request. And, we see the action key-value pair added to our Request Body.
Conclusion
We’ve covered a lot of examples in this walkthrough. We’ve transformed query string parameters, headers, bodies and even request methods. The Request Transformer plugin offers several ways to transform parts of the request. While we’ve covered many of them, combining multiple transformations can yield powerful results that uniquely shape a request to fit your API.
We didn’t cover URI transformations, which allow you to transform the upstream request URI based on the incoming request. You can imagine taking a long REST-compliant URI, chock-full of a chain of resources, ids, sub-resources and more ids. Using URI capturing and template values, the Request Transformer plugin can capture all of that data in the URI and write it into the request body instead, sending the request upstream to a general all-purpose endpoint.
Ready for Advanced Request Transformations?
The Kong Konnect enterprise-level Request Transformer Advanced plugin provides even more targeted transformations with regular expression matching, variables and substitutions.
All in all, the ability to transform your requests at the gateway level before they hit your upstream service is critical. Whether it’s because of deprecated specs, documentation-reality misalignment or data privacy and security concerns, having the flexibility to shape incoming requests—and to do so quickly and simply—might be exactly what your DevOps team needs to save the day in a pinch.
Once you’ve successfully set up API gateway request transformation policies, you may find these other tutorials helpful:
- How to Use the Kong Gateway JWT Plugin for Service Authentication
- 4 Steps to Authorizing Services With the Kong Gateway OAuth2 Plugin
- Getting Started With Kuma Service Mesh
The post 8 Common API Gateway Request Transformation Policies appeared first on KongHQ.
Top comments (0)