The Marketing Team Has a Request

TLDR - Learn how to set up a reverse proxy using Cloudflare Workers to integrate your Webflow & flask servers

Your marketing team has decided they need a platform that will allow them to update the website themselves.

Sounds great - one less engineering workstream for you to juggle against the rest of the roadmap. But what's the start up cost? How much will it take to stand that up?

Having just wrapped up this process here at Block Party, I’ll share some reflections on the tradeoffs we considered, challenges we faced, and even some code samples to accelerate your efforts if you choose to go in this direction.

Do we really need this?

We built the initial version of our web application all-in-one, with both static and interactive pages hosted on the same web server. The rule of thumb was to keep things simple to start, and to optimize only as needed.

And as our company has matured, our needs have indeed changed: in particular, our growth & marketing teams now want - and need - to be able to iterate more quickly on design and copy, as well as spin up new landing pages to test concepts and target specific audiences.

While these changes are relatively simple since they only involve static pages, they still require engineering time, slowing down the growth & marketing teams and adding to the strain on our limited engineering resources.

Fortunately, there are many web page builders out there, so it is quite straightforward to empower our colleagues to make their own changes and move quickly on growth & marketing experiments, updates, and campaigns.

At Block Party, we decided to fill this need by setting up a Webflow server. In the following sections, we'll discuss the additional work we did to integrate Webflow with the rest of our website.

The technical task at hand

Our website lives at www.blockpartyapp.com, and both marketing and application pages are served off of this domain.

This is all currently handled by the same server, built in Python using flask. What we want to do going forward is to route requests for our marketing pages to the new Webflow server.

Specifically, we need to build a reverse proxy that does the following:

1. examines each incoming request's URL,
2. sends a subrequest to the appropriate server based on the incoming URL, and
3. returns the subrequest's response to the client.

Using Cloudflare Workers for routing

A common solution for this problem is to use something like NGINX, but that would introduce new infrastructure for us to deploy and maintain. Since we use Cloudflare for DNS already, it makes sense for us to implement this reverse proxy with Cloudflare Workers.

Cloudflare Workers allow you to run a Javascript or Typescript script in front of your web servers. The script will run every time a request is submitted to the hostname at which the worker is registered, allowing you to decide how those requests get handled.

Conveniently, scripts written for Cloudflare Workers can use all APIs accessible to Web Workers. This makes it easy to implement the reverse proxy's key functionality without installing any additional libraries. In particular, we'll use the following APIs:

• URLPattern API - allows us to create and execute URL-matching patterns
• Fetch API - allows us to execute subrequests

We can tie these together by creating objects with the following two methods:

• isMatch: (string) => bool and
• makeResponse: (Request) => Promise<Response>.

We call these objects RouteRule's.

When the reverse proxy receives a request, we can iterate through an ordered array of RouteRule's, checking isMatch for each rule and running makeResponse for the first rule for which isMatch evaluates to true.

This setup gives us most of what we need to get our Webflow pages up and running. Here's a simplified example of our reverse proxy code, using https://www.blockpartyapp.com/about-us/ as an example of a marketing page that we want to serve from Webflow:

// Define an interface describing the RouteRule object
interface RouteRule {
    isMatch: (string) => bool
    makeResponse: (Request) => Promise<Response>
}

// Create a rule for the /about-us/ page
const AboutUsRule = {

    // Match this rule for all incoming requests to /about-us/
    isMatch: href => {
        const pat = new URLPattern({ path: '/about-us/' })
        return pat.test(href)
    },

    // When we get a match, send a subrequest to Webflow
    makeResponse: request => {
        const subrequestUrl = new URL(request.url)
        subrequestUrl.host = WEBFLOW_HOST

        // Provide the original request object to pass on headers, etc.
        return fetch(subrequestUrl.href, request)
    },
}

// Create a collection of rules (just one for this example)
const routeRules: RouteRule[] = [
    AboutUsRule,
]

// Cloudflare Workers require your script to export a "fetch" method
export default {
    async fetch(request) {
        let response;

        // Check the condition for each rule, 
        // exiting the loop after we find a match
        for (let i = 0; i < routeRules.length; i++) {
            const rule = routeRules[i]
            if (rule.isMatch(request.url)) {
                response = rule.makeReponse(request)
                break
            }
         }

         // If there is no match, default to sending a
         // subrequest to the flask server
         if (response === undefined) {
             const subrequestUrl = new URL(request.url)
             subrequestUrl.host = FLASK_HOST

             response = fetch(subrequestUrl.href, request)
         }
         return response
     }
}

Handling authentication

In addition to marketing pages like our /about-us/ page, we also want to host our landing page on Webflow.

Currently, our home page (https://www.blockpartyapp.com/) shows logged-in users their application dashboard while unauthed users see the standard landing page. Deciding which page to show is easy to do when everything is on one server: we can just check to see if the user is logged in, and return the correct view. It is more complicated for the reverse proxy, as it isn't as easy for it to determine if the user is logged in or not.

We can accomplish this by taking advantage of how authentication works on our flask server. In particular, the server uses a cookie to determine if a user is logged in. By forwarding the cookie to our flask server and examining the server's response, we can determine if our user is logged in and therefore how to handle the request:

const LandingPageRule = {

    isMatch: href => {
        const pat = new URLPattern({ path: "/" })
        return pat.test(href)
    },

    makeResponse: request => {
        // First, we send a request to the flask server,
        // passing on the cookies from the original request.
        const loginSubrequestResponse = await fetch(
            'some-endpoint-on-the-flask-server', 
            request,
        )

        // By examining the response, we can determine if the user is logged in.
        // The exact way this works will depend on how your server does authentication.
        const isLoggedIn = ....

        // We can now create the appropriate subrequest
        const subrequestUrl = new URL(request.url)
        if (isLoggedIn) {
            subrequestUrl.host = FLASK_HOST
        } else {
            subrequestUrl.host = WEBFLOW_HOST
        }
        return fetch(subrequestUrl.href, request)
    }

We can then include this rule in our list of route rules:

...

const routeRules: RouteRule[] = [
    AboutUsRule,
    LandingPageRule,
]

...

And that's it! These two rules handle most of the behavior we need our reverse proxy to be able to perform, and provide a pattern for how we can add future rules (e.g., to handle forwarding for blog posts with paths like /blog/some-post-name).

Potential corner cases

However, for us, that turned out not to be the whole story. After initial development was complete, we deployed our reverse proxy to our staging environment and quickly found some unintended behavior:

• https://flask-host/about-us/ returned the expected content, but only after redirecting us to https://webflow-host/about-us
• https://flask-host/about-us returned a 404 instead of the expected content

We had expected requests to both of those URLs to return the correct content without altering the requested URLs; we also had not expected including or excluding a trailing slash to have made a difference.

Experimenting a bit, we discovered that on our main application server, all our endpoints were defined with a trailing slash, and flask automatically redirected paths without the trailing slash. On the Webflow server, the opposite was true - all endpoints were defined without trailing slashes, and paths including trailing slashes were automatically redirected.

The RouteRule's we had defined only included the /about-us/ path (nothing for /about-us) and issued a subrequest to https://webflow-host/about-us/ - which, as we now understood, returned a redirect that made the browser issue a second request to the Webflow domain. Since we had not defined a rule for /about-us, that request was forwarded directly to our application server, which no longer had an About Us page, and therefore returned a 404.

While these redirects were unexpected, the fix was fortunately easy. We were able to handle this with two changes:

1. Create a new rule that matches on paths without a trailing slash & issues redirects to include the slash.
2. Remove the trailing slash when forwarding requests to Webflow.

This looked as follows:

...

// New rule to redirect requests to /about-us => /about-us/
const AboutUsTrailingSlashRule = {

    isMatch: href => {
        const pat = new URLPattern({ path: '/about-us' })
        return pat.test(href)
    },

    makeResponse: request => {
        const redirectUrl = new URL(request.url)
        redirectUrl.pathname = '/about-us/'

        const response = Response.redirect(redirectUrl.href, 301)
        return new Promise(resolve => resolve(response))
    },

}

// Modify the existing rule to forward /about-us/ => /about-us
const AboutUsRule = {

    isMatch: href => {
        const pat = new URLPattern({ path: '/about-us/' })
        return pat.test(href)
    },

    makeResponse: request => {
        const subrequestUrl = new URL(request.url)
        subrequestUrl.host = WEBFLOW_HOST
        subrequestUrl.pathname = '/about-us'

        return fetch(subrequestUrl, request)
    },

}

...

// Add the new rule to the route rules
const routeRules: RouteRule[] = [
    AboutUsTrailingSlashRule,
    AboutUsRule,
    LandingPageRule,
]

...

Depending on how your main application server was built, you may find similar corner cases specific to how your server handles certain requests.

Handling HTTP requests

Another corner case you may want to be aware of is that your server needs to handle some HTTP requests.

We initially set up our reverse proxy to reject all HTTP requests, figuring that that would make our servers more secure. However, a few months in, we had a serious incident - an SSL certificate had failed to renew, taking down our website. While we were able to find a mitigation & get the site running again within a couple hours, we were unable to identify the root cause until several weeks later.

It turns out that our certificate was set up to renew through a HTTP-01 Challenge. In a HTTP-01 Challenge, a certificate validation service sends a request to http://yourDomain/.well-known/acme-challenge/some-long-token. The challenge is successful if the server responds with the correct answer, verifying its identity. Since we had set up our server to reject all HTTP requests, our server failed the challenge, and therefore our certificate failed to renew.

This is fortunately easy to accommodate once you know about it. We only need to update the reverse proxy to include two additional rules - passing HTTP-01 Challenge requests forward to the flask server and redirecting all HTTP requests (instead of rejecting them):

// Forward HTTP-01 Challenges to the flask server
const HttpChallenge01PassThroughRule = {

    isMatch: href => {
        const pat = new URLPattern({ protocol: 'http', path: '/.well-known/acme-challenge/*' })
        return pat.test(href)
    },

    makeResponse: request => {
        const subrequestUrl = new URL(request.url)
        subrequestUrl.host = FLASK_HOST

        return fetch(subrequestUrl.href, request)
    },

}

// Redirect all HTTP requests to HTTPS
const HttpRedirectRule = {

    isMatch: href => {
        const pat = new URLPattern({ protocol: 'http' })
        return pat.test(href)
    },

    makeResponse: request => {
        const redirectUrl = new URL(request.url)
        redirectUrl.protocol = 'https'

        const response = Response.redirect(redirectUrl.href, 301)
        return new Promise(resolve => resolve(response))
    },

}

...

const routeRules: RouteRule[] = [
    HttpChallenge01PassThroughRule,
    HttpRedirectRule,
    AboutUsTrailingSlashRule,
    AboutUsRule,
    LandingPageRule,
]

...

Lessons learned

Hopefully, the failed HTTP-01 Challenge will be the last of the issues we experience with our reverse proxy. In the meantime, I appreciate that even for a piece of technology as common as a reverse proxy there are many small details to learn and appreciate; I came away from this project with more knowledge and intuition for how web servers work and a reminder to appreciate how complex our software systems are. (And our marketing team is delighted that they can create, publish, and update pages on their own, too!)