Setting Up a Vercel "Log Drain" with AWS

Mannie Schumpert — Mon, 23 Jan 2023 13:46:58 +0000

If you're hosting apps on Vercel, you might have noticed that they have a number of quality logging integrations in their Marketplace. But if those don't suit your needs, or if you're already using AWS services (especially Lambda and CloudWatch) and want to keep things lean and simple, this article is for you!

This might seem a bit involved, but the code is simple and it really only takes about 15 minutes! 😎
Also, searching API logs from Vercel in CloudWatch is a breeze! 🏄🏻 I'll tell you more about that at the end of the article.

1. Create a Lambda to process your logs

In your AWS console, go to Services -> Lambda -> Functions, and click "Create Function". Name your function as appropriate, but leave everything else in this first section at the defaults.

Under "Advanced Settings", you need to select "Enable function URL", choose "NONE" for "Auth type" (we'll be handling authentication ourselves), and check the CORS button.

Now click "Create Function". Once you're redirected to your new lambda, find "Function URL" in the upper right under "Function Overview", and copy it to your clipboard. You'll need this for the next step.

⛔️ Don't close this tab! You've got more work to do here very soon.

2. Create a new Log Drain in Vercel

In a different browser tab, go to your Vercel dashboard, then go to Settings (this is settings for your whole team in Vercel), then click "Log Drains". Here you'll see the "Add a Log Drain" form.

Choose your desired sources. (I just wanted to log Next.js API requests, so I only chose "Lambda").
For delivery format, we will use "NDJSON" format which is easy for us to process in our logging lambda. (More on that later.)
Choose "all projects" or specific projects as you wish. Note that you can easily filter logs by project later from the CloudWatch console. (See end of article for details.)
Under "Endpoint" paste your lambda function URL. (You'll now see a new panel with a "Verify" button. ✋ Don't do this yet!)
Click the toggle for "Custom Headers". We will be authenticating requests to our logging lambda via an "authentication" header.
Enter "authentication" in the Header Key. Then create a token (using something like the LastPass generator), and paste it into "Header Value", prepending "Bearer ". (This is an authentication header convention.) Now click "Add" to add the header to the Log Drain.

⛔️ Don't close this tab!

3. Verify your logging URL

Vercel requires that you verify your ownership of your logging URL, and to do this, you'll need to modify your lambda to return a header that Vercel specifies.

Go back to your lambda console in AWS, and replace index.mjs with this:

export const handler = async(event) => {
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
        headers: {
            'x-vercel-verify': '<vercel-verification-string>'
        },
    };
    return response;
};

🟢 Be sure to click "Deploy" to deploy your changes to your live lambda.

Now go back to Vercel and click "Verify". The verification panel should now go away and you'll see a blue check mark next to your URL.

🙌 We're almost ready to "Add Log Drain", but not quite yet!

4. Complete your lambda, adding authentication and logging functions

Back in your lambda console tab, you'll need to add the authentication token you generated earlier to your lambda's environment variables. Go to "Configuration -> Environment Variables" then click "Edit". Add your token with the key "AUTH_TOKEN" and click "Save".

Now, replace your function's index.mjs with the following:

/* global atob */

export const handler = async(event, context) => {
    // Return unauthorized if missing auth header
    if (!event.headers.authentication || event.headers.authentication !== `Bearer ${process.env.AUTH_TOKEN}`) {
        console.log('🔴 Unauthorized request');
        return {
            statusCode: 401,
        };
    }
    /**
     * We convert the request body from Base64,
     * then split the resulting string by new lines.
     * (The NDJSON format sends line-delimited JSON logs.)
     */
    const logs = atob(event.body).split('\n');

    // We create a log entry for each item received.
    logs.forEach((log) => {
        /**
         * The split function will leave an empty item
         * at the end of the array. We don't want to log this.
         */
        if (log !== ''){
          console.log(log);
        }
    });

    return {
        statusCode: 200,
    };
};

🟢 Again, be sure to click "Deploy" to deploy your changes to your live lambda.

Your lambda is now complete! 🕺🏻

FINAL STEP: Activate the Log Drain in Vercel

Finally, go back to your Vercel tab and click the "Add Log Drain" button.

Done! 🎉

(Note that Vercel will show a dialog with information about securing your logging lambda, but we've already handled this, so you can ignore these instructions unless you prefer the authentication method they suggest.)

🚨 Bonus: Working with API logs in CloudWatch and Logs Insights

To see the logs you've ingested from Vercel, go to your lambda's console, click "Monitor" then click "View CloudWatch logs". To view all logs, click "Search all log streams", then apply filters as needed to find the logs you're looking for.

The way we've set up our logging, with one entry per log sent by Vercel, logs can be filtered as though they are JSON objects using CloudWatch filter syntax. For example, to view all GET requests, add this to the search/filter input:
{$.proxy.method = "GET"}

Or to filter by project, use:
{$.projectName = "my-nextjs-project"}

In Log Insights, you can create some valuable visualizations, like how many requests missed the cache per hour:

fields @message, @timestamp
|  filter(proxy.vercelCache = "MISS")
|  stats count() by bin(1h)

...or stats for requests for specific project grouped by request method:

fields @message, @timestamp
|  filter(projectName = "my-nextjs-project")
|  stats count() by proxy.method

I hope this helps!