Setting Up a Vercel "Log Drain" with AWS

If you're hosting apps on Vercel, you might have noticed that they have a number of quality logging integrations in their Marketplace. But if those don't suit your needs, or if you're already using AWS services (especially Lambda and CloudWatch) and want to keep things lean and simple, this article is for you!

This might seem a bit involved, but the code is simple and it really only takes about 15 minutes! 😎
Also, searching API logs from Vercel in CloudWatch is a breeze! 🏄🏻 I'll tell you more about that at the end of the article.

1. Create a Lambda to process your logs

In your AWS console, go to Services -> Lambda -> Functions, and click "Create Function". Name your function as appropriate, but leave everything else in this first section at the defaults.

Under "Advanced Settings", you need to select "Enable function URL", choose "NONE" for "Auth type" (we'll be handling authentication ourselves), and check the CORS button.

Now click "Create Function". Once you're redirected to your new lambda, find "Function URL" in the upper right under "Function Overview", and copy it to your clipboard. You'll need this for the next step.

⛔️ Don't close this tab! You've got more work to do here very soon.

2. Create a new Log Drain in Vercel

In a different browser tab, go to your Vercel dashboard, then go to Settings (this is settings for your whole team in Vercel), then click "Log Drains". Here you'll see the "Add a Log Drain" form.

1. Choose your desired sources. (I just wanted to log Next.js API requests, so I only chose "Lambda").
2. For delivery format, we will use "NDJSON" format which is easy for us to process in our logging lambda. (More on that later.)
3. Choose "all projects" or specific projects as you wish. Note that you can easily filter logs by project later from the CloudWatch console. (See end of article for details.)
4. Under "Endpoint" paste your lambda function URL. (You'll now see a new panel with a "Verify" button. ✋ Don't do this yet!)
5. Click the toggle for "Custom Headers". We will be authenticating requests to our logging lambda via an "authentication" header.
6. Enter "authentication" in the Header Key. Then create a token (using something like the LastPass generator), and paste it into "Header Value", prepending "Bearer ". (This is an authentication header convention.) Now click "Add" to add the header to the Log Drain.

⛔️ Don't close this tab!

3. Verify your logging URL

Vercel requires that you verify your ownership of your logging URL, and to do this, you'll need to modify your lambda to return a header that Vercel specifies.

Go back to your lambda console in AWS, and replace index.mjs with this:

export const handler = async(event) => {
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
        headers: {
            'x-vercel-verify': '<vercel-verification-string>'
        },
    };
    return response;
};

🟢 Be sure to click "Deploy" to deploy your changes to your live lambda.

Now go back to Vercel and click "Verify". The verification panel should now go away and you'll see a blue check mark next to your URL.

🙌 We're almost ready to "Add Log Drain", but not quite yet!

4. Complete your lambda, adding authentication and logging functions

Back in your lambda console tab, you'll need to add the authentication token you generated earlier to your lambda's environment variables. Go to "Configuration -> Environment Variables" then click "Edit". Add your token with the key "AUTH_TOKEN" and click "Save".

Now, replace your function's index.mjs with the following:

/* global atob */

export const handler = async(event, context) => {
    // Return unauthorized if missing auth header
    if (!event.headers.authentication || event.headers.authentication !== `Bearer ${process.env.AUTH_TOKEN}`) {
        console.log('🔴 Unauthorized request');
        return {
            statusCode: 401,
        };
    }
    /**
     * We convert the request body from Base64,
     * then split the resulting string by new lines.
     * (The NDJSON format sends line-delimited JSON logs.)
     */
    const logs = atob(event.body).split('\n');

    // We create a log entry for each item received.
    logs.forEach((log) => {
        /**
         * The split function will leave an empty item
         * at the end of the array. We don't want to log this.
         */
        if (log !== ''){
          console.log(log);
        }
    });

    return {
        statusCode: 200,
    };
};

🟢 Again, be sure to click "Deploy" to deploy your changes to your live lambda.

Your lambda is now complete! 🕺🏻

FINAL STEP: Activate the Log Drain in Vercel

Finally, go back to your Vercel tab and click the "Add Log Drain" button.

Done! 🎉

(Note that Vercel will show a dialog with information about securing your logging lambda, but we've already handled this, so you can ignore these instructions unless you prefer the authentication method they suggest.)

---

🚨 Bonus: Working with API logs in CloudWatch and Logs Insights

To see the logs you've ingested from Vercel, go to your lambda's console, click "Monitor" then click "View CloudWatch logs". To view all logs, click "Search all log streams", then apply filters as needed to find the logs you're looking for.

The way we've set up our logging, with one entry per log sent by Vercel, logs can be filtered as though they are JSON objects using CloudWatch filter syntax. For example, to view all GET requests, add this to the search/filter input:
{$.proxy.method = "GET"}

Or to filter by project, use:
{$.projectName = "my-nextjs-project"}

In Log Insights, you can create some valuable visualizations, like how many requests missed the cache per hour:

fields @message, @timestamp
|  filter(proxy.vercelCache = "MISS")
|  stats count() by bin(1h)

...or stats for requests for specific project grouped by request method:

fields @message, @timestamp
|  filter(projectName = "my-nextjs-project")
|  stats count() by proxy.method

I hope this helps!

Comments

[Gustavo Adolfo Mejía Sánchez]

Awesome explanation, It's what I need. Just a little feedback

There is an error in the code. When you set the custom header you named it as authentication, but in the code you are asking for event.headers.authorization

Again Thanks for the excellent post :D

[Mannie Schumpert]

Thanks for the heads up!

[Charanjit Singh]

Thank you for the post. If you want to just copy paste and not do to and fro tab switching and checking if everything is set up correctly. You can use following script:

/* global atob */


export const handler = async (event, context) => {

  // Check if authorization header is present

  if (!event.headers.authentication || event.headers.authentication !== `Bearer ${process.env.AUTH_TOKEN}`) {
    console.log('🔴 Unauthorized request');
    return {
      statusCode: 401,
    };
  }


  if (event.body === undefined) {
    // No body i.e. check request

    return {
      statusCode: 200,
      headers: {
        'x-vercel-verify': process.env.INTEGRATION_SECRET
      }
    };
  }

  /**
   * We convert the request body from Base64,
   * then split the resulting string by new lines.
   * (The NDJSON format sends line-delimited JSON logs.)
   */
  try{

    const logs = atob(event.body).split('\n');

    // We create a log entry for each item received.
    logs.forEach((log) => {
      /**
       * The split function will leave an empty item
       * at the end of the array. We don't want to log this.
      */
     if (log !== '') {
       console.log(log);
      }
    });
  } catch (e) {
    console.log('🔴 Error decoding body', e);
    // Continue to return 200 to avoid retries
  }

  return {
    statusCode: 200,
    headers: {
      'x-vercel-verify': process.env.INTEGRATION_SECRET
    }
  };
};

and add INTEGRATION_SECRET and AUTH_TOKEN in Lambda env variables

INTEGRATION_SECRET is provided by vercel and AUTH_TOKEN is your random password from lastpass

[Sebastian Goos]

Thanks for sharing this!

After setting up based on the tutorial we receive an error on the final setup "Add Log Drain".

Cannot validate endpoint url: either the url is not reachable or the x-vercel-verify header in the response did not contain the expected value

The verification in step #3 just works fine and we can still see the blue verified tick/checkmark.

Any hint of what we are doing wrong?