Integration Testing Vercel Serverless Functions with OpenTelemetry

Today you’ll learn how to create production-ready serverless functions using Vercel. It will include adding distributed tracing with OpenTelemetry for troubleshooting and integration testing with Tracetest for trace-based testing.

Once you’re done reading you’ll have a complete boilerplate project to use for your own Vercel serverless functions interacting with Vercel Postgres as storage. More importantly I’ll explain how to locally test your serverless functions in a typical development lifecycle and create integration tests for CI pipelines packaged in Docker.

If you can’t wait until the end, clone the example from GitHub, get your API keys and tokens after signing up at [app.tracetest.io](https://app.tracetest.io), and create a Vercel Postgres instance. Add the Vercel Postgres env vars to your .env files and spin up the tests with Docker. Make sure to have Docker and Docker Compose installed for the quick start!

git clone [e-mail]:kubeshop/tracetest.git
cd tracetest/examples/integration-testing-vercel-functions
docker compose up -d --build
docker compose run integration-tests

Why Serverless Functions?

Serverless functions are server-side code run on cloud servers without the need to be deployed to a server. They are instead executed in cloud environments and eliminate traditional infrastructure needs.

Platforms like Vercel that host serverless functions and front-end code offer developers scalability and flexibility with no infrastructure overhead.

Observability for Serverless Architecture

Serverless Architecture often struggles with visibility. However, observability tools can help. They trace events from start to finish, collect metrics, and evaluate how systems manage these events.

OpenTelemetry, an open-source observability framework, is one such tool. It helps gather, process, and export data like traces, metrics, and logs. Traces are especially useful as they provide insights into how distributed systems perform by tracing requests across various services.

Luckily for us, Vercel has built-in support for OpenTelemetry! 🔥

Testing Serverless Architecture

How does this tie into testing? Running tests against serverless can be quite tiresome since you can only run black box tests.

But, what if you can use distributed tracing for testing as well? Now you can. Using OpenTelemetry and Tracetest in your development lifecycle enables both local testing and integration testing in your CI pipelines. Let’s jump in and you’ll learn exactly how to do it yourself!

Vercel Function Architecture

Today you’ll build a Vercel function that imports a Pokemon into a Pokedex!

The function will fetch data from an external API, transform the data and insert it into a Vercel Postgres database. This particular flow has two failure points that are difficult to test.

1. Validating that an external API request from a Vercel function is successful.
2. Validating that a Postgres insert request is successful.

Install Vercel and Create a Boilerplate

Deployment and development is managed with the Vercel Command Line Interface (CLI), and I'll be using Next.js to simplify the setup and development of serverless functions.

The initial step involves creating a Vercel account, installing the CLI, and authenticating through the CLI.

npm i -g vercel@latest
vercel login

Next create the Next.js boilerplate.

vercel init nextjs

Rename the nextjs directory to integration-testing-vercel-functions. Now you’re ready to deploy it to Vercel.

vercel deploy

This will create a project in the Vercel Dashboard.

Create a Vercel Postgres Database

Go to the Storage tab in your Vercel account and create a Postgres database. Give it a name. I’ll use pokedex since I’ll show how to catch some Pokemon!

Use psql to connect to the database.

psql "postgres://default:************@ep-morning-wood-76425109.us-east-1.postgres.vercel-storage.com:5432/verceldb

And, create a table.

CREATE TABLE IF NOT EXISTS pokemon (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  "createdAt" TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

Proceed to connect the database to your project and pull the environment variables to your local development.

vercel link
vercel env pull .env.development.local

This will create a .env.development.local file in your local directory that contains all the Postgres connection details.

# Vercel Postgres
POSTGRES_DATABASE="**********"
POSTGRES_HOST="**********"
POSTGRES_PASSWORD="**********"
POSTGRES_PRISMA_URL="**********"
POSTGRES_URL="**********"
POSTGRES_URL_NON_POOLING="**********"
POSTGRES_USER="**********"

# ...

Finally install the Vercel Postgres SDK to interact with the database via code

npm install @vercel/postgres

You’re ready to start building!

Create a Serverless Function

You’ll create an import flow. Send the ID of a Pokemon to the serverless function, it handles getting the Pokemon info from an external API and stores it in the pokedex database.

In the root directory create a /pages directory with an /api directory inside of it, and create pokemon.ts there.

Your structure should look like this.

/pages
  /api
    /pokemon.ts

Every file within the /api directory maps to a specific API route. The function you’ll create will be accessible at the URL [http://localhost:3000/api/pokemon](http://localhost:3000/api/pokemon).

Each function takes a request as input and is expected to return a response. Failing to return a response will result in a timeout.

To return JSON data, you’ll use the res.status(200).json({...}) method. Async/Await flows are enabled by default as well!

Here’s an example of a POST request with a GET request to an external API from within the serverless function and then inserting the data into Postgres. It’s a common point-of-failure that is hard to troubleshoot and test.

import type { NextApiRequest, NextApiResponse } from 'next'
import { sql } from '@vercel/postgres'

export async function addPokemon(pokemon: any) {
  return await sql`
    INSERT INTO pokemon (name)
    VALUES (${pokemon.name})
    RETURNING *;
  `
}

export async function getPokemon(pokemon: any) {
  return await sql`
    SELECT * FROM pokemon where id=${pokemon.id};
  `
}

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  try {
    const requestUrl = `https://pokeapi.co/api/v2/pokemon/${req.body.id || '6'}`
    const response = await fetch(requestUrl)
    const resPokemon = await response.json()

    const { rowCount, rows: [addedPokemon, ...addedPokemonRest] } = await addPokemon(resPokemon) 
    res.status(200).json(addedPokemon)

  } catch (err) {
    res.status(500).json({ error: 'failed to load data' })
  }
}

Go ahead and run the function.

npm run dev

Navigate to http://localhost:3000/api/pokemon to see the response from your serverless function.

{"id":13,"name":"charizard","createdAt":"2024-01-07T13:56:12.379Z"}

Configure Troubleshooting with OpenTelemetry and Distributed Tracing

OpenTelemetry libraries for Node.js are stable and include auto-instrumentation. I’d like to say it’s automagical since it gives you distributed tracing out of the box by just adding a few modules and a single preloaded JavaScript file.

Note: To learn more check out the official Vercel docs about OpenTelemetry or take a look at the official OpenTelemetry docs.

Firstly you need to install OpenTelemetry packages:

npm install \
    @opentelemetry/sdk-node \
    @opentelemetry/resources \
    @opentelemetry/semantic-conventions \
    @opentelemetry/sdk-trace-node \
    @opentelemetry/exporter-trace-otlp-http \
    @opentelemetry/exporter-trace-otlp-grpc \
    @opentelemetry/api \
    @opentelemetry/auto-instrumentations-node \
    @opentelemetry/instrumentation-fetch

Note: OpenTelemetry APIs are not compatible with the edge runtime, so you need to make sure that you are importing them only when process.env.NEXT_RUNTIME === 'nodejs'. The official Vercel docs recommend creating a new file instrumentation.node.ts to import only when using Node.

Start by enabling OpenTelemetry instrumentation in the Next.js app by setting the experimental.instrumentationHook: true in the next.config.js like below.

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    instrumentationHook: true,
  },
    // ...
}

module.exports = nextConfig

Then, create a file called instrumentation.ts to initialize the NodeSDK for OpenTelemetry in your serverless functions.

// instrumentation.ts

export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    await import('./instrumentation.node')
  }
}

Since you need a custom configuration and a dedicated file for the Node.js runtime, create another file called instrumentation.node.ts and paste this code into it.

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { Resource } from '@opentelemetry/resources'
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { FetchInstrumentation } from '@opentelemetry/instrumentation-fetch'

const sdk = new NodeSDK({
  // The OTEL_EXPORTER_OTLP_ENDPOINT env var is passed into "new OTLPTraceExporter" automatically.
  // If the OTEL_EXPORTER_OTLP_ENDPOINT env var is not set the "new OTLPTraceExporter" will
  // default to use "http://localhost:4317" for gRPC and "http://localhost:4318" for HTTP.
  // This sample is using HTTP.
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [
    getNodeAutoInstrumentations(),
    new FetchInstrumentation(),
  ],
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'integration-testing-vercel-functions',
  }),
})
sdk.start()

This will configure the OpenTelemetry libraries and trace exporters. The OTLPTraceExporter will look for an environment variable called OTEL_EXPORTER_OTLP_ENDPOINT. If not found it’ll default to using [http://localhost:4318](http://localhost:4318) for HTTP traffic. Add it to your .env.development.local file.

# OTEL
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

Now, you need to add a tracer to the Vercel function. Replace the code in your pages/api/pokemon.ts with this.

import { trace, SpanStatusCode } from '@opentelemetry/api'
import type { NextApiRequest, NextApiResponse } from 'next'
import { sql } from '@vercel/postgres'

export async function addPokemon(pokemon: any) {
  return await sql`
    INSERT INTO pokemon (name)
    VALUES (${pokemon.name})
    RETURNING *;
  `
}

export async function getPokemon(pokemon: any) {
  return await sql`
    SELECT * FROM pokemon where id=${pokemon.id};
  `
}

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  const activeSpan = trace.getActiveSpan()
  const tracer = await trace.getTracer('integration-testing-vercel-functions')

  try {

    const externalPokemon = await tracer.startActiveSpan('GET Pokemon from pokeapi.co', async (externalPokemonSpan) => {
      const requestUrl = `https://pokeapi.co/api/v2/pokemon/${req.body.id || '6'}`
      const response = await fetch(requestUrl)
      const { id, name } = await response.json()

      externalPokemonSpan.setStatus({ code: SpanStatusCode.OK, message: String("Pokemon fetched successfully!") })
      externalPokemonSpan.setAttribute('pokemon.name', name)
      externalPokemonSpan.setAttribute('pokemon.id', id)
      externalPokemonSpan.end()

      return { id, name }
    })

    const addedPokemon = await tracer.startActiveSpan('Add Pokemon to Vercel Postgres', async (addedPokemonSpan) => {
      const { rowCount, rows: [addedPokemon, ...rest] } = await addPokemon(externalPokemon)
      addedPokemonSpan.setAttribute('pokemon.isAdded', rowCount === 1)
      addedPokemonSpan.setAttribute('pokemon.added.name', addedPokemon.name)
      addedPokemonSpan.end()
      return addedPokemon
    })

    res.status(200).json(addedPokemon)

  } catch (err) {
    activeSpan?.setAttribute('error', String(err))
    activeSpan?.recordException(String(err))
    activeSpan?.setStatus({ code: SpanStatusCode.ERROR, message: String(err) })
    res.status(500).json({ error: 'failed to load data' })
  } finally {
    activeSpan?.end()
  }
}

What is going on here?

1. You first import the @vercel/postgres SDK.
2. Then, define two functions for interacting with Vercel Postgres, addPokemon and getPokemon.
3. The code instantiates a tracer with trace.getTracer.
4. You get the activeSpan and instantiate a tracer with trace.getTracer. The active span in this context is the span from the Vercel function itself.
5. It uses the tracer.startActiveSpan to wrap the external HTTP request and a Vercel Postgres database call. This will generate a trace span that will attach any child spans to it. What I want you to care about is making sure the fetch request’s trace spans are attached the correct parent span. This happens automatically with the tracer.startActiveSpan.
6. It uses span.setStatus and span.setAttribute to attach values to the span itself.
7. It uses span.recordException(String(err)) in the exception handler to make sure to record an exception if it happens.
8. Finally, pun intended, it ends the span with the span.end method.

With this, your Next.js app will emit distributed traces! But, you’re not sending them anywhere. Let’s fix that by adding Tracetest to the development lifecycle.

The Modern Way of Testing Serverless Functions Locally

Spin up your Next.js app.

npm run dev

This starts the server and function on http://localhost:3000/api/pokemon.

Tracetest is a trace-based testing tool for building integration tests in minutes using OpenTelemetry traces. Create test specs against trace data at every point of a request transaction. It’s open source and has a managed cloud offering as well.

Since it’s easier to get started with no dependencies, I’ll show how to use Tracetest instead of Tracetest Core. But, you can still get the same functionality with Tracetest Core!

To get started with Tracetest:

1. You’ll need to download the CLI for your operating system. Sample for MacOS below.

brew install kubeshop/tracetest/tracetest

1. And, sign up for an account. Go ahead and do that now.

The CLI is bundled with Tracetest Agent that runs in your infrastructure to collect responses and traces for tests. Learn more in the docs here.

To start Tracetest Agent add the --api-key flag manually at startup or, just run tracetest start and pick your env from the menu.

You can find the organization and environment values in the Settings > Agent in the Tracetest app.

Now, go ahead and start Tracetest Agent.

tracetest start --api-key <ttagent_key>

With the Tracetest Agent started, go back to [app.tracetest.io](http://app.tracetest.io) and trigger the serverless function.

Switch to the Trace tab to see the full preview of the distributed trace.

From here you can add test specs to validate that the HTTP request never fails. Including both the downstream API and both Vercel Postgres database connections.

Click the Test tab and add a test spec from the snippets. Select All HTTP Spans: Status code is 200.

Save the test spec. You now see the test is passing since both the external HTTP requests and database interactions are passing and returning status code 200.

All this enabled by OpenTelemetry tracing and Tracetest! What’s also awesome is that these tests are stored in your Tracetest Account and you can revisit them and run the same tests again every time you run your dev environment!

This is awesome for your development lifecycle and API testing while developing Vercel functions, but also in pre-merge testing, and integration testing.

Let me explain how to enable integration testing in CI pipelines next!

Integration Testing Vercel Serverless Functions

Finally, check out the Automate tab.

Every test you create can be expressed with YAML. I know you love YAML, quit complaining! 😄

With this test definition you can trigger the same test via the CLI either locally or in any CI pipeline of you choice.

To try it locally, create a directory called test in the root directory.

Paste this into a file called api.pokemon.spec.development.yaml.

# api.pokemon.spec.development.yaml

type: Test
spec:
  id: kv8C-hOSR
  name: Test API
  trigger:
    type: http
    httpRequest:
      method: POST
      url: http://localhost:3000/api/pokemon
      body: "{\n  \"id\": \"6\"\n}"
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="http"]
    name: "All HTTP Spans: Status  code is 200"
    assertions:
    - attr:http.status_code = 200

Since you already have the Tracetest CLI installed, running it is as simple as one command.

tracetest run test -f ./test/api.pokemon.spec.development.yaml --required-gates test-specs --output pretty

[Output]
✔ Test API (https://app.tracetest.io/organizations/ttorg_e66318ba6544b856/environments/ttenv_0e807879e2e38d28/test/-gjd4idIR/run/22/test) - trace id: f2250362ff2f70f8f5be7b2fba74e4b2
    ✔ All HTTP Spans: Status code is 200

What’s cool is you can follow the link and open the particular test in Tracetest and view it once it’s saved in the cloud.

Let’s introduce Docker into the mix and set up a proper CI environment.

Start by creating a .env.docker file.

# OTLP HTTP
OTEL_EXPORTER_OTLP_ENDPOINT="http://tracetest-agent:4318"

# Vercel Postgres
POSTGRES_DATABASE="**********"
POSTGRES_HOST="**********"
POSTGRES_PASSWORD="**********"
POSTGRES_PRISMA_URL="**********"
POSTGRES_URL="**********"
POSTGRES_URL_NON_POOLING="**********"
POSTGRES_USER="**********"

This will make sure we set up the trace exporter endpoint to send traces to Tracetest Agent.

Then, create a docker-compose.yaml in the root directory.

version: "3"

services:
  next-app:
    image: foobar/next-app:v1
    build:
      context: .
      dockerfile: ./Dockerfile
    env_file:
      - .env.docker
    restart: always
    ports:
      - 3000:3000
    networks:
      - tracetest

  tracetest-agent:
    image: kubeshop/tracetest-agent:latest
    environment:
      - TRACETEST_API_KEY=ttagent_<api_key> # Find the Agent API Key here: https://docs.tracetest.io/configuration/agent
    ports:
      - 4317:4317
      - 4318:4318
    networks:
      - tracetest

  integration-tests:
    image: foobar/integration-tests:v1
    profiles:
      - tests
    build:
      context: ./
      dockerfile: ./test/Dockerfile
    volumes:
      - ./test/:/app/test/
    depends_on:
      tracetest-agent:
        condition: service_started
    networks:
      - tracetest

networks:
  tracetest:

Next, create a Dockerfile for the Next.js app.

FROM node:20-alpine AS base
FROM base AS builder
WORKDIR /app
RUN apk add --no-cache g++ make py3-pip

COPY package.json package-lock.json ./
RUN npm i

COPY app ./app
COPY pages ./pages
COPY public ./public
COPY next.config.js .
COPY tsconfig.json .
COPY instrumentation.ts .
COPY instrumentation.node.ts .

RUN npm run build

# Step 2. Production image, copy all the files and run next
FROM base AS runner
WORKDIR /app
# Don't run production as root
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
USER nextjs

COPY --from=builder /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
CMD ["node", "server.js"]

Don’t forget to set the next.config.js to include output: standalone.

/** @type {import('next').NextConfig} */
const nextConfig = {
  // ...
  output: 'standalone',
}

module.exports = nextConfig

Go back to the test directory and create a Dockerfile in it. This Dockerfile is for a container running the integration tests. It will include an installation of the Tracetest CLI and contain a bash script called run.bash to run tests.

FROM alpine
WORKDIR /app
RUN apk --update add bash jq curl
RUN curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash
WORKDIR /app/test/
ENTRYPOINT ["/bin/bash", "/app/test/run.bash"]

Next, create the run.bash.

#/bin/bash

# Add a Tracetest token here
# https://docs.tracetest.io/concepts/environment-tokens
tracetest configure -t tttoken_<token>
tracetest run test -f ./api.pokemon.spec.docker.yaml --required-gates test-specs --output pretty
# Add more tests here! :D

Finally, create a api.pokemon.spec.docker.yaml file in the test directory.

# api.pokemon.spec.docker.yaml

type: Test
spec:
  id: p00W82OIR
  name: Test API
  trigger:
    type: http
    httpRequest:
      method: GET
      url: http://next-app:3000/api/pokemon # Note: Using Docker networking!
      body: "{\n  \"id\": \"6\"\n}"
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="http"]
    name: "All HTTP Spans: Status code is 200"
    assertions:
    - attr:http.status_code = 200

Now, go ahead and give it a run!

docker compose up -d --build

And, trigger the integration tests.

docker compose run integration-tests

[Ouput]
[+] Creating 1/0
 ✔ Container integration-testing-vercel-functions-tracetest-agent-1  Running                                                                                             0.0s
 SUCCESS  Successfully configured Tracetest CLI
✔ Test API (https://app.tracetest.io/organizations/ttorg_e66318ba6544b856/environments/ttenv_82af376d61da80a0/test/p00W82OIR/run/8/test) - trace id: d64ab3a6f52a98141d26679fff3373b6
    ✔ All HTTP Spans: Status code is 200

Running this particular Docker Compose stack in any CI pipeline of choice is a breeze. The only dependencies you need are Docker and Docker Compose. It’s self-contained and standalone. Easily transferrable and runnable in your pipeline of choice.

Beyond Integration Testing

In conclusion, today you learned a step-by-step approach to developing, troubleshooting, and testing serverless functions using Vercel and Next.js. You started from a boilerplate Next.js project, created a serverless function, and integrated OpenTelemetry for distributed tracing.

Then you added Tracetest for trace-based testing, in both your development workflow and CI pipelines. With these tools at your disposal you can build, test, and optimize serverless functions efficiently for developing robust serverless applications.

If you get stuck along the tutorial, feel free to check out the example app in the GitHub repo, here.

Stay tuned for the next 2 parts of this series coming soon:

• Part 2: I’ll dive into end-to-end testing by integrating Cypress with Tracetest.
• Part 3: You’ll learn how to configure production troubleshooting and testing by using observability tools on the Vercel Marketplace.

Would you like to learn more about Tracetest and what it brings to the table? Visit the Tracetest docs and try it out by downloading it today!

Also, please feel free to join our Slack community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.