DEV Community

Cover image for How to Run Automated API Tests in Buildkite
Hassann
Hassann

Posted on • Originally published at apidog.com

How to Run Automated API Tests in Buildkite

Run automated API tests in Buildkite by adding a command step to .buildkite/pipeline.yml: install the Apidog CLI, execute apidog run against an Apidog environment, and upload the generated HTML report as a Buildkite artifact. This guide assumes your Apidog test scenarios already exist and focuses on wiring them into CI without leaking your access token.

Try Apidog today

One clarification before the setup: Buildkite is a CI/CD platform. It is not Docker BuildKit, the image-build backend inside Docker. This article is about Buildkite CI/CD.

What Buildkite is

Buildkite is a CI/CD platform with a hybrid architecture:

  • A hosted control plane handles the dashboard and build orchestration.
  • Agents run the actual jobs.

Buildkite overview

That split matters for API testing. Your pipeline is coordinated by Buildkite, but your tests can run on agents inside your own infrastructure, cloud account, or network boundary. You can also use Buildkite-hosted agents if managed compute is enough for your workflow.

This makes Buildkite useful when API tests need access to internal staging services, private networks, test databases, or environment-specific dependencies.

The Buildkite agent is open source, written in Go, and released under the MIT License. The Buildkite platform and hosted control plane are commercial SaaS.

What Buildkite is used for

Teams use Buildkite to automate jobs triggered by code changes, including:

  • Building artifacts
  • Running unit and integration tests
  • Running API and end-to-end tests
  • Deploying services
  • Running jobs on specific hardware or private networks

Buildkite usage

API testing fits this model well. A typical flow is:

  1. Push code.
  2. Buildkite starts a pipeline.
  3. An agent runs API tests against staging.
  4. The pipeline uploads the test report.
  5. A failed test blocks deployment.

If you are comparing CI options for API testing, see the best continuous integration tools for API teams.

How Buildkite pipelines are defined

Buildkite pipelines are usually defined in:

.buildkite/pipeline.yml
Enter fullscreen mode Exit fullscreen mode

When a build starts, Buildkite looks for a .buildkite directory containing pipeline.yml. A non-hidden buildkite/ directory at the repository root also works.

A basic pipeline starts with the top-level steps: key:

steps:
  - label: ":test_tube: Run tests"
    command: "npm test"
Enter fullscreen mode Exit fullscreen mode

For API testing, you will mostly use these Buildkite step types:

Step type Use case
Command step Run shell commands on an agent
Wait step Pause until previous steps finish
Block step Require manual approval before continuing

Common command-step keys include:

label: "Human-readable step name"
key: "api-tests"
command: "shell command"
commands:
  - "command 1"
  - "command 2"
env:
  NODE_ENV: "test"
agents:
  queue: "default"
secrets:
  - APIDOG_ACCESS_TOKEN
artifact_paths:
  - "reports/**/*"
parallelism: 5
matrix:
  setup:
    env:
      - "staging"
      - "production"
Enter fullscreen mode Exit fullscreen mode

The agents key routes jobs to agents with matching tags. For example:

agents:
  queue: "tests"
Enter fullscreen mode Exit fullscreen mode

Use this when API tests must run on agents with specific network access, installed tools, or permissions.

A copy-paste Buildkite pipeline for Apidog API tests

Create .buildkite/pipeline.yml:

steps:
  - label: ":test_tube: API tests (Apidog)"
    key: "api-tests"
    command: |
      npm install -g apidog-cli
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 637132 \
        -e 358171 \
        -r html,cli
      buildkite-agent artifact upload "apidog-reports/**/*"
    agents:
      queue: "default"
    secrets:
      - APIDOG_ACCESS_TOKEN
Enter fullscreen mode Exit fullscreen mode

This step does four things:

  1. Installs the Apidog CLI.
  2. Runs an Apidog test scenario.
  3. Generates CLI and HTML reports.
  4. Uploads the HTML report as a Buildkite artifact.

The key Apidog CLI flags are:

Flag Meaning
--access-token Apidog access token used by the CLI
-t Test scenario or scenario directory ID
-e Runtime environment ID
-r html,cli Generate both HTML and terminal reports

The Buildkite agent already has the buildkite-agent binary available. You only need to install apidog-cli.

For more Apidog CLI options, see the Apidog CLI complete guide.

If you want to test from your terminal before CI, follow the Apidog CLI command-line testing tutorial.

Pass the Apidog access token with Buildkite secrets

Your Apidog access token should not be committed to pipeline.yml or printed in logs.

Use Buildkite secrets instead.

Buildkite secrets is an encrypted key-value store. Secret values are encrypted at rest and in transit over TLS, scoped per cluster, and auto-redacted if they appear in logs. It works with both Buildkite-hosted and self-hosted agents.

Option 1: Inject the secret into the command step

If your Buildkite secret is named APIDOG_ACCESS_TOKEN, inject it like this:

steps:
  - label: ":test_tube: API tests"
    command: |
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 637132 \
        -e 358171 \
        -r html,cli
    secrets:
      - APIDOG_ACCESS_TOKEN
Enter fullscreen mode Exit fullscreen mode

The environment variable name matches the secret key.

Option 2: Map a secret key to a different environment variable

If your stored secret is named apidog_token, but you want the environment variable to be APIDOG_ACCESS_TOKEN, use the hash form:

steps:
  - label: ":test_tube: API tests"
    command: |
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 637132 \
        -e 358171 \
        -r html,cli
    secrets:
      APIDOG_ACCESS_TOKEN: apidog_token
Enter fullscreen mode Exit fullscreen mode

The format is:

ENV_VAR_NAME: buildkite_secret_key
Enter fullscreen mode Exit fullscreen mode

Option 3: Fetch the secret inside the script

You can also retrieve the secret with the Buildkite agent CLI:

APIDOG_ACCESS_TOKEN="$(buildkite-agent secret get apidog_token)"

apidog run \
  --access-token "$APIDOG_ACCESS_TOKEN" \
  -t 637132 \
  -e 358171 \
  -r html,cli
Enter fullscreen mode Exit fullscreen mode

This is useful when you want explicit control over when the secret is read.

Other secret-loading options

On self-hosted agents, you can also use an environment agent hook to export secrets at job startup. For example, you can load a token only when the job matches a specific pipeline or step:

if [[ "$BUILDKITE_PIPELINE_SLUG" == "api" && "$BUILDKITE_STEP_KEY" == "api-tests" ]]; then
  export APIDOG_ACCESS_TOKEN="..."
fi
Enter fullscreen mode Exit fullscreen mode

You can also fetch secrets from external systems such as AWS Secrets Manager, HashiCorp Vault, or GCP through Buildkite plugins. In that setup, the secret does not need to be stored in or sent to Buildkite.

Use plain env: only for non-sensitive configuration. Do not put access tokens there.

For more on CLI authentication, see Apidog CLI authentication.

Upload the HTML report as a Buildkite artifact

The Apidog HTML reporter writes a report file into the agent workspace. That workspace is temporary, so upload the report before the job ends:

buildkite-agent artifact upload "apidog-reports/**/*"
Enter fullscreen mode Exit fullscreen mode

Keep the glob pattern quoted:

"apidog-reports/**/*"
Enter fullscreen mode Exit fullscreen mode

Quoting prevents your shell from expanding the pattern before buildkite-agent receives it.

By default, uploaded artifacts go to Buildkite-managed storage. Buildkite’s default artifact storage has a 6-month retention window and a 5GB-per-artifact limit. You can also pass a destination as a second argument if you want to send artifacts elsewhere.

After the build completes, open the build page and go to the Artifacts tab to download the HTML report.

For report format details, see the Apidog CLI test reports guide.

Run tests in parallel across environments

Use a Buildkite matrix when the same Apidog scenario should run against multiple environments.

steps:
  - label: ":test_tube: API tests {{matrix.env}}"
    command: |
      npm install -g apidog-cli
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 637132 \
        -e "{{matrix.env}}" \
        -r html,cli
      buildkite-agent artifact upload "apidog-reports/**/*"
    secrets:
      - APIDOG_ACCESS_TOKEN
    matrix:
      setup:
        env:
          - "358171"   # staging environment id
          - "358172"   # production environment id
Enter fullscreen mode Exit fullscreen mode

Buildkite expands this into one job per matrix value.

In this example:

{{matrix.env}}
Enter fullscreen mode Exit fullscreen mode

is substituted into:

  • The step label
  • The Apidog -e environment flag

Use parallelism instead if you want identical copies of a job:

steps:
  - label: ":test_tube: API tests"
    command: |
      npm install -g apidog-cli
      apidog run --access-token "$APIDOG_ACCESS_TOKEN" -t 637132 -e 358171 -r html,cli
    secrets:
      - APIDOG_ACCESS_TOKEN
    parallelism: 5
Enter fullscreen mode Exit fullscreen mode

For data-driven Apidog runs, use the Apidog CLI -d flag instead of a Buildkite matrix. See the Apidog CLI data-driven testing guide.

Gate a deploy behind API tests

A common production pipeline looks like this:

  1. Run API tests.
  2. Wait for all test jobs.
  3. Require manual approval.
  4. Deploy.

Buildkite models this with wait and block steps:

steps:
  - label: ":test_tube: API tests"
    command: |
      npm install -g apidog-cli
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 637132 \
        -e 358171 \
        -r html,cli
    secrets:
      - APIDOG_ACCESS_TOKEN

  - wait

  - block: ":rocket: Deploy?"
    branches: "main"

  - label: ":rocket: Deploy"
    command: "scripts/deploy.sh"
Enter fullscreen mode Exit fullscreen mode

How it works:

  • wait stops the pipeline until previous steps complete.
  • If the API test step fails, later steps do not continue.
  • block requires a human approval click.
  • branches: "main" restricts the approval gate to the main branch.
  • The deploy runs only after approval.

For broader CI/CD patterns, see CI/CD best practices for API testing.

Add a Buildkite annotation with the test summary

Build logs can be long. Use a Buildkite annotation to put a short test summary at the top of the build page.

cat << 'EOF' | buildkite-agent annotate --style "success" --context "apidog"
### API test results

All Apidog scenarios passed.

[Download the full HTML report](artifact://apidog-reports/index.html)
EOF
Enter fullscreen mode Exit fullscreen mode

Useful flags:

Flag Purpose
--style Controls the annotation style
--context Gives the annotation a unique ID

Supported --style values include:

info
warning
error
success
Enter fullscreen mode Exit fullscreen mode

The --context value lets later steps update the same annotation instead of creating duplicates.

The artifact:// link points reviewers directly to the uploaded report.

Where Apidog fits

The Buildkite YAML should stay small. The API test logic should live in Apidog.

Apidog is an all-in-one API platform for designing, debugging, testing, mocking, and documenting APIs. You can build test scenarios visually, chain requests, pass data between steps, and add assertions without writing scripts.

Apidog test scenarios

The CLI is the bridge between Apidog and CI.

The workflow is:

  1. Build an API test scenario in Apidog.
  2. Get the scenario ID.
  3. Get the environment ID.
  4. Store your access token as a Buildkite secret.
  5. Run one command in CI: 
apidog run \
  --access-token "$APIDOG_ACCESS_TOKEN" \
  -t 637132 \
  -e 358171 \
  -r cli,html
Enter fullscreen mode Exit fullscreen mode

When you update the scenario in Apidog, the next Buildkite run uses the updated test without requiring YAML changes.

The same CLI command can run in other CI systems too. See the Apidog CLI CI/CD pipeline guide and the Apidog CLI with GitHub Actions walkthrough.

To try the workflow locally before wiring it into Buildkite, run:

apidog run \
  --access-token "$APIDOG_ACCESS_TOKEN" \
  -t 637132 \
  -e 358171 \
  -r cli,html
Enter fullscreen mode Exit fullscreen mode

Then move the same command into .buildkite/pipeline.yml.

Frequently Asked Questions

What is Buildkite?

Buildkite is a CI/CD platform with a hybrid design. A hosted control plane runs the dashboard and orchestrates builds, while agents run the actual jobs. Agents can run on your own infrastructure or on Buildkite-hosted compute. It is unrelated to Docker BuildKit.

What is Buildkite used for?

Teams use Buildkite to automate jobs triggered by code changes, including builds, tests, and deployments. For API teams, Buildkite can run automated tests against staging or production environments and block a deploy when tests fail.

Is Buildkite open source?

The Buildkite agent is open source. It is written in Go and released under the MIT License, with source code on GitHub. The hosted platform and control plane are commercial SaaS.

Is Buildkite free?

Yes. Buildkite has a free Personal plan with no credit card and no expiry. It includes 3 concurrent jobs, 1 user, 90-day retention, 500 Linux hosted-agent minutes per month, and 50,000 test executions per month. Buildkite also offers a 30-day All Access trial for evaluating paid features.

How do you upload artifacts in Buildkite?

Run this inside a command step:

buildkite-agent artifact upload "<pattern>"
Enter fullscreen mode Exit fullscreen mode

For Apidog HTML reports:

buildkite-agent artifact upload "apidog-reports/**/*"
Enter fullscreen mode Exit fullscreen mode

Quote the glob pattern so the Buildkite agent performs the matching. Uploaded reports appear in the build’s Artifacts tab.

How do you run API tests in a Buildkite pipeline?

Add a command step to .buildkite/pipeline.yml that installs the Apidog CLI, runs apidog run, injects the Apidog access token through Buildkite secrets, and uploads the HTML report:


yaml
steps:
  - label: ":test_tube: API tests"
    command: |
      npm install -g apidog-cli
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 637132 \
        -e 358171 \
        -r html,cli
      buildkite-agent artifact upload "apidog-reports/**/*"
    secrets:
      - APIDOG_ACCESS_TOKEN
Enter fullscreen mode Exit fullscreen mode

Top comments (0)