How to Set Up Automated Nightly API Tests in GitHub Actions, GitLab & Jenkins

Your API tests pass on every pull request. The build is green. You ship. Then at 9 a.m., someone in another timezone reports that checkout returns a 500. Nobody changed checkout code. The failure came from something outside the commit: a third-party payment sandbox changed, an overnight migration shifted data, or a token expired. Per-commit tests missed it because nothing in your repo changed.

Try Apidog today

That is the gap a nightly build closes. A nightly build is a scheduled full test run, usually overnight, that checks your real API workflows against real environments. For APIs, it catches failures between commits: data drift, flaky integrations, expired credentials, performance creep, and contract breaks introduced by services you do not control.

This guide shows how to build a nightly API regression workflow with Apidog. You will create a regression suite, run it with the Apidog CLI, schedule it in GitHub Actions, GitLab CI/CD, and Jenkins, and archive a report before your team starts the day.

Why per-commit tests are not enough

CI tests answer one question:

Did this code change break something?

That is useful, but narrow. Per-commit suites are intentionally fast and scoped so developers are not blocked.

A nightly build answers a broader question:

Is the system still healthy right now, even if nobody changed the code?

That matters because production-like systems have moving parts outside your Git history:

• External dependencies drift. A payment provider rotates a sandbox key or an API removes a field.
• Data changes without code. ETL jobs, migrations, and content updates can create states your fast tests never hit.
• Tokens and certificates expire. OAuth refresh tokens, TLS certs, and API keys have lifetimes.
• Performance regressions creep in. A query can move from 200 ms to 1.2 s without failing a functional assertion.
• Full suites are too slow for every push. The deep 400-case regression suite belongs on a schedule, not on every PR.

Use a two-tier pattern:

1. Fast smoke tests on every commit.
2. Deep regression tests on a schedule.

Nightly is where you put checks that are too slow, too broad, or too dependent on live integrations for the inner loop.

What to include in a nightly API regression suite

Before scheduling anything, define what the nightly run should prove.

A useful nightly API suite covers:

• Critical user journeys end to end. Example: sign up, log in, create a resource, read it, update it, delete it.
• Authentication and authorization. Valid login, expired-token rejection, role-based access, missing credentials.
• Contract conformance. Validate responses against your OpenAPI schema so removed fields or changed types fail the run. See why Swagger docs and collections keep drifting.
• Boundary and error cases. Bad input, missing resources, rate limits, unauthorized requests.
• Data integrity across requests. Create something, then verify later requests can read it correctly.
• Performance thresholds. Assert response-time budgets on critical endpoints.

If you need a structure for test cases, start with this API test case template and example. For response validation, see API assertions.

Step 1: Build the regression suite in Apidog

Apidog lets you design endpoints and chain them into test scenarios. A scenario is an ordered workflow where each step is an API request with assertions, and data can flow from one request to the next.

Example checkout regression scenario:

1. POST /auth/login

   • Send credentials.
   • Assert 200.
   • Extract accessToken.

2. POST /carts

   • Use the token.
   • Assert 201.
   • Extract cartId.

3. POST /carts/{cartId}/items

   • Add a product.
   • Assert 200.
   • Assert total equals the expected price.

4. POST /checkout

   • Submit the cart.
   • Assert 200.
   • Assert order.status equals "confirmed".

5. GET /orders/{orderId}

   • Read the order back.
   • Assert the order persisted with the correct line items.

In Apidog, you can add assertions visually, such as:

• Response status is 200
• JSONPath $.order.status equals confirmed
• Response time is under a threshold
• Response matches the saved OpenAPI schema

Use these rules to keep the suite maintainable:

• Use environments instead of hardcoded URLs. Define staging, production-readonly, or release-candidate environments.
• Parameterize with variables. Keep base URLs, usernames, credentials, and IDs outside the scenario.
• Group scenarios into a test suite. The suite is the unit you will run from CI.

For more on chaining workflows, read the API test orchestration guide. If you are new to Apidog testing, start with how to test an API with Apidog. You can also download Apidog and build one scenario before wiring it into CI.

Step 2: Run the suite from the command line

A nightly build must run unattended. Use the Apidog CLI to run saved scenarios from a terminal or CI job.

Install the CLI:

npm install -g apidog-cli

The CLI requires Node.js v16 or later.

To run an online scenario or suite from your Apidog project, you need:

• An Apidog access token
• A scenario ID, suite ID, or scenario folder ID
• An environment ID

Generate the access token in Apidog from the CI/CD settings. Store it as a CI secret, not in your repository.

Example command:

apidog run \
  --access-token $APIDOG_ACCESS_TOKEN \
  -t 123456 \
  -e 789 \
  -r cli,html \
  --out-dir ./apidog-reports

Useful CLI flags:

Flag	Purpose
--access-token <token>	Authenticates the run. Pass it from an environment variable.
-t, --test-scenario <id>	Runs one test scenario.
--test-suite <id>	Runs a full test suite.
-f, --test-scenario-folder <id>	Runs a folder of scenarios.
-e, --environment <id>	Selects the environment.
-r, --reporters [reporters]	Sets output formats, such as cli and html.
--out-dir <dir>	Writes reports to a directory CI can archive.

Other useful options:

# Repeat a run to expose flaky behavior
-n 3

# Run the same scenario with CSV or JSON data
-d ./test-data.json

# Inject runtime variables
--env-var "username=$TEST_USERNAME"
--global-var "buildId=$GITHUB_RUN_ID"

You can generate the exact command from Apidog: open the CI/CD panel, select your scenario or suite, and copy the generated apidog run command. Run it locally once before scheduling it.

Step 3: Schedule it as a nightly build

A nightly build is the CLI command on a timer.

The CI pattern is the same everywhere:

1. Trigger on a cron schedule.
2. Install Node.js and the Apidog CLI.
3. Read the Apidog token from a secret.
4. Run the suite.
5. Archive the report, even on failure.

GitHub Actions

This workflow runs every day at 02:00 UTC and also supports manual runs.

name: Nightly API Regression

on:
  schedule:
    - cron: '0 2 * * *'   # 02:00 UTC daily
  workflow_dispatch:        # allow manual runs

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Apidog CLI
        run: npm install -g apidog-cli

      - name: Run nightly regression suite
        run: |
          apidog run \
            --access-token $APIDOG_ACCESS_TOKEN \
            --test-suite ${{ vars.APIDOG_SUITE_ID }} \
            -e ${{ vars.APIDOG_ENV_ID }} \
            -r cli,html \
            --out-dir ./apidog-reports
        env:
          APIDOG_ACCESS_TOKEN: ${{ secrets.APIDOG_ACCESS_TOKEN }}

      - name: Upload HTML report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: apidog-nightly-report
          path: ./apidog-reports

if: always() is important. If the test fails, you still need the report.

GitHub scheduled jobs run in UTC and can be delayed during peak load, so avoid relying on exact-second execution.

GitLab CI/CD

In GitLab, create the schedule in the UI under CI/CD > Schedules. Then use rules so the job only runs for scheduled pipelines.

nightly-regression:
  image: node:20
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
  before_script:
    - npm install -g apidog-cli
  script:
    - apidog run
        --access-token "$APIDOG_ACCESS_TOKEN"
        --test-suite "$APIDOG_SUITE_ID"
        -e "$APIDOG_ENV_ID"
        -r cli,html
        --out-dir ./apidog-reports
  artifacts:
    when: always
    paths:
      - apidog-reports/
    expire_in: 30 days

Set these as GitLab CI/CD variables:

• APIDOG_ACCESS_TOKEN as masked and protected
• APIDOG_SUITE_ID
• APIDOG_ENV_ID

Then create a schedule such as:

0 2 * * *

Jenkins

In Jenkins, use a pipeline with a cron trigger. Store the Apidog token as a Jenkins credential.

pipeline {
    agent any
    triggers {
        cron('H 2 * * *')   // around 02:00, H spreads load
    }
    stages {
        stage('Install CLI') {
            steps { sh 'npm install -g apidog-cli' }
        }
        stage('Nightly regression') {
            steps {
                withCredentials([string(credentialsId: 'apidog-token',
                                        variable: 'APIDOG_ACCESS_TOKEN')]) {
                    sh '''
                        apidog run \
                          --access-token $APIDOG_ACCESS_TOKEN \
                          --test-suite $APIDOG_SUITE_ID \
                          -e $APIDOG_ENV_ID \
                          -r cli,html \
                          --out-dir ./apidog-reports
                    '''
                }
            }
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'apidog-reports/**', allowEmptyArchive: true
        }
    }
}

The H in H 2 * * * lets Jenkins pick a stable minute within the 2 a.m. hour. That avoids starting every scheduled job at the same time.

Common cron examples:

0 2 * * *      # every day at 02:00
0 2,14 * * *   # every day at 02:00 and 14:00
0 2 * * 1-5    # weekdays at 02:00

Pick a time when your staging environment is quiet and required third-party sandboxes are available.

Step 4: Make failures impossible to ignore

A nightly run that fails silently is not useful. Wire the result into the channels your team already watches.

apidog run exits non-zero when a scenario fails. CI will mark the job red automatically.

From there, add one or more actions:

• Use native CI notifications. Enable failed scheduled-run notifications in GitHub Actions, GitLab, or Jenkins.
• Post to chat. Send a Slack or webhook message with the run URL and archived HTML report.
• Keep reports. Store reports as artifacts so you can compare failures over time.
• Track repeat failures. One failure may be noise. The same endpoint failing three nights in a row is a signal.

Treat a nightly failure as a real bug until proven otherwise. If the suite becomes flaky and ignored, it stops being a safety net.

To debug flakiness:

apidog run \
  --access-token $APIDOG_ACCESS_TOKEN \
  --test-suite $APIDOG_SUITE_ID \
  -e $APIDOG_ENV_ID \
  -n 3 \
  -r cli,html \
  --out-dir ./apidog-reports

Repeated runs help separate a deterministic regression from unstable data, timing, or environment dependencies.

Why Apidog fits the nightly pattern

You can build a nightly API pipeline from separate tools: one for design, one for tests, one for headless runs, and another for schema validation. That works until the pieces drift.

Apidog keeps the endpoint definition, test scenario, schema validation, and CLI run in the same workflow. The scenario you debug visually is the same scenario you run headlessly in CI.

That reduces the chance that your runner executes an outdated collection or validates against a stale contract. If you have dealt with Postman collections that are not a real source of truth, this single-source workflow is the main benefit.

Frequently asked questions

What is the difference between a nightly build and continuous integration?

CI runs on code changes. A nightly build runs on a fixed schedule, usually overnight, even if no code changed.

Use both:

• CI catches regressions introduced by commits.
• Nightly runs catch expired tokens, dependency drift, changed data, integration failures, and slow performance regressions.

What time should a nightly build run?

Choose a window when:

• Your test environment is quiet.
• Required third-party services are available.
• Scheduled migrations or ETL jobs are complete.
• The team can review results in the morning.

For many teams, 1 a.m. to 4 a.m. local time works well.

Can I run a nightly suite against production?

Yes, but be careful.

Use this split:

• Staging/pre-production: full read-write regression suite.
• Production: small read-only smoke suite.

If you must write to production, use idempotent operations and cleanup steps.

How is this different from monitoring?

Monitoring answers:

Is the API up?

A regression suite answers:

Is the API correct?

Monitoring usually checks availability and status codes. A nightly regression suite checks workflows, response bodies, schemas, auth boundaries, business rules, and performance budgets.

They are complementary.

Do I need to write code to schedule API tests?

Not much. You build scenarios visually in Apidog, then copy the generated apidog run command into your CI config.

The only code you need is the CI YAML or pipeline definition.

For command-line runner comparisons, see Postman CLI vs Newman and running collections in CI without Newman.

Set up your first nightly run

Start with one critical workflow:

1. Pick login, checkout, billing, or another high-impact path.
2. Build it as an Apidog scenario.
3. Add assertions for status codes, response bodies, schema conformance, and response time.
4. Run it locally with the Apidog CLI.
5. Store the access token as a CI secret.
6. Schedule the command in GitHub Actions, GitLab, or Jenkins.
7. Archive the HTML report.
8. Send failures to your team chat.

Once that works, expand the suite with more journeys, boundary cases, schema checks, and performance thresholds.

A small nightly suite can be running in an afternoon. Within a week, you can have a regression net that catches failures your per-commit tests were never designed to see.