DEV Community

Cover image for Apidog CLI vs Newman: Run API Tests in CI (2026)
Hassann
Hassann

Posted on • Originally published at apidog.com

Apidog CLI vs Newman: Run API Tests in CI (2026)

Your API tests passing locally is not enough. The real requirement is running them on every pull request, merge, and scheduled build without anyone opening a GUI. A command-line runner does that: it executes API tests headlessly, returns an exit code your CI system can trust, and writes reports your pipeline can publish.

Try Apidog today

For years, the default CLI runner for Postman collections has been Newman. It is open source, mature, and widely used. If you author tests in Apidog, you also have another option: the Apidog CLI, which runs the same visual test scenarios you build in Apidog directly inside CI.

This guide compares Newman and the Apidog CLI at the implementation level: install commands, run commands, reports, exit codes, GitHub Actions examples, and how to choose the right runner for your workflow.

TL;DR

  • Newman runs exported Postman collection JSON files. 
  npm install -g newman
  newman run api-tests.postman_collection.json -e staging.postman_environment.json
Enter fullscreen mode Exit fullscreen mode
  • Apidog CLI runs Apidog test scenarios by ID from your Apidog project. 
  npm install -g apidog-cli
  apidog run --access-token "$APIDOG_ACCESS_TOKEN" -t 605067 -e 1629989
Enter fullscreen mode Exit fullscreen mode
  • Both work in GitHub Actions, GitLab CI, Jenkins, and any CI environment with Node.js.
  • Both return non-zero exit codes when tests fail.
  • Both can generate JUnit reports for CI dashboards.
  • Choose Newman if your tests already live as Postman collections.
  • Choose Apidog CLI if you want visual test authoring, request chaining, shared environments, and no collection export step.

The real problem: API tests that exist but never run

A manually triggered API test suite decays quickly. Someone creates it, runs it once, and then the API changes while the tests sit untouched.

CI fixes that only if the test runner can:

  1. Run without a GUI.
  2. Fail the build with a non-zero exit code.
  3. Generate machine-readable reports.
  4. Fit into your existing pipeline.

Both Newman and the Apidog CLI satisfy those requirements. The main difference is where the test definition lives.

  • Newman runs an exported Postman collection file.
  • Apidog CLI runs a live scenario from an Apidog project.

If you are designing your broader automation flow, this guide pairs well with how to automate API tests in CI/CD.

What Newman does well

Newman is the official open-source command-line runner for Postman collections. If your team already maintains tests in Postman, Newman is usually the fastest path to CI.

Install it globally with npm:

npm install -g newman
Enter fullscreen mode Exit fullscreen mode

Run an exported collection:

newman run api-tests.postman_collection.json
Enter fullscreen mode Exit fullscreen mode

Run it with an environment file:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json
Enter fullscreen mode Exit fullscreen mode

That is the basic Newman workflow:

  1. Build requests and tests in Postman.
  2. Export the collection JSON.
  3. Export the environment JSON.
  4. Commit those files or store them as CI artifacts.
  5. Run them with Newman in CI.

For a step-by-step setup, see how to install Newman and run Postman collections.

Run data-driven tests with Newman

Newman supports CSV and JSON data files:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json \
  -d users.csv \
  -n 5
Enter fullscreen mode Exit fullscreen mode

Useful flags:

Flag Purpose
-e Environment file
-d Data file
-n Iteration count
-r Reporters
--bail Stop after first failure

Example with reporters:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json \
  -r cli,junit \
  --reporter-junit-export ./results/junit.xml
Enter fullscreen mode Exit fullscreen mode

The JUnit file can then be uploaded as a CI artifact or parsed by your test reporting tools.

Where Newman adds maintenance cost

Newman is reliable, but the maintenance model matters.

The main trade-off is the export step. A Postman collection in CI is usually a JSON snapshot. If someone updates the test in Postman but forgets to export and commit the new JSON, your CI run executes an older version of the test.

That creates two sources of truth:

  • The collection being edited in Postman.
  • The exported JSON being run in CI.

The second trade-off is scripting. Postman test logic often lives in JavaScript snippets attached to requests. That is flexible, but larger suites can become harder to maintain as chaining, assertions, variable extraction, and setup logic spread across many scripts.

This does not make Newman a bad tool. It means Newman fits best when your team is comfortable with:

  • Postman as the authoring tool.
  • JavaScript snippets for test logic.
  • Exported JSON as the CI artifact.
  • Manual or automated discipline around keeping exports current.

For related constraints, see Postman Collection Runner restrictions and how to work around them.

What the Apidog CLI does differently

The Apidog CLI runs test scenarios created in Apidog. Instead of exporting a collection file, you reference a scenario by ID.

Install it:

npm install -g apidog-cli
Enter fullscreen mode Exit fullscreen mode

Run a scenario:

apidog run \
  --access-token "$APIDOG_ACCESS_TOKEN" \
  -t 605067 \
  -e 1629989 \
  -n 1 \
  -r html,junit
Enter fullscreen mode Exit fullscreen mode

In this model:

  1. You create a test scenario visually in Apidog.
  2. You chain requests in the scenario.
  3. You configure assertions.
  4. You select an environment.
  5. You generate a CI command from the scenario’s CI/CD tab.
  6. You store the access token as a CI secret.
  7. CI runs the scenario by ID.

The important difference is that there is no exported JSON file to keep in sync. The scenario in Apidog is the test definition. The CLI fetches it and runs it.

If you need to verify supported flags for your installed version, run:

apidog run --help
Enter fullscreen mode Exit fullscreen mode

Newman vs Apidog CLI: side-by-side

Dimension Newman (newman) Apidog CLI (apidog)
Package newman apidog-cli
Install npm install -g newman npm install -g apidog-cli
Run command newman run <collection.json> apidog run
Test source Exported Postman collection JSON Apidog test scenario fetched by ID
Authoring Postman app and JavaScript test scripts Visual scenario builder and no-code assertions
Sync model Export JSON after changes Run live scenario from project
CI authentication None for local files; API key if fetching from cloud Access token via --access-token
Environment -e <environment.json> -e <environmentId>
Data-driven runs -d <file.csv or file.json> -d <path>
Iterations -n <count> -n <count>
Reporters cli, json, junit, html cli, html, json, junit
Fail-fast behavior --bail --on-error end
Open source Yes No; npm CLI runs scenarios from your Apidog plan
Account required Postman account for cloud collections Apidog account for project access

The flags are similar enough that the CI wiring feels familiar. The bigger difference is the test lifecycle:

  • Newman: export a collection, run the file.
  • Apidog CLI: build a scenario, run it by ID.

For another comparison from the Postman ecosystem, see Postman CLI vs Newman.

Reports and exit codes

In CI, the two most important runner outputs are:

  1. The process exit code.
  2. The test report file.

If the runner exits with a non-zero status, CI marks the step as failed. That is what blocks a merge or deployment.

Newman reports

Example:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json \
  -r cli,junit \
  --reporter-junit-export ./results/junit.xml
Enter fullscreen mode Exit fullscreen mode

Use --bail if you want the run to stop after the first failure:

newman run api-tests.postman_collection.json \
  -e staging.postman_environment.json \
  --bail
Enter fullscreen mode Exit fullscreen mode

Apidog CLI reports

Example:

apidog run \
  --access-token "$APIDOG_ACCESS_TOKEN" \
  -t 605067 \
  -r html,junit \
  --out-dir ./apidog-reports
Enter fullscreen mode Exit fullscreen mode

Control failure behavior with --on-error:

apidog run \
  --access-token "$APIDOG_ACCESS_TOKEN" \
  -t 605067 \
  --on-error continue
Enter fullscreen mode Exit fullscreen mode

Common modes:

Mode Behavior
end Stop at the first error
continue Continue running and collect all failures
ignore Skip past an error

Avoid this in CI:

newman run api-tests.postman_collection.json || true
Enter fullscreen mode Exit fullscreen mode

or:

apidog run --access-token "$APIDOG_ACCESS_TOKEN" -t 605067 || true
Enter fullscreen mode Exit fullscreen mode

That hides the non-zero exit code and makes your API test gate useless.

For more on data-driven runs and reports, see data-driven API testing with CSV or JSON.

Run Newman in GitHub Actions

Here is a minimal GitHub Actions workflow for Newman.

name: API tests

on:
  pull_request:
    branches: [main]

jobs:
  api-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Newman
        run: npm install -g newman

      - name: Run API tests
        run: |
          newman run ./tests/api-tests.postman_collection.json \
            -e ./tests/staging.postman_environment.json \
            -r cli,junit \
            --reporter-junit-export ./results/junit.xml

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: newman-report
          path: ./results
Enter fullscreen mode Exit fullscreen mode

This works well when your collection and environment JSON files are already committed to the repo.

The key implementation detail is:

if: always()
Enter fullscreen mode Exit fullscreen mode

Keep that on the artifact upload step so you can inspect the report when tests fail.

Run Apidog CLI in GitHub Actions

The Apidog version has the same structure, but uses a secret access token and a scenario ID.

name: API tests

on:
  pull_request:
    branches: [main]

jobs:
  api-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Apidog CLI
        run: npm install -g apidog-cli

      - name: Run API test scenario
        run: |
          apidog run \
            --access-token "$APIDOG_ACCESS_TOKEN" \
            -t 605067 \
            -e 1629989 \
            -r html,junit \
            --out-dir ./apidog-reports
        env:
          APIDOG_ACCESS_TOKEN: ${{ secrets.APIDOG_ACCESS_TOKEN }}

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: apidog-report
          path: ./apidog-reports
Enter fullscreen mode Exit fullscreen mode

Implementation checklist:

  1. Create or open your Apidog test scenario.
  2. Open the scenario’s CI/CD tab.
  3. Generate an access token.
  4. Copy the generated command.
  5. Add the token to GitHub repository secrets as APIDOG_ACCESS_TOKEN.
  6. Use the scenario ID with -t.
  7. Use the environment ID with -e.
  8. Upload the report directory as an artifact.

Because the scenario is fetched at runtime, there is no collection JSON file to export or commit.

For similar CI patterns, see how to automate API tests in GitHub Actions.

How to choose

Choose Newman if:

  • Your tests already exist as Postman collections.
  • Your team is comfortable maintaining exported JSON files.
  • You want an open-source runner.
  • You prefer committing test definitions directly into the repo.
  • Your test logic already lives in Postman scripts.

Choose Apidog CLI if:

  • You want to build API test scenarios visually.
  • You want request chaining without maintaining many scripts manually.
  • You want the CI runner to use the same scenario your team edits and debugs.
  • You want to avoid exporting and syncing collection JSON files.
  • You already use Apidog for API design, debugging, mocking, or testing.

For more context on Newman and Postman, see what is the difference between Newman and Postman.

For teams evaluating alternatives, Apidog is also covered as a Postman alternative for API testing.

A practical migration path

You do not have to switch everything at once.

A common setup is:

  1. Keep existing Newman jobs for legacy Postman collections.
  2. Add Apidog CLI for new chained or data-driven scenarios.
  3. Run both as separate CI steps.
  4. Let each step fail independently.
  5. Move tests gradually as your team standardizes on one workflow.

Example pipeline shape:

steps:
  - name: Run legacy Postman tests with Newman
    run: |
      newman run ./tests/legacy.postman_collection.json \
        -e ./tests/staging.postman_environment.json \
        -r cli,junit \
        --reporter-junit-export ./results/newman.xml

  - name: Run Apidog scenario tests
    run: |
      apidog run \
        --access-token "$APIDOG_ACCESS_TOKEN" \
        -t 605067 \
        -e 1629989 \
        -r html,junit \
        --out-dir ./apidog-reports
    env:
      APIDOG_ACCESS_TOKEN: ${{ secrets.APIDOG_ACCESS_TOKEN }}
Enter fullscreen mode Exit fullscreen mode

That gives you a low-risk path: keep the old tests running, add new scenarios where Apidog’s workflow helps, and retire the Newman step when you are ready.

Final recommendation

Use the runner that matches where your tests live.

If your team already maintains Postman collections and wants a simple open-source CLI, use Newman.

If your team wants visual scenario authoring, shared environments, request chaining, and CI runs without exported snapshots, use the Apidog CLI.

Either way, the important part is the same: make the API test runner part of every pull request, publish the reports, and let the exit code block bad changes before they ship.

Top comments (0)