DEV Community

Cover image for Apidog CLI vs Postman CLI: The Better CI Test Runner
Hassann
Hassann

Posted on • Originally published at apidog.com

Apidog CLI vs Postman CLI: The Better CI Test Runner

Your API tests passing on your laptop only proves they work when someone clicks Run. The useful test is the one that runs automatically on every pull request, merge, and nightly build. That is where a CLI runner belongs: it executes your existing API tests headlessly, returns an exit code CI can understand, and writes reports your pipeline can publish.

Try Apidog today

Two runners usually come up when teams automate this workflow: the Postman CLI and the Apidog CLI. They solve the same CI problem from different starting points. The Postman CLI runs collections from Postman. The Apidog CLI runs visual test scenarios from Apidog. Both install quickly, work in GitHub Actions, GitLab CI, and Jenkins, and fail the build when tests fail. The main differences are how you author tests, how CI authenticates, and where the test definition lives.

💡 This is a command-level comparison. The Postman CLI does several things well, especially if your team already works in Postman. Apidog is an all-in-one API platform for design, debugging, testing, mocking, and documentation, and its CLI is the automation layer that runs your Apidog scenarios in CI.

TL;DR

  • Postman CLI runs collections stored in your Postman workspace or exported collection files. It is based on Newman, officially supported by Postman, and authenticates with a Postman API key.
  • Apidog CLI runs test scenarios designed visually in Apidog. You reference a scenario by ID and authenticate with an access token.
  • Both support JUnit, JSON, HTML, and terminal reports.
  • Both return a non-zero exit code when a test fails.
  • Use Postman CLI if your tests already live in Postman collections and you want Postman cloud reporting.
  • Use Apidog CLI if you want visual scenario authoring, request chaining, environment management, and data-driven runs from one source of truth.

The CI problem: tests that exist but do not run

A test that only runs manually will eventually drift. Someone writes it, it passes once, and then the API changes. If the test is not part of CI, nobody notices until much later.

A CLI runner fixes that by doing three things:

  1. Runs without a GUI.
  2. Exits non-zero when a test or assertion fails.
  3. Writes machine-readable reports such as JUnit XML.

That makes the test result usable by CI. The pipeline can block a merge, show failed assertions, and archive reports without requiring a developer to rerun the test locally.

If you are building this setup from scratch, the broader CI/CD patterns in automating API tests in CI/CD are worth reading. This article focuses on the two CLI runners.

What the Postman CLI does well

The Postman CLI is not the same thing as Newman.

Newman is the older open-source npm runner. The Postman CLI is newer, built on Newman’s foundation, signed and officially supported by Postman. If you are deciding between those two Postman-side tools, see Postman CLI vs Newman.

The Postman CLI is useful when your team already stores collections, environments, and shared variables in Postman. You do not need to convert the tests into another format. You authenticate, select a collection, and run it.

Install it on macOS or Linux with the official script:

curl -o- "https://dl-cli.pstmn.io/install/unix.sh" | sh
Enter fullscreen mode Exit fullscreen mode

Or install it with npm:

npm install -g postman-cli
Enter fullscreen mode Exit fullscreen mode

The binary is:

postman
Enter fullscreen mode Exit fullscreen mode

In CI, authenticate with a Postman API key:

postman login --with-api-key $POSTMAN_API_KEY
Enter fullscreen mode Exit fullscreen mode

Run a collection by ID:

postman collection run $POSTMAN_COLLECTION_ID -e $POSTMAN_ENV_ID
Enter fullscreen mode Exit fullscreen mode

You can also run an exported local collection file:

postman collection run ./collections/api-tests.json -e ./environments/staging.json
Enter fullscreen mode Exit fullscreen mode

When authenticated, the Postman CLI sends run results to the Postman cloud. That gives teams run history and shared visibility inside the Postman workspace without extra reporting setup.

What the Apidog CLI does well

Apidog takes a different approach. You build API tests visually in the Apidog app as test scenarios. A scenario can chain multiple requests, assert responses, extract values from one response, reuse them in later requests, and run with iteration data.

The Apidog CLI is the headless executor for those scenarios. It does not require you to rewrite visual scenarios as scripts. CI runs the same scenario you built in the app.

That helps with common multi-step API flows such as:

  1. Log in.
  2. Create a resource.
  3. Read the created resource.
  4. Update it.
  5. Delete it.
  6. Assert each response.

The mechanics of building those flows are covered in the guide to test scenarios for API test automation.

Install the CLI with npm:

npm install -g apidog-cli
Enter fullscreen mode Exit fullscreen mode

The binary is:

apidog
Enter fullscreen mode Exit fullscreen mode

A typical run looks like this:

apidog run --access-token $APIDOG_ACCESS_TOKEN \
  -t 605067 \
  -e 1629989 \
  -n 1 \
  -r html,junit
Enter fullscreen mode Exit fullscreen mode

Where:

  • --access-token authenticates the CLI.
  • -t selects the test scenario ID.
  • -e selects the environment ID.
  • -n sets the iteration count.
  • -r selects report formats.

You do not need to manually discover those IDs. Open the scenario in Apidog, go to the CI/CD tab, generate an access token, and Apidog builds the full command for you. Copy the command into your CI job, move the token into a CI secret, and reference it as $APIDOG_ACCESS_TOKEN.

Apidog also lets you decide where reports go. By default, reports are local. Use --out-dir to choose the artifact directory, and add --upload-report only when you want to upload an overview to Apidog.

Side-by-side comparison

Dimension Postman CLI (postman) Apidog CLI (apidog)
Package / install Install script, signed installer, or npm i -g postman-cli npm i -g apidog-cli
Run command postman collection run <id|file> apidog run -t <scenarioId>
Test source Collections in your Postman workspace or exported files Test scenarios in your Apidog project, fetched by ID
Authoring Collection editor and Postman app Visual scenario builder in Apidog
Auth in CI Postman API key via postman login --with-api-key Access token via --access-token
Select what runs Collection ID or file path -t scenario, -f folder, --test-suite
Environment -e, --environment -e, --environment <environmentId>
Data-driven runs -d, --iteration-data with JSON or CSV -d, --iteration-data with JSON or CSV
Iterations -n, --iteration-count -n, --iteration-count
Reporters cli, json, junit, html cli, html, json, junit
Fail-fast behavior --bail --on-error end; default stops on first failure
Cloud reporting Auto-sends results when signed in Opt-in via --upload-report
Built on Newman Apidog’s own engine

The CI-level capabilities are similar: both support environments, iteration data, JUnit reports, HTML reports, and failing builds on failed tests.

The bigger difference is test ownership:

  • Postman CLI runs Postman collections.
  • Apidog CLI runs Apidog visual scenarios.

Choose based on where your team wants the source of truth to live.

Reports and exit codes

CI primarily cares about two outputs:

  1. The process exit code.
  2. The test report artifact.

For the Postman CLI, use reporters and --bail when you want the run to stop on the first failure:

postman collection run $POSTMAN_COLLECTION_ID \
  -e $POSTMAN_ENV_ID \
  --reporters cli,junit \
  --bail
Enter fullscreen mode Exit fullscreen mode

The junit reporter produces XML that CI dashboards can parse. If any request, test, or assertion fails, the CLI exits non-zero and the build fails.

For the Apidog CLI, use -r for reporters and --out-dir for local artifacts:

apidog run --access-token $APIDOG_ACCESS_TOKEN \
  -t 605067 \
  -r html,junit \
  --out-dir ./apidog-reports
Enter fullscreen mode Exit fullscreen mode

Use --on-error to control scenario behavior:

apidog run --access-token $APIDOG_ACCESS_TOKEN \
  -t 605067 \
  -r cli,junit \
  --on-error continue
Enter fullscreen mode Exit fullscreen mode

Supported behavior:

  • end: stop on the first failure. This is the default.
  • continue: keep running and collect all failures.
  • ignore: continue past failures.

The process still exits non-zero when the run fails, so CI can block the merge.

GitHub Actions examples

The structure is the same for both tools:

  1. Check out the repo.
  2. Install the CLI.
  3. Authenticate with a secret.
  4. Run the tests.
  5. Upload or archive reports if needed.

Store API keys and access tokens in GitHub Actions secrets. Do not hardcode them in workflow files.

Postman CLI in GitHub Actions 

- name: Run API tests with Postman CLI
  run: |
    curl -o- "https://dl-cli.pstmn.io/install/unix.sh" | sh
    postman login --with-api-key ${{ secrets.POSTMAN_API_KEY }}
    postman collection run $POSTMAN_COLLECTION_ID \
      -e $POSTMAN_ENV_ID \
      --reporters cli,junit \
      --bail
Enter fullscreen mode Exit fullscreen mode

Apidog CLI in GitHub Actions 

- name: Run API tests with Apidog CLI
  run: |
    npm install -g apidog-cli
    apidog run --access-token ${{ secrets.APIDOG_ACCESS_TOKEN }} \
      -t 605067 \
      -e 1629989 \
      -r cli,junit
Enter fullscreen mode Exit fullscreen mode

To write Apidog reports to a known directory:

- name: Run API tests with Apidog CLI
  run: |
    npm install -g apidog-cli
    apidog run --access-token ${{ secrets.APIDOG_ACCESS_TOKEN }} \
      -t 605067 \
      -e 1629989 \
      -r cli,junit,html \
      --out-dir ./apidog-reports
Enter fullscreen mode Exit fullscreen mode

For a deeper GitHub Actions walkthrough, see API test automation with GitHub Actions. For Jenkins, see integrating Apidog automated tests with Jenkins.

Which CLI should you use?

Use the Postman CLI when:

  • Your tests already live in Postman collections.
  • Your environments and variables are already managed in Postman.
  • You want run history in the Postman cloud.
  • Your team is not planning to move away from Postman.

Use the Apidog CLI when:

  • You want to author tests visually as scenarios.
  • You need request chaining for multi-step API flows.
  • You want one source of truth for design, debugging, testing, mocking, and docs.
  • You want local reports by default and cloud upload only when you choose it.
  • You are evaluating alternatives to a Postman-centered workflow.

If you are comparing the platforms beyond the CLI layer, read the full Apidog vs Postman comparison.

A practical Apidog setup looks like this:

  1. Build a test scenario in Apidog.
  2. Add assertions and request chaining.
  3. Open the scenario’s CI/CD tab.
  4. Generate an access token.
  5. Copy the generated apidog run command.
  6. Store the token as a CI secret.
  7. Run the command in your pipeline.

FAQ

Is the Postman CLI the same as Newman?

No. Newman is the older open-source npm runner. The Postman CLI is newer, built on Newman’s foundation, signed and supported by Postman, and integrated with Postman cloud reporting. See Postman CLI vs Newman for the detailed comparison.

Do I need an account or token for either CLI in CI?

Yes. The Postman CLI uses a Postman API key:

postman login --with-api-key $POSTMAN_API_KEY
Enter fullscreen mode Exit fullscreen mode

The Apidog CLI uses an access token:

apidog run --access-token $APIDOG_ACCESS_TOKEN -t 605067
Enter fullscreen mode Exit fullscreen mode

Store both as CI secrets.

Do both runners fail the build when a test fails?

Yes. Both return a non-zero exit code when a test or assertion fails. That is what tells CI to mark the build as failed.

Can I keep reports local?

With Apidog CLI, yes by default. Use --out-dir to choose the report directory and --upload-report only if you want to upload a report overview.

With Postman CLI, you can generate local reports too, but when authenticated it also sends run results to the Postman cloud.

How do I get the exact Apidog command for my scenario?

Open the test scenario in Apidog, switch to the CI/CD tab, generate an access token, and copy the generated apidog run command. It includes the scenario ID and environment ID. Move the token into a CI secret before committing the workflow.

For all available options, run:

apidog run --help
Enter fullscreen mode Exit fullscreen mode

Which one should a team migrating off Postman cloud pick?

If the goal is to reduce dependence on a cloud-centered Postman workflow, the Apidog CLI is the better fit. It runs visual scenarios from your Apidog project and lets you choose whether reports stay local or get uploaded. Start with the Apidog vs Postman comparison if you need a broader platform-level view.

Top comments (0)