Originally published on the Postman blog.
Your load test passed. One of its requests returned a 404. If you already have Postman collections and want to turn them into load tests, this is the trap to know about. Postman doesn't count an HTTP status as a failure on its own. A green run can hide broken requests, and a script that works for one user can quietly misbehave at 50. This guide shows how to get the script right at one virtual user (VU) first: real assertions, a single sign-in, steady pacing and a p95 gate in CI. Only then do you scale.
A Postman performance test is only as good as the script behind it. A collection that works for one request can quietly misbehave when 50 virtual users run it at once: a variable that was never set, an assertion that only holds for one record, a sign-in that every user fires in the same second.
This post walks through building that script. You get a collection, chain requests, write assertions that mean something under load, sign in and clean up with setup and teardown, and feed the test data. Just as important, you debug it before you add load. Once it runs cleanly, you can try it in the app and then take it to CI with the Postman CLI or Postman Cloud.
The workflow at a glance
- Get a collection: import your OpenAPI spec, or record it with the Postman proxy, fork one, or write it.
- Script it: chain values between requests and control the pace.
- Assert on correctness and on a response-time budget.
- Sign in and clean up with setup and teardown collections.
- Feed it data with a data file or a dataset.
- Debug it at one user, before you add load.
You will loop between Debug and Script until a one-VU run is clean, and only then run it at scale.
What a performance test script is
There is no separate scripting language for performance tests. A performance test runs the requests in your collection (or one folder) in order, in a repeating loop, for every VU at once, for a set duration. Your pre-request scripts, post-response scripts and pm.test() assertions run on every pass of that loop, the same way they run in a functional run. That has two consequences:
- Your script runs thousands of times. A slow or noisy script is multiplied by every request, every VU and every iteration. On a local run, heavier scripts reduce the number of VUs your machine can simulate.
- Many copies run at once. Anything that assumes it is the only user (a shared record, a one-time sign-in, a counter) behaves differently when it is not.
Prerequisites
- The Postman desktop app, where you also run a local performance test, and a workspace.
- Docker, to run the demo app.
- For Step 5: a paid plan if you want datasets or data files in a performance test.
Set up the demo app
The examples use Swagger Petstore, an open source (Apache-2.0) sample REST API with an OpenAPI spec. It runs on your own machine, which matters: never point a load test at a shared public service, and never at production.
The app listens on 8080 inside the container, and the setup maps it to 18080 on your machine. Port 8080 is one of the most commonly used ports, and a clash fails quietly (see Step 6, pointer 4). To check that 18080 is free first (macOS and Linux), run the command below. No output means the port is free.
lsof -nP -iTCP:18080 -sTCP:LISTEN
Give the app a few seconds to start. The OpenAPI spec is at http://localhost:18080/api/v3/openapi.json. Four things are worth knowing before you start:
-
Pin the tag. The README suggests
:unstable, which is a March 2023 build.1.0.28is the current release. - Timings are not benchmarks. The image is built for amd64 only, so on Apple Silicon it runs under emulation.
- State lives in memory. Restart the container for a clean slate.
-
Auth is declared but not enforced. The spec lists OAuth2 and an
api_keyheader, but the server accepts requests without either.
Step 1: Get a collection
A performance test needs one thing: a collection (or one folder of one). The rest of this post uses a single collection, Petstore Load, built from the app's OpenAPI spec.
Import the OpenAPI spec
If your API has an OpenAPI spec, importing it is usually the fastest route, and it doubles as a check that the spec matches reality. In the app:
- Click Import.
- Add Petstore's spec: paste
http://localhost:18080/api/v3/openapi.json, or save it withcurl -o openapi.json http://localhost:18080/api/v3/openapi.jsonand drag the file in. - Leave the import option as it is (OpenAPI 3.0).
- Press Enter to import, then rename the new collection to Petstore Load.
Petstore's spec has 19 operations, so you get 19 requests grouped into folders by path, each with saved examples. A collection generated from an OpenAPI 2.0, 3.0 or 3.1 spec can be kept in sync with the spec afterward, so when the spec changes you update the collection instead of rebuilding it.
Prefer the command line? This builds the same kind of collection as local files (add --folder-strategy Tags to group by tag):
postman spec generate collection ./openapi.json -n "Petstore Load"
Fix what the import gets wrong
-
Set the base URL. Petstore's spec declares its server as the relative path
/api/v3, so the collection'sbaseUrlhas no host. Change it tohttp://localhost:18080/api/v3. - Check the request bodies. The generated "Add a new pet" request uses the form-encoded body variant. Step 2 replaces it with a clean JSON body.
- Watch the order. Requests run in folder order, which is alphabetical here, so the sign-in request runs last. Step 4 moves sign-in into a setup collection, which runs before everything else.
- Add assertions. A generated collection has no tests. Step 3 adds them.
Trim it to the requests you will test
Load-testing all 19 requests is not the goal. Keep the four that make up one realistic flow, put them in a folder called Pet Lifecycle, and delete the rest. Sign-in is not among them: it moves to its own setup collection in Step 4.
Run it once before you load it
Send the requests once before you add load. A performance test repeats the collection in a loop for every VU, so anything that should happen only once will happen thousands of times. Move one-time work into a setup collection (Step 4).
Read the result with suspicion. With baseUrl fixed and nothing else changed, the four-request collection reports 4 requests, 0 failed. Yet one of those four, GET /pet/4808, came back 404 Not Found. Nothing failed because nothing was asserting anything. Postman does not treat an HTTP status as a failure on its own, so a collection without assertions can only fail on a transport problem. Add assertions (Step 3) before you trust a green run.
Other ways to get a collection
-
A cURL command or HAR file: paste it into Import, or run
postman collection import <curl or file.har> --name "...". - Real traffic from a client: the Postman proxy (Tools → Proxy) records what a client sends and saves it as a collection.
- A collection someone else built: fork it or import the file.
-
From scratch: + → Collection, or
postman collection new "Orders API".
Step 2: Script it
Pre-request and post-response scripts run under load exactly as they do locally, so your functional scripts carry over. Every script runs once per request per VU, so keep them lean.
Know your variables first
Postman has five variable scopes. From broadest to narrowest, they are global, collection, environment, data and local. When the same name exists in two scopes, the narrowest one wins.
Who else sees a variable's value? Collection, environment and global variables are local to your copy of Postman by default. The value you type is used when you send requests, and it is not synced to the Postman Cloud. A variable stays yours until someone with Editor access chooses to share its value.
Changing a variable from a script changes your local value only.
Don't confuse teammates with virtual users. During a performance test, each VU runs with its own copy of the collection variables. A value one VU sets (a token, a counter, an iteration start time) is never visible to another VU. That is why per-VU sign-in and the pacing timestamp below are safe. It is also why collection variables cannot coordinate VUs.
The shared value matters for load tests in two ways:
- Cloud features use the shared value, not your local one. That includes cloud performance runs, scheduled runs, monitors and the Postman CLI. A variable that is only local works on your laptop and comes up empty when the same collection runs in the cloud or in CI. Share the values your test reads before you run it anywhere but your own machine.
- Shared values are not private. Never share a password or token as a plain value. Mark it as a secure variable, which stores the value in Postman Vault.
Rule of thumb:
- Put test settings (think time, pacing, flags) in shared collection variables.
- Put the base URL in an environment.
- Put per-iteration input in a data file.
- Put credentials in secure variables.
Make the settings variables, not code
Built-in think time and pacing settings are not part of the performance test configuration yet, and the Postman team is working on them. Until then, scripts can do the job. Put every tunable in a collection variable with a sensible default, and have the scripts read it. Document the knobs in the collection description.
Chain values between requests
The Petstore flow is add a pet, get it, find pets by status, and delete it. Each request needs the ID the first one chose, so give every iteration its own pet ID. That way concurrent users don't step on one another's data.
In a pre-request guard, use pm.execution.skipRequest(). In a pre-request script, pm.execution.setNextRequest(null) does not stop the request it runs in. In a post-response script, setNextRequest(null) is the right call, because the request has already been sent. Later requests read the same variables, such as GET {{baseUrl}}/pet/{{petId}} and DELETE {{baseUrl}}/pet/{{petId}}.
pm.test("add pet returns 200", () => {
pm.response.to.have.status(200);
});
pm.test("add pet echoes the id we sent", () => {
pm.expect(String(pm.response.json().id)).to.eql(pm.collectionVariables.get('petId'));
});
Sign in once, and ramp the users up
If every VU signs in at the same instant, you are testing your login endpoint's tolerance for a stampede, not the flow you care about. The Postman CLI has two built-in answers. Both are in Postman CLI 1.75.0: check postman performance run --help for --setup-collection and --load-profile.
Sign in once, for the whole test. Put the sign-in request in a setup collection. It runs once before any VU starts, and Postman passes the environment it updated on to every VU. No sign-in request is left in the loop, so there is no stampede. Step 4 shows the details.
Ramp the VUs up. With the CLI, the default load profile is ramp-up:
--load-profile ramp-up --base-vu 2 --peak-vu 50 --duration 10 --load-segments 1m,3m
That is 2 VUs for the first minute, a climb to 50 over the next three, and 50 until minute 10. The base VUs still start together, so keep the base small.
When one token is not enough. If each VU needs its own session, or the token can expire mid-run, sign in once per VU inside the loop:
pm.test("sign in returns 200", () => { pm.response.to.have.status(200); });
// Petstore answers with plain text: "Logged in user session: "
const match = pm.response.text().match(/session:\s*(\S+)/);
if (match) {
pm.collectionVariables.set('accessToken', match[1]);
} else {
console.log('Sign in did not return a session, ending this iteration.');
pm.execution.setNextRequest(null);
}
Add think time
Without think time, every VU fires its next request the instant the last one returns. Add a random pause in each request's post-response script:
const min = Number(pm.collectionVariables.get('THINK_TIME_MIN_MILLIS')) || 0;
const max = Number(pm.collectionVariables.get('THINK_TIME_MAX_MILLIS')) || 0;
if (max > 0) {
const delay = Math.floor(Math.random() * (max - min + 1)) + min;
await new Promise(resolve => setTimeout(resolve, delay));
}
Because the pause is gated on max > 0, one variable turns it off for a quick smoke test.
Hold a steady pace
Think time spaces out requests within an iteration. Pacing fixes how long the whole iteration takes. Without pacing, a fast API makes each VU loop faster, so the load you send quietly grows with how well your API performs. A pacing floor makes every iteration take at least a fixed time, so the request rate stays predictable.
// First request of the loop
const pacing = Number(pm.collectionVariables.get('PACING_DURATION_MILLIS')) || 0;
if (pacing > 0) {
pm.collectionVariables.set('iterationStartMillis', String(Date.now()));
}
// Last request of the loop
const pacingTarget = Number(pm.collectionVariables.get('PACING_DURATION_MILLIS')) || 0;
const pacingStart = Number(pm.collectionVariables.get('iterationStartMillis'));
if (pacingTarget > 0 && Number.isFinite(pacingStart)) {
const remaining = pacingTarget - (Date.now() - pacingStart);
if (remaining > 0) {
await new Promise(resolve => setTimeout(resolve, remaining));
}
}
Once pacing is on, the request rate is roughly VUs × requests per iteration ÷ pacing time. That is how you scale VUs without scaling the rate by accident. Measure to confirm the knobs do what you think. Sleep time is not counted as response time.
Keep scripts cheap
- Skip
console.logon the hot path. - Parse the body once and reuse it.
- Fail loudly on missing inputs. A thrown error counts as an error in the run.
Step 3: Add test assertions that mean something under load
Think in two layers: correctness (status code, shape) and budget (response time).
pm.test("Status is 200", () => pm.response.to.have.status(200));
pm.test("Responds within budget", () => {
pm.expect(pm.response.responseTime).to.be.below(800);
});
pm.test("Returns the pet we added", () => {
pm.expect(pm.response.json().name).to.eql(pm.collectionVariables.get('petName'));
});
Assertions are the only thing that can make an individual request fail. A 404 or a 500 is not an error on its own. If you want a status code to count, assert on it.
The run also has test-level pass/fail thresholds: average, p90, p95, p99, error rate and requests per second. Per-request assertions tell you which endpoint broke. Run-level thresholds tell you whether the test as a whole passed, and you need both. In the CLI, set them with --pass-if, for example --pass-if "less_than(p95, 500)". Repeat the flag to combine conditions; every condition must hold. The command exits 0 when the conditions are met and 1 when they are not.
Step 4: Sign in and clean up with setup and teardown collections
A setup collection and a teardown collection run once before and once after the whole test. The main collection runs in between, in the VU loop.
- Setup runs once, before the load starts. If setup fails, the test is skipped, teardown is still attempted, and the run is marked failed.
- Teardown runs on every outcome, including cancel, fail and timeout.
- State passes through a per-run copy of the environment you select.
// Petstore Setup — Sign In post-response script
pm.test("sign in returns 200", () => { pm.response.to.have.status(200); });
const match = pm.response.text().match(/session:\s*(\S+)/);
if (match) {
pm.environment.set('accessToken', match[1]);
}
// Petstore Teardown — Sign Out post-response script
pm.test("sign out returns 200", () => pm.response.to.have.status(200));
Setup writes the token to the environment, not to a collection variable, which is what lets the main collection see it. The main requests send Bearer {{accessToken}}. Check that the token actually arrives:
pm.test("Authorization header carries a token", () => {
const authorization = pm.variables.replaceIn(pm.request.headers.get('Authorization') || '');
pm.expect(authorization).to.match(/^Bearer \S+$/);
pm.expect(authorization).to.not.include('{{');
});
From the CLI:
postman performance run <collection-id> \
--environment <environment-id> \
--setup-collection <setup-collection-id> \
--teardown-collection <teardown-collection-id> \
--peak-vu 5 --duration 5
Step 5: Drive the test with data
One user hammering one record tests your cache, not your API. Feed each VU different data with a data file or a Postman Dataset (both need a paid plan in performance tests).
In a CSV, the first row holds variable names. Use Unix line endings and keep every row the same width.
Watch the names: a data column beats a collection variable with the same name. Give script-created values names no column uses, such as createdPetId.
# Data file (local only)
postman collection run <collection-id> --data-file ./pets.csv
# Dataset (local or cloud)
postman performance run <collection-id> \
--dataset-id <dataset-id> \
--dataset-view-id <view-id> \
--dataset-distribution round-robin
Step 6: Debug the script before you add load
Debug at one user, with nothing else going on. A script bug under 50 VUs shows up as a wall of identical errors. At one VU it shows up as one error you can read.
- Open the Postman Console (⌘+Option+C / Ctrl+Alt+C) and run once. It shows the request sent, the raw response and your logs.
- Print what you would otherwise guess. Log the value and its type.
- Read the error before you fix it.
-
Check the final URL, and who answered. An undefined
{{variable}}produces an invalid address. A 404 on the first request can mean another service owns the port, so compare theServerheader. - A bad status code is not an error. Assert on it.
- Make missing input loud. Guard every input, and stop or throw when it is missing.
- Know which bucket a failure lands in. The CLI's live progress counts only transport errors. Script errors show up only in the final summary, so do not gate on the live number.
-
Don't count on logs once the load starts. Use
console.logfor the one-VU pass, then rely on assertions and thrown errors. - When everything fails at once, check the server. Send one valid request by hand and read the server's own log.
-
Shrink the problem.
postman performance runalways runs the whole collection, so run a trimmed copy or addskipRequest()guards.
Once the script works: run it
- In the app: Run → Performance → App. Start with a handful of VUs for a minute.
- In CI: use the Postman CLI with a pass/fail threshold so a slow build fails the job.
- At scale: Postman Cloud (paid plan).
postman performance run <collection-id> --peak-vu 5 --duration 1 --pass-if "less_than(p95, 500)"
Set think time and pacing before you scale. They are what let you raise the VU count without raising the request rate by accident.
Wrapping up
If you take one thing from this post, make it this: get the script right at a small scale first, because adding virtual users will not fix a script that is wrong at one. Native think time and pacing settings, and richer logging, are on the way. Until then, the scripts above do the job.
Walk the six steps on your own collection, and get one clean one-VU run before you scale. The full post with every screenshot and measurement is on the Postman blog.


Top comments (0)