DEV Community: Simon Raynaud

Test a login flow without writing Playwright

Simon Raynaud — Wed, 14 Oct 2026 18:08:07 +0000

Pathly watches the path. Ping if it's up. Flow if a user can walk it. Chain if the services behind it hold. Login is a Flow. A Ping on /login only proves the URL answered. This article is the steps a real Chrome run already understands, including the login template that ships in the product. You can try a journey without writing a script from try Flow.

A Flow opens Chromium, follows a list of at most 50 steps, and stops. The default run cap is two minutes. A step timeout you do not set falls back to two seconds for a selector wait. Two clicks in a row wait two seconds between them unless you set clickDelayMs to something else, including 0. The viewport you do not set is desktop 1280 by 720. Locale defaults to fr-FR and the timezone to Europe/Paris. Set those when the page changes language or the price depends on the clock.

Why a green Ping is not a login

/login returns 200 when the button does nothing. A third-party script fails to load. An identity provider changes its redirect. A release renames the field. The document is fine. The sign-in is not. The check that matches a visitor opens the page, types an email and a password, clicks, and asserts a text that exists only after sign-in. The account name, or the words "Log out", are the usual asserts. An assert on the word "Login" passes on the form you failed to leave.

Record that in the product by signing in as usual. Each click and each fill becomes a step. If a selector changes later, edit that step. You do not have to record the whole journey again. A failed run keeps a screenshot of the failing step. Open the run and read the step name before you change a selector from memory.

The login template, step by step

The built-in template id is login. Its steps are the ones below. The host example.com is a placeholder. applyTemplateStartUrl rewrites each goto onto the origin you pass, and keeps the template path.

{
  "steps": [
    { "op": "goto", "url": "https://example.com/login" },
    {
      "op": "wait_for",
      "selector": "input[type='email'], input[name='email']",
      "timeoutMs": 2000
    },
    {
      "op": "fill",
      "selector": "input[type='email'], input[name='email']",
      "value": "user@example.com"
    },
    {
      "op": "fill",
      "selector": "input[type='password']",
      "value": "ChangeMe123!"
    },
    {
      "op": "click",
      "selector": "button[type='submit'], button:has-text('Connexion'), button:has-text('Sign in')"
    },
    { "op": "assert_text", "text": "Mon compte" }
  ]
}

goto navigates with domcontentloaded, then waits up to five seconds for network idle and continues if that wait misses. After the navigation the runner tries known cookie-accept buttons for up to five seconds. That pass is a convenience. It is not a guarantee that your banner uses one of those buttons. Add your own step when it does not.

wait_for requires the selector to show. timeoutMs here is 2000, which is also the default when you omit it. The maximum you can set on this op is 60000. A login form that appears after a slow identity widget needs a higher timeout. A timeout of two seconds on a widget that takes four will fail on every run, and you will blame the password.

fill types into the selector. The value is part of the scenario. Replace user@example.com and ChangeMe123! before you save this against a real site. The template password is a placeholder in source control, not an account.

click tries the selector, then can fall back to visible text or an href when you recorded those. The template uses a CSS list: a submit button, or a button whose text is "Connexion" or "Sign in". A redesign that leaves the text and changes the element still matches the text fallback if you recorded text. Add retryTimes between 0 and 5 when the control mounts late. Retries are not a substitute for wait_for.

assert_text requires the string to be visible. If you also pass selector, the text must sit in that element, not in a hidden node of the document. ignoreCase and regex are optional. "Mon compte" is French because the template was written that way. Change it to the string your signed-in page actually shows. An assert that matches the login page is a green failure.

Creating this as a browser scenario is POST /v1/scenarios with type: "browser" and the steps under scenario. scenarios:write is the scope. enabled: false saves it without alerting until you turn it on. The same call takes Idempotency-Key, 8 to 128 characters from [A-Za-z0-9._:-], remembered for 24 hours.

{
  "type": "browser",
  "name": "Sign in",
  "intervalSec": 300,
  "severity": "critical",
  "tags": ["prod", "login"],
  "scenario": {
    "steps": [
      { "op": "goto", "url": "https://shop.example.com/login" },
      {
        "op": "wait_for",
        "selector": "input[type='email']",
        "timeoutMs": 8000
      },
      {
        "op": "fill",
        "selector": "input[type='email']",
        "value": "monitor@shop.example.com"
      },
      {
        "op": "fill",
        "selector": "input[type='password']",
        "value": "replace-with-the-test-password"
      },
      { "op": "click", "selector": "button[type='submit']" },
      { "op": "assert_text", "text": "Log out" }
    ]
  }
}

A read of that scenario does not return the steps. The list leaves the steps out because they carry login credentials. You get scenarioFingerprint instead. It changes when the steps change. That is enough to notice drift. It is not enough to recover the password. Keep the steps in your own private source if you create them through the API. Do not commit the test password next to production passwords.

A test account, never a real customer

The typed password is stored with the scenario. Create an account that exists only for this run. No admin rights. No customer data. A password used nowhere else. If the scenario leaks, that account opens nothing that matters. Do not record your own staff login because it was convenient on a Thursday.

Rotate that password on the site and in the scenario together. A fingerprint change on the next read tells you the steps moved. It does not tell you whether the site still accepts them. Run once after the rotation and read the screenshot if it fails.

basicAuth on the scenario is a different field. It is HTTP basic credentials for the browser context, not the form fill. Use it only when the site gates the whole host with basic auth. The form fill remains the login you are testing.

The cookie banner

Three mechanisms exist. Use the one that matches the banner. Do not stack all three and then wonder which click won.

After every goto, the runner calls a dismiss helper. It walks a list of known accept selectors for at most five seconds, clicks the first that is visible, and waits 200 milliseconds. If your banner is OneTrust-shaped, this often clears it before your steps run. If your banner is custom, it does nothing, and your next step must handle it.

The conditional step is if_visible. The runner looks for the selector for up to 2.5 seconds. If it is visible, the next step runs. If it is not visible, the next step is marked ok with the detail "skip (si…)" and is not executed. A missing banner must not fail the login.

{
  "steps": [
    { "op": "goto", "url": "https://shop.example.com/login" },
    {
      "op": "if_visible",
      "selector": "button:has-text('Accept all')"
    },
    {
      "op": "click",
      "selector": "button:has-text('Accept all')"
    },
    { "op": "wait_for", "selector": "input[type='email']", "timeoutMs": 8000 }
  ]
}

The click immediately after if_visible is the step that gets skipped. Do not put the email fill in that position. The fill would be skipped on a day the banner is absent, and the click on "Sign in" would land on an empty form.

There is also a template id cmp that waits 1500 milliseconds, clicks a list of accept buttons, and runs assert_no_cmp. That assert fails while a known overlay is still visible. Use it when the journey is "the banner goes away", not when the journey is "sign in". A login Flow can end with assert_no_cmp if a leftover overlay would hide the account menu. It should not replace assert_text of the signed-in page. An overlay can be gone on an error page.

if_text works the same way as if_visible, matching text in a selector or in the body, and skips the next step when the text is absent. Use it when the banner has no stable button selector and does have a stable sentence.

Captcha and one-time codes

A captcha or a one-time code exists to stop robots. This runner is a robot. Do not solve the captcha, do not read the mailbox, and do not replay a stolen cookie from a real user. Exempt the test account on your side, or recognise the monitoring traffic with a header you choose.

Headers live on the scenario, not on a single step. At most ten. Each name matches ^[A-Za-z0-9-]+$, is at most 64 characters, and must not be host, content-length, connection or transfer-encoding. The value is at most 500 characters. The runner copies them to Chromium's extra HTTP headers for every request of that run. You pick the name. There is no reserved Pathly header you can assume your site already trusts.

{
  "steps": [
    { "op": "goto", "url": "https://shop.example.com/login" }
  ],
  "headers": {
    "X-Shop-Monitor": "login-flow"
  }
}

On the site, skip the captcha when that header has the value you configured, and only for the test account. A header anyone can send is not an admin backdoor. Pair it with the test account, or with a network rule you already use for other monitors. If you cannot exempt the account, this Flow cannot pass the captcha, and it should not. Monitor the page up to the captcha with an assert that the widget rendered, and test the API behind it with a Chain from a network that your API allows.

Two-factor on the test account has the same rule. Disable it for that account, or use a method your policy allows a monitor to complete without a human. Do not store a colleague's phone number in a step.

Asserts that mean you are in

assert_text was the template's last step. Add assert_url when the signed-in page has a stable path.

{
  "op": "assert_url",
  "path": "/account",
  "ignoreQuery": true,
  "ignoreHash": true
}

includes matches a substring. regex is a pattern of at most 200 characters. Ignoring the query and the hash avoids a failure on a tracking parameter. Do not ignore the path. A redirect to /login?error=1 must fail.

assert_visible checks a selector and can retry. assert_amount checks a number in a selector against min, max or equals, with an optional currency. Use it on a cart, not on a login. wait_for_response waits for a URL substring and an optional status, which is how you tie the click to POST /session returning 200. wait_for_networkidle waits for the network to settle, up to the timeout you set, capped at 60 seconds.

{
  "op": "wait_for_response",
  "urlIncludes": "/session",
  "status": 200,
  "timeoutMs": 10000
}

Put that after the submit click, before the text assert. A 200 from /session plus the text "Log out" is the pair. The status alone can be a 200 error payload. The text alone can be left over in a cached layout. Together they are the sign-in.

press sends Enter, Escape, Tab, ArrowDown, ArrowUp or Space, optionally on a selector. Use Enter when the form submits from the password field and the button selector is unstable. You still want assert_text after it.

A click can set newTab when the control opens a popup. The runner switches to that page. switch_tab with urlIncludes comes back to a tab whose URL contains the string. Identity providers that bounce through a second window need this. A Ping never will.

Viewport, locale and timezone belong on the scenario next to steps when the mobile menu or the translated button is what you are proving.

{
  "viewport": "mobile",
  "locale": "en-US",
  "timezone": "Europe/Paris",
  "steps": []
}

mobile is a real viewport id. Desktop is the default, 1280 by 720. Record the size your customers use for that form. A banner that covers the button on a phone and not on a desktop will stay green if you only record the desktop.

When login is an API, use Chain

Flow sees the button. Chain sees the token. A front end that calls an auth API can break in either place. Chain does not open Chrome. One hop extracts a JSON field into a name. The next call uses it only where you write {{token}}, in the URL, the body, or an Authorization header. It is cheaper and it does not prove the button.

Hops are one to ten. Methods are the usual HTTP set. extractJson.path is at most 200 characters. extractJson.as is a name of at most 64 characters, starting with a letter or underscore.

import { PathlyClient } from "@pathly/sdk";

const pathly = new PathlyClient({ token: process.env.PATHLY_API_TOKEN! });

await pathly.createChain({
  name: "Login then /me",
  intervalSec: 300,
  severity: "critical",
  hops: [
    {
      name: "Login",
      method: "POST",
      url: "https://api.example.com/login",
      body: JSON.stringify({
        email: process.env.API_USER,
        password: process.env.API_PASSWORD,
      }),
      assertStatus: 200,
      extractJson: { path: "token", as: "token" },
    },
    {
      name: "Me",
      method: "GET",
      url: "https://api.example.com/me",
      headers: { Authorization: "Bearer {{token}}" },
      assertStatus: 200,
    },
  ],
});

extractCookie on a hop stores a cookie name instead of a JSON path, when the session is a cookie rather than a bearer token. waitMs pauses up to 30 seconds before the next hop. expectText fails the hop when the body does not contain the string. Put the test password in the CI secret, as API_PASSWORD above, not in a screenshot of this article.

Keep the Flow for the button and the Chain for the token when both are on the critical path. A green Chain with a broken submit button is the outage this article started from. A green Flow with a token your mobile app cannot refresh is the outage the Chain is for.

POST /v1/scenarios/{id}/run with scope runs:trigger queues a run and returns jobId. Poll GET /v1/runs?scenarioId= with runs:read. Finished statuses are ok, fail and error. The browser cap is two minutes unless that account was given a higher one, up to ten minutes. A poll that waits longer than the cap will only see the timeout.

The GitHub Action pathlyhq/pathly-action@v1 can run-scenario or assert-scenario on the id you saved. Assert reads the last status and does not start a browser. Use assert in a pull request when you only need to know the scheduled Flow is currently green. Use run after a deploy when you need a fresh sign-in. The token scopes are scenarios:read and runs:read, plus runs:trigger for the run operation.

What not to put in the steps

Do not put a production admin password in fill. Do not put a card number in a checkout Flow. Stop the journey before payment, on a text that proves the payment page rendered, unless you have a payment provider test card that cannot move money. Do not add a step that clicks "delete account". The runner will click it on schedule.

Fifty steps is the maximum. A journey that long is two journeys. The default click delay of two seconds times a long click list will also hit the two-minute cap. Shorten the path to the assert that matters.

Selectors are at most 500 characters. Prefer an id or a name. A selector that is a copy of the entire form will break on the next class hash. text and href on a click are the fallbacks for pages that have neither.

When the run fails, the step result carries the op, the duration, the page URL, and the screenshot of that step. Fix the selector in the list. Run again. A Ping on the same host can stay green the whole time. That is not a contradiction. They are different instruments.

Viewports you can name

viewport is one of desktop, tablet, mobile, iphone_se, iphone_14, pixel_7, galaxy_s21, ipad_mini. desktop is 1280 by 720 and it is the default when the field is absent. tablet is 768 by 1024. Pick mobile or a named phone when the login form collapses into a menu. Record and replay the same id. A journey recorded on desktop and replayed on iphone_14 will miss buttons that moved. Set the id in the scenario, do not rely on the recorder window being maximised.

Locale and timezone are strings on the same object. locale is at most 16 characters. timezone is at most 64. fr-FR and Europe/Paris are what you get if you omit them. An English submit button will not match button:has-text('Connexion') on an en-US locale. Align the assert text, the click text, and the locale. A mismatch here looks like a product outage and is a configuration error.

A scheduled run versus a run you start

intervalSec: 300 checks the login every five minutes and spends browser time on each pass. A login that only needs to be proven after a deploy can stay on a long interval, and the pipeline calls POST /v1/scenarios/{id}/run when the deploy finishes. Plans do not share the same Flow quota. Do not copy a five-minute interval from this example onto every journey because the JSON looked tidy. Browser minutes are the meter. A Ping on the same URL is the cheap signal that the host answers. The Flow is the expensive signal that a person can sign in.

The GitHub Action below starts one fresh run and fails the job if it does not finish ok. The id is a variable, not a secret. The token is a secret.

- name: Pathly sign-in
  uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_API_TOKEN }}
    operation: run-scenario
    scenario_id: ${{ vars.PATHLY_LOGIN_SCENARIO_ID }}
    timeout_sec: "120"

timeout_sec of 120 matches the default browser cap. Raising the action timeout above the scenario cap does not give the browser more time. assert-scenario is the other operation. It fails the job when the last status is not ok, and it does not start a browser. Use it on pull requests. Use run-scenario after the deploy.

Reading a failure without guessing

The step list in the run is the order the runner executed, including a skipped step after if_visible. A skipped step is ok and its detail says it was skipped. Do not "fix" a skip by deleting if_visible. The skip is the banner being absent, which is the success case for that pair.

A failed fill usually means the selector matched nothing, or matched an element that cannot be typed into. A failed click after a successful fill often means a banner appeared late and covered the button. Put if_visible and the accept click after goto and before wait_for on the email field, not after the fill. A failed assert_text with a screenshot of the login form means the click did not navigate. Look at wait_for_response for /session. A 401 or a timeout there is the password or the exempt rule, not the CSS.

assert_no_cmp fails while a known overlay is still visible. That is a banner the dismiss list did not click. Add an explicit click. Do not lengthen the goto and hope.

Console messages and a short network list can be attached to the step when the runner captured them. They explain a script that did not load. They are not a second monitoring product. The screenshot is still the first thing to open.

What the API will not echo back

Create the scenario once, store the id, and stop sending the password on every deploy. A later GET does not include the steps. scenarioFingerprint changes when they change. If you need to rotate the test password, patch the scenario from a secret store, then run once. Do not print the patch body in the job log. The fingerprint moving, plus a green run, is the confirmation.

enabled: false on create lets you save the steps and read a fingerprint before any alert goes out. Patch enabled to true after the first manual run is green. An enabled scenario with a wrong assert will page you every interval. That is the correct behaviour, and it is a bad first hour.

Headers you set for the captcha exemption are part of the scenario object too. Rotate the header value the same way you rotate the password, on the site and in the scenario together. A header that has been in a ticket and a screenshot is not a secret. Use a fresh value per environment.

This is the whole login check. A test account, a goto, a banner step that is allowed to skip, two fills, a click, a response wait, and a text that only exists once signed in. Ping stays on the URL. Chain stays on the token. Flow is the walk between them.

Pathly as an API, and a Terraform provider

Simon Raynaud — Wed, 14 Oct 2026 18:07:15 +0000

Pathly watches the path. Ping if it's up. Flow if a user can walk it. Chain if the services behind it hold. This article describes the HTTP contract those three share, and why a Terraform provider can sit on it without a second source of truth. The developer pages, including the contract they render, are on the Pathly developer docs.

A Ping is one HTTP check on a timer. A Flow is a Chrome session with steps. A Chain is several HTTP calls in order, on the same scenario resource, in the httpChain field. The provider declares the Ping, the availability target, the maintenance window and the webhook. It does not invent a parallel API.

The published contract is the one that runs

The API serves its OpenAPI document at a public address, with no authentication. The document is not written beside the code. It is produced from the table that declares the routes. That same table decides, on every call, which scope is required. A route that is missing from the table does not start. The developer page reads the contract when it renders, so a removed endpoint disappears from the documentation. A hand-copied page drifts from the route table.

Each operation carries x-pathly-scope. A generator, or a human reading the JSON, can see the scope without opening the handler. The public document is also tagged with x-pathly-language for the locale of the text. The routes do not change with the language. The sentences do.

GET /v1/scenarios requires scenarios:read. POST /v1/scenarios requires scenarios:write. POST /v1/scenarios/{id}/run requires runs:trigger. Those three are the split you feel first. Listing is not creating. Creating is not executing. An execution spends quota. A read does not.

The create call says what each type needs. type: http requires url or httpChain. type: browser requires scenario. enabled: false prepares a scenario without raising an alert. The same call accepts an Idempotency-Key header so a retry does not create a second row.

A minimal Ping body, using a host reserved for examples:

{
  "type": "http",
  "name": "Home",
  "url": "https://www.example.com",
  "intervalSec": 300,
  "expectedStatus": 200,
  "maxLatencyMs": 2000,
  "severity": "major",
  "tags": ["prod"]
}

intervalSec is seconds between checks. maxLatencyMs fails the run when the response is slower than that, even if the status matches. severity is critical, major or minor. Tags are the labels a Terraform data source can filter on later. If you omit expectedStatus, the server stores 200. If you omit maxLatencyMs, it stores 5000. Send the values you mean.

Narrow scopes, and four refusals

A key carries scopes shaped as resource then action. scenarios:read lists and reads. scenarios:write creates, updates, deletes, mutes and sets a baseline, and it includes read, because a tool always re-reads what it just wrote. runs:read lists runs. runs:trigger queues a run and includes runs:read. alerting:write includes alerting:read. maintenance:write includes read. sla:write includes read. org:read is only read. Writing does not include triggering. A provisioner that creates checks should not hold runs:trigger unless that same job also launches them.

Four actions stay out of every key, by decision.

A key cannot issue or revoke a key. A key that could would cancel the scopes of every other key.

A key cannot invite a member. The new account would issue whatever keys it wants. That is the same escalation one hop later.

A key cannot change plan or touch billing. A continuous integration token must not start a recurring charge.

A key cannot export the organisation. The export contains members, so it contains personal data, and it does not belong in a job log.

The official TypeScript client does not widen any of this. A missing scope is a 403 whose message names the scope. Create the key with the minimum you need. A leaked provisioner token still cannot become an organisation takeover, because the four actions above are not scopes you can grant.

Keys are minted in the product settings, by a person. Copy the value once. It starts with sp_. The client rejects anything else before the first request, so a truncated paste fails in your process instead of as a 401 that looks like a revoked key.

An id from another organisation is a 404

If the identifier belongs to another organisation, the API answers 404, never 403. A 403 would confirm that the id exists. A 404 is the same answer as a deleted id or a typo. Log the status. Do not branch on "forbidden" to decide that a scenario is real and hidden. Your code should treat 404 as "not in this org." Recreate only when you are sure the id was yours and was deleted.

The SDK exposes that as PathlyApiError with notFound.

import { PathlyClient, PathlyApiError } from "@pathly/sdk";

const pathly = new PathlyClient({ token: process.env.PATHLY_API_TOKEN! });

try {
  await pathly.getScenario("mon-unknown");
} catch (e) {
  if (e instanceof PathlyApiError && e.notFound) {
    // Gone from this org. Create it again, or fix the id.
  }
  throw e;
}

status, path, the API message, retryAfter and validation details are on the same error. A 400 puts the field errors in details. Do not retry a 400. The body will be refused again, and you will spend rate limit on a typo.

Idempotency for twenty-four hours

Writes accept Idempotency-Key. The key must match ^[A-Za-z0-9._:-]{8,128}$. Shorter than 8 characters is ignored, which means the call is not replayable. A missing header is the same. A key with a space or a slash in it is ignored, so that call is not replayable. Spaces around an otherwise valid key are trimmed. Use a job id or a UUID.

The server stores the path, a hash of the body, and the response. The row counts for 24 hours. The same key and the same body return the original response, including the original id. The same key and a different body return 409 with the message that the key was already used with another body. A pipeline that retries after a dropped response must send the same bytes, not a "fixed" name.

The SDK attaches Idempotency-Key on writes. runScenario opts out (idempotent: false), because two triggers are two runs, and replaying the previous verdict would hide the second one. The default key function builds a new id each time it is called, and the client calls it on each attempt. A transport retry inside one process can therefore send a new key. When the retry is your pipeline starting over, pass idempotencyKey so it returns the same job id for that deploy. A fresh random key creates a second scenario. The option exists for that stable id. It does not invent one for you.

curl -sS -X POST "https://api.pathlyhq.com/v1/scenarios" \
  -H "Authorization: Bearer $PATHLY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: deploy-2026-10-14-home" \
  -d '{"type":"http","name":"Home","url":"https://www.example.com","intervalSec":300,"expectedStatus":200,"severity":"major"}'

Run that twice with the same body. You get one scenario. Change name and keep the key. You get 409. Wait, or use a new key, when the body is intentionally new.

POST /v1/scenarios/{id}/run is not a create of a configuration object. It queues a run and returns jobId. It spends quota. Give it runs:trigger and do not confuse it with idempotent create. Triggering twice is two runs.

Cursor pages, not page numbers

List endpoints take limit and cursor. The default page size is in the schema, not only in a sentence, so a generator can read it. The next page is the nextCursor value from the previous response. The last page omits it.

Page numbers break when someone creates a row while you are walking. Later pages shift, and you either read a row twice or skip one. A cursor does not. An integration that stops after the first page believes it knows the estate, and then deletes what it never saw.

for await (const scenario of pathly.paginate((cursor) =>
  pathly.listScenarios({ cursor }),
)) {
  console.log(scenario.id, scenario.name);
}

Pass the cursor through. Do not request limit so high that you think one page is the world. When you compare this list to a Terraform configuration, you are doing the same job as the pathly_scenarios data source. Filter by tag in your own code if you only care about one environment. The data source's filter_tag is that filter for HCL.

Retry-After, and what is never retried

When the rate limit is hit, the API sends Retry-After. Without that header an automated client retries on its own clock and stays rate-limited. The SDK honours the header, caps the wait at 90 seconds, and stops after four attempts. If the header is absent on a 429 or a 5xx, the wait is attempt * 500 milliseconds. A 400 or a 403 is thrown immediately. Both would fail the same way on the next try.

The client also refuses a base URL that is not https, except http://localhost or http://127.0.0.1. The key is in the Authorization header. Clear text to a remote host is readable by every intermediary. baseUrl defaults to https://api.pathlyhq.com. Override it only for a local server.

userAgent defaults to pathly-sdk-ts. Set it to the name of your integration so API logs can tell your job from someone else's.

What a read never returns

An authentication password, a webhook address, and the steps of a Flow are written and not returned. In their place the API returns a fingerprint. For a webhook it is twelve characters, and it changes when the address changes. For a browser scenario the field is scenarioFingerprint, and it changes when the steps change. The list description says the Flow steps are not returned, because they carry login credentials. The fingerprint is enough to see drift. It is not enough to rebuild the secret.

Terraform uses that for the webhook. The destination edited by hand in the console shows on the next plan, and the address is not stored as a read-back attribute you could print. You still keep the URL in your own file, because the API will not remind you of it. The fingerprint only tells you it moved.

Do not log a create body that contains a Flow password or a Chain body with a test password. The request you sent is the copy. The response is the safe one.

A Chain on the same route

Chain is not a second product hostname. createChain posts type: http to /v1/scenarios and sets httpChain to the hops. The first hop's URL is also the scenario URL. At least one hop is required. The client rejects an empty list before the request, with a 400-shaped error, so you do not spend a round trip on it.

const chain = await pathly.createChain({
  name: "Login then /me",
  intervalSec: 300,
  hops: [
    {
      name: "Login",
      method: "POST",
      url: "https://api.example.com/login",
      body: JSON.stringify({
        email: process.env.API_USER,
        password: process.env.API_PASSWORD,
      }),
      assertStatus: 200,
      extractJson: { path: "token", as: "token" },
    },
    {
      name: "Me",
      method: "GET",
      url: "https://api.example.com/me",
      assertStatus: 200,
    },
  ],
});

Each hop has a method in GET, POST, PUT, PATCH, HEAD, DELETE, a URL, an optional name, optional headers, an optional body up to 8000 characters, an optional waitMs up to 30000, an optional assertStatus, an optional expectText, an optional extractJson and an optional extractCookie. extractJson.as must match ^[A-Za-z_][A-Za-z0-9_]*$ and is at most 64 characters. The path is at most 200 characters. That extracted name is how the next hop can use the token. A Chain sees a broken token. It does not see a broken button. Flow is the Chrome run for the button. Use both when both can fail.

Ten hops is the maximum on the schema. A chain longer than that is two scenarios, not one hop array you hope the server will truncate.

Trigger, then read the run

runs:trigger queues. It does not wait. listRuns with scenarioId and limit: 1 is how you see the latest. Status ok passes. fail and error are finished and not ok. Anything else is still running.

await pathly.runScenario(chain.id);
const started = Date.now();
for (;;) {
  const { items } = await pathly.listRuns({ scenarioId: chain.id, limit: 1 });
  const run = items[0];
  if (run && (run.status === "ok" || run.status === "fail" || run.status === "error")) {
    if (run.status !== "ok") throw new Error(`Pathly ${run.status}`);
    break;
  }
  if (Date.now() - started > 120_000) throw new Error("Pathly run timeout");
  await new Promise((r) => setTimeout(r, 3000));
}

A browser run's default cap is two minutes, which is why this loop stops at 120 seconds. A Ping is usually faster. Do not poll tighter than a few seconds. You will hit the same rate limit the SDK is trying to back off from.

For a pipeline, the published GitHub Action pathlyhq/pathly-action at v1 already implements assert and run-and-wait. Prefer it to a hand-rolled poll when the job is GitHub Actions. The snippet above is the same idea when the caller is the SDK. Scopes for the action's assert path are scenarios:read and runs:read. Add runs:trigger only for run-scenario. Add scenarios:write only for ensure-scenario.

Mute and baseline are writes, not runs

POST /v1/scenarios/{id}/mute takes mutedUntil as a date-time, or null to wake the scenario. The deadline is capped at 30 days. Mute is scenarios:write. It does not spend a run. Use it for a known bad deploy, and wake it. A mute that nobody wakes leaves a failing scenario silent.

POST /v1/scenarios/{id}/baseline sets the comparison baseline to the last successful run. Same scope. It does not change the URL or the interval. A patch does that.

{ "intervalSec": 600, "maxLatencyMs": 3000, "severity": "critical" }

That patch body is the in-place update. The id stays. History stays. A client that implements "change the URL" as delete plus create will drop incidents. The provider's rule is the same rule. After apply, the next plan is empty, and a changed interval must not force a new id.

How the provider sits on this

Four resources, and data sources that re-read the inventory. pathly_scenario is this create and patch. pathly_sla_target is the SLA routes under sla:write. pathly_maintenance_window is maintenance:write. pathly_webhook is alerting:write, with events run.failed and run.recovered only. The data source pathly_scenarios is GET /v1/scenarios plus a tag filter. API keys and members are absent because the API refuses those actions.

The provider block stays empty. PATHLY_API_TOKEN is the same sp_ key. Version ~> 0.1 while the schema is 0.x. Import by id when the scenario already exists. The fingerprint fields are how plan sees a webhook URL or a Flow's steps move without the API echoing the secret.

If you are writing a language the SDK does not ship, you do not wait for a provider either. One key, https://api.pathlyhq.com, and the OpenAPI document are enough. Generate the client from that document, or call the routes in this article. Keep the idempotency key, follow nextCursor, honour Retry-After, and treat 404 as "not in this org."

Page size, order, and a client that stops too soon

The default page size is 100. The maximum is 500. Both numbers are in the pagination schema the contract publishes, so a generated client can read them instead of hard-coding a guess. Asking for more than 500 is clamped to 500. Asking for 500 on every call because you can will make each response heavy, and you will hit Retry-After sooner. 100 is enough for a small org. A large org still needs the cursor.

Order is created_at DESC, id DESC. The timestamp alone is not a cursor. Two rows created in the same millisecond are tied, and the id breaks the tie. If you sort again on the client by name, do it after you have walked every page. Sorting one page and then fetching the next mixes two orders and hides gaps.

A safe walk stores the last nextCursor you successfully finished. If the process dies, resume from that cursor. Starting over from the first page is correct only when you are building a full set in memory and you discard the partial one. Starting over and merging by name will double-count a scenario that was renamed between the two walks.

paginate on the SDK is that walk. It stops when nextCursor is absent. Do not add a counter "just in case" that breaks after three pages. The day the estate passes 300 scenarios, the counter is the bug.

A provisioner key and a pipeline key are not the same key

Split them.

The Terraform key needs scenarios:write, sla:write, maintenance:write and alerting:write. Those include the matching reads. It does not need runs:trigger. Apply should not launch Chrome because a plan was saved.

The deploy key needs runs:trigger if the job starts a run, and runs:read if it only asserts the last status. runs:trigger already includes runs:read. It does not need scenarios:write unless the job also creates checks. ensure-scenario on the GitHub Action is the operation that needs scenarios:write. assert-scenario does not.

org:read is for usage and org surfaces. The smoke configuration leaves it off so a data source that reaches outside the managed resources fails closed. Copy that. A key that can read every member list is not required to declare a Ping.

Neither key can invite a teammate or change the plan. If a runbook says "the CI token can also add an admin," the runbook is wrong about this API.

What a 409 is, and what a 429 is

409 on create means the Idempotency-Key was already stored with a different body hash, inside the last 24 hours, for this organisation. Fix the body or change the key. Retrying the same wrong body keeps the 409. The SDK will not retry it, because it is not a 429 and not a 5xx.

429 means slow down. Read Retry-After. The SDK waits that many seconds, and never more than 90. Your own curl loop should do the same. A loop with no sleep will stay at 429 until the job times out, and the scenario will not exist.

5xx is the other retry. Four attempts, then the error. If the first attempt actually created the row and the response was lost, a stable idempotency key is what returns the original id on the next attempt. A new key on each attempt cannot do that. That is the whole reason to pass a job id.

400 is a body the schema refused. details names the fields. Common causes on a scenario are a URL that is not a URL, an intervalSec of the wrong type, a hop list that is empty, an extractJson.as that starts with a digit, or more than ten headers. Fix the JSON. Do not rotate the key.

401 is a missing or revoked token, or a token that does not start with sp_ and never reached the API because the client refused it locally. 403 names the scope. Those two are not interchangeable. Adding scopes does not fix a revoked token. Minting a new token does not fix a 403 if you copy the same scopes.

Patch versus replace, in one call

await pathly.updateScenario(id, {
  intervalSec: 600,
  maxLatencyMs: 3000,
  severity: "critical",
});

updateScenario is PATCH /v1/scenarios/{id}. The id in the path is the id you keep in state. Fields you omit stay as they are. A client that sends the full object on every sync must still send the same URL and the same name, or it will change them. Terraform's empty-plan rule is this patch behaving as a no-op when the file matches the read-back.

Delete is DELETE /v1/scenarios/{id}, also scenarios:write. The SDK returns an empty object when the body is empty, so the caller does not dereference null. Deleting a scenario deletes the history you were keeping. Import and patch instead, unless the check is actually gone.

Mute, when you need silence without a delete:

await pathly.muteScenario(id, "2026-10-15T06:00:00.000Z");
await pathly.muteScenario(id, null);

The first call silences until that instant, and the API caps the horizon at 30 days. The second call wakes it. Leave a comment in the job with the incident id. A mute left in place keeps a failing scenario silent.

The Pathly Terraform provider, in practice

Simon Raynaud — Wed, 14 Oct 2026 18:07:11 +0000

Pathly watches the path. Ping if it's up. Flow if a user can walk it. Chain if the services behind it hold. This article is about the Terraform provider that declares the HTTP check, the availability target, the maintenance window and the webhook. The browser journey and the HTTP chain are real products, and they are not what this file creates. If you want the contract those resources call, read the API article. The pages a developer actually uses live on the Pathly developer docs.

The provider is published as pathlyhq/pathly on the public Terraform Registry. You do not compile it and you do not host a mirror. A version constraint plus terraform init is the install. The schema is still 0.x. That is a precise statement, not a shrug. Existing resources will not move without a reason, and a schema field can still change before 1.0. Pin ~> 0.1 so a later 0.2 does not land in a pipeline that never asked for it.

The smallest file that does something

terraform {
  required_version = ">= 1.6"
  required_providers {
    pathly = {
      source  = "pathlyhq/pathly"
      version = "~> 0.1"
    }
  }
}

provider "pathly" {}

resource "pathly_scenario" "site" {
  name         = "Home page"
  url          = "https://shop.example.com"
  interval_sec = 300
  expect_text  = "Our products"
  tags         = ["prod"]
}

The provider block is empty on purpose. The API key is read from the environment variable PATHLY_API_TOKEN. If you pass that key through a Terraform variable, the value is written in clear into the state file. State is copied, backed up and sometimes committed by mistake. The environment variable stays out of the plan. The provider also refuses plain http toward anything except localhost, because the key travels in an Authorization header and every proxy on a clear-text hop can read it.

pathly_scenario with a url is a Ping. It requests that address on a timer. It does not open Chrome, it does not click, and it does not call a second URL. interval_sec = 300 means one check every five minutes. expect_text is the line that separates a live server from a live site. An application error page is very happy to answer HTTP 200. The expected text is what exposes it. tags are free labels. Later a data source filters on one of them, which is how you find checks that exist in the console and not in this file.

shop.example.com is a documentation host. Use your real origin when you apply this. Do not point a production check at a host you do not control.

What the CI smoke actually applies

The repository ships a smoke configuration that creates every managed resource, reads it back, and destroys it. It is the file to copy when you want the shapes, not a file to apply against a customer site. The identifier of the run is a variable so two applies do not collide.

variable "smoke_id" {
  description = "Unique id for this CI pass, so two applies do not collide."
  type        = string
}

locals {
  tag = "pathly-tf-smoke"
}

resource "pathly_scenario" "api_health" {
  name            = "CI smoke API health ${var.smoke_id}"
  url             = "https://api.pathlyhq.com/health"
  interval_sec    = 86400
  expect_text     = "pathly-api"
  expected_status = 200
  severity        = "critical"
  tags            = [local.tag, "ci"]
  runbook         = "https://pathlyhq.com/en/developers"
}

interval_sec = 86400 is one check a day. That is a long interval so a CI smoke does not poll all day, and it is the wrong interval for a checkout. A Ping does not spend the browser budget. That budget is Flow. A public home page usually wants 300 seconds. A health URL that only proves the process is up can wait longer. expected_status = 200 fails the check when the origin answers anything else. expect_text = "pathly-api" fails it when the status is 200 and the body is the wrong service. Both arguments belong together. Status alone is how a parking page stays green.

severity is critical, major or minor. It does not change the HTTP request. It changes how loud the failure is, and it is the predicate the SLA target below uses to decide which journeys get an objective. runbook is a URL shown with the alert. Put the page your on-call already trusts. Do not put a secret, a ticket id that contains a customer name, or a link that only works on a VPN you will have forgotten in a year.

The name includes smoke_id because Terraform names are labels for humans, not primary keys. Two smokes with the same name and different ids are two resources. The API id, not the name, is what terraform import uses later.

An availability target on that Ping

A Ping that fails is an incident. An SLA target is the budget you are willing to spend before the month is considered broken. The smoke attaches one target to the scenario it just created.

resource "pathly_sla_target" "api_health" {
  scenario_id          = pathly_scenario.api_health.id
  name                 = "CI smoke SLA ${var.smoke_id}"
  objective_pct        = 99.0
  window_days          = 7
  exclude_maintenance  = true
  warn_at_budget_ratio = 0.8
}

scenario_id is the id Terraform just received. Do not paste a UUID from the console into this block unless you are importing. A pasted id from another workspace is how a plan updates the wrong check.

objective_pct = 99.0 over window_days = 7 is a loose objective for a smoke. A payment path in production is usually tighter and longer, for example 99.9 over 30 days. The percentage is not a marketing claim about Pathly. It is the number you asked the product to measure on your own runs.

exclude_maintenance = true only helps if the maintenance window is also in this configuration. A window declared in the console, and a target that ignores it, spends the error budget on an outage you announced. The alert then punishes a planned job. warn_at_budget_ratio = 0.8 fires when eight tenths of that budget is already gone, while there is still time to stop shipping risky changes. Waiting for 100 percent means the warning and the breach arrive together.

On a real set of Pings, attach the target only to the critical ones. A marketing page that is allowed to be briefly red should not share the payment objective.

resource "pathly_sla_target" "objective" {
  for_each = {
    for key, j in var.journeys : key => j if j.severity == "critical"
  }

  scenario_id          = pathly_scenario.journey[each.key].id
  objective_pct        = 99.9
  window_days          = 30
  exclude_maintenance  = true
  warn_at_budget_ratio = 0.8
}

The for_each filter is ordinary Terraform. Rows whose severity is not critical get a Ping and no objective. That is a product decision expressed as data, which is the reason to leave the console.

A maintenance window that does not mute the whole organisation

The smoke window is scoped to one scenario. Sunday, starting at minute 180 of the day, for 30 minutes. Minute 180 is 03:00. Weekday 7 is Sunday in this schema.

resource "pathly_maintenance_window" "weekly" {
  scenario_id  = pathly_scenario.api_health.id
  weekday      = 7
  start_minute = 180
  duration_min = 30
  reason       = "CI smoke weekly ${var.smoke_id}"
}

scenario_id matters. Without it, a window can cover more than the check you meant to silence. The smoke sets the id so a CI pass does not mute every Ping in the organisation for half an hour. reason is the sentence a human reads on Monday. Write the job name, not "test".

A weekly backup that should not spend the error budget looks like this. It is the shape from the practice article, and it is not scoped in that example, so copy the scenario_id line from the smoke when the backup belongs to one journey.

resource "pathly_maintenance_window" "backup" {
  weekday      = 7
  start_minute = 3 * 60
  duration_min = 120
  reason       = "Weekly backup"
}

start_minute is minutes from midnight in the window's clock, not a cron expression. 3 * 60 is the same integer as 180. Prefer the multiplication when the hour is the thing you want a reviewer to see. duration_min = 120 is two hours. A duration that covers the whole night will also cover incidents you wanted to know about. Keep it as long as the job and no longer.

Failures inside the window do not spend the error budget when the target has exclude_maintenance = true. They can still be visible in the run history. Maintenance is not a delete. When the window ends, the next failed run alerts again.

The webhook, and the URL that never comes back

resource "pathly_webhook" "alerts" {
  url    = "https://example.com/hooks/pathly-smoke/${var.smoke_id}"
  events = ["run.failed", "run.recovered"]
}

Those two event names are the set the API accepts on this resource: a run failed, and a run recovered. There is no third event to subscribe to. If your receiver only handles failure, pass a one-element list. An empty list is not a useful webhook.

The URL is written on create and on update. A later read does not return it. The API returns a twelve-character fingerprint instead. The fingerprint changes when the address changes. It is not enough to reconstruct the address, and it is enough for the next plan to show that someone edited the destination in the console.

output "webhook_fingerprint" {
  value = pathly_webhook.alerts.url_fingerprint
}

Treat a fingerprint change as drift. Do not try to reverse it. Put the real receiver URL in the configuration, in a private module or a private variable file, and let Terraform set it. The smoke uses example.com because that name is reserved for documentation. A real hook must be HTTPS on a host you operate.

Deliveries are signed. The signature header is X-Pathly-Signature and the value starts with sha256=. Verify that header with the webhook secret before you trust the body. Do not log the secret next to the payload. The secret stays in state on the machine that applied the configuration. That is another reason state belongs in an encrypted backend, not in git.

A Ping going red is run.failed. The same Ping going green again is run.recovered. Your receiver should close or resolve on the second event, otherwise the incident stays open after the site is back.

An estate as data

Past three or four checks, one block per resource becomes a list you re-read on every addition. A variable plus for_each makes a new journey one row.

resource "pathly_scenario" "journey" {
  for_each = var.journeys

  name            = each.value.name
  url             = each.value.url
  interval_sec    = each.value.interval_sec
  expect_text     = each.value.expect_text
  max_latency_ms  = each.value.max_latency_ms
  expected_status = 200
  severity        = each.value.severity
  tags            = concat(["terraform", var.environment], each.value.tags)
}

max_latency_ms is per journey because a cached home page and a payment page do not share a budget. A threshold copied from the home page will page you for a checkout that was always slower, and you will mute it. expected_status = 200 is fixed in the loop on purpose. If one URL is allowed to answer 204, give it its own resource instead of a nullable field half the team will forget.

concat puts a common tag, the environment, and the row's own tags on every check. The common tag is what the data source filters on. If each row invents its own tag and nothing is shared, the drift check in the next section cannot see them as one set.

Keep URLs, names and thresholds in the variable. Keep the rules in the resource. A reviewer then diffs a row, not a copy of the whole block.

Spotting what was created by hand

The useful data source is the one that lists scenarios the API can see, then subtracts the ids this configuration owns.

data "pathly_scenarios" "prod" {
  filter_tag = var.environment
  depends_on = [pathly_scenario.journey]
}

output "scenarios_outside_terraform" {
  value = [
    for s in data.pathly_scenarios.prod.scenarios : s.name
    if !contains([for j in pathly_scenario.journey : j.id], s.id)
  ]
}

filter_tag limits the list to one environment. Without it you compare production code to every sandbox check in the org and the output is noise. depends_on matters on the first apply, because the data source can be read before the resource it should see has been created. After that, Terraform usually orders them by the reference in the output. Leave the dependency in. A green first apply is the one people trust.

A name in that output is a Ping somebody created in the console. Nobody reviews it in pull requests, and nobody deletes it when the environment is destroyed. Import it, or delete it on purpose. Do not ignore the list until it is long.

The smoke checks the same idea with four booleans. Each created id must appear in the matching data source: scenarios filtered by tag, all SLA targets, maintenance windows filtered by scenario id, and all webhooks. pathly_sla reads the measured budget for that scenario. On the first run the measure can be empty. The smoke only requires the read to succeed.

Org-wide reads are absent from that smoke on purpose. The CI key is scoped to the resources it manages. It does not have org:read. A data source that calls a route the key cannot read fails the apply. That failure is the scope working.

Import, then an empty plan

You do not rebuild monitoring to move it into code. Each resource imports by the id you already have. For a scenario, that id is the one in the console address.

terraform import pathly_scenario.checkout mon_01H8ZK

Replace mon_01H8ZK with the real id. After the import, terraform plan shows the gap between the live object and the file. While the plan is not empty, apply changes the existing scenario in place. It does not create a second one beside it. A recreate would drop history and incidents, which is the record you were keeping.

The same import exists for the SLA target, the maintenance window and the webhook. Import the scenario first, because the others store its id.

The usability test is one sentence. After a successful apply, the next plan is empty. A provider that proposes a change on every pass teaches people to stop reading plans. The day the plan is true, nobody sees it. In-place update of the URL, the interval and the latency threshold is what makes that empty plan possible. Those fields are readable back as the same values you wrote. The webhook URL is the exception, and it is represented by the fingerprint so a real change still shows up.

If a plan wants to replace a scenario after you only changed interval_sec, stop. Do not apply a replacement to accept the diff. That is the bug the empty-plan rule is there to catch.

What this provider will never declare

Four actions stay out of a key, and therefore out of HCL.

An API key cannot issue or revoke API keys. A key that could do that would cancel the scopes of every other key. A configuration file must not be able to widen itself.

An API key cannot invite a member. The new account would then issue whatever keys it wants. That is the same escalation one step later.

An API key cannot change plan or touch billing. A continuous integration token must not start a recurring charge.

An API key cannot export the organisation. The export contains members, which is personal data, and it does not belong in a plan log.

The provider follows those refusals. You will not find a pathly_api_key or a pathly_member resource. If a module offers one, it is not this provider.

The token you export as PATHLY_API_TOKEN still needs the scopes of the resources you declare. Writing scenarios needs scenarios:write, which includes read. An SLA target needs sla:write. A maintenance window needs maintenance:write. A webhook needs alerting:write. Triggering a run is a different scope, runs:trigger, and this file does not trigger runs. Do not add it because it was on a checklist.

Flow and Chain stay out of this file

A Ping answers "did this URL respond the way the file says." A Flow is a real Chrome session: click, type, assert a text that only exists once a person is signed in. A Chain is several HTTP calls in order, where one hop can extract a token for the next. Do not invent a pathly_flow resource. Flow is steps on pathly_scenario, with type = "browser". Chain is http_chain on the same resource. This file only declares the Ping.

Changing a Ping's address in this file updates that Ping. It does not migrate a Flow's recorded steps. Those steps are not returned on read, because a fill or an http_auth step can contain a password. State can still hold them, marked sensitive, and scenario_fingerprint shows a console edit. This file stays on the Ping fields so the first example has nothing write-only in it.

Apply order on a new workspace

terraform init downloads pathlyhq/pathly at the 0.1 line you pinned. terraform plan should show four adds the first time: scenario, SLA target, maintenance window, webhook. terraform apply creates them in dependency order because the target, the window and any output reference the scenario id. terraform plan a second time should show no changes. Then read scenarios_outside_terraform. An empty list means the tag you filtered on has no console-only checks. A non-empty list is the work queue.

Store state in a backend you already encrypt. Do not commit terraform.tfstate. The webhook secret and the fingerprint live there. The API key should not, as long as it came from PATHLY_API_TOKEN and not from a variable.

When you rotate the key, export the new value and re-run the plan. An empty plan means the rotation did not touch the resources. A 401 means the new key is missing a scope the old one had. Add the scope. Do not fall back to a key that can also invite members, because no such key exists, and a personal session token does not belong in CI either.

This is the whole loop. One provider block, one Ping that checks text, one objective that excludes maintenance you also declared, one window, one webhook whose URL you keep in the file because the API will not remind you of it, and a data source that lists everything else.

The outputs the smoke keeps

The smoke publishes the four ids it created, the webhook fingerprint, and one boolean that is true only when every created id was read back.

output "created_ids" {
  value = {
    scenario           = pathly_scenario.api_health.id
    sla_target         = pathly_sla_target.api_health.id
    maintenance_window = pathly_maintenance_window.weekly.id
    webhook            = pathly_webhook.alerts.id
  }
}

output "all_resources_visible" {
  value = (
    local.scenario_visible &&
    local.sla_visible &&
    local.window_visible &&
    local.webhook_visible &&
    local.sla_measure_ok
  )
}

all_resources_visible is the assertion a scheduled job can fail on. A create that returns 200 and a later list that omits the id is a broken read, and an empty plan will not catch it, because the plan compares state to config, not state to a filtered list. The data source is what compares state to the API. Wire that boolean into the job. Do not only archive the ids.

visible_in_inventory remains in the smoke as the older scenario-only check, so a job that already reads that output does not break. New jobs should read all_resources_visible.

When you copy the smoke into a real workspace, delete smoke_id from the names, set interval_sec to the interval you mean, replace the health URL, and keep the read-back outputs. The destroy at the end of CI is correct for a smoke and wrong for production. Remove the destroy step before the first real apply.

Fail your GitHub Actions deploy if Pathly Ping or Chain is red

Simon Raynaud — Wed, 07 Oct 2026 11:04:15 +0000

Unit tests pass. Prod still breaks on the path users actually hit.

Pathly: Ping if HTTP is up, Flow if a user can walk the journey, Chain if the API hops behind it hold.

Wire a CI gate so the job fails when that path is red:

- uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_API_TOKEN }}
    operation: run-scenario
    scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}
    timeout_sec: "120"

Use run-scenario for a fresh check. Use assert-scenario for the last known status (faster).

Live: https://pathlyhq.com/try-flow