Nightly checkout timeouts: Ping green, Flow red, Chain lagging

This is a reconstituted, composite case. It aggregates patterns we see when an agency runs an e-commerce store for a merchant: HTTP uptime stays green, server dashboards look calm, and yet checkout dies in a narrow overnight window. No customer brand is named. The point is the split between “the site responds” and “a buyer can pay.”

Pathly names three instruments — Ping, Flow, and Chain — and keeps those names in every language. Ping watches HTTP surface. Flow walks a browser journey. Chain follows the API calls behind the journey. Together they explain incidents that classic uptime tools cannot see.

Canonical write-up on the Pathly blog: Nightly checkout timeouts.

The business problem before the stack

Between roughly 02:00 and 02:15, buyers hit the payment step and timed out. Carts were abandoned. Support tickets arrived the next morning with screenshots of spinners and “something went wrong” pages. Revenue for that slice of the night was effectively zero.

For the agency, the status page and the uptime tool said the site was up. Health endpoints returned 200. Homepages loaded. CDN cache hit ratios looked normal. For the merchant, the register was closed exactly when a portion of traffic still converted — late-night mobile shoppers, overseas time zones, people finishing a purchase after a show or a shift.

That mismatch is the whole story. Ops and commerce were both looking at truthful signals. They were not looking at the same object.

If you only measure “is the origin answering HTTP?”, you will close the ticket as noise. If you measure “can a buyer complete cart → shipping → payment?”, you will see a recurring red window that lines up with something else on the calendar: a backup job, a batch ETL, a lock-heavy maintenance window, a third-party settlement batch. The functional problem comes first. The tooling comes second.

Why classic monitoring stayed green

HTTP checks on the storefront and on /health (or equivalent) kept returning 200 within the SLA. CPU, RAM, and disk on the app nodes did not sustain an alert. Database CPU had a bump, but nothing that pager duty treated as a hard outage. Synthetic “click home” checks passed because they never entered the write-heavy checkout path.

Checkout is not a ping. Until you run the browser journey — add to cart, choose shipping, enter payment, wait for confirmation — you can reassure the ops team while sales leak. The leak is time-boxed, so weekly averages still look fine. Dashboards that aggregate over an hour wash out a fifteen-minute failure.

Two rules of thumb we use when reading green boards:

1. HTTP uptime ≠ buyer journey. A 200 on / does not imply a successful payment authorization.
2. Server metrics ≠ business latency on a critical step. A database under backup I/O can slow commits without flipping a binary “down” flag.

Agencies feel this pain twice: once for the merchant’s revenue, once for their own credibility. “Your monitoring said green” is a hard conversation when the merchant’s bank settlement report shows a hole.

How to read this reconstituted case

We reconstruct a typical store-plus-agency incident: timing, symptoms, and how Pathly’s three instruments read it. Treat it as a field pattern, not as a named testimonial. The value is the diagnostic sequence:

1. Confirm the business symptom (timeouts at checkout, abandoned carts, support volume).
2. Check whether Ping agrees with the symptom (usually it does not).
3. Run or schedule a Flow on the exact buyer path during the window.
4. Attach a Chain on the payment-related HTTP sequence to see which hop slowed down.
5. Correlate the red window with ops calendars (backups, reindexes, partner cutovers).
6. Fix the cause, keep Flow on guard, optionally gate deploys with the same scenario in CI.

That sequence is deliberately boring. It is also what stops the argument between “green uptime” and “lost sales.”

What Pathly showed: Ping, Flow, Chain

Ping stayed green

Ping is the HTTP / uptime instrument. In this pattern, the storefront and health endpoints stayed inside thresholds all night. From Ping’s point of view, the path’s edge was fine. That is useful information: you are not dealing with a total outage, a DNS failure, or a TLS expiry. You are dealing with a journey failure under load or lock contention.

Teams sometimes disable Ping after the first Flow alert. Do not. Ping still answers “is the surface reachable?” when Flow goes red for a flaky third-party script or a payment provider blip. The triad is the point.

Flow went red on checkout

Flow is the browser journey instrument. Every night in the same window, Flow failed on the checkout step. The run included a browser capture of the timeout — not a vague log line, but the failed journey at the exact minute. That artifact changes the incident review: product, ops, and the agency account manager can look at the same screenshot instead of debating whether “the site felt slow.”

A Flow scenario for checkout usually looks like a short scripted path: open product, add to cart, go to checkout, fill shipping fields (test data), proceed to payment, assert a success selector or URL. You do not need a full Playwright suite in the client repo to start. Pathly’s product UI is built so non-specialists can record or assemble the path; developers can still automate around the same scenario id from CI.

Chain showed lag behind payment

Chain watches the API sequence behind the journey. While Ping said “up” and Flow said “buyer stuck,” Chain showed latency climbing on the calls behind payment — inventory reservation, payment intent, order create — without a clean uptime outage. That is the signature of lock or I/O pressure: endpoints still answer, but too slowly for the browser timeout budget.

Reading the three together:

Instrument	Signal in this case
Ping	HTTP surface OK
Flow	Buyer journey KO + screenshot
Chain	APIs behind checkout slowed down

If you only had Ping, you would have closed the night as healthy. If you only had Flow, you would know the buyer failed but not which hop slowed. If you only had Chain, you might chase API SLOs without proving the cart UX broke. The composite case exists to show why the triad is not marketing garnish.

Cause: full database backup locking the path

The red window matched a full database backup. Locks and saturated I/O slowed checkout writes without taking health checks down. Ops saw green. The business saw timeouts. Both were right on their own signal, not on the same object.

This is a common shape for Magento, WooCommerce, Shopify-adjacent custom stacks, and headless setups that still write orders to a primary Postgres or MySQL. Backup tools that take a consistent snapshot can hold locks or hammer disks. Logical dumps that scan large order tables compete with checkout inserts. Even “online” backups can push I/O wait high enough that a 10–15 second browser timeout fires.

Other calendars produce the same shape: nightly search reindex, warehouse sync, fraud batch, payment settlement file pull, certificate rotation that restarts a connection pool poorly. The diagnostic move is the same: align Flow’s red buckets with the ops calendar, then prove causality by shifting the job and watching Flow return to green at the same clock time.

Fix: shift the backup, keep Flow on guard

The team moved the backup off the overnight conversion window, then confirmed Flow went green again at the same hour. Keeping Flow scheduled on that journey avoids replaying the same hole next quarter when someone re-enables an aggressive backup flag.

A green Ping is no longer enough to close a business incident. Close when Flow is green through the former window for several consecutive nights, and when Chain latency on the payment hops is back under your budget. Document the calendar change so the next on-call does not “optimize” the backup back into the conversion hour.

For agencies: put the Flow scenario in the client’s runbook with the scenario id, the expected window, and the CI gate (next section). That is how you stop the incident from becoming a he-said-she-said between merchant and ops.

Keep checkout behind a CI gate

The same scenario can block a pipeline after deploy. If a release changes checkout selectors, payment provider config, or API contracts, you want the failure in CI — before overnight traffic redistributes the incident.

Pathly publishes a GitHub Action mirror at pathlyhq/pathly-action@v1. In the monorepo it lives under integrations/github-action. Operations include ping, assert-scenario, run-scenario / run-and-wait, and ensure-scenario.

Assert last status (fast gate)

Use this when a scheduled Flow already ran recently and you only need to fail the job if the last status is not ok:

- name: Pathly assert checkout Flow
  uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_API_TOKEN }}
    operation: assert-scenario
    scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}

assert-scenario does not trigger a new run. It reads the last status and exits non-zero when that status is not ok. That makes it cheap on the critical path of a deploy workflow.

Run scenario after deploy (smoke)

Use this when you want a fresh browser journey against the environment you just shipped:

- name: Pathly smoke checkout
  uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_API_TOKEN }}
    operation: run-scenario
    scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}
    timeout_sec: "120"

A minimal workflow that combines both ideas (assert on every push, optional smoke on main) matches the seed under integrations/ci-template/.github/workflows/pathly.yml:

name: Pathly CI
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  pathly-assert:
    runs-on: ubuntu-latest
    steps:
      - name: Assert last status
        uses: pathlyhq/pathly-action@v1
        with:
          api_token: ${{ secrets.PATHLY_API_TOKEN }}
          operation: assert-scenario
          scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}

  pathly-smoke:
    runs-on: ubuntu-latest
    if: github.event_name == 'workflow_dispatch' || github.ref == 'refs/heads/main'
    steps:
      - name: Run scenario
        uses: pathlyhq/pathly-action@v1
        with:
          api_token: ${{ secrets.PATHLY_API_TOKEN }}
          operation: run-scenario
          scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}
          timeout_sec: "120"

Secrets and scopes: store PATHLY_API_TOKEN as a repository secret (sp_…). For assert you need scenarios:read and runs:read. For run-scenario add runs:trigger. Put the scenario UUID in a Actions variable such as PATHLY_CHECKOUT_SCENARIO_ID so you can rotate scenarios without editing YAML.

Docs: GitHub Actions on Pathly.

GitLab CI component (same gate, different forge)

If the merchant’s store deploys from GitLab, the same idea exists as CI/CD components in the Pathly GitLab catalog (pathly-ping, pathly-scenario, pathly-assert, pathly-run). The monorepo seed under integrations/ci-template/.gitlab-ci.yml shows the shape:

include:
  - component: $CI_SERVER_FQDN/pathly1/provider/gitlab-ci/pathly-assert@~latest
    inputs:
      api_token: $PATHLY_API_TOKEN
      scenario_id: $PATHLY_SCENARIO_ID
  - component: $CI_SERVER_FQDN/pathly1/provider/gitlab-ci/pathly-run@~latest
    inputs:
      api_token: $PATHLY_API_TOKEN
      scenario_id: $PATHLY_SCENARIO_ID

Mask PATHLY_API_TOKEN in GitLab CI/CD variables. Prefer the provider project tags for releases; the monorepo integrations/ folder is a reference copy so examples stay close to the app docs.

Terraform: declare the watch as code

Agencies that already manage infra as code can keep the checkout Flow next to the store definition with the Pathly Terraform provider (Pathlyhq/pathly on the registry, source in pathly1/provider/terraform-provider). The mental model matches Ping / Flow / Chain: resources describe scenarios and related objects, then CI asserts them after apply.

A typical flow:

1. Apply Terraform so the scenario exists with the expected steps and schedule.
2. Deploy the store.
3. Run pathlyhq/pathly-action@v1 with assert-scenario or run-scenario against the scenario id output by Terraform.

That keeps the “buyer can pay” check in the same review as the infrastructure change that might have moved the backup window or the payment worker size. You are not inventing a second source of truth in a wiki page that ages out.

Provider docs and examples ship with the provider repository; Pathly’s developer hub also walks through authentication and scopes used by both the Action and Terraform.

API sketch: assert from any runner

If you are not on GitHub or GitLab, the Action is a thin client over the Pathly HTTP API. The SDK package @pathly/sdk in the monorepo wraps the same contract. Conceptually, an assert gate is:

1. Authenticate with an org API token that has scenarios:read and runs:read.
2. GET the scenario by id.
3. Fail the job if lastStatus !== "ok".

A run gate adds runs:trigger, POSTs a run, then polls until ok, fail, or error within timeout_sec.

Keeping the gate dumb is intentional. The intelligence lives in the scheduled Flow and Chain during the night. CI only asks: “is the last known buyer path still healthy before we ship?” or “run it once against what we just shipped.”

How an agency should operationalize this

1. Name the journey. “Checkout happy path — production” with a stable scenario id shared with the merchant.
2. Schedule Flow through the risky window. If sales happen at 02:00, measure at 02:00. Do not only measure at 10:00 when the backup is idle.
3. Attach Chain to payment hops. When Flow is red, open Chain before blaming the frontend theme.
4. Keep Ping. Surface outages still happen; the triad prevents single-signal blindness.
5. Gate deploys with assert-scenario or run-scenario in GitHub Actions or GitLab CI.
6. Align the ops calendar. Backup, reindex, and partner batches should not sit on conversion hours without an explicit risk acceptance.
7. Close incidents on Flow, not on Ping alone.

This is also how you speak to the merchant without theater: show the Flow capture, the Chain latency bump, the backup window, the green after the shift. No need for invented certifications or “100% EU” slogans — just the instruments and the calendar.

Try the journey, not only uptime

If you run a store for a client, run a Flow on checkout and a quick HTTP surface scan. In a few minutes you will see whether Ping, Flow, and Chain tell the same story.

• Try Flow without an account: pathlyhq.com/try-flow
• Scan a site: pathlyhq.com/scan
• Product page for Flow: pathlyhq.com/flow
• CI gate docs: pathlyhq.com/developers/github-actions

Pathly watches the path. Ping if it is up. Flow if a user can walk it. Chain if the services behind it hold.

Appendix: reproducing the diagnostic without Pathly product UI

Even if you prototype with your own Playwright job, keep the same separation of concerns:

• A cheap HTTP probe (Ping-shaped) every minute on / and /health.
• A browser journey (Flow-shaped) on a schedule that includes the suspect window.
• A timed HTTP sequence (Chain-shaped) for payment intent → capture → order create.

When the browser journey fails and the HTTP probe does not, you have already disproved “the site is down.” When the HTTP sequence shows one hop spiking, you have a target for DBA or the payment provider. When both journey and sequence fail only inside a backup window, you have a calendar fix, not a rewrite of checkout.

Pathly’s product packages that triad with captures, alerting, and the CI Action so agencies do not maintain three bespoke harnesses per client. The composite case above is what those harnesses are for: a functional failure that never lights a classic uptime pager.

Appendix: what we explicitly did not do

• We did not declare the site “down” because Ping was green.
• We did not ask the merchant to accept lost overnight sales as monitoring noise.
• We did not replace a backup with “no backups.”
• We did not claim a compliance badge we do not hold.
• We did not spam directories with the case. The write-up lives on the blog and here as a technical narrative for people who ship stores and CI.

If your overnight graph looks calm while finance says checkout died, start with the buyer journey at the clock time of the complaints. Then read Ping, Flow, and Chain as three answers to three different questions — not as three ways to say “uptime.”

Appendix: timeline of a night that looked calm

Here is a concrete timeline you can reuse in an incident doc. Times are local to the store.

• 01:45 — Backup job starts. Disk util rises. Ping still green.
• 02:02 — First Flow run in the window fails on payment confirmation. Screenshot stored.
• 02:05 — Chain shows p95 on order-create climbing past the browser timeout budget.
• 02:08–02:14 — Buyer support chat and abandoned-cart webhooks spike. Homepage Ping remains ok.
• 02:20 — Backup ends. Chain latency falls. Next Flow run passes.
• 09:10 — Merchant asks why overnight revenue dipped. Classic uptime report shows 100%.

Without Flow and Chain aligned to that clock, the morning review becomes opinion. With them, the review becomes a calendar change and a CI gate. That is the entire value of separating Ping from Flow from Chain: three questions, three answers, one shared night.

When you write the postmortem, paste the Flow run ids and the Chain hop chart for 01:45–02:20 next to the backup job name. Resist the urge to rewrite checkout CSS first. Most of these nights are infrastructure calendars colliding with conversion calendars, not theme bugs. Fix the collision, keep the triad watching, and let CI refuse deploys that break the same journey again.

Agencies can also keep a one-page client brief: scenario id, schedule, CI workflow link, and who owns the backup calendar. When the merchant hires a new ops person, that brief prevents the backup from sliding back onto the conversion hour without anyone noticing. The brief is dull on purpose. Dull runbooks survive handoffs better than heroic Slack threads.

That handoff cost is why the CI gate and the scheduled Flow both matter in production, not only in a demo environment.